Big Data Applications - anyone out there?

If you read my post last week we discussed how Big Data and Hadoop is no longer running on commodity hardware. This is a good thing, provided you are prepared to accept the higher Hardware cost, because it lays the groundwork for the next big move in Big Data - off the shelf, high value applications.

If you are at all involved in Big Data you know several things :-

1. It's moving very fast, new Apache Projects / new Cloudera Labs projects and significant upgrades in real-time/streaming/analytics capabilities.
2. Right now, Big Data is a bunch of tools - infrastructure tools, ingestion tools, analytic tools, go to any Big Data conference and what you see is a lot of customer specific use cases and a lot of talk about the latest 'tools'.
3. Make any kind of search for 'Big Data Applications' or Hadoop Applications' or even 'Cloudera Applications' and you come up empty. What you will find is a lot of SaaS Analytics offerings that leverage a Big Data infrastructure. To me, these are not new Big Data Apps, but simply BI/Analytics apps using additional data sources. Here's an example  http://www.cio.com/article/2917433/startups/can-a-startup-democratize-big-data-apps.html Looks interesting but it's not a Big Data app.

I believe there is a second Big Data wave coming soon that will be just as important as the ERP and CRM and Mobile / online waves were. That is - Big Data Apps that move beyond just analytics and start offering services to customers based on real time, streaming data ingestion matched up with geo-location and customer preference information. This is not new and it's certainly not an original idea. What's new is that Big Data and the Hadoop ecosystem in particular can now deliver what businesses and governments have aimed for - customer/client/citizen segmentation that delivers individual services/offers/capabilities to to specific individual people.

Very simple example - multi channel business (bricks and mortar, online, etc) targets a micro segment of couples under 30, no kids living in the top 50 income neighborhoods in the US. Every time one of those customers or potential customers passes by a multi -channel outlet (either physically or online) they receive a co-ordinated marketing campaign just for them (promotional offer, 2 for 1 offer, groupon offer or whatever the analytics part of the Big Data process has defined as the right offer. There are many other Apps that we will discuss in later posts.

When I review the use cases that we have developed for customers, in the beginning it was simple batch oriented analytics for customer analysis, maybe telco infrastructure analysis. Then maybe 2 years ago the discussion moved to real time analytics. What I see now is a definite move to discussing not jut analytics but new services to customers (B2C, B2B, G2C etc) that can only be developed using Big Data paradigms.

Critical in this (and in current Big Data projects) is security and we will dsicuss that next week.

What do you think? Are we still in the era of Big Data tools? Or are we entering the next phase of Big Data Apps?