Big Data and Security - the next big disruptor?

Last quarter I was invited to a Cloudera sales event in Las Vegas. Some impressive stats on last year's performance, a lot of enthusiasm and in particular a great session from Charles Zedlewski @zedlewski outlining some of the product and Apache initiatives coming soon.

Two in particular are now announced 

• the Open Network Insight (ONI) https://vision.cloudera.com/introducing-open-network-insight-accelerating-cybersecurity-analytics-solutions/ ONI is effectively Cloudera's move into core security functions such as threat detection.
• Apache Arrow https://arrow.apache.org/ Arrow leverages latest SIMD (Single input multiple data) operations optimization of analytical data processing. Arrow is Cloudera's attempt to gain control of in-memory columnar data processing.

So far, so good, but these two announcement will make a huge impact in the IT Security market. for sometime now there has been little innovation in Security. the main players are all offering incremental enhancements to technology that has been around for years.

Big Data and the Hadoop eco-system can (and already has) disrupt the ITSec market. Principally it's a cost/scale dynamic. SIEM's, Vulnerability Management, Configuration Management tools and others are essentially about reacting to events that have already happened. they also use Metadata structured repositories to normalize, correlate and report. Look at any SIEM vendors details and you will see this common theme. Detect and fix something that has already happened. 

With Hadoop and it's various components and, in particular, the continuing path to maturity in machine learning products, this old style architecture is going to disappear. Sometime between now and 2020 the Enterprise Security Warehouse concept will be widely adopted. All data from all sources poured into a massive data lake (in real-time of course), with an HDFS/Kudu style repository for persistence and machine learning algorithms constantly monitoring what is happening and taking appropriate action as the threats happen  not after they happen. Gartner predicted this back in 2014 so it must be true..... http://www.gartner.com/newsroom/id/2778417 

In our discussions with clients we see a gradual realization, usually in the biggest clients first, that the old style Security Architectures have failed to keep up and new architectures built on big Data eco-systems and machine learning in particular, offer the greatest potential for the next disruptor. Look at how Splunk has built a $600m business on just this premise but without the machine learning part.

For an alternative view of ML and Security read Matt Harrigan's post@mattharrigan at Tech Crunch. http://techcrunch.com/2016/02/29/machine-learning-is-not-the-answer-to-better-network-security/ 

What do you think, is Machine learning already the big disruptor in Cyber Security?

Machine Learning and Spark - get ready for the next big disruptor

There are lots of articles, blogs, reports and noise at the moment about Spark and machine learning - driven primarily by the rapid adoption of MLlib (Spark's general machine learning library) that is leading developers to use R and Python in particular for Advanced Analytics. For a great overview go to Infoworld - Why you should use Spark for Machine learning.

It's generally recognized that Spark has a long way to go before it is fully Enterprise ready. Almost every client I talk to follows a very familiar pattern - they want to try it for speed and scale, they try it and get disappointed in particular by it's scaleability and then decide to wait.

However, when Machine Learning comes into the discussion, Spark adoption is rapid, visible and highly successful. Customers are now recognizing the growing power of Spark/MLLib, particularly with the growing number of algorithms Spark MLLib supports.  ML has been around since 1979 and more recently the 'not very good' Mahout implementation has led to a lot of disappointed projects. 

We don't have space here to go into the details of ML but I notice four key trends that will help customers see strong and rapid time to value in their machine learning projects :-

1. Customer 360 views are one of the most common Big Data use cases. Using ML and Spark MLLib in particular, customers can leverage massive data volumes to make product recommendations to customers in real time using ads or other recommendation platforms. ML can take Recommendation and Monetization engines to whole new level of predictability and relevance in real-time
2. Similarly in Mobile Networks, ML can be used to predict and manage Network Optimization - a critical cost element in Mobile Network profitability. Think about it like a river. Use ML to maximize the flow of water through the narrowest channels while maintaining speed and volume. Maximum benefit flows from predicting in near real time how the flows (Wireless traffic) should be managed. 
3. With Geolocation services, massive data volumes and ML, Retailers can tailor specific offers to individuals. Imagine a scenario where you go into a Nordstrom's type store, the Store ML system picks up (from the Store's already installed  Mobile App) that you have entered the store. As you wander round the various departments the ML system is rapidly choosing products you will be interested in (and presenting them on your mobile device) and, when you press the 'Get Help' button on your phone, the Sales assistant glides over, already armed with all your previous purchase history and set of suggestions on what to buy. They open the conversation with 'Good Morning Mr. Bennett, let's take a look at that Emile Staub Cocotte that you looked at last time you were here'.....
4. Data Wrangling is still a big issue, Machine learning based companies like Trifacta are starting to get a lot of traction inside the Enterprise. Once large companies understand how ML apps can change their entire Big Data ecosystem, ML will become a mainstream technology during 2016.

Want to know more about Machine learning - take a look at this Infoworld slideshare
What do you think? Is Machine Learning the next big disruptor?

The Open Source Vendor game

The latest in a long line of struggling Data Warehouse companies announced their results  on 5th November. Teradata had flat revenues and flat to slightly declining revenues predicted for the year. Look at the Dell/EMC idea, HP''s continuing machinations and flat revenues, IBM's problems (including the strange decision to buy .... The Weather Channel , Oracle's well documented problems  and it's clear the seismic shift in Enterprise software that started with AWS and continues with Hadoop by way of open source continues to grind down the Proprietary Enterprise software vendors.

As Mike Olson put it over 2 years ago - the Proprietary vendors now have no hope of competing. They don't have the resources, the knowledge or the skills to compete with globalized software development teams running Apache projects or similar.

So when a client asks us - what does this all mean? Why is everything moving so fast? Why can't we get back on the old familiar proprietary software merry go round? All this uncertainty makes customers very nervous about what to do, and in some cases they therefore..... do nothing.

When a client asks this question we tell them three things :-

1. Look at what is happening in the critical Apache projects. Take Hadoop, or Spark, or Kudu or many others. Look at the trend in committers to see which projects have a solid bench of committers and is growing.
2. Secondly - look at where these committers are doing their day jobs. Here is a good example of Spark. Scroll down the list and you will see it's solidly dominated by Databricks, with a good mix of UC Berkeley (where Spark was developed), Intel (watching their investment in Cloudera) and, of course Cloudera.
3. Thirdly - if you are intent on looking at projects still in incubation - be careful. Hadoop Development tools  seemed like a good idea but failed to get any sort of traction with the community.

The last piece of advice we give clients is always look at what the market leaders are donating to Apache. Last week Cloudera announced they want to donate Impala and Kudu to Apache. Impala has been widely adopted and so Cloudera want the community to take over some of the development work. Kudu is a big complex project so, now that it's architecture has been broadly set, Cloudera want the bigger, more agile, more skilled worldwide community to shoulder some of the development load.  

For many clients, used to easy decision making provided by Gartner and others, Open Source adoption in the enterprise and in the data center in particular is changing everything. 5 years ago and certainly 10 years ago clients would insist on Roadmaps under NDA, long term support with access to source code if the vendor went out of business, long complex discussions about pricing and licensing and entry into elite customer user groups. Not one of those types of discussion happens any more in Open Source and Big Data. No roadmaps (it's up to the community to direct the roadmap so join in!), source code is already freely available, licensing is easy - it's free!, 

We believe clients need to adapt not just their technology options but also their vendor outlook to get the most out of the Open source world.

What do you think - are customers able to adapt to the changing market and technology landscape? Or they stuck in a time warp - working in a style and with a market view that no longer exists?

And take a look at the Platinum sponsors of ASF - they are sponsoring for a reason of course. 

Kudu - the end of MapReduce?

Last week at Strata Cloudera announced Kudu, http://blog.cloudera.com/blog/2015/09/kudu-new-apache-hadoop-storage-for-fast-analytics-on-fast-data/ 

Kudu is going to fill in the gaps in Hadoop's storage layer and be almost as good as HDFS at what HDFS is good at (high speed writes and scans) and, at the same time, almost as good as HBase at what HBase does best (random access queries).  Although it's a long way from being enterprise ready it's clear that Kudu can avoid architectures that we have implemented in several customers :-

• Persistence of data in HDFS and
• A subset of data in HBase for real time access / analytics / GIS services etc.

Not only is this architecture expensive in Hardware and Professional services but it introduces high levels of complexity that  a lot of customers are uneasy about. So - when Kudu is ready - it will spell the end of HDFS and HBase for many customers.

So now it becomes clear that the architectural goal is to replace HDFS and HBase with Kudu and, as we have all ready heard about before now - replace MapReduce with Spark.

So then you have the target architecture 2-3  years out - Kudu and Spark replacing HDFS/MapReduce/HBase. As I and others have written in previous posts this will then allow for full real-time streaming analytics and services on massively scaleable clusters. It is this architecture that will lead to an explosion in IoT use cases.

Now - one last point - in all this change there are some Dinosaurs out there - yes EMC I am looking at you, and you HP and IBM and Teradata and even NetApp. These businesses are in flat or declining markets (take a look at this Forbes article for more detail http://www.forbes.com/sites/greatspeculations/2015/01/02/how-emc-lines-up-against-netapp-hp-ibm-hitachi-in-storage-systems-market/ ). As Kudu gains traction these older vendors with old style technology will become less and less relevant. EMC has been reinventing itself for a while but it will be interesting to see how the decline or even disappearance of Enterprise storage will reshape the landscape.

Maria Deutscher also makes some great points about Spark and the Hadoop Ecosystem over at SiliconANGLE  http://siliconangle.com/blog/2015/09/28/apache-kudu-how-cloudera-wants-to-save-hadoop-by-killing-it/ 

Spark / Cloudera One Platform initiative - important? Maybe...probably

Last week Cloudera announced the One Platform initiative  http://vision.cloudera.com/one-platform/ . This is an interesting development -  we know that Spark replaced Hadoop as the most popular Apache project and that Cloudera first backed Spark publicly over a year ago.  (Hadoop still has more committers though (https://projects.apache.org/projects.html?number) .

In making the announcement Cloudera also confirmed the gaps that exist in Spark to-day, somewhat similar to lists I have shared here :-

• Security
• Scale
• Management
• Streaming

We know that Cloudera intends to replace Map-Reduce with Spark. The only way this can happen is for Spark to  scale. Our experience with Spark is that it does work and has a lot of advantages in use cases like Machine Learning, way faster than Hive, alerting and aggregating in streaming analysis, joining multiple data sources in memory. But the one thing Spark does not do well is scale. Compared to the scale advantages that Mapreduce provides Spark is a long way behind.

But most interesting is the fact that Enterprises want to have a single data lake/warehouse/repository/whatever you want to call it. Until Spark can scale then all those pesky structured data systems that have been built by customers using Oracle and all their competitors will continue to be used and deployed. Therefore - if Cloudera succeed - and I think they will - the hollowing out of the traditional database vendors is going to really take off.

What do you think? Is the One Platform initiative just marketing hype? Will Cloudera succeed in replacing Oracle and their like? Let us know and check out our new website at www.exceleratesystems.com. 

Big Data Applications - anyone out there?

If you read my post last week we discussed how Big Data and Hadoop is no longer running on commodity hardware. This is a good thing, provided you are prepared to accept the higher Hardware cost, because it lays the groundwork for the next big move in Big Data - off the shelf, high value applications.

If you are at all involved in Big Data you know several things :-

1. It's moving very fast, new Apache Projects / new Cloudera Labs projects and significant upgrades in real-time/streaming/analytics capabilities.
2. Right now, Big Data is a bunch of tools - infrastructure tools, ingestion tools, analytic tools, go to any Big Data conference and what you see is a lot of customer specific use cases and a lot of talk about the latest 'tools'.
3. Make any kind of search for 'Big Data Applications' or Hadoop Applications' or even 'Cloudera Applications' and you come up empty. What you will find is a lot of SaaS Analytics offerings that leverage a Big Data infrastructure. To me, these are not new Big Data Apps, but simply BI/Analytics apps using additional data sources. Here's an example  http://www.cio.com/article/2917433/startups/can-a-startup-democratize-big-data-apps.html Looks interesting but it's not a Big Data app.

I believe there is a second Big Data wave coming soon that will be just as important as the ERP and CRM and Mobile / online waves were. That is - Big Data Apps that move beyond just analytics and start offering services to customers based on real time, streaming data ingestion matched up with geo-location and customer preference information. This is not new and it's certainly not an original idea. What's new is that Big Data and the Hadoop ecosystem in particular can now deliver what businesses and governments have aimed for - customer/client/citizen segmentation that delivers individual services/offers/capabilities to to specific individual people.

Very simple example - multi channel business (bricks and mortar, online, etc) targets a micro segment of couples under 30, no kids living in the top 50 income neighborhoods in the US. Every time one of those customers or potential customers passes by a multi -channel outlet (either physically or online) they receive a co-ordinated marketing campaign just for them (promotional offer, 2 for 1 offer, groupon offer or whatever the analytics part of the Big Data process has defined as the right offer. There are many other Apps that we will discuss in later posts.

When I review the use cases that we have developed for customers, in the beginning it was simple batch oriented analytics for customer analysis, maybe telco infrastructure analysis. Then maybe 2 years ago the discussion moved to real time analytics. What I see now is a definite move to discussing not jut analytics but new services to customers (B2C, B2B, G2C etc) that can only be developed using Big Data paradigms.

Critical in this (and in current Big Data projects) is security and we will dsicuss that next week.

What do you think? Are we still in the era of Big Data tools? Or are we entering the next phase of Big Data Apps?

The Internet of Things and Big Data

We are seeing increasing interest from a number of clients in IoT. While it's still more of a hype phrase than reality there are a number of Organizations that can point to real success with IoT and Big Data. One of them is Transport for London who use heir thousands of sensors in their transport network to build actionable insights. Lauren Sager Weinstein of TfL explains more here https://tfl.gov.uk/cdn/static/cms/documents/big-data-customer-experience.pdf 

To me it seems that Public sector and particularly State and Local Government are the leading adopters of IoT and Big Data - interesting idea - local government as technology leaders. In particular we see this trend in Europe rather than the USA. In multiple regional and EU projects where government is trying to leverage their massive array of sensors to improve delivery of services to citizens.

The area where there is a lot of discussion but not a lot of real use cases is in the Enterprise. This week I was at a very large retailer who is a client of ours. Clearly Retailing is the next obvious sector for IoT. Lots of locations, lots of customers, lots of channels, lots of assets producing data. David Dorf at Oracle put out a good list of possible use cases in January - you can read it here  http://www.forbes.com/sites/oracle/2015/01/09/how-the-internet-of-things-will-shake-up-retail-in-2015/ 

So - what next? It's clear IoT is going to change Big Data Architectures again, just as we get used to Spark possibly replacing MapReduce, the massive increase in data volumes and types from IoT, is, for sure, going to lead to another architecture pivot. 

What are your thoughts? Do you see IoT use cases in the Enterprise happening to-day or is it just talk. What about Public sector - are they the leaders in IoT?

Spark and Hadoop - replace or complement?

I recently read a survey report from Typesafe of 2,136 respondents who were asked about Spark vs Hadoop. You can see the report here for yourself (registration required) https://info.typesafe.com/COLL-20XX-Spark-Survey-Report_LP.html?lst=RW&lsd=COLL-20XX-Spark-Survey-Trends-Adoption-Report

The most interesting part of the report for me was that 78% of respondents were using Spark for fast processing of BATCH data sets! Think about that. Spark can work with HDFS as the persistent data store but Spark is really good at processing streaming, transactional data - but - most respondents are just using it to make batch go faster.

This is our experience too - when we talk to customers they want to consider Spark, they know they have to think about future use cases which will almost certainly involve streaming data, transactional data sets and - most importantly - real time analytics and machine learning. But - for now, even ten years after Doug Cutting and Mike Cafarella invented Hadoop - we are still seeing the vast majority of use case focused on batch processing. It really is - back to the 80's!

So - in my view - Spark is not replacing Hadoop but is simply complementing what is already out there. What do you think?

This Aptuz blog also summarizes neatly the Spark vs Hadoop discussion http://aptuz.com/blog/is-apache-spark-going-to-replace-hadoop/