Big Data and Security - the next big disruptor?

Last quarter I was invited to a Cloudera sales event in Las Vegas. Some impressive stats on last year's performance, a lot of enthusiasm and in particular a great session from Charles Zedlewski @zedlewski outlining some of the product and Apache initiatives coming soon.

Two in particular are now announced 

• the Open Network Insight (ONI) https://vision.cloudera.com/introducing-open-network-insight-accelerating-cybersecurity-analytics-solutions/ ONI is effectively Cloudera's move into core security functions such as threat detection.
• Apache Arrow https://arrow.apache.org/ Arrow leverages latest SIMD (Single input multiple data) operations optimization of analytical data processing. Arrow is Cloudera's attempt to gain control of in-memory columnar data processing.

So far, so good, but these two announcement will make a huge impact in the IT Security market. for sometime now there has been little innovation in Security. the main players are all offering incremental enhancements to technology that has been around for years.

Big Data and the Hadoop eco-system can (and already has) disrupt the ITSec market. Principally it's a cost/scale dynamic. SIEM's, Vulnerability Management, Configuration Management tools and others are essentially about reacting to events that have already happened. they also use Metadata structured repositories to normalize, correlate and report. Look at any SIEM vendors details and you will see this common theme. Detect and fix something that has already happened. 

With Hadoop and it's various components and, in particular, the continuing path to maturity in machine learning products, this old style architecture is going to disappear. Sometime between now and 2020 the Enterprise Security Warehouse concept will be widely adopted. All data from all sources poured into a massive data lake (in real-time of course), with an HDFS/Kudu style repository for persistence and machine learning algorithms constantly monitoring what is happening and taking appropriate action as the threats happen  not after they happen. Gartner predicted this back in 2014 so it must be true..... http://www.gartner.com/newsroom/id/2778417 

In our discussions with clients we see a gradual realization, usually in the biggest clients first, that the old style Security Architectures have failed to keep up and new architectures built on big Data eco-systems and machine learning in particular, offer the greatest potential for the next disruptor. Look at how Splunk has built a $600m business on just this premise but without the machine learning part.

For an alternative view of ML and Security read Matt Harrigan's post@mattharrigan at Tech Crunch. http://techcrunch.com/2016/02/29/machine-learning-is-not-the-answer-to-better-network-security/ 

What do you think, is Machine learning already the big disruptor in Cyber Security?

Would you like a Monopoly with your Data or just a cartel?

Two weeks ago I was fortunate to attend the CES show in Las Vegas. I experienced the joys of 3x Uber surge pricing on Thursday afternoon as well as some great discussions. Two things struck me from CES :-

1. Try as they might, no-one can make the Internet of things very interesting. I visited a home automation demo in the Qualcomm booth. the demo was good and you can see some of the content here in this promo from Qualcomm. But the problem is - the technology is not new, not very innovative and has been around quite a while. More interesting was the La Poste booth where there was a platform for IoT traffic (Hub Numerique) that is being used by 20 innovative startups (from smart shoes, to intelligent drinking glasses). Hub Numerique (French). Using the platform idea, La Poste takes away the complexity and frees start ups to innovate.
2. Even though it is not very interesting, IoT is really going to take off when bandwidth makes the next step. 4G mobile networks are not powerful enough to handle the vast amounts of data that IoT will generate. 5G - which is more of an idea than an emerging standard  - will provide mobile networks that are 100 times fast than 4G. (Download an HD movie in 1 second). So my thought is that IoT is coming but maybe not quite as fast as the vendors would like it to. Tech Republic have a great post on 5G and it's background including the usual issue about spectrum for new wireless networks.

However, it will happen, and, when it does, there will be an enormous explosion in data volumes. Already we create 5 Exabytes every two days, by 2019 it will be 5 Exabytes every day (including 1 Exabyte in mobile platforms alone). by way of comparison - From the beginning of civilization to 2003 the human race created..... 5 Exabytes. Now we create the same volume of data in 48 hours. And this growth is happening without massive IoT adoption. Cisco's annual forecast is informative for those of us whose like lots of numbers. This year's will be out in February and it will be interesting to see what the growth will be to 2020.

So all this data and who will control it? This is a key question for markets, customers, solution providers and the whole Big Data ecosystem. In the European Union for example, the online search market is completely dominated by Google with a 90% share. Which has given birth to a new term 'Data dominance'. just like the old monopolies and cartels of Oil, Railways, Steel making and so on shaped the 19th and 20th Century economies so - the theory goes - Data dominance is the key metric for commercial success in the 21st Century. Then it becomes clear that even if you have a free market with limited regulation and access to as much information as you want - whoever controls that information and data controls everything.  So Big Data becomes a platform for centralization and consolidation of market power.

What do you think -  is Data Dominance important? Do you think we should be concerned? For a (Vodafone sponsored) survey on this question and some interesting insights into European perspectives on Big data take a look at this recently published report or the summary at Forbes.com