Is Docker Ready for Prime Time?

Docker - ready for prime time or not? It's a question that has been asked (and answered) hundreds if not thousands of times already. So - rather than repeat that long and somewhat tired conversation I want to focus on one piece of the debate - Security.

There's three issues that I see with Docker and Security

1. Because Docker is opensource it has been widely adopted and is almost certainly already deployed, whether you like it or not, inside your organization. This makes for all sorts of security nightmares that the CISO's and their teams are unable to control. What if an employee introduces uncontrolled code to a mission critical stack? What does that do for Compliance, internal audit and Corporate governance issues not to mention liability problems.
2. Many others have covered the point about large attack surface. Thousands of containers vs hundreds of apps, VMs etc. The larger the attack surface the more vulnerable your organization is to internal and external breaches.
3. The very flexibility that Docker and containerization in general provide gives it a massive security hole. What if a rogue employee or external intruder plants a container that launches and East-West attack? Good luck finding that single container in the thousands you have already deployed.

There is plenty of advice out there on how to implement Docker security effectively - this article from Amir Jerbi co-founder and CTO of Aqua Security, is a good basis.

In my discussions with customers about Docker it's clear that, at the Enterprise level, they are just not comfortable yet in adopting Docker. Typical responses include 'Maybe next year,' 'let's wait and see', 'who else is using Docker across their infrastructure?' All good points with limited answers. Look at the list of Docker customers at docker.com. Are these all in production? Let's hope so.

When customers ask me about Docker security I always tell them 'Be careful, move forward in a considered way and you might just end up where you expect to be. If you let it get out of control you will spend a lot of time and money getting Docker under control.' Full disclosure - we offer Docker/Containerization as a service from Alauda We do this because Containerization as a service is intrinsically more secure running on AWS or Azure than letting Docker loose in your Datacenter.

What do you think? Is Docker ready for Prime time?

Kudu - the end of MapReduce?

Last week at Strata Cloudera announced Kudu, http://blog.cloudera.com/blog/2015/09/kudu-new-apache-hadoop-storage-for-fast-analytics-on-fast-data/ 

Kudu is going to fill in the gaps in Hadoop's storage layer and be almost as good as HDFS at what HDFS is good at (high speed writes and scans) and, at the same time, almost as good as HBase at what HBase does best (random access queries).  Although it's a long way from being enterprise ready it's clear that Kudu can avoid architectures that we have implemented in several customers :-

• Persistence of data in HDFS and
• A subset of data in HBase for real time access / analytics / GIS services etc.

Not only is this architecture expensive in Hardware and Professional services but it introduces high levels of complexity that  a lot of customers are uneasy about. So - when Kudu is ready - it will spell the end of HDFS and HBase for many customers.

So now it becomes clear that the architectural goal is to replace HDFS and HBase with Kudu and, as we have all ready heard about before now - replace MapReduce with Spark.

So then you have the target architecture 2-3  years out - Kudu and Spark replacing HDFS/MapReduce/HBase. As I and others have written in previous posts this will then allow for full real-time streaming analytics and services on massively scaleable clusters. It is this architecture that will lead to an explosion in IoT use cases.

Now - one last point - in all this change there are some Dinosaurs out there - yes EMC I am looking at you, and you HP and IBM and Teradata and even NetApp. These businesses are in flat or declining markets (take a look at this Forbes article for more detail http://www.forbes.com/sites/greatspeculations/2015/01/02/how-emc-lines-up-against-netapp-hp-ibm-hitachi-in-storage-systems-market/ ). As Kudu gains traction these older vendors with old style technology will become less and less relevant. EMC has been reinventing itself for a while but it will be interesting to see how the decline or even disappearance of Enterprise storage will reshape the landscape.

Maria Deutscher also makes some great points about Spark and the Hadoop Ecosystem over at SiliconANGLE  http://siliconangle.com/blog/2015/09/28/apache-kudu-how-cloudera-wants-to-save-hadoop-by-killing-it/ 

Don't believe the (Big Data) hype?

Two Gartner reports recently about Big Data

1. Your Big Data projects will fail and here's why http://ht.ly/Sj41q (ZDNet Summary)
2. Big Data ROI remains elusive http://ht.ly/Sj41q (ZDNet summary).

Both reports and summaries try to inject a dose of reality into Big Data adoption. I don't plan to summarise them here, (ZDNet already did that...) but I notice two important points :-

1. Between one-third and almost 50% of respondents don't know if ROI will be positive or negative.
2. According to Gartner / ZDNet - two thirds of Big Data projects will fail.

ROI is obviously an elusive measurement, sometimes it's used as a predictive tool to justify investments and then measurement doesn't take place. We have seen plenty of Big Data projects that goo through an elaborate internal dance of justification using ROI calculations that are then forgotten or ignored once the project is up and running.

For sure many Big Data projects will fail (lots of our successful projects have been built on the back of failed projects from other suppliers), but most of the customers we talk to are becoming knowledgeable enough to avoid basic pitfalls like business linkage, skills building etc. The challenge for them is understanding the frenetic pace of change in the technology, I am sure this is the number 1 factor (among many others) that causes projects to fail.

If you feel out of control in setting your Big Data priorities there is one activity above all others that avoids the pitfalls Gartner outlines. In many ways it's a paradigm shift as Open Source has moved into mainstream Enterprise adoption. A pre-requisite for success is - reading and lots of it. Very basic, but in such a fast moving environment with relatively high levels of complexity and still very tools/platform based - to avoid failure you must (must) spend maybe 1 hour a day reading blogs, product announcements, Apache project details, white papers. It's that simple. Read lots and don't fail..... don't read and fail (lots)

What do you think - do technologists and non-technologists have to up their reading time? What are the other critical factors for success in Big Data.?

Going to Strata NY next week? Please let us know - we would love to met you there.  http://exceleratesystems.com/contact/ 

Spark / Cloudera One Platform initiative - important? Maybe...probably

Last week Cloudera announced the One Platform initiative  http://vision.cloudera.com/one-platform/ . This is an interesting development -  we know that Spark replaced Hadoop as the most popular Apache project and that Cloudera first backed Spark publicly over a year ago.  (Hadoop still has more committers though (https://projects.apache.org/projects.html?number) .

In making the announcement Cloudera also confirmed the gaps that exist in Spark to-day, somewhat similar to lists I have shared here :-

• Security
• Scale
• Management
• Streaming

We know that Cloudera intends to replace Map-Reduce with Spark. The only way this can happen is for Spark to  scale. Our experience with Spark is that it does work and has a lot of advantages in use cases like Machine Learning, way faster than Hive, alerting and aggregating in streaming analysis, joining multiple data sources in memory. But the one thing Spark does not do well is scale. Compared to the scale advantages that Mapreduce provides Spark is a long way behind.

But most interesting is the fact that Enterprises want to have a single data lake/warehouse/repository/whatever you want to call it. Until Spark can scale then all those pesky structured data systems that have been built by customers using Oracle and all their competitors will continue to be used and deployed. Therefore - if Cloudera succeed - and I think they will - the hollowing out of the traditional database vendors is going to really take off.

What do you think? Is the One Platform initiative just marketing hype? Will Cloudera succeed in replacing Oracle and their like? Let us know and check out our new website at www.exceleratesystems.com. 

Big Data and Security - still a problem?

When we talk to clients about Big Data it's often assumed that there is a strong security infrastructure in place to secure all that data either at rest or in motion. While that assumption is starting to be true it's important to understand where the Hadoop Eco-system is strong in security and where there is additional work to do.

1. Go back 2 years and the the answer to the 'Security?' question was very simple - Kerberos. While that sounds a little silly now, it really was that basic. Any requirements for Role Based Access control, encryption, audit trails / governance / compliance, intrusion detection and all the other requirements of Enterprise security were handled in a single somewhat evasive response 'Those aspects are covered by the operating systems and / or the networking systems.' What this actually meant was that Big Data implementations had gaping holes in security.
2. We worked with one client whose Internal Audit team worked out this lack of security and kept a large Big Data project on hold for a full 12 months before they would allow the Hadoop cluster to be joined in any way to the Enterprise Infrastructure.
3. If we look at where Security in Hadoop stands today then customers can implement 
1. Authentication using LDAP or AD. (this is the Kerberos stuff)
2. Encryption at rest or in motion inside the cluster.
3. Role Based Access control (for example using Apache Sentry)
4. Data redaction (thus avoiding the problem of admins having access to all data). This is critical when using Hadoop clusters to PCI use cases so that PII (Personally Identifiable information) can be redacted.
5. Data Governance that provides Auditing, data lineage, data life-cycle management.
6. Key management to manage encryption keys, certificates etc.

These are all significant advances that have been implemented in various Apache projects such as Sentry. However, it's also clear in our discussions with clients that most CIO's and CISO's still don't feel 100% comfortable with Big Data Security. This is particularly true in Europe as a recent survey by Forrester showed. http://eandt.theiet.org/news/2015/apr/big-data-privacy.cfm 

So - when we talk to clients about their Big Data strategy and how they should design and architect what, for many of them, is totally new, we now ask a set of simple questions :-

1. Is the CISO part of the Big Data strategy team, if not why not?
2. As part of the Discovery process, we need to make sure the client has the rights to use all the data they plan to use.
3. Will the client implement data redaction?
4. Is the client willing to encrypt everything?
5. Who owns the cluster security profile?.

This is not meant to be a complete list but it simply makes the client consider Security as an issue at the design phase not as an afterthought.

What do you think? Is Security in Big Data a big problem? Do you think the progress in the last 2 years has allowed Hadoop implementations to catch up with more traditional designs?