Big Data has a Big Problem

One of the keys to success in Big Data is having the skills to make Big Data projects successful. While some of these skills are 'soft skills', the core requirement is to know what we are doing with complex, highly interrelated and fast moving technology. I recently read a Cap Gemini report on How Successful Companies Make Big Data Operational 

The report itself made for interesting reading but also highlighted one big problem - 

Global organizational spending on Big Data exceeded $31 billion in 2013, and is predicted to reach $114 billion in 2018.

Almost 300% growth in 5 years. Now - think about the critical issues this creates - not enough Data Scientists, Admins, Developers, NoSQL experts (plus many other critical skill sets) are being trained right now to come anywhere close to satisfying this demand.  As long ago as 2011 McKinsey estimated that there will a 1.5 million person hole in the US workforce alone of managers and analysts capable of using Big Data to make effective decisions.

So - as others have pointed out, if you are not Facebook/Google/Linkedin etc what can you do. One interesting set of results lies in some research conducted at Microsoft earlier this year. The full report is here and a great summary is at the Register  The study effectively shows that Big Data - the technology - in Microsoft is not failing but - equally - the results are not yet packaged in a way that helps the consumers of the information to use it.

This is the critical point - when we talk to clients we discuss the details of the technology we believe will meet their needs but there is absolutely no substitute for experience. And experience in the customer is critical. For this reason - whenever a client asks for help on Big Data we always tell them - get started, get started right now and start building your own institutional knowledge. That way you will not get trapped in a skills shortage dead-end.

What do you think? Is the skills shortage going to stifle the adoption of Big Data? For an interesting slant on this question take a look at this Venturebeat article from Cameron Sim at Crewspark.