Every Angle Blog
Big Data: Big Disappointment or Massive Success?
My view on Big Data challenges
I've worked in IT since 1978 and during this time I've seen numerous hypes and fuzzes introduced to the world of IT - but not all have managed to live up to their initial hype. With that said, the buzz surrounding big data is definitely one of the biggest. It goes without saying that with proper application, big data can bring with it numerous competitive advantages. However, I tend to furrow the eyebrows when I read it's being positioned as a new silver bullet. So what causes my eyebrows to fluctuate so vigorously? Why do I think it will take time to become a success? Allow me to share my thoughts…
Big Data should be in “The Top 10 CIO concerns for 2016”, shouldn’t it?
The American society for Information Management published a study concerning 'The Top 10 CIO concerns for 2016' and the ability to leverage the promise of big data is often mentioned. I would therefore have expected to see big data initiatives included as one of the major concerns for CIO’s - yet it wasn’t explicitly mentioned. The study based on a poll of 785 organizations, yielded the following top 10 results:
- Alignment of IT with the Business
- Security and Privacy
- Speed of IT Delivery and Time-to-Market
- Business Productivity and Efficiency
- IT Value Proposition to the Business
- IT Agility and Flexibility
- IT Cost Reduction and Controls
- Business Agility and Flexibility
- Business Cost Reduction and Controls
As you can see, this list mainly focuses on improving the agility and flexibility of an organization - and achieving better cooperation between the business and IT. The often long and frustrating IT lead times to provide the business with answers makes this entirely understandable.
Although you could argue that big data falls under the ‘Innovation’ category, the fact IT departments typically spend around 95% of their budgets on just keeping existing stuff running, this doesn’t exactly leave much left for other innovative pursuits. If big data is so hot, then why don’t we see it on the list? Are the 785 organizations a non-representative statistical sample? Are there only few big data initiatives currently in circulation? Or perhaps we’re going to have to wait a couple of years to see the boom?
Turning structured data into information is already quite cumbersome…
Anyone with a smattering of business and IT knowledge will have noticed that companies all over the world still struggle to distill and extract key information from their ERP system. This is why the words alignment, speed, productivity, efficiency, IT value, cost, agility and flexibility are all littered across the above list. ERP systems consist of structured information, stored in many hundreds of tables and many thousands of fields (the traditional relational databases). Lack of data is therefore not the problem here, but data complexity and application logic complexity is. The question that has been buzzing in my mind since I first learned about big data is; “how can adding more unstructured data to the IT ecosystem be the answer?”. It simply isn’t.
In an SAP environment I can show you proof that it is perfectly possible to get end-to-end supply chain visibility and predictive analysis information with less than 5% of the total amount of data stored. That is not more data, but less - it's also smarter data. The old adage work smarter not harder springs to mind. Unfortunately smart data, in the case of SAP, is not available through a simple flick of a wand. With Every Angle it is the result of many man-years of hard analysis and development work by a team of people with a full understanding of operations management, SAP data structures, and software development.
Conversations I’ve had with professionals within my own personal network have made me aware that big data potential is definitely not to be ignored; a pharmaceutical company I know is using social media information to improve their medicines and supporting documentation for example. This ‘pool of unstructured data’ can indeed become an ocean of information if the proper tooling is employed, but if, and only if, one knows what you are looking for. The common denominator for all the key successes I've heard is that business experts with a wealth of interest and experience with information systems, a.k.a. Data Scientists, were employed throughout.
If turning structured data into actionable information has been found to be cumbersome, then I find it only logical to assume the same process for unstructured data will be even more tiresome and much harder work. Unless big data tooling alleviates this problem dramatically, for which I am yet to see any proof, then this is why I am quite convinced that it will take a couple of years to see the boom.
Need for and Availability of Data Scientists
Data Scientists are required to turn data into information - but is it a science to derive information from data? It sure is…
An ideal Data Scientist masters IT development (data modelling, data structuring, computing and coding) but also statistics (the math, the methods, pattern recognition, etc.) and visualization methods. With proper tooling and technical knowledge, it is not that difficult to pinpoint correlations showing statistical relations. Yet statistical relations are not the same as causal relationships. The data showing a reduction of the population of storks and another set of data showing a drop in the number of births in a certain region, during the same period, may suggest a correlation, yet there is no causal relationship. This is of course assuming most people appreciate that babies are not delivered by storks! The ability to find a causal relationship is key, and that can only be achieved with in-depth domain knowledge (a.k.a. business knowledge) and solid reasoning.
Well-educated IT experts are knowledgeable on algorithms, coding, databases and statistics, but the ones that also understand the underlying business logic and can really communicate with the business (by really understanding the what and why of the questions and being able to discuss this with the business) are the proverbial needle in a haystack. Tamara Dull, Director of Emerging Technologies and Thought Leader at SAS, describes such an omnipotent expert as a unicorn, or to be more precise, “a quant who can find that insightful needle in your data haystack while delivering a charismatic TED talk”. Let's face it, good Data Scientists are very, very scarce. You will need time and a well-filled wallet to hire someone of this ilk I’m afraid. Yet another reason to postpone the start of the big data boom by a couple of years.
Balancing my own view on Big Data
Big data potential is certainly promising - powerful data visualization, data discovery and data exploration tools are already available. The same goes for in-memory technology, which is mandatory for high speed analysis and reporting, and agile data extraction and model building; mandatory for ever changing data source structures and data source content. However, all of this new stuff cries for those that are blessed with technical and business knowledge to support it - both of which are hard to find and will remain so for quite some time. It is also important to remember that none of these new technologies will be enforced if it is not backed and made a major priority by the C-suite.
So I think it’s fair to say that I’m not currently a big bata believer. I need to be convinced by facts, logic, observations, tangible business cases and true stories about the efforts and the costs. Until now, this evidence is as scarce as the available Data Scientists.
On Wednesday September 21, I will host a presentation on this topic at the Big Data Expo 2016 event, which will take place at the Jaarbeurs in Utrecht. You can subscribe for a free full conference ticket via this link. If you’re open for a conversation on this topic, come and meet me at the Every Angle booth (B42). Follow the scent of freshly baked lolly waffles!
Recent Posts in Data
Business Analytics (19)
Every Angle (11)
Industry Perspective (2)
IT and business alignment (3)
Supply Chain (7)
The self-service struggle for IT
Control Towers: Why you need to be able to control the planes in the air
Every Angle announces R2017: the power to know even more, the freedom to act even faster!
Wanna Be Agile? Give your users ‘Real Business User Self-Service’
Santa: Deconstructing The World's Best Supply Chain
Report on the Past or Control the Future? What Would You Choose?
Supply Chain Control Towers – Reporting is Not Enough...
IT – Business Enabler or another Functional Silo?
Measuring what matters
Big Data: Big Disappointment or Massive Success?