Apache Hadoop emerged on the IT scene in 2006 with the promise to provide organizations with the capability to store an unprecedented volume of data using cheap, commodity hardware. In this brave new world, businesses could store as much data as they could get their hands on in Hadoop-based repositories known as data lakes and worry about the analysis later.

These data lakes were accompanied by a number of independent open source compute engines – and on top of that, “open source” meant free! What could go wrong? 

Monte Zweben, CEO of Splice Machine, says there were three main reasons behind the downfall of Hadoop. They are:

Schema-on-Read was a mistake
First, the so-called best features of Hadoop turned out to be its Achilles heel. With the schema-on-write restriction lifted, terabytes of structured and unstructured data began to flow into the data lakes. With Hadoop’s data governance framework and capability still being defined, it became increasingly difficult for businesses to determine the lineage of their data, causing them to lose trust in their data and data lakes to turn into data swamps.

Hadoop complexity and duct-taped compute engines
Second, Hadoop distributions provided a number of Open Source compute engines like Apache Hive, Apache Spark and Apache Kafka to name a few, but this turned out to be too much of a good thing. These compute engines were complex to operate and required specialized skills to duct-tape together that were difficult to find in the market.

The wrong focus – the data lake vs. the application
Third and most importantly, data lake projects began to fail because Hadoop clusters often became the gateways of enterprise data pipelines that filter, process, and transform data that is then exported to other databases and data marts for reporting downstream and almost never find their way to a real business application in the operating fabric enterprise. As a result, the data lakes ended up being a massive set of disparate compute engines, operating on disparate workloads, all sharing the same storage. This is very hard to manage. The resource isolation and management tools in this ecosystem are improving but they still have a long way to go. Enterprises were not able to shift their focus away from using their data lakes as inexpensive data repositories to platforms that consume data and power mission-critical applications.

 Many organizations are concerned about the recent developments in the Hadoop ecosystem and are under pressure to demonstrate the value of their data lake. I would love to put you in touch with Mr. Zweben to comment on how enterprises can successfully modernize their applications after the fall of Hadoop, and the best strategy for getting there.