Kubernetes may be the current darling of the open source crowd, but Hadoop was no less revered before it. because it was incredibly hard to use. , though making strides, remains “no picnic to operate,” as Capital One’s . That’s a very diplomatic way of saying, as , that the Kubernetes “experience [can be] a pain in the ass.”
Is Kubernetes heading toward a Hadoop-style exit?
Probably not. While Hadoop got more complicated with age, Kubernetes keeps getting easier. While Kubernetes will likely never be “easy,” per se, its complexity differs from that of Hadoop in critical ways, paving the way for Kubernetes to remain an industry standard for years to come.
Hadoop, the complex gift that kept on taking
Let’s first be clear about Hadoop. Or not clear, as the case may be. Apache Hadoop was complicated enough when it roughly translated to “MapReduce.” Over time, however, it kept evolving, and while that evolution led to more powerful options, those options proliferated. They also weren’t necessarily easy to get working together. As , “What does Hadoop actually do? MapReduce was replaced by Spark was replaced by other stuff and so on. Of course you can plug a lot into it but it’s still clunky.”
Why clunky? VMware’s nicely: “Hadoop’s complexity comes from the fact that a typical Hadoop setup basically consists of dozens of independent and complicated systems that had different lifecycles and management models.” Flume, Chukwa, Hive, Pig, ZooKeeper, and so on. Clever names but a nightmare to get everything working together. Hadoop is a “complex stack of solutions,” Host Analytics CEO Dave Kellogg, and all that complexity is born by the user.
Perhaps most differently from Kubernetes, however, is the model used to extend Hadoop. As , “Hadoop didn’t think about how people would extend it and the result was an ecosystem of incompatible extensions.” By contrast, he continues, “One thing Kubernetes gets extremely right is structuring the way it gets extended. Operators, CRI/CSI/CNI, ensure that as more vendors pile on, they do so in sane ways.” In other words, unlike Hadoop and its incompatible extensions, “Kubernetes with dozens of operators is still Kubernetes.”
, “Kubernetes is a complex system.” That complexity is somewhat necessary, he goes on, because “It does a lot and brings new abstractions.” Does everyone need all of those abstractions (and bells and whistles) all of the time? No. “I’m sure that there are plenty of people using Kubernetes that could get by with something simpler.”
But for those who need Kubernetes, Beda stresses, it’s not necessarily more complex than other system with which people are already familiar. It may simply be “new” complex versus “old and comfortable” complex:
, “You can run Kubernetes locally much, much easier (Docker Desktop, Kind, MicroK8s) than the other similar examples. Lowering the barrier to entry makes it easier to become familiar, which combats perceived complexity.”
It also helps, as Cloud Native Computing Foundation executive , that while “distributed systems are inherently complex, the upside with Kubernetes is that every major worldwide cloud provider and multiple vendors offer a managed conformant/certified version of it (no forks) which helps most users with complexity of managing at scale.” Even so, perhaps the right question, , is whether “Kubernetes [is] complex given the problem it tries to solve.” For him, the answer is no.
That is the same answer to the question, “Will Kubernetes get Hadooped?” Kubernetes is already well past that stage. Yes, as , Kubernetes is “a complex orchestration tool, and not ideal for all use cases. Like so many of the tools in our space, it also takes time to learn, use, and understand. ‘A few hours’ isn’t going to be sufficient.” It’s a complex tool solving a complex problem. But there is “intentional complexity and accidental complexity,” as Beda argues. Hadoop suffered from the latter, while Kubernetes involves the former.
For these and other reasons, we should see Kubernetes continue to thrive as the industry standard for container orchestration.