“We’ve built a 70,000-node Mesos cluster for our developers, but they won’t use it. Can you help?” This was the beginning of a conversation with the VP of infrastructure operations in a very large and famous company. While an impressive feat to accomplish, it was also by far the largest containerized infrastructure setup I had seen that had gone unused—nor, sadly, was it an isolated incident.
I’ve talked about this encounter with a large number of customers, analysts, friends, colleagues, partners, venture capitalists, and competitors. We all expressed similar experiences, and all wanted to know why this is so. After all, if so many resources are being wasted in our industry, we are all risking a great deal by not understanding and solving the problem. Otherwise, the next wave of adopters might start to doubt containers can help their businesses, and we would all need to starting polishing our resumes.
I have to be honest here: I am a developer, an engineer and a technologist who loves to build products and use the latest technologies. So, the first place I looked, in my quest to find an answer to this 70,000-node question, was the technologies used. Was Mesos the wrong technology? Was it implemented the wrong way? Did they use open source or closed source? Was there an SI involved? Questions like these came first to mind. In hindsight I think those were probably the wrong questions.
The answer came to me when I remembered a day in my career 15 years ago: