The widespread misconception that Kubernetes was not ready for stateful applications such as MySQL and MongoDB has had a surprisingly long half-life. This misconception has been driven by a combination of the initial focus on stateless applications within the community and the relatively late addition of support for persistent storage to the platform.
Further, even after initial support for persistent storage, the kinds of higher-level platform primitives that brought ease of use and flexibility to stateless applications were missing for stateful workloads. However, not only has this shortcoming been addressed, but Kubernetes is fast becoming the preferred platform for stateful cloud-native applications.
Today, one can find first-class Kubernetes storage support for all of the major public cloud providers and for the leading storage products for on-premises or hybrid environments. While the availability of Kubernetes-compatible storage has been a great enabler, Kubernetes support for the Container Storage Interface (CSI) specification is even more important.
The CSI initiative not only introduces a uniform interface for storage vendors across container orchestrators, but it also makes it much easier to provide support for new storage systems, to encourage innovation, and, most importantly, to provide more options for developers and operators.
. StatefulSets automatically handle the hard problems of gracefully scaling and upgrading stateful applications, and of preserving network identity across container restarts. StatefulSets provide a great foundation to build, automate, and operate highly available applications such as databases.
Second, to make it easier to manage stateful applications at scale and without human intervention, the “Operator” concept was introduced. A encodes, in software, the manual playbooks that go into operating complex applications. The benefits of these operators can be clearly seen in the operators published for , , and .
In conjunction with these orchestration advances, the flourishing of , the equivalent of a package manager for Kubernetes, has made it simple to deploy not only different databases but also higher-level applications such as GitLab that draw on multiple data stores. Helm uses a packaging format called “charts” to describe applications and their Kubernetes resources. A single-line command gets you started, and can be easily embedded in larger applications to provide the persistence for any stack. In addition, multiple reference examples are available in the form of open source charts that can be easily customized for the needs of custom applications.
, an open-source project, has been driven by the increasing need for a universal and application-aware data management plane—one that supports multiple data services and performs data management tasks at the application level. Developers today frequently draw on multiple data sources for a single app (polyglot persistence), consume data services that are eventually consistent (e.g., Cassandra), and have complex requirements including consistent data capture, custom data masking, and application-centric backup and recovery.
Kanister addresses these challenges by providing a uniform control plane API for data-related actions such as backup, restore, masking, etc. At the same time, Kanister allows domain experts to capture application-specific data management actions in blueprints or recipes that can be easily shared and extended. While Kanister is based on the Kubernetes Operator pattern and Kubernetes , those details are hidden from developers, allowing them to focus on their application’s requirements for these data APIs. Instead of learning how to write a Kubernetes Controller, they simply author actions for their data service in whatever language they prefer, ranging from Bash scripts to Go. Today, public examples cover everything from MongoDB backups to deep integration with PostgreSQL’s Point-in-Time-Recovery functionality.
Whereas Kanister handles data at an application level, significant operator challenges also exist for managing data within multiple applications and microservices spread across clusters, clouds, and development environments. We at Kasten introduced the to make it easy for enterprises to build, deploy, and manage stateful containerized applications at scale. With a unique application-centric view, K10 uses policy-driven automation to deliver capabilities such as compliance, data mobility, data manipulation, auditing, and global visibility for your cloud-native applications. For stateful applications, K10 takes the complexity out of a number of use cases including backup and recovery, cross-cluster and multi-cloud application migration, and disaster recovery.
, , and mature, expect to see even more innovation in this space.
As we turn the page on this first chapter of the evolution of stateful cloud-native applications, the future holds both a number of opportunities as well as challenges. Given the true cloud portability being offered by cloud-native platforms such as Kubernetes, moving application data around multi-cluster, multi-cloud, and even planet-scale environments will require a new category of distributed systems to be developed.
Data gravity is a major challenge that will need to be overcome. New efficient distribution and transfer algorithms will be needed to work around the speed of light. Allowing enterprise platform operators to work at the unprecedented scale that these new cloud-native platforms enable will require a fundamental, application-centric rethinking of how the data in these environments is managed. What we are doing at Kasten with our K10 enterprise platform and Kanister not only tackles these issues but also sets the stage for true cloud-native data management.
Niraj Tolia is co-founder and CEO of , an early-stage startup working on cloud-native storage infrastructure. Previously, he was the senior director of software engineering at EMC/Maginatics where he was responsible for the CloudBoost family of products that focused on in-cloud data protection. Prior to EMC’s acquisition of Maginatics, he was a founding member of the Maginatics team and played multiple roles within the company including vice president of engineering, chief architect, and staff engineer. Niraj received his PhD, MS, and BS degrees in computer engineering from Carnegie Mellon University.
New Tech Forum provides a venue to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to .