Jul 10, 2019
Seth Mason
D2iQ
3 min read
Running distributed stateful applications is never an easy job, and one of those key areas is providing the optimal storage backend for your workload. Run the entire application on SSD and your costs are too high, run it all on HDD and your performance takes a hit. Leveraging a seamless method for matching your workload to your storage backend's capabilities is key.
Today we are proud to announce the general availability of Policy-driven storage on Mesosphere DC/OS Enterprise. Policy-driven storage, powered by the DC/OS Storage Service (DSS) enables Policy-driven Storage Orchestration for Distributed Services that provide Application-Level Redundancy.
DC/OS Storage Service
The DC/OS Storage Service (DSS) is a foundational service for DC/OS that enables you to simplify your storage operations for stateful workloads. If you are managing a complex set of workloads, you're facing many different requirements for the type of storage required. Even within the same application, there are different storage needs. For example, you're likely to use a different type of storage for your production data vs. log data. DSS enables this flexibility.
Centralized control plane for storage volume lifecycle management of direct attached storage.
- Automation of volume lifecycle management: volume provisioning, allocation, de-allocation, and de-provisioning.
- Enable policy-driven consumption of differentiated storage types by distributed services
- Automatically match workload requirement to storage providing node without manual configuration
- Initial support for Marathon applications, plus Cassandra, Elasticsearch, Kafka, HDFS and dcos-monitoring (Prometheus/Grafana) frameworks
Consider one of your larger distributed applications - one that requires different types of volumes for its subcomponents. You don't want to be faced with manual configuration or manually matching a given task to a specific volume on a specific node. Defining policies, for example Production Logs and Production Data, allows your application developers to easily pick the policy that meets the needs of their data set. As the cluster operator, you're building out a catalog of storage policies that your users, the Application Developers, may choose from.
Application developers do not need to be aware of which cluster node or storage system is providing the storage. Instead, in their application definition, they just need to specify the Policy that they require and DC/OS takes care of locating a volume that meets that policy with CPU/Memory requirements so that the application can be scheduled to that node.
From an Operator's perspective, management of DSS is easily automated via JSON configuration, and adding capacity is as simple as configuring a new DC/OS agent to provide more capacity to an existing profile/policy. DC/OS will immediately begin scheduling workloads to that node. Before hitting day-2 operations, there are pre-built, Grafana dashboards leveraged by the dcos-monitoring service to provide you with insight into the operational state and performance of DSS.
Access node-mounted, industry standard, volumes provided by CSI (Container Storage Interface)
- Industry Standard interface
- Dynamically add storage to consolidated storage resources at cloud-scale
DC/OS Storage Service, available for DC/OS Enterprise, builds upon the open source CSI API to leverage the strength of the community in areas such as storage provisioning and managing local storage for distributed applications. As part of the CNCF landscape, the same CSI plugins can be used across other platforms such as Kubernetes. Additionally, as your application storage requirements grow and your data needs increase, you'll be able to seamlessly add capacity to your cluster and therefore add capacity to your applications to accommodate their insatiable demand for storage.
DSS is now available today for General Availability and is installable via an online download into your DC/OS Catalog. Running on DC/OS Enterprise, test it out with your favorite distributed application and see for yourself the ease provided by Policy-driven storage.
For more information, including examples, see the documentation.
