Jul 30, 2015
Michael Hausenblas
D2iQ
3 min read
Apache Mesos is the kernel of our Datacenter Operating System (DCOS), and it's just getting better. The latest Mesos release, 0.23.0, brings to the table some exciting features that advance the DCOS and enable you to do even more cool stuff.
Here's a look at just a few of them.
Oversubscription
High-priority, user-facing services are typically provisioned on large clusters for peak load and unexpected load spikes. This means that for most of the time the cluster is under-utilized. The new 0.23 oversubscription feature takes advantage of temporarily unused resources to execute best-effort tasks such as background analytics, video/image processing, chip simulations, and other low priority jobs. Oversubscription adds two new slave components: a Resource Estimator and a Quality of Service (QoS) Controller. The Resource Estimator taps into the resource monitor and periodically gets usage statistics to identify the amount of oversubscribed resources. The QoS Controller informs the Mesos slave that particular corrective actions—containing information about executor or task and the type of action—need to be made.
In the context of the DCOS, this means that you will be able to get even more bang for your bucks: more applications with the same hardware investments or lower TCO.
Persistent volumes
Mesos now provides a mechanism to create a persistent volume from disk resources. Up to 0.23, stateful services such as distributed filesystems (HDFS, Quobyte, etc.), NoSQL databases (Cassandra, ArangoDB, etc.) and message queues like Kafka had to use out-of-band techniques -- network-mounted EBS volumes or unmanaged disk resources in a well-known location, for example -- to store their data within Mesos.
With persistent volumes, these stateful services have now become first-class citizens, making the life easier for DCOS service developers and providing a more robust experience for DCOS end-users.
Dynamic reservations
Experimental support for dynamically reserving resources on specific Mesos slaves has been added with 0.23: rather than requiring a human operator to upfront specify a fixed, pre-calculated set of static reservations on slave startup, a DCOS service can now reserve resources as they are being offered, without requiring a slave restart. No breaking changes were introduced with dynamic reservation, which means the existing static reservation mechanism continues to be fully supported.
This new feature means for DCOS operators less manual interventions when business requirements change priorities concerning the cluster usage.
And even more …
Other noteworthy improvements with 0.23 include:
- Per-container network isolation. Mesos now provides support for per-container network monitoring and isolation. Network isolation prevents a single container from exhausting the available network ports, consuming an unfair share of the network bandwidth or significantly delaying packet transmission for others.
- SSL. Encrypting traffic between the Mesos master and its slaves and services is important for information security, as it prevents eavesdropping and impersonation.
- Fetcher caching. The fetcher can be instructed to cache URI downloads in a dedicated directory for reuse by subsequent downloads.
Hundreds of other bug fixes/improvements are included in Mesos 0.23.0 and rolling upgrades from a Mesos 0.22.x cluster to Mesos 0.23 are straightforward, but there are a few caveats/deprecations.
While some of the above mentioned features are still experimental, we aim at updating the DCOS with Mesos 0.23 in the next few weeks. Of course, you can already now download the tested packages and install on your own.
