Apr 19, 2016
D2iQ
D2iQ
6 min read
It's been 5 years since Marc Andreessen coined the phrase software is eating the world, but indeed it is. And the types of software we're talking about here go far beyond the ubiquitous mobile apps and websites now attached to nearly every product we buy. Doing business in the 21st century means embracing a world where software quite literally powers everything—from televisions to cars, and from social media to banking systems.
If you're not at least competent, you can't compete. If you want to master business in the 21st century, you have to excel at software. It's a fact of which today's smartest companies—Amazon, Apple, Facebook, GE, Google, Samsung and Uber as a small sample—are well aware.
That's why today, we're proud to announce the open source DC/OS project. The product of years of work by Mesosphere, and backed by dozens of partners big and small, DC/OS gives all companies the foundation they need to thrive in a software-defined economy. Keep reading to learn more about how DC/OS works, who's involved and how the project will grow.
Built for today's applications—and tomorrow's
There has been a lot of recent excitement around containers and Docker. DC/OS supports these modern packaging systems natively, having been built around containers from day one. But one of the driving forces behind DC/OS is the new reality that, while important, containers and container orchestration are not sufficient.
Software doesn't start and stop with the bits in containers, any more than it starts and stops with the bits that are installed directly on a mobile device or control system in the dashboard of a car. Whereas in the past an application might have been entirely local or connected to a single server, today's best applications are often built on distributed systems comprised of myriad individual services, storage engines and data-processing tools. Even a cloud-based HR application inside a large enterprise is just the consumer touch point of a much larger software system.
Today's modern enterprise applications include microservices running in containers and stateful big data services. Modern apps make extensive use of open source software, enabling enterprises to benefit from community contributions to core technologies, including those yet to come.
Applications are built like this so that they can handle today's demands around performance, features, reliability and continuous improvement. And the better all these pieces work together, the better the user, operator and developer experience. That part is hard work, which is why the companies mentioned above have spent untold engineer-hours and dollars building out the software infrastructure platforms that can run these types of applications.
DC/OS eliminates much of this work, allowing users to easily deploy a single platform for running everything that modern applications require —from Docker containers to big data systems—all on the same set of shared resources.
Banding together to deliver DC/OS
DC/OS is an open source technology (more on that in a bit), but it's also an ecosystem. It was spearheaded by Mesosphere with participation from more than 60 partner companies as of today, all of which understand the importance of developing new types of infrastructure software for new types of applications. Partners include cloud computing providers such as Microsoft—whose dedication to open source software was a catalyst for the project's creation—as well as leading systems integrators such as Accenture, and some of the world's most innovative consumer-technology companies, such as Yelp.
A small sample of other partners includes Autodesk, Cisco, Confluent, EMC, Equinix, Hewlett-Packard Enterprise, NGINX, Puppet and Verizon.
Every DC/OS partner received early access to the code, and is committed to helping grow and shape the project in their own ways. Some will integrate their technologies with DC/OS, while others will build new products on top of it. Others will run DC/OS inside their datacenters and contribute code to ensure it continues to be the most reliable, scalable and easiest platform for running containers, microservices, big data systems and everything else a modern enterprise application requires.
Organizations and individuals that want to get involved with DC/OS can learn more and sign up on the project website. Watch the video below to see current partners talk about running production container environments on DC/OS.
DC/OS: Mesos and a whole lot more
DC/OS derives from Mesosphere's Datacenter Operating System, a commercial product built around Apache Mesos. Open sourcing DCOS has always been part of our strategic roadmap and we're proud to have collaborated with our launch partners for today's unveiling.
DC/OS is a software platform that's 100 percent open source, comprised of more than 30 component technologies, including Apache Mesos and Marathon. Some of the technologies were always open source, including Mesos, while others were previously proprietary code developed by Mesosphere, such as the GUI and our Minuteman load balancer.
Features include:
- The industry's most-proven, most production-ready container orchestration platform, based on Mesos and Marathon.
- Fast, push-button, app-store-like installation of complex distributed systems (including HDFS, Apache Spark, Apache Kafka, Apache Cassandra and more) with DC/OS Universe.
- Built-in high-availability and fault-tolerance for all types of applications, services and workloads.
- GUI-based monitoring and management. Significant simplification of datacenter management, by turning thousands of machines into a single logical computer.
- Intelligent workload scheduling with automated placement of tasks and containers to increase utilization.
- Standard user experience whether deployed on bare metal, virtual machines or in the cloud.
DC/OS is not a toy—the technologies on which it is built already power production applications and massive container environments at some of the world's largest and most-innovative companies. Partner and user Autodesk has seen drastic improvements since moving its streaming and big data ingestion pipeline services ton DC/OS:
- 66 percent reduction in AWS Instances
- Up to 57 percent cost improvements
- 40-second time to deploy a new build, with zero downtime
- 3-minute time to stand up a new region
- 100 percent uptime
- 1 DevOps engineer to manage it all
Watch the video below to get a feel for just how powerful—and easy—DC/OS is.
Why we open sourced our Datacenter Operating System
DC/OS is an open source project because we want to put its power into as many people's hands as possible. Specifically, DC/OS helps companies adopt the advanced operational and architectural practices of organizations such as Apple, Yelp, Netflix and Twitter (and, at a broader level, Facebook and Google) without having to reinvent the wheel or hire scores of distributed systems engineers. DC/OS brings this type of advanced datacenter environment and application platform to anyone, anywhere they wish to run it.
At Mesosphere, we are big believers in open source. Open source software helps its creators overcome the limitations of their own vision. Having a vibrant community of users, partners and contributors means we can continue to advance DC/OS as new requirements and use cases emerge.
The heart of DC/OS is Mesos
At the heart of DC/OS is the Mesos distributed systems kernel, and DC/OS utilizes Mesos to handle job scheduling, resource management and abstraction, high availability, and other infrastructure-level processes. Apple, Yelp, Samsung, Twitter and other companies have scaled Mesos to remarkable proportions and run some truly innovative applications on top of it.
However, it's the technologies built on top of and around Mesos that combine with Mesos to make DC/OS what it is. These include the native container-orchestration and application platform (Marathon); the DC/OS installer and Universe experience; the powerful but intuitive user interfaces for management and monitoring; and improvements around networking, security, load balancing, service discovery and much more.
DC/OS is the only open source project that bundles all these things together into a single software distribution. DC/OS gives every company the ability to do what these early Mesos adopters did—and then some—but with a fraction of the engineering effort.
Mesos will always be at the core of DC/OS, and Mesosphere will continue to be a leader in the Mesos community (come meet us at MesosCon in June). Our engineering team will continue to make large contributions to Mesos and Mesos co-creator and Mesosphere co-founder Ben Hindman will continue to lead the Apache project. Every improvement and feature added to Mesos will, by its very nature, be incorporated into DC/OS, as well.
For more thoughts from Ben about open source DC/OS, check out his blog post.
DC/OS is containers—and so much more than Docker
Because of its strong application-container focus, it might be tempting to compare DC/OS with other popular container technologies, such as Kubernetes and Docker Swarm. While that might be fair if your sole focus is the DC/OS Marathon technology and its ability to run Docker containers, the reality is that DC/OS as a whole stands out in three very important ways:
1. DC/OS is great for Docker, but also great for all containers. DC/OS, and its Marathon container orchestrator component in particular, is an outstanding platform for running Docker containers. But with multiple container formats emerging, users need flexibility.
DC/OS is built around Mesos, which uses pluggable containerizers in order to handle existing and emerging container formats and already supports both Docker containers as well as "native" Mesos containers. The latter utilize Linux cgroups and namespaces, and are the default method of isolating resources when launching a DC/OS service such as Spark or HDFS. The beautiful thing about the native Mesos containerizer is that users don't really have to think about containers at all. When users install a new service via the Universe GUI or the command line (for example, by typing ‘dcos package install kafka'), it's automatically scheduled and isolated inside Linux containers—running like on bare metal with no image creation required! Complex distributed systems can literally be deployed in minutes.
2. DC/OS delivers on container operations rather than mere container orchestration. The Marathon service that's included with DC/OS is the most-production-proven container orchestration engine on the planet. It includes a collection of powerful features for launching, managing, scaling and networking containers (and also long-running non-containerized apps), and has an aggressive roadmap for new feature development. Large production Marathon users include Verizon, Samsung, Yelp, Autodesk, Disney, Mattermark and more.
3. DC/OS has a unique two-level scheduler that can run many systems, including Kubernetes and Swarm, as services. Another critically important thing to remember about DC/OS is that it can actually run any distributed system that has been optimized to run on it—including such popular services as Spark, Kafka, Cassandra and Jenkins. These services appear in the Universe and can be installed on DC/OS and run on the same cluster as all the other workloads
Available services
Packages presently included in Universe and ready to install as DC/OS services today include: ArangoDB, Avi Networks, Cassandra, Chronos, Confluent, Crate, DataDog, Elasticsearch, Etcd, Exhibitor, HDFS, Hue, Jenkins, Kafka, Linkerd, Marathon-lb, Marathon, MemSQL, mr-redis, Namerd, NGINIX, OpenVPN, Project Calico, Quobyte, Riak, Ruxit, Spark, Spark Notebook, Storm, Swarm, Weave and Zeppelin. More will be added shortly, and even more will come online as the community matures.
DC/OS is optimized for innovation
DC/OS is presently organized by Mesosphere with contributions and roadmaps from community committers. This is a proven path to success for open source projects—Linux as one shining example—and we think it is especially important in the early days in order to sustain development momentum and establish a vibrant DC/OS community.
We have created a separate organization in GitHub for DC/OS and we are releasing it under the Apache License 2.0, which allows users to download and use the software freely. We will be exploring different long-term stewardship options, including options where DC/OS is overseen by a third party such as the Apache Software Foundation, Linux Foundation or the Cloud Native Computing Foundation. We will work together as a community to figure out the long-term home, and in the meantime we're optimizing for shipping code to users and making the platform as stable possible.
Thank you, Microsoft
The DC/OS community is indebted to Microsoft for its early commitment and deep support for a one hundred percent open source DC/OS. Microsoft, with its early commitment to building Azure Container Service around DC/OS and generally backing open source software, was a catalyst for the creation of DC/OS and is a close collaborator. If there's one thing that Mesosphere, Microsoft and all the DC/OS partners share, it is a commitment to giving developers, ops teams and all IT stakeholders an open standard on which to run the applications that will power their companies from here on out. Especially with Azure, Microsoft has been quick to add support for proven open source technologies such as Linux, Hadoop and R, and DC/OS is the latest chapter in that story.
Watch Microsoft explain the importance of DC/OS in the video below.
Mesosphere's business remains largely the same
Mesosphere is still a software company selling our products to Global 2000 enterprises. Nothing has changed on that front. Our Mesosphere Enterprise DC/OS product (previously called DCOS Enterprise Edition) enhances the open source by offering key enterprise features around security, performance, networking, compliance, monitoring and multi-team support that the DC/OS project does not include. The two versions will remain distinct as we continue to add new features to both DC/OS and Mesosphere Enterprise DC/OS in order to make each as valuable as possible to their respective users.
Mesosphere also has targeted software products, Infinity and Velocity, that are designed to work in conjunction with Enterprise DC/OS and will be generally available soon. These are pre-integrated solutions for real-time data processing and CI/CD (or application development lifecycle management), respectively, that drastically simplify the deployment, operation and scalability of complex pipeline systems involving numerous components.
How to get DC/OS and join the community
You can learn more about DC/OS on its website at dcos.io, and in its documentation. The code is also available on the DC/OS GitHub page. For a full list of deployments method and installation instructions, from your laptop to the cloud, check out the DC/OS installation page.
If we haven't made it clear yet, DC/OS is all about community. You can sign up and become a contributor on GitHub. Or hop into our Slack channel to ask questions and meet the community. We'll also post regular updates on Twitter, where we're @dcos_io.
Please do download it, hack on it, try to break it, install Spark and Zeppelin and start doing some data analysis, deploy a bunch of Docker containers, or whatever you please. We're happy to share DC/OS with the world. Datacenters and the applications that run inside them are changing, and DC/OS provides the tools to start taking advantages of these changes today.
More information
You can get more information about DC/OS and follow its progress in the following places:
