Mar 20, 2015
Elizabeth Lingg
D2iQ
5 min read
Mesosphere has announced HDFS on Mesos, Version 0.1.0, an open source project to fully integrate HDFS with Mesosphere's stack. We are ready for beta testing in the community. The HDFS Datacenter Service enables the Mesosphere community to integrate the data storage layer with Mesos. It also allows for a datacenter stack with HDFS integrated with Mesos to be used by big data solutions such as Hadoop and Spark, which also seamlessly run on your Mesos cluster.
HDFS on Mesos is being fully integrated into the Mesosphere Datacenter Operating System (DCOS) and Mesosphere is accepting applications from community members who want to be part of the DCOS Early Access Program.
Mesosphere partnered with Nic Grayson and Luke Amdor from Banno, part of Jack Henry and Associates. We would also like to thank the community members and partners who helped with early testing prior to this release.
Ease of Installation and Administration
For organizations that run HDFS, conventional practice has been for operations teams to manually install HDFS on their cluster with resources, such as disk space, statically partitioned. In this siloed model, setup and maintenance is painful. If a NameNode, DataNode, or one of their host machines goes down, an engineer or operator needs to manually run the appropriate HDFS command based on the type of failure. They may need to set up new instances and account for failed tasks manually.
Instead of manually changing configurations on each machine in the cluster, the HDFS on Mesos service updates the configurations and automatically detects changes in the cluster. We also have a configuration option to run HDFS with Mesos DNS, which allows operators to utilize Mesos DNS service discovery in the configuration files.
Fault Tolerance
With HDFS running on Mesos, setup is done when the service launches and when nodes are launched on hosts with the appropriate resources. If a task fails, each failure is accounted for by a custom scheduler and a state machine transitions between all the appropriate states, JournalNodes launching, NameNodes launching, NameNode initialization, and DataNodes launching.
We are able to run HDFS in Super High Availability mode. Not only are we running HDFS in high availability mode, but we are benefiting from the HA and fault tolerance features inherent in Mesos and leveraging them to our advantage. If one NameNode fails, for example, we first try to relaunch it on the machine where it is was running. If that fails, it is then relaunched on any host with available resources and is bootstrapped off the original NameNode.
Resource Management and Full Stack Installation
Resource management is done within the HDFS on Mesos service and utilizes the underlying Mesos APIs. Efficiently utilizing cluster resources along with the ability to run alongside other datacenter services such as Cassandra, Myriad, Chronos, Marathon, and Kubernetes is a significant benefit.
Running a full stack for big data workloads can be be achieved by installing HDFS on Mesos alongside Spark or Hadoop on the same Mesos cluster. Resource management, fault tolerance, and ease of administration are provided via Mesos and the Mesosphere DCOS.
Join the Community
We are looking to build a strong HDFS community. If you have an interest in big data and data storage in the datacenter, please test out the beta version and use the HDFS GitHub project to submit tickets, pull requests, or become a contributor.
