Nov 05, 2015
Roger Ignazio
D2iQ
4 min read
Today, we're excited to share some of the work we've done to reliably deploy Jenkins masters using Mesosphere's Datacenter Operating System (DCOS) and Marathon. Up until this point, Jenkins has been usable on Apache Mesos, but deploying it on the DCOS was non-trivial.
To help organizations better scale their Jenkins environments—and to avoid the statically partitioned infrastructure that is typically associated with multiple Jenkins clusters—the team at Twitter developed an open source Mesos plugin for Jenkins. This plugin allows multiple Jenkins masters to share a single pool of resources to run builds, with the ability to dynamically create and destroy Jenkins agents as demand increases or decreases.
Perhaps most notably, eBay uses the Mesos plugin for Jenkins to improve resource utilization in its clusters while still providing thousands of individual Jenkins masters for each of its developers.
Now, DCOS users can reap the same benefits.
How it works
When starting this project, we faced two major hurdles:
- Jenkins persists all configuration and build history as individual files on disk, and this does not map well to the Mesos concept of disposable working directories (the task's sandbox, to put it in Mesos terms).
- Jenkins—specifically the Mesos plugin for Jenkins—must be configured with the full URL of the instance. This URL can change depending on the Mesos agent and port that the instance is running on.
We got around the persistence problem by introducing a new requirement: users must provide a mount point to a shared file system on each of their DCOS or Mesos agents. There are a number of existing shared file system solutions, including NFS, HDFS (plus its NFS gateway), Ceph and more. There are even some efforts to run HDFS and Ceph on top of Mesos, making use of the new storage primitives recently introduced into the core Mesos project. For the purposes of this post, we'll use NFS as the example.
For the second issue—reconfiguring specific Jenkins parameters based on the current host and port—we opted for a bootstrap script that executes at container runtime. This bootstrap script either populates the host volume with the base configuration and plugins that we've provided within the Docker image, or if user data already exists, it modifies the two necessary Jenkins configuration options so that the master can successfully register as a Mesos framework and launch tasks on the cluster.
Installation
You can easily configure and install Jenkins for DCOS (and open source Marathon) by using the DCOS CLI. As mentioned previously, you must have a single NFS share mounted at the same path on each of your DCOS or Mesos agents. Since there are plenty of NFS resources available on the Internet, we won't cover that part here. There already some great NFS guides online for Red Hat Enterprise Linux, Ubuntu, and CoreOS.
First, create the file options.json with any instance or site-specific information, such as the framework name and the base path to the NFS share on the DCOS or Mesos agent:
{ "jenkins": { "framework-name": "jenkins-myteam", "host-volume": "/mnt/nfs/jenkins_data" } }By choosing a unique framework name, you can run multiple Jenkins instances on the same cluster, sharing the cluster's resources among all Jenkins masters. Each Jenkins instance creates a subdirectory inside the directory that you specified for the host volume. In this particular case, the data is stored on the NFS server at /mnt/nfs/jenkins_data/jenkins-myteam.
You will need to update your local cache of the DCOS package repository to pull in the Jenkins package:
$ dcos package update
Then, install Jenkins just as you would any other DCOS service:
$ dcos package install jenkins --options=options.json
That's it! The first time Jenkins runs, it populates the directory on your NFS share with a basic Jenkins configuration and a small set of plugins. If Marathon needs to restart the task on a different host, the container automatically mounts your existing data directory, including job configurations, build history and installed plugins. For future versions of DCOS, we are planning add support for other distributed file systems and leverage the persistence primitives available in recent versions of Mesos.
More information
We plan to support Jenkins masters running on DCOS beginning in DCOS version 1.4, which is scheduled to be released later this year. If you'd like to try it out sooner, we have a hotfix available for DCOS 1.2 and 1.3 that resolves an issue with accessing the Jenkins web interface. For more information, contact the team at support@mesosphere.io.
In the spirit of promoting open software, we've made this project publically available on GitHub. For more information on using Jenkins with DCOS or open source Mesos and Marathon, or to learn how to contribute to the project, check out https://github.com/jenkinsci/mesos-plugin.
