Apr 02, 2015
Sunil Shah
D2iQ
12 min read
Datacenter automation is one of the strongest features offered by the Mesosphere technology. This post is a follow on from our recent presentation at the Bay Area Infracoders meetup where we demonstrated how an organization can use Mesosphere to easily construct a simple continuous deployment pipeline from source code repository to your datacenter.
Goal
Continuous deployment is the holy grail of datacenter automation. In this post, we demonstrate a simplified example of how we use Mesos, Marathon, TeamCity, and Docker internally to deploy applications automatically to our internal staging environment. The team is automatically notified of new deployments by using Slack.
The following flow chart shows the high level sequence of actions. A check-in to GitHub triggers a build of a Docker image in TeamCity. On a successful build, a special TeamCity Deploy build is triggered, which uses Marathon's REST API to trigger a new deployment.
Implementation
Our example project is a reveal.js presentation, a developer friendly HTML + JavaScript presentation format that Mesosphere engineers use for talks. We check these into a GitHub repository at github.com/mesosphere/presentations.
Project Setup
For this example, we are using the presentation located at ba-infracoders-2015:
The app folder contains our presentation along with a Dockerfile and marathon.json which describe how to build and deploy our presentation.
The simple Dockerfile reuses the nginx base container which provides the nginx webserver. Finally, we ADD the presentation directory to the container to be served by nginx.
FROM nginx
MAINTAINER Mesosphere support@mesosphere.io
EXPOSE 80
ADD app/ /usr/share/nginx/html
The simple Marathon app definition in marathon.json specifies our mesosphere/cd-demo-app DockerHub repository as the source of our Docker image. Note the $tag placeholder, which is required by TeamCity to deploy the latest built tag of our presentation.
We also specify health checks in this definition, which allows Marathon to gauge when the newly deployed container is up and running correctly. The health checks use HTTP requests to a defined endpoint. You can read more about how Marathon health checks work here.
{"id": "/mesosphere/cd-demo-app",
"instances": 1,
"cpus": 1,
"mem": 512,
"container": {"type": "DOCKER",
"docker": {"image": "mesosphere/cd-demo-app:$tag",
"network": "BRIDGE",
"portMappings": [
{"servicePort": 28080,
"containerPort": 80,
"hostPort": 0,
"protocol": "tcp"
}
]
}
},
"healthChecks": [
{"gracePeriodSeconds": 120,
"intervalSeconds": 30,
"maxConsecutiveFailures": 3,
"path": "/",
"portIndex": 0,
"protocol": "HTTP",
"timeoutSeconds": 5
}
],
"constraints": [
[ "hostname", "GROUP_BY" ]
]
}
TeamCity Setup
In TeamCity, we have a build that builds and pushes the Docker image for our presentation and another build that deploys the most recently built image to Marathon.
Tag Scheme
Before setting up the build you should develop a scheme to use for generating the Docker image tags.
Here's the scheme we're going to use for our demo: %env.BUILD_NUMBER%-%teamcity.build.branch%.%env.BUILD_VCS_NUMBER%.
The components of the tag are as follows:
- %env.BUILD_NUMBER%: The build number so that we can always track down the TeamCity build that created the Docker Image
- %teamcity.build.branch%: The branch of the Git repo that the Docker Image was built from
- %env.BUILD_VCS_NUMBER%: The Git SHA of the code that was added to the Docker Image
This scheme ensures that we have a new tag for every build, meaning that we are creating immutable containers. Also, the scheme ensures that we have visibility into the branch that the code came from and the Git SHA of the commit.
Docker Image Build
The first build in TeamCity has these steps:
- docker login: log in to our DockerHub account
- docker build: build the specified Dockerfile with a new tag
- docker push: push the built image to DockerHub
- Report tag into new marathon.json: use jq to write the tag into the Marathon app definition
The first three steps are fairly standard, but the fourth step is the magic that makes this work. Our script looks like this:
mkdir -p target
echo %TAG% > target/docker-tag
cat marathon.json | \
jq '.container.docker.image |= "%DOCKER_IMAGE%:%TAG%"' > target/marathon.json
After the build completes you can view the generated artifacts on the Artifacts tab for the specific build:
Deploy
After the first build completes successfully and there are new artifacts, TeamCity automatically triggers a second build where we do two things:
- PUT to Marathon's REST API
- Send a message to the team on Slack
Marathon's REST API makes it easy to kick off new deployments of an application. Using HTTP PUT with the newly generated marathon.json artifact creates a new deployment. You can read more about Marathon deployments here. The deployment starts a new instance of the application and stops the old instance only when the new application is deemed "healthy".
APP_ID=$(cat %MARATHON_JSON_FILE% | jq -r '.id')
http --check-status PUT \
%SCHEME%://%MARATHON_HOST%:%MARATHON_PORT%/v2/%MARATHON_JSON_TYPE%/$APP_ID < %MARATHON_JSON_FILE%
Sending a message on Slack is easy because of their accessible API. The process involves pushing a JSON encoded message to a pre-configured Slack webhook URL (see Slack's documentation for more details).
echo '{"username" :"%SLACK_USERNAME%",
"channel" :"%SLACK_CHANNEL%",
"text" :"%SLACK_MESSAGE%",
"icon_emoji" :"%SLACK_EMOJI%",
"mrkdwn" :%SLACK_MARKDOWN%
}' | http --print=HhBb --json POST %SLACK_WEBHOOK_URL%
Of course, by parameterizing all of these values, it is easy to create a TeamCity template and extend this to other services.
NamevalueMARATHON_HOST |
MARATHON_JSON_FILE | marathon.json
MARATHON_JSON_TYPE | apps
MARATHON_PORT | 8080
SCHEME | http
SLACK_CHANNEL | #demo
SLACK_EMOJI | :teamcity:
SLACK_MARKDOWN | true
SLACK_MESSAGE | Heads Up! %teamcity.build.triggeredBy% just deployed <https://teamcity.mesosphere.io/viewLog.html?buildId=%teamcity.build.id%&tab=buildResultsDiv&buildTypeId=%system.teamcity.buildType.id%|%system.teamcity.projectName% :: %system.teamcity.buildConfName% #%build.counter%>
SLACK_USERNAME | TeamCity
SLACK_WEBHOOK_URL | https://hooks.slack.com/services/
MARATHON_JSON_FILE | marathon.json
MARATHON_JSON_TYPE | apps
MARATHON_PORT | 8080
SCHEME | http
SLACK_CHANNEL | #demo
SLACK_EMOJI | :teamcity:
SLACK_MARKDOWN | true
SLACK_MESSAGE | Heads Up! %teamcity.build.triggeredBy% just deployed <https://teamcity.mesosphere.io/viewLog.html?buildId=%teamcity.build.id%&tab=buildResultsDiv&buildTypeId=%system.teamcity.buildType.id%|%system.teamcity.projectName% :: %system.teamcity.buildConfName% #%build.counter%>
SLACK_USERNAME | TeamCity
SLACK_WEBHOOK_URL | https://hooks.slack.com/services/
Results
Each time a successful Docker image build completes, a deploy build is triggered, the marathon.json is PUT to Marathon, and the app begins rolling out across your cluster. A notification that an application rollout is happening is sent to the team.
Artifact marathon.json
Here you can see the generated marathon.json that is sent to Marathon.
{"id": "/mesosphere/cd-demo-app",
"instances": 1,
"cpus": 1,
"mem": 512,
"container": {"type": "DOCKER",
"docker": {"image": "mesosphere/cd-demo-app:34-master.3fcfaa664108e75120c65c26e69fb57b8fefce3d",
"network": "BRIDGE",
"portMappings": [
{"servicePort": 28080,
"containerPort": 80,
"hostPort": 0,
"protocol": "tcp"
}
]
}
},
"healthChecks": [
{"gracePeriodSeconds": 120,
"intervalSeconds": 30,
"maxConsecutiveFailures": 3,
"path": "/",
"portIndex": 0,
"protocol": "HTTP",
"timeoutSeconds": 5
}
],
"constraints": [
[ "hostname", "GROUP_BY" ]
]
}
Marathon Rollout
The new version of the application is scaled up, the new task is shown in the Staged state here:
After Marathon considers the new instance healthy, it cleans up old instances of the application:
Slack Message
A notification is then sent to the team indicating that a deploy has been triggered:
Key Takeaways
- Set up an automatic build for your project and generate a new Docker tag for each build.
- Generate marathon.json as a build artifact.
- Configure TeamCity to collect the generated marathon.json.
- Configure firewalls to allow TeamCity build agents to communicate with Marathon.
- Create a TeamCity build to PUT marathon.json to Marathon.
- Notify your team of the deployment.
Summary
This post shows how to easily set up automated builds that deploy Dockerized applications automatically to a Mesosphere cluster. We use this same procedure in many of our projects to make life easier for our engineers. Give continuous deployment on Mesosphere a try today!
