Jan 19, 2018
Justin Lee
D2iQ
6 min read
The dynamic nature of containers and microservices has introduced new security challenges. Traditionally app owners had complete control over where applications resided and used secure file systems on hosts managed by access control lists to control access to apps. However, due to their dynamic nature, containers and microservices are not pinned to a specific host. This presents challenges in securing them because as they elastically scale up and down you don't know where they are or how many you have. Centralized secrets management tools such as Hashicorp Vault are gaining popularity because they provide a common location to store and retrieve the secrets needed to secure containers and microservices.
Mesosphere DC/OS integrates this ability by default and combines it with strong access control. The result is that operators can securely provide sensitive information (items such as username/passwords, configuration files, and certificates) to applications and services. This technology is integral to running a secure, production DC/OS cluster.
Many enterprises are required to perform full configuration backups on a regular basis, either to a disaster recovery (DR) site or to some long-term storage location. We've developed a simple command line tool to achieve this goal which can be used as is, or as a basis for your own project. Please note this is an open source project which is not supported or warrantied by Mesosphere. For issues or feature requests, please raise them directly on the Github repository.
Backup and Restore Background
All secrets are available through the DC/OS Secrets API . The dcos-secrets-backup tool obtains a list of all the secrets, retrieves them all, encrypts them, and stores them in a portable .tar file. It can also be used to restore a full set of secrets from the same .tar file to either the same or a different cluster.
The latest version of the dcos-secrets-backup tool and all documentation are available here: https://github.com/dcos-labs/professional-services/tree/master/tools/dcos-secrets-backup (the code is available in the accompanying Github repository).
Using the DC/OS Secrets Backup Tool
Step 1. Obtain the tool (Linux or OSX binaries provided):
bash
# Linux:
curl -LO https://github.com/dcos-labs/professional-services/raw/master/tools/dcos-secrets-backup/binaries/dcos-secrets-backup-linux
# macOS:
curl -LO https://github.com/dcos-labs/professional-services/raw/master/tools/dcos-secrets-backup/binaries/dcos-secrets-backup-darwin
Step 2. Set proper permissions (and rename it for convenience):
bash
mv dcos-secrets-backup-* dcos-secrets-backup
chmod +x dcos-secrets-backup
Step 3. Back up the secrets from your cluster:
bash
$ ./dcos-secrets-backup \
--hostname 34.216.178.11 \
--username bootstrapuser \
--password deleteme \
--cipherkey TvP5zCEPtRBuvEr2enFAM7cRzJxcKcnT \
--destfile /tmp/backup/secrets-2018-01-10.tar \
backup
Logging into cluster [https://34.216.178.11]
Getting secret 'test'
Getting secret 'edgelb-secret'
Getting secret 'dev/test'
Writing to tar at /tmp/backup/secrets-2018-01-10.tar
Notes:
- Hostname should be the hostname of your master (or master load balancer)
- Username and Password should be local users in the cluster with permissions to read the secrets you want to back up (in this case, the `bootstrapuser` has `dcos:superuser` permissions)
- Cipherkey is optional (it has a default of `ThisIsAMagicKeyString12345667890`, and should be a string that is a multiple of 32 characters long (this limitation may be removed in the future)
- Destfile indicates where you want the tar file to be placed. The directory must be pre-existing (this may be changed in the future)
Step 4. Restore the secrets to a different cluster:
bash
$ ./dcos-secrets-backup \
--hostname 54.214.120.255 \
--username bootstrapuser \
--password deleteme \
--cipherkey TvP5zCEPtRBuvEr2enFAM7cRzJxcKcnT \
--sourcefile /tmp/backup/secrets-2018-01-10.tar \
restore
Logging into cluster [https://54.214.120.255]
Queueing secret [dev/test] ...
Queueing secret [edgelb-secret] ...
Queueing secret [test] ...
Secret [dev/test] successfully updated.
Secret [edgelb-secret] successfully updated.
Secret [test] successfully updated.
Notes:
- Hostname should be the hostname of your master (or master load balancer)
- Username and Password should be local users in the cluster with permissions to write the secrets you want to back up (in this case, the `bootstrapuser` has `dcos:superuser` permissions)
- Cipherkey is optional (it has a default of `ThisIsAMagicKeyString12345667890`, and should be a string that is a multiple of 32 characters long (this limitation may be removed in the future)
- Sourcefile which tarfile to be used. The directory must be pre-existing (this may be changed in the future)
- This will create secrets that do not exist
- This will *overwrite* secrets that already exist.
