Cloud News‎ > ‎

Using CloudFormation, Packr, etc. for Immutable Infrastructure to build DC/OS and deploy it to Amazon Web Services

posted Nov 4, 2016, 2:26 PM by Rick Hightower   [ updated Nov 4, 2016, 4:55 PM ]

We setup a dev env for DC/OS in AWS (subnets, multi-az, ags groups, ami images, etc), tagged everything as dcos-dev, and then used Cloudformer to generate a starter AWS CloudFormation script. Cloudformer allows you to reverse engineer your AWS environment into cloudformation scripts. We then modified what cloudformer produced (to make it work as cloudformer just gets you about 90% of the way there), and then we added mappings, parameters and outputs to our cloudformation script.

Included are the cloudformation and packer scripts. Hope it helps you get setup. Feedback is welcome.

We in this case is our client, DC/OS support, Amazon support, and Mammatus. We did this instead of using the canned Amazon support because we needed to run masters, and agents in multiple AZs per region.

Production servers are setup, and integration servers are also setup. It is all immutable infrastructure (for the most part). We have scripts that build a DC/OS cluster from nothing to the full network, security, ELBauto-scale groupsfor public agents & private agents, master instances, user-data etc. stack using AWS cloud formation.

The two parts that are not yet automated are the enterprise marathon-lb setup (you have to fire up a micro into bastion security group, install DCOS cli and then run one script which is checked into git), and the CNAME/DNS bits which are controlled by IT. We might install an NS record to delegate to AWS Route53 and then control virtual host setup with Route53 aliases, which means that we can automate via cloud formation. Then we might add a Lambda/SNS/event to our cloud formation to handle the marathon-lb install when the cloud formation is done running.

All the AMI image creation is automated as well. The cloud formations point to AMI images which we create using Packr that have all of the advance prequisites for DC/OSPackr has excellent support for Amazon AMIs as well as Docker, and just about every other image container that exists. We choose CentOS over Amazon Linux so we are less locked in and could move images to other clouds or containers more readily.

The cloudformation script has a drop down cloudformation parameter which picks which env (prod, int, dev) and it sets up all of the CIDR addresses etc. via cloudformation mappings.

The DC/OS advanced install tar files and config were uploaded to S3. We use IAM roles, and s3 bucket policies to lock down the S3 bucket where the DC/OS advanced install params are. (We run sudo bash dcos_generate_config.ee.sh and aws s3 sync genconf/serve/ s3://somebucket-dcos-install-bootstrap/prod to upload the install packages from the DC/OS advanced install.)

The cloudformation script supports multiple AWS regions and multiple AZs. To do this support we use cloudformation mappingsparameters and outputs.

We plan on improving the cloudformation scripts and packer script as follows:


CloudFormation diagram:



{
"variables": {
"aws_access_key": "",
"aws_secret_key": "",
"aws_region": "us-east-1",
"aws_ami_image": "ami-6d1c2007",
"aws_instance_type": "m4.large"
},
"builders": [
{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "{{user `aws_region`}}",
"source_ami": "{{user `aws_ami_image`}}",
"instance_type": "{{user `aws_instance_type`}}",
"ssh_username": "centos",
"ami_name": "base-centos-7-dcos-{{timestamp}}",
"tags": {
"Name": "ami-centos7-dcos-v1",
"OS_Version": "LinuxCentOs7",
"Release": "7",
"Description": "Base CentOs7 image with prerequisites for DC/OS"
},
"user_data_file": "files/user-data.txt"
}
],
"provisioners": [
{
"type": "file",
"source": "files/overlay.conf",
"destination": "/home/centos/install/overlay.conf"
},
{
"type": "shell",
"inline": [
"sudo mkdir -p /etc/modules-load.d",
"sudo mv /home/centos/install/overlay.conf /etc/modules-load.d/overlay.conf",
"sudo reboot"
]
},
{
"type": "file",
"source": "files/docker.repo",
"destination": "/home/centos/install/docker.repo"
},
{
"type": "file",
"source": "files/override.conf",
"destination": "/home/centos/install/override.conf"
},
{
"type": "shell",
"inline": [
"echo installing Docker -----------------------------------",
"sudo mkdir -p /etc/systemd/system/docker.service.d",
"sudo mv /home/centos/install/override.conf /etc/systemd/system/docker.service.d/override.conf",
"sudo mv /home/centos/install/docker.repo /etc/yum.repos.d/docker.repo",
"sudo yum install -y docker-engine-1.11.2",
"sudo systemctl start docker",
"sudo systemctl enable docker",
"sudo docker ps",
"echo DONE installing Docker -----------------------------",
"sudo yum install -y tar xz unzip curl ipset nano tree",
"sudo sed -i s/SELINUX=enforcing/SELINUX=permissive/g /etc/selinux/config",
"sudo groupadd nogroup",
"sudo reboot"
]
},
{
"type": "shell",
"inline": [
"echo installing aws cli -------------------------------",
"mkdir /tmp/awscli",
"cd /tmp/awscli",
"curl https://s3.amazonaws.com/aws-cli/awscli-bundle.zip -o awscli-bundle.zip",
"unzip awscli-bundle.zip",
"sudo ./awscli-bundle/install -i /usr/lib/aws -b /usr/bin/aws",
"echo DONE installing aws cli -------------------------------"
]
},
{
"type": "shell",
"inline": [
"echo DONE installing packages for CentOS7 DC/OS"
]
}
]
}


#!/bin/bash
sed -i -e '/Defaults requiretty/{ s/.*/# Defaults requiretty/ }' /etc/sudoers
sed -i -e '/%wheel\tALL=(ALL)\tALL/{ s/.*/%wheel\tALL=(ALL)\tNOPASSWD:\tALL/ }' /etc/sudoers
mkdir /etc/modules-load.d/
mkdir -p /home/centos/install
chown -R centos /home/centos







Comments