Cloud News

Data safety with EBS using Snapshots - Backup/Recovery

posted Nov 11, 2016, 1:16 AM by Rick Hightower   [ updated Nov 11, 2016, 10:04 AM ]

Data safety with EBS - Backup/Recovery (Snapshots)

Amazon EBS allows you to easily backup data. You do this by taking snapshots. Snapshots are point-in-time backups. Data written to an EBS volume can be periodically used to create a snapshot. Snapshots provide incremental backups of your data. Snapshots just saves the blocks that have changed. Only changed blocks since the last snapshot are saved in the new snapshot.

Even though snapshots are saved incrementally, only the last snapshot is needed in order to restore the volume. You can delete older snapshot, and still use the latest snapshot.

Taking Snapshots

Snapshots are done with:

  • AWS Management Console
  • Scheduled snapshots
  • AWS API
  • AWS CLI

EBS snapshots are backed by S3 in AWS-controlled storage. You can see these snapshots your account’s Amazon S3 buckets. It is hidden from you but backed by S3. You use the snapshot tools to manage snapshots not S3.

Snapshots are stored per region. You use snapshots to create new EBS volumes. Snapshots can be copied to other regions.

Using snapshots

Snapshots are used to create new EBS volumes. The volume is created and the data is transferred lazily into the EBS volume. Data accessed before the transfer is restored on request. If you need to increase the size of a volume, you create a snapshot and then recreate a larger volume from the snapshot. Then replace the original volume with the new volume from the snapshot.

Tag snapshots to help manage them later. Describing the original volume of the snapshot with the device name (/dev/sdd).

The AWS console can be used to take snapshots or the command line.

Using command line to create a snapshot of a volume

aws ec2 create-snapshot --volume-id vol-1234567890abcdef0 --description "Hadoop RAID 0 volume 5."

Restoring volumes from snapshots

Amazon EBS volumes persist beyond the EC2 instance lifecycle. Thus you can recover data of an instance that fails. Before the instance is terminated, the volume should be detached. Then the volume can attached as a data volume to another instance and then the data can be recovered.

Backing up root devices

To create a snapshot from a root devices EBS volume, should unmount the volume before taking the snapshot so the OS or application that have outstanding / cached blocks can flush them.

To unmount the volume in Linux, use the following command:

umount -d /dev/sdc

Best practices for Backup and Recovery

There are best practices for dealing with EBS. Back up your EC2 instance regularly using Amazon EBS snapshots. Test the process of recovering your instances from snapshots if the Amazon EBS volumes fail. Use separate volumes for the operating system versus your data. Make sure that the data persists after instance termination. Don't use instance store for database storage, unless you are using replication.

AWS Elastic Block Store (EBS) - Getting the most bang for your buck

posted Nov 9, 2016, 11:31 PM by Rick Hightower   [ updated Nov 11, 2016, 9:57 AM ]

Getting the most bang for your buck with AWS Elastic Block Store (EBS)

Amazon Elastic Block Store

Amazon Web Services (AWS) provides Amazon Elastic Block Store (Amazon EBS) for EC2 instance storage. EBS is the virtual hard drives and SSDs for your servers running in the cloud. Amazon EBS volumes are automatically replicated, and it is easy to take snapshots of volumes to back them up in a known state. The replication happens within an availability zone (AZ).

AWS EBS has lots of advantages like reliability, snapshotting, and resizing.

AWS provides four volume types. It provides two types of Hard Disk Drives (HDD), and two types of SSDs. Volumes differ in price and performance. An EC2 instance can have many volumes attached to it, just like a server can have many drives. A volume can only be attached to one EC2 instance at a time. If you wanted to share files between EC2 instances than you would use Amazon Elastic File System or S3.

EBS Volumes Types

There are many types of volumes. Different types have different performance characteristics. The trick is to pick the most cost-efficient for the workload of your service.

Magnetic Volumes - Hard Disk Drives (HDD)

Magnetic volumes have the lowest performance for random access. However, they have the least cost per gigabyte. But, they have the highest access for throughput (500 MB/s) for sequential access. Magnetic volumes average 100 IOPS, but can burst to hundreds of IOPS.

IOPS are Input/output operations per second (pronounced eye-ops). IOPS are used to characterize storage devices.

Services like Kafka which writes to a transaction log in long streams, and databases which use log structured storageor an approximate of that using some sort of log structured merge tree (examples LevelDBRocksDBCassandra) might do well with HDD EBS - Magnetic volumes. Application that might employ streaming, or less operations per second but larger writes could actually benefit from using HDDs throughput performance.

In general, magnetic volumes do best with sequential operations like:

  • streaming workloads which require cost effective, fast, consistent I/O
  • big data
  • data warehouses
  • log processing
  • Databases that employ log structured merge tree

Magnetic volumes can't be used as a boot volume.

There are two types of HDD - Magnetic Volumes:

  • st1 - Throughput Optimized HDD
  • sc1 - Cold HDD and most cost effective

General-Purpose SSD (gp2)

General-purpose SSD (gp2) volumes are cost effective, and useful for many workloads. It is the minivan of EBS. Not sexy but works for a lot of applications, and is common.

Performance of gp2 is three IOPS per gigabyte provisioned, but capped at 10,000 IOPS. The sizes range from 1 GB to 16 TB. Databases that use some form of BTrees (MongoDB, MySQL, etc.) can benefit from using SSD. But gp2 would be more geared to a lower volume database or one that has peak load times but long periods at rest where IOPS credits can accumulate.

Under 1 TB these volumes burst to 3,000 IOPS for extended periods of time. For example, if you have a 250 GB volume you can expect a baseline of 750 IOPS. When those 750 IOPS are not used, they are accumulated as IOPS credits. Under heavy traffic, those IOPS credits will be used and this is how you can burst up to 3,000 IOPS. IOPS credits is like a savings account. You use this savings when you get hit hard by a user tornado. But as you are using it, the bank account is being withdrawn from.

  • Development and test environments
  • A server than does periodic batch or cron jobs
  • Recommended for most workloads
  • Can be used for boot volumes
  • Low-latency interactive apps
  • Medium-sized databases

Provisioned IOPS SSD (io1)

Provisioned IOPS SSD volumes are for I/O-intensive workloads. These volumes are for random access I/O throughput. They are the most expensive Amazon EBS volume type per gigabyte. And, they provide the highest performance of random access of any Amazon EBS volume. With this volume type you can pick the number of IOPs (pay to play). The IOPs can be up to 20,000. These volumes are great for high-volume databases or just databases that need a constant level of performance. High volume databases that use some form of BTrees (MongoDB, MySQL, etc.) can benefit from using this SSD volume. The io1 IOPS can be ramped up.

Provisioned IOPS SSD volumes are more predictable (don't have to store up IOPS like gp2), and for application with higher performance needs like:

  • Mission critical business applications that require sustained IOPS performance
  • Databases with large, high-volume workloads

Overcoming the performance problems by using Provisioned IOPS is expensive.

Some companies have employed RAID-0 striping using a 4-way stripe and used EnhanceIO to effectively increased throughput by over 50% with no more additional expense.

EnhanceIO driver is based on EnhanceIO SSD caching software product developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source volumes throughout this document). --[EnhanceIO]https://github.com/stec-inc/EnhanceIO

RAID-0 can be employed to increase size constraints of EBS and to increase throughput.

RAID

AWS allows configuring EBS volumes into RAID. EBS volumes can be setup in RAID 0 (stripe multiple volumes together) configuration for even more throughput and size. RAID 0 volume capacity is the sum of the capacities of the disks mounted. RAID 0 does not provide added redundancy for disk failures. Striping distributes the data of files among all disks mounted. In general, this speeds read and write operations almost by a factor number of disks mounted into a RAID 0 set. Increased throughput (and size for EBS) is the big benefit of RAID 0.

Striping a group of IOPS for RAID 0

#!/bin/bash
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
echo BEGIN

#Raid if mutliple devices are mapped
if [ -e '/dev/xvdb' ] && [ -e '/dev/xvdc' ] ; then
 yum -y install mdadm
 umount /mnt/
 # create the array and format the drive
 yes | mdadm --create /dev/md0 --level=0 --name=RAID0 --raid-devices=2 /dev/xvdb /dev/xvdc
 mkfs.ext4 -L RAID0 /dev/md0
 mount LABEL=RAID0 /mnt
 # update fstab
 sed -i '/\/mnt/d' /etc/fstab
 echo "LABEL=RAID0       /mnt/   ext4    defaults,nofail,noatime,barrier=0        0       2" >> /etc/fstab
 mount -a
fi

Performance testing and monitoring

The best way to make an educated guess to pick the right EBS is to know your tool. If you are deploying Kafka or Cassandra or MongoDB then you must understand how to configure the tool and EBS to get the most bang for the buck. When in doubt, test.

You can make educated guesses about which EBS will fit your application or service the best. However using Amazon CloudWatch and watching the IOPs and IO throughput while load testing or watching production workloads could be the quickest way to to pick the best EBS volume type and get the most bang for your buck. This can also help you decide whether or not to use RAID 0, HDDs (st1 or sc1), provisioned IOPS SSD (io1), SSD general purpose (gp2) or not. There is no point in overpaying, and you do not want a laggy service or application that do not meet their SLAs.


Conclusion:


Amazon has HDDs which have fewer IOPs but are much better at streaming data. There is a good chance the HDDs would be better at Cassandra and Kafka due to the way Cassandra uses its version of log structured storage and updates and the way that Kafka does distributed logs. You basically get twice the throughput when streaming data over HDDs then you get when using provisioned IOPs. Since most NoSQL solutions use some form of log structured storage, then most would actually benefit from mounting a second EBS volume that is magnetic. The exception would be MongoDB, but you could still mount an HDD for MongoDBs transaction log whilst leaving the BTree indexed document storage on provisioned IO. 


Using AWS CloudWatch Metrics to report OS metrics of an EC2 instance

posted Nov 8, 2016, 2:44 PM by Rick Hightower   [ updated Nov 8, 2016, 2:46 PM ]

If you use Amazon CloudWatch to enable EC2 instance metrics, you may have noticed that the instance metrics it reports leaves out useful OS metrics like "how much disk space do I have left". This of course is a great metric to know about so you can learn things like my XYX cluster is running out of disk space and in about four days, there will be hell to pay. You know ops and DevOps sorts of things.

It is a shame because although CloudWatch is missing these OS metrics, these are often times the most useful metrics when you want to setup CloudWatch Alarms and trigger CloudWatch events.

Well Amazon does provide an API to CloudWatch Metrics so you can report your own application metrics or in our case OS metrics. In fact they even provide a working example that reports critical Linux OS metrics to CloudWatch Metrics.



Amazon CloudWatch Metrics DashBoard

AWS CloudWatch DashBoard

Before you can use this tool, you need to setup an AMI Role that the instance will use, and the create an IAM policyfile that is associated with that IAM Role.

The policy file looks like this:

AllowMetrics IAM Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:GetMetricStatistics",
                "cloudwatch:ListMetrics",
                "cloudwatch:PutMetricData",
                "ec2:DescribeTags"
            ],
            "Resource": "*"
        }
    ]
}

In this example, we will install the AWS metrics monitor as a cron job the executes every five minutes.

install_cron.sh used to Install mon-put-instance-data.pl as a cron job

echo "Installing metrics watcher"
sudo yum install perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https perl-Digest-SHA -y
sudo yum install zip -y
mkdir /tmp/aws-mon-install
cd /tmp/aws-mon-install
curl http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip -O
unzip CloudWatchMonitoringScripts-1.2.1.zip
sudo mv aws-scripts-mon/ /usr/local/lib/

echo "Installed metrics watcher to `ls /usr/local/lib/aws-scripts-mon`"
crontab /home/centos/install/cronjobs

rm -rf /tmp/aws-mon-install

The cronjob file looks like this:

~/install/cronjobs

*/5 * * * *  /usr/local/lib/aws-scripts-mon/mon-put-instance-data.pl  --mem-util  --mem-used \
--mem-avail --disk-path=/ --disk-space-util --disk-space-used --disk-space-avail \
--disk-space-units=megabytes --memory-units=megabytes --auto-scaling --from-cron


Of course, we don't actually install these things by hand but use packer to build a base AMI image (in this case a base image for DC/OS).

We tell mon-put-instance-data.pl to track memory and disk usage. It will do this per instance and per Amazon auto-scale group. This allows you to see the metrics as a whole via the auto-scale group.

The packer file to build this image is as follows:

Packer to build EC2 image that has mon-put-instance-data to monitor Linux OS metrics

{
  "variables": {
    "aws_access_key": "",
    "aws_secret_key": "",
    "aws_region": "us-east-1",
    "aws_ami_image": "ami-6d1c2007",
    "aws_instance_type": "m4.large"
  },
  "builders": [
    {
      "type": "amazon-ebs",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "region": "{{user `aws_region`}}",
      "source_ami": "{{user `aws_ami_image`}}",
      "instance_type": "{{user `aws_instance_type`}}",
      "ssh_username": "centos",
      "ami_name": "base-centos-7-dcos-{{timestamp}}",
      "tags": {
        "Name": "ami-centos7-dcos-v1",
        "OS_Version": "LinuxCentOs7",
        "Release": "7",
        "Description": "Base CentOs7 image with prerequisites for DC/OS"
      },
      "user_data_file": "files/user-data.txt"
    }
  ],
  "provisioners": [
    {
      "type": "file",
      "source": "files/overlay.conf",
      "destination": "/home/centos/install/overlay.conf"
    },
    {
      "type": "shell",
      "inline": [
        "sudo mkdir -p /etc/modules-load.d",
        "sudo mv /home/centos/install/overlay.conf /etc/modules-load.d/overlay.conf",
        "sudo reboot"
      ]
    },
    {
      "type": "file",
      "source": "files/docker.repo",
      "destination": "/home/centos/install/docker.repo"
    },
    {
      "type": "file",
      "source": "files/override.conf",
      "destination": "/home/centos/install/override.conf"
    },
    {
      "type": "shell",
      "inline": [
        "echo installing Docker -----------------------------------",
        "sudo mkdir -p /etc/systemd/system/docker.service.d",
        "sudo mv /home/centos/install/override.conf /etc/systemd/system/docker.service.d/override.conf",
        "sudo mv /home/centos/install/docker.repo /etc/yum.repos.d/docker.repo",
        "sudo yum install -y docker-engine-1.11.2",
        "sudo systemctl start docker",
        "sudo systemctl enable docker",
        "sudo docker ps",
        "echo DONE installing Docker -----------------------------",
        "sudo yum install -y tar xz unzip curl ipset nano tree",
        "sudo sed -i s/SELINUX=enforcing/SELINUX=permissive/g /etc/selinux/config",
        "sudo groupadd nogroup",
        "sudo reboot"
      ]
    },
    {
      "type": "shell",
      "inline": [
        "echo installing aws cli -------------------------------",
        "mkdir /tmp/awscli",
        "cd /tmp/awscli",
        "curl https://s3.amazonaws.com/aws-cli/awscli-bundle.zip -o awscli-bundle.zip",
        "unzip awscli-bundle.zip",
        "sudo ./awscli-bundle/install -i /usr/lib/aws -b /usr/bin/aws",
        "sudo rm -rf /tmp/awscli",
        "echo DONE installing aws cli -------------------------------"
      ]
    },
    {
      "type": "file",
      "source": "files/journald-cloudwatch.conf",
      "destination": "/home/centos/install/journald-cloudwatch.conf"
    },
    {
      "type": "file",
      "source": "files/journald.unit",
      "destination": "/home/centos/install/journald.unit"
    },
    {
      "type": "shell",
      "inline": [
        "echo install log agent -------------------------------",
        "mkdir /tmp/logagent",
        "cd /tmp/logagent",
        "curl -OL  https://github.com/saymedia/journald-cloudwatch-logs/releases/download/v0.0.1/journald-cloudwatch-logs-linux.zip",
        "unzip journald-cloudwatch-logs-linux.zip",
        "sudo mv journald-cloudwatch-logs/journald-cloudwatch-logs /usr/bin",
        "sudo mkdir -p /var/lib/journald-cloudwatch-logs/",
        "sudo mv /home/centos/install/journald-cloudwatch.conf /etc/",
        "sudo mv /home/centos/install/journald.unit /etc/systemd/system/journald-cloudwatch.service",
        "sudo chmod 664 /etc/systemd/system/journald-cloudwatch.service",
        "sudo chown -R centos /var/lib/journald-cloudwatch-logs/",
        "sudo systemctl enable journald-cloudwatch.service",
        "sudo rm -rf /tmp/llogagent",
        "echo DONE installing log agent -------------------------------"
      ]
    },
    {
      "type": "file",
      "source": "files/install_cron.sh",
      "destination": "/home/centos/install/install_cron.sh"
    },
    {
      "type": "file",
      "source": "files/cronjobs",
      "destination": "/home/centos/install/cronjobs"
    },
    {
      "type": "shell",
      "inline": [
        "echo install aws metrics cron -------------------------------",
        "/usr/bin/bash /home/centos/install/install_cron.sh",
        "echo done installing aws metrics cron -------------------------------"
      ]
    },
    {
      "type": "shell",
      "inline": [
        "echo DONE installing packages for CentOS7 DC/OS"
      ]
    }
  ]
}

To learn more about this packer build and immutable-infrastructure in AWS see Using CloudFormation and Packer to build immutable infrastructure in AWS, and setting up an AWS log agent to send journald & DC/OS logs to Amazon CloudWatch Log Streams.

Setting up an AWS log agent to send journald & DC/OS logs to Amazon CloudWatch Log Service

posted Nov 7, 2016, 8:28 PM by Rick Hightower   [ updated Dec 5, 2016, 12:44 PM ]

In this example, we are using Centos7journalctl and systemctl so that we can monitor logs from DC/OSinstances (masters, agents and public agents). It is useful for anyone using systemdjournald in an AWS EC2enviroment that wants logging. The nice thing about Amazon CloudWatch is that it integrates well with Amazon EMRand Amazon Elasticsearch. (For more background on this subject see this article which covers using CloudFormation, Packr, etc. for Immutable Infrastructure to build DC/OS and deploy it to Amazon Web Services.)

We will install systemd-cloud-watch. We are going to setup a daemon into systemd that forwards logs to Amazon CloudWatch log streams.

This utility journald-cloudwatch-logs monitors the systemd journal, managed by journald, and writes journal entries into AWS Cloudwatch Logs. You can find out more about it at their website journald-cloudwatch-logs.

This program is an alternative to the AWS-provided logs agent which works only with sending text log files into AWS Cloudwatch. Conversely the journald-cloudwatch-logs utility reads directly from the systemd journal.

Step 1) Create an IAM role to start your DC/OS instances.

Step 2) Create an IAM policy as follows and associate it with your the DC/OS IAM role.

IAM policy AllowLogs

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:*",
                "arn:aws:logs:*:*:log-group:*:log-stream:*"
            ]
        }
    ]
}

Step 3) Download and Install the agent software and setup config files

Install agent software

mkdir /tmp/logagent
cd /tmp/logagent
curl -OL  https://github.com/advantageous/systemd-cloud-watch/releases/download/v0.1.0/systemd-cloud-watch_linux
sudo mv journald-cloudwatch-logs/systemd-cloud-watch_linux /usr/bin/journald-cloudwatch-logs
sudo mkdir -p /var/lib/journald-cloudwatch-logs/
sudo mv /home/centos/install/journald-cloudwatch.conf /etc/
sudo mv /home/centos/install/journald.unit /etc/systemd/system/journald-cloudwatch.service
sudo chmod 664 /etc/systemd/system/journald-cloudwatch.service
sudo chown -R centos /var/lib/journald-cloudwatch-logs/
sudo systemctl enable journald-cloudwatch.service

install/journald.unit

[Unit]
Description=journald-cloudwatch-logs
Wants=basic.target
After=basic.target network.target

[Service]
User=centos
Group=centos
ExecStart=/usr/bin/journald-cloudwatch-logs /etc/journald-cloudwatch.conf
KillMode=process
Restart=on-failure
RestartSec=42s


[Install]
WantedBy=getty.target

install/journald-cloudwatch.conf

log_group = "dcos-logstream"

log_priority = "WARNING"

We created packer file to automate the creation of our base AMI image for DC/OS.

Packer creation script

{
  "variables": {
    "aws_access_key": "",
    "aws_secret_key": "",
    "aws_region": "us-east-1",
    "aws_ami_image": "ami-6d1c2007",
    "aws_instance_type": "m4.large"
  },
  "builders": [
    {
      "type": "amazon-ebs",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "region": "{{user `aws_region`}}",
      "source_ami": "{{user `aws_ami_image`}}",
      "instance_type": "{{user `aws_instance_type`}}",
      "ssh_username": "centos",
      "ami_name": "base-centos-7-dcos-{{timestamp}}",
      "tags": {
        "Name": "ami-centos7-dcos-v1",
        "OS_Version": "LinuxCentOs7",
        "Release": "7",
        "Description": "Base CentOs7 image with prerequisites for DC/OS"
      },
      "user_data_file": "files/user-data.txt"
    }
  ],
  "provisioners": [
    {
      "type": "file",
      "source": "files/overlay.conf",
      "destination": "/home/centos/install/overlay.conf"
    },
    {
      "type": "shell",
      "inline": [
        "sudo mkdir -p /etc/modules-load.d",
        "sudo mv /home/centos/install/overlay.conf /etc/modules-load.d/overlay.conf",
        "sudo reboot"
      ]
    },
    {
      "type": "file",
      "source": "files/docker.repo",
      "destination": "/home/centos/install/docker.repo"
    },
    {
      "type": "file",
      "source": "files/override.conf",
      "destination": "/home/centos/install/override.conf"
    },
    {
      "type": "shell",
      "inline": [
        "echo installing Docker -----------------------------------",
        "sudo mkdir -p /etc/systemd/system/docker.service.d",
        "sudo mv /home/centos/install/override.conf /etc/systemd/system/docker.service.d/override.conf",
        "sudo mv /home/centos/install/docker.repo /etc/yum.repos.d/docker.repo",
        "sudo yum install -y docker-engine-1.11.2",
        "sudo systemctl start docker",
        "sudo systemctl enable docker",
        "sudo docker ps",
        "echo DONE installing Docker -----------------------------",
        "sudo yum install -y tar xz unzip curl ipset nano tree",
        "sudo sed -i s/SELINUX=enforcing/SELINUX=permissive/g /etc/selinux/config",
        "sudo groupadd nogroup",
        "sudo reboot"
      ]
    },
    {
      "type": "shell",
      "inline": [
        "echo installing aws cli -------------------------------",
        "mkdir /tmp/awscli",
        "cd /tmp/awscli",
        "curl https://s3.amazonaws.com/aws-cli/awscli-bundle.zip -o awscli-bundle.zip",
        "unzip awscli-bundle.zip",
        "sudo ./awscli-bundle/install -i /usr/lib/aws -b /usr/bin/aws",
        "echo DONE installing aws cli -------------------------------"
      ]
    },
    {
      "type": "file",
      "source": "files/journald-cloudwatch.conf",
      "destination": "/home/centos/install/journald-cloudwatch.conf"
    },
    {
      "type": "file",
      "source": "files/journald.unit",
      "destination": "/home/centos/install/journald.unit"
    },
    {
      "type": "shell",
      "inline": [
        "echo install log agent -------------------------------",
        "mkdir /tmp/logagent",
        "cd /tmp/logagent",
        "curl -OL  https://github.com/saymedia/journald-cloudwatch-logs/releases/download/v0.0.1/journald-cloudwatch-logs-linux.zip",
        "unzip journald-cloudwatch-logs-linux.zip",
        "sudo mv journald-cloudwatch-logs/journald-cloudwatch-logs /usr/bin",
        "sudo mkdir -p /var/lib/journald-cloudwatch-logs/",
        "sudo mv /home/centos/install/journald-cloudwatch.conf /etc/",
        "sudo mv /home/centos/install/journald.unit /etc/systemd/system/journald-cloudwatch.service",
        "sudo chmod 664 /etc/systemd/system/journald-cloudwatch.service",
        "sudo chown -R centos /var/lib/journald-cloudwatch-logs/",
        "sudo systemctl enable journald-cloudwatch.service",
        "echo DONE installing log agent -------------------------------"
      ]
    },
    {
      "type": "shell",
      "inline": [
        "echo DONE installing packages for CentOS7 DC/OS"
      ]
    }
  ]
}

Using CloudFormation, Packr, etc. for Immutable Infrastructure to build DC/OS and deploy it to Amazon Web Services

posted Nov 4, 2016, 2:26 PM by Rick Hightower   [ updated Nov 4, 2016, 4:55 PM ]

We setup a dev env for DC/OS in AWS (subnets, multi-az, ags groups, ami images, etc), tagged everything as dcos-dev, and then used Cloudformer to generate a starter AWS CloudFormation script. Cloudformer allows you to reverse engineer your AWS environment into cloudformation scripts. We then modified what cloudformer produced (to make it work as cloudformer just gets you about 90% of the way there), and then we added mappings, parameters and outputs to our cloudformation script.

Included are the cloudformation and packer scripts. Hope it helps you get setup. Feedback is welcome.

We in this case is our client, DC/OS support, Amazon support, and Mammatus. We did this instead of using the canned Amazon support because we needed to run masters, and agents in multiple AZs per region.

Production servers are setup, and integration servers are also setup. It is all immutable infrastructure (for the most part). We have scripts that build a DC/OS cluster from nothing to the full network, security, ELBauto-scale groupsfor public agents & private agents, master instances, user-data etc. stack using AWS cloud formation.

The two parts that are not yet automated are the enterprise marathon-lb setup (you have to fire up a micro into bastion security group, install DCOS cli and then run one script which is checked into git), and the CNAME/DNS bits which are controlled by IT. We might install an NS record to delegate to AWS Route53 and then control virtual host setup with Route53 aliases, which means that we can automate via cloud formation. Then we might add a Lambda/SNS/event to our cloud formation to handle the marathon-lb install when the cloud formation is done running.

All the AMI image creation is automated as well. The cloud formations point to AMI images which we create using Packr that have all of the advance prequisites for DC/OSPackr has excellent support for Amazon AMIs as well as Docker, and just about every other image container that exists. We choose CentOS over Amazon Linux so we are less locked in and could move images to other clouds or containers more readily.

The cloudformation script has a drop down cloudformation parameter which picks which env (prod, int, dev) and it sets up all of the CIDR addresses etc. via cloudformation mappings.

The DC/OS advanced install tar files and config were uploaded to S3. We use IAM roles, and s3 bucket policies to lock down the S3 bucket where the DC/OS advanced install params are. (We run sudo bash dcos_generate_config.ee.sh and aws s3 sync genconf/serve/ s3://somebucket-dcos-install-bootstrap/prod to upload the install packages from the DC/OS advanced install.)

The cloudformation script supports multiple AWS regions and multiple AZs. To do this support we use cloudformation mappingsparameters and outputs.

We plan on improving the cloudformation scripts and packer script as follows:


CloudFormation diagram:



{
"variables": {
"aws_access_key": "",
"aws_secret_key": "",
"aws_region": "us-east-1",
"aws_ami_image": "ami-6d1c2007",
"aws_instance_type": "m4.large"
},
"builders": [
{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "{{user `aws_region`}}",
"source_ami": "{{user `aws_ami_image`}}",
"instance_type": "{{user `aws_instance_type`}}",
"ssh_username": "centos",
"ami_name": "base-centos-7-dcos-{{timestamp}}",
"tags": {
"Name": "ami-centos7-dcos-v1",
"OS_Version": "LinuxCentOs7",
"Release": "7",
"Description": "Base CentOs7 image with prerequisites for DC/OS"
},
"user_data_file": "files/user-data.txt"
}
],
"provisioners": [
{
"type": "file",
"source": "files/overlay.conf",
"destination": "/home/centos/install/overlay.conf"
},
{
"type": "shell",
"inline": [
"sudo mkdir -p /etc/modules-load.d",
"sudo mv /home/centos/install/overlay.conf /etc/modules-load.d/overlay.conf",
"sudo reboot"
]
},
{
"type": "file",
"source": "files/docker.repo",
"destination": "/home/centos/install/docker.repo"
},
{
"type": "file",
"source": "files/override.conf",
"destination": "/home/centos/install/override.conf"
},
{
"type": "shell",
"inline": [
"echo installing Docker -----------------------------------",
"sudo mkdir -p /etc/systemd/system/docker.service.d",
"sudo mv /home/centos/install/override.conf /etc/systemd/system/docker.service.d/override.conf",
"sudo mv /home/centos/install/docker.repo /etc/yum.repos.d/docker.repo",
"sudo yum install -y docker-engine-1.11.2",
"sudo systemctl start docker",
"sudo systemctl enable docker",
"sudo docker ps",
"echo DONE installing Docker -----------------------------",
"sudo yum install -y tar xz unzip curl ipset nano tree",
"sudo sed -i s/SELINUX=enforcing/SELINUX=permissive/g /etc/selinux/config",
"sudo groupadd nogroup",
"sudo reboot"
]
},
{
"type": "shell",
"inline": [
"echo installing aws cli -------------------------------",
"mkdir /tmp/awscli",
"cd /tmp/awscli",
"curl https://s3.amazonaws.com/aws-cli/awscli-bundle.zip -o awscli-bundle.zip",
"unzip awscli-bundle.zip",
"sudo ./awscli-bundle/install -i /usr/lib/aws -b /usr/bin/aws",
"echo DONE installing aws cli -------------------------------"
]
},
{
"type": "shell",
"inline": [
"echo DONE installing packages for CentOS7 DC/OS"
]
}
]
}


#!/bin/bash
sed -i -e '/Defaults requiretty/{ s/.*/# Defaults requiretty/ }' /etc/sudoers
sed -i -e '/%wheel\tALL=(ALL)\tALL/{ s/.*/%wheel\tALL=(ALL)\tNOPASSWD:\tALL/ }' /etc/sudoers
mkdir /etc/modules-load.d/
mkdir -p /home/centos/install
chown -R centos /home/centos







1-5 of 5