Cloud News‎ > ‎

Using AWS CloudWatch Metrics to report OS metrics of an EC2 instance

posted Nov 8, 2016, 2:44 PM by Rick Hightower   [ updated Nov 8, 2016, 2:46 PM ]

If you use Amazon CloudWatch to enable EC2 instance metrics, you may have noticed that the instance metrics it reports leaves out useful OS metrics like "how much disk space do I have left". This of course is a great metric to know about so you can learn things like my XYX cluster is running out of disk space and in about four days, there will be hell to pay. You know ops and DevOps sorts of things.

It is a shame because although CloudWatch is missing these OS metrics, these are often times the most useful metrics when you want to setup CloudWatch Alarms and trigger CloudWatch events.

Well Amazon does provide an API to CloudWatch Metrics so you can report your own application metrics or in our case OS metrics. In fact they even provide a working example that reports critical Linux OS metrics to CloudWatch Metrics.



Amazon CloudWatch Metrics DashBoard

AWS CloudWatch DashBoard

Before you can use this tool, you need to setup an AMI Role that the instance will use, and the create an IAM policyfile that is associated with that IAM Role.

The policy file looks like this:

AllowMetrics IAM Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:GetMetricStatistics",
                "cloudwatch:ListMetrics",
                "cloudwatch:PutMetricData",
                "ec2:DescribeTags"
            ],
            "Resource": "*"
        }
    ]
}

In this example, we will install the AWS metrics monitor as a cron job the executes every five minutes.

install_cron.sh used to Install mon-put-instance-data.pl as a cron job

echo "Installing metrics watcher"
sudo yum install perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https perl-Digest-SHA -y
sudo yum install zip -y
mkdir /tmp/aws-mon-install
cd /tmp/aws-mon-install
curl http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip -O
unzip CloudWatchMonitoringScripts-1.2.1.zip
sudo mv aws-scripts-mon/ /usr/local/lib/

echo "Installed metrics watcher to `ls /usr/local/lib/aws-scripts-mon`"
crontab /home/centos/install/cronjobs

rm -rf /tmp/aws-mon-install

The cronjob file looks like this:

~/install/cronjobs

*/5 * * * *  /usr/local/lib/aws-scripts-mon/mon-put-instance-data.pl  --mem-util  --mem-used \
--mem-avail --disk-path=/ --disk-space-util --disk-space-used --disk-space-avail \
--disk-space-units=megabytes --memory-units=megabytes --auto-scaling --from-cron


Of course, we don't actually install these things by hand but use packer to build a base AMI image (in this case a base image for DC/OS).

We tell mon-put-instance-data.pl to track memory and disk usage. It will do this per instance and per Amazon auto-scale group. This allows you to see the metrics as a whole via the auto-scale group.

The packer file to build this image is as follows:

Packer to build EC2 image that has mon-put-instance-data to monitor Linux OS metrics

{
  "variables": {
    "aws_access_key": "",
    "aws_secret_key": "",
    "aws_region": "us-east-1",
    "aws_ami_image": "ami-6d1c2007",
    "aws_instance_type": "m4.large"
  },
  "builders": [
    {
      "type": "amazon-ebs",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "region": "{{user `aws_region`}}",
      "source_ami": "{{user `aws_ami_image`}}",
      "instance_type": "{{user `aws_instance_type`}}",
      "ssh_username": "centos",
      "ami_name": "base-centos-7-dcos-{{timestamp}}",
      "tags": {
        "Name": "ami-centos7-dcos-v1",
        "OS_Version": "LinuxCentOs7",
        "Release": "7",
        "Description": "Base CentOs7 image with prerequisites for DC/OS"
      },
      "user_data_file": "files/user-data.txt"
    }
  ],
  "provisioners": [
    {
      "type": "file",
      "source": "files/overlay.conf",
      "destination": "/home/centos/install/overlay.conf"
    },
    {
      "type": "shell",
      "inline": [
        "sudo mkdir -p /etc/modules-load.d",
        "sudo mv /home/centos/install/overlay.conf /etc/modules-load.d/overlay.conf",
        "sudo reboot"
      ]
    },
    {
      "type": "file",
      "source": "files/docker.repo",
      "destination": "/home/centos/install/docker.repo"
    },
    {
      "type": "file",
      "source": "files/override.conf",
      "destination": "/home/centos/install/override.conf"
    },
    {
      "type": "shell",
      "inline": [
        "echo installing Docker -----------------------------------",
        "sudo mkdir -p /etc/systemd/system/docker.service.d",
        "sudo mv /home/centos/install/override.conf /etc/systemd/system/docker.service.d/override.conf",
        "sudo mv /home/centos/install/docker.repo /etc/yum.repos.d/docker.repo",
        "sudo yum install -y docker-engine-1.11.2",
        "sudo systemctl start docker",
        "sudo systemctl enable docker",
        "sudo docker ps",
        "echo DONE installing Docker -----------------------------",
        "sudo yum install -y tar xz unzip curl ipset nano tree",
        "sudo sed -i s/SELINUX=enforcing/SELINUX=permissive/g /etc/selinux/config",
        "sudo groupadd nogroup",
        "sudo reboot"
      ]
    },
    {
      "type": "shell",
      "inline": [
        "echo installing aws cli -------------------------------",
        "mkdir /tmp/awscli",
        "cd /tmp/awscli",
        "curl https://s3.amazonaws.com/aws-cli/awscli-bundle.zip -o awscli-bundle.zip",
        "unzip awscli-bundle.zip",
        "sudo ./awscli-bundle/install -i /usr/lib/aws -b /usr/bin/aws",
        "sudo rm -rf /tmp/awscli",
        "echo DONE installing aws cli -------------------------------"
      ]
    },
    {
      "type": "file",
      "source": "files/journald-cloudwatch.conf",
      "destination": "/home/centos/install/journald-cloudwatch.conf"
    },
    {
      "type": "file",
      "source": "files/journald.unit",
      "destination": "/home/centos/install/journald.unit"
    },
    {
      "type": "shell",
      "inline": [
        "echo install log agent -------------------------------",
        "mkdir /tmp/logagent",
        "cd /tmp/logagent",
        "curl -OL  https://github.com/saymedia/journald-cloudwatch-logs/releases/download/v0.0.1/journald-cloudwatch-logs-linux.zip",
        "unzip journald-cloudwatch-logs-linux.zip",
        "sudo mv journald-cloudwatch-logs/journald-cloudwatch-logs /usr/bin",
        "sudo mkdir -p /var/lib/journald-cloudwatch-logs/",
        "sudo mv /home/centos/install/journald-cloudwatch.conf /etc/",
        "sudo mv /home/centos/install/journald.unit /etc/systemd/system/journald-cloudwatch.service",
        "sudo chmod 664 /etc/systemd/system/journald-cloudwatch.service",
        "sudo chown -R centos /var/lib/journald-cloudwatch-logs/",
        "sudo systemctl enable journald-cloudwatch.service",
        "sudo rm -rf /tmp/llogagent",
        "echo DONE installing log agent -------------------------------"
      ]
    },
    {
      "type": "file",
      "source": "files/install_cron.sh",
      "destination": "/home/centos/install/install_cron.sh"
    },
    {
      "type": "file",
      "source": "files/cronjobs",
      "destination": "/home/centos/install/cronjobs"
    },
    {
      "type": "shell",
      "inline": [
        "echo install aws metrics cron -------------------------------",
        "/usr/bin/bash /home/centos/install/install_cron.sh",
        "echo done installing aws metrics cron -------------------------------"
      ]
    },
    {
      "type": "shell",
      "inline": [
        "echo DONE installing packages for CentOS7 DC/OS"
      ]
    }
  ]
}

To learn more about this packer build and immutable-infrastructure in AWS see Using CloudFormation and Packer to build immutable infrastructure in AWS, and setting up an AWS log agent to send journald & DC/OS logs to Amazon CloudWatch Log Streams.

Comments