Setting up an AWS log agent to send journald & DC/OS logs to Amazon CloudWatch Log Service

posted Nov 7, 2016, 8:28 PM by Rick Hightower [ updated Dec 5, 2016, 12:44 PM ]

In this example, we are using Centos7, journalctl and systemctl so that we can monitor logs from DC/OSinstances (masters, agents and public agents). It is useful for anyone using systemd, journald in an AWS EC2enviroment that wants logging. The nice thing about Amazon CloudWatch is that it integrates well with Amazon EMRand Amazon Elasticsearch. (For more background on this subject see this article which covers using CloudFormation, Packr, etc. for Immutable Infrastructure to build DC/OS and deploy it to Amazon Web Services.)

We will install systemd-cloud-watch. We are going to setup a daemon into systemd that forwards logs to Amazon CloudWatch log streams.

This utility journald-cloudwatch-logs monitors the systemd journal, managed by journald, and writes journal entries into AWS Cloudwatch Logs. You can find out more about it at their website journald-cloudwatch-logs.

This program is an alternative to the AWS-provided logs agent which works only with sending text log files into AWS Cloudwatch. Conversely the journald-cloudwatch-logs utility reads directly from the systemd journal.

Step 1) Create an IAM role to start your DC/OS instances.

Step 2) Create an IAM policy as follows and associate it with your the DC/OS IAM role.

IAM policy AllowLogs

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:*",
                "arn:aws:logs:*:*:log-group:*:log-stream:*"
            ]
        }
    ]
}

Step 3) Download and Install the agent software and setup config files

Install agent software

mkdir /tmp/logagent
cd /tmp/logagent
curl -OL  https://github.com/advantageous/systemd-cloud-watch/releases/download/v0.1.0/systemd-cloud-watch_linux

sudo mv journald-cloudwatch-logs/systemd-cloud-watch_linux /usr/bin/journald-cloudwatch-logs

sudo mkdir -p /var/lib/journald-cloudwatch-logs/
sudo mv /home/centos/install/journald-cloudwatch.conf /etc/
sudo mv /home/centos/install/journald.unit /etc/systemd/system/journald-cloudwatch.service
sudo chmod 664 /etc/systemd/system/journald-cloudwatch.service
sudo chown -R centos /var/lib/journald-cloudwatch-logs/
sudo systemctl enable journald-cloudwatch.service

install/journald.unit

[Unit]
Description=journald-cloudwatch-logs
Wants=basic.target
After=basic.target network.target

[Service]
User=centos
Group=centos
ExecStart=/usr/bin/journald-cloudwatch-logs /etc/journald-cloudwatch.conf
KillMode=process
Restart=on-failure
RestartSec=42s


[Install]
WantedBy=getty.target

install/journald-cloudwatch.conf

log_group = "dcos-logstream"

log_priority = "WARNING"

We created packer file to automate the creation of our base AMI image for DC/OS.

Packer creation script

{
  "variables": {
    "aws_access_key": "",
    "aws_secret_key": "",
    "aws_region": "us-east-1",
    "aws_ami_image": "ami-6d1c2007",
    "aws_instance_type": "m4.large"
  },
  "builders": [
    {
      "type": "amazon-ebs",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "region": "{{user `aws_region`}}",
      "source_ami": "{{user `aws_ami_image`}}",
      "instance_type": "{{user `aws_instance_type`}}",
      "ssh_username": "centos",
      "ami_name": "base-centos-7-dcos-{{timestamp}}",
      "tags": {
        "Name": "ami-centos7-dcos-v1",
        "OS_Version": "LinuxCentOs7",
        "Release": "7",
        "Description": "Base CentOs7 image with prerequisites for DC/OS"
      },
      "user_data_file": "files/user-data.txt"
    }
  ],
  "provisioners": [
    {
      "type": "file",
      "source": "files/overlay.conf",
      "destination": "/home/centos/install/overlay.conf"
    },
    {
      "type": "shell",
      "inline": [
        "sudo mkdir -p /etc/modules-load.d",
        "sudo mv /home/centos/install/overlay.conf /etc/modules-load.d/overlay.conf",
        "sudo reboot"
      ]
    },
    {
      "type": "file",
      "source": "files/docker.repo",
      "destination": "/home/centos/install/docker.repo"
    },
    {
      "type": "file",
      "source": "files/override.conf",
      "destination": "/home/centos/install/override.conf"
    },
    {
      "type": "shell",
      "inline": [
        "echo installing Docker -----------------------------------",
        "sudo mkdir -p /etc/systemd/system/docker.service.d",
        "sudo mv /home/centos/install/override.conf /etc/systemd/system/docker.service.d/override.conf",
        "sudo mv /home/centos/install/docker.repo /etc/yum.repos.d/docker.repo",
        "sudo yum install -y docker-engine-1.11.2",
        "sudo systemctl start docker",
        "sudo systemctl enable docker",
        "sudo docker ps",
        "echo DONE installing Docker -----------------------------",
        "sudo yum install -y tar xz unzip curl ipset nano tree",
        "sudo sed -i s/SELINUX=enforcing/SELINUX=permissive/g /etc/selinux/config",
        "sudo groupadd nogroup",
        "sudo reboot"
      ]
    },
    {
      "type": "shell",
      "inline": [
        "echo installing aws cli -------------------------------",
        "mkdir /tmp/awscli",
        "cd /tmp/awscli",
        "curl https://s3.amazonaws.com/aws-cli/awscli-bundle.zip -o awscli-bundle.zip",
        "unzip awscli-bundle.zip",
        "sudo ./awscli-bundle/install -i /usr/lib/aws -b /usr/bin/aws",
        "echo DONE installing aws cli -------------------------------"
      ]
    },
    {
      "type": "file",
      "source": "files/journald-cloudwatch.conf",
      "destination": "/home/centos/install/journald-cloudwatch.conf"
    },
    {
      "type": "file",
      "source": "files/journald.unit",
      "destination": "/home/centos/install/journald.unit"
    },
    {
      "type": "shell",
      "inline": [
        "echo install log agent -------------------------------",
        "mkdir /tmp/logagent",
        "cd /tmp/logagent",
        "curl -OL  https://github.com/saymedia/journald-cloudwatch-logs/releases/download/v0.0.1/journald-cloudwatch-logs-linux.zip",
        "unzip journald-cloudwatch-logs-linux.zip",
        "sudo mv journald-cloudwatch-logs/journald-cloudwatch-logs /usr/bin",
        "sudo mkdir -p /var/lib/journald-cloudwatch-logs/",
        "sudo mv /home/centos/install/journald-cloudwatch.conf /etc/",
        "sudo mv /home/centos/install/journald.unit /etc/systemd/system/journald-cloudwatch.service",
        "sudo chmod 664 /etc/systemd/system/journald-cloudwatch.service",
        "sudo chown -R centos /var/lib/journald-cloudwatch-logs/",
        "sudo systemctl enable journald-cloudwatch.service",
        "echo DONE installing log agent -------------------------------"
      ]
    },
    {
      "type": "shell",
      "inline": [
        "echo DONE installing packages for CentOS7 DC/OS"
      ]
    }
  ]
}

Comments

	Search this site