CloudWatch Agent and Collecting of metrics and Logs

Installing Amazon CloudWatch Agent and Collecting Metrics and Logs from Amazon EC2 Instances

CloudWatch is an Amazing service in AWS that will collect Metrics and Logs from the Server instances and other AWS services. The same can be integrated with Notification Service (SNS) or Email Service (SES). However, CloudWatch is having default pre-defined metrics and logs within the AWS services. This Limitation can be overcome and increase the metrics and log stream availabilities, we have an excellent solution called CloudWatch Agent. This will collect the Metrics and Log streams from EC2 instance or on-premise servers. In this article, we will discuss Installing Amazon CloudWatch Agent and Collecting Metrics and Logs from Amazon EC2 Instances.

Pre-Requisites

Before installing and configuring the CloudWatch Agent we need to get some pre-configuration and setups both from the AWS side and server-side. We will see the configurations one by one.

AWS IAM Role for EC2 Instance.

In order to send and receive CloudWatch related data from Instance to CloudWatch service, we need to give the EC2 instance the proper permission with AWS IAM Role. So, Let’s configure the IAM role for the same.

To Create IAM Role Follow the below steps,

  • STEP 1: Go to Services
  • STEP 2: Select IAM
  • STEP 3: Select Roles from the left Panel
  • STEP 4: Click Create Role
  • STEP 5: Select type AWS service and Choose a use case EC2 from Common use case
  • STEP 6: Click Next: Permission
  • STEP 7: Search Below Policies and check those
  • STEP 8: Click Next: Tags
  • STEP 9: Add Tags optionally
  • STEP 10: Click Next: Review
  • STEP 11: Fill Role Name (cloudwatch-custom-metrics), Role description
  • STEP 12: Click Create role.

The Created Role will look like this:

AWS IAM Role
AWS IAM Role

Assign the Role to EC2 Instance.

Now we need to assign the created IAM Role to the EC2 Instance so that the EC2 Instance can able to send and receive the data from CloudWatch to CloudWatch Agent.

For the same Follow the steps,

  • STEP 1: Go to Services
  • STEP 2: Select EC2
  • STEP 3: Click Instance from the left panel
  • STEP 4: Select and Check the instance you want to assign
  • STEP 5: Click Actions
  • STEP 6: Scroll down to Security and Select it
  • STEP 7:-> Select Modify IAM Role
  • STEP 8: Type in the Above Created Role (cloudwatch-custom-metrics)
  • STEP 9: The Click Save.

Once you finish the above steps, you will see the role in the instance status block as mentioned in the picture

Assigning IAM Role to AWS EC2
Assigning IAM Role to AWS EC2

Installing the CloudWatch Agent

Now, lets start with the Installing the CloudWatch Agent.

CloudWatch Agent can be installed using

  1. Command Line
  2. AWS System Manager
  3. AWS CloudFormation

But to understand the working method of the CloudWatch Agent, we will proceed with CloudWatch Agent.

Installing from Command line is very easy by following the simple steps.

 Download Agent for Linux.

  • STEP 1: Download CloudWatch Agent package
[email protected]:~/CWAgent$ wget https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip
  • STEP 2: UnZip the package
[email protected]:~/CWAgent$ unzip AmazonCloudWatchAgent.zip
  • STEP 3: Run the Install Script
[email protected]:~/CWAgent$ ./install.sh 

Download Agent For Windows

  • STEP 1: Download CloudWatch Agent package
https://s3.amazonaws.com/amazoncloudwatch-agent/windows/amd64/latest/AmazonCloudWatchAgent.zip
  • STEP 1: Unzip the Package
unzip AmazonCloudWatchAgent.zip
  • STEP 1: Run the Installation Script
./install.ps1

Create the Configuration File

To run the CloudWatch Agent as Daemon on the Instance, we need to create a configuration file as JSON first. We can manually create the Configuration file but, it is advised to create it from Configuration Wizard. So, go to /opt/aws/amazon-cloudwatch-agent/bin the Run the configuration wizard by passing the following command and answer the questions one by one.

Remember the default metrics are Basic, Standard, and Advanced which are explained In this Article. And Here I provided the Log group name as access.log as I want to see the apache server logs.

[email protected]:~# cd /opt/aws/amazon-cloudwatch-agent/bin
[email protected]:~# /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
=============================================================
= Welcome to the AWS CloudWatch Agent Configuration Manager =
=============================================================
On which OS are you planning to use the agent?
1. linux
2. windows
default choice: [1]:
1
Trying to fetch the default region based on ec2 metadata...
Are you using EC2 or On-Premises hosts?
1. EC2
2. On-Premises
default choice: [1]:
1
Which user are you planning to run the agent?
1. root
2. cwagent
3. others
default choice: [1]:
1
Do you want to turn on StatsD daemon?
1. yes
2. no
default choice: [1]:
2
Do you want to monitor metrics from CollectD?
1. yes
2. no
default choice: [1]:
2
Do you want to monitor any host metrics? e.g. CPU, memory, etc.
1. yes
2. no
default choice: [1]:
1
Do you want to monitor cpu metrics per core? Additional CloudWatch charges may apply.
1. yes
2. no
default choice: [1]:
2
Do you want to add ec2 dimensions (ImageId, InstanceId, InstanceType, AutoScalingGroupName) into all of your metrics if the info is available?
1. yes
2. no
default choice: [1]:
1
Would you like to collect your metrics at high resolution (sub-minute resolution)? This enables sub-minute resolution for all metrics, but you can customize for specific metrics in the output json file.
1. 1s
2. 10s
3. 30s
4. 60s
default choice: [4]:
1
Which default metrics config do you want?
1. Basic
2. Standard
3. Advanced
4. None
default choice: [1]:
1
Current config as follows:
{
        "agent": {
                "metrics_collection_interval": 1,
                "run_as_user": "root"
        },
        "metrics": {
                "append_dimensions": {
                        "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
                        "ImageId": "${aws:ImageId}",
                        "InstanceId": "${aws:InstanceId}",
                        "InstanceType": "${aws:InstanceType}"
                },
                "metrics_collected": {
                        "disk": {
                                "measurement": [
                                        "used_percent"
                                ],
                                "metrics_collection_interval": 1,
                                "resources": [
                                        "*"
                                ]
                        },
                        "mem": {
                                "measurement": [
                                        "mem_used_percent"
                                ],
                                "metrics_collection_interval": 1
                        }
                }
        }
}
Are you satisfied with the above config? Note: it can be manually customized after the wizard completes to add additional items.
1. yes
2. no
default choice: [1]:
2
Which default metrics config do you want?
1. Basic
2. Standard
3. Advanced
4. None
default choice: [1]:
1
Current config as follows:
{
        "agent": {
                "metrics_collection_interval": 1,
                "run_as_user": "root"
        },
        "metrics": {
                "append_dimensions": {
                        "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
                        "ImageId": "${aws:ImageId}",
                        "InstanceId": "${aws:InstanceId}",
                        "InstanceType": "${aws:InstanceType}"
                },
                "metrics_collected": {
                        "disk": {
                                "measurement": [
                                        "used_percent"
                                ],
                                "metrics_collection_interval": 1,
                                "resources": [
                                        "*"
                                ]
                        },
                        "mem": {
                                "measurement": [
                                        "mem_used_percent"
                                ],
                                "metrics_collection_interval": 1
                        }
                }
        }
}
Are you satisfied with the above config? Note: it can be manually customized after the wizard completes to add additional items.
1. yes
2. no
default choice: [1]:
1
Do you have any existing CloudWatch Log Agent (http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html) configuration file to import for migration?
1. yes
2. no
default choice: [2]:
2
Do you want to monitor any log files?
1. yes
2. no
default choice: [1]:
1
Log file path:
/var/log/apache2/access.log
Log group name:
default choice: [access.log]

Log stream name:
default choice: [{instance_id}]

Do you want to specify any additional log files to monitor?
1. yes
2. no
default choice: [1]:
2
Saved config file to /opt/aws/amazon-cloudwatch-agent/bin/config.json successfully.
Current config as follows:
{
        "agent": {
                "metrics_collection_interval": 1,
                "run_as_user": "root"
        },
        "logs": {
                "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                                "file_path": "/var/log/apache2/access.log",
                                                "log_group_name": "access.log",
                                                "log_stream_name": "{instance_id}"
                                        }
                                ]
                        }
                }
        },
        "metrics": {
                "append_dimensions": {
                        "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
                        "ImageId": "${aws:ImageId}",
                        "InstanceId": "${aws:InstanceId}",
                        "InstanceType": "${aws:InstanceType}"
                },
                "metrics_collected": {
                        "disk": {
                                "measurement": [
                                        "used_percent"
                                ],
                                "metrics_collection_interval": 1,
                                "resources": [
                                        "*"
                                ]
                        },
                        "mem": {
                                "measurement": [
                                        "mem_used_percent"
                                ],
                                "metrics_collection_interval": 1
                        }
                }
        }
}
Please check the above content of the config.
The config file is also located at /opt/aws/amazon-cloudwatch-agent/bin/config.json.
Edit it manually if needed.
Do you want to store the config in the SSM parameter store?
1. yes
2. no
default choice: [1]:
2
Program exits now.

This will store the configuration file as config.json (/opt/aws/amazon-cloudwatch-agent/bin/config.json)

Running the CloudWatch Agent

Now, Start the CloudWatch Agent and Make it run as Daemon process. So, Run the following command.

[email protected]:~/CWAgent$ /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

In the above command,

  • amazon-cloudwatch-agent-ctl is the executable script with parameter.
  • -a is the action parameter which can be start, stop, status, fetch/append/remove configuration.
  • -m is the mode parameter that will select the instance type as EC2 or onPremise or auto.
  • -c is the location for the configuration file. We can have a configuration in AWS SSM or in a local file. Here we selected the local file.
  • -s is to optionally restart the CloudWatch Agent.

Once after the above command, check the running CloudWatch Agent by running the follow command

[email protected]:~/CWAgent$ service amazon-cloudwatch-agent status

This will show whether the service us running or not.

Verify the CloudWatch Metrics and Logs in AWS Console

As part of the Installing Amazon CloudWatch Agent and Collecting Metrics and Logs from Amazon EC2 Instance and the final step, verify whether the setup is working or not by login into the AWS Console.

CloudWatch Metrics

 For checking the metrics from CloudWatch Agent, Follow the bellow steps

  • STEP 1: Go to Services
  • STEP 2: Click CloudWatch
  • STEP 3: Click Metrics From left panel
  • STEP 4: Click All Metrics
  • Step 5: Click Custom Namespaces You can see CWAgent.

Then You can see metrics as mentioned in the below screenshot.

AWS CloudWatch Agent Metrics
AWS CloudWatch Agent Metrics

CloudWatch Logs

For checking the metrics from CloudWatch Agent, Follow the bellow steps

  • STEP 1: Go to Services
  • STEP 2: Click CloudWatch
  • STEP 3: Click Logs From the left panel
  • STEP 4: Click View All Log Groups button
  • STEP 5: In the filter, search for the Log group you name in the configuration file. Here “access.log”.
  • STEP 6: Click the Log Group name (access.log)
  • STEP 7: Click the Instance ID

Then You can see the Log Stream as mentioned in the below screenshot.

AWS CloudWatch Log Stream By CloudWatch Agent
AWS CloudWatch Log Stream By CloudWatch Agent

Conclusion

In this article, we have discussed Installing Amazon CloudWatch Agent and Collecting Metrics and Logs from Amazon EC2 Instances. Hope this article explains how to install and configure the same. But we haven’t discussed much how to utilize the metrics and collected Logs for Alert and other processes. We will discuss the same in our upcoming article. Stay tuned and subscribe DigitalVarys for more articles and study materials on DevOps, Agile, DevSecOps, and App Development.

2 thoughts on “Installing Amazon CloudWatch Agent and Collecting Metrics and Logs from Amazon EC2 Instances”

Leave a Reply