CloudWatch is an Amazing service in AWS that will collect Metrics and Logs from the Server instances and other AWS services. The same can be integrated with Notification Service (SNS) or Email Service (SES). However, CloudWatch is having default pre-defined metrics and logs within the AWS services. This Limitation can be overcome and increase the metrics and log stream availabilities, we have an excellent solution called CloudWatch Agent. This will collect the Metrics and Log streams from EC2 instance or on-premise servers. In this article, we will discuss Installing Amazon CloudWatch Agent and Collecting Metrics and Logs from Amazon EC2 Instances.
Table of Contents
Pre-Requisites
Before installing and configuring the CloudWatch Agent we need to get some pre-configuration and setups both from the AWS side and server-side. We will see the configurations one by one.
AWS IAM Role for EC2 Instance.
In order to send and receive CloudWatch related data from Instance to CloudWatch service, we need to give the EC2 instance the proper permission with AWS IAM Role. So, Let’s configure the IAM role for the same.
To Create IAM Role Follow the below steps,
The Created Role will look like this:
Assign the Role to EC2 Instance.
Now we need to assign the created IAM Role to the EC2 Instance so that the EC2 Instance can able to send and receive the data from CloudWatch to CloudWatch Agent.
For the same Follow the steps,
Once you finish the above steps, you will see the role in the instance status block as mentioned in the picture
Installing the CloudWatch Agent
Now, lets start with the Installing the CloudWatch Agent.
CloudWatch Agent can be installed using
- Command Line
- AWS System Manager
- AWS CloudFormation
But to understand the working method of the CloudWatch Agent, we will proceed with CloudWatch Agent.
Installing from Command line is very easy by following the simple steps.
Download Agent for Linux.
root@awsinstance:~/CWAgent$ wget https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip
root@awsinstance:~/CWAgent$ unzip AmazonCloudWatchAgent.zip
root@awsinstance:~/CWAgent$ ./install.sh
Download Agent For Windows
https://s3.amazonaws.com/amazoncloudwatch-agent/windows/amd64/latest/AmazonCloudWatchAgent.zip
unzip AmazonCloudWatchAgent.zip
./install.ps1
Create the Configuration File
To run the CloudWatch Agent as Daemon on the Instance, we need to create a configuration file as JSON first. We can manually create the Configuration file but, it is advised to create it from Configuration Wizard. So, go to /opt/aws/amazon-cloudwatch-agent/bin the Run the configuration wizard by passing the following command and answer the questions one by one.
Remember the default metrics are Basic, Standard, and Advanced which are explained In this Article. And Here I provided the Log group name as access.log as I want to see the apache server logs.
root@awsinstance:~# cd /opt/aws/amazon-cloudwatch-agent/bin
root@awsinstance:~# /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
=============================================================
= Welcome to the AWS CloudWatch Agent Configuration Manager =
=============================================================
On which OS are you planning to use the agent?
1. linux
2. windows
default choice: [1]:
1
Trying to fetch the default region based on ec2 metadata...
Are you using EC2 or On-Premises hosts?
1. EC2
2. On-Premises
default choice: [1]:
1
Which user are you planning to run the agent?
1. root
2. cwagent
3. others
default choice: [1]:
1
Do you want to turn on StatsD daemon?
1. yes
2. no
default choice: [1]:
2
Do you want to monitor metrics from CollectD?
1. yes
2. no
default choice: [1]:
2
Do you want to monitor any host metrics? e.g. CPU, memory, etc.
1. yes
2. no
default choice: [1]:
1
Do you want to monitor cpu metrics per core? Additional CloudWatch charges may apply.
1. yes
2. no
default choice: [1]:
2
Do you want to add ec2 dimensions (ImageId, InstanceId, InstanceType, AutoScalingGroupName) into all of your metrics if the info is available?
1. yes
2. no
default choice: [1]:
1
Would you like to collect your metrics at high resolution (sub-minute resolution)? This enables sub-minute resolution for all metrics, but you can customize for specific metrics in the output json file.
1. 1s
2. 10s
3. 30s
4. 60s
default choice: [4]:
1
Which default metrics config do you want?
1. Basic
2. Standard
3. Advanced
4. None
default choice: [1]:
1
Current config as follows:
{
"agent": {
"metrics_collection_interval": 1,
"run_as_user": "root"
},
"metrics": {
"append_dimensions": {
"AutoScalingGroupName": "${aws:AutoScalingGroupName}",
"ImageId": "${aws:ImageId}",
"InstanceId": "${aws:InstanceId}",
"InstanceType": "${aws:InstanceType}"
},
"metrics_collected": {
"disk": {
"measurement": [
"used_percent"
],
"metrics_collection_interval": 1,
"resources": [
"*"
]
},
"mem": {
"measurement": [
"mem_used_percent"
],
"metrics_collection_interval": 1
}
}
}
}
Are you satisfied with the above config? Note: it can be manually customized after the wizard completes to add additional items.
1. yes
2. no
default choice: [1]:
2
Which default metrics config do you want?
1. Basic
2. Standard
3. Advanced
4. None
default choice: [1]:
1
Current config as follows:
{
"agent": {
"metrics_collection_interval": 1,
"run_as_user": "root"
},
"metrics": {
"append_dimensions": {
"AutoScalingGroupName": "${aws:AutoScalingGroupName}",
"ImageId": "${aws:ImageId}",
"InstanceId": "${aws:InstanceId}",
"InstanceType": "${aws:InstanceType}"
},
"metrics_collected": {
"disk": {
"measurement": [
"used_percent"
],
"metrics_collection_interval": 1,
"resources": [
"*"
]
},
"mem": {
"measurement": [
"mem_used_percent"
],
"metrics_collection_interval": 1
}
}
}
}
Are you satisfied with the above config? Note: it can be manually customized after the wizard completes to add additional items.
1. yes
2. no
default choice: [1]:
1
Do you have any existing CloudWatch Log Agent (http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html) configuration file to import for migration?
1. yes
2. no
default choice: [2]:
2
Do you want to monitor any log files?
1. yes
2. no
default choice: [1]:
1
Log file path:
/var/log/apache2/access.log
Log group name:
default choice: [access.log]
Log stream name:
default choice: [{instance_id}]
Do you want to specify any additional log files to monitor?
1. yes
2. no
default choice: [1]:
2
Saved config file to /opt/aws/amazon-cloudwatch-agent/bin/config.json successfully.
Current config as follows:
{
"agent": {
"metrics_collection_interval": 1,
"run_as_user": "root"
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/apache2/access.log",
"log_group_name": "access.log",
"log_stream_name": "{instance_id}"
}
]
}
}
},
"metrics": {
"append_dimensions": {
"AutoScalingGroupName": "${aws:AutoScalingGroupName}",
"ImageId": "${aws:ImageId}",
"InstanceId": "${aws:InstanceId}",
"InstanceType": "${aws:InstanceType}"
},
"metrics_collected": {
"disk": {
"measurement": [
"used_percent"
],
"metrics_collection_interval": 1,
"resources": [
"*"
]
},
"mem": {
"measurement": [
"mem_used_percent"
],
"metrics_collection_interval": 1
}
}
}
}
Please check the above content of the config.
The config file is also located at /opt/aws/amazon-cloudwatch-agent/bin/config.json.
Edit it manually if needed.
Do you want to store the config in the SSM parameter store?
1. yes
2. no
default choice: [1]:
2
Program exits now.This will store the configuration file as config.json (/opt/aws/amazon-cloudwatch-agent/bin/config.json)
Running the CloudWatch Agent
Now, Start the CloudWatch Agent and Make it run as Daemon process. So, Run the following command.
root@awsinstance:~/CWAgent$ /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s
In the above command,
Once after the above command, check the running CloudWatch Agent by running the follow command
root@awsinstance:~/CWAgent$ service amazon-cloudwatch-agent status
This will show whether the service us running or not.
Verify the CloudWatch Metrics and Logs in AWS Console
As part of the Installing Amazon CloudWatch Agent and Collecting Metrics and Logs from Amazon EC2 Instance and the final step, verify whether the setup is working or not by login into the AWS Console.
CloudWatch Metrics
For checking the metrics from CloudWatch Agent, Follow the bellow steps
Then You can see metrics as mentioned in the below screenshot.
CloudWatch Logs
For checking the metrics from CloudWatch Agent, Follow the bellow steps
Then You can see the Log Stream as mentioned in the below screenshot.
Conclusion
In this article, we have discussed Installing Amazon CloudWatch Agent and Collecting Metrics and Logs from Amazon EC2 Instances. Hope this article explains how to install and configure the same. But we haven’t discussed much how to utilize the metrics and collected Logs for Alert and other processes. We will discuss the same in our upcoming article. Stay tuned and subscribe DigitalVarys for more articles and study materials on DevOps, Agile, DevSecOps, and App Development.
Experienced DevSecOps Practitioner, Tech Blogger, Expertise in Designing Solutions in Public and Private Cloud. Opensource Community Contributor.
Your blog has so many great articles. I was impressed by the great job you have done with the way each topic is presented in your writing.
AWS Training in Hyderabad
AWS Course in Hyderabad
Impressive blog with lovely information. Really a very useful article for us thanks for sharing such a wonderful blog.
https://espirittech.com/php-development-services/
Nice blog. I finally found great post here Very interesting to read this article and very pleased to find this site. Great work!
https://espirittech.com/dot-net-application-development-company/
Really enjoyed reading your article, the information you delivered in this post was damn good. Keep sharing your post with efficient news.
Software Development Services
Software Development Company