Server Monitoring Using Prometheus and Grafana.

Prometheus is a popular, open-source monitoring system. Prometheus is unique because it is collecting the metrics from the resources and stores them in a time-series database. Basically, Time Series Database will store time and value pair as data, which is helpful in terms of storing trends, curves, traces, etc. Also, it will give us a powerful Query model called PromQL. Prometheus offers great Visualization, Storage, Alert management, and Many Integrations. Grafana and Prometheus will make a fantastic combination as Grafana is so good at having many panels and visualizations. In this article, we will discuss How to Install and Configure Prometheus, node, and Grafana. Then we will have a detailed discussion on Server Monitoring Using Prometheus and Grafana using node_exporter.

Plan of action

Before getting into deep, Let’s discuss the architectural design of our plan of server monitoring using Prometheus and the Grafana.

As mentioned in the figure, the target server which is the server we need to be monitored should be installed with the piece of so software called Prometheus node_exporter. This will connect all kernel and node-level metrics like disk space, memory, CPU, and more. Then it will send it to the Prometheus Server. Then, Grafana will pull data from Prometheus and have dashboards.

So, As the Plan of action, we will see the following steps to accomplish this.

1. Prometheus Installation.
2. Prometheus node_exporter Installation
3. Grafana Installation and Configuration.

Download and Install the Prometheus.

Prometheus provides binaries to install on various platforms. So as in Linux Platform. As the first step, download the tar file which contains the Prometheus binary executable that can be moved directly to the binary folder. Otherwise, simply follow the following steps to install the Prometheus.

STEP 1: Download the Prometheus using wget command.

wget https://github.com/prometheus/prometheus/releases/download/v2.25.0/prometheus-2.25.0.linux-amd64.tar.gz

If you need to have different versions download it from the distribution list.

STEP 2: Unpack the tar file.

tar -xzf prometheus-2.25.0.linux-amd64.tar.gz

STEP 3: Move all the binary files to bin folder.

mv prometheus-2.25.0.linux-amd64/prometheus /usr/local/bin/

STEP 4: Configure the Prometheus Configuration file in etc folder.

mv prometheus-2.25.0.linux-amd64/Prometheus/prometheus.yml /etc/

this file will be modified later after the node_exporter Installation

STEP 5: Create Prometheus daemon process.

Create a Unit file

sudo nano /etc/systemd/system/prometheus.service

add the following lines

[Unit]

Description=Prometheus

Wants=network-online.target

After=network-online.target

[Service]

Type=simple

ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus.yml

[Install]

WantedBy=multi-user.target

Now we just need to restart the systemctl daemon process

sudo systemctl daemon-reload

then we need to enable the Prometheus process

sudo systemctl enable Prometheus

sudo systemctl start Prometheus

Thants it. Promotheus installation is completed. You can check the installation from the port number 9090 (http://<your-IP-ADDRESS:9090>)

Prometheus node_exporter Installation.

We have our Prometheus Server up and running. Now we need to send metrics of the target servers to Prometheus. Node_exporter is one of the Prometheus utility that will be installed on every target server which we need to be monitored.

Now, let’s see how to install node_exporter in the following steps.

STEP 1: Update the apt repo

sudo apt-get update

STEP 2: Install node_exporter using apt repo

sudo apt-get install prometheus-node-exporter

The advantage of using the apt repo is, we don’t have to configure the service as we did for the installing Prometheus server. But for verifying, run the following command to know whether the node_exporter is running or not.

sudo systemctl status node_exporter.service

That’s all the installation. Now let’s configure the node_exporter in such a way that it will communicate with the Prometheus server to export the required metrics.

Remember we talked about Pull based metrics collection of Prometheus?, Yes, Since Prometheus is working in pull-based metrics collection, we are going to configure the Prometheus server’s configuration file to pull the metrics from node_exporter.

So, Let’s follow the steps to configure the node_exporter to the prometheus.yml

STEP 1: Open the prometheus.yml file in the Prometheus server.

sudo nano /etc/prometheus/prometheus.yml

STEP 2: Add the following lines in the scrap_configs: section in the above file,

scrape_configs:

- job_name: '<name-the-target-server>'

static_configs:

- targets: ['<target_server_IP_address>:9100']

STEP 3: Once done, Restart the server.

sudo systemctl restart prometheus.service

Now, you can see the metrics exported from the target server using node_exporter in the drop down as mentioned in the below screen shot.

That’s all. We have configured the Prometheus server with node_exporter.

Installing and configuring Grafana

Configuring Prometheus server and node_exporter itself a milestone of server monitoring configuration. In addition to this, we will install and configure Grafana on another server.

To install the Grafana, follow the steps.

STEP 1: Download the latest package from the Grafana distribution list.

wget https://dl.grafana.com/oss/release/grafana_7.4.3_amd64.deb

STEP 2: unpack the package

sudo dpkg -i grafana_7.4.3_amd64.deb

STEP 3: Enable and Start the Grafana service

sudo systemctl enable grafana-server.service
sudo systemctl start grafana-server.service

That’s all. Grafana Installation is completed. Now, open the browser and hit <Grafana-server-IP-address>:3000 and follow the steps.

STEP 1: From the Home page, Click “Data Source” Configuration from the left panel.

STEP 2: Now click the “Add data source” button.

STEP 3: Select “Prometheus” from the reviled list

STEP 4: Enter the URL of the Prometheus Server, <Prometheus_ip>:9090

STEP 5: Leave the other field as it is and click “Save & Test”.

STEP 6: Check if it is giving a success message. Else, verify the endpoint and whether the Prometheus server and Grafana server can talk in the port or not using telnet.

That’s all. The Grafana Installation and configuration are done.

Conclusion

In this article, we will discuss How to Install and Configure Prometheus, node, and Grafana. Then we will have a detailed discussion on Server Monitoring Using Prometheus and Grafana using node_exporter. In our upcoming article, we will discuss how to configure Prometheus on various other target systems, like Jenkins, SonarQube, and more DevOps tools. Also, we will discuss more on how to create Dashboard and Panels in Grafana to make use of the Prometheus Metrics. Stay tuned and subscribe DigitalVarys for more articles and study materials on DevOps, Agile, DevSecOps and App Development.