Monitor Docker metrics using Prometheus & Grafana

Prometheus and Grafana are great tools for both metric data collection and visualization. Using Prometheus and Grafana, here we shall create a dashboard to monitor Docker container. 

Assume, we have already a setup with Grafana & Prometheus  and a docker container.

To setup Prometheus & Grafana, Click here.

To setup Docker container, Click here

Step-1: Check docker service is active or not

[root@docker docker]# service docker status| grep active

Redirecting to /bin/systemctl status docker.service

     Active: active (running) since Thu 2023-03-30 06:03:24 UTC; 47s ago

[root@docker docker]#

Step-2 : Add below file in docker server/ machine

vi /etc/docker/daemon.json

{

 "metrics-addr" : "172.31.37.33:9393",

 "experimental" : true

}

[root@docker docker]# service docker restart

Step3: Check running containers if any

[root@docker docker]# docker ps -a

CONTAINER ID   IMAGE     COMMAND       CREATED         STATUS          PORTS     NAMES

deb6378efe5b   ubuntu    "/bin/bash"   2 minutes ago   Up 6 seconds              container5

e2967e62a390   ubuntu    "/bin/bash"   2 minutes ago   Up 14 seconds             container4

d044849ad38e   ubuntu    "/bin/bash"   2 minutes ago   Up 30 seconds             container3          container1

[root@docker docker]#

Step-4: Edit prometheus.yml file and restart Prometheus service

[root@grafana ~]# cd /home/ec2-user/prometheus/prometheus_setup/

# vi prometheus.yml

static_configs:

      - targets: ["localhost:9090"]

      - targets: ["172.31.37.33:9100"]

      - targets: ["172.31.37.33:9323"]   ## added for docker

!wq!

# systemctl restart prometheus.service

Step-5 : Now connect to Prometheus server and check this target status.

It should be "Up"

Step-6: Create new dashboard/ panel

use below PromoSQL

sum by(state)(engine_daemon_container_states_containers{})

Now you will have data in your panel:

Manage dashboards - Grafana Dashboard Exporter/Importer

 Importing monitoring template from Grafana lab to grafana dashboard

Add custom Grafana dashboards:

You must have a Grafana account with admin rights to perform the steps described in this section.

To add a custom Grafana dashboard:

• Login to Grafana and create a new dashboard.
• Export your dashboard to json file. See Grafana documentation to find out how to do this.
• Put the exported file under the grafana-dashboards/ folder.

Use below official website to download json file:

https://grafana.com/grafana/dashboards/1860-node-exporter-full/

    node-exporter-full_rev30.json

Grafana Home --> Dash board --> Import --> Import Json file --> Upload Json file (node-exporter-full_rev30.json) --> choose Data source as Prometheus --> Import

Manually I added load to one target machine which is configured here. You can use last 15 mins data and see the chnages.

-- cpu load

# while true; do /bin/true; done

-- memory load

# cat <( </dev/zero head -c 500m) <(sleep 120) | tail

So, the take away is, no need to create all dash boards manually. Many are already developed and available in grafana official website. Just use it with documentation.

Try to create your another dashboard using below json file:

https://grafana.com/grafana/dashboards/13978-node-exporter-quickstart-and-dashboard/

    node-exporter-quickstart-and-dashboard_rev2.json

Grafana & Prometheus - Setup (Part-2)

 Grfana Dashboard setup(Part-1)

Here we can continue to setup grafana. so that we can have stunning visualizations of Prometheus server data collection.

 For my lab setup, I shall use same Prometheus server for grafana setup. But you can use different server also in real case scenario.

Go to grafana official website 

https://grafana.com/ --> Downloads --> Self-managed --> select Grafana --> Download --> select (OSS) Open source softwae --> select Linux version

[root@grafana ec2-user]# mkdir grafana

[root@grafana ec2-user]# cd grafana/

[root@grafana grafana]# pwd

/home/ec2-user/grafana

# wget https://dl.grafana.com/oss/release/grafana-9.4.7-1.x86_64.rpm

# sudo yum install grafana-9.4.7-1.x86_64.rpm

[root@grafana grafana]# service grafana-server restart

Restarting grafana-server (via systemctl):                 [  OK  ]

-- Enable always on for both grafana and Prometheus

[root@grafana grafana]# chkconfig grafana-server on

[root@grafana grafana]# systemctl enable prometheus.service

-- in targets/ nodes also enable node exporter

[root@target node_exporter]# systemctl enable node_exporter.service

Note: Grafana runs on default 3000 port. make sure this port allowed in security group.

Now open grafana URL.

http://54.169.181.69:300

It ll ask you username and password. By default username and password will be "admin".

As soon as you logged in, it ll prompt you to change password:

Now you need to configure data source.. Grafana support lots of data sources:

select Configuration --> Data source

Now add prmotheus data souce like below:

URL: http://<private-ip>:9090/

Then "save & Test". You can see message like "Data souce is working". If error need to check whether IP address used wrongly etc.

Now create your dashboard:

Select Dash board --> New Dash board --> New Panel 

Now you need to choose data source. Here we have only Prometheus data source as of now.

then, use same PromoQL. for that select "Code" and use the query and click on "Run queries"

Here is my first dashboard. You can give name to this panel also. You can create many panels and dashboards as per your requirements.

Grafana & Prometheus - Setup (Part-1)

 About Gaafana:
Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources. Most high lighted part is "Grafana provides you with tools to turn your time-series database (TSDB) data into insightful graphs and visualizations".

About Prometheus:
Prometheus is an open-source technology designed to provide monitoring and alerting functionality for cloud-native environments, including Kubernetes. It records real-time metrics in a time series database built using a HTTP pull model, with flexible queries and real-time alerting.. It can also collect and record labels, which are optional key-value pairs.
Let us proceed with a lab setup to configure Prometheus and Grafana. Here I am using free tier AWS account with EC2 instances to setup my lab environment.

My architecture diagram will be like below: (simple monitoring architecture)

Installation steps:

Installation of Prometheus on Linux Server:

# sudo hostname grafana

# sudo su

[root@grafana ~]# cd /home/ec2-user/

# mkdir prometheus

# cd prometheus

# 

Step#1 : Install Prometheus 

https://prometheus.io/  --> download --> prometheus-2.43.0.linux-amd64.tar.gz (copy this link)

[root@grafana prometheus]# wget https://github.com/prometheus/prometheus/releases/download/v2.43.0/prometheus-2.43.0.linux-amd64.tar.gz

...

prometheus-2.43.0.linux-amd64 100%[=================================================>]  86.87M  17.2MB/s    in 6.4s

[root@grafana prometheus]# tar -xzvf prometheus-2.43.0.linux-amd64.tar.gz

[root@grafana prometheus]# mv prometheus-2.43.0.linux-amd64 prometheus_setup

[root@grafana prometheus]# # cd prometheus_setup/

# pwd

/home/ec2-user/prometheus/prometheus_setup

Step#2 : Configure Prometheus

# cd /etc/systemd/system

# vi prometheus.service

[Unit]

Description=Prometheus Server

Documentation=https://prometheus.io/docs/introduction/overview/

After=network-online.target

[Service]

User=root

Restart=on-failure

ExecStart=/home/ec2-user/prometheus/prometheus_setup/prometheus --config.file=/home/ec2-user/prometheus/prometheus_setup/prometheus.yml

[Install]

WantedBy=multi-user.target

:wq

Restart Prometheus

[root@grafana prometheus_setup]# pwd

/home/ec2-user/prometheus/prometheus_setup

[root@grafana prometheus_setup]# systemctl restart prometheus.service

verify status

#  systemctl status prometheus.service

prometheus.service - Prometheus Server

     Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; preset: disabled)

     Active: active (running) since Sat 2023-03-25 19:03:44 UTC; 57s ago

Step#3 : Allow default Prometheus port 9090 in security group

security Group -->  select your security group --> Inbound rules --> Edit inbound rules --> Add rule --> Custom TCP --> 9090 --> 0.0.0.0/0 (internet) --> save 

Step#4: Use Prometheus from browser

copy the public IP and used 9090 port

http://54.169.181.69/9090

Now you have Prometheus Home page

Setup Target

Now we can configure target machines to start monitoring. Here at least we can configure one target machine.

Go to Prometheus official website --> Click on node exporter 

https://prometheus.io/download/#node_exporter

copy node_exporter-1.5.0.linux-amd64.tar.gz link

[root@target ~]# cd /home/ec2-user/

[root@target ec2-user]# mkdir node_exporter

[root@target ec2-user]# cd node_exporter

[root@target node_exporter]# wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz

[root@target node_exporter]# tar xzvf node_exporter-1.5.0.linux-amd64.tar.gz

# mv node_exporter-1.5.0.linux-amd64 node_exporter

[root@target node_exporter]# pwd

/home/ec2-user/node_exporter/node_exporter

[root@target node_exporter]# ls -l

total 19336

-rw-r--r--. 1 3434 3434    11357 Nov 29 19:05 LICENSE

-rw-r--r--. 1 3434 3434      463 Nov 29 19:05 NOTICE

-rwxr-xr-x. 1 3434 3434 19779640 Nov 29 18:59 node_exporter

[root@target node_exporter]#

Now create node exporter service file like previous. Name of files are fixed. You can't use custom names.

# touch /etc/systemd/system/node_exporter.service

# vi /etc/systemd/system/node_exporter.service

[Unit]

Description=Node_exporter Server

Documentation=https://prometheus.io/docs/introduction/overview/

After=network-online.target

[Service]

User=root

Restart=on-failure

ExecStart=/home/ec2-user/node_exporter/node_exporter/node_exporter

[Install]

WantedBy=multi-user.target

:wq!

Now restart node exporter and check status:

[root@target node_exporter]# systemctl restart node_exporter.service

[root@target node_exporter]# systemctl status node_exporter.service

● node_exporter.service - Node_exporter Server

     Loaded: loaded (/etc/systemd/system/node_exporter.service; disabled; preset: disabled)

     Active: active (running) since Sat 2023-03-25 19:59:12 UTC; 15s ago

Now attach this target to prometheus server:

Go to prometheus server

Edit prometheus.yml and node/target machine details

[root@grafana prometheus_setup]# pwd

/home/ec2-user/prometheus/prometheus_setup

[root@grafana prometheus_setup]# ls prometheus.yml

prometheus.yml

Note: Use private IP address for the targets as our ab setup resides on same netwrok.

# vi prometheus.yml

 static_configs:

      - targets: ["localhost:9090"]

      - targets: ["172.31.37.33:9100"]

Note: Node exporter service runs on 9100 port and you need to allow this port as well.

Now again Prometheus service.

Now check whether you able to see node/ target details in Prometheus dashboard or not.

Yes we can see it.

Prometheus Home --> Status --> Targets 

Here we can see our target machine/ node machine.

You can monitor targets using PromoSQL (queries)

e.g.,

Use below query in search query box and click on execute and select graph

sum by(mode)(irate(node_cpu_seconds_total{mode!="idle"}[5m])) > 0

Here we can see this CPU load metric values with interval of 5 mins as per our PromoSQL query.

But, this graph is not so much user friendly and informative. For this w can setup and use grafana dash board .

Click on  below grafana dashboard setup link :

http://facedba.blogspot.com/2023/03/grafana-prometheus-setup-part-2.html

Grafana & Prometheus - Setup (part-2)