Grafana & Prometheus - Setup (Part-1)

 About Gaafana:
Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources. Most high lighted part is "Grafana provides you with tools to turn your time-series database (TSDB) data into insightful graphs and visualizations".

About Prometheus:
Prometheus is an open-source technology designed to provide monitoring and alerting functionality for cloud-native environments, including Kubernetes. It records real-time metrics in a time series database built using a HTTP pull model, with flexible queries and real-time alerting.. It can also collect and record labels, which are optional key-value pairs.
Let us proceed with a lab setup to configure Prometheus and Grafana. Here I am using free tier AWS account with EC2 instances to setup my lab environment.

My architecture diagram will be like below: (simple monitoring architecture)

Installation steps:

Installation of Prometheus on Linux Server:

# sudo hostname grafana

# sudo su

[root@grafana ~]# cd /home/ec2-user/

# mkdir prometheus

# cd prometheus

# 

Step#1 : Install Prometheus 

https://prometheus.io/  --> download --> prometheus-2.43.0.linux-amd64.tar.gz (copy this link)

[root@grafana prometheus]# wget https://github.com/prometheus/prometheus/releases/download/v2.43.0/prometheus-2.43.0.linux-amd64.tar.gz

...

prometheus-2.43.0.linux-amd64 100%[=================================================>]  86.87M  17.2MB/s    in 6.4s

[root@grafana prometheus]# tar -xzvf prometheus-2.43.0.linux-amd64.tar.gz

[root@grafana prometheus]# mv prometheus-2.43.0.linux-amd64 prometheus_setup

[root@grafana prometheus]# # cd prometheus_setup/

# pwd

/home/ec2-user/prometheus/prometheus_setup

Step#2 : Configure Prometheus

# cd /etc/systemd/system

# vi prometheus.service

[Unit]

Description=Prometheus Server

Documentation=https://prometheus.io/docs/introduction/overview/

After=network-online.target

[Service]

User=root

Restart=on-failure

ExecStart=/home/ec2-user/prometheus/prometheus_setup/prometheus --config.file=/home/ec2-user/prometheus/prometheus_setup/prometheus.yml

[Install]

WantedBy=multi-user.target

:wq

Restart Prometheus

[root@grafana prometheus_setup]# pwd

/home/ec2-user/prometheus/prometheus_setup

[root@grafana prometheus_setup]# systemctl restart prometheus.service

verify status

#  systemctl status prometheus.service

prometheus.service - Prometheus Server

     Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; preset: disabled)

     Active: active (running) since Sat 2023-03-25 19:03:44 UTC; 57s ago

Step#3 : Allow default Prometheus port 9090 in security group

security Group -->  select your security group --> Inbound rules --> Edit inbound rules --> Add rule --> Custom TCP --> 9090 --> 0.0.0.0/0 (internet) --> save 

Step#4: Use Prometheus from browser

copy the public IP and used 9090 port

http://54.169.181.69/9090

Now you have Prometheus Home page

Setup Target

Now we can configure target machines to start monitoring. Here at least we can configure one target machine.

Go to Prometheus official website --> Click on node exporter 

https://prometheus.io/download/#node_exporter

copy node_exporter-1.5.0.linux-amd64.tar.gz link

[root@target ~]# cd /home/ec2-user/

[root@target ec2-user]# mkdir node_exporter

[root@target ec2-user]# cd node_exporter

[root@target node_exporter]# wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz

[root@target node_exporter]# tar xzvf node_exporter-1.5.0.linux-amd64.tar.gz

# mv node_exporter-1.5.0.linux-amd64 node_exporter

[root@target node_exporter]# pwd

/home/ec2-user/node_exporter/node_exporter

[root@target node_exporter]# ls -l

total 19336

-rw-r--r--. 1 3434 3434    11357 Nov 29 19:05 LICENSE

-rw-r--r--. 1 3434 3434      463 Nov 29 19:05 NOTICE

-rwxr-xr-x. 1 3434 3434 19779640 Nov 29 18:59 node_exporter

[root@target node_exporter]#

Now create node exporter service file like previous. Name of files are fixed. You can't use custom names.

# touch /etc/systemd/system/node_exporter.service

# vi /etc/systemd/system/node_exporter.service

[Unit]

Description=Node_exporter Server

Documentation=https://prometheus.io/docs/introduction/overview/

After=network-online.target

[Service]

User=root

Restart=on-failure

ExecStart=/home/ec2-user/node_exporter/node_exporter/node_exporter

[Install]

WantedBy=multi-user.target

:wq!

Now restart node exporter and check status:

[root@target node_exporter]# systemctl restart node_exporter.service

[root@target node_exporter]# systemctl status node_exporter.service

● node_exporter.service - Node_exporter Server

     Loaded: loaded (/etc/systemd/system/node_exporter.service; disabled; preset: disabled)

     Active: active (running) since Sat 2023-03-25 19:59:12 UTC; 15s ago

Now attach this target to prometheus server:

Go to prometheus server

Edit prometheus.yml and node/target machine details

[root@grafana prometheus_setup]# pwd

/home/ec2-user/prometheus/prometheus_setup

[root@grafana prometheus_setup]# ls prometheus.yml

prometheus.yml

Note: Use private IP address for the targets as our ab setup resides on same netwrok.

# vi prometheus.yml

 static_configs:

      - targets: ["localhost:9090"]

      - targets: ["172.31.37.33:9100"]

Note: Node exporter service runs on 9100 port and you need to allow this port as well.

Now again Prometheus service.

Now check whether you able to see node/ target details in Prometheus dashboard or not.

Yes we can see it.

Prometheus Home --> Status --> Targets 

Here we can see our target machine/ node machine.

You can monitor targets using PromoSQL (queries)

e.g.,

Use below query in search query box and click on execute and select graph

sum by(mode)(irate(node_cpu_seconds_total{mode!="idle"}[5m])) > 0

Here we can see this CPU load metric values with interval of 5 mins as per our PromoSQL query.

But, this graph is not so much user friendly and informative. For this w can setup and use grafana dash board .

Click on  below grafana dashboard setup link :

http://facedba.blogspot.com/2023/03/grafana-prometheus-setup-part-2.html

Grafana & Prometheus - Setup (part-2)