Prometheus is a software package that can watch and collect statistics from many different k8s pods and services. It then provides those statistics in a simple html/text format, often referred to as the "scraper page", meaning that it scrapes the statistics and presents them as a simple text-based web page.

Grafana is a data visualization tool, which contains a time series database and graphical web presentation tools. Grafana imports the Prometheus scraper page statistics into it's database, and allows you to create Dashboards of the statistics that are important to you. There are a large number of pre-built dashboards provided by both Grafana and the k8s community, so there are many available to use. And of course, you can customize them as needed or build your own.

Prerequisites

To get started, ensure you have the following:

A Debian 12 server.

A non-root user with sudo administrator privileges.

Installing Grafana on Debian 12

Grafana is an open-source visualization and analytics software. It allows you to query, visualize, alert, and explore your metrics, logs and traces no matter where they are stored. It provides you with tools to turn your time-series database (TSDB) data into insightful graphs and visualizations.

Install Grafana

Complete the following steps to install Grafana from the APT repository:

1. Install the prerequisite packages:

sudo apt-get install -y apt-transport-https software-properties-common wget

2. Import the GPG key:

sudo mkdir -p /etc/apt/keyrings/
sudo wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | tee /etc/apt/keyrings/grafana.gpg > /dev/null

3. To add a repository for stable releases, run the following command:

sudo echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | tee -a /etc/apt/sources.list.d/grafana.list

4. Run the following command to update the list of available packages:

sudo apt-get update

5. To install Grafana OSS, run the following command:

sudo apt-get install grafana -y

Start the Grafana Server

To start the service, run the following commands:

sudo systemctl start grafana-server
sudo systemctl status grafana-server

 

To verify that the service is running, run the following command:

sudo systemctl status grafana-server

Configure the Grafana server to start at boot using systemd

To configure the Grafana server to start at boot, run the following command:

sudo systemctl enable grafana-server.service

Sign in to Grafana

To sign in to Grafana for the first time:

1. Open your web browser and go to http://<your-ip-address>:3000. The default HTTP port that Grafana listens to is 3000 unless you have configured a different port.
2. On the sign-in page, enter admin for both the username and password.
3. Click Sign in. If successful, you’ll see a prompt to change the password.
4. Click OK on the prompt and change your password.

Note: We strongly recommend that you change the default administrator password

Run Grafana behind a reverse proxy

When running Grafana behind a proxy, you need to configure the domain name to let Grafana know how to render links and redirects correctly.

In the Grafana configuration file /etc/grafana/grafana.ini, change the domain to the domain name you’ll be using:

# The public facing domain name used to access grafana from a browser
;domain = localhost

Restart Grafana for the new changes to take effect.

sudo systemctl restart grafana-server

Nginx configuration example

upstream grafana {
  server localhost:3000;
}

server {
    listen 80;
    #listen 443 ssl;

    root /var/www/grafana;
    index index.html index.htm index.nginx-debian.html;

    server_name grafana.localhost;

    #ssl_certificate     /etc/letsencrypt/live/grafana.localhost/fullchain.pem;
    #ssl_certificate_key /etc/letsencrypt/live/grafana.localhost/privkey.pem;

    location / {
        proxy_set_header Host $http_host;
        proxy_pass http://grafana;
    }
}

Reload the NGINX configuration via the command line:

/etc/init.d/nginx reload

Navigate to port 80 on the machine NGINX is running on. The Grafana login page greets you.

 

Installing Prometheus and Node Exporter on Debian 12

Prometheus is an open-source monitoring system for collecting systems metrics. Prometheus uses an exporter for collecting system metrics, such as Node Exporter which allows you to collect metrics for your system.

First, update and refresh your Debian package index by executing the following command as a non-root user

sudo apt update -y

Now install Prometheus and Node Exporter via the apt command below.

sudo apt install prometheus prometheus-node-exporter -y

After installation is finished, both Prometheus and Node Exporter will be running as a systemd service and enabled automatically. The prometheus will have the service prometheus, and the Node Exporter will have the service prometheus-node-exporter.

Verify the prometheus service using the following command.

sudo systemctl is-enabled prometheus 
sudo systemctl status prometheus

The displayed output should reveal that the prometheus service is running and enabled. And by default, it is running on port 9090.

Now verify the prometheus-node-exporter service by executing the following command.

sudo systemctl is-enabled prometheus-node-exporter
sudo systemctl status prometheus-node-exporter

The displayed output will be similar, which reveals the prometheus-node-exporter service is running and enabled. The default port for Node Exporter is 9100.

Next, run the prometheus command below to verify the Prometheus version.

prometheus --version

Then, check the binary path of promtool and its version like this. The promtool is a command line for managing the Prometheus monitoring system.

which promtool
promtool --version

Lastly, verify the Node Exporter binary path prometheus-node-exporter and its version using the following command.

which prometheus-node-exporter
prometheus-node-exporter --version

Configuring Prometheus

After installing Prometheus and Node Exporter, the next step you will configure Prometheus by editing the default configuration file prometheus.yml which is located in the prometheus configuration directory /etc/prometheus.

Open the file with your choice of editor, here we are using the always installed vi editor

sudo vi /etc/prometheus/prometheus.yml

Within the scrape_configs section, add a new job prometheus with the target endpoint <your-ip-address>:9090, which is the Prometheus server itself.

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["<your-ip-address>:9090"]

Save the file and exit the editor when you're finished by typing :wq!

Now run the systemctl command below to restart the prometheus service and apply the changes you've made.

sudo systemctl restart prometheus

Now launch your web browser and visit your Prometheus installation, such as http://<your-ip-address>:9090

If everything is working correctly, you should see the Prometheus dashboard

Click on the menu Status > Targets to get the list of endpoints of the target monitoring system. You should see the endpoint prometheus with the status Up.

Lastly, you can also verify the Prometheus metrics by visiting the path URL /metrics, such as http://<your-ip-address>:9090/metrics

Adding Node Exporter to Prometheus

Now that you've configured Prometheus, the next step is to add Node Exporter to your Prometheus installation. The Node Exporter will gather metrics from your server. To achieve that, you must complete the following:

• Checking Node Exporter Status: This will ensure that Node Exporter is running before going further.
• Adding New Job to Prometheus: This will show you how to add a target monitoring system to Prometheus.

Now let's begin.

Checking Node Exporter Status

Before adding Node Exporter to Prometheus, you must ensure that Node Exporter is running without any errors. This can be achieved by checking the prometheus-node-exporter service status, checking the port 9100 that is used by Node Exporter, and accessing the Node Exporter metrics from your browser.

Check the prometheus-node-exporter service status by executing the following command.

sudo systemctl status prometheus-node-exporter

Now run the command below to ensure port 9100 is in the LISTEN state, which the Node Exporter uses.

ss -tulpn | grep 9100

Lastly, open your web browser and visit the Node Exporter metrics URL, such as http://<your-ip-address>:9100/metrics

Adding New Job to Prometheus

Now that the Node Exporter is running, you're ready to add Node Exporter to the Prometheus.

Open the Prometheus configuration /etc/prometheus/prometheus.yml using the following vi editor command.

sudo vi /etc/prometheus/prometheus.yml

Within the scrape_configs section, add a new job prometheus-node-exporter with the endpoint of the Node Exporter metrics like the following.

  - job_name: 'prometheus-node-exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['<your-ip-address>:9100']

Save the file and exit the editor when finished by typing :wq!

Next, execute the following systemctl command to restart the prometheus service and apply the changes.

sudo systemctl restart prometheus

Lastly, back to the Prometheus dashboard, then click Status > Targets menu. If everything goes well, you should see the Node Exporter on the target endpoint.

Basic Usage of Prometheus Dashboard

In this section, you will learn the basic query of Prometheus and Node Exporter, which can be executed from the Prometheus dashboard.

Type the query node_os_info and click the Execute button. The node_os_info is a query language provided by Node Exporter for checking operating system details.

On the Console section, you should see detailed host information

Next, type another query such as node_memory_Active_bytes to check active memory on the target server.

Furthermore, you can also use PromQL (Prometheus Query Language) to get specific data, such as node_memory_Active_bytes[5] that will show you data for the last 5 minutes.