==================================
Monitoring Dashboard of the Server
==================================

Prerequisites
=============

The **Prometheus Node Exporter** exposes a wide variety of hardware- and kernel-related metrics. Then add the metrics endpoint as a target to your **Prometheus server**. 
You can do this by editing the prometheus configuration file on your prometheus server.  

source: https://pimylifeup.com/raspberry-pi-prometheus/
https://prometheus.io/docs/guides/node-exporter/


1. Install Node_Exporter on RaspberryPi
---------------------------------------

Step 1 Download Node Exporter
_____________________________


In this step we are simply downloading a release of the node exporter. Releases are published on projects releases page on Github

.. note:: 

    The node exporter release binaries are architecture specific. 


Log into your Raspberry Pi and run the following wget command to download node exporter for the ArmV7 architecture.

.. code-block:: bash

    wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-armv7.tar.gz

Now un-tar the release using this command.

.. code-block:: bash

    tar -xvzf node_exporter-0.18.1.linux-armv7.tar.gz

This will un-tar the files into a sub-directory that looks like this.

.. code-block:: bash

    node_exporter-1.8.1.linux-armv7/
    node_exporter-1.8.1.linux-armv7/node_exporter
    node_exporter-1.8.1.linux-armv7/NOTICE
    node_exporter-1.8.1.linux-armv7/LICENSE

Step 2 Install node_exporter binary and create required directories
___________________________________________________________________

The only file we need out of the expanded tarball is the node_exporter binary. Copy that file to /usr/local/bin.

.. code-block:: bash

    sudo cp node_exporter-0.18.1.linux-armv7/node_exporter /usr/local/bin

Use the chmod command to make the node_exporter binary executable.

.. code-block:: bash

    chmod +x /usr/local/bin/node_exporter

Create a service account for the node_exporter.

.. code-block:: bash

    sudo useradd -m -s /bin/bash node_exporter

Make a directly in /var/lib/ that will be used by the node_exporter. Change the ownership to the service account we just created.

.. code-block:: bash

    sudo mkdir /var/lib/node_exporter
    chown -R node_exporter:node_exporter /var/lib/node_exporter

You have completed the node_exporter binary installation and setup of required directories!


Step 3 Setup systemd unit file
______________________________

Next step, setting up the unit file. The unit file will allow us to control the service via the systemctl command. Additionally it will ensure node_exporter starts on boot.

Create a file called node_exporter.service in the /etc/sytemd/system directory. The full path to the file should be:

.. code-block:: bash

    /etc/systemd/system/node_exporter.service

Put the following contents into the file:

.. code-block:: bash

    [Unit]
    Description=Node Exporter

    [Service]
    # Provide a text file location for https://github.com/fahlke/raspberrypi_exporter data with the
    # --collector.textfile.directory parameter.
    ExecStart=/usr/local/bin/node_exporter --collector.textfile.directory /var/lib/node_exporter/textfile_collector

    [Install]
    WantedBy=multi-user.target

Reload systemd, enable and start the service.

.. code-block:: bash

    sudo systemctl daemon-reload 
    sudo systemctl enable node_exporter.service
    sudo systemctl start node_exporter.service

Use the `systemctl status node_exporter` command to verify.

2. Install Prometheus on RaspberryPi
------------------------------------

Step 1 Download Prometheus
__________________________

We can download the pre-compiled version of Prometheus for the ARMv7 architecture.

To download this software, run the following command on your software.

.. code-block:: bash

    wget https://github.com/prometheus/prometheus/releases/download/v2.22.0/prometheus-2.22.0.linux-armv7.tar.gz

This command will use wget to download the 2.22.0 version of Prometheus to your Raspberry Pi.

Extract the binaries outside of the archive you downloaded by running the following command.

.. code-block:: bash

    tar xfz prometheus-2.22.0.linux-armv7.tar.gz

Our next step is to rename the extracted folder to remove the version from the folder name.

Doing this makes it easier to reference the files within the directory. Use the mv command to rename the directory to prometheus.

.. code-block:: bash

    mv prometheus-2.22.0.linux-armv7/ prometheus/


Your locally running Prometheus instance needs to be properly configured in order to access Node Exporter metrics. 
The following prometheus.yml example configuration file will tell the Prometheus instance to scrape, and how frequently, from the Node Exporter via localhost:9100:

.. code-block:: bash
        
    global:
    scrape_interval: 15s

    scrape_configs:
    - job_name: node
    static_configs:
    - targets: ['localhost:9100']
    

Step 2 Setting up a Service for Promotheus
__________________________________________

To create a service, we need to create a new file within the “/etc/systemd/system/” directory.

This directory is where services are handled by default.

.. code-block:: bash

    $ sudo nano /etc/systemd/system/prometheus.service

Within this file, enter the following text.

The text defines how the service works and how it should run the Prometheus software.

.. code-block:: bash

    [Unit]
    Description=Prometheus Server
    Documentation=https://prometheus.io/docs/introduction/overview/
    After=network-online.target

    [Service]
    User=user
    Restart=on-failure

    ExecStart=/home/fabgt/prometheus/prometheus \
    --config.file=/home/user/prometheus/prometheus.yml \
    --storage.tsdb.path=/home/user/prometheus/data

    [Install]
    WantedBy=multi-user.target

With the way this service file is written, it will run the Prometheus software on your Raspberry Pi once the network has come online.

Upon starting up, it will run the Prometheus executable located at “/home/pi/prometheus/prometheus“.

We pass in both the config file location and a storage location for the database that the monitoring software requires.

If you ever need to modify the config file, you can find it at “/home/pi/prometheus/prometheus.yml“.

With the new service created, we can now go ahead and enable it and start it

.. code-block:: bash

    sudo systemctl enable prometheus
    sudo systemctl start prometheus

Verify with `status`:

.. code-block:: bash

    $ sudo systemctl status prometheus
    ● prometheus.service - Prometheus Server
        Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; preset: enabled)
        Active: active (running) since Sun 2024-06-16 15:03:34 CEST; 4s ago
        Docs: https://prometheus.io/docs/introduction/overview/
    Main PID: 2613 (prometheus)
        Tasks: 9 (limit: 9252)
            CPU: 77ms
        CGroup: /system.slice/prometheus.service
                └─2613 /home/fabgt/prometheus/prometheus --config.file=/home/fabgt/prometheus/prometheus.yml --storage.tsdb.path=/home/fabgt/prometheu>

    Jun 16 15:03:34 pi prometheus[2613]: ts=2024-06-16T13:03:34.272Z caller=head.go:684 level=info component=tsdb msg="Replaying WAL, this may take a >
    Jun 16 15:03:34 pi prometheus[2613]: ts=2024-06-16T13:03:34.278Z caller=head.go:755 level=info component=tsdb msg="WAL segment loaded" segment=0 m>
    Jun 16 15:03:34 pi prometheus[2613]: ts=2024-06-16T13:03:34.279Z caller=head.go:755 level=info component=tsdb msg="WAL segment loaded" segment=1 m>
    Jun 16 15:03:34 pi prometheus[2613]: ts=2024-06-16T13:03:34.279Z caller=head.go:792 level=info component=tsdb msg="WAL replay completed" checkpoin>
    Jun 16 15:03:34 pi prometheus[2613]: ts=2024-06-16T13:03:34.281Z caller=main.go:1040 level=info fs_type=EXT4_SUPER_MAGIC
    Jun 16 15:03:34 pi prometheus[2613]: ts=2024-06-16T13:03:34.282Z caller=main.go:1043 level=info msg="TSDB started"
    Jun 16 15:03:34 pi prometheus[2613]: ts=2024-06-16T13:03:34.282Z caller=main.go:1224 level=info msg="Loading configuration file" filename=/home/fa>
    Jun 16 15:03:34 pi prometheus[2613]: ts=2024-06-16T13:03:34.282Z caller=main.go:1261 level=info msg="Completed loading of configuration file" file>
    Jun 16 15:03:34 pi prometheus[2613]: ts=2024-06-16T13:03:34.282Z caller=main.go:1004 level=info msg="Server is ready to receive web requests."
    Jun 16 15:03:34 pi prometheus[2613]: ts=2024-06-16T13:03:34.282Z caller=manager.go:995 level=info component="rule manager" msg="Starting rule mana>

Start the web interface:

localhost:9090/graph



Exploring Node Exporter metrics through the Prometheus expression browser

Now that Prometheus is scraping metrics from a running Node Exporter instance, you can explore those metrics using the Prometheus UI (aka the expression browser). Navigate to localhost:9090/graph in your browser and use the main expression bar at the top of the page to enter expressions. The expression bar looks like this:

Prometheus expressions browser

Metrics specific to the Node Exporter are prefixed with node and include metrics like node_cpu_seconds_total and node_exporter_build_info.

3. Install Grafana on RaspberryPi
---------------------------------

Grafana is a fast tool as it offloads most of the demanding tasks, such as the rendering of graphs to the client. This allows the software to focus on providing the data for graphs and helps make it a perfect fit for the Raspberry Pi as there is less data to process.

The Grafana software has support for a wide way of different data inputs and even allows you to define alert rules on essential metrics so you can be automatically notified if something is not right.

One of these potential Grafana data inputs is the popular InfluxDB. InfluxDB is a popular database to accompany Grafana as it is fast and is “time series” based meaning each record comes with a timestamp.

.. note::

    To change the default 3000 port, edit the .ini configuration file located at /etc/grafana/grafana.ini 
    Grafana uses semicolons (the ; char) to comment out lines in a .ini file. You must uncomment each line in the custom.ini or the grafana.ini file that you are modify by removing ; from the beginning of that line. Otherwise your changes will be ignored.

For example:

.. code-block:: bash

    # The HTTP port  to use
    ;http_port = 3000

1 Add the Grafana repository
____________________________

To install Grafana to the Raspberry Pi, we need to add the Grafana package repository.

Before we can add the repository, we have to add the APT key. The APT key is used to verify the packages actually came from the Grafana package server and have been signed correctly.

To add the Grafana APT key to your Raspberry Pi’s keychain, run the following command.

.. code-block:: bash

    curl https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/grafana-archive-keyrings.gpg >/dev/null

With the key added, we can now safely add the Grafana repository to our Pi’s list of packages sources.

Use the following command on your Raspberry Pi to add the repository to the list.

.. code-block:: bash

    echo "deb [signed-by=/usr/share/keyrings/grafana-archive-keyrings.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

When you run and update your Raspberry Pi will now automatically read from the Grafana repository for packages.

2 Install and configure Grafana
_______________________________

.. code-block:: bash

    sudo apt update
    sudo apt install grafana

Our next step is to get Grafana to start at startup. Luckily for us, Grafana comes with a systemd service file.

To enable Grafana to start at boot, all we need to do is run the following command.

.. code-block:: bash

    sudo systemctl enable grafana-server

This command will tell the systems service manager to enable the service file called “grafana-server.service”.

.. code-block:: bash

    sudo systemctl start grafana-server

Connect to the web interface on port `3000` and start by adding a new source (Prometheus) add a new Dashboard and new Visualizations (instruments) to it.