So this is my first post ever. No introduction about me. Just straight into grafana. Why grafana? The first time I heard about it was at work. My task was to build different panels and dashboards to visualize all the data we collect from devices in a data center. I had no prior experience with grafana nor have I heard about it. Instead of watching videos or doing a course about it, I just started building stuff and read the documentation if I had no idea what I am doing.

Grafana is pretty easy. It can get somewhat complicated if you manage a lot of devices or have a very complicated infrastructure but besides that, it’s straightforward.

Docker

I won’t go into that much detail, but if it comes to self-hosting stuff and managing different services and applications, docker is your best friend. I am pretty sure it cannot get easier than with docker. Instead of manually installing dependencies and managing everything by yourself. Docker just does the job. You only need to tinker a bit.

Config

Filestructure:

grafana/

├── docker-compose.yml

├── grafana-storage

└── prometheus.yml

To manage my grafana instance, I use a docker-compose file with three different services. I will go into details later on. Here is the whole docker-compose.yml file:

services:
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - 3000:3000
    networks:
      - monitoring
    volumes:
      - grafana_data:/var/lib/grafana

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - 9090:9090
    networks:
      - monitoring
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'

  node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node_exporter
    restart: unless-stopped
    network_mode: host
    pid: host
    volumes:
      - '/:/host:ro,rslave'
    command:
      - '--path.rootfs=/host'

networks:
  monitoring:
    driver: bridge

volumes:
  prometheus_data:
  grafana_data:

and the prometheus.yml file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['100.115.196.121:9100']

Grafana

Let’s take the first block of this big file:

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - 3000:3000
    networks:
      - monitoring
    volumes:
      - grafana_data:/var/lib/grafana

This part just defines the grafana service. There is nothing really special about it. image: grafana/grafana:latest is the actual image from docker. The container_name could be anything but it obviously makes sense to call it grafana. With restart: unless-stopped we tell docker to restart this image. This is pretty neat because this starts the image after a system reboot. Than we define the port-mapping to be 3000:3000 which makes the WEB-UI available on this port. The rest is just the docker network grafana operates on and the volume where grafana stores important data. These are default options, which can be changed.

Prometheus

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - 9090:9090
    networks:
      - monitoring
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'

Than there is the prometheus section. It follows the same style as the grafana section. We define the image image: prom/prometheus:latest, give the container a name container_name: prometheus, tell docker when to automatically restart the container restart: unless-stopped, map some ports ports: - 9090:9090, tell which network to operate on networks: - monitoring and where to store some persistant data volumes: - .... New is the command section. You can set commands to be executed when running this container. '--config.file=/etc/prometheus/prometheus.yml' tells prometheus where the prometheus.yml file is located. '--storage.tsdb.path=/prometheus' tells prometheus where the storage path is. This section is important for prometheus, since these are prometheus relevant commands. The config command points to this file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['100.115.196.121:9100']

We tell prometheus in which interval to scrape the data and where the data can be found. In this case it’s our node_exporter.

Node_exporter

node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node_exporter
    restart: unless-stopped
    network_mode: host
    pid: host
    volumes:
      - '/:/host:ro,rslave'
    command:
      - '--path.rootfs=/host'

We get all our metrics from node exporter. We tell the node-exporter to target the host system. That’s how we receive all the metrics, which we than can use in grafana/prometheus.

Conclusion

I did not go into that much detail, but i hope this config can maybe help out a bit. Since this is just one docker-compose.yml file, copy all the content and than just run docker compose up -d. Node exporter runs on port 9100, prometheus on 9090 and grafana on 3000. You need to add prometheus to the data sources in grafana and then you are ready to go. Either build your own dashboards and panels or import the preconfiguerd node dashboard with the id 1860. Have fun!