Requirements

First, we obviously need Docker. Install it using sudo apt install docker-compose and verify, if the installation was successful with docker run hello-world. In my case, I needed to add my user to the docker group to get it running: sudo usermod -aG docker ${USER}.

When Docker is running, we need to prepare it to expose its metrics for Prometheus. Create a file at /etc/docker/daemon.json and edit it as follows:

{
  "metrics-addr": "0.0.0.0:9323"
}

Restart Docker using sudo service restart docker

The Docker Stack

Here is the docker-compose.yml File for the Stack:

networks:
  proxy:
    name: proxy

services:
  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    networks:
      - proxy
    ports:
      - 9090:9090
    volumes:
      - prometheus-config:/etc/prometheus
      - prometheus-data:/prometheus/
    extra_hosts:
      - host.docker.internal:host-gateway
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    ports:
      - 9100:9100
    networks:
      - proxy
      
  cadvisor:
    image: unibaktr/cadvisor:latest
    container_name: cadvisor
    ports:
     - 8083:8080
    networks:
      - proxy
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    devices:
      - /dev/kmsg
    restart: unless-stopped
    
  grafana:
    image: grafana/grafana-enterprise
    container_name: grafana
    restart: unless-stopped
    networks:
     - proxy
    ports:
     - 3002:3000
    volumes:
     - grafana:/var/lib/grafana
     
volumes:
  prometheus-config:
    name: prometheus-config
  prometheus-data:
    name: prometheus-data
  grafana:
    name: grafana

Now spin that up in the background using docker-compose up -d or some Docker-Manager like portainer.io.

Prometheus should list at least the Docker-Endpoint at http://%DOCKER_HOST_IP%:9090/targets?search=.

Prometheus Configuration

Now we need to Configure Prometheus to request metrics from Docker, Node-Exporter and cAdvisor. Edit the Config-File at /var/lib/docker/volumes/prometheus-config/_data/prometheus.yml as follows:

global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: "codelab-monitor"

# Alertmanager configuration
#alerting:
#  alertmanagers:
#    - static_configs:
#        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
#rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: "prometheus"
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "docker"
    static_configs:
      - targets: ["host.docker.internal:9323"]
  - job_name: "node"
    static_configs:
      - targets: ['node-exporter:9100']
  - job_name: "cadvisor"
    static_configs:
      - targets: ['cadvisor:8083']

After a restart, Prometheus should also list the cAdvisor and node-exporter-Endpoints at http://%DOCKER_HOST_IP%:9090/targets?search=.

Grafana Configuration

The Grafana configuration is rather simple, as we can use the Import-Function and the GrafanaLabs. So after the initial login (admin/admin), go to Connections -> Add new Connection and add a DataSource for Prometheus.

Now go to Dashboards -> New -> Import and import the Dashboards Cadvisor exporter (ID: 14282) and Node Exporter Full (ID: 1860)

Et voilá!