/ docker

How to Monitor a Traefik Reverse Proxy with Prometheus

Recently, I started using the reverse proxy Traefik as a default for my projects. Out of the box, Traefik supports Docker Swarm, Kubernetes, and many others. What's unique about Traefik compared to NGINX or Apache is that it dynamically listens to your Orchestrator like Docker and knows each time a container is added, removed, killed or upgraded, and can generate its configuration automagically. Really? Tell me more.

I'm running Docker as an Orchestrator (obviously). Traefik listens to the Docker API for containers starting or stopping on Docker and it dynamically registers this information with Traefik. How cool is that? Traefik comes out of the box with nice dashboards for your services and the health of these services.

Traefik Architecture

The Traefik dashboards are nice but I want this information in a central location with the rest of my monitoring information. After quite some digging through Traefik configurations, GitHub issues, and tons of side discussions I finally found that Traefik exports metrics in a Prometheus friendly format. Wow, this is great news.

Deploy Traefik

The time has come to start playing with Docker (My favorite time). We will launch 2 separate Docker stacks. Stack 1 will contain our Prometheus stack for monitoring Traefik and Stack 2 will be the Traefik stack. This might be new for some as we will join the Traefik container to the Prometheus network. This allows us to have 2 different docker-compose files yet all containers can interact with each other via Docker networking. Cool, right?

First, we need to clone the Traefik repo

$ git clone https://github.com/vegasbrianc/docker-traefik-prometheus

Let's quickly take a look at the docker-compose.yml file to understand how Traefik will communicate to Prometheus. The docker-compose.yml file is very simple in terms of compose files go.

The command key is where both the Docker endpoint and Prometheus metrics are enabled.

At the bottom of the docker-compose file is the networks section. Here is where we connect to the network of the Prometheus stack. This connects the Traefik container to the Prometheus network or additional networks if desired.

version: '3'

services:
  traefik:
    image: traefik
    command: --web --web.metrics.prometheus --web.metrics.prometheus.buckets="0.1,0.3,1.2,5.0" --docker --docker.domain=docker.localhost --logLevel=DEBUG
    networks:
      - prometheus
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

networks:
 prometheus:
   external:
     name: dockerprometheus_back-tier

Time to start Traefik

From the Traefik directory let's start the Traefik stack.

$ docker-compose up -d

Once Traefik is up and running open a browser tab and open the Trafik Dashboard with http://0.0.0.0:8080Since nothing is running the dashboard should be blank. Click on the health tab at the top of the page and this will display some basic health statistics about Traefik.

Deploy Prometheus

Next, we need to deploy a Prometheus stack. Lucky for you I have a Docker stack containing Docker, Prometheus, and Grafana ready for such a task.

Clone the below repository to your Docker host. This repository contains instructions in case you have never set this up Prometheus before.

$ git clone https://github.com/vegasbrianc/prometheus

Before we can launch the stack we need to adjust the targets which Prometheus will scrape. A target is an endpoint which Prometheus listens to for metrics.

In the Prometheus project directory edit the following file prometheus/prometheus.yml.

Navigate to the end of the file and replace localhost with Traefik's metric endpoint traefik:8080

    static_configs:
     - targets ['traefik:8080','cadvisor:8080','node-exporter:9100']

Once we have made this change we can start the Prometheus stack. From the Prometheus project directory run:

$ docker-compose up -d

Traefiking Magic

Now that both Traefik and Prometheus have been deployed, let's take a look at what Traefik is doing. Open up the Traefik dashboard again and have a look http://0.0.0.0:8080. We should now see that Traefik has automatically populated the Providers tab of the dashboard with all the running containers. Traefik even assigned a local domain name to all of our services (Mind blown).

Traefik Providers

We can now choose a hostname from one of our services and access it directly via the hostname. For demo purposes, let's use Grafana as this is a service we know that is exposed. In your browser, open http://grafana.dockerprometheus.docker.localhost. This should now open the Grafana login page.

If you have never setup Grafana be sure to check the setup instructions in the Prometheus Repo.

Setup the Traefik Grafana Dashboard

We covered a lot of ground. The next steps will start to show how this is all comes together inside Grafana. In the previous section, you configured Grafana and Prometheus as a Data Source.

Several starter dashboards are available via Grafana. I took the liberty of combining a couple dashboards into a new one to offer a little more functionality. Open the Grafana menu -> Dashboards -> Import. You will now be prompted with an import screen. Enter 2870 for the Dashboard ID to import. This will import the Dashboard directly from Grafana.

Next, select the Data Source for Prometheus which we created earlier and click Import
Grafana Dashboard for Traefik Metrics

Great, now you should see the Traefik dashboard inside Grafana. Have a look around and don't forget to write a comment if you have a question.

Grafana Dashboard Traefik Metrics

Brian Christner

Brian Christner

Brian's spent a large portion of his career in the casino industry where he made sure the house always won. Brian is a nominated member of the Docker Captain’s program and a seasoned cloud architect.

Read More