Stand and Deliver

God of Fire

The next step is to connect Prometheus to Grafana. Again, I'll go down the Docker route with the official documentation [9]. Before beginning, create a configuration file and save it locally as /root/prometheus.yml (Listing 2). Now you can start up the Prometheus container with:

global:
  scrape_interval:     15s # Scrape frequency
  evaluation_interval: 15s # Check rules
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['127.0.0.1:9090'] # Open up Prometheus on TCP port 9090

$ docker run -p 9090:9090 -v /root/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

The command shows a volume is mounted for a specific file at the local /root file path. From that command, you are left with a running Docker container, but because you didn't detach the container (with the -d switch), if you close that terminal, it will stop; anything else you do will have to be in a second terminal. The STDOUT information is useful (Listing 3, abridged).

level=info ts=2020-11-15T19:11:23.675Z caller=web.go:516 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-11-15T19:11:23.697Z caller=head.go:642 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2020-11-15T19:11:23.697Z caller=head.go:656 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.766µs

The logging output looks relatively sane, so you should try to connect the two pieces of software. For proof that Prometheus is indeed working, however, point your browser to its web interface at http://localhost:9090/graph (Figure 5). If you want, you can run queries in the field above the Execute button, called the expression browser. You can find instructions on how this works on the main website [10].

Figure 5: Happiness is a working Prometheus instance.

Jigsaw Puzzle

By now, you should have a recognizable data source that Grafana can ingest. Figure 6 shows where to navigate to choose plugins: Click Plugins and the cog icon in the left sidebar. However, after typing Prometheus into the search box at the top of the page, the entry simply states "Grafana ships with built in support for Prometheus, the open-source service monitoring system and time series database" and encourages you to look at this in more detail on the Data Sources page [11]. In other words you are being politely redirected; therefore, you will instead try and use the Prometheus data source directly.

Figure 6: Plugins for data sources.

If you go back to the settings cog icon and choose Data Sources this time, click Add Data Source (Figure 7). After clicking Prometheus , assuming the Name box is already populated with the name of your data source, you can add the URL of your Prometheus server and continue.

Figure 7: The Prometheus data source.

To find the correct URL, you should open a new terminal (or the Prometheus container will stop) and run the command:

$ docker ps

Once you have found the hash ID for the Prometheus container on the left side (mine started with da33 ), you can find its IP address with:

$ docker inspect da33 | grep IPAddress
"SecondaryIPAddresses": null,
           "IPAddress": "172.17.0.3",
           "IPAddress": "172.17.0.3",

Now you can add that IP address to the URL field in the Grafana GUI as http://172.17.0.3:9090 . Click Save & Test , and if all is going well, you should see a big green banner that says Data source is working . Now you can set up some dashboarding to show off Grafana.

Showing Off

To begin, navigate to the Dashboards tab at the top of the screen and click import on the right for the three example dashboards provided. If you want to delve deeper, you can find the Prometheus Stats JSON file online [12].

Once the import has completed, the name of the dashboard becomes a link that you can mouse over and select with a click. As if by magic, you have multiple metrics on display (Figure 8).

Figure 8: The Prometheus Stats 2.0 dashboard.

To speed up the data refresh frequency, mouse over the top right pull-down menu and select 5s (as opposed to 1m ). If your graphs are lightly populated, fret not; waiting for a few minutes might resolve the issue. If using a tablet, try the "pinch" trick to focus on a specific time period and dig deeper into a specific display and practice with some of the settings to get used to how Grafana interacts.

If you click up near the top left on the name of the dashboard, you will see a list of dashboards you have created. The others might not be set up for you, so don't despair if they offer empty displays.