Watching the Shop

Alerting with Alertmanager

Prometheus allows for PromQL queries to become conditions and, on those bases, trigger alarms (alerts) to warn about abnormal states by creating rules that are periodically evaluated against the content of the Prometheus TSDB. Listing 3 shows three rules that provide notifications about the most common service downtimes:

01 groups:
02 - name: Web application alerts
03  rules:
04    - alert: target_server root disk partion is getting full
05      expr: 100 - ((node_filesystem_avail_bytes{job="node_exporter",mountpoint="/",fstype!="rootfs"} * 100) / node_filesystem_size_bytes{job="node_exporter",mountpoint="/",fstype!="rootfs"}) > 80
06      for: 30m
07      labels:
08        severity: warning
09      annotations:
10        summary: Disk space on target_server is getting full, please check and free some space.
11
12    - alert: Webserver unavailable
13      expr: probe_http_status_code{job="blackbox_exporter"} != 200
14      for: 1m
15      labels:
16        severity: critical
17      annotations:
18        summary: Webserver is responding to anything other than 200 for more than 1 minute
19
20    - alert: MariaDB unavailable
21      expr: mysql_up{job="mysql_exporter"} != 1
22      for: 1m
23      labels:
24        severity: critical
25      annotations:
26        summary: The database is not responding for more than 1 minute

• disk space running out on target_server,
• target_server web server responding to something other than the 200/OK HTTP status, and
• target_server MariaDB being unable to respond to queries.

Now create a file called alert.rules in the same directory as the prometheus.yml config file so that Prometheus will pick it up automatically. Once the file is created, wait a few seconds and check http://localhost:9090/alerts again to ensure that the rules have been parsed.

As a final touch, you definitely want the alerts to be forwarded through some communication medium (e.g., email, instant messaging, etc.). This job is done by the Alertmanager companion service. To download, enter:

wget https://github.com/prometheus/alertmanager/releases/download/v0.23.0/alertmanager-0.23.0.linux-amd64.tar.gz -O alertmanager.tar.gz
tar -zxf alertmanager.tar.gz
./alertmanager --config.file=alertmanager.yml &

The sample YAML config file in Listing 4 sends email by Alertmanager when an alert fires. A user interface will be available at http://localhost:9093 .

01 global:
02  resolve_timeout: 5m
03
04 route:
05  group_by: ['alertname']
06  group_wait: 10s
07  group_interval: 10s
08  repeat_interval: 10s
09  receiver: 'email'
10 receivers:
11 - name: 'email'
12  email_configs:
13  - to: 'helpdesk@mycompany.com'
14    from: 'monitoring@mycompany.com'
15    smarthost: smtp.mycompany.com:587
16    auth_username: 'monitoring@mycompany.com'
17    auth_identity: 'monitoring@mycompany.com'
18    auth_password: 'mysecuremailpassword'

To finish the alerting part, "sabotage" your system (e.g., stop the MariaDB service; Figure 3) and check whether Prometheus reports the alert as "fired." Immediately afterward, Alertmanager takes care of sending the email.

Figure 3: Alerts fire when one of the alerting rules is matched.

Dashboards with Grafana

Previously, I wrote about Grafana Loki [3] and that, once data is available, a graphical tool is needed to visualize it and make it more meaningful for your purposes. Grafana natively supports Prometheus as a data source, as well. Grafana is not available as a single binary, so you use Docker to run an instance:

docker run --network host -d --name grafana grafana/grafana

Now point your browser to http://localhost:3000 (admin:admin), click on the Settings wheel in the left panel, choose Data Sources | Add data source , select Prometheus , and set http://localhost:9090 as the URL. To finish, click Save & Test . Creating Grafana graphs is a topic that would require an article on its own [4].

To kickstart your engine, limit yourself to adding a premade dashboard that visualizes data coming from Node Exporter (machine metrics): Click on the plus (+ ) symbol on the left, then Import . Once there, enter 13978 in the Import via grafana.com textbox. Click on Load and voilà – a full-blown set of graphs reporting useful information about the target_server (Figure 4).

Figure 4: Machine metrics graphs with Grafana.

The same approach can be taken for the other two exporters by heading to https://grafana.com/dashboards and searching for mysql_exporter or blackbox_exporter .

Conclusions

With little effort you can set up a scrape-based monitoring and evaluation system that can help you in two ways: by giving you a complete overview of your web application infrastructure and making your first steps toward the fascinating world of observability.

Throughout the article, for the sake of simplicity, I executed all services by just running binaries in the background.

If you're planning a stable deployment, the best way to continue would be to use official Docker images or to install each application through your favorite distro package manager.

The Author

Stefano Chittaro manages multicloud deployments with a special focus on automation and observability. Sometimes he rants about technology on https://nevarsin.blog.