Watching the Shop

Blackbox Exporter

Now it's time to monitor the application HTTP endpoint. The Blackbox exporter, installed with

wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.19.0/blackbox_exporter-0.19.0.linux-amd64.tar.gz -O blackbox_exporter.tar.gz
tar -zxf blackbox_exporter.tar.gz
./blackbox_exporter &

will be executed on the local machine and will be instructed to perform HTTP requests (either GET or POST), collect the results, and provide metrics on probe success, status code received, latency, and so on. This will be of great use as it will monitor the application from a visitor point of view.

The Blackbox exporter can send requests over a multitude of protocols, including ICMP, SMTP, and SSH. If your goal is to monitor any kind of TCP service from an external point of view, this is the exporter to use.

The default port is 9115. Again, cURL it and append your target_server as the target:

curl http://localhost:9115/probe?target=http://target_server

This command delivers metrics regarding the TLS protocol in use, the latency to perform the request, the SSL certificate expiration time, and more.

Database Exporter

As your last objective, you should take a look at your database engine. This last exporter connects, on your behalf, to the MariaDB server and – you guessed it – fetches metrics.

To begin, log in to the database server and create a user with the necessary permissions. The SQL queries are:

CREATE USER 'exporter'@'%' IDENTIFIED BY 'mysecurepassword' WITH MAX_USER_CONNECTIONS 3;
GRANT SLAVE MONITOR, PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';

Now, to get the exporter itself, enter:

wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.13.0/mysqld_exporter-0.13.0.linux-amd64.tar.gz -O mysqld_exporter.tar.gz
 tar -zxf mysqld_exporter.tar.gz
export DATA_SOURCE_NAME='exporter:mysecurepassword@(target_server:3306)/'
./mysqld_exporter &

The default port is 9104, so you should test it:

curl http://localhost:9104/metrics

You will receive operational, performance, and configuration metrics.

Now that all of your infrastructure elements are configured to produce metrics, you can proceed to collect them and make something out of them.

Installing Prometheus

All of your components are exposing values, so it's time for Prometheus to be deployed and configured to scrape and store them. To install, enter:

wget https://github.com/prometheus/prometheus/releases/download/v2.29.2/prometheus-2.29.2.linux-amd64.tar.gz -O prometheus.tar.gz
tar -zxf prometheus.tar.gz
./prometheus --config.file=prometheus.yml &

Prometheus requires a YAML configuration file that lists all the targets from which you want to collect data. Call this file prometheus.yml and specify it as a parameter during launch (Listing 2).

01 global:
02  scrape_interval: 15s
03  evaluation_interval: 15s
04
05 ########## ALERTING CONFIGURATION ##########
06
07 alerting:
08  alertmanagers:
09  - static_configs:
10    - targets:
11      - localhost:9093
12
13 # Load and evaluate rules in this file every 'evaluation_interval' seconds.
14 rule_files:
15  - alert.rules
16
17 ########### SCRAPING CONFIGURATION #########
18
19 scrape_configs:
20
21 # scrape machine metrics
22  - job_name: node
23    static_configs:
24    - targets: ['target_server:9100']
25
26 # scrape MariaDB metrics
27  - job_name: mysql
28    static_configs:
29    - targets: ['localhost:9104']
30
31 # scrape HTTP endpoint check metrics
32  - job_name: blackbox
33    metrics_path: /probe
34    params:
35      module: [http_2xx]  # Look for a HTTP 200 response.
36    static_configs:
37      - targets:
38        - "http://target_server"    # Target to probe with http.
39    relabel_configs:
40      - source_labels: [__address__]
41        target_label: __param_target
42      - source_labels: [__param_target]
43        target_label: instance
44      - target_label: __address__
45        replacement: localhost:9115  # blackbox exporter's real hostname:port.

When the service is running, you can reach the web interface by pointing your web browser at http://localhost:9090 . Through this user interface Prometheus will allow you, among other things, to:

• perform PromQL queries against the collected data (getting results either in table or graph format),
• check target health (answering the question: Am I actually collecting data?),
• evaluate the alert status, and
• verify the Prometheus integrated TSDB status.

Before moving on, check whether all of your targets are being scraped successfully by heading to http://localhost:9090/targets and making sure that every entry is reported as UP (Figure 2). If that's not the case, you should go back to the specific exporter's configuration steps and verify its correctness and subsequent reachability.

Figure 2: The Prometheus web user interface lists healthy targets.