Time-series-based monitoring with Prometheus

Lighting Up the Dark

Instrumenting Your Own Application

The Prometheus project offers ready-made libraries for some programming languages (e.g., Go, Java, Ruby, Python, and others) to simplify the instrumentation of your own applications with metrics. Listing 2 shows an example of how to instrument a Python application for Prometheus [22]. After you launch the demo application, you check out the generated metrics at http://localhost:8000 .

Listing 2

Instrumenting a Python Application

01 from prometheus_client import start_http_server, Summary
02 import random
03 import time
04
05 # Create a metric to track time spent and requests made.
06 REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')
07
08 # Decorate function with metric.
09 @REQUEST_TIME.time()
10 def process_request(t):
11     """A dummy function that takes some time."""
12     time.sleep(t)
13
14 if __name__ == '__main__':
15     # Start up the server to expose the metrics.
16     start_http_server(8000)
17     # Generate some requests.
18     while True:
19         process_request(random.random())

Sending metrics from batch jobs to Prometheus also is very easy. An example is provided by speedtest_exporter [23]. The simple Perl script processes the output of another program (e.g., speedtest_cli) and uses curl to send the results to the Pushgateway. Prometheus picks up the collected data from there. To do so, it uses an HTTP POST to send a string with one or more metrics line by line to the Pushgateway. The metrics sent by speedtest_exporter, for example, look like this:

speedtest_latency_ms 22.632
speedtest_bits_per_second {direction="downstream"} 52650000speedtest_bits_per_second{direction="upstream"} 29170000

The URL specifies a unique job name, such as http://localhost:9091/metrics/job/speedtest_exporter .

Conclusions

Prometheus offers simplicity and the versatility of PromQL. For developers who want to monitor their applications or container environment, Prometheus is certainly a very good choice. For example, it can monitor a complete Kubernetes cluster easily and simply [24].

In business use, however, you need to put up with a few quirks: Long-term archiving of metrics is still a construction site, and you do need to build any required encryption or access controls around Prometheus. If you can live with this, what you get is a powerful and effective tool that will also almost certainly address the above-mentioned shortcomings in the next few years.

Infos

Nagios: https://www.nagios.org
Icinga: https://www.icinga.com
Prometheus: https://prometheus.io
Kubernetes: https://kubernetes.io
Official Prometheus announcement: https://developers.soundcloud.com/blog/prometheus-monitoring-at-soundcloud
Prometheus on GitHub: https://github.com/prometheus
CNCF: https://www.cncf.io
Prometheus download: https://prometheus.io/download/
Brian Brazil's blog: https://www.robustperception.io/blog/
Keynotes at PromCon 2016: https://www.youtube.com/playlist?list=PLoz-W_CUquUlCq-Q0hy53TolAhaED9vmU
Weaveworks blog: https://www.weave.works/blog/ (enter "Prometheus" in the Search box)
node_exporter: https://github.com/prometheus/node_exporter
List of exporters: https://prometheus.io/docs/instrumenting/exporters/
Jobs and instances: https://prometheus.io/docs/concepts/jobs_instances/
Service discovery: https://prometheus.io/docs/operating/configuration/
Project architecture diagram: https://github.com/prometheus/prometheus/blob/master/documentation/images/architecture.svg
Grafana: https://prometheus.io/docs/visualization/grafana/
Prebuilt Grafana dashboards for Prometheus: https://grafana.com/dashboards?dataSource=prometheus
Prometheus and FritzBox: https://labs.consol.de/monitoring/2017/03/08/prometheus-und-die-fritzbox.html [in German]
Best practices in the use of Prometheus: https://prometheus.io/docs/practices/naming/
PromQL functions: https://prometheus.io/docs/querying/functions/
Demo application instrumenting Python: https://github.com/prometheus/client_python
speedtest_exporter: https://github.com/RichiH/speedtest_exporter
Monitoring Kubernetes with Prometheus: https://coreos.com/blog/prometheus-and-kubernetes-up-and-running.html

The Author

Michael Kraus is a senior consultant for open source monitoring at ConSol in Munich, Germany. Since 2005, he has developed and managed the monitoring environments with Nagios and its successors. His main focus is integrating new components in a meaningful monitoring environment based exclusively on open source software.

« Previous 1 2 3 4