Backups and Clustering

Influx is clearly ahead of Prometheus when it comes to data storage. The Prometheus approach is simple: Individual Prometheus instances record and store the data on local hard drives. If you have to monitor more than a single instance, you just run several instances, each of which covers a part of the setup. Clustering can be achieved with some restrictions. The developers explicitly advise against mirroring the content of multiple Prometheus instances in a central database. If you want to create backups of Prometheus, you have to set up a second instance that collects the same measurement values as the first.

InfluxDB is far more agile. If you need clustering to interconnect instances of InfluxDB, you get exactly this feature for a price in the commercial version. InfluxDB follows an open source core model, wherein the basic functionality is available free of charge as open source software, and additional features, such as clustering, come at a price.

Data retention is also more flexible with InfluxDB: Instead of just one database, like Prometheus, InfluxDB can contain multiple databases, each using its own retention policy. InfluxDB has a genuine backup function that uses either binary or raw files – just like a classic database such as MariaDB.

InfluxDB also works better for downsampling (i.e., reducing the sampling points of a time series). Instead of defining rules at the beginning to stipulate which data is to be kept, InfluxDB continuously queries data with InfluxQL and stores the result at any location (e.g., in another database). Downsampling is therefore simple and flexible, specifically because the queries can be changed easily during operation.

Alerts by Kapacitor

Although InfluxDB has a number of advantages over Prometheus, unlike Prometheus, it cannot trigger events from stored readings. In a MAT setup, this is a problem, because a database that recognizes a difficulty by date but cannot sound an alarm is not helpful. Enter TICK Stack Kapacitor [5] (Figure 2), the InfluxDB watchdog that can generate alarms according to defined parameters and then pass to various alerting frameworks. If you are looking for the counterpart in Prometheus, the most likely candidate is Alert Manager.

Figure 2: Kapacitor triggers alarms according to various events and thus retrofits InfluxDB with alerting functionality.

Kapacitor can do much more: In addition to detecting anomalies in the metrics, Kapacitor also features automatic detection of the environment in which it runs, which saves configuration work. If Kapacitor notices by itself that it is running in an AWS environment, various alarms that are preconfigured for this scenario take effect. The same applies to the Microsoft Azure Cloud or Kubernetes environments.

Versatility Is Key

If you think Kapacitor is a pure alarm generator, you would be wrong. With a plugin interface, the tool can be extended almost arbitrarily and connected to various external systems. The functionality is then by no means limited to plain vanilla monitoring. Conceivably, you could transfer collected metrics to a connected system for machine learning, to draw conclusions from the available information. Kapacitor is clearly superior to Prometheus' Alert Manager, at least in this respect.

Finally, it is worth mentioning that Kapacitor has his own domain-specific language (DSL), TICKscript, a scripting language with which you can define alarm conditions. The content of a TICKscript should be a task description that contains the condition that leads to an action and describes the action Kapacitor should take. Although this step has a learning curve, the Kapacitor DSL is well documented.