Case 1: Service Monitoring

From the admin's point of view, it is not complicated to integrate a service into the Consul service database. A simple command to the Consul API is sufficient. Alternatively, services can be stored directly on the hosts in the form of configuration files for Consul. If you want to tell Consul that Keystone, the OpenStack authentication component, is running on port 5000 on the controllers, this shell command will do the trick:

echo '{"service": {"name": "keystone", "tags": ["auth"], "port": 5000}}' | sudo tee /etc/consul.d/keystone.json

Afterward, you just need to restart the Consul agent on the respective host: Consul already knows the Keystone service for this host. Clients can discover the existing services in a directory, either with a DNS request or by asking Consul directly with a RESTful request.

If you want to add monitoring to this configuration, you have several options: an HTTP-based check, a check in which Consul calls arbitrary external scripts, and a classic Ping test, wherein Consul checks that a service responds. You can add the checks again either via a REST request or a JSON configuration on the respective system.

For the Keystone process described above, this might look like:

echo '{"service": {"name": "keystone", "tags": ["auth"], "port": 5000, "check": {"script": "curl localhost:5000 > /dev/null 2>&1", "interval": "10s"}}}' >/etc/consul.d/keystone.json

Assuming Keystone listens on localhost:5000, the described check would attempt to open the Keystone homepage every 10 seconds with an HTTP request. Consul offers its own web interface (Figure 2) that lists the results of the monitoring queries. It also offers the possibility of defining alerts. Consul then escalates according to the established rules, including the ability to send email.

Figure 2: Consul offers simple service monitoring, but if you are dealing with real bare metal, this depth of detail will hardly suffice.

Case 2: Connecting with Prometheus

One advantage of the described monitoring solution with Consul alone is that it manages without bloated monitoring systems like Icinga or Prometheus. On the other hand, at the hardware level, these systems usually exist in some form anyway; usually they were present before Consul came into the picture. Therefore, Consul should not be regarded as competition to the existing monitoring systems, but as an extension of the current setup.

Prometheus – the king of the hill for monitoring, alerting, and trending – can obtain the list of hosts to be monitored directly from Consul, because the current list of all hosts is always available there. At the same time, Prometheus, with its various exporters, now offers much more granular monitoring than would be possible with Consul alone. Especially with bare metal, Consul is often not the best choice; for example, monitoring SMART messages from your storage devices cannot be achieved reliably with Consul. Prometheus, on the other hand, offers a SMART exporter.

The Prometheus developers are aware that the combination of Prometheus and Consul is attractive. In fact, Prometheus now comes with an interface out of the box to dock to an existing Consul cluster and obtain the list of existing hosts and their services (Figure 3).

Figure 3: Prometheus comes with a scraper that can talk to Consul natively.

The rest is just the daily grind: If the admin rolls out Consul along with the Prometheus exporters for collecting metrics on the affected hosts and registers these as local services in Consul, Prometheus receives all the relevant data from Consul and can query all exporters there directly.

At this point, you would handle alerting exclusively via Prometheus; if the Ping metrics for a host are missing, Prometheus can sound the alert (Figure 4). The combination of automation, Consul, and Prometheus allows fully automated monitoring in this scenario, because once you have automatically rolled out Consul on new hosts with all exporters and the necessary service definitions, the host is a valid monitoring target for Prometheus.

Figure 4: The Prometheus Alert Manager takes care of alerting in the Consul-Prometheus team.

If you prefer to use a more classic monitoring system than Prometheus, you still have options: With its template function, Consul can create files on the local filesystem that contain details from the Consul host and service database. Details can be found in the instructions on the HashiCorp GitHub site [6]. A bridge between Icinga, Zabbix, or other monitoring systems and Consul will also work.

Consul in the Cloud

Consul shows its various strengths even better when used as a component of VMs in the cloud, because it was originally developed for this use case. As already mentioned, the monitoring standards in virtual setups are different from those for bare metal, because the focus is always on the availability of a specific service to the outside world. The primary goal of the administrator is to integrate Consul with cloud-integrated load balancers. Several examples illustrate this case.

First, the automatic service detection included in Consul plays a prominent role. If you build your virtual setup in such a way that the IP address of the usable database is always stored in Consul, you can automatically let your clients talk to this IP.

Consul's DNS back end shows its strengths, as well: The individual VMs within the cloud use Consul as the primary service for DNS queries, so the admin simply stores the value for accessing the required database in Consul as the target host for all database queries.

When setting up a virtual environment, however, this scenario means extra work. In the beginning, you have to think the setup through, and then implement it accordingly (e.g., when creating a template for the cloud orchestration solution). This includes providing the database service in Consul with appropriate service checks right from the start and building the database in the background so that several front ends are available (e.g., Galera and MariaDB).