Integrating Consul with External Services

Even if you have configured your virtual Consul-based environment perfectly for high availability and secured it against failure, you still have one problem left to solve: seamless scalability. Although not impossible, it might require you to rethink various points.

Many clouds offer Load Balancing as a Service (LBaaS). On the one hand, this is practical because the cloud architect no longer has to worry about the load balancer; instead, the cloud provisions it completely automatically. On the other hand, the load balancer system in such a setup is usually hidden from the eyes of the user and can only be configured via an API, although it is usually just a normal Linux VM under the hood.

If you want to use Consul to build an automatically scalable and monitored setup, it makes sense to take the somewhat painstaking path of building your load balancer as a separate VM (e.g., with HAProxy). The part that automatically starts new VMs when the system load is correspondingly high remains in the cloud.

If the newly started VMs automatically become part of the Consul cluster with the additional web front ends, then HAProxy can be built using the Consul template function so that it automatically transfers new VMs into the load balancer, as described in detail online [7].

The big advantage of such a solution is now obvious: A setup based on this principle can be used in virtually any cloud, because it does not depend on cloud-specific functions.

Conclusions

Monitoring in the cloud works differently from conventional monitoring systems – at least with regard to VMs. In the case of bare metal, things haven't changed that much: The admin still wants to know when individual servers fail, to take quick countermeasures. The combination of Consul, Prometheus, and Grafana offers ideal conditions (Figure 5). In the virtual environment, however, the availability of services, rather than individual systems, has become far more the focus.

Figure 5: All services registered with Consul can be monitored conveniently and automatically with Prometheus. Grafana then handles the visualization of the performance data.

Consul is good for both scenarios: At the data center, it supports automated monitoring of huge networks of systems with manageable overhead. In virtual environments, Consul is the link between the VMs running services and the users accessing those services.