Service discovery, monitoring, load balancing, and more with Consul
Cloud Nanny
The cloud revolution has brought some form of cloud presence to most companies. With this transition, infrastructure is dynamic and automation is inevitable. HashiCorp's powerful tool Consul, described on its website as "… a distributed service mesh to connect, secure, and configure services across any runtime platform and public or private cloud" [1], performs a number of tasks for a busy network. Consul offers three main features to manage application and software configurations: service discovery, health check, and key-value (KV) stores.
Consul Architecture
As shown in Figure 1, each data center has its own set of servers and clients. All nodes within the data center are part of the LAN gossip pool, which has many advantages; for example, you do not need to configure server addresses, because server detection is automatic, and failure detection is distributed among all nodes, so the burden does not end up on the servers only.
The gossip pool is also used as a messaging layer between nodes within the data center to communicate with each other. The server nodes elect one node among themselves as the Leader node, and all the RPC requests from the client nodes sent to other server nodes are forwarded to that Leader node. The leader selection process uses a consensus protocol based on the Raft consensus algorithm.
When it comes to cross-data-center communication, only server nodes from each data center participate in a WAN gossip pool, which uses the gossip protocol to manage membership and broadcast messages to the cluster. Requests are relatively fast, because the data is not replicated across data centers. When a request for a remote data-center resource is made, it's forwarded to one of the remote center's server nodes, and that server node returns the results. If the remote data center is not available, then that resource also won't be available, although it shouldn't affect the local data center.
Raft
For this article, I consider a simple data center, named Datacenter-A
, with three server and two client nodes – a web server node and an application node. I chose three server nodes because of the Raft protocol, which implements the distributed consensus. In a nutshell, the Raft protocol uses the voting process to elect a leader among the server nodes that announce themselves as candidates to become the leader node. To support this voting process, a minimum of three nodes are needed in the server group. A one-server-node system does not need to implement this distributed consensus. In a two-server-node group, no node can gain a majority, because each node votes for itself as soon as it becomes a candidate.
Installation and Configuration
Consul installation is fairly simple and easy: Just extract the binary file into your local filesystem and run it as a daemon:
- After extracting the consul binaries to each node, create the
/etc/consul.d/config.json
configuration file (Listing 1). When theserver
key is settrue
, that particular node acts as the Server node; afalse
value sets that instance to a Client node.
Listing 1
Config for Server Node
01 { 02 "acl_datacenter": "Datacenter-A", 03 "acl_default_policy": "allow", 04 "acl_enforce_version_8": false, 05 "acl_master_token": "", 06 "check_update_interval": "60s", 07 "data_dir": "/var/consul", 08 "datacenter": "Datacenter-A", 09 "disable_update_check": true, 10 "enable_script_checks": true, 11 "enable_syslog": true, 12 "encrypt": "xxxxxxxxxxxxxx", 13 "log_level": "INFO", 14 "node_meta": { 15 "APPLICATION": "consul", 16 "DATACENTER": "Datacenter-A", 17 "NAME": "Datacenter-A-consul", 18 "SERVICE": "consul", 19 "VERSIONEDCONSULREGISTRATION": "false" 20 }, 21 "retry_join": [ 22 "consul.datacenter-a.internal.myprivatedns.net" 23 ], 24 "server": true 25 }
- The Amazon EC2 instance tag can be referenced through the
node_meta
key. - The important
datacenter
key decides the data center that particular node joins. - After creating the config file, you start the consul daemon (Listing 2).
Listing 2
Starting the Daemon
$ /usr/bin/consul agent -ui -config-dir /etc/consul.d -pid-file /var/consul/consul.pid -client 0.0.0.0 >> /var/log/consul/consul.log 2>&1 &
- Once the
consul
process starts, you can use theconsul members
command (Listing 3) to check all the member details of that particular node's data center.
Listing 3
Data Center Members
[ec2-user@ip-172-31-20-189 ~]$ consul members Node Address Status Type Build Protocol DC Segment ip-172-31-16-22 172.31.16.22:8301 alive server 1.2.0 2 datacenter-a <all> ip-172-31-67-70 172.31.67.70:8301 alive server 1.2.0 2 datacenter-a <all> ip-172-31-23-194 172.31.23.194:8301alive server 1.2.0 2 datacenter-a <all> ip-172-31-27-252 172.31.27.252:8301alive client 1.2.0 2 datacenter-a <default> ip-172-31-31-130 172.31.31.130:8301alive client 1.2.0 2 datacenter-a <default>
- To check all the options available with the
consul
command, enterconsul help
. - For proper communication between the nodes, make sure ports 8300, 8301, 8302 (both TCP and UDP), and 8500 are open in the data center for all nodes. The consul UI can be accessed through port 8500.
All six nodes will be started with this Consul agent: three nodes as server nodes and the remaining three nodes as client nodes.
Buy this article as PDF
(incl. VAT)