The cloud revolution has brought some form of cloud presence to most companies. With this transition, infrastructure is dynamic and automation is inevitable. HashiCorp's powerful tool Consul, described on its website as "… a distributed service mesh to connect, secure, and configure services across any runtime platform and public or private cloud" [1], performs a number of tasks for a busy network. Consul offers three main features to manage application and software configurations: service discovery, health check, and key-value (KV) stores.

Consul Architecture

As shown in Figure 1, each data center has its own set of servers and clients. All nodes within the data center are part of the LAN gossip pool, which has many advantages; for example, you do not need to configure server addresses, because server detection is automatic, and failure detection is distributed among all nodes, so the burden does not end up on the servers only.

Figure 1: Consul's high-level architecture within and between data centers (image source: HashiCorp [2]).

The gossip pool is also used as a messaging layer between nodes within the data center to communicate with each other. The server nodes elect one node among themselves as the Leader node, and all the RPC requests from the client nodes sent to other server nodes are forwarded to that Leader node. The leader selection process uses a consensus protocol based on the Raft consensus algorithm.

When it comes to cross-data-center communication, only server nodes from each data center participate in a WAN gossip pool, which uses the gossip protocol to manage membership and broadcast messages to the cluster. Requests are relatively fast, because the data is not replicated across data centers. When a request for a remote data-center resource is made, it's forwarded to one of the remote center's server nodes, and that server node returns the results. If the remote data center is not available, then that resource also won't be available, although it shouldn't affect the local data center.

Raft

For this article, I consider a simple data center, named Datacenter-A, with three server and two client nodes – a web server node and an application node. I chose three server nodes because of the Raft protocol, which implements the distributed consensus. In a nutshell, the Raft protocol uses the voting process to elect a leader among the server nodes that announce themselves as candidates to become the leader node. To support this voting process, a minimum of three nodes are needed in the server group. A one-server-node system does not need to implement this distributed consensus. In a two-server-node group, no node can gain a majority, because each node votes for itself as soon as it becomes a candidate.

Installation and Configuration

Consul installation is fairly simple and easy: Just extract the binary file into your local filesystem and run it as a daemon:

• After extracting the consul binaries to each node, create the /etc/consul.d/config.json configuration file (Listing 1). When the server key is set true, that particular node acts as the Server node; a false value sets that instance to a Client node.

01 {
02     "acl_datacenter": "Datacenter-A",
03     "acl_default_policy": "allow",
04     "acl_enforce_version_8": false,
05     "acl_master_token": "",
06     "check_update_interval": "60s",
07     "data_dir": "/var/consul",
08     "datacenter": "Datacenter-A",
09     "disable_update_check": true,
10     "enable_script_checks": true,
11     "enable_syslog": true,
12     "encrypt": "xxxxxxxxxxxxxx",
13     "log_level": "INFO",
14     "node_meta": {
15         "APPLICATION": "consul",
16         "DATACENTER": "Datacenter-A",
17         "NAME": "Datacenter-A-consul",
18         "SERVICE": "consul",
19         "VERSIONEDCONSULREGISTRATION": "false"
20     },
21     "retry_join": [
22         "consul.datacenter-a.internal.myprivatedns.net"
23     ],
24     "server": true
25 }

• The Amazon EC2 instance tag can be referenced through the node_meta key.
• The important datacenter key decides the data center that particular node joins.
• After creating the config file, you start the consul daemon (Listing 2).

$ /usr/bin/consul agent -ui -config-dir /etc/consul.d -pid-file /var/consul/consul.pid -client 0.0.0.0 >> /var/log/consul/consul.log 2>&1 &

• Once the consul process starts, you can use the consul members command (Listing 3) to check all the member details of that particular node's data center.

[ec2-user@ip-172-31-20-189 ~]$ consul members
Node             Address           Status Type   Build Protocol DC            Segment
ip-172-31-16-22  172.31.16.22:8301 alive  server 1.2.0 2        datacenter-a  <all>
ip-172-31-67-70  172.31.67.70:8301 alive  server 1.2.0 2        datacenter-a  <all>
ip-172-31-23-194 172.31.23.194:8301alive  server 1.2.0 2        datacenter-a  <all>
ip-172-31-27-252 172.31.27.252:8301alive  client 1.2.0 2        datacenter-a  <default>
ip-172-31-31-130 172.31.31.130:8301alive  client 1.2.0 2        datacenter-a  <default>

• To check all the options available with the consul command, enter consul help.
• For proper communication between the nodes, make sure ports 8300, 8301, 8302 (both TCP and UDP), and 8500 are open in the data center for all nodes. The consul UI can be accessed through port 8500.

All six nodes will be started with this Consul agent: three nodes as server nodes and the remaining three nodes as client nodes.