Service discovery, monitoring, load balancing, and more with Consul

Cloud Nanny

Service Registration and Discovery

Service discovery is an important feature in Consul and can be used effectively in many ways to run your services in the dynamic cloud infrastructure. For example, you can use Consul's service discovery and health check features to create your own load balancers, along with the use of Nginx, Apache, or HAProxy.

In the example here, Apache is used as a reverse proxy. Instead of using Amazon Elastic Load Balancing (ELB) between Apache and the application layer, the Apache layer can use Consul's service discovery and health check feature to render application node information dynamically.

To use the service discovery feature, each node should have a config file (Listing 4) to register a particular service running on that node with Consul. Once the consul process is started, the node registers the service MyTomcatApp with Consul and shares the service status with the Consul server. The health check path, port, interval, and timeout details are configured under checks (lines 11-19).

Listing 4

Config File

01 {
02   "service": {
03     "name": "MyTomcatApp",
04     "tags": ["http","lbtype=http","group=default","service=MyTomcatApp"],
05     "enable_tag_override": false,
06     "port": 8080,
07     "meta": {
08       "lb_type": "http",
09       "service": "MyTomcatApp"
10     },
11     "checks": [
12       {
13         "name": "HTTP /myapp on port 8080",
14         "http": "http://localhost:8080/myapp",
15         "tls_skip_verify": true,
16         "interval": "30s",
17         "timeout": "25s"
18       }
19     ]
20   }
21 }

Load Balancing

The service discovery feature also can be used to configure Apache or Nginx proxy load balancers. Apache is set up as a reverse proxy in Listing 5 and the application servers as the back ends in Listing 6. This back-end information can be rendered dynamically through the Consul template:

$ consul-template -template "/etc/httpd/conf/httpd.conf.ctmpl:/etc/httpd/conf/httpd.conf:/sbin/service httpd reload" &

Listing 5

Apache Reverse Proxy

<Proxy balancer://mycluster>
{{range $index, $element := service "MyTomcatApp" -}}
BalancerMember http://{{.Address}}:{{.Port}}
{{end -}}
</Proxy>
ProxyPass / balancer://mycluster/myapp/
ProxyPassReverse / balancer://mycluster/myapp/

Listing 6

Back Ends

<Proxy balancer://mycluster>
BalancerMember http://172.31.20.189:8080
BalancerMember http://172.31.31.130:8080
</Proxy>
ProxyPass / balancer://mycluster/myapp/
ProxyPassReverse / balancer://mycluster/myapp/

As shown in Figure 2, once the Tomcat service starts up, it is registered, and its health status is periodically updated. The config files can be rendered from the consul templates through the consul-template binary and be run in the background to keep the back-end node information up to date.

Figure 2: Service starts up.

For example, if the application's EC2 instance crashes, that node will be marked down on the Consul service list (Figure 3). Once the application health status is marked down (Figure 4), the consul-template daemon running on the Apache node will update the config file and reload Apache. In this way, Apache can handle and keep its proxy balancer config up to date in the dynamically changing cloud infrastructure.

Figure 3: After a crash.

Figure 4: A node is marked unhealthy.

Consul KV Store

Consul's KV store feature lets you manage not only the application configurations, but the configurations of all software components (e.g., Apache, Tomcat, Nginx, databases). Listing 7 manages the Apache configuration. If you want to change the Apache worker process settings, you can just update the Consul KV store, instead of making the changes to a config file and rebuilding the EC2 instance.

Listing 7

KV Store for Apache Config

01 # worker MPM
02 # StartServers: initial number of server processes to start
03 # MaxClients: maximum number of simultaneous client connections
04 # MinSpareThreads: minimum number of worker threads which are kept spare
05 # MaxSpareThreads: maximum number of worker threads which are kept spare
06 # ThreadsPerChild: constant number of worker threads in each server process
07 # MaxRequestsPerChild: maximum number of requests a server process serves
08 <IfModule worker.c>
09 StartServers        {{ keyOrDefault (printf "%s/%s/apache/startservers" $myenv $myapp ) "4" }}
10 MaxClients          {{ keyOrDefault (printf "%s/%s/apache/MaxClients" $myenv $myapp ) "300" }}
11 MinSpareThreads     {{ keyOrDefault (printf "%s/%s/apache/MinSpareThreads" $myenv $myapp ) "25" }}
12 MaxSpareThreads     {{ keyOrDefault (printf "%s/%s/apache/MaxSpareThreads" $myenv $myapp ) "75" }}
13 ThreadsPerChild     {{ keyOrDefault (printf "%s/%s/apache/ThreadsPerChild" $myenv $myapp ) "25" }}
14 MaxRequestsPerChild {{ keyOrDefault (printf "%s/%s/apache/MaxRequestsPerChild" $myenv $myapp ) "0" }}
15 </IfModule>

When consul-template sees the missing keys in the Consul KV store, it just hangs; no output config files will be generated, and ultimately the service startup fails. To avoid this situation, keyOrDefault should specify default values to be used when the keys are missing in the KV store. Variables can be used in the key path. In the example in Listing 7, $myenv and $myapp segregate the keys for particular environment and application spaces. Listing 8 shows the values in the Consul KV store for the example, and the rendered config file is shown in Listing 9.

Listing 8

Consul KV Store

$ consul kv get -recurse
dev/myapp/apache/MaxClients:400
dev/myapp/apache/MaxRequestsPerChild:2
dev/myapp/apache/MaxSpareThreads:80
dev/myapp/apache/MinSpareThreads:30
dev/myapp/apache/ThreadsPerChild:30
dev/myapp/apache/startservers:4

Listing 9

Apache Config File

# Usual KV store comments as in Listing 7
<IfModule worker.c>
StartServers         4
MaxClients         400
MinSpareThreads     30
MaxSpareThreads     80
ThreadsPerChild     30
MaxRequestsPerChild  2
</IfModule>

In Listing 10, only the key is being used. If the requested key in the template exists, the template renders successfully, but if it is missing (e.g., lines 3 and 4), the template rendering just hangs.

Listing 10

Only the key

01 # ...
02 <IfModule worker.c>
03 StartServers        {{ printf "%s/%s/apache/startservers" $myenv $myapp | key }}
04 MaxClients          {{ printf "%s/%s/apache/MaxClients" $myenv $myapp | key }}
05 MinSpareThreads     {{ keyOrDefault (printf "%s/%s/apache/MinSpareThreads" $myenv $myapp ) "25" }}
06 MaxSpareThreads     {{ keyOrDefault (printf "%s/%s/apache/MaxSpareThreads" $myenv $myapp ) "75" }}
07 ThreadsPerChild     {{ keyOrDefault (printf "%s/%s/apache/ThreadsPerChild" $myenv $myapp ) "25" }}
08 MaxRequestsPerChild {{ keyOrDefault (printf "%s/%s/apache/MaxRequestsPerChild" $myenv $myapp ) "0" }}
09 </IfModule>

Listing 11 shows that the dev/myapp/apache/startservers key is missing, so the template rendering will hang. In this case, you literally have to press Ctrl+C to exit the command or kill the consul-template daemon if it runs in the background.

Listing 11

Missing key

$ consul kv get -recurse
dev/myapp/apache/MaxClients:400
dev/myapp/apache/MaxRequestsPerChild:2
dev/myapp/apache/MaxSpareThreads:80
dev/myapp/apache/MinSpareThreads:30
dev/myapp/apache/ThreadsPerChild:30

Mistakes are difficult to avoid when creating or updating a KV store, which will make troubleshooting difficult. To avoid this situation, always use keyOrDefault. On the other hand, how do you troubleshoot missing keys in the template? The consul-template daemon comes with a -log-level option to enable debug logging, but as you can see in the final line of Listing 12, the output is not particularly helpful. All you know from the missing data for 1 dependency message is that one key is missing in the KV store, but you don't know which one. In this situation, the Unix commands awk and grep come in handy:

$ for i in `grep printf httpd.conf1.ctmpl | grep -v keyOrDefault | awk -F"\"" '{print $2}' | awk -F"%s" '{print $3}'`; do echo "key" $i":"; consul kv get dev/myapp$i; done
key /apache/startservers:
Error! No key exists at: dev/myapp/apache/startservers
key /apache/MaxClients:
400

Listing 12

Debug Output Snippet

consul-template -once -template "httpd.conf1.ctmpl:out" -log-level=debug
2018/07/10 01:03:09.580356 [INFO] consul-template v0.19.5 (57b6c71)
2018/07/10 01:03:09.580415 [INFO] (runner) creating new runner (dry: false, once: true)
2018/07/10 01:03:09.580838 [INFO] (runner) creating watcher
2018/07/10 01:03:09.581454 [INFO] (runner) starting
2018/07/10 01:03:09.581511 [DEBUG] (runner) running initial templates
2018/07/10 01:03:09.581532 [DEBUG] (runner) initiating run
2018/07/10 01:03:09.581552 [DEBUG] (runner) checking template 88b8c1d0821f2b9beb481aa808a359bc
2018/07/10 01:03:09.582159 [DEBUG] (runner) was not watching 7 dependencies
2018/07/10 01:03:09.582200 [DEBUG] (watcher) adding kv.block(dev/myapp/apache/startservers)
2018/07/10 01:03:09.582255 [DEBUG] (watcher) adding kv.block(dev/myapp/apache/MaxClients)
...
2018/07/10 01:03:09.582441 [DEBUG] (runner) diffing and updating dependencies
2018/07/10 01:03:09.582476 [DEBUG] (runner) watching 7 dependencies
2018/07/10 01:03:09.588619 [DEBUG] (runner) missing data for 1 dependencies

« Previous 1 2 3 Next »