On-Screen

When administrators monitor containers, they first need to consider what expectations they have with respect to the monitoring tool. Is it important to know the load the containers generate on the host resources, or is availability of the containerized services of more interest? Existing monitoring solutions can be used in part for both use cases.

Rough Overview

Docker [1] offers a variety of information related to the containers [2], but only as a starting point for tools with more complex monitoring approaches. Docker does not format the data, but it does support a couple of commands that at least allow administrators to establish rudimentary container monitoring.

The docker ps command lists all active containers – or those that have run – on a host (Figure 1). This command gives you a rough overview of whether containers are still running and an application is therefore available. Additionally, the command reveals other details, such as open and forwarded ports. If the system terminates an application, you get to see a return code.

Figure 1: The docker ps command provides an overview of the active containers.

The docker logs command queries what the applications running in the containers send to stdout and stderr. If you scan these logs, you can also establish a simple monitoring setup. For the command to work, you first need to enable the logging driver, json-file, which is installed by default. Alternatively, you can select syslog or journald as the logging target, which gives you better options for processing the output.

The docker stats command (Figure 2) shows CPU usage, memory usage, and network I/O. The data display is refreshed automatically. If you want to automate the process, you are better off disabling this feature with the --no-stream parameter, so you only see the first result for each case.

Figure 2: docker stats shows rudimentary statistics for the selected container.

If you want to process the data downstream, it might be more useful to tap directly into the source of the resource statistics, cgroups, which organizes processes hierarchically to manage them in groups. Cgroups also allows logging of resource statistics, which then form the basis for the docker stats command.

By default, systemd stores the cgroups data on the filesystem below /sys/fs/cgroup. Each active cgroup subsystem has a folder here, in which the system maps the hierarchical structure of the processes. If you look below /sys/fs/cgroup/memory/system.slice/docker-<container-ID>.scope/memory.stat, for example, you will find the RAM statistics for the container.

The tools provided by Docker give you a rough overview for a host, but it is difficult to use them for widespread monitoring. Admins need to use different tools for this purpose.

Live Container Statistics

One tool comes from the Google cAdvisor [3] project. Originally designed for Google's own lmctfy (let me contain that for you), it now also supports Docker, assuming that Docker uses the libcontainer library (which is enabled by default since version 1.0) as its execution driver. Installing as a Docker container is the easiest way of starting a cAdvisor instance.

The tool visualizes the performance and load data in a web interface that automatically refreshes (Figure 3). The interface shows the results for the entire host or for individual containers; the statistics include CPU load, memory consumption, network traffic, and occupied disk space, giving the admin a rapid overview of the host's load state and the container states.

Figure 3: Google's cAdvisor comes with an attractive web interface and uses InfluxDB as its database.

cAdvisor refreshes data on the fly with graphics that visualize the captured data. By default, the software does not cache the data. To compensate, Google lets you write the performance data to an InfluxDB [4] database, which stores the data for later evaluation; however, this is not handled by cAdvisor. Instead, other tools step in.

One of the tools that generates graphics from the statistics in InfluxDB is Grafana [5], a fork of Kibana [6] used in the ELK (Elasticsearch, Logstash, Kibana) stack. Grafana impresses with a configurable dashboard on which data is visualized. Users can generate and display their own graphs from the existing data. Various chart types are supported (e.g., dot, bar, or line diagrams). Additional options can be configured, such as data point stacking or percentage displays. Plugins let you add additional chart types.

The combination of InfluxDB and Grafana allows you to use cAdvisor for a large number of hosts. Additionally, it consolidates the display of data in graphs or dashboards and outputs them to the screen for a useful overview, helping admins keep track of their Docker landscape at all times.

The development roadmap contains plans to add more advanced features to cAdvisor and proposes container performance optimizations. Moreover, cAdvisor will be able to fine-tune containers automatically for the best possible performance. The load predictions required for this will also be available for cluster managers.

In the Cockpit

The Red Hat Cockpit [7] project not only manages servers, it also adds a container manager to the service. The manager starts and stops containers and provides an overview of the resources the containers use. It captures the generated CPU load, RAM usage, and disk space occupied by the images (Figure 4).

Figure 4: Cockpit shows the load for containers and for the images from which they are derived.

Cockpit monitors CPU and RAM usage and visualizes the results in graphs. It additionally lists the resource usage for each container in an overview. Unfortunately, Cockpit lacks a function for sorting containers by CPU usage or memory load, which means that the list – on a host with many containers – can quickly become cluttered. Additionally, the data is displayed in graphs with graduated blue tones. If you manage a large number of containers, it can be difficult to see which container is using what resources.

Cockpit is not suitable as a monitoring solution in a large environment because it lacks the ability to aggregate multiple hosts. What's more, it has no options for issuing alerts if a container stops running (i.e., if a service is no longer available). On smaller systems, however, Cockpit offers a good, although not very detailed, overview of the active containers and the resources they use.