Perfect Fit

On the Lookout

The monitoring software automatically detects applications that provide metrics in Prometheus. To do so, you must first provide specific annotations in key-value format, as described by the example in Listing 6 [3].

[...]
metadata:
    annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9100'
[...]

The next component, node_exporter, makes use of these annotations [16] and collects data about the cluster nodes, such as storage usage, network throughput, and CPU usage. If you want to make sure the software is running on every single node, you need to launch the node_exporter as a DaemonSet [17]. This step simply ensures that a separate instance of node_exporter runs on each node: If a new node is added, Kubernetes automatically calls a new instance.

Add the above-mentioned annotation to node_exporter to help Prometheus find all its instances without further configuration; in this way, you noticeably reduce your manual configuration work.

kubectl create -f 05-node-exporter.yml

For the node_exporter to have access to the information of all host systems, you must provide it with extended privileges by extending the YAML file:

securityContext:
  privileged: true

This privilege gives the node_exporter instance access to the host's resources and lets it read, for example, its /proc filesystem (Listing 7).

[...]
hostPID: true
hostIPC: true
hostNetwork: true
[...]
  volumeMounts:
    - name: proc
      mountPath: /host/proc
[...]
volumes:
  - name: proc
    hostPath:
      path: /proc
[...]

The Aim

After these few steps, Prometheus is ready for use; following this call,

minikube service prometheus --namespace=monitoring

Prometheus delivers the metrics, and Grafana [18] provides a nice graphical overview of the Kubernetes cluster:

kubectl create -f 06-grafana-deployment.yml
kubectl create -f 07-grafana-service.yml
minikube service grafana --namespace=monitoring

A script I wrote helps set up Grafana [11], which creates a data source for Prometheus and imports two useful dashboards for Kubernetes [19] [20]:

./configure_grafana.sh

If you log in with the same usernames and the password admin, you can select the newly created dashboards from the drop-down menu, and you can then browse the information that the Minikube cluster reveals (Figure 3).

Figure 3: The Grafana dashboard helps visualize the data collected during monitoring.

A series of dashboards for Kubernetes [21] is available from the Grafana website [18]; however, some trial and error is in order: Sometimes the developers seem to use other relabeling rules, and all fields remain empty. Adjusting the queries can be quite complex, so these dashboards are more suitable as a good starting point for your own programming.

What Else?

So far, you have gathered a lot of information about the Kubernetes cluster, but there is still more. One interesting Kubernetes subproject named kube-state-metrics [22] retrieves information relating to existing objects from the Kubernetes API and generates new metrics:

kubectl create -f 08-kube-state-metrics-deployment.yml
kubectl create -f 09-kube-state-metrics-service.yml

It provides these metrics in a form compatible with Prometheus [23]. Thus, it can notify administrators, for example, if nodes are not accepting any new pods (unschedulable) or if pods are on the kill list. Complete monitoring in the Kubernetes dashboard is shown in Figure 4.

Figure 4: At the end of the little experiment, the Minikube dashboard displays various values for Kubernetes.