Lead Image © bowie15, 123RF.com

Collecting application logfiles with Kubernetes

Attentive Co-Driver

Article from ADMIN 66/2021

By Andreas Stolzenberger

Modern scale-out environments with containers make log collection difficult. We present concepts and methods for collecting application logfiles with a sidecar container in Kubernetes environments.

When a large, monolithic virtual machine (VM) with all services is replaced by a group of several cooperating containers, each running only one service, containerized applications scale the number of respective containers as required to improve availability and performance and allow for the individual components of an application to be changed separately (e.g., to complete an update).

Web applications are the classic example of a modern scale-out architecture. A number of containers handle the database back end, and still others host a classic network filesystem for static content. A scaling group runs the application's web front end. Redundant containers with a message bus or a key value store provide communication for all the components involved. The front-end application developers can dynamically change and update their part of the application without disturbing the functionality of the back end or other components.

However, a scale-out architecture also comes with a whole series of challenges – and not just for application developers. Administrators of a scaling environment also need to stay on top of things: both metrics and logs. In monolithic scenarios, the administrator can simply integrate a log collector client into the application VM and provide it with static configurations. However, this scenario no longer works for dynamically scaling environments, where logs come from a great many containers with ever-changing names and addresses.

Most commercial Kubernetes implementations take care of metrics out of the box because the management layer needs information such as the CPU load and memory usage of individual pods and containers (e.g., to trigger scale-up or scale-down tasks). However, logs are a different story because users have to take care of collecting the application logs themselves.