Small Packages

Among the number of burgeoning Kubernetes distributions available today is the excellent production-ready K3s [1], which squeezes into a tiny footprint and is suitable for Internet of Things (IoT), thanks to a binary of just 100MB. The perfect laboratory companion that offers immediate access to Kubernetes is the clever minikube [2].

Another distribution caught my eye recently when I was arriving, very late to the party, and tinkering with some fascinating Raspberry Pi tech: MicroK8s [3]. Two things jumped out when I spotted Ubuntu's mini Kubernetes documentation. First, as with K3s and minikube, a software build suitable for ARM64 processors (e.g., which a Raspberry Pi uses) is available; second, the documentation prominently notes zero-ops infrastructure.

I hadn't seen terminology about hands-off operations software since Amazon Linux 2 extolled its virtues about live kernel patching [4], which I started using on critical production servers about a decade ago. I can attest to the fact that it's a great feature that saved many 4:00am reboots (after kernel security updates were applied to a critical running system) and was an exceptionally welcome addition to my toolbox because it reduced maintenance windows and downtime considerably.

In this article, I look at how a Raspberry Pi 4 (Model B) with 4GB of RAM stands the tests of an installed Ubuntu MicroK8s (minimal production Kubernetes), courtesy of Canonical. Note that 4GB of RAM is definitely recommended; although, as you will see, you might possibly get away with 2GB.

Pie-O-My

As with other Ubuntu documentation, the different routes of getting started with MicroK8s are clearly written in welcome detail. The marketing strapline offers the high-availability banner along with the description: "Low-ops, minimal production Kubernetes, for devs, cloud, clusters, workstations, Edge and IoT" [3].

The documentation talks about how MicroK8s doesn't have any of the standard Kubernetes APIs removed, and you are encouraged to enter your email address for a research whitepaper [5] that walks through security, operations, and where IoT workloads make the most demands from a Kubernetes cluster. You are then gently reminded that enterprise support is available, should it be required.

The zero-ops claim is explained with the statement: "MicroK8s will apply security updates automatically by default, defer them if you want" [3]. Apparently you can upgrade MicroK8s with just one command – an impressive statement in terms of reducing downtime and admin overhead.

For the Raspberry Pi ARM64 support, the MicroK8s website reminds you where IoT devices are deployed these days: "Under the cell tower. On the racecar. On satellites or everyday appliances, MicroK8s delivers the full Kubernetes experience on IoT and micro clouds" [3].

Blueberry Pie

Clearly, you have to have an Ubuntu version, whether a desktop build or a server, to test MicroK8s. In my case, I had the latest Long Term Support (LTS) version, Ubuntu Server 20.04. Older LTS releases are equally suitable (i.e., 18.04 and 16.04).

Fret not if you can't meet these requirements, however. In addition to offering some welcome troubleshooting advice, the Alternative Installs page [6] is well worth a look and offers information about offline cluster creation and Windows installations, among other things.

On the Raspberry Pi, I was running Docker containers for an automation project, so to prevent any breakages of that build, I cloned the SD card running that project.

As you will see in a second, Ubuntu is a big proponent of Snappy package management [7], commonly known as "snap" (the package is snapd), which cleverly packages up software so that it can run on almost any device [8].

To install snap on the Raspberry Pi, you should run as root the commands:

$ apt update; apt install -y snapd

The packages pulled down for installation (in this case) were snapd and squashfs-tools.

The Raspberry Pi whirs away for a minute or two during this process. Remember the hardware specification that is responsible for running the installation and be warned that a modicum of patience is needed.

According to the docs [9], you need to reboot:

$ reboot

Although I hadn't asked snap to install or run anything yet, I took a quick check of available RAM (Listing 1). Even with Docker running, the Raspberry Pi still had about 2.9GB of free RAM.

Popo ~ # free -m
          total       used       free      shared  buff/cache   available
Mem:      3827        548        2620         173         658        2977
Swap:       99          0          99

To start the MicroK8s installation, simply enter the command:

$ snap install microk8s --classic --channel=1.19

The docs explain exactly what channel refers to [10]. If you intend to continue using MicroK8s beyond testing, you should definitely understand a little more about the release cycle, which you can find on that page, as well. The stable release version, for example, wouldn't follow that channel but would instead be installed with the command:

$ snap install microk8s --classic

That's not always the version you want, so you could use a specific version:

$ snap install microk8s --classic --channel=1.18/stable

I continued with the 1.19 channel (--classic without mentioning a channel, unless you run into problems, which in my case pulled down the 1.20 version). Again, having run the snap install command, you will need a little patience, so water the plants and polish your shoes while you wait. The download takes a few minutes and then the setup of the Snap Core follows before MicroK8s is downloaded and then installed.

Once the process has finished, expect to see output something like:

microk8s (1.19/stable) v1.19.7 from Canonical\u00e2 installed

In true snap style, you don't get much information.

The docs then follow a route similar to what a standard Docker installation might. I won't follow those steps here but will continue using the root user. The docs suggest adding your own login user to the microk8s system group:

$ usermod -a -G microk8s chris
$ chown -R chris ~/.kube

The docs are not very clear about the ~/.kube cache access for the less privileged user, so I stuck with the root user and not chris (which you should replace with your own username if you followed the docs).

At this stage, you should open another terminal, log in to your Raspberry Pi, and check the installation progress in the original window:

$ microk8s status --wait-ready

With the top command in the other terminal window, you can watch the MicroK8s processes running. My device showed a one-minute average load of between 3.0 and 3.5, even with Docker running dutifully in the background. Nearer the end of the installation, the one-minute load jumped to about 5.0. The first time I tried MicroK8s's own version of the kubectl command not much happened, even after waiting an age (Listing 2).

$ microk8s kubectl get pods -A
NAMESPACE     NAME                                  READY  STATUS   RESTARTS  AGE
kube-system   calico-kube-controllers-[snip]-pnnkc  0/1    Pending  0         66m
kube-system   calico-node-m79sg                     0/1    Pending  0         66m

To make the MicroK8s build friendlier to Raspberry Pi, I followed some cgroups (control groups) instructions on the Alternative Installs page [6]. Control groups help manage, typically on Linux containers, the limits on resources consumed in terms of RAM and I/O and are a feature of the kernel. It seems that the Raspberry Pi needs a configuration change to play nicely with MicroK8s. The fix is to add

cgroup_enable=memory cgroup_memory=1

to the end of the first line already present in the /boot/firmware/cmdline.txt file and reboot. The docs point out that some Raspberry Pi versions use the file /boot/firmware/nobtcmd.txt instead. Use that if it's required.

Next I removed the existing iptables rules I had installed and made sure that after a reboot they weren't applied. You can put the following entries in a file and run it as a script to flush iptables rules:

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X

To check your rules are cleared, run the command:

$ iptables -nvL

A handy tip (because Kubernetes uses iptables extensively) is to check that MicroK8s is starting up as hoped by re-running that command every 30 seconds or so during start-up to see multiple Kubernetes and Calico [11] rules filling up your iptables chains. You can also use the watch command in a new terminal:

$ watch -n1 iptables -nvL

The next thing I realized was that Docker wasn't necessarily playing that nicely with MicroK8s, so I stopped the service and removed it completely with the command:

$ systemctl purge docker-ce

Your package name might be docker.io instead of docker-ce.

To make sure MicroK8s was completely happy, I then ran an exceptionally useful command,

$ microk8s inspect

which, as noted on the MicroK8s Troubleshooting page [12], you can run at any time during the installation. (It might just tell you, for example, that you haven't started the cluster yet, more on that below.)

The abbreviated output from the inspect command is shown in Listing 3. Note that this command also gives a warning if you don't get the control groups file entry quite right, which is really handy. Even after removing Docker from the system, this command offers advice about how Docker should be configured to access the sophisticated built-in image registry that is available for MicroK8s on localhost:32000 .

Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-flanneld is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running

Raspberry Pie

The proof of success is, of course, demonstrating that a workload is running on Kubernetes. As a test, I chose one of the most popular container images on the Internet, the lightweight Nginx webserver. You can create an Nginx deployment with the relatively standard Kubernetes command:

$ microk8s kubectl create deployment nginx --image=nginx

Within just a few seconds the command completes, and an Nginx pod has been created. It will also be re-created if it fails for some reason, because the deployment keeps an eye on it.

Once live, you can run a command that describes all pods in the default namespace (which by default is no pods, other than the newly created web server pod). To see the details of the Nginx pod, enter:

$ microk8s kubectl describe po

Next, you can take the IP address of the pod and use the curl command to connect to it, producing the abbreviated output in Listing 4. In this way, you've proven that you have a workload running courtesy of Kubernetes on your Raspberry Pi.

$ curl -k http://10.1.167.10
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
       body {
              width: 35em;
              margin: 0 auto;
              font-family: Tahoma, Verdana, Arial, sans-serif;
        }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>