Daily Grind

In everyday life, you see more details when comparing the two solutions: Kpatch works more thoroughly than kGraft. In the kGraft example, the patch for a function only fields calls to the function that occur after applying the patch. Functions that were running before remain unaffected, which means that several versions of the same function could exist in kernel space. This will probably be a problem if a kernel patch fixes a critical problem that affects, for example, central drivers for the network hardware. You cannot unload and reload these drivers easily to benefit from the corrected functions.

Red Hat follows a different philosophy; once a Kpatch patch is loaded, it replaces all active function calls in the kernel with the new version of the function, so that multiple versions do not exist.

The following instructions show how to run kGraft on openSUSE 13.1 and Kpatch on Fedora 20 with quite new kernels (Figure 1). Although you could follow these instructions for other distributions, you might find that the installation overhead is considerably greater.

Figure 1: Lab systems: (top) Fedora 20 with a brand new 3.14 kernel; (bottom) SUSE 13.1 with kernel 3.13.

Kpatch on Fedora

On Fedora 20, you first need to install some additional packages. The following instructions assume that all packages are up to date across the system. The kernel I used was version 3.14.3-200.fc20, the latest at the cut-off date for this issue.

For a functioning Kpatch, the following packages are required:

• gcc
• kernel-devel
• elfutils and elfutils-devel
• rpmdevtools
• pesign
• yum-utils

After installing these packages with yum install, the yum-builddep command makes sure the appropriate kernel headers find their way onto the system. If you want to use ccache, you should install this package as well. Next, you should download Kpatch by mirroring the repository on your local disk and putting the modules in place:

git clone https://github.com/dynup/kpatch
cd kpatch && make && make install

To test whether Kpatch is working as desired, just apply a small patch. The Red Hat example suggests modifying /proc/meminfo so that the kernel VMallocChunk line displays uppercase characters from now on. For Fedora  20, this is done with the patch in Listing  1.

$ diff -ruN orig/fs/proc/meminfo.c new/fs/proc/meminfo.c
--- orig/fs/proc/meminfo.c 2014-03-31 05:40:15.000000000 +0200
+++ new/fs/proc/meminfo.c  2014-05-11 16:33:19.148771809 +0200
@@ -131,7 +131,7 @@
  "Committed_AS:   %8lu kB\n"
  "VmallocTotal:   %8lu kB\n"
  "VmallocUsed:    %8lu kB\n"
-    "VmallocChunk:   %8lu kB\n"
+    "VMALLOCCHUNK:   %8lu kB\n"
 #ifdef CONFIG_MEMORY_FAILURE
  "HardwareCorrupted: %5lu kB\n"
 #endif

After editing, the

Kpatch-build meminfo-string.patch

command initiates the process to build the patch. Before it can build a Kpatch module, the helper tool first builds the original kernel. In the second step, it then creates the patch. Finally, the Kpatch module is created on the basis of the delta of the two trees. This process explains why the use of ccache makes sense.

Working with Kpatch for the first time, you might be annoyed at how long the procedure takes, compared with compiling a kernel with countless reboots for dozens of computers, the loss in time is quickly put into perspective.

At the end of the build process you will see a Kpatch-meminfo-string.ko file. To load this into the kernel, run

sudo Kpatch load Kpatch-meminfo-string.ko

A look at /proc/meminfo then reveals that the change worked (Figure 2). Following the same pattern, changes to other functions in the kernel should now also work.

Figure 2: Before deploying the patch with Kpatch (left), /proc/meminfo shows the usual picture. After loading the patch, you see VMALLOCCHUNK – now with uppercase letters – at the same location.

kGraft on openSUSE

As expected, our lab revealed that the barriers to installing kGraft on openSUSE are somewhat higher (Figure  3), because kGraft cannot be built as a module for Linux but requires a specially patched kernel. Before you can use kGraft, you always need to build a kernel.

Figure 3: SUSE's kGraft also lets you load individual kernel patches at run time, but the barriers to entry are much higher.

Whereas Red Hat provides detailed Kpatch installation guides in the README.md file in the Kpatch GitHub repository, documentation for kGraft is comparatively sparse. Although SUSE provides a link [4] you can follow to the kGraft-patched version of the currently active SUSE kernel, this only gives you the source code; detailed instructions are not to be found (Figure 4). In the quite non-trivial process of performing openSUSE brain surgery, admins are initially left on their own.

Figure 4: SUSE gives you the kGraft tree, but it fails to provide documentation telling you exactly what to do with it. In the end, much manual work is called for.

If you do find documentation, you can look forward to an intensive bout of tinkering: After a Git checkout of developer Jiri Slaby's repository, you can start by compiling the source code. In a separate blog entry [5], SUSE describes how a kernel can be produced SUSE-style, but replacing the kernel is nevertheless not trivial.

Even after the successful launch of a DIY kernel, your tribulations are not over yet. As with Kpatch, it is then up to the users to build the appropriate module files with the aid of various scripts by Slaby. At the end of the process, you then need to load the results into the kernel.

It is beyond the scope of this article to deliver a detailed explanation of the steps required to build a ready-to-use kGraft system, but if you are interested in the details, check out Slaby's repository.