Kpatch and kGraft

Patches as Kernel Modules

In kGraft's case, then, how is the code to which the handler needs to jump compiled into the kernel? The SUSE developers decided to provide patched kernel code in the form of ordinary kernel modules. For users to replace code on the fly, they need a matching kernel module with a newer version of the desired function. Only functions in the kernel can be replaced by kGraft; you cannot modify the internal data structures of the kernel with kGraft.

You load the kGraft kernel module just like any other module. In this way, kGraft updates can be integrated into the existing package architecture of many distributions. For SUSE, Red Hat, Debian, and Ubuntu, it is ultimately nothing new to deliver kernel functions in the form of additional module packages. Until now, the process has affected only drivers that were not previously in the kernel, but if you look at the kGraft module interfaces, they are identical to normal kernel modules.

This principle makes it possible to distribute kGraft patches in the form of packages via any installer system and load them into the running kernel with insmod. If you use Puppet or Chef, the entire process can even be automated. It's hard to imagine a more convenient way to implement in-place updates of kernel functions. Thus, you gradually collect modules until you reboot. Whether or not this ever-growing stack of modules causes any problems has yet to be seen.

Red Hat's Kpatch

Red Hat also presents a solution that lets you replace kernel code on the fly. The Red Hats call it Kpatch [2], and just as SUSE developed kGraft on its own, Red Hat developed Kpatch from scratch. Neither kGraft nor Kpatch borrow directly from Oracle's Ksplice, although in some respects Kpatch follows the fundamental decisions that kGraft also uses.

First, the similarities: Like SUSE's kGraft, Kpatch latches into the kernel via the Ftrace subsystem that GCC automatically includes at compile time. Ultimately, Kpatch thus fields function calls in a very similar way. Unlike kGraft, however, Kpatch is a kernel module itself; admins do not need to patch the host kernel.

This hot patch mechanism provides benefits in scenarios in which admins want to replace the patch mechanism itself with a newer version. With SUSE, this implies a reboot, but not necessarily for Red Hat.

Kpatch injects two modules into the kernel: hot patch module contains the Kpatch functionality and kpatch core module provides the user interface by which patches are then injected into the kernel later on.

Et tu, Kpatch?

Even Kpatch expects updates for functions in the form of kernel modules that are loaded into the host kernel. When you call a function that is available in a more recent version of a Kpatch patch, the function call is then forwarded to the "new" version of the function. The Kpatch core module serves as a central patch registry and knows whether Kpatch has installed a patch for a specific function; depending on this knowledge, it executes the old or new code.

Incidentally, Red Hat delivers a tool called kpatch-build that generates an updated module from a patch for Linux, and Kpatch also includes a utility that conveniently manages the existing hot patches. You can even use this tool to define patches to be reloaded after a system reboot.

At first, this method sounds as if it were contrary to the Kpatch approach as a whole, but in reality, it is very smart: If an admin has a working combination built from a stable kernel and additional patches, and the power then fails – making a reboot unavoidable – they might not want to face an additional problem caused by a kernel update. The better way might be just to recreate the known environment and load the update modules with the Kpatch utility.