When it comes to kernel version or security updates in Linux, most admins trust an ancient binary procedure: They install the updated kernel packages provided by their distributor of choice, or they build a new kernel and restart the system.

Anyone who has followed kernel updates of the various distributions in recent months and years will come to the conclusion that the legendary Linux uptime is only feasible if you do not install kernel patches and thus accept the associated vulnerabilities and other risks.

No Way! Rebooting a Cluster

To provide new kernel functions or security fixes, you need to reboot, but although this process is performed in the background thousands of times a day all over the world, it can create havoc that any administrator would prefer to avoid.

If the server you need to restart belongs to a cluster, for example, you need to take great care to avoid Pacemaker or some other cluster manager unintentionally identifying a failure and initiating an emergency response. Cluster admins will usually want to migrate running services manually to other systems before the reboot.

The reboot not only means more work but often downtime as well, and admins always need to mitigate the effect of service downtime. For this reason and others, IT professionals around the world seek to avoid reboots, even if they "only installed a new kernel."

Other groups would also be happy to avoid reboots. Kernel and driver developers could work more efficiently if they did not have to reboot after each code update, so hot patches are at the top of their wishlist.

Ksplice

Until now, hot patching was a fantasy on Linux. Recently, though, both SUSE and Red Hat launched solutions that will make kernel patching possible during operation. However, neither SUSE nor Red Hat invented the principle: Oracle has offered Ksplice  [1]  – a kernel patching solution – for some time, but it suffers from various health problems and its license conditions do not exactly inspire confidence. Additionally, Oracle now reserves the tool exclusively for its own business customers; the open source variant of Ksplice has not been under development for some time.

Both Red Hat and SUSE offer alternatives to Ksplice that claim to be equivalent in terms of functionality: Kpatch [2] and kGraft [3]. Technically, the two approaches have major differences with widely divergent functionality. Perhaps this divergence is why Red Hat and SUSE continue to press ahead with their own approaches, rather than agreeing on a single development path.

kGraft by SUSE

kGraft comes from SUSE's own development department. The concept has achieved production maturity, and the vendor has therefore gone public. kGraft leverages a number of functions that modern versions of the Linux kernel support, including int 3 trap calls, ready-copy updates (RCUs), and memory profiling by mcount. At the end of the day, this potpourri of approaches gives kGraft the skills it takes to replace code in the kernel with other code on the fly.

What then follows is technically highly complex. For example, kGraft draws massively on the fact that the profiling code from GCC leaves some space at the beginning of each function when compiling. This area is initially used by calls to the ftrace() function via an __fentry__ call. However, these are replaced by NOP entries at boot time so that each function starts with a deliberate space.

kGraft uses precisely this space, replacing it with int 3 handlers that can immediately jump to a different part of the kernel code when calling a specific function without taking any intermediate steps. That is, calling a function in a kernel with support for kGraft always means that kGraft itself is called, so kGraft is omnipresent.