Managing Linux Memory

Memory Hogs

Faster Reads

The page cache is thus a cache for file data that has been stored in pages. Its use speeds up disk access by orders of magnitude. Therefore, Linux makes heavy use of this cache and always tries to keep it at the maximum possible size [5].

Two other areas, which were managed separately in legacy Linux versions (kernel 1.x and 2.x), are integrated into the page cache. First is the buffer cache, which manages blocks belonging to disks and other block devices in main memory. Today, however, the Linux kernel no longer uses blocks as storage units, but pages. The buffer cache thus only stores page-to-disk block mappings, because they do not necessarily have to be the same size as pages. It is of very little importance today.

The second area is the swap cache, which is also a cache for pages that originate from a disk. However, it does not work with pages that reside in regular files but with pages that reside on the swap device – the anonymous pages. When a page is swapped out to the swap partition and later put back into storage, it is initially just a regular disk page. Thus, it should be cached.

The swap cache now primarily contains only pages that have been swapped in but not subsequently modified. Linux can detect from the swap cache whether swapping out again requires a write to disk (to the swap space). One of the swap cache's most important tasks is thus avoiding (write) access. A clear distinction to the actual page cache exists here, which primarily accelerates reads [5].

In the case of frequent disk access, the kernel now tries to adjust the page cache to meet these requirements. In the above-outlined test, it grows as the file being written becomes larger. However, because the number of available page frames is limited (to 10GB of free memory in the test), this growth comes at the expense of the pages that applications can keep for themselves in physical memory.

Old Pages

As the page cache grows, the kernel's page replacement strategies [1] try to create space. Linux uses a simplified LRU (least recently used pages) strategy for this. More specifically, it identifies outsourcing candidates by finding pages that have remained unused for a longer period of time. The kswapd kernel thread handles the actual swapping process.

Unfortunately, many application-specific buffer areas fall into this category. A typical SAP application server can manage many gigabytes in these buffers and temporary areas. The diversity of requests, however, leads to large parts of these not being used very frequently or very quickly, although they are still used from time to time. The request locality is fairly small.

Depending on how aggressively it swaps, the Linux kernel sacrifices these pages in favor of page cache growth and swaps them out. However, this approach degrades application performance, because, under certain circumstances, Linux memory access is no longer served directly from main memory, but only after swapping back in from swap space.

These considerations provide the framework for explaining the empirically observed behavior discussed previously. When backups or copying large files cause substantial disk access, the operating system can swap out application buffer pages in favor of a larger page cache. After this has happened, when the application needs to access the paged memory, this access will be slower. Increasing the physical memory size by installing additional memory modules does not fundamentally solve the problem; at best, it just postpones its occurrence.

Finding Solutions

The background from the previous sections shows that the observed performance degradation is due to the lack of physical memory. Possible solution approaches can be classified in terms of how they ensure sufficient space for an application in physical memory:

Focusing on your own application: Preventing your own pages from being swapped out.
Focusing on other applications: Reducing or limiting the memory usage of other applications.
Focusing on the kernel: Changing the way the kernel manages physical memory.

These different approaches are summarized graphically in Figure 3.