Behind Time

Going Further: NUMA Issues

Google has also highlighted SchedViz's ability to do more than just look at how processing resources are shared according to the CPU time given to each. SchedViz also provides a powerful way to visualize the way larger systems work with NUMA nodes.

Larger servers often have several NUMA nodes that are a subset of DRAM memory assigned to particular cores that can be accessed more quickly than the general memory pool. Cores can access NUMA nodes assigned to other cores, and often do if the cores are overworked; however, this process is much slower than if the cores stuck to their own NUMA node. This nonuniformity is a practical consequence of growing core count, but it brings challenges.

If a core jumps to a different NUMA node, performance can be affected significantly, because it will then have to pay an extra tax for each DRAM access. SchedViz can help identify cases like this, making it clear when a thread has had to migrate across NUMA boundaries. Moreover, SchedViz can show you which NUMA nodes are in use at any one time, helping to identify situations in which one part of your machine is overtaxed while the other part sits idle. A typical trace of that situation will look like Figure 4. SchedViz can identify an unbalanced system like this, so the sys admin can adjust the NUMA behavior [11] to fix the issue.

Figure 4: All available NUMA nodes are at the right, with their usage on the left. Apparently, this system is very unbalanced.

Further Resources

If you want to explore the features that come packaged with SchedViz, take a look at the detailed features walkthrough [12] provided by Google. This document will show you how to collect various types of traces and how to use the tools available for analyzing them.

Another very useful feature provided by the kernel is a debug feature [13] that can analyze trace data and stream it to a buffer for later analysis, providing you with a quick way of highlighting scheduling or scheduling rules problems.

At the moment, SchedViz is primarily useful for tracking scheduling errors and fine-tuning the way you assign computing resources. Although that's pretty useful in itself, plans are in progress to make it even more powerful in the future. Beyond using SchedViz for figuring out kernel scheduler defects, Google is also looking at using it to visualize other kernel tracepoints to analyze other kernel behavior that could be optimized for better efficiency, so watch this space.