System Profiling

At this point, you can glean some timing numbers from applications and some tracing information from the hardware performance counters that are common in modern processors. However, often you want to learn how your application is running from the system itself. This is similar to profiling the application, but it is extended to profile the system as a whole and can create quite a bit more data; however, it also gives you more information about how your application is performing.

Probably the most common system profiling tool is called OProfile [19]. OProfile gathers system statistics using sampling techniques (similar to gprof and other profiling tools) over a period of time. It gathers processor information gathered using Linux-perfctr along with other data; then, you can run reports against the gathered data to analyze whatever you want. Using OProfile is fairly easy. Begin by starting up OProfile (opcontrol) then run your application (preferably on a quiet system, so you don't get the effects of other applications running). After the application is done, you run a report to summarize the information in which you are interested. The steps are summarized in a cheat sheet online [20].

For help on using OProfile, look at the GitHub tutorial [21], among others online. Also, a nice GUI for interacting with OProfile, called Visual OProfile [22], comes with a distribution called STLinux, but I can't tell whether it's open source or not (little information about it can be found).

System Tracing

In addition to system profiling, you can "trace" or watch system events or values over time. This generally applies to the system as a whole, but you can gain some reasonable insight into what the system is doing to support the application.

Probably the most common set of system tracing tools are the Sysstat utilities [23]. Sysstat comprises several tools that come with almost all Linux distributions and are very useful for tracing system activities:

• sar
• sadf
• mpstat
• iostat
• nfsiostat
• cifsiostat
• pidstat
• sa, which comprises sadc, sa1, and sa2

Some very good tutorials on using sysstat tools, particularly sar, can be found on the Sysstat page, as well as on HowtoForge [24], which also talks about a graphical tool, ksar [25] that graphically plots the sar data, and on Linux.com [26], which includes a discussion about how to use MySQL to store the sar data.

A commercial tool, SarCheck [27], acts like sar but it goes further by creating a report of the monitored data and making recommendations on system tuning parameters to improve performance. Although I don't personally have any experience with SarCheck, it might be worth investigating.

Another tool that can be very useful for system tracing is called collectl, a fairly comprehensive package that is oriented a little more toward HPC because it adds the ability to trace InfiniBand and Lustre components. I have written about collectl in the past on the ADMIN HPC website [28], so I won't cover it here.

MPI Profiling and Tracing

For HPC, it's appropriate to discuss how to profile and trace MPI (Message-Passing Interface) applications. A number of MPI profiling tools are available, but you should make sure they work with the MPI library you are using.

Examples of tracing tools include MPE [29], which is associated with MPICH but works with other MPI implementations, TAU (mentioned under Processor Tracing), and Paraver [30]. Remember, tracing really refers to collecting event histories so you can get an idea of how performance or events evolve over time. Although MPI profiling tools such as mpiP, FPMPI-2, and IPM focus more on aggregating statistics during run time to get totals rather than time histories, some of these tools add tracing as well.

Scalasca [31] (see the System Tracing section) is a bit more than a profiling or tracing tool for MPI applications, because it goes beyond the normal set of tools to provide guidance on how the causes of performance bottlenecks might be improved.