Hidden Treasures

Syscall Tracing

The libiotrace library extends tracing beyond function call wrapping by adding support for syscall tracing, which has upsides and downsides.

The upside is that no file I/O will escape the library. File I/O services can only be requested from the kernel through the syscall interface. Hence, whether the application invokes a syscall manually with in-line assembly or by calling a libc wrapper, the file I/O event will be traceable.

The downside is that mapping the observed syscall to a function call in the log of the traced application is not always feasible. For syscall tracing purposes, libc file I/O functions can be categorized into two categories: (1) I/O functions that are merely syscall wrappers (e.g., write, documented in section 2 of the system calls man pages) and (2) functions that provide additional buffering in the user space (e.g., fwrite, documented in section 3 of the library functions man pages). The second type makes mapping on the basis of timestamps unfeasible because those functions won't immediately trigger a syscall, unlike the first type.

The Price of Tracing

Linux syscalls used to be fairly cheap (on the order of hundreds of clock cycles); however, since mitigations for CPU vulnerabilities (e.g., Spectre and Meltdown), syscalls have become more expensive (on the order of thousands of clock cycles). Tracing syscalls however is even more expensive. The incurred slowdown depends on many factors: for instance, how many syscalls the application triggers; how many processes, threads, or both are being traced; and whether the stack is unwound.

According to benchmarks with only one tracee, the additional overhead in terms of execution time for syscalls (caused by the libiotrace syscall tracing implementation) is approximately 1,400%. This massive slowdown can be attributed mostly to the expensive stack unwinding operation.

Goals

Keeping these limitations in mind, the following goals were set for the development of the libiotrace's syscall tracing implementation, called "stracer":

• Detect missed I/O events. Stracer should detect whether libiotrace traced the function call that triggered the current syscall. If the function isn't traced, a warning is written to console and logfile.
• Pass traced "syscall events" to libiotrace. As a dynamically linked library, libiotrace uses the same address space as the traced application. The stracer, however, will run as a separate process because the stracer should not be traced by libiotrace. Thus, an interface is used for passing traced syscalls to libiotrace, which then updates mapping data for file handles.