Appendix – MPI Application I/O Report from MPI Strace Analyzer

Strace Report

Introduction

This report summarizes the statistics in analyzing the strace results for a series of files that have been run through the strace_analyzer. This output allows you to compare multiple strace files such as those resulting from an MPI application. The number of files used in this analysis is 8. The files are:

file_18590.pickle
file_18591.pickle
file_18592.pickle
file_18593.pickle
file_18594.pickle
file_18595.pickle
file_18596.pickle
file_18597.pickle

This report is broken into several sections. In each section a summary of the results is presented and a statistical analysis of the files relative to each other is made. In addition, the statistics for each file are also presented.

1. Time Statistics

Table 1 below is a table of the total elapsed times and IO times in seconds for each of files. 

Table 1 - Total time and IO Time (seconds)

Please note that the numbers in red are the maximum times (maximum total elapsed time and maximum IO time). The number below the IO time, in green, is the percentage of the total elapsed time used by IO.

The following results are the overall statistics for all of the files.
 

Maximum Elapsed Time: 3,376.819 seconds   file: file_18591.pickle
Minimum Elapsed Time: 3,376.810 seconds   file: file_18596.pickle
Average Total Elapsed Time = 3,376.815 seconds
Standard Deviation of Total Elapsed Time = 0.003117 seconds
Average IO Time = 12.768351 seconds (0.378% of Total Time)
Standard Deviation of IO Time = 9.919384 seconds

Figure 1 below plots the total elpased time (yellow) and the IO time (red) in seconds for all of the input files.
 

Figure 1 - Total time and IO Time (seconds)

Figure 1 - Total time and IO Time (seconds)

2. I/O Function Count

This section lists the I/O function command count (how many times a particular I/O command is called) for the various files. Table 2 below contains this information.

Table 2 - I/O Function Command Count

The data listed in red are the maxium values. For example the data in the write row that is red is the file with the largest number of write function calls.

Figure 2 below plots the IO function count for all of the input files focusing on read(), write(), lseek() - all lseek functions, open(), and close() IO functions.

Figure 2 - IO function count for all input files for read, write, lseek, open, and close

Figure 2 - IO function count for all input files for read, write, lseek, open, and close

3. Write Statistics

This section presents statistical information about the write functions in the application. It focuses on use of the write(2) function in glibc. The pertinent details of the man pages are:

NAME
write - write to a file descriptor

SYNOPSIS
#include

ssize_t write(int fd, const void *buf, size_t count);

DESCRIPTION
write() writes up to count  bytes from the buffer pointed buf  to the file referred to by the file descriptor fd .

The number of bytes written may be less than count  if, for example, there is insufficient space on the underlying physical medium, or the RLIMIT_FSIZE resource limit is encountered (see setrlimit(2)), or the call was interrupted by a signal handler after having written less than count  bytes. (See also pipe(7).)

For a seekable file (i.e., one to which lseek(2) may be applied, for example, a regular file) writing takes place at the current file offset, and the file offset is incremented by the number of bytes actually written. If the file was open(2)ed with O_APPEND, the file offset is first set to the end of the file before writing. The adjustment of the file offset and the write operation are performed as an atomic step.

POSIX requires that a read(2) which can be proved to occur after a write() has returned returns the new data. Note that not all file systems are POSIX conforming.

The information and statistics in this section focus on the write() aspects of the application. Both tabular and graphical information is provided.

Table 3 below contains information on the total number of write function calls as a function of the data transfer size. The data is presented in

Table 3 - Total Write Function calls vs. data size

Table 4 below contains the number of write function calls for each file.

Table 4 - Write Function calls for each file as a function of size

The next 5 figures plot the data contained in Table 4. The first figure, Figure 3, plots the write IO Function call Count for all of the intervals. The next 4 figures plot subsets of Figure 3. Figure 4 plots the write IO function call count for the KB range intervals. Figure 5 plots the write IO function call count for the MB range intervals. Figure 6 plots the write IO function call count for the GB range intervals. Figure 7 plots the write IO function call count for the TB range intervals.

Figure 3 - Write IO function count for all input files for all intervals

 Figure 3 - Write IO function count for all input files for all intervals
Figure 4 - Write IO function count for all input files for KB intervals

Figure 4 - Write IO function count for all input files for KB intervals
Figure 5 - Write IO function count for all input files for MB intervals

Figure 5 - Write IO function count for all input files for MB intervals
Figure 6 - Write IO function count for all input files for GB intervals

Figure 6 - Write IO function count for all input files for GB intervals
Figure 7 - Write IO function count for all input files for TB intervals

Figure 7 - Write IO function count for all input files for TB intervals

The following statistics are for the write() function for all of the files in the analysis.

Total number of Bytes written = 3,092,316,937 (3,092.317 MB)
Number of Write function calls = 290,969
Average (mean) bytes per call = 10,634.815 (bytes) (0.011 MB)
  Standard Deviation bytes per call = 151,511.125208 (bytes) (0.151511 MB)
  Mean Absolute Deviation = 159,176.444887 (bytes) (0.159176 MB)
Median Bytes per call = 58 (bytes) (0 MB)
  Median Absolute Deviation = 10,601.735722 (bytes) (0.010602 MB)
Time for slowest write function (secs) = 0.255515   file:   file_18597.pickle
  Line location in file = 5,661

Table 5 below contains the write function statistics on a per file basis.

Table 5 - Write Function statistics for each file

 Figure 8 plots the total megabytes written for each file.   

Figure 8 - Per File Write Amount for all input files

Figure 8 - Per File Write Amount for all input files

In Figure 8 the solid blue line is the average amount written in MB across all files. The two dashed blue lines show the +/- one standard deviation in the amount of data written in MB.

Figure 9 plots the number of write function calls for each file.

Figure 9 - Per File Write() function all input files

Figure 9 - Per File Write() function all input files

In Figure 9 the solid blue line is the average number of write() function calls across all files. The two dashed blue lines show the +/- one standard deviation in the number of write() functions called.

4. Read Statistics

This section presents statistical information about the read functions in the application. It focuses on use of the read(2) function in glibc. The pertinent details of the man pages are:

NAME
read - read from a file descriptor

SYNOPSIS
#include

ssize_t read(int fd, void *buf, size_t count);

DESCRIPTION
read() attempts to read up to count bytes from file descriptor fd into the buffer starting at buf.

If count is zero, read() returns zero and has no other results. If count is greater than SSIZE_MAX, the result is unspecified.

The information and statistics in this section focus on the read() aspects of the application. Both tabular and graphical information is provided.

Table 6 below contains information on the number of read function calls as a function of the data transfer size. The data is presented in  

Table 6 - Read Function calls vs. data size

Table 7 below contains the number of read function calls for each file.

Table 7 - Read Function calls for each file

The next 5 figures plot the data contained in Table 7. The first figure, Figure 10, plots the Read IO Function Count for all of the intervals. The next 4 figures plot subsets of Figure 10. Figure 11 plots the Write IO function count for the KB range intervals. Figure 12 plots the Write IO function count for the MB range intervals. Figure 13 plots the Write IO function count for the GB range intervals. Figure 14 plots the Write IO function count for the TB range intervals.

Figure 10 - Read IO function count for all input files for all intervals

Figure 10 - Read IO function count for all input files for all intervals
Figure 11 - Read IO function count for all input files for KB intervals

 Figure 11 - Read IO function count for all input files for KB intervals

Figure 12 - Read IO function count for all input files for MB intervals

Figure 12 - Read IO function count for all input files for MB intervals

 
Figure 13 - Read IO function count for all input files for GB intervals

Figure 13 - Read IO function count for all input files for GB intervals
Figure 14 - Read IO function count for all input files for TB intervals

Figure 14 - Read IO function count for all input files for TB intervals

Overall summary of read functions. These statistics are for all of the files in in the analysis.

Total number of Bytes read = 7,850,388,044 (7,850.388044 MB)
Number of Read function calls = 9,108
Average (mean) bytes per call = 880,878.371185 (bytes) (0.880878 MB)
  Standard Deviation bytes per call = 721,848.996629 (bytes) (0.721849 MB)
  Mean Absolute Deviation = 573,996.685684 (bytes) (0.573997 MB)
Median Bytes per call = 672,127 (bytes) (1 MB)
  Median Absolute Deviation = 573,051.721499 (bytes) (0.573052 MB)
Time for slowest read function (secs) = 0.063513   file:   file_18595.pickle
  Line location in file = 835

Table 8 below contains the read function statistics on a per file basis.

Table 8 - Read Function statistics for each file

Figure 15 plots the total megabytes read for each file.
 

Figure 15 - Per File Read Amount for all input files

Figure 15 - Per File Read Amount for all input files

In Figure 15 the solid blue line is the average amount read in MB across all files. The two dashed blue lines show the +/- one standard deviation in the amount of data read in MB.

Figure 16 plots the number of read function calls for each file.
 

Figure 16 - Per File read() function all input files

Figure 16 - Per File read() function all input files

In Figure 16 the solid blue line is the average number of read() function calls across all files. The two dashed blue lines show the +/- one standard deviation in the number of read() functions called.

5. Close Statistics

This section presents statistical information about the close() functions in the application. Below are the statistics across all files in the analysis.
 

Total number of close function calls for all files = 617
Average time for close function calls (secs) = 3.32437395659e-05
  Standard Deviation for close function calls (secs) = 0.000109286534399
Median close function time (secs) = 5e-06
Time for fastest close() across all files (secs): 3e-06
  file: file_18592.pickle line number: 28
Time for slowest close() across all files (secs): 0.001728
  file: file_18592.pickle line number: 12308

This section lists the close() function statistics for the input files. It also lists the fastest and slowest close() function for each file. Table 9 below contains this information

Table 9 - Close Function details - per file basis

6. Open Statistics

This section presents statistical information about the open() functions in the application. Below are the statistics across all files in the analysis.

Total number of open function calls for all files = 646
Average time for open function calls (secs) = 0.000136857585139
  Standard Deviation for open function calls (secs) = 0.000511730042175
Median open function time (secs) = 8e-06
Time for fastest open() across all files (secs): 5e-06
  file: file_18593.pickle line number: 182
Time for slowest open() across all files (secs): 0.008109
  file: file_18596.pickle line number: 48265

This section lists the open() function statistics for the input files. It also lists the fastest and slowest open() function for each file. Table 10 below contains this information

Table 10 - Open Function details - per file basis

7. Lseek Statistics

This section presents statistical information about the lseek() functions in the Table 11 below contains information about which files experienced a lseek() operation while performing I/O. This is done for each input file.

Table 11 - Lseek Function details - per file basis

8. IOPS Statistics

This section presents statistical information about the IOPS performance while the application was running. The following statistical data is for the peak IOPS across all files.

Maximum Peak Write IOPS = 2444    file: file_18595.pickle
Maximum Peak Read IOPS = 153    file: file_18591.pickle
Maximum Peak Total IOPS = 2444    file: file_18595.pickle

Average Peak Write IOPS = 2056.75
    Standard Deviation Peak Write IOPS = 247.330623862
Average Peak Read IOPS = 139.75
    Standard Deviation Peak Read IOPS = 12.8038080273
Average Peak Total IOPS = 2058.75
    Standard Deviation Peak Total IOPS = 245.587942497

The following statistical data is for the Overall IOPS across all files. Remember that the Overall IOPS are computed as the total IOPS divided by the total application time.

Average Overall Write IOPS = 10.763581995
    Standard Deviation Overall Write IOPS = 2.36588573648
Average Overall Read IOPS = 0.329896633154
    Standard Deviation Overall Read IOPS = 0.315014380446
Average Overall Total IOPS = 11.8788340116
    Standard Deviation Overall Total IOPS = 2.9381477474

Table 12 below contains this information.

Table 12 - IOPS Summary for all files

Figure 17 plots the peak write IOPS for each file.
 

Figure 17 - Per File Peak Write IOPS for all input files

Figure 17 - Per File Peak Write IOPS for all input files

In Figure 17 the solid blue line is the average Peak Write IOPS across all files. The two dashed blue lines show the +/- one standard deviation in the Peak Write IOPS.

Figure 18 plots the peak read IOPS for each file.
 

Figure 18 - Per File Peak Read IOPS for all input files

Figure 18 - Per File Peak Read IOPS for all input files

In Figure 18 the solid blue line is the average Peak Read IOPS across all files. The two dashed blue lines show the +/- one standard deviation in the Peak Read IOPS.

Figure 19 plots the peak total IOPS for each file.

Figure 19 - Per File Peak Total IOPS for all input files

Figure 19 - Per File Peak Total IOPS for all input files

In Figure 19 the solid blue line is the average Peak Total IOPS across all files. The two dashed blue lines show the +/- one standard deviation in the Peak Total IOPS.

Figure 20 plots the average write IOPS for each file.

Figure 20 - Per File Average Write IOPS for all input files

Figure 20 - Per File Average Write IOPS for all input files

In Figure 20 the solid blue line is the average write IOPS across all files. The two dashed blue lines show the +/- one standard deviation in the Average Write IOPS.