The Lay of the Land

The vast majority of CPU cycles that HPC systems at the Leibniz Supercomputing Center (LRZ) [1] run in the course of daily calculations involves highly optimized binary code. The underlying source code is written and compiled in one of three classic HPC languages: Fortran [2], C++ [3], or C [4].

Why these languages? Because they have enabled generation of very efficient code for a long time. Figure 1 shows statistics for languages used on the SuperMUC [5] system in the first half of 2017 (see the box titled "SuperMUC"). Column lengths in the figure are proportional to the number of CPU cycles used by the projects. Because many projects use a mixture of languages (only 32 percent of all cycles belong to projects that use only one language), these mixtures are represented as separate categories. Fortran is involved in about 72 percent of all allocated project resources, while C++ and C have shares of 54 and 41 percent, respectively.

The compilers used on SuperMUC for all three languages are from Intel, which are bundled with other HPC software components and are available as the Parallel Studio Cluster Edition. The "other" category in Figure 1 includes dynamic languages such as Java, Python, and R, and the "unknown" category includes projects that use special, mostly commercial applications for which the language of implementation is not known.

Figure 1: Programming language share in active projects run on SuperMUC in the first half of 2017.

All of these classic HPC languages are subject to a standardization process that ensures the integration of new language features at regular intervals. Table 1 provides an overview of the last three standard versions of all HPC languages and, if applicable, upcoming new editions.

Programming Models and HPC Software

Most projects on the high-performance computers at the LRZ are based on code developed by scientists over many years. Program authors usually choose the appropriate programming language in the design phase at the beginning of the development process. The decision is often based on their previous knowledge or preferences. As a rule, the selection is rather conservative – on the one hand in terms of the language itself, and on the other hand with regard to the language features on which the programmers rely.

Recently defined language features are only used in production code when it is assured that they have been ported to other platforms and work there without error and as efficiently as possible. Such new language features (e.g., object orientation) tend to aim for an efficient programming methodology rather than for maximum performance, so it is important to use them wisely – and preferably not at all in data- or computation-intensive contexts.

The programming language is by no means the only factor on which the success of a project depends. Also essential is support for parallel programming models such as Open Multiprocessing (OpenMP, a directive-based model for parallelization with threads in a shared main memory) [6] and Message Passing Interface (MPI, a library-based system for parallelization to distributed main memory, typically via a high-speed network connecting the nodes) [7]. Implementations of these models are available for all HPC languages.

Standard interfaces make frequently used functions available as libraries that are optimized for the target platform. For example, BLAS/LAPACK is the standard for linear algebra operations, and FFTW is the standard for Fourier transformations of all kinds. Corresponding libraries for data I/O (e.g., MPI-IO, HDF5, NetCDF) are usually available for all HPC languages.

The implementation of scalable C++ containers in Threading Building Blocks (TBB) [8] has achieved an exceptional status. Initially it was driven by Intel, who later made it available on other platforms as part of an open source project.

Similarly, Fortran implements parallel functionality with Coarrays, an single program, multiple data (SPMD) model with one-way communication for appropriately annotated objects that is much easier to handle than MPI, potentially speeding up the development cycle for parallel applications. However, this is not available in the other HPC languages – or only in a limited form.

The Procurement Process

Because of high investment volumes, in Europe, parallel computers of the highest performance class have to be procured in accordance with legal regulations. The basis is a detailed specification of all requirements for the computing system. In addition to many other criteria, one requirement is that the manufacturer provide a compiler suite (mainline compiler) that is optimal for the architecture offered for all three classic HPC languages.

The manufacturer must describe the degree of standards conformity of the implementations and also provide information on future development of the compilers during the planned operating period. This information is compared among all offers and contributes to the qualitative evaluation of the respective offer.

The quantitative evaluation of the offers is derived from the achievable computing power of the system. Each provider must provide performance projections for their system for a set of benchmark programs specified in advance. These benchmarks are either synthetic and measure specific system aspects (e.g., main memory or network bandwidths), or they are real applications based on programs running on the current legacy operating system that are scaled up for the problem sizes to be expected in the future (e.g., by appropriately increasing the size of the datasets to be processed).

As a rule, the mainline compilers offered are used to compile the computation programs; therefore, all classic HPC languages are represented in the benchmark suite. The manufacturer can optimize the execution speed by appropriate selection of implementation-specific compiler options and compiler directives (pragmas). As a rule, such options also include hardware-specific optimizations, as described below.

Although the selection of the compiler plays a key role from the provider's point of view, in comparisons among manufacturers, benchmark performance is the primary focus of the evaluation. Language implementations and support for language features only represent a small correction factor. However, the procurement specification always requires a minimum of standard support for both the HPC languages and the accompanying programming models. In most cases, the feature set grows between system generations.

Consequences of a System Change

A procurement that results in a switch to a new system architecture often carries considerable consequences. The relevant standards for HPC languages, programming models, and libraries guarantee that the codebase can be ported to a new platform. However, the capability to achieve the same or better performance is far from obvious. With every system change in the past, scientists had to check their program code for performance weaknesses and – possibly with the support of the data center – optimize again.

Recently, the introduction of newer architectures has shown that the performance portability problem has become increasingly worse. Regardless of whether you consider a switch to many-core (Intel Xeon Phi) systems or those that use GPU acceleration, in switching, you will often lose more than an order of magnitude of your computing power, rather than a fraction, as in the past, if the memory access patterns or memory requirements of the application are not precisely tailored to the sweet spot of the new architecture.

For example, discontinuously stored field elements in memory cannot be vectorized on current Intel processors, which can result in a performance loss factor of up to 32 when using current Single Instruction, Multiple Data Processing (SIMD) units. Similarly, users whose working data sets do not completely fit into the relatively small local main memory of an accelerator card can suffer painful performance losses that often result from the need to introduce offload data transfers. To recover performance, programmers are then forced to change previously effective data layouts, which can be a very time-consuming process for large applications.

Additionally, programmers typically have to use newer language features (e.g., the directives for asynchronous offloading of data from the host processor to an accelerator card or SIMD directives for vectorization defined in OpenMP 4.5) to create a GPU-enabled or vectorized application on many-core processors. Efficiently implementing these OpenMP concepts (or alternative models such as OpenACC) in the selected compiler suite is necessary for successful optimization.

With the high complexity of programming models, it may well be necessary, depending on the application profile, to consider a compiler alternative. Depending on the platform, the LRZ offers one or two such alternatives on its HPC systems.