Using benchmarks to your advantage

Node Check

From my perspective as a user, customer, developer, and administrator on one hand and as a vendor on the other, one of the most contentious issues from both sides in the HPC industry has been benchmarks.

As a user and customer, I used benchmarks to get an idea of performance and to compare product metrics, such as performance/price or performance/watt. However, benchmarks require time and effort, both to create them and to interpret the results, delaying the request for proposal while vendors take the time to produce benchmarks, and thus delaying the introduction of a new system. Moreover, after the system is installed, the requested benchmarks are often re-run to make sure it meets the vendor's guarantees. This again delays putting the system into production. On the vendor side, I used benchmarks to improve my understanding of how new systems performed, so I could make good recommendations to customers. It also helped me explain to customers how much work would be needed to port applications to these new systems.

To achieve this, standard benchmarks and commercial applications are run on the new systems and the results are published in a series of articles and blog posts. Any customer-specific benchmarks typically took a great deal of work. Because of the enormous amount of effort required in this process, both sides – customer and vendor – view benchmarks as a necessary evil. Neither side really wants them; nonetheless, they use them. That said, perhaps I can find a way to use them that isn't so evil. To begin this quest, I'll examine the benchmarks typically run when installing a system.

Installation Benchmarks

During installation, the system is reconstructed on the customer site, which includes racking and cabling the hardware and installing or checking the system software. Once the system is up and running, benchmarks are run to determine two things: Are the nodes and

... comments powered by Disqus