How Linux and Beowulf Drove Desktop Supercomputing

Summary

Open source software development has had a huge effect on high-performance computing (HPC, ie, supercomputing). Beginning with the free software movement, epitomized by the GNU project, high-quality source code was available for use, study, modification, and extension, which led to even better software and techniques that improved HPC.

After the establishment of the free software movement, the open software movement developed, which is a bit different in that it is focused on a development model, not just fulfilling the definition of “free.” The movement showed the world that publishing source code can have a great effect on the quality of software and the resulting economic impact.

From these movements, a complete and powerful operating system was born: Linux. When Linux first came out, it was thought that it wouldn’t last very long or have much of an effect, but it’s safe to say that opinion was wrong. Linux is now arguably the dominant operating system for servers, particularly in HPC. By November 2017, the TOP500 list reported that 100% of HPC systems ran a Linux or Linux-family operating system.

Equally important to Linux in the HPC world is the Beowulf cluster. This project used commodity components coupled with Linux, other software tools, best practices, and a thriving community willing to help and share to create HPC systems that were orders of magnitude better than the previous generation of monolithic centralized supercomputers. Beowulf clusters then became the dominant HPC architecture, as evidenced by the TOP500 list.

An important aspect of Beowulf clusters is that their origin was targeted at workstations. The focus was on putting more performance in the hands of individual users and moving away from centralized single systems. Keep this in mind for the next article in the series.

Additional Resources

Levy, Steven. Hackers: Heroes of the Computer Revolution. Anchor Press/Doubleday, 1984

Raymond, Eric. The Cathedral and the Bazaar. O’Reilly Media, 1999

Wikipedia History of Linux

Digital Ocean concise history of Linux

Beowulf on the NASA website

ClusterMonkeyHPC (Doug Eadline) history of Beowulf clusters