AlmaLinux and HPC

Linux dominates the world of high performance computing (HPC), with CentOS, a free alternative to Red Hat Enterprise Linux (RHEL), historically being the system of choice for HPC clusters used in noncommercial settings like universities and research institutions.

That all changed in late 2020 when Red Hat (a subsidiary of IBM) announced it would replace CentOS with CentOS Stream. Whereas CentOS sat downstream of RHEL making it a reliable alternative to RHEL, CentOS Stream sits upstream and serves as a developmental platform for Red Hat partners and community members who want to contribute to RHEL. Compared to CentOS’s 10-year life cycle, CentOS Stream has a 5-year life cycle and updates software versions more frequently, making it a poor choice for anyone who needs long-term support and consistency. Consequently, CentOS Stream is not intended for production environments. It lacks the reliability required for the HPC ecosystem.

Luckily, AlmaLinux, an open source, community-driven Linux, quickly arose to fill this gap by delivering a CentOS alternative in early 2021. Originally a downstream build of RHEL with 1:1 (or bug-to-bug) compatibility, AlmaLinux pivoted to being application binary interface (ABI) compatible in the summer of 2023 when Red Hat announced restrictions to RHEL-related source code. Today, AlmaLinux provides HPC users with a production grade, enterprise operating system that offers the stability and security required in HPC environments.

In terms of stability, each AlmaLinux release has a 10-year life cycle, and AlmaLinux attempts to update releases within one business day of RHEL updates. In addition, AlmaLinux’s nonprofit status, with a guaranteed $1 million in sponsorship from CloudLinux and other sponsors, assures HPC users that AlmaLinux will have the necessary funding for future updates.

AlmaLinux uses firewalld, iptables, and SELinux for security mechanisms. It also helps users build more secure environments by allowing the integration of security tools like OpenSCAP and Center for Internet Security (CIS) Benchmarks, a set of guidelines that help secure systems, software, and networks against evolving cyberthreats. With help from CloudLinux, AlmaLinux 9 OS also has achieved Federal Information Process Standard (FIPS) 140-3 validation, which means AlmaLinux 9 is using approved algorithms for encryption, hashing, assigning, and so on.

AlmaLinux’s move to ABI compatibility has freed it from being bound to bug-to-bug compatibility with RHEL. As a result, bug fixes can potentially be addressed more quickly than in RHEL. In her 2023 recap, benny Vasquez, AlmaLinux OS Foundation Chair, noted “As a result of this change we have introduced new repositories for AlmaLinux and have been able to ship critical security and bug fixes sooner than any other enterprise Linux distro.”

As a community-governed nonprofit, the AlmaLinux OS Foundation owns and manages the operating system, intellectual property, and direction of the project. Foundation membership is open to AlmaLinux OS users, project contributors, mirror maintainers/sponsors, service providers to the community, and official project partners. Not only do members vote on the AlmaLinux Board of Directors, but they also help decide the direction of AlmaLinux, including any important issues that may arise. Ultimately, the community, not a corporation, owns AlmaLinux.

As a result of AlmaLinux’s stability, security, and community governance, HPC community members have turned to AlmaLinux, including MEGWARE, CERN, and University College London (UCL). We had the opportunity to interview all three companies about their adoption of AlmaLinux in an HPC environment.

TOP500 with MEGWARE

MEGWARE, a leading European supercomputer specialist, develops and installs HPC systems and Linux clusters. Several of MEGWARE’s supercomputers have made the TOP500, a biannual ranked list of the world’s most powerful non-distributed computer systems.

In looking for a CentOS replacement in December 2020, a customer pointed MEGWARE toward AlmaLinux, which already had a beta version available. MEGWARE started their first HPC cluster with AlmaLinux in March 2021 and have been using it for almost all of their HPC cluster deployments since then.  Peter Großöhme, Chief Operating Officer and Head of Engineering at MEGWARE, says “If there is no preference from the customer’s side, MEGWARE goes with AlmaLinux.”

As for the TOP500 list, six MEGWARE HPC clusters running AlmaLinux made the TOP500 list in 2023, demonstrating AlmaLinux’s viability in the HPC marketplace:

• #84 “Goethe-NHR” at Goethe University Frankfurt – NHR@SW, Germany running AlmaLinux 9.2 (November 2023)
• #187 “Alex” at University of Erlangen-Nuremburg – NHR@FAU, Germany running AlmaLinux 8.5 (November 2023)
• #213 “Fritz” at University of Erlangen-Nuremburg – NHR@FAU, Germany running AlmaLinux 8.5 (November 2023)
• #416 “VSC-5” at Vienna University of Technology, Austria running AlmaLinux 8.5 (November 2023)
• #467 “Marvin” (Scalable GPU Partition) at University of Bonn, Germany running AlmaLinux 9.1 (November 2023)
• #483 “amplitUDE (GPU Partition)” at University of Duisburg-Essen, Germany running AlmaLinux 9.1 (June 2023)

Due to updates and security patches since making the list in 2023, the VSC-5, Alex, and Fritz clusters currently use AlmaLinux 8.9, while the Marvin and amplitUDE clusters have updated to AlmaLinux 9.3.

Being free and open source makes AlmaLinux a good fit for MEGWARE’s HPC clusters, which are used exclusively for research and education, as they offer a free alternative to RHEL. Großöhme notes, “Especially our academic customers won’t spend additional money for subscription and/or support fees for each node.”

Another advantage to AlmaLinux is its quick release cycle, even after Red Hat made it more difficult to access the official RHEL source code. Großöhme says:

“AlmaLinux provides updates and security patches very quickly – the last major update to AlmaLinux 9 was released within a few days after the official announcement by Red Hat. This is an outstanding performance compared to previous CentOS community releases and other current RHEL-based Linux distributions.”

Finally, Großöhme appreciates the way the AlmaLinux team works. He likes “the whole team and their mindset” and, in particular, AlmaLinux’s willingness “to fight for open source software after Red Hat restricted the source code access.”

CERN

CERN, the European Organization for Nuclear Research, is also using AlmaLinux. In addition to RHEL, CERN’s IT department began supporting AlmaLinux in production a little over a year ago.

Dr. Arne Wiebalck, Head of Compute and Devices Group at CERN IT, said they started offering AlmaLinux in December 2022 for CERN use cases in "close collaboration with other laboratories from the High Energy Physics community.” By early 2023, the appropriate infrastructure was in place to support AlmaLinux.

CERN mainly uses AlmaLinux for services offering interactive access to non-CERN sites as well as for virtual machine and container images that will be distributed outside of CERN. CERN also coordinates the Worldwide Large Hadron Collider (LHC) Computing Grid. This global computing infrastructure handles the vast amount of data generated by the LHC. Wiebalck notes:

“As not all participating sites have support contracts for an Enterprise Linux distribution, the deployment of a meaningful fraction of the CERN compute capacity with a free and open source Linux distribution is an important aspect of CERN's role in this community.”

When looking for an Enterprise Linux clone, Wiebalck said:

“The CERN IT Linux team did a thorough review of the characteristics of the various options. Eventually, the long life cycles for major versions, the extended architecture support, the rapid release cycle, the upstream community contributions, and the support for security advisory data made AlmaLinux the logical choice for our use cases.”

Because the LHC has multi-year run phases, CERN needs a system that offers stability and predictability. For CERN, Wiebalck says “the setup of AlmaLinux as a 501(c)(6) nonprofit organization to ensure the processes are open, transparent, and allow for participation reduces the risk of surprises and was clearly among the reasons to go with AlmaLinux.”

AlmaLinux’s move to ABI compatibility (rather than bug-to-bug compatibility) and the subsequent potential ability to address bugs and security issues before releases in other offerings gave AlmaLinux an advantage over other options.

Finally the AlmaLinux community in general was a draw for CERN. Wiebalck said:

“The CERN IT Linux team felt extremely welcome from day one; we could openly discuss issues and concerns (e.g., around support for older custom-built hardware) and always felt the AlmaLinux team was trying to find the best solution, balancing our specific needs with the requirements from the community as a whole in a very transparent way.”

In fact, AlmaLinux welcomed Alex Iribarren, a systems engineer at CERN and leader of CERN’s Cloud and Linux Platform Engineering teams, to the AlmaLinux OS Foundation board in December 2023. In the announcement, Vasquez pointed out that Iribarren’s experience “managing a large and diverse user base and his insights into the High Energy Physics community will help the AlmaLinux Foundation to meet the needs of the scientific community and contribute to the overall success of the project's mission.”