Warewulf 4 is ready to help you build an HPC cluster.
Warewulf 4
Some of the early articles I wrote for HPC ADMIN were about Warewulf 3, a cluster provisioning and management tool. The first article was published a little more than nine years ago. People seemed to enjoy the article series, so I thought I would bring you up to date with Warewulf 4.
The Warewulf tool allows you to configure a cluster head node (at least one) and boot compute nodes into a stateless (i.e., “diskless,”) configuration. It’s all open source with a good number of people behind it. Warewulf 3 is used by OpenHPC, and in an upcoming version, OpenHPC will switch over to Warewulf 4.
Warewulf 3 was very widely used and still is. Early attempts to create a Warewulf 4 did not really stick; then, the original developer, Greg Kurtzer, started working on a new version and formed a company, CIQ, to develop it and the next generation of HPC tools, Fuzzball. While doing all of this work, CIQ also started Rocky Linux.
Kurtzer wanted to update Warewulf 4 with more modern concepts. For example, the Virtual Node File System (VNFS), which manages the node operating systems (OSs) in Warewulf 3, now can be a container. However, you can still use the VNFS from a chroot filesystem if you want and convert it to a container. Warewulf 4 has gone beyond just a simple group of compute nodes that create a cluster to defining many “roles” and using them to boot various nodes in the cluster that have different functions (e.g., login, storage, or visualization nodes). You can have different types of compute nodes on the basis of hardware differences or node OS images based on a scheduler so that some nodes use Slurm or some nodes use Kubernetes.
Currently in Warewulf 4, only stateless nodes are accommodated. In the past, Warewulf allowed for stateful nodes (i.e., nodes that put the image on local storage). Of course, a few requests have been made for this in Warewulf 4, but stateless booting is much more popular than stateful. You can read a bit more about stateless nodes in the Warewulf documentation.
In this article, I cover how to configure a head node and boot stateless compute nodes for a simple configuration.
Configuration
The physical configuration I used to illustrate the installation of Warewulf was very simple (Figure 1): just two nodes – a head node and one compute node – and a single internal Gigabit Ethernet (GigE) switch (unmanaged in this case).
Figure 1: Physical configuration.
The head node has two GigE network interface cards (NICs), with one on the motherboard connected to the local LAN using the Dynamic Host Configuration Protocol (DHCP). This NIC is my home network (192.168.x.x). The second NIC is connected to a private Ethernet network (10.0.0.1, subnet mask 255.255.252.0), which follows the network addressing discussed in the Warewulf documentation. I only have this one cluster network, with no intelligent platform management interface (IPMI), storage, or additional nodes. I will use the head node as a network filesystem (NFS) server.
One step I did have to take, which I learned the hard way, was to check the BIOS of all the nodes, except the head node, to make sure they would boot as the first choice in the preboot execution environment (PXE). You also need to make sure that PXE can use the Unified Extensible Firmware Interface (UEFI) instead of 32-bit Legacy booting. Many times the container for the compute image is too large for Legacy booting. If you have trouble, use the Warewulf mailing list to ask for help.
Head Node Installation
I began with a freshly installed Rocky 8.6 distribution. I accepted the defaults to a Server with a GUI installation. I did make some changes to the OS. I can’t stand the Gnome desktop, so I installed the Mate desktop, but this change shouldn’t affect anything. Once I installed the OS, I disabled the firewall on head node:
$ sudo systemctl stop firewalld $ sudo systemctl disable firewalld
I believe there are some simple steps for configuring Warewulf 4 while keeping the firewall up (see the documentation), but for illustrative purposes, I disabled the firewall.
Additionally, I disabled SELinux for the purposes of this article, but it’s possible to keep it functioning while using Warewulf. Disabling SELinux is the subject of a number of articles online. It requires a reboot to complete.
Don’t be afraid to add packages to your head node installation. They are unlikely to interfere with Warewulf. However, Warewulf can take care of installing and configuring three packages it uses for provisioning nodes:
- dhcp
- tftp
- nfs-utils
One more thing, when you create a user such as administrator or some other account that will be the primary Warewulf administrator, be sure to allow that user to be an administrator on the system so they can use sudo to run commands.
The next step is to install Warewulf. The documentation covers several different methods, from cloning the Git repo to downloading the source. In general, the head node needs access to the outside world (i.e., to a local repository that holds the RPMs and their dependencies).
I chose to use binary RPMs because they are a little easier for me to update Warewulf. Listing 1 shows the results of the documentation guidance.
Listing 1: Warewulf Installation
$ sudo yum install -y https://repo.ctrliq.com/rhel/8/ciq-release.rpm $ yum install -y warewulf CIQ Stable 12 kB/s | 32 kB 00:02 Last metadata expiration check: 0:00:01 ago on Fri 04 Nov 2022 02:42:11 PM EDT. Dependencies resolved. ================================================================================================= Package Architecture Version Repository Size ================================================================================================= Installing: warewulf x86_64 4.3.0-1.git_235c23c.el8 ciq 8.3 M Installing dependencies: bind-export-libs x86_64 32:9.11.36-3.el8_6.1 baseos 1.1 M dhcp-common noarch 12:4.3.6-47.el8 baseos 206 k dhcp-libs x86_64 12:4.3.6-47.el8 baseos 147 k dhcp-server x86_64 12:4.3.6-47.el8 baseos 529 k tftp-server x86_64 5.2-24.el8 appstream 49 k Transaction Summary ================================================================================================= Install 6 Packages Total download size: 10 M Installed size: 31 M Downloading Packages: (1/6): dhcp-common-4.3.6-47.el8.noarch.rpm 902 kB/s | 206 kB 00:00 (2/6): dhcp-libs-4.3.6-47.el8.x86_64.rpm 3.1 MB/s | 147 kB 00:00 (3/6): tftp-server-5.2-24.el8.x86_64.rpm 123 kB/s | 49 kB 00:00 (4/6): dhcp-server-4.3.6-47.el8.x86_64.rpm 3.9 MB/s | 529 kB 00:00 (5/6): bind-export-libs-9.11.36-3.el8_6.1.x86_64.rpm 579 kB/s | 1.1 MB 00:02 (6/6): warewulf-4.3.0-1.git_235c23c.el8.x86_64.rpm 746 kB/s | 8.3 MB 00:11 ------------------------------------------------------------------------------------------------- Total 799 kB/s | 10 MB 00:13 CIQ Stable 1.6 MB/s | 1.7 kB 00:00 Importing GPG key 0xD71AB47F: Userid : "Ctrl IQ, Inc. " Fingerprint: 0BF7 3422 0FF7 326C 5DE9 E196 E97E 8F85 D71A B47F From : /etc/pki/rpm-gpg/RPM-GPG-KEY-ctrliq Key imported successfully Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : dhcp-libs-12:4.3.6-47.el8.x86_64 1/6 Installing : dhcp-common-12:4.3.6-47.el8.noarch 2/6 Installing : bind-export-libs-32:9.11.36-3.el8_6.1.x86_64 3/6 Running scriptlet: bind-export-libs-32:9.11.36-3.el8_6.1.x86_64 3/6 Running scriptlet: dhcp-server-12:4.3.6-47.el8.x86_64 4/6 Installing : dhcp-server-12:4.3.6-47.el8.x86_64 4/6 Running scriptlet: dhcp-server-12:4.3.6-47.el8.x86_64 4/6 Installing : tftp-server-5.2-24.el8.x86_64 5/6 Running scriptlet: tftp-server-5.2-24.el8.x86_64 5/6 Running scriptlet: warewulf-4.3.0-1.git_235c23c.el8.x86_64 6/6 Installing : warewulf-4.3.0-1.git_235c23c.el8.x86_64 6/6 Running scriptlet: warewulf-4.3.0-1.git_235c23c.el8.x86_64 6/6 Verifying : tftp-server-5.2-24.el8.x86_64 1/6 Verifying : bind-export-libs-32:9.11.36-3.el8_6.1.x86_64 2/6 Verifying : dhcp-common-12:4.3.6-47.el8.noarch 3/6 Verifying : dhcp-libs-12:4.3.6-47.el8.x86_64 4/6 Verifying : dhcp-server-12:4.3.6-47.el8.x86_64 5/6 Verifying : warewulf-4.3.0-1.git_235c23c.el8.x86_64 6/6 Installed: bind-export-libs-32:9.11.36-3.el8_6.1.x86_64 dhcp-common-12:4.3.6-47.el8.noarch dhcp-libs-12:4.3.6-47.el8.x86_64 dhcp-server-12:4.3.6-47.el8.x86_64 tftp-server-5.2-24.el8.x86_64 warewulf-4.3.0-1.git_235c23c.el8.x86_64 Complete!
I apologize for the length, but I wanted you to see what Warewulf installed (it’s not too much). For this article, Warewulf 4.3 was the current stable release, but the Warewulf 4.4 release candidate is out now.
Once installed, you can configure Warewulf 4 on the head node (also called the host node).
Configuring Warewulf on the Head Node
The configuration files for Warewulf are in /etc/warewulf. The primary command you will use to interact with Warewulf is wwctl. The first file you should examine is /etc/warewulf/warewulf.conf. My version of this file is shown in Listing 2.
Listing 2: warewulf.conf
WW_INTERNAL: 43 ipaddr: 10.0.0.1 netmask: 255.255.252.0 network: 10.0.0.0 warewulf: port: 9873 secure: false update interval: 60 autobuild overlays: true host overlay: true syslog: false datastore: "" dhcp: enabled: true template: default range start: 10.0.1.1 range end: 10.0.1.255 systemd name: dhcpd tftp: enabled: true tftproot: "" systemd name: tftp nfs: enabled: true export paths: - path: /home export options: rw,sync mount options: defaults mount: true - path: /opt export options: ro,sync,no_root_squash mount options: defaults mount: false systemd name: nfs-server
The default warewulf.conf can be different because it will be based on the IP addresses of the primary NIC in the head node. You will likely have to edit the file, so I’ll walk you through that.
If you refer back to Figure 1 for my setup, the internal cluster network should be 10.0.0.1 (Listing 2, line 2). I also changed the netmask so I’d have room to add nodes, although this is my home cluster, so I won’t have many nodes. This netmask must also match the host’s subnet mask for the cluster’s private interface. Otherwise, you run the danger of the various networks not being compatible, and nodes may not be able to communicate.
I made the network 10.0.0.0 so that it can address anything I am likely to put on the cluster network. You can pick whatever you want to match the number of nodes you think you are likely to need or use.
The next thing I changed was the range of cluster IP addresses. In the dhcp section of warewulf.conf, I changed the range start point to 10.0.1.1 and the last point to 10.0.1.255. If you need to go larger than 254 nodes, you can change the range end to a different IP address (e.g., 10.0.255.255, which would give you 65,636 nodes). Just be sure the subnet mask matches the address range.
For this tutorial, I left everything else the same. Take a look the subsequent sections, particularly the nfs section. By default, Warewulf exports /home and /opt. You can NFS export whatever you want. You can also change whether the compute nodes mount filesystems from nodes other than the head node. You can easily change NFS mountpoints later and just reboot the nodes.
The Warewulf documentation explains the other variables in /etc/warewulf/warewulf.conf.
Another file in /etc/warewulf is worth mentioning: nodes.conf is a simple, flat text file in YAML format that acts as the database of the compute nodes (pretty much anything that gets an image from Warewulf). You can edit this file by hand, but you should know what the various fields mean before you proceed, because you could fairly easily change or remove things in the file that will cause corruption, forcing you to redo all of the compute node configuration. If you are going to edit the file by hand, I would recommend using a good YAML lint tool to make sure at least the YAML is correct. Overall, wwctl is recommended to make changes to nodes. This tool updates the nodes.conf file and preventscorruption. Keeping a copy of nodes.conf in a safe place is a good idea, so if it gets corrupted, you can restore it easily.
Initializing Warewulf
At this point, Warewulf has been installed and configured. Now you can initialize it and have it start the needed system services. The general command with some sample output is shown in Listing 3.
Listing 3: Initializing Warewulf
$ sudo wwctl configure --all [sudo] password for laytonjb: Building overlay for localhost.localdomain: host Enabling and restarting the DHCP services Created symlink /etc/systemd/system/multi-user.target.wants/dhcpd.service --> /usr/lib/systemd/system/dhcpd.service. Building overlay for localhost.localdomain: host Enabling and restarting the NFS services Created symlink /etc/systemd/system/multi-user.target.wants/nfs-server.service --> /usr/lib/systemd/system/nfs-server.service. Updating system keys Setting up key: ssh_host_rsa_key Setting up key: ssh_host_dsa_key Setting up key: ssh_host_ecdsa_key Setting up key: ssh_host_ed25519_key Skipping, authorized_keys already exists: /root/.ssh/authorized_keys Writing PXE files to: /var/lib/tftpboot/warewulf Enabling and restarting the TFTP services Created symlink /etc/systemd/system/sockets.target.wants/tftp.socket --> /usr/lib/systemd/system/tftp.socket.
This command does quite a bit, and you can use it to great effect when administering clusters. I’ll walk through the output so you can gain a better understanding of what it did.
To begin, it builds an overlay for the local host. (I didn’t give it a name, so it’s known as localhost.localdomain) then configures and starts the DHCP service from a template in /etc/warewulf/dhcpd and updates the head node’s /etc/hostsfile according to the template in /etc/warewulf/hosts.tmpl. Next, it configures the NFS server on the head node from the entries near the end of /etc/warewulf/warewulf.conf. Note that it will update /etc/fstab in the system overlay, as well. Finally, it starts the NFS server service and exports the filesystem (exportfs -a).
Warewulf authenticates with SSH keys and uses passwordless SSH for communicating between nodes. After configuring NFS, it will configure keys, creating the appropriate one in /etc/warewulf/keys. Look in that directory to see what Warewulf created.
The last service the wwctl configure -all command configures is TFTP by copying the appropriate binary PXE/iPXE blobs to the tftp root directory and enabling and starting the tftp service.
Warewulf has accomplished quite a lot, and the defaults should be good enough to start. However, you can easily change anything, if needed, including changing the templates and rerunning the command. Only do this if you know what you’re doing, though.
Checking the Services
System services don’t always come up the way I intend, so I always, always check them. I think this is very important, particularly after a reboot of the head node. I’ll start with DHCP (Listing 4). The first lines of data with date stamps can be ignored. They are only saying that it is ignoring requests from the enp2s0 NIC, which is the NIC to the outside world; enp3s0 is the NIC to the cluster network. Overall, the DHCP system service is up and functioning.
Listing 4: Checking DHCP Service
$ sudo systemctl status dhcpd dhcpd.service - DHCPv4 Server Daemon Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2022-11-05 08:51:10 EDT; 35s ago Docs: man:dhcpd(8) man:dhcpd.conf(5) Main PID: 4866 (dhcpd) Status: "Dispatching packets..." Tasks: 1 (limit: 48440) Memory: 9.7M CGroup: /system.slice/dhcpd.service |-4866 /usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid Nov 05 08:51:10 localhost.localdomain dhcpd[4866]: Nov 05 08:51:10 localhost.localdomain dhcpd[4866]: No subnet declaration for enp2s0 (192.168.5.18). Nov 05 08:51:10 localhost.localdomain dhcpd[4866]: ** Ignoring requests on enp2s0. If this is not wh> Nov 05 08:51:10 localhost.localdomain dhcpd[4866]: you want, please write a subnet declaration Nov 05 08:51:10 localhost.localdomain dhcpd[4866]: in your dhcpd.conf file for the network segment Nov 05 08:51:10 localhost.localdomain systemd[1]: Started DHCPv4 Server Daemon. Nov 05 08:51:10 localhost.localdomain dhcpd[4866]: to which interface enp2s0 is attached. ** Nov 05 08:51:10 localhost.localdomain dhcpd[4866]: Nov 05 08:51:10 localhost.localdomain dhcpd[4866]: Sending on Socket/fallback/fallback-net Nov 05 08:51:10 localhost.localdomain dhcpd[4866]: Server starting service.
Listing 5 shows the output for the TFTP service. Now that TFTP is up and running, check NFS (Listing 6).
Listing 5: Checking TFTP Service
$ sudo systemctl status tftp tftp.service - Tftp Server Loaded: loaded (/usr/lib/systemd/system/tftp.service; indirect; vendor preset: disabled) Active: active (running) since Sat 2022-11-05 08:51:11 EDT; 1min 8s ago Docs: man:in.tftpd Main PID: 4961 (in.tftpd) Tasks: 1 (limit: 48440) Memory: 236.0K CGroup: /system.slice/tftp.service |-4961 /usr/sbin/in.tftpd -s /var/lib/tftpboot Nov 05 08:51:11 localhost.localdomain systemd[1]: Started Tftp Server.
Listing 6: Checking NFS Service
$ sudo systemctl status nfs-server nfs-server.service - NFS server and services Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled) Drop-In: /run/systemd/generator/nfs-server.service.d |-order-with-mounts.conf Active: active (exited) since Sat 2022-11-05 08:51:10 EDT; 1min 29s ago Main PID: 4924 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 48440) Memory: 0B CGroup: /system.slice/nfs-server.service Nov 05 08:51:10 localhost.localdomain systemd[1]: Starting NFS server and services... Nov 05 08:51:10 localhost.localdomain systemd[1]: Started NFS server and services.
You can also verify with the sudo exportfs command. By default, you should see two filesystems exported: /home and /opt.
If the services are not running, the commands
$ sudo systemctl enable --now tftp $ sudo systemctl start tftp
enable and start the services. If you don’t use the --now option, they start on the next boot. Strictly speaking, the second command shouldn't be necessary, but I like to use it to be sure. (It won’t hurt to use it.)
All of the dependent services have been started and are running at this point, so it’s time to start the Warewulf daemon itself (Listing 7).
Listing 7: Starting the Warewulf Daemon
$ sudo systemctl enable --now warewulfd Created symlink /etc/systemd/system/multi-user.target.wants/warewulfd.service --> /usr/lib/systemd/system/warewulfd.service. $ sudo wwctl server status SERV : Warewulf server is running at PID: 5127
As part of your system checks, you should also check warewulfd. Although I’ve never had any problems with it, it is always good to check.
At this point, all services should be up and running. If you like, look at the Warewulf logs, which should be pretty empty because the cluster is newly created:
$ sudo more /var/log/warewulfd.log [Sat Nov 5 08:53:25 EDT 2022] SERV : Starting HTTPD REST service on port 9873
As expected, nothing is really in there. Now you're ready to tackle the compute nodes.
Container Management for Compute Nodes
If you have used Warewulf before, you are familiar with VNFS. As the name suggests, it’s a virtual filesystem that is basically the OS image for the compute nodes. In Warewulf 3 and before, it was created in a chroot directory, which in effect is a container but is really a directory on the head node system that serves as a root directory. Warewulf 4 has moved to using the term “container” more explicitly and using the container apparatus (container) for creating and using compute node OS images.
Warewulf 4 does not restrict you to using a container for your OS image. You can still use chroot to create the VNFS and then very easily convert it to a container.
Warewulf supports importing an image from any Open Container Initiative (OCI)-compliant registry, meaning you can import from a public registry or from a private registry – either Docker or Singularity (Apptainer, too, I suppose, but I don’t know for sure).
The container for the compute nodes is not just an ordinary container. It has to be bootable and should have a limited version of systemd to make it lighter weight for container purposes.
If you pull and try to boot a container such as docker://rockylinux, it will not boot properly; rather, it will boot into single-user mode and get stuck. It is better to look for containers in Docker Hub and even ask on the Warewulf mailing list or Slack channel about bootable containers. Right now (November 2022), I see five containers under the Warewulf account.
I chose to pull a Rocky 8 container from the warewulf repo on Docker Hub (Listing 8). A quick note in case it is subtle: the head node needs access to the outside world to pull the container unless you have added it to a local repository.
Listing 8: Pull Container on Docker Hub
[laytonjb@localhost ~]$ sudo wwctl container import docker://warewulf/rocky rocky-8 Getting image source signatures Copying blob 1a930d163dca done Copying blob d3ca234f568b done Copying blob 3b38e1016650 done Copying config aff218cbc1 done Writing manifest to image destination Storing signatures 2022/11/05 08:56:13 info unpack layer: sha256:1a930d163dcafa193dc2c3c005d9c220ae1c07a48cad5f7feed0066ada0b998f 2022/11/05 08:56:15 info unpack layer: sha256:d3ca234f568b088b991388a0e9e8b61b05ac8627522f10fe16df2b81d51c0748 2022/11/05 08:56:22 info unpack layer: sha256:3b38e1016650813dc5e3bb98cabcc9bf6167399caee7806bc1ca9f3df9825f2e Updating the container's /etc/resolv.conf uid./gid not synced, run wwctl container syncuser --write rocky-8 to synchronize uid/gids. Building container: rocky-8 Created image for VNFS container rocky-8: /var/lib/warewulf/container/rocky-8.img Compressed image for VNFS container rocky-8: /var/lib/warewulf/container/rocky-8.img.gz
The Warewulf container documentation has good information on importing a container from a private repository, creating containers from scratch with something like chroot, and building a container from Singularity.
$ sudo wwctl container list CONTAINER NAME NODES KERNEL VERSION rocky-8 0 4.18.0-372.13.1.el8_6.x86_64
Houston, we have our first container. Before going any further, I want to show you that you can update, change, and modify your container very easily. Just like other containers, you can exec into that container, poke around, and make changes if you like (Listing 9). Notice that when you exit the container, it is automatically rebuilt, even if you have made no changes. Later in this article and in future articles, modifying containers on the head node will be important.
Listing 9: Modifying Containers on the Head Node
[laytonjb@localhost keys]$ sudo wwctl container exec rocky-8 /bin/bash [rocky-8] Warewulf> cat /etc/rocky-release Rocky Linux release 8.6 (Green Obsidian) [rocky-8] Warewulf> exit exit + LANG=C + LC_CTYPE=C + export LANG LC_CTYPE + dnf clean all 0 files removed Rebuilding container... Created image for VNFS container rocky-8: /var/lib/warewulf/container/rocky-8.img Compressed image for VNFS container rocky-8: /var/lib/warewulf/container/rocky-8.img.gz
Compute Node Kernel
Compute nodes need a Linux kernel to boot. Warewulf allows you to separate the kernel from the container, which can give you great flexibility; however, you have to check that the container can boot and function with a different kernel.
The simplest way is to install the desired kernel in the container or use the one that comes with the container. Warewulf will locate the kernel in an uploaded container by default and use it for any node that uses that container. Note that it can use the same container for nodes but have them boot with different kernels, but please test the combination before putting it in production.
Combining a container with a kernel is done with a profile. Personally, I like building containers with specific kernels rather than mixing and matching, but I can see the utility of being able to use separate kernels.
You can see the kernel used in the container with the command:
$ sudo wwctl container list CONTAINER NAME NODES KERNEL VERSION rocky-8 0 4.18.0-372.13.1.el8_6.x86_64
To get a list of available kernels, use the command:
$ sudo wwctl kernel list KERNEL NAME KERNEL VERSION NODES 4.18.0-372.9.1.el8.x86_64
I won’t spend more time discussing kernels, so I suggest you read the appropriate section in the Warewulf documentation and perhaps ask questions on the Warewulf mailing list or Slack channel.
Configuring Compute Nodes
You’re almost ready to start booting compute nodes, but you need to configure them in Warewulf before doing so. Recall that /etc/warewulf/nodes.conf is the database of compute nodes in a simple flat text YAML format. You can get the list of defined compute nodes with the command:
$ sudo wwctl node list -a
The command at this point should have no output, unless you or someone else with Warewulf access has created one. If you see that any node has been created that you don’t expect, you might want to uninstall Warewulf and start over.
Adding a node is as simple as:
$ sudo wwctl node add --netdev eth0 --hwaddr XX:XX:XX:XX:XX:XX --ipaddr 10.0.1.1 --netmask 255.0.0.0 n0001
You don't have to know the MAC address (-hwaddr …); however, I like knowing them for my nodes before booting. This information can be found by booting the nodes and going into the BIOS and looking for the MAC address. (If you have more than one NIC, it is likely the first.)
If you are using Warewulf with servers, your hardware provider should have a list of MAC addresses. If they rack the servers, they should have a list as they are installed in the rack.
In my case, I knew the MAC address of the first node I was to boot (I looked in the BIOS), so I used it to associate the MAC address to the IP address and the node (n0001). I like to start with node 1 rather than node 0, but that is personal preference and is a little different than in the Warewulf documentation. One important thing to note is that the IP address matches the one I used in /etc/warewulf/warewulf.conf to define the start of the address range.
If you don't know the MAC address of the compute node, you can boot the nodes one by one and have Warewulf capture the MAC–node name association. When you define a node in Warewulf for which you don’t know the MAC, you use the command:
$ sudo wwctl node add NODENAME.CLUSTER --ipaddr aaa.bbb.ccc.ddd --discoverable
The key is to add the option --discoverable, which marks a field in nodes.conf named Discoverable to true so that a MAC address can be assigned to a node that doesn't currently have one. You can also leave out the --netdev option, which defaults to eth0. You can do this for any or all nodes in the cluster.
Now boot the next node. Warewulf assigns the next discovered MAC to the next node listed as discoverable in nodes.conf. The MAC gets added to the entry for the node and discovery is set to false so it won’t be assigned to a different node.
The key point is to do this one node at a time. Power on a node and make sure it gets the proper IP address and continues to boot, and you can access it over SSH. Be sure to check the nodes under Warewulf’s control with:
$ sudo wwctl node list -a
You should see the current node, the MAC address, and that the discoverable flag is false.
If you are using Warewulf to provision and manage nodes other than compute nodes, you might want to add an extension to the node name, such as node0001.cluster. You can also use node name extensions to help delineate what nodes are doing what. For example, you could label the nodes by rack such as n0001.rack1.cluster, and so on. Choose a naming scheme that works for you and helps you administer the system. Just be sure to document it!
When first booting a Warewulf cluster, I like to be cautious. Therefore, I used an extremely large subnet mask to make sure the head node and the compute node can communicate. After I use this large netmask, I go back and edit the node properties to have it match the head node netmask. You can get a list of the defined nodes as in Listing 10.
Listing 10: List of Defined Nodes
$ sudo wwctl node list -a ################################################################################ NODE FIELD PROFILE VALUE n0001 Id -- n0001 n0001 Comment default This profile is automatically included for each node n0001 Cluster -- -- n0001 Profiles -- default n0001 Discoverable -- false n0001 Container -- -- n0001 KernelOverride -- -- n0001 KernelArgs -- (quiet crashkernel=no vga=791) n0001 SystemOverlay -- (wwinit) n0001 RuntimeOverlay -- (generic) n0001 Ipxe -- (default) n0001 Init -- (/sbin/init) n0001 Root -- (initramfs) n0001 AssetKey -- -- n0001 IpmiIpaddr -- -- n0001 IpmiNetmask -- -- n0001 IpmiPort -- -- n0001 IpmiGateway -- -- n0001 IpmiUserName -- -- n0001 IpmiInterface -- -- n0001 IpmiWrite -- false n0001 default:DEVICE -- eth0 n0001 default:HWADDR -- c8:5a:cf:0a:a8:68 n0001 default:IPADDR -- 10.0.1.1 n0001 default:IPADDR6 -- -- n0001 default:NETMASK -- (255.0.0.0) n0001 default:GATEWAY -- -- n0001 default:TYPE -- -- n0001 default:ONBOOT -- true n0001 default:PRIMARY -- true
The values listed in parentheses are values that can be changed with wwctl.
At this point, I like to assign the container to the node with a simple command:
$ sudo wwctl node set --container rocky-8 n0001 Are you sure you want to modify 1 nodes(s): y $ sudo wwctl node list -a n0001 | grep Container n0001 Container -- rocky-8
One thing a friend recommended that I thought was a really good idea was to add a password to the root account in the container so I could log in to the compute node with a monitor and keyboard. This step really helps with debugging, particularly with misconfigured networks.
To create a password for the root account, you must install the passwd tool in the container. I won’t show all the output of this process because it can be a bit long, but I least wanted to show you the steps. By the way, this is a great way to illustrate how to update the container.
To add packages to the container, exec into the container:
$ sudo wwctl container exec rocky-8 /bin/bash [rocky-8] Warewulf>
Once “inside” the container, you can use yum to install passwd, the package for the password tool (Listing 11), then create a password for the root user (Listing 12). Again, notice that on exit, the container is rebuilt.
Listing 11: Installing passwd
[rocky-8] Warewulf> yum install passwd Failed to set locale, defaulting to C.UTF-8 Rocky Linux 8 - AppStream 2.2 MB/s | 11 MB 00:05 Rocky Linux 8 - BaseOS 1.4 MB/s | 9.0 MB 00:06 Rocky Linux 8 - Extras 12 kB/s | 12 kB 00:00 Dependencies resolved. ...
Listing 12: Creating a Password Root
[rocky-8] Warewulf> whereis passwd passwd: /usr/bin/passwd /etc/passwd /usr/share/man/man1/passwd.1.gz [rocky-8] Warewulf> passwd root Changing password for user root. New password: Retype new password: passwd: all authentication tokens updated successfully. [rocky-8] Warewulf> exit exit + LANG=C + LC_CTYPE=C + export LANG LC_CTYPE + dnf clean all 25 files removed Rebuilding container... Created image for VNFS container rocky-8: /var/lib/warewulf/container/rocky-8.img Compressed image for VNFS container rocky-8: /var/lib/warewulf/container/rocky-8.img.gz
The passwd tool is installed and you have a password for root that you can use to check the compute nodes.
Booting Nodes
At this point, you can boot compute nodes. For the first compute node, I would plug in a monitor and keyboard to watch the boot process and debug what is happening. After that, you can just turn on the power to the node.
You should see the node get the correct the IP address over DHCP; then you should see iPXE get copied over, and the node starts booting. The container is copied over the node and pivot boots into the container. Finally, you should see a prompt on the monitor plugged into the node indicating that the node is ready to be used.
Because only SSH is used for authentication, you won’t be able to log in to the compute node as a user; however, the rootuser has a password, so you can log in as root with the monitor and keyboard. When you do this, poke around the compute node and make sure everything looks correct. For example,
- check that the NFS exported files have mounted (use the mount command),
- look at the local IP address (ip addr show),
- ping the host node with the IP address (ping 10.0.0.1), and
- check the hostname of the node (more /etc/hostname).
These are just a few things you can do to check the compute node, but you can do whatever you like to satisfy that the compute node is up and working correctly.
At this point, you might go back to the host node and check out the compute node. The first thing to do is ping the IP address of the compute node (Listing 13) then ping the hostname of the compute node (Listing 14).
Listing 13: Pinging the Compute IP Address
$ ping 10.0.1.1 PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data. 64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=1.50 ms 64 bytes from 10.0.1.1: icmp_seq=2 ttl=64 time=2.56 ms ^C --- 10.0.1.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 1.503/2.032/2.562/0.531 ms
Listing 14: Pinging the Compute Hostname
$ ping n0001 PING n0001-default (10.0.1.1) 56(84) bytes of data. 64 bytes from n0001-default (10.0.1.1): icmp_seq=1 ttl=64 time=2.20 ms 64 bytes from n0001-default (10.0.1.1): icmp_seq=2 ttl=64 time=2.44 ms ^C --- n0001-default ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 2.200/2.321/2.443/0.130 ms
Being able to pin the hostnames is a good sign because the default host overlay that takes care of the hostnames doesn’t always work in Warewulf version 4.3. If you can’t ping the hostname, you need to rerun the wwctl configure -a command and reboot the compute node (Listing 15). This problem should be fixed in Warewulf 4.4 or a later version.
Listing 15: Enabling Services
$ sudo wwctl configure -a Building overlay for localhost.localdomain: host Enabling and restarting the DHCP services Building overlay for localhost.localdomain: host Enabling and restarting the NFS services Updating system keys Skipping, key already exists: ssh_host_rsa_key Skipping, key already exists: ssh_host_dsa_key Skipping, key already exists: ssh_host_ecdsa_key Skipping, key already exists: ssh_host_ed25519_key Skipping, authorized_keys already exists: /root/.ssh/authorized_keys Writing PXE files to: /var/lib/tftpboot/warewulf Enabling and restarting the TFTP services
After pinging the compute nodes, try to SSH into the node as a user to make sure authentication is working:
$ ssh 10.0.1.1 [laytonjb@n0001 ~]$ logout Connection to 10.0.1.1 closed.
That operation was successful, so now try using the hostname:
$ ssh n0001 Warning: Permanently added 'n0001' (ECDSA) to the list of known hosts. Last login: Thu Nov 17 13:58:01 2022 from 10.0.0.1 [laytonjb@n0001 ~]$ ls -s total 24 0 Desktop 0 Downloads 0 Pictures 0 Templates 4 ww4-2_notes.txt 0 Documents 0 Music 0 Public 0 Videos 20 ww4_notes.txt
That was successful too, and you can see the files on the compute node, so I would say you can successfully boot compute nodes now.
Review and Debugging Tips
The work in this article accomplished some great things. Overall, on the head node Warewulf 4 was configured for booting one group of compute nodes. To do this, Warewulf was installed, configured, and initialized on the head node; a container OS was imported for use on the compute nodes; and after one small update to allow root to log in locally, the first compute node was configured with the container, IP address, MAC address, and kernel in the container. Finally, the first compute node booted. You could do even more if you wanted.
If you need to debug as you are bringing up the compute node:
- Be sure to check the BIOS of your compute nodes so the first boot option is DHCP and it can use UEFI. Legacy mode for DHCP is not likely to work because the size of the container is too large (Legacy mode is 32-bit). You can test this on a compute node by rebooting it even if Warewulf is not configured. The node should come up looking for a DHCP server (plug in to the node a monitor, keyboard, and mouse).
- Check the netmask defined for your compute nodes with the netmask for the host node’s cluster interface. (Be sure it is the network interface used for the cluster network.) Make sure they are the same or at least compatible. This bit me a few times when I didn’t pay close enough attention and fat-fingered the subnet mask.
- Always, always, always check the status of dhcpd, tftp, nfs-server, and warewulfd system services, as described in a subsection in this article. I found that if I shut down the head node for a bit and then rebooted, one of the services almost always failed to start. I don’t know why, even though I enabled them with the enable --now option every time and checked them. For some reason, though, they didn’t always start.
- Plug a monitor and keyboard into your compute node when you are booting it for the first time. It can save you a lot of headaches, especially if you give the root account a password by installing the passwd tool. You can check whether DHCP is working, whether tftp is working, and whether the node is booting into the container. If the node comes up with a prompt, you can log in as root locally on the node and test network connectivity, NFS, and a few other things. Use ping to check the IP address and the node name.
- After booting your first compute node, check that the network on the host and compute node use hostnames and not just IP addresses. If this doesn’t work, run the command wwctl configure -a on the head name again and reboot the compute node.
Summary
Booting compute nodes is a huge step forward in building your cluster. Without this step, everything following is kind of meaningless. However, I am still a few steps away from having what I think is a modern cluster with compilers, MPI libraries, and a resource manager (aka a job scheduler). Moreover, my compute nodes have GPUs in them, and they are currently not configured.
In subsequent articles I will explore how to configure GPUs in relation to Warewulf 4 to illustrate how you build a modern cluster with all of the tools and accelerators (GPUs).