It’s the Little Things

SSH and PDSH

It might sound rather obvious, but two of the tools I rely on the most are a simple secure tool for remote logins, ssh , and a tool that uses ssh to run commands on remote systems in parallel or in groups, pdsh . When clusters began, the tool of choice for remote logins was rsh . It had been around for a while and was very easy to use. However, it was very insecure.

It transmitted data from the host machine to the target machine with no encryption – including passwords. Therefore, a very simple network sniff could gather lots of passwords. Using rsh , rlogin , or telnet between systems across the Internet, such as when users logged into the cluster, left you wide open to attacks and password sniffing. Very quickly people realized that something more secure was needed.

In 1995, researcher Tatu Ylönen created Secure Shell (SSH) because of a password sniffing attack on his network. It gained popularity, and SSH Communications Security was founded to commercialize it. OpenBSD community grabbed the last open version of SSH and developed it into OpenSSH. After gaining popularity in the early 2000s, the cluster community grabbed it and started using it to help secure clusters.

SSH is extremely powerful. Beyond just remote logins and running commands on a remote node, it can be used for tunneling or forwarding other protocols over SSH. Specifically, you can use SSH to forward X from a remote host to your desktop and copy data from one host to another (scp ), and you can use it in combination with rsync  to back up, mirror, or copy data.

SSH was a great development for clusters and HPC, because there was finally a way to log into systems and send commands securely and remotely. However, SSH could only do this for a single system. HPC systems can comprise hundreds or thousands of nodes, which needed a way to send the same command to a number of nodes or a fixed set of nodes.

In a past article, I wrote about a class of tools that accomplishes this goal. These tools are parallel shells, and a number of them meet different needs. The most common is pdsh. In theory, it is fairly simple. It uses a specific remote command to run the same command on the specified nodes. You have a choice of underlying tools to use when you build and use pdsh. I prefer to use SSH because of the security if offers.

To create a simple file containing the list of hosts you want PDSH to use as default:

export WCOLL=/home/laytonjb/PDSH/hosts

WCOLL is an environment variable that points to the location of the file with the list of hosts. You can put this command in your .bashrc file in your home directory, or you can put it in the global .bashrc file.

SSHFS

I've written about SSHFS in the past. It has to be one of the most awesome filesystem tools I have ever used. It is a FUSE-based userspace client that mounts and interacts with a remote filesystem as though the filesystem were local (i.e., shared storage). It uses SSH as the underlying protocol and SFTP as the transfer protocol, so it’s as secure as SFTP.

SSHFS can be very handy when working with remote filesystems, especially if you only have SSH access to the remote system. Moreover, you don’t need to add or run a special client tool on the client nodes or a special server tool on the storage node. You just need SSH active on your system. Almost all firewalls allow port 22 access, so you don’t have to configure anything extra (e.g., NFS or CIFS); you just need one open port on the firewall – port 22. All the other ports can be blocked.

Many filesystems have encryption of data at rest. Using SSHFS in combination with an encrypted filesystem ensures that your data is encrypted at rest and “over the wires,” which prevents packet sniffing of data within or outside the cluster, an important consideration in our current mobile society where users want to access their data from multiple places with multiple devices.

A quick glance at SSHFS performance indicates that the sequential read and write performance is on par with NFS. However, random I/O performance is less efficient than NFS. Fortunately, you can tune SSHFS to reduce the effect of encryption on performance. Furthermore, you can enable compression to improve performance as well. Using these tuning options, you can recover SSHFS performance so that it matches and even exceeds NFS performance.