Magical Mail

A good pedigree is an important aspect of a software package for several reasons – especially if the software is installed on a valuable server. Often, it will help ensure that operational facets of the package have been carefully considered, such as its integrity, reliability, compatibility, and security.

I mention a package's pedigree because I recently stumbled across a tiny piece of software that I used extensively some years ago, and it reminded me how steeped in history that package was. Despite its diminutive size and unquestionable sophistication, it is shrouded with confusion harking back to its provenance.

Along with other key services associated with the Internet of the past, historians looking back will always include email as a cornerstone. Without email, the Internet might not have gained as much traction and might have failed to conjure the staggering uptake it has achieved.

With that in mind, you might consider the inbound and outbound aspects of email at a more technical level, and how the Post Office Protocol (POP) in its various versions and incarnations was all pervasive until a few more features were required, such as encrypted login details. This was just one of a variety of reasons that the Internet Message Access Protocol (IMAP) gained popularity. Also, I would be remiss not to include the Simple Mail Transfer Protocol (SMTP), which surfaced as far back as 1982 and still remains the champion for outbound email on today's very different Internet.

As a sys admin, providing services so that end users might enjoy picking up and sending email messages is a relatively well-documented procedure. Popular mail servers and mail clients slot together nicely, and the docs read clearly and concisely.

Consider, however, when you need do something slightly off-piste and a mail client won't just plug straight into an out-of-the-box mail server's services to solve your problem. Sys admins since time immemorial have hand-crafted scripted solutions for abstract issues and then emailed the results to a number of people. On Unix-like systems, the magical utility that does one thing and does it well (with a doff of the cap to the Unix philosophy), in terms of communicating efficiently with a preconfigured SMTP server, is the mail utility.

It's important to note that throughout the history of the mail utility confusion has reigned. That's the understatement of the day. Thus, when I reference the "mail utility," I am more than likely referring to the modern mailx or similar utilities, because few systems use the older, deprecated versions.

In this article, I'll look at how the magical mail utility has evolved, somewhat dramatically, over the years and explore some ways in which you can integrate it with simple, day-to-day, sys admin command-line tasks along with adding it to your scripts.

Enough with the Scenarios Already

I'll begin with some simple examples and scenarios. Imagine you have just spent a morning working with screeds of headache-inducing, fiddly text files containing comma-separated this and tab-delimited that. Finally, you've reached a point where you think your boss will be suitably happy with the result, and you breath a sigh of relief. Then, it occurs to you that you've been working on a remote server and the resulting text file is about 8,000 lines long, several columns wide, and you don't have an easy way to retrieve it from the remote server. Copying and pasting just won't cut it.

The epiphany dawns when you realize that you can email the file to yourself directly from the server. Step forward the mail utility.

In its simplest form, the mail utility lives in /usr/bin/mail, and if you want to send an empty email with just a subject line, you could populate the body of the email with a reference to /dev/null. The -s switch allows a subject line to be specified (although even such a simple feature isn't available on the oldest versions).

# mail -s "Subject Matter" chris@binnie.tld < /dev/null

The outcome of this command is an email with an empty body, but otherwise a perfectly formed email is sent (as long as you've configured an SMTP mail host to point your server at, which I will assume is done for now and explore further a little later).

As much as I enjoy sending myself and others brief email messages with short subject lines and empty bodies, at some point, I'll want to add some text to the email body, so first I'll look at how to enter text into the body field by omitting the /dev/null reference.

You can enter text arbitrarily (even by copying and pasting) and then tell the mail utility that you've finished the body of the email by entering a dot on a new line, which is translated as EOT, meaning "end of transmission" or "end of text." This triggers the sending, which means the mail is shipped off right away. You can see in Figure 1 how this might look on a command line.

Figure 1: The figure demonstrates possibly the most user-friendly way of sending an email ever invented.

More pragmatic readers might be thinking about how to send a file that is not typed into the email directly. You would be forgiven for desiring such functionality. For simple text files, you would "insert" text (note that it's not an "attachment") from a previously created file into the body of your email:

# mail -s "The one and only Billy Shears" chris@binnie.tld \
  < file_full_of_lyrics

Another simple option for a super-short message might be achieved with this method, too:

# echo "The act you've known for all these years" | mail -s "Cellophane \
  flowers of yellow and green" chris@binnie.tld

Now, back to the frustrating scenario that is threatening your employment status. The problem often encountered with the less modern mail utility is that of "attaching" files to an email and not just "inserting" them (although the methodology I demonstrate is not really an "attachment" as it is perceived today). Thankfully, there's a simple solution that I will come to in a moment.

For now, please take it on faith that to use the mail utility in any sensible way you are probably going to need to install its feature-filled successor heirloom mailx or at least mailx . The former is a fully featured MIME-enabled mail client (MIME stands for Multipurpose Internet Mail Extensions). Simply put, heirloom mailx will happily attach files for you without any rigmarole.

If you're curious, the package actually used to be called nail , but I'll explore its history later. Assuming you're looking at a modern Linux operating system, the complexities of which command to use and which features are available to you (depending on which package is installed: mail , mailx , nail , or heirloom mailx ) can be put to one side.

Debian/Ubuntu users can install it as follows:

# apt-get install heirloom-mailx

The package description aptly describes the package as "feature-rich BSD mail" – which sums up its raison d'être nicely. Far from just being able to send email from the command line, heirloom mailx can spin plates and dance a jig, too. Next, I'll peer into its provenance in more detail.

Wake Up, You at the Back

All the way back to well before Sinclair C5s were the rage – 1971 or so – an early edition of Unix incorporated a command called mail. So nascent was this embryonic operating system that it couldn't speak to other machines; thus, mail was more like the write or talk commands in the sense that you could send lines of text to other users on the system that you were logged in to.

The talk command (for those unfamiliar it) could be used to speak to users on your machine or remotely on other machines. I certainly don't want to get into the intricacies of a chicken-and-egg debate, but I think it's of interest to show how similar the two commands are for comparison.

For example, to speak to someone on the same machine, you could use:

# talk pepper

If networking of some sort were present, you could use this command:

# talk rita@some_other_machine

Rather than examining the mail utility's rich history in detail, I will skip rapidly onward. In brief, the mail utility grew a number of arms and legs and eventually included an interactive shell of sorts that could speak to local mailboxes. During this evolution, the mail utility also learned to speak to varying SMTP servers, such as the most popular, and undoubtedly archetypal, Sendmail.

Having mercilessly skipped the tool's rich tapestry, I feel somewhat vindicated by the example below. As mentioned, it's an understatement to say that the mail utility's background is an age-old, bubbling cauldron of confusion. Here's that example, shown in Figure 2.

Figure 2: You can see the lengths to which modern operating systems must go to support the legacy mail command, shown here with a symlink indirectly to the heirloom-mailx command.

Anyone, Anyone

Now, I'll return to that scenario in which you need to attach some relatively large files to an email directly from your server, and, for whatever reason, a modern version of the mail utility is not readily available.

Can you guess what obstacles might need to be surmounted before you can attach certain files properly? The example scenario involved manipulating large amounts of text, but what about attaching an image file like a JPG or a smallish binary, such as the /usr/bin/mail command, which is about 350KB in size?

Before mail clients had all the requisite capabilities necessary to send files across the Internet, the files had to be encoded (wrapped or encapsulated) in a way that could then be easily unwrapped at the other end. In its simplest form, the excellent uuencode command can do exactly that. So that you can see it in action, I will show you a tiny script in a second.

First, here's a reminder how you would send plain text in the body of an email:

# mail -s "Filling in a ticket in her little white book" chris@binnie.tld \
  < text_in_the_body

If you did that with an image file, you would effectively insert it into the body making extra copying and pasting work for the recipient. The email would just appear nonsensical to a mail client upon receipt if it was sent like this:

# mail -s "Sitting on the sofa with a sister or two" chris@binnie.tld \
  < bad_idea.jpg

Incidentally, you might need to install uuencode (around 142KB) as follows with the sharutils package:

# apt-get install sharutils

To use uuencode alongside the outbound email functionality, you can use the mail utility like this:

(cat EdgarAllenPoe.txt; uuencode ringo.jpg first.jpg; uuencode \
  george.jpg second.jpg) | mail-s "Expert textpert choking smokers" \
  chris@binnie.tld

Take a look at this command more closely. Before the pipe, I've enclosed the commands in brackets to be fastidiously neat (and they're needed, actually). You're likely familiar with the layout of the commands after the pipe, so take a look inside the brackets. The text file is first presented to the mail utility. That cleverly gets picked up as the body of the email.

The uuencode options appear to be written incorrectly at first glance, which is why I'm drawing your attention to them. The ringo.jpg entry is the local file name of the attachment that you're picking up, and the first.jpg entry is the file name that will be presented to the email recipient. This action is rinsed and repeated for george.jpg and second.jpg and then piped into the mail command with a -s for the subject, followed by a recipient's email address.

A few CPU cycles later, you should see the results either in your logfile /var/log/mail.log or your inbox – if you've emailed yourself for the test. If all went well, included are the binary attachments with the correct file names and a well-formed email body.

I promised to show you a very simple script to bundle up lots of attachments for inclusion with an email – a reminder that email was never designed to be used for large attachments, which is a perennial discussion I seem to have with end users. As a guide, one well-known and widely adopted webmail provider currently limits 20MB as the outgoing file size for its SMTP, per email. I tend to be stricter and limit the size to 10MB on a smaller infrastructure.

If you want to test this method, you can run the touch command to create a few empty JPG files, as I have here (you should test with a real JPG image later to confirm it's working):

# touch {1..5}.jpg

I have filled up a text file called file_list with a list of JPGs in my directory as follows:

# ls *.jpg > file_list

I also created a file called bodytxt, which contains the text content of my email, as you might have guessed. Make the script shown in Listing 1 executable with

#!/bin/bash
files="file_list"
( while read uniq; do
    ( uuencode $uniq new.$uniq )
done < $files; cat bodytxt) | mailx -s "Go to a show, you hope she goes" chris@binnie.tld
exit

chmod +x scriptname

and try it yourself with ./scriptname. The new file names arrive as attachments named new.1.jpg, new.2.jpg, and so on.

You could potentially attach a hundred, tiny, uuencoded image files to an email very swiftly with a method like this. As ever with shell scripts, your mileage might vary, but you can easily adjust this method to send an email for each attachment, too.