More Ham and Less Spam

With reports of spam reaching pandemic levels and ham (solicited email as opposed to unsolicited email) conservatively only being around 5 to 10 percent of the email on the Internet as a whole (depending on which study you believe), then it’s hardly surprising that vendors continue to develop new defences against one of the Internet’s most notorious pests.

Spammers only continue to send unsolicited email because there’s something in it for them – namely, profit gleaned from recipients unwittingly clicking on links in email messages to, and then buying, pharmaceutical products and the like. It’s been said that if no one clicked, hardly any spam would be left.

Silver Bullet

For those of you who have dealt with the Internet for a while, prepare yourselves for a sentence that might be hard to come to terms with: I can instantly and reliably remove 95% of your spam. An incredulous response would be a perfectly natural and warranted reaction. I’ll even go so far as to say that after applying that magic bullet, simply by adding a further five lines or so to your mail server configuration can increase that percentage to about 97% or 98%. Hard to believe I realise, but read on.

Aim for the Heart

The sliver bullet to which I refer is known as greylisting . Its simplicity makes it eye-wateringly effective.

How does it work you ask? Well, the receiving SMTP server simply responds to first-time email senders with a temporary SMTP error code. Even though it’s really old, SMTP is actually pretty clever. These error codes might say something like “I’m really sorry but I’m out of disk space, try back in a little while” or “I’ve got internal configuration issues and there’s no way I can deliver your mail at the moment so please come back later.”

Greylisting works on the premise that spammers will try to send as many thousands of email as quickly as possible. This is because spammers are usually using the resources of compromised mail servers, which might be patched and defended at any time, or they are using IP addresses that might be banned by antispam authorities within an unknown time frame.

As a result, these nefarious types set their mail software to fire off email as quickly as possible before they lose the ability to send any more. Clearly, a sensible sys admin would try to use that fact to their advantage to help defend against the deluge of spam online these days.

Horses for Courses

A few implementations of greylisting are out there, and they’re usually extremely fast to deploy. Jon Atkins has a clever Qmail version, for example. In this article, however, I’m going to look closely at Postgrey , which integrates seamlessly with Ubuntu’s default mail server, Postfix, and is fully compatible with Debian.

Garlic and a Cross

Spammers flood our poor mail servers every second of every day with email as fast as their software will allow, but how does greylisting actually work? When a new connection is made to a mail server from a remote delivery agent (think along the lines of an SMTP server, mail client, or bulk email software), greylisting does a quick local lookup (from a directory full of symlinks, flat files, or sometimes a database – but I’ll just refer to it as a database) to see whether the IP address of that delivery agent has ever connected in the past.

It checks to see whether the mail server admin has whitelisted that IP address (or the sender’s email domain name), as well as the last time that IP address connected.

If that IP address isn’t listed as having connected recently, then it is added to the greylisting database with a time stamp. If the IP address then tries to reconnect within a specified interval (such as five minutes), it’s considered to be a spammer and not allowed to connect again. It’s beautifully simple and perfectly effective. The idea is that legitimate mail servers will retry after a sensible period of time, but not bulk mailers who are desperate to spew out as much email as they can in their limited operational time frame.

Deviating off course on a tangent for a second, consider Table 1, a wonderful, slightly confusing, and some might say counterintuitive accounting of Qmail’s delivery retry time intervals according to a respected website.

Table 1: Qmail Delivery Retry Events

As you can see, it’s important to adjust your greylisting retry allowances depending on which mail server you use; otherwise, you could be waiting quite a while for delivery from some mail servers!

Installation

On Ubuntu and Debian servers, once you have a functioning Postfix installation, the addition of a popular implementation of greylisting, called Postgrey, is no great shakes. The Postgrey site has a great graph showing the effectiveness of greylisting (Figure 1).

Figure 1: What happens to spam when you enable mysteriously magical greylisting (from http://postgrey.schweikert.ch/)

To install the very lightweight Postgrey software, which runs in addition to Postfix, enter:

# apt-get install postgrey

Integrating with Postfix

As promised, the integration of Postgrey with Postfix is a breeze – as simple as locating the smtpd_recipient_restrictions line inside /etc/postfix/ main.cf and adding an entry to the end of that list of commands, as shown here. Clearly your settings could look different; just append the last line to your configuration:

smtpd_recipient_restrictions =
            reject_non_fqdn_sender,
            reject_non_fqdn_recipient,
            reject_rbl_client bl.spamcop.net,
            reject_rbl_client dnsbl.sorbs.net,
            check_policy_service inet:127.0.0.1:60000

Essentially, you’re asking Postfix to check with whatever is running on port 60000 before it completes an SMTP transaction. You can alter that port number if you like, but because it’s only listening on localhost , you probably don’t need to firewall access off to that port. Of course, that’s entirely at your discretion.

Now, all you need do is restart Postfix. In this case, a reload would probably suffice, but go ahead and do a full-fledged restart to be sure it refreshes all of the main.cf   configuration:

# /etc/init.d/postfix restart

Postgrey can now talk to Postfix in email dialogs with other servers.

Setting Up Postgrey

To get a full functioning setup, you really only have three configuration files of concern.

The first and lesser used configuration file is /etc/postgrey/whitelist_recipients , which offers a quick and easy way to circumvent Postgrey completely by allowing some addresses to bypass all greylisting mechanisms. This might be handy for a secret address that receives email from multiple, diverse senders or a public address that must always instantly receive email, such as postmaster@ or abuse@ ,

The second file, which which you will probably spend more time working, is named /etc/postgrey/whitelist_clients . This powerful file lets you control the whitelisting of all sorts of different aspects of incoming mail. For example, you can filter individual email addresses and entire domain names or IP addresses or regular expressions (regex). Apparently, you can even whitelist prefixes from sender email addresses, such as chrisbinnie@, but this is quite targeted and assumes you have a sender with the same email address at multiple domain names.

The whitelist client file might look like this:

Domain Names:
debian.org
gmail.com

Regular Expressions (all outbound Gmail boxes):
/^.*-out-.*\.google\.com$/

CIDR IP Addresses:
112.113.0.0/24
112.113.0.10

Many examples are included post-installation.

The last configuration file on Ubuntu and Debian systems lives at /etc/default/postgrey . I’ll explore a couple of changeable parameters in this file, but more can be altered.

The POSTGREY_OPTS line

POSTGREY_OPTS="--inet=60000 --delay=199 --auto-whitelist-clients=5 --greylist-text"

sets the port on which to run. The next parameter is worth testing with different values until you find the number that suits your needs. The allowed delay threshold is the time a returning mail server needs to exceed to be able to connect again. In this case, any remote delivery agent that connects sooner than 199 seconds after the last attempt will be told to come back later or dropped if it has returned too many times previously. The default is five minutes, usually, but to cut the delay down for first-time emailers, dropping it as low as 199 seconds isn’t too troublesome.

The --auto-whitelist-clients switch is a bit tricky to get right, and you might want to disable it entirely. It kicks in when a remote delivery agent has sent a number of email messages and is automatically whitelisted after a certain threshhold. This can defeat Postgrey’s defences easily, so use it with caution. Again, the volume of inbound mail and the type of spam you receive will determine how you want to use this, if at all.

One final parameter to set is

POSTGREY_TEXT="Sorry, your mail server has been greylisted for five minutes. Please come back later."

which is simply the configurable message that remote delivery agents receive during the temporary delay they are subject to if they’re new senders.

Come In, Tokyo

Now that you’ve made your configuration changes, you need to refresh the daemon so it picks them up:

# /etc/init.d/postgrey restart

To make sure it’s up and running, you can use your friend “list open files,” lsof , to make sure it’s listening on port 60000 correctly. If you don’t have lsof installed, it’s a simple as apt-get install lsof .

# lsof -i :60000

COMMAND   PID     USER   FD  TYPE DEVICE SIZE/OFF NODE NAME

postgrey 1081 postgrey   5u  IPv4   4663      0t0  TCP Riley:60000 (LISTEN)

That’s it. You can see an entry listed for TCP port 60000, so you’re in luck. After some tweaking and trial and error to get your settings as you need them, you should be up and running.

RBLs

If you want to top up the percentage of blocked spam from around 95% to as high as 98%, you should take a brief look at Real-time Blackhole Lists (RBLs) in Postfix. RBLs are third-party lists maintained by individuals and businesses and contributed to by users who report unwanted email. Your mail server simply looks up an RBL or two – in this case, at the start of the conversation to keep server load to a minimum – and drops the connection if someone has reported that mail server for spamming. RBLs are real-time, reliable, and free to use in most cases.

The two reject_rbl_client entries inside the Postfix file main.cf   allow the following servers to be queried in real time at the start of the SMTP dialog.

smtpd_recipient_restrictions =
            reject_non_fqdn_sender,
            reject_non_fqdn_recipient,
            reject_rbl_client bl.spamcop.net,
            reject_rbl_client dnsbl.sorbs.net,
            check_policy_service inet:127.0.0.1:60000

More Ham, Less Spam

Now you should be receiving far less spam and feel fully empowered to use your email again with confidence. Email is still a fantastic method of communication, and it’s a relief when you can use it without hindrance. The most attractive aspect of greylisting has to be the negligible level of false positives it produces. Every now and again when a remote mail server is told to go away for a while and retry later, it might go away for a few hours instead of a few minutes; otherwise, it’s a truly reliable way of perturbing spam without load-hungry software clogging up your mail server as it does so. As with all great pieces of technology, once you’ve used it for a while you’ll undoubtedly wonder how you used to manage without it.