Object Stores to the Rescue

The situation changed dramatically, however, with the advent of object stores. These solutions were given their name because they handle all data in the same way internally – as binary objects. The trick here is that binary objects can be split up and put back together as desired, as long as this happens in the same order.

This trick lets object stores offer real horizontal scaling, because the object store itself "only" has to make sure that the binary objects are split correctly and distributed neatly across the existing hard drives. If more hard drives are added to the installation, the object store automatically uses them and thus expands scalability limits to theoretical dimensions.

The existing cloud computing solutions have brought a whole wave of different stores into the limelight. Red Hat acquired Ceph [1]-[3] and introduced its own Storage Server [4] as a solution for storing objects. OpenStack entered the game with Swift, which is also an object store in the classical sense. Moreover, you have those who provide object stores as a service for users, such as Amazon S3 or Dropbox.

Setting up scalable storage systems with all the services is certainly possible, and it would be great for email platform administrators if such a storage solution could be harmonized with the email architecture referred to previously. After all, there is no obstacle to treating an email message like a binary object. That is what Timo Sirainen, the author of the Dovecot secure IMAP mail server [5], probably thought and drew his own conclusions: The enterprise version of Dovecot offers an Amazon S3 plugin that perfectly exploits the benefits of the object store.

Dovecot with S3

Sirainen has offered the Dovecot S3 plugin for some time. Importantly, the plugin only runs with the enterprise version of the Dovecot mail server (Figure 1). The license for the enterprise-grade Dovecot Pro edition costs around EUR5,000 per year for 10,000 mailboxes. Admittedly, that is not exactly cheap, but the number has little meaning.

Figure 1: The license for the S3 plugin with Dovecot is not inexpensive; however, the costs can usually be offset by cheap hardware.

Dovecot Pro might cost more than the free version, but using it with an object store for backup will, in many cases, mean significant hardware discounts in the enterprise, because it removes the need for SAN storage, so off-the-shelf hardware will do. Companies should certainly take such considerations into account if they are considering using Dovecot with the S3 plugin.

How, specifically, does the S3 plugin work for Dovecot? Sirainen explains this in detail in the documentation for the plugin. Generally, anyone who wants to use the Dovecot S3 back end needs access to an object store as per the Amazon S3 standard. Login credentials in the form of two values are usually attached to such accounts: The access key acts as a kind of username, and the secret key is the password. Anyone who creates an account with Amazon receives both pieces of information automatically.

To storing email with Dovecot, you also need to create your own bucket in S3. At first, you might be a little uneasy and feel insecure because not all users get their own buckets – but this is an illusion. Not all users have their own filesystem with a regular mail server, after all; the responsibility for enforcing access rights lies with Dovecot as the mail server in both cases.

Dovecot Configuration

The next step involves the configuration of Dovecot itself: Anyone who already uses the program for IMAP or IMAPS will be familiar with the paragraph-like structure of the configuration files. Registering an additional paragraph that takes over the plugin configuration is all it takes for the Amazon S3 plugin. The example in Listing 1 is taken directly from the Dovecot documentation (Figure 2).

plugin {
# Use 100 GB cache for mails in /var/lib/dovecot/cache. The cache directory is \
  the same for all users.
obox_fs = fscache 100G:/var/lib/dovecot/cache:s3:https://Accesskey:Secret@\
          Bucket-Name.s3.amazonaws.com/
}

Figure 2: The Dovecot website provides a PDF file that describes how to set up the enterprise repository and the S3 plugin.

In this case, Dovecot uses a local cache with 100GB of storage space to process local access to frequently used objects as quickly as possible. Clearly, it is not very difficult to dock Dovecot onto S3, as long as the required Dovecot license is available for the plugin.

Those who would prefer to store their data in Microsoft's Azure cloud can do just that – a plugin for Azure is available too. Dropbox support is available on top of that; thus, Dropbox can also be used as back-end storage for Dovecot.

This very fact, however, leads to a discussion that is much more legal than technical: Do companies actually want to use Dovecot to store their email on Amazon, Microsoft, or Dropbox? Skepticism is perfectly understandable in light of the Snowden revelations.