Mail server administrators have a hard life. On one hand, few other services produce as many complaints about improper use as mail operations – incorrect blacklisting is an example. On the other hand, people are beginning to realize that email is a product from another age that is now struggling to deal with today's fast-paced IT. Users' demands for receiving and sending email have grown steadily in recent years.

Several years ago, it was quite common to allow email attachments only up to a particular, very low size, but this practice is frowned upon today: Users now expect to be able to send their 20MB PowerPoint presentations by email. Moreover, the number of email users is still growing steadily, and the amount of data stored by all of a provider's users is growing proportionally.

This problem might not be specific to email, because other Internet services are also experiencing the effect of the steady growth of the data volumes that need to be handled. However, working around this problem for email setups is relatively complex, primarily because of the architecture inherent in the "email" principle.

Architecture

Normal setups usually follow the same old pattern: On one side, SMTP daemons ensure that customers can send email and that incoming messages reach the recipients. On the other side, the services that deliver email to customers carry out the majority of the work. Some providers still exclusively rely on POP3 or only offer IMAP against payment; however, on the whole, the IMAP protocol has asserted itself.

This situation means additional worries for email service providers: Customers quite rightly expect to find their email on the provider's servers at any time. These customers rarely even download local copies of their email and simply rely on their providers. The providers then have to figure out how to cope with the burden and memory requirements of such a setup. Even if each user only has a small mailbox, the need for storage is already substantial if you have a large numbers of users.

Sooner or later, you will need to adjust your infrastructure to accommodate the increased number of users, which means growing your infrastructure. From experience, load issues with services can be dealt with relatively simply by using multiple mail exchanger (MX) entries (on the SMTP side) and load balancers for IMAP or POP3 daemons. The storage problem is trickier.

The Storage Problem

When email administrators wanted to add storage space to their servers in the past, they mainly used central storage systems. SAN models from the major manufacturers are the classic candidates. However, those things are expensive and scale vertically at best. Remember that vertical scalability means repowering the existing hardware, which is the opposite of horizontal scalability, for which new machines are simply added to existing machines. The load is then distributed across the devices that are part of the installation.

Sharding

Traditional storage systems can be expanded by installing new hard drives, but the end of the road is always reached eventually, and additional storage space can only be managed by purchasing multiple storage systems. The process of storing data records across multiple machines (known as sharding) has been the standard approach for many years, with users being assigned a variety of storage systems.

This approach brings a raft of disadvantages: First, SAN storage is significantly more expensive than regular, off-the-shelf hardware; second, multiple storage islands make centralized administration more difficult. Admins previously just had to grin and bear this because of the lack of alternatives; the storage market seems to have been in a deep sleep when it came to horizontal scaling.