S3 Load Balancing

The road to a finished Ceph cluster with the S3 front end is actually not as rocky as it looks at first glance. Inktank [4] has a powerful tool up its sleeve in the form of ceph-deploy, which Ceph provides on systems with common distributions in just a few minutes. Anyone who first wants to test on three virtual machines can, of course, do just that.

Ceph is now integrated in common automation tools like Puppet or Chef. The Ceph cluster itself is therefore quick. Just a little more overhead and the Ceph Object Gateway is connected: Inktank itself provides detailed documentation that explains the entire process [6]. In the end, you have an instance of the Ceph Object Gateway that speaks S3 and can be used as a starting point for a Dovecot instance.

You should, however, bear in mind that the S3 gateway can break quite a sweat with large setups. No wonder: If there is only a single gateway instance, all connections from the Dovecot front ends to Ceph have to squeeze through the eye of a needle. This is not just a problem from the perspective of performance, but also a classic single point of failure (Figure 6).

Figure 6: A simple Ceph gateway setup: The single gateway instance, however, is a single point of failure.

However, because S3 is based on HTTP and there are enough high-availability approaches for HTTP, the solution for the problem is obvious: It makes sense to start with two object gateways, for which a (highly available) load balancer (LB) redirects incoming connections accordingly (Figure 7). This architecture is also scalable; additional object gateways can be integrated at any time, and the process can be done with no downtime.

Figure 7: That's better: The setup can be scaled seamlessly in width at all levels.

Conclusions

With the S3 plugin for Dovecot, Sirainen has expanded the scalability of mail at the storage level. Thus, a useful alternative is now available to previous solutions, such as sharding, whose inflexibility often causes problems in today's agile IT structures. The license fees the author charges for the plugin are undoubtedly a drawback. With a sensible setup, however, the costs can be recouped quickly because they remove the need for an expensive SAN on the storage side; instead, Ceph runs on off-the-shelf hardware. In terms of performance, Ceph – except the single-serial-write scenario – usually proves to be the superior solution and certainly has no reason to fear direct comparison with commercial storage solutions.

Those who operate an email setup and regularly face challenges when creating more space for user data should certainly take a closer look at the combination of Dovecot, the S3 plugin, and Ceph with Object Gateway. Instructions [7] explain how to set up Ceph with Object Gateway so that it cannot be distinguished from "real S3" clients.