Locked Away

By law, enterprises are required to retain email for a certain period of time. The archiving solutions discussed here, Piler, Benno MailArchiv, and MailArchiva, promise both legally compliant storage and added benefits for corporations.

Many countries provide a comprehensive system of legal regulations for governing the delivery and storage of email. For instance, some mail might contain commercial correspondence or tax-relevant information that forms an official legal record and must stay around for a predetermined time for auditing purposes.

The auditability requirement often dictates that the archived email be immutable and thus protected against access by staff – even the omnipotent administrator. The laws governing email pose several technical problems: Many messages are stashed away in local mailboxes of mail clients where they are stored in an unstructured way so that pertinent messages cannot easily be distinguished from other mail. Targeted searching thus involves a large amount of manual work. The requirements for digital auditing, and possibly also internal compliance policies, are thus not met. Additionally, the daily flood of spam that swamps mailboxes makes it very difficult to sort the digital chaff from genuine content and hugely bloats the data volumes.

More Than Compliance

State-of-the-art archiving solutions address these tasks and problems with a varying degree of internal overhead. They take care of automated, permanent email storage, make the content easy and quick to find through centralized search solutions, and ensure auditable storage of the data. Ideally, these solutions will integrate seamlessly with the enterprise network, collaborate nicely with all popular mail servers, offer web-based access, provide granular authorization, and be able to store any kind of data transparently on any popular kind of storage medium, or even on specialized archiving systems. Ideally, too, the operator will have a choice of infrastructure between on-premises or cloud storage.

A positive side effect of such a solution is the reduced storage space requirement, because mail can be compressed and deduplicated. Additionally, it will be beneficial for business continuity, because mail remains in the archive even if mail servers fail or mail data is lost. Full-text searching against mail content and in mail attachments also makes it possible to find mail content years after archiving a message. See the "Legal Framework Example" box for more information.

Powerful Open Source Applications

In this article, I examine three standalone mail archiving products from the open source camp: Piler, Benno, and MailArchiva; Table 1 compares the features of these archiving tools.

All of these products have a fundamentally similar approach: Email is either actively transmitted to the archive (using SMTP) or passively polled by the archive system, that is, retrieved from the mail or groupware server – typically using POP3 or IMAP calls to a journaling mailbox. As you can see in Figure 1, the messages are permanently stored either on the archive filesystem or in the database, including attachments.

Figure 1: Mail archiving solutions store all incoming and outgoing mail in a central archive.

All systems can be managed using a web client and support audits. The tools come with rights management features, including optional directory integration. The systems listed here all claim to be audit-proof and legally compliant. Of course, any technical solution must be accompanied by appropriate organizational policies to guarantee it as a compliant and comprehensive solution. Also, different jurisdictions have different rules for mail archiving. You should familiarize yourself with laws for your own country: Don't depend on the software to know your legal requirements.

Piler

Piler [1] is completely open source software from Hungary; its feature scope has grown immensely in the past two years so that it can now be regarded as a complete solution for mail archiving.

Email can be retrieved from SMTP servers by a variety of manufacturers, including Microsoft Exchange, and imported from different formats. The data is encrypted using the Blowfish algorithm and stored on the filesystem as compressed files. The matching metadata is stored in a MySQL database. The duplication rules are applied to both messages and attachments. Searching is handled by a Sphinx search engine.

The software takes legal requirements into consideration for the most part: Auditing options are in place, as is logging throughout. When saved, email is digitally signed to be able to check for manipulation or demonstrate a lack of it. Piler can connect to a large number of mail servers, including Lotus Notes, Zimbra, Google Apps, and Office 365. Authentication can be handled by LDAP or Active Directory or controlled by an IMAP server.