Hardening mail servers, clients, and connections

Special Delivery

DMARC, DANE, and More

The third technology in the group is somewhat more controversial: The Domain-based Message Authentication, Reporting and Conformance (DMARC) protocol, which also involves storing a TXT entry on your domain's DNS server. However, the approach is far more complex than that of SPF and DKIM; in fact, DMARC even incorporates both technologies. In addition to the matching modes for DKIM and SPF, a DMARC entry for a domain contains specific, standardized instructions (e.g., regarding email forwarding). If these rules are violated, a DMARC-enabled mail server drops the message.

This outcome occasionally leads to problems in everyday life, because DMARC can cause legitimate mail to fail. Mailing lists, for example, forward email from the sender to many recipients. Anyone who has ever received email from a mailing list in which the sender is the mailing list itself, with User via … in the From line instead of the original author now knows the reason. These constructs are needed for DMARC to work. Because of the catastrophic flood of spam, however, DMARC has now also become widely established, and new mail servers should use it if at all possible.

Finally, DANE stands slightly apart from the other three solutions and has nothing to do with authenticating the sender; rather, it enforces the use of an SSL-encrypted connection from the sender's point of view. To this end, it makes use of DNSSEC, the standard intended to secure the authenticity of DNS entries through digital signatures, and attempts to solve a problem similar to that DKIM tackles. DNS is an ancient protocol; its original form had virtually no protection against misuse.

DNSSEC corrects this problem by using DANE to ensure that a remote mail server can only be considered for delivery of mail to a specific domain if its signed DNS entry explicitly allows an SSL-encrypted connection to be opened by STARTTLS. If the domain does not support it, the outgoing mail server immediately cancels sending. This arrangement prevents man-in-the-middle attacks and can even be a building block for making your setup more compliant with the European Union's General Data Protection Regulation (GDPR). It should therefore be a standard tool on new mail servers. In return, you have to accept the fact that mail might not be delivered because of a lack of DANE support on the receiving end.

Finally, TLS reporting (TLS-RPT), which acts similarly to DANE, is now considered an accepted standard and relies on state-of-the-art technology. Put simply, with TLS-RPT in place, an outgoing mail server checks whether the message transfer agent (MTA) on the other side supports the STARTTLS command. If so, it uses TLS to encrypt the message and sends it to the other end, where it is unpacked and delivered. By doing so, TLS-RPT enforces encryption of the content to be transmitted. In particular, it rejects clients frequently found on the network that try a hack to downgrade communication from the encrypted variant to plain text while establishing a connection. This hack is still regularly used today, particularly in cases of attempted fraud.

Reporting also plays an important role in TLS-RPT. When TLS-RPT is active, the server admin receives regular reports of incidents (Figure 4), such as when a client has attempted to intervene, as described above. This standard helps detect malfunctions and missed email at an early stage and, where needed, helps the admin know when to change the configuration.

Figure 4: TLS-RPT generates reports on failed email deliveries, ensuring that admins are immediately notified of potential configuration issues.

Protecting the Receiver

Securing and verifying receiving and sending addresses does not change the fact that billions of malicious email messages find their way into the inboxes of potential victims every day. At the end of the day, doing something about this situation is one of the mail server administrator's obligations. Conveniently, you can draw on tried and tested solutions that have proven their value over many years.

Most admins have probably heard of SpamAssassin, a tool typically used in conjunction with ClamAV, a virus scanner. Just one more component is missing for happy mail delivery: the Amavis daemon, Amavisd, which acts as a bridge between Postfix, SpamAssassin, and other components that play a role in the mail delivery process.

The whole setup is basically simple: Postfix receives an incoming email and forwards it to Amavisd, which then calls SpamAssassin, ClamAV, and other components and lets them check the entire content of the email. If one of the called services finds malware, or SpamAssassin identifies unwanted content, it marks the mail header accordingly before Amavisd returns the message to Postfix along with all the new headers. The final decision on what happens to the mail then rests on Postfix and its configuration.

SpamAssassin grades email according to various factors that confirm or allay the suspicion of spam. As the admin, you determine the limits by which a local setup classifies email as definitely spam, suspected spam, or probably okay, but it is important to train a setup of this type yourself and ideally also have it trained by your users. Solutions such as SpamAssassin are based on Bayesian filters that cannot work properly without appropriate training. In an environment like this, it makes sense – from the user's point of view – to move email that has been incorrectly identified as spam to the correct inbox and to configure the filter so that it uses the contents of the spam folder for its own regular training (Figure 5).

Figure 5: Modern antispam tools such as Rspamd only work efficiently and well if you train them properly to help them evaluate messages with realistic assumptions.

An alternative solution based on Rspamd (a compact replacement for Amavisd), SpamAssassin, ClamAV, and other components is far more modern. Rspamd is installed as a mail filter in Postfix and has a strictly modular structure, with modules adding features such as an antivirus option and a filter for potential spam messages. In principle, Rspamd works similarly to SpamAssassin. Unlike Amavisd, it also has a local delivery agent (LDA) mode. It does not dock onto the Postfix mail server, but to the delivery service – usually Dovecot as the IMAP server. The disadvantage of this solution is that email first checked in the message delivery agent is already considered accepted and can no longer be rejected absolutely. Where this is not a problem, Rspamd offers a modern and powerful alternative to SpamAssassin.

If you want to containerize your mail server as described earlier, you need to make sure it also runs the additional services in the container and establishes a communication path to them. Again, some established automation providers offer ready-made solutions for this scenario.

Conclusions

If you want to make sure your mail server works well and is secure, you can look forward to a fair amount of work. The patchwork of extensions to the original email standard contributes significantly to the confusion and makes operation more complex, but there's no point in complaining about the inevitable. If you have to run a mail server, you have no choice but to jump through several burning hoops. Thanks to the wide choice of ready-made automation solutions and modern technologies such as containers [1], you can now get started far faster than ever before.

Infos