Open Relays and Mail Sinks

Email is a “store and forward” protocol. The sender doesn’t connect directly to the recipient to send the mail with just one network hop, rather the sender connects to a mailserver (usually referred to as an “MTA”, short for Mail Transfer Agent) and sends the message there. Once that MTA has received the message it sends it on to another MTA, and so on until it reaches the recipient.
Mail clients typically don’t have any intelligence built in to them to decide which MTA to send an email to. Instead they’re configured to blindly send every message to one particular local MTA, the smarthost, which then does all the proper SMTP work to decide where to send it on to.

Many years ago it was considered polite and neighborly to offer your smarthost to people nearby, so they could send mail. The very early generations of spammers abused that resource by picking some legitimate smarthost and injecting their spam into it, from where it would be delivered to the spammers targets.

This is what became known as an “open relay” – a mailserver that would accept a mail from anyone and forward it on to wherever the sender wanted it to go. Mail operators started to lock down access to their smarthosts to only legitimate users via various sorts of authentication – IP based, so that only customers on their network could use it, or password based, so that only users with a valid username and password could use it, and many variations on those.

Many early blacklists targeted primarily open relays, as they were the second major wave of spam sources. Today, true open relays are pretty rare, as most mailservers are secure “out of the box” and will only end up as an open relay if they’re misconfigured (or misguidedly configured intentionally) to be one. In the mid-90s, though, there was a lot of checking mailservers to see if they could be convinced to behave as an open relay (often by sending intentionally malformed SMTP traffic, attempting to exploit bugs or common misconfiguration problems in late 80s and early 90s mailservers).

A very simple test to see if a server is an open relay is to send email to it for relaying, from an IP address that shouldn’t have access, and see if it rejects the connection or whether it accepts it for delivery. That will sometimes work, but there are many cases where the mailserver will accept a mail “for relay”, but then won’t do anything with it. That’s not uncommon in a wide range of special-purpose mailservers – bounce handlers, feedback list handlers, ticketing systems, spamtraps and many more. They’re “mail sinks” because they’ll accept mail from almost anyone to anyone, but won’t then send it on to the intended recipient. What they will do with it depends on what the mail sink is intended to do. Many will just silently discard unexpected mail – feedback loop handlers and bounce handlers, for example, typically look for specific strings in the mail, or the recipent email address, and will ignore anything that doesn’t match. Others will do something “wrong” with the mail, such as turn it into an entry in a ticketing system, which may be annoying to the owner of the mail sink, but doesn’t allow it to be abused to send spam.

A relay tester that uses this approach will wrongly decide that the mail sink is an open relay. The only valid way to test for an open relay is to actually show that it will relay mail for anyone, by sending a message to it targeting an email address you control, and see that message actually delivered to that email address.

A well behaved relay tester will send messages to both the mail sink and the open relay, but it will then wait to see if it gets mail back from them. The open relay forwards the mail on to the recipient, so is an open relay. As the mail sink never forwards the mail on, the relay tester doesn’t consider it an open relay.
Are open relays an actual concern? No. They haven’t been for many years. There are a tiny scattering of them out there, but they’re so few (and will get blocked so quickly by blacklists if they’re abused) that it’s not worth anyones time to abuse them.
So why am I talking about them? With the rise of email, there are many companies that will offer you ways to improve your mail setup, audit your infrastructure, probe your systems for security holes and so on. Checking for an open relay is an easy additional “check box” test to add to an audit suite – it’s pretty much useless, but a longer list of tests adds the appearance of value to paying customers. If those tests are implemented by people who have never been exposed to actual 1990s era open relay problems, then they will implement them badly. If they don’t do a thorough test (which, if I recall correctly, requires attempting to relay 21 different messages) and mistakenly give an actual open relay a clean bill of health then – other than you not getting what you paid for – it’s not really a problem.
If, on the other hand, their audit claims that your mail server is an open relay then that can be alarming. The first thing you should do is ask the auditor to show you the mail being sent from the relay tester, to your mail server, and back to an email address of the auditors choice. If they can show you that, then you really do have an old-school open relay – which isn’t a big concern, but you should probably fix it. If they can’t show you that, then you don’t have an open relay – and the only thing you need to be concerned about is whether the rest of the auditors service is as inaccurate as their open relay test.