SMTP Level Rejections

While discussing a draft of a Deliverability BCP document the issue came up of what rejections at different phases of the email delivery transaction can mean. That’s quite a big subject, but here’s a quick cheat sheet.
At initial connection
Dropped or failed connection:

  1. your reputation with the receiver is so bad that they don’t want to see any email from you, ever
  2. their mail server is badly overloaded and can’t accept any connections right now
  3. the server you’re connecting to isn’t a mail server – this is common with bad email addresses

Temporary (4xx) failure:

  1. your reputation with the receiver is poor or unknown and they’re waiting for more information before they’ll accept your mail
  2. the mail server is overloaded and can’t accept any new mail right now
  3. your reputation is “grey”, the mail server is overloaded and it’s accepting mail preferentially from other senders right now
  4. there is something technically wrong with your mailserver, such as non-existent reverse DNS, and your mail won’t be accepted until you fix that

Permanent (5xx) failure:

  1. a 554 response can mean “there’s no mailserver here”. If someone has set up a mailserver specifically to reject connections this way, it’s probably been done for a domain that has a lot of bad delivery attempts. That might mean that the recipient email address is one that’s commonly used for fake signups, or it may be a common thinko address (e.g. something.com vs something.net)
  2. they’ve explicitly blocked all mail from your mailserver due to a poor reputation – this is a common way of implementing DNS blacklist based blocking

Any time

  1. when a mailserver is shut down, it may drop existing sessions with a 421 response
  2. if you violate SMTP – send a bad command, bad parameters, lines too long – you might get 5xx responses at any time

After HELO or EHLO

  1. a syntactically bad or obviously forged (e.g. yahoo.com) HELO hostname – unless you’re a spammer or have a badly misconfigured server, this probably shouldn’t happen

After MAIL FROM

  1. mail from a domain (or, in theory, return path) might be blocked here, but it’s more likely to be rejected at RCPT TO time

After RCPT TO
This is a favorite point to apply reputation based filtering, for several reasons. It’s the first point at which you know who the intended recipient is, and that’s a very useful thing to record to help with diagnosing spam filtering false positives. So even the mailserver has decided it’s going to reject a message, based on the IP address it’s being sent from, waiting until the RCPT TO to reject if will give better diagnostics. There may be special recipient addresses that are exempt from global filtering (such as role addresses), so that filtering can’t be applied until this step. Also, some mailing list managers historically don’t handle rejections before RCPT TO gracefully – they’ll keep retrying and won’t unsubscribe the user forever.
If a mailserver waits until after this point it either has to accept the entire email or unceremoniously crash the connection (which violates SMTP and will provoke a redelivery attempt, so is rarely seen).
Temporary (4xx) failure:

  1. the server is out of resources, and just needs you to come back later
  2. the server is low on resources and mail from you isn’t as important as mail from other senders right now,
  3. the overall volume of mail you’re sending has exceeded short term limits the receiver is applying to you (based on reputation) and it’s deferring mail so as to throttle your traffic
  4. there’s a temporary problem (DNS blacklist resolution, perhaps) in looking up your reputation
  5. your reputation is “grey”, the mailserver hasn’t decided whether your mail is likely to be wanted or not, and it’s letting a small fraction of your mail through, to see how recipients respond
  6. your reputation is “grey” or there isn’t enough information about you to make a delivery decision, so your mail is being deferred in the hope that there’ll be a clearer reputation decision when you try to redeliver it
  7. some combination of all the above

Permanent (5xx) failure:

  1. the recipient doesn’t exist – sometimes a bad address is just a bad address
  2. the source (IP address or domain) of the email has a poor reputation and mail is blocked
  3. there’s a per-recipient block on this email

After DATA

  1. the content of the message “looks like spam” – either because it looks similar to recently received unwanted messages or based on heuristics such as those used by spamassassin
  2. URLs in the message have a poor reputation (or resolve to IP addresses with poor reputation)
  3. DKIM signatures will be checked at this point, so if the mail would have been delivered due to DKIM-based whitelisting and the DKIM signature doesn’t validate that may cause rejection here
  4. DMARC failures
  5. virus or malicious content filters
  6. any of the reasons given for RCPT TO – some filters may defer any rejection until after data, either so as to be able to analyze the content of rejected messages, or because they use a simple architecture that does all it’s filtering at once,
  7. score-based filtering – based on combining everything the mailserver knows about this message, including the content

There’s probably something I’ve missed – call it out in the comments – but these are some of the things I think about when working out why a message has been rejected with an unhelpful error message.

Related Posts

Does email have a guarantee of delivery?

A client asked me earlier this week what SLAs ISPs provided for email delivery. The short answer is that there isn’t a SLA and that the only guarantee is that the email will get there when it gets there.
But as I was mentioning this to Steve, he pointed out that there was a recent change in the RFCs for email. In both RFC 821/2 and RFC 2821/2 (the original email related RFCs and the update in the early 2000’s) the RFCs stated that once a receiving MTA accepted an email that that MTA was required to either delivery the mail or generate an asynchronous bounce. While this isn’t a standard SLA, it does mean that a 2xy response after DATA meant the email would either be delivered to the user or be sent back to the sender. Despite the RFC requirements some receivers would still drop mail on the floor for various reasons, sometimes intentionally and sometimes not.
RFC 5321/2, the current SMTP standard, still says that once a server accepts the mail it must not lose that mail ‘for frivolous reasons.’ The RFC goes on to admit, though, that in recent years, SMTP servers are under a range of attacks and dropping mail on the floor is not frivolous in those cases.
 

Read More

ISP Relationships

Delivra has a new whitepaper written by Ken Magill talking about the value (or lack thereof) of relationships with ISPs. In Ken’s understated way, he calls baloney on ESPs that claim they have great delivery because they have good relationships with ISPs.
He’s right.
I get a lot of calls from potential clients and some calls from current clients asking me if I can contact an ISP on their behalf and “tell the ISP we’re really not a spammer”. My normal answer is that I can, but that there isn’t a place in the spam filtering process for “sender has hired Laura and she says they’re not a spammer.” I mean, it would be totally awesome if that was the case. But it’s not. It’s even the case where I’m close friends with folks inside the ISPs.
I’m pretty sure I’ve told the story before about being at a party with one of the Hotmail ISP folks. There was a sender that had hired me to deal with some Hotmail issues and I’d been working with Barry H. (name changed, and he’s not at Hotmail any more) to resolve it. During the course of the party, we started talking shop. Barry told me that he was sure that my client was sending opt-in mail, but that his users were not reacting well for it. He also told me there was no way he could override the filters because there wasn’t really a place for him to interfere in the filtering.
Even when folks inside the ISPs were willing and able to help me, they usually wouldn’t do so just because I asked. They might look at a sender on my request, but they wouldn’t adjust filters unless the sender met their standards.
These days? ISPs are cutting their non-income producing departments to the bone, and “sender services” is high up the list of departments to cut. Most of the folks I know have moved on from the ISP to the ESP side. Ken mentions one ISP rep that is now working for a sender. I actually know of 3, and those are just employees from the top few ISPs who are now at fairly major ESPs. I’m sure there are a lot more than that.
The reality is, you can have the best relationships in the world with ISPs, but that won’t get bad mail into the inbox. Filters don’t work that way anymore. That doesn’t mean relationships are useless, though. Having relationships at ISPs can get information that can shorten the process of fixing the issue. If an ISP says “you are blocked because you’re hitting spam traps” then we do data hygiene. If the ISP says “you’re sending mail linking to a blocked website” then we stop linking to that website.
I have a very minor quibble with one thing Ken said, though. He says “no one has a relationship with Spamhaus volunteer, they’re all anonymous.” This isn’t exactly true. Spamhaus volunteers do reveal themselves. Some of them go around openly at MAAWG with nametags and affiliations. A couple of them are colleagues from my early MAPS days. Other do keep their identities secret, but will reveal them to people they trust to keep those identities secret. Or who they think have already figured it out. There was one drunken evening at MAAWG where the nice gentleman I was joking with leaned over and says “You know I am elided from Spamhaus, right?” Uh. No? I didn’t. I do now!
But even though I have the semi-mythical personal relationship with folks from Spamhaus, it doesn’t mean my clients get preferential treatment. My clients get good advice, because I know what Spamhaus is looking for and can translate their requirements into solid action steps for the client to perform. But I can think of half a dozen ESP delivery folks that have the same sorts of relationships with Spamhaus volunteers.
Overall, relationships are valuable, but they are not sufficient to fix inbox delivery problems.

Read More

Questions on Google lawsuit post

A couple questions in the previous discussion thread about the Google privacy case. Both concern permission granted to Google to scan emails.
Google’s stance about this is fairly simple.
Gmail users give explicit permission for their mail to be scanned.
People who send mail to Gmail users give implicit permission for their mail to be scanned.
The plaintiff’s lawyers are alleging that some subset of gmail users – specifically those at Universities that use Google apps and ISP customers like CableOne – did not give explicit permission for their mail to be scanned by Google. They’re also arguing no senders give permission.
In addition to the lack of permission, the plaintiffs lawyers are arguing that Google’s behaviour is in violation of Google’s own policies.
Google thinks scanning is part of the ordinary course of business and they’re doing nothing wrong.
This is an interesting case. I think anyone who knows about email understands that the people who run the mail server have the ability to read anything that goes through. But a lot of us trust that most postmaster and admin types consider it unprofessional to look at mail without a decent reason. There are good reasons an admin might need to go into a mail spool.
Automated filtering is simply a part of life on the internet these days. Mails have to be scanned for viruses, spam and, yes, they are scanned for targeted advertising. I’m not convinced Google is outside the norm when they say that any emails sent through Google is personal information given too Google and therefore Google can use that information in accordance with their policies.

Read More