SMTP Level Rejections

While discussing a draft of a Deliverability BCP document the issue came up of what rejections at different phases of the email delivery transaction can mean. That’s quite a big subject, but here’s a quick cheat sheet.
At initial connection
Dropped or failed connection:

  1. your reputation with the receiver is so bad that they don’t want to see any email from you, ever
  2. their mail server is badly overloaded and can’t accept any connections right now
  3. the server you’re connecting to isn’t a mail server – this is common with bad email addresses

Temporary (4xx) failure:

  1. your reputation with the receiver is poor or unknown and they’re waiting for more information before they’ll accept your mail
  2. the mail server is overloaded and can’t accept any new mail right now
  3. your reputation is “grey”, the mail server is overloaded and it’s accepting mail preferentially from other senders right now
  4. there is something technically wrong with your mailserver, such as non-existent reverse DNS, and your mail won’t be accepted until you fix that

Permanent (5xx) failure:

  1. a 554 response can mean “there’s no mailserver here”. If someone has set up a mailserver specifically to reject connections this way, it’s probably been done for a domain that has a lot of bad delivery attempts. That might mean that the recipient email address is one that’s commonly used for fake signups, or it may be a common thinko address (e.g. something.com vs something.net)
  2. they’ve explicitly blocked all mail from your mailserver due to a poor reputation – this is a common way of implementing DNS blacklist based blocking

Any time

  1. when a mailserver is shut down, it may drop existing sessions with a 421 response
  2. if you violate SMTP – send a bad command, bad parameters, lines too long – you might get 5xx responses at any time

After HELO or EHLO

  1. a syntactically bad or obviously forged (e.g. yahoo.com) HELO hostname – unless you’re a spammer or have a badly misconfigured server, this probably shouldn’t happen

After MAIL FROM

  1. mail from a domain (or, in theory, return path) might be blocked here, but it’s more likely to be rejected at RCPT TO time

After RCPT TO
This is a favorite point to apply reputation based filtering, for several reasons. It’s the first point at which you know who the intended recipient is, and that’s a very useful thing to record to help with diagnosing spam filtering false positives. So even the mailserver has decided it’s going to reject a message, based on the IP address it’s being sent from, waiting until the RCPT TO to reject if will give better diagnostics. There may be special recipient addresses that are exempt from global filtering (such as role addresses), so that filtering can’t be applied until this step. Also, some mailing list managers historically don’t handle rejections before RCPT TO gracefully – they’ll keep retrying and won’t unsubscribe the user forever.
If a mailserver waits until after this point it either has to accept the entire email or unceremoniously crash the connection (which violates SMTP and will provoke a redelivery attempt, so is rarely seen).
Temporary (4xx) failure:

  1. the server is out of resources, and just needs you to come back later
  2. the server is low on resources and mail from you isn’t as important as mail from other senders right now,
  3. the overall volume of mail you’re sending has exceeded short term limits the receiver is applying to you (based on reputation) and it’s deferring mail so as to throttle your traffic
  4. there’s a temporary problem (DNS blacklist resolution, perhaps) in looking up your reputation
  5. your reputation is “grey”, the mailserver hasn’t decided whether your mail is likely to be wanted or not, and it’s letting a small fraction of your mail through, to see how recipients respond
  6. your reputation is “grey” or there isn’t enough information about you to make a delivery decision, so your mail is being deferred in the hope that there’ll be a clearer reputation decision when you try to redeliver it
  7. some combination of all the above

Permanent (5xx) failure:

  1. the recipient doesn’t exist – sometimes a bad address is just a bad address
  2. the source (IP address or domain) of the email has a poor reputation and mail is blocked
  3. there’s a per-recipient block on this email

After DATA

  1. the content of the message “looks like spam” – either because it looks similar to recently received unwanted messages or based on heuristics such as those used by spamassassin
  2. URLs in the message have a poor reputation (or resolve to IP addresses with poor reputation)
  3. DKIM signatures will be checked at this point, so if the mail would have been delivered due to DKIM-based whitelisting and the DKIM signature doesn’t validate that may cause rejection here
  4. DMARC failures
  5. virus or malicious content filters
  6. any of the reasons given for RCPT TO – some filters may defer any rejection until after data, either so as to be able to analyze the content of rejected messages, or because they use a simple architecture that does all it’s filtering at once,
  7. score-based filtering – based on combining everything the mailserver knows about this message, including the content

There’s probably something I’ve missed – call it out in the comments – but these are some of the things I think about when working out why a message has been rejected with an unhelpful error message.

Related Posts

Questions on Google lawsuit post

A couple questions in the previous discussion thread about the Google privacy case. Both concern permission granted to Google to scan emails.
Google’s stance about this is fairly simple.
Gmail users give explicit permission for their mail to be scanned.
People who send mail to Gmail users give implicit permission for their mail to be scanned.
The plaintiff’s lawyers are alleging that some subset of gmail users – specifically those at Universities that use Google apps and ISP customers like CableOne – did not give explicit permission for their mail to be scanned by Google. They’re also arguing no senders give permission.
In addition to the lack of permission, the plaintiffs lawyers are arguing that Google’s behaviour is in violation of Google’s own policies.
Google thinks scanning is part of the ordinary course of business and they’re doing nothing wrong.
This is an interesting case. I think anyone who knows about email understands that the people who run the mail server have the ability to read anything that goes through. But a lot of us trust that most postmaster and admin types consider it unprofessional to look at mail without a decent reason. There are good reasons an admin might need to go into a mail spool.
Automated filtering is simply a part of life on the internet these days. Mails have to be scanned for viruses, spam and, yes, they are scanned for targeted advertising. I’m not convinced Google is outside the norm when they say that any emails sent through Google is personal information given too Google and therefore Google can use that information in accordance with their policies.

Read More

Holiday mailing advice from mailbox providers

Christine Borgia has a post on the Return Path blog where she interviews a number of different groups (spamfilters, DNSBLs, mailbox providers) about their filtering strategy for the holidays. Overall, no one changes their filtering during the Holiday Mailing Season. On the other hand, many marketers do change their marketing strategies in ways that trigger more filtering and blocking.
The take home message? Pay attention to what is being sent and who it is being sent to. This is nothing new, but many marketers seem to forget it in the effort to get into their customers’ inboxes.

Read More

Abuse it and lose it

Last week I blogged about the changes at ISPs that make “ISP Relations” harder for many senders. But it’s not just ISPs that are making it a little more difficult to get answers to questions, some spam filtering companies are pulling back on offering support to senders.
For instance, Cloudmark sent out an email to some ESPs late last week informing them that Cloudmark was changing their sender support policies. It’s not that they’re overwhelmed with delisting requests, but rather that many ESPs are asking for specific data about why the mail was blocked. In December, Spamcop informed some ESPs that they would stop providing data to those ESPs about specific blocks and spam trap hits.
These decisions make it harder for ESPs to identify specific customers and lists causing them to get blocked. But I understand why the filtering companies have had to take such a radical step.
Support for senders by filtering companies is a side issue. Their customers are the users of the filtering service and support teams are there to help paying customers. Many of the folks at the filtering companies are good people, though, and they’re willing to help blocked senders and ESPs to figure out the problem.
For them, providing information that helps a company clean up is a win. If an ESP has a spamming customer and the information from the filtering company is helping the ESP force the customer to stop spamming that’s a win and that’s why the filtering companies started providing that data to ESPs.
Unfortunately, there are people who take advantage of the filtering companies. I have dozens of stories about how people are taking advantage of the filtering companies. I won’t share specifics, but the summary is that some people and ESPs ask for the same data over and over and over again. The filtering company rep, in an effort to be helpful and improve the overall email ecosystem, answers their questions and sends the data. In some cases, the ESP acts on the data, the mail stream improves and everyone is happy (except maybe the spammer). In other cases, though, the filtering company sees no change in the mail stream. All the filtering company person gets is yet another request for the same data they sent yesterday.
Repetition is tedious. Repetition is frustrating. Repetition is disheartening. Repetition is annoying.
What we’re seeing from both Spamcop and Cloudmark is the logical result from their reps being tired of dealing with ESPs that aren’t visibly fixing their customer spam problems. Both companies are sending some ESPs to the back of the line when it comes to handling information requests, whether or not those ESPs have actually been part of the problem previously.
The Cloudmark letter makes it clear what they’re frustrated about.

Read More