Message not compliant with the RFCs

Every once in a while we’ll see a rejection from Yahoo that says RFCs 554 5.0.0 Message not accepted due to failed RFC compliance. What does that mean and what can we do about it?

It really does mean exactly what it says on the label: there’s something about the message that is not in compliance with any number of RFCs and are not going to accept the message in its current state. When trying to help a colleague diagnose the issue I came up with a list of things to check.

Troubleshooting in the email

  • Is there any high ASCII without quoted printable or Base64 encoding in the body or the headers?
  • Is there a Date header?
  • Is there any duplication in header fields?
  • Is there a bare IP address in a link somewhere?
  • Are the line lengths inside the message shorter than 998 characters?
  • Are lines correctly terminated with CR/LF?
  • Is the DKIM signature field correct?
  • Is the MIME type correct?

Troubleshooting outside the email

  • Is the sending domain configured for DNSSec? Did something break?
  • Is DKIM, SPF and DMARC correct or did a DNS update get pushed and break something?
  • Does the SPF domain have a MX and is it answering on port 25?
  • Does the From domain have a MX and is it answering on port 25?
  • Is there a problem with TLS negotiation?
  • Is there a network problem somewhere preventing DNS lookups from happening?

There are so many different technical ratholes to go down when troubleshooting technical errors, but this gives people a place to start with issue. The most common problems I see are:

  • Line lengths too long or incorrectly terminated. Yahoo isn’t the only receiver to refuse mail for this problem.
  • Header duplication. There are some headers that cannot and should not show up more than once in a message and sometimes there are duplications that happen.
  • Lack of a Date header. I’ve seen this in mail coming in here just recently.

Overall, most receivers are fairly liberal in what they receive and don’t reject mail simply because it’s not fully in compliance with the RFCs. Some, like Yahoo, are much pickier about the standards.

The underlying solution is the same: fix whatever it is that’s broken and resend the messages.

Edit: After this was posted a representative of Yahoo contacted me to let me know that their requirements for RFC compliance were to address some types of exploits they’ve seen in the past.

Command Line Tools
Stop with the incorrect SPF advice

Related Posts

Censorship, email and politics

Spamfiltering blocks email. This is something we all know and understand. For most people, that is everyone who doesn’t manage an email server or work in the delivery field or create spamfilters, filtering is a totally unseen process. The only time the average person notices filters is when they break. The breakage could be blocking mail they shouldn’t, or not blocking mail they should.
Yesterday, a bunch of people noticed that Yahoo was blocking mail containing references to a protest against Wall Street. This understandably upset people who were trying to use email as a communication medium. Many people decided it was Yahoo (a tool of the elites!) attempting to censor their speech and stop them from organizing a protest.
Yeah. Not so much.
Yahoo looked into it and reported that the mail had gotten caught in their spam filters. Yahoo adjusted their filters to let the mail through and all was (mostly) good.
I don’t think this is actually a sign of filters being broken. The blocked mail all contained a URL pointing to a occupywallst.com. I know there was a lot of speculation about what was being blocked, but sources tell me it was the actual domain. Not the phrase, not the text, the domain.
The domain was in a lot of mostly identical mail coming out of individual email accounts. This is a current hallmark of hijacked accounts. Spammers compromise thousands of email accounts, and send a few emails out of each of them. Each email is mostly identical and points to the same URL. Just like the protest mail.
There was also a lot of bulk mail being sent with that URL in it. I’ve been talking to friends who have access to traps, and they were seeing a lot of mail mentioning occupywallst.com in their traps. This isn’t surprising, political groups have some horrible hygiene. They are sloppy with acquisition, they trade names and addresses like kids trade cold germs, they never expire anything out. It’s just not how politics is played. And it’s not one party or another, it’s all of them. I’ve consulted with major names across the political spectrum, and none actually implement best practices.
As I have often said the secret to delivery is to not have your mail look like spam. In this case, the mail looked like spam. In fact, it looked like spam that was coming from hijacked accounts as well as spam sent by large bulk mailers. I suspect there was also a high complaint rate as people sent it to friends and family who really didn’t want to hear about the protests.
To Yahoo!’s credit, though, someone on staff was on top of things. They looked into the issue and the filter was lifted within a couple hours of the first blog post. A human intervened, overruled the algorithm and let the mail out.
I bet this is one of the few times anyone has seen that Yahoo does outbound filtering. Given it’s a politically charged situation, I can see why they assume that Yahoo is filtering because of politics and censorship. They weren’t though.
More on politics, filtering and censorship.

They’re not blocking you because they hate you
It really can be your email
More on Truthout
Another perspective on the politico article

Read More

Same MX, different filters

One of the things I do for clients is look at who is really handling mail for their subscribers. Steve’s written a nifty tool that does a MX lookup for a list of domains. Then I have a SQL script that takes the raw MX lookup and categorizes not by the domain or even the MX, but by the underlying mail filter.

Read More

Why do ISPs do that?

One of the most common things I hear is “but why does the ISP do it that way?” The generic answer for that question is: because it works for them and meets their needs. Anyone designing a mail system has to implement some sort of spam filtering and will have to accept the potential for lost mail. Even the those recipients who runs no software filtering may lose mail. Their spamfilter is the delete key and sometimes they’ll delete a real mail.
Every mailserver admin, whether managing a MTA for a corporation, an ISP or themselves inevitably looks at the question of false positives and false negatives. Some are more sensitive to false negatives and would rather block real mail than have to wade through a mailbox full of spam. Others are more sensitive to false positives and would rather deal with unfiltered spam than risk losing mail.
At the ISPs, many of these decisions aren’t made by one person, but the decisions are driven by the business philosophy, requirements and technology. The different consumer ISPs have different philosophies and these show in their spamfiltering.
Gmail, for instance, has a lot of faith in their ability to sort, classify and rank text. This is, after all, what Google does. Therefore, they accept most of the email delivered to Gmail users and then sort after the fact. This fits their technology, their available resources and their business philosophy. They leave as much filtering at the enduser level as they can.
Yahoo, on the other hand, chooses to filter mail at the MTA. While their spamfoldering algorithms are good, they don’t want to waste CPU and filtering effort on mail that they think may be spam. So, they choose to block heavily at the edge, going so far as to rate limit senders that they don’t know about the mail. Endusers are protected from malicious mail and senders have the ability to retry mail until it is accepted.
The same types of entries could be written about Hotmail or AOL. They could even be written about the various spam filter vendors and blocklists. Every company has their own way of doing things and their way reflects their underlying business philosophy.

Read More