Message not compliant with the RFCs

Every once in a while we’ll see a rejection from Yahoo that says RFCs 554 5.0.0 Message not accepted due to failed RFC compliance. What does that mean and what can we do about it?

It really does mean exactly what it says on the label: there’s something about the message that is not in compliance with any number of RFCs and are not going to accept the message in its current state. When trying to help a colleague diagnose the issue I came up with a list of things to check.

Troubleshooting in the email

  • Is there any high ASCII without quoted printable or Base64 encoding in the body or the headers?
  • Is there a Date header?
  • Is there any duplication in header fields?
  • Is there a bare IP address in a link somewhere?
  • Are the line lengths inside the message shorter than 998 characters?
  • Are lines correctly terminated with CR/LF?
  • Is the DKIM signature field correct?
  • Is the MIME type correct?

Troubleshooting outside the email

  • Is the sending domain configured for DNSSec? Did something break?
  • Is DKIM, SPF and DMARC correct or did a DNS update get pushed and break something?
  • Does the SPF domain have a MX and is it answering on port 25?
  • Does the From domain have a MX and is it answering on port 25?
  • Is there a problem with TLS negotiation?
  • Is there a network problem somewhere preventing DNS lookups from happening?

There are so many different technical ratholes to go down when troubleshooting technical errors, but this gives people a place to start with issue. The most common problems I see are:

  • Line lengths too long or incorrectly terminated. Yahoo isn’t the only receiver to refuse mail for this problem.
  • Header duplication. There are some headers that cannot and should not show up more than once in a message and sometimes there are duplications that happen.
  • Lack of a Date header. I’ve seen this in mail coming in here just recently.

Overall, most receivers are fairly liberal in what they receive and don’t reject mail simply because it’s not fully in compliance with the RFCs. Some, like Yahoo, are much pickier about the standards.

The underlying solution is the same: fix whatever it is that’s broken and resend the messages.

Edit: After this was posted a representative of Yahoo contacted me to let me know that their requirements for RFC compliance were to address some types of exploits they’ve seen in the past.

Related Posts

New Spamhaus lists

Spamhaus announced today they are publishing two new BGP feeds: Extended DROP and the Botnet C&C list. These lists are intended for use inside routers in order to stop all traffic to or from listed IP addresses. This is a great way to impact botnet traffic and hopefully will have a significant impact on virus infections and botnet traffic.
In other news I’ve been hearing rumbling about changes at Yahoo. It looks like they have changed their filters and some senders are feeling lots of pain because of it. It looks like senders with low to mid range reputations are most affected and are seeing more and more of their mail hit the bulk folder. This afternoon I’m hearing that some folks are seeing delivery  improvements as Yahoo tweaks the changes.

Read More

Why do ISPs do that?

One of the most common things I hear is “but why does the ISP do it that way?” The generic answer for that question is: because it works for them and meets their needs. Anyone designing a mail system has to implement some sort of spam filtering and will have to accept the potential for lost mail. Even the those recipients who runs no software filtering may lose mail. Their spamfilter is the delete key and sometimes they’ll delete a real mail.
Every mailserver admin, whether managing a MTA for a corporation, an ISP or themselves inevitably looks at the question of false positives and false negatives. Some are more sensitive to false negatives and would rather block real mail than have to wade through a mailbox full of spam. Others are more sensitive to false positives and would rather deal with unfiltered spam than risk losing mail.
At the ISPs, many of these decisions aren’t made by one person, but the decisions are driven by the business philosophy, requirements and technology. The different consumer ISPs have different philosophies and these show in their spamfiltering.
Gmail, for instance, has a lot of faith in their ability to sort, classify and rank text. This is, after all, what Google does. Therefore, they accept most of the email delivered to Gmail users and then sort after the fact. This fits their technology, their available resources and their business philosophy. They leave as much filtering at the enduser level as they can.
Yahoo, on the other hand, chooses to filter mail at the MTA. While their spamfoldering algorithms are good, they don’t want to waste CPU and filtering effort on mail that they think may be spam. So, they choose to block heavily at the edge, going so far as to rate limit senders that they don’t know about the mail. Endusers are protected from malicious mail and senders have the ability to retry mail until it is accepted.
The same types of entries could be written about Hotmail or AOL. They could even be written about the various spam filter vendors and blocklists. Every company has their own way of doing things and their way reflects their underlying business philosophy.

Read More

TWSD: Adapt to filters

This morning the new Yahoo! CEO posted about changes to Yahoo! mail. I logged into one of my Yahoo accounts to check and see if I had access to the new Yahoo! mail client yet. I don’t, but I did notice that spammers have adapted to the new Yahoo model of disabling filters in the mail folder. Most of the mail in my inbox has, at the very top of the message “Click not spam to enable links!”
My favorite has to be the animated gif of how to click “not spam.”
Spammers spend so much time and energy compensating for filters, hopping IP addresses, rotating through domains, and specially creating mail for different ISPs. I have to wonder, though, if they would waste less time by sending opt-in mail.

Read More