Optimize your SPF records

I talked on Monday about the SPF rule of ten and how it made it difficult for companies to use multiple services that send email on their behalf.
Today I’m going to look at how to fix things, by shrinking bloated SPF records. This is mostly aimed at those services who send email on their customers behalf and ask their customers to include an SPF record as that’s the biggest pain point, but some of it is also useful to people publishing their own SPF records.
email_vice
Get rid of costly SPF directives
First, rethink using the “mx” directive. It’s often used in example SPF records, because it makes them look simpler. But an MX directive always triggers a DNS lookup that counts against your limit of ten, and it’ll also trigger a DNS lookup for each hostname in your MX record – they don’t count against the SPF limit but may increase the latency of your delivery a little. Better than using “mx” is to use explicit “ip4” and “ip6” records to list the addresses your smarthost and MX send mail from. Even though this makes your SPF record look longer it’ll actually make it smaller, as measured by DNS queries, as a single “mx” directive costs more than 20 “ip4” directives.
Similarly, avoid the “a” directive. It’s much less commonly seen but again can usually be replaced with “ip4” or “ip6” directives.
Don’t use “ptr” directives. They’re deprecated by the current SPF RFC.
Check the address ranges
If you have many “ip4” and “ip6” directives, make sure they’re not redundant. Are there any address ranges that you’re not using any more? Are there any adjacent address ranges that can be merged? For example, “ip4:x.y.z.4/24” and “ip4:x.y.z.5/24” can be replaced with “ip4:x.y.z.4/23” (note that you can’t always replace adjacent address blocks of the same size – read up on CIDR notation).
If you’ve generated your CIDR blocks from address ranges you can sometimes have very inefficient representations. The address range 10.11.12.1-10.11.12.254 needs 14 “ip4” directives to represent precisely. Instead you can use the single directive “ip4:10.11.12.0/24”, even if you’re not sending any email from the .0 or .255 addresses.
You don’t need a “~all” or “-all” at the end of a TXT record that is only included in another SPF record, not used directly. It won’t do any harm but it wastes a few characters.
Once you’ve got your list of SPF directives cleaned up the next thing to do is to pack them into one or more DNS TXT records.
Use as few TXT records as you can
Some SPF tutorials say that you can’t put more than 255 characters of SPF data into each TXT record. That’s not quite true, though.
A TXT record contains one or more strings of text and each string can contain no more than 255 characters. But an SPF checker will take all of the strings in a TXT record and concatenate them together in order before it starts looking at the content. So you can have more than 255 characters of SPF data in a TXT record by splitting it into more than one string. (Some low-end DNS management web front ends don’t really understand TXT records and won’t let you include multiple strings – you should check that your DNS management system does before relying on this).
How much more than 255? That’s where you have to get a little familiar with the DNS protocol, as the real limitation is that you don’t want your DNS packets to be more than 512 bytes long. (Why 512 bytes? That’s a long story of protocol changes and incompatibility, but 512 bytes are about as big as you can reliably use over UDP. Just trust me.)
The DNS overhead for a reply that contains a single TXT record with two strings is about 34 bytes, plus the length of the hostname that’s being queries (e.g. “spf.example.com” is 15 bytes). So to keep within the 512 byte limit you need to break your SPF into chunks of no more than 478 minus the length of the hostname. Then you need to break that SPF data into two strings (remembering that they’ll be concatenated with no white space added, so if you break it at a space you need to include the space at the end of the first string or the beginning of the second).
That’ll give you a TXT record that looks something like this:

spf.example.com 3600 IN TXT “v=spf1 ip4:10.11.12.13 … more spf” ” yet more spf ~all”

TXT records other than SPF
Note that the size of the DNS reply is driven by all the matching TXT records. For an SPF record designed to be included – such as spf.example.com – that’s not a problem, but for the actual SPF record for a domain you need to be aware of other TXT record pollution at the domain root. “spf2.0/pra”, “v=msv1”, google validation records, all that junk makes your DNS response bigger. You can use the dig commandline tool – dig example.com txt – to see what records you have and how big your DNS reply is. This also means that when you’re providing an SPF record for your users to include it’s important that you provide it explicitly – for example “include:spf.example.com” – rather than telling them to include your whole SPF record as “include:example.com”. This avoids a lot of other potential problems with unexpected nested includes.
How much it can help
Making these changes to slim down the SPF records you ask customers to include is important to the health of SPF. If a customer cannot add your SPF record because they don’t have enough available DNS queries to do so then either they’re not going to use you as a vendor, or they won’t be able to use SPF to authenticate mail you send and delivery is likely to suffer (unless you also support delegated DKIM signing – if you do that’s great, but it’s much more work than cleaning up your SPF).
The two vendors I currently have in my SPF record each consume three DNS lookups. I could add one more similar vendor before it broke my SPF record. If the vendors I use shrank their included SPF records down from costing three lookups to one lookup then I could use ten vendors before I needed to worry about SPF size.
(Think your customers won’t be aware of how costly their SPF records are? Just this morning we got a call from someone whose ESP is using our SPF checking tool to check the health of new customers SPF as part of their onboarding process.)
A tool to do the hard work for you
Optimizing your SPF to this degree is important, but it’s hard. We’ve created a prototype tool to help with optimizing your included SPF records at tools.wordtothewise.com/spf/minimize. Enter your SPF include record, hit the Minimize button and read the “Read Me First” tab to see how to use it.
Checking some email vendors I use or like there’s definitely room for improvement.
Zoho shrank from 3 queries to 1
MailChimp shrank from 3 queries to 1
SalesForce shrank from 3 queries to 1
Google Apps shrank from 4 queries to 1
Outlook shrank from 3 queries to 2
(If you grok SPF or DNS and you see any issues with what I’m suggesting, please leave a comment here or send me email)

Related Posts

SPF: The rule of ten

Some mechanisms and modifiers (collectively, “terms”) cause DNS queries at the time of evaluation, and some do not. The following terms cause DNS queries: the “include”, “a”, “mx”, “ptr”, and “exists” mechanisms, and the “redirect” modifier. SPF implementations MUST limit the total number of those terms to 10 during SPF evaluation, to avoid unreasonable load on the DNS. If this limit is exceeded, the implementation MUST return “permerror”.

Read More

What to expect in 2016

WttWColorEye_forBlogI don’t always do predictions posts, even though they’re  popular. Most years I skip them because I don’t see major changes in the email space. And, I’m not the type to just write a prediction post just to post a prediction.
This year, though, I do see changes for everyone in the email space. Most of them center on finally having to deal with the technical debt that’s been accumulating over the past few years. I see ISPs and ESPs spending a lot of development effort to cope with the ongoing evolution authentication requirements.
When people started seriously looking at how to authenticate email, the first goal was getting organizations to implement the protocols. This was a practical concession; in order for a new protocol to be used it needs to be widely implemented. Phase one of authenticating email was simply about publishing protocols and getting organizations to use them.
During phase one, the organization that authenticated a mail hasn’t been important. In fact, the SPF spec almost guarantees that the ESP domain is the authenticated domain. In DKIM, the spec says any domain could sign as long as they could publish a public key in that domain’s domainkeys record.
ESPs took full advantage of this and lowered their own development overhead by taking most of the authentication responsibility on themselves. Their domains were in the 5321.from and they published the SPF records. Domains they control were in the d= and they generated and published the DKIM keys. Mail was authenticated without ESP customers having to do much.
We’ve hit the end of phase one. Most of the major players in the email space are authenticating outbound email. Many of the major players are checking authentication on the inbound. Phase one was a success.
We’re now entering phase two, and that changes thing. In phase two, SPF and DKIM are used as the foundation for user visible authentication. Neither SPF nor DKIM were designed to be user visible protocols. To understand what they’re authenticating you have to understand SMTP and email. Even now there are days when I begin talking about one of them and have to take a step back and think hard about what is being authenticated. And I use these things every day!
DMARC is the first of these end user visible protocols built on SPF and DKIM. It uses the established and widespread authentication to validate the user visible from address. This authentication requires that the d= value or the 5321.from address belong belong to the same domain in the visible from address. While you can pick whether the alignment between the visible from and the authentication is “strict” or “relaxed” you have no choice about the alignment.
Prior to DMARC no one really paid much attention to the domain doing the authentication. Authentication was a yes or a no question. If the answer was yes, then receivers could use the authenticated domain to build a reputation. But they weren’t really checking much in the way of who was doing the authentication.
In the push to deploy authentication, ESPs assumed the responsibility for authentication deployed ESPs took the responsibility and did most of the work. For many or most customers, authentication was as simple as clicking a checkbox during deployment. Some ESPs do currently let customers authenticate the mail themselves, but there’s enough overhead in getting that deployed that they often charged extra to cover the costs.
DMARC is rapidly becoming an expectation or even a full on requirement for inbox delivery. In order to authenticate with DMARC, the authenticating domain must be in the same domain space as the visible from. If senders want to use their own domain in the visible from, DNS records have to be present in that domain space. Whether it’s a SPF TXT record or a domainkeys record the email sender customer needs to publish the correct information in DNS. Even now, if you try to authenticate with DKIM through google apps, they require you to publish DNS records.
ESPs aren’t in a situation where they can effectively manage authentication alignment for all their customers. Hosting companies are in even worse shape when it comes to letting customers authenticate email. Developers are facing the fact they need to go back and rework their authentication code. Businesses are facing the fact they need to change their processes so customers can authenticate with DMARC.
It’s not just the infrastructure providers that are facing challenges with authentication. Senders are going to discover they can no longer hand authentication off to their ESPs and not worry about it. They’re going to have to get DNS records published by their own staff.
Getting DNS updates through some big companies is sometimes more difficult than it should be. I had one client a few years ago where getting rDNS changed to something non-generic took over a month. From an IT standpoint, changing DNS should require approvals and proper channels. Marketers may find this new process challenging.
And, if organizations want to publish reject policies for their domains, then they will have to publish records for every outside provider they use. Some of those providers can’t support DMARC alignment right now.
In 2016 a lot of companies will discover their current infrastructure can’t cope with modern authentication requirements. A lot of effort, both in terms of product development and software development, will need to be spent to meet current needs. This means a lot of user visible features will be displaced while the technical debt is paid.
These changes will improve the security and safety of email for everyone. It won’t be very user visible, which will give the impression this was a slow year for email development. Don’t let that fool you, this will be a pivotal year in email.

Read More

TXTing

txt
On Friday I talked a bit about the history behind TXT records, their uses and abuses.
But what’s in a TXT record? How is it used? When and where should you use them?
Here’s what you get if you query for the TXT records for exacttarget.com from a unix or OS X command line with dig exacttarget.com txt

Read More