One of these things is just like the other
Canonicalization is about comparing things to see if they’re the same. Sometimes you want to do a “fuzzy” comparison, to see if two things are interchangeable for your purposes, even if they’re not exactly identical.
As a concrete example, these two email addresses:
- (Steve) firstname.lastname@example.org
- “Also Steve” <steve@WORDTOTHEWISE.COM>
They’re clearly not identical, but they’ll deliver to the same maibox.
I could compare them with a set of comparison rules (if the string to the left of the @ sign up to white space is the same in both, or one of them has a less than sign in front of it, and the string to the right of the @ is the same, compared case-insensitively, ….).
Or I could canonicalize both email addresses and see if the results are identical. A simple canonicalization algorithm might be “Remove anything in parentheses. Remove any quoted strings. Strip any whitespace and any greater than or less than signs at each end. Convert anything after the @ sign to lower-case”. That’ll give two canonicalized email addresses:
They’re trivially identical, so I know that the two email addresses I started with are interchangeable.
DKIM validation is all about comparing whether things have changed or not. The DKIM-Signature header contains a “fingerprint” of the canonicalized message body and of the canonicalized headers from when the mail was sent. If you canonicalize the body and headers that you received, take the fingerprint in the same way and that’s identical to the one in the DKIM-Signature header, you know the headers and body haven’t been modified since the message was signed (and then you can do some DNS lookups and some cryptography to find who signed the message).
Unfortunately, email was never designed to send messages unchanged. Intermediate servers will often “fix up” messages – by folding long lines, normalizing whitespace, adding missing headers or fixing up invalid ones, by re-encoding content to different encodings and all sorts of other changes. That would break a byte-for-byte comparison of the mail as sent and as received. And because we don’t have a copy of the mail as sent to compare with – we only have a “fingerprint” of it – we can’t do any fancy comparison. So we have to rely on the canonicalization, and hope that even after the “fix ups” made during delivery the canonicalized forms – and hence the fingerprints – will be identical.
DKIM defines two canonicalization algorithms for the body of the message, simple and relaxed.
Simple body canonicalization does very little: it just strips any blank lines at the end of the body. Relaxed body canonicalization strips those blank lines, and then replaces any run of white space – spaces or tabs – in the body with a single space. This means that any change in whitespace in the body, such as converting tabs to spaces, won’t affect the relaxed canonicalized body.
DKIM also defines two canonicalization algorithms for the headers of the message. They’re also called simple and relaxed, despite doing quite different things. (Yes, this is confusing.)
Simple header canonicalization is as simple as you can get. It makes no changes, so the headers must be byte-for-byte identical to match. Relaxed header canonicalization converts all header names to lower case, unfolds headers so each is a single line, replaces any run of white space with a single space character and removes any trailing whitespace on each line.
(See the DKIM spec if you want all the details.)
The simple takeaway from this is that simple canonicalization makes DKIM signatures that are broken by even trivial modifications in transit, while relaxed canonicalization makes them more robust.
The really simple takeaway is “use relaxed canonicalization”.
The c= field
The canonicalization you use is recorded in the c= field of the DKIM-Signature header, with the two canonicalization names separated by a slash, header first.
So “c=strict/relaxed” means to use strict canonicalization for the headers and relaxed for the body.
You can also use just a single canonicalization type in the c= field. This does not do what you expect.
“c=strict” is exactly the same as “c=strict/strict”. “c=relaxed” is exactly the same as “c=relaxed/strict”.
Yes, this makes no sense. But it’s what the spec says. If you use just “c=relaxed” you’re using strict canonicalization for the body of the message, and any change to whitespace in the body will break your signature.
Microsoft have a long history of modifying email in transit, often to “fix up” differences between standard Internet email and the expectations of their internal code. This article goes into some of how that breaks DKIM in some cases.
It appears that some paths through outlook.com from MX to inbox are converting tabs in the body of the message into spaces at the moment, while other paths aren’t. If you’re using strict DKIM body canonicalization – either intentionally or accidentally with “c=relaxed” – that means you’ll see apparently random DKIM failures for mail sent to recipients hosted by outlook.com, but only for messages where the body is susceptible to whitespace damage.
Using the right (“c=relaxed/relaxed” unless you have a good reason not to) canonicalization is a good start but you should also look at making the content you send as clean as possible, beyond just complying with the email standards avoid structures that risk being rewritten in transit. But that’s another post.
If only relaxed canonicalization could fix the DKIM forwarding problems.
I sent a message to an Office 365, and the equivalent to a Gmail account from a 3rd email account with the text “How does this message look in Gmail raw text vs. Exchange raw text?”
Exchange adds the following to the HTML:
It makes a lot of other similar changes to the headers and body, other than line wrapping. That additional insertion will change DKIM’s body hash regardless of what type of canonicalization is used.
Great article Laura and thank you!
How many ESPs actually use simple? That would be great to know. Identifying their clients and signing up to their mail is the way I know in answering that question in order to find out and compare.
We send using “c=relaxed/relaxed”, yet we will see consistent “body hash did not verify” failures at Microsoft, while no other provider seems to have issues with them.
In certain cases, I have been able to modify the message to influence the pass/fail result when sent to Microsoft, so it does seem to be making other changes to the body content. I have not yet been able to (spend the time to) determine exactly what changes to content influence the result.
I have an open support ticket that is now with their product group, and I have provided them two sample messages I sent to myself for analysis; first is a pass result with dead simple content and the second is a fail (body hash did no verify) due to additional content in the message body. Both messages were sent via the same smtp session, and I have provided them with the log data from my side that includes the hex and human readable data from that session.
I am curious to see what they respond with after analyzing the sent vs received content.
I may do some more testing to see if I can find what content changes influence the result, but there are only so many hours in the day.
Even using “c=relaxed/relaxed” is failing DKIM for messages to Microsoft because the body hash does not verify for HTML/Multi-part messages. Plain text does not have that issue. I have looked at various signers who use Power MTA to Mimecast.
Is “strict” an actual canonicalization type or did you mean “simple” in your examples?