Apple MPP

You’ve probably heard about Apple Mail Privacy Protection. Email marketing chat has been all a-twitter about it since it was announced in June.

Skipping over all the “Openpocalypse” panic, what is it and what does it do?

Image Loads

It’s all about images in email and how they’re loaded (particularly invisible one pixel images that are used solely for tracking).

Why do we care about image loads? Email marketers and ESPs have used the metadata included with image loads for years to grab information and metrics from their recipients. By using a unique name for an image they can tell when a particular recipient loads that image. That image load correlated with a particular recipient is what ESPs describe as “an Open”, and it means that means the sender knows the recipient read the email. They’ll often use an invisible, single pixel image that can be easily added to every email they send without affecting the design, and track image loads solely of that image. The notorious “tracking pixel”.

Email marketers use these image loads as a crude signal as to whether an email is going to a recipients Inbox and being interacted with by the recipient, or not. They’ll often refer to this as “engagement”, but it’s very, very different to the “engagement” measured by mailbox providers who own the mail client, such as webmail providers, who can observe the user’s behaviour in a way that email marketers have no access to.

Image loads do let email marketers get a broad view of whether more mail might be going to the junk folder this month than last. And it’s also used to segment out more active and less active users, as measured by their loading images or not, with the hope that if you stop sending mail to people who aren’t loading images that’ll reduce the number of “unengaged” users as measured by the mailbox provider, improving delivery.

Image loads haven’t been a great measure for years, and Apple MPP is going to make it even less useful.

(There are other neat tricks you can do with image loads – you can keep the web session for the image open and see how long a user has the mail open in their client, or you can generate the image dynamically to give countdown timers or geoip localized custom information in the image. They’re not going to be as useful going forward either.)

Apple Mail Privacy Protection

MPP is technology that’s built into the mail client – Apple Mail on iOS 15 or macOS Monterey – that provides a little more privacy to recipients by changing the way images in email are loaded.

The user can choose several different ways of using it. They can load images by default, or not (this isn’t new, nor unique to Apple). They can load images directly from their ‘phone or via an anonymising proxy. And they can configure Mail.app to fetch images even if the message they’re in hasn’t been opened.

It’s not anything to do with the email servers or domains Apple host, and it will work with any mail provider. Someone receiving their mail at Gmail and reading it on their phone or mac with Apple Mail can use it. That means you can’t tell just by someone’s email address whether they’re using it or not. Some @gmail.com and @yahoo.com and corporate email addresses will be using it, some won’t.

If a user chooses to load images through a proxy then senders can still tell whether there was an “open” or not, but they lose access to some other metadata, such as the type of platform or mail client being used and geoip based location of the user.

But if they also choose to preload images even when the mail isn’t being read then marketers will see an “open” from any mail they send to that user, even if the user never reads it. That makes that “open” metric meaningless for MPP users.

How does the user see it?

Right now the user doesn’t see it unless they’re installing beta pre-releases of iOS15 or macOS Monterey. It’s not been released to the general public yet. That’s likely to happen soon, in weeks rather than months. Once it’s released it’ll end up deployed to a large fraction of iPhones very quickly based on past releases, while macOS machines will be upgraded more slowly.

When they upgrade the user sees a single switch in the “Privacy Protection” of their settings, labeled “Protect Mail Activity”. And, at least on my devices, that switch has defaulted to on. This being on turns on both preloading images and loading images via the proxy. Unless they turn that off they don’t even see any options for other combinations of behaviour.

The setting is per-device, not per-email account. And there’s no (obvious) way to enable image preloading without also enabling proxies.

Other than that there’s no obvious change in email behaviour. Messages look just the same, images will always load when they open a mail. If the mail has preloaded images then the message will load a little quicker, as the image is already there, but other than the “feels a bit snappier” there’s no difference.

How does it work?

I’ve set up a test bed with an iPhone running iOS15 beta and a mac mini running macOS Monterey beta. They’re both connected to the net via a dedicated wifi access point where I can sniff all the traffic they’re sending (wireshark makes that easy), and a commercial VPN so I can make that traffic enter the internet from anywhere in the world.

And I wrote some code that made it easy to send an email with a uniquely named image in it, serve the image and store all the results in a database. It saved a lot of time and typing when sending hundreds of different mails. It’s called pixelcheck, grab a copy if you’re playing with tracking images.

Preloading images

This isn’t particularly interesting behaviour. Every so often a background process on the iPhone wakes up. If it’s connected to wifi and not in low power mode it will look for new emails that haven’t had their images loaded, and it will load all of them via the proxy, then go back to sleep.

If an email is downloaded some time after it was delivered the images may be preloaded immediately. There only appears to be enough delay intended so that preload image loads aren’t so easily identifiable.

Images are preloaded from the inbox and from other folders (I’m assuming that includes the junk folder, but haven’t tested that).

If you’ve sent multiple emails to a recipient over a short period you may see a burst of image loads for multiple emails within a fraction of a second, but otherwise it looks exactly like the proxied image load you’d see when a user opens the mail.

Proxy image loading

This is where it gets interesting technically.

Whether the image is being preloaded or loaded when an email is viewed the proxy process is the same.

Some readers will care about the technical details, so this may be a bit of a deep dive. Feel free to skip forward to the diagram if you just want a summary.

The ‘phone asks Apple’s DNS for two hostnames, mask.icloud.com and mask-h2.icloud.com. It asks for both A records and HTTPS records. The latter is a draft standard from Google and Akamai that lets a client initialise a https connection in a much faster, more secure, more privacy protecting way than just grabbing an A record and poking port 443. For our purpose the differences don’t really matter, other than to show that Apple and Akamai have done the work to make the initial proxy QUIC connection fast and secure.

Those hostnames are CNAMEd to some apple-dns.net hostnames which end up pointing at a cluster of Akamai servers. The DNS resolution is using geolocation magic to point at the local CDN cluster. They’re close in network terms, less than 15 ms round trip away. So far, so good.

That’s all from Dublin. If I’m in Dallas then the DNS resolves to a similar looking cluster, but hosted on Apple’s own network. I’m assuming that doesn’t make any difference to their behaviour, but it shows it’s a geographically distributed, performance optimised network maintained by, at least, Apple and Akamai.

Then Apple Mail opens a QUIC connection to one of those machines. QUIC is sort of the next generation of TCP/TLS/HTTP. Most modern browsers support it, and it’s probably how you’re connecting to Google or Facebook, amongst other places. We don’t really care about all it’s very, very cool features other than to know it’s very fast and it’s very encrypted.

After a fraction of a second we see the log entry on our webserver as the image is fetched, and Akamai sends it back to Apple Mail via the QUIC connection, and it’s displayed.

I’d love to sniff inside this connection and see what’s going on, but it’s TLS encrypted. If it were coming from a web browser I’d inject a fake certificate into the browser or OS registry, set it up in an environment with transparent proxies and man-in-the-middle the poor thing. But it’s an app, not a browser, and one that connects to a small, well-defined group of servers. It’s going to have the certificate pinned, and there’s no way to easily override that.

But … we can still see how much traffic is going back and forth just by watching the UDP packets go to and fro. QUIC has some padding in it, and there’s session overhead and so on so we can’t necessarily get hard information, but we can get an idea. I patch the webserver to wait five seconds before it returns the requested image. Everything still works fine, but now the QUIC session is split into two sections, before and after the big five second gap in the middle.

Apple Mail sends about 12k of traffic to Akamai, to fetch a link that’s less than 40 characters long, so there’s plenty of space for Apple Mail to be sending metadata, if it felt so inclined. After the gap it looks like Akamai just sending the image content.

And the other proxy

But when we look at what the webserver is doing to serve the image it’s not seeing a request from Akamai or Apple network space. Instead it’s seeing requests come in from a couple of Cloudflare hosted addresses and one Fastly address.

The image requests from all three look exactly like this:

Accept: image/webp,image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5
User-Agent: Mozilla/5.0
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en;q=0.9Code language: HTTP (http)

Very vanilla request headers for a client that has a vague preference for modern image formats. The User-Agent is about as bland as it’s possible to be. It’s expressing a vague preference for English content, which is interesting.

The Cloudflare accesses are coming from 104.28.9.181 and 182, while Fastly is from 146.75.162.37. They look like they’re hosted in Frankfurt and Amsterdam, or somewhere close to them in network terms. That makes sense if you’re serving users in Ireland as hosting and bandwidth is much more available and much cheaper there. (The VPS I’m using to serve images from is in Frankfurt for exactly that reason, and it’s about 300 microseconds away from the Cloudflare server as the packet flies).

So how does it all work? Apple Mail contacts a local Akamai (or Apple) hosted CDN node and tells it, amongst other things, that it wants to fetch an image. That CDN node runs some code to decide an appropriate server to use to do the image fetch, and chooses a local Cloudflare server (based, presumably, on location, server availability and load and all the other things it might be interested in). Running fragments of code for each request is something CDNs are good at. That server fetches the image from the senders webserver and returns it to the Akamai CDN, which in turn sends it back to Apple Mail.

This is a fairly clever architecture, that allows Apple to scale out infrastructure relatively cheaply by relying on third party expertise. It allows them to add and remove the servers that fetch the images, making it more tedious for marketers to identify proxied image loads, without requiring privacy trust in those third parties. Apple only needs to trust Akamai with (potential) access to the connection between the recipients device IP and metadata and the images they’re fetching. There’s much less need to trust the Cloudflares and the Fastlys – and the cabinets they’re hosted in – as they only have access to the URL being fetched.

Wait, that’s interesting…

So MPP chose some relatively local servers to fetch images for me, but it really doesn’t look like they’re on the Island of Ireland, rather they’re on the other end of some fast pipes in Germany or the Netherlands or therabouts.

What does geolocation of those IPs give?

104.28.9.181:
Ireland
Leinster
Dublin
D02
53.3382
-6.2591
Cloudflare, Inc.
Cloudflare WARP
AS13335 Cloudflare, Inc.

146.75.162.37:
Ireland
Leinster
Dublin
D02
53.3382
-6.2591
Fastly
Fastly, Inc
AS54113 Fastly

I, and my iPhone, are in Dublin. I’m pretty sure those servers aren’t. But the commercial GeoIP database is claiming not only that both of those servers – run by different companies – are in Ireland, but are at the exact same location. And that location isn’t some warehouse on the outskirts of the city, it’s here, right in the middle of Stephen’s Green in the center of Dublin:

That means that if you’re sending mail to me and using GeoIP based location to customize the content, you’ll get roughly the right customization based on a commercial GeoIP lookup of the IP address that requested the content.

This looks like Apple (or someone they’re partnered with) has requested that the GeoIP database contain false data, in a way that benefits everyone. It’s clever, and will minimize breakage, assuming that’s really what’s going on.

I’m fairly sure that’s what’s going on. Unfortunately I’ve not been able to test it, as something in the system seems to persist where my location is. I’ve used a VPN to run Mail from an IP address in a different country, disabled location services, set my device to a different location and I still get image loads from the same fake-Dublin servers.

In brief

Apple Mail with Mail Privacy Protection turned on, on iOS or macOS devices, will not load images directly. It will go through two layers of proxy before reaching your image load tracking webserver.

You won’t see loads from the recipients device, so won’t have any platform or device information. It appears you will have rough geolocation information, though. (This is probably better than you already see for gmail recipients).

Emails that are opened immediately will show image loads. Emails that are left unread for a while will also show image loads. An image load still tells you something, but it’s not that a user “opened” an email.

Recipients are likely to be happy, as it gives them increased privacy and faster loading emails with no drawbacks.

This hasn’t started yet, but it will soon.

More about MPP infrastructure and geolocation.