BLOG

Industry News & Analysis

Apple MPP

You’ve probably heard about Apple Mail Privacy Protection. Email marketing chat has been all a-twitter about it since it was announced in June.

Skipping over all the “Openpocalypse” panic, what is it and what does it do?

Image Loads

It’s all about images in email and how they’re loaded (particularly invisible one pixel images that are used solely for tracking).

Why do we care about image loads? Email marketers and ESPs have used the metadata included with image loads for years to grab information and metrics from their recipients. By using a unique name for an image they can tell when a particular recipient loads that image. That image load correlated with a particular recipient is what ESPs describe as “an Open”, and it means that means the sender knows the recipient read the email. They’ll often use an invisible, single pixel image that can be easily added to every email they send without affecting the design, and track image loads solely of that image. The notorious “tracking pixel”.

Email marketers use these image loads as a crude signal as to whether an email is going to a recipients Inbox and being interacted with by the recipient, or not. They’ll often refer to this as “engagement”, but it’s very, very different to the “engagement” measured by mailbox providers who own the mail client, such as webmail providers, who can observe the user’s behaviour in a way that email marketers have no access to.

Image loads do let email marketers get a broad view of whether more mail might be going to the junk folder this month than last. And it’s also used to segment out more active and less active users, as measured by their loading images or not, with the hope that if you stop sending mail to people who aren’t loading images that’ll reduce the number of “unengaged” users as measured by the mailbox provider, improving delivery.

Image loads haven’t been a great measure for years, and Apple MPP is going to make it even less useful.

(There are other neat tricks you can do with image loads – you can keep the web session for the image open and see how long a user has the mail open in their client, or you can generate the image dynamically to give countdown timers or geoip localized custom information in the image. They’re not going to be as useful going forward either.)

Apple Mail Privacy Protection

MPP is technology that’s built into the mail client – Apple Mail on iOS 15 or macOS Monterey – that provides a little more privacy to recipients by changing the way images in email are loaded.

The user can choose several different ways of using it. They can load images by default, or not (this isn’t new, nor unique to Apple). They can load images directly from their ‘phone or via an anonymising proxy. And they can configure Mail.app to fetch images even if the message they’re in hasn’t been opened.

It’s not anything to do with the email servers or domains Apple host, and it will work with any mail provider. Someone receiving their mail at Gmail and reading it on their phone or mac with Apple Mail can use it. That means you can’t tell just by someone’s email address whether they’re using it or not. Some @gmail.com and @yahoo.com and corporate email addresses will be using it, some won’t.

If a user chooses to load images through a proxy then senders can still tell whether there was an “open” or not, but they lose access to some other metadata, such as the type of platform or mail client being used and geoip based location of the user.

But if they also choose to preload images even when the mail isn’t being read then marketers will see an “open” from any mail they send to that user, even if the user never reads it. That makes that “open” metric meaningless for MPP users.

How does the user see it?

Right now the user doesn’t see it unless they’re installing beta pre-releases of iOS15 or macOS Monterey. It’s not been released to the general public yet. That’s likely to happen soon, in weeks rather than months. Once it’s released it’ll end up deployed to a large fraction of iPhones very quickly based on past releases, while macOS machines will be upgraded more slowly.

When they upgrade the user sees a single switch in the “Privacy Protection” of their settings, labeled “Protect Mail Activity”. And, at least on my devices, that switch has defaulted to on. This being on turns on both preloading images and loading images via the proxy. Unless they turn that off they don’t even see any options for other combinations of behaviour.

The setting is per-device, not per-email account. And there’s no (obvious) way to enable image preloading without also enabling proxies.

Other than that there’s no obvious change in email behaviour. Messages look just the same, images will always load when they open a mail. If the mail has preloaded images then the message will load a little quicker, as the image is already there, but other than the “feels a bit snappier” there’s no difference.

How does it work?

I’ve set up a test bed with an iPhone running iOS15 beta and a mac mini running macOS Monterey beta. They’re both connected to the net via a dedicated wifi access point where I can sniff all the traffic they’re sending (wireshark makes that easy), and a commercial VPN so I can make that traffic enter the internet from anywhere in the world.

And I wrote some code that made it easy to send an email with a uniquely named image in it, serve the image and store all the results in a database. It saved a lot of time and typing when sending hundreds of different mails. It’s called pixelcheck, grab a copy if you’re playing with tracking images.

Preloading images

This isn’t particularly interesting behaviour. Every so often a background process on the iPhone wakes up. If it’s connected to wifi and not in low power mode it will look for new emails that haven’t had their images loaded, and it will load all of them via the proxy, then go back to sleep.

If an email is downloaded some time after it was delivered the images may be preloaded immediately. There only appears to be enough delay intended so that preload image loads aren’t so easily identifiable.

Images are preloaded from the inbox and from other folders (I’m assuming that includes the junk folder, but haven’t tested that).

If you’ve sent multiple emails to a recipient over a short period you may see a burst of image loads for multiple emails within a fraction of a second, but otherwise it looks exactly like the proxied image load you’d see when a user opens the mail.

Proxy image loading

This is where it gets interesting technically.

Whether the image is being preloaded or loaded when an email is viewed the proxy process is the same.

Some readers will care about the technical details, so this may be a bit of a deep dive. Feel free to skip forward to the diagram if you just want a summary.

The ‘phone asks Apple’s DNS for two hostnames, mask.icloud.com and mask-h2.icloud.com. It asks for both A records and HTTPS records. The latter is a draft standard from Google and Akamai that lets a client initialise a https connection in a much faster, more secure, more privacy protecting way than just grabbing an A record and poking port 443. For our purpose the differences don’t really matter, other than to show that Apple and Akamai have done the work to make the initial proxy QUIC connection fast and secure.

Those hostnames are CNAMEd to some apple-dns.net hostnames which end up pointing at a cluster of Akamai servers. The DNS resolution is using geolocation magic to point at the local CDN cluster. They’re close in network terms, less than 15 ms round trip away. So far, so good.

That’s all from Dublin. If I’m in Dallas then the DNS resolves to a similar looking cluster, but hosted on Apple’s own network. I’m assuming that doesn’t make any difference to their behaviour, but it shows it’s a geographically distributed, performance optimised network maintained by, at least, Apple and Akamai.

Then Apple Mail opens a QUIC connection to one of those machines. QUIC is sort of the next generation of TCP/TLS/HTTP. Most modern browsers support it, and it’s probably how you’re connecting to Google or Facebook, amongst other places. We don’t really care about all it’s very, very cool features other than to know it’s very fast and it’s very encrypted.

After a fraction of a second we see the log entry on our webserver as the image is fetched, and Akamai sends it back to Apple Mail via the QUIC connection, and it’s displayed.

I’d love to sniff inside this connection and see what’s going on, but it’s TLS encrypted. If it were coming from a web browser I’d inject a fake certificate into the browser or OS registry, set it up in an environment with transparent proxies and man-in-the-middle the poor thing. But it’s an app, not a browser, and one that connects to a small, well-defined group of servers. It’s going to have the certificate pinned, and there’s no way to easily override that.

But … we can still see how much traffic is going back and forth just by watching the UDP packets go to and fro. QUIC has some padding in it, and there’s session overhead and so on so we can’t necessarily get hard information, but we can get an idea. I patch the webserver to wait five seconds before it returns the requested image. Everything still works fine, but now the QUIC session is split into two sections, before and after the big five second gap in the middle.

Apple Mail sends about 12k of traffic to Akamai, to fetch a link that’s less than 40 characters long, so there’s plenty of space for Apple Mail to be sending metadata, if it felt so inclined. After the gap it looks like Akamai just sending the image content.

And the other proxy

But when we look at what the webserver is doing to serve the image it’s not seeing a request from Akamai or Apple network space. Instead it’s seeing requests come in from a couple of Cloudflare hosted addresses and one Fastly address.

The image requests from all three look exactly like this:

Very vanilla request headers for a client that has a vague preference for modern image formats. The User-Agent is about as bland as it’s possible to be. It’s expressing a vague preference for English content, which is interesting.

The Cloudflare accesses are coming from 104.28.9.181 and 182, while Fastly is from 146.75.162.37. They look like they’re hosted in Frankfurt and Amsterdam, or somewhere close to them in network terms. That makes sense if you’re serving users in Ireland as hosting and bandwidth is much more available and much cheaper there. (The VPS I’m using to serve images from is in Frankfurt for exactly that reason, and it’s about 300 microseconds away from the Cloudflare server as the packet flies).

So how does it all work? Apple Mail contacts a local Akamai (or Apple) hosted CDN node and tells it, amongst other things, that it wants to fetch an image. That CDN node runs some code to decide an appropriate server to use to do the image fetch, and chooses a local Cloudflare server (based, presumably, on location, server availability and load and all the other things it might be interested in). Running fragments of code for each request is something CDNs are good at. That server fetches the image from the senders webserver and returns it to the Akamai CDN, which in turn sends it back to Apple Mail.

This is a fairly clever architecture, that allows Apple to scale out infrastructure relatively cheaply by relying on third party expertise. It allows them to add and remove the servers that fetch the images, making it more tedious for marketers to identify proxied image loads, without requiring privacy trust in those third parties. Apple only needs to trust Akamai with (potential) access to the connection between the recipients device IP and metadata and the images they’re fetching. There’s much less need to trust the Cloudflares and the Fastlys – and the cabinets they’re hosted in – as they only have access to the URL being fetched.

Wait, that’s interesting…

So MPP chose some relatively local servers to fetch images for me, but it really doesn’t look like they’re on the Island of Ireland, rather they’re on the other end of some fast pipes in Germany or the Netherlands or therabouts.

What does geolocation of those IPs give?

I, and my iPhone, are in Dublin. I’m pretty sure those servers aren’t. But the commercial GeoIP database is claiming not only that both of those servers – run by different companies – are in Ireland, but are at the exact same location. And that location isn’t some warehouse on the outskirts of the city, it’s here, right in the middle of Stephen’s Green in the center of Dublin:

That means that if you’re sending mail to me and using GeoIP based location to customize the content, you’ll get roughly the right customization based on a commercial GeoIP lookup of the IP address that requested the content.

This looks like Apple (or someone they’re partnered with) has requested that the GeoIP database contain false data, in a way that benefits everyone. It’s clever, and will minimize breakage, assuming that’s really what’s going on.

I’m fairly sure that’s what’s going on. Unfortunately I’ve not been able to test it, as something in the system seems to persist where my location is. I’ve used a VPN to run Mail from an IP address in a different country, disabled location services, set my device to a different location and I still get image loads from the same fake-Dublin servers.

In brief

Apple Mail with Mail Privacy Protection turned on, on iOS or macOS devices, will not load images directly. It will go through two layers of proxy before reaching your image load tracking webserver.

You won’t see loads from the recipients device, so won’t have any platform or device information. It appears you will have rough geolocation information, though. (This is probably better than you already see for gmail recipients).

Emails that are opened immediately will show image loads. Emails that are left unread for a while will also show image loads. An image load still tells you something, but it’s not that a user “opened” an email.

Recipients are likely to be happy, as it gives them increased privacy and faster loading emails with no drawbacks.

This hasn’t started yet, but it will soon.

3 Comments

About the Apple thing

A lot of folks are talking about Apple’s recent announcement about building privacy protection into email. I have somewhat stayed out of the conversation and I’m not sure what I really think about it. This is a change to how a lot of folks use email and no one really likes change.

I actually have a post I’ve been quietly working on talking about open rates. From my perspective, they’re an increasingly useless metric for deliverability and too many senders put too much emphasis on them and they’re not telling us what we think they’re telling us. I might work a little bit more on that, but at this point it kinda feels moot given Apple’s new announcement.

I do have some thoughts and opinions on the Apple change but I’m still thinking about it and looking at the longer term implications. I know there’s a rush to be the first with an opinion but there are a lot of angles to this. I’ve mostly been reading about what other folks have to say, so I have a bigger picture before even thinking about ‘next steps’. I know folks are upset about this and they have every right to be. But I don’t think it’s the end of deliverability or the end of email marketing. We don’t have to be happy about it, but we do have to cope with it. Just like we coped with the first spam filters and the first blocklists and the first this-is-spam button.

In no particular order here are some of my initial thoughts and impressions related to things other folks have said. Not pointing out who said what or where because, honestly, this conversation is going on in so many places I don’t even know where I saw something or who said it.

Opinion 1: Basing things like nurture campaigns and whether or not recipients get mail based on opens loses some percentage of customers. We know there are people who block remote image loading. Most webmail providers still give recipients the ability to turn it off (but I think they’ve moved away from off by default in the last few years.) Only sending mail to people who load images omits a certain type of customer / recipient from your mail stream.

Opinion 2: I’m seeing a lot of marketing statements (particularly recently) that seem to completely remove any agency from the recipient. “How do I make people open my mail” “What tricks are there to increasing engagement” “Only 30% of my recipients are opening, and how do I change the subject lines to get 35% to open?” All of these presume that something the sender does can make someone open and email or perform an action. I’ve been increasingly uncomfortable with these types of questions – it feels to me like it’s more that the recipient is a passive object to manipulate rather than a partner. A lot of what I do in delivery is treat the relationship as equal and balanced, the recipient has at least as much control as the sender.

Opinion 3: Open rates are an easy metric to measure and quantify. But that doesn’t make them a good metric. The idea that we’re now actively ‘cleaning’ open data – by trying to discover which opens are bots and which are real people, is one of the clearer signs that open rates are becoming a poorer and poorer metric. At any point where you have to remove, edit or modify data to be more accurate in your conclusions the data itself is a problem. 

Opinion 3a: Apple is increasing the noise in open rates here. But we’ve known for a while that other providers (Yahoo has said they’re doing it and my personal belief is that Google may be doing this as well) are prefetching images for some emails – just like Apple says they’re going to do. The post I was working on had a lot to do with how increasingly inaccurate open rates are and that there are too many image loads that have zero to do with whether or not anyone actually opened an email.

Opinion 4: Privacy is an issue and a lot of people think it’s creepy that they are being tracked without their knowledge. Apple has positioned themselves as a company that is going to protect the privacy of their users*, even to the point of refusing to help governmental authorities to break into apple hardware to prosecute and capture terrorists. Personally, I’m waiting to see if this mail update is going to stop image loads at all. Initial information seems to indicate Apple may be removing the ‘block all remote content’ button – which is a big problem to me. I don’t want images loaded remotely for reasons that aren’t all tracking related.

(*Yes, they’re not doing this uniformly and have decided to do different things in some different countries. The world is complicated and while I disagree with their decision in general, I don’t have all the data they’re working with or know what choices they’re actually making here.)

Opinion 4a: Companies have often pushed the line of privacy invasion and tracking people. This is bigger than email – the whole big-box-store knows when you’re pregnant and will market to you before you even know you are. Said big-box-store started simply being more subtle, they didn’t stop the underlying tracking activity. They’ve also increased the amount of tracking they’re doing in store. People are, understandably, concerned that companies can pull this level of data out of their activity and that they know what you may have put in your physical cart and then removed.

Opinion 5: This is not going to be the end of deliverability and the way to reach the inbox. B2B mailing has never been reliant on engagement and open rates the way consumer mail is. In many cases business filters can’t measure engagement and in other cases engagement isn’t important for determining what mail belongs in the inbox.

Opinion 5a: In fact, the B2B space is a model for what the consumer space is starting to look like. For B2B mail we don’t have all of the data we do for consumers and yet, still, companies are successfully marketing to business domains every day. ie, we can’t fix deliverability to a business domain based on sending mail to engaged users – that’s now how the filters work. And business filters are much more prone to doing what Apple is doing – following all links in emails. We know that marketing works in this case. It’s a little more challenging, yes. But it’s not impossible.

Personally, I’ve been trying to work out other ways to capture information about subscribers for a while. And as I’ve been working more and more in the business space have been really discovering the overall limitations of ‘sending mail only to people who open messages’.

In some ways I feel like the hey day of deliverability was 2009 – 2015: probe addresses worked, panel addresses worked, open rates were easy to measure, filters were consistent across providers, FBLs were widespread, it was cool for companies to have postmaster pages, dedicated IPs and certification and the reliance on IP reputation and simple authentication all made deliverability a problem we could solve. We had a plethora of tools at our disposal and a lot of knowledge and data to base our actions and recommendations on.

Many of those tools have gradually being removed from our toolboxes in the last few years. Google and Yahoo stopped with the panel accounts. Probe accounts are not always representative of what are individual subscribers are seeing in their inbox. Postmaster pages disappeared for a lot of providers and weren’t replaced. Domain reputation got way more important and that means we have to look at increasingly complex mail through many channels to determine what is going on with delivery. We can’t just look at one channel in isolation.

I totally understand why people are upset. This is a big deal given Apple’s dominance in the email client space. Increasingly people are using the Apple mail client and they have the ability to fundamentally change email marketing by changing their mail client behavior. This is hard and scary and challenging. What do we do next? I don’t have any real insightful answers. Some of what we’re going to do is spend a couple days (or weeks for some of us) adapting to the mental shift that we’re losing such a big tool. Then we’re going to adapt.

I think, though, there’s a larger piece to think about here. The adaptation and changes coming to marketing aren’t just about email. There’s a bigger picture around online advertising and marketing and tracking and what it means for individuals. A lot of what the advertising and marketing industry has been doing is really invasive and most people wouldn’t be happy with it if they understood it. They’re starting to figure it out and governments are starting to step in and impose some rules. The power imbalance is changing and marketers are losing many of the things they’ve been taking advantage of. As a consumer, I’m OK with this. As a professional in the space I am thinking about how to adapt without violating consumer privacy.

5 Comments

Domains and Reputation

(Copied and lightly edited from a Facebook post)

It occurred to me as I was commenting elsewhere that there is a lot of confusion about domains / subdomains and where they’re used in emails.
I get confused when people talk about ‘domain reputation’ because I have at least 4 distinct places where domains show up in an email that heavily influence delivery. But a lot of other folks talk about domain reputation as a single thing. I can never work out which place they’re talking about and thus find it difficult to comment on the domain reputation.

Domain Types:

  • SPF domain
  • DKIM domain
  • Visible From domain
  • URL / Image hosting domains

SPF is the Envelope From / Return Path / Bounce domain / 5321.from. The end user does not see this domain unless they go look for it. It is the domain that is checked by SPF and the one that must match the Visible From domain for DMARC to pass. It does not need to be a domain controlled by the sending entity.

DKIM domain is the value in the d= of the DKIM signature. The end user does not see this domain unless they go look for it. This is intended to be a ‘domain that takes responsibility for the email’. This is the one that must match the Visible From domain for DMARC to pass. It does not need to be a domain controlled by the sending entity.

Visible From domain is what most non-email-geek people think of as the From domain. This is what is visible to the end user when they read their mail (assuming their mail client doesn’t hide it like all too many of them do). This is what consumer filters use to help drive delivery to individual user inboxes. This is the domain that is verified by DMARC.

URL / Image hosting domains are in the body of the message. This includes any links to CSS files or outside images (fonts.googleapis.com comes to mind as the big ‘shared’ domain that so much marketing mail uses).

Each of these categories develops reputation individually and then the overall email reputation is determined, in part, by how these reputations interact.

In this case I use ‘domain’ to include disparate subdomains. So I might have an email with the following ‘domains’ in it:

  • SPF Domain: bounce.wttwmail.com
  • DKIM Domain: tr.wttwmail.com
  • Visible From: domain: wordtothewise.com
  • URL / Image domains: wttw.me, image.wttwmail.com, facebook.com, click.wttwmail.com, linkedin.com, font.googleapis.com

Every single one of those domains has their own reputation and the reputation is monitored both individually and in a group.

1 Comment

What’s the best opt-in method?

Kickbox interviewed a bunch of us to find out what methods of opt-in we recommend. Go check it out.

What’s your favourite method of opt-in?

1 Comment

Step by Step guide to fixing Gmail delivery

I regularly see folks asking how to fix their Gmail delivery. This is a perennial question (see my 2019 post and the discussions from various industry experts in the comments). Since that discussion I haven’t seen as much complaining about problems.

There are steps that work to get delivery fixed at Gmail.

  1. Verify that your mail is actually going to bulk. I had one client that had a bad / medium reputation at Google, but their mail was actually inboxing for the most part. We spent a lot of time trying to fix the reputation without success but it didn’t matter as they were reaching the folks they needed to reach. 
  2. Cut way back on your mail to google. Stop sending to anyone who is currently receiving the mail in their bulk folder. About the only way to know who’s getting mail in bulk is to focus on those folks who are opening or clicking on mail. Send only to people who have opened or clicked in the recent past (you can pick the timeline, but I don’t recommend going back more than 90 days for this). Do this for a minimum of a month. 
  3. Monitor both delivery and your reputation. The reputation graphs at google are a lagging indicator and they take between 3 and 4 weeks to reflect changes in behavior. 
  4. If you don’t see improvement: investigate what mail that you don’t know is using your domain and ensure they implement the same level of hygiene. 
  5. If you don’t see improvement still then look at what other mail you are sending to google. There are lots of small domains that use GSuite to host their mail. Mail to those domains does affect reputation. Sometimes there’s enough volume that it breaks remediation and you need to apply the same hygiene to the hosted domains before you get an improvement in delivery.
  6. Once you start to see improvement in inboxing and reputation you can start to re-engage with the addresses that you removed for the reputation repair process. Do not drop them back into the feed all at once, start a warmup process to get you back up to full sends. You may need to permanently remove some unengaged recipients from the list.

It does take time to see improvements reflected in Google Postmaster tools. The good news is that when you’re on the right track mail will start to go to the inbox before you see your reputation improve.

It takes patience to fix delivery at Gmail, but it can be done. Focus on sending mail to the people you know are getting mail in their inbox and who are actively interacting with that mail. Eventually, the ML filters will learn this is wanted mail and know all of your mail should go to the inbox.

No Comments

Current events and filters

That was a longer than intended hiatus from blogging. I’ll be honest, though, talking about email just seemed so trivial in the face of what was and is continuing to happen. I posted this over on slack, earlier, and Steve pointed out I should make it public on the blog. It’s as good a way as any to come back to the blog.

With everything going on in the US, people are applying the brakes to some types of content and speech.  These are not, at the moment, going to be nuanced or careful. They’re trying to stop violence, insurrection and sedition. This is potentially a place of ‘block it all and we’ll sort it out later’

I think folks should expect filters to tighten down on content – particularly political content – in the next few days and lasting for at least a few weeks. I don’t think this will be permanent and I don’t know that it’s going to affect email as much as social media and advertising. But I do think that some email systems will be affected.

There’s also a lot coming out of the martech end of things (look at what Nandini and her crew are coming out with particularly with how much fraud is in the ad networks, see https://branded.substack.com for details and links to news articles). One of the things she isn’t saying, but which is blindingly obvious to me, is a lot of the same people running the fraudulent ad networks are also in email – they’re your affiliate marketers and co-reg vendors. The fallout from the work she and her group are doing will spill into email, too.

No Comments

Cyber Monday

@TwistedDoodles

No Comments

#ltdelivery: Maintaining reputation

At tomorrow’s #ltdelivery session we’ll continue talking about session: Maintaining and warming up reputations.

Invitations are going out end of the day (Dublin) today. Want to join dozens of your colleagues talking about Reputation? Sign up on our #ltdelivery page.

No Comments

The care and feeding of reputation

The next #letstalkdelivery session is Wednesday September 16 at 5pm. Invites went out today so if you signed up for our mailing list, you have the invite in your inbox. OK, if you’re on gmail it went to the promotions tab, but that’s OK, I’m promoting our call.

Check out our schedule and sign up for our mailing list so you don’t miss the next session at our #letstalkdelivery webpage.

No Comments

Lets Talk Delivery

Hope everyone had a good break.

The Let’s Talk Delivery sessions are restarting. I’ve set up a schedule and a page where you can subscribe to invites. Our next session is September 16th and we’re talking Reputation: Warmup, developing and nurturing. We talked a little last week about identity and you can follow along with the notes.

A couple things have changed about how we’re handling the sessions.

We’ve set up a LetsTalk page. There you can see upcoming questions, links to the docs and notes from current and past sessions and sign up to receive invites and notices.

Instead of collecting questions prior to the session on the Google Doc, we’ll collect them on our Facebook page. I’ll open up question pages for each session and you can add questions there.

Invites will still go out on the Monday before the session. We’ll also be opening up the Google doc for questions and feedback during the session.

No Comments

Recent Comments

Archives