Prefetches and Proxies

steve
Technical
August 6, 2024

Jody asks “Are ‘prefetch opens’ and ‘proxy opens’ the same thing?”

Non-human opens

An “open” is just someone (or something) fetching a remote image.

A non-human-interaction (NHI) open is where some sort of automation fetches the image without human interaction – i.e. it’s fetched when the automation feels like it, not triggered by a user opening a message.

Prefetched images

A pre-fetched open is a NHI open where the image is retrieved by some sort of automation prior to the user seeing the email. Describing it as prefetched implies that the users mail client won’t fetch the image again when the mail is read by a human, rather it’ll use that copy of the image it’s already got. (But it’s not always that simple, especially with users with multiple mail clients).

Prefetches provide uncertainty about whether and when the image was seen by a human.

Proxied opens

A proxied open is where the entity that wants an image (maybe a humans mail client, maybe some automation) doesn’t fetch the image directly, rather it asks another system – the proxy – to fetch the image on it’s behalf. That means that the IP address fetching the image will be that of the proxy, and the browser metadata such as user-agent will be that of the proxy.

(The proxy may also modify the image, cache it across multiple users, compress it or do other things – but we don’t care about that so much when we’re talking about opens.)

Proxies provide uncertainty about what IP address a human was using when they saw the email, and what browser or mail client they use (and also eliminate some of the protocol-level tricks that can be used to track users over time and across different websites and mail senders).

Proxy opens are not necessarily NHI – the proxy may be doing work on behalf of a human, or they may be doing it for some automation.

Why not both?

You will often see systems that use both prefetches and proxies. They’re working together to provide uncertainty about whether the image was seen by a human, when it was seen by a human, the IP address (and hence network provider and geographic location) of the user, and any information that could be used to fingerprint and track the user.

All that uncertainty helps to protect the privacy of the user, making it more difficult to profile them and to track their behaviour.

Apple MPP is one of the higher profile mailbox providers that does this, but there are similar elements in other mail systems.

Meaningless metrics

I’ve been having some conversations with fellow delivery folks about metrics and delivery and bad practices. Sometimes, a sender will have what appear to be good metrics, but really aren’t getting them through any good practices. They’re managing to avoid the clear indicators of bad practices (complaints, SBL listings, blocks, etc), but only because the metrics aren’t good.
This made me laugh when a friend posted a link to a Business Insider article about how many website metrics aren’t useful indicators of the business value of a website. Then I found the original blog post referenced in the article: Bullshit Metrics. It’s a great post, you should go read it.
I’d say the concluding paragraph has as much relevance to email marketing as to web marketing.

Clickthrough forensics

When you click on a link in your mail, where does it go? Are you sure?
HTTP Redirects
In most bulk mail sent the links in the mail aren’t the same as the page the recipients browser ends up at when they click on it. Instead, the link in the mail goes to a “click tracker” run by the ESP that records that that recipient clicked on this link in this email, then redirects the recipients web browser to the link the mail’s author wanted. That’s how you get the reports on how many unique users clicked through on a campaign.
In the pay-per-click business that’s often still not the final destination, and the users browser may get redirected through several brokers before ending up at the final destination. I walked through some of this a few years ago, including how to follow link redirection by hand.
HTTP Forensics
Evil spammers sometimes deploy countermeasures against that approach, though – having links that will only work once or twice, or redirects that must be followed within a certain time, or javascript within an intermediate page or any of a bunch of other evasions. For those you need something that behaves more like a web browser.
For serious forensics I might use something like wireshark to passively record all the traffic while I interact with a link from inside a sandboxed browser. That’s not terribly user-friendly to use or set up, though, and usually overkill. It’s simpler and usually good enough to use a proxy to record the web traffic from the browser. There are all sorts of web proxies, used for many different things. What they have in common is that you configure a web browser to talk to a proxy and it’ll send all requests to the proxy instead of to the actual website, allowing the proxy to make any changes it wants as it forwards the requests on and the results back.
For investigating what a browser is doing the most useful proxies are those aimed at either web developers debugging web apps or ~~crackers~~ penetration testers compromising web apps. Some examples are Fiddler (Windows), Cellist (OS X, commercial), mitmdump (OS X, linux, Windows with a little work), Charles (anything, commercial) or ZAP (anything).
I’m going to use mitmdump and Firefox. You don’t want to use your main browser for this, as the proxy will record everything you do in that browser while you have it configured – and I want to keep writing this post in Safari as I work.

When opens hurt reputation

Podia has scraped the Word to the Wise blog and I’m currently receiving an ongoing drip campaign from them absolutely begging me to mention them in my blog post on cold emails.