Dodgy PDF handling at Gmail

We sent out some W-9s this week. For non-Americans and those lucky enough not to have to deal with IRS paperwork those are tax forms.
They’re simple single page forms with the company name, address and tax ID numbers on them. Because this is the 21st Century we don’t fill them in with typewriters and snail mail them out, we fill in a form online at the IRS website which gives us PDFs to download that we then send out via email.

We started to get replies from people we’d sent them to that we hadn’t included the tax ID number. Which was odd, because it was definitely there in the PDFs we’d sent.
The reports of missing numbers came from Google Apps users, so we sent a copy to one of our Gmail addresses to see. Sure enough, when you click on the attachment it’s mostly there, but some of the digits of the tax ID number are missing.

And all the spaces have been stripped from our address.

The rest of the form looked fine, but the information we’d entered was scrambled. Downloading the PDF from Gmail and displaying it – everything is there, and in the right place.
Weird. After a brief “Are gmail hiding things that look like social security numbers?” detour I realized that the IRS website was probably generating the customized forms using PDF annotations.
PDF is a very powerful, but very complex, file format. It’s not just an image, it’s a combination of different elements – images, lines, vector artwork, text, interactive forms, all sorts of things – bundled together into a single file. And you can add elements to an existing PDF file to, for example, overlay text on to it. These “annotations” are a common way to fill in a PDF form, by adding text in the right place over the top of an existing template PDF.
I cracked the PDF open with some forensics tools and sure enough, the IRS had generated the PDF form using annotations.
 

<< /Type /Annot /DV (Palo Alto, CA) /T (topmostSubform[0].Page1[0].Address[0].f1_8[0])
/Rect [ 57.6 539.968 388.8 553.969 ] /AP 81 0 R /FT /Tx /DA (/Helvetica-Bold 9 Tf 0 g)

And the Gmail PDF viewer isn’t rendering that annotated text correctly.
I’ve filed a bug sent feedback to Google, so hopefully it’ll be fixed. Meanwhile, if you’re sending customized content to recipients using PDF you should probably check that it renders correctly when previewed in Gmail.
 

Related Posts

Google Postmaster Tools: Last Chance!

I’ll be closing down the Google Postmaster Tools survey Oct 31. If you’ve not had a chance to answer the questions yet, you have through tomorrow.
This data will be shared here. The ulterior motive is to convince Google to make an API available soon due to popular demand.

Read More

Google makes connections

One of the client projects I’m working on includes doing a lot of research on MXs, including some classification work. Part of the work involves identifying the company running the MX. Many of the times this is obvious; mail.protection.outlook.com is office365, for instance.

There are other cases where the connection between the MX and the host company is not as obvious. That’s where google comes into play. Take the domain canit.ca, it’s a MX for quite a few domains in this data set. Step one is to visit the website, but there’s no website there. Step 2 is drop the domain into google, who tells me it’s Roaring Penguin software.
In some cases, though, the domain wasn’t as obvious as the Roaring Penguin link. In those cases, Google would present me with seemingly irrelevant hosting pages. It didn’t make sense until I started digging through hosting documentation. Inevitably, whenever Google gave me results that didn’t make sense, they were right. The links were often buried in knowledge base pages telling users how to configure their setup and mentioning the domain I was searching for.
The interesting piece was that often it was the top level domain, not the support pages, that Google presented to me. I had to go find the actual pages. Based on that bit of research, it appears that Google has a comprehensive map of what domains are related to each other.
This is something we see in their handling of email as well. Gmail regularly makes connections between domains that senders don’t expect. I’ve been speaking for a while about how Gmail does this, based on observation of filtering behavior. Working through multiple searches looking at domain names was the first time I saw evidence of the connections I suspected. Gmail is able to connect seemingly disparate hostnames and relate them to one another.
For senders, it means that using different domains in an attempt to isolate different mainstreams doesn’t work. Gmail understands that domainA in acquisition mail is also the same as domainB in opt-in mail is the same as domainC in transactional mail. Companies can develop a reputation at Google which affects all email, not just a particular mail stream. This makes it harder for senders to compartmentalize their sends and requires compliance throughout the organization.
Acquisition programs do hurt all mail programs, at least at Gmail.
 

Read More

Gmail survey rough analysis

I closed the Google Postmaster Tools (GPT) survey earlier today. I received 160 responses, mostly from the link published here on the blog and in the M3AAWG Senders group.
I’ll be putting a full analysis together over the next couple weeks, but thought I’d give everyone a quick preview / data dump based on the analysis and graphs SurveyMonkey makes available in their analysis.
Of 160 respondents, 154 are currently using GPT. Some of the folks who said they didn’t have a GPT account also said they logged into it at least once a day, so clearly I have some data cleanup to do.
57% of respondents monitored customer domains. 79% monitored their own domains.
45% of respondents logged in at least once a day to check. Around 40% of respondents check IP and/or domain reputation daily. Around 25% of respondents use the authentication, encryption and delivery errors pages for troubleshooting.
10% said the pages were very easy to understand. 46% said they’re “somewhat easy” to understand.
The improvements suggestions are text based, but SurveyMonkey helpfully puts them together into a word cloud. It’s about what I expected. But I’ll dig into that data. 
10% of respondents said they had built tools to scrape the page. 50% said they hadn’t but would like to.
In terms of the problems they have with the 82% of people said they want to be able to create alerts, 60% said they want to add the data to dashboards or reporting tools.

97% of respondents who currently have a Google Postmater Tools account said they are interested in an API for the data. I’m sure the 4 who aren’t interested won’t care if there is one.
47% of respondents said if there was an API they’d have tools using it by the end of 2017. 73% said they’d have tools built by end of Q1 2018.
33% of respondents send more than 10 million emails per day.
75% of respondents work for private companies.
70% of respondents work for ESPs. 10% work for retailers or brands sending through their own infrastructure.
That’s my initial pass through the data. I’ll put together something a bit more coherent and some more useful analysis in the coming week and publish it. I am already seeing some interesting correlations I can do to get useful info out.
Thank you to everyone who participated! This is interesting data that I will be passing along to Google. Rough mental calculation indicates that respondents are responsible for multiple billions of emails a day.
Thanks!

Read More