BLOG

Industry News & Analysis

Gmail tab weirdness

Lots of reports today about mail being delivered to unusual tabs today. Mail that normally goes to promotions is in updates, updates are in the inbox, things like that.

It’s not just you.

Update: The Washington Post reported on the bug.

https://www.washingtonpost.com/technology/2018/12/07/gmail-promotions-bug-reveals-an-algorithmic-mystery/?utm_term=.fadd3485a612

No Comments

Whois silliness from Tucows

In the wake of GDPR, public whois records are 100% redacted. There is lots of work going on to attempt to provide the data without violating privacy laws, but no one is there yet.

This came up because today I got email from Tucows asking  me to verify and, if necessary, update my whois data. Now, Tucows is the registrar, so they know all of the data. But they sent me thisGee, thanks. That’s so helpful.

No Comments

Never 100% inbox

No matter how great an email program deliverability is, no one can guarantee that 100% of the email sent will reach the recipient’s inbox. Why? Recipients can make decisions about where mail goes in their own inbox. Every mail client has a way for users to control where mail is delivered.

This is good for delivery, when the mail means so much to people that they override spam filters and put mail in their inbox. This is problematic for delivery when the final recipient throws mail away or filters it into spam.

Of course, there’s no way to know what individual recipients are doing with mail in general. Sure, there’s currently panel data but that is only for the subset of users that installs a 3rd party app into their mailbox. There’s no way to know where 100% of email is delivered.

For me, I consider any email program with a >95% inbox delivery rate to be an excellent program. I also don’t think there’s much the sender can do in order to get that last 5% to the inbox. That 5% is just not reachable. Not by improving data, not by double opt-in, not by any of the things we do to improve delivery. Some small percentage of mail is just never going to get in front of the user.

The primary reason for these delivery failure is the end user. End users can, and do, create their own filters. While ISPs do curate the inbox, end users have the ability to filter email. If an enduser sets up a filter for a particular email, the ISP isn’t going to overrule that. ISPs want the end users to have a pleasant inbox experience. Thus, the end user’s wants and needs rule. Nothing is clearer to the ISP whether a particular user wants an email than that user directly setting up a filter.

No Comments

Thinking about the concept of best practices

In 2010 Chad White declared best practices dead.

Frankly, the term has always been too “big tent” to be truly useful. When “don’t buy email lists” and “use buttons for primary calls-to-action” are both best practices, it’s no wonder there’s confusion. What we need is new language that differentiates those practices that are a litmus test for legitimate email marketers vs. spammers, from practices that are simply wise.

He went on to define a set of Ethical Imperatives that all legitimate marketers should follow in order to separate themselves from spammers. These imperatives entered around permission and gaining informed consent from recipients. At the time this article was published, I wrote a blog post agreeing with him about how the term best practices is meaningless. This is reflected in my blogging, over the years I’ve written fewer and fewer posts tagged with Best Practices.

Many of the posts I have written about best practices mention permission and then focus on the technical things every mailer should do as a responsible mailer. But even then, much of the time I write about them as minimum practices, not best practices. They are the bare minimum practices that bulk mail should conform to.

But are those really best practices? As I think about it more and more the answer is really no. Best practices are not the minimum practices. They’re actually being thoughtful and doing things right. To me, best practices should take some effort to get right. They need to be more than simply the minimum standard. What do I mean by that? Let’s look at some specific examples.

I state that every email should be authenticated. At a minimum that means either having SPF or DKIM and it doesn’t mention what domains are authenticated. Many small companies using Mailchimp or other mail providers send authenticated mail, but all the mail is authenticated with Mailchimp’s domain(s). A better practice is authenticating with both SPF and DKIM and having the d= point directly to you. A best practice is to have both SPF and DKIM authenticated with your own domains and aligning with the 5322.from.

We can go through the same thought process when discussing permission. Email addresses should be collected with permission. At a minimum that means having an opt-in notice on a website where the address is collected. Better practices are to require the user to actually take an action to opt-in, like checking a box. Best practices are to do a round trip confirmation to link the person giving the email address on the website with the person receiving the email.

Another feature (?) of best practices is that doing them doesn’t necessarily improve deliverability. They’re good things to do, they’re the right thing to do, they’re even the best thing to do. But they’re not always going to get your mail to the inbox. Many of the minimum practices will get >95% of mail to the inbox. Improving practices isn’t necessarily going to get you that last 5%.

Best practices are about going the extra mile to do things correctly, whether or not they actually benefit you.

 

No Comments

How much is too much?

Anecdotally I’m hearing a few different things about recent mail sends.

  1. Multiple ESPs are reporting that their customers, combined, sent more than 2 Billion emails on Friday. SendGrid was close to 3 Billion. Mailchimp was over 2 billion. When all is said and done, I wouldn’t be surprised if the final volume totals topped 20 billion emails in a single day.
  2. One person reported they went to bed with an empty inbox and woke up to almost 400 emails for Black Friday.
  3. Another person reported pages of emails in their promotions tab “too many to read!”
  4. Anecdotally I’m seeing some marketers report lowered open rates compared to normal sends.

Statista says there are somewhere between 230 and 250 million Americans that use email. Taking the higher figure, and a conservative 15 billion emails sent on black friday, that’s 60 emails per person. Conservatively, every US consumer received 60 emails last Friday.

Back in the dark ages, when we were trying to convince people spam was a problem, one of the arguments was that mail was cheap and spammers would send more mail than users could handle. The solution, as we saw it, was to turn email into a permission based system. If people only received mail they asked for, then they wouldn’t be overwhelmed with volume. Now, we’ve reached the point where a single day’s permission based email is overwhelming recipients to the point where they can’t read all of them.

Does that mean I think Black Friday mail is spam? Absolutely not. It was all wanted and asked for mail. But, when a recipient is getting 50 or 100 or 300 or 500 emails in their inbox in a day it takes time to read all of it. They may be excited the first few dozen emails, but as the day goes on

I’m not surprised that some marketers are seeing a lowering of open rates when inboxes are so crowded. I expect that what happened is recipients actually opened more email than their usual amount. But due to the significant increases in volume, each individual sender saw lower open rates.

It seems there’s no limit to the amount of email retailers can send. Is there a limit to the amount of email recipients can read?

 

 

2 Comments

Successful sends on Black Friday

Last year a number of ISPs mentioned the Black Friday email volume was congesting their systems and causing delays. While anecdotally it seems that volume is up over last year I also haven’t heard any ISPs talking about congestion. Likewise, most of the delivery folks I’ve spoken too today and over the weekend are saying there were no major problems.

How’d the busiest email weekend of the year work for all of you?

No Comments

Ready. Set…

No Comments

Send Actual SMTP

It’s rare I find mail that violates the SMTP spec (rfc5321 and rfc5322). I’ve even considered removing “send mail from a correctly configured mail server” from my standard Best Practices litany.

But today I got mail asking me to respond to a survey.

This whole email is a mess of problems, and it’s claiming to be from the California Secretary of State.  It’s also discussing the June Primary, which isn’t the election we just had. The from address doesn’t reassure me, they’re claiming to be: VotersChoice.SoS.Ca.Gov@mailservices6.com. The mail is being sent to the address I gave California when I registered as an overseas voter, but those lists are public.

In the course of trying to decide if this was real or was just some way to steal private information, I discovered this particular mail server isn’t actually sending real SMTP.

Now, quite honestly, I suspect this is actually legitimate mail. A few google searches and I discover mailservices6.com belongs to California Survey Research Services, Inc. They manage data collection for a lot of different government agencies. Looking at information around them this is exactly the kind of vendor that I expect a government agency to use.

I have to wonder, though, how well their email surveys actually perform. They’re not sending actual SMTP. The non-ASCII character is in their own internal handoff to a server running an obsolete version of Sendmail. While our mail server is somewhat forgiving of non-SMTP mail not all mail servers are. Even if that isn’t enough to tank their delivery, there are multiple similar but not identical domain names in the body of the message. The link to ‘research.net’ doesn’t actually go to research.net, it points to yet another random domain name. Put all this together with the unsolicited nature of the email I’d be amazed if any of their mail was reaching the inbox at the consumer ISPs.

Looks like I’ll be keeping the “and make sure you send SMTP” in the list of recommendations, because there are still groups out there who are not sending valid SMTP. If my mail is to be believed, some of them are being paid by the state of California to do so.

No Comments

Email addiction survey

The great folks over at Zettasphere and Emailmonday have released their Email Addiction Survey. Nothing surprising in the data that I can see, although I suspect one particular data point is going to surprise folks.

Yup, more than 70% of people don’t really care about a do not reply address in a message. Honestly, I’m not surprised. Most users don’t really care. In all honesty it probably doesn’t affect delivery that much any longer, either. I still think it’s generally a bad idea but as long as you’re providing a communications channel for recipients to connect with your company it’s probably not going to be a giant problem.

 

No Comments

Why do my URLs have two dots?

You take a turn, I take a turn

At the SMTP level email is very much a simple line-by-line text based protocol. The client sends a command on a single line, the server responds with one or more lines (the last one marked by having a space in the fourth column), and then the client sends another command.

The main exception to that is when the client sends the payload of the email. Once the server has said it’s ready to receive the email the client sends the whole thing, then tells the server it’s finished by sending a line consisting of just a single period “.”.

But what if you have a line in your email that’s just a single period? The developers of SMTP thought of that! Whenever the SMTP client sees a line that begins with a period, it has to add another one. And whenever the SMTP server sees a line that begins with a period, it has to remove it.

This is called “dot-stuffing”. If an email has the line “.”, it’ll be sent on the wire as “..”. Email with the line “.yakko.wakko.dot.” will be sent as “..yakko.wakko.dot.”.

If everything works properly, nobody will ever notice the dot-stuffing. Often it doesn’t, though. One way to break things is for a sender to use a message composition API that produces mail that’s been dot-stuffed and is ready to send, then to send it via an API that expects non-dot-stuffed content, and will stuff it again before sending. Another is for the receiving server or intermediate server to just not do un-stuffing properly.

Understuffed, overstuffed, wombling free

The symptoms of this vary, depending on whether the mail is over-stuffed or under-stuffed. If it’s not stuffed enough, and there’s a line in the email which consists of a single dot then the receiving server will see that line as the end of the message, and accept the (truncated) message for delivery at that point – and then throw an error for every line the client sends after that, as it’s expecting SMTP keywords, not a message body.

If it’s over-stuffed then the problem is a bit less obvious. Any line that begins with a single dot when it’s sent will begin with two dots when it’s received.

You might think that it’d be pretty rare for a line to begin with a dot, and even rarer for anyone to notice a problem if that dot is doubled. But there’s one case where it’s quite likely.

Quoted-Printable encoded HTML

HTML sent quoted-printable encoded is the normal format for commercial emails. The HTML gives us all the rich content we like, while the quoted-printable encoding lets us not have to worry so much about sending non-ascii characters or violating SMTP specs by accidentally sending lines longer than 998 characters.

Quoted-printable encoding is pretty simple. Some characters are encoded as three ascii characters – an equals sign, followed by two hex digits representing the character. So “=” is quoted-printable encoded as “=3D”. And long lines are wrapped at 76 characters, with an = sign to mark the wrapping. There are a few other rules, but that’s about it.

(We have a tool that’ll decode quoted-printable, if you ever need that, on tools.wordtothewise.com).

If you have a line that looks like …

about forty-five characters of text <a href=”http://whatever.example.com”>click here</a>

… it will quoted-printable encode to look like …

about forty-five characters of text  <a href=3D”http://whatever.example=
.com”>click here</a>

… then overstuffing will mean it ends up at the recipient as …

about forty-five characters of text  <a href=3D”http://whatever.example=
..com”>click here</a>

… and the mail client will quoted-printable decode it to be …

about forty-five characters of text <a href=”http://whatever.example..com”>click here</a>

That looks identical, but the link won’t work because of the doubled dot in the URL.

The symptoms of this will be that some links in the mail you send won’t work when it’s received. If the stuffing bug is at your end then it’ll mean that some links, depending on the content and layout of your email, won’t work. If the stuffing bug is at the recipient mailserver or a forwarding middlebox it means that some links, depending on the content and layout of your email, won’t work for some recipients.

Mitigation

If your mail generation and delivery pipeline isn’t handling dot-stuffing correctly you should fix that. You can test it by sending yourself a plain text email with a line that starts with a dot – if it’s doubled, there’s a problem.

If the issue is at the receiver it’s harder to identify that it’s happening, let alone get it fixed. One way to mitigate in that case would be to configure your quoted-printable encoder to encode “.” as “=2E”. That way no raw “.” will appear in your messages, so there’s no way it can get overstuffed.

No Comments