Recent Posts

Yahoo trying to cope with misdirected email

Techcrunch says Yahoo is announcing a new “this is not me” button for email sent to recovered addresses.

Recycled Yahoo addresses and PII leaks

Infoweek interviewed a number of people who acquired new Yahoo addresses during Yahoo’s address recycling and reuse process. It seems that at least for some small percentage of former Yahoo users, there is a major risk of information going to the wrong people.

Does mail volume contribute to blocking?

There are two extreme opinions I see among marketing agencies and email senders when it comes to volume.
One group seems to think that volume alone triggers blocks. Another group thinks volume never affects delivery.
As with many things in delivery reality is at neither extreme.
Sending lots of mail isn’t the problem. Sending lots of mail your recipients aren’t interested in getting is the problem. Last year during the US political elections the Obama campaign, for instance, sent lots and lots of mails. Their list was an order of magnitude larger than the Romney campaign and there were days they were sending 10s of mails per subscriber. It was a deluge. But they were smart, and they did a lot of data mining and they did it in a way that got recipients to act on the mail. That mail was a deluge, but it was a wanted deluge by most of the receivers.
For a lot of vendors, too, increasing volume does increase response and revenue and all the things you want to drive with email marketing. But there will be people who don’t like the increase in volume. If they’re not valuable customers, no great loss. If they are valuable customers, then the increase in volume may drive a decrease in revenue.
In terms of inbox delivery, it’s not the volume it’s how wanted the mail is. Send wanted, interesting and engaging mail, you can send dozens of times a day.
No, volume alone doesn’t contribute to delivery problems.

Ad-hoc analysis

I often pull emails into a database to analyze them, but sometimes I want something simpler. Emails are typically stored in one of two ways: mbox format, where an entire mailbox is stored in a single file, and maildir format, where a mailbox is a directory with one file in it for each email.
My desktop mail application is Mail.app on OS X, and it stores messages in a maildir-ish format, so I’m going to work with that here. If you’re using mbox format mailboxes it’s a little trickier (but you can use a tool called formmail to split an mbox style format into a maildir directory and go from there).
I want to gather some statistics on mail I’ve sent to abuse desks, so the first thing I do is open up a terminal window and change directory to where my “Sent Messages” mailbox is:
cd Library/Mail/V2/IMAP-steve@misc.wordtothewise.com/Sent Messages.mbox
(Tab completion is really useful for navigating through the mailbox hierarchy.)
Then I need to go through every email (file) in that directory, for each file find the “To:” header and check to see if it was sent to an abuse desk. If it was sent to an abuse desk I want to find the email address for each one, count how many times I see that email address and find the top twenty or so abuse desks I send reports to. I can do all that with a single command line:
find . -type f -exec egrep -m1 '^To:' {} ; | egrep -o 'abuse@[a-zA-Z0-9._-]+' | sort | uniq -c | sort -nr | head -20
(Enter that all as a single line, even though it’s wrapped into two here).
That’s a bit much to understand all at once, so lets redo that in several stages, with an intermediate file so we can see what’s going on.
find . -type f -exec egrep -m1 '^To:' {} ; >tolines.txt
The find command finds all the files in a directory and does something with them. In this case we start looking in the current directory (“.”), look just for files (“-type f”) and for each file we find we run that file through another command (“-exec egrep -m1 ‘^To:’ {} ;”) and write the result of that command to a file (“>tolines.txt”). The egrep command we run for each file goes through the file and prints out the first (“-m1”) line it finds that begins with “To:” (“‘^To:'”). If you run that and take a look at the file it creates you can see one line for each message, containing the “To:” header (or at least the first line of it).
The next thing to do is to go through that and pull out just the email addresses – and just the ones that are sent to abuse desks:
egrep -o 'abuse@[a-zA-Z0-9._-]+' tolines.txt
This uses egrep a second time, this time to look for lines that look like an email address (“‘abuse@[a-zA-Z0-9._-]+'”) and when it finds one print out just the part of the line that matched the pattern (“-o”).
Running that gives us one line of output for each email we’re interested in, containing the address it was sent to. Next we want to count how many times we see each one. There’s a command line idiom for that:
egrep -o 'abuse@[a-zA-Z0-9._-]+' tolines.txt | sort | uniq -c
This takes all the lines and sorts (“sort”, reasonably enough) them – so that identical lines will be next to each other – then counts runs of identical lines (“uniq -c”). We’re nearly there – the result of this is a count and an email address on each line. We just need to find the top 20:
egrep -o 'abuse@[a-zA-Z0-9._-]+' tolines.txt | sort | uniq -c | sort -nr | head -20
Each line begins with the count, so we can use sort again, this time telling it to sort by number, high to low (“sort -nr”). Finally, “head -20” will print just the first 20 lines of the result.
The final result is this:

SpamArrest Loses in Court

Internet law expert Eric Goldman points out that winning anti-spam lawsuits is hard. SpamArrest just learned that the hard way, he explains. If you weren’t aware, SpamArrest (whose website proclaims “SPAM ARREST WORKS!”) is a vendor of a Challenge/Response-based anti-spam filtering system. The way that works is, if you’re using a C/R-based system, any time somebody sends you an email, the system sends the person back a “challenge” email that the sender must now respond to, usually by clicking on a link. By engaging in this “response,” the sender is proving that they’re not a robot. The theory being that by doing so, the sender must not be a spammer. It’s a flawed system, for multiple reasons. First, the internet is global, and it’s possible for bad guys to very cheaply hire people from a far away land to click these links all day long, every day. Even worse, legitimate senders aren’t going to take the time to bother to click through these links; they’re going to write it off as too time intensive. Do you really think Amazon is going to bother responding to challenge requests, to push through your shipping notification? For years, I’ve been telling senders to ignore C/R challenge emails, because it’s a self resolving problem, those people don’t want to receive emails, so let those people not get their emails. Even worse than that, those of us who actually care about the email ecosystem find C/R abhorrent because of its inherent backscatter problem. Spammers forge sending addresses. C/R systems send challenge emails back to those forged sending addresses. Thus, unrelated people often receive C/R challenge emails, when they didn’t even initiate the original message. It doesn’t solve the spam problem; it just exchanges spam mail for misdirected junk mail. I’m not a fan of SpamArrest, but I’m also not a fan of anything that makes it hard to use legal means to go after people sending unsolicited email. So my emotions are mixed on this one. Regardless, I wouldn’t be happy if I were one of the 600 SpamArrest users who received the alleged spam message in question. And with regard to the rest of their users, I worry that bad guys will now interpret the court’s ruling as making it acceptable to set up “C/R approval farms” and respond to every challenge message received. This would seriously undermine both SpamArrest’s business strategy and anti-spam strategy. And to the rest of the anti-spam community, allow me to echo something Eric says in his article: Anti-spammers don’t win in court just by showing up.

Things real people don't say about advertising

My favorite is about the call to action, but it’s probably not quite safe for work.
Sadly, I do sometimes try and figure out what segment I’m in that caused me to get a particular ad.

ISP Relationships

Delivra has a new whitepaper written by Ken Magill talking about the value (or lack thereof) of relationships with ISPs. In Ken’s understated way, he calls baloney on ESPs that claim they have great delivery because they have good relationships with ISPs.
He’s right.
I get a lot of calls from potential clients and some calls from current clients asking me if I can contact an ISP on their behalf and “tell the ISP we’re really not a spammer”. My normal answer is that I can, but that there isn’t a place in the spam filtering process for “sender has hired Laura and she says they’re not a spammer.” I mean, it would be totally awesome if that was the case. But it’s not. It’s even the case where I’m close friends with folks inside the ISPs.
I’m pretty sure I’ve told the story before about being at a party with one of the Hotmail ISP folks. There was a sender that had hired me to deal with some Hotmail issues and I’d been working with Barry H. (name changed, and he’s not at Hotmail any more) to resolve it. During the course of the party, we started talking shop. Barry told me that he was sure that my client was sending opt-in mail, but that his users were not reacting well for it. He also told me there was no way he could override the filters because there wasn’t really a place for him to interfere in the filtering.
Even when folks inside the ISPs were willing and able to help me, they usually wouldn’t do so just because I asked. They might look at a sender on my request, but they wouldn’t adjust filters unless the sender met their standards.
These days? ISPs are cutting their non-income producing departments to the bone, and “sender services” is high up the list of departments to cut. Most of the folks I know have moved on from the ISP to the ESP side. Ken mentions one ISP rep that is now working for a sender. I actually know of 3, and those are just employees from the top few ISPs who are now at fairly major ESPs. I’m sure there are a lot more than that.
The reality is, you can have the best relationships in the world with ISPs, but that won’t get bad mail into the inbox. Filters don’t work that way anymore. That doesn’t mean relationships are useless, though. Having relationships at ISPs can get information that can shorten the process of fixing the issue. If an ISP says “you are blocked because you’re hitting spam traps” then we do data hygiene. If the ISP says “you’re sending mail linking to a blocked website” then we stop linking to that website.
I have a very minor quibble with one thing Ken said, though. He says “no one has a relationship with Spamhaus volunteer, they’re all anonymous.” This isn’t exactly true. Spamhaus volunteers do reveal themselves. Some of them go around openly at MAAWG with nametags and affiliations. A couple of them are colleagues from my early MAPS days. Other do keep their identities secret, but will reveal them to people they trust to keep those identities secret. Or who they think have already figured it out. There was one drunken evening at MAAWG where the nice gentleman I was joking with leaned over and says “You know I am elided from Spamhaus, right?” Uh. No? I didn’t. I do now!
But even though I have the semi-mythical personal relationship with folks from Spamhaus, it doesn’t mean my clients get preferential treatment. My clients get good advice, because I know what Spamhaus is looking for and can translate their requirements into solid action steps for the client to perform. But I can think of half a dozen ESP delivery folks that have the same sorts of relationships with Spamhaus volunteers.
Overall, relationships are valuable, but they are not sufficient to fix inbox delivery problems.

Questions on Google lawsuit post

A couple questions in the previous discussion thread about the Google privacy case. Both concern permission granted to Google to scan emails.
Google’s stance about this is fairly simple.
Gmail users give explicit permission for their mail to be scanned.
People who send mail to Gmail users give implicit permission for their mail to be scanned.
The plaintiff’s lawyers are alleging that some subset of gmail users – specifically those at Universities that use Google apps and ISP customers like CableOne – did not give explicit permission for their mail to be scanned by Google. They’re also arguing no senders give permission.
In addition to the lack of permission, the plaintiffs lawyers are arguing that Google’s behaviour is in violation of Google’s own policies.
Google thinks scanning is part of the ordinary course of business and they’re doing nothing wrong.
This is an interesting case. I think anyone who knows about email understands that the people who run the mail server have the ability to read anything that goes through. But a lot of us trust that most postmaster and admin types consider it unprofessional to look at mail without a decent reason. There are good reasons an admin might need to go into a mail spool.
Automated filtering is simply a part of life on the internet these days. Mails have to be scanned for viruses, spam and, yes, they are scanned for targeted advertising. I’m not convinced Google is outside the norm when they say that any emails sent through Google is personal information given too Google and therefore Google can use that information in accordance with their policies.

No expectation of privacy, says Google

I spent yesterday afternoon in Judge Koh’s courtroom listening to arguments on whether or not the class action suit against Google based on their scanning of emails for advertising purposes can go forward. This is the case that made news a few weeks ago because Google stated in their brief that users have “no expectation of privacy” in using online services.
That does appear to be what Google is actually saying, based on the arguments by attorney Whitty Somvichian. He made it clear that Google considers everything that passes through their servers, including the content of emails, covered under “information provided to Google” in the privacy policy. Google is arguing that they can read, scan, and use that content to display ads and anything else they consider to be in the normal course of business.
I have pages and pages of notes but I have some paying work to finish before I can focus on writing up the case. There were multiple reporters and bloggers in the courtroom, but I’ve not found many article. Some I’ve found are:

Patent trolling, meet RPost

Yesterday I mentioned Ubicomm and their patent trolling based on an ancient Xerox patent they acquired earlier this year. I think the mere fact that Xerox sold the patent says all we need to know about how applicable it is.
The other patent troll in the email space right now is RPost. Steve did a blog post about RPost patent trolling about a year ago.
This summer, RPost’s legal team started calling different companies in the email space. I got a call the first week in July. After introducing himself as their lawyer and reassuring me he was not sending me legal threats, he started to ask all sorts of questions about our technology. I declined to answer any of them.
The lawyer then said he had some paperwork to send me and asked for an email address. I told him we do not accept legal service by email and that he could send me any relevant paperwork to our address of record. If I had any questions about RPost having a real product, it was answered when the lawyer didn’t tell me that RPost technology is all about secure delivery of legal papers.
Others in the email space started reporting similar calls and letters from RPost around the same time.
It’s been 2 months (almost to the day) since RPost’s lawyer called me and we have yet to receive anything from them. Clients of mine, however, have received papers from RPost. The papers instruct recipients to read RPost’s patents and notify RPost if they are infringing.
Yes, RPost are such cheapskates they expect their target companies to do the work identifying any potential infringement. Or possibly it’s just that they have so little money they can’t afford to pay their legal team. Certainly my experience is that telling them to send us postal mail is enough expense? time? to stop them from moving forward.
My recommendations to anyone receiving a letter from RPost (or anyone else claiming patent infringement) are pretty simple.