Recent Posts

BlueHornet spun off from Digital River

Earlier this week, the investment firm Marlin Equity Partners announced they purchased BlueHornet Networks from Digital River. BlueHornet has been around for quite a while. In 2004 they were acquired by Digital River and run as a wholly owned subsidiary.
Congrats to the folks working at BlueHornet.

But my purchased list is TARGETED!!!

listshoppingcart I hear this all the time. But, y’know what? It’s BS. Total BS.
In the last month, I’ve gotten “targeted” messages (that escaped my filters) from the following companies who purchased lists.

Clickthrough forensics

When you click on a link in your mail, where does it go? Are you sure?
HTTP Redirects
In most bulk mail sent the links in the mail aren’t the same as the page the recipients browser ends up at when they click on it. Instead, the link in the mail goes to a “click tracker” run by the ESP that records that that recipient clicked on this link in this email, then redirects the recipients web browser to the link the mail’s author wanted. That’s how you get the reports on how many unique users clicked through on a campaign.
In the pay-per-click business that’s often still not the final destination, and the users browser may get redirected through several brokers before ending up at the final destination. I walked through some of this a few years ago, including how to follow link redirection by hand.
HTTP Forensics
Evil spammers sometimes deploy countermeasures against that approach, though – having links that will only work once or twice, or redirects that must be followed within a certain time, or javascript within an intermediate page or any of a bunch of other evasions. For those you need something that behaves more like a web browser.
For serious forensics I might use something like wireshark to passively record all the traffic while I interact with a link from inside a sandboxed browser. That’s not terribly user-friendly to use or set up, though, and usually overkill. It’s simpler and usually good enough to use a proxy to record the web traffic from the browser. There are all sorts of web proxies, used for many different things. What they have in common is that you configure a web browser to talk to a proxy and it’ll send all requests to the proxy instead of to the actual website, allowing the proxy to make any changes it wants as it forwards the requests on and the results back.
For investigating what a browser is doing the most useful proxies are those aimed at either web developers debugging web apps or ~~crackers~~ penetration testers compromising web apps. Some examples are Fiddler (Windows), Cellist (OS X, commercial), mitmdump (OS X, linux, Windows with a little work), Charles (anything, commercial) or ZAP (anything).
I’m going to use mitmdump and Firefox. You don’t want to use your main browser for this, as the proxy will record everything you do in that browser while you have it configured – and I want to keep writing this post in Safari as I work.

November 2015: The month in email

As we head into the last month of the year, we look back at our November adventures. I spoke twice this month, first at Message Systems Insight in Monterey (my wrap-up post is here) and then with Ken Magill at the at the 2015 All About eMail Virtual Conference & Expo (a short follow-up here, and a longer post on filters that came out of that discussion here.). Both were fun and engaging — it’s always great to get a direct sense of what challenges are hitting people in the email world, and to help clear up myths and misconceptions about what works and doesn’t work in email marketing and delivery. I’m putting together my conference and speaking schedule for 2016 — if you know of anything interesting that should be on my radar, please add it in the comments, thanks!
In industry news, we noted a sharp uptick in CBL listings, and then posted about the explanation for the false positives. Steve wrote about an interesting new Certificate Authority (CA) called Let’s Encrypt, which looks to be a wonderful (and much-needed) alternative for certificates, and I put together some thoughts on SenderScore.
Steve and I did a few posts in parallel this month. First, Steve posted an interesting exercise in SPF debugging. Are you seeing mail from legitimate senders flagged as spam? This might be why. My investigative post was about ISP rejections, and how you can figure out where the block is occurring. In each case, you’ll get a glimpse of how we go about identifying and troubleshooting issues, even when we don’t have much to go on.
We each also wrote a bit about phishing. Steve posted a timely warning about spear phishing — malware attacks disguised as legitimate email from within your organization — and reminds all of us to be careful about attachments. With all of the more secure options for document sharing these days, it’s a lot easier to avoid the risk by maintaining a no-attachments policy in your company. And I wrote about how the Department of Defense breaking HTML links in email to help combat phishing. If your lists include military addresses (.mil), you may want to come up with a strategy for marketing to those recipients that relies less on a clickthrough call to action.
We amused ourselves a bit with a game of Deliverability Bingo, then followed up with a more serious look at the thing we hear all the time — “I’m sure they’ll unblock me if I can just explain my business model.” While an ESP abuse desk is unlikely to be swayed by this strategy, it is actually at the core of how we think about deliverability at Word to the Wise. Legitimate senders have many kinds of lists, many kinds of recipients, many kinds of marketing strategies, and many kinds of business goals. For us to help marketers craft sustainable email programs, we need to understand exactly what matters most to our clients.

CASL botnet take down

biohazardmail The CRTC served its first ever warrant as part of an international botnet takedown. The warrant was to take down a C&C (command and control) server for Win32/Dorkbot. International efforts to take down C&C servers take a lot of effort and work and coordination. I’ve only ever heard stories from folks involved but the scale and work that goes into these take downs is amazing.
Bots are still a problem. Even if we manage to block 99% of the botnet mail out there people are still getting infected. Those infections spread and many of the newer bots steal passwords, banking credentials and other confidential information.
This kind of crime is hard to stop, though, because the internet makes it so easy to live in one country, have a business in a third, have a shell corp in a fourth, and have victims in none of those places. Law enforcement across the globe has had to work together and develop new protocols and new processes to make these kinds of takedowns work.

Looking forward

The nice folks over at Sparkpost asked me and other email experts for some thoughts on what we think the most important issues in email will be in 2016.
I do think security is going to be a major, major change in delivery. From what I’ve seen there’s been a shift in the mindset of a lot of people. Previously a lot of folks in the email space were very accommodating to old systems and unauthenticated mail and were not quite ready to cut off senders that didn’t meet modern standards.

There were a lot of people who didn’t want to take any action that would break email. There are still a lot of people who think that breaking email is a bad thing and changes should be backwards compatible.
Then people started realizing not every change had to be backwards compatible.

There are a few reasons I think this attitude shift happened.

Buying lists costs more than just money

I’ve been talking to a lot of companies recently who are dealing with some major delivery challenges probably related to their practice of purchasing lists and then sending advertising to every address on the list. They assure me that their businesses would be non-viable if they didn’t purchase lists and it has to be that way.
Maybe that’s true, maybe it is more cost effective to purchase lists and send mail to them. I know, though, that their delivery is pretty bad. And that a lot of the addresses they buy never see their email. And that they risk losing their ESP, or they risk being SBLed, or they risk being blocked at Gmail, or they risk bulk foldering at Hotmail. There are a lot of risks to using purchased lists.
The reality is it’s only getting harder to mail to purchased lists and it’s getting more expensive to mail purchased lists. Paying for the list is a small part of the cost of using them.
Other costs incurred by companies using purchased lists include:
1) Having multiple ESPs. There are certainly legitimate reasons for companies to use different ESPs but there is a cost associated with it. Not only do they have to pay for duplicate services, but they spend a lot of employee time moving lists and recipients around to see who might have the better delivery today.
2) Multiple domains and brand new websites for every send. Landing pages are good marketing and are normal. But some ISPs track the IPs of the landing sites, and those IPs can get their own poor reputation. To get around it, senders using purchased lists often have to create new websites on new IPs for every send.
3) Complicated sending schedules. Sending schedules aren’t dictated by internal needs, they’re dictated by what ISP is blocking their IPs or domains (or even ESP) right now.
All of these costs are hidden, though. The only cost on the actual bottom line is the money they spend for the addresses themselves and that’s peanuts. Because, fundamentally, the folks selling addresses have no incentive to take any care in collecting or verifying the data. In fact, any verification they do only cuts into their profit, as buyers won’t actually pay for the verification and data hygiene and it also reduces the size of the lists they can sell.
And, no, data hygiene companies that look for traps and bounces and “bad addresses” don’t take a bad list and make it good. They just take a bad list and make it a little less bad. If the recipients don’t want the mail, all the hygiene in the world isn’t going to get that message into the inbox.
Outsourcing address collection to list selling companies is more expensive than it looks on paper. That doesn’t stop anyone from building a business around purchased lists, though.

Lets Encrypt Everything

Using ~~SSL~~ TLS to protect data in transit and authenticate servers you contacted originally required specialized software, complex configuration and expensive and complicated to require certificates.
The need for specialized software is long since gone. Pretty much every web server and mail server will support SSL out of the box.
Basic server configuration is now pretty simple – give the server a couple of files, one containing the TLS certificate and the other the associated private key. (Configuring a server securely, avoiding a variety of attacks on weak parts of the SSL/TLS protocol, can be a bit more work, but there are a lot of tools and documentation to help with that).
But acquiring a certificate from a reputable provider is still expensive enough (one Certificate Authority’s list price for entry level certificates is $77/year, though the same certificates are available for <$10/year from resellers, which tells you a lot about the SSL certificate market) that you might not want to buy one for every endpoint.
And the process of buying an SSL certificate is horrible.
First you have to find a Certificate Authority or, more likely, a cheap reseller. Then it requires generating a “Certificate Signing Request” – something that can be done in dozens of different ways, depending on the device it’s being generated for. The user-friendliest way I’ve found to do it is to use the openssl commandline tools – and that’s really, really not very friendly.
Then you need to log in to the CA, upload the CSR, follow a bunch of directions to confirm that you are who you say you are and you’re entitled to a certificate for the domain you’re asking for. That can be as simple uploading a file to a webserver that serves the domain you’re getting a certificate for. Or it can turn into an inquisition, where the CA requires your home address and personal phone number before they’ll even talk to you.
Then you need to use the same browser you submitted the CSR request from to get your certificate – as the CA doesn’t do all the cryptography itself, it relies on your web browser for some of it. There are some security-related reasons why they chose to do that, but it makes for a fragile – and near impossible to automate – process. And the attempts to upsell you to worthless “security” products are never-ending.
help
There’s probably more horribleness, but it’s 9 months since I last bought a certificate and I may have forced myself to forget some of the horror.
There’s no need for it to be that painful. It’s easy to mechanically prove to a Certificate Authority that you control a domain in the same way you prove ownership to other companies – put a file the CA gives you onto the webserver, or add a special CNAME to your DNS.
And once you’ve proved you own a domain a certificate for that domain could be generated completely automatically. For the most common web setups you could even prove domain ownership, generate a new certificate and install that certificate into the webserver entirely automatically.
I’m sure there’s some reason other than “because we’re milking the SSL cash cow for all it’s worth” that dealing with most Certificate Authorities is so painful and expensive. But there’s no need for them to be that way.
This year there’ve been several groups who have stepped forward to escape the pain of dealing with legacy Certificate Authorities, at least for basic domain verified TLS certificates (as opposed to the “green bar” extended verification certificates you might want for, e.g. an eCommerce site).
Lets Encrypt has been one of the higher profile new CAs who are driving this effort. With the help of ACME, an open protocol being developed for issuing certificates automatically, they’re providing zero-cost TLS certificates. They’re hoping that, to use their phrase, you’ll encrypt everything, using TLS to encrypt traffic everywhere it makes sense to do so.
So, how does it work?
coyote
It works wonderfully well.
Lets Encrypt is still in public beta testing for a few more weeks, but I just got beta access. Installing their client tool on a reasonably recent Linux box just took copy-pasting two commands and waiting a minute.
The tool supports a variety of different ways to request a certificate – from entirely automated for a vanilla Apache installation through to the most complicated, entirely manual.
To see how difficult it was I decided to install a certificate for blighty.com, a domain on one of our very old webservers – one that was too old to install the letsencrypt tool itself. Fully manual! For a website on a different server!
I ran the letsencrypt tool, telling it I wanted a certificate for blighty.com. It gave me a the contents and location of a file to put on the blighty.com webserver – creating that required copying and pasting two commands from one shell window to another. Then it created and gave me the new certificate and key. I copied those across to the webserver and reloaded it’s configuration. And I have TLS!
This is going to be very simple to completely automate, so you could easily build it in to existing automation to create TLS protected action and tracking links for branded domains. Supporting thousands of TLS protected hostnames wouldn’t be difficult.
And how about using the certificates for things other than webservers? Mail servers, say? You do need a webserver to be live while you’re generating or renewing a certificate – but that’s actually easier to do for a host that doesn’t usually serve web pages than one that does. Once the certificate is generated you can use it for any service, including mail servers.
ACME-movie

What happened with the CBL false listings?

The CBL issued a statement and explanation for the false positives. Copying it here because there doesn’t seem to be a way to link directly to the statement on the CBL front page.

Increase in CBL listings

Update: As of Nov 24, 2015 11:18 Pacific, Spamhaus has rebuilt the zone and removed the broken entries. Expect the new data to propagate in 10 – 15 minutes. Delivery should be back to normal.
The CBL issued a statement, which I reposted for readers that find this post in the future. I think it’s important to remember there is a lot of malicious traffic out there and that malicious traffic affects all of us, even if we never see it.
Original Post from 10am pacific on Nov 24

Mid-morning west coast time, I started seeing an uptick in reports from many ESPs and marketers that they were getting listed on the XBL/CBL. Listings mentioned the kelihos spambot.