Recent Posts

Network glitches and corrupted VMs

I had a bit of a interesting Friday. I was so glad it was finally the weekend. Saturday we did a bunch of errands, including go visit our servers. See, we’ve been upgrading infrastructure to implement a second type of backup system. Saturday we were doing the last set of upgrades so we could install over the weekend.
Yes, we do all our own networking and racking.

Saturday evening Steve is installing the new backup software. This is awesome backup software. It backs up the entire virtual machine. If we lose a virtual machine, we can just reload the entire thing and it will be back again.
Except while installing the software, there is a weird network glitch. Said network glitch caused the system to crash. The system crashes hard. The system crash corrupts some of the data on disk. The data on disk is our virtual machine files. Files are in read only mode and won’t fsck automatically.
We lose most of our production virtual machines. We’re off the air.

Possibly this was tragic, not ironic. I dunno, it’s been a long weekend.
We lost a bunch of production virtual machines to the disc corruption. We haven’t lost any data, but it’s taking some time to rebuild the machines and pull data from the other backup system and get it installed.
That means some of our websites and services, like tools.wordtothewise.com are down. It may mean you saw some bounces if you sent us mail over the weekend. Mail is back and we are communicating with the outside world again.
Steve’s working through our other services as fast as possible to get them back up and running.
(If massive server issues weren’t enough, one of the cats got a UTI so we’re having to pill her twice a day. Then last night managed to puke so hard she passed out briefly. Poor thing. She’s doing better this morning.)

What a week!

Yesterday, after 5pm, I was so happy. I was telling folks to have a great weekend. To take time off and relax. Have fun! Don’t work! Enjoy the weather!
Then someone pointed out it was only Thursday.

But! I got up this morning and got lots of happy Facebook notifications from friends about how TODAY was Friday. I was ready to have an awesome and productive day and go into the weekend with a clean todo list and a well planned next week.
Then I broke my mail client. Trying to add an attachment would crash everything. That got fixed that somewhere around noon.
So! I’ll just grab some lunch and get ready for a productive afternoon!
Then I broke finder.

Yes, that is a picture of my 27 inch monitor with hundreds of Windows opening. I was trying to delete some of the 39,000 .jpgs from my mail client. My finger slipped on the trackpad, though, and instead of “move to trash” I clicked “show in containing folder.” Ooops. I finally crashed finder manually and it restarted and didn’t try and reopen all the windows.
OK. Fine. I’ll go to the bank and pick up mail and drop off tax (ugh, ow) payments.
On the way there, construction screwed up traffic and it took me more than 20 minutes to go 2 miles. (It’s not a safe place to walk, or I would have). On the way back, I went the Other Way. Only to discover a firetruck across 4 lanes of traffic and half a dozen cop cars showing up to a very recent accident.
Then, while writing this blog post I managed to somehow move widgets around and lose them on the wordpress editor.
Apparently I should have taken my friend’s advice and just decided today was not a work day. Because, wow, was it a mess. What all this means is I’m not going to try and blog anything substantial. I’d probably make some total boneheaded mistake and that wouldn’t be any good.
Instead, I will share the song KFOG played every Friday at 5pm (before Cumulus decided to fire everyone). Because I am really in the need of this week to be over.
Have a good weekend. Next week will be better!

A DKIM primer resurrected

I was looking for some references today back in old blog posts. This means I discover some old links are dead, blog posts are gone or moved, and information is lost.
In this case it’s a post by J.D. Falk on deliverability.com. The link is dead (it looks like the whole website is dead), but I found a copy of his post and am reproducing it here. I don’t have permission, because I can’t get permission from him, but the content is extremely useful and I don’t want it lost.

Podbox Expert Interview Series

Last month I did an interview with Podbox about email, deliverability and how I became an email expert (breaking things, lots of breaking things… and having to pick up the pieces and fix them…)
Check out the interview over on their website.

I’ve been thinking a lot about history and longevity. Next year will mark 10 years of the Word to the Wise blog and 20 years of me entering the anti-spam / deliverability space. That’s a lot of time. When I first started fighting spam it was really about my mailbox and getting rid of the junk I was receiving. At the time, a lot of people thought it was silly to spend so much effort fighting spam.
But as time as gone on, email spam and fraud became a big deal. Criminals realized they could use spam to further their gains at the expense of people. Spam is a network problem. Spam is a danger.
Personally, I’ve moved away from fighting spam. I’m now working more on making and keeping email a useful tool. Yes, that does include commercial email. Yes, it does include bulk email. Helping people get the mail they want in their inbox is a part of keeping the email ecosystem healthy. It’s the part I can do and the part I am good at.
Seeing email become such an important part of commerce, communication and modern life has been a journey. I look forward to seeing where the next 20 years takes us.

My panels from #EEC16

I had the privilege to be a part of two panels at EEC16, with some of the best folks in the business.
The first panel was “Everything You Ever Wanted to Know About Deliverability, but Were Afraid to Ask.”
We had a lot of great audience questions.
The first question, which was awesome (and I don’t think planted) was: “What is the most important thing we can do to improve our deliverability?”
All of us had really similar answers: pay attention to your data and your acquisition. Deliverability starts with your data: good data = good deliverability, poor data = poor deliverability. How you acquire addresses is vital to any email program.
I’ve had dozens of sales calls with potential clients over the years. Most of them tell me lots of stuff about their marketing program. I hear details of engagement, data hygiene, response rates, CTRs, bounce handling. But very, very few people spontaneously tell me how they’re acquiring addresses. That’s so backwards. Start with acquiring addresses the right way. Deliverability is all in the acquisition step. Of course, you need to nurture and care for those subscribers, sent the right message at the right time and all the good things we talk about. None of that matters if you don’t start with good data.
Another question was about spamtraps. The panel had me take this one. I’ve written extensively about spamtraps and what they do and what they mean. The important thing to remember, though, is that a spamtrap is a signal. If you have spamtraps on your list, then there is a problem with your data acquisition. Somehow, people are getting addresses that do not belong to them on the list.
Spamtraps are a problem, but not for the reasons many people think they are problems. Folks get upset when their mail is blocked because of spamtraps. Blocking isn’t the only damage, though. For every spamtrap on a list that is one less responsive addresses. It’s one customer who you are not reaching. If there are spamtraps on a list, it’s likely there are deliverable addresses that don’t belong to your customers, too. These recipients are going to view that mail as spam. They didn’t sign up, they didn’t ask for it, they don’t want it. They’re going to complain, hurting your reputation. Too many of these recipients and delivery will suffer.
Spamtraps are a warning that something is wrong. That something is usually your data acquisition process.
Questions went on through the session and ranged from things like how to get mail to B2B inboxes and is there value in certification. We also had some insightful questions about authentication.
The second panel I was on was the closing keynote panel: “ISP Postmasters & Blacklist Operators: Defending Consumer Inboxes.” This was where I got to show my incoming mail chops, a bit. I was a last minute fill in for the panel and I am honored that Dennis and Len thought I could represent the incoming mail folks. It’s not like I’m out there writing filters, but I do pay attention to what the filter operators are saying and doing.
I think it is important for marketers to get a feel for what’s really going on at the ISPs. They aren’t trying to stop real mail, they’re trying to stop malicious mail. Matt from Comcast talked a lot about how marketers and ISPs share customers and the ISPs are trying to keep those customers safe and happy. Jaren discussed some of the decision making processes his company goes through deciding whether to err on the side of letting spam through or filtering good mail. Tom discussed how his blocklist works with some brands to help stop phishing attacks against those brands.
Overall, I think the session was a great success. The conference was great and I am looking forward to going back next year.
Were you at either panel? What did you think?
+eddc

Insight into Gmail filtering

Last week I posted a link to an article discussing how Gmail builds defenses to protect their users from malicious mail. One of the things I found very interesting in that article was the discussion about how Gmail deploys many changes at once, to prevent people from figuring out what the change was.
Let’s take a look at what Gmail said.

Ask Laura: Can you help me understand no auth / no entry?

Dear Laura,
I’m a little confused by the term “no auth / no entry”. Gmail and other major receivers seem to be moving towards requiring authentication before they’ll even consider delivery.
Does this just mean SPF and DKIM, or does this mean the much more stringent DMARC, as well?
Thanks,
No Shirt, No Shoes, No What Now?

Thoughts on filters

One of the questions we received during the EEC16 closing keynote panel was why isn’t there a single blocklist that everyone uses and why don’t ISPs share data more. It would be so much easier for senders if every ISP handled mail the same as every other. But the world isn’t that simple, and it’s not always clear which mail stream is spam and which is good mail.

Dueling data

One of the things I miss about being in science is the regular discussions (sometimes heated) about data and experimental results. To be fair, I get some of that when talking about email stuff with Steve. We each have some strong view points and aren’t afraid to share them with each other and with other people. In fact, one of the things we hear most when meeting folks for the first time is, “I love it when you two disagree with each other on that mailing list!” Both of us have engineering and science backgrounds, so we can argue in that vein.

One of the challenges of seemingly contradictory data is figuring out why it seems to disagree. Of course, in science the first step is always to look at your experimental design and data collection. Did I do the experiment right? (Do it again. Always do it again.) Did I record the data correctly? Is the design right? So what did I do differently from what you did? For instance, at one of my labs we discovered that mixing a reagent in plastic tubes created a different outcome from mixing the reagent in glass vials. So many variables that you don’t even think of being variables that affect the outcome of an experiment.

Thoughts from #EEC16

EEC16 was my first Email Experience conference. I was very impressed. Dennis, Len, and Ryan put together a great program. I made it to two of the keynotes and both took me out of an email focused place to look at the bigger picture.

Patrick Scissons discussed his experiences creating marketing and advertising campaigns for good and to share messages. Some of the campaigns were ones I’d seen as a consumer, or on the news. One of the campaigns he talked about specifically was for the group Moms Demand Action, looking at sensible gun control in the US. The images and symbology used in those campaigns were striking and very effective.
Kelly McEvers talked about her experiences as a correspondent in the middle east during the Arab Spring. She is an engaging speaker, as one who does radio should be. Her overall message and theme was that sometimes events are such that you need to throw the list away and go with it. As someone who lives by “the list” and tries to make sure I’m prepared for every eventuality I found that a very useful message. Particularly when throwing away “the list” turned into some massively successful stories.
In terms of sessions, I found the email content session fascinating. I blogged about content in email last week and did some live tweeting, too. What really hit me after that session was that good marketing drives deliverability. Everything that Carey Kegel was talking about in terms of better marketing, sounded like things I recommend to clients to drive deliverability.
Back in 2012 I was writing posts about how delivery and marketing were somewhat at odds with each other. The premise was that marketing was about creating mindshare, and repeating a message so often a recipient couldn’t forget it. In email, repetition can cause recipient fatigue and drive delivery problems. But what I’m hearing now, from the leading minds of email marketers, is that email marketing works better if you send relevant and useful information to consumers. Recipients are key and you can’t just keep hammering them, you have to provide them with some value.
It seems marketing has finally come around to the delivery point of view.