Happy Fourth
- laura
- Uncategorized
- July 4, 2017
Related Posts
Network glitches and corrupted VMs
I had a bit of a interesting Friday. I was so glad it was finally the weekend. Saturday we did a bunch of errands, including go visit our servers. See, we’ve been upgrading infrastructure to implement a second type of backup system. Saturday we were doing the last set of upgrades so we could install over the weekend.
Yes, we do all our own networking and racking.
Saturday evening Steve is installing the new backup software. This is awesome backup software. It backs up the entire virtual machine. If we lose a virtual machine, we can just reload the entire thing and it will be back again.
Except while installing the software, there is a weird network glitch. Said network glitch caused the system to crash. The system crashes hard. The system crash corrupts some of the data on disk. The data on disk is our virtual machine files. Files are in read only mode and won’t fsck automatically.
We lose most of our production virtual machines. We’re off the air.
Possibly this was tragic, not ironic. I dunno, it’s been a long weekend.
We lost a bunch of production virtual machines to the disc corruption. We haven’t lost any data, but it’s taking some time to rebuild the machines and pull data from the other backup system and get it installed.
That means some of our websites and services, like tools.wordtothewise.com are down. It may mean you saw some bounces if you sent us mail over the weekend. Mail is back and we are communicating with the outside world again.
Steve’s working through our other services as fast as possible to get them back up and running.
(If massive server issues weren’t enough, one of the cats got a UTI so we’re having to pill her twice a day. Then last night managed to puke so hard she passed out briefly. Poor thing. She’s doing better this morning.)
And… we're back from London
The Email Innovations Summit in London was a good conference. Much smaller than Vegas, but with a number of very interesting talks. I got to meet a number of folks I’ve only known online and we had some interesting conversations at the conference and at the pub-track in the evenings.
I had so many grand plans for doing some work while in London. So many plans. And then I actually mostly disconnected and ignored anything I “should” be doing. Instead, Steve and I did some touristing, some relaxing, some family time and some connecting with his college friends. We also (over)heard a lot of conversations about the US Election. One night at dinner every table around us was talking about our candidates and what they thought of them. It’s always interesting to hear what non-Americans think about our country.
In addition to missing two debates, it seems we missed some online news, too. I think the biggest thing was another large DDoS attack against that took out many major websites. I’m starting to see some comments that spam levels were down during the attack, too, but haven’t dug into that yet.
I did have an article published in the Only Influencers newsletter last week: Marketers Can’t Learn from Spam. All too often marketers think spammers are better at unboxing because they see spam in their inbox. But spammers are just more criminal and spend a lot of effort trying to bypass filters. These aren’t lessons marketers can learn from.
Unfortunately, due to our London trip, we are going to miss M3AAWG in Paris, which starts today. Two weeks between conferences was exactly the wrong time for going to both. Never fear, many folks will be tweeting what they can using #m3aawg38.
We’re both slowly getting back into the swing (and timezone!) of back to work. Blogging will pick up over the next few days. And I have new castle pictures to share.
Blogging
It’s been a wild week here in the US. I have to admit, the current political climate is affecting my ability to blog about email. I’ve always said email is not life or death. And how can I focus on the minutia of deliverability when things are in such turmoil and uncertainty? There are many things I want to write about, including some resources for those of us who are struggling with the current administration and changes in the US. What we can do. What we must do. It just takes work and focus I don’t have right now.