Network glitches and corrupted VMs

I had a bit of a interesting Friday. I was so glad it was finally the weekend. Saturday we did a bunch of errands, including go visit our servers. See, we’ve been upgrading infrastructure to implement a second type of backup system. Saturday we were doing the last set of upgrades so we could install over the weekend.
Yes, we do all our own networking and racking.
12974536_10206263292444901_7498678361263518784_n
Saturday evening Steve is installing the new backup software. This is awesome backup software. It backs up the entire virtual machine. If we lose a virtual machine, we can just reload the entire thing and it will be back again.
Except while installing the software, there is a weird network glitch. Said network glitch caused the system to crash. The system crashes hard. The system crash corrupts some of the data on disk. The data on disk is our virtual machine files. Files are in read only mode and won’t fsck automatically.
We lose most of our production virtual machines.  We’re off the air.
IronyBlog
Possibly this was tragic, not ironic. I dunno, it’s been a long weekend.
We lost a bunch of production virtual machines to the disc corruption. We haven’t lost any data, but it’s taking some time to rebuild the machines and pull data from the other backup system and get it installed.
That means some of our websites and services, like tools.wordtothewise.com are down. It may mean you saw some bounces if you sent us mail over the weekend. Mail is back and we are communicating with the outside world again.
Steve’s working through our other services as fast as possible to get them back up and running.
(If massive server issues weren’t enough, one of the cats got a UTI so we’re having to pill her twice a day. Then last night managed to puke so hard she passed out briefly. Poor thing. She’s doing better this morning.)

Related Posts

Spam illustrated

Portraits of Spammers
It’s been a long week, so enjoy some art (and spam). Next week we’ll get back to discussing the many faults of Gmail. And senders. And receivers. And, well, everyone has faults. And email is Dead. Tabs killed it.

Read More

Lavabit shuts down

Lavabit is a secure mail system. Today their CEO announced he was shutting down the service immediately.

Read More

And we're back

Happy New Year!
I am back and ready to talk email with folks.
December is always a busy time, both between the holidays and all associated personal stuff, but also for delivery consulting. There are senders that suddenly discover their email going to the bulk folder and needing help and assistance. But now it’s January and email marketing gets a brief break.
The beginning of the new year and the lull after the Christmas season marketing storm is a good place for folks to think about marketing and email goals for the upcoming years. Many senders get so wrapped up in the day to day details of email that they fail to think strategically about email and their business.
It works much that way for me, as well. I hate it when my clients have bad delivery and do everything I can to fix their problems. If their mail isn’t getting to the inbox, then it’s as much my problem as theirs. I’m thinking and working to get to the root of their problem and come up with solutions to get their mail sent. This sometimes means my own strategic planning gets pushed aside while I focus on client needs. January is a fun time of year for me, because it’s all a little more relaxed and I can look at the new year and how to improve services and share more of my knowledge with folks.
You’ll start to see some of those improvements in the upcoming months. I’ll also be blogging regularly. We should be getting some research and white papers out over the next few months. I’ll be catching up on the Google privacy cases and updating on some other email related lawsuits.
2014 is looking like a year of growth and excitement.

Read More