Related Posts

Network glitches and corrupted VMs

I had a bit of a interesting Friday. I was so glad it was finally the weekend. Saturday we did a bunch of errands, including go visit our servers. See, we’ve been upgrading infrastructure to implement a second type of backup system. Saturday we were doing the last set of upgrades so we could install over the weekend.
Yes, we do all our own networking and racking.
12974536_10206263292444901_7498678361263518784_n
Saturday evening Steve is installing the new backup software. This is awesome backup software. It backs up the entire virtual machine. If we lose a virtual machine, we can just reload the entire thing and it will be back again.
Except while installing the software, there is a weird network glitch. Said network glitch caused the system to crash. The system crashes hard. The system crash corrupts some of the data on disk. The data on disk is our virtual machine files. Files are in read only mode and won’t fsck automatically.
We lose most of our production virtual machines.  We’re off the air.
IronyBlog
Possibly this was tragic, not ironic. I dunno, it’s been a long weekend.
We lost a bunch of production virtual machines to the disc corruption. We haven’t lost any data, but it’s taking some time to rebuild the machines and pull data from the other backup system and get it installed.
That means some of our websites and services, like tools.wordtothewise.com are down. It may mean you saw some bounces if you sent us mail over the weekend. Mail is back and we are communicating with the outside world again.
Steve’s working through our other services as fast as possible to get them back up and running.
(If massive server issues weren’t enough, one of the cats got a UTI so we’re having to pill her twice a day. Then last night managed to puke so hard she passed out briefly. Poor thing. She’s doing better this morning.)

Read More

Phone call of the week

I phoneforblogreceived a message on our 800 number. “This is Mark from a-website.example. Your customers are complaining to me that they are not getting my mail. And you’re blocking mail from me. Explain this to me!”
 
 
I called him back and left a message: “I think you’re confused and I probably can’t help you.”
A few minutes later, Mark returns my call.
L: Hi, this is laura.
M: Who are you? You called me, you must be from Clearwire!
L: No, I’m not with Clearwire, I’m with WttW.
M: Then why is your phone number on the Clearwire website?
L: I don’t know, but this isn’t Clearwire. The Clearwire website is redirecting to Sprint. They got bought out a while ago.
M: Redirecting to Sprint? What does that mean? Your phone number is on Clearwire’s website. You must be with Clearwire.
L: No, really, I’m not.
M: Why is your phone number on their website?
L: I don’t know. But this is not Clearwire. (I start searching the blog because I remember some post somewhere about Clearwire.)
M: Well, who are you?
L: I run a delivery consulting firm. Is it possible you found my website and the blog post that says all clearwire.net addresses are being discontinued April 15, 2015?
M: They’re gone?
L: Yes, for more than a year now.
M: Oh.
scene
That blog post is the #1 google hit if you search for clearwire.net.
 

Read More

Why care about email?

I got my first email address in the very late 80s. I was an intern at a government agency. I learned a lot there: how to sequence DNA, how to handle radioactive material, how to handle human pathogens, and how to send email. I got my first non-work non-school address in the mid-90s. One of the first things I did was join some mailing lists.
One of them was a list for folks who had pet rabbits. I met a lot of people there, both online and in person. As with many people we meet through a shared interest as our interest wanes the relationships change. Some relationships were maintained, but some of us lost touch with one another. Moves, job changes, email address changes, they all affect our ability to maintain relationships online. I kept in touch with some, one was the maid of honor at my wedding and a few years ago I was the maid of honor at hers. I lost track of others.
 

Read More