Alexandria Online

Sunday, September 21, 2014

The Devil Needs a Database

In this sentence from Annie Proulx's short story, "I've Always Loved This Place," the devil complains because he has no idea whether the famous painter Signorelli, who did such a lovely job painting The Last Judgment, resides in Hell or not:

"They had to start compiling a database of the damned and their particular niches; it was impossible to find anyone in Hell."

Even the devil, whose very mojo is chaos, needs a database.

But I think the devil, or perhaps Proulx, has it wrong. What the devil really needs is a couple of good Cisco certified internet engineers who can give him the kind of bandwidth and rock-solid wifi that the blessed would enjoy in heaven. That way he can google Signorelli and anyone else he's trying to find in hell.

But good luck finding a top notch network guy in hell. All the ones I've worked with are obviously going to heaven for all their good deeds.

Saturday, September 20, 2014

What Happens When the Network Goes Down?

Short answer: no internet, no email, no phones, no printing, no clocks, no bells, no public announcement system at pickup time, can't adjust the heating/cooling system, can't change the door lock schedule, can't operate the outdoor sprinkler system on the fields. The water still runs and the lights stay on, but everything else is a yard sale.

We learned a bit later that one of our two, core Cisco 6500 switches had experienced a tiny power failure that partially scrambled its instructions without bringing it to its knees. The two switches are redundant, meaning that if one goes down, the other takes over. The problem is that the one did not completely go down, so the backup did not take over.

The Cisco 6500 is one of the great workhorses of the world. It has been around in some form since 1999, at the core of campus networks all over the world, running day and night directing traffic.

This happened yesterday at Episcopal between 12:30 and 5:00 on a beautiful, mild, Friday afternoon in early Fall. As Director of Technology, I was very stressed out. But I was surprised by how relaxed my colleagues were. In fact, there was a bit of a holiday spirit around school.

When I was sure about what had happened, I scurried over to the head of school's office with my heart pounding. He smiled when he saw me. I said, "the network is down." (I didn't say "sir" but I wanted to.) He said "yes, I know" and laughed. He said, "you probably don't know how long it will take to fix. Good luck."

Later in the day EA's assistant head of school stopped by. She too was in a fine mood. A former network administrator herself, she and I had an in-depth discussion of what we thought had happened and what sort of follow up we would need after the system was repaired. She understood my position better than anyone else on campus and shares my passion for getting things fixed and solving problems. To some extent, this is what we live for: putting out fires.

Many other colleagues smiled at me in the halls and expressed support and sympathy. EA is known for its competitive athletics, and I felt like our network disaster and the ensuing effort to restore service was just another contest, another opportunity for everyone to root for the home team.

I am grateful to the good people I work with for their support and understanding.

Maybe someday the students who attended school yesterday, who couldn't go to their writing lab, watch the video, research their projects, get the handouts, will say to their children, "back when I was in school we lost the internet for a whole day once."

Maybe their children will reply, "what was the internet?"