Alexandria Online

Saturday, September 20, 2014

What Happens When the Network Goes Down?

Short answer: no internet, no email, no phones, no printing, no clocks, no bells, no public announcement system at pickup time, can't adjust the heating/cooling system, can't change the door lock schedule, can't operate the outdoor sprinkler system on the fields. The water still runs and the lights stay on, but everything else is a yard sale.

We learned a bit later that one of our two, core Cisco 6500 switches had experienced a tiny power failure that partially scrambled its instructions without bringing it to its knees. The two switches are redundant, meaning that if one goes down, the other takes over. The problem is that the one did not completely go down, so the backup did not take over.

The Cisco 6500 is one of the great workhorses of the world. It has been around in some form since 1999, at the core of campus networks all over the world, running day and night directing traffic.

This happened yesterday at Episcopal between 12:30 and 5:00 on a beautiful, mild, Friday afternoon in early Fall. As Director of Technology, I was very stressed out. But I was surprised by how relaxed my colleagues were. In fact, there was a bit of a holiday spirit around school.

When I was sure about what had happened, I scurried over to the head of school's office with my heart pounding. He smiled when he saw me. I said, "the network is down." (I didn't say "sir" but I wanted to.) He said "yes, I know" and laughed. He said, "you probably don't know how long it will take to fix. Good luck."

Later in the day EA's assistant head of school stopped by. She too was in a fine mood. A former network administrator herself, she and I had an in-depth discussion of what we thought had happened and what sort of follow up we would need after the system was repaired. She understood my position better than anyone else on campus and shares my passion for getting things fixed and solving problems. To some extent, this is what we live for: putting out fires.

Many other colleagues smiled at me in the halls and expressed support and sympathy. EA is known for its competitive athletics, and I felt like our network disaster and the ensuing effort to restore service was just another contest, another opportunity for everyone to root for the home team.

I am grateful to the good people I work with for their support and understanding.

Maybe someday the students who attended school yesterday, who couldn't go to their writing lab, watch the video, research their projects, get the handouts, will say to their children, "back when I was in school we lost the internet for a whole day once."

Maybe their children will reply, "what was the internet?"

No comments:

Post a Comment