Call a plumber, quick!
This week was gonna be a tough one, no matter what. My company was launching a major update to one of our systems at The Modesto Bee and so a gaggle of us were on site and were planning to spend the night when…
“Um, where did Fresno go?”
Suddenly all the systems at The Fresno Bee dropped offline… followed by all the Fresno staffers I have in my IM buddy list disappearing. One of our guys was on the phone to someone down there and said, “I heard the fire alarm and then the line went dead.”
Holy crap!
It turns out that a fire sprinkler pretty much blew apart and flooded a small room where all the network and phones came into the building.
I wound up driving down that night because with the network out web updates — my domain — would be awfully tricky and while they would probably have the network back up sometime that night, if they didn’t by the time we knew for sure it would be too late to drive down and implement any wild hair ideas.
When I got there I immediately set up my laptop on the local network and popped in a Verizon EVDO card. Worst case I’d move updates to my laptop from the internal net and then bring up the EVDO card and move the updates manually to the web staging server.
Then I went to check out the damage. There had been 6″ to 8″ inches of water in this tiny little data room and most of the equipment had been either submerged or completely sprayed with this mucky 25-year-old sprinkler water. When I first walked in I didn’t have my camera, which was too bad because there was muck on the walls and guys were basically laying in it trying to get equipment pulled out to dry it off.
Then I sat in a pretty grim meeting of powerful people. The print operation was unaffected as all the internal systems and networks were fine, they’d still put out a paper. The major concern was the phones, the next morning people would be calling to place ads, to make complaints, to do any amount of business all via the phones. For an hour we discussed a variety of ways to get the phone traffic handled in the event they couldn’t get the phone system back up by morning.
Interestingly what was NOT discussed was the myriad of internet feeds than any newspaper has to put stuff on the web in various places including daily news feeds, classified feeds and the like. By exclusion, I got to see what really mattered.
So after the first edition went to bed, I dutifully juggled the data from one machine to another over the EVDO card and as expected 5 minutes later the network came up. While I called it a night, other folks were still at it using hair dryers to dry the remaining equipment.
Amazingly, they had most systems restored by morning, which is a testament to the hard work a lot of people did (but not me, I was in a hotel room working on the aforementioned system upgrade).