Sunday, November 19, 2006

Broke Down, Busted, Brutal Week

I'm submitting a timesheet for 64.5 hours tomorrow. I don't often get overtime, and when I do, it's generally 3 or 4 hours in a week. A couple of times I've submitted 50 hour timesheets, but this week I worked over 1.5 weeks in 6 days.

My Life in Hell started Tuesday night, with a fun-filled evening of backing up 162GB of data, replacing one rack of old & busted servers with a new rack of shiny new servers, and moving an extremely heavy RAID system. I was at work until 1am, and when I tried to leave work, I learned that my car (the Miata) was busted, too. I couldn't engage any gear with the car running, and the car wouldn't start with a gear engaged. The problem? A leaky clutch slave cylinder. I had a coworker help push me out of my parking spot, then push my car to a running start, whereupon I jumped in and wrangled it into gear. I couldn't bring the car back to a complete stop without killing the engine, which would mean getting it back to a running start, so I had to run a red-light on my way home.

Fortunately, the Miata's transmission is very amenable to clutch-less shifting, so I got home quite safely (it was, after all, after 1am, so there was very little traffic). I remembered on Wednesday afternoon that I have access to the Mazda Motorsports Development Program, which gets me steep discounts on parts, so I ordered up a new slave cylinder and a braided stainless clutch line.

But the week was just warming up.

Wednesday afternoon began the joy of conference calls. We were migrating a set of systems from being hosted locally to a remote site, which called for testing in a staging enclave, removing the systems from our domain, dropping the remote systems in the production network, joining them to the domain, and then ensuring connectivity.

Only nobody ever bothered to analyze the infrastructure systems to see what hurdles we might encounter. As soon as our proxy servers were stood up in the production network, we lost them. Countless hours were spent on a "war line" with reps from our site, their site, and even some outside experts who tried to lend a hand.

I got home every night from Wednesday to Friday sometime between 5 and 6 pm, got on conference calls from 7 to 8:30, and then back on more calls around 9pm, which would, in turn, end at around 10 or 10:30. Friday I got to work early, still trying to troubleshoot our busted proxy servers, but finally with a plan.

It was 10pm Friday night before I finally got one of our load-balanced proxy servers working correctly, and much later before I got the other busted one taken off the network.

Then Saturday morning I got up and tried to get to the one remaining system to get configured and redeployed. It wasn't until several hours later that I finally get a tech on the phone and got him to reset the system before I could log onto it. 5 hours after that, the system was essentially hosted.

Throughout the week, I think I saw my son for a grand total of about 6 or 7 hours. Saturday was better, but I was on the phone while interacting with him, which is less than ideal.

I'm so pooped I think I'll be in bed by 9:30 tonight. But first, I have to get through my 7pm conference call. Fun.

No comments: