Friday, January 8, 2010

Switching to the Switch

I'm home today feeling under the weather, on the couch, typing on the lap-hot. No that's not a typo. The MacBook makes my lap hot, so that's what I'm calling it.

I picked a swell day to stay home. Yesterday, at work, I got some weird complaints about our network. I arranged for a tech to come by today and troubleshoot. Here's my definition of weird: each office computer is connected to the network by a single cable, which delivers e-mail, internet access, and access to our company database. One user, then another and another, complained that they couldn't get onto the internet. However, they were able to access our database. And the other 95% of the computers had no problem accessing the internet. So the internet connection was working and the cables were working... what other component could be faulty? I had no idea. Well, we can live without internet for a day, no biggie.

This morning while I was lolling around in my p.j.'s, taking my temperature for the 8th time (Do I have a fever now? How about now?) and feeling sorry for myself, I got a call from work. Apparently the whole network was crashing, slowly and painfully, one computer at a time. User A's computer was fine first thing in the morning, then lost all connectivity for an hour, then came back online for 20 minutes, then went offline again. User B's computer could get e-mail but no database. User C, who couldn't access the internet yesterday, was one of the few computers that could get the internet today. None of it made any sense, and the users were panicking.

I figured this must all be caused by the changes that have recently been made to our network.

In a nutshell, a year ago I noted that our one and only server, which we are completely dependent upon, was out of warrantee and needed to be replaced. I started a process of getting quotes for an upgrade, but due to getting screwed around by two different companies before we got a decent proposal, and due to the need to continuously give priority to putting out fires instead of working on the upgrade project, it took a full year to finally sign off on a proposal. By the time work was due to begin, I was very concerned that this poor old server, at the end of its natural lifespan, could choke any day, leaving us high and dry.

The installation of the new server systems began a couple of weeks ago. First we got the physical components. Then the techs came in and spent many days assembling the new systems. Finally, just this week, the whole rack of new goodies got plugged in and powered up. The head tech told me that nothing on the old system would be affected until the scheduled cut-over date, but I've worked with I.T. long enough to know that it's never that simple. The new components are on our network, so they must be affecting it in some way. I figured one of these new bits and pieces must be to blame for all the problems.

Alternately, and equally likely in my experience, was human error on the part of one of the techs. The company we used to work with sent guys who were sloppy. They'd come in and change settings in order to troubleshoot problems, but they wouldn't track all the changes they'd made, and then when they didn't reset everything back to the regular settings at the end, we'd have more problems. I knew that the new techs were monkeying around with our firewall, which controls internet access, so I figured they may have screwed it up. They've given themselves remote access, so they can make changes to our systems at any time and I wouldn't know it.

The solution to the problem turned out to be much more satisfying than either of those scenarios.

Our hub has always been a bit touchy. (The hub is the big box that serves as a cross-connection point for all the network cables in the office. Cables come from the back of each computer, run through the office walls and into the server room. There they all plug into the hub.) If you so much as breathe on any network cable on that hub, it may lose its connection to the network. I have to figure out which port is affected and gently push the cable back in, very carefully, otherwise in the process of plugging one node back in I'll lose another one. It's not ideal.

The techs installed a new hub this week. Actually, it's a switch, which serves the same function as a hub, except it's better. It's the difference between a four-way stop and an intersection with traffic lights. The switch will manage our network traffic more efficiently.

The techs had moved a few test computers from the hub to the switch already.

I got a call later today with the diagnosis and fix for the network problems. It was the hub. That testy old thing had finally decided to go completely senile only three days after the new switch went live. What fantastic timing!

If the hub had died any sooner, we'd have been completely screwed. The whole network would have gone kerflooey, and business would have ground to a standstill. Instead, the techs simply unplugged all the network cables from the hub, popped them into the switch, and Bob's your uncle!

Talk about synchronicity.

My accountant boss is going to be thrilled with this story. We couldn't have squeezed even one more day of use out of the old hub. It's a very efficient use of resouces, you have to admit. It makes me look good. I warned my bosses for months that we couldn't drag our heels implementing the new equipment because the old components were living on borrowed time. They wanted me to wait until the summer of 2010. I told them we couldn't afford to wait. Now I can say "I told you so." Darn straight I'm going to say it right to them. It's called building credibility as a manager.

Now if you'll excuse me, I'm going to go take my temperature again.


LL Cool Joe said...

I think you have lap-hot fever. ;)

I used to call a laptop a flat top. Yes I was drunk.

Get well soon mate!

darcknyt said...

Nice! Excellent work!

I was reading your story about wonky connections and going, "OH, I remember this one. This is what happens when a switch or router needs rebooting or replacing. Yeah, yeah -- I've heard this one before."

Imagine how gratified I was when you said the problem was a dying hub. :)

Brilliant work on that feather in your very-fine cap, dear.

powdergirl said...

Yeah, whew, I know the lap tops do put out a lot heat!

Yay! You get to say 'I told you so."

Thats always fun : )

Well? Fever? No fever?

Ily said...

When you're right, you're right...and it's good to let them know who they need to listen to in the future to prevent any more disasters!

Feel better soon!!

NicoleB said...

Hope you don't get sick!
And Congrats on diagnosing the @work computer Probs right ;)

Sparkling Red said...

LL Cool Joe: Well, it does have a flat top. So, according to my standards of naming things, that is a perfectly acceptable name. :-)

darcknyt: Thank you! I never ran into this particular problem before. I didn't realize how crazy things could get from one confused hub. I promise, I'll never forget this lesson!

Powdergirl: My temperature spiked up into fever range a couple of times through the day, but mostly it hovered just below official fever territory. I guess that's good news, although I always feel more justified lying around all day if the thermometer's showing higher numbers.

Ily: Thanks! I already feel quite a lot better today, Saturday. :-)

Nicole: Thanks! I think if I take good care of myself, I'll get better, not sicker.

DarcsFalcon said...

Network crashes and fevers - not a good day for you!

I'm glad you're feeling better and that you apparently haven't been getting sicker, and that now the network is all fixed and upgraded. That one did end up getting timed perfectly. :)

G said...

Congrats for proving once again that you are indispensable. Get well soon!

Dianne said...

I have a tiny network set-up and I fear my hub ;)

how's your temp?

Jameil said...

lol. yay you! i think you should take your temperature again for good measure.

San said...

At first I read "bundling credibility as a manager." That works too!

Hope you're feeling better, Spark...

Jenski, PhD said...

Nice foresight and close call!