Tech Mess

On Saturday February 20th, I supervised a major computer upgrade at work. Two technicians, ten hours, many frustrations, and at the end of the day it still wasn't all done to spec. There just wasn't enough time. But it was a good try, and I was satisfied that everyone would be able to get their work done on Monday, with a few minor inconveniences.

Come Monday, as expected, there were issues. I called in to work at 6:15 am to help my early shift person get logged in, and checked in every half hour from then on, until I got there in person at 8:30. From that point, I did not sit down. It was run here, run there, what's my new password, I can't print, where's my e-mail, etc. and so forth. No problem. That's what I was expecting. Give it a couple of days; everyone gets used to the new system; and things will simmer down.

Tuesday wasn't much better than Monday. Wednesday, I thought, is the day we'll turn the corner.

Wednesday wasn't better. In fact, Wednesday was worse. The users were starting to adjust to the new way of working, but the technical problems kept stacking up. Printers that I had personally tested after the upgrades stopped working. I would spend half an hour on the phone with remote tech support getting a printer working, and the next day it would be kaputsky again.

So not cool.

We are not a paperless office. We are a paper-plus office. Almost everyone has at least one printer at their desk; two if they need a label printer. I spent all day Wednesday dealing with printer issues.

By Thursday I started giving up on the new system. I rolled a couple of users back to the old system, just so that they could print. As the day progressed, I rolled back more and more people. We paid umpteen thousands of dollars for four brand-spanking-new servers, for what? Some upgrade. More of a downgrade, methinks.

By Friday I'd pretty much thrown in the towel. I don't have the time to spend 7 hours every day playing helpdesk. Our entire workplace is relocating in approximately 8 weeks. I have to make decisions on moving quotes, telephony system quotes, office furnishings for a dozen rooms in my department, etc. Most of these require a decision with six weeks' lead time to our move date, which means I have only two weeks to have all my meetings and make all my major decisions.

I came in on Saturday to catch up on some work, including itemizing a list of a dozen technical problems that people had asked for help with on Friday that I hadn't had time to address. I forwarded this to the owner of our tech consultant company, along with a frazzled plea for help.

On Monday I was reassured that help was on the way. Oh me of little faith. I'd believe it when I see it.

On Tuesday my tech guy came on site with his boss. By then there were over 20 problems on my list. They put their heads together in the server room. Then they met with me, pen and paper in hand, and made lots of little diagrammatic notes to themselves that I didn't understand. Then we called our specialized software provider on speakerphone.

We had consulted the software provider prior to the upgrades. At that time, they said everything should be fine. But on this call, they changed their tune. Apparently their software doesn't work with our printer system. Apparently. Apparently we would need to buy multi-thousands of dollars' worth of print servers and switches and install them around the entire organization in order for our new system to work.

Gaaaaaah!

I swilled that idea around in my head, hating it. At least now we knew were we stood. But we did not stand in a good place. I worried about this place all afternoon.

Finally, my brilliant tech guy's brilliant boss came up with a brilliant solution. It's a little piece of software that he wrote just for us. If it works, and we're testing it now on six PC's, then we don't have to buy a ton of additional equipment just to be able to function. So far so good. Please cross your fingers that this solution goes the distance. We should be able to roll it out to everyone by the end of the week.

Switching to the Switch

I'm home today feeling under the weather, on the couch, typing on the lap-hot. No that's not a typo. The MacBook makes my lap hot, so that's what I'm calling it.

I picked a swell day to stay home. Yesterday, at work, I got some weird complaints about our network. I arranged for a tech to come by today and troubleshoot. Here's my definition of weird: each office computer is connected to the network by a single cable, which delivers e-mail, internet access, and access to our company database. One user, then another and another, complained that they couldn't get onto the internet. However, they were able to access our database. And the other 95% of the computers had no problem accessing the internet. So the internet connection was working and the cables were working... what other component could be faulty? I had no idea. Well, we can live without internet for a day, no biggie.

This morning while I was lolling around in my p.j.'s, taking my temperature for the 8th time (Do I have a fever now? How about now?) and feeling sorry for myself, I got a call from work. Apparently the whole network was crashing, slowly and painfully, one computer at a time. User A's computer was fine first thing in the morning, then lost all connectivity for an hour, then came back online for 20 minutes, then went offline again. User B's computer could get e-mail but no database. User C, who couldn't access the internet yesterday, was one of the few computers that could get the internet today. None of it made any sense, and the users were panicking.

I figured this must all be caused by the changes that have recently been made to our network.

In a nutshell, a year ago I noted that our one and only server, which we are completely dependent upon, was out of warrantee and needed to be replaced. I started a process of getting quotes for an upgrade, but due to getting screwed around by two different companies before we got a decent proposal, and due to the need to continuously give priority to putting out fires instead of working on the upgrade project, it took a full year to finally sign off on a proposal. By the time work was due to begin, I was very concerned that this poor old server, at the end of its natural lifespan, could choke any day, leaving us high and dry.

The installation of the new server systems began a couple of weeks ago. First we got the physical components. Then the techs came in and spent many days assembling the new systems. Finally, just this week, the whole rack of new goodies got plugged in and powered up. The head tech told me that nothing on the old system would be affected until the scheduled cut-over date, but I've worked with I.T. long enough to know that it's never that simple. The new components are on our network, so they must be affecting it in some way. I figured one of these new bits and pieces must be to blame for all the problems.

Alternately, and equally likely in my experience, was human error on the part of one of the techs. The company we used to work with sent guys who were sloppy. They'd come in and change settings in order to troubleshoot problems, but they wouldn't track all the changes they'd made, and then when they didn't reset everything back to the regular settings at the end, we'd have more problems. I knew that the new techs were monkeying around with our firewall, which controls internet access, so I figured they may have screwed it up. They've given themselves remote access, so they can make changes to our systems at any time and I wouldn't know it.

The solution to the problem turned out to be much more satisfying than either of those scenarios.

Our hub has always been a bit touchy. (The hub is the big box that serves as a cross-connection point for all the network cables in the office. Cables come from the back of each computer, run through the office walls and into the server room. There they all plug into the hub.) If you so much as breathe on any network cable on that hub, it may lose its connection to the network. I have to figure out which port is affected and gently push the cable back in, very carefully, otherwise in the process of plugging one node back in I'll lose another one. It's not ideal.

The techs installed a new hub this week. Actually, it's a switch, which serves the same function as a hub, except it's better. It's the difference between a four-way stop and an intersection with traffic lights. The switch will manage our network traffic more efficiently.

The techs had moved a few test computers from the hub to the switch already.

I got a call later today with the diagnosis and fix for the network problems. It was the hub. That testy old thing had finally decided to go completely senile only three days after the new switch went live. What fantastic timing!

If the hub had died any sooner, we'd have been completely screwed. The whole network would have gone kerflooey, and business would have ground to a standstill. Instead, the techs simply unplugged all the network cables from the hub, popped them into the switch, and Bob's your uncle!

Talk about synchronicity.

My accountant boss is going to be thrilled with this story. We couldn't have squeezed even one more day of use out of the old hub. It's a very efficient use of resouces, you have to admit. It makes me look good. I warned my bosses for months that we couldn't drag our heels implementing the new equipment because the old components were living on borrowed time. They wanted me to wait until the summer of 2010. I told them we couldn't afford to wait. Now I can say "I told you so." Darn straight I'm going to say it right to them. It's called building credibility as a manager.

Now if you'll excuse me, I'm going to go take my temperature again.

The Dirt on Bootleg DVDs

The first time we bought illegal movies on DVD at Pacific Mall, we were really excited. I picked "March of the Penguins" and Ken chose "Borat". We paid $2.50 for each disc.

Back at home, I went straight to the TV to watch my movie. Oh boy! There they were, tubby little penguins trudging stoically across the snow. Cue Morgan Freeman's voice-over:

"Hmmmmmnnn. Mmm nnnn-nnn, hmmm wahhh wah mwaaaaah."

I fiddled with the volume.

"Mrrrrrrm hhmn-nweh, mwaaah-ah-ah mwaaarm," boomed Morgan Freeman.

Yeah, so. Excepting the experience of those gifted, special people who can understand Charlie Brown's teacher, the audio was useless.

Borat was marginally better, as though the audio were being broadcast to us via a badly-tuned AM radio station. You had to do the aural equivalent of squinting to follow the dialogue. Ken gamely watched it all the way through, but I couldn't handle the static.

Months later, we were speaking to acquaintances at a party. We got onto the topic of Pacific Mall and all the wonders to be found therein.

"We love the illegal DVD's!" they said. "We have over 600 at home!"

They revealed to us the secrets of illegal DVD purchasing. Apparently there are several standards of DVD's, and the one we should have bought was DVD 9. Anything less would not do.

Armed with that knowledge, we returned to the mall. DVD 9's only, please! Things went much better. From then on, most of the DVD's we bought were indistinguishable from their legal counterparts.

We'll still occasionally find a disc that won't play, or one that was recorded on a handheld Camcorder from a movie theatre screen. I bought "The Queen", only to find that it was dubbed in Italian and the "English" subtitles were a random mash of non-words. But at that price, you throw a few discs in the trash and still feel that you got a fantastic deal.

Now if only I had time to watch all the movies I bought...

Ain't as smart as I thunk I wuz

At work, I am known as The Most Technical Person. I have earned this title by being most consistently able to unjam the photocopier and coerce reluctant printers to actually print. However, all things are relative, and tonight I'm really not feeling like a genius.

Back in May of this year, I bought a digital camera, a Sony DSC-H9 Cybershot, if you care. It was a big investment, but my photos are precious to me and I wanted the best that I could afford. I bought it at a Serious Camera Store (Henry's, if you must know), where knowledgeable experts go to get all their camera gizmos.

The first few of batches of photos turned out really well. The detail, the focus, the colour - top notch. Then, sometime near the end of the summer, I noticed that some photos I had taken out in the sun had a whitish, blurry patch in the middle of them. I put it down to overexposure, and fiddled around in iPhoto to minimize the damage.

Next time I went out, I screwed a sunshade attachment onto the lens. The same problem recurred. Then I started noticing the blurry effect in indoor photos too. Because I had already diagnosed the problem as over-exposure, I figured that there must be something wrong with the photo-receptors inside the camera. It was a manufacturer's defect, and I'd have to get the whole camera replaced under the warrantee.

Today I finally got around to calling Henry's to ask how I should go about returning my camera. I even took a series of pictures to show the obvious blurred area. I was just about ready to pack the camera back into the box it came in, when Ken said:

Maybe there's something wrong with the lens. Have you checked the lens?

Well, no, I had been so confident of my previous assessment that it never occurred to me to take a look at the lens. And guess what?

There was a greasy smudge on the lens. Right in the middle, where the blurry spot kept showing up.

So, basically, I had been worrying about the state of my photo-receptors and taking bad photos for four months because I WAS TOO STUPID TO CLEAN MY LENS. Boy howdy, am I ever glad I figured that out in the safety and comfort of my own home, and not in the Fancy Pants Camera Store.

"Did you hear about that woman who came in this morning? Yeah, she wanted to return her camera because their was a smudge on the lens. No, seriously! Can you believe it? Some people are so dumb."