Our bit is done, what's the holdup?
Last Sunday I left my accomodation at 07:00 to drive to one of the other offices, where I was covering the upgrade of one of the storage arrays. I arrived on site just before 09:00 with my first task due at 09:30 – so pretty good timing. The day was going relatively well up until around 10:30, when the engineer arrived from the vendor, he was late and I had decided not to start my work until he arrived. The rational behind this was simple, if he didn’t turn up I could see no point in closing down the production servers and annoying the customers. I had been ready for quite a while and was expecting the go notification at any moment, but was disapointed when the vendor engineer turned up and looked like a little boy! However did I let this deter me – not a bit of it, I shut down the servers and got ready for the day’s play. Now as I’ve been in this game for a long time, I’ve gradually become more easy going about things, want me to do any thing – well I can mostly do it. Will it be as good as an expert doing the same thing, almost certainly not – but then the way that most companies work nowadays this isn’t really a major consideration.
The Lord of Darkness – You’re in my power!

As the day moved forward things gradually got worse, I had got to where I was supposed to be and was looking for more things to do. I called one of the other members of the team, she has very kindly agreed to be the standby person for the work I was doing. Unfortunately my call was answered by an other person, this person is commonly known as the Lord of Darkness – I can’t think why. But it did throw me for a few minutes and also landed me with some extra work.  However after a pained conversation I had the number I required and contacted the very nice lady, questions asked and answered quickly it was onward and upward. 

Anyway, after handing the array over the the member of the SAN team and the little boy. I left to travel to one of the other sites at the bidding of the Lord of Darkness, all this to bring up two servers that had fallen over. The servers were returned to a functioning condition and I then went back to the first site, here I was greeted the SAN specialist and the Little Boy both wearing a slightly quizical look. On investigation this turned out to be due to the fact that the first firmware upgrade had seemed to remove all the disks from the array, I must admit that this wasn’t really what I’d been expecting!

Still after a while it was decided that the best approach may be to power cycle the array, when this was done one of the breakers tripped leaving the system running on a single supply. This isn’t really how these things are meant to work, so we called out the duty electrician – as the breaker is in a locked cabinet and we didn’t have a key for it. Dual power supplies restored and lost disks back, we carried on – it was after one A.M. when I left the site and almost three by the time I got back to my digs – so much for “It’ll only be a few minutes” I thought.