I just joined the Muppets!

Well it’s now just over 10 years since my last major dropped bollox, true to form I managed to drop an other. So I managed a cockup in 1992, 2002 and 2012 am I seeing a pattern here – well it looks like it. As I’ve already said in this blog I’ve been a Systems admin for a long time, I’ve done jobs that would surprise many. However this time I managed to bollox up in a big way, why well there was a distinct lack of RTFM all of it down to me. I had acknowledged that I had received the new instructions and indeed I had, however I had already transcribed my notes for the work and didn’t notice the change.

So what happened I hear you ask, well one of my tasks was to make a change to the /etc/fstab file to mount a file system on a new mount point and to restart some daemons that should have recreated the /dev tree. If it failed I was supposed to reboot the server with the “-r” option – pretty simple stuff really. However before the reboot I was supposed to regress the changes, on that score I failed miserably. The server failed to reboot, well that was unexpected – doh! Never mind we’ll get it sorted out was the consensus at 14:00, at 20:00 there were still some issues getting resource sorted out.

Had this been a normal environment, the problem would have been sorted out in no more that 30 minutes. However after a number of hours on the telephone and a number of international calls, including Holland, India and The UK the issue still wasn’t resolved. In line with the best SA traditions we had a local Guru that decided he could hack the system – so after 7 attempts at hacking the system. Each one with a reboot on a system, which had a potential corruption of the root file system and maybe others – there had been no progress. It was requested that I leave the site and go home, generally not a great omen. But I complied, calling in to check progress later, wish I hadn’t bothered – still two to three hours to run on the fsck. I can’t really describe how bad that this made me feel on the day.

Anyone who knows me or who has worked with me in the IT industry over the past 30 years will tell you that I certainly don’t give up easily, I’m dogged if nothing else. So being sent home in disgrace was pretty demeaning, as the new boy on the team I didn’t want to suggest that it may be better to wait until we had a proper resolution. Especially as we’d already rebooted the server several times and in my experience, each time you do that it adds to the grief. Anyway, I’m just surfing the internet looking for a new class of Sombrero, something classy – you know. Large enough to land a Helicopter on, stop the Rain drowning you because of your petted bottom lip or for you to hide in. I’m sure I’ll find it, especially as the alternative is pretty horrible – as it is I’d probably be within my rights to go postal!

Still I’ll just have to wait till the morning – if I’m back here before 18:00 tomorrow I’m going to guess that the doggie bolloxes are still floating around and I’m going to have to make a career move for the benefit of everyone concerned Muppets, Goons and Smurfs included.