Just thinking about calling it a night, when a severity one call came in. How did that work I thought, but it was just a fleeting thought – time enough to think after it was fixed. The fix turned out to be quite simple, but finding the problem didn’t – even after carrying out a substantial investigation I was still none the wiser. Still this can be the lot of a unix systems admin, some times you know how you fixed it – some times you don’t. This was beginning to look like one of the times where you don’t.
However I did all the right things, sent out a detailed email to all who might be able to shed some light on the problem. Went out into the car park and performed the headless chicken dance, considered taking the next step to becoming an arsonist (but the computer room is too far away). I have however requested that my European colleagues provide some input, that obviously carries the “if you can” proviso. Still as this was a severity one incident I had a nice chat with the incident management team, I’d forgotten how many of them were real fun loving people.
Still it didn’t take too long to fix, although I still don’t have an understanding of how I fixed it. I’ll no doubt have sone feedback to post on this tomorrow, with the process being what it is where I work – I sometimes wish I’d let the more gullible people answer the phone. Still I don’t really have that in me and I hate the annoying ring that a desktop phone has. Still it was only a severity one incident no less, what else was a body supposed to do when it all came down to it – but fix it!