little problem
Operations idea of a little problem.

After the power outage on Friday morning, the systems were returned to production after some six hours. The call center began taking customer calls, the other users began thier daily tasks.

The other systems began to feed the updates to the databases, all was as it should be. This wasn’t like the last outage at all, there has been all sorts of problems with the disks for almost a week but eventually all was well in the mechanical garden.

After the online day all that had to be done was the backup and run the batch, just how tricky can it be operating a large billing system.

A few minutes into the batch, there was a disk problem. The disks were already associated, the flash copy couldn’t proceed. Best call out the on call sysadmin, what a lucky break we’ve got one still on site. He’ll be able to fix it, he almost did!

 Right up to the point where he overwrote the last good backup of the day, quickly followed by trashing the database. Brilliant result, call everyone in for the whole weekend. To be fair to the Admin he had no idea of how the application works, couldn’t have known that running the job deleted the oldest backup only keeping the last three – he does now!

 Now that we’ve managed to turn a small drama into a major crisis, the application support team and their management will be checking the notes that they were given on the the last “it wasn’t me” course that they were on. The operations team will be checking the notes that they got on the “it was him” course, The Systems Administrators will be looking for one of their number to buy cakes and biscuits, in order to maintain a silence.

It’s a well known fact that computers run on smoke, it’s been observed many times that if the smoke escapes the computer stops working. An other observation that has been frequently made is that it’s impossible to put the smoke back in properly.

In line with tradition and because we are all falible, this systems admin is awarded The Three Amigos Sombrero. There is no need for a presentation, it will just ad to the humiliation.