Today’s task reboot two domains, how hard can that be I thought. Went to the local office first, did some bits and pieces. Had a couple of small jobs to check and thought it would be better to check first, then drove the 90 miles to the other office. I arrived in good time, I had only a few minutes work but was required on site for peace (or should it be piece) of mind.
Job went as follows, shut down all dependent application – tick in the box. Edit the /etc/system files – tick in the box. Reboot the domains (at this point one of the Poo Poo Pixies sneaked in) – tick in the box. Wait for systems to come back – tick in the box. Check databases back up (Poo Poo Pixie had disabled shutdown/startup script for oracle) – cross in box. Manually start databases and check, change cross to tick and still on time.
Restart applications yet an other tick in the box, how good can this get I think to myself. Start business checks, ticks all the way. Action final test sequence, first test is good – tick in the box. Second test is good, bloody hell – an other tick in the box. Wait for third test, wait 30 minutes – contact tester to find out that the Muppet has taken his dog for a walk before doing the third test.
Advise tester that there may be a better time to take his dog for a walk, advice and death threats are taken on board. Tester returns to house and completes his testing. Results dont appear, we check various things for four hours without finding anything. Eventually the problem seems to point to an application problem, we are then advised that we should have applications support on site – person who proffers that advice is well known and will be dealt with in due courese.
After a short discussion with the business decision is made to leave the changes in place and implement a fix on Monday, where there will be further discussions on what has gone wrong. I would imagine that the discussion will go something like this. Sysadmin Reply.