Patches aren’t just for Pirates!

Yesterday morning I left the house at 5:00 AM secure in the knowledge that it would be a long day, I had a two hour drive to the office where all the action was going to be – then it would be a pretty full day as I had two planned activities. The first was an Oracle upgrade on a SUN F-15K domain, the second was to upgrade ClamAV on two SUN T5140 servers. As I neared my destination I pulled into some services to grab a coffee as I had some extra time, my timing was impeccable – there in the car park were six coaches. The passengers of these coaches had all got into the queue in front of me, each one of the what seemed like 300 old ladies meticulously searching their purse for exactly the right change to pay for their tea and toast! Still I was ahead of schedule and I would have time to get my coffee, after all my contingency time had been used up. I left empty handed and a firm believer in euthanasia, I managed to collect my coffee from the Petrol fourcourt and continue my journey arriving only a few minutes late. Still although I had said that I would arrive at 07:00 I didn’t actually have any tasks to do until 07:45, the Oracle upgrade had already started by the time that I arrived and was a little ahead of schedule, could this be a portent of what the rest of the day was going to be like?

By 10:00 we were in fact an hour ahead of schedule and I decided to tackle the ClamAV upgrade on the other servers, I quickly discovered that the people who used to administer these servers were likely to take the odd short cut. The short cut in question was not one that I expected, in that the ClamAV upgrade is available in the SUN Datastream format. The packages had been installed from a source bundle, where the code is compiled down and the configuration switches and the library paths weren’t documented at all. A quick search on the server revealed that I would need to be a lot more prepared, as most of the source files and the build files didn’t actually have any assigned owner and the missing path for the user I had meant that it would be unlikely that I could complete the task in the time slot that I had.

Not to worry the big one was the Oracle upgrade, it was progressing nicely, by the time we got to 12:00 the entries on the plan were saying that we were now well over two hours ahead of schedule. So as I would have little to do for the next hour or so I decided to read one of the glossy Howto Manuals that were lying on one of my coleagues desks, however before I had time to start looking at the manual – there was a problem with one of the applications that would be critical to the production environment. This problem would turn out to be a show stopper and after trying many things to reolve the problem, at about 19:00 it was decided that it was time to chuck it in!

After the regression, there was a post mortem (A mercifully short one). During the course of the checks it was discovered that this problem had been picked up, however as the problem was only visible on the develoment server and not on the UAT and test servers and as this was a not a problem for the testers (The application worked fine in test and UAT) it was decided that the implementation would go ahead. This was thought to be a problem for the development team, however further investigation during the upgrade indicated that the patch levels were a little out of date. Infact the problem probably became apparent, with only two of the four servers having the same patch level (At least 5 years old) and the others being much newer. As it was most likely that the problem relates to the patch set ages on these servers, (they should be kept up to date) these are not the eye patches that Pirates wear. They are the software patches, the ones that include library updates etc.

When I left the office to drive home I was very tired, by the time I got to the closed part of the A1 I was exhausted. From the day I’d just had it had been specially closed just to put a cherry on top of the cake. A detour delayed me even more, I finally arrived back home at 00:30 so I was right about the day being a long one.