Blitzkreig A’la Muppet

Well today was just F{}c#1%* wonderful, things just gradually got better and better. When a colleague was doing the early checks he noticed that one of the system logfiles was filling very quickly, not surprising as all the messages were being logged multiple times. Then minutes later we found out that there were problems running any commands that had any kind of privilege, some scripts that we had been working on were failing and in general every thing was turning to S#!+. All of this just 48 hours before a major go live, one that has been designated a major milestone by our European masters. In honour of this I have decided to leave the site in Red, it seems to fit the feeling that I have about the move of some of our regulated applications to our European Data Center! You may think that the title of the post is a little unkind, you may actually be right – but had you been in the office with us today. You too would be looking to dispatch these people in the most painful way possible, probably – as I’ve said before with a cheese grater only this time you may want to have one of those squirty things of Jiff lemon juice handy as well.

Just to bring you up to date and to give you a better understanding of the problems that we had, I’ll provide you with some information that has some historical significance. This go live is extremely important, it is part of a massive project and has been designated a milestone by the people who matter. It has been tested to death based on the fact that the systems had been baselined with the agreement of all parties, it hasvisibility at the highest level throughout the company. The project is being managed by people that could be outsmarted by a ring-pull can, these people have already pissed me off to the extreme and if it wasn’t unprofessional I’d be tempted to let them bollox the whole thing up!

The changes on the systems that are about to go live were in them selves quite insignificant, these included but were probably not limited to. Removing “/usr/sbin” from everyone’s path, changing the logging on the system to the lowest level and setting it to log everything twiceto two files then remove one of the files and link it to the other so as it logs everything four times. Changing the priviledged users management file to ensure that nothing of any consequence could be run, all in all quite an achievment for a few minutes work.

Now I understand that computer security is important, I really do! Anyone who has followed this blog will know that I’ve already had a run in with the UK based security team, which led to some minor changes in policy and working practices. If you want to bring yourself up to speed than “Garry Rotter and the Scans of Doom” is a good place to start, my opinion of the UK security team has risen significantly (by an order of magnitude in fact).

To me there is a simple issue here, the UK team is supposed to be the lead on the Solaris systems. But it doesn’t seem to be working that way, we always seem to be playing catch up with the changes that are made by our European colleagues. They are in charge of the jump-start image (on average changes twice a week or so since the project was started), they are in charge of what services run, what packages are added at build time, what generic file systems are started etc. Yet this would seem to be our fault, although we have only used what has been set up for us. Our main input has been to report back when things don’t work the way we expect, often the reports are not even acknowledged – but to be fair the changes do occur (although there doesn’t seem to be any change control).