Very occasionally there is a situation most commonly referred to as the shit hitting the fan, suffice to say I’m glad I wasn’t there to witness the phrase being uttered for that very first time. I first came across a variant of the saying when working for British India a long time ago, the variant being “When the pooh hits the Punka Wallah”. The original saying has been around for quite a bit longer than that, but I don’t have any information regarding it’s origin. But suffice to say that it’s indicative of things not going quite right, or even of a bad situation getting worse.
Other comedic references to things going wrong include little ditties like, “the (magic) smoke has escaped!”, “it’s all gone Pete Tong”, “it’s gone Tango Uniform” and one of my favorites “steer clear or you’ll bust your foo foo (fufu) valve” – meaning that getting involved could cause your sphincter to give way!
As a rule this last term is never used in the presence of management, they are usually smart enough to be nowhere in the vicinity when things go awry.
When things do go wrong in the Unix world and they do, the severity of the situation can overwhelm you very quickly – especially when things go wrong at three in the morning when you are probably not at your best. Watching everything collapse no matter what you try in response is not a pleasant place to be, I’ve been there and it really sucks. I had a storage array fail – resident on the array were around 50 production applications, it was supposed to be resilient but a number of things conspired to undo the resilience. I came across the issue late on a Sunday evening and backed out, but not before every server lost access to the array – and with it being a Sunday there was no damage. It was a pain rebooting every server and restarting every application, but no damage done.
I had communicated the information out to all and sundry specifically the Unix team including what had triggered the failure, with a don’t do anything message. But as is sometimes the way a pimply faced youth, thirty five years my junior decided to have a go in the middle of the production day – because they were sure that I’d made a mistake.
Finding out that you are wrong under such circumstances is a painful life lesson, especially when you drop those same servers and databases – in some places where I have worked it would have been a career buster. But being the cause of an eight day business outage because you thought that someone was wrong does focus your mind, the person concerned left the IT world a couple of years later to enter teaching – watch out children!