I work for a large multi-national company, I’m based in the UK but administer systems in locations in Europe as well. In November last year we had a power outage at our main UK data center, this was not just a little blip but a full blown outage that lasted for several days.
Almost immediately after the power failure to the systems, all the administrators based at the data center went out to the car park and performed the “formation headless chicken dance”.
Our illustrious leaders decided that such a thing could never be allowed to happen again, with the thoroughness of a grooming chimpanze they planned. They would install a second supply to the building from a separate substation, the UPS was upgraded and a superior generating set was deployed. That would prevent such a thing from happening again they said, secure in the knowledge that these ocurances are few and far between.
At 05:20 on Friday morning the power from one substation was interupted, along with the power from the other substation. But the UPS picked up the load without interuption, the emergency generating set kicked in and all was well on the operations bridge. There was much self congratulation and back slapping, all the people who had to be notified were notified and it was smiles all round.
Those smiles were still on faces 25 minutes later when all the lights went out and a deathly silence decended on the computer suit, the generator although it had started wasn’t powering the building – it was just running and heating it’s cooling water. Still at least the UPS had worked long enough to allow tha systems admins to get back to sleep, time to form up for the dance again then.