Poor Planning on your part does not constitute an Emergency on my part.
(Except when it does.)
Recently I was involved in a major hardware failure at work.
There had been indications our disk storage was under duress for quite some time. It had been in my status reports for months, I finally quit reporting on it since none of our leadership team even acknowledged the concern.
They would say, yeah we know we need to do something there. Or even better: Your predecessor complained about that as well.
There had been indications our disk storage was under duress for quite some time. It had been in my status reports for months, I finally quit reporting on it since none of our leadership team even acknowledged the concern.
They would say, yeah we know we need to do something there. Or even better: Your predecessor complained about that as well.
Over a long weekend involving most of my team, teams from other groups, vendors from infrastructure support and hardware vendors we fought for our customers not management.
War room calls were set up to run around the clock with people stepping in and out of the meeting we finally were able to get things ironed out.
War room calls were set up to run around the clock with people stepping in and out of the meeting we finally were able to get things ironed out.
But at what cost?
To proactively take care of this issue would have required potentially spending a bit of money up front to either get new infrastructure or upgrade it.
However to be down for the amount of time that we were down we broke faith with our customers.
I don't know the long term impacts of that on our organization.
However to be down for the amount of time that we were down we broke faith with our customers.
I don't know the long term impacts of that on our organization.
In cases like this it is never a good idea to say "I told you so."
A current joke going around says: "At the start of every disaster movie there's a scientist being ignored"
When you are in the middle of a disaster even if you were the one that could have prevented it, the only thing that you can do is to work hard on the recovery.
There is a tendency to shoot the messenger when dealing with issues, but that should be the last thing on anyones mind. The messenger is the one who knew about the problem the longest. Usually they are the person that has been kept up at night stressing over what to do.
The messenger probably has the most ideas about how to solve the problem.
Knee-jerk reactions do not create long term solutions. Careful design, planning, testing, and proactively building in resiliency and stability create long term solutions.
None of these are cheap, but as the saying goes: "Pay me now, or pay me later."
Proactive "pay me now" situations appear to be expensive.
Reactive "pay me later" situations make the proactive look like nickels and dimes.
When you are getting warnings about things that need to be addressed, don't ignore them. If you are the ones giving the warnings, don't give up.
Keep warning.
Keep telling.
Above all, Keep planning.
When the disaster strikes, someone has to be the voice of reason.
A current joke going around says: "At the start of every disaster movie there's a scientist being ignored"
When you are in the middle of a disaster even if you were the one that could have prevented it, the only thing that you can do is to work hard on the recovery.
There is a tendency to shoot the messenger when dealing with issues, but that should be the last thing on anyones mind. The messenger is the one who knew about the problem the longest. Usually they are the person that has been kept up at night stressing over what to do.
The messenger probably has the most ideas about how to solve the problem.
Knee-jerk reactions do not create long term solutions. Careful design, planning, testing, and proactively building in resiliency and stability create long term solutions.
None of these are cheap, but as the saying goes: "Pay me now, or pay me later."
Proactive "pay me now" situations appear to be expensive.
Reactive "pay me later" situations make the proactive look like nickels and dimes.
When you are getting warnings about things that need to be addressed, don't ignore them. If you are the ones giving the warnings, don't give up.
Keep warning.
Keep telling.
Above all, Keep planning.
When the disaster strikes, someone has to be the voice of reason.
Sooooo true, Doug! I've got a few war stories of my own. Someone once told me, "At times, the kindest thing you can do, is let someone fail.". I've taken that advice to heart, because honestly, you can't fix stupid.
ReplyDelete