I recently participated in my first contingency exercise for an application. It was what we call a "Tabletop Test", and while I would rather have had a physical exercise, it was nonetheless informative. This is something that I have been pushing for a very long time. The problem is that people are often so consumed with day-to-day operations that we never make time to actually run through a simulation of what to do when bad things happen.
I picked the very unlikely scenario of a hurricane knocking out operations of a data center in the upper midwest. I know it is not reasonable that a hurricane is going to do that, but a tornado is much more likely, only it wasn't on my scenario list and the hurricane still allowed me to run through what people should do for the other, more likely situation. We actually identified a couple of areas that need some improvement. For example, there are several applications that run out of this particular data center, we have to take some time to prioritize the order in which these applications will be restored and assume that we don't have the resources to bring them all up at once. Also, you already know that I am a Green IT fan. This means that if I don't have to print it, I won't. But Contingency Plans and Disaster Recovery Plans must be available in hard copy. I didn't have them in paper before.
Overall though, it was a worthwhile exercise and we found some things that can be improved for the next time. And the next time will be in 6 months. I think that the more frequently you run through these types of scenarios, the better you become at it. Practice makes perfect (well, at least better than before), drill drill drill. So we'll be doing it again 6 months from now, only I want to have a physical exercise, bring down the servers and restore them at the alternate location and bring the application up. I know that we'll find other useful information that will help us to perform more efficiently if we ever had to do it for real.
No comments:
Post a Comment