Help! I’m an observer at my first disaster recovery exercise...

Sean Doherty, BCM Adviser for RBC Royal Bank IT division, provides some advice.

I was recently asked by a colleague to share checklists which would help a new observer at a business continuity or disaster recovery exercise. The request forced me to think back on my own observer role, when I began to work in BCM. If I could change one thing from then, what would it be? The answer was deceptively simple: I wished I had known what questions to ask. So, as you read on, look for the questions sprinkled throughout the article. Perhaps you could extract them and build your own observer checklist.

Prior to the exercise itself, request a copy of the exercise plan. As you read it, ask yourself. “Is this scripted for success or reality?” For example, you may notice there is a limited scope, almost ensuring success. The exercise plan should grow in complexity as time goes on, until it aims for a full recovery, or to the point where adding complexity may impact production.

You may wonder, “Does the exercise plan include client validation of recovered applications?” This is an important validation of your disaster recovery exercise that some IT staff may overlook. When clients verify their application recoveries, it helps them understand that IT can deliver in an emergency, to say nothing of the confidence it will instil in your audit team.

Resist the temptation to comment on the exercise plan prior to the exercise. You may be drawn across the line between observer and participant, losing any objectivity. Save it for your final report.

During the exercise, other questions emerge, such as, “Are detailed technology recovery scripts being followed and updated?” and, “Do these reside at the recovery site?” Not to be confused with the exercise plan, these documents are operations-level instructions on the recovery of subsystems and applications. It is important that these are meticulously maintained, so that they can be followed if key staff members are missing from any recovery.

Things will get fairly hectic during a typical large-scale disaster recovery exercise. As an observer, you must know “Are Issues, gaps and problems being recorded for follow-up?” While the exercise is in progress is the best time to ensure that an owner is assigned to be responsible for tracking and resolving these. Once the exercise is over, it will be more difficult to assign ownership. Also, “Are previous issues or gaps being re-examined and closed?”, and “Are any recurring during this exercise?” Scrutinise how incidents and problems are resolved and ask, “Is one person or group dominating the incident/problem management process?” This may indicate an imbalance in the concentration of recovery expertise in the team. An effort to cross-train new or inexperienced team members may be appropriate to resolve this concern.

Presumably, a key assumption for your disaster recovery exercise is that the primary data centre is destroyed or inaccessible in some way. As an observer, you should be on alert for anything that violates this assumption. For example, “Are any computer or network resources being used from the primary site?” Pay particular attention to missing data files. “Are these being recovered from production?” This may be allowed for the exercise to continue, and would not invalidate the exercise. However, these occurrences would have serious consequences in a real emergency, and must be documented with the highest priority for resolution prior to any subsequent exercise.

Occasionally, external limitations can impose unrealistic constraints on the exercise. For example, the recovery vendor’s contracted exercise time may run out prior to completion of your DR exercise. This would leave questions hanging over the entire exercise, like “Could we have recovered if we had some more time?” Fortunately, the observer is not responsible for judging such a question, but it would probably be answered by the consensus of the team.

Although the questions and examples included here would be typical of a technology recovery (DR), the tips could be readily adapted to other types of exercises.

On your first observer assignment, try to look like you belong there, even if you feel like an outsider. Circulate and talk to participants, develop contacts and ask your questions, rather than isolating yourself, trying to understand what is happening from a distance.

After it’s over, gather your notes and write your report. Remember to solicit a review of your report from a senior participant before it goes to a wider audience, as you may have misinterpreted something major during the excitement. Good luck.

Disclaimer: The content of this article does not represent any official policy or procedure at RBC Royal Bank. Errors and omissions are the responsibility of the author.

Author: Sean Doherty is a BCM Adviser for RBC Royal Bank IT division. He has worked in business continuity for 14 years. He may be contacted at sean.doherty@rbc.com

•Date: 27th Nov 2008• Region:UK/World •Type: Article •Topic: BC testing
Rate this article or make a comment - click here