The art of the disaster simulation

By Mark Armour

Art - definition: skill acquired by experience, study, or observation (i.e. the art of making friends)

No simulation is perfect. There are technical details that get missed. There are uninvolved and disinterested parties. There are complete no-shows. There are overzealous leaders who threaten to hijack the event. There are challenges to the scenario. There are rabbit-holes that waste time and activities that are poorly executed. There’s equipment that doesn’t work. There’s food that fails to arrive. Schedules get thrown off and individuals go missing-in-action. Coordinators get second-guessed and late arrivals demand to be brought up-to-speed.

No two simulations are alike. The types of tests we conduct and how we go about them can vary greatly. They can involve stepping through detailed procedures. They can be about big picture decision-making during a crisis. They can be loose discussions or tightly controlled activities. They can include small teams or a multitude of individuals across a wide spectrum of job functions, disciplines, locations and management levels. They can be carefully guided or freewheeling.

Each and every simulation provides opportunities and challenges not only for participants but for seasoned business continuity practitioners. All tests can be improved upon and everyone involved can learn something different as a result of their contribution. What may be an exceptional experience for one may be a major disappointment for another. Simulations are not a science. In fact simulations are, for better or for worse, an art.

Value - definition: relative worth, utility, or importance

Great art challenges the observer. Lasting works leave an impression or elicit strong feelings from those they touch. Most art works carry a value, one which is greatly subjective and can vary drastically from one seemingly similar piece to another. It is true that one person’s work of beauty can end up on another’s trash heap. So it goes with simulations.

The value of a properly planned and executed simulation goes far beyond simply working completed plans. An exercise is the only opportunity, outside of an actual event, to put plans into action. The exercise process provides the chance to incorporate a wealth of additional activities to further enhance the program. With that in mind, we need to devote as much attention to our follow-through as to our preparation. It is also important to be mindful of the value others place on the activity by their level of participation. Rank-and-file engineers and developers may be required to attend, but yours is usually just one step in seeing their job to completion. Line-level managers may have other things riding on their attendance (requirements and incentives) while senior managers may actually see the benefit, and consequently, have the highest expectations.

Simulation - defined: examination of a problem often not subject to direct experimentation

Without going into the details of each, there are a number of different types of simulations one can conduct - from table-top and component recovery to functional and even full scale exercises. Each provides its own unique set of challenges and opportunities. Preparing for and conducting any one of these involves largely the same steps, though some require much more time and effort than others. Remember, too, that the more time you devote to preparation the more attention will be required following the exercise.

Putting on a proper simulation takes time to master and, like any skill, you will improve your abilities with each one. The first step is in managing the simulation process. Like many components of business continuity, testing is largely a matter or proper project management.

There are, of course, peripheral activities that require their own specific skill-sets. You want management to be involved and you require buy-in from various support areas, whether internal or external. You need to properly sell the vision of the exercise, explaining to participants the benefits of their time and effort. Properly motivated participants make all the difference in an exercise. Part of the process also involves a bit of criticism. Remember that the primary point of any test is to identify the problems and a good simulation coordinator keeps this in mind at all times. Every test is not only an opportunity to evaluate an organization’s preparedness, but also to improve the next exercise. Bottom line: never be satisfied.

Opportunity - definition: a good chance for advancement or progress

Every test provides a forum for identifying multiple opportunities to improve your overall program. The first step in preparing your exercise should be to identify your objectives:

Test the plan
A simulation is your chance to exercise completed plans and procedures already in place. This means using the simulation to work those processes in a realistic setting. You may be aware of oversights or issues within the organization that get little notice. This is the chance to highlight those issues by developing your scenario around those gaps. It does not hurt to over prepare in such situations. It’s possible your scenario will bring attention to omissions, but there may be reasons that glaring issues have gone unnoticed. Be prepared to put extra emphasis where it should belong during the course of the simulation itself.

Re-prioritize
Less subtle than gaping holes in a plan are misplaced priorities. For example, it is not uncommon for many leaders to skip past emergency response and employee issues in order to go right to the task of recovering resources and resuming business. Use your exercise to force a re-examination and identify priorities.

Training and awareness
Every test is an opportunity to promote awareness within the organization. Advertising preparedness for preparedness’ sake seldom does the trick. Use the exercise to bring focus to your campaign. A good scenario, properly presented, will involve your participants. This is the time to provide comprehensive training and make your management aware of resources that may otherwise go unnoticed. By testing the process, you emphasize the need to promote those resources to members of the team or even employees at large.

Certainly, there is nothing like an exercise for acquainting managers with the plans that support them. Building a plan is one thing, trying to put it into action is another and this is frequently the time to ensure management can actually implement what is in place. In many cases this may be upper management’s first exposure to such materials.

Maintenance
There’s nothing like a test to force your management team into maintenance mode. Unless you have a fairly robust and well enforced maintenance program, regular exercises may be one of the few chances to ensure plans are being reviewed and updated. Even if you have a comprehensive program to ensure regular maintenance, exercises provide re-enforcement. They also ensure that management and individual contributors alike are seriously evaluating the effectiveness of those plans and not just performing cursory updates.

Participation
Every successful recovery is more than just the sum of its individual steps. Major incidents require the involvement of teams beyond the departments impacted. At a minimum, there are IT groups, facilities and property management, security, safety and human resources involved. Larger organizations will include corporate communications or marketing teams responsible for managing the messages to employees, customers, shareholders and the public. They should be just as involved during the exercise phase so nobody is unclear about his or her role in the event of a disaster.

Include audit. This demonstrates transparency and fosters a positive working relationship. This is an opportunity to work with them outside of a formal audit setting. By proactively soliciting input and feedback you ensure that their concerns can be incorporated into the program, thereby improving your chances of positive findings when audit day does come around.

External agencies
It is almost certain that external emergency responders are going to be involved, if not managing, your response and recovery efforts during the initial phases of a disaster. Any good program needs to include these folks beforehand and the best way to bring them to the table is to invite them to your test – either as observers or true contributors. As with your internal participants, the level of external involvement is yours to dictate and will depend on the maturity of your program.

There may also be support organizations outside of emergency response. You may feel confident that they’ve been thoroughly vetted to perform their duties in the event of a crisis, but has that capability ever been tested? Exercises represent just one more chance to bring those critical service providers into the fold and work with you on a comprehensive response.

Prove value
When all is said and done, the exercise is proof of business continuity’s value to the organization – short of an actual catastrophe. Up until the time of testing, you’ve probably done 90 percent of the work, but without the final 10 percent, you might as well throw it out the window. This is business continuity’s chance to shine. A properly executed exercise always provides a chance to look good. By setting proper expectations, a well-run test will reveal gaps and action items that need to take place, thereby proving the need to continue improving your program.

Expectation - definition: basis for considering probable or certain; assurance

Failing to properly prepare your participants or neglecting to show them what success looks like is a sure fire way to hobble your exercise from the start. Be realistic about the time and effort necessary from those involved. This means making them aware that a good test should result in problems and issues. Remember that your participants are system owners and business unit managers. They expect positive results and they will attend your simulation with the same belief that success is measured by speed and accuracy. If those involved expect your simulation to result in a smooth recovery, then there is every chance that that is what you will get. Most people would rather feign success than risk the perception of failure. If that’s the case then there’s little need to conduct the test in the first place. Take the time to emphasize the ‘positives’ of a ‘negative’ finding. Remind everyone that their job is to find problems. If necessary, provide incentives for identifying issues.

Old school
Don’t use the old pass/fail criteria. When presented in that context, any test can fail. If a system is not brought back within a specific time frame or, heaven forbid, cannot be recovered at all, there’s every chance that participants will feel disheartened and unlikely to embraced future activities. If a table-top results in endless rabbit-holes and discussions without solid resolution, those involved can feel cheated of their time. Instead, make sure everyone you bring to the table understands that this is not about doing everything correctly, but identifying the issues that could come up in a real-world situation. Every lengthy side-bar – if addressed - is one less distraction following an actual crisis.

Budget your time
Stick to timeframes and always schedule more time than may be necessary. If a recovery time is two hours, schedule four. When presenting a scenario, time your discussion periods. Use an audible timer if necessary so things get back on track when they need to and you aren’t rushed to squeeze everything in at the end. Use additional time to address issues that arise or at the end of your simulation debriefing your participants. If things are fully vetted during the table-top, then you can let individuals go early – always preferable to keeping people longer than they anticipated.

Conduct - definition: to direct the performance of (i.e. to conduct an orchestra)

Any orchestra conductor will tell you that a great performance only comes about through regular rehearsal and preparation. Only proper planning and practice can get the simulation coordinator through all the nuances and potential stumbling blocks a simulation presents. Active participation of those invited will also go a long way to ensure the results are top-notch.

Presentation
Public speaking sometimes presents one of the greatest challenges to an otherwise flawless simulation. If your simulation requires you or someone on your team to address a group, the best way to ensure things go smoothly is to be properly prepared.

There is a common misperception that good public speaking is an innate talent rather than an acquired skill. Nothing could be further from the truth. Great presenters are not born. Know your weaknesses then prepare accordingly. You will be surprised at the results.

Resources
Always maintain a thorough list of the resources needed to conduct your exercise. Run through it as part of your preparation and always get setup well ahead of time. Make you also have contacts should things go South during the simulation.

Having your own set of tools is important but make sure those involved have everything they need as well. To the extent possible, ensure the materials being used are consistent and specific to the task at hand. Pen and paper are a good start. Custom printed materials work far better. Tailored forms allow you to direct your participants to consider specific components of their plan.

Be prepared
How many worthwhile simulations have gone awry because a projector failed to work properly or the conference line was full of static? Have a backup plan. That could mean a printed copy of your presentation that you can read from should the technology fail you. It could also mean an alternate conference line. Remember – preparedness is your job and by having a means to quickly get running despite setbacks only demonstrates your abilities as a practitioner.

If, despite your best efforts, things do go wrong, remember that those eventualities are also part and parcel of an actual event. Respond as best you can and remind your participants that it is all part of what they can expect following an actual disaster.

Support roles
The larger the exercise the more support you should have on-hand. If you are conducting a simulation, no matter how small, you should designate at least one other individual to assist. This person should be available to help others with questions and issues during the presentation in order to prevent the entire simulation from being sidetracked.

Delegate someone to act as the exercise scribe. This person should record the overall activities while also evaluating the events and the coordinator’s abilities. During smaller scenarios, this may be the same as your support person. If not, make a point of ensuring that those acting in other support roles keep your scribe informed as to the issues being experienced and questions being asked.
Lastly, ensure there is someone who can provide technical assistance. At a minimum, ensure there is a means of obtaining that assistance at a moment’s notice.

Be open to criticism and listen to what your support folks are telling you. Opportunities always abound for the business continuity professional to take corrective action and apply lessons learned to future simulations.

Result - definition: something obtained by calculation or investigation; also : beneficial or tangible effect: fruit

All of this effort and all of the resources devoted to it are for naught if you do not dedicate commensurate time and effort to capturing your results. This includes the good and the bad. The benefit of recording the good is obvious: demonstrate to management, audit and your participants that your program is working. The positives attributable to you and your team are also those elements you want to ensure you keep for future activities. Keeping track of what you did well is a means of ensuring you do the same thing again (and again and again). The successes can also be used as a benchmark. Look to improve upon them in future sessions. Be ready to sound alarms in the event you fall short the next time around.

If recording positive results is important, tracking your negatives is a necessity. Identifying issues and shortfalls is the primary purpose of any test. At a minimum, you want to make sure you identify where plans and strategies are not adequate to recover in the time necessary. More broadly, you also want to find where the weaknesses are within the organization and where opportunities for improvement lie. If you already have a sense of where problems exist and have adequately prepared then your exercise should bear them out in the results. Lastly, you’ll want to evaluate the exercise itself. Just as you want to repeat the things you do well, you want to identify those things that went poorly so you can make appropriate modifications the next time.

Capturing lessons learned
Just like conducting sessions is a skill that must be practiced and honed, objectively evaluating your exercises and capturing lessons learned is not something that comes naturally. Only through effort, practice and simple trial and error can you learn to identify all of the valuable lessons that come from a test. There are two basic means of capturing this information: in a live session or via feedback form.

The live session
As mentioned previously, you should set aside an extra 30 minutes to an hour as part of your simulation. This provides time to conduct a quick evaluation session with your participants while the exercise is still fresh in their minds. Memories fade over time and things that felt important at the moment tend to seem trivial when people return to their day-to-day work.

Conversely, a second session held a short time following the exercise allows individuals to think of things that hadn’t occurred to them during the simulation. Sometimes the extra time allows everyone involved to more clearly formulate and communicate the difficulties experienced. You may even find that, given some time, participants will develop and propose solutions.

In addition to group sessions, it’s also possible to conduct one-on-one sessions with each of the people involved. While this can be resource-intensive it can sometimes yield the most informative results. Members may be willing to speak up more in a private session then in the middle of a group. This also allows you or your team to devote the time necessary to discussing the input.

The survey
The alternate means of gathering input is via the survey or questionnaire. As with the live sessions, the means by which you do this can vary greatly. You can provide materials for your participants to record their feedback and turn it in at the end of the sessions. Using this to augment the debrief sessions allows those who are not comfortable speaking in a group to provide their input. A follow-up survey accomplishes the same thing as the delayed debrief session. Individuals can consider their input. The written or web-based approach also eliminates potential time constraints associated with scheduled sessions.

Combined approach
Employing all means of capturing input is ideal. It takes advantage of the individual strengths of all those involved. The drawback is the time and effort involved. As a result you’ll need to consider the size, scope and importance of the simulation and devote the appropriate effort to capturing this information. Remember, this is the most important aspect of any simulation. Don’t cut corners at this juncture.

Evaluate - definition: to determine the significance, worth, or condition of, usually by careful appraisal and study.

By now you’ve established goals, honed your scenario, prepared your presentation, set expectations, conducted the session and gathered lessons learned. Now is the time to turn that feedback into meaningful steps to take.

Program improvement
The goal of any exercise is to improve the overall program. This includes evaluating and improving strategies as well as changing plans themselves. Look for gaps in understanding which represent opportunities to improve training and awareness. Depending on the scope of the simulation you may find that your recovery environments (both technical and work area) fall short and require investment. Your communication process – from the crisis management plan to the tools being used – should all be open to scrutiny.

Admittedly, turning varied input from multiple sources and media into concrete action steps can be a daunting task. Where necessary, get clarification from the individuals who provided the feedback.

Involvement and interest
Sometimes lost in the effort to close real gaps are the issues with the exercise itself. Don’t hesitate to look for opportunities to improve the testing program. The goal here should be active participation and positive contribution. If those attending your exercise exhibit negative reactions they are less likely to be involved the next time. While they may be required to attend, don’t expect any meaningful input if you have not addressed their concerns.

Remember that this starts with setting the proper expectations. This means going in with the proper mindset yourself. A mountain of issues should not be looked at as a failure. Quite the contrary – this is indicative of active participation and points to a willingness to take on problems. Each difficulty is an opportunity to improve the program and such challenges should be readily acknowledged and forthrightly addressed. Embrace those pointed questions!

Implement - definition: to give practical effect to and ensure of actual fulfillment by concrete measures.

Now is the phase where you put it all into action. This can be simultaneously the easiest and the most difficult part of the entire process. It is the easiest because many of the activities can be handed off – to your participants, your support partners and your team-members – to be carried out. It is the most difficult precisely because you don’t always have direct control. Activities can sometimes languish following an exercise. Participation in the test may be mandatory but follow-through is not the first priority once individuals get back to their desks.

Follow-up
Step one is to make sure the results of your evaluation are communicated. This should include the action being taken to remediate issues and the individuals responsible for execution. Rather than following up individually, list owners and action items in a single distribution to all participants. Seeing everyone else on the distribution is sometimes all that’s needed to prompt action. This also ensures nobody can claim they weren’t aware of their role. At the very least it’s a confirmation of who is responsible. If you’ve designated the wrong person or team with an action item you are sure to hear back from them and can make corrections as a result.

Regular reminders along with a checklist of accomplishments can help reinforce expectations across the groups.

Escalation
Remember, the activities you’ve defined are all necessary to address issues and improve the program. In the case of known gaps to recovery, ensuring they are closed should be the first priority. This means escalating when there is little or no movement. Sometimes this can be a difficult task. Seldom do individuals refuse to take action or refute the evidence of the findings. Instead there are competing priorities and resource constraints to contend with. That’s when escalation is most critical.

As the practitioner it is contingent upon you to communicate the importance of the activity. This means making leadership aware of the strategy and plans in place as well as (and sometimes most importantly) the time, effort and cost involved in implementing that strategy. Why would senior management put forth such expense when the result falls short of expectations? Making the necessary corrections ensures that the expenditure is justified and the results are maximized.

Future exercises
Don’t forget those lessons learned that apply to the exercise. Was there feedback indicating issues or problems with the session itself? Is there input on the preparation activities? The follow-through? As the one who manages the process it is easy to become defensive or discount such input. If there is agreement from multiple individuals then you can be assured there is a problem to address. In those instances where one disgruntled individual has a bone to pick, don’t shy away from confronting the issue. Set up a separate session to talk through the person’s experience. In almost all situations, just demonstrating interest in the problem and a willingness to discuss it is enough to turn around a negative perception.

Again, it’s important to recognize that not everyone is alike and not all people will see the same benefits of the exercise you’ve just been through. This is an art after all. Making the effort to educate your participants and spending time understanding their perspective will go a long way in helping you make your exercises meaningful to the most people.

In summary
This wasn’t meant to be a book and it isn’t meant to be the final word on disaster simulations. The idea here has been to bring a lot of considerations to the table. In an ideal world we’d all have unlimited time and budgets to conduct exercises properly. Short of that, all we can do is maximize the results of the tests we do conduct. If I had to synopsize the critical components of a comprehensive exercise it would be thus:

- Identify opportunities to strengthen your program and highlight deficiencies.

- Set expectations among your participants. This includes yourself and your team.

- Conduct your session by being as prepared as possible and through regular practice.

- Gather results from your participants by as many means as realistically possible.

- Evaluate input for improvement opportunities and action items.
Implement changes using diligent follow-up and escalation.

It sounds simple but – like art itself – there is always more to it than meets the eye. Good luck and happy testing!

Author
Mark Armour has been in the business continuity profession for over nine years. Contact
mnjarmour@sbcglobal.net

•Date: 21st June 2012 • US/World •Type: Article • Topic: BC testing & exercising

Sign up for Continuity Briefing, our weekly roundup of business continuity news. For news as it happens, subscribe to Continuity Central on Twitter.

How to advertise on Continuity Central.

To submit news stories to Continuity Central, e-mail the editor.

Want an RSS newsfeed for your website? Click here