Preparing a power failure response strategy

By Dr. Jim Kennedy.

According to a recent Forrester Research survey nearly half of the declared disasters reported over a one-year period were due to power failures. To ensure that this type of event does not escalate into a disaster that results in business disruption and lost revenue for your organization you need to be prepared. And this requires a power failure response strategy as part of a wider business continuity plan.

This article explains how to develop such a power failure response strategy.

Determine what you have
The first step in preparing a power failure response strategy is to identify and understand the electrical distribution system layout and design in the facility to be protected. Most building or office electrical power distribution systems are quite complex. In order to properly understand them, wiring documentation will need to be reviewed and brought up to date. If up-to-date documentation is not available when conducting the review, it needs to be developed as soon as possible and prior to developing the power failure response strategy. An electrician or electrical engineer can aid in this effort.

An up-to-date electrical system schematic or wiring diagram will clearly identify how the main components of the electrical system are connected. It should include any special systems, such as uninterruptible power supplies or power protection devices. Specifics regarding equipment voltage and current ratings should be identified and documented. It is also important to identify each major component with a unique name describing its location and important power ratings.

In addition to identifying the equipment on the drawings, it is equally important to physically match the system or component to the drawings by affixing a permanent label to the actual equipment for identification purposes. This, along with a flashlight, facilitates finding the equipment quickly in the case of an emergency.

Determine what your organization’s mission critical requirements are for emergency power
Critical electrical power requirements, or ‘loads’ as they are often called, are usually developed as part of a business impact analysis and risk assessment. However, if these assessments have not been developed as part of an on-going business continuity or disaster recovery program, then now, before a crisis occurs, is a good time to identify the organization’s critical need for electrical power.

Once the electrical power distribution system is documented and understood, the next step is to identify business critical loads that will require emergency power in the event of a power failure. At this point it is important to note that uninterruptible power supplies and emergency generators may not normally provide for all of an organization’s electrical needs. So, decisions will need to be made to determine what equipment in the operation in time of a disaster or incident absolutely need to have power maintained during a power outage.

As I indicated earlier, this is usually done as part of an overall business impact analysis, where business critical systems and devices within the organization are identified. These are systems for which special considerations need to be made in case of any type of failure — of which power failures are only one example.

After critical systems have been identified their sources of power within the facilities need to be identified on the electrical drawings to make sure it is understood that they are critical to the operation (i.e., critical computing systems, LAN/WAN network devices, PBXs, production and/or security equipment). If the electrical distribution system has been properly designed, all business equipment should be connected to common circuits and separated from non-critical equipment. This makes it much easier to isolate mission critical circuits and to properly connect them to emergency power sources during an emergency. It is also important to keep the documentation up-to-date for the equipment contained on these circuits, so that the electrical load (current required to keep the equipment running) is properly identified and emergency power generation equipment can be properly sized.

Determine what type of emergency power your organization should use
Typically if the power failure lasts less than an hour an uninterruptible power supply (UPS) may be able to sustain power – if properly sized and maintained. However, if the power outage lasts in excess of an hour, as has been the case with several US, UK, and European blackouts in the last few years, emergency generators will be needed. If it is absolutely critical to maintain power for an entire operation throughout a blackout, regardless of the length of that blackout, then consideration to possibly installing permanent emergency generators should be made. It should be noted that this type of installation is quite expensive and complex to install, operate and maintain, and may not be viable due to local building codes or environmental constraints. If it is decided that the organization wants to install permanent electrical generation equipment, then an experienced professional’s help should be solicited.

Portable electric generators can also be used. They can be installed after the power has failed and before the UPSs would shut down. If a portable generating unit is to be used, it is absolutely critical that the electrical load of each emergency circuit (measured in amperes) be known so that the generator(s) can be correctly sized (be large enough to supply the rated current of all devices to be powered during the power failure). The wattage and voltage ratings for each generator should be defined prior to the outage and procedures documented, so that when acquiring them, the load does not have to be calculated under adverse conditions: which usually leads to error and further disaster. If it is decided that the use of portable generation is the best alternative, it is prudent to arrange a pre-emergency contract to rent or lease the generators.

If a pre-rental or lease agreement is undertaken, be sure that the contract specifies delivery time and maintenance if a provided system/component fails to operate correctly. It is also important to understand in what priority order you your generators will be delivered especially in case of a widespread emergency outage. However, even with a pre-contracted agreement in place, there is no real guarantee that the generators will be available if the power outage is widespread.

It may be in your organization’s best interest to acquire portable generators ahead of time and store them for a time when they are needed. If you choose this course, it is important to establish and implement proper maintenance procedures (e.g., running the generators periodically and changing the gas and oil as recommended by the manufacturer) to make sure that the generators function when needed.

Making the generators ready for the next outage
Before any power failure occurs is the time to plan how each generator will be connected during an emergency. Here is where an electrical engineer, electrician or experienced technician is critically important. Manual transfer switches can be placed near the emergency panels ahead of time so that switching to emergency power, once the generators are connected and started, is made easier.

All of the necessary connection equipment (e.g., cables and connectors, etc.) should be stored near the place where the generators will be used. In addition, fuel and operational supplies need to be planned for as generators will consume large supplies of gasoline and oil if they need to run for many hours. Emergency generator procedures need to be well documented beforehand and attached to the generator or the transfer switch. This ensures that the generators will be connected properly and will not damage any wiring or critical equipment.

If the connection is very complex then part of the emergency response procedures should include an on-staff or contracted electrician or experienced technician to perform the connection of the generators.

Ensure the generators are functional when needed
To ensure that the emergency generation equipment (permanent or portable) is ready in case of an emergency, regular testing is essential. Testing procedures should be established (and internal audits conducted to regularly review compliance) to ensure that the generators will work properly when called upon for service.

The generators should be tested under actual circumstances and full load, if possible, but, if not, then a simple functional test will have to suffice. This functional test should include starting and running each generator for a minimum of thirty minutes at least every month or minimally, once each quarter. Also make sure that each generator’s fuel tank is full upon completion of each test and that there are documented procedures to obtain fuel during an emergency (perhaps the local gas station may not be the best source). Manufacturers of generating equipment typically recommend that fuel be replaced that is more than one year old.

Formal process and procedures are critical to success
Experiences of business continuity and disaster recovery professionals have shown that proper planning in advance of any incident makes response to that incident much more effective and electrical power failures are no exception. The group responsible for reacting during a blackout should make sure that well documented and easily understood procedures are developed and maintained.

During a crisis, people react differently than when there is no stress, so the procedures should be clearly written. One method to make the procedures easier to follow would be to use step-by-step check lists or flow-charts. Having these procedures available at each generator site to connect each portable generator will save a great deal of time during an actual crisis event. It is also important to note that the procedures should include both how to connect the generator and how to disconnect the generator when power has been restored.

Running generator installation tests periodically can help make sure everyone is comfortable and prepared for the time when the drill may be replaced by a real power outage situation.

About the Author
Dr. Jim Kennedy, MRP, MBCI, CBRM, CHS-IV has over 30 years' experience in the information and cyber security, business continuity, disaster recovery, and incident management fields. He is the co-author of three books, ‘Security in a Web 2.0+ World, A Standards Based Approach’, ‘Blackbook of Corporate Security’ and ‘Disaster Recovery Planning: An Introduction’ and author of an e-book, ‘Business Continuity & Disaster Recovery – Conquering the Catastrophic’.

In addition, Dr. Kennedy has authored numerous information/cyber security and business continuity articles for US and international publications, has presented at numerous national conferences and seminars, and has taught courses in these disciplines at the university level. He has also developed many information security management and assurance programs for pharmaceutical, educational, financial, petrochemical, energy, transportation, and telecommunications organizations world-wide.

The author can be reached at: Recovery-Solutions@xcellnt.com

•Date: 11th November 2011 • Region: US/World •Type: Article • Topic: BC plan development

Sign up for Continuity Briefing, our weekly roundup of business continuity news. For news as it happens, subscribe to Continuity Central on Twitter.

How to advertise on Continuity Central.

To submit news stories to Continuity Central, e-mail the editor.

Want an RSS newsfeed for your website? Click here