Please note that this is a page from a previous version of Continuity Central and is no longer being updated.

To see the latest business continuity news, jobs and information click here.

Business continuity information

Business continuity and disaster recovery lessons from the response to Hurricane Sandy

SunGard Availability Services recently won a DRI Award of Excellence for its response to Hurricane Sandy. This article looks at how SunGard managed over one hundred disaster declarations related to Sandy and what business continuity and disaster recovery lessons were learned as a result.

On October 29, 2012, Sandy made landfall. Staten Island and New Jersey took the brunt of the storm and for SunGard Availability Services it was the start of one of the busiest periods in the company’s history.

The SunGard AS disaster recovery team received 342 alerts related to Sandy and 117 disaster declarations. It deployed nearly one-third of its staff, five mobile recovery units and nine workgroup facilities and provided 1,500 workgroup seats for customers’ employees. Additionally customers were able to keep their employees up-to-date with critical alerts and ongoing notifications using SunGard’s award-winning message delivery platform throughout the entire duration and post effects of the storm.

Throughout the incident, SunGard AS maintained 100 percent up-time.

SunGard’s Carlstadt, N.J., center served as an impromptu community command center for local law enforcement, city government, medical and first-response teams. It helped more than 100 evacuees by providing medical supplies, food and shelter. Indeed, the center was the only facility with dependable electricity during and after Sandy’s impact in the area.

To deal with regional fuel supply issues SunGard arranged for a gasoline tanker to station itself at the Carlstadt center to fuel up workers’ vehicles.

As such events often do, Hurricane Sandy left in its wake several key lessons learned and best practices to follow should another disaster strike and threaten IT data centers and facilities. It certainly challenged assumptions, such as that working from home is always a viable business continuity strategy. (It isn’t if you don’t have power or a backup generator.)

Here are some of the lessons learned:

Transporting tapes can prove a challenge when flooding and road closures make travel practically impossible. Many of SunGard’s customers considered moving to a disk-based backup or to a real-time data mirroring and replication as an alternative data-protection strategy.

Tape requires the longest recovery times because the data is stored in a different format and requires rehydration. As an alternative, companies can move to managed backup or vaulting services that leverage storage-area network replication technologies.

If a company remains tape-based, it can take advantage of best practices in parallel processing to shorten recovery times significantly. It can consider a data-recovery provider whose standby operating system offers a service where the customer’s O/S copies can be stored inside the provider. This can significantly speed tape-based recovery.

System-recovery lessons emerged. SunGard’s customers tend to run hybrid or mixed environments, although their applications can have complex interdependencies. This means that managing change becomes vital. If the recovery environment isn’t compatible with the production environment, the recovery will fail.

In a disaster situation, a company must avoid negotiating last–minute changes to its contracted hot site as this will just delay its recovery. During the Hurricane Sandy incident, nearly one-third of SunGard customers required significant changes to their recovery system’s specifications. These changes included: having to obtain higher-end servers; additional disk capacity; different tape technology, firewall configurations and LAN bridges.

Regarding people, if a company must comply with federal regulations that mandate data privacy, it should be sure to contract for dedicated workspace where its people and the data being recovered can be kept apart from other companies’ employees. Don’t under-contract for workgroup space.

It’s critical to concentrate on internal resource planning and communicate to employees in advance about their taking personal disaster measures. This includes how to get to the data center, personal family evacuation plans, etc. If possible, a company should try to get recovery personnel to arrive at recovery sites before the storm hits since closed-down roads, public transportation and bridges can thwart access to many locations.

When it comes to processes, recovery runbooks must be kept up-to-date and procedures based on current production configurations.

Continuously upgrade and audit mass communications tools. The best practice is to pre-establish customer communications and communications with vital contacts specifically for disaster recovery purposes. Identify and train on alternative communication methods so if one form of communication isn’t accessible, a fallback is available.

What SunGard AS took away from Hurricane Sandy is that disaster recovery is a multilayer challenge. It encompasses coordinating and maximizing your strategies along the three fronts of data protection, system recovery and people, processes and program. The data protection challenges prompted its customers to think more about acceptable recovery time objectives and, potentially, about moving toward real-time mirroring and replication technologies from tape-based backup. Because so many of them required drastic changes to their recovery environments, this highlighted the importance of change management and keeping production and recovery in sync.


•Date: 14th June 2013 • US •Type: Article • Topic: Recovery facilities

Business Continuity Newsletter Sign up for Continuity Briefing, our weekly roundup of business continuity news. For news as it happens, subscribe to Continuity Central on Twitter.

How to advertise How to advertise on Continuity Central.

To submit news stories to Continuity Central, e-mail the editor.

Want an RSS newsfeed for your website? Click here