A white paper supplied by SunGard Availability Services.
The first sign of a business in trouble can be a phone that goes unanswered. Customers don’t know if the phone system is down or if you have just gone out of business. This paper outlines how firms can recover their telephony in the fastest possible time using business continuity providers who have the in-house capability to recover even the most complex telephony systems. It also highlights how recovery can be achieved and how to identify the ‘providers’ from the ‘pretenders’.
“Could you recover your telephony in 10 minutes?”
This is perhaps a more pertinent question than it might at first appear. In our experience, even industry acolytes and award-winning business continuity managers can be shocked to discover that, when placed under scrutiny, their telecoms recovery leaves much to be desired. Many organizations have been sobered to find out just how limited their telecoms recovery provision was and how much more could be achieved.
Without a doubt, telecoms have been incredibly reliable. On average, firms experience just four minutes telephony downtime per year and because of this reliability it’s not given the due attention it deserves, with many important aspects often overlooked in recovery plans. BC managers tend to shy away from telecoms because it has been perceived as something of a dark art; an aspect of recovery which has traditionally been separate from IT and the sole preserve of the telecoms manager. But as the convergence of IT and telecoms continues such shyness is even less acceptable for any organization keen to secure maximum communications availability and associated flows of business information.
So, challenge yourself and your recovery provider. Is your telecoms recovery as robust and seamless as you – and, more importantly, your customers – would like it to be?
Telephony change is coming. Are you prepared?
The telecoms transition from Time-Division Multiplexing (TDM) to Internet Protocol (IP) Telephony-based platforms has significantly impacted telecoms recovery to the point where telephony more closely resembles an enterprise application than the standalone box that sits hidden in the corner of the office.
Whilst the previous TDM systems had significant functionality, it was never fully exploited by business. Now the application is IP based and can be managed within the IT function it is far more pervasive and is now one of the most intricate applications to successfully recover. If you want your recovery plans to better reflect the location of your staff, or if your business relies on voice contact with customers, you cannot ignore this evolution of telephony.
As well as the change of platforms, firms have also pursued a converged network strategy, where their data and telecoms networks have been consolidated into one single network, to leverage both cost savings and the functional benefits of a single IP infrastructure.
Not only does joining two networks together create a single point of failure that could bring down both voice and data at the same time, the running of two forms of data down the same network also increases complexity. This potentially amplifies the risk of confliction and corruption between the two data forms.
From the 20 percent or so of firms who completed initial IP Telephony pilot projects, the value of such networks has been proven. Telecom suppliers end-of-lifed traditional TDM systems and so full-scale IP telephony implementations were rolled out across business estates. This enabled them to leverage lower total cost of ownership (TCO), when integrated with office rationalisation and more efficient ways of working such as hot-desking and intelligent handset applications and directory services. These items were typically found in call centres, but IPT has made this functionality more widely available.
Telephony change is coming. Are you prepared?
Telephony therefore should no longer be considered in isolation, as an easy to recover, standalone platform. It is now a complex application, spanning both IT and telecoms departments – and especially so in the context of its recovery in the event of business interruption or disaster scenario. BC managers are strongly advised to increase their level of understanding of telephony changes – in SunGard’s opinion it is imperative that the impact of those changes on recovery plans in both positive and negative terms be understood.
Don’t allow your provider to design your PBX during a disaster
If you are feeling reluctant to drill down and understand the principles of telephony recovery – you are not alone. There are business continuity providers who are also reluctant to offer a full recovery service in this space because it is a challenge if you are not familiar with the technology. This might explain why some service providers do not keep an image of your telecoms. Instead they prefer to build it on the day of your invocation or on the day before your test. In the case of a recovery test they may well have plenty of time to prepare your PBX, but how will your business fare in a real disaster when such providers don’t have the intellectual property or the luxury of a day to set up beforehand?
At time of disaster, an unanswered phone is going to be one of the first things customers notice when they try to contact you. Will your failure to take that call, or manage it with a staged recorded message, meet with sympathy or a loss of confidence? Will they trust that you are on the road to resolution or will the next call be made to your competition? You really need your telephony back up again as soon as possible, especially in an age when we tend to become impatient waiting for the microwave at home to ping!
Don’t accept an outbound call only service
UK recovery models have been changing in line with Tripartite Authority guidelines, particularly in the London region. In the capital, multi-site recovery is growing to help dilute recovery centre concentrations and deal with the wide-scale metropolitan incidents that can lock the city down.
So when evaluating your recovery supplier’s capability, you may wish to test their service delivery in terms of:
- Recovering telephony so your staff can call out – that should be easy;
- Recovering telephony so customers can call you – this is a step change in difficulty;
- Recovering telephony during multiple invocations – this is especially difficult to execute without disrupting the systems of those customers who have just recovered their telephony;
- Recovering telephony during multiple invocations across multiple recovery centres – this requires the provider to have a co-ordinated multi-point recovery capability;
- Recovering telephony during multiple invocations across multiple recovery centres, spanning multiple shifts of your staff – the most difficult challenge of all!
To recover against the last three scenarios, your provider will need to have spent several £million in hardware, software and network capability – not to mention investing in the human skills that can design and deliver such a service. As a benchmark, look for evidence of significant telephony investment, specifically:
- New infrastructure that is standardised everywhere
- People resources to deliver the service and provide first, second and third level support to ensure they can recover against all of the above scenarios.
Otherwise the success of your recovery could be dependent on a minor subcontractor fulfilling their part of the process.
Don’t accept an outbound call only service
So, as a customer in the market for telephony recovery, you need to ensure that your recovery time objective (RTO) is not limited to outbound calls only. You also need to question the assurances service providers give you. This is not to suggest any lack of honesty, but to enable you to identify any lack of capability. Many simply do not understand telecoms recovery sufficiently or may prefer to defer the question in the hope that presales or product management can answer the question for them.
Given the available technology, there is no need to compromise your telephony recovery or customer service performance. Even when you take into account all the nuances and variables within typical contracts, it is possible to achieve an RTO within 10 to 40 minutes providing all preparatory measures are in place.
Complete telephony recovery
Telephony recovery should form an integral part of your workplace recovery model and your recovery provider should run a suitable national network (a network that offers enterprise level bandwidth and speed). Currently dual diverse multi-gig connectivity between workplace recovery centre locations and Terabit capacities between resilient data centres lead the marketplace.
The principle behind telephony recovery is simple: to provide you with the same telephony experience that you are used to at work … without compromise and to manage your recovery both professionally and more quickly than doing it in house. A bold claim but this is how it can be done.
1) First your provider should capture and store your call configurations in a planning tool so it is ready for deployment at time of disaster (ATOD) so that the process is automated and accurately executed ATOD; thereby achieving a faster RTO.
Some providers may baulk at automation because they cannot accommodate all of the invocations and so will ask companies to share facilities. Therefore any predetermined recovery and telephony plans are immediately compromised and will need to be changed on the fly.
BC managers should consider the implications of this in the context of your provider building the PBX in conditions that are highly pressurised in a multiple invocation scenario: i.e. what services and functionality will need to be eliminated in order to reduce the time of the PBX re-build. How long will it take your provider to build the PBX? How reliable will it be? What are the security and regulatory implications if I end up sharing a suite with another business or department with a conflict of interest?
2) Recovery PBXs should be partitioned or divided, with your requirements ring-fenced, to ensure your inbound calls stay within your recovery suites and within your business. And there should be ample DDI numbers to ensure they can provide you with your usual familiar department extension numbers and coverage paths which can send calls out to staff working at home which all adds up to making life that little bit easier. Top tier business continuity providers should have sufficient DDI and be able to map numbers to desks and sites enabling them to offer customers significant flexibility and resilience.
3) During the first stages of telephony recovery your provider should initially manage your inbound calls with a pre recorded customer message or a sequence of messages, until your staff arrive at your designated workplace recovery centre or centres.
4) A bulk configuration tool enables your provider to load your call configurations into the PBXs without interrupting other customers that have already recovered. This means your newly recovered telephony service isn’t disrupted should a fellow customer require their telephony system to be recovered.
5) The Auto Call Distribution (ACD) feature provides and manages the handset logins for your staff, so if your working culture is hot desking based they can sit anywhere in your suite, access their phone features and their calls still reach them.
6) When your staff begin to arrive at the recovery centre, an initial holding message can be turned off and calls (according to your chosen configuration) can be sent to hunt groups or skill sets so you can begin to answer calls using your people.
7) When your required amount of staff have reached the recovery centre the hunt group setting can be modified and normal service resumed, with the option of call coverage or overflow to manage busy lines: i.e. directing calls to other staff or voice mail.
8) If you have opted to split recovery across two or more sites, a central PBX can be used or you can opt to a local PBX configured to split calls instead. Some firms also prefer to take over the LAN switch at a Recovery Suite level for greater physical security control.
9) If any recovery sites become unavailable, your calls can be quickly managed and redirected to another nearby workplace recovery centre using a facility called ’rollback’. This means that large-scale incident risks can also be mitigated. This capability is not available with all BC providers. And to measure any supplier claims in this respect, you need to be confident that a, “yes we can do it” answer translates into a proven, robust process that is standardised and repeatable without any degradation to your recovery as opposed to engineering it ATOD.
10) So when it comes to recovering your telephony think about some of the questions you need answers to and challenge your provider for the answer:
1 Do you have a plan for inbound calls?
a) Individual DDI numbers
b) Service desks intelligent numbers (0800, 0845 …)
2 Do you have a plan for home workers?
3 Do you have a voice plan that is portable?
4 Do you have multiple recovery scenarios?
5 Do you have a staged voice plan?
6 Do you have a plan to communicate with; customers, staff and suppliers?
7 What have you pre-planned?
a) Call announcements
b) Hot desking
c) Tenant partitioning and rollback
d) Staff Incident Hotline
e) New phones for the Incident Management Team to ensure effective and focused team communication which can for example help to avoid ‘disaster tourist’ calls.
When deciding upon who your workplace recovery provider might be, recovery of the telephony element should be uppermost in your mind because any recovery service limitations can have a detrimental effect on the quality and the likely success of your overall recovery, especially in a multiple invocation scenario.
Service provider’s telephony and LAN limitations mean you could end up having to unexpectedly cut the amount of people you plan to recover. Some providers are not able to manage large-scale invocations, or cannot rollback customers easily to another site should your first choice sites become unavailable.
If your recovery provider’s telephony is limited you may be required to share a recovery suite with another company which can create significant security and compliance risk issues.
If your provider builds your PBX from scratch ATOD, you may have to wait some considerable time before your telephony is recovered, encounter limitations in functionality, or experience additional call downtime whilst other customers are having their telephony systems recovered.
Finally, set your telecoms recovery expectations high. You don’t need to artificially suppress your telecoms RTO by adopting the old recovery mantra of keeping it simple. If your service desk is a high performance, complex telephony environment you can still replicate it. But your business continuity provider has to have the tools and experience needed to build in the complexity in advance without having to worry about it ATOD. So don’t accept the restraints that some recovery providers may place on you just because they find it a bit difficult. Instead find a partner who can give you the recovery your business requires.
•Date: 20th Nov 2009• Region: UK/World •Type: Article •Topic: Telecoms continuity
Rate this article or make a comment - click here