Tag Archives: Disaster Planning and Recovery

Backup Power Systems

By Peter Lyle DeHaan, PhD

Author Peter Lyle DeHaan

Most call centers know the importance of having a good and reliable backup power source. Unfortunately not all centers have taken steps to obtain that protection or to ensure that it is adequate and properly maintained. Given rolling blackouts in California, the recent unprecedented outage affecting most of the Northeast United States reaching into Canada, and the interruptions that generally follow hurricanes, there is ample reason to move backup power to the top of the priority list. Add to this possible quality issues with utility power, such as brownouts, sags, surges, spikes, and so forth (see the Power Glossary for an explanation), the need to be covered with backup power is all the more critical.

“During the hurricane, the power was real dirty,” stated Steve Kenny, owner of The Best Answer, in Ocean City, MD “and many people lost power or their power was intermittent. Our service kept running because of the UPS backups and our propane generator. This gave us clean, uninterrupted power while everyone else lost power.” One of their clients, a home health company, was without power. “We were able to answer their calls, dispatch oxygen, and medicine to the nurses, and act as the clearing house for them. Their competitor was without power, too, and [our client] picked up a few new customers because of our reliability.” With good, reliable backup power systems, you too can gain new business and better serve existing clients.

Whether your center needs to install backup power for the first time, add more capacity, or replace aging systems there is a general strategy to follow. Use UPS systems for immediate, short-term protection supported by electrical generators for long-term outages.

UPS for short-term outages and problems: Uninterruptible Power Supplies (UPS) provide a quick source of power for short durations. Since most power outages or problems last less than two hours, UPSs are the best first line of defense. In general, a UPS takes DC power from batteries and converts it into the AC power needed to run call center switches, computers, and networks. The more batteries, the longer power can be supplied but at a greater cost.

There are different types of UPS systems, which produce different qualities of voltage. While manufacturers may use different terminology, UPS fall into two general categories. The first are those that run power through them, conditioning and filtering it all the time. When there is a problem with the utility power, these UPSs are already on the job working. The second type monitors the line, but does not filter or condition it. At the sign of a problem, they turn on, take over, and provide power until the utility power is restored and stable.

Both approaches have their strengths and weaknesses. If you have any doubt about which is best for you, have an in-depth discussion with your vendor, preferably one that sells both kinds.

Also, the quality of the power provided by UPS systems varies. More advanced systems produce a pure waveform, called a sine wave. Simpler and more basic units may have a “square-wave” or some other non-sinusoidal output. For some equipment, this won’t matter, while for other things, such as sensitive electronics, this could present a problem or shorten the life span of the equipment. When considering both the style of UPS and the type of output generated, there are cost-performance issues to be carefully weighed.

When installing UPS systems, there are three strategies for consideration: The first, and historically most common, approach is a single large UPS system, in a central location. It will typically be in the phone or computer room and will supply power to all critical systems, as well as a requisite number of agent stations and perhaps some minimal overhead lighting. The length of run time is determined by the number of batteries and the load, or amount of current that the devices plugged into it require. The advantage of a single centralized system is that there is a common maintenance site, while the disadvantage is that it provides a single point of failure. Also, an electrician is generally needed to install these larger units and rewire the circuits that need protection.

The second methodology is the distributed approach. Here, multiple smaller UPSs are installed throughout the call center and equipment rooms, at the point of need, to provide back up power to a small number of devices or even a single unit. Often, one or two agent stations will be serviced by one small UPS. The advantages are that they are easy to install (just plug them into an outlet and plug the computers into them) and there is no single point of failure. The disadvantages are that with multiple units, maintenance is often harder to track and more time consuming. Smaller UPS units can be found from a variety of sources. “Amazingly, Best Buy has a large selection of individual usage UPSs,” stated Thomas D. Larvin, President, Creative Switching Designs, Inc. “They have saved us numerous times during power fluctuations, [which are] common here in Houston.”

The third strategy is the combination approach. Since each UPS system is rated for a certain range of load (the amount of power it can supply) as call centers grow and more things are connected to a main UPS, often its rating can be exceeded. This can cause a UPS to underperform or to not work at all. Rather than discard an existing system that otherwise is in fine working condition and replacing it with something bigger and more expensive, an option is to supplement it with secondary UPS or a number smaller distributed units. In this way the original investment is maintained, while increasing the overall power protection.

Generators for long-term protections: Once the batteries in a UPS run down, it can no longer power the equipment that is connected to it, so it too shuts down. For this reason, most modern call centers also have a generator to cover power needs for longer durations. Generators are essentially an engine, powered by gasoline, natural gas, or propane, which produces an AC voltage, similar to the electrical utility. Since they are an engine, they must be started and the voltage output must be given time to stabilize. This generally takes two to five minutes to stabilize.

Generators are run in parallel to the utility company lines, but are not connected to them. To use a generator’s power, the power company must be disconnected and then the generator connected. This sounds complicated, but it is easily accomplished with a transfer switch. A transfer switch, which must be installed by an electrician, can be either manual or automatic. While manual switches are inexpensive, someone must activate it and know how and when to do so. In the midst of a power outage, this seemingly simple task can introduce some unexpected problems. An automatic transfer switch handles all of this without human intervention. It is recommended for all but the most basic (and non-critical) installations.

The question of which generator fuel source is ideal can be a perplexing issue, but the answer is simple. Merely find out which fuel source is the most readily available and practical in your area.

Too many people have made the mistake of buying a generator that is too small. Most experts use the rule of thumb that your generator should have three times the capacity of your UPS and any other non-UPS loads that it needs to handle. The reason for this is so that the generator can handle the peak current requirements when devices cycle up and when the generator first comes on line, switching from no load to full load.

Maintenance and testing: UPSs and generators are not pieces of equipment that you buy, install, and then forget about. If you neglect them, they will fail, but you won’t know until you need them because of a power outage. Then, you are no better off than if you hadn’t purchased them in the first place.

For UPSs, the primary maintenance issue is to periodically replace the batteries. Closely follow the manufacturers’ recommendations on this. Yes, you may end up replacing batteries that still have some life left in them, but you don’t want them to fail when you need them most. Two to five years are commonly recommended intervals, even if you have never needed the UPS. It is also good to periodically test UPSs to make sure they do in fact work. Schedule testing during a non-critical time, so that if there is a problem, the impact is minimal.

Generators, just like cars, need to have their oil changed, filters cleaned or replaced, and battery periodically replaced. Again, adhere to the manufacturer’s recommended maintenance schedule. The maximum time between maintenance is annually, even if the generator has not been used. Maintenance needs to be done more frequently with heavy usage.

Many generators come with an “exerciser” module that will automatically start the generator each week and let it run for about 10 to 20 minutes. If the generator doesn’t start, runs poorly, or produces inadequate output, a warning will be produced. This allows users to seek the necessary repairs before the generator is needed for a real power outage.

(Also see Battery Recycling, Backup Power Vendors, and Power Glossary.)

Peter Lyle DeHaan, PhD, is the publisher and editor-in-chief of Connections Magazine. He’s a passionate wordsmith whose goal is to change the world one word at a time.

[From Connection MagazineNovember 2003]

The Phoenix

By Frank McKeown

We operate a small inbound call center near the University of South Florida in Tampa, Florida. Most of our CSRs are students at the University. We service national clients by providing order taking, help desk and other customer service functions.

On September 11 we witnessed the national horror, outrage, and sorrow. Our mourning was interrupted later that day when we were selected to participate in the national Red Cross Call Center Grid to accept donations and pledges and provide information on blood donations. The outpouring of support from people of all situations made us extremely proud of our country. We received individual donations ranging from $3 to $300,000, all given with a fervent prayer for America and the victims’ families. We tripled our normal staff and still had calls in queue. What a great show of support for the country!

In the middle of this campaign I was awakened at 5:00 A.M. on Friday September 14 by one of my employees who told me that our building was on fire. I arrived at the scene in the middle of a virtual tropical storm, with four fire engines training water cannons on our building.

First, we ascertained that none of our people were hurt. Fortunately, we had a disaster recovery plan with another center who also used the PI-2000 Order Entry System. They had all of our account programming and had assigned DIDs for each account. While the firemen finished their job, I was on my cell with the owner of the other center and the long distance carrier to transfer our lines. We were out of business for only 37 minutes.

The next challenge to overcome was to relocate our center. Our landlord was at the scene and we negotiated immediate occupancy of another vacant suite in the same complex. We then not only persuaded the fire captain to allow us to enter the burned out suite to get our computers, telephony system, furniture, and records, but the firemen actually helped us. A “Fire Recovery” team suddenly appeared (they monitor the police band); we were able to persuade our landlord (who owned the business that had caused the fire) to hire this roving band to help move our gear.

In our new location we had to get back in business immediately because we were unable to transfer the Red Cross calls and felt an obligation to resume this support as soon as possible. We kept all management and labor on duty for the weekend. Everybody understood the importance of this and all volunteered to work as long as necessary. Our folks worked from 7:00 A.M. to 10 P.M. Friday, Saturday, and Sunday. We also needed our computer and telephone vendors to help. They arrived Friday morning and virtually lived with us for the entire weekend, re-establishing our networks and systems. They ordered new equipment, on credit, for weekend overnight delivery and agreed to wait for the insurance settlement to receive payment.

The insurance company contacted me on Saturday and told me not to move, replace, or re-build any equipment until an adjuster could inspect our situation in ten days. After making her aware of our need to minimize our business loss, the obligation we had to the Red Cross, and the fact that we were about to be interviewed by two local TV stations, she dispatched an adjuster Saturday afternoon and agreed that we should rebuild immediately and just provide inventory information to the adjuster.

We were fully back in business Monday at 8:00 A.M.

While I am extremely proud of our “weekend recovery”, this story was written not to chest beat but to point out some key principles of operation which carried us through this near disaster.

Have A Disaster Recovery Plan: Keep it up-to-date so you can transfer your business immediately without advanced notice.

Share Your Mission With Your Employees and Management: Through good communication, training, and motivation make your employees into your greatest asset.

Treat Your Vendors Professionally: You never know how much you will need them.

Act Like An Owner: It’s your business and when it’s on the line don’t be afraid to do what you have to do to save it.

Be Grateful For Divine Providence: More than ever, this particular week showed us how fortunate we were.

Now, if I can only make the insurance company understand!

[Editor’s note: The Phoenix is a mythical bird that rose from the ashes after being burned.]

[From Connection MagazineMarch 2002]

Disaster Recovery for the Call Center

By Jim Becker

Not every disaster is as horrific as the World Trade Center, but Call Centers can also be affected by lightning, fire, water damages, telephone and electrical outages, and other natural and man-made catastrophes. All can impact your center and especially your clients.

How can you lessen the affect? First of all, good planning and communications. No one expects a problem, but if one occurs, a center should be prepared. Trying to cope with an outage or worse, without prior planning, can be disastrous for a center and its client base.

Where do you start? Start with involving your staff in preparing a detailed plan. They can help identify specific areas of your business and clients that need to be addressed in case of an emergency. Then perform regular system backups. Depending on your center and the type of clients, once a week might be acceptable. Those with “critical” clients, such as medical, specialty service accounts, and emergency-type calls should obviously back up their system more often; maybe even daily. Store the back-up information away from your center. Alert your managers and key staff as to where it is and what to do if there is a need to use the backup.

Power outages due to lightning and power interruptions can be a problem. Make sure you have reliable back-up power. Test the back-up power at least once a month. It is not uncommon for batteries and other parts of the system to fail over time; be sure and replace batteries according to your manufacturer’s recommended schedule. Again, make sure your managers know what to do if you lose power. If the power goes out for extended periods of time, you may have to supplement your battery-based power back-up unit with a generator. Not all off-the-shelf generators will meet the needs of a center. You may need to run the power through a line conditioner to help eliminate power spikes, surges, and variations in voltage.

Also key components in your system could fail. Power spikes, heat, and time take their toll on computer components. Be sure to have spares of critical components like the processor board, power supply, and disk drives. You may even opt for key telephony interface cards for operator station ports, T1, ISDN, or DID trunks, and business lines. You may also want to consider pooling spares with other centers, in your area, who use the same system. Don’t count on counter-to-counter deliveries of key components. Weather and the renewed attention to airport security could make this more difficult or even impossible.

You may consider a small back-up telephone system as a standby unit. If your primary system fails, having a small back-up system can provide your center the opportunity to take calls for key clients. Again, make sure your managers know the procedure if your primary system fails – the procedure to contact your local telephone company to redirect lines to your back-up system or to another location. You may want to set up an agreement with another center that if they have a problem, or if you have a problem, either could redirect lines to the other center so that service would be continued.

Having remote agents may also be part of your plan. Inclement weather could prevent employees from traveling to a central location. Agents who have the ability to work from home would still be able to process calls and provide your clients with the service they expect. Remote stations can work via direct telephone lines or over the Internet. Some system manufactures provide both options. Neither can be set up on short notice. Plan ahead and have the capabilities in place before an emergency so that remote access is available when you need it.

Considering today’s complex computer-based systems, having a service contract in place with your system vendor(s) could be some of the best insurance available. There is nothing more frustrating than having a problem with your system and then finding that technical support is not immediately available or if it is available, it is real expensive.

Keep in touch with your power company and telephone company. Have key contact information readily available. Make sure your managers know how to contact them in case of an emergency.

If you rent or lease space, make sure you work with the property manager so you can get immediate access to the power relay area and the telephone junction area.

Make sure you have adequate insurance to cover fire, water, or electrical problems that could cripple or destroy your system and operations.

Have some working capital available or at least a line of credit with your financial institution to help cover initial expenses in case of an emergency. An insurance policy will not always provide immediate cash necessary to repair or replace damaged facilities and equipment.

Be sure to share your emergency plan with your managers and employees. Keep a copy of the plan with all contact information readily available at your center and another copy off premise. You may need to share your plan with your key accounts and your financial institution. Review the plan every six-to-nine months and update contact information for the telephone, power, building management, insurance, and technical service immediately when it changes.

Having a well thought out and readily available emergency plan can be some of the best insurance your center can have.

You may never need it…but if you do, it could save your business!

Jim Becker is the Vice President of Sales and Marketing at Amtelco. He can be reached at 800-356-9148.

[From Connection Magazine – November 2001]

An Emergency Power Solution for a Small Business

By Don Benton

Late last year I upgraded my 10 operator positions to PC’s. The PC’s consume more power than the terminals I was using and my UPS could no longer handle the additional load. It had to be replaced. I chose an APC Matrix 5000.

After the choice of which UPS to use, the next major decision is, “How may batteries should I buy?” The system comes with two batteries, which would power the system for approximately 30 minutes. Each additional battery will add 15 minutes to the time at a cost of about $500.00 per battery. The difficult question is “How long will the UPS need to power the system?” Since I live in California it appears that I can expect a number of outages and they may last for a while. Add to this the fact that the batteries will last from “3 to 5 years” at which time there will be another considerable expense of replacing batteries. I decided I should look for a generator that would supply the needed power when I am unable to use utility power.

There is a generator on the market that appeared to be an excellent solution to the problem. The Guardian generator made by Generac comes in 6, 8 and 10 kW size. It is a standby generator built to be permanently installed outside. It has a battery starter and all the electronics to sense a power outage, start itself and switch power to the load through a transfer switch which is also supplied. The system comes with all of the hardware and pre-wired cables. It runs on either natural gas or propane. I prefer propane to gas because propane will not get “old” when it is stored. I purchased the 8 kW generator from Northern Tool and Equipment for $3,500.00, which I thought was a good price. Since my purchase the price has been reduced to $2,500.00.

There are some factors which should be considered when selecting a generator. The UPS may have a problem “synchronizing” to the generator. The generator needs to be 2 to 3 times the kVA load drawn from the UPS. This problem can also be relieved by placing a small load on the generator, such as a couple of 100W incandescent light bulbs on a separate circuit to the generator. This load will help the generator to regulate its voltage and frequency before the UPS load is applied.

The use of this generator with a UPS gives you the best of both worlds. The UPS provides power immediately when the utility power fails. The generator senses the loss of power, starts itself and after it gets up to speed the transfer switch is activated and the UPS gets its power from the generator.

For my load the generator will run for about five hours on one 5 gallon propane bottle. The generator has been installed and tested. We have not yet had a rolling blackout, but I now feel like I can leave town without worrying about my system if there is a loss of power.

Don Benton of Custom Communications, Inc. can be reached at 925-373-7482.

[From Connection Magazine – September 2001]