Where Do I Start?

by Craig Foote

Disaster Recovery Planning. Just the sound of it can make a company president cringe, and make the MIS manager want to cry. Naturally, everyone goes along thinking a disaster will not happen to them - which is the first reason that most companies do not have a formalized, tested disaster recovery plan (DRP). But there are important reasons to have such a plan. A disaster never happens on a schedule - it is by definition a sudden, unplanned calamitous event which brings about great damage or loss. In a public company, management can be held responsible if events force a company out of business and no disaster recovery plan was in effect. And it is worth mentioning that insurance companies may give substantial discounts to companies with a plan in place. It has been projected that 43% of all companies that experience a total site loss will never reopen, and that 49% will close within two years.

So what does a manager do? After getting past the reluctance to consider the subject a serious priority, he or she must set to the task of planning for a set of "what if" circumstances never-before experienced. In this article we will provide some guidance for the process of working out a reliable DRP. A complete data recovery plan will include a detailed analysis of all parts of your DP operations, such as checksum protection, disk mirroring, and UPS needs.

To begin with, consider the story of The Case-Hoyt Corporation, headquartered in Rochester, New York, which has a large System/38. This system at one time had communications channels to subsidiary plants in Virginia, North Carolina, Georgia, and another printing factory in New York. When a high-ranking executive was asked what he would do if he were to drive past the office and see his computer room in flames, he replied, "I'd get out of my car, loosen my tie, and walk into the flames!" Anyone who has been assigned the task of picking up the pieces after an unanticipated catastrophe has struck might have considered a similar course of action.

To create an effective plan, one must first and foremost establish support for the philosophy of DRP at the top management level in the organization, and downwards. If a disaster occurs, it is ultimately senior management who will have to answer questions from the shareholder right on down to the fireman about what actions are being taken to handle the situation. It is most important that senior management gets involved.

A team of people should be appointed to oversee the creation of your disaster recovery plan. When the key players are in one room to formulate the framework of a plan, they will tend to represent their own interests, with the goal that the final plan will be complete. However, it may be inevitable that a key area gets overlooked, so consider the plan to be a draft that will evolve as it is tested. Certain activities such as training backup personnel to implement the plan will be ongoing. To get you started, some guidelines are set forth here.

Make sure you conduct regularly scheduled meetings. Appoint someone to take minutes (eventually, the DRP coordinator will perform this task). Write down all suggestions, comments and concerns. Within two months, you should have a DRP team established. Within one month more, each team member should have been assigned their responsibilities. By the fifth month, a draft plan should be under review. In the six-to-seven month timeframe, the plan should have most of the problems worked out, and by month eight it should be ready to test. Spend about three months performing tests, evaluation and documentation. Define what constitutes a disaster and determine who has the authority to declare one. And of course, modify this schedule as you see fit.

The responsibilities of each proposed DRP team member are described below. Small organizations may have to assign more than one task to a particular DRP team member.

Chief Financial Officer or V.P. of Finance - The bottom-line dollar impact of a disaster, whether a short-term hardware failure or a complete catastrophe, should be sized so that critical operations in the business can be identified and assigned priority. In addition to estimating the cost of being unable to do business in any given area, financial experts should size the cost of recovering, with the objective of minimizing costs overall.

DRP Coordinator - A coordinator should be assigned to ensure that a plan is completed. The coordinator can "plan the planning" process: coordinating all plan input, documentation, and training activities. The coordinator works out a schedule for completion and runs activities so that the plan is completed on schedule. All decisions made by other members of the DRP team should be reviewed by the DRP coordinator, who assesses the item's total impact on the project. The coordinator's assignment would also include conducting tests of the DRP, and logging test dates, times and results. Depending on the complexity of the organization and its plan, it may be appropriate to have two or more individuals share this job.

Computer Operations Manager - This individual will determine which hardware is deemed critical to maintaining the business, and the installation time required for the equipment. Another function this person serves is implementing priorities within the plan: determining when and in what order critical jobs will be run after a disaster. The overall selection of which jobs are critical will have been made by the entire DRP team. It is the computer operations manager's responsibility to schedule the jobs to reflect the priorities set by the team.

Project Manager - A member of the DRP team must be responsible for keeping track of the current projects (such as programming, testing and new development) and the current phases of development for each. He must also know the extent to which deadlines will slip if a disaster occurs. This is the responsibility of the project manager. The duties of the project manager should not be confused with the duties of the DRP coordinator, who oversees the entire DRP process. The project manager designs plans for resuming the progress of project development, revises time lines, and re-prioritizes the project list. The project manager can also be made responsible to develop project teams, made up of personnel who are assigned to anticipate the unique problems of specific areas under project development. The project manager may also be involved in reviewing the software and the operating system as a whole, and be responsible for having current software (CPF, SSP, SCPF, and so on) on-hand and off-site.

Communications Manager - All communications needs of the company, including telephone, data communications, audio and even video must be planned for by a person familiar with those requirements. The communications manager assumes the duty of working with each long distance or local carrier in scheduling the repair or maintenance of equipment necessary to get the computer system back on-line.

Data Entry Manager - This person will schedule and prioritize the entry of data into the system and coordinate these activities with the computer operations manager. This operation may require additional staffing and equipment. The data entry manager should work with temporary agencies to assure that personnel will be available as needed in a disaster.

Building Maintenance Personnel - Without the involvement of these personnel, many details could impede a resumption of operations after a disaster. It is possible to have another recovery site internally within the organization. While this may not sound logical or realistic, it is, in fact, the most cost-effective. This group will be in charge of providing electrical power, heating, cooling, and office furniture needed on the new site. The most important job of the building maintenance personnel is to design and help select a new site on which reconstruction of the computer facility could take place. Their input on availability and costs of heating and air conditioning units and electrical power will be instrumental.

Security Personnel - At the time of an actual disaster, security personnel will be busy with a myriad of responsibilities - evacuating personnel, ensuring that buildings may be re-occupied safely, coordinating emergency efforts with fire or law enforcement agencies, investigating evidence of sabotage, and so on. But they can be called upon in the planning stages to provide valuable advice and assistance. In terms of their role in your DRP, security personnel will be involved at a minimum in controlling the passage of personnel according to their security clearances, and enforcing the general security measures normally followed on the premises. Security should also be involved in monitoring tests of your DRP.

Risk Management Personnel - Your organization probably does not maintain "risk managers" on staff under that title - but in fact, all line managers are risk managers in this effort. In this capacity, managers must provide support and specific input to the CEO and CFO to help determine the amount of risk associated with any one situation, and to estimate its direct ability to stall a return to normal on-line business operations.

Outside Consultant - An outside disaster recovery consultant can check for "loopholes" in your plan and should be able to provide an objective view of the process as a whole. It may be a good "check" on the credentials of the consultant you are considering to ascertain what experience the individual has with real-life disaster recovery efforts.

Purchasing Agent - The DRP is dependent on the services of a purchasing agent to ensure that critical supplies for the business operation are maintained. Supplies should include everything from paper stocks, diskettes and tape cartridges, to computer forms such as payroll and accounts payable checks. Supplemental supply should be stored offsite for disaster purposes only.

Local IBM Sales Representative or Branch Manager - This person may be able to provide a variety of kinds of assistance, but most importantly, he or she is made part of the DRP team to facilitate the replacement of the CPU as quickly as possible. This individual is the direct contact between you and the IBM manufacturing plant. Of course, if you purchased your hardware from a third-party vendor, you would choose a representative from that company.

Each of these team members should be assigned specific responsibilities in the planning and execution of a DRP. The same individuals who have created the plan will have a grasp on these matters when it is time to perform tests, and ultimately, to recover your site after a disaster has struck. Assigning alternate personnel and training them on the DRP is just as important as having a primary DRP team - and it is a good idea to rotate personnel in actual testing modes to practice recovery. In the event of a disaster, the availability of individual people is uncertain. Practicing with the aid of the most qualified, and later without them, may be a valuable exercise.

After you have decided who should contribute to the disaster recovery plan, you must define your plan's purpose and scope. Generally, disaster recovery planning can be seen as the process of continually providing assurance of the ability to continue processing data and providing management information in the event of disruption of normal facilities. Your DRP should document current responsibilities for what actions must be taken and by whom, before, during, and after a disruption occurring at any level. Your plan should specify requirements for:

A) Information availability - how quickly can data be recaptured for input into the system? What is its timeliness? When is it no longer valid? B) Critical processing - what jobs MUST be done to keep the business functioning and supply senior management with vital information? C) Timing - in what order will jobs be run? How will running jobs out of the usual order affect validity of files? D) Providing critical resources - personnel, software and hardware must be allocated properly to ensure that critical areas of the business are addressed first. E) Planning, conducting, and documenting results of tests - DRP tests should be conducted regularly until all of the "bugs" are worked out. Each success and failure should be documented fully and corrections put in place as needed. F) Assuring that the DRP is kept current as systems, equipment, and personnel changes occur - new equipment means new procedures. New software means updated source files. New personnel require training about the system. The DRP plan must be updated to keep reasonably current with all changes.

After documenting the initial recovery, the plan should prescribe an orderly return to full processing capabilities and standard operating procedure, planned to occur over a timeline of days, weeks or months.

The needs of each company vary, and thus the approach to a disaster recovery plan will vary as well. For example, hospitals with a large-scale AS/400 or System/36 or S/38 typically demand up-time as high as 21-22.5 hours per day. But small offices with a Baby/36 can probably afford to be down for a longer period of time. The characteristics of your business (budget, location, and so on) should be considered by your DRP team.

For the all-important part of your DRP, namely, what approach you will use to resume operations when your facility is disrupted, various options are available to you:

Reciprocal Agreements - This concept requires that you sign an agreement to run your most critical jobs, defined in your DRP Handbook, on another system similar to yours, in your area. In turn, you allow your partner to use your facility if they experience a disaster.

Pro's - All around, this is the most inexpensive way to provide recovery of minimal operations. The key word here is MINIMAL.

Con's - Eventually your reciprocal partner needs the machine back, so you must plan to complete only critical processing. Minimal time could be remaining after files are restored and indexed file keys are built. You're also in danger of crashing the borrowed backup system, and if that occurs, who is the first person to be blamed? Your company may be setting itself up for a big lawsuit. If the goal is to return to normal processing as soon as possible, this alternative is the least effective in that it defeats the purpose: all of the emphasis is on what should be done in a short amount of time, and no progress is made on resuming "normal" processing. Until the original facility is restored or duplicated, normal operations are diminished.

Hot Sites - This concept is based upon buying time at an alternate facility that has the equipment and resources to recover your business functions affected by a disaster. Hot sites vary in the type of facilities offered, such as types of DP equipment or communications equipment.

Pro's - A 'hot site coordinator' will work with your DRP team in designing a plan to accommodate your company's needs. The system is available on the vendor's site, and, nine times out of ten, the system is ready and waiting for your use. (That one time out of ten is another story.)

Con's - This means of recovery is great at the outset of "starting over," but stop there. 'Hot sites' contract for periods of time. COMDISCO, Leasing Dynamics Inc., XL/Datacomp, and others plan on your company staying for awhile, but the day may come too soon for your preference when your agreement must end.

And what happens if two companies experience a disaster at the same time - who gets the machine?

Which brings us to what's behind door number three: most 'hot sites' have one machine. Let's assume you are a large AS/400 shop running a model B50 with 40MG of memory. The 'hot site' has a model B35 with 30MG of memory. Processing times are going to decrease simply because of the smaller size of their machine. If you are fortunate enough to find a hot site with multiple CPUs, congratulations. Another problem is that you are eventually going to have to replace your old system. You can't buy the one you are currently using, so you probably will have to wait. The precious commodity of time is slipping away, and with it, the prospect for resuming normal business in a short period of time.

If 'hot sites' are the way to go for your company, spec out the facility, and ask questions about their machine. Find out its size, capacity, availability, rates, and most importantly, how multiple disasters are handled. This means is far better than the reciprocal agreement in that more normal processing utines and schedules can be met.

Cold Sites - This concept is based upon buying time at an alternate facility that DOES NOT have any resources or equipment, except raised flooring and air-conditioning. Cold sites vary; some have UPS systems, communications facilities, and so on. You may also contract for a specific machine type (generally, one similar to your company's system) and some consulting services.

Pro's - With this type of arrangement you can see exactly what you are paying for, so you have exact control of expenses. You are contracting for a machine, facility, and consulting expertise. The 'cold site' owners many times are VARs, IBM Business Partners, or other new-and-used equipment dealers. After you have established what system you need, you can contract for it and then buy or lease that machine outright and place it back in your home installation at a future date. This save a lot of time in backing up your files after you have completed processing and have restored them to a new machine.

Con's - One major drawback to this approach is that a system may not be immediately available to your company when a disaster occurs. Most companies offering cold sites will stipulate the length of time it will take to deliver a suitable system to the cold site. Usually, it ranges from 12 to 48 hours. Once the system is delivered to the cold site, you may have to load the operating system and perform configuration tasks which could add costly time to your recovery - time you usually do not have during a disaster.

Mobile Disaster Recovery Systems - A system is delivered to your site to use after a disaster has occurred. Depending upon the condition of your facility, the new system can be set up in existing facilities, or if your facilities are unusable, some disaster recovery vendors offer mobile data centers. Mobile data centers are self-contained, fully functional data processing centers that can be taken to your company and used until your facilities are running again.

Pro's - This system will allow you to run your most critical jobs in a hurry. It will also eliminate the immediate need for an alternative site (i.e., a physical building) as the mobile data center is specially equipped with lighting, air conditioning, and power. Perhaps the greatest advantage to this approach is that your end user routines will be least disrupted. Because you will not have to move your end users to another facility, you minimize user productivity losses.

Con's - Many of theses machines are "on the road" most of the time. Their availability is not reliable. Because most mobile disaster recovery systems are most by truck, verify with your disaster recovery vendor that your disaster recovery system is not located in an entirely different geographical area of the country. A contract can be established to have one arrive within a specified timeframe, but this guarantee will sometimes be made at an additional cost to the customer. And such an arrangement does not solve the problem of a machine replacement, since this machine will be back "on the road" as soon as your contract expires or you company is operational again.

Alternative Sites - This is the best approach to disaster recovery. If you are fortunate enough to be running a multiple-site shop, you can transfer your data from one system to another. Or if your company has money to burn, you can place one machine in development and one in production and switch from one system to another as the need arises. On the System/38 or AS/400, files and programs can be transferred via APPC or some other communications node on a regular basis.

Pro's - This method of operation can be the most useful in disaster situations. It provides greater flexibility and minimal mean downtime for production.

Con's - This method is clearly expensive. Having two CPU's at your disposal is a wonderful idea, but the costs for duplicate or compatible systems are enormous. For those companies which are serious about Disaster Recovery (such as hospitals, insurance companies, credit bureaus, banks, and government agencies), the need justifies the expense. This concept will allow you to make by far the easiest transition back to normal processing, compared with other DRP alternatives. It also gives the company the best possible means of a fast and complete recovery.

Once you have determined which alternative suits your needs, it is time to start planning and testing. Start putting some of your ideas down on paper. Talk to your top management and see what their ideas are. Start finding out who will be team members. Most importantly, GET STARTED!

 ------------------------------------------------------------------------------- Some Disaster Recovery Service Providers 

The following companies provide disaster recovery services which may be useful for the IBM midrange system user. The list is not exhaustive - there are other companies in the business - but we have tried to include those which are appropriate for the midrange market. These companies provide consulting services, PC-based planning aids, alternative site locations with IBM equipment, or miscellaneous disaster-related services. We wish to thank the staff of the Disaster Recovery Journal, a quarterly magazine specializing in corporate recovery planning, published in St. Louis, Missouri, for some assistance with this list of resources.

 Advanced Information Management, Inc., Woodbridge VA (703) 643-1002 Aeroscopic Engineers, Inc., Los Angeles CA (213) 245-3024 Agway Data Services, Syracuse NY (315) 449-6523 Andersen Consulting, Chicago IL (800) 637-5244 ( CK Classifieds) Arcus Data Security, Inc., Union City CA (800) GO ARCUS ARMA International, Prairie Village KS (913) 341-3808 AT&T Crisis Management Services, Warren NJ (800) 447-0012 Bekeen Computer Corp., Mississauga, ON, Canada (416) 826-7780 BMS Catastrophe, Inc., Fort Worth TX (800) 433-2940 CHI/COR Information Management, Inc., Chicago IL (800) 448-TRPS Comdisco Disaster Recovery Services, Inc., Rosemont IL (708) 698-3000 Computer Performance, Inc., Tolland CT (203) 872-1672 Computer Security Consultants, Inc., Ridgefield CT (203) 431-8720 ComputerSite Engineering, Danvers MA (508) 750-6700 Continental Computer Assurance Corp., Newton PA (215) 968-6000 Contingency Planning & Recovery Institute, Wellesley Hills MA (617) 235-2895 Contingency Planning Reseach, Inc., Jericho NY (516) 997-1100 Data Base Recovery Services, Inc., Columbia MD (301) 995-6433 Dataguard Recovery Services, Inc., Louisville KY (800) 325-3977 Data Processing Security, Fort Worth TX (817) 457-9400 DataSafe Recovery Services Ltd., Calgary, Alberta Canada (403) 236-1066 Disaster Masters, Inc., New York NY (212) 724-9225 Disaster Recovery, Inc., Minneapolis MN (612) 823-0416 Disaster Recovery Services, Inc., Long Beach CA (213) 432-0559 Disaster Survival Planning, Inc., Port Hueneme CA (805) 984-9547 Document Reprocessors, San Francisco CA (800) 4 DRYING E-Net, Greenbrae CA (415) 925-1888 EDP Security, Inc., Littleton MA (508) 486-8080 El Camino Recovery Services, Houston TX (713) 588-4901 First Recovery, Inc., Troy NC (919) 576-0901 Health Plus, San Francisco CA (415) 864-2969 Hotsite (r) Division of CompuSource, Cary NC (919) 460-1234 IBM Corp., National Services Div., Franklin Lakes NJ (201) 848-4243 Integrity Solutions, Inc., Littleton CO (800) 289-9900 K. N. Myers & Associates, Annapolis MD (301) 224-0900 L/A Distributing Company, Brooklyn Park MN (612) 560-0203 LDI Disaster Recovery Corp., Solon OH (216) 248-0991 McGladrey & Pullen, St. Paul MN (612) 293-8440 M-PLUS Consulting Services, Arlington Heights IL (800) 35 MPLUS NCS Contingency Services, Roswell GA (404) 740-2460 National Underground Storage, Inc., Boyers PA (412) 794-8474 Newtrend Disaster Recovery Services, Orlando FL (407) 880-9050 Page Assured Systems, Inc., Fort Lee NJ (201) 767-9411 Perm-A-Store, Inc., Wichita KS (800) 366-7535 Perrson Associates, Deerfield IL (708) 940-1660 Premiere Network Services, Inc., Dallas TX (214) 733-6870 Provident Recovery Systems, Cary NC (919) 481-0011 Raymond Professional Management, Inc., Alpharetta GA (404) 641-9264 Recovery Management, Inc., Littleton MA (508) 486-8866 Resource Management Associates, Inc., Plantation FL (305) 742-8882 Rothstein Associates, Inc., Ossining NY (914) 941-6867 Softworks, Inc., Clinton MD (800) 638-9254 Sun Data, Norcross GA (404) 449-6116 Sungard Recovery Services, Wayne PA (215) 341-8700 Survivor Industries, Inc., Newbury Park CA (805) 498-6062 The Systems Audit Group, Inc., Newton MA (617) 332-3496 TAMP Computer Systems, Inc., Merrick NY (516) 623-2038 Thompson Systems Group, Reston VA (703) 709-8646 United Software Security, Inc., Vienna VA (703) 556-0007 Vance International, Oakton VA (703) 385-6754 Velcro USA Specialty Fasteners, Capistrano Beach CA (800) 666-4302 Weyerhaeuser Information Systems, Tacoma WA (800) 654-9347 XL/Datacomp, Inc., Hinsdale IL (708) 323-1200 

Vision Solutions, Inc. is the world’s leading provider of information availability software and services for Windows, Linux, IBM Power Systems and Cloud Computing markets. Vision’s trusted MIMIX® and iTERA™ high availability and disaster recovery brands support business continuity, satisfy compliance requirements and increase productivity. Affordable and easy-to-use, Vision products are backed by certified worldwide 24X7 customer support centers and a global partner network that includes IBM (NYSE:IBM), HP (NYSE:HPQ), Microsoft (NASDAQ:MSFT), VMware (NYSE:VMW) and Dell (NASDAQ:DELL). Privately held by Thoma Bravo, Vision Solutions is headquartered in Irvine, California, USA with offices worldwide. For more information, visit visionsolutions.com, search for VSIChannel on popular social networks or call 1 800.957.4511 (toll-free U.S. and Canada) or 801.799.0300.

Also Read

Can You Overcome an IT Outage?

Three Compelling Drivers for Implementing an HA Solution on an IBM i Cloud with MIMIX

Maxava's Enterprise+ "Saves the Ship" for Averitt Express