Charley, Frances, Ivan and Jeanne are unpopular names in the Burdock Group's office in Vero Beach, Fla.
It has been four months since the hurricanes hit that office, but IT operations are still not back to normal, said Tim Fritz, IT manager for the toxicology consulting group.
Damage to equipment and data was avoided because Fritz had prepared his IT center for disaster, but Mother Nature outfoxed him in a few ways.
Disaster planning is critical for The Burdock Group because it also has an office not far from the White House. In setting up and managing the company's IT systems, Fritz has to take into account the threat of hurricanes, tornadoes and lightning strikes in Florida and possible terrorism threats in our nation's capital.
For the most part, Fritz handles the design and management of the company's IT systems on his own. The Florida office has three servers and the Washington office has one. Three of the servers run Red Hat Linux Enterprise System and handle file and application serving. One critical application is CG Information's Biblioscape 5.4. One server, in Florida, runs Windows 2000 Server Edition and is dedicated to running Deltek Vision, an accounting and project/contact management software from Deltek Systems Inc. The company's desktops run Windows 2000 SP4 or Windows XP Professional SP1 and Microsoft Office 2000 Professional. Each office has an Inter-Tel phone system and Polycom videoconference system integrated into the network to provide seamless connection and communication between the two offices.
Backup for each office is done automatically, with data being saved on digital linear tape. Thinking that the likelihood of one office and its saved data could be put out of commission, the Burdock Group sought another way to protect its data.
"We [had] to prepare for potential catastrophes before they happened," Fritz said.
After some research, the company decided that data should be instantly replicated between the two sites. "We desired instant backup because our professionals are constantly drafting large technical documents that would be difficult to recreate if we only backed up at the end of the day," Fritz said.
The Burdock Group uses Linux for file and applications serving because it's an economical and relatively maintenance-free platform. Those qualities and support for Linux were important decision criteria for choosing replication software. After looking at several products, the company chose Constant Replicator from Constant Data Inc., Hopkins, Minn.
Constant Replicator had all the right features, Fritz said. It would keep the Washington and Florida servers and about 30 clients in sync in real time over a VPN. Also, the product runs Linux and is both economical and easy to maintain.
The expertise and responsiveness of Constant Data's staff also played a key role in the purchasing decision. The Burdock Group's in-house expertise in implementing applications on Linux was limited. "Fortunately, the support staff at Constant Data was more than willing to talk us through the entire process from start to finish," Fritz said.
Once the selection was made, Constant Replicator was installed on two Linux-based servers for real-time replication between remote sites. "Because we planned for this prior to the January 2004 opening of the second office in D.C., we were able to purchase the second server and have it at the same location as the first server for installation, configuration and testing of the systems," Fritz said. "Once everything was in working order, we shipped the second server to the other office, plugged it in and addressed any minor issues that had not been previously attended to."
As with the implementation of any new hardware or software product, there were some initial problems. "We quickly realized that some of our data files were of such enormous size or were changing on such a frequent basis that Constant Replicator was running nonstop, consuming most of our available bandwidth," Fritz said.
Constant Data's staff found that some of Burdock Group's client data was being backed up on one server every night, but was not being replicated until the next day. These were very large files, and that's why bandwidth was compromised. Two separate jobs were set up: one replicated data in real time, the large backup files were replicated at night, which rectified the problem.
For the most part, Constant Replicator runs in the background, doing its job with little maintenance. "The reliability of Constant Replicator running on a Linux platform translates into peace-of-mind, not only from the perspective that each office has the same real-time data, but also because we know each site serves as a backup of the other in the event of a disaster," Fritz said.
Speaking of disaster, data replication was going along swimmingly until Charley hit. Fritz encased all of the Vero Beach office's IT equipment, including PCs, in several layers of plastic. The office was on the third floor, so he didn't expect flooding to be a problem.
"We were expecting broken windows, and rain coming in from the side," he said.
Instead, the deluge came from the roof. Carpets, furniture and walls were soaked. The ceiling fell in. The whole building became uninhabitable. Ironically, power was restored to the office's neighborhood first, but had to be turned off in that condemned building.
Constant Replicator had replicated all data up until the power outage, so no data was lost. "We were required to establish a temporary office at another location, and we haven't resumed replicating data between the sites," said Fritz, speaking from temporary office last week.
By the end of this month, the Burdock Group will move into a new and more watertight office in Vero Beach. Then, Fritz will be able to restore the VPN and resume replicating data between the two sites. Until then, the staff is keeping data moving and up to date between the two offices using e-mail and tape backups. "It's a little tricky," Fritz said. He's looking forward to getting back to normal.