In an instant, all the screens went black and the steady hum from the servers' cooling fans ceased, and with each passing second their precious data was unaccounted for, Peter Dominguez faced the potential loss of money and clientele.
Dominguez is the CEO of Yonkers, N.Y.-based First Network Mortgage and what he experienced was every CEO's worst nightmare. A freak loss of power at his company's building had crashed the Web and e-mail server despite the proper uninterruptible power supply (UPS) being in place.
Following the latest blackout over the holiday break in December, Dominguez decided to implement a disaster recovery plan for First Network Mortgage. The damage, including the loss and subsequent recreation of e-mails that had been acquired during the previous week, had cost the company too much for such an event to happen again.
Dominguez said the company's new building owners had made changes to the electrical system that apparently knocked computers on two floors offline, while simultaneously electrifying a ground line that caused file systems to become corrupted.
"It was like if [the changes] happened over the weekend, they thought it would minimize the effects, but basically all they did was short out two floors of computers," Dominguez said.
Ironically, the outage was an unfortunate case of dÉjÀ vu for Dominguez, who said that a similar loss of power occurred in 2003. A mammoth string of blackouts spread from parts of Ohio to New York City and into parts of Canada in a matter of minutes, as 21 separate power plants stopped producing electricity, including the one that supplied the power to First Network Mortgage.
"Here I am in this giant blackout sitting there all myself and no one to talk to saying, 'what good is all this money,'" Dominguez said.
To make such disruptions don't happen again, Dominguez turned to Arkeia on Linux and the Arkeia Network Backup application because of an existing partnership with Arkeia. His IT shop was running Novell's SuSE Linux Enterprise Server on one server while a second one ran SuSE Linux Openexchange 8. A third back-end server was set aside as a dedicated Arkeia backup server.
"[The Arkeia application] started right up," Dominguez said. "But it was one of those things where I was coming from the area of cloning disks and using USS Dump so it was completely foreign to me."
The main challenge that Dominguez said was involved with implementing a back-end server that ran Arkeia Network Backup was the learning curve.
"It had a nice GUI [interface] but the thing is, you want to have a sense of what's going on behind the scenes," Dominguez said.
However, even with the learning curve, the implementation went off without incident. Dominguez said Arkeia's tech support helped him get his shop up and running. It's only been a month since the electricity surge, but Dominguez said things continue to run smoothly.
"Now we periodically drill with disaster recovery procedures," Dominguez said. He added that if the worst should occur, then it takes only a few hours to recover data, and he no longer has to hire professional recovery firms to retrieve lost e-mails and information.