Creating blue lasers and BioGlass isn't a nine-to-five job. So, 24/7 uptime is the rule for the IT systems that...
serve the creators of these technological wonders, the scientists at University of Florida's department of Materials Science and Engineering (MSE).
That rule posed a big problem for Ed Bailey, director of information systems, because MSE's full backups took two to three hours, and nightly incremental backups called for up to two hours of downtime.
"Getting a good backup means freezing the state of the data, and that means - for the most part - taking the application offline till the backup is done," Bailey said.
Rapid growth of data stores complicated and intensified Bailey's need for downtime-free backup. MSE's scientists had to use more than a gigabyte of data to create blue lasers, which enable 30 times the digital storage of the original red laser CD, or BioGlass, the first manmade substance to promote bone and tissue growth.
Student-generated data added to the backup burden. MSE offers Bachelor and Master of Science degrees, as well as a Ph.D. The students and scientists overuse, in Bailey's opinion, e-mail as a data backup option.
"We have around 150 gigs of IMAP mail data to backup alone, and it is growing everyday," he said. "E-mail has become a poor man's file server."
Naturally, MSE wants to put most of its money into scientific discovery, not IT systems, so Bailey needed a low cost, no-downtime backup system.
Fortunately, Bailey is well-versed in making an IT budget's ends meet by using open source technologies such as Red Hat Linux, MySQL database software, Cyrus IMAP server and a 240-node Beowulf-style cluster. Three servers run Novell Netware and 25 run Linux, but MSE IT has to support 11 operating systems in all.
"The other operating systems only exist to support scientific applications that are custom made for the operating system and, in most cases, are simply too expensive to upgrade," he said.
With a long list of criteria in hand, Bailey evaluated several backup products, including PeerFS from Radiant Data, HP's Lustre clustering technologies and Veritas Volume Replicator.
PeerFS turned out to be less expensive than the other options, largely because it's designed to work with Linux and IMAP software. PeerFS is a POSIX-compatible, peer-to-peer replicating file system that runs on either SuSE or Red Hat Linux.
Bailey's evaluation showed that PeerFS was reliable and met MSE's need for real-time replication to multiple nodes. PeerFS eliminated the backup window in disk-to-disk backup, he said.
Getting PeerFS up on a server took about 20 minutes; not counting time spent copying live data. Bailey explained how he set up his e-mail server:
1. Added RAID 10 volume to the server.
2. Installed PeerFS and format the new volume with the PeerFS file system
3. Configured the e-mail server to replicate to the backup server and then detached the servers.
4. Synced the live volume and the PeerFS formatted volume with Rsync.
5. During the normal backup window, he froze the data store, re-synced the volumes and then pointed the e-mail server at the PeerFS formatted volume.
6. Re-attached the e-mail server to the backup server and PeerFS proceeded to sync the backup volume with the primary volume.
Besides backing up e-mail, PeerFS backs up data for MSE's 25 MySQL database systems. The databases are used for student and financial data and for MSE's 51 Web sites.
Generally, Bailey was pleased with the implementation process.
"The only change I would like to see is that it be possible to upgrade a file system to work with PeerFS," he said. "That would make the migration process very easy indeed."
A few months down the line, PeerFS has increased uptime.
"PeerFS helps me to provide 24/7 uptime by replicating the data to another server," Bailey said. "I can backup the secondary server without taking the primary server offline, plus the secondary server provides me with cheap near-term storage."