Back in the mid-1990s, Automated Trading Desk, a financial services and stock trading company in Mount Pleasant, S.C., switched to Linux on almost of its servers and desktops and began living happily ever after.
In the new millennium, however, the fairy tale took a bad turn, as support and product quality from ATD's Linux hardware vendor began to wither. As plot twists would have it, the vendor faltered just when ATD needed top-flight reliability and performance more than ever.
To turn that around, ATD went with a new vendor and a new hardware platform, and kept the fairy tale from turning tragic.
When ATD was founded in 1987, its applications ran mostly Unix and BSD, according to Eric Hunter, ATD's current system administrator. In the early 1990s, the company hired a new software developer who asked why the company was not using Linux. He pointed out some problems related to using Unix and BSD, particularly with lack of flexibility, complex development and cost. He explained that developing on and using Linux was easier. As soon as the IT team tested it, they agreed with him, and ATD has been a Linux shop ever since.
Today, Linux rules on most of ATD's server spread, which consists of about 300 servers and 120 clients.
"We run Linux on 95% or more of our production servers," said Hunter. As for other infrastructure particulars, ATD's network is built on Cisco switches and routers, and its internal network is built on 13 Cisco 6500 series switches (6509s and 6513s) with routing modules. An Oracle database runs on Altus 4200 servers connected to an EMC SAN. Management is done on HP OpenView NMS. All other production software is written internally.
When choosing Linux, ATD didn't go halfway. About 70% of the end-users have Linux desktops. "Even if someone has a Windows box, they typically have a Linux box right beside it, Hunter said. "All of our traders and programmers and most of our IT staff work exclusively on Linux running OpenOffice."
Sharing documents and doing other office processes on OpenOffice is a breeze for Hunter and ATD's users. "Anything that I have to do that has to be done quickly, I do on my Linux box," Hunter said. "I have a Windows desktop here because I have to support our Windows systems; but I would be happy just using Linux, and most of our users agree with me on that." The price makes Hunter and ATD's executives happy, too.
Into every life, however, a little rain must fall.
"When we first started buying from VA Linux, we were getting support and good products," said Hunter. "As time went on, a lot of the products that we were purchasing would come in DOA, or we would have hardware problems very soon after deployment."
He found it difficult to get replacements, on-site service or even a suggestion on how to fix something. "We were usually turned away," he said. "Or, we were told, "We will call you back." Then, we'd never hear from them."
As fate would have it, the timing of the vendor's lapse was terrible. ATD was ramping up its customer trading services, taking orders from individuals. "With that came more complexity and responsibility to be more reliable," said Hunter.
ATD's financial and trading applications use complex statistical algorithms to make predictions on stock movements. "If our systems can't process information fast enough then we are not able to make stock predictions as efficiently, which has a direct impact on profits," said Hunter.
Hunter is talking about thousands of transactions taking place in a matter of minutes. Consider, he said, that ATD's business accounts for about 5% of what the NASDAQ reports in total volume trade. That percentage is increasing, and that puts an extra burden on ATD's systems and system administrators.
"As a result of this growth, we are constantly adding new software pieces and new connections from customers to our infrastructure, and when we do, the new pieces eat up that many more resources from each system," Hunter said. "So, we are constantly having to get new machines to replace ones that just can't keep up anymore."
And those servers and desktops were showing up DOA. So, the search for a new hardware vendor was on. The IT staff evaluated products from almost every vendor who shipped Linux boxes. ATD internal application teams were called in to determine if products were compatible with their technologies. Price was a factor, of course, but so was vendors' ability to respond quickly to service needs.
ATD chose and now uses Penguin Computing to provide and service desktops and servers. "The folks at Penguin have been very helpful in helping us maintain upgrades or replacing parts if need be getting us new servers," said Hunter. "They take responsibility, and we work together to make sure that the systems are running smoothly."
Beside providing support and products, Penguin has helped ATD get 30% to 50% improvements in application performance. ATD's legacy hardware run on Intel Pentium and Xeon processors, but replacements and new boxes with Red Hat Enterprise Linux pre-installed from Penguin use AMD Opteron processors. "We recently moved our Oracle database to a few of the Penguin Opteron servers with equally impressive results," said Hunter.
Next on Hunter's agenda is moving some production applications into a clustered environment to increase redundancy. Fortunately, Linux and clustering are very congenial mates, so Hunter expects ATD's happy life with Linux to go on for ever after.