Los Alamos Builds Innovative Supercomputer Network With Ultra-Fast 3Com Gigabit Ethernet Switches - 3Com's SuperStack II Switch3900 - Company Operations

Edge: Work-Group Computing Report, August 16, 1999

3Com Corporation (Nasdaq:COMS) Tuesday announced that Los Alamos National Laboratory, Los Alamos, NM, is using 3Com Gigabit Ethernet switches in its Avalon supercomputer. This device, which is among the most powerful computers in the world, was built in-house using off-the-shelf components from 3Com and Digital Equipment Corporation (DEC), now Compaq. As a result, Avalon cost only about $275,000. In addition, the laboratory's Center for Nonlinear Studies (CNLS) has upgraded its local area network (LAN) using 3Com Gigabit Ethernet systems.

Los Alamos is a scientific institution owned by the U.S. Department of Energy. It will use Avalon for research ranging from astrophysics and global climate modeling to materials and weapons technologies. CNLS, one of the research centers operating within the laboratory, is devoted to the identification and study of fundamental nonlinear phenomena and promote their use in applied research. The center coordinates the efforts of the Laboratory's premier researchers from many disciplines and many divisions, mobilizing theoreticians and experimentalists in applied math, statistical physics, solid state and materials physics, polymer chemistry, structural biology and biochemistry.

"Avalon sets a new standard for cost-effective supercomputing," said Michael Warren of Los Alamos National Laboratory's Theoretical Astrophysics Group and Avalon's chief architect. "We're able to process huge amounts of data reliably and without latency, even at supercomputer speeds."

Connectivity lies at the core of Avalon's supercomputer design, known as Beowulf architecture. This approach deploys multiple processors, or nodes, that are networked together. Computations are distributed among the nodes via the network, enabling all the processors to work simultaneously on tasks for much quicker results.

Supercomputing with Gigabit Ethernet Performance

Avalon is powered by a 3Com network based on a Gigabit Ethernet backbone. The supercomputer's nodes are comprised of 144 DEC desktop computers with 533 megahertz, 64-bit processors and 256 megabytes of RAM. In total, Avalon has nearly 40 gigabytes of RAM. The nodes are all linked at full-duplex 100 megabits-per-second (Mbps) Fast Ethernet speeds to four 3Com 36-port SuperStack II 3900 Gigabit Ethernet switches with three 1000 Mbps Gigabit Ethernet uplink modules. Using their Gigabit Ethernet uplinks, these switches connect to a 12-port SuperStack II 9300 Gigabit Ethernet switch, which serves as the network backbone. This robust design features 12 gigabits of connectivity between the SuperStack II 9300 switch and the SuperStack II 3900 switches.

"One of Avalon's key enabling technologies is the ability of the SuperStack II 3900 switch to trunk its three discrete Gigabit Ethernet uplinks into one," said David Neal, systems administrator for Los Alamos' CNLS and a co-developer of Avalon. "This allows each switch to deliver, in effect, a three gigabit link to the SuperStack II 9300 switch, ensuring the network can sustain extremely high traffic volumes between the 144 nodes."

To reduce costs, Avalon's development team deployed the open source Red Hat Linux 5.1 operating system and other software that are freely available on the Internet.

When Avalon was first built last spring, it featured 70 nodes, but the scalability and high-density of the SuperStack II switches permitted Avalon to attain even greater speeds. "Just half the ports on the SuperStack II 3900 switches had processors linked to them, which meant the network was operating at only half its capacity," added Warren. "The wire-speed performance of the 3900s and the full-duplex 12 gigabit backplane of the 9300 let us expand Avalon to 144 processors, which multiplies its computing power."

With 70 nodes, Avalon reached speeds of 19.2 gigaflops, which is 19.2 billion floating point operations per second. At that speed, Avalon was the 315th most powerful computer in the world. With 144 nodes, the supercomputer's performance is expected to approach 60 gigaflops, which will place it within the top one hundred most powerful computing devices. "Avalon began with much more processing power per dollar than any other supercomputer and we improved upon that considerably," said Neal.

Like many supercomputers, Avalon will be used for exotic scientific research, including exploration into the origins of the universe. Avalon, however, demonstrates that Beowulf class supercomputing is accessible to many organizations. "In campus environments, workstations can be linked together and used at night for supercomputing tasks," said Neal. "A high-speed network is essential so the workstations don't compete for bandwidth, undermining performance."

"When we first set up Avalon, we received all the components on a Friday and connected them over the weekend," said Warren. "On Monday, we had one of the most powerful computers on the planet. Supercomputing is no longer super-costly."

Enhancing the Network to Handle Supercomputing