On the performance and use of dense servers

IBM Journal of Research and Development, Sep-Nov 2003 by Felter, Wesley M, Keller, Tom W, Kistler, Michael D, Lefurgy, Charles, Et al

Up to 32 blades can be mounted in a CompactPCI chassis that supplies power, cooling, and backplane connections to the blades and support hardware mounted in the chassis. The chassis is a 6-U enclosure, where "U" is the unit of height (1.75 inches) for rack-mounted components. Standard data center equipment racks can accommodate 42 U of components. We designed the SDS blade to be 3 U high, and the connections in the backplane of the chassis were slightly modified to allow two blades to be mounted into a single 6-U-high backplane slot. The CompactPCI backplane carries the Ethernet, I2C, and power connections for the blades, eliminating any need for cables to them. We do not currently use the backplane PCI bus. We modified the connections in the backplane slightly in order to accommodate the half-height blades, since the standard connections arc for 6-U-high blades. We chose this packaging format to reduce development time and expense, since it eliminated the need to design a custom mechanical enclosure, reduced the cabling requirements, and allowed us to leverage standard system management and network switching components.

Figure 3 shows a diagram of the resulting configuration. The standard 6-U-high chassis contains slots for two Ethernet switch blades, two system management blades, and four independent power-supply cards. The remaining slots in the chassis are used to hold SDS blades. The Ethernet connections provide the communications among the blades through the Ethernet switches, which in turn connect the chassis to the external network. The I2C bus connections allow the system management blades to communicate with the service processors on each of the blades for control and management.

Although resource constraints allowed us to build only 11 working blades, our design point potentially allows up to 360 systems per standard rack of 42-U height, 19-inch width, and 30-inch depth, including the necessary cooling fans, system management blades, power supplies, and Ethernet switches. The SDS uses conservative packaging for cost reasons, but a more aggressive design could substantially increase the implementation density. The maximum power rating for the rack is within 7 kW, where 5.7 kW of power is allocated for the servers and the remaining power operates the fans, the Ethernet switches, and the power supply. Our design yields a density of 8.57 server blades per 1 U. In raw processing terms, this translates to 180 GHz of x86 processing, 184 GB of main memory, 71.4 Gb/s of Ethernet bandwidth, an aggregate of 92 MB of L2 caches, and 360 independent memory and I/O buses. To put this density in perspective, a rack of 42 traditional high-performance servers that fit in the same space and use contemporary server technology can accommodate raw processing resources of up to 100 GHz of x86 processing, 168 GB of main memory, 84 Gb/s of Ethernet bandwidth, an aggregate of 42 MB of L2 caches, and 42 independent memory and I/O buses.

Our system differs from other dense server designs [1-4] in two important aspects: It does not contain a disk or keyboard, video, and mouse (KVM) connections. Eliminating the disk provides four important advantages. First, it reduces maintenance costs because we eliminate the only source of mechanical failure on the blade. Second, it allows the hardware designer to focus on increasing the design density. Third, it conforms with trends in data centers of consolidating storage into an independent storage area network (SAN) or network attached storage (NAS) for improved reliability and simplified storage management. Fourth, it allows the easy re-tasking of a particular blade by simply pointing its root file system to a different external partition during boot. The elimination of KVM connections also offers several important advantages. The first of these is elimination of cables, which reduces cost, improves reliability, and simplifies server installation and management. The second advantage is improved server density, since the physical connectors and support chips for KVM devices would otherwise reduce space for other components. Elimination of KVM connections is consistent with typical server deployments in modern data centers, where the video capabilities of a server are never used. In fact, the KVM connections seem to be used only when the system administrator has to walk to a malfunctioning machine and connect I/O devices to reboot or reinstall software, which is not a viable solution with the increased blade density.