High-density server cooling

Engineered Systems, Sept, 2005 by Christopher Kurkjian, Doug McLellan

Computer equipment in most computer rooms can vary significantly, both in power output and size. The large water cooled mainframe computers of 20 years ago have given way to distributed processing through a collection of networked servers. Although tape and air cooled mainframe computers can be a significant portion of the computer room, often the servers stacked on open racks or cabinets set the design cooling requirement and the design power density of the raised floor.

A server is a computer connected to a network with components similar to the typical PC such as a microprocessor, memory chips, and disc drives. But because the server communicates with other computers, the keyboard and video display are not necessary. Also like the typical PC, the server has a power supply and needs to dissipate heat roughly equal to the total electrical power input to the device.

RACK LOADING

Atypical rack or cabinet is 24- to 30-in. wide, 36- to 44-in. long and 76-in. high and can incorporate up to 42 U of computing equipment (Figure 1).

[FIGURE 1 OMITTED]

"U" is a standard unit indicating the height of the computer server. Each "U" is 1.75 in. A 5 U server is 8.75 in. tall, therefore eight 5 Us fill a 42 U rack. Like most electronic devices, the processing power and space efficiency of the servers has increased considerably in the last several years. As the servers have become more compact and efficient, more servers can fit into the rack or cabinet rack. A casual glance at Figure 2 in ASHRAE's latest publication on data centers (1) would give the impression that the rate of increase heat load per square foot of data equipment appears to be leveling off in 2005.

[FIGURE 2 OMITTED]

On closer scrutiny, the vertical scale is logarithmic, so that a straight line with a positive slope reflects an increasing rate of growth. A 42 U cabinet installed five years ago with three U servers has a total cooling load of 3 kW to 4 kW, about one ton in six sq ft. Today, the same cabinet can be filled with 7 U blade servers having a total power consumption over 20 kW, or more than 5.5 tons of cooling for the same six sq ft.

SATISFYING THE NEEDS FOR HIGHER DENSITY LOADS

Challenged with replacing older lower-powered computers with modern higher-powered equipment, IT and facility engineers are often faced with determining how to install the new equipment within the confines of existing power and cooling systems. Generally, if the servers are spread out by lightly filling the cabinets, cooling and power issues are more easily solved. If the cabinets are filled to capacity with the new high-powered servers, the computer room will have large open areas or "white space." This space cannot be occupied due to lack of power and cooling capability.

A typical corporate data center can have several hundred cabinets. For example, a legacy computer room designed for 400 2.0-kW racks has an equipment-cooling load of 800 kW of cooling. If the legacy servers in the 400 racks are replaced with 200 racks at say 12 kW each, the equipment load increases from less than 250 tons to over 680 tons with half as many racks. If all 400 racks are upgraded to 12 kW, the cooling system capacity climbs to 1,365 tons! When planning for a new computer room, it is imperative to master plan for ultimate power and cooling capability as well as to set an upper limit on the maximum power consumption in a single rack or cabinet.

Overloading an existing system is not the only problem that high-density racks present. Even when cooling systems are sized properly for the high-power consumption cabinets, getting cooling to the servers reside the racks can be difficult. A computer room can be designed for an average load of 60 W/sq ft, but may not adequately cool a load that is not evenly distributed throughout the space. One end of the room may have tape drive equipment with a 20 W/sq ft load, while the other end is filled with high-density cabinets having a load of 120 W/sq ft. The space in between could he left unpopulated, with no load but a few watts per square foot for lights. If the design does not have the flexibility to provide more airflow to the heavily loaded areas, hot spots are unavoidable.

Just evenly distributing the high-density cabinets may not solve all the problems. A row of equipment 3-ft wide and 40-ft long with 20 cabinets each dissipating 10 kW is a lot of heat in a very small area. This row of cabinets can be repeated with 4-ft wide aisle spacing. Ten rows total 200 racks in less than 3,000 sq ft or about 5.3 sq ft/ton. Compare this to a typical office building around 300 to 400 sq ft/ton. The 3,000 sq ft with 200 cabinets needs over 80 cfm/sq ft based upon a 25[degrees]F differential temperature.

Computer rooms have always needed special cooling equipment to meet their high airflow requirements, but even with standard computer room cooling equipment, the high loads can be difficult to cool. Most computer equipment has maximum inlet air temperatures of no more than 95[degrees]. ASHRAE has recommended temperature and relative humidity guidelines to help engineers and facility personnel ensure computing equipment outages are not environmentally related2 (Figure 2).