Packaging the IBM eServer z990 central electronic complex

IBM Journal of Research and Development, May-Jul 2004 by Parrilla, J C, Bosco, F E, Corbin, J S, Loparco, J J, Et al

The z990 eServer(TM) central electronic complex (CEC) houses four multichip-module-based processor units instead of one, as in the previous-generation z900 eServer. The multichip module (MCM) input/output pin density in z990 processor units is more than twice that of the MCMs in z900 processor units. This increase in packaging density and the consequent tripling of the current drawn by the processor units were accommodated by the first-time use of land grid array (LGA) MCM-to-board interconnections in an IBM zSeries� eServer. This was done by using innovative refrigeration cooling of the MCM with air cooling as backup, and by a new mechanical packaging and power distribution scheme. This paper describes the mechanical engineering of the CEC cage, the LGA MCM-to-board interconnection scheme, and the mechanical isolation of the MCM evaporator-heat-sink mass from the LGA contacts. The paper also describes the electrical power and the cooling solutions implemented to meet the more demanding requirements of the denser CEC package.

Introduction

The IBM z990 eServcr*, an evolution of the z900 eServer [1], consists of two frames, A and Z, as shown in Figure 1. Frame Z houses the bulk power assemblies (BPAs) which convert the utility power into 350 Vdc. The 350 Vdc is distributed throughout the system to point-of-load converters, called distributed converter assemblies (DCAs), which convert the 350 Vdc to dc low voltages required by the input/output (I/O) and the CEC cages. The BPAs also feed the 350 Vdc power to the drive circuits for the compressors in the modular refrigeration units (MRUs) and the blowers in the air-moving assemblies (AMAs). The Z frame also houses up to two I/O cages and the associated AMAs. The A frame houses the CEC cage, an I/O cage, two MRUs, and two AMAs.

A CEC is a specialized device that performs all of the high-speed computing associated with a computer. The z990 CEC cage, while not much changed in overall dimensions compared with that in the z900 eServer, contains up to four processor unit (PU) books, each PU book containing one multichip module (MCM). In comparison, the z900 CEC cage houses only one MCM. The z990 MCMs have 2.3 (= U.6/0.26) times higher I/O density than the z900 MCMs, as shown in Table 1. As a consequence, the z990 cage requirement for electric current drawn is triple that of the z900 cage.

This paper describes the z990 eServer power, packaging, and cooling solutions that overcome the challenge of quadrupling the number of MCMs in a CEC cage, more than doubling the MCM I/O density and tripling the current draw by the CEC cage compared with its z900 eServer predecessor.

PU books and the CEC cage

To meet the on-demand needs of today's business environment, the z990 eServer provides a significant increase in system scalability over its z900 predecessor by taking an MCM processor-in-book design approach with a CEC cage housing up to four PU books. From a processor and memory perspective, the z990 PU book design approach yields four times the performance of the zQOO in a comparable 24-in.-wide (610-mm-widc) rack. Each book contains an MCM, two memory books, each with up to 64 GB of memory, and up to 12 new high-performance enhanced self-timing interconnects (eSTIs) for communication with peripheral devices. To achieve efficient packaging density and performance, a hybrid refrigeration cooling system was developed that features a specially designed evaporator-finned-hcat-sink combination attached to the MCM hat [2]. The PU book is fully surrounded by a plated steel sheet enclosure for effective immunity from electromagnetic interference. The PU book (Figure 2) delivers balanced system performance in the new on-demand era of e-business.

The CEC cage provides the physical structure and the interconnections to support up to four PU books at the front portion of the enclosure with their associated DCAs at the rear. The general design and packaging approach was governed by many factors, including functional density, processor timing, system cooling, electromagnetic compatibility, shipping shock and vibration, cost, and ease of assembly and service. To manage these packaging considerations, the CEC enclosure, with its modular design, has been mounted in a standard 24-in.-wide (610-mm-wide) rack. The z990 CEC cage mechanically supports separate modules for the PU and the DCA books and occupies 13 standard Electronic Industries Association (EIA) vertical units. Each EIA unit equals 44.5 mm. The cage also contains air inlet and exhaust ducts. The PU and DCA book modules are merged with a center midplanc circuit-board assembly. The center midplane circuit-board assembly consists of a mother board mechanically supported between two aluminum stiffencrs that mechanically stabilize the individual modules into a cohesive structure. The design allows interconnection access to both sides of the midplanc board, provides manageable modules from a human-factors perspective, and minimizes the time required for attaching components and circuit-board assembly while providing a robust structure for the CEC components. An AMA located under the CEC cage houses the blowers. In contrast to previous air-cooling approaches, the AMA is in a completely separate enclosure occupying four EIA vertical units. By making the AMA an independent and separate enclosure, it was possible to optimize the design from a cost and weight perspective. Moreover, the separate-AMA-enclosure approach provided future cooling design enhancements without necessarily affecting the basic CEC structure. The separate AMA enclosure, located under the CEC cage, also acts as a structural pedestal that has proved to be effective in helping to mechanically support a fully populated CEC weighing -250 kg during shipping and earthquake scenarios. The AMA houses two front-located primary blowers to cool the memory and a memory bus adapter within the PU books at the front of the CEC, as well as the DCAs at the rear. Also housed in the rear of the AMA are two backup blowers. Should the modular refrigeration unit (MRU) cooling subsystem fail, the two backup blowers are activated to provide adequate air flow for cooling the processors until the refrigeration system can be serviced.