On the performance and use of dense servers

IBM Journal of Research and Development, Sep-Nov 2003 by Felter, Wesley M, Keller, Tom W, Kistler, Michael D, Lefurgy, Charles, Et al

Boot-time firmware

We used a version of Linux BIOS [20] to initialize the blade hardware and execute the DHCP protocol to obtain the Linux-DSA kernel image to be booted on the blade. With Linux BIOS, our SDS blades complete the boot process much faster than a traditional server. Even with the Linux-DSA boot sequence over the Ethernet, a typical boot from initiation to an operational Web server takes about twenty seconds. This rapid boot time dramatically reduces the time needed to recover or reconfigure a blade server. It also benefits our PARD policy, since it reduces the amount of extra capacity that must be kept in the active state to handle rapid increases in workload. We also considered fully embedding the OS in electrically erasable programmable read-only memory (EEPROM), which would have reduced boot time to only five seconds, but rejected this alternative because it would require reprogramming the EEPROM for every kernel change. One limitation of our implementation of Linux BIOS and Linux-DSA is that the blade has to run DHCP twice, once at the BIOS level to get the kernel and once at the kernel level to configure the IP information: It would be better to pass the IP information from the BIOS to the kernel and run DHCP only once.

Console and logging over Ethernet

To reduce space and cabling requirements, the SDS blades do not have standard KVM or serial ports. To support normal console access to the SDS blades, we developed console over Ethernet, which extends the recently developed Linux netconsole feature [21]. Here we provide an overview of our console over Ethernet support; a detailed description is available in [22].

We have extended netconsole in three important ways. First, console over Ethernet is built into the kernel and enabled at boot time, and thus starts operation much sooner than netconsole so that messages generated during system boot can be observed remotely. Messages generated prior to network activation are buffered and transmitted once the network is available. We also added console over Ethernet support to our Linux BIOS code to relay power-on self-test and BIOS messages over Ethernet. This is useful for fault isolation. Second, we have added tty support, so that we can connect to the system console through a slightly modified terminal emulator program that allows full console input and output, including special console key combinations, escape sequences, and tty functions. Third, while netconsole transmits messages as UDP packets, our implementation sends messages using a special Ethernet frame type. This means that console over Ethernet does not depend on the correct configuration and operation of the full network stack. A console monitor program receives and displays messages contained in these special Ethernet packets and allows us to selectively monitor any particular blade or all of them.

Much of the disk output generated by the SDS blades during normal operation goes to system and application log files. In the Linux-DSA environment, these log files reside in the per-blade /var file system on the MetaServer and arc accessed by the blades using NFS. This allows each blade to have its own set of log files. Postprocessing techniques can merge these files into a single set if desired. With the exception of the Web-server logs, most of this log information passes through the syslog daemon, which can alternatively be configured to filter and relay the log messages over the network.