On the performance and use of dense servers

IBM Journal of Research and Development, Sep-Nov 2003 by Felter, Wesley M, Keller, Tom W, Kistler, Michael D, Lefurgy, Charles, Et al

The MetaServer provides several functions in support of the SDS blades. It acts as a dynamic host configuration protocol (DHCP) server that assigns a fixed IP address to each blade on the basis of its media access control (MAC) address, giving the blade an infinite lease. In addition to the IP address, the DHCP server informs the blade which kernel to boot and what to mount as the root file system. It also serves as a boot server for each of the server blades, transmitting the kernel using the standard trivial file transfer protocol (TFTP). For storage access, we used the Network File System (NFS) for root and other system-related file access, and a new network block driver that we have developed, called the Ethernet block driver (EBD) [16], for swapping. EBD implements a high-performance protocol for block data transfer that is optimized for modern high-speed local area networks. We have used this protocol successfully to support the performance-critical swapping from the server blades running Linux-DSA to and from the MetaServer or an alternative swap server. Finally, the MetaServer is also the master of the I2C bus, allowing it to collect system status from individual blades, issue system resets, select blade operating frequency, and individually power on or off each blade. These features are critical to our cluster-based power management techniques, which are described later in this section.

Fife system layout

To simplify system management and software maintenance while ensuring a uniform level of software across all of the blades, Linux-DSA uses a file system layout that maximizes the amount of standard Linux file space that is shared by all of the blades. Files stored on the MetaServer that are used by the diskless blades are kept in a separate subtree, /dsa. Rather than maintain a separate root file system image on the MetaServer for each blade, files are installed only once in a common root file system, /dsa/root. The root file system used by the blades during runtime is a standard Red Hat Linux 7.1 root file system image. It is mounted on a non-root directory on the MetaServer and is exported using NFS and is mounted read-only by the blades. Thus, the vast majority of the standard system and subsystem binaries, libraries, and configuration files are common across all blades. This greatly simplifies system management, especially software maintenance and upgrades.

However, it is necessary to maintain separate versions of certain files because they contain configuration information that is specific to a particular blade. Each blade also requires its own copy of certain files written during normal system operation such as logs, subsystem lock, and process ID (PID) files. Since the majority of these files are already under the /var directory, Linux-DSA uses a separate /var file system for each blade, which the MetaServer exports read-write to the particular blade. The /var file systems are configured as directories within a single physical file system on the MetaServer. There are a few cases in which the standard Linux implementation writes into a file in the root file system. Most of these are easily handled using symbolic links. However, most of the configuration files, including most of those found in the /etc directory hierarchy, are common across all of the blades. Finally, /tmp space is also provided by the /var file system by linking /tmp to /var/tmp.