DAFS: Enabling Data Center Applications - DAFS Collaborative - Industry Trend or Event

Computer Technology Review, Dec, 2000 by Jeffrey Carter, Malt DeBergalis, John Gillono, Arthur Lent

DAFS is a lightweight protocol enabling apps to directly access transport resources.

This is the second in a series of columns authored by members of the DAFS Collaborative, a new industry group formed to create a protocol specification for direct, memory-to-memory data networking.

The Direct Access File System (DAFS) protocol is a new file access method designed to provide application servers with high performance low latency access to shared storage pools over Fibre Channel, Gigabit Ethernet, InfiniBand, and other VT-compliant transports in data center environments.

Designed from the ground up to take full advantage of these next generation interconnect technologies, DAFS is a lightweight protocol that enables applications to directly access transport resources. Consequently, a DAFS-enabled application can transfer data from its application buffers to the network transport, bypassing the operating system while still preserving file semantics. In addition, since DAFS is designed specifically for data center environments, it provides data integrity and availability features such as consistent high speed locking, graceful recovery and fail-over of clients and servers, fencing, and enhanced data recovery.

All of this translates into high-performance file I/O, significantly improved CPU utilization, and greatly reduced system overhead due to data copies,, user/kernel context switches, thread context switches, interrupts, and network protocol processing.

DAFS vs. Traditional File Access Methods

As mentioned above, DAFS greatly reduces the overhead normally associated with file access methods. Figure 1 compares three file access methods: local file system access, network file system access, and DAFS access. In the case of local or network file systems, data is copied from the disk or network subsystem into a buffer cache, and then copied into the application's private buffer. File access over network file systems incurs additional data copies in the networking stack. Some operating systems can bypass the buffer cache copy in certain cases, but all reads over a traditional network file system require at least one data copy.

DAFS has a fundamental advantage over other file access methods when reading data. By using the remote memory addressing capability of transports like VT (Virtual Interface Architecture) and InfiniBand, an application using the DAFS API can read a file without requiring any copies on the client side. Using the "direct" DAFS operations, a client's read or write request causes the DAFS server to issue remote DMA requests back to the client, so data can be transferred to and from a client application's buffers without any CPU overhead at all on the client side. The DAFS write path is also efficient; to avoid extra data copies on write requests, a traditional local or remote file system must lock down the application's I/O buffers before each request. A DAFS client allows an application to register its buffers with the NIC once, which avoids the per-operation registration overhead.

DAFS Client Implementations

Applications can take advantage of these capabilities in several ways. The first is through a user library that implements the DAFS protocol and is loaded as a DLL or shared library. Alternatively, an application may access a DAFS server transparently through a load-able kernel module. These two client implementations are shown in Figure 2.

The first method shown is a user library that implements the DAFS protocol. A user space implementation of DAFS offers the greatest potential for increased I/O performance, by taking advantage of its transport's ability to directly access the networking hardware from user space and providing an appropriate API for asynchronous zero-copy I/O. The DAFS library calls into the kernel to set up an initial end-to-end connection, which can then be safely accessed from user space. The NIC (Network Interface Card) implements the necessary protection to prevent any other processeses from accessing or interfering with the connection. A, so applications can transfer data directly to the NIC, avoiding the overhead of data copies and system call context switches while still enjoying the security benefits of traditional kernel-based networking implementations. The combination of direct initiation of I/O and the elimination of data copies reduces operation latency and increases bulk data transfer throughput.

To maximize performance, the DAFS library exports an API that gives the user application explicit control over NIC DMA access to its address space. The library API also exposes the asynchronous nature of the server interaction, allowing an intelligent application to manage overlapped data transfers.

The disadvantage of the user library approach is its lack of compatibility with the usual system call interface to the host OS file systems, requiring applications to be modified to take advantage of these capabilities. The benefit is optimal performance. This approach, therefore, is intended for high-performance applications that are sensitive to either throughput or latency, or applications that can make use of the extended DAFS services made available through the user API.