IP-SAN performance: best practices

Computer Technology Review,  Feb, 2005  by Zophar Sante

• E-mail
• Print
• Link

SAN benefits of improved storage utilization, high availability and data protection are well understood. Today, there are two protocols available for building block-based SANs: FC (Fibre Channel) and iSCSI. Both protocols use SCSI commands generated by the file systems of the servers. These SCSI commands are converted by the iSCSI or FC protocol so they can move through a network to and from a centralized data storage system (usually one or several large disk arrays) where the commands are executed. In the case of FC, the network equipment is specific to the protocol. In the case of iSCSI, the network equipment type is anything that will handle IP packets--1GB Ethernet is the most popular.

There are many IT professionals considering iSCSI-based IP-SANs as a means for centralizing storage for their application servers. As opposed to FC SANs, IP-SANs have the benefit of being based on TCP/IP allowing businesses to use standard Ethernet equipment, NICs, tools and the knowledge base within their IT staff. But when data is being read and written across an IP-SAN instead of to internal disk drives, users are concerned that network latencies will degrade server performance. Similar to an FC SAN, when using an IP-SAN the server, the network and the storage system all play a part in application performance and client satisfaction. It's important to understand how to identify and eliminate latency bottlenecks to ensure superior application performance. In many cases, a properly designed IP-SAN can deliver better performance than internal disk drives.

Attaching a Server to the IP-SAN: Server CPU-Induced Latency

Today, most popular operating systems support an iSCSI software initiator. The iSCSI initiator is software responsible for encapsulating the SCSI command into TCP/IP and placing it onto the network. iSCSI in itself is fairly low CPU intensive software and even during heavy loads uses very little of the CPU power. But TCP/IP can consume noticeable CPU resources. If you want to eliminate latency at the server layer and not dedicate much of the CPU/s for driving IP-SAN traffic, it is recommended to use an iSCSI TCP/IP TOE NIC. A TOE (TCP/IP Offload Engine) NIC is a special interface card specifically designed for interfacing a server to the IP-SAN and will offload iSCSI as well and TCP/IP encapsulation from the server CPU/s. As a general rule, an iSCSI TOE NIC (also called an iSCSI HBA) is recommended if your average CPU utilization before the use of iSCSI is higher than 50% during usual business hours, above 65% during peak use periods, or nearing 75% during backup or mirroring operations. Most iSCSI TOE NICs come with their own initiators so check compatibility with your target operating systems before selecting a TOE. Network boot is also a feature supplied by some iSCSI TOE NICs.

Attaching a Server to the IP-SAN: Saturating the Ethernet Link to the LAN

"Server Fan-in" describes how many servers you can run through a single GB Ethernet port within an IP-SAN before you start experiencing latency caused by excessive storage traffic. Average business servers do not generate more than 50-200 IOPs (input/output operations) toward the storage drives, and in most cases do not generate more than 5MBs of storage traffic. As a general rule, assume a GB connection can support 80MBs of storage traffic and 10,000 IOPs. Using these figures you can attach up to 16 servers on a single GB connection, assuming each server is not generating more than 5MBs/second of storage traffic. It's important to note that iSCSI is encapsulated into TCP/IP and thus any network that supports TCP/IP can be used as part of an IP-SAN to move storage traffic including 10/100 connections, wireless, infrared LAN/MAN/WAN and even the Internet. Naturally, performance across these types of networks will vary greatly depending on connection speed--but all have been tested and work. Form new IP-SAN deployments; 1GB is recommended.

Understanding your server statistics relative to storage I/O, MBs and CPU usage is very important when determining what equipment to purchase and what size to make your IP-SAN solution. Most operating systems have performance monitoring tools and logs you can use to collect statistics on server CPU and storage usage. For example, Windows performance monitoring can be done using "perfmon" and in Linux you can use "sysstat". This information, in addition to your expansion plans, will help you determine the proper configuration and equipment needed to build an IP-SAN solution that will exceed the performance requirements of your business application servers and be able to scale into the future.

IP-SAN Network Speed: Selecting an Ethernet Switch

The main criterion for an Ethernet switch is that the switch is non-blocking. It can be either a layer 2 or layer 3 switch. Having a layer 3 switch is generally preferred for easier SAN management and monitoring. Today's IP networks are extremely fast and scalable. For example, it takes only a quarter second to send and receive a "ping command" from half way around the world. This is miniscule when compared to the time it takes to seek and read a 10K file from any disk drive. In all shared iSCSI/IP storage models, the performance of the network is usually insignificant when compared to the performance of the storage solution (usually 5-10x faster than the storage system).