The case of the dropped packets - Netcom Systems' SmartBits system utilized at Paine Webber - Product Information

Communications News, August, 1999

Tests solve firewall mystery.

Some people treat networks the same way they treat cars--they drive them until something breaks and then try to get it fixed. At PaineWebber, Inc., New York, N.Y., the network is too vital a part of its business to allow anything less than tip-top performance. Traders, brokers, and other specialists rely on the network to get instant access to news and financial data for their customers.

So when traders on the network began to complain about slow application response and dropped messages, Michael Weintraub, divisional vice president in the Information Systems Delivery's Network Architecture group, took action running sophisticated tests on the system.

An intranet, Internet access, and extranet comprise PaineWebber's network. These three components were logically separated from one another by a firewall infrastructure, centered on the Check Point 3.0 product, known for providing a robust, reliable firewall.

PaineWebber runs the firewall on four Sun Enterprise 3000 servers, two in New York City, two in New Jersey.

On the extranet, many vendors bring market data and trading applications to PaineWebber's trading floors and back offices. One of the many important applications is the Open Bloomberg System, which provides an umbrella of applications, the most important of which are real-time market and trading data. "We are a very heavy user of Open Bloomberg. It is one of our most critical and important applications," Weintraub says.

Open Bloomberg uses the UDP/IP protocol. UDP (user datagram protocol) is the vendor's transport-layer protocol of choice. UDP was chosen by the vendor to guarantee accurate, speedy delivery of the market information. For a brokerage, time literally is money. "Time is of the essence. People want to be on time when news surfaces or prices change. It is important for brokers to be up to the moment on the ticker," Weintraub continues.

The system worked well in the conventional WAN (wide area network) routing environment but had problems when it had to pass through the PaineWebber firewall. Bloomberg controls the main server and delivers information over T1s to the PaineWebber firewall, but many of the traders said the information was coming in slow and complaints mounted.

Weintraub began thorough troubleshooting. "With some surprise, we noted a great number of packets were being dropped and not all data requests were being delivered from the trader back to the server. Not every reply from Bloomberg was received by the user. We started investigating," Weintraub continues.

He and his colleagues used conventional methods at first, logging into every piece of network equipment and checking for error messages, but found nothing. Eventually, the only unknown was the firewall on the 3000s. "At that point, we remembered that we had acquired a SmartBits network performance analysis test system from Netcom Systems about a year and a half earlier to test and evaluate switches," he says.

The SmartBits system, which generates full wire-rate traffic, was used to load the firewall and then capture and analyze data movement, checking to assure that all elements on the network were performing properly. In the lab, Weintraub has a setup like the one on the trading floor. He set up the SmartBits and ran the firewall program. The users were proved correct. "We found that, yes, the firewall did begin to drop UDP packets," he says.

This market-data application averages about 3,000 packets per second on the network. Packets are short--200 to 400 bytes--so, at first sight, the load on the firewall did not appear to be excessive. Yet, the testbed found packets dropped at rates similar to what was experienced on the floor.

"What we failed to realize is that the 90 Mbps didn't mean much. It's the number of packets per second which is the limitation," Weintraub says. "We started playing with the configurations of the Sun box and the configurations of the security policy. We realized that the policy, as it was, presented a problem for the system overall," Weintraub says. "When we started reducing the number of rules in the firewall policy, it started to speed things up. The rules seemed to create the biggest bottleneck." There were two problems here: first, the rules were important and go hand-in-hand with the level of security the firewall offers. Second, there was supposed to be little, if any, limitation on the number of policy rules set up, according to the firewall manufacturer. But PaineWebber's particular combination of policy rules, objects, and sequences seemed to create a problem.

Check Point responded promptly with an on-site analysis in Dallas, performed by test-lab technicians who tried to simulate PaineWebber's problem. Netcom pitched in by bringing a portable SmartBits test system to the lab where the exact problem was replicated.

"The goal wasn't to find a problem but to fix it," Weintraub says. The solution was to upgrade the firewall from version 3.0 to 4.0. "By going to 4.0, and without any changes to our existing rules policy, we were able to increase performance dramatically," he says.