Microsoft Resubmits Withdrawn Benchmarks - Product Development

ENT, August 16, 2000 by Scott Bekker

Microsoft Corp. is reemerging from the database benchmarking doghouse.

Benchmark results that Redmond unveiled at the February launch of Windows 2000 had shattered all previous performance records on the Transaction Processing Performance Council's (TPC) OLTP performance benchmark, the TPC-C. But the results were disputed.

In late June, the TPC disqualified the Microsoft numbers. Saying the ruling was over a technicality, Microsoft (www.microsoft.com) vowed to resubmit audited results shortly.

In late July, Microsoft announced that it reran the tests, and once again, the benchmarks are posted on TPC's site (www.tpc.org).

The company achieved the record benchmark in February with Windows 2000 Advanced Server and prerelease code of SQL Server 2000 on a cluster of Compaq Computer Corp. (www.compaq.com) eight-processor ProLiant servers. Since then, IBM Corp. (www.ibm.com) reclaimed the top spot in raw performance with Microsoft's help using a cluster of 32 Netfinity servers running Windows 2000 Advanced Server on IBM's DB2 database software.

The resubmitted numbers from Microsoft and Compaq are an improvement over the February results, but remain far behind IBM's achievement.

The SQL Server 2000 on Windows 2000 Advanced Server results are important to Microsoft. For the first time, a Microsoft database on a Microsoft operating system was at the top of the raw performance chart: A rank Microsoft products never came close to competing for before.

To get there, Microsoft and Compaq built a complex cluster using a new feature of SQL Server 2000 called Distributed Partition Views. In that architecture, each server in the cluster controls its own piece of the larger database. Microsoft posted two record results for first and second place on TPC-C performance. One configuration used 12 server nodes, the other used eight nodes. Some observers question whether the architecture is feasible for most IT departments to set up and support.

Microsoft got tripped up when a challenge was mounted within the TPC over a rule that covers the primary key of a database.

The primary key is the field that uniquely identifies a record and often links records in different tables of a relational databases. Some database experts contend that a primary key should be something that never changes, but the challenge came over the inability of SQL Server to update primary keys in the distributed configuration.

There has been some fallout since the TPC decision. IBM found the primary key interpretation surprising, and it needed to add the primary key update capability to DB2 to get its 32-node configuration submitted to the TPC. In the month since the Microsoft results came down, another 10 TPC-C benchmark results dating back to 1996 have been withdrawn. Those tests had been performed using Oracle and Sybase databases.

For the resubmission round of tests, Microsoft and Compaq chose to rerun the original systems with their 550-MHz Pentium III Xeon processors for a slight gain in performance over the original results. A second test on 700 MHz improved the results, but they were well shy of IBM's mark (see chart above).