Orchestrating the Great Global Grid

Software Magazine, August, 2001 by John P. Desmond

Scientific and academic communities are solving large-scale computing challenges, which could speed the evolution of Web services.

In efforts ranging from earthquake engineering to surveying the sky, the first prototypes of a Great Global Grid are being built today. Pioneered by the scientific and academic communities, the Grid is an ongoing effort to create testbeds on top of the Internet to provide access to tremendous volumes of data and computational resources in a practical and secure way.

The Grid Forum sees the potential in working jointly with the Object Management Group (0MG) and the World Wide Web Consortium (W3C) to make their experiences more applicable to the commercial industry.

These Grid experiences and the infrastructure software the Grid community has developed to support its work have the potential to accelerate the evolution of standards for Web services by providing practical lessons on the necessary requirements for robust "virtual enterprises." (See www.globus.org/research/papers/anatomy.pdf for more details.)

While this type of large-scale collaboration seems futuristic in the e-business world, it is a reality today for participants in the Grid. Work is going on in earthquake engineering, experimental physics, and simulation. Infrastructure software has been developed to support distributed data collection, data grids, federated digital libraries, and security.

Examples of ongoing projects follow:

* In a distributed data collection effort, the All Sky Survey project stores 10 Terabytes of data, each a 2MB image of the sky, for five million images in all. This requires 130,000 containers to hold all the images. The data is housed at the San Diego Supercomputer Center and Caltech, and is accessible via the Web.

* An example data grid is the Astronomy Sky Survey, which is an effort to integrate the All Sky Survey with astronomy collections. The total collection includes images of two billion stars, stored in 500 million images. The collection consumes up to 20 Terabytes of space. Some 500 million sets of attributes from one catalog are correlated in various research efforts. The collection is stored on multiple sites.

* In a federated digital library effort, a branch of the National Science Foundation is working on making collections available for classroom use, from kindergarten through university levels.

* In another, the National Archives has been directed to store 700 collections of various federal agencies. The agency is required to "ingest," process, and reference those collections from a central repository. One repository cannot hold all the collections so a catalog tracks the data over multiple sites.

* In a security effort, the Department of Defense is working on a Grid Security Infrastructure, an authentication system that would support the integration of resources across multiple security realms.

* In earthquake engineering, the Network for Earthquake Engineering Simulation is an effort of scientists worldwide to collaborate on what instruments are picking up about movements within the earth.

Where the Grid Began

The Grid Problem is formally defined as the "flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resources."

The earliest efforts in Grid computing were gigabit testbeds between Los Alamos, San Diego, and Pasadena in the early 1990s. Early software projects including Globus (www.globus.org) and application experiments were conducted in the mid to late 1990s. In 2001, major application communities are emerging, major infrastructure deployments are under way, and a rich technology base has been constructed.

The first Grid Forum Workshop (www.gridforum.org) was held in June of 1999 at NASA's Ames Research Center, Moffett Field, Calif. Subsequent meetings have been held on the Chicago campus of Northwestern University, on the University of California San Diego campus, at Microsoft's campus in Redmond, Wash., and at the Verizon Learning Center in Marlborough, Mass. At the last meeting held in the Netherlands in March, 350 people attended, representing 28 countries and 190 organizations.

Potentially, the five-year experience of the Grid Forum participants could speed the evolution of Web services, according to Bob Marcus, chief technology officer of Rogue Wave Software, Boulder, Cob. He was the lead organizer for a Software Services Grid Workshop held as part of the Object Management Group's meeting in Danvers, Mass., in July. "In the next generation of Web services inside companies and across the Internet, you will have lots of different databases and computational resources, with interfaces all describing their own resources so that applications can take advantage of them," Marcus says. "We think this is going to be a key issue in the next five to 10 years of distributed computing." Other organizers included Erick Von Schweber of Cacheon, Paul Kogut of Lockheed-Martin, Shel Sutton of Mitre, and Craig Thompson of Object Services and Consulting. (The proceedings for the Workshop are at www.roguewave.com/groups/CTO/workshop/agenda.htm.)