Digging Deep

Sea Power, Oct 2005 by Munns, David

The use of data mining reportedly helped unmask a terrorist leader months before 9/11, but there are concerns about coordination and privacy

26 Terabytes of Data

The Navy mines large volumes of data each day, but converting it into intelligence is still the work of human analysts.

* New software tools cannot determine the significance of data.

* An executive office to foster coordination among data mining programs could be helpful.

* Coming soon: Project Rockwell will plumb the depths of news reports.

Recent reports by The New York Times and Fox News that the Pentagon identified 9/11 ring-leader Mohammed Atta as part of a U.S.-based terrorist cell months prior to the attacks on Washington and New York have sparked new interest - and controversy - about the Defense Department's relatively nascent abilities to assess huge volumes of data for patterns of behavior that are indicative of terrorists and their activities.

According to press reports, Atta was identified in early 2000 by several military officers, including Navy Capt. Scott J. Phillpott, who managed a Pentagon program called "Able Danger" that employed an analytical process called "data mining." The process allows intelligence analysts armed with specially designed software to aggregate multiple data sources, such as lists of terrorists and decades of reporting by the Associated Press, and search for specific patterns of behavior, anomalies and relationships. The findings become the basis for refined analyses by intelligence specialists.

The New York Times reported in August that Defense Department lawyers forced three meetings to be canceled where military officials involved with "Able Danger" were to report Atta's name to the FBI after the program identified him. These claims have not been confirmed by the Pentagon.

U.S. Rep. Curt Weldon, R-Pa., who arranged a meeting between the news agencies and Phillpott, released a statement in late August describing the program's objective as "to identify and target al Qaeda on a global basis, and, through the use of cutting-edge technology ... to manipulate, degrade or destroy the global al Qaeda infrastructure."

After the public speculation about "Able Danger," the 9/11 Commission stated Aug. 12 that it had learned about the program in October 2003. Initial informants did not mention Atta or any other future highjackers. In July 2004, a different informant knowledgeable about "Able Danger" told the Commission he had seen Atta's name and photo in another analyst's notes. However, this informant was not able to substantiate that assertion to the satisfaction of the Commission, and "Able Danger" was not mentioned in the Commission's final report.

The alleged identification of Atta has attracted high-profile attention to the potential of data mining technologies and processes as intelligence tools. However, the usage and processes of data mining remain relatively immature in the military arena.

One official told Seapower that coordination of data-mining efforts and requirements between federal agencies should be much improved. Also, implementation and oversight issues remain a key challenge in balancing the use of data-mining tools with privacy concerns.

Data mining is not new. Industry has reaped benefits from it in sectors such as health care, insurance and banking. But the lack of coordination between government agencies sometimes creates barriers that prevent valuable intelligence from reaching the proper authorities.

At the forefront of acquisition and development of Navy data-mining tools are the Space and Naval Warfare Systems Command, the Naval Research Laboratory and the Office of Naval Intelligence (ONI). There is little to no coordination between these commands to acquire data-mining tools in concert, a Navy official said, adding that one of the biggest problems with Navy data-mining tools is the number of various commands working on acquiring these tools, "some of which overlap, and it's not always as well coordinated as it could be."

The official suggested establishing a maritime domain awareness program executive office as a means to "deconflict" some of the divergent acquisition of data-mining tools between commands, which leads to conflicts in data and hardships in comparing data sets. The Navy had no comment on the plausibility of this suggestion.

"There have been times where ONI needed information that existed in other agencies' data sources" and it was not available, the Navy official said. "It's certainly not seamless and it's not as well integrated as it could be. Today, there are still lots of places where things can fall through the cracks and where connections might not be made.

"For example, there is not a single source of, or a single list of, terrorists" that all intelligence commands share, the official said. "If someone boards a ship in the Mediterranean and gets a crew list of people who are on that ship and that ship's en route to the United States, we can take that crew list but we have to run it against multiple lists to see if anybody who's on that ship pops up as a bad guy. ... It could be easy to not check against somebody's database."