Pattern Analysis Gains Traction

The hot new thing in government data-mining now is "pattern analysis."

"Link analysis" is such old hat.

The Department of Homeland Security is testing a data-mining program that would attempt to spot terrorists by combing vast amounts of information about average Americans, such as flight and hotel reservations. Similar to a Pentagon program killed by Congress in 2003 over concerns about civil liberties, the new program could take effect as soon as next year.

But researchers testing the system are likely to already have violated privacy laws by reviewing real information, instead of fake data, according to a source familiar with a congressional investigation into the $42.5 million program.

Bearing the unwieldy name Analysis, Dissemination, Visualization, Insight and Semantic Enhancement (ADVISE), the program is on the cutting edge of analytical technology that applies mathematical algorithms to uncover hidden relationships in data. The idea is to troll a vast sea of information, including audio and visual, and extract suspicious people, places and other elements based on their links and behavioral patterns.

The privacy violation, described in a Government Accountability Office report that is due out soon, was one of three by separate government data mining programs, according to the GAO. "Undoubtedly there are likely to be more," GAO Comptroller David M. Walker said in a recent congressional hearing. ...

The issue lies at the heart of the debate over whether pattern-based data mining -- or searching for bad guys without a known suspect -- can succeed without invading people's privacy and violating their civil liberties. ...

The Disruptive Technology Office, a research arm of the intelligence community, is working on another program that would sift through massive amounts of data, such as intelligence reports and communications records, to detect hidden patterns. The program focuses on foreigners. Officials declined to elaborate because it is classified.

Officials at the office of the director of national intelligence stressed that pattern analysis research remains largely theoretical. They said the more effective approach is link analysis, or looking for bad guys based on associations with known suspects.