To find meaningful patterns in transactional data, analysts need a lot of it. They must set baselines about what constitutes "normal" behavior versus "suspicious" activity. Administration officials have said that the NSA doesn't intercept the contents of a communication unless officials have a "reasonable" basis to conclude that at least one party is linked to a terrorist organization.
To make any reasonable determination like that, the agency needs hundreds of thousands, or even millions, of call records, preferably as soon as they are created, said a senior person in the defense industry who is familiar with the NSA program and is an expert in the analytical tools used to find patterns and connections. Asked if this means that the NSA program is much broader and less targeted than administration officials have described, the expert replied, "I think that's correct."...
Such was the state of affairs when the NSA started looking for terrorist patterns in a telephonic ocean. So, instead of looking for a tool that could cull through the data, the agency decided to "reverse" the process, starting with the data set and working backward, looking for algorithms that could work with it.
The NSA has made some breakthroughs, the industry expert said, but its solution relies in part on a technological "trick," which he wouldn't disclose. Another data-mining expert, who also asked not to be identified because the NSA's work is classified, said that computer engineers probably started with the telecom companies' call data, looked for patterns, and then wrote algorithms to detect them as they went along, tweaking the algorithms as needed.
As your junior correspondent here at SMC, and no stranger to the spooky world of skullduggery, I am awe-struck by the prescience routinely shown in this pixelated domain by M1.