Most people were introduced to the arcane world of data mining when US National Security Agency contractor Edward Snowden allegedly leaked classified documents that detail how his government uses the technique to track terrorists. The security breach revealed that the US government gathers billions of pieces of data – phone calls, e-mails, photos and videos – from Google, Facebook, Microsoft and other communications giants, then combs through the information for leads on national security threats. The disclosure caused a global uproar over the sanctity of privacy, the need for security, and the perils of government secrecy. People rightfully have been concerned about where governments get data – from all of us – but equal attention has not been paid to what they actually do with it. Here’s a guide to big-data mining, NSA-style.
* Veins of useful information
The concept behind the NSA’s data-mining operation is that this digital information can be analysed to establish connections between people, and these links can generate investigative leads. But in order to examine data, it has to be collected – from everyone. As the data-mining saying goes: to find a needle in a haystack, you first need to build a haystack.
* Data has to be tagged before it’s bagged
Data mining relies on metadata tags that enable algorithms to identify connections. Metadata is data about data – for example, the names and sizes of files on your computer. In the digital world, the label placed on data is called a tag. Tagging data is a necessary first step to data mining because it enables analysts (or the software they use) to classify and organise the information so it can be searched and processed. Tagging also enables analysts to parse the information without examining the contents. This is an important legal point in NSA data mining because the communications of US citizens and lawful permanent resident aliens cannot be examined without a warrant. Metadata on a tag has no such protection, so analysts can use it to identify suspicious behaviour without fear of breaking the law.
Read more in PM’s November 2013 issue – on sale 21 October.