Across the world, law enforcement and intelligence agencies are facing an increasingly complex range of threats from a myriad of different sources; Nation States, groups and individuals with completely different motivations and a growing arsenal of modus operandi. Whether tackling cross-border organized crime or lone wolf terrorist threats, the need to continually adapt countermeasures continues. Agencies have invested heavily in collection platforms and technologies to capture and analyze information, the volume of which is increasing exponentially. When faced with the many types of data now available, the potential for information overload is high. This is especially true for intelligence professionals charged with turning this data into actionable intelligence to help counter these threats, increasing the risk of missing key pieces of information leading to reputational damage at the least, and at the worst failure to prevent a real and credible threat.

Since the attacks on New York and Washington in September 2001, we have seen growing urgency across the Western world and beyond to tackle the ever-present threat of terrorism, a need that has been underlined in the intervening years by high-profile incidents such as the London suicide attacks of July 7, 2005, Anders Breivik’s twin attacks in Norway in 2011, and the Boston Marathon bombings in the United States in April 2013. Developments in Syria and Iraq indicate that this trend is likely to continue. Add to this the increase in organized criminal activities such as arms trading, human trafficking, and drug smuggling, coupled with tough financial constraints, and it is clear that agencies need to improve their effectiveness whilst driving economic efficiencies.

In the past, most agencies worked with data that they held internally within their organization and the intelligence they were generating themselves in the form of intelligence reports or information gathered from the investigations they were carrying out. In this tightly controlled environment, data growth was largely manageable.

The situation today has changed radically because data volumes have increased in all elements of our lives. The same applies to agencies that now have data available to them that increasingly originate from or are held outside their own organization. In the Internet age, to gain a truly holistic view of activities, new and emerging technology solutions such as social media, automatic number plate recognition, telecommunications, and financial and other sensor data have an increasingly important role in agency life.

Analysts need to call on all of the information available to them and assess it to obtain a clear picture of how a situation is evolving over time. This is increasingly difficult in an environment where adversaries are exploiting the Internet, social media networks and other digital channels to great effect. Analysts often have to review huge volumes of information, looking for that golden nugget of relevant information that could bring an investigation to a positive conclusion.

At the same time, much of the data being generated no longer take the form of easy-to-manage structured data saved in a tabular format within relational databases. Instead, a significant proportion of data are unstructured in the form of word documents, transcripts, witness statements, or Internet content, which presents another key Big Data challenge. These data can be extremely valuable—especially given the latest advances in text analytics to automatically generate keywords and topics, categorize content, manage semantic terms, unearth sentiment, and put all of that into context (see Chapters 11–13).

Overwhelmed by the scale and complexity of the data being generated, most agencies’ legacy databases simply cannot cope. There is a growing acceptance that siloed systems are a thing of the past and that such silos prevent users from seeing the bigger picture that their data could potentially reveal. To help address this challenge, agencies must consider using the latest data storage and processing technology, such as Hadoop, which uses lower-cost commodity hardware to reliably store large quantities of data (see also Chapters 9–11) and supports the power of parallel processing enabling rapid access to the data. The inability of an organisation to access the entire Big data store in a timely manner and identify useful and relevant data inevitably leads to missed opportunities and potentially poor intelligence.

Once the data is available in your ‘data lake’ the ability to carry out comprehensive searches across data sources is a critical part of an analysts Big Data arsenal, but also assumes that there is a starting point. This might be a known organization or individuals who can lead the analyst to further valuable sources of intelligence. When faced with organized threats, national security agencies are frequently in a position to ask such targeted questions about an organization, its methods of operation, who its members are and with whom they interact. Exploring the data available and gathering new data is the key to success. However, in these examples, that success is achieved through asking specific questions about the vast quantities of data available to build the intelligence picture, identifying the known unknowns and then gathering further data to fill in the gaps.

The next step to truly unlocking the Big Data gold mine requires the adoption of high-performance analytics (HPA) solutions, creating the ability to process vast volumes of data quickly and efficiently, to uncover hidden patterns, anomalies, and other useful information that can be used to make better decisions. High-performance analytics has the potential to enable law enforcement or national security agencies to integrate and analyze huge volumes and varieties of data, structured and unstructured, to review and anticipate changes in criminal and terrorist activity. For example, Figure 2.2 shows an analysis that was performed on casualties inflicted by different types of explosive attack. The figure indicates a gradual reduction in numbers of wounded or killed people in all types of attack owing to better preventative measures, but also that improvised devices were still responsible for several hundreds of wounded people. These data can be graphically presented to users in a matter of seconds and manipulated to predict future possibilities.

High-performance analytics delivers the potential to help agencies quickly access and analyze every bit of relevant data and thereby move from a pay and chase approach, in which agencies put in place technology to react to events that have already happened, to a more proactive predict and prevent environment.

The real benefit of applying HPA to Big Data is that agencies do not need to know what they are looking for before they start. Instead, analytical techniques will model the data and push information of interest back to them, drawing attention to relevant content, effectively pushing the needle out from inside the haystack. This information can then be processed through standard analysis and investigation processes to determine whether it is viable intelligence, effectively converting Big Data into actionable intelligence.

However, agencies have become increasingly clear about the value of Big Data in preventing and solving crime and threats to national security. We have seen an ongoing shift in mindsets, in which data are increasingly seen as an opportunity rather than a problem, and the latest technologies are available now to enable agencies to start taking advantage of that opportunity.

Without the right tools, pinpointing relevant data in Big Data that might potentially be useful can be resource-intensive and unaffordable. With the right solutions, irrelevant information can be sifted out and areas of interest highlighted.

Today’s latest analytics technologies enable faster, better decision making by improving analysis of the huge and ever-growing volume of data. The ability to scour Big Data using High Performance analytics and data management will become increasingly crucial in enabling intelligence professionals to reveal hidden insights and produce better decisions. A prerequisite for processing and analyzing all available information is the ability to extract it, preferably automatically, from many different sources and formats and to apply suitable data quality processes. The final step is then to use the insight gained in support of operational effectiveness.