Requests for “actionable information¹” are recurrent. Actionable information is required at every level by military forces, so that well-informed decisions can be made. However, these requests are difficult to meet within a suitable timeframe, and the information often arrives too late for the execution of specific missions, which are defined by speed, lack of casualties, clear and instant information on the battle field, and a restricted budget. Maximizing and optimizing a mission requires additional and efficient data processing. The data processing chain consists, at first sight, of simple operations: filtering, merging, aggregating, and simple correlation of data. These are simple processing operations, but have to be applied on a different scale with low latency to achieve the level of intelligence required by today’s missions.

Beyond intelligence requirements, the huge quantity of data available on a battlefield is a consequence of the ever increasing use of sensors, technologies essential in warfare such as nano network sensors, tactical sensors, global positioning systems for vehicles and men, and tags/codes and communications.

The military looks for an agile, responsive, and swift reaction, because this is usually a decisive element for victory. But that is not all. A battlefield is not isolated from its geopolitical, social, and cultural environment. The preparation of military action is based on thorough intelligence groundwork requiring the gathering of a large quantity of data from the environment surrounding the battlefield. Asymmetrical wars have reinforced these links. It is therefore also a matter of processing data originating worldwide: from the web, social networks, communication flows; i.e., from a “sea of data,” which provides a digital picture of the world updated every second with hundreds of thousands of events. The COP²—a tool favored by the military when making decisions, giving orders or taking action—is a snapshot that is valued only because it is up-to-date. Such a snapshot is even more difficult to obtain due to primary event flows that are rocketing in terms of quantity and speed.

Even if the Big Data field brings solutions to this new situation, the military knows that technology cannot solve everything on its own. A major risk arises if the data is analyzed by people without experience of analyzing 3’Vs data³ or without the right tools and modes of data visualization. The human factor is crucial when interpreting complex data structures (see also Chapters 17 and 18). Let us imagine that instead of handling a few hundreds or thousands of lines in an Excel spreadsheet with sorting macros and filters, it is necessary to analyze manually several thousand Excel spreadsheets with hundreds of thousands of lines. For instance, a paper by the UK’s Government Business Council estimates that the number of unmanned aerial systems (UAS) in use has risen from 50 a decade ago to “thousands” now, with spending on them increasing from less than $300 million in 2002 to more than $4 billion in 2011. These UAS, it is estimated, have flown more than one million combat hours during this period.

At the more global level, it took years of effort to set up “network centric warfare” (NCW). It is a concept popularized under the name of networked warfare, which appeared in military doctrines at the end of the 20th century. As a term of American origin, it describes a way to conduct military operations by using the capabilities of the information systems and networks available at the time. The main evolution in relation to the prior doctrines concerns the sharing of information. It is about the capacity to link together the different armed forces (Army, Air Force, Navy), as well as armed forces of allied countries, to gather information using drones or satellites and to disseminate it to units in real-time so as to strike quicker and more efficiently. During the past few years, the network centric concept has expanded into the civil and business sector in the United States, and today, it supports the US influence in large companies and international organizations.

Unfortunately, the twilight of the concept began in 2010 for several reasons: the cost of such a system, the complexity of its implementation, and, finally, the nature of complex threats that do not seem, at first sight, to be consistent with the concept. However, a large amount of military needs have been formalized in NCW, and today some elements are extracted by armed forces. These armed forces are also, by indirect means, in possession of a light version of NCW. We feel that the advent of Big Data updates NCW under new auspices. Our objectives are more specifically to allow the rapid rise of networked warfare best practices, at a low cost and adapted to the new threats: agility, swift reaction, and short loops between sensors and effectors.

In this chapter, instead of focusing only on Big Data’s theoretical capacity to meet needs or on evaluating the positive impact of technology on the military profession, we will focus on a very specific vision of a Big Data system rendered usable by integrating use cases and construction within the same thinking process. This chapter proposes a solution, aiming for quick implementation and immediate cost-effective benefits. In a time of austerity in military budgets, it becomes even more interesting to set priorities with results achievable in less than 18 months and then evolve in increments without forgetting the long-term vision. Wanting to design a full Big Data system can be extremely expensive. It is preferable to set priorities and implement them in stages. For example, it is better to make concessions on precision/accuracy rather than computation time, and then with longer calculation time to increase the result’s degree of accuracy. Actions on the ground require a short loop of less than 2 minutes. In that case, any result, however partial, is meaningful, as long as it is delivered swiftly. This is the reason why we are looking to use Big Data systems, which are presently capable of providing results in quasi real-time, and then gradually improve either the precision or the richness of the results.

We will explain how to create a system capable of processing around 10,000 to 1 million events per second. Processing means to filter, enrich, and correlate data sufficiently to obtain results useful on the battlefield. With 3’Vs flows, this system is designed to process an event with a low latency, of about a few seconds, to execute simple or complex business rules, and run machine learning on the fly. We will also explain how to make the system flexible, fault tolerant, and capable of evolving. We shall give a few examples highlighting the advantages of the type of architecture, which will be developed.

In its final stage, a Big Data solution has to be compared to a reactive weapon system that has flexibility and adaptability when faced with rapid threat. It is modular, and it grows incrementally in power and service. The implementation and configuration of thousands of computer nodes is not trivial, but it is possible today. A soldier still finds it difficult to compare the value of a fighter plane and that of a superscalar Big Data datacenter. It is an old debate, illustrated during past conflicts such as World War II, when the tenacious work of a British team of cryptanalysts averted two years of war.

Finally, it is to be noticed that police force intelligence and military intelligence tend to work more closely together than before. In this time of austerity, it is reasonable to assume that the sharing of means and sources of intelligence will become more common within the same country, while agreements with allied private operators will continue to increase. In other words, what we describe here can also be implemented for police intelligence or for the collaboration of both.

The first class (class I) consists of the use cases already known and implemented in conventional applications that are struggling with the amount of data. It is a matter of adapting these applications or running them after applying some other treatments first, so as to reduce the magnitude with different processing treatments. It is also through Big Data batch processing that it is possible to rewrite part of these applications. These technologies are well known in the Big Data field, and a part of them are described later in the technical chapters (see Chapters 9–14).

As noted, the ability to process Big Data opens horizons inaccessible until now. Information, which used to be hidden in the sea of data, becomes accessible thanks to simple or complex treatments run mostly in MapReduce. This first class is well known, and the “conventional” Big Data systems based on large warehouses and batch processing are fairly well mastered. It is, nevertheless, necessary to review the algorithms to recode them in parallel mode. The infrastructures also need to be fully reviewed and resized to receive Big Data treatments.

The increasing number of events (data points) provided by sensors and other means, which describe individuals and their environment improves the digital version of the world. The challenge is to exploit it through Big Data. The different stages of the processing chain decrease the magnitude of the information along with the enrichment process, so that once it has reached the user’s desk, it can be handled, analyzed, and interpreted. This first stage already allows, for instance, the analyst to take all the videos recorded during the MQ-9 Reaper’s flight (not just 5%) and to apply treatments for filtering, shape recognition, analysis of differences between two missions, etc. The treatments, applied to each image of the video, are parallelized. For each image, the results are compared with the subsequent computation cycle, until they provide information of interest for the analyst. Although the treatments do not replace the human eyes and brain, they can highlight specific sequences containing relevant elements. To summarize, only a minute fraction of the video will be viewed by the analyst, but it will be the right one. Nonetheless, these treatments are applied after the flight, with batch processing with a duration that will vary from a few minutes to several hours.

Use cases in class II attempt to exploit the sea of available data within a fixed period. This second class regroups use cases that were inaccessible until now, thanks to technologies able to process flows “on the fly,” i.e., streaming processing. These use cases are characterized by short processing time and short latency. The computation is applied on a subset of data: The most recent data is used in streaming processing. They are kept within a time window that moves continuously, defined by the current time T and T-delta. Delta can vary from a few minutes to a few hours.

Let us return to the video recording of the MQ-9 Reaper’s mission. Imagine that a radio channel enables the retrieval of the images in streaming mode. This flow of images is analyzed in real-time. Each image injected in the Big Data system is processed immediately. As the images are being processed, the results are compared with each previous image. The algorithms of the image analysis are prioritized according to operational priorities: Which strategic information, even incomplete, does the soldier need before taking action? Is it enough to r-program the flight of the drone?

“On the fly” processing provides tactical information, which is at the heart of what is happening on the battlefield and to the action. You need to know about it is at the time the event occurs. Big Data are progressively building up toward the real-time analysis of data. Since 2007, Big Data technologies and systems allow the processing of all data over periods that vary from a few minutes to several hours. Over the past three years, with ever increasing volumes of data, the need of the market is moving toward a demand for reactive systems and short processing times. A need for interactivity with the data is growing beyond batch treatments, because these are not in the “battle rhythm.” This requires system architectures and new technologies, which are gradually emerging, to:

To make the most of real-time mass processing, the implementation of Big Data causes a radical change in the way a need is expressed. This change seems close to the military’s operational concerns.

The collection of structured and unstructured datasets involves reconsidering the way data are gradually distilled through several stages, until information for decision-making is produced. To benefit better from it, it is always important to know whether all the data is necessary. A particular organization of the data in RAM could provide useful results with a sufficient degree of accuracy and quality.

Depending on the case, it is not always necessary to obtain maximum accuracy for “actionable information.” The needs should be recorded and then prioritized.

It is possible to obtain an interesting result without realizing all iterations of the calculation on the whole data. This is why it is desirable to determine and prioritize the needs according to the objective to be achieved.

Faced with the Big Data phenomenon, we are trying to build systems, methods, and processes at the crossroad of the constraints of the military profession and the constraints of the computer systems.

The subclasses in Figure 7.1 are now described from a different angle:

In Figure 7.1, we also note that the separation into subclasses is linked to the localization of the data either in RAM or externally, i.e., stored on disk or accessible through the network on another physical server. The larger the amount of data used for processing located in memory, the quicker the treatments. Conversely, the larger the amount of data stored on disk or remotely, the longer it takes to access it. Depending on this localization, we can notice that the accessible subclasses are different. The size of the data sets used also determines which subclasses are accessible. Obviously, the larger the set, the slower the treatments on a Big Data scale.

A vast majority of these cases has been collected into 20 canonic cases. These 20 cases determine the architecture and the function of components of the system to be built. They are divided into the classes and subclasses described above.

The system we describe later is capable of handling these canonic use cases (see Table 7.1). It is highly likely that more than 80% of a military intelligence agency’s needs can be met by this system. The rest can be met by additional applications included in this system.

In the remaining part of the chapter, we present several canonic use cases to highlight the importance of some components.

One has to consider that this data explosion is also an opportunity, because the world has never been digitalized to such an extent. To mention only one sub-set, our communications and movements, as well as the behaviors of the connected objects that we use, all generate a constantly updated digital image of the world. Not only is the volume of data huge, but reconstituting the real world is difficult; because event sources are diverse, some of those captured are of an uneven quality, are captured with a time lag, or come from different geographical locations.

We have identified an intermediary stage, which consists in rebuilding a coherent digital version of the real world. The construction of a digital version of the world is feasible, if we consider that:

The soldier of today and tomorrow uses more and more connected objects and sensors. Soldiers are now able to shoot at enemies without seeing them directly by using augmented reality glasses connected to his or her viewfinder. It is conceivable that part of the augmented digital world could be superimposed onto these glasses. Within a few seconds, it is possible to provide additional information in the viewfinder by using massive real-time processing to clarify images of newly discovered enemy positions.

The sensors of the soldier’s Nano network not only inform the soldier directly, but they also provide data about the environment that could only be correlated by central systems. From a more thorough perspective:

There are some paradoxes, which are not only constraints but also strengths:

To address these paradoxes, we use a building device. “On the fly” processing or streaming allows the launch of immediate processing on the flow of events describing one part of the real world. The resulting information may be partial or complete, partially reliable or totally reliable. However, at the time, it may prove vital to the success of an operation. Indeed, the early emergence of a piece of tactical information enables the early preparation of the means, of the decision to open fire or not, the prepositioning of forces, the selection of targets, etc. It is therefore natural to design processing, which ranges from the simplest to the most complex and from the quickest to the slowest. For example, upstream of the processing, the data are purged by filtering the nonuseful events, sorting or counting them. By examining only a few fields of the event, it is possible to quickly identify the useful events, so as to regroup and correlate them. At every stage, added value appears, which brings more elements to support the decision.

The batch, streaming, and publishing layers are loosely coupled with strong coherency between updating and data timeline coverage. Beyond the field of streaming capabilities, all layers give access to the 20 canonic uses cases over the whole data. The most generic of capabilities is contained in the equation: Query = F(All Data). We explain hereafter the optimum choices for the layers with the best Big Data open-source software.

rawdata = LOAD ‘myhdfsevent’ AS (srcip, evtdate,dstip);
ipaddresses = FOREACH rawdata GENERATE srcip, evtdate;
grps = GROUP ipaddresses BY srcip;
count = FOREACH grps GENERATE count(ipadresses) as frequency, ipaddresses.srcip;
order = ORDER count BY frequency DESC;
result = LIMIT order 50
 
STORE result INTO ‘output’ USING PigStorage(‘;’);

For these tools to support the heavy load of these data, three options are available:

We recommend option 3. It is necessary to apply part of the business rules upstream in the Big Data system. For conventional data mining tools, this option presents some difficulties on entry with regard to the integration and understanding of the added value and use of intermediate results.

This is the reason why it is recommended that a superscalar datacenter (see Figure 7.4) be made available on the battlefield to drastically shorten the loop between sensors and users and to obtain real-time results with very low latency. Services rendered by the datacenter are so important that any break in the link with a central level becomes critical. Consequently, a loose cooperative mode is envisaged between the main datacenter and the battlefield datacenter, as well as a disconnected operational mode. In that case, there is an impact on the use: any user should be informed of the operational status of the systems used for the conduct of operations.

The Big Data technologies we have briefly described are operational. The architecture of the proposed system combine these technologies to bring them to the attention of our readers and provide pointers on how to build the new generation of Big Data systems, capable of processing the large quantities of data available within a time scale ranging from tenths of a second to several minutes. This chapter opens the way to quickly implemented solutions, sometimes breaking away from the traditional approaches of military intelligence, command systems, and logistics systems.

Breaking away does not mean throwing out existing applications, but rather replacing or building some radically different links, as well as dealing with the change in the approach to the battlefield. This change of approach has the support of the younger soldiers, who are familiar with the digital world and fully used to superposing it onto the real world of action and the triggering of fire. It is also certain that the multiplication of data sources—as previously described—is accompanied by a multiplication of ways to use the results of Big Data processing at all levels in the armed forces. The mobility of sources and terminals is an essential element, which should be factored into the impact of Big Data when exploiting results. Thanks to its potential capability for zero latency, the real-time Big Data system, as described in this chapter, adds decisive sharpness and augmented reality to the battlefield if, but only if, the information is instantly exploited by military analysts in an operation center and by troops close to the frontline.