Big Data is at the heart of the smart revolution. The basic idea behind the phrase ‘Big Data’ is that everything we do is increasingly leaving a digital trace (or data), which we (and others) can use and analyse to become smarter. The driving forces in this brave new world are access to ever-increasing volumes of data and our ever-increasing technological capability to mine that data for commercial insights.

There is little doubt that Big Data is changing the world. It is already completely transforming the way we live, find love, cure cancer, conduct science, improve performance, run cities and countries and operate business. As a result there is a huge amount of hype and fuss over Big Data. Everyone is discussing it. It is THE hot topic discussed in every boardroom, every business publication from The Economist to Fortune to the Harvard Business Review. Big Data is even making its way into mainstream media.

But despite the noise around Big Data most people still don't really understand it and very few people know what to do about it. Personally, I don't like the term because it's too simplistic and potentially misleading. Granted, we are now tracking and storing data on everything so we potentially do have access to large volumes of data – hence the term Big Data. But the real value is not in the large volumes of data but what we can now do with it. It is not the amount of data that is making the difference but our ability to analyse vast and complex data sets beyond anything we could ever do before. Innovations such as cloud computing combined with improved network speed as well as creative techniques to analyse data have resulted in a new ability to turn vast amounts of complex data into value. What's more, the analysis can now be performed without the need to purchase or build large supercomputers. This means that any business, government body, or indeed anyone can now use Big Data to improve their decision-making.

Especially powerful is our ability to analyse so called ‘unstructured data’ (more on this in Chapter 3). Basically, unstructured data is the data we can't easily store and index in traditional formats or databases and includes email conversations, social media posts, video content, photos, voice recordings, sounds, etc. Combining this messy and complex data with other more traditional data is where a lot of the value lies. Many companies are starting to use Big Data analytics to complement their traditional data analysis in order to get richer and improved insights and make smarter decisions.

In effect what Big Data should really stand for is SMART Data and whilst I think the term Big Data will disappear in time, the increasing production and use of SMART Data is definitely here to stay.

Who is using Big Data?

The big players in the space, including Amazon, Google, Walmart, and Facebook, are already making a splash. Walmart, for example, handles more than a million customer transactions each hour and imports those into databases estimated to contain more than 2.5 petabytes of data.¹ The company is now able to combine data from a variety of sources including customers' past purchases and their mobile phone location data, Walmart internal stock control records, social media and information from external sources such as the weather, and initiate tailored sales promotions. For example, if you have bought any BBQ-related goods from Walmart, happen to be within a 3 mile radius of a Walmart store that has the BBQ cleaner in stock, and the weather is sunny, you might receive a voucher for money off a BBQ cleaner delivered to your smart phone!

In another example a client of mine, a leading telecom company, is using Big Data analytics to predict customer satisfaction and potential customer churn. Based on phone and text patterns as well as social media analytics, the company was able to classify customers into different categories. The analytics showed that people in one specific customer category were much more likely to cancel their contract and move to a competitor. This extremely useful information now helps the company closely monitor the satisfaction levels of these customers and prioritize actions that will prevent them from leaving and keep them happy.

Even mid-tier cars today have about 40 microprocessors that measure performance. These electronics usually account for about one-third of the cost of a new car. Of course, all this data that is being generated, collected and analysed by the car manufacturers offer them significant competitive advantages. One car maker working with an external analytics company noticed that a sensor in the fuel tank made by a German supplier was not working well at all. The manufacturer could have told the supplier and asked them to fix it but then the improvement would have been passed on to other car manufacturers that use that supplier. So instead the manufacturer invented a software patch that fixed the issue, received a patent on the fix and sold the patent to the supplier.²

Big Data is changing the very nature of business, from manufacturing to healthcare to retail to agriculture and beyond. The rate that data is and can be collected on every conceivable activity means that there are increasing opportunities to fine-tune procedures and operations to squeeze out every last drop of efficiency.

How companies are using Big Data

Different industries have responded to the call in different ways. Retail and sales are seeking to collect as much information about their customers' lives as possible so as to fulfil their changing needs more effectively. Manufacturing are seeking to streamline operations. Equipment calibration settings can be recorded and refined, and product storage environments monitored to determine the optimum conditions that lead to minimum spoilage and waste.

For global companies this can mean collecting and analysing data from plants across the world, allowing minor variances to be studied and their results understood.

In 2013, for example, pharmaceutical giants Merck used analysis to dramatically cut the amount of waste caused by variance in manufacturing environment conditions. It took three months and involved 15 billion calculations on individual production data from 5.5 million vaccine batches. This allowed them to discover the optimum conditions during the fermentation process, and should greatly increase their yield, once the FDA has approved the proposed changes to the manufacturing process.

In the automotive industry a 2014 report by the Centre for Automotive Research stated that advances made possible through advanced IT solutions and Big Data represented ‘an engine of innovation’. The report highlighted the growing complexity of cars and the industry as the biggest challenge faced by automotive manufacturers.

The efficiency of every machine – and human – involved in the manufacturing process can be recorded so companies know what is working, and can make improvements where they are needed.

And in agriculture, data analysis is helping the industry meet the challenge of increasing the world's food production by 60%, as forecasters have said will be necessary by 2050 due to the growing population. Tractor and agricultural machinery manufacturer, John Deere, already fits sensors to its machinery. The data that is available to the farmers via its myjohndeere.com and Farmsight services helps them to establish optimum conditions for their crops. Plus the data is also used by John Deere to forecast demand for spare parts.

Of course, in business once a product has been grown or manufactured it needs to be sold and distributed. The petabytes of customer data, including you and me, already gathered by big retailers tells them who will want to buy what, where and when. Amazon, for example, uses its S3 system to keep track of millions of stock items across dozens of warehouses and distribution centres scattered around the globe. Operatives can track deliveries in real-time to see what is where, and where it should be going.

At the point of sale, retailers can use data to determine where stock should be displayed, which stores will sell most of which particular product and track customer movements around stores. Loyalty cards are not new but ever more sophisticated analysis of customer habits will lead to an increase with which retailers can predict what you will buy. This has advanced to the point where Amazon believes it will soon be able to predict what you will buy accurately enough to despatch it toward you before you have even bought it!

The connectivity that is now possible is also changing business. In 2014 Cisco announced a $150 million fund for start-ups working on improving integration between the virtual and physical world. For a business, the ability to have its production, stock control, distribution and security systems all connected and talking to each other will mean greater efficiency and less waste. GE refers to this convergence of data and machinery as the ‘Industrial Internet’, and claims it can save global industry £150 billion in wastage.

Every area of industry is learning to reap the benefits of Big Data analysis, and it looks certain that finding innovative methods of gathering, recording and analysing data is going to play a big part of business in the foreseeable future.

Even something as subjective and ‘human’ as Human Resources is being transformed by Big Data and analytics. Finding and keeping the right people is a major bugbear for most businesses. Talent management is fraught with challenges and the cost of failed management and leadership is enormous. It is estimated that the average cost of executive failure is $2.7 million.³ Published estimates into the extent of poor leadership range from 33%⁴ to 67%.⁵ In other words between one- and two-thirds of all current leaders will fail in their role.

But it's not just a financial cost. Unsuccessful executive appointments alone incur significant hidden costs, which can include lost opportunities, poor public relations, brand damage, poor productivity and employee disengagement and alienation. The impact of poor leadership on employee morale can be severe: 40 per cent of American workers classified their jobs as stressful and 75 per cent of working adults said the most stressful part of their job was their immediate supervisor.⁶

Getting the wrong person in any job can be a disaster. Get the wrong executive or leader and it can be catastrophic.

Considering that employees are the greatest asset of a business and, as the statistics confirm, potentially its greatest liability, it's easy to see how companies are getting excited by Big Data solutions such as Evolv.

Evolv is a software tool which helps assess and understand employees and candidates by crunching half a billion data points across 18 industries in 13 different countries on everything from gas prices, unemployment rates and social media use, to how long a person takes to travel to work, or to how often they speak to their managers. Although data collection methods include the controversial ‘smart badges’ that monitor employee movements and track which employees interact with each other, Evolv clients such as Bank of America are impressed.

Bank of America have reportedly improved performance metrics by 23% and decreased stress levels (measured by analysing workers' voices) by 19%, simply by allowing more staff to take their breaks together.⁷

The software is being used to predict a range of things including how long an employee is likely to stay in his or her job. Evolv has also gleaned some remarkable and unexpected insights such as the fact that in some careers, such as call centre work, employees with criminal records perform better than those without! Or the fact that employees who change the default browser on their computer to a nonstandard browser such as Firefox or Chrome perform better across the board than those who use a standard browser such as Internet Explorer and Safari.⁸ (Of course now this is public knowledge people could ‘game’ the predictor and change their default browser prior to interviews that will render the predictor useless.)

Although this type of Big Data analytics is currently focused on customer-facing roles it's only a matter of time before it reaches the upper echelons of management. Certainly improving the performance of top executives has a ‘disproportionate effect on the company’ so Big Data solutions are certain to be considered. According to the Economist Intelligence Unit more than half of HR departments have already reported an increase in data analytics since 2010.

Don't panic!

The challenge of course is that when business leaders read stories like these or hear about the cool – and a little scary – things that Big Data Gods like Google, Amazon and Facebook are doing, they panic!

Most business leaders know about Big Data – they'd have to be living under a rock not to. They understand its inherent promise and they may even be fully aware of the fact that their business is data rich. But most business leaders have no idea what to do with it! We have been told for years that we live in the Information Age; we are reminded of the importance of information, knowledge and the role of knowledge workers. We know that we need to find a way to access and use the information we already have and manage the explosion of information we could have, or are being told we should have, moving into the future. Information is gathering momentum and pace, it's growing exponentially and yet our research suggested that less than 20 per cent of the data companies currently hold is used to inform decision-making. And this 20 per cent only took traditional structured KPI or financial data into account. If that is true of the structured data which is relatively easy to extract, insight from the unstructured data represents a rich untapped vein of information gold that is currently largely ignored.

Of course this escalation of data and endless information possibilities poses its own set of problems. If we are already drowning in data that we don't use then what on earth are we supposed to do with the rest?

Some stand on the sidelines feeling the pressure of inaction growing with every article they read about the Big Data revolution. The brave (or crazy) business leaders decide to dive in and work out what they can get access to and how they can use it but inevitably they get completely lost and end up drowning in their own information, unable to convert it into insight and meaning. Unfortunately, in this case the result of either action or inaction is the same – overwhelm and confusion!

This book is designed to help you change that outcome.

Focus to reap the rewards

Big Data offers business an unparalleled opportunity to extract insight into the behaviour of their customer that can in turn transform business results. BUT just because we can measure, monitor and access everything doesn't mean we should. It is much too easy to get bamboozled by the proliferation of smart technology and endless possibilities that send business down resource-sapping rabbit holes without any definable or useful output. The danger therefore is that we get lost in a sea of data that delivers no value whatsoever.

So on one hand Big Data is changing the world because we now have so much more data and new data formats. But on the other nothing much has changed because we are still seeking to use data and information to inform corporate decision-making. The only real difference is that we now have new data formats that we can use and new technology to actually analyse that data and leverage it.

As business leaders we need to understand that lack of data is not the issue. Most businesses have more than enough data to use constructively; we just don't know how to use it. The reality is that most businesses are already data rich, but insight poor. It may be true that companies like Amazon, Google and Facebook enjoy a considerable competitive advantage because of the data they now have access to but they also have vast budgets and teams of data scientists whose only job is to analyse that data. For most businesses that is not possible, realistic or necessary. There is probably more than enough data in your business right now for you to tap into the power of Big Data without stellar tech or eye-watering budgets. And even if your business hasn't kept very good records or doesn't hold a huge amount of existing data there is definitely enough external sources to harness the power of Big Data in your business.

So essentially it doesn't matter whether you already have access to unfathomable amounts of information or your data collection systems have been a little sketchy up until now, Big Data can revolutionize your business – but only if we focus on SMART Data not Big Data. In order to do that we need a practical framework that can help us to wrestle the Big Data monster so that we can harness it in order to gain new insights that will guide the business into the future.

We need a way to navigate the oceans of data to find the pockets of meaning. Like the modern fisherman we need a sophisticated, but practical way of working out what customers we are trying to catch, finding out what we need to know to locate those customers, predict their behaviour and deliver bottom line results.

This book provides that urgently needed navigation system (see Figure 1.1) that will allow you to create a SMART business and harness the awesome power of Big Data regardless of your size or budget.

The SMART Model will mirror the structure of the book. Each chapter will unpack each part of the model and provide a practical structure that you can use to take advantage of Big Data in your business.

In order to cut through the chaos, confusion and sheer volume of data that can or does exist we must therefore ‘Start with strategy’. Instead of starting with the data, start with your business objectives and what you are specifically trying to achieve. This will automatically point you toward questions that you need to answer which will immediately narrow data requirements into manageable areas.

Once you know what you are trying to achieve you need to work out how you could access that information so you can ‘Measure metrics and data’. Once you know what type of data is available and have accessed that data, you need to ‘Apply analytics’ to extract useful insights from the data that can help you to answer your strategic questions. Of course, the insights alone are useless unless you ‘Report results’. These three stages of SMART business are underpinned by technology. Technology will help you to collect the data that you need to measure, it will facilitate analytics in ways that you have probably never considered before and it will allow you to convert the insights into data visualizations that can be easily and quickly understood and acted on.

When you approach data (big and small) and analytics from this narrower more focused and practical perspective you can get rid of the stress and confusion surrounding Big Data, reap the considerably rewards and ‘Transform your business’.