CHAPTER 21

EXECUTIVE SUMMARY

There are many details in each chapter of this book that you will likely find beneficial to your big data efforts and in developing your business strategy. There are so many changes wrought by big data in so many industries that the entire market-scape is changing as a result. Therefore, it behooves any business leader to be aware of changes in industries other than their own in order to predict, adapt, and profit from the fluidity of disruption and the continuance of cross-industry convergence happening now and continuing into the foreseeable future. To this end, the following summary is provided for those pressed for time or seeking a quick refresher after an initial read.

This is not an overly short summary; rather it is an attempt to flesh out the skeleton of the book so you can walk away with a working understanding. The following headings correspond to the chapter titles so that you can easily go to that chapter for details, real-world examples, and data resources.

WHAT IS BIG DATA REALLY?

The term big data is a bit of a misnomer as there is no consensus on the minimal size data must be in order to qualify for the designation. Further, the current definition is relative to current computing capabilities. This means that any data considered “big” today will likely not be deemed as such in the future when computing capabilities grow. Therefore the term is considered by technologists to be largely useless. Instead the technical world tends to favor a definition more attuned to data characteristics, that is, “containing volume, velocity, variety, and veracity.” However, it is important to note that data is indeed getting bigger every day and extreme data is on its way.

Big data tools are highly adept at handling data sets of various sizes so business people should not think they only have value if the business owns very large data sets. Indeed, much of the data that is useful to any business, even very small businesses, comes from outside the organization and is quite large. One example is social media data and another would be publicly available data offered by governments and open data communities. Further the tools are inexpensive, some are even free, and thus well within the grasp of any business, even a microbusiness.

Big data is not universally viewed the same by business executives. This is important to understand if you are tasked with making a business case for big data before executives or in communicating analytical results to them. At the extreme ends of the spectrum are the “big data is omnipotent” and the “big data is just a spreadsheet upgrade” groups. Many executives fall somewhere between these two extremes.

It is important to assess where the executives you work with fit in this spectrum and assess how each executive learns and absorbs information in order to effectively manage expectations and expediently communicate outcomes via visualizations. If you don’t know how the executive best absorbs information, you may choose the wrong visualization. Conversely, if you prefer more modern visualizations, you may discount executive preferences before realizing those preferences may not be as outdated as you thought.

HOW TO FORMULATE A WINNING BIG DATA STRATEGY

Strategy is everything. No amount of data or collection of big data tools will help you if you don’t have a strong strategy in place first. Big data does not replace business acumen. Start at the end by identifying what you need to know and why you need to know it before you do anything else. Decide if that information is actionable, that is, something you can take action on immediately. If it is, computing and delivering ROI, and choosing tools and data sets will be infinitely easier.

IT can get things rolling by improving existing processes and workflows using big data. These changes typically relate to increased efficiencies of different categories and business divisions; as a result, metrics and ROI can thus be quickly measured and, in a best-case execution, actually realized.

Beyond that, you should develop an overall big data strategy as well as more pointed project specific strategies. The overall data strategy should be focused on continuously discovering ways to improve through innovation and solid returns both in the short and long terms. Project specific strategies should lead to a successful and actionable end to be immediately followed with ideas on what can be done from there, which in turn should ultimately lead to satisfying the goals in the overall big data strategy. Successful projects will also lead to refinements and changes in your overall strategy, as it is a living document.

Be aware that the most vexing challenges you are likely to encounter are people problems, not technical ones. People are simply unaccustomed to thinking in terms of using data to decide the way forward and to predict business impact. Further, people tend to limit their use of big data to familiar types of analysis already performed in the course of their work rather than imagine new uses. Fear is also a common problem that must be overcome. Common fears include a loss of power, loss of job security, or inadvertently creating a public relations backlash. Data literacy rates are also low. These skills must be taught throughout the organization to optimize data-driven decision-making and innovation but also to prevent errors in analysis that can lead to costly mistakes and embarrassments for the company.

Look for “big data diviners” in your organization to help speed the adoption of big data tools internally. These are employees who possess sufficient intellectual curiosity, analytical and critical thinking skills, and fearlessness of change to power true innovation through data use. They tend to be natural teachers who peers look to for answers about how to complete tasks. They also tend to be goal-oriented rather than clock watchers and they easily detect patterns in everything they see. Be sure to also cross-pollinate your interpretive team. To do list:

1. Consider adding a Chief Data Officer to manage your data.

2. Use prototype and iteration strategies to increase adoption throughout your organization.

3. Grow your own experts. Start with small projects, learn from those, and proceed to bigger projects.

4. Remember to also use big data to answer questions that no one has yet asked in order to find new threats and opportunities early.

5. Avoid the pitfall of focusing only on immediately profit-generating projects. You must also use big data to explore and discover the mid and far futures.

6. Watch out for cognitive biases and lurking hidden agendas in your team.

HOW TO ASK THE “RIGHT” QUESTIONS OF BIG DATA

While data analysis has always been important to business decisions, it has traditionally only been possible to do so after the fact. For example, analysis of the effectiveness of a marketing campaign is historically done after the campaign has run its course. The great thing about today’s analytical tools is you can see what is happening in real-time, or near real-time, and therefore you can change course as needed to affect outcomes much sooner than was possible in the past. However, immediate actions can lead to problems in the mid- to long-term too, so be sure to use predictive analytics to consider those impacts as well before you take the action now.

Avoid data paralysis by forming the questions your company most needs answers too right now. Think in terms of what instant knowledge will make the biggest difference in moving the company or the department ahead immediately. For that moment forget about the data and the tools and focus only on what insights would offer the most business value.

All data analysis must be purpose driven to be useful. And that purpose is defined by the question asked of the data. Make sure that question is highly relevant to the business at hand and that the answer to that question will be something you can take action on or use in meaningful ways. Most of the time forming the right questions to ask will require a collaborative effort with people outside the data science team.

HOW TO PICK THE “RIGHT” DATA SOURCES

In the minds of data scientists and IT pros who deal with the intricacies of analyzing and storing data, the source of any given data set is not nearly as important as its form: structured or unstructured. And that is as it should be from a technical perspective, but from a business perspective the source of data matters a great deal.

Finding key insights and solutions to problems that can be solved with data analysis depends in large part on the quality of the data used, both in terms of its accuracy and the reliability of its source. Not all data brokers are created equal but we are now officially in a new data economy wherein data is the new currency and all and sundry are trying to cash in. Remember always “caveat emptor”—let the buyer beware.

A considerable amount of data is available free. You should check first to see if the data you need is available from a free source before you go off to purchase it. It is not uncommon for public data to be repurposed and then resold by commercial interests. Be careful that you don’t buy that which is free for the taking.

A data seller’s reputation for providing quality data must be taken into full consideration before a purchase is made but the buyer must also seek hard assurances that the data is regularly cleaned and protected from corruption. Further, since most of the data bought and sold today is not raw data but an analysis of that data, buyers need to examine and evaluate the algorithm(s) the seller used. It is no longer workable for algorithms to be trade secrets because data buyers need to know the modeling is sound in order to know the ultimate value of the data they are buying. Transparency is everything in determining data value.

Choosing which data brokers to buy data from becomes an easier task when you know what data you need before you go shopping. In many big data projects you will be adding information to existing formulas. The key to using big data successfully is not in just putting more data behind each of the variables you use now but in adding more variables to the equation.

For projects where no formula already exists, you must first define the problem you are trying to solve and then determine what information is pertinent to solving the problem. This will result in the formation of an algorithm. Double check the variables to ensure that you have enough inputs and that all inputs are relevant to the question. From there you can determine what data you need and whether such exists internally or externally. That enables you to define what data you need to purchase.

WHY THE ANSWER TO YOUR BIG DATA QUESTION RESEMBLES A RUBIK’S CUBE

The key point to remember is that “data-driven decision making” does not mean that analytics actually make decisions for you. They usually don’t, with a few exceptions such as with prescriptive analytics. Using analytics generally means you have better and faster information with which you or others in your company can make a decision. Real-time analytics in particular, but high frequency runs of other analytics too, will send computed data results to you rapidly and frequently. What you do with that information is entirely up to you and beyond the scope of a machine.

Not all questions asked of data tender straightforward results. Many questions will have to refined, sometimes repeatedly, before you can ultimately learn whatever it is that you need to know and do. Sometimes you’ll need to combine the results from two or more questions to get the answer you seek. You need to make sure you align everything correctly to solve the puzzle. Even if you make another algorithm to work that out for you, you still have to figure out how to weight each of those former results.

On still other occasions, the results will be so bizarre that you will feel compelled to test them and the algorithm to make sure they are correct. Testing and double-checking is important, as is making sure the data you are using is clean and relevant.

The most useful analytics are dynamic and not static. It generally is nonsensical to run analytics once and stop unless of course you are looking at a one-time event, such as a unique marketing campaign that had a definite beginning and ending. But not only is the work dynamic, so is the algorithm. A common mistake is to continue using the same algorithm as it is. Revisit and revise algorithms often or you risk missing important information.

In any event, analytics often spawn as many new questions as they do answers. Or at least they should because you should be learning continuously and the new knowledge should spark more questions. Data analysis is all about learning—machine learning and human learning. That being the case, you will never see a time when you have all the answers. It’s impossible to know all the answers when you can never know all of the questions.

THE ROLE OF REAL-TIME ANALYTICS IN ROLLING YOUR STRATEGY

When real-time analytics should be used is an individual call made by every company based on its overall strategy. Look at the advantages and shortcomings in real-time analytics in order to determine where they might best fit into any particular big data strategy.

Chief among the many concerns in real-time analysis is the speed in which the data is collected and whether the data itself is current enough to be considered real-time. With direct streaming of social media, for example, the speed and immediacy is readily evident. Not so much with other data.

In addition to the speed and immediacy of incoming data, one must consider reaction time on the other end of things. If, for example, marketing can do a real-time analysis on customer reactions to a marketing campaign but cannot act on the analysis for days, weeks or perhaps even months, then they are in effect reacting to a past event and not to a real-time analysis.

You must be selective in determining what information you need in real-time and how fast your company can react to it before you go to the effort and expense of analyzing it in real-time. Sometimes the slower and cheaper past-time analytics work just as well and are far more practical.

If you use real-time analysis too much, you’re burning up resources needlessly. However, if you skip using it where needed you are hobbling the enterprise and putting it at a huge competitive disadvantage. Find the timing sweet spot for any given information and adjust your tactics accordingly.

However, even real-time analytics can reveal past events or decisions already made, so you may discover that damage is already done or opportunity is already lost. For example, you may find that a startup has produced a wildly popular product similar to the one you were planning to introduce six months from now or that completely disrupts your industry. In other words, the analysis you do in real-time may result in bad news that is difficult to influence or change to any significant degree.

With real-time analytics you can pivot mid-stride. Whatever you find in real-time analysis, you can find a way to cope with it and even profit from it despite the obstacles because you are now aware of the situation.

THE BIG DATA VALUE PROPOSITION AND MONETIZATION

Divining the value proposition of big data depends considerably on how well your company wields it. Therefore it is exceedingly difficult for a big data vendor to specifically answer your questions pertaining to ROI. It is not difficult, however, for you to calculate ROI for your company as presumably you do know the rest of the equation: how well your company strategizes and implements internal, market, and customer intelligence.

The value in big data is always found in the outcome. Unfortunately, that poses a problem for big data champions trying to make a business case for big data investment long before projects are started much less completed. Hence the recommendations by some experts to run small projects first and then make the case to scale that particular project, or to use the value found in the small test case as evidence that value does exist in other big data projects by extension. Many big data champions bank their business case on resolving existing data problems and costs and/or on increasing efficiencies in processes.

Outputs are intended for different audiences with different goals and the values of each are therefore determined differently. In other words, there is no one ROI calculation, but many. You will have to choose the formula that fits your expected business outcome. See the full chapter for a variety of formulas you can use.

Inevitably the question of data monetization surfaces in the ROI discussion. Data has already become a commodity and the prices are now so low that it is doubtful that selling your data would be worth the effort. The problem is that the data most data buyers want is the very data you would exclude or anonymize to oblivion in order to protect your company’s competitive edge, secrets, and profitability. What is left after you do all that redacting is worth little to anyone. Raw data then is rarely of much value. Insights however are another story. There is a market for big data insights that big data users cannot arrive at on their own.

You can monetize your data by selling it if you create unique value in some way. Is it cooked to offer unique insights that are so fresh that they are a hot attraction? It’s generally not of value if you serve it like sushi: raw and cold.

RISE OF THE COLLABORATIVE ECONOMY AND WAYS TO PROFIT FROM IT

Big data’s biggest impact is model shattering. One example of that is the emerging collaborative economy wherein people value access to goods and services over ownership and where they get such from each other rather than from retailers, manufacturers and other corporations.

The collaborative economy has three basic components: sharing economy, maker movement, and co-innovation. How much this new market shift will affect your company depends entirely on how well you respond to it. That means you will have to use big data to predict impact but also to deduce the means to compete in an ever changing environment. Successful adaptations will more likely than not be in the form of revised or new business models often put into effect on the fly. This is an example of why your big data strategy must be far broader than just improving how you do business now. It must also include a focus on the mechanisms and processes for how to change or adapt your business model as necessary to survive and capitalize on changes brought about by ever-evolving industry and market disruptors.

The definition of agile is quickly becoming antiquated. The new goal will be to become fluid, that is, to change the shape of the entire organization as needed to fit an ever-evolving market.

It will be the companies that master big data to the point that they can successfully use predictive analytics to change their overall company, processes, services and products rapidly to snare fast moving opportunities that will ultimately survive and profit. This is why you must use big data for more than just short-term goals. This is why ROI should not be your main focus or motivation in using big data. Make sure that your big data strategy is big enough to do all that must be done.

THE PRIVACY CONUNDRUM

Four major shifts in how data is collected and used have sparked public ire. The first is that data collection tactics are far more invasive than in years past. Secondly, more and a larger variety of data is collected, correlated and inferred which reveals far more telling details than most people are comfortable with others knowing. Thirdly, silos have crumbled, leaving the average person more exposed and vulnerable in data integration. Last but not least, companies and governments are gathering data far beyond the traditional business or mission scope and often doing so through surreptitious means.

The privacy conundrum is further complicated by the unfettered use of data in the private sector. If future regulation focuses on government use of data alone then the private sector will continue on unabated, thereby guaranteeing individual privacy will not exist. Further, the private sector is likely to profit from regulations forbidding governments to collect data as the corporate world will simply collect the same information and sell it, or sell the analysis of it on demand, to governments. Indeed, many are already doing so.

In addition, numerous corporate entities such as Facebook and Google have already collected sufficient information on individuals and businesses to infer behaviors for decades to come. It is also likely that the United States and foreign federal governments have as well. There is no realistic way to ensure data is permanently erased in any of these collections.

Lastly, there is no clear definition of who owns what data, making the argument to delete data on individuals harder to make and win.

This brings you to an important aspect in developing big data strategies for your organization: are your data collection processes and uses respectful of individual privacy and if not, is your organization prepared to cope with the inevitable repercussions?

Certainly liability issues should be forefront on your mind in determining what data to collect, how you collect it, and what you infer. However, few companies understand that they can also potentially be liable on other counts because of their efforts to protect individual privacy. Not reporting a clearly suspicious or illegal act, in the name of protecting customer privacy, could conceivably make a company complicit in the crime. A very good way to help limit this exposure is to make absolutely sure you collect no more data than is necessary to conduct your business in a “privacy by design” framework. Another good approach is in supporting the development and implementation of sensible privacy regulations and clear guidelines, so you know better how to safely proceed.

The one truth all data practitioners should hold fast is this—Big Brother is us and we are the individual. It is our data practices, and the practices of others like us, that collectively give life to the Big Brother concept. But each of us is also an individual and whatever data use horrors that we allow to exist will haunt each of us in turn. Together we know privacy or none of us do.

USE CASES IN GOVERNMENTS

Very few local governments in the United States are working in big data now due largely to funding cutbacks and a lack of talent capable of doing big data projects. For the most part local governments are using traditional data collection and analysis methods largely contained to their own databases and local and state sources. Large cities tend to be far ahead of the curve in using big data and in offering public data sets. Eventually local governments will benefit from the growing spread of data from myriad sources and simpler analytics that can then be more easily used at the local level.

State governments vary greatly in their use of big data. Some states rely on a mix of their own existing data and that which they pull from the federal government or is pushed to them. Other states are busy integrating data from cities, townships, counties, and federal agencies as well as integrating data across state agencies. Most states are making at least public records data readily available online but you usually have to find that information in a state by state online search. Some state agencies also offer mobile apps that contain specialized data. As of this writing, no regional data sets have been created by state governments that are voluntarily combining their data with that of adjacent states for the benefit of their region—at least none that we have found.

The U.S. government has made substantial strides in making more government data available to citizens, commercial and non-profit interests and even to some foreign governments. The Obama administration’s stated goal is to make the government more transparent and accountable. But the data provided also often proves to be an invaluable resource for researchers and commercial interests. Data.gov is the central site for U.S. government data and is part of the Open Government Initiative. The Data.gov Data Catalog contains 90,645 data sets as of this writing. You can expect that count to climb higher over time and for the data sets to individually grow as well.

The federal government is using big data to inform its decisions but also sharing data as a means of fueling or aiding private and commercial efforts as well as a public relations initiative. See the full chapter for more details.

USE CASES IN THE DEPARTMENT OF DEFENSE AND INTELLIGENCE COMMUNITY

There is nothing more frightening to the public psyche than the prospect of an all seeing, all knowing Big Brother government. The recent and ongoing leaks from Edward Snowden, a former Central Intelligence Agency (CIA) employee and a former contractor for the National Security Agency (NSA) turned whistle-blower, continue to feed that fear. Many of the resulting concerns are covered in the chapter on the privacy conundrum, but this chapter looks at the other side of those issues, from the government’s perspective.

Given their urgent and important national security mission needs, it is no surprise that technologists in the Department of Defense (DoD) and Intelligence Community (IC) have sought out new approaches to analyzing and using data. After all, big data is uniquely suited to quickly and efficiently searching mega-sized data for markers that could indicate criminal and terrorist activity. Big data tools also make it possible to see and note personal relationships between criminals and terrorists that might otherwise remain hidden.

Modern data analysis can even accurately predict rising nation aggressors and impending war. Big data tools can also make it easier to find and respond to cyber-attacks in progress. Further, big data can power artificial intelligence and drive automated war machines to attack faster and more precisely than their human counterparts. Indeed, there are already so many ways to use big data in national defense that it boggles the mind. Even so, more uses for it will appear as mastery of big data improves.

Currently, the types of big data solutions being fielded include situational awareness and visualization, enhanced healthcare, information search and discovery in overwhelming amounts of data, logistical information including asset catalogs, big data in weaponry and war, and many more.

The DoD and IC Community’s work with big data is more complex by nature, but they do share some data openly. For example, DARPA has made its Open Catalog available on a public website. It lists DARPA-sponsored software and peer-reviewed publications for interested parties to see and review. Obviously, such openness could lead to efforts designed to thwart DARPA’s and other agencies’ work as easily as it could enlist aid. Just as obviously, agencies don’t list anything overly sensitive in their open data collections for precisely that reason.

Indeed, as of this writing, terrorist groups were already seen to be using intelligence gained from the Snowden revelations to implement encryption and other ways to shield their actions. Suffice it to say that the Department of Defense (DoD) and Intelligence Community (IC) are now severely challenged in detecting and preventing threats. For all practical purposes, they must now create new ways to work from scratch. Big data will continue to play a central role in these new efforts.

USE CASES IN SECURITY

The cybersecurity and physical/facility security subsectors, for all practical purposes, have blended into one, given the digitalization of devices and sensors and the advent of the Internet of Things, which refers to all things being connected to the Internet. It is in this context that security experts must now work whether they are charged with cybersecurity, that is, protecting data, or in physical security (protecting facilities, events, and persons). Very little in security work relies on direct human observation anymore.

That’s not to say that security work has gotten any easier. Despite all this data, all these additional mechanical eyes and other extensions of the security expert’s manifested presence, the exponential growth of threats and the steady rise in the sophistication of the attacks is still outpacing security efforts.

Privacy problems increase when data is collected for security purposes. Once data is gathered, stored and shared it is impossible for that information to be retrieved, deleted or otherwise guarded by the person that information is about. This leaves the door wide open for future abuse.

On the other hand, security professionals can only do so much in identifying immediate threats in real time. It is often the case that forensics teams and infosec professionals need to backtrack through metadata, log files, and sometimes even generalized data to find the trail of evildoers and to predict what they are likely to do next. Hence, there is value in collecting and storing massive amounts of data over time.

Therein lies the rub. That which strengthens privacy is often “bad” for security and vice versa. Yet those who seek privacy also wish to be secure and those who seek to provide security also want privacy for themselves. Big data, then, is both friend and foe to both causes.

Use cases for big data in this category include those in any other cybersecurity effort as well as in the development of innovative protective measures. Such includes but is not limited to offline overrides in medical equipment that react to unauthorized changes in commands or performance and offline secondary and compartmentalized or containerized emergency and backup systems such as for electric grids, water and banking systems.

One of the biggest benefits of big data in security is in its ability to identify likely threats and afford the opportunity to design both prevention and responses.

USE CASES IN HEALTHCARE

These are already data-driven industries, albeit in a highly siloed and splintered fashion. Pooling data and analyzing disparate data sets for additional insights is the next logical step. Unfortunately, that is easier said than done. For one thing, many of the industry players are highly competitive and their information is proprietary.

For another thing, the industry is highly regulated. Revealing data could lead them to compliance troubles with existing privacy regulations such as the Health Insurance Portability and Accountability Act of 1996 (HIPAA) and its counterparts in other nations. How does one go about sharing meaningful data on thousands if not millions of patients when the barrier is so high on sharing data on just one patient?

Despite these challenges, efforts are underway to share data in order to speed cures, treatments, and vaccines. Big data helps speed the shift to personalized medicines to improve patient outcomes and to genetic-based medicines to replace current antibiotics that will soon no longer be effective. It also is invaluable in finding new and more effective ways to prevent health issues from developing in the first place and in innovating entire healthcare systems on a scale never before seen.

There are problems with using only data shared on the Internet. Consider the case of Google Flu Trends vs the CDC’s traditional methods of tracking outbreaks of the flu wherein Google Flu Trends handily lost. This happened largely because that data which is online, or collected digitally online, is too limiting for accurate analysis. Further, for such data to be useful it must be exacting and precise. It must follow strict protocols and include traditional data collection methods and refined statistical analysis. Fortunately traditional health agencies are beginning to share more data online to aid the development of more comprehensive efforts elsewhere.

On the side of more informal data sharing and collaborations is the biohacker community. As is the case with computer hackers, they come in both the white hat and black hat varieties. Like computer hackers before them, the vast majority of biohackers in these early days of the movement are indeed white hats. They create cheap antibiotics in the field and hack DNA to experiment with creating genetic-based medicines, cures, and biological defenses. When they gather in groups, they learn from one another as well as collaborate on projects. Such gatherings happen all over the world and very frequently.

Data is used in healthcare to cure humankind of what ails it and to break bottlenecks in discoveries and innovations. It is used to better patient outcomes and to battle rising costs. Big data is used to inform policymakers and to proactively warn providers of impending compliance issues. Data reveals all from the drug abuser visiting multiple emergency rooms to get a fix to fraudulent insurance and Medicare/Medicaid claims to bad medical practices and price gouging by the industry.

It enlightens and improves healthcare but also illuminates and destroys healthcare as we know it. Those providers who wield data analysis well and take smart action will prosper. Those who don’t, won’t.

USE CASES IN SMALL BUSINESSES AND FARMS

Big data gives SMBs incredible business advantages that were once only in reach of the biggest companies. But it also makes large companies more agile and innovative, and their offerings more personalized—strengths that formerly belonged to smaller businesses. This means that large companies using big data can better compete with the SMBs’ legendary personalized service and innovative capabilities. This poses a new threat to SMBs.

While big data can give you important and even vital insights, it can’t think for you. Indeed, it is the best strategists that win using big data, not those who own the most expensive, fanciest big data tools. The three things SMBs should understand are that a) big data is not just for big companies, b) the information has little to no value unless you act on it, and c) it is your strategy borne out in this use that benefits or harms your company.

Fortunately for small businesses and farms on tight budgets, many big data tools are free or available for a very minimal fee. Specific examples and resources can be found in the full chapter.

One of the big advantages to using big data tools is the ability to augment your own data with data from other sources. There are many public data sets available for free and more showing up every day such as those found on Amazon Web Services (AWS), Yahoo! Labs, and Open Science Data Cloud (OSDC). Some of the most bountiful sources for free data sets come from government agencies from the federal level to the county or city level. Small businesses can use this information for a variety of purposes from identifying best locations for new stores to finding loans, grants, and a variety of other resources. Data is readily accessible and available almost everywhere. Decide what information you need first and let that guide your search for data sets.

Deciding whether to share your own company data is a bit trickier. Small businesses and farms should seek resolution and clarity before committing to any data sharing agreement or smart machine purchase. Conflicts of interest often do exist.

While small businesses are busy figuring out how best to use big data and whether or not to share or pool their data and if so then with whom, other industries are busy using big data to figure out how best to serve and profit from SMB customers. In many cases, big-data-spawned disruptors have arrived to change circumstances for the better for small businesses.

USE CASES IN ENERGY

Countries, states, provinces, cities and townships, utility companies, and businesses of all kinds and sizes seek to lower their energy use and costs. Each also seeks to know how energy trends and future predictions impact them and their constituents or customers. Big data and advanced analytics can help each meet their goals and reduce risks.

Several entities provide energy data for industry and public use. Among them is the U.S. Energy Information Administration (EIA). The EIA has possession of energy data that is otherwise unobtainable by other parties but shares it publicly and freely through reports and the eia.gov website.

If you’re looking for data from smart meters, you’re in for a long wait. While most utilities, particularly electric companies, are quickly moving to use smart meters throughout their network, few have a clue as to what to do with the resulting data. But there are also a number of problems in streaming smart meter data to the EIA including a lack of standardization in outputs and a variety of local regulations.

Utility companies look to data from smart meters and other sources first to improve cash issues both in raising revenue and cutting costs. The second most common use case is to detect and resolve network issues such as breaks, stressed components, inefficient components and weak points, likelihood of outages, and future capacity needs. With the advent of new smart sensors, meters, and other equipment capable of generating a constant stream of data, and the analytics needed to make sense of all that data, utilities can now work in a proactive state thereby preventing most outages and containing repair costs.

A third use case is in opening the network to receive more energy supplies, such as in customer buy-back programs where customers augment utility use with an alternative energy source, such as solar panels on a business or home. The utility can buy the customer’s excess energy and add it to its general supply for resale. It will take big data tools to correctly track and monitor new incoming energy supplies and to properly track those costs and resale dollars.

A fourth use case is in early detection of a threat to the system, be that a heavy squirrel population near a transformer, a developing weather system, or a terrorist or activist threat. As utilities increasingly turn to digital information to drive operations, those systems will be vulnerable to attack. It will take big-data-driven security efforts to detect, block, respond, and deter such efforts.

A fifth use case is in innovating the utility company overall. From changing inefficient internal processes to improving energy output and alternative energy input, analytics will eventually change everything in how a utility company and the energy industry overall operates.

By using data so readily available from reliable sources such as the EIA your analysis of energy costs and availability impact becomes infinitely more accurate. APIs make it easy to feed the information into your calculations and dashboards. Further, businesses can use this information to inform their analysis on store, plant, or office locations as energy costs affect cost of living, and therefore payroll costs, as well as overall operational costs.

USE CASES IN TRANSPORTATION

At first, in-vehicle data collection was thought of in terms of advantages to the automakers, most notably in developing new revenue streams from drivers, insurance and lien holder sales. For example, features that offered emergency services for drivers, that is, those of the OnStar ilk, also provided data that was valuable to car insurance companies and vehicle repossession companies working for lien holders.

Car dealers also found new data-generating, in-car features to be financially rewarding. They discovered that by reading data on cars they could cut diagnostic and repair costs and increase the number of service department visits via automated customer diagnostic messaging and thereby increase revenue in the service centers.

However, the focus in using data from cars and in-car technologies has broadened beyond the auto manufacturers and dealers scope. Governments, from the township and county level all the way to the federal level, are also looking to this data to improve roadways, safety, and traffic flow and to eventually move to a connected transportation infrastructure.

While most people think of “connected vehicles” in terms of cars being connected to their owners’ devices both in and out of the car, and connected to the Internet, the term actually has a different meaning. Connected vehicles as a specific term means connectivity between vehicles and road infrastructures, usually achieved with the aid of wireless technologies.

Our existing roads and highways are now being digitalized through the use of cameras and sensors. Data is collected and fed to different government agencies, some focused on law enforcement and others on improving traffic patterns and future transportation planning. Our vehicles are also using and generating more digitalized data everyday using a range of sensors and technologies. Data analytics and reporting is done in-vehicle, between vehicles, and soon with the road infrastructure and surrounding structures, signs, and buildings.

Trains, planes, ships and other forms of transportation including everything from drones and intelligent, autonomous war machines to ferries and golf carts, are being transformed. In each case, data generation, analysis and data-driven innovation are at the heart of the transformation.

Increasingly, data is being publicly shared. Entities outside the transportation arena can make use of this data in myriad ways. Already plans are afoot to redesign cities into smart cities and to change transportation completely. You can expect wearable computing and mobile apps to become more tightly integrated with transportation. The possibilities and opportunities in data convergence and fast analytics are nearly endless.

All of these changes bring pressures to bear on the transportation industry, which must keep up in order to stay in business. Shipping and delivery companies are relying heavily on data and analytics to improve their operations and remain competitive. They will also need to use data analysis to reshape their entire business model and offerings in order to adapt and survive all these market changes.

Insurance coverage will still be needed but such will likely be tied more to malfunction coverage rather than individual driver actions and skills. The shift in what risks auto insurance will cover in the future will not lessen the value of the data they gather on drivers today, however. Such data can be used for other purposes useful to other types of insurance coverage, employers, and other industries and business functions. For example, driver data can be used to assess the level of risk an individual will take. Such could be useful to life insurance companies in assessing risk in insuring the life of an individual or to employers seeking personalities uniquely suited to taking risks in their work that may lead to new innovations.

In short, this industry, like many others, is now data-centric in all facets. This will only become increasingly true over time.

USE CASES IN RETAIL

Big data tools, unlike any other, can show retailers what is happening now rather than merely dissecting what happened in the past. They can uncloak emerging threats and opportunities and spur changes in innovation, processes, and future business model needs.

The problem retailers have run into so far using big data is two-fold. For one thing, “information certainty” gets fuzzier outside of structured data (the retailer’s own transactional data). Even the retailer’s own data can be “dirty,” that is, incorrect or out-of-date and incomplete, as much of their data is still locked in siloes.

Second in the retailers’ two-fold problem is in the lack of creativity in forming big data questions. Like many organizations in every industry, retailers typically ask the same old questions of big data that they asked of smaller data. While it’s perfectly understandable to ask these questions repeatedly, they give too narrow a view if that is all you ask. The goal before retailers today is to change the organization into a highly distinctive, fully differentiated, profit-generating machine. How are they going to do that by doing the same things over and over again ad nauseam?

In a nutshell, these typical problems with using big data, or in plotting the path forward in general, is why retailers are having a tough time in finding their way. Instead, retailers find themselves continuing to resort to cliché advertising messaging, personalized marketing that is often anything but, and broadly offered, deep discounts they can ill afford. If status quo continues to be projected into big data projects in retailing, status quo will be the result of the efforts, that is, small gains if there are any gains at all.

The rise of such disrupters as Amazon and eBay should have been a wake-up call to the industry and in some ways it was. But many retailers have yet to regain their footing since. Now there are more disruptors on the horizon and retailers will have to really scramble to catch up and surpass these challenges.

There are other changes coming as well such as 3D printers. Already manufacturers are using 3D printers to produce goods faster, cheaper and more precisely. In the near future, 3D printers will be in wider use. Instead of shipping goods assembled in a manufacturing plant or from a retailer to a customer, soon design codes can be sent for many goods to an on-site 3D printer instead and produced there.

Aiding that shift to 3D printing is the advent of home body scanners, such as can be done via an Xbox Kinect game console now. Such precise body measurements can be used to virtually “try-on” and order clothing and other items made to fit perfectly. This will decrease or eliminate the need to return items after purchase to retailers because of fit or “look” issues.

Obviously these changes, separately and together, will abruptly alter how retailing is done. It behooves the industry to prepare for such now. Some retailers may want to strive to be the first to offer such since first to market has its advantages. Others may prefer to incorporate upcoming market changes into their model to enhance customer experience and continue to entice customers to shop at their brick-and-mortar stores.

If the brick-and-mortar retail sector is to save itself, it must hasten and improve its use and mastery of big data and other technologies. It must also reinvent itself in ways previously never imagined.

USE CASES IN BANKING AND FINANCIAL SERVICES

Banking and other financial services institutions are a study in contradictions. On the one hand they traditionally excel at calculating and containing risks, which means of course that they excel at using data effectively. On the other hand, they’re not doing so well in using big data.

Big data tools are not so much the problem for this group, although some are certainly struggling with those, rather they generally suffer from a brain rut. Their thinking on the various aspects of their business models, product lines, risk calculations, and investments is so deeply rooted in tradition that they often have difficulty branching out into more creative approaches.

The problem with the traditional lending approach is three-fold. For one thing, shrinking one’s customer base purposefully is almost always a bad idea. Fewer customers generally equates to fewer sales, smaller profits, and a future lost to customer attrition. For another, this approach opens the door wide for competitors to encroach and even seize the bank’s market share. Hunkering down in a fearful position encourages aggressive competitors to move in on your territory. And third, while traditional financial institutions spurn potential customers and new alternative lenders welcome them with open arms, the public perception of banks shifts from essential financial service providers to nonessential, perhaps irrelevant, players in a crowded field.

New alternative credit sources are also beginning to pressure banks. They include in-house financiers such as PayPal Working Capital and Amazon Capital Services, peer-to-peer lenders, and co-branded credit cards such as those offered by major retailers like Target, Macy’s, and Sears in partnership with companies such as Visa, MasterCard and American Express. Further, these cards are generating a considerable amount of data for competing entities that was formerly contained and controlled by banks and traditional lending institutions.

As more consumers learn it’s easier to qualify for credit cards from airlines, retailers and other non-bank companies—and where the loyalty rewards are typically better too—fewer consumers will turn to banks for these services. Combined these trends erode market share for banks and other traditional financial institutions.

Additionally, traditional lending institutions place too much faith in outside traditional risk-assessment sources, namely traditional credit bureaus. Therefore the lenders are further stymied by the problems in traditional credit bureau data such as too little data, irrelevant data, outdated credit rating systems, and virtually no distinction between potential borrowers who are truly are a credit risk and those who were set back temporarily by the recession but otherwise pose little risk. To add insult to injury, lending institutions pay credit bureaus for data that may have little to no actual value.

For banks and other traditional financial institutions to better assess their risk and realistically expand revenue from loans and other credit and financial services, they have to import significant data from sources other than credit bureaus and develop better risk assessment algorithms. Further, they have to develop the means to assess the credibility, accuracy and validity of data sources and the data they supply. The days of meekly accepting credit bureau data, or any data, at face value are long gone. The business risk in that behavior is unacceptably high.

Financial institutions can also use data on existing customers to help identify characteristics they need to look for in prospects in order to add profitability and sustainability for their business. Other use cases include strategic planning, new revenue stream discovery, product innovation, trading, compliance, and risk and security.

USE CASES IN MANUFACTURING

There are a number of overall market and technology trends now pressuring manufacturing. Chief among them is the larger market trend toward personalization and the emerging trend of 3D printing in manufacturing. The two trends are related even though they may not appear to be at first glance.

The trend toward personalization is seen everywhere in the market today from personalized marketing to personalized medicine. In every case, personalized means tailored to an individual and not bulk delivery to the masses.

Manufacturing is currently undergoing a disruptive shift from subtractive to additive production, which neatly dovetails with the personalization trend. Additive manufacturing, aka 3D printing, is conducive to personalized production as it adds materials in layers to form a given shape or a finished product, the specs of which can be easily, rapidly and cheaply changed between printings. Additive manufacturing also produces little to no waste in the printing process; indeed, it can use industrial waste as “ink.” Further, it is very much faster and cheaper than subtractive manufacturing.

Traditional manufacturers will have to increasingly compete against manufacturers that are using faster, cheaper and greener 3D printing production techniques. Therefore, it will soon be no longer profitable to merely turn to more automation to eek small gains from increasing efficiencies. Traditional manufacturers will have to discover and deploy new ways to remain competitive.

Big data is among the manufacturer’s best tools in finding their way forward on what will increasingly feel like an alien market-scape. Unfortunately, most manufacturers have so far ignored or failed to master these tools.

Because 3D printing enables personalized manufacturing, everything downstream from manufacturing will change too. For example, homes can be printed faster and cheaper than they can be built in the traditional way. As architects and construction companies increasingly turn to 3D printing to produce their work, their need for traditional building materials diminishes and then disappears. That will bring about massive changes in demands for everything from sheetrock manufacturing to secondary manufacturing such as engineered wood product production. Even home improvement stores may shift their buying patterns from traditional building supplies to stocks of “ink” suitable for printing whatever is needed. The stores may even provide the printing services.

Similar changes will occur in other types of manufacturing. For example, car dealers in the future may no longer stock a large number of vehicles on their lots. Instead, they may have only a handful on a tinier showroom floor. In this way customers can see the finished product and take it for a test drive. Consumers will also be able to order exterior and interior colors in any combinations they can imagine and even add personalized art elements too.

The advantages to consumers include being able to purchase the precise vehicle they desire rather than choose from those on a dealer’s lot. But even that may have limited appeal to buyers. A new collaborative economy is emerging. In a collaborative economy the emphasis is on access to, rather than ownership of, material goods.

Manufacturers are better served in using big data analytics to not only note upcoming sea changes but to predict the likelihood of success in various models to improve their competitiveness before they choose a path forward. In this way, risks are better managed and innovative ideas are made more certain.

It is prudent for manufacturers to look to analytics they place on and within their products with an eye to how this information can be used for the manufacturer’s benefit in ways other than reselling the data to other parties or placing ads. There are numerous ways such can be done, such as in product innovation and reducing product liability and product recall costs.

By studying how, when, and where consumers use their products, ideas for profitable innovations are more easily found. By comparing dealer, reseller or retailer performance, a manufacturer can increase profits by better managing performance details in each and by identifying new requirements they should seek or impose on their distributors.

EMPOWERING THE WORKFORCE

The typical big data project suffers from a host of ills including bottlenecks and communication snafus between technologists, executives and business users. Because the process is often clogged, output is often sluggish, results are frequently confusing or error-laden, and valuable time is habitually lost.

Empowering the workforce through big data analysis and solutions can dramatically change everything. And improving the current sub-optimized processes will be just the beginning. Fortunately big data is already beginning to trend toward data democratization and self-service tools and that is a clear indication that workforce empowerment is on the near horizon.

Big data will likely follow the path of every other major enterprise technology—from telephone to e-mail to business intelligence—all of which were optimized when put in the hands of the workforce. As big data follows suit, organizations that accelerate this trend benefit from faster decision-making and marked competitive advantage. In order for that to happen, big data must be accessible to users in a self-service, readily available, user-friendly model.

Democratization of data is rich in its promise to empower increasing numbers of knowledge workers in big and small enterprises. Adoption rates are likely to be lean in the early years simply due to low data literacy rates and varying levels of interest and ability in analytical thinking among the rank and file. The overall effect will be significant as first-adopters begin to reshape how the company functions and competes. That effect will be magnified as more workers become comfortable with using the tools.

True to expectations, self-service data access and analysis is already showing great results. Allowing workers to interact directly with information, and thus eliminating the need for highly trained data scientists or IT departments to get involved, adds agility to the organization overall by eliminating bottlenecks and speeding reaction times. Innovation becomes a common occurrence since every ounce of talent on the organization’s payroll comes into play. IT also becomes more productive since resources are freed for other projects.

Expect to see a sharp rise in big data self-service soon and across all industries. If you are not empowering your team now, they will be at a huge competitive disadvantage in the near-term.

Now that you can see the big picture in big data, you can fashion your own strategies and projects to move your company meaningfully ahead. Remember you—and your competitors—are only limited by the imagination and business acumen at hand.

Good luck to you all!