You have read about it in the papers, you have seen it on the evening news, you have heard about it from your friends – big data and data science are everywhere and they are here to stay.

The success stories from Silicon Valley have made the effect even more pronounced. Who would have thought that a ride-sharing and ride-hailing phone application, Uber, could become one of the most popular companies in the world with an estimated valuation of close to $70 billion. Sites and apps such as Airbnb turned apartment-sharing into a booming business, becoming the second most valued company at $30 billion.

These and other similar events transformed the topics of big data and data science from being purely theoretical and technical subjects into common terminology that people have come to associate with unbounded investment success.

Since nearly all major technology vendors have started adding features categorized as big data, nearly all companies that invest in technology today are using some facets of big data, knowingly or otherwise.

The process of implementing however, is very loosely defined. As such, there is no definitive framework, other than perhaps Hadoop, which has been the de facto framework that most companies have adopted. Senior level management are often aware of the big picture, namely, the value that big data can bring to their organizations. However, the path to realizing the vision is challenging and as such there is no definitive solution that can guarantee success. 

Broadly speaking, there are three stages of implementation:

• Dormant: When the company has not yet established a firm mandate, but there are discussions about big data
• Passive: The discussions start taking a more formal shape, usually leading to the delegation of a team/teams to assess the impact and value to the organization
• Active: The company starts assessing technologies and engages in active implementation

The ownership of the big data and/or data science strategy can be somewhat confusing. This is due to the fact that the field encompasses elements of both analytics as well as technology. The former, analytics, is usually owned by the business-facing divisions of the organization, whereas technology is owned by IT departments. However, in data science, both elements are required. Individuals who are data specialists, that is, those who understand the domain very well and have experience with the data used in the domain, can be great business subject matter experts. They may also be able to comprehensively identify the ideal use cases and how the data can be best utilized. However, without having strong technical acumen, it would be difficult to identify the right tools to realize the vision.

In similar terms, an IT manager may be very knowledgeable about big data-related technologies, but would require the feedback of business stakeholders to effectively determine which of the various solutions will meet the specific immediate and long-term needs of the organization.

In other words, multiple cross-disciplinary streams need to collaborate in order to implement a truly effective organizational big data ecosystem.

The process of realizing the strategy is generally either top-down or bottom-up. However, rather than adopting a rigid directional approach, a collaborative, iterative, and agile process is often the best solution. There will be decisions, and changes to decisions based on new requirements and discoveries during the course of assessing big data needs, and prior evaluations may need to be modified in order to meet modified objectives.

The bottom-up approach involves structuring decisions starting at the IT level. The top-down approach, which is arguably more common, involves making decisions starting at the management level. Neither is generally optimal. The ideal approach is one where there is a continuous feedback loop that adjusts requirements based on a process of discovery:

In contrast, the top-down approach is as follows:

Neither the bottom-up nor the top-down approach is optimal for a successful big data initiative. A better option is a collaborative process that takes into account the shifting needs and diverse requirements of different departments that stand to benefit from the implementation of a big data platform: