Data science is a relatively new topic in terms of enterprise IT and analytics. Traditionally, researchers and analysts belonged broadly to one of two categories:
- Highly technical researchers who used complex computing languages and/or hardware for their professional tasks
- Analysts who could use tools such as Excel and BI platforms in order to perform both simple and complex data analysis
Organizations started looking into Big Data and, more generally, data science platforms in the late 2000s. It had gained immense momentum by 2013, when solutions such as Hadoop and NoSQL platforms were released. The following table shows the developments in data science:
|
Year |
Developments |
|
1970s to late 1990s |
Widespread use of relational database management systems. Entity relationship model, structured query language (SQL), and other developments eventually led to a rapid expansion of databases in the late 90s. |
|
Early 2000s |
The anti-climatic, yet expensive, non-event of Y2K, coupled with the collapse of the dot-com bubble led to a period of stagnation. In terms of databases, or more generally, data mining platforms, this meant that companies were less focused on new innovations than they were on keeping the business running. |
|
2005-2010 |
The industry slowly recovered, but it was not until 2005 that newer developments began to emerge. Some notable events included:
|
|
2010-2012 |
2010: NoSQL conferences and related events start gaining popularity and NoSQL becomes a commonly accepted technical term. At the same time, Hadoop becomes widely popular, and nearly all major companies begin the process of implementing Hadoop-related technologies. 2011: Market leaders start adopting Big Data and forming Big Data strategies. Numerous articles and research papers claiming the huge potential of Big Data makes it very popular. McKinsey publishes a paper on Big Data and calls it the next frontier of innovation, competition, and productivity. The October 2012 edition of, Harvard Business Review includes a very positive outlook on data scientists, which becomes immediately popular. |
|
2013-2015 |
The growth of Big Data technologies leads to the development of a concept called data science, which moves the focus from just the data to the value of the data. Coupled with developments in machine learning and the rise of the popularity of R, Python, and other data science-oriented platforms, the industry shifts attention to getting insights from data as opposed to merely managing data. Machine learning is the new buzzphrase. |
|
2016- |
The evolution of smart devices, wearables, AI-enabled cell phones, autonomous driving cars, and other such innovative solutions adds a new component of artificial intelligence to the existing trend of Big Data and machine learning. Manufacturers start broadly advertising the intelligent capabilities, as opposed to merely the machine learning capabilities, of technical solutions. |
The responsibility for implementing a Big Data platform or, more generally, a Big Data Initiative, is generally delegated to the IT or Analytics Department of a company, if such a department exists.
In a general survey of Big Data and data science delegations in organizations, we observed that, in most cases, the Chief Information Officer or the Chief Data/Digital Officer was responsible for the Enterprise Big Data Strategy, as shown in the following figure:
Although analytics and IT teams played a significant role, not surprisingly, the final responsibility was delegated to the C-level management of the company.
Investment in Big Data within the enterprise was also varied, with most organizations in the $100k to $1M range. An analysis of mid/large-scale organizations produced an expected result. What was evident, though, was that nearly all respondents had made at least some investment in Big Data:
Most organizations also reported having a corporate mandate for Advanced Analytics. This helped in securing the required budget to implement and advance the state of analytics within the organization.
Furthermore, the predicted forecast of the revenue potential of data science greatly helped in making the case to senior management that a suitable investment in Big Data was essential to the future growth of a company.
With a current Big Data and Business Analytics revenue that has grown exponentially to more than $150 billion, the pressure on corporations to implement such capabilities, at least at a preliminary level, has been immense.
Another aspect of organizational awareness and acceptance of Big Data as a corporate mandate is the cultural perception of the utility of such tools. In a survey conducted with C-level management at large companies, most respondents stated that analytics was being used by managers in their departments, but there wasn't a uniform level of engagement across all departments, as shown in the following figure:
Furthermore, it appeared that the partnership strategies for Big Data across the organization were also not structured to the extent needed for commercial success. Respondents to the survey indicated that the partnership, as in, the cross-functional collaboration of analytics initiatives, was loosely defined:
While the responses to some of the questions were skewed, with one category being the overwhelming majority, the feedback on organizational challenges with Big Data and analytics in general had a broad uniform consensus. The following chart shows the feedback on analytics challenges from each participant in the survey:
All of this leads to an interesting but paradoxical analytics dilemma in the enterprise. Although the merit of Big Data and analytics is widely understood and accepted, there is a sense of ambiguity regarding the appropriate approach, as shown in the following diagram:
