Although it may feel like Hadoop is just a few years old going by the market buzz, in truth it is more than 10 years old. In 2003, Yahoo was doing a project called Nutch to address their web-search performance and scalability issue.

In 2003 and 2004, Google published its paper on Google's filesystem and MapReduce, respectively.

Nutch's developers started making the change using the design from Google's papers, which helped the process of scaling to more than just one machine. The solution worked to a very large extent, but still it wasn't a web-scale solution.

In 2006, Yahoo hired Doug Cutting to improve the web-search solution. He liaised with Apache to develop an open source project and took out the storage and processing parts from Nutch to create a generic reusable framework. Interestingly, Doug named the open source project Hadoop after his son's toy elephant and even he couldn't have thought that the name would become the top buzzword in the world of information technology a few years later. Doug's team improved the Hadoop framework and released a truly web-scale solution in 2008.

Yahoo kept on migrating its applications onto Hadoop and by 2011 it was running its search engine on 42,000 nodes containing hundreds of petabytes of data storage.

Almost every large web giant, such as Facebook, Google, Microsoft, and eBay, is exploiting web data using Hadoop as its framework. According to an IDC survey, 63.3 percent of all organizations have either deployed or are planning to deploy Hadoop in the next year. The remaining 36.6 percent are planning to implement it as well but may take more than a year to do so.

There are numerous Hadoop distribution companies such as Cloudera and HortonWorks that make the deployment and management of Hadoop a lot easier. They also provide software support and help in the integration of Hadoop into an organization's IT landscape.

IDC research forecasts data storage will see the biggest growth at 53.4 percent compound annual growth rate for at least another few years.

They also forecast that the big data technology and services market will grow at 27 percent compound annual growth rate to $32.4 billion through 2017—about six times the growth rate of the overall information and communication technology market.

At the moment, there are at least dozens of start-ups with very unique products and they have secured hundreds of millions of venture capital.

However you define it, though, there's no denying that big data will be a huge market and will attract the majority of technology expenditure for an organization.