One of the biggest problems that mid enterprise-sized organizations face is that data resides everywhere. Over the years, data has been accumulated usually by different systems, third-party, or in-house developed applications. Many vendors have set up a requirement to segregate their database servers in order to ensure performance, security, and management of their systems. Also, third-party vendors did not or do not want to take responsibility for their systems in a shared environment.
Organizations are starting to realize, or are already in the process of realizing, that consolidation is a must, both from the cost perspective as well as for easier manageability. However, in many cases, the vendors or developers are no longer to be found, which makes it very hard to make decisions to upgrade and/or migrate to the cloud. What could complicate things even further is the fact that shared or centralized data may be replicated everywhere and there may not even be one source of truth for the centralized data.
The bottom line is that if you have managed to complete a successful consolidation project with one source of truth, you are lucky and one of the few who have been able to achieve this goal!
On the other hand, it's the era of data analytics and reports, spanning multiple systems. Single sources of the truth are becoming more and more important. There are huge amounts of data to scan, summarize, analyze...
And so Microsoft came up with the Azure Data Lake, which is, in a nutshell, a cloud offering for big data that integrates with other Azure services such as: SQL database, SQL Server, SQL data warehouse, machine learning, Power BI, and Cortana. It also allows us to import and export data from almost any data source. Its main goals are ease of use and cost-effectiveness. The service has two main components:
- Data Lake Store (static)
- Data Lake Analytics component (paid on demand)
According to Microsoft at https://azure.microsoft.com/en-ca/solutions/data-lake/:
This chapter will primarily focus on the components of the Azure Data Lake and basic implementation of those components.
We will cover the following:
- Creating and configuring the Data Lake Store resource
- Creating and configuring the Data Lake Analytics resource
- Using data factory to create and configure Data Lake Store/Analytics
- Uploading data from an SQL Server Azure VM database into the data lake using blob storage
- Calling a U-SQL script to summarize data into a new file in the data lake using blob storage (this part requires Data Lake Analytics)
- Running U-SQL from a Data Lake Analytics job to do the following point
- Summarizing data into a new file in the data lake using blob storage (this part requires Data Lake Analytics)