Update Reference Data Input File for Azure Stream Analytics Jobs

In this section, we walk through the steps necessary to create an Azure Data Factory copy pipeline job that will copy data from an Azure DocumentDB “NoSQL” database to a text-based CSV file that is persisted in Azure blob storage.

Note that the CSV file will implement a specific file naming convention so that the Azure Stream Analytics job knows to utilize the latest version of the file for use in stream analytics jobs that require this reference data. Figure 6-8 illustrates the INPUT REFERENCE DATA parameter for the Azure Stream Analytics job that describes the file naming convention.

Figure 6-8. Stream Analytics reference data file naming convention

By utilizing this file naming convention, the reference data used as input in Azure Stream Analytics job will always reflect the most recent version of the data. Another advantage of using this approach is that future reference data updates can be easily made without adversely impacting any stream analytics jobs currently in process.

The next job that runs simply locates and ingests the latest reference data file that exists in Azure blob storage, based on the file naming convention. In this simple, but very effective, way, the stream analytics job will always use the latest version of the reference data at runtime.

Create Azure Data Factory Job

To get started, navigate to the resource group for your deployment via the Azure Portal. Click on the + Add button and search for Data Factory, as shown in Figure 6-9.

Figure 6-9. Searching and adding a Data Factory job to a resource group

After selecting Azure Data Factory, click on the next screen to create the new job, as shown in Figure 6-10.

Figure 6-10. Create Data Factory job

The next screen allows you to enter the specific parameters for creating a new Azure Data Factory job, as shown in Figure 6-11.

Figure 6-11. Data Factory create job parameters

Fill in your choices for the corresponding parameter values:

• Name: Enter a unique name for your new Data Factory. Note that the name of the Azure Data Factory must be globally unique.

• Subscription: The Azure subscription to use for this job.

• Resource Group: The Azure Resource Group to create this service in.

• Location: The Azure Data Center location.

Once you are done, click on the Create button at the bottom of the screen. Your input will then be validated and the new Azure Data Factory job will be created after a brief period of time. It should take less than one minute via the Azure Portal.

After your job has been provisioned, navigate to the new Data Factory via the Azure Portal and select the Copy Data (PREVIEW) option, as shown in Figure 6-12.

Figure 6-12. Data Factory: Copy Data Wizard

This will invoke the Azure Data Factory Copy Data Wizard to launch and will walk you through the steps necessary to create a basic copy pipeline. Behind the scenes, Azure Data Factory is generating JSON files to reflect your choices in the Copy Wizard.

Figure 6-13 depicts the first screen of the Copy Data Wizard and allows you to specify the properties for the copy job.

Figure 6-13. Data Factory Copy Data Wizard: specify properties

For this example, enter a task name of CopyReferenceData and keep the remaining defaults for the schedule, stat, and end dates. Click Next to advance to the next step.

Figure 6-14 depicts the Source Data screen, where you select Azure DocumentDB for the reference implementation scenario.

Figure 6-14. Data Factory Copy Data Wizard: specify source data store of Azure DocumentDB

After selecting Azure DocumentDB, you will then see a detailed screen similar to Figure 6-15, where you can specify the parameters for your DocumentDB instance to pull the reference data from.

Figure 6-15. Data Factory Copy Data Wizard: specify Azure DocumentDB parameters

Click the Next button after entering the DocumentDB parameters. You will see a screen similar to Figure 6-16.

Figure 6-16. Data Factory Copy Data Wizard: specify copy from tables or query

Click on the option to Use Query instead of the existing tables. You want to dynamically select the fields you need from the DocumentDB table. You will see a screen similar to Figure 6-17.

Figure 6-17. Data Factory Copy Data Wizard: specify query parameters

In the Query window, type in the following SQL statement:

SELECT
c.authid,
c.companyname,
c.firstname,
c.lastname,
c.username,
c.imageUrl,
c.type,
c.address.address1,
c.address.address2,
c.address.address3,
c.address.city,
c.address.state,
c.address.zip,
c.address.country,
c.social.phone,
c.social.email,
c.social.linkedin,
c.social.facebook,
c.social.twitter,
c.social.blog,
c.healthInformation.age,
c.healthInformation.height,
c.healthInformation.weight,
c.healthInformation.gender,
c.healthInformation.race,
c.location.longitude,
c.location.latitude,
c.id,
c.cachettl,
c._rid,
c._self,
c._etag,
c._attachments,
c._ts
FROM
c
WHERE
c.type <> 1

For the table name, select Profile . Then click on Validate Query to test your SQL syntax. If there are no errors, the button will transition from Validating… back to Validate Query.

Click on the Next button to advance to the next screen where you will specify the destination data store, as shown in Figure 6-18.

Figure 6-18. Data Factory Copy Data Wizard: specify destination data store

Select Azure Blob Storage and then click on the Next button to advance to the next screen. The next will ask for your Azure blob storage account specifics, as shown in Figure 6-19.

Figure 6-19. Data Factory Copy Data Wizard: specify Azure blob storage account properties

Enter your Azure blob storage account specifics and then click on the Next button to advance to the next screen, as shown in Figure 6-20. This is where you specify the folder and file names for the destination in Azure blob storage.

Figure 6-20. Data Factory Copy Data Wizard: specify output file or folder properties

Enter refdata-team as the folder path and TeamReferenceData.csv as the file name for the destination outputs. Then click on the Next button to advance to the next screen, as shown in Figure 6-21, where you will specify the file format settings.

Figure 6-21. Data Factory Copy Data Wizard: specify file format settings

Keep the defaults for the file format settings options, but be sure to check the option for Add Header to File so that the column names are preserved.

Click on the Next button to advance to the next screen, performance settings, as shown in Figure 6-22.

Figure 6-22. Data Factory Copy Data Wizard: specify performance settings

For the performance settings, keep the defaults and click the Next button. At this point, you will see a summary page that recaps all of the properties and settings you specified for this new copy job, as shown in Figure 6-23.

Figure 6-23. Data Factory Copy Data Wizard summary page

If all of the settings look good, click on the Finish button. The new Data Factory copy job will be validated and deployed. When that’s complete, you will see a screen similar to Figure 6-24.

Figure 6-24. Data Factory Copy Data Wizard, deployment complete

Modify JSON Parameters for Copy Job

At this point, you have created the basic Azure Data Factory copy pipeline job for the reference implementation to refresh the reference data used in the Stream Analytics job in Chapter 5. This copy pipeline will select data from of the Azure DocumentDB database (via a SQL query statement) and then write it out to a CSV file in Azure blob storage.

The only minor change left is to tweak the JSON output parameters to create the output file name in Azure blob storage using a file naming convention pattern as the one shown:

TeamReferenceDatayyyy-MM-ddhh-mm.csv

Note the following the date/time naming format:

• yyyy = Year

• MM = Month

• dd = Day

• hh = Hour

• mm = Minute

This naming convention change is crucial to ensure that you don’t try to update the file while it is in use and may be locked. It also allows you to build a historical inventory of the previous versions.

To make the change, you need to the JSON of the new Azure Data Factory pipeline job, navigate to the new pipeline job via the Azure Portal, and select the Author and Deploy option, as shown in Figure 6-25.

Figure 6-25. Data Factory, Author and Deploy option

After you select this option, you will see a screen similar to Figure 6-26. Here, you can click on each of the components of the pipeline and expand the parameters underneath each section .

Figure 6-26. Data Factory, view data factory components

As you click on each section, you will see the corresponding JSON template and parameters on the right side of the Azure Portal web page.

Click on the OutputDataset-cdw under the Datasets section to see the corresponding JSON. Scroll all the way down to the bottom and you should see a section of code similar to Figure 6-27.

Figure 6-27. Data Factory: default output file naming in JSON

Note the fileName parameter, which is set to the value of TeamReferenceData.csv. This will be the section of JSON code you will modify to meet the file naming convention of TeamReferenceData" + "yyyy-MM-ddhh-mm.csv". To do this, find and replace the previous JSON code with the following JSON code:

"typeProperties": {
    "fileName": "TeamReferenceData{slice}.csv",
    "folderPath": "refdata-team",
    "format": {
        "type": "TextFormat",
        "columnDelimiter": ",",
        "firstRowAsHeader": true
    },
    "partitionedBy": [
        {
            "name": "slice",
            "value": {
                "type": "DateTime",
                "date": "SliceStart",
                "format": "yyyy-MM-ddhh-mm"
            }
        }
    ]
},

Note that the JSON code you replaced will use a dynamic file naming convention based on the Date and Time attributes for the specific processing slice that is created from the Data Factory pipeline job.

After you make this update to the JSON, the Deploy option will become available, as shown in Figure 6-28.

Figure 6-28. Data Factory: deploy updated JSON

After selecting the option to deploy your updated pipeline JSON data to Azure, the JSON will be saved, validated, and then deployed to the Azure Data Factory service.

Run Data Factory Copy Job On-Demand and Check Results

To run this new Azure Data Factory pipeline job, navigate to the job via the Azure Portal and select the Monitor & Manage option, as shown in Figure 6-29.

Figure 6-29. Data Factory’s Monitor & Manage option

After selecting this option, a new tab will open in your browser. The Data Factory Resource Explorer App will open and display your Data Factory jobs and their corresponding pipeline definitions and activities, as shown in Figure 6-30.

Figure 6-30. Data Factory, re-run job activity

To run this updated copy pipeline job “on demand,” do the following:

• Select the latest pipeline activity.

• Click on the Rerun icon, as highlighted in red in Figure 6-30.

• Choose either Rerun or Rerun with Upstream Data.

Note When you select the Rerun with Upstream in Pipeline option, it reruns all upstream activity windows as well.

After a few minutes, the copy pipeline job will run to completion. At this point, you can check the Azure blob output destination container refdata-team and look for a file with a naming convention that follows TeamReferenceDatayyyy-MM-ddhh-mm.csv. The screenshot in Figure 6-31 displays successful output of several Team Reference Data CSV files over a period of three days using the Visual Studio Cloud Explorer tool to view the output blob container.

Figure 6-31. Data Factory: successful daily outputs of team reference data CSV output files

To summarize this exercise, you just walked through all the steps necessary to update the reference data that you used for the Azure Stream Analytics job in Chapter 5.

Implementing the Data Factory Azure Ml Update Resource Activity

In this next section, we tackle the second objective for the Azure Data Factory pipeline tasks, which is to re-train the Azure Machine Learning model via an Azure Data Factory pipeline job. The updates to the Azure Machine Learning model will come from the results of medical stress tests that are administered to team members on a periodic basis and then uploaded into Azure blob storage for re-training the model.

We cover the detailed specifics of implementing the Azure Machine Learning model and associated Web Services in Chapter 9.

For this exercise, we assume that the Azure ML predictive model has already been extended with an additional web service endpoint. The additional endpoint will allow for re-training the model (in batch mode) based on recently updated training data.

Data Factory AML Retraining: High-Level Design

At a high level, we are going to create a Data Factory pipeline job that will accomplish two objectives in order to fully re-train our Azure ML model :

1. Process the updated Machine Learning training data and produce an .iLearner ML output file. This file then becomes the input to the Update Resource activity in the next step.

2. Add a second Update Resource Activity to the pipeline to update the existing Azure ML Web Service with the updated trained model via the .iLearner ML output file .

Create Data Factory Job to Retrain Azure ML Model

To get started, navigate to the resource group for your deployment via the Azure Portal. Click on the + Add button and search for Data Factory, as shown in Figure 6-32.

Figure 6-32. Searching for and adding a Data Factory job to a resource group

After selecting Azure Data Factory, you will see a Data Lake overview screen. Click on the Create button to create the new Data Factory job.

The next screen allows you to enter the specific parameters for creating a new Azure Data Factory job, as shown in Figure 6-33.

Figure 6-33. Data Factory create job parameters

Fill in your choices for the corresponding parameter values:

• Name: Enter a unique name for your new Data Factory. Note that the name of the Azure Data Factory must be globally unique. We use ReTrainMLModel.

• Subscription: The Azure subscription to use for this job.

• Resource Group: The Azure Resource Group to create this service in.

• Location: The Azure Data Center location.

Once you are done, click on the Create button at the bottom of the screen. Your input will then be validated and the new Azure Data Factory job will be created after a brief period of time.

After your job has been provisioned, navigate to the new Data Factory via the Azure Portal and select the Author and Deploy option, as shown in Figure 6-34.

Figure 6-34. Data Factory Author and Deploy option

Next, you are going to build the Data Factory pipeline components using just JSON to define the individual elements of the Data Factory Azure ML re-training pipeline job.

Define Linked Service: Azure Storage

To define the Linked Service for Azure Storage, click on the New Data Store icon in the top navigation bar and then select Azure Storage, as shown in Figure 6-35.

Figure 6-35. Data Factory: add a new data store

After your new linked service has been created, replace the default JSON with the JSON code shown here. Note that you need to have your specific Azure blob storage credentials to fill in.

{
    "name": "AzureStorageLinkedService",
    "properties": {
        "description": "",
        "hubName": "retrainmlmodel_hub",
        "type": "AzureStorage",
        "typeProperties": {
            "connectionString": "DefaultEndpointsProtocol=https;AccountName=<YourAccountName>;AccountKey=<YourAccountKey>"
        }
    }
}

Next, click on the Deploy icon in the top navigation bar and your JSON will be uploaded, validated, and deployed.

Define Input Dataset: Updated Azure ML Training Data

To define a new dataset, click on the …More icon and then click on the New Dataset icon, as shown in Figure 6-36.

Figure 6-36. Data Factory: add new dataset, training data

Next, select the option for Azure Blob Storage, as shown in Figure 6-37.

Figure 6-37. Data Factory: add new dataset, Azure blob storage option

After your new dataset has been created, replace the default JSON with the JSON code shown here.

{
    "name": "trainingData",
    "properties": {
        "published": false,
        "type": "AzureBlob",
        "linkedServiceName": "AzureStorageLinkedService",
        "typeProperties": {
            "fileName": "RETRAIN_Teammates_AML_Training_Data.csv",
            "folderPath": "aml-retrain-data",
            "format": {
                "type": "TextFormat"
            }
        },
        "availability": {
            "frequency": "Week",
            "interval": 1
        },
        "external": true,
        "policy": {
            "externalData": {
                "retryInterval": "00:01:00",
                "retryTimeout": "00:10:00",
                "maximumRetry": 3
            }
        }
    }
}

Make sure the fileName and folderPath parameters are set for your environment. Note that the frequency will be once per week.

Next, click on the Deploy icon in the top navigation bar and your JSON will be uploaded, validated, and deployed to the Data Factory definition.

Define Output Dataset: Updated Azure ML Training Model

This dataset definition will represent the output .iLearner file from the Azure ML training web service. The Azure ML Batch Execution Activity produces this dataset. This dataset will also serve as the input file for the Azure ML Update Resource activity.

Create an additional dataset for our Data Factory job by following the same instructions to create a new dataset and summarized here:

• Click on the …More icon and then click on the New Dataset icon.

• Select the option for Azure Blob Storage.

After your new dataset has been created, replace the default JSON with the JSON code shown here.

{
    "name": "trainedModelBlob",
    "properties": {
        "published": false,
        "type": "AzureBlob",
        "linkedServiceName": "AzureStorageLinkedService",
        "typeProperties": {
            "fileName": "model.ilearner",
            "folderPath": "aml-retrain-data",
            "format": {
                "type": "TextFormat"
            }
        },
        "availability": {
            "frequency": "Week",
            "interval": 1
        }
    }
}

Make sure the folderPath parameter is set correctly for your environment.

When finished, click on the Deploy icon in the top navigation bar and your JSON will be uploaded, validated , and deployed to the Data Factory definition.

Define Linked Service: Azure ML Training Endpoint

Next, you will create a linked service that points to the default endpoint of the Azure ML training web service.

To get started, click on the …More icon and then click on the New Compute icon. Then select the Azure ML option, as shown in Figure 6-38.

Figure 6-38. Data Factory: add new compute Azure blob storage option

After your new linked service has been created, replace the default JSON with the JSON code shown here. Note that you will need to have your specific Azure ML URL endpoint and API key to fill in the parameters:

{
    "name": "trainingEndpoint",
    "properties": {
        "hubName": "retrainmlmodel_hub",
        "type": "AzureML",
        "typeProperties": {
            "mlEndpoint": "<YourEndPointURL>",
            "apiKey": "<YourAPIKey>"
        }
    }
}    }
}

Next, click on the Deploy icon in the top navigation bar and your JSON will be uploaded, validated , and deployed.

Define Linked Service: Azure ML Updatable Scoring Endpoint

Next, you will create a linked service that defines an Azure Machine Learning linked service that points to the non-default updatable endpoint of the scoring Azure ML Web Service.

Note Before creating and deploying an Azure ML linked service, follow the steps in this link to create a second (non-default and updatable) endpoint for the Azure ML Scoring Web Service.

https://docs.microsoft.com/en-us/azure/machine-learning/machine-learning-create-endpoint .

To get started, click on the …More icon and then click on the New Compute icon. Then select the Azure ML option, as shown in Figure 6-38.

After your new linked service has been created, replace the default JSON with the JSON code shown here. Note that you will need to have your specific Azure ML URL endpoint and API key to fill in the parameters:

{
    "name": "updatableScoringEndpoint2",
    "properties": {
        "hubName": "retrainmlmodel_hub",
        "type": "AzureML",
        "typeProperties": {
            "mlEndpoint": "<YourMLRetrainingEndpoint>",
            "apiKey": "<YourMLRetrainingAPIKey>",
            "updateResourceEndpoint": "<YourMLRetrainingURLEndpoint>"
        }
    }
}

Next, click on the Deploy icon in the top navigation bar and your JSON will be uploaded, validated, and deployed.

Define Output Dataset: Dummy Azure Blob Output

At the time of this writing, when you include an Azure ML Update resource activity in a Data Factory pipeline job, it does not generate any output.

However, Azure Data Factory requires an output dataset in order to drive the schedule of a pipeline. Therefore, we will implement a dummy/placeholder Azure blob dataset to handle this use case.

To define a new Dataset, click on the …More icon and then click on the New Dataset icon, as shown in Figure 6-39.

Figure 6-39. Data Factory: add new dataset, dummy output data

Next, select the option for Azure Blob Storage, as shown in Figure 6-40.

Figure 6-40. Data Factory: add new dataset, Azure blob storage option

After your new dataset has been created, replace the default JSON with the JSON code shown here.

{
    "name": "DummyPlaceholderBlob",
    "properties": {
        "published": false,
        "type": "AzureBlob",
        "linkedServiceName": "AzureStorageLinkedService",
        "typeProperties": {
            "fileName": "dummyfile.csv",
            "folderPath": "aml-retrain-data",
            "format": {
                "type": "TextFormat"
            }
        },
        "availability": {
            "frequency": "Week",
            "interval": 1
        }
    }
}

Make sure the fileName and folderPath parameters are set for your environment. Note that the frequency will be once per week.

Next, click on the Deploy icon in the top navigation bar and your JSON will be uploaded, validated, and deployed to the Data Factory definition.

Define Data Factory Pipeline Job with Two Activities

Now, you will combine all the previously defined linked services and dataset definitions as you define a new Data Factory pipeline job.

The new Data Factory pipeline job will have two activities defined:

• AzureMLBatchExecution: The Azure ML Batch Execution activity takes the updated team health training data from Azure blob storage as input, and then produces an .iLearner file as an output.

• AzureMLUpdateResource: This activity takes the .iLearner file as input and then sends it to the Azure ML Training web service to update the ML model.

Note: The placeholderBlob is just a dummy output dataset that is required by the Azure Data Factory service to run the pipeline.

To define the new pipeline, right-click on the Pipelines section of the left navigation bar of the Authoring pane and then select New Pipeline, as shown in Figure 6-41.

Figure 6-41. Data Factory: add new pipeline

After your new pipeline has been created, replace the default JSON with the JSON code shown here.

{
    "name": "retrainmlpipeline",
    "properties": {
        "activities": [
            {
                "type": "AzureMLBatchExecution",
                "typeProperties": {
                    "webServiceInput": "trainingData",
                    "webServiceOutputs": {
                        "output1": "trainedModelBlob"
                    },
                    "webServiceInputs": {},
                    "globalParameters": {}
                },
                "inputs": [
                    {
                        "name": "trainingData"
                    }
                ],
                "outputs": [
                    {
                        "name": "trainedModelBlob"
                    }
                ],
                "policy": {
                    "timeout": "02:00:00",
                    "concurrency": 1,
                    "executionPriorityOrder": "NewestFirst",
                    "retry": 1
                },
                "scheduler": {
                    "frequency": "Week",
                    "interval": 1
                },
                "name": "retraining",
                "linkedServiceName": "trainingEndpoint"
            },
            {
                "type": "AzureMLUpdateResource",
                "typeProperties": {
                    "trainedModelDatasetName": "trainedModelBlob",
                    "trainedModelName": "Training Exp for ADF ML [trained model]"
                },
                "inputs": [
                    {
                        "name": "trainedModelBlob"
                    }                                                                                                    
                ],
                "outputs": [
                    {
                        "name": "DummyplaceholderBlob"
                    }
                ],
                "policy": {
                    "timeout": "01:00:00",
                    "concurrency": 1,
                    "retry": 3
                },
                "scheduler": {
                    "frequency": "Week",
                    "interval": 1
                },
                "name": "AzureML Update Resource",
                "linkedServiceName": "updatableScoringEndpoint2"
            }
        ],
        "start": "2016-02-13T00:00:00Z",
        "end": "2016-02-14T00:00:00Z",
        "isPaused": false,
        "hubName": "retrainmlmodel_hub",
        "pipelineMode": "Scheduled"
    }
}

Make sure the fileName and folderPath parameters are set for your environment. Note that the frequency will be once per week.

Next, click on the Deploy icon in the top navigation bar and your JSON will be uploaded, validated, and deployed to the Data Factory definition.

At this point, you have manually created a complete Data Factory pipeline job (described via JSON) to update the Azure ML training service.

The Data Factory pipeline is composed of the following components , as shown in Figure 6-42.

• Three linked services

• Three datasets

• One pipeline

Figure 6-42. Data Factory pipeline components

In order to get a visual representation of the Azure Data Factory and components you have created, navigate to your Data Factory job in the Azure Portal and select the Diagram option, as shown in Figure 6-43.

Figure 6-43. Data Factory Diagram view

After clicking on the Diagram icon, you will see a representation of your Data Factory pipeline, as shown in Figure 6-44.

Figure 6-44. Data Factory diagram, open pipeline option

However, you will notice that this view is still collapsed, as the pipeline icon states it has two activities. To expand the pipeline, right-click on the pipeline icon and select Open Pipeline, as shown in Figure 6-44.

When you click on the Open Pipeline option, the screen will expand to auto-fit the viewing window in order to reveal the entire pipeline. Figure 6-45 illustrates the new view.

Figure 6-45. Data Factory diagram, open pipeline view

Note that this view is extremely helpful when assembling multi-step pipeline jobs using Azure Data Factory. You can also adjust the zoom levels and drill down into individual components.

To summarize, in this section, you created a new Data Factory pipeline job to automatically update the Machine Learning Web Service using the following JSON definitions :

• Linked Service: Azure Storage

• Linked Service: Azure ML Training Endpoint

• Linked Service: Azure ML Updatable Scoring Endpoint

• Input Dataset: Updated Azure ML Training Data

• Output Dataset: Updated Azure ML Training Model

• Output Dataset: Dummy Azure Blob Output

• Data Factory Pipeline Job: With two Activities

This piece of the reference implementation provides a completely automated method of retraining the Azure Machine Learning Web Service.

The updates are based on updated team health data that is generated from periodic stress test results administered to team members. This creates a “full lifecycle” solution to maintaining an updated Azure ML training models based on physical data updates.

Note See the following link for more details about configuring an Azure Data Factory pipeline job to retain an Azure ML Web Service:

https://docs.microsoft.com/en-us/azure/data-factory/data-factory-azure-ml-batch-execution-activity#updating-azure-ml-models-using-the-update-resource-activity .

Move Data From Blob Storage to Data Lake

The third and last Data Factory job we will implement in this chapter focuses on moving data from Azure blob storage to Azure Data Lake.

This Data Factory will prepare the reference implementation data for the next subject covered in Chapter 7.

As a refresher, the data in Azure blob storage that you will move to Azure Data Lake was created as an output result of the Azure Stream analytics job created in Chapter 5.

You will move this data to Azure Data Lake for several reasons, including the following:

• Data Archival: Keep the data in its original form as when it was received from the IoT Hub.

• Deep Analytics: Azure Data Lake is both a powerful storage and powerful analytics platform, as we will explore more in Chapter 7.

• Regression Analysis: Oftentimes, there is value in being able to re-run Big Data analysis over historical data to evaluate alternative outcomes or make other make other (historical) improvements.

Create Azure Data Lake Store Account

Before you can copy your data from Azure blob storage to Azure Data Lake, you need to create an Azure Data Lake Store account.

To get started, navigate to the resource group for your deployment via the Azure Portal. Click on the + Add button and search for Data Lake, as shown in Figure 6-46.

Figure 6-46. Searching for and adding a Data Lake Store to a resource group

Next, you will see an overview page about Azure Data Lake Store. Click on the Create button to advance to the next screen.

Figure 6-47. Parameters for creating a new Data Lake Store

Fill in your choices for the corresponding parameter values:

• Name: Enter a unique name for your new Data Lake Store.

• Subscription: The Azure subscription to use for this job.

• Resource Group: The Azure Resource Group to provision this resource in.

• Location: The Azure Data Center location.

Once you are done, click on the Create button at the bottom of the screen. Your input will then be validated and the new Azure Data Lake Store will be created in a few minutes at most.

Create Data Factory Job: Copy from Azure Blob to Data Lake

To get started, navigate to the resource group for your deployment via the Azure Portal. Click on the + Add button and search for Data Factory, as shown in Figure 6-48.

Figure 6-48. Searching for and adding a Data Factory job to a resource group

After selecting Azure Data Factory, click on the Create button to create the new job.

The next screen will allow you to enter the specific parameters for creating a new Azure Data Factory job, as shown in Figure 6-49.

Figure 6-49. Data Factory create job parameters

Fill in your choices for the corresponding parameter values:

• Name: Enter a unique name for your new Data Factory job. Note that the name of the Azure Data Factory must be globally unique.

• Subscription: The Azure subscription to use for this job.

• Resource Group: The Azure Resource Group to create this service in.

• Location: The Azure Data Center location.

Once you are done, click on the Create button at the bottom of the screen. Your input will then be validated and the new Azure Data Factory job will be created after a brief period of time. It should take less than one minute via the Azure Portal.

After your job has been provisioned, navigate to the new Data Factory via the Azure Portal and select the Copy Data (PREVIEW) option, as shown in Figure 6-50.

Figure 6-50. Data Factory Copy Data Wizard

This will invoke the Azure Data Factory Copy Data Wizard to launch and will walk you through the steps necessary to create a basic copy pipeline. Behind the scenes, Azure Data Factory is generating JSON files to reflect your choices in the Copy Wizard.

Figure 6-51 depicts the first screen of the Copy Data Wizard and allows you to specify the properties for the copy job.

Figure 6-51. Data Factory Copy Data Wizard: specify properties

For this example, enter a task name of CopyBlobToDataLake and keep the remaining defaults for the schedule, stat, and end dates. Click Next to advance to the next step.

Figure 6-52 depicts the Source Data screen, where you should select Azure Blob Storage for the reference implementation scenario.

Figure 6-52. Data Factory Copy Data Wizard: specify source data store for Azure blob storage

Next, specify the Source Data Store properties for Azure blob storage, as shown in Figure 6-53.

Figure 6-53. Data Factory Copy Data Wizard: specify source data store for Azure blob storage properties

Next, choose the folder in Azure blob storage that will contain the source dataset, as shown in Figure 6-54.

Figure 6-54. Data Factory Copy Data Wizard: specify a source folder in blob storage

After selecting the streamingdata folder and clicking on Choose, click on the Next button. You will see the screen in Figure 6-55.

Figure 6-55. Data Factory Copy Data Wizard: specify a source folder and an option for copying files recursively

Select the option to Copy Files Recursively, as shown in Figure 6-55, then click on the Next button.

At this point, the Data Factory Copy Wizard will attempt to connect to the Azure blob storage folder and automatically detect the file format of the files in blob storage, as shown in Figure 6-56.

Figure 6-56. Data Factory Copy Data Wizard: file format settings

If the settings do not match, you may need to review the Azure blob storage output from the Stream Analytics job defined in Chapter 5.

If the file format settings match your expected inputs, click on the Next button to select the destination data store, as shown in Figure 6-57.

Figure 6-57. Data Factory Copy Data Wizard: select Data Lake destination

Click on the Azure Data Lake Store icon and then click on the Next button to select Data Lake as the output destination.

The next option you’ll define in the Copy Activity Wizard is for the destination Azure Data Lake Store account, as shown in Figure 6-58.

Figure 6-58. Data Factory Copy Data Wizard: select Data Lake destination

Select your Azure subscription and Azure Data Lake Store Name and then click on the Next button.

The next screen in the Copy Wizard will allow you to select the output destination folder and file path for the destination data in Azure Data Lake.

Set the Filename parameter to the value of inputfolder/{year}/{month}/{day}, as shown in Figure 6-59.

Figure 6-59. Data Factory Copy Data Wizard: select a Data Lake Destination folder and file name values

Click on the Next button to advance to the next screen in the Copy Wizard. Here, you will see options for specifying the file format settings. Select the option for Add Header to File, as shown in Figure 6-60.

Figure 6-60. Data Factory Copy Data Wizard: specify Data Lake destination file format values

Next, you will see a screen where you can adjust the performance settings for the copy job. Accept the default values of Auto for Cloud Units and Parallel Copies and click on the Next icon, as shown in Figure 6-61.

Figure 6-61. Data Factory Copy Data Wizard: specify Data Lake destination file format values

The last screen in the Copy Wizard is the summary screen, as shown in Figure 6-62.

Figure 6-62. Summary screen of the Data Factory Copy Data Wizard

First, click on the Authorize button, as shown in Figure 6-62 so that you can sign in to the Azure subscription again and the copy job can capture the credentials.

Finally, click on the Finish button to create the new Data Factory pipeline job.

After a brief period of time, the new Data Factory copy job will be registered and deployed, as shown in Figure 6-63.

Figure 6-63. Deployment complete

After your new pipeline job has been deployed, it will run and you can check for the output in your destination Azure Data Lake Store.

Figure 6-64 depicts an example of the updated Data Lake Store copy results using the Visual Studio Cloud Explorer to view the destination output in the Data Lake Store folder.

Figure 6-64. Visual Studio Cloud Explorer: view Data Lake Store

At this point, you have now completed the third and final Data Factory use case scenario for the reference implementation.

Note See the following link for more detailed information.

“Move Data to and from Azure Data Lake Store Using Azure Data Factory”: https://docs.microsoft.com/en-us/azure/data-factory/data-factory-azure-datalake-connector .