Copying data using our data factory

OK, we're now ready to implement our first pipeline, that is, copy data from our SQL Server to our Azure storage account. We'll go back to the Azure portal and open our data factory. Once in there, we'll navigate to its details by clicking on Author & Monitor. As shown in the following screenshot, we'll select Copy Data from the Let's get started section:

The Copy Data wizard appears. As shown in the following screenshot, we'll fill in the Properties section and click Next:

The details to be filled in are as follows:

  • Task name: CopyPipeline_ADFV2Book
  • Task description: Enter a description, as shown in the preceding screenshot
  • Task cadence or Task schedule: Select Run once now

We're now directed to the Source blade. Select Azure SQL Database as our source, the one we created earlier in this chapter. Click Next:

Fill in the Connection properties as shown in the following list and click Next once done:

  • Connection name: Source_SQLAzure_wwimporters
  • Network environment: Public network in Azure Environment
  • Server/database selection method: From Azure subscriptions
  • Azure subscription: The current subscription used to create the factory
  • Server name: ADFV2Book; you should use another name since server names are unique in Azure
  • Database name: WideWorldImporters-Standard
  • Authentication: SQL Authentication

In the table selection blade, we'll click on USE QUERY and we'll enter the following query:

SELECT [CustomerID] 
      ,[CustomerName] 
      ,[CustomerCategoryName] 
      ,[PrimaryContact] 
      ,[AlternateContact] 
      ,[PhoneNumber] 
      ,[FaxNumber] 
      ,[BuyingGroupName] 
      ,[WebsiteURL] 
      ,[DeliveryMethod] 
      ,[CityName] 
  FROM [Website].[Customers] 
Table selection blade

You should get something similar to that shown in the following screenshot. Click Next:

Destination data store for copy task

In the Destination selection blade, we'll choose Azure Blob Storage and click Next, as shown in the following screenshot.

In the Dataset blade, fill in the properties as shown in the following screenshot the following list:

  • Connection name: Destination_BlobStorage_Customers
  • Network Environment: Public Network in Azure Environment
  • Account selection method: From Azure subscriptions
  • Azure subscription: The current subscription used to create the factory
  • Storage account name: Name of the storage account selected

Clicking Next will bring us to the destination file or folder choice. We'll use website-customer as the filename.

Clicking Next will bring us to the File format settings blade. Fill out the properties, as shown in the following screenshot and described later:

  • File format: We have several choices between text, Avro, JSON, ORC, and Parquet format; we'll use the Text format.
  • Column delimiter: We'll use the Pipe(|) delimiter. We could also use a custom delimiter if we check the Use custom delimiter checkbox.
  • Row delimiter: We'll use the default, Carriage return + Line feed ( ). Again, we could use a custom delimiter here. We'll check the Add header to file option.

Clicking Next will bring up the Settings blade. There, we can adjust fault tolerance, performance, and parallel settings. We'll keep the default values, as shown in the following screenshot:

Clicking Next will direct us to the Summary blade. As shown in the following screenshot, we can see a summary of our pipeline:

Summary blade (summary of pipelines)

Clicking Next will deploy the factory and it'll run the pipeline. Once the pipeline is triggered, we can click on the Monitoring icon at the left of the factory blade to go into the pipeline's run details, as shown in the following screenshot:

Factory blade (ADFV2Book)

Once the pipeline has run successfully, we'll return to our blob storage account. We click on Blobs in the blob service, as shown in the following screenshot:

This will open the Blob service pane. As shown in the following screenshot, we can see two containers. We'll select the import-data container to see its content. The website-customer blob appears on the right pane of the blade:

When we right-click on the file, a contextual menu appears. We'll select Edit, as shown in the following screenshot:

The edit blade opens for our website-customer blade, as shown in the following screenshot:

 

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.147.65.247