Example 1 – Debatching in Logic Apps with the splitOn property and Azure Functions change feed with Cosmos DB

The splitOn property in Logic Apps debatches an item array and starts a new instance of Logic Apps for each item in the array list. In this example, we will be using the Cosmos graph change feed as an example of an item array and use logic apps and Azure Functions together to build an integration that will write the changed feed data into the Azure SQL database.

Whenever there is a change to one or more user documents in the Cosmos graph database, Azure Functions will trigger a Logic Apps endpoint, which will have the splitOn property set on the request trigger.

Logic Apps will send an email notification to a user to indicate that their profile has been updated. These change requests will also be synced with the Logic Apps unique identifier into an Azure SQL database for reporting purposes. The following diagram shows the pattern that we are going to develop as part of this exercise, and the steps for doing so are listed here:

  1. Multiple users update their profiles in the Cosmos graph database.
  2. Azure Functions listen to the change feed array list item.
  1. Azure Functions call the Logic Apps HTTP endpoint, which has the splitOn feature enabled for the request trigger.
  2. The request messages are debatched, and for each profile update, an automatic trigger email is sent. 
  3. Each item's data is also published to a Service Bus topic, where Logic Apps will poll the request message and update the Azure SQL database:

The first part of this exercise involves developing a Cosmos DB change feed function that will invoke the Logic Apps endpoint in real time. We will extend the Cosmos database change feed code developed in Chapter 3, Introduction to Azure Event Gridto include the HTTP method. This HTTP method will POST the change feed document to the Logic Apps HTTP request trigger endpoint with the splitOn property set on triggerBody():

[FunctionName("cosmosgraphchangefeed")]
public static void Run([CosmosDBTrigger(
databaseName:"",
collectionName: "",
CreateLeaseCollectionIfNotExists = true,
ConnectionStringSetting = "cosmosgraph",
LeaseCollectionName = "leases")]IReadOnlyList<Document> documents, TraceWriter log)
{
if (documents != null && documents.Count > 0)
{
using (var client = new HttpClient())

{
var jsonString = JsonConvert.SerializeObject(documents);
var content = new StringContent(jsonString, Encoding.UTF8, "application/json");
var response = client.PostAsync(System.EnvironmentVariable("logicappsuri"), content).Result;
}
}
}

After creating Functions to listen for the change feed, the next step is to develop logic apps with the HTTP trigger and use Office 365 to send emails along with the Service Bus send message action. The overall Logic Apps flow diagram is shown here:

In the code of the Logic Apps workflow template, you need to set the splitOn property to split repetitive mode. In this case, as the Cosmos change feed is itself an array, we can split the array item at the triggers section:

The second Logic Apps workflow definition will poll the Service Bus topic for any new messages with a peek lock, so that the message is not lost if the change feed fails to be inserted into the SQL database:

Now, to test the splitOn function in Logic Apps, we have performed a bulk update on the users' profiles in the Cosmos graph database with the Gremlin query, which updates Azure Functions with an array of change feed documents. The run details are shown here:

When navigating to the Logic Apps run history, we can verify that multiple runs with splitOn on documents coming through Cosmos DB are being executed:

This can also be verified in SQL server. When you open up SQL server management studio and perform a select statement against the table, you can verify the run history along with the user profile information:

As you can see, with the splitOn feature, you can use the simple Logic Apps workflow to split the incoming JSON array data and have Logic Apps run the ID as a unique identifier for the debatch operation. When you have a central depository, such as an Azure data lake, a Cosmos DB instance, or a SQL server, you can use gain insights from data using the Logic Apps unique run ID for the debatch operations.

In the next section, we will cover concurrency control and singleton patterns, along with request data validation.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.144.12.14