Hour 13 Learning to Track Workflows
What You’ll Learn in this Hour:
Tracking architecture
How to create TrackingProfiles with the TrackingProfileDesigner
How to manually modify TrackingProfiles and update them to the tracking database
How to extract and query workflow and activity properties
How to create and query user tracking records
Tracking underpins WF’s runtime transparency. Tracking’s runtime transparency capabilities also play a major role in WF’s capability to offer agility. Dynamic Update discussed in the next hour, graphical workflow construction, and XAML workflows (which do not require compilation) all aid agility. Although useful, these features can wreak havoc without a control mechanism. Through Tracking, these workflows can be logged, audited, and versioned, which enables agile, governable processes. This hour covers Tracking, and the next hour shows you how Tracking teams with Dynamic Update to allow constantly changing applications.
Tracking was looked at in Hour 5, “Creating an Escalation Workflow.” As you saw, registering the out-of-the-box (OOB) SqlTrackingService with the WF runtime extracts workflow information and saves it to a SQL database, where it can be monitored, analyzed, and used for other purposes. You populated a text box with all workflow and activity events. Then you used the WorkflowMonitor SDK application that graphically displays previously executed and the already executed activities on active workflows. The WorkflowMonitor application highlighted the runtime transparency Tracking offers.
This hour first begins with an overview of the WF Tracking architecture and a description of business activity monitoring (BAM). The main emphasis of this hour is using the TrackingProfileDesigner SDK sample to create a custom TrackingProfile. Custom TrackingProfiles are used to control what information is tracked. You will use the tool to create TrackingProfiles that both limit and enrich what data is tracked. Limiting generally involves tracking only selected events, such as closed and executing. Enrichment involves including workflow and activity properties, such as order number and order amount. You will also learn to manually modify the TrackingProfiles the tool produces and to update them to the tracking database via a Windows Forms app. This is useful because, like most tools, there are scenarios where the TrackingProfileDesigner will get you part way, but not all the way, to a solution. Then you will learn to create and query UserTrackingRecords. UserTrackingRecords are similar to the WorkflowTrackingRecords and ActivityTrackingRecords you worked with in Hour 5. They are generated through custom code and RuleSets. Finally, you will learn another form of filtering using predicates to determine what data is tracked.
If you are wondering why you would ever want to limit or enrich the information tracked, following are three reasons:
There is a performance cost associated with emitting all workflow, activity, and user events. Therefore, it may be worthwhile limiting the activities and events tracked to those you need. For instance, you only need to track the executing and closed events to use the WorkflowMonitor SDK sample. Tracking the unload/reload/persistence cycle may be unnecessary overhead if you want to use only the WorkflowMonitor.
You may want to include contextual information, such as purchase order and amount.
You may want to include a customized tracking service in one or more workflows that, for instance, listen for aborted events. You could then email or take other actions. You can register multiple tracking services with one workflow, so you could register an aborted tracking service with all workflows.
Let’s take a look at how Tracking works. The WF runtime evaluates running workflows to determine which information has been selected for tracking. Records that match the criteria are sent to the host. The host is then responsible for storing the information. The reason records are evaluated by the runtime is to avoid incurring unnecessary overhead when sending data to the host that it does not want. The host can persist to any medium it chooses, although the only OOB implementation uses SQL Server. Figure 13.1 illustrates the WF Tracking architecture.
TrackingProfiles (the text above the diamond) are used to specify what information should be tracked.
Matching TrackingRecords are sent over a TrackingChannel to the host.
The TrackingService registered with the host then stores the received TrackingRecord to the storage medium specified by the TrackingService. (If multiple TrackingServices are registered, each may have its own profile that dictates what events are routed to it.)
You can store information to Oracle, MySQL, file systems, and other destinations by creating a custom TrackingService. Creating a custom TrackingService is not covered in this book. If you are interested in learning how to create custom TrackingServices to persist tracking information to an alternative medium, you can look at the ConsoleTrackingServiceSample and FileTrackingService SDK samples.
Static models are useful. For many reasons, binding execution to models is frequently more useful. You can also use the same models, or permutations thereof, for analytical and reporting needs. For one, this helps create one version of the truth, because both the executable process and reports are produced from the same information source. In fact, many business process management systems feature sophisticated process monitoring capabilities that combine running process information with business intelligence and other tools to provide real-time business reporting, query capabilities, and alerts. Business Process Management (BPM) products generally refer to this feature as BAM, and in other parlors they are referred to as digital dashboards.
BAM consists of three parts: extraction, persistence, and reporting. The first two parts are covered very well by WF Tracking, where its extensibility makes it shine. WF is not a major player in the third. The SqlTrackingQuery and the WorkflowMonitor SDK application are not full-fledged reporting and analytics tools, like the ones found in BAM products. As a platform, WF must ensure that rich BAM reporting tools can be built on it—which it does with excellent extraction and solid persistence capabilities—but leaves the actual rich reporting steps to others to build on top of it.
The TrackingProfileDesigner tool simplifies creating TrackingProfiles. You click an activity or the workflow and specify which events and properties you want to track.
Functional TrackingProfiles can be created using the TrackingProfileDesigner. The TrackingProfiles it produces can also be analyzed to understand the inner workings of TrackingProfiles. They can also be modified to produce a workable solution when the TrackingProfileDesigner comes close but cannot quite meet your objective (a common pattern with many graphical tools).
Throughout this section, you will create various custom TrackingProfiles in the TrackingProfileDesigner tool. Using the TrackingProfileDesigner requires loading the TrackingProfileDesigner SDK application itself. Then you tell the application which workflow you want to track. The last step is to save the TrackingProfile to the tracking database. The workflow will now be associated with the TrackingProfile you created. The information tracked for this workflow will be determined by the TrackingProfile you created.
You will create two TrackingProfiles in this section. In the first, you specify which workflow and activity events are to be tracked. In the second, you incorporate order number and other workflow properties, and you apply conditions to what is tracked.
Before creating a custom TrackingProfile, follow the next steps to take a look at the solution that will be used to support our effort this hour:
1. Open the TrackingWorkflow in the TrackingWorkflows project in design mode. Then click the codeActivityUserData activity and set its Enabled property to false.
2. Open the TrackingSolution in the C:SamsWf24hrsHoursHour13TrackingTrackingSolution directory.
3. Open TrackingForm.cs in the TrackingHostForm project. It serves a dual role as the host for the tracking workflow that will be used in this hour. It also holds the text box where tracking information can be displayed.
4. The TrackingLocalServices project holds the local service for host-workflow communication.
5. Form1.cs in the TrackingProfileForm project is used to manually update TrackingProfiles to the tracking database.
6. The TrackingWorkflows project holds the tracked workflow. It contains specific activities that force persistence, and it creates UserTrackingRecords.
7. Run the project (make sure the TrackingHostForm is set as the startup project), enter any order number, any order amount, and click Submit Order.
8. Wait until the workflow completion dialog is spawned, and then click OK. (It will take a little while because the workflow contains a Delay activity.)
9. Click the Get Tracking button and you should see results similar to Figure 13.2. This is what is tracked by the default TrackingProfile. You get this by registering the SqlTrackingService with the runtime.
The key is that all workflow events, all activities, and all activity events are tracked. User events (which produce UserTrackingRecords) are tracked as well, but are not shown because we have not produced any yet. We add user events later in this hour.
Follow the next steps to open the TrackingProfileDesigner and use it to graphically create a custom TrackingProfile.
1. Open the TrackingProfileDesigner by double-clicking the TrackingProfileDesigner.sln file in your sample installation directory WFApplicationsTrackingProfileDesignerCS.
2. Open the App.Config file and change the ConnectionString value to point to your server and tracking database. If you followed the book default, the database is named WFTrackingAndPersistence.
See Hour 1, “Understanding Windows Workflow Foundation,” for more details on setting up connection strings and using other databases such as SQL Express.
3. Make sure the TrackingProfileDesigner project is set as the startup project, and then start the project.
4. You should now see the Workflow Tracking Profile Designer form shown in Figure 13.3.
5. Click File, Open, Workflow from File, and select the TrackingWorkflows.dll file from the C:SamsWf24hrsHoursHour13TrackingTrackingSolutionTrackingWorkflowsinDebug directory to load the workflow to work with.
6. The Workflow Tracking Profile Designer form should now look like Figure 13.4.
At the workflow level, you can track the workflow life cycle events. These events include Created, Started, Loaded, Suspended, and Aborted. These are the same events the host uses to interact with the workflow. We will track the workflow Executing and Completed events at the workflow activity level. It may seem hard to keep straight, but workflows are activities. Therefore, the only events we are interested in capturing at the workflow level are Exception, Terminated, and Aborted. We want this to be a light profile and are therefore omitting Loaded, Unloaded, and other workflow life cycle events.
1. Click the workflow away from any activity on the Workflow Tracking Profile Designer form.
2. You should see two buttons under the File menu choice on the Workflow Tracking Profile Designer form. The first button should have a large green plus sign followed by Tracking Workflow. The second button should have a lightning mark and read Workflow Events. This means that you are at the workflow level. See Figure 13.5 for an example.
3. Click the Workflow Events drop-down and select the Exception event.
4. Click the Workflow Events drop-down and select the Terminated event.
5. Click the Workflow Events drop-down again and select the Aborted event. You are now tracking the workflow Exception, Terminated, and Aborted events.
6. Click the Tracking Profile Markup button under the Workflow Events button, and you should see that the Exception, Terminated, and Aborted events are being tracked, as shown in Figure 13.6.
Now you can be thankful for the TrackingProfileDesigner tool. Without it, you would have to manually produce the XML or create it through the CodeDom, which is used to generate code in the .NET Framework. Neither is that intuitive. The nice thing is that because the tool generates the XML, you can modify the XML as needed and learn it over time.
Tracking at the activity level in the TrackingProfileDesigner is done at the type level. For instance, clicking the ReceiveOrder activity and specifying tracking information applies to all HandleExternalEvent activities on the workflow. Specifying all activities or a single activity by name cannot be accomplished with the TrackingProfileDesigner (although both can be done by manually modifying the TrackingProfile or using the CodeDom). You can use the Match Derived Types to include activities derived from the specified activity, as discussed in the Annotate and Match Derived Types Options section.
Follow the next steps to configure the HandleExternalEvent activities to be tracked.
1. Click the Tracking Profile Designer button to return to graphical view.
2. You need to switch from workflow tracking to the activity tracking. To do this, click the ReceiveOrder activity at the top of the workflow.
3. You should see a button with a large green plus sign and the text Track HandleExternalEventActivity.
4. Click the Track HandleExternalEventActivity button to specify that you want to track this type of activity.
5. You should now see new menu choices to the right of the button you just clicked; also, the ReceiveOrder activity contains a thumbtack and an exclamation mark, as shown in Figure 13.7.
6. Click the Activity Events drop-down and select the Executing event.
7. Click the Activity Events drop-down again and select the Closed event. You are now tracking when HandleExternalEvent activities on the workflow are in the executing and closed states. The ReceiveOrder activity should no longer have an exclamation mark, because you have specified events. The thumbtack, however, should remain.
8. Click the Tracking Profile Markup button under the Track HandleExternalEventActivity button, and you should see the Executing and Closed events as shown in Figure 13.8.
Follow the next steps to configure the CallExternalMethod activities to be tracked.
1. Click the Tracking Profile Designer button to return to graphical view.
2. Click one of the CallExternalMethod (blue) activities near the bottom of the workflow.
3. You should now see a button with a large green plus sign and the text Track CallExternalMethodActivity.
4. Click the Track CallExternalMethodActivity button to specify that you want to track this type of activity.
5. You should now see new menu choices to the right of the button you just clicked, and both CallExternalMethod activities contain a thumbtack and an exclamation mark.
6. Click the Activity Events drop-down and select the Executing event.
7. Click the Activity Events drop-down again and select the Closed event. You are now tracking when CallExternalMethod activities on the workflow are in the executing and closed states.
8. Click the Tracking Profile Markup button under the Track HandleExternalEventActivity button. You should see a new set of Executing and Closed events for the CallExternalMethod activities.
You are now tracking the workflow level Exception, Terminated, and Aborted events. You are also tracking the Executing and Completed events for all HandleExternalEvent and CallExternalMethod activities.
If you wanted to track the workflow Executing and Completing events, you would click the workflow surface away from an activity. Then click the Track TrackingWorkflow button. This gives you the standard activity menu. You could then select the Executing and Completed events from the Activity Event menu. Using this method, you are accessing the workflow as an activity. You will access the workflow as an activity when you specify workflow properties a little later in the Access Properties in Tracking section.
You have created a valid TrackingProfile, and you must upload it to the tracking database before the SqlTrackingService will use it. The “Examine How Profile Works” section discusses how the SqlTrackingService locates TrackingProfiles.
Follow the next steps to upload the TrackingProfile you created to a database and run the workflow.
1. Click the File in the menu in the upper-left corner of the form. Then choose Save, Profile to SQL Tracking Database. Click OK.
2. If you are requested to enter a profile number, enter a number one larger in the third digit than the Current Profile Version Number, as shown in Figure 13.9.
3. Run the TrackingHostForm again, enter any order number, any order amount, and click Submit Order.
4. Wait for the workflow to finish and click the Get Tracking button.
5. You should now see the results shown in Figure 13.10. (No workflow events are shown because none of our tracked workflow events occurred).
You now have a custom TrackingProfile assigned to the TrackingSolution workflow. You can see how this works by examining the tables in the tracking database (WFTrackingAndPersistence).
Follow the next steps to look at the TrackingProfile tables and see how the runtime selects the proper TrackingProfile.
1. This workflow is now associated with the TrackingProfile you just created. To see how, open your tracking database and look at the TrackingProfile table. It contains a record for your workflow with a non-null TrackingProfileXml field, which contains your custom profile. It is the same markup you saw when clicking the Tracking Profile Markup button. See Figure 13.11.
2. Open the DefaultTrackingProfile table as well and examine its one record. Until you created the custom TrackingProfile, this profile had been applied to all tracked workflows. It still applies to all workflows other than the TrackingSolution one that is now bound to a custom TrackingProfile. If you wanted to change the default tracking behavior, you would change its entry here, and all current and future workflows that use the default profile would adhere to its new configuration.
If you add, remove, or change the name of activities on your workflow and rebuild without versioning the assembly, Tracking will no longer function. The reason is that the SqlTrackingService uses a couple of tables that contain one record for each activity on a workflow. The tables are constructed on initial build and are not updated during rebuilds. Reversioning your assembly will fix this problem. Alternatively, if you choose, you can also modify the Activity and ActivityInstance tables. This has worked for me but is not supported by Microsoft. If you choose this route, you can obtain the type used in tables from the Type table.
In this section, you learn to access workflow properties such as order number. You also learn to restrict which information is shown by adding conditions, such as only showing approved orders.
Although it’s useful to know event enumerations at the workflow and activity level, it’s frequently helpful to see order number, amount, and other properties as well. These properties can be used to augment the tracking information for operational needs. They can also be used for analytical purposes and by BAM tools.
Follow the next steps to track Status and other properties on the workflow.
1. Close the TrackingProfileDesigner form.
2. Run the Tracking Profile Designer project again and select File, Open, Workflow from File, and load the TrackingWorkflows again.
3. Click the Track TrackingWorkflow button. You are now at the workflow level and will utilize activity tracking against it. You could, for instance, specify that the Closed and Activity events be tracked at the workflow level here. We will track properties here instead.
4. Click the Extract Data drop-down, select TrackingWorkflow, and then select Status from the menu of members, as shown in Figure 13.12.
5. Click the Extract Data button, select TrackingWorkflow, and then select OrderAmount.
6. Click the Extract Data button again, select TrackingWorkflow, and select OrderNumber.
7. Notice the exclamation mark in the upper-right corner of the workflow. This is because you need to select an event in which to access the selected tracking data.
8. Select the Activity Events button and then select the Closed event. The exclamation marks should disappear. This means that you will access all the values you selected when the workflow is closed (or completed).
9. Click the Tracking Profile Markup button under the Track TrackingWorkflow button, and you should see that the Closed activity event is selected. In addition, you’ll see an Extracts element that contains all the extracted values, as shown in Figure 13.13:
Follow the next steps to filter the profile to show only approved orders. Filtering can be useful, if you want only very large or otherwise sensitive orders tracked and potentially sent to an alerting mechanism or analytical system. Filtering can also be useful to lessen the load. Later in this hour, you will learn to apply less than and larger conditions by manually modifying the profile, because you can only select = and != using the tool.
1. Click the Tracking Profile Designer button to switch back to design view.
2. Select Conditions, Add New Condition, click the Select member drop-down and select Status; leave the == default, enter Approved in the text box. See Figure 13.14 for an example.
3. Click Save.
4. You will now see a Conditions element in your TrackingProfile, as shown in Figure 13.15.
Click a CallExternalMethod activity on the workflow surface, and click the Track CallExternalMethod activity button. Note that you can also Extract Data (access the properties) of the activity type. Any property native to CallExternalMethod activity can be tracked here. If you created a custom activity (Hours 20–21) with the properties you are interested in, they could be tracked here. Click the Track CallExternalMethod activity button again to remove CallExternalMethod activities from the TrackingProfile.
Follow the next steps to create both approved and nonapproved orders to see how the emitted tracking data differs between the two.
1. Click the File option in the menu in the upper-left corner of the Tracking Profile Designer form. Then choose Save, Profile to SQL Tracking Database.
2. Click OK. If you are requested to enter a profile number, enter a number one larger in the third digit than the Current Profile Version number.
3. Run the TrackingSolution again. Enter any order number, enter 800 for the order amount, and click the Submit Order button.
4. After the order is rejected, click OK in the Workflow Completed dialog. Also, when you click Get Tracking, it generates an error.
5. This time you should see that only the activity Closed event and the three extracted properties were selected as shown in Figure 13.16.
6. Enter any new order number, enter 1500 for the order amount, and click the Submit Order button. (You can overwrite the current values.)
7. After the order is approved, click OK in the Workflow Completed dialog and click the Get Tracking button.
8. You should get the message No tracking events located and an Extracted Events header. The order is now rejected because it no longer conforms to the TrackingProfile.
There are times when the TrackingProfileDesigner alone is not sufficient. Maybe you want to specify conditional criteria that uses less than or larger than—for instance, orders > 2000. WF stores certain tracking information in UserTrackingRecords. This information consists of custom information that can be specified in the otherwise opaque Code activity, or in a custom activity. The Policy activity (which holds RuleSets) is also tracked at the user level. Hence the reason for the Policy activity on the workflow. The rules data will begin to show after we specify that UserTrackingRecords display. The TrackingProfileDesigner does not support adding user tracking criteria to tracking profiles. This section will demonstrate the following:
Inject tracking data into the workflow (UserTrackingRecords)
How to update custom TrackingProfiles to the tracking database
How to manually modify TrackingProfiles
Follow the next steps to track data in the Code activity’s handler.
1. Click the codeActivityUserData and set its Enabled property to true.
2. Double-click the codeActivityUserData activity on the TrackingWorkflow workflow and add the following to its handler:
CodeActivity ca = sender as CodeActivity;
this.TrackData(ca.Name + " is executing line 1.");
this.TrackData(ca.Name + " is executing line 2.");
Using this method, you may emit any information you want to Tracking. User tracking data is frequently specified inside Code activities to specify what function the otherwise opaque activity performs. It is also used by the Policy activity to emit rules data. Finally, it is frequently used in custom activities.
Follow the next steps to retrieve the profile you created and manually modify it to include user tracking events.
1. Close the TrackingProfileDesigner application if it is running.
2. Run the TrackingProfileDesigner application.
3. Select File, Open, and From SQL Tracking Database.
4. Click the TrackingWorkflows.TrackingWorkflow choice in the Choose a Workflow step of the Load Workflow and Profile from Store form.
5. Click the last version of the profile in Choose a Profile section of the form (as shown in Figure 13.17).
6. Click OK, and click OK in the dialog. Browse to the TrackingWorkflows.DLL and select it if you receive a message that the assembly cannot be found.
7. The workflow and TrackingProfile are now both loaded into the TrackingProfileDesigner.
8. Select File, Save, and Profile as File.
9. Select the C:SamsWf24hrsHoursHour13Tracking directory and name the file StartingDesignerProfile.xml.
Tracking profiles can be created using the CodeDom, or they can be hand rolled in XML, as you have seen when observing the output created by the Tracking Profile Designer tool. However, both of those methods are a bit complex. In this section, you will “mash up” a new profile based on the one you saved to a file and the default profile. You will insert the default profile’s UserTrackPoint (the element in the TrackingProfile that retrieves UserTrackingRecords) element into the profile you just saved to a file. This way you will enable your custom profile to track UserTrackPoints.
Follow the next steps to manually modify the TrackingProfile.
1. Open the StartingDesignerProfile.xml file you just created.
2. Open the DefaultTrackingProfile table in the tracking database, copy the contents of the TrackingProfileXml field, and paste the content into Notepad. If more than one record exists, copy the data from the last record.
3. Page down near the end of the Notepad document, and you should see a UserTrackPoint element.
4. Copy the UserTrackPoint element and its children, and then insert it into the StartingDesignerProfile.xml file you created between the ActivityTrackPoint and TrackPoints ending elements, as shown in Figure 13.18.
5. Save the updated file.
6. You now have to get this updated profile into the TrackingProfile table. You could manually update the proper record in the TrackingProfile table; however, we will use a stored procedure named UpdateProfile in the tracking database instead. This way you can automate the update and use it in cases when the profile is hand-coded or augmented via the CodeDom or XML.
Follow the next steps to upload the manually modified TrackingProfile to the database.
1. There is another form in this project named UpdateTrackingProfile. It permits a profile to be pasted into a text box, a profile number to be entered, and it then updates the ProfileTable (via the UpdateProfile stored procedure) with this information.
2. The code to call the UpdateProfile proc is provided by Microsoft in the SqlDataMaintenance WF SDK sample. See the InsertTrackingProfile method of the TrackingHostForm.cs file, if you are interested in its content. The only parts of this code that are specific to a given workflow are the typFullName.SqlValue and assemblyFullName.SqlValue properties that must be set to the tracked workflow type. I modified the code to accept these two values as parameters.
3. Set the TrackingProfileForm project (in the TrackingSolution) as the startup project and run it.
4. Look in the TrackingProfile table to see the last profile for our TrackingWorkflows project. It should be 1.0.2, but check yours to make sure.
5. Enter a profile number one greater than the current (1.0.3 in my case), paste the contents of the StartingDesignerProfile.xml file into the text box and click Submit. See Figure 13.19 for an example:
6. Close the TrackingProfileForm project.
There is code in the ShowTracking method in TrackingForm.cs that displays UserTrackingRecords. The code to access the UserTrackingRecords is identical in structure to the code that accesses the ActivityTrackingRecords and WorkflowTrackingRecords, with one exception. The rules data is stored as properties of the UserTrackingRecord.UserData type returned from the query. Therefore, you must test whether the type returned is RuleActionTrackingEvent. If so, you access the ConditionName and RuleName properties of the returned value. Look at the source if you want more details.
Follow the next steps to uncomment the UserTrackingRecords Query Code and then run the workflow
1. Open TrackingForm.cs and search for "#if false" to access the code that searches for UserTrackingRecords.
2. Remove the #if false and its corresponding #endif entries located about 20 lines below.
3. Run the WorkflowTracking solution. Enter any order number, enter 800 for the order amount, and click the Submit Order button.
4. Click the Get Tracking button. Scroll to the bottom of the tracking text box to see the UserEvents section, and you should see the results shown in Figure 13.20.
Your “mashed-up” TrackingProfile now shows the Code activity and rule UserTrackingRecords. This is in addition to the extracted properties and workflow activity Closed event previously specified in the TrackingProfile from last section.
You may have noticed the Annotate and Match Derived Types buttons on the TrackingProfileDesigner form menu. Annotate permits you to inject custom text into tracking at the specified location. Match Derived Types will, as the name says, match all derived types. If you look at the DefaultTrackingProfile, you will see that MatchDerivedTypes is true. The activity is set to the base WF activity. This how the DefaultTrackingProfile is able to include all activities on any workflow without knowing what they are in advance and enumerating them all.
Tracking is a very broad, intricate, and important component of workflow. We started in this hour reviewing WF’s Tracking architecture and where Tracking fits in the workflow world. Then you used the TrackingProfileDesigner to create various TrackingProfiles that both limited and enriched the data provided by the DefaultTrackingProfile. You created a mashup and updated the tracking database with the new profile. There is still much more to Tracking, such as programmatically creating profiles with the CodeDom and creating custom TrackingServices. You should look to the WF SDK for examples if you are interested. In the next hour, you will learn how Tracking and Dynamic Update combine to allow interesting new applications.