Chapter 14. BizTalk Server Integration

When considering integration with other systems, remember that Microsoft has long recommended their own stack: BizTalk Server.

BizTalk Server is a middleware application designed to facilitate the communication with systems that generally don’t have integration options/adapters out of the box.

What this means is that you can use BizTalk Server to communicate with another system, with BizTalk acting as the conduit between the two applications.

In simpler terms, if we consider that the Atlantic Ocean is one system (CRM) and the Pacific Ocean is another system (ERP) and the Panama Canal is the mechanism for interacting with each (via a series of rules [locks, measured capacity, days/times of operation, and so on]), then the Panama Canal is closely analogous to BizTalk Server.

In this chapter, we review how BizTalk Server can be used to perform an integration with CRM and an ERP system. It should be stressed that the examples in this chapter are specific for an integration with Microsoft Dynamics GP. However, they can be applied to other systems following similar steps.

In summary, BizTalk can be used for such things as the following:

• Messaging: Adapters can be used to handle proprietary protocols and can support business data processes/conversions.

• Radio frequency identification (RFID): Supporting a variety of hardware devices, it provides device abstraction, management, and simulation, which can be used to integrate into existing business processes or backend systems.

• Business rules: Support for authoritative business rules via the Rules Framework using .NET components or defined XML documents or database tables.

• Business activity monitoring (BAM): The BAM infrastructure monitors events inside the BizTalk application and publishes those events for metrics and reporting. The reporting can be achieved via the use of KPIs (key performance indicators), automatic alerts, or by Microsoft Office BI tools. BAM includes a set of tools that allows for the management, aggregation, and modification (via APIs) by customer code development.

• Management/operations tools: These include support for scaling, fault tolerance, load balancing, and the ability to configure and monitor the databases, hosts, and services that are within the scope of the implementation.

Basically, BizTalk can be used to allow Microsoft Dynamics CRM to communicate back and forth with Microsoft Dynamics AX, Microsoft Dynamics GP, Microsoft Dynamics NAV, and other databases such as Oracle, Siebel, PeopleSoft, and SAP.

In this chapter, we look at two such applications: Microsoft Dynamics CRM and Microsoft GP (Great Plains).

BizTalk Versions, Licensing, and Requirements

Microsoft BizTalk server is available in four different versions:

• Enterprise

• Standard

• Branch

• Developer

All versions are licensed on a per-processor basis, with the exception of the Developer version, which is licensed on a per-user basis.

Typically, organizations with high volume requirements use the Enterprise version, whereas organizations with relatively moderate volume and deployment scenarios use the Standard version. The Branch version is a unique version designed for hub-and-spoke deployment (including RFID), and the Developer edition is available for development and testing purposes.

Originally introduced for use with Windows Server 2000, BizTalk Server has had several iterations, leveraging .NET code since .NET 1.0, and its latest version, BizTalk Server 2006 R2, which leverages Windows Communication Foundation (WCF), and Visual Studio 2005 for designing, mapping, and manipulation.

The following is necessary for a deployment of BizTalk Server 2006 R2, but additional components might be necessary depending on the defined requirements:

• Windows Server 2003/Vista/XP

• Internet Information Services (IIS) 6.0

• Microsoft Office Excel 2003 with Service Pack 2 or Microsoft Office Excel 2007

• Visual Studio 2005

• SQL Server 2005 with Service Pack 2

• SQL Server 2005 Analysis Services with Service Pack 2

• SQL Server 2005 Notification Services with Service Pack 2

Architecture

BizTalk is based on a “pub/sub” (publish/subscribe) model. Events in one system are defined, and the data from the event is made available by publication. Other applications are considered subscribers, and if the published data is something they’re interested in, they can consume it.

The idea behind this is to provide a level of abstraction whereby the two systems don’t have to be connected by any direct mechanism, thereby providing a level of fault tolerance and ease of change/modification.

Figure 14.1 describes this scenario.

Figure 14.1 Direct application connections versus pub/sub connections.

The model outlined is not dependent on systems running on your network and may be extended to external systems, too.

BizTalk services have recently been extended to the Internet because there is no reason why the pub/sub can’t be extended to the cloud. Although still in beta, we have included information about using this cloud service in Chapter 15, “Azure Web Services Integration.”

BizTalk Server uses connectors and adapters as its means of establishing links to the underlying systems.

Adapters

An adapter is specific to a single product, such as Microsoft Dynamics CRM 4.0. With the adapter, you can integrate Microsoft Dynamics CRM with other applications easily using the BizTalk Server mapping capabilities.

It is important to understand that adapters can be used to push data into the target system, but they cannot be used to pull data from the system. The reason for this is that the events that cause data to be sent from the target system vary depending on the integrated solution, and without a separate adapter, there is no way to know whether the event data is valid/necessary.

An example of this is when an order is created in Microsoft Dynamics CRM. This can be done either in Microsoft Dynamics CRM or it can be done in a corresponding enterprise resource planning (ERP) system. However, consider the ERP system: When the order creation occurs in Microsoft Dynamics CRM, the effects of re-creating the order in the ERP system (without an adapter/connector) can be far-reaching.

Connectors

A connector is a device that has been predefined and is specific for two (or more) applications.

Connectors are preestablished “templates” that use adapters for the underlying applications and include a series of business rules that allow for the quick deployment of an integration solution with minimal development.

They provide ongoing synchronization of data between the applications that the connector is defined for.

A common connector is the Microsoft Dynamics CRM Connector that was released for Microsoft CRM 3.0. This connector uses the Microsoft CRM 3.0 adapter and the adapter for Microsoft Dynamics GP.

Message Queuing

Message queuing is a huge component of the reliability, redundancy, and auditing of a system. Without it, data messages that need to be consumed by a system can fail for a variety of reasons (such as network/database failure, permissions, and so forth) and then never be read and written.

Microsoft Message Queuing is included in the latest version of BizTalk Server.

Integration with Microsoft Dynamics CRM and GP 10 Example

Because of the lack of a connector for Microsoft Dynamics CRM and GP 10, integration using BizTalk Server can be performed in a number of different ways.

The typical implementation involves the use of the Microsoft Dynamics CRM adapter to push data into CRM, and the creation of custom business rules to push data out when necessary. These custom business rules can be defined as either workflow or plug-ins that monitor an entity or attribute in Microsoft Dynamics CRM. As necessary, the data is pushed from Microsoft Dynamics CRM and made available for consumption by other applications through BizTalk in XML format.

Figure 14.2 describes this scenario.

Figure 14.2 Recommended integration with Microsoft Dynamics CRM to GP 10.

For less complex integrations, or where it may be acceptable to only update GP (and not push data back to CRM from GP), Figure 14.3 illustrates a one-way integration option. Of course, this kind of integration is less sophisticated, and it could be argued that if you do not need to integrate other data or perform complex data manipulation you could just bypass BizTalk Server and create the integration using .NET components.

Figure 14.3 Recommended integration with Microsoft Dynamics CRM to GP 10, one way.

The concepts of Microsoft Dynamics CRM plug-ins and web services are adequately documented in this book, but we need to explain some of the components to GP:

• eConnect allows applications to programmatically interact with Microsoft Dynamics GP using .NET, message queuing (MSMQ), and the Microsoft BizTalk Application Integration Component (AIC).

• eConnect simplifies development effort, because it has a number of integration points for Microsoft Dynamics GP.

• As part of BizTalk Server, Orchestration Designer is a Microsoft Visio-based design tool that enables you to create business process drawings that can be compiled and run. (See, for example, Figure 14.4.)

Figure 14.4 Sample orchestration.

To install the Microsoft BizTalk Adapter for Microsoft Dynamics CRM 4.0, follow the steps as outlined in the Microsoft BizTalk Adapter for Microsoft Dynamics CRM 4.0 – Installation and Usage Guide.doc, which you can download from http://www.microsoft.com/downloads/details.aspx?FamilyID=4628fca6-388d-45bc-a154-453b920dbcb8&displaylang=en.

Once installed, Microsoft Dynamics CRM 4.0 appears in the Administration Console as an available adapter, as shown in Figure 14.5.

Figure 14.5 Adapters in the BizTalk console.

The Connector for Microsoft Dynamics CRM 3.0 enforced a number of business rules that are recommended as best practices when considering deploying an integrated solution consisting of quotes, orders, and invoices. These business rules include the following:

• In general, invoices should not be created in Microsoft Dynamics CRM, but rather created in the ERP system and then pushed as read-only entities into Microsoft Dynamics CRM. Therefore, the ability to create invoices in Microsoft Dynamics CRM should be removed.

• Orders can be optionally created, updated, or deleted in either system depending on your organizational business rules. Generally, it is acceptable to allow full functionality in either system, with updated data moving from one system to the other.

In addition, it is common to enforce business rules associated with inventory when orders are submitted, but this is discretionary and is dependent on the desires of the organization and level of integration required for inventory management.

• In an integrated environment, pricing options for quotes, orders, and invoices should be given careful thought, because updating these entities from the back office ERP is highly recommended. (Tax calculations alone warrant this.)

In the following example, we perform a simple integration between GP 10 and Microsoft Dynamics CRM 4.0 whereby an order can be created in Microsoft Dynamics CRM and pushed to GP.

For this example, we are assuming the following environment:

• BizTalk Server 2006

• GP 10

• CRM Adapter

• eConnect 10 (with latest service packs and eConnect SDK)

• Visual Studio 2005

In addition, we need to identify the CRM schema and GP schema that will be mapped. In our scenario, we’re going to map CRM order information and GP order information.

Schema Mappings Between CRM and GP

1. Download and install the CRM Adapter in BizTalk Server.

2. Open Visual Studio and select a new project, as shown in Figure 14.6.

Figure 14.6 Selecting a new project in Visual Studio.

3. Select Empty BizTalk Server Project from the available templates under the Project Types for BizTalk Projects, as shown in Figure 14.7.

Figure 14.7 Select Empty BizTalk Server Project Solution.

4. You now have an empty solution file. Right-click the solution and select Add Generated Items, as shown in Figure 14.8.

Figure 14.8 Add generated items to the solution.

5. Select Add Adapter Metadata from the available options, as shown in Figure 14.9. It is at this step that the connector is utilized.

Figure 14.9 Add adapter metadata to the project.

6. Select the Microsoft Dynamics CRM 4.0 adapter, as shown in Figure 14.10.

Figure 14.10 Add the Microsoft Dynamics CRM 4.0 adapter.

7. Enter the credential information for your Microsoft Dynamics CRM deployment. The CRM server URL is in the format of http://<<server name>>:<<port>>). Figure 14.11 illustrates the required credential information.

Figure 14.11 Microsoft Dynamics CRM 4.0 credential information.

8. Select the actions or the entities that you want to publish in the schema (as shown in Figure 14.12).

Figure 14.12 Select Microsoft Dynamics CRM actions or entities to publish.

9. In our example, we want to fetch the Microsoft Dynamics CRM schema information for the Create action (see Figure 14.13).

Figure 14.13 Select the Microsoft Dynamics CRM actions or entities Create action.

10. We can now navigate the CRM schema file. Figure 14.14 shows the Order schema file.

Figure 14.14 Microsoft Dynamics CRM Order schema file.

11. Now we need to find the GP schema. Assuming that eConnect 10 and the eConnect SDK are installed on the same machine, the GP schema can usually be found in the following location: C:Program FilesCommon FilesMicrosoft SharedeConnect 10XML Sample DocumentsIncoming XSD Individual Schemas.

To add the schemas to Visual Studio, right-click the project, select Add, Existing Item, and then navigate to the location specified, selecting SOPTransaction.xsd.

Figure 14.15 shows the GP schema added to Visual Studio.

Figure 14.15 Microsoft Dynamics GP schema file.

12. We now have both the Microsoft Dynamics CRM and the Microsoft Dynamics GP schemas. The next step is to map both of the schemas, which may require some functoids depending on the integration.

To create the map file, right-click the project and select Add, New Item, Map. Name it appropriately for your project, and then select Add. Once added, you can select the source schema on the left side, and the right side will be the destination schema.

Selecting either Open Source Schema or Open Destination Schema opens the schemas we have available. After we select both schemas, they are ready for mapping. If it is a direct mapping between the two fields, you can just drag and link between the two fields. In cases where we want to apply logic to the mapping, we would reference a functoid.

Figure 14.16 shows the two schemas mapped, with three functoids.

Figure 14.16 Microsoft Dynamics CRM and GP schemas mapped.

13. Deploy the application.

14. After the application has been deployed, it is available within the BizTalk Administration Console, as shown in Figure 14.17.

Figure 14.17 Deployed solution in the BizTalk Administration Console.

Web Service Creation and SOAP Adapter Configuration

The next step is to create a web service and configure it using a SOAP adapter. This is necessary to receive data from CRM and pass it into BizTalk Server. Because the schemas are already identified for CRM, we need to expose them as web services so that we can pass the values using the plug-in on the CRM side (as discussed later in this chapter).

To create the web service, follow these steps:

1. Deploy the application that we created in the previous steps.

2. Open the same solution and go to Tools, and select BizTalk Web Services Publishing Wizard (see Figure 14.18).

Figure 14.18 Selecting the BizTalk Web Services Publishing Wizard in BizTalk.

The BizTalk Web Services Publishing Wizard starts and guides you through the process of deploying a web service.

Select Publish Schemas as Web Services when prompted for the method of creating the web service, and when asked to describe the web service, right-click the WebMethod and select Select Schema Type. Browse for the DLL files of the previously deployed project, specifying the target namespace and location of the project.

3. After the web service has been created, it needs to be configured using a SOAP adapter. Figure 14.19 shows the General tab information for the adapter that is configured using the BizTalk Server Administration Console and selecting the Receive Ports.

Figure 14.19 SOAP adapter General configurations.

4. Set the receive location, as shown in Figures 14.20 and 14.21.

Figure 14.20 SOAP adapter receive location configurations.

Figure 14.21 SOAP adapter receive location General configurations.

5. Click Configure to specify the web service virtual directory that you created in the first step (see Figure 14.22).

Figure 14.22 Specify the virtual directory for the web service .asmx file.

6. The mapping file we created earlier will automatically populate the column for the Inbound Maps tab (see Figure 14.23).

Figure 14.23 SOAP adapter inbound maps.

eConnect Configuration

Our next step is to configure the send adapter using eConnect so that we can integrate values into GP from the source. Figure 14.24 shows the configuration for eConnect.

Figure 14.24 eConnect configuration.

1. Right-click the eConnect adapter and select Properties.

2. Select Properties (see Figure 14.25) and set the Transport Properties Connection string (see Figure 14.26).

Figure 14.25 eConnect properties.

Figure 14.26 eConnect connection string.

Because eConnect can be installed anywhere, we can select the database we want to use that is on our network. In our example, we’re connected locally (the same computer is running GP), but we can point it to any connected computer using a standard connection string.

Configuring the Send Adapter

1. Figure 14.27 shows the send adapter configuration options.

Figure 14.27 Send adapter configuration options.

2. The transport connection string can be configured after you click Configure (see Figure 14.28).

Figure 14.28 Configure the connection string.

3. Select Outbound Maps (see Figure 14.29).

Figure 14.29 Configure the outbound maps.

4. Set the filter options. In this example, we set the BTS.ReceivePortName = the web service we created previously. Figure 14.30 illustrates how to set this.

Figure 14.30 Filter options.

CRM Plug-In Creation

The CRM plug-in will call the web service and pass the expected values during our selected event (in this case, the Create event).

Use the following code to create the plug-in:

Results

When we create a new order in Microsoft Dynamics CRM, either directly as an order or when we convert a quote, the information is reflected in GP.

Figure 14.31 shows a new order in Microsoft Dynamics CRM, and Figure 14.32 shows the corresponding information in GP.

Figure 14.31 New order in Microsoft Dynamics CRM.

Figure 14.32 Microsoft Dynamics GP.

Notice that the customer name and the amount flow from Microsoft Dynamics CRM to GP. In this example, we’ve kept the unique order number created in each application, but you can integrate that if necessary based on your requirements.

When developing a solution similar to this, you need to consider a number of things. Because the options are extensive and our goal is to provide an overview template on how to create an integration using BizTalk, we don’t list them all here. However, consideration should be given to the following:

• Validation of account credit limits/holds

• Account creation permissions on the ERP side

• Transport of data from the ERP to the CRM

We highly recommend a serious study of your particular business rules before attempting this kind of integration.

Summary

This chapter covered the options associated with using BizTalk Server as an integration application between CRM and GP.

BizTalk Server is certainly not limited to these two applications, and many organizations have successfully used BizTalk Server to integrate more than two disparate applications successfully.

It is important to note that BizTalk Server is an application with considerable complexity and options and, therefore, is recommended only for experienced integrators.

Of the concepts that you should be familiar with after reading this chapter, the main one is creating the necessary components for this type of integration, consisting of CRM plug-in development and eConnect configuration. Although BizTalk Server configuration options are covered in this chapter, it is an enterprise product that enables you to perform a number of additional functions. Therefore, further independent review is recommended should you require these types of configurations.

Finally, although BizTalk Server has been the integration recommendation by Microsoft for its Dynamics products, it is not the only solution available. We have included information about how to perform other integrations in Chapter 3, “Extending Microsoft Dynamics CRM.” In addition, be sure to check our chapters on Scribe before selecting a solution that is right for your organization.