Integrating Data Across Environments

As soon as you start dealing with the cloud, you must establish a way to deal with the fact that you have data that potentially spans multiple environments. Of course, your data probably does span multiple environments today, but it’s under your control. How do you integrate all this data? Most companies very quickly find that they must contend with many different integration scenarios.

Three integration scenarios

The three most common cloud integration scenarios are

On-premises data center to cloud

Connectivity between (or among) clouds

Connectivity in clouds

We discuss each of these in this section.

On-premises data center to cloud

Connectivity from the data center to the cloud is one of the basic uses of cloud integration. The typical IT organization manages its enterprise resource planning (ERP) system within its data center and uses a SaaS environment to manage sales leads. Sales, order, invoice, and inventory data must be synchronized across these systems for the company to function properly. This can be a major cultural shift for an organization that’s used to having full control over its line-of-business applications.

There is little or no control over the architectural structure of the SaaS environment. Consequently, the IT organization needs to establish new processes to institute management between a data center application and a cloud-based application. IT management needs to separate the data elements within the line-of-business applications from unnecessary dependencies. For example, there may be a business process that controls a specific circumstance that interferes with your ability to easily connect between data sources on the cloud.

In addition, specific issues related to using cloud computing environments affect the style of integration. For example, although your company gains huge value from using a SaaS-based customer relationship management (CRM) system, governance requirements demand that customer data be stored behind your firewall. So, when a prospect becomes a customer, the company moves the data into the data center for additional security. This company now has a hybrid environment to manage. The company needs to automate data mobility across clouds in order to transfer and transform customer data to migrate between public and private clouds.

Connectivity among clouds

Companies may need to integrate among a private cloud and public clouds. One common example of this occurs when private cloud resources are insufficient to support peak demand. In this situation, select workloads are allowed to burst into a public cloud environment.

For instance, an entertainment organization is testing the introduction of a new game that supports on-demand group participation. The online gaming community has already shown a great deal of interest surrounding this new capability. The entertainment company wants to test how its web application scales from 20,000 to 1 million concurrent users before going live. They know they need more cycles and more power than they have available on-premises, so they expand their environment by leveraging public cloud resources, such as IBM’s Smart Business Development and Test Cloud or Amazon EC2.

Connectivity in clouds

A third type of integration occurs when you need to create bidirectional integration with multiple SaaS applications in order to support a business process. In this case, the connectivity capability itself is in the cloud. For example, a services organization uses sales automation to keep track of its prospects and a different SaaS application to manage commission and salary payments. Many sales situations exist where a cross-brand sales team collaborates in closing a large sales opportunity.

As a result, the sales commission must be split across different sales people. The data in the CRM system needs to be consistent with the data in the payment application, or the people who worked to close the deal won’t be paid accurately. Automating this process requires the synchronization of the data between the two SaaS applications.

Because both of these applications are in the cloud, the most efficient approach to synchronizing the data is to use a cloud-based integration capability. Public cloud offerings include connectivity in the cloud for this type of situation.

Choosing an integration method wisely

When considering a hybrid cloud, you need to understand that you lose control over how many things are done. Use an integration method to monitor these connections and make sure standards are met. For example, the SSL/TLS industry standards handshake will help to authenticate using X.509 certificates to ensure that users are legitimate. Part of this handshake process includes creating an encrypted tunnel; some of the security threats this process attempts to protect against are eavesdropping and man-in-the-middle attacks. Secure communication protocols exist that should be followed when communicating with endpoint applications and databases.

Options for cloud data integration

Various options are emerging for cloud data integration. The option that you choose may depend on the business problem you’re trying to solve and the kind of cloud deployment you’re dealing with:

Software solution: In this approach, vendors can provide a preconfigured integration pattern or template that jump-starts the effort of integration between applications. One of the benefits of working with a standardized template is that the same template can be reused for other integration projects. The template is typically designed to cover about 60 percent of the requirements for a particular integration. The packages usually also provide a way to visually map the data between source and target systems.

For example, in your on-premises ERP system, your customer ID might be called ID, and in your cloud application, it’s called CUST ID, the visual mapping interface makes this easy to specify that the two fields refer to the same entity. You just draw a line between the two to specify that they’re the same. Some packages also allow you to work with more complex mappings as well as provide a way to set up rules for data integration.

Cloud-based tool: This option is similar in many ways to traditional tools, such as connectors, that can be used to connect specific applications. In this case, end users can buy different components from a provider based on what they need to do. For example, you could buy a component for database connectivity or to transform data going into a database.

Cloud-based solutions: In this case, data integration is offered as a service or set of services. For example, a vendor might offer a data replication service to copy data from one source to another and then automatically update it. Or it might offer data quality and assessment services, or services to load data from various format types (such as flat files or databases) into target applications. Or it might offer a packaged web application server and database.

You can find a number of vendors offering solutions in this space, including companies such as Dell Boomi, IBM, Informatica, Pervasive Software, Liaison Technologies, and Talend. Some offer packaged solutions, some offer cloud solutions, and some offer both. The solution you choose will depend on the problem you’re trying to solve.

No matter what approach you use, an overriding issue is going to be to make sure that your data maintains its integrity by being complete, accurate, and up to date. You will still have to make sure that you have a master version of your data in place that serves as what is often called a “single version of the truth.” In other words, an agreed upon golden master.