Chapter 19: Managing Integrations

For many NetSuite clients, the ERP system is the central hub for a collection of applications they use to run their business. Data needs to move securely and efficiently between each application for the business to function. A consultant needs to know how to talk to clients about integrations, as well as how to see them through development and testing.

In this chapter, we will cover the following topics:

• Talking to clients about integrations
• Collecting information about and documenting integrations
• Managing data coming into NetSuite
• Creating custom exports from NetSuite

With these skills, you will be able to talk intelligently to clients about their integration needs and help them use the right solutions.

Just as in Chapter 18, Managing Gaps and Creating Custom Automations, when we're talking about integrations with clients, we need to understand the main features of the SuiteCloud platform, and any SuiteTalk and/or SuiteScript training will help the client stay with you as we discuss development.

Talking to clients about integrations

We need integration whenever we need to bring data into NetSuite or send data out to another system. Integrations are very common in the NetSuite world, so every consultant helping a client through an implementation should be at least a little familiar with the topic. Gathering a client's requirements around any integration can be a challenge since the client doesn't usually have technical experience with integrations (most of my clients had someone else build the integrations they're using in their legacy ERP system) and we don't have experience with running their business. But so long as someone from the business can speak about the types of data they need to see coming into and going out of the system, and you bring your technical understanding to the conversation, it usually works out just fine. We, as consultants, need to always remember to speak without letting technical terms get in the way of the client's understanding, and we also need to be excellent listeners to pick up on every nuance of the requirements we're gathering (as always).

The following diagram shows a typical integration flow:

Figure 19.1 – A typical integration flow with data coming into and going out of NetSuite

Whether we're getting expense data from an external employee portal or integrating NetSuite's financial data into another ERP accounting system (as is often the case when a relatively small line of business in a very large enterprise is using NetSuite and sending their journals into a centralized server), I like to stick to terms everyone can understand. It's important to always be clear about what types of data we're focusing on, and in which direction the data is being moved. It's not uncommon for a new client to still refer to transaction names they use in their legacy ERP; when that happens, we can help them translate those terms into NetSuite record names. I avoid terms such as upload and download since they're relative, and just saying them doesn't tell you which direction the data is flowing in. I'll talk about imports into NetSuite or exports from NetSuite instead.

At the project management level, we should always call out any new integration requirements we learn of as they come up. If the integration requires development and that might cause a delay in the rest of the project's schedule, the project managers and the client need to know that so that they can plan the custom development work accordingly. Once the entire team (your business and the client) are clear on what's involved, then new timelines can be put together. This can cause a change, for instance, in the go-live date, if the integration is critical and the work to develop and test it takes longer than the rest of the implementation.

Every time we integrate with another system, we're moving data from one database to another, so we should always expect that a transformation of some kind must be performed on the data between the two systems. For instance, when we're bringing data from a Warehouse Management System (WMS) into NetSuite, the WMS will have some data points that will be useful in the ERP system, but we won't have fields for them. An Item Fulfillment transaction might include details about the packages that were packed in the shipment. In these cases, we can find native fields to store the data in, but we might need to create new custom fields instead. And the same situation can occur when data is headed from NetSuite to another application. The other application might need to be able to store some of our data in their system in whatever their equivalent of a custom field is.

Important Tip

In some cases, we can talk to a client about using what we call a middleware service provider to meet their integration needs. These are companies that have built software systems that can take data from many sources and send them to any other source. They support all the standard data formats, plus most have pre-built connectors for the most popular applications, such as NetSuite, SalesForce.com, Workday, and so on. If a client is already subscribed to one of these services, it's usually fairly easy to add flows to them to support new use cases. But otherwise, they can be expensive to sign up with and time-consuming to get started with. We should always keep this option in mind, but also be mindful of the problems that it can add to a project.

I talk to my clients about endpoints (a connection between two computer systems, where NetSuite is one of those systems) when I'm trying to understand their integration needs. This gives us a simple framework where we can discuss all of the data that needs to come into NetSuite or be sent out from it. Once we understand the full list of endpoints, we can start to map out how each will be accomplished. I'll explain a few options for doing that next.

Collecting information about and documenting integrations

Some NetSuite clients just need to integrate with one other application, for some relatively small part of their business. We always start to help these clients by searching the SuiteApp Marketplace to find out whether the other business already has a package for accomplishing this. With things such as taxes and warehouse integrations, as well as invoicing and payment handling, there are usually multiple options. But when we find that no app exists for their use case or the client needs to define and create integrations with multiple endpoints, we must take a couple of additional steps to make sure the work is planned and executed properly.

First, we must build a document listing all of the endpoints and their most important details. I usually do this in a spreadsheet since this allows me to easily build a table of information and I can extend the listing any time I need. We always want to be able to refer to each endpoint by a common, easy-to-remember name such as Subscription Charges in NetSuite or Item Updates to the Data Warehouse, so that's the first thing we include here. Doing this avoids a common issue I see at the beginning of these projects where the client will say something like, "All of NetSuite's data has to be sent to another system." That's not specific enough to work on, so breaking it down and naming each record involved is what's needed in these situations before we can start to make any headway. Having this list will also make it easier for everyone involved to stay focused later as we build out whatever is needed to deliver each integration from NetSuite's perspective.

For each endpoint, we like to list a few things in a summary fashion, based on what the client tells us is required:

• Name: Something that makes it clear what's transferred and usually, its destination system.
• Source system: This is where is the data is created initially.
• Source data: Which record type in NetSuite or the other system this is about.
• Destination system: Which system needs to receive the source data.
• Destination data: Which record type will store the received data.
• Authentication method: How will the calling process prove who they are to the other system? NetSuite supports a few industry-standard options such as OAUTH and token-based authentication.
• Frequency: Real-time, hourly, daily, and so on.
• Estimated transaction volume: How many records of the stated type are expected to be integrated each hour, day, and so on.
• Remaining: Whatever else is needed on a case-by-case basis.

Here's what that spreadsheet might look like, though you should apply your creativity to this task and make something that will work for you:

Figure 19.2 – An example of a spreadsheet for integration tracking

For instance, if we know some endpoints will use SuiteApps and that others will need custom solutions, we should include that detail in this document too. Getting a document like this started and fully populated can be a challenge on larger implementations but it's worth doing. When we kick off this effort, I explain to the client that the document will be modified over time and won't be finished until they're live on NetSuite.

Once you have started that list, we like to create an official integration long-term planning guide. This is typically done in the form of a Word document, with sections and details we think are relative to all of the client's integration needs – whether they will be processed manually, whether we're creating automation, or whether the client will use an off-the-shelf solution. This gives them a single place that the business' users can refer to later to answer questions such as, Where do our Sales Orders come from? or How often should our Customer records be synced with SalesForce.com?. We also include other details in these documents, including topics such as integration security issues and how the integration work will tie in with the implementation's data migration plans or any related customizations being developed. It's also a good idea to incorporate any special data handling routines into this document. We try to nail down things such as date or phone number formatting in one place and explain how lists such as Items and Accounting Periods within any of the integrations will be handled. For instance, if a banking integration requires NetSuite to send payment files to a secure remote server, the high-level details of that connection can be described in this document. Lastly, for each integration endpoint, it's a good idea to make it clear which system is thought of as the system of record; new data will be created in that system, whereas the other system just inherits those records once they're created.

By having the integration listing and the overall architecture documented as much as possible early on, we can launch into the work of defining the new scope in a contract with the client (if that's needed) and creating whatever deliverables are called for.

In the next two sections, we'll break this work down into two separate streams – imports and exports from the system.

Managing data coming into NetSuite

When we talk about integrating data into NetSuite, there are several things to decide on as we initially plan the work:

• Will NetSuite pull the data from another system, or will the other system push it into NetSuite?
• What are the technical options for accomplishing this integration?
• Does the integration need to be automated or can a user perform a CSV import to bring the data into NetSuite instead?
• What actions are needed on the NetSuite side? (In other words, will we be creating, modifying, or deleting records?)
• Who will create the integration that performs these actions?

In some cases, these things are easy to map out – the other system has a connector already and we just install and configure it in NetSuite and we're done. In other cases, though, this can be tricky. When we need to build some custom integration in NetSuite, the first topic that comes up is usually one regarding scope from our professional services contract. It's common for integrations like this for the scope to be called out in our initial Statement of Work, for instance, but many times it isn't. We might need to negotiate with the client to add services work (at some new cost) to the contract before building something new. But this depends on your delivery business model – how the initial contract was set up. If you're working on an open-ended time and materials contract already, you can get approval for the work and then go for it.

Getting two systems to talk to each other can occasionally be time-consuming and difficult. As I mentioned previously, when we integrate another system's data into NetSuite, the two separate databases might not have a lot in common, and that usually means we need to create a few or a lot of custom records or fields that we can store that data in. In some cases, we might want to create a custom record type for this purpose, allowing us to define a place where all of the data will reside within NetSuite, without adding a lot of unusual fields to a native record type. Then, we can leave the data there and include it in reports or whatever we need, or we can incorporate some of those fields into native records as needed. For instance, if a business uses a third-party system to track most of their day-to-day services operations work, they might want to import that data into a custom record called something like Daily Services. Then, that data can be synced to invoice lines or just reported on to make it useful within the ERP system.

As I mentioned earlier in this section, we always want to find out how frequently the data will need to be brought into NetSuite. When the frequency is low, such as once per month, we will suggest that a user performs a CSV import rather than create automation. This saves on development and users can usually work with this approach with just a single training session. When the frequency warrants it, though, we have a few other options, where some type of automation running in NetSuite needs to import the data for the users. We'll look at this in the next few sections.

Option 1 – Common SuiteScripts

The most common option in these situations is to create a SuiteScript of the User Event or Map/Reduce types. Depending on the need, we can create a script that runs in real time (if the triggering event happens in NetSuite) or on a predefined schedule. Either way, these scripts can be made to reach out to an external system, such as a web service or a secure file transfer server, gather the intended data, and integrate it into NetSuite wherever it belongs. This is technically possible but requires a good deal of work.

Option 2 – SuiteTalk SOAP

When the owners of the external system are OK with pushing their data into NetSuite – that is, they can develop the integration themselves – then we talk about the two SuiteTalk web services options that are currently supported. SuiteTalk SOAP is the more mature option, having been available for many years. It's based on an older XML-formatted technology but it's fully featured and is used by many, many integrations already. The SOAP standard is used by many applications (not just NetSuite), so developers should find it easy enough to begin bringing data into NetSuite using easily accessed and configured development tools. NetSuite publishes a sort of data dictionary called the Web Services Description for SuiteTalk SOAP and that gets updated twice a year, along with the rest of the product.

Option 3 – SuiteTalk REST

SuiteTalk REST is a newer option and as of 2021, it's only been partially released. Some records are generally available (that is, GA, such as Customers and Sales Orders), which means NetSuite fully supports their use with the REST API; the API's details won't change without an announcement in advance. But some record types are still in what we call beta status. Those records are still subject to changes as the API has not been finalized for them, and that makes it hard to build production-ready integrations upon them. I can't make any firm predictions on this, but I expect that most of the commonly used record types will be moved to the GA status in the short term, so we're expecting more and more integrations to be built on the REST API over time. REST is a standard that most web developers will be familiar with since it's become the most popular communication method among modern web applications and services.

With either the SOAP or REST options, though, someone outside NetSuite has to develop and test an integration from an external server. This is usually a challenge since those people probably won't have NetSuite experience or even access an account, other than the clients'. You can help them with this by providing them with answers to questions about NetSuite and the APIs, to make sure they complete their tasks on time.

Option 4 – SuiteScript RESTlet

When the other party is OK with developing the integration, but the data they want to send cannot come into NetSuite through SuiteTalk (which is rare), then we can create another type of script known as a RESTlet. I described them in Chapter 18, Managing Gaps and Creating Custom Automations. These scripts are like mini web services running in NetSuite all the time. They listen, as we say, for another system or service to call them, with either data in the request or a request for us to send them NetSuite data, and respond accordingly. In the case of data coming into NetSuite, the other party in the endpoint might be an external e-commerce web store that wants to send data to NetSuite, and then we might create a RESTlet that receives those requests and creates Sales Orders in NetSuite using the data.

Next, let's go over the options a client has when they need to send data from NetSuite to another system.

Creating custom exports from NetSuite

When a client needs to export data from one system to any other system, we usually push for the simplest options we can find and only resort to building something custom when those tools just don't fit the bill. As described in Chapter 17, Analytics, Reports, and Data Exports, our first choice for exports should always be SuiteAnalytics Connect. As a reminder, this is the option that allows a client to connect to NetSuite with one of three industry-standard methods (ODBC, JDBC, or ADO.NET). They then execute queries to collect any sets of data they need from the system. The client's technical people can create automation that runs outside NetSuite in a variety of databases, data warehouses, or applications to pull data out on whatever schedule they need. The NetSuite Help page explains how to use the Connect feature and it's very popular for this use case. Most developers already know how to use Connect's SQL queries, so this is usually something they can do themselves, once we help them get started and get their tools connected to the account.

When Connect doesn't work (not very often), we can fall back to something custom running in NetSuite for exports. We start by suggesting that the users run a search and select one of the export options from the results screen (CSV, Excel, and PDF options are always available), or sometimes we will set up a saved search that runs on a schedule and sends its results to an email address. But when these options are rejected, we have the option of creating a script.

For instance, say a customer needs to have certain details captured from every new (and approved) Purchase Order, and those details must be sent to some other system for additional processing. We could do that with a User Event script running to save the record in NetSuite if we need to export data in near-real time. This is not ideal since we would need to include additional logic in the script to handle any errors which occur, which will depend largely on how critical this operation is to the business and since this kind of process can slow the system from the user's perspective.

Something more common for exports would be some sort of batch process. A Scheduled or Map/Reduce script can be created to gather a set of records that have been created in the system at the end of every day, for example, and send those records to another system, in the form of a file or a stream of data sent to a web service.

For all of the export options mentioned in this section, keep in mind that scripts are tricky to develop and get running correctly, and they add cost and delays to the schedule, so we want to reserve this option as the last one we'll consider for most client implementations. However, they can be really powerful and if the frequency and data volume warrant the work, they can be well worth the effort.

Summary

Successfully running a business in the 21st century almost always requires integrations between systems. We like to say that NetSuite eliminates the need for a lot of integrations that used to be required, but that doesn't mean NetSuite can operate in total isolation. We want to help our clients choose the best option they can find for every business process, though sometimes that means someone will need to make two or more systems share data. In this chapter, we learned how to assist the client in the planning stages for their integrations by helping them understand each integration endpoint their business requires. Once we've done that, we can help them choose the right technologies to meet each need.

Data can be imported into NetSuite by customizations running in the account via middleware solutions, or by third parties developing custom solutions based on NetSuite's APIs. We can export data from the system via searches, CSV exports, SuiteAnalytics Connect, middleware solutions, or scripts, depending on all the variables in each situation. No matter which option we choose, we must plan for the complexities involved in any custom development work to keep our implementation projects on track and schedule.

In the next chapter, Managing Data Migrations, we'll talk about bringing the client's legacy data into the system and how this is not something we wait until the last minute to address. Migrations need to be planned from the beginning of the project to make sure they're just as successful as the rest of the project.

Self-assessment

Here are a few more practice questions for you to consider. As in the previous chapters, think about how you would respond in each situation:

1. Your client is about 1 month away from going live when they let you know that they need help getting data from their expense tracking system into NetSuite. In what order should you research the options available for making this integration happen?
2. You've started to build a list of the integration endpoints your very large client needs with NetSuite in a spreadsheet. They mention that one particular integration will need to transfer something like 1 million records from their legacy accounting system into NetSuite. What sorts of questions should you ask at this stage to make sure you understand and address this requirement properly?
3. Just as you begin working with a new client, you learn that they will only have five accountants working in the system; there will be no other NetSuite users. They intend to integrate all of the data those people will work with from another system; that is, the application everyone else in the company works in every day. How should you adjust your consulting to support this situation?
4. A client is thinking about how to export their very high daily sales transaction data to another system. They want to know whether the SuiteAnalytics Connect feature will be able to perform their transfers as quickly as they need. They say that they might need to send around 10,000 orders out of the system every day and each order might have 100 lines or more. They need to know that this data can be synchronized from NetSuite once every 15 minutes. How can you help them set up enough test data in NetSuite to run a few timed tests so that they can answer this question?
5. You've been planning to use a middleware partner to transfer journal entries from NetSuite to another system with your client, at which point they tell you that their contracted developer is not going to be able to complete the work. They ask how you can help them resolve this issue. Assuming you and your company lack the technical expertise to finish the work yourselves, how else might you assist them in completing this task? (What other resources can you call upon, for instance?)