Is this a core competency?

Always weigh whether a solution/integration under consideration is both your core competency as well as the company’s core competency.

Third parties offer specialized solutions and have teams of engineers working to expand their value proposition by building additional features and services. A Software as a Service (SaaS) solution for log aggregation might be more effective in the long run than trying to adopt an open source solution such as Elasticsearch. You have to decide whether Elasticsearch is really something that you, your team, and your company need to focus on.

Integration timeline?

However long you think it’s going to take to complete an integration, double it. Then add a little more. A PoC for an open source solution might be easy to implement, but this does not mean that the project is production ready. The definition of production readiness for SREs should encompass testing/staging, monitoring, logging, configuration management, deployment procedure, disaster recovery, and beyond.

With things like CDN, DNS, monitoring, CA, and messaging solutions, implementation of these production readiness factors can prove to be incredibly challenging, and in some cases, impossible. Fortunately, with third-party integrations, especially at larger companies, a variety of teams will evaluate the solution as we work on the integration. You should expect that Legal, Security, and Procurement² teams will play a role during the purchase process. Each of these teams will introduce its own processes, which can delay getting the third-party solution into production.

Project Operating Expense and Abandonment Expense

Beyond consideration for the initial cost to purchase goods and services (CapEx, such as hardware, software, and one-time licenses) and the ongoing costs to run and maintain (OpEx, such as monthly server prices, recurring service fees), additional expense categories should be a part of the consideration process.

Project Operating Expense (PrOpEx)

Implementation costs for a solution. Typically a single line item in the CapEx table, implementation costs can prove to be a more significant factor in the decision over build, buy, or adopt choices. Implementation costs to consider include services that will be utilized as the solution is implemented, such as legal fees, vulnerability probing, additional monitoring and performance infrastructure, and professional or consulting services. These costs can also expose overlooked OpEx costs.

Abandonment expense (AbEx)

This is the cost to abandon a solution to implement another altogether. This is sometimes referred to by the softer term migration cost. There is always the potential that adopting an open source solution might result in abandonment in order to buy a fleshed-out SaaS solution, or vice versa. Costs will differ between abandonment scenarios. In the buy-to-adopt scenario, you might be required to pay out contractual obligations such as monthly usage commitments in addition to the incurred costs of restarting a project. In the adopt-to-buy scenario, cost is less of an issue; however, time is a much larger factor. In the previous section, we saw that Legal, Security, and Procurement processes can prolong time to production. It’s in our best interest to plan accordingly and abandon early.

You should consider AbEx throughout the lifetime of a project. Working with third parties means that we should also consider the buy-to-buy abandonment scenario, as well—that is, leaving one vendor to use a cheaper, more robust vendor’s solution. If you’re not careful, one vendor’s custom solution could cause vendor lock-in, ultimately increasing abandonment expense. Vendor lock-in can also tip the scales in the vendor’s favor when it comes time to renegotiate contracts.

After deliberating the risks and benefits to every solution, you may reach the decision to buy. Therefore, it should come as no surprise to the higher-ups that there’s work involved in setting up the third party beyond simple configuration. SREs have an obligation to build world-class solutions, and to that end, we should explore some best practices to move third parties from being ancillary to your tech stack to being an extension of it.

Direct impact

Quite often, the third-party providers that we work with have a direct impact on site availability, site reliability, and performance. In fact, these third parties are obviously impactful to the overall reliability. Here is a short list of providers with direct impact:

• DNS is the first system that end users experience. When DNS is down, you’re down.

• Edge products, such as site accelerators, wide-area firewall, forward proxies/load balancers, and so on, can suffer congestion issues, human configuration error, and routing failures.

• CDNs for static objects, such as JavaScript and stylesheets, play a role in perceived user experience; improper configuration can lead to ineffective cacheability and high page load times, leading to increased bounce rates. The ability to diagnose and triage performance has become increasingly complicated with Single-Page Application (SPA) frameworks.⁴

Indirect impact

Some providers impact site reliability in less obvious ways. Based on experience, these are third parties that process transactions on the backend of the technology stack. The following are examples of providers with indirect impact:

• Payment processors outages—for instance, failed API key rotation or incorrect billing codes—should not delay user experience. An optimistic approach might use previous billing history to indicate future payment processing success and will simply queue the transaction.

• Synthetic and Real-User Monitoring (RUM) are necessary for telemetry; however, use caution when utilizing either for automation.⁵

• SMS and email integrations generally have no direct touchpoints between your end users and your tech stack. However, end users have come to expect instant gratification; therefore, undelivered payment receipts or delayed two-factor verification codes will lead to a negative perception of your company.

SREs are not only seeking excellent uptimes, but also a consistent user experience overall. A theoretically simple mitigation strategy for mission-critical services is to establish redundancy. Whether direct or indirect, these mission-critical third-party integrations should be treated as first-class citizens and run like any other service in your tech stack.

But if third parties are categorized as black boxes, we shouldn’t expect much from them in terms of data, right? That’s simply not true in modern day. More and more third parties are building APIs that allow us to take advantage of running their infrastructure as code, which means we are able to manage configuration, reporting, and a suite of other features to run our third-party products more like services.

SLIs on black boxes

SLIs for services within our own tech stack are objectively easy to collect. In many cases, it’s a matter of running a daemon to collect operating system/virtual machine/container stats or a library/SDK that we import to emit metrics to our metrics collection cluster. Easy enough. Vendors run their services within their own tech stack. They will not typically emit server-level metrics; however, they do offer a wealth of data in a couple of different forms.

SLOs

SREs typically focus on meeting their SLOs—the agreement between SRE and product owners for how often a service’s SLIs can be out of spec. As an SRE working with a provider, SLOs can be difficult to calculate and even more difficult to define. It’s not simply a matter of carrying over the SLA from the vendor. SREs responsible for third-party integrations might be dealing with multiple vendors simultaneously and support a broad range of product teams, as is the case with payments, ads, CDN, and DNS. In this scenario, the product owners end up being your sibling SRE teams. The SLOs we define in this scenario allow our sibling SRE teams to calculate risk and formulate error budgets.

Testing and staging

Quarterly releases are ancient history. These days, Continuous Integration (CI) and Continuous Deployment (CD) are standard. We’ve become accustomed to maintaining a deployment pipeline similar to one that is triggered by committing code to master and allowing our automation to run a suite of unit tests, followed by deployment to a staging environment, followed by Selenium testing, followed by canary deployment, and finally production deployment.

Third-party integrations can pose a different set of challenges with regard to properly testing functionality within staging environments. It’s unlikely that CDN, DNS, certificates, and edge proxy configurations are considered as part of a CI/CD pipeline. It’s even less likely that your staging environment resembles production, if that staging environment even exists. Unfortunately, as these products have been seen as ancillary, little attention has been given to how these technologies play a role in the deployment process.

With the emergence of new players in the CDN, DNS, certificate, edge, and APM space, progress is being made as to how we can configure and manage their respective products via code. As a result, replication of environments has become more feasible. Additionally, some vendors offer some form of staging environment. For example, Akamai offers configuration staging as a standard part of its configuration deployment procedure; the staged configuration is triggered by modifying hostname resolution to target the staging hosts. Staging environments ultimately encourage the creation of unit, regression, and Selenium test regimens.

Monitoring

There are two categories of monitoring that we should consider with third-party solutions: integration monitoring (is the integration working?) and impact monitoring (what impact does the solution have on our users?). Obviously, we want to ensure that the integrated service (CDN, message processor, etc.) is stable; hence, we should be looking at whether the third party is up or down. Additionally, we want to know when the integrated service is dropping connections, becoming overloaded, or introducing processing delays. Some third-party integrations rely entirely on APIs; thus, monitoring the integration is a bit more straightforward, as is the case with an SMS or email provider.

Impact monitoring is a bit different. When we relinquish some control of our site to third parties—for example, content delivery by CDN or steering to data centers by DNS providers—monitoring a third party’s impact on our users becomes a bit more challenging. To understand how end users are impacted by these critical path integrations, we should rely on synthetic monitoring or RUM.

Tooling

Tooling is a critical part of our third-party integration, especially in a larger organization. As an SRE responsible for third-party integrations, we want to avoid being a choke point as much as possible. To that end, we should build systems that take advantage of everything the third party has to offer via APIs. These APIs facilitate integrations by allowing you to build the tools needed to effectively manage your third-party integrations. As these APIs are employed, we will want to consider AbEx and avoid the use of custom features that could create vendor lock-in. We can see an example of this in CDN purge APIs—some vendors utilize surrogate keys, some utilize some form of regular expression, some utilize wildcards, and some utilize batching. Any of these implementations into our regular workflow could cause an issue if we need to replace or add another vendor’s solution.

Further removing us from being a choke point, we want to avoid any integration that requires we play a role in access control. Managing access to vendor portals for every engineer is overly cumbersome and has the potential to lead to security issues and technical debt.

Finally, as we create tooling around our vendor’s API, it’s critical that we consider abandonment. Try to maintain a modular architecture to support a variety of third parties. This layer of abstraction allows for additional providers and swapability should a third-party vendor need to be replaced.

Automation

As an extension of tooling, we want to be able to automate some of the functionality of our third-party solutions as much as possible. The possibilities here are endless and fully depend on how you make use of your third party’s APIs. To give you an idea of what we can do, let’s look at the following examples:

Automating content removal on account closure

When a user closes their account, certain privacy laws apply and require that user PII is deleted. As a result, it’s possible to utilize your company’s data pipeline (or similar message queue) to identify such events to automate the image removal process. This will be necessary for GDPR compliance.

Automating data center/endpoint selection

In the event that a critical service is down behind an endpoint, we can use DNS APIs to steer traffic away from the affected endpoint.

Automating certificate renewal

In the event that certificates are ready to expire, you could easily use your CA’s API to reinstall certificates to your endpoints rather than relying on manual intervention or outage.

Logging

Third-party reporting APIs are generally not great. Often, the reporting data we receive via pull APIs are aggregated, processed, and scrubbed. Aggregation delays are common with pull API and should not necessarily be used for raising alerts. As an SRE, we need to know what’s happening now, not what happened 15 minutes ago. To that end, third-party integrations have often been looked at as black boxes with little to no visibility into their operations.

But times have changed. Many third parties offer real-time data feeds that include all sorts of useful data; CDNs are offering delivery receipts for every object; DNS providers are offering query data; and synthetic monitoring providers offer granular test data.

The benefit to all of this available data is that we’re able to build with it as we see fit. However, this is also a risk. Just because we have the data does not mean that it’s entirely useful. As we utilize real-time logging data, we need to be sure we’re utilizing what we need to supplement our monitoring as well as provide a way to track down issues. For example, CDN logs can include a browser type, extension details, client IP details, cookies, and other request headers. Much of this is unnecessary for most applications, such as regional isolation of HTTP response codes.

Finally, logging can be absurdly expensive. Not only do some vendors charge for the delivery of this data, but you’re also responsible for consuming, indexing, processing, and retaining it. This can be a large undertaking in and of itself, and you should approach it with caution.

Disaster planning

Although we don’t like to think about it, working with a third-party vendor incurs some risk. As part of our playbook, we will need to consider what to do when the vendor is hard down. Disaster planning should include considerations for the removal of the third party altogether. You will likely have to consider the following:

• Maintaining an additional (as active or standby) provider

• Having capacity to serve requests ourselves

• Queueing requests to be handled later

Removal of a CDN might require that we accept a certain amount of latency. Removal of a payment processor might require an optimistic approach to ensure a good customer experience. Some delay in a welcome message or payment receipt via email might not be entirely noticeable to an end user. Remember, slow is better than down.

Communication

With third-party integrations, it’s important to not live in a silo. Talking to a solutions or sales engineer actually plays to your advantage for discussing product enhancements, discovering unknown or underutilized features, understanding their product roadmap, and preparing third parties for future events such as product releases. Additionally, it pays to maintain bidirectional communication to align priorities. Third-party vendors should be under NDA; therefore, it’s helpful to communicate any upcoming projects that might alter the relationship.

Establishing a regular call cadence can be useful but might not be ideal for all SREs. Often, these calls are tedious are time-consuming, and can conflict with production issues. Additionally, there is a bit of organization required, especially when you’re working with multiple vendors. To that end, it’s often a good idea to work with a project manager or someone similar who can facilitate these calls and keep detailed notes. Support from a project manager will help you keep your vendors aligned with your team’s and company’s priorities.

Decommissioning

When it comes time to decommission a service, we consider all of our dependencies. The same is true of dealing with third parties. We obviously need to consider removal of tooling, service monitoring, alerts, and data collection services/endpoints. In addition, we are obliged to consider a few other dependencies that play a role in the integration.

Sometimes, termination is not as simple as stating “you’re fired.” Terms of the termination are generally stated in your service agreements. If you’re deciding to terminate a contract for financial reasons, you might be able to negotiate in advance of termination. In these cases, use caution when evaluating contracts that include Most Favored Customer (MFC) clauses because these often make it harder to negotiate a better deal. Some agreements also allow for autorenewal months prior to a contract’s end date. Your legal and procurement teams might have prevented this, but it’s important to look at the contract and discuss termination with them.

Communication obviously played a role in the purchase and ongoing relationship of your third-party integration. Communication is even more critical during contract termination. If, for instance, the third party was not meeting SLAs but that was never communicated, it’s not necessarily fair to that vendor and could prove costly for the third-party sales teams.

The tech industry is small, and stories of a bad breakup will haunt you and your company. These stories could play a role in your future integration attempts. Sales organizations remember which customers were most difficult to work with, and salespeople move between companies, especially within the same industry. Failure to communicate during the decommissioning process can tarnish reputations and complicate future relationships.

The best way to avoid this is to communicate your intent to terminate a third party well in advance of the end date—at least a quarter ahead. This gives the third-party sales team an opportunity to rectify any problems that might have persisted during the integration life cycle. We vote with our dollars. Losing a large enough customer can incentivize a third party to prove themselves.