Bound Resources

Every application we build needs resources. In the traditional world of deploying apps and services to specific servers (virtual or physical), we’re used to our applications needing things like files on disk. These apps also have configuration, credentials, and URLs for accessing other services, and any number of other dependencies that often tightly couple the application to the server on which it is supposed to run.

When we’re running our services in the cloud, we need to build our applications with a slightly more abstract notion. Every resource needed by our application should be considered a bound resource, and accessed in a way that doesn’t violate any of the rules of cloud-native applications.

For example, if our application needs to read and write binary files, we can’t assume that we can use System.IO.File to read and write bytes to disk. This is because the disk in the cloud must be considered ephemeral. It is subject to complete removal without our application knowing. This is part of what allows our services to rapidly and dynamically scale—instances can be brought up and shut down anywhere in the world on demand. If it expects a file to exist on a local disk between requests or process starts, our app is going to fail in unpredictable and potentially catastrophic ways.

The solution is to assume that everything, including the filesystem, is a service. Backing services are bound to our application through some mechanism likely facilitated by a cloud provider (PaaS, or Platform as a Service). Instead of opening a file, we communicate with a generalized persistence service. This could be something we build ourselves, or it could be an Amazon Web Services S3 bucket, or it could be any number of other brokered persistence services available.

Likely one of the most common types of bound resources is a  database connection. The binding of this resource contains things we should all be familiar with, such as a connection string and credentials. We’ll see more about bound resources and database connections in Chapter 5.

Lastly, as we’ll see in the samples in this chapter, other microservices are also bound resources. The URLs and credentials to the services on which our own service depends should be considered part of the resource binding.

I should note that the concept of a resource binding is an abstraction, and the implementation of it will vary depending on what cloud platform is hosting your application. These service bindings could just as easily come from environment variables injected by your platform or from an external configuration provider.

Whether you’re using Google Cloud Platform, AWS, Azure, Heroku, or just running a bunch of Docker images manually, the key to enabling communications between services is the combination of externalized configuration and treating everything as a bound resource.

Strategies for Sharing Models Between Services

There are a few things that are required for an environment to be considered a microservice ecosystem. The first, obviously, is that you need more than one service. The second is that the services within this ecosystem communicate with each other. Without the latter, you’re just standing up an array of isolated and unrelated services.

If we’re being diligent about following some cloud-native best practices like API First, then all of our services will have documented, versioned, well-understood public APIs. We might be using a YAML standard like Swagger to document our APIs, or we could be using one based on Markdown, like API Blueprint. The mechanism of defining and documenting our APIs is not nearly as important as the discipline we put into designing our APIs before we write our code.

With a well-defined, versioned API that we know isn’t going to break out from underneath us, the services within our ecosystem can be built by different teams. Consuming the API from those services then becomes merely a matter of writing simple REST clients.

If it’s so simple, then why are we dedicating a section of the chapter to the concept of model sharing? The reason is because as people build ecosystems following the API First rule, once they get into writing the code, they often allow the API boundary to become a soft or blurred boundary.

Teams frequently make some architectural decisions early on during a project that won’t cause trouble until far into the future, when the cost of untangling the mess can get exorbitant.

As an example, let’s say that you’ve got two services in your suite that both operate on invoices. One accepts an invoice from a queue, performs processing, and then submits an updated invoice to another downstream service.

When we look at this solution on paper, it’s very easy to say something like, “Let’s just extract the invoice model and share it among services.” Seems like a great idea, and it’s used frequently enough that it is a named pattern, often called the canonical model pattern.

Fast-forward a few months, and developers on both service teams have been adding features. The invoice model and its validation rules and (de)serialization code have been factored out into a nice shared module. Because it’s easy and gets the job done, both services eventually end up performing their internal processing against the canonical or public model.

Now when one service changes the model in order to accommodate what should be an internal concern, the other service is affected and potentially has builds and tests broken as a result. They’ve lost the flexibility of true independence, and instead of being a source of flexibility, the canonical model is now a source of tight coupling and is preventing independent team deployment schedules.

It is entirely possible to maintain a canonical model without internal pollution, but you absolutely must be ruthless about maintaining a “pure” canonical model and forcing the internals of a service to use an internal model and convert back and forth with an anti-corruption layer (ACL). This is often perceived as a lot of work for little to no benefit, so many teams skip this discipline and lapse into tightly coupling internals to a public model, the consequences of which grow worse exponentially as more and more services adopt this anti-pattern.

Put another way, two services that are tightly coupled to the same shared internal model are as tightly coupled as if they resided within the same monolith.

In my opinion, based on years of building new software and untangling the messes of legacy software, the real answer is to share nothing. A microservice is an embodiment of the Single Responsibility Principle (SRP) and the Liskov Substitution Principle (LSP). A change to one service should never have any impact on any other service. A change to the internal model should be possible without corrupting the service’s public API or any external models.

Lastly, before getting into the nuts and bolts of the code for this chapter, I leave you with this quote from Sam Newman about the perils of sharing in microservices:

The evils of too much coupling between services are far worse than the problems caused by code duplication

Sam Newman, Building Microservices

Configuring Service URLs with Environment Variables

As mentioned, there are a number of different ways we can “bind” connection information for backing services to our application. The most important thing for us to remember when doing this is that this information must come from the environment. It cannot be information checked in with our codebase.

The simplest way to do this is to set some reasonable defaults in an appsettings.json file, and then allow those defaults to be overridden with environment variables. The defaults are here only to make it easier to work on the code from our workstations, and should never be left intact in real environments:

{
    "location": {
        "url": "http://localhost:5001"
    }
}

With this in place, we can override this setting with an environment variable called LOCATION__URL. Note that there are two underscores in this environment variable. Regardless of how the variable was set by the environment, we can query it by checking for the "location:url" configuration setting, thanks to ASP.NET Core’s configuration system creating a universal abstraction around the data hierarchy.

We can modify our startup so that we register an HttpLocationClient instance with the appropriate URL (we’ll see the implementation of this class shortly):

var locationUrl = Configuration.GetSection("location:url").Value;
logger.LogInformation("Using {0} for location service URL.", 
     locationUrl);
services.AddSingleton<ILocationClient>(
   new HttpLocationClient(locationUrl));

With just a single URL that never changes, this kind of environment-fed configuration is pretty easy. We’ll talk about more robust methods of configuring your applications later in the book.

Consuming a RESTful Service

Now that we know how to set up the order of precedence configuration settings allowing our file settings to be overridden with environment variables, we can focus on implementing a location client that talks to our location service.

Since we want to be able to unit test a controller method in our team service without making HTTP calls, we know we’re going to start off with creating an interface for our location client (Example 4-5).

public interface ILocationClient
{
   Task<LocationRecord> GetLatestForMember(Guid memberId);
}

And Example 4-6 is our implementation of a location client that makes simple HTTP requests. Note that the URL to which this client connects is passed in the constructor we saw in our Startup class earlier.

using System;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using StatlerWaldorfCorp.TeamService.Models;
using Newtonsoft.Json;

namespace StatlerWaldorfCorp.TeamService.LocationClient
{
  public class HttpLocationClient : ILocationClient
  {
    public String URL {get; set;}

    public HttpLocationClient(string url)
    {
        this.URL = url;
    }

    public async Task<LocationRecord> 
      GetLatestForMember(Guid memberId) 
    {
        LocationRecord locationRecord = null;

        using (var httpClient = new HttpClient())
        {
            httpClient.BaseAddress = new Uri(this.URL);
            httpClient.DefaultRequestHeaders.Accept.Clear();
            httpClient.DefaultRequestHeaders.Accept.Add(
              new MediaTypeWithQualityHeaderValue(
                 "application/json"));

            HttpResponseMessage response = 
                await httpClient.GetAsync(
                 String.Format("/locations/{0}/latest",
                     memberId));

            if (response.IsSuccessStatusCode) {
              string json = 
                await response.Content.ReadAsStringAsync();
              locationRecord = 
                JsonConvert
                 .DeserializeObject<LocationRecord>(json);
              }
          }
          return locationRecord;
      }       
  }
}

With a location service client available, we can now modify the controller method in the team service responsible for querying member details. I didn’t explicitly cover the code for this controller in the previous chapter, so if you didn’t write your own you can find a copy in the GitHub repository.

We’ll modify the controller to invoke the location client so we can append the most recent location for the member to the response (Example 4-7).

[HttpGet]
[Route("/teams/{teamId}/[controller]/{memberId}")]    
public async virtual Task<IActionResult> 
   GetMember(Guid teamID, Guid memberId) 
{
   Team team = repository.GetTeam(teamID);
      
   if(team == null) {
     return this.NotFound();
   } else {
     var q = team.Members.Where(m => m.ID == memberId);
     if(q.Count() < 1) {
       return this.NotFound();
     } else {
       Member member = (Member)q.First();

       return this.Ok(new LocatedMember {
         ID = member.ID,
         FirstName = member.FirstName,
         LastName = member.LastName,
         LastLocation = 
           await this.locationClient.GetLatestForMember(member.ID)
       });
     }
   }
}

It’s also worth pointing out here that the LocationRecord model class we’re using is private to the team service. Per the earlier discussion on model sharing, the team service and location service are not sharing models, which allows the team service to remain coupled only to the public API of the location service, and not the internal implementation.

We’re getting away with what could almost be called an implicit anti-corruption layer here because we’re using the fact that two JSON payloads look the same on both sides of a conversation.

In more typical scenarios, we would invoke some form of translation utility to convert from the location service’s public API format to the type of information we need for our own internal model.

Running the Services

Before we continue, here’s a quick recap of what we’ve done so far. We decided that we wanted to add the ability to maintain a history of location check-ins for each person using our application. To do this, we created a location service that is the sole owner of location data and exposes a convenience endpoint for checking a member’s most recent location.

The new location service is in GitHub, in the no-database branch. The team service we modified to consume the location service can be found at the team service’s location branch.

You can also run both of these branches directly from tagged Docker images on docker hub:

• Team service: dotnetcoreservices/teamservice:location
• Location service: dotnetcoreservices/locationservice:nodb

First let’s start up the team service. We need to give it two different configuration parameters via environment variables:

• Port number, using the PORT variable. We will have to give our services two different ports if we’re running them locally so they don’t collide.
• Location URL, using the LOCATION__URL variable (remember, two underscores).

Run the following command:

$ docker run -p 5000:5000 -e PORT=5000 
  -e LOCATION__URL=http://localhost:5001 
  dotnetcoreservices/teamservice:location
... 
info: Startup[0]
      Using http://localhost:5001 for location service URL.
Hosting environment: Production
Content root path: /pipeline/source/app/publish
Now listening on: http://0.0.0.0:5000
Application started. Press Ctrl+C to shut down.

This starts the team service on port 5000, maps port 5000 inside the container to port 5000 on localhost, and tells the team service that it can find the location service at http://localhost:5001.

If you’re on a Mac, you can also pass a one-time environment variable straight through to the dotnet run command as shown here:

LOCATION__URL=http://localhost:5001 dotnet run

With the team service up and running, let’s start the location service:

$ docker run -p 5001:5001 -e PORT=5001 
   dotnetcoreservices/locationservice:nodb
...
Status: Downloaded newer image for dotnetcoreservices/locationservice:nodb
starting
Hosting environment: Production
Content root path: /pipeline/source/app/publish
Now listening on: http://0.0.0.0:5001
Application started. Press Ctrl+C to shut down.

Now we’ve got two services running. You can see the Docker configuration for both of those services by using the docker ps command. Next we’re going to need to run a series of commands to see everything working:

1. Create a new team.
2. Add a member to that team.
3. Query the team details to see the member.
4. Add a location to that member’s location history.
5. Query the member’s details from the team service to see their location added to the response.

If you’re using Windows, you can achieve the same effect by using your favorite REST client.

Create a new team:

$ curl -H "Content-Type:application/json" -X POST -d 
 '{"id":"e52baa63-d511-417e-9e54-7aab04286281", 
    "name":"Team Zombie"}' http://localhost:5000/teams

Add a new member by posting to the /teams/{id}/members resource:

$ curl -H "Content-Type:application/json" -X POST -d 
 '{"id":"63e7acf8-8fae-42ce-9349-3c8593ac8292", 
   "firstName":"Al", 
   "lastName":"Foo"}' 
 http://localhost:5000/teams/e52baa63-d511-417e-9e54-7aab04286281/members

To confirm that everything has worked so far, query the team details resource:

$ curl http://localhost:5000/teams/e52baa63-d511-417e-9e54-7aab04286281

{"name":"Team Zombie",
  "id":"e52baa63-d511-417e-9e54-7aab04286281",
  "members":[{"id":"63e7acf8-8fae-42ce-9349-3c8593ac8292",
    "firstName":"Al","lastName":"Foo"}]}

Now that the team service has been properly primed with a new team and a new member, we can add a location to the location service. Note that we could have just added an arbitrary location, but the team service wouldn’t be able to find it without at least one team with one member with a location record:

$ curl -H "Content-Type:application/json" -X POST -d 
 '{"id":"64c3e69f-1580-4b2f-a9ff-2c5f3b8f0e1f", 
   "latitude":12.0,"longitude":12.0,"altitude":10.0, 
   "timestamp":0, 
   "memberId":"63e7acf8-8fae-42ce-9349-3c8593ac8292"}' 
 http://localhost:5001/locations/63e7acf8-8fae-42ce-9349-3c8593ac8292

{"id":"64c3e69f-1580-4b2f-a9ff-2c5f3b8f0e1f",
  "latitude":12.0,"longitude":12.0,
  "altitude":10.0,"timestamp":0,
  "memberID":"63e7acf8-8fae-42ce-9349-3c8593ac8292"}

Finally everything is set up to truly test the integration of both the team and the location service. To do this, we’ll query for the member details from the teams/{id}/members/{id} resource:

$ curl http://localhost:5000/teams/e52baa63-d511-417e-9e54-7aab04286281 
/members/63e7acf8-8fae-42ce-9349-3c8593ac8292

{
  "lastLocation":
    {"id":"64c3e69f-1580-4b2f-a9ff-2c5f3b8f0e1f",
      "latitude":12.0,"longitude":12.0,
      "altitude":10.0,"timestamp":0,
      "memberID":"63e7acf8-8fae-42ce-9349-3c8593ac8292"},
  "id":"63e7acf8-8fae-42ce-9349-3c8593ac8292",
  "firstName":"Al",
  "lastName":"Foo"
}

I apologize for the lack of a shiny interface to these services. This book is all about building services and not about presentation to users. Additionally, given my lack of artistic ability, you really are better off not being subjected to my user interfaces and sticking with curl or generic REST clients.