Chapter 8. Working with web services
- Making REST requests
- Detecting timeouts and resuming downloads
- Passing errors over HTTP
- Parsing JSON, including arbitrary JSON structures
- Versioning REST APIs
REST APIs are a cornerstone of the modern internet. They enable cloud computing, have been a pillar in the DevOps and automation movements, and set up client-side web development, among other things. They’re one of the great enablers on the internet.
Although plenty of tutorials about creating and consuming simple APIs are available, what happens when things don’t go as planned? The internet was designed to be fault-tolerant. API requests and servers need to enable that fault tolerance to work.
This chapter starts with the basics of REST APIs and quickly moves on to handling cases that don’t go as planned. You’ll look at detecting timeout failures, including those that Go doesn’t formally flag as timeouts. You’ll also look at resuming file transfers when timeouts happen, and you’ll learn how to pass errors between an API endpoint and a requesting client.
Many APIs pass information as JSON. After a quick look at how JSON parsing works in Go, you’ll learn about handling JSON structures when you don’t know the structure of the data ahead of time. This is useful when you need to work with poorly defined or undefined JSON data.
Functionality within applications changes over time, and this often causes APIs to change. When APIs change, they need to be versioned. How can APIs be versioned? You’ll learn a couple of methods for versioning REST APIs.
From this chapter, you’ll learn how to move from the basics of API handling into more robust functionality.
8.1. Using REST APIs
The Go standard library includes an HTTP client that’s pretty straightforward for most common use cases. After you move beyond the common use cases, you’ll see rarer but still regularly needed cases without a clear solution. Before we touch on a couple of those, let’s look at how the HTTP client works.
8.1.1. Using the HTTP client
The HTTP client is found in the net/http library within the standard library. It has helper functions to perform GET, HEAD, and POST requests, can perform virtually any HTTP request, and can be heavily customized.
The helper functions are http.Get, http.Head, http.Post, and http.PostForm. With the exception of http.PostForm, each function is for the HTTP verb its name suggests. For example, http.PostForm handles POST requests when the data being posted should be posted as a form. To illustrate how these functions work, the following listing shows a simple use of http.Get.
Listing 8.1. A simple HTTP get
The helper functions are all backed by the default HTTP client that’s accessible and can perform any HTTP request. For example, the following listing shows how to use the default client to make a DELETE request.
Listing 8.2. DELETE request with default HTTP client
Making a request is broken into two separate parts. The first part is the request, contained in http.Request instances. These contain the information about the request. The second part is the client that performs a request. In this example, the default client is used. By separating the request into its own object, you provide a separation of concerns. Both of these can be customized. The helper functions wrap creating a request instance and executing it with a client.
The default client has configuration and functionality to handle things like HTTP redirects, cookies, and timeouts. It also has a default transport layer that can be customized.
Clients can be customized to allow you to set up the client any way you need to. The following listing shows the creation of a simple client with a timeout set to one second.
Listing 8.3. A simple custom HTTP client
Custom clients allow numerous elements to be customized, including the transport layer, cookie handling, and the way that redirects are followed.
8.1.2. When faults happen
The internet was designed with fault tolerance in mind. Things break or don’t work as expected, and you try to route around the problem while reporting it. In the age of cloud-native computing, this characteristic has been used to allow applications to move between locations and to be updated in place. When you’re working with HTTP connections, it’s useful to detect problems, report them, and try to fix them automatically when possible.
Technique 49 Detecting timeouts
Connection timeouts are a common problem and useful to detect. For example, if a timeout error occurs, especially if it’s in the middle of a connection, retrying the operation might be worthwhile. On retry, the server you were connected to may be back up, or you could be routed to another working server.
To detect timeouts in the net package, the errors returned by it have a Timeout() method that’s set to true in the case of a timeout. Yet, in some cases, a timeout occurs and Timeout() doesn’t return true, or the error you’re working with comes from another package, such as url, and doesn’t have the Timeout() method.
Timeouts are typically detected by the net package when a timeout is explicitly set, such as in listing 8.3. When a timeout is set, the request needs to complete in the time-out period. Reading the body is included in the timeout window. But a timeout can also happen when one isn’t set. In this case, a timeout in the network occurs while the timeout checking isn’t actively looking for it.
Problem
How can network timeouts be reliably detected?
Solution
When timeouts occur, a small variety of errors occurs. Check the error for each of these cases to see if it was a timeout.
Discussion
When an error is returned from a net package operation or a package that takes advantage of net, such as http, check the error against known cases showing a timeout error. Some of these will be for the explicit cases where a timeout was set and cleanly detected. Others will be for the cases where a timeout wasn’t set but a timeout occurred.
The following listing contains a function that looks at a variety of error situations to detect whether the error was caused by a timeout.
Listing 8.4. Detect a network timeout from error
This function provides the capability to detect a variety of timeout situations. The following snippet is an example of using that function to check whether an error was caused by a timeout:
res, err := http.Get("http://example.com/test.zip")
if err != nil && hasTimedOut(err) {
fmt.Println("A timeout error occured")
return
}
Reliably detecting a timeout is useful, and the next technique highlights this in practice.
Technique 50 Timing out and resuming with HTTP
If a large file is being downloaded and a timeout occurs, starting the download from the beginning isn’t ideal. This is becoming truer with the growth of file sizes. In many cases, files are gigabytes or larger. It’d be nice to avoid the extra bandwidth use and time to redownload data.
Problem
You want to resume downloading a file, starting from the end of the data already downloaded, after a timeout occurs.
Solution
Retry the download again, attempting to use the Range HTTP header in which a range of bytes to download is specified. This allows you to request a file, starting partway through the file where it left off.
Discussion
Servers, such as the one provided in the Go standard library, can support serving parts of a file. This is a fairly common feature in file servers, and the interface for specifying ranges has been a standard since 1999, when HTTP 1.1 came out:
This snippet creates a local file location, downloads a remote file to it, displays the number of bytes downloaded, and will retry up to 100 times when a network timeout occurs. The real work is inside the download function spelled out in the following listing.
Listing 8.5. Download with retries
Although the download function can handle timeouts in a fairly straightforward manner, it can be customized for your cases:
- The timeout is set to five minutes. This can be tuned for your application. A shorter or longer timeout may provide better performance in your environment. For example, if you’re downloading files that typically take longer than five minutes, a timeout longer than most files take will limit the number of HTTP requests needed for a normal download.
- If a hash of a file is easily available, a check could be put in to make sure that the final download matches the hash. This integrity check can improve trust in the final download, even if it takes multiple attempts to download the file.
Checking for errors and attempting to route around the problem can lead to fault-tolerant features in applications.
8.2. Passing and handling errors over HTTP
Errors are a regular part of passing information over HTTP. Two of the most common examples are Not Found and Access Denied situations. These situations are common enough that the HTTP specification includes the capability to pass error information from the beginning. The Go standard library provides a rudimentary capability to pass errors. For example, the following listing provides simple HTTP generating an error.
Listing 8.6. Passing an error over HTTP
This simple server always returns the error message An Error Occurred. Along with the custom message, served with a type of text/plain, the HTTP status message is set to 403, correlating to forbidden access.
The http package in the standard library has constants for the various status codes. You can read more about the codes at https://en.wikipedia.org/wiki/List_of_HTTP_status_codes.
A client can read the codes the server responds with to learn about what happened with the request. In listing 8.5, when the res.StatusCode is checked, the client is looking for a status in the 200 range, which signifies a successful request. The following snippet shows a simple example of printing the status:
res, _ := http.Get("http://example.com")
fmt.Println(res.Status)
fmt.Println(res.StatusCode)
The res.Status is a text message for the status. Example responses look like 200 OK and 404 Not Found. If you’re looking for the error code as a number, res.Status-Code is the status code as an int.
Both the response code and the error message are useful for clients. With them, you can display error messages and automatically handle situations.
8.2.1. Generating custom errors
A plain text error string and an HTTP status code representing an error are often insufficient. For example, if you’re displaying web pages, you’ll likely want your error pages to be styled like your application or site. Or if you’re building an API server that responds with JSON, you’ll likely want error responses to be in JSON as well.
The first part of working with custom error responses is for the server to generate them.
Technique 51 Custom HTTP error passing
You don’t have much room for customization when using the Error function within the http package. The response type is hardcoded as plain text, and the X-Content-Type-Options header is set to nosniff. This header tells some tools, such as Microsoft Internet Explorer and Google Chrome, to not attempt to detect a content type other than what was set. This leaves little opportunity to provide a custom error, aside from the content of the plain text string.
Problem
How can you provide a custom response body and content type when there’s an error?
Solution
Instead of using the built-in Error function, use custom functions that send both the correct HTTP status code and the error text as a more appropriate body for your situation.
Discussion
Providing error responses that are more than a text message is useful to those consuming an application. For example, someone viewing a web page gets a 404 Not Found error. If this error page is styled like the rest of the site and provides information to help users find what they’re looking for, it can guide users rather than only provide a surprise that what they’re looking for wasn’t found and that they can’t easily find it.
A second example involves REST API error messages. APIs are typically used by software development kits (SDKs) and applications. For example, if a call to an API returns a 409 Conflict message, more detail could be provided to guide the user. Is there an application-specific error code an SDK can use? In addition to the error message, is there additional guidance that can be passed to the user?
To illustrate how this works, let’s look at an error response in JSON. We’ll keep the same response format as the other REST API responses that provide an application-specific error code in addition to the HTTP error. Although this example is targeted at API responses, the same style applies to web pages.
Listing 8.7. Custom JSON error response
This listing is conceptually similar to listing 8.6. The difference is that listing 8.6 returns a string with the error message, and listing 8.7 returns a JSON response like the following:
{
"error": {
"code": 123,
"message": "An Error Occurred"
}
}
After errors are passed as JSON, an application reading them can take advantage of the data being passed in this structured format. Using errors passed as JSON can be seen in the next technique.
8.2.2. Reading and using custom errors
Any client can work with HTTP status codes to detect an error. For example, the following snippet detects the various classes of errors:
res, err := http.Get("http://goinpracticebook.com/")
switch {
case 300 <= res.StatusCode && res.StatusCode < 400:
fmt.Println("Redirect message")
case 400 <= res.StatusCode && res.StatusCode < 500:
fmt.Println("Client error")
case 500 <= res.StatusCode && res.StatusCode < 600:
fmt.Println("Server error")
}
The 300 range of messages has to do with redirects. You’ll rarely see these because the default setting for the HTTP client is to follow up to 10 redirects. The 400 range represents client errors. Access Denied, Not Found, and other errors are in this range. The 500 range of errors is returned when a server error occurs; something went wrong on the server.
Using the status code can provide insight into what’s going on. For example, if the status code is a 401, you need to log in to see the request. A user interface could then provide an opportunity to log in to try the request again, or an SDK could attempt to authenticate or re-authenticate and try the request again.
Technique 52 Reading custom errors
If an application responds with custom errors, such as those generated by technique 51, this presents an API response with a different structure from the expected response in addition to there being an error.
Problem
When a custom error with a different structure is returned as an API response, how can you detect that and handle it differently?
Solution
When a response is returned, check the HTTP status code and MIME type for a possible error. When one of these returns unexpected values or informs of an error, convert it to an error, return the error, and handle the error.
Discussion
Go is known for explicit error handling, and HTTP status codes are no different. When an unexpected status is returned from an HTTP request, it can be handled like other errors. The first step is to return an error when the HTTP request didn’t go as expected, as shown in the next listing.
Listing 8.8. Convert HTTP response to an error
This code replaces the http.Get function for making a request to a server with the get function, which handles custom errors. The Error struct, which holds the data from the error, has the same structure as the error in technique 51. This custom error handling is designed to work with a server that emits errors in the same way as technique 51. These two techniques could share a common package defining the error.
Adding the Error() method to the Error type implements the error interface. This allows instances of Error to be passed between functions as an error, like any other error.
The main function in the following snippet illustrates using the get function instead of http.Get. Any custom errors will print the custom error details from the JSON and exit the application:
func main() {
res, err := get("http://localhost:8080")
if err != nil {
fmt.Println(err)
os.Exit(1)
}
b, _ := ioutil.ReadAll(res.Body)
res.Body.Close()
fmt.Printf("%s", b)
}
Using this technique for getting and passing HTTP errors around applications allows these errors to get the benefits of other error handling in Go. For example, using switch statements to test the type of error and reacting appropriately, as listing 8.4 showed, will work for the custom errors.
8.3. Parsing and mapping JSON
When communicating over REST APIs, the most common format to transfer information is JSON. Being able to easily and quickly convert JSON strings into native Go data structures is useful, and the Go standard library provides that functionality out of the box via the encoding/json package. For example, the following listing parses a simple JSON data structure into a struct.
Listing 8.9. A simple custom JSON-parsing example
Although the standard library provides everything you need for the foundational JSON-parsing use cases, you may run into some known and common situations without an obvious solution.
Technique 53 Parsing JSON without knowing the schema
The structure of JSON is often passed along via documentation, examples, and from reading the structure. Although schemas exist for JSON, such as JSON Schema, they’re often not used. Not only is JSON schemaless, but API responses may vary the structure, and in some cases you may not know the structure.
When JSON data is parsed in Go, it goes into structs with a structure defined in the code. If you don’t know the structure when the structs are being created, or the structure changes, that presents a problem. It may seem as though it’s difficult to introspect JSON or operate on documents with a varying structure. That’s not the case.
Problem
How can you parse a JSON data structure into a Go data structure when you don’t know the structure ahead of time?
Solution
Parse the JSON into an interface{} instead of a struct. After the JSON is in an interface, you can inspect the data and use it.
Discussion
A little-known feature of the encoding/json package is the capability to parse arbitrary JSON into an interface{}. Working with JSON parsed into an interface{} is quite different from working with JSON parsed into a known structure, because of the Go type system. The following listing contains an example of parsing JSON this way.
Listing 8.10. Parse JSON into an interface{}
The JSON parsed here contains a variety of structure situations. This is important because working with the interface{} isn’t the same as working with JSON parsed into a struct. You’ll look at working with this data in a moment.
When JSON data is parsed into a struct, such as the example in listing 8.9, it’s easily accessible. In that case, the name of the person from the parsed JSON is available at p.Name. If you tried to access firstName on the interface{} in the same way, you’d see an error. For example:
fmt.Println(f.firstName)
Accessing firstName like a property would generate an error:
f.firstName undefined (type interface {} is interface with no methods)
Before you can work with the data, you need to access it as a type other than interface{}. In this case, the JSON represents an object, so you can use the type map[string] interface{}. It provides access to the next level of data. The following is a way to access firstName:
m := f.(map[string]interface{})
fmt.Println(m["firstName"])
At this point, the top-level keys are all accessible, allowing firstName to be accessible by name.
To programmatically walk through the resulting data from the JSON, it’s useful to know how Go treats the data in the conversion. When the JSON is unmarshaled, the values in JSON are converted into the following Go types:
- bool for JSON Boolean
- float64 for JSON numbers
- []interface{} for JSON arrays
- map[string]interface{} for JSON objects
- nil for JSON null
- string for JSON strings
Knowing this, you can build functionality to walk the data structure. For example, the following listing shows functions recursively walking the parsed JSON, printing the key names, types, and values.
Listing 8.11. Walk arbitrary JSON
Although it’s handy to be able to parse and work with JSON when you don’t know the structure, it’s useful to have known structures or to handle the version changes when those structures change. In the next section, you’ll learn about versioning APIs that includes changes in JSON structures.
8.4. Versioning REST APIs
Web services evolve and change, which leads to changes in the APIs used to access or manage them. To provide a stable API contract for API consumers, changes to the API need to be versioned. Because programs are the users of an API, they need to be updated to account for changes, which takes time after an update is released.
APIs are typically versioned by major number changes such as v1, v2, and v3. This number scheme signifies breaking changes to the API. An application designed to work with v2 of an API won’t be able to consume the v3 API version because it’s too different.
But what about API changes that add functionality to an existing API? For example, say that functionality is added to the v1 API. In this case, the API can be incremented with a point version; feature additions can increment the API to v1.1. This tells developers and applications about the additions.
The following two techniques cover a couple of ways to expose versioned APIs.
Technique 54 API version in the URL
A change in the API version needs to be easy to see and work with. The easier it is for developers to see, understand, and consume, the more likely they are to work with it and to fully use services.
Versioned APIs that easily work with existing tools are also important. For example, the ability to quickly test API calls with cURL or Postman, a popular API extension for Chrome, makes it easier for developers to develop and test APIs.
Problem
What is an easily accessible method to provide versioned APIs?
Solution
Provide the API version in the REST API URL. For example, instead of providing an API of https://example.com/api/todos, add a version to the path so it looks like https://example.com/api/v1/todos.
Discussion
Figure 8.1 illustrates an incredibly popular method for versioning APIs: via the URL. Google, OpenStack, Salesforce, Twitter, and Facebook are a few examples that use APIs versioned this way.
Figure 8.1. REST API version in the URL
As the following listing shows, implementing this URL structure is done when the mapping between path and handlers occurs.
Listing 8.12. Register the API path including a version
In this example, the way the handler function is mapped to the path doesn’t allow you to easily handle different request methods such as POST, PUT, or DELETE. If an endpoint represents a resource, the same URL typically handles all these requests. You can find techniques for handling multiple HTTP methods being mapped to the same URL in chapter 2.
Although this is an easy method for passing an API version, it’s not technically semantic. A URL doesn’t represent an object. Instead, it represents accessing an object within a version of an API. The trade-off is developer ease. Specifying an API version in the URL is easier for developers consuming the API.
Technique 55 API version in content type
Although the previous technique focused on a method that was easy for developers, the method wasn’t semantic. Part of the original theory of REST was that a URL represented something. That could be an object, list, or something else. Based on the details in the request, such as the requested content type or HTTP method, the response or action to that object would be different.
Problem
How can API versions be handled in a semantic manner?
Solution
Instead of referencing JSON in the request and response, use a custom content type that includes the version. For example, instead of working with application/json, use a custom content type such as application/vnd.mytodo.v1.json or application/vnd.mytodo.json; version=1.0. These custom types specify the intended schema for the data.
Discussion
To handle multiple API versions at a single path, as seen in figure 8.2, the handling needs to take into account the content type in addition to any other characteristics. Listing 8.13 showcases one method for detecting the content type and using that to generate the response.
Figure 8.2. Differences between semantic URLs and API version in URL
Listing 8.13. Pass the API version in the content type
When a client requests the content, it can specify no content type to get the default response. But if it wants to use API version 2, it will need to forego a simple GET request and specify more details. For example, the following snippet requests version 2 and prints out the response:
Although this method provides the capability to have multiple API versions from a single endpoint, you need to be aware of the following considerations:
- Content types in the vnd. namespace are supposed to be registered with the Internet Assigned Numbers Authority (IANA).
- When making a request for a nondefault version, you need to add extra steps to specify the content type for the version. This adds more work to applications consuming the API.
8.5. Summary
In this chapter, you started with the basics of working with web services such as making REST requests. You quickly moved from the basics to elements of building robust web service interactions that included the following:
- Detecting network timeouts, even when the network layer doesn’t formally flag them, and resuming downloads when timeouts occur.
- Passing errors between API endpoints and client requestors by using and going beyond the HTTP status header.
- Parsing JSON, even when you don’t know the structure ahead of time.
- Using two methods for versioning REST APIs and working with versioned APIs.
In the next chapter, you’ll learn about working with cloud services. Running applications effectively in the cloud involves more than working with the APIs that let you configure them. In chapter 9, you’ll learn techniques to help your Go applications be effective in the cloud.