Chapter 2. A solid foundation

This chapter covers

• Working with command-line flags, options, and arguments
• Passing configuration into an application
• Starting and gracefully stopping a web server
• Path routing for web and API servers

The foundation an application is built upon can be as important as any feature. Handling the way an application shuts down so you don’t lose data or create a bad user experience is an example of a step you take when building a strong foundation.

This chapter covers four foundational areas. You’ll start with console applications, also known as CLI applications. You’ll learn about handling command-line options, sometimes called flags or getopts, in a way that’s consistent with modern applications on Linux and other POSIX systems. As part of this, you’ll explore a setup that enables you to focus on application code rather than structure while developing console applications.

You’ll follow that up by looking at several ways to pass configuration into an application. This includes various popular file and content formats used to store configuration.

From there, you’ll move onto servers and practical measures for starting and stopping them. This may seem fairly straightforward, but addressing certain situations early (for example, stopping a server before data has finished being saved) can cut down on future problems.

The chapter ends with path-matching techniques for servers. URL path matching is common for websites and servers providing a Representational State Transfer (REST) API. You’ll learn how and when to implement a few of the common methods for path matching.

By the end of this chapter, you’ll know the best practices for building command-line clients and web servers that can serve as a foundation for building robust applications in Go.

2.1. Working with CLI applications, the Go way

Whether you’re using a console application (such as the source-control management system Git or an application such as the MySQL database or the Apache web server), command-line arguments and flags are part of the application user interface. The Go standard library has built-in functionality for working with these, but it comes with a twist.

2.1.1. Command-line flags

The argument and flag handling in the Go standard library is based on Plan 9, which has a different style from the systems based on GNU/Linux and Berkeley Software Distribution (BSD), such as Mac OS X and FreeBSD, that are in wide use today. For example, on Linux and BSD systems, you can use the command ls -la to list all files in a directory. The -la part of the command contains two flags, or options. The l flag tells ls to use the long form listing, and the a flag causes the list to include hidden files. The Go flag system won’t let you combine multiple flags, and instead sees this as one flag named la.

GNU-style commands (such as Linux’s ls) support long options (for example, --color) that require two dashes in order to tell the program that the string color isn’t five options, but one.

A debate has arisen over whether the Go (or Plan 9) system offers any real advantages over the style that dominates GNU, Linux, and BSD. Honestly, it boils down to a matter of preference. GNU-style double-dashes feel clunky and error-prone to some. But Go’s single-dash long options eliminate the ability (as you just saw) of grouping single-character options into one.

This built-in flag system doesn’t differentiate between short and long flags. A flag can be short or long, and each flag needs to be separated from the others. For example, if you run go help build, you’ll see flags such as -v, -race, -x, and -work. For the same option, you’ll see a single flag rather than a long or short name to use.

To illustrate the default flag behavior, the following listing shows a simple console application using the flag package.

Listing 2.1. Hello World CLI using flag package

$ flag_cli
Hello World!
$ flag_cli –s –name Buttercup
Hola Buttercup!
$ flag_cli –-spanish –name Buttercup
Hola Buttercup!

Each flag is separate from the rest and begins with either a - or --, as they’re interchangeable. A method is available for defining a flag with either a short or long name, but as you can see in the next listing, it’s an implicit method.

Listing 2.2. Source of Hello World using flag package: flag_cli.go

Here you can see two ways to define a flag. In the first, a variable can be created from a flag. In this example, it’s done using flag.String() . flag.String takes a flag name, default value, and description as arguments. The value of name is an address containing the value of the flag. To access this value, you’ll need to access name as a pointer .

The second method for handling a flag is the one that implicitly lets you have a long and short flag. Start by creating a normal variable of the same type as the flag. This will be used to store the value of a flag. Then use one of the flag functions that places the value of a flag into an existing variable . In this case, flag.BoolVar is used twice, once for the long name and once for the short name.

Tip

Flag functions exist for each variable type. To learn about each of them, see the flag package (http://golang.org/pkg/flag).

Finally, for the flag values to be in the variables, flag.Parse() needs to be run .

Tip

Command-line arguments aren’t registered with the flag package but are available via the Args or Arg function from the flag package.

The flag package doesn’t create help text for you but does help with the flags. This package has two handy functions you can use. The PrintDefaults function generates help text for flags. For example, the line of help text for the preceding name option reads as follows:

-name string
    A name to say hello to. (default "World")

This is a nicety in Go that makes it easier to keep your user informed about how your program works.

Flags also have a VisitAll function that accepts a callback function as an argument. This iterates over each of the flags executing the callback function on it and allows you to write your own help text for them. For example, the following listing would instead display -name: A name to say hello to. (Default: 'World').

Listing 2.3. Custom flag help text

flag.VisitAll(func(flag *flag.Flag) {
     format := "	-%s: %s (Default: '%s')
"
     fmt.Printf(format, flag.Name, flag.Usage, flag.DefValue)
})

Technique 1 GNU/UNIX-style command-line arguments

Although the built-in flag package is useful and provides the basics, it doesn’t provide flags in the manner most of us have come to expect. The difference in user interaction between the style used in Plan 9 and the style in Linux- and BSD-based systems is enough to cause users to stop and think. This is often due to problems they’ll encounter when trying to mix short flags, such as those used when executing a command like ls -la.

Problem

Those of us who write for non-Windows systems will likely be working with a UNIX variant, and our users will expect UNIX-style flag processing. How do you write Go command-line tools that meet users’ expectations? Ideally, you want to do this without writing one-off specialized flag processing.

Solution

This common problem has been solved in a couple of ways. Some applications, such as Docker, the software container management system, have a subpackage containing code to handle Linux-style flags. In some cases, these are forks of the Go flag package, in which the extra needs are added in. But maintaining per-project implementations of argument parsing results is an awful lot of duplicated effort to repeatedly solve the same generic problem.

The better approach is to use an existing library. You can import several standalone packages into your own application. Many are based on the flag package provided by the standard library and have compatible or similar interfaces. Importing one of these packages and using it is faster than altering the flag package and maintaining the difference.

Discussion

In the following examples, you’ll see two packages with slightly different approaches. The first attempts to keep API compatibility with the Go flag package, whereas the second breaks from it.

gnuflag

Similar to the flag package, the launchpad.net/gnuflag package brings GNU-style (Linux) flags to Go. Don’t let the name fool you. The license isn’t the GPL license, typical of GNU-based programs and libraries, but rather a BSD-style license in line with the Go license.

Several forms of flags are supported when using gnuflag, including these:

• -f for a single-letter or short flag
• -fg for a group of single-letter flags
• --flag for a multiletter or long flag name
• --flag x for a long flag with a value passed in as x
• -f x or -fx for a short flag with a passed-in value

The gnuflag package has almost the exact same API as the flag package, with one notable difference. The Parse function has an additional argument. Otherwise, it’s a drop-in replacement. The flag package will parse flags between the command name and the first nonflag argument. For example, in listing 2.4, the -s and -n flags are defined, and -n takes a value. When flag parses foo, it has reached a nonflag argument and stops parsing. That means -bar won’t be parsed to see if it’s a flag. The gnuflag package has an option to continue parsing that would allow it to find flags positioned where -bar is.

Listing 2.4. flag package flag parsing

$ flag_based_cli –s –n Buttercup foo -bar

To switch from the flag package to the gnuflag package, you need to change only two things in an application. First, instead of importing the flag package, import launchpad.net/gnuflag. Then, add an additional first argument to the Parse function of either true or false. A value of true looks for flags anywhere in the command, and false behaves like the flag package. That’s it.

Note

lanuchpad.net uses the Bazaar version-control system. You need to have this installed in order for Go to fetch the package. More details are available at http://bazaar.canonical.com/.

go-flags

Some community flag packages break from the conventions used in the flag package within the standard library. One such package is github.com/jessevdk/go-flags. It provides Linux- and BSD-style flags, providing even more features than those in gnuflag, but uses an entirely different API and style. Features include the following:

• Short flags, such as –f, and groups of short flags, such as -fg.
• Multiple-letter or long flags, such as --flag.
• Supporting option groups.
• Generating well-formed help documentation.
• Passing values for flags in multiple formats such as -p/usr/local, -p /usr /local, and -p=/usr/local.
• The same option can, optionally, appear more than once and be stored multiple times on a slice.

To illustrate using this flag package, the following listing shows our Spanish-capable Hello World console application rewritten using go-flags.

Listing 2.5. Using go-flags

The first big difference you’ll notice from previous techniques is how flags are defined. In this instance, they’re defined as properties on a struct . The property names provide access to the values in your application. Information about a flag, such as short name, long name, default value, and description, are gathered using reflection to parse the space-separated key-value pairs following a property on a struct. This key-value-pair parsing capability is provided by the reflect package in the standard library. For example, opts.Name is mapped to a flag that has a short name -n, a long name --name, a default value of World, and a description used for help text.

For the values to be available on the opts struct, the Parse function needs to be called with the opts struct passed in . After that, the properties on the struct can be called normally with flag values or their defaults available .

2.1.2. Command-line frameworks

Processing flags certainly isn’t the only important part of building a command-line application. Although Go developers learn as a matter of course how to begin with a main function and write a basic program, we often find ourselves writing the same set of features for each new command-line program we write. Fortunately, tools can provide a better entry point for creating command-line programs, and you’ll look at one of those in this section.

Technique 2 Avoiding CLI boilerplate code

You know the drill: You need a quick command-line tool for doing some light processing. You start from scratch and build that same old, bare-minimum boilerplate code. And it works. Unfortunately, it works well enough that you soon find yourself writing a more advanced version. This one has new options. But to add these new features, you refactor the boilerplate code into something just flexible enough to get the new job done. And the cycle repeats.

Problem

Repeatedly, we find ourselves writing the same kinds of command-line programs. At first, we think of them as disposable, but some of the tools we write end up growing far beyond our initial expectations. Why keep writing the same boilerplate, only to be faced later with refactoring the top level of the program? Are there tools that can provide a simple and repeatable pattern for rapidly building CLI programs?

Solution

If you want something more opinionated and full of features for building a console application, frameworks are available to handle command routing, help text, subcommands, and shell autocompletion, in addition to flags. A popular framework used to build console-based applications is cli (https://github.com/urfave/cli). This framework has been used by projects such as the open source platform-as-a-service (PaaS) project Cloud Foundry, the container management system Docker, and the continuous integration and deployment system Drone.

Discussion

Combining routing, flag and option parsing, and help documentation into a setup that’s self-documenting, cli.go is one of the easiest ways to get started building a console application that works the way you’d expect it to.

A Simple Console Application

Before looking at a console application with multiple commands, it’s useful to look at a console application that executes a single action. You’ll use this foundation in an expansion to multiple commands and subcommands. Figure 2.1 shows the structure that a single-action application can use from the command line.

Figure 2.1. Usage structure of a simple application

The following listing is a simple console application that displays Hello World! or says hello to a name you choose.

Listing 2.6. Hello World CLI: hello_cli.go

$ hello_cli
Hello World!
$ hello_cli --name Inigo
Hello Inigo!
$ hello_cli –n Inigo
Hello Inigo!
$ hello_cli –help
NAME:
   hello_cli - Print hello world


USAGE:
   hello_cli [global options] command [command options] [arguments...]

VERSION:
   0.0.0

COMMANDS:
   help, h   Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --name, -n 'World'      Who to say hello to.
   --help, -h              show help
   --version, -v           print the version

Using cli.go, the application-specific code needed to generate this application is only 26 lines long and can be seen in the following listing. The rest of the work and user interface is handled by cli.go.

Listing 2.7. Hello World CLI: hello_cli.go

After cli.go is imported, a new application is created using cli.NewApp. The returned variable is the basis for the application. Setting the Name and Usage properties is useful for documentation .

Tip

If the Name property is left unset, cli.go will attempt to determine the name by using the name of the application. If an application may be renamed, it’s useful to use the default behavior.

The Flags property is a slice of cli.Flag containing the global flags . Each flag has a long name, short name, or both. Some flags, such as a cli.StringFlag, have a Value property for the default value. cli.BoolFlag is an example of a flag without a default value. Its presence sets the value to true, and its absence sets the value to false. The Usage property is used for documentation that can be seen on the help screen.

cli.go provides the ability to have a default action when the application is run, to have commands, and to have nested subcommands. We cover commands and subcommands in the next section. Here you set the Action property that handles the default action . The value of Action is a function that accepts *cli.Context as an argument and returns an error. This function contains the application code that should be run. In this case, it obtains the name to say hello to by using c.GlobalString("name") before printing hello to the name. If you want the returned error to cause the application to exit with a nonzero exit code, return an error of cli.ExitError, which can be created using the cli.NewExitError function.

The final step is to run the application that was just set up. To do this, os.Args, the arguments passed into the application, are passed into the Run method on the application .

Tip

Arguments are available in the Action function. In this case, a string slice of arguments can be obtained by calling c.Args. For example, the first argument would be c.Args()[0].

Commands and Subcommands

Applications built with cli.go often have commands and subcommands, as illustrated in figure 2.2. Think of an application such as Git, in which you can run commands such as git add, git commit, and git push.

Figure 2.2. Usage structure of an application with commands and subcommands

To illustrate using commands, the following listing is a simple application with commands to count up and count down.

Listing 2.8. Count up and down: count_cli.go

Instead of defining the Action, a slice of cli.Command is defined on the Commands property with details about each command . Action, by default, will be set to show the help screen. That way, if no commands are used with the application, help will be displayed. Each command has a Name and Action along with optional Flags, Usage, and ShortName. Just as the base application can have a Commands property with a list of commands, each command can have a Commands property with a list of subcommands to that command.

Flags defined on a command are similar to those defined globally. The pattern of definition is the same with the Name and Usage, and in some cases, a default Value. Global options, as shown in figure 2.2, are available to every command and come before the command itself. Command-specific options, defined on the command , are placed after the command. Additionally, retrieving them is done from cli .Context by using a function such as String, Int, or Bool . Global flags are similar but have Global prepended to them.

We started off this section by showing how Go command-line programs differ from the UNIX norm. First we presented command-line flags. We introduced libraries that provide UNIX-style command-line flags. Then we introduced a higher-level issue: how do you quickly and repeatedly build command-line applications without rewriting the same old boilerplate code? You looked at a fantastic library for that. Now you’re ready to explore a topic relevant to almost all programs: configuration.

2.2. Handling configuration

A second foundational area that complements flag handling is persistent application configuration. Examples of this form of configuration include the files in the etc directory on some Linux installations and user-specific configuration files such as .gitconfig or .bashrc files.

Passing configuration to an application is a common need. Virtually every application, including console applications and servers, can benefit from persistent configuration. Operation toolchains such as Ansible, Chef, and Puppet have quite a bit of functionality for managing configuration in a distributed manner. How can configuration information be easily passed into and made available in your applications?

Whereas the preceding section talked about passing in configuration via command-line options, this section looks at passing in configuration via files, or in the case of 12-factor apps, using environment variables. In this section, we add support for common configuration formats, including JSON, YAML, and INI. Then we present the approach favored in the 12-factor pattern, which passes in configuration through environment variables.

Technique 3 Using configuration files

Command-line arguments, like those in the preceding section, are good for many things. But when it comes to one-time configuration of a program installed into a particular environment, command-line arguments aren’t the right fit. The most common solution is to store configuration data in a file that the program can load at startup.

In the next few sections, you’ll look at the three most popular configuration file formats and how to work with them in Go.

Problem

A program requires configuration above and beyond that which you can provide with a few command-line arguments. And it would be great to use a standard file format.

Solution

One of today’s most popular configuration file formats is JavaScript Object Notation (JSON). The Go standard library provides JSON parsing and handling. It should be no surprise that JSON configuration files are common.

Discussion

Consider the JSON file config.json with the following:

{
     "enabled": true,
     "path": "/usr/local"
}

The following listing showcases a JSON configuration parser.

Listing 2.9. Parsing a JSON configuration file: json_config.go

This method has only a few parts. First, a type or collection of types needs to be created representing the JSON file . The names and nesting must be defined and map to the structure in the JSON file. The main function begins by opening the configuration file and decoding the JSON file into an instance of the configuration struct . If no errors occur, the values from the JSON file won’t be available on the conf variable and can be used in your application.

Note

JSON parsing and handling has many features and nuances. Chapter 6, which covers working with JSON APIs, presents the JSON features in more detail.

Storing configuration in JSON is useful if you’re already familiar with JSON, using a distributed configuration store such as etcd, or want to stick with the Go standard library. JSON files can’t contain comments, which can be a common complaint when creating configuration files or generating examples of them. In the following two techniques, comments are available.

Solution

YAML, a recursive acronym meaning YAML Ain’t Markup Language, is a human-readable data serialization format. YAML is easy to read, can contain comments, and is fairly easy to work with. Using YAML for application configuration is common and a method we, the authors, recommend and practice. Although Go doesn’t ship with a YAML processor, several third-party libraries are readily available. You’ll look at one here.

Discussion

Consider this simple YAML configuration file:

# A comment line
enabled: true
path: /usr/local

The following listing is an example of parsing and printing configuration from a YAML file.

Listing 2.10. Parsing a YAML configuration file: yaml_config.go

To work with YAML files and content, this listing imports the github.com/kylelemons/go-gypsy/yaml package . This package, which we’ve used and recommend, provides features to read YAML as a string or from a file , deal with different types, and turn configuration into YAML output.

Using the function ReadFile, the configuration file is read in and returns a File struct from the yaml package. This struct provides access to the data in the YAML file. Using the Get method, the value of a string can be obtained . For other types, such as a Boolean, you can use methods such as GetBool. Each type has its own method to ensure proper handling and return type.

Solution

INI files are a format in wide use and have been around for decades. This is another format your Go applications can potentially use. Although the Go developers didn’t include a processor in the language, once again libraries are readily available to meet your needs.

Discussion

Consider the following INI file:

; A comment line
[Section]
enabled = true
path = /usr/local # another comment

In the following listing, this file is parsed, and you can see how to use the internal data.

Listing 2.11. Parsing an INI configuration file: ini_config.go

In this case, the third-party package gopkg.in/gcfg.v1 handles parsing the INI file into a data structure . This package provides a means to parse INI files and strings similar to JSON handling in the standard library.

Before the INI file can be parsed, a variable needs to exist to receive the values from the file. As in the JSON technique, the structure of this variable needs to map to the structure in the INI file. In this case, a struct is created, though a new type similar to the JSON example could be used instead, with an internal struct for a section . This struct is where the configuration values will reside after parsing.

The function ReadFileInto reads the file into the struct that was created . If an error occurs, it will be available. After this has passed, the configuration from the INI file is available to be used .

The gopkg.in/gcfg.v1 package has several useful features such as tags and reading strings, files, or anything implementing the io.Reader interface. For more information, see the package documentation.

Technique 4 Configuration via environment variables

The venerable configuration file certainly provides a great vehicle for passing configuration data to programs. But some of today’s emerging environments defy some of the assumptions we make when using traditional configuration files. Sometimes the one configuring the application won’t have access to the filesystem at the level we assume. Some systems treat configuration files as part of the source codebase (and thus as static pieces of an executable). This removes some of the utility of configuration files.

Nowhere is this trend clearer than in the emerging PaaS cloud services. Deploying into these systems is usually accomplished by pushing a source-code bundle to a control server (like a Git push). But the only runtime configuration you get on such servers is done with environment variables. Let’s take a look at a technique for working in such an environment.

Problem

Many PaaS systems don’t provide a way to specify per-instance configuration files. Configuration opportunities are limited to a small number of environmental controls, such as environment variables.

Solution

Twelve-factor apps, commonly deployed to Heroku, Cloud Foundry, Deis, and other PaaS or container cluster managers (covered in chapter 11), are becoming more common. One of the factors of a twelve-factor app is storing the configuration in the environment. This provides a way to have a different configuration for each environment an application runs in.

Discussion

Consider, for example, the environment variable PORT containing the port a web server should listen to. The following listing retrieves this piece of configuration and uses it when starting a web server.

Listing 2.12. Environment variable–based configuration: env_config.go

This example uses the http package from the standard library. You may remember it from the simple Hello World web server in listing 1.16. In the following section, we cover more on web servers.

Retrieving configuration from the environment is fairly straightforward. From the os package, the Getenv function retrieves the value as a string . When no environment variable is found, an empty string is returned. If you need to convert the string to another type, you can use the strconv package. For example, if the PORT in this example needed to be an integer, you could use the ParseInt function.

Warning

Be careful with the information in environment variables and the processes able to obtain the information in them. For example, a third-party subprocess started by your application could have access to the environment variables.

2.3. Working with real-world web servers

Although the Go standard library provides a great foundation for building web servers, it has some options you may want to change and some tolerance you may want to add. Two common areas, which we cover in this section, are matching URL paths to callback functions and starting and stopping servers with an interest in gracefully shutting down.

Web servers are a core feature of the http package. This package uses the foundation for handling TCP connections from the net package. Because web servers are a core part of the standard library and are commonly in use, simple web servers were introduced in chapter 1. This section moves beyond base web servers and covers some practical gotchas that come up when building applications. For more information on the http package, visit http://golang.org/pkg/net/http/.

2.3.1. Starting up and shutting down a server

Starting a server in Go is fairly easy. Using the net or http packages, you can create a server, listen on a TCP port, and start responding to incoming connections and requests. But what happens when you want to shut down that server? What if you shut down the server while users are connected, or before all the data (such as logs or user information) has been written to disk?

The commands used to start and stop a server in the operating system should be handled by an initialization daemon. Using go run on a codebase is handy in development and may be used with some systems based on twelve-factor apps but isn’t typical or recommended. Manually starting an application is simple but isn’t designed to integrate nicely with operations tools or handle problems such as an unexpected system restart. Initialization daemons were designed for these cases and do them well.

Most systems have a default toolchain used for initialization. For example, systemd (https://freedesktop.org/wiki/Software/systemd/) is common on Linux distributions such as Debian and Ubuntu systems. If a script is used with systemd, you’ll be able to use commands like those in the following listing.

Listing 2.13. Start and stop applications with upstart

A wide variety of initialization daemons are available. They vary depending on your flavor of operating system, and numerous ones exist for the various versions of Linux. You may be familiar with some of their names, including upstart, init, and launchd. Because configuration scripts and commands can vary widely among these systems, we don’t cover them here. These tools are well documented, and many tutorials and examples are available.

Note

We recommend that you don’t write your applications as daemons but rather use an initialization daemon to manage the execution of the application.

A common antipattern: a callback URL

A simple pattern (or rather antipattern) for development is to have a URL such as /kill or /shutdown, that will shut down the server when called. The following listing showcases a simple version of this method.

Listing 2.14. Callback shutdown URL: callback_shutdown.go

We don’t recommend this method, but we share it because it’s a pattern easily found on the internet. Although its simplicity is an advantage, the list of disadvantages is problematic and includes the following:

• The URL needs to be blocked in production, or more likely, removed before going to production. Needing to have code differences for development and production is prone to introducing bugs. If this URL were left in for production, anyone who discovered it could easily take down your service.
• When the callback URL receives a request, the server shuts down immediately. Any actions in progress are immediately stopped. Any data not saved to disk is lost because there’s no opportunity to save it before exiting.
• Using a URL sidesteps typical operations tooling such as Ansible, Chef, and Puppet or initialization toolchains. More-appropriate tools exist that manage updates and running applications.

We don’t recommend using this method. It’s most useful for development when the process is in the background or being run as a daemon. Go applications typically shouldn’t be daemons, and better methods are available for starting and stopping the server, even for development.

Technique 5 Graceful shutdowns using manners

When a server shuts down, you’ll often want to stop receiving new requests, save any data to disk, and cleanly end connections with existing open connections. The http package in the standard library shuts down immediately and doesn’t provide an opportunity to handle any of these situations. In the worst cases, this results in lost or corrupted data.

Problem

To avoid data loss and unexpected behavior, a server may need to do some cleanup on shutdown.

Solution

To handle these, you’ll need to implement your own logic or use a package such as github.com/braintree/manners.

Discussion

Braintree, a division of PayPal, created the manners package that gracefully shuts down while maintaining the same interface for ListenAndServe that the core http package uses. Internally, the package uses the core http server while keeping track of connections by using WaitGroup from the sync package. WaitGroup is designed to keep track of goroutines. The following listing takes a look at a simple manners-based server.

Listing 2.15. Graceful shutdown using manners: manners_shutdown.go

The main function begins by getting an instance of a handler function capable of responding to web requests . This handler is a simple Hello World responder . In its place, a more complex one handling routing rules, such as the path or regular expression handlers covered later in the chapter, could be used.

To gracefully shut down, you need to know when to do so. The signal package provides a means to get signals from the operating system, including signals to interrupt or kill the application. The next step is to set up a channel that receives interrupt and kill signals from the operating system so the code can react to them . Listen-AndServe, like its counterpart in the http package, blocks execution. To monitor signals, a goroutine needs to run concurrently. The function listenForShutdown waits until it receives a signal on the channel . After a signal comes in, it sends a message to Shutdown on the server. This tells the server to stop accepting new connections and shut down after all the current requests are completed.

Calling ListenAndServe in the same manner as the http package starts the server.

Tip

The server waits only for request handlers to finish before exiting. If your code has separate goroutines that need to be waited on, that would need to happen separately, using your own implementation of WaitGroup.

This approach has several advantages, including the following:

• Allows current HTTP requests to complete rather than stopping them mid-request.
• Stops listening on the TCP port while completing the existing requests. This opens the opportunity for another application to bind to the same port and start serving requests. If you’re updating versions of an application, one version could shut down while completing its requests, and another version of the application could come online and start serving.

A couple of disadvantages also exist under some conditions:

• The manners package works for HTTP connections rather than all TCP connections. If your application isn’t a web server, the manners package won’t work.
• In some cases, one version of an application will want to hand off exiting socket connections currently in use to another instance of the same application or another application. For example, if you have long-running socket connections between a server and client applications, the manners package will attempt to wait or interrupt the connections rather than hand them off.

2.3.2. Routing web requests

One of the fundamental tasks of any HTTP server is to receive a given request and map it to an internal function that can then return a result to the client. This routing of a request to a handler is important; do it well, and you can build web services that are easily maintainable and flexible enough to fit future needs. This section presents various routing scenarios and solutions for each.

We start with simple scenarios and provide simple solutions. But we encourage you to plan ahead. The simple solution we talk about first is great for direct mappings, but may not provide the flexibility that a contemporary web application needs.

Technique 6 Matching paths to content

Web applications and servers providing a REST API typically execute different functionality for different paths. Figure 2.3 illustrates the path portion of the URL compared to the other components. In the Hello World example from chapter 1, listing 1.16 uses a single function to handle all possible paths. For a simple Hello World application, this works. But a single function doesn’t handle multiple paths well and doesn’t scale to real-world applications. This section covers multiple techniques to handle differentiating paths and, in some cases, different HTTP methods (sometimes referred to as verbs).

Figure 2.3. The path portion of the URL used in routing requests

Problem

To correctly route requests, a web server needs to be able to quickly and efficiently parse the path portion of a URL.

Solution: multiple handlers

To expand on the method used in listing 1.16, this technique uses a handler function for each path. This technique, presented in the guide “Writing Web Applications” (http://golang.org/doc/articles/wiki/), uses a simple pattern that can be great for web apps with only a few simple paths. This technique has nuances that you’ll see in a moment that may make you consider one of the techniques following it.

Discussion

Let’s start with a simple program that illustrates using multiple handlers, shown in the following listing.

Listing 2.16. Multiple handler functions: multiple_handlers.go

Note

Content collected from an end user should be sanitized before using. That includes displaying the content back to a user. This functionality is part of the templating Go package covered in chapter 5.

Here you use three handler functions for three paths or path parts . When a path is resolved, it tries to go from the most specific to the least specific. In this case, any path that isn’t resolved prior to the / path will resolve to this one.

It’s worth noting that paths ending in / can have redirection issues. In this listing, a user who visits /goodbye will be automatically redirected to /goodbye/. If you have query strings, they may be dropped. For example, /goodbye?foo=bar will redirect to /goodbye/.

The way resolution works by default is important to know as well. The handler registered to /hello will work only for /hello. The handler registered to /goodbye/ will be executed for /goodbye (with a redirect), /goodbye/, /goodbye/foo, /goodbye/foo/bar, and so on.

The handler function hello is mapped to the path /hello . As arguments, the handler functions receive an http.ResponseWriter and an http.Request. Optionally, a name to say hello to can be in a query string with a key of name . The requested URL is a property on http.Request as url.URL. The Query method on the URL returns either the value for the key, or an empty string if no value is available for the key. If the value is empty here, it’s set to Inigo Montoya.

Tip

The net/url package, which contains the URL type, has many useful functions for working with URLs.

Note

To differentiate between HTTP methods, check the value of http.Request.Method. This contains the method (for example, GET, POST, and so on).

The goodbye function handles the path /goodbye/, including the case where additional text is appended . In this case, a name can be optionally passed in through the path. For example, a path of /goodbye/Buttercup will set the name to Buttercup . To achieve this, the URL is split by using the strings package to find the part of the path following /goodbye/.

The homePage function handles both the / path and any case where a page isn’t found . To decide whether to return a 404 Page Not Found message or home page content, the http.Request.Path needs to be checked . The http package contains a NotFound helper function that can optionally be used to set the response HTTP code to 404 and send the text 404 page not found.

Tip

The http package contains the Error function that can be used to set the HTTP error code and respond with a message. The NotFound function takes advantage of this for the 404 case.

Using multiple function handlers is the core way to handle different functions alongside different paths. The pros of this method include the following:

• As the basic method in the http package, it’s well documented and tested, and examples are right at hand.
• The paths and their mappings to functions are easy to read and follow.

Alongside the pros are some cons that lead many, including the authors, to use other methods. The cons are as follows:

• You can’t use different functions for different HTTP methods on the same path. When creating REST APIs, the verb (for example, GET, POST, or DELETE) can require significantly different functionality.
• Wildcard or named sections to a path, a common feature or feature request for mapping systems, aren’t available.
• Virtually every handler function needs to check for paths outside their bounds and handle returning a Page Not Found message. For example, in listing 2.16 the handler /goodbye/ will receive paths prepended with /goodbye. Anything returned by any path is handled here, so if you want to return a Page Not Found message for the path /goodbye/foo/bar/baz, that would need to be handled here.

Using multiple handlers is useful for simple cases. Because it doesn’t require packages outside the http package, the external dependencies are kept to a minimum. If an application is going to move beyond simple use cases, one of the following techniques is likely to be a better fit.

Technique 7 Handling complex paths with wildcards

The previous technique is straightforward, but as you saw, it’s decidedly inflexible when it comes to path naming. You must list every single path that you expect to see. For larger applications or for applications that follow the REST recommendations, you need a more flexible solution.

Problem

Instead of specifying exact paths for each callback, an application may need to support wildcards or other simple patterns.

Solution

Go provides the path package with functionality to work with slash-separated paths. This package isn’t directly designed to work with URL paths. Instead, it’s a generic package intended to work with paths of all sorts. In fact, it works well when coupled with an HTTP handler.

Discussion

The following listing builds a router that uses path matching to map URL paths and HTTP methods to a handler function.

Listing 2.17. Resolve URLs using path package: path_handlers.go

The main function starts off quite differently by getting an instance of a pathResolver . The pathResolver, which you’ll look at in a moment, contains the core logic for matching functions to paths. After an instance of the pathResolver has been created, two mappings of HTTP verbs and their paths are added to the resolver . The format for these is the HTTP method name followed by a path, with a space separating the two. You can use an asterisk (*) as a wildcard character for the HTTP method or in the path.

The pathResolver is set as the handler function for the built-in HTTP server when the server is started . For pathResolver to work as a handler function, it needs to implement the ServeHTTP method and implicitly implement the HandlerFunc interface. The ServeHTTP method is where path resolving happens.

When a request comes into the server, the ServeHTTP method iterates over the paths registered with the pathResolver . For each path, it checks the current HTTP method and path to see if a function is mapped to the combination . This is a check you construct because with REST servers you’ll often need to handle different HTTP methods (for example, a DELETE or GET request) with entirely different functions. When a match is found, the handler function registered for that case is executed . If no matched paths are found in the lookup, you default to a 404 Page Not Found error .

You should be aware of the pros and cons of path resolution using the path package. Here are the pros:

• Easy to get started with simple path matching.
• Included in the standard library, the path package is well traveled and tested.

The cons have a common thread in that the path package is generic to paths and not specific to URL paths. The cons are as follows:

• The wildcard abilities of the path package are limited. For example, a path of foo/* will match foo/bar but not foo/bar/baz. Using * for a wildcard stops at the next /. To match foo/bar/baz, you’d need to look for a path like foo/*/*.
• Because this is a generic path package rather than one specific to URLs, some nice-to-have features are missing. For example, in listing 2.17 the path /goodbye/* is registered. Visiting the path /goodbye in a browser will display a Page Not Found message, whereas visiting /goodbye/ works. Although there’s a technical path difference (the trailing /), the common web use case isn’t transparently handled. You’ll need to identify and handle cases such as this one.

This method is useful for simple path scenarios and it’s one that we, the authors, have successfully used.

Technique 8 URL pattern matching

For most REST-style apps, simple pattern matching with regular expressions is more than sufficient. But what if you want to go beyond that and do something fancy with your URLs? The path package isn’t well suited for this because it supports only simple POSIX-style pattern matching.

Problem

Simple path-based matching isn’t enough for an application that needs to treat a path more like a text string and less like a file path. This is particularly important when matching across a path separator (/).

Solution

The built-in path package enables simple path-matching schemes, but sometimes you may need to match complex paths or have intimate control over the path. For those cases, you can use regular expressions to match your paths. You’ll combine Go’s built-in regular expressions with the HTTP handler and build a fast but flexible URL path matcher.

Discussion

In the next listing you’ll walk through using paths and a resolver based on regular expressions.

Listing 2.18. Resolve URLs using regular expressions: regex_handlers.go

The layout of the regular-expression-based path resolution (listing 2.18) is the same as the path resolution example (listing 2.17). The differences lie in the format of the path patterns registered for a function and in the ServeHTTP method handling the resolution.

Paths are registered as regular expressions . The structure is the same as the path package technique, with an HTTP method followed by the path, separated by a space. Whereas GET /hello showcases a simple path, a more complicated example is (GET|HEAD) /goodbye(/?[A-Za-z0-9]*)?. This more complicated example accepts either a GET or HEAD HTTP method. The regular expression for the path will accept /goodbye, /goodbye/ (the trailing / matters), and /goodbye/ followed by letters and numbers.

In this case, ServeHTTP iterates over the regular expressions looking for a match . When the first match is found, it will execute the handler function registered to that regular expression. If more than one regular expression matches an incoming path, the first one added would be the first one checked and used.

Note

Compiled versions of the regular expressions are built and cached at the time they’re added. Go provides a Match function in the regexp package that can check for matches. The first step for this function is to compile the regular expression. By compiling and caching the regular expression, you don’t need to recompile the regular expressions each time the server handles a request.

Using regular-expression checking for paths provides a significant amount of power, allowing you to finely tune the paths you want to match. This flexibility is paired with the complicated nature of regular expressions that may not be easy to read, and you’ll likely want to have tests to make sure your regular expressions are matching the proper paths.

Technique 9 Faster routing (without the work)

One criticism of Go’s built-in http package is that its routing and multiplexing (muxing) is basic. In the previous sections, we showed some straightforward ways of working with the http package, but depending on your needs, you may not be satisfied with the configurability, performance, or capabilities of the built-in HTTP server. Or you may just want to avoid writing boilerplate routing code.

Problem

The built-in http package isn’t flexible enough, or doesn’t perform well in a particular use case.

Solution

Routing URLs to functions is a common problem for web applications. Therefore, numerous packages have been built and tested, and are commonly used to tackle the problem of routing. A common technique is to import an existing request router and use it within your application.

Popular solutions include the following:

• github.com/julienschmidt/httprouter is considered a fast routing package with a focus on using a minimal amount of memory and taking as little time as possible to handle routing. It has features such as the ability to have case-insensitive paths, cleaning up /../ in a path, and dealing with an optional trailing /.
• github.com/gorilla/mux is part of the Gorilla web toolkit. This loose collection of packages provides components you can use in an application. The mux package provides a versatile set of criteria to perform matching against, including host, schemes, HTTP headers, and more.
• github.com/bmizerany/pat provides a router inspired by the routing in Sinatra. The registered paths are easy to read and can contain named parameters such as /user/:name. It has inspired other packages such as github.com/gorilla/pat.

Each package has a different feature set and API. Numerous other routing packages exist as well. With a little investigation, you can easily find a quality third-party package that meets your needs.

2.4. Summary

After the foundational elements for an application are decided on and in place, it’s easier to dive into application-specific situations and the elements that will make your application useful. This chapter covered several foundational elements and options:

• Handling command-line options in a comfortable and accessible manner. This ranges from lightweight solutions to a simple framework for building console-based applications and utilities.
• Retrieving configuration information from files and the environment in various ways and data formats.
• Starting and stopping a web server that works with ops tooling and graceful shutdowns to avoid a bad user experience or loss of data.
• Several ways to handle resolving URL paths for an application, and route to handler functions.

In the next chapter, you’ll look at concurrency in Go. Concurrency is a cornerstone and building block of Go applications. You’ll learn how to effectively use it.