This chapter will equip you to use the Go HTTP client to talk to other systems over the internet.
You will start by learning to use the HTTP client to get data from a web server and to send data to a web server. By the end of the chapter, you will be able to upload a file to a web server and experiment with a custom Go HTTP client to interact with web servers.
Introduction
In the previous chapter, you looked at SQL and databases. You learned how to execute queries, how to create tables, how to insert data into tables and fetch data, how to update data, and how to delete data within a table.
In this chapter, you will learn about the Go HTTP client and how to use it. An HTTP client is something that is used to get data from or send data to a web server. Probably the most well-known example of an HTTP client is a web browser (such as Firefox). When you enter a web address into a web browser, it will have an HTTP client built in that sends a request to the server for data. The server will gather the data and send it back to the HTTP client, which will then display the web page in the browser. Similarly, when you fill out a form in a web browser, for example, when you log in to a website, the browser will use its HTTP client to send that form data to the server and then take appropriate action depending on the response.
This chapter looks at how you can use the Go HTTP client to request data from a web server and send data to a server. You will examine the different ways you can use the HTTP client to interact with a web server and the various use cases for those interactions. The web browser example will be useful in explaining the different interactions. As part of this chapter, you will create your own Go programs that make use of the Go HTTP client to send and receive data from a web server.
The Go HTTP Client and Its Uses
The Go HTTP client is part of the Go standard library, specifically the net/http library. There are two main ways to use it. The first is to use the default HTTP client that is included in the net/http library. It's simple to use and allows you to get up and running quickly. The second way is to create your own HTTP client based on the default HTTP client. This allows you to customize the requests and various other things. It takes longer to configure, but it gives you much more freedom and control over the requests you send.
When using an HTTP client, you can send different types of requests. While there are many types of requests, we will discuss the two main ones, the GET request and the POST request. For instance, if you wanted to retrieve data from a server, you would send a GET request. When you enter a web address in your web browser, it will send a GET request to the server at that address and then display the data it returns. If you wanted to send data to the server, you would send a POST request. If you wanted to log into a website, you would POST your login details to the server.
In this chapter, there are a few exercises to teach you about the Go HTTP client. They will teach you how to request data from a server in various formats using GET requests. They will also teach you how to POST form data to a web server, similar to how a web browser would send a POST request when you log in to a website. These exercises will also show you how to upload a file to a web server and how to use a customized HTTP client to have more control over the requests you send.
Sending a Request to a Server
When you want to retrieve data from a web server, you send a GET request to the server. When sending a request, the URL will contain the information on the resource you want data from. The URL can be broken down into a few key parts. These include the protocol, the hostname, the URI, and the query parameters. The format of it looks like this:
Figure 14.1: URL format breakdown
In this example:
- The Protocol tells the client how to connect to the server. The two most common protocols are HTTP and HTTPS. In this example, we have used https.
- The Hostname is the address of the server we want to connect to. In this example, it is example.com.
- The URI is the Uniform Resource Identifier (URI), and this tells the server the path to the resource we want. In this example, it is /downloads.
- The Query Parameters tell the server of any additional information it needs. In this example, we have two parameters. These are filter=latest and os=windows. You will notice they are separated from the URI by ?. This is so the server can parse them from the request. We join any additional parameters to the end of the URI with the & symbol, as seen with the os parameter.
Exercise 14.01: Sending a Get Request to a Web Server Using the Go HTTP Client
In this exercise, you will be getting data from a web server and printing out that data. You will send a GET request to https://www.google.com and display the data the web server returns:
Note
For this topic, you will need to have Go installed and GOPATH set up on your system. You will also need an IDE that you can use to edit .go files.
- Open your IDE and create a new directory, Exercise14.01, on your GOPATH. Within that directory, create a new Go file called main.go.
- As this is a new program, you will want to set the package of the file to the main() function. Import the net/http library, the log library, and the io/ioutil library. Type the following code:
package main
import (
"io/ioutil"
"log"
"net/http"
Now that you have the package set up and the imports you need, you can start creating a function to get data from a web server. The function you are going to create will request data from a web server.
- Create a function that returns a string:
func getDataAndReturnResponse() string {
- Within that function, you can then use the default Go HTTP Client to request data from a server. In this exercise, you will request data from https://www.google.com. To request data from the web server, you use the GET function in the http library, which looks as follows:
r, err := http.Get("https://www.google.com")
if err != nil {
log.Fatal(err)
}
- The data the server sends back is contained within r.Body, so you just to read in that data. To read the data within r.Body, you can use the ReadAll function within the io/ioutil library. The two together would look like this:
defer r.Body.Close()
data, err := ioutil.ReadAll(r.Body)
if err != nil {
log.Fatal(err)
}
- After you have received the response from the server and read the data, you just need to return that data as a string, which looks like this:
return string(data)
}
The function you have now created will now look like this:
func getDataAndReturnResponse() string {
// send the GET request
r, err := http.Get("https://www.google.com")
if err != nil {
log.Fatal(err)
}
// get data from the response body
defer r.Body.Close()
data, err := ioutil.ReadAll(r.Body)
if err != nil {
log.Fatal(err)
}
// return the response data
return string(data)
}
- Create a main function. Within the main function, call the getDataAndReturnResponse function and log the string it returns:
func main() {
data := getDataAndReturnResponse()
log.Println(data)
}
- To run the program, open your terminal and navigate to the directory that you created the main.go file in.
- Run go run main.go to compile and execute the file:
go run server.go
The program will issue a GET request to https://www.google.com and log the response in your terminal.
While it may look like gibberish, if you were to save that data to a file called response.html and open it in your web browser, it would resemble the Google home page. This is what your web browser will do under the hood when you open a web page. It will send a GET request to the server and then display the data it returns. If we do this manually, it will look as follows:
Figure 14.2: Request HTML response when viewed in Firefox
In this exercise, we saw how to send a GET request to a web server and get data back. You created a Go program that sent a request to https://www.google.com and got back the HTML data for the Google home page.
Structured Data
Once you have requested data from a server, the data returned can come in various formats. For example, if you send a request to packtpub.com, it will return HTML data for the Packt website. While HTML data is useful for displaying websites, it isn't ideal for sending machine-readable data. A common data type used in web APIs is JSON. JSON provides a good structure for data that is both machine-readable and human-readable. Later, you will learn how to parse JSON and make use of it using Go.
Exercise 14.02: Using the HTTP Client with Structured Data
In this exercise, you will parse structured JSON data in Go. The server will return JSON data and you will use the json.Unmarshal function to parse the data and put it into a struct:
- Create a new directory, Exercise14.02, on your GOPATH. Within that directory, create two more directories, server and client. Then, within the server directory, create a file called server.go and write the following code:
package main
import (
"log"
"net/http"
)
type server struct{}
func (srv server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
msg := "{"message": "hello world"}"
w.Write([]byte(msg))
}
func main() {
log.Fatal(http.ListenAndServe(":8080", server{}))
}
This creates a very basic web server that sends back JSON data. We will explain in more detail how this works in the next chapter. For now, we will just use it as an example.
- Once you have created the server, navigate to the client directory and create a file called main.go. Add package main and import the packages needed for the file:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
)
- Then, create a struct with a string parameter that can accept the response from the server. Then, add JSON metadata to it so it can be used to unmarshal the JSON message parameter:
type messageData struct {
Message string `json:"message"`
}
- Next, create a function that you can call to get and parse the data from the server. Use the struct you just created as the return value:
func getDataAndReturnResponse() messageData {
When you run the web server, it will listen on http://localhost:8080. So, you need to send a GET request to that URL and then read the response body:
r, err := http.Get("http://localhost:8080")
if err != nil {
log.Fatal(err)
}
defer r.Body.Close()
data, err := ioutil.ReadAll(r.Body)
if err != nil {
log.Fatal(err)
}
- This time, however, you will parse the response instead of simply returning it. To do that, you create an instance of the struct you created, then pass it along with the response data to json.Unmarshal:
message := messageData{}
err = json.Unmarshal(data, &message)
if err != nil {
log.Fatal(err)
}
This will populate the message variable with the data returned from the server.
- You then need to return the struct to complete the function:
return message
- Finally, call the function you just created from the main() function and log the message from the server:
func main() {
data := getDataAndReturnResponse()
fmt.Println(data.Message)
}
- To run this, you need to do two steps. The first is navigate to the server directory in your terminal and run the following command. This will start the web server:
go run server.go
- In a second terminal window, navigate to the client directory and run go run main.go. This will start the client and connect to the server. It should output the message from the server:
Figure 14.3: Expected output
In this exercise, you sent a GET request to the server and got back structured data in JSON format. You then parsed that JSON data to get the message from it.
Activity 14.01: Requesting Data from a Web Server and Processing the Response
Imagine you are interacting with a web API. You send a GET request for data and get back an array of names. You need to count those names to find out how many of each you have. In this activity, you will do just that. You will send a GET request to the server, get back structured JSON data, parse the data, and count how many of each name you got back in the response:
- Create a directory called Activity14.01.
- Create two sub-directories, one called client and another called server.
- In the server directory, create a file called server.go.
- Add the server code in server.go.
- Start the server by calling go run server.go in the server directory.
- In the client directory, create a file called main.go.
- In main.go, add the necessary imports.
- Create structs to parse the response data.
- Create a function called getDataAndParseResponse that returns two integers.
- Send a GET request to the server.
- Parse the response into a struct.
- Loop through the struct and count the occurrences of the names Electric and Boogaloo.
- Return the counts.
- Print the counts.
The expected output is as follows:
Figure 14.4: Possible output
Note
The solution for this activity can be found on page 752.
In this activity, we have requested data from a web server and processed the data it returned using the Go HTTP client.
Sending Data to a Server
In addition to requesting data from a server, you will also want to send data to a server. The most common way of doing this is via a POST request. A POST request comes in two main parts: the URL and the body. The body of a POST request is where you put the data you want to send to the server. A common example of this is a login form. When we send a login request, we POST the body to the URL. The web server then checks that the login details within the body are correct and updates our login status. It responds to the request by telling the client whether it succeeded or not. In this chapter, you will learn how to send data to a server using a POST request.
Exercise 14.03: Sending a Post Request to a Web Server Using the Go HTTP Client
In this exercise, you will send a POST request to a web server containing a message. The web server will then respond with the same message so you can confirm that it received it:
- Create a new directory, Exercise14.03, on your GOPATH. Within that directory, create two more directories, server and client. Then, within the server directory, create a file called server.go and write the following code:
package main
import (
"encoding/json"
"log"
"net/http"
)
type server struct{}
type messageData struct {
Message string `json:"message"`
}
func (srv server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
jsonDecoder := json.NewDecoder(r.Body)
messageData := messageData{}
err := jsonDecoder.Decode(&messageData)
if err != nil {
log.Fatal(err)
}
jsonBytes, _ := json.Marshal(messageData)
log.Println(string(jsonBytes))
w.Write(jsonBytes)
}
func main() {
log.Fatal(http.ListenAndServe(":8080", server{}))
}
This creates a very basic web server that receives a JSON POST request and returns the message sent to it back to the client.
- Once you have the server created. Navigate to the client directory and create a file called main.go. Add package main and the imports needed for the file:
package main
import (
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
)
- Next, you need to create a struct for the data we want to send and receive. This will be the same as the struct used by the server to parse the request:
type messageData struct {
Message string `json:"message"`
}
- You then need to create the function to POST the data to the server. It should accept a messageData struct parameter as well as return a messageData struct:
func postDataAndReturnResponse(msg messageData) messageData {
- To POST the data to the server, you need to marshal the struct into bytes that the client can send to the server. To do this, you can use the json.Marshal function:
jsonBytes, _ := json.Marshal(msg)
- Now that you have the bytes, you can use the http.Post function to send the POST request. Within the request, you just need to tell the function what URL to post to, what kind of data you are sending, and the data you want to send. In this case, the URL is http://localhost:8080. The content you are sending is application/json and the data is the jsonBytes variable you just created. Together, it looks like this:
r, err := http.Post("http://localhost:8080", "application/json", bytes.NewBuffer(jsonBytes))
if err != nil {
log.Fatal(err)
}
- After that, the rest of the function is the same as in the previous exercise. You read the response, parse out the data, and then return the data, which looks like this:
defer r.Body.Close()
data, err := ioutil.ReadAll(r.Body)
if err != nil {
log.Fatal(err)
}
message := messageData{}
err = json.Unmarshal(data, &message)
if err != nil {
log.Fatal(err)
}
return message
- Then, you just need to call the postDataAndReturnResponse function from your main function. This time, however, you need to pass the message you want to send to the function. You just need to create an instance of the messageData struct and pass that to the function when you call it, which looks like this:
func main() {
msg := messageData{Message: "Hi Server!"}
data := postDataAndReturnResponse(msg)
fmt.Println(data.Message)
}
- To run this exercise, you need to carry out two steps. The first is to navigate to the server directory in your terminal and run go run server.go. This will start the web server. In a second terminal window, navigate to the client directory and run go run main.go. This will start the client and connect to the server. It should output the message from the server:
Figure 14.5: Expected output
In this exercise, you sent a POST request to the server. The server parsed the request and sent the same message back to you. If you change the message sent to the server, you should see the response from the server sending back the new message.
Uploading Files in a Post Request
Another common example of data you might want to POST to a web server is a file from your local computer. This is how websites allow users to upload their photos and so on. As you can imagine, this is a little more complex than sending simple form data. To achieve this, the file needs to be read first, then wrapped in a format that the server can understand. It can then be sent in a POST request to the server in what's called a multipart form. You will learn how to read in a file and upload it to a server using Go.
Exercise 14.04: Uploading a File to a Web Server via a Post Request
In this exercise, you will read in a local file and then upload it to a web server. You can then check that the web server saved the file you uploaded:
- Create a new directory, Exercise14.04, on your GOPATH. Within that directory, create two more directories, server and client. Then, within the server directory, create a file called server.go and write the following code:
server.go
9 func (srv server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
10 uploadedFile, uploadedFileHeader, err := r.FormFile("myFile")
11 if err != nil {
12 log.Fatal(err)
13 }
14 defer uploadedFile.Close()
15 fileContent, err := ioutil.ReadAll(uploadedFile)
16 if err != nil {
17 log.Fatal(err)
18 }
The full code for this step is available at: https://packt.live/2SkeZHW
This creates a very basic web server that receives a multipart form POST request and saves the file within the form.
- Once you have created the server, navigate to the client directory and create a file called main.go. Add package main and the imports needed for the file:
package main
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"log"
"mime/multipart"
"net/http"
"os"
)
- You then need to create a function to call that you will give a filename to. The function will read in the file, upload it to the server, and return the server's response:
func postFileAndReturnResponse(filename string) string {
- You need to create a buffer that you can write the file bytes to, then create a writer to allow bytes to write into it:
fileDataBuffer := bytes.Buffer{}
multipartWritter := multipart.NewWriter(&fileDataBuffer)
- Open the file from your local computer using the following command:
file, err := os.Open(filename)
if err != nil {
log.Fatal(err)
}
- Once you have opened the local file, you need to create a formFile. This wraps the file data in the right format to upload it to the server:
formFile, err := multipartWritter.CreateFormFile("myFile", file.Name())
if err != nil {
log.Fatal(err)
}
- Copy the bytes from the local file into the form file, then close the form file writer so that it knows no more data will be added:
_, err = io.Copy(formFile, file)
if err != nil {
log.Fatal(err)
}
multipartWritter.Close()
- Next, you need to create the POST request you want to send to the server. In the previous exercises, we used shortcut functions such as http.Post. However, in this exercise, we need more control over the data being sent. That means we'll need to create an http.Request. In this case, you're creating a POST request that you will send to http://localhost:8080. As we are uploading a file, the bytes buffer also needs to be included in the request. That looks as follows:
req, err := http.NewRequest("POST", "http://localhost:8080", &fileDataBuffer)
if err != nil {
log.Fatal(err)
}
- You then need to set the Content-Type request header. This tells the server about the content of the file, so it knows how to handle the upload:
req.Header.Set("Content-Type", multipartWritter.FormDataContentType())
- Send the request as follows:
response, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
- After you have sent the request, we can read in the response and return the data within it:
defer response.Body.Close()
data, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal(err)
}
return string(data)
- Finally, you just need to call the postFileAndReturnResponse function and tell it what file to upload:
func main() {
data := postFileAndReturnResponse("./test.txt")
fmt.Println(data)
}
- To run this, you need to carry out two steps. The first is to navigate to the server directory in your terminal and run go run server.go. This will start the web server:
go run server.go
- Next, in the client directory, create a file named test.txt and put a few lines of text in it.
- In a second terminal window, navigate to the client directory and run go run main.go. This will start the client and connect to the server:
go run server.go
- The client will then read in test.txt and upload it to the server. The client should give the following output:
Figure 14.6: Expected client output
Then, if you navigate to the server directory, you should see that the test.txt file has now appeared:
Figure 14.7: Expected client output
In this exercise, you sent a file to a web server using the Go HTTP client. You read in a file from disk, formatted it into a POST request, and sent the data to the server.
Custom Request Headers
Sometimes there is more to a request than simply requesting or sending data. This information is stored within the request headers. A very common example of this is authorization headers. When you log into a server, it will respond with an authorization token. In all future requests sent to the server, you would include this token in the request's headers so the server knows you are the one making the requests. You will learn how to add an authorization token to requests later.
Exercise 14.05: Using Custom Headers and Options with the Go HTTP Client
In this exercise, you will create your own HTTP client and set custom options on it. You will also set an authorization token in the request headers, so the server knows it is you requesting the data:
- Create a new directory, Exercise14.05, on your GOPATH. Within that directory, create two more directories, server and client. Then, within the server directory, create a file called server.go and write the following code:
package main
import (
"log"
"net/http"
"time"
)
type server struct{}
func (srv server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
auth := r.Header.Get("Authorization")
if auth != "superSecretToken" {
w.WriteHeader(http.StatusUnauthorized)
w.Write([]byte("Authorization token not recognized"))
return
}
time.Sleep(10 * time.Second)
msg := "hello client!"
w.Write([]byte(msg))
}
func main() {
log.Fatal(http.ListenAndServe(":8080", server{}))
}
This creates a very basic web server that receives a request, checks the authorization header is correct, waits 10 seconds, then sends back data.
- Once you have created the server, navigate to the client directory and create a file called main.go. Add package main and the imports needed for the file:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
)
- Then, you need to create a function that will create an HTTP client, set the timeout limitations, and set the authorization header:
func getDataWithCustomOptionsAndReturnResponse() string {
- You need to create your own HTTP client and set the timeout to 11 seconds:
client := http.Client{Timeout: 11 * time.Second}
- You also need to create a request to send it to the server. You should create a GET request with the URL http://localhost:8080. No data will be sent in this request, so the data can be set to nil. You can use the http.NewRequest function to do this:
req, err := http.NewRequest("POST", "http://localhost:8080", nil)
if err != nil {
log.Fatal(err)
}
- If you look at the server code again, you will notice that it checks for the Authorization request header and it expects its value to be superSecretToken. So, you need to set the Authorization header in your request as well:
req.Header.Set("Authorization", "superSecretToken")
- You then get the client you created to do the request:
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
- Then, you need to read in the response from the server and return the data:
defer resp.Body.Close()
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
return string(data)
- Finally, you need to call the function you just created from the main function and log the data it returns:
func main() {
data := getDataWithCustomOptionsAndReturnResponse()
fmt.Println(data)
}
- To run this exercise, you need to carry out two steps. The first is navigate to the server directory in your terminal and run go run server.go. This will start the web server.
- In a second terminal window, navigate to the directory you created the client in.
- To execute the client, run the following command:
go run main.go
This will start the client and connect to the server. The client will send the request to the server and after 10 seconds it should output the following:
Figure 14.8: Expected output
Note
Change the timeout settings in the client to be under 10 seconds and see what happens. You can also change or remove the authorization header on the request and see what happens.
In this exercise, you learned how to add custom headers to a request. You learned about the common example of adding an authorization header, which is required by many APIs when you want to interact with them.
Activity 14.02: Sending Data to a Web Server and Checking Whether the Data Was Received Using POST and GET
Imagine you are interacting with a web API and you wish to send data to a web server. You then want to check whether the data was added. In this activity, you will do just that. You will send a POST request to the server, then request the data back using a GET request, parse the data, and print it out.
Follow these steps to get the desired outcome:
- Create a directory called Activity14.02.
- Create two sub-directories, one called client and one called server.
- In the server directory, create a file called server.go.
- Add the server code to the server.go file.
- Start the server by calling go run server.go in the server directory.
- In the client directory, create a file called main.go.
- In main.go, add the necessary imports.
- Create structs to host the request data.
- Create structs to parse the response data.
- Create an addNameAndParseResponse function that posts a name to the server.
- Create a getDataAndParseResponse function that parses the server response.
- Send a POST request to the server, to add names.
- Send a GET request to the server.
- Parse the response into a struct.
- Loop through the struct and print the names.
This is the expected output:
Figure 14.9: Possible output
Note
The solution for this activity can be found on page 754.
In this activity, you saw how to send data to a web server using a POST request and then how to request data from the server to ensure it was updated using a GET request. Interacting with a server in this way is very common when programming professionally.
Summary
HTTP clients are used to interact with web servers. They are used to send different types of requests to a server (for example, GET or POST requests) and then react to the response returned by the server. A web browser is a type of HTTP client that will send a GET request to a web server and display the HTML data it returns. In Go, you created your own HTTP client and did the same thing, sending a GET request to https://www.google.com and then logging the response returned by the server. You also learned about the components of a URL and that you can control what you request from a server by changing the URL.
There is also more to web servers than simply requesting HTML data. You learned that they can return structured data in the form of JSON, which can be parsed and used in your code. Data can also be sent to a server using POST requests, allowing you to send form data to a server. However, the data sent to a server isn't limited to just form data: you can also upload files to a server using a POST request.
There are also ways to customize the requests you send. You learned about the common example of authorization, where you add a token to the header of HTTP requests so that a server can tell who is making that request.
In this chapter, you used some basic web servers in the exercises. However, you didn't learn about the details of what they were doing. In the next chapter, you will learn about web servers in more detail.