Downloading from non-HTTP sources

We'll start by writing the function that will download the data, as follows:

func download() io.Reader {
client, err := ftp.Dial("aftp.cmdl.noaa.gov:21")
dieIfErr(err)
dieIfErr(client.Login("anonymous", "anonymous"))
reader, err := client.Retr("products/trends/co2/co2_mm_mlo.txt")
dieIfErr(err)
return reader
}

The NOAA data sits on a publicly accessible FTP server: ftp://aftp.cmdl.noaa.gov/products/trends/co2/co2_mm_mlo.txt. If you visit the URI via a web browser, you would see the data immediately. To access the data programmatically is a little tricky, as this is not a typical HTTP URL.

To handle FTP connections, we will be using the github.com/jlaffaye/ftp package. The package can be installed using the standard go get method: go get -u github.com/jlaffaye/ftp. The documentation for the package is a little sparse and somewhat requires you to understand the FTP standards. But, fear not, using FTP to acquire the file is relatively simple.

First we need to dial in to the server (you would need to do the same if you were working with HTTP endpoints—net/http merely abstracts out the dialing in so you wouldn't necessarily see what's happening in the background). Because dialing in is a fairly low-level procedure, we would need to supply the ports as well. Just like the convention for HTTP is for the server to listen on port 80, the convention for an FTP server is to listen to port 21, so we'd have to connect to a server specifying that we want to connect on port 21.

An additional oddity to those not used to working with FTP is that FTP requires a login to the server. For servers with anonymous read-only access, the convention is typically to use "anonymous" as the username and password.

After successfully logging in, we retrieve the requested resource (the file that we want) and download the file. The fttp library at github.com/jlaffaye/ftp returns io.Reader. Think of it as a file that contains the data.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.145.177.39