Handling non-standard data

Parsing the data is a piece of cake with only the standard library:

func parse(l loader) (dates []string, co2s []float64) {
s := bufio.NewScanner(l())
for s.Scan() {
row := s.Text()
if strings.HasPrefix(row, "#") {
continue
}
fields := strings.Fields(row)
dates = append(dates, fields[2])
co2, err := strconv.ParseFloat(fields[4], 64)
dieIfErr(err)
co2s = append(co2s, co2)
}
return
}

The parsing function takes a loader, which when called, returns a io.Reader. We then wrap the io.Reader in a bufio.Scanner. Recall that the format is not standard. There are some things that we want and some things we don't. The data however is in a fairly consistent format—we can use the standard library functions to filter the ones we want and the ones we don't.

The s.Scan() method scans io.Reader until it encounters a newline. We can retrieve the string using s.Text(). If the string starts with #, we skip the line.

Otherwise, we use strings.Fields to split the string into fields. The reason why we use strings.Fields instead of strings.Split is because the latter does not handle multiple spaces well.

Following the splitting of the row into fields, we parse things that are necessary:

type loader func() io.Reader

Why do we need a loader type?

The reason is simple: we want to be good citizens— we should not be repeatedly requesting data from the FTP server while we are developing the program. Rather, we would cache the file and work with that single file while in development mode. This way, we wouldn't have to download from the internet all the time.

The corresponding loader type that reads from the file looks something like this, and is rather self-explanatory:

func readFromFile() io.Reader {
reader, err := os.Open("data.txt")
dieIfErr(err)
return reader
}
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.145.97.170