Using the StructArrays package

The ugliness of the preceding columnar struct left us in a very unsatisfied state. Not only do we need to create a new data type with tons of Vector fields, we also have to create a constructor function to convert our array of structs into the new type.

We can recognize the power of Julia when we get to use powerful packages in its ecosystem. To fully implement this pattern, we will introduce the StructArrays.jl package, which automates most of the mundane tasks in turning an array of structs into a struct of arrays.

In fact, the usage of StructArrays is embarrassingly simple:

using StructArrays
sa = StructArray(records)

Let's take a quick look at the content. First of all, we can treat sa just like the original array—for example, we can take the first three elements of the array as before:

If we pick just one record, it comes back with the original TripPayment object:

Just to make sure that there is no mistake, we can also check the type of the first record:

Hence, the new sa object works just like before. Now, the difference comes in when we need to access all of the data from a single field. For example, we can get the fare_amount field as follows:

Because the type is already materialized as a dense array, we can expect superb performance when doing numerical or statistical analysis on this field, as follows:

What is a DenseArray? It is actually an abstract type for which all elements in the array are allocated in a contiguous block of memory. DenseArray is a super-type of array.

Julia supports dynamic arrays by default, which means the size of the array can grow when we push more data into it. When it allocates more memory, it copies existing data over to the new memory location. 

To avoid excessive memory reallocation, the current implementation uses a sophisticated algorithm to increase the size of memory allocation—fast enough to avoid excessive reallocation but conservative enough to not over-allocate memory.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.191.223.123