NumPy Arrays are Different

The fundamental data structure introduced by NumPy is the ndarray or N-dimensional numerical arrays. For beginners in Python, sometimes these arrays look similar to a Python list. But they are anything but similar. Let’s demonstrate this using a simple example.

The treatment of the elements in the lists feel object-like, not very numerical, doesn’t it? If these were numerical vectors instead of a simple list of numbers, you would expect the + operator to act slightly different and add the numbers from the first list to the corresponding numbers in the second list element-wise.

What is np.array? It is nothing but the array method called from the NumPy module (the first line of the code did that with import numpy as np).

There are so many more (and different looking) functions and attributes available with the NumPy array object. In particular, take note of methods such as mean, std, and sum, as they clearly indicate a focus on numerical/statistical computing with this kind of array objects. And these operations are fast too. How fast? You will see that now.

NumPy Array vs. Native Python Computation

NumPy is much faster due to its vectorized implementation and the fact that many of its core routines were originally written in the C language (based on the CPython framework). NumPy arrays are densely packed arrays of homogeneous types. Python lists, by contrast, are arrays of pointers to objects, even when all of them are of the same type. So, we get the benefits of the locality of reference.

Many NumPy operations are implemented in the C language, avoiding the general cost of loops in Python, pointer indirection, and elementwise dynamic type checking. In particular, the boost in speed depends on what operation you are performing. For data science and ML tasks, this is an invaluable advantage because it avoids looping in long and multi-dimensional arrays.

Locality of reference (www.geeksforgeeks.org/locality-of-reference-and-cache-operation-in-cache-memory/) is one of the main reasons behind NumPy arrays being much faster and more efficient than Python list objects. Spatial locality in memory access patterns results in performance gains notably due to the CPU cache operations. The cache loads bytes in chunks from RAM to the CPU registers (the fastest memory in a computer system, located next to the processor). Adjacent items in memory are then loaded very efficiently.

Conversion First, Operation Later

Although not a guaranteed outcome, it is almost always better to vectorize your data first (Figure 3-1). In other words, convert it to NumPy arrays as early as possible and run the mathematical operations on those array objects rather than running native Python functions and then converting them to an array.

If you test the execution time, you will see the second option is 2X to 3X faster for this data. For a bigger data size, this much improvement may prove significant.

Data in real-life situations comes from business operations and databases. Data comes either in streaming or batch mode. Data can also come in web APIs in the format of JSON or XML. It almost will never come in a nicely NumPy-formatted manner. This is why it is so important to understand the pros and cons of array conversion, operations like appending to and updating an array, back conversion to a Python list in case you must stream the data back to another API through a JSON interface, and so on.

Vectorize Logical Operations

The second line of this code uses the Boolean indexing with NumPy where you create a Boolean NumPy array with array_of_nums%7==0 and then use this array as an index of the main array. This effectively creates an array with only the elements that are divisible by 7. Finally, you run your operation on this shorter array_div7. In a way, this is a filtering operation too where you filter the main array into a shorter array based on a logical check.

Use the Built-In Vectorize Function

Here you pass on the custom function object myfunc as the first argument in the np.vectorize and define the object types it should expect by the otypes parameter. The great thing is that although the main myfunc works on individual floating point numbers x and y, the resulting vectfunc can accept any array (or even a Python list) with the np.float64 data type (or even native Python floating point data, which will be coerced into the np.float64 type automatically).

Avoid Using the .append Method

Appending new or incoming data to an array is a common data science operation. Often the situation is that the data is generated by a stochastic or random process (e.g., a financial transaction or a sensor measurement) and it has to be recorded in a NumPy array (for later use in an ML algorithm, for example).

You can see this is directly contrary to what we discussed in the subsection “Conversion First, Operation Later.” However, the situation is subtly different here because, in this case, you are updating the array with incoming data that results from an unknown process, so you don’t know what precise mathematical operation to perform on the array.

As discussed, because of the uncertainty in the length of the data or the process that generates it, it is advisable to use a Python list to append the data as it comes in. When the data collection is finished, go back to the “conversion first, operation later” principle and convert the Python list to a NumPy array before doing any sophisticated mathematical operation over it.

Utilizing NumPy Reading Utilities

How would you read a text file where numerical data is stored in a CSV format into a NumPy array? This situation is extremely common in a regular data science pipeline as CSV (comma-separated value) remains one of the most popular file formats in use across all platforms (Windows, Linux, Mac OS, etc.).

Of course, you can use the csv module that comes with Python and read line by line. But, conveniently enough, NumPy provides many utility functions to read from file or string objects. Using them makes the code cleaner and thereby more productive. These routines are well-optimized for speed too, so your code remains efficient.

Utility for Tabular Data in a Text File

Numpy offers another similar text-reading utility called loadtxt, which is even more powerful and feature-rich. It works with text file where data is written in tabular format (i.e., in rows and columns) and loads data directly into a multi-dimensional array as long as the number of entries in each row remains same. Figure 3-2 illustrates this.

For example, suppose you have a CSV text file with three rows and three columns of data, as shown in Figure 3-3.

One line of code can read the contents of this file into a 3x3 NumPy array/matrix:

np.loadtxt('npread.txt',delimiter=',')

>> array([[  9.2,  22.1, -33.6],

       [  6.4,   2.3,  -5.4],

       [ 12.2,   4.5,   7.2]])

You can even read selective columns from the file. This is particularly useful if you always get a massive data file from a customer, but you know that only certain specific columns are useful for your data science work. Then, you can load only selective data into memory and make your pipeline fast and efficient.

np.loadtxt('npread.txt',delimiter=',',usecols=(0,2))

>> array([[  9.2, -33.6],

       [  6.4,  -5.4],

       [ 12.2,   7.2]])

Imagine the amount of custom text-reading code you would have to write if you did not have this utility function from NumPy. In the spirit of productive data science and keeping your code clean and readable, use these utilities whenever possible.

Setting Values in a New DataFrame

pandas provides a variety of options to index, select particular data, and set it to a given value. In many situations, you will find yourself with a Python list or NumPy array that you want to set at a particular position (row) in your DataFrame.

You have a few options to insert this data to the rows of a DataFrame. Note that in reality you will have a dictionary or a few thousand such lists (all different). Just for the speed demonstration, I show inserting the same list data in the DataFrame.

Now comes the part where you iterate and insert the data into one row after another.

Specify Data Types Whenever Possible

Making pandas guess data types is one of the most frequent inefficient code patterns and it happens with almost all data scientists. It is inefficient because when you import data into a DataFrame without specifically telling pandas the datatypes of the columns, it will read the entire dataset into memory just to figure out the data types. Quite naturally, it hogs the system memory and results in a highly wasteful process that can be avoided with more explicit code.

So, how do you do it as a standard practice? Reading data from the disk is often done with some sort of plain text file like a CSV. You can read just the first few lines of the CSV file, determine the data types, create a dictionary, and pass it on for the full file read, or use it repeatedly for reading similar files (if the column types are unchanged). You can use the dtype parameter in various pandas reading functions to specify the expected data types.

Figure 3-4 shows a visual illustration of the idea of reading sample data first, determining the data type, and then utilizing it for the full reading of the data.

Iterating Over a DataFrame

It is a quite common situation where you are given a large pandas DataFrame and are asked to check some relationships between various fields in the columns, in a row-by-row fashion. The check could be some logical operation or some conditional logic involving a sophisticated mathematical transformation on the raw data.

Essentially, it is a simple case of iterating over the rows of the DataFrame and doing some processing at each iteration. You can choose from the following approaches. Interestingly, some of the approaches are much more efficient than others.

Using Modern, Optimized File Formats

CSV is a flat-file format used widely in data analytics. It is simple to work with and performs decently in small to medium data regimes. However, as you do data processing with bigger files (and also, perhaps, pay for the cloud-based storage of them), there are some excellent reasons to move towards file formats using the columnar data storage principle (www.stitchdata.com/columnardatabase/). The basic idea of columnar data storage (vs. the traditional row-based storage) is illustrated in Figure 3-5.

Apache Parquet is one of the most popular of these columnar file formats. It’s an excellent choice in the situation when you have to store and read large data files from disk or cloud storage. Parquet is intimately related to the Apache Arrow framework. But what is Apache Arrow?

As per their website, https://arrow.apache.org/, “Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enable big data systems to process and move data fast. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware.”

Therefore, to take advantage of this columnar storage format, you need to use some kind of Python binding or tool to read data stored in Parquet files into the system memory and possibly transform that into a pandas DataFrame for the data analytics tasks. This can be accomplished by using the PyArrow framework.

Impressive Speed Improvement

PyArrow is a Python binding (API) for the Apache Arrow framework. Detailed coverage of Apache Arrow or PyArrow (https://arrow.apache.org/docs/python/) is far beyond the scope of this book, but interested readers can refer to the official documentation at https://arrow.apache.org/ to get started.

Using the PyArrow function read_table, you can demonstrate considerable improvement of the reading speed of large data files over the commonly used pandas read_csv method. For example, Figure 3-6 shows the ratio of pandas and PyArrow reading times of the same data, stored in CSV and Parquet, respectively. The ratio goes up as the data size increases; PyArrow performs considerably better with larger file sizes.

This is something truly astonishing to ponder. pandas is based on the fast and efficient NumPy arrays, yet it cannot match the file-reading performance shown by the Parquet format. If we think about it deeply, the reason becomes clear that the file-reading operation has almost nothing to do with how pandas optimize the in-memory organization of the data after it is loaded into the memory. Therefore, while pandas can be a fast and efficient package for in-memory analytics, we don’t have to stay dependent upon traditional file formats like CSV or Excel to work with pandas. Instead, we should move towards using more modern and efficient formats like Parquet.

Read Only What Is Needed

Often, you may not need to read all the columns from a columnar storage file. For example, you may apply some filter on the data and choose only selected data for the actual in-memory processing. With CSV files or regular SQL databases, this means you are choosing specific rows out of all the data. However, for the columnar database, this effectively means choosing specific columns. Therefore, you do have an advantage in terms of reading speed when you are reading only a small fraction of columns from the Parquet file.

Figure 3-7 shows the reading advantage as the number of columns increases for the same CSV vs. Parquet comparison. When you read a very small fraction of columns, say < 10 out of 100, the reading speed ratio becomes as large as > 50 (i.e., you get 50X speedup compared to the regular pandas CSV file reading). The speedup tapers off for large fractions of columns and settles down to a stable value.

Reading selected columns from a large dataset is an extremely common scenario in data analytics and machine learning tasks. Often, subject matter experts advise data scientists with domain knowledge and can preselect a few features from a large dataset although the default data collection mechanism may store a file with many columns/features. In these situations, it makes logical sense to read only what is needed and process those columns for the ML workload. Storing the data in a columnar data format like Parquet pays handsomely for these cases.

PyArrow to pandas and Back

While the results shown above are impressive, the central question is about how to take advantage of this fast and efficient file format for pandas-based data analytics tasks. This has been made extremely simple by PyArrow utility methods, as this simple boilerplate code illustrates:

import pyarrow as pa

import pandas as pd

df = pd.DataFrame({"a": [1, 2, 3],

                   "b":[2.7,-1.2,5.4],

                   "c": ['abc','xyz','pqr']})

# Convert from pandas to Arrow

table = pa.Table.from_pandas(df)

# Convert back to pandas

df_new = table.to_pandas()

So, there are ready functions to convert PyArrow tables and pandas DataFrame back and forth. You can take advantage of this in the scenario illustrated in Figure 3-8.

Suppose you have a large CSV file of numeric quantities with ~1 million rows and 14 columns, and you want to calculate the basic descriptive stats on this dataset. Not so surprisingly, if you only use pandas code, the majority of the time will be taken by the file reading operation, not by the statistical calculation. You can make this task efficient by storing the file in the Parquet format instead of CSV, reading it using the read_table method, converting to pandas using the to_pandas method, doing the statistical calculation, and then just storing the result back in CSV or Parquet. The output consists of only a few rows/columns as it is just the descriptive stats, so the file format does not matter much. A demo example with speed comparison is shown in the accompanying Jupyter notebook with this book.

Other Miscellaneous Ideas

pandas is such a vast and storied library that there are thousands of ways to improve upon inefficient and non-productive code patterns while using it. A few miscellaneous suggestions are mentioned here.

Embrace the Object-Oriented Nature of Matplotlib

Matplotlib is built in a thoughtful manner (www.aosabook.org/en/matplotlib.html) following multiple layers of abstractions and object-oriented design hierarchy (as shown in Figure 3-9). Almost always, a data scientist deals with the scripting layer to draw quick plots (e.g., plt.scatter(x,y)) and change the look and feel of that graphical output (e.g., plt.xlabel("The x-axis variable", fontsize=15)). Sometimes, they venture into the middle artistic layer, creating custom Axes and setting the properties of Figure objects. Usually, a data scientist does not need to work directly with the backend layer for regular data analytics tasks.

However, it is a great education for a data scientist to have deep knowledge about this layered architecture and follow the best practices that leverage the strength of a solid object-oriented design. In particular situations such as those involving subplots, this becomes prominent.

Two Approaches for Creating Panels with Subplots

A simple example of a good practice is to not to use the following old style of code to create two subplots or panels stacked vertically:

# Create the main figure

plt.figure()

# The first of two panels

plt.subplot(2, 1, 1) # (rows, columns, panel number)

plt.plot(x, np.sin(x))

# The second panel

plt.subplot(2, 1, 2)

plt.plot(x, np.cos(x));

A better alternative is to use the following code:

# Create an array of two Axes objects

fig, ax = plt.subplots(2)

# Call plot() method on the appropriate object

ax[0].plot(x, np.sin(x))

ax[1].plot(x, np.cos(x))

They produce identical graphical output, as shown in Figure 3-10.

But why is the second approach better or more productive? Think about the cognitive load you might have to carry if it were 5 or 15 plots instead of 2 and the chances of bugs that could have been introduced writing code like plt.subplot(3, 1, 3) or plt.subplot(4, 4, 13). How would you keep track of all those parameters inside the plt.subplot() function? The second approach frees you from these considerations by allowing it to pass in a single number like 2 or 15 and repeat the plot statement that many times.

However, an even better approach is to put this code in a proper function that has a little more intelligence to handle any number of plots and that refactors the plotting statements using a loop.

A Better Approach with a Clever Function

Consider the following code defining a function that can produce a panel with an arbitrary number of plots (always in a three-column format respecting the natural width of the webpage or a book), dynamically adjusting the number of rows with the number of total subplots:

def plot_panels(n):

    """

    Produce a panel consisting of variable number of rows and 3 columns

    """

    if n%3==0:

        nrows =  int(n/3)

    else:

        nrows = n//3+1

    ncols = 3

    fig, ax = plt.subplots(nrows, ncols, figsize=(15,nrows*3))

    axes = ax.ravel()

    for i in range(n):

        axes[i].plot(x, np.sin(x))

Here, you can change the variable n to any value. Internally, the function will always calculate the appropriate number of rows with the logic in the code and set ncols = 3. Here, ax is a (multi-dimensional) list of Matplotlib Axes objects (https://matplotlib.org/stable/api/axes_api.html) and therefore can be indexed with axes[i] within a loop after you flatten the list with an axes = ax.ravel() statement.

When you call this function with plot_panels(5), you get the result shown in Figure 3-11.

Note the blank canvas in the last row. This is because the plots must be arranged in a rectangular grid and for placing five plots on a 3 x 2 grid, so the last one will be left blank. When you call the same function with plot_panels(15), you get the result shown in Figure 3-12.

It is the object-oriented style of programming pattern you embraced in your function definition that resulted in this scalable and efficient mechanism of generating any number of plots without worrying about potential bugs. This type of practice makes the codebase productive in the long run.

Set and Control Image Quality

Matplotlib interacts with the user’s graphical output system (web browser or stand-alone window) in a complex manner and optimizes the output image quality with a balanced set of internal settings. However, it is possible to tweak those settings as per the user’s preference to get the most optimum quality that they desire.

This becomes particularly important for using Matplotlib in the Jupyter notebook environment, which is an extremely common scenario. The quality of the default image, rendered in a Jupyter notebook web browser, may not be good enough for publication in a book or further processing. Data scientists often spend additional time and effort enhancing the quality of the visualizations they produce as part of the data science tasks. However, Matplotlib provides a simple and intuitive workaround to accomplish the same.

Adjust Global Parameters

The Matplotlib back end provides the ultimate flexibility in terms of setting global parameters that control the look and feel of your visualization. The rcParams settings (https://matplotlib.org/stable/api/matplotlib_configuration_api.html#matplotlib.RcParams) have all the possible varieties you can think of. Here is a code example:

import matplotlib as mpl

# Data

x = np.arange(-10,10,0.1)

y = x**2

# Set all backend parameters

mpl.rcParams['lines.linewidth'] = 3

mpl.rcParams['text.color'] = 'red'

mpl.rcParams['lines.linestyle'] = '--'

mpl.rcParams['axes.facecolor'] = '#c3e2e6'

mpl.rcParams['figure.dpi'] = 120

mpl.rcParams['font.style'] = 'italic'

mpl.rcParams['font.weight'] = 'heavy'

# Plot

plt.plot(x,y)

plt.title("Parabola", fontsize=16)

plt.xlabel('x-axis')

plt.ylabel('y-axis')

Note how you had to import the Matplotlib module itself with the statement import matplotlib as mpl and not just use the matplotlib.pyplot as plt. Also note the figure.dpi as one of the many settings you set in this code. A typical result from this setting is shown in Figure 3-13.

If you have decided on a set of image quality and styling settings, you can store them in a local config file and just read the values at the beginning of your Jupyter notebook or Python script while importing Matplotlib. That way, every image produced by that script or in that Jupyter session will have the same look and feel. The output of the code above should look something like Figure 3-13.

Did you notice that the axes.facecolor was set to a hex string #c3e2e6 in the code above? Matplotlib accepts regular color names like red, green, or blue, or hex strings in its various internal settings. You can simply use an online color picker tool (https://imagecolorpicker.com/) and copy-paste the hex code for better styling of your image.

Tricks with Seaborn

Seaborn is a Python library built on top of Matplotlib with a concentrated focus on statistical visualizations like boxplots, histograms, and regression plots. Naturally, for data scientists, it is a great tool to use in a typical exploratory data analysis (EDA) phase. However, using Seaborn with a couple of simple tricks can improve the productivity of your EDA tasks.

Use pandas Correlation with Seaborn heatmap

This is a trick to quickly visualize the correlation strengths between multiple features of your dataset with just two lines of code. This kind of trick should be standard part of your efficient data science toolkit.

Here is a code snippet:

df_mpg = sns.load_dataset('mpg')

mpg_corr = df_mpg.corr()

sns.heatmap(mpg_corr,cbar=True,cmap='plasma')

plt.show()

This loads the famous Auto MPG dataset (https://archive.ics.uci.edu/ml/datasets/auto+mpg) and produces the correlation heatmap shown in Figure 3-14, demonstrating the positive and negative correlation strengths between various numerical features of the dataset. The bright colors and italic/bold axis names of this plot are the result of the Matplotlib style settings you did in the previous section. Unless you change them explicitly or start a new Jupyter notebook session, they remain in effect.