Out-of-Core (a.k.a. Out of Memory)

pandas is the most popular data analysis library in Python, and it is at the front end of any standard data science pipeline. However, if you have ever tried to work with data files larger than a few GB, you may have seen the memory error that is thrown by pandas (Figure 9-1).

Of course, this error depends on the exact state of the system memory such as how many other processes are running alongside the pandas code and what type of memory they are blocking. Nonetheless, it is a well-known fact that pandas cannot handle multi-GB datasets (no matter how simple in structure they may be) efficiently.

Furthermore, this inefficiency and limitation can rear its ugly head even with a large dataset that could be somehow loaded in the memory without any memory error at the beginning. Due to the way pandas handles in-memory objects and calculations, it is quite easy to run into the same memory error in your data science code. This can be exacerbated by code that produces many large in-memory DataFrames in quick succession with intermediate calculations.

This is a generic code snippet, but you get the idea that this code is inefficient, particularly when dealing with large pandas DataFrame objects. It produces multiple intermediate DataFrames and does not purge them from memory when their job is done. At the end, it may use only the final DataFrame for a machine learning modeling, but the system memory is already clogged with so many useless objects that it will result in a memory error and the whole pipeline will crash. This is illustrated in Figure 9-2.

Of course, one way to get around this issue is to rigorously maintain a good coding habit where unused objects are tracked and purged regularly. However, while doing prototyping on their Jupyter notebooks, data scientists are bound to write quick and dirty code without following this practice, and this will hinder their scalability options with large datasets.

Python Single Threading

The GIL was one of the earlier design choices in the Python language and it solved quite a few important problems related to memory leaks and racing conditions. Put simply, it is a locking mechanism that allows only one thread to hold the control of the Python interpreter. This means that only one thread can be in a state of execution at any given point in time.

Generally, its impact isn’t visible to programmers executing single-threaded programs. In fact, many data science tasks can run just fine without worrying about GIL as they execute a series of tasks one after another and do not employ many parallel processing tricks. However, it can become a performance bottleneck in CPU-bound and multi-threaded code.

For larger datasets, sometimes it makes sense to divide the data into multiple chunks and utilize a parallel processing execution pipeline. The idea is to send the chunked data to each core of the CPU and execute the analysis as much in parallel as possible. When the executions are done, the results can be combined to get back a transformed dataset. While this does not necessarily help to fit a larger dataset in memory, it can make analysis of the same dataset faster by the parallel execution.

The beauty here is that this approach can speed up data science exploration and prototyping tasks even without paying for large CPU clusters on the cloud. It is really a matter of taking advantage of the 8 or 16 cores that routinely come with the single modern-day CPU inside a data scientist’s laptop. However, you must make sure that the data science code and libraries are not getting in the way, and that you are using libraries that can take full advantage of the multi-core hardware platform.

Cloud Instances

For larger and larger datasets, there is always a brute-force solution of renting out a cloud instance with a large RAM attached. As an example, these days you can rent out an AWS (Amazon Web Service) Elastic Compute (EC2) instance with 128GB of RAM for less than a dollar per hour. Figure 9-3 shows the pricing for a r6g.4xlarge instance (a so-called memory optimized EC2 instance, www.amazonaws.cn/en/ec2/instance-types/#Memory_Optimized_Instances).

Once set up, you can install all your favorite Python libraries, read large data files stored locally (e.g., to a mapped SSD) or from an AWS S3 folder, and do pandas data transformation without worrying about memory errors. While it may still seem expensive to a causal user, organizations or teams who need that much memory to process pandas DataFrames regularly probably won’t mind paying ~$0.8 an hour for a smooth and error-free data science task flow.

However, remember that pandas will still be limited to use only one CPU core at a time and, by default, it will exhibit slowness while loading and dealing with large files. Just running run-of-the-mill pandas code on a large-memory cloud instance may stop some frequent memory error situations, but it may not fundamentally make the data science pipeline productive or efficient at scale.

Google Colab

Google Colaboratory (or Colab, as it is known popularly; https://research.google.com/colaboratory/faq.html) is also a cloud service at its core. Basically, it runs a Jupyter notebook service that is hosted on Google cloud servers. You can use a CPU, GPU, or even a TPU (if you are lucky) for free just by having a Google account.

The greatest advantage of Colab, as compared to AWS or GCP, is its ease of use and low barrier of entry. If you have your data science Python code in a Jupyter notebook, Colab can help get you started on this cloud instance instantly (as soon as you upload your notebook to the instance). Unlike AWS or GCP barebone instances, there is no setup or installation needed. You can directly access Colab notebooks through your browser and start running your code in a matter of minutes.

For some specific situations, this may indeed increase productivity and efficiency. For example, if the local laptop does not have a good enough CPU or a GPU card installed, or the RAM is under 8GB, then switching to Google Colab should enhance the productivity instantly.

The typical instance (free of cost) has ~12-13 GB of RAM and a CPU equivalent to an Intel Xeon processor. Getting a GPU instance is also quite easy, with the most common GPU being a Tesla K80 (compute 3.7, having 2496 CUDA cores and 12GB GDDR5 VRAM). While the CPU core count is nothing boast about, having a larger RAM and GPU memory may help data science exploration, especially if it involves GPU-intensive tasks like training a deep neural network or even vectorized computation involving NumPy arrays. If 12GB RAM seems too little, you can upgrade to Colab Pro (https://colab.research.google.com/signup), which offers double the RAM for only $10/month (a whole lot cheaper than paying for an equivalent EC2 instance with 24GB of RAM).

However, despite its attractive features, Colab does have some serious limitations for practicing data scientists who are trying to explore larger datasets and scale up their data science workflow. At the outset, it puts a time limit on the running time of the notebook, so if you leave it idle for a certain amount of time, the instance will die (along with any variables and internal states). Basically, you must plan your code execution carefully and be ever vigilant to take full advantage of Colab.

Also, file loading (whether uploading from local drive or reading from the Web) is painfully slow (most probably a deliberate choice to control the bandwidth usage over the Google Cloud infrastructure). Therefore, while you can do in-memory analytics and data transformations rather quickly, the initial loading can take an inordinate amount of time or may even crash your notebook. Upgrading to Colab Pro or Pro+ (from a completely free account) alleviates these issues to some extent but not fully.

pandas-Specific Tricks

Since I started the scalability discussion by pointing out the out-of-core issues in pandas, it makes sense to loop back to ground zero and examine what suggestions the pandas developers have to address this issue.

It goes on to point out some useful tricks and techniques for coping with memory issues. I discuss some of them below and add a few more.

Column-Specific Functions (If Applicable)

Following the same idea as above, it is a good practice (wherever applicable) to write separate functions that deal with specific columns/features in the dataset as needed (Figure 9-4). For example, a dataset may have the following:

• String data corresponding to name and address. This can be handled by a specific function.

• Datetime data corresponding to some business transaction. This should be handled by another specific function that loads and process only these columns.

• Pure numeric data, which can be handled many ways, even read as a pure NumPy array and utilizing vectorizing tricks (as discussed elsewhere in this book) to speed up the data transformation process.

Explicitly Specify/Convert Data Types

The default data types in pandas are not designed to be the most memory efficient. This is especially true for text/string data columns with relatively few unique values (alternatively known as low-cardinality data). By using more efficient and targeted data types, you can significantly reduce the memory usage and process larger datasets.

There is a dataset called loan_data.csv on file (supplied with the book). Let’s see how explicitly specifying the data type can reduce the memory usage while working with this dataset:

import pandas as pd

df = pd.read_csv("../loan_data.csv")

df.memory_usage(deep=True)

The function memory_usage() shows the true memory usage by the in-memory object. The output is shown in Figure 9-5. The output of df.info() is shown in the same figure, indicating that the default loading assigned the general-purpose object data type to that column while others were assigned data types like int64 or float.

You might also have noticed that the credit.policy is an unsigned integer taking on values 1 or 0. Why do you need a 64-bit integer data type to represent that? So, let’s also type convert that column:

df['credit.policy'].unique()

>> array([1, 0], dtype=int64)

Here is the code for doing the data type conversions (explicit specifications):

df2 = df.copy()

df2['purpose'] = df2['purpose'].astype('category')

df2['credit.policy'] = df2['credit.policy'].astype('uint8')

del(df)

Here, first you do a copy on the existing DataFrame. Then, you use the astype function to assign the category data type to the purpose column and unit8 (8-bit unsigned integer) to the credit.policy column. Lastly, as a good practice, you delete the old DataFrame object from the memory since you no longer need it for your data science pipeline.

You can see the stark difference in Figure 9-6.

This memory saving may seem trivial for this example, but small savings like this add up quickly for a long and data-intensive pipeline and can reduce the total overhead significantly.

Libraries for Parallel Processing

Parallel computing is an extensive field of its own. It is not trivial to implement optimized code in Python that will execute parallel threads/processes flawlessly and with high performance. Fortunately, there are some fantastic Python frameworks for doing parallel processing with minimal learning curves.

I will discuss a couple of them, Dask and Ray, with hands-on examples in the next chapter, so I won’t get into those details here.

Libraries for Handling Out-of-Core Datasets

There are special libraries to handle out-of-core datasets. Vaex and Modin are two such frameworks. Let’s discuss them in more detail with hands-on examples next.

A Note About the Preferred OS

Features at a Glance

Basic Usage Example

When you run this code first time, it will download the dataset from the Web, so an Internet connection is required while running this code first time. It will store the dataset (a .hdf5 file) in a folder called data.

You will see something like Figure 9-7.

It will give you the expected output (Figure 9-8).

No Unnecessary Memory Copying

The best thing about Vaex is that it does not create unnecessary copies of DataFrame objects while doing simple filtering operations or intermediate calculations. Even the base DataFrame has minimal memory impact. The computations are done in a lazy, on-the-fly (when necessary) manner.

You can run all the necessary calculations on the Vaex dataframe object with much less worry about memory errors cropping up.

Expressions and Virtual Columns

Computation on a Multidimensional Grid

The result is nothing but a NumPy array with the number counts in 32 bins distributed between x = -5, and x = 5. The key thing to note here is the binby argument inside the function that works similar to GroupBy in SQL or even pandas. Here the data was grouped by the x column (binby=df.x).

Figure 9-9 shows the visualization.

Figure 9-10 shows the result.

Dynamic Visualizations Using Widgets and Other Plotting Libraries

The N-dimensional grid-based computation is designed to be fast with Vaex. This allows you to extend the visualization to be dynamic using widgets and third-party libraries like bqplot. Unfortunately, these dynamic visualizations are not possible to render in the pages of a book. However, some code and results are shown in the Jupyter notebook.

For example, this simple code creates a plot widget in the Jupyter notebook where you can pan and zoom around and choose a few data transformations from the drop-down menu. (Figure 9-11 shows a static snapshot of the widget.)

The usefulness of such utility methods cannot be overemphasized for large-scale data analysis. You can plot complex transformations on large, out-of-core datasets (say 10GB or 20GB in size) with only a few lines of code to visualize the hidden patterns. This increases the productivity and efficiency of such a data analysis pipeline far beyond what would have been possible with only pandas and Matplotlib code.

Vaex Preferred HDF5 Format

The magic in Vaex happens because of internal optimization and data representation. One of the design choices is to work with HDF5 file formats as much as possible. Therefore, the best way to work with Vaex is to load other types of data into this format before you start exploring. For convenience, Vaex provides many utility methods to convert other files or data structures to this format. You can convert from CSVs, Arrow tables, Python dictionaries, NumPy arrays, JSON, and more.

When you run this code, another file named Large-data.csv.hdf5 gets created in the folder where the Large-data.csv file resides. You must not forget to set convert=True for this to happen.

In today’s world of data science, a dataset with a million rows is not a particularly large one. Even this modest sized file caused pandas to take over 2 seconds to read from the disk. With Vaex, after conversion to HDF5, it becomes so much faster. Therefore, for a data processing pipeline utilizing the power of libraries like Vaex, it makes sense to convert (in a systematic manner) all the text-based data files (CSV or even JSON if that makes sense) to HDF5 and read them as a Vaex DataFrame as much as possible, as illustrated in Figure 9-12.

Single CPU Core to Multi-Core

From a simple change in one line of code, the benefit that you get is enormous. This comes from the fact that despite all the great features and capabilities, the core implementation of pandas is inherently single-threaded. This means that only one of the multiple CPU cores can be utilized at any given time for executing normal pandas code. In a single CPU machine (e.g., a data scientist’s laptop), it would look something like Figure 9-13.

However, just wrapping the pandas code with Modin (a single line of code change), you can utilize all the cores (by setting up a Dask or Ray backend cluster/worker system), as shown in Figure 9-14.

Out-of-Core Processing

Let’s now demonstrate the out-of-core processing capability of Modin. Here, the phrase “core” does not refer to the CPU core but really to the system memory or RAM.

When you execute this code for the first time, you may see some user warnings and message (Figure 9-15) about the Dask cluster setup (assuming that you are using the Dask back end for the parallel processing/ clustering). In the next chapter, you will see how to start and monitor a Dask cluster. The good thing with Modin is that all of this gets taken care of under the hood, and the user doesn’t have to write the cluster setup code.

So, it does have over 20 million rows and 256 columns. This would be almost impossible to handle as a persistent in-memory object with pure pandas.

So, each column’s memory usage is over 80MB. In total, for 256 columns (and one index), this represents over 20GB of memory usage. My laptop has only 16GB of RAM and surely the Jupyter notebook, where this code is being run, did not take up all the memory. This is the direct demonstration of out-of-core computing with Modin.

So, the time goes from 2.7 seconds to 37.7 seconds. Not a 256X increase in the computing time, but much less. This is the fruit of parallel processing and allocating data chunks optimally to each worker that the Dask cluster has set up in the background.

Other Features of Modin

For more details and updates, interested readers should definitely check out the official documentation at https://modin.readthedocs.io/en/stable/index.html.