Now that we have seen how Python works in Jupyter, including the underlying encoding, how does Python access a large dataset of work in Jupyter?

I started another view for pandas, using Python data access as the name. From here, we will read in a large dataset and compute some standard statistics on the data. We are interested in seeing how we use pandas in Jupyter, how well the script performs, and what information is stored in the metadata (especially if it is a larger dataset).

Our script accesses the iris dataset that is built-in to one of the Python packages. All we are looking to do is read in a slightly large number of items and calculate some basic operations on the dataset. We are really interested to see how much of the data is cached in the .pynb file.

The Python code is as follows: 

# import the datasets package 
from sklearn import datasets 
 
# pull in the iris data 
iris_dataset = datasets.load_iris() 
# grab the first two columns of data 
X = iris_dataset.data[:, :2] 
 
# calculate some basic statistics 
x_count = len(X.flat) 
x_min = X[:, 0].min() - .5 
x_max = X[:, 0].max() + .5 
x_mean = X[:, 0].mean() 
 
# display our results 
x_count, x_min, x_max, x_mean 

I broke these steps into a couple of cells in Jupyter:

Now, run the cells (using Cell | Run All) and we will get the following display. The only difference is the last out line where our values are displayed:

It seemed to take longer to load the library (the first time I ran the script) than to read the data and calculate the statistics.

If we look in the .pynb file for this Notebook, we can see that none of the data is cached in the .pynb file. We simply have code references to the library, our code, and the output from when we last calculated the script:

{ 
 "cells": [ 
  { 
   "cell_type": "code", 
   "execution_count": 2, 
   "metadata": {}, 
   "outputs": [], 
   "source": [ 
    "# import the datasets package
", 
    "from sklearn import datasets" 
   ] 
  }, 
  { 
   "cell_type": "code", 
   "execution_count": 3, 
   "metadata": {}, 
   "outputs": [], 
   "source": [ 
    "# pull in the iris data
", 
    "iris_dataset = datasets.load_iris()
", 
    "# grab the first two columns of data
", 
    "X = iris_dataset.data[:, :2]" 
   ] 
  }, 
  { 
   "cell_type": "code", 
   "execution_count": 4, 
   "metadata": {}, 
   "outputs": [], 
   "source": [ 
    "# calculate some basic statistics
", 
    "x_count = len(X.flat)
", 
    "x_min = X[:, 0].min() - .5
", 
    "x_max = X[:, 0].max() + .5
", 
    "x_mean = X[:, 0].mean()" 
   ] 
  }, 
  { 
   "cell_type": "code", 
   "execution_count": 5, 
   "metadata": {}, 
   "outputs": [ 
    { 
     "data": { 
      "text/plain": [ 
       "(300, 3.8, 8.4, 5.843333333333334)" 
      ] 
     }, 
     "execution_count": 5, 
     "metadata": {}, 
     "output_type": "execute_result" 
    } 
   ], 
   "source": [ 
    "# display our results
", 
    "x_count, x_min, x_max, x_mean" 
   ] 
  } 
 ]...