Python is very good at handling reading and writing files or file-like objects. For example, if you try to load big files, say a few hundred MB, assuming you have a modern machine with at least 2 GB of RAM, Python will be able to handle it without any issue. It will not try to load everything at once, but play smart and load it as needed.
So even with decent file sizes, doing something as simple as the following code will work straight out of the box:
with open('/tmp/my_big_file', 'r') as bigfile:
for line in bigfile:
# line based operation, like 'print line'But if we want to jump to a particular place in the file or do other nonsequential reading, we will need to use the handcrafted approach and use IO functions such as seek(), tell(), read(), and next() that allow enough flexibility for most users. Most of these functions are just bindings to C implementations (and are OS-specific), so they are fast, but their behavior can vary based on the OS we are running.
Depending on what our aim is, processing large files can sometimes be managed in chunks. For example, you could read 1,000 lines and process them using Python standard iterator-based approaches, as shown here:
import sys
filename = sys.argv[1] # must pass valid file name
with open(filename, 'rb') as hugefile:
chunksize = 1000
readable = ''
# if you want to stop after certain number of blocks
# put condition in the while
while hugefile:
# if you want to start not from 1st byte
# do a hugefile.seek(skipbytes) to skip
# skipbytes of bytes from the file start
start = hugefile.tell()
print "starting at:", start
file_block = '' # holds chunk_size of lines
for _ in xrange(start, start + chunksize):
line = hugefile.next()
file_block = file_block + line
print 'file_block', type(file_block), file_block
readable = readable + file_block
# tell where are we in file
# file IO is usually buffered so tell()
# will not be precise for every read.
stop = hugefile.tell()
print 'readable', type(readable), readable
print 'reading bytes from %s to %s' % (start, stop)
print 'read bytes total:', len(readable)
# if you want to pause read between chucks
# uncomment following line
#raw_input()We call this code from the Python command-line interpreter, giving the filename path as the first parameter:
$ python ch02-chunk-read.py myhugefile.dat
We want to be able to read blocks of lines for processing without reading the whole file in the memory.
We open the file and read in lines in the inner for loop. The way we move through the file is by calling next() on the file object. This function reads the line from the file and moves the file pointer to the next line. We append lines in the file_block variable during the loop execution. In order to simplify the example code, we don't do any processing but just add file_block to complete the output variable readable.
We do some printing during execution just to illustrate the current state of certain variables.
The last comment line in the while loop raw_input() can be uncommented and we can pause the execution and read the printed lines above it.
This recipe is, of course, just one of the possible approaches to reading large (huge) files. Other approaches could include specific Python or C libraries, but they all depend on what we aim to do with data and how we want to process it.
Parallel approaches like the MapReduce paradigm have become very popular recently as we get more processing power and memory for a low price.
Multiprocessing is also a feasible approach sometimes as Python has good library support for creating and managing threads with several libraries such as multiprocessing, threading, and thread.
If processing huge files is a repeated process for a project, we suggest building your data pipeline so that every time you need data ready in a specific format on the output end, you don't have to go to the source and do it manually.