Probably the biggest limitation of DBM keyed files is in
what they can store: data stored under a key must be a simple text
string. If you want to store Python objects in a DBM file, you can
sometimes manually convert them to and from strings on writes and
reads (e.g., with str
and eval
calls), but this takes you only so far.
For arbitrarily complex Python objects such as class instances and
nested data structures, you need something more. Class instance
objects, for example, cannot be later re-created from their standard
string representations. Custom to-string conversions are error prone
and not general.
The Python pickle
module, a
standard part of the Python system, provides the conversion step
needed. It converts nearly arbitrary Python in-memory objects to and
from a single linear string format, suitable for storing in flat
files, shipping across network sockets between trusted sources, and so
on. This conversion from object to string is often called
serialization—arbitrary data structures in memory
are mapped to a serial string form.
The string representation used for objects is also sometimes referred to as a byte stream, due to its linear format. It retains all the content and references structure of the original in-memory object. When the object is later re-created from its byte string, it will be a new in-memory object identical in structure and value to the original, though located at a different memory address. The re-created object is effectively a copy of the original.
Pickling works on almost any Python datatype—numbers, lists, dictionaries, class instances, nested structures, and more—and so is a general way to store data. Because pickles contain native Python objects, there is almost no database API to be found; the objects stored are processed with normal Python syntax when they are later retrieved.
Pickling may sound complicated the first time you encounter it, but the good news is that Python hides all the complexity of object-to-string conversion. In fact, the pickle module ’s interfaces are incredibly simple to use. For example, to pickle an object into a serialized string, we can either make a pickler and call its methods or use convenience functions in the module to achieve the same effect:
P = pickle.Pickler(
file
)
Make a new pickler for pickling to an open output file
object file
.
P.dump(
object
)
Write an object onto the pickler’s file/stream.
pickle.dump(
object, file
)
Same as the last two calls combined: pickle an object onto an open file.
string = pickle.dumps(
object
)
Return the pickled representation of object
as a character string.
Unpickling from a serialized string back to the original object is similar—both object and convenience function interfaces are available:
U = pickle.Unpickler(
file
)
Make an unpickler for unpickling from an open input file
object file
.
object = U.load(
)
Read an object from the unpickler’s file/stream.
object = pickle.load(
file
)
Same as the last two calls combined: unpickle an object from an open file.
object = pickle.loads(
string
)
Read an object from a character string rather than a file.
Pickler
and Unpickler
are exported classes. In all of
the preceding cases, file
is
either an open file object or any object that implements the same
attributes as file objects:
Pickler
calls the
file’s write
method with a
string argument.
Unpickler
calls the
file’s read
method with a
byte count, and readline
without arguments.
Any object that provides these attributes can be passed in to
the file
parameters. In
particular, file
can be an
instance of a Python class that provides the read/write methods
(i.e., the expected file-like interface). This
lets you map pickled streams to in-memory objects with classes, for
arbitrary use. For instance, the StringIO
standard library module discussed
in Chapter 3 provides classes
that map file calls to and from in-memory strings.
This hook also lets you ship Python objects across a network, by providing sockets wrapped to look like files in pickle calls at the sender, and unpickle calls at the receiver (see the sidebar "Making Sockets Look Like Files,” in Chapter 13, for more details). In fact, for some, pickling Python objects across a trusted network serves as a simpler alternative to network transport protocols such as SOAP and XML-RPC; provided that Python is on both ends of the communication (pickled objects are represented with a Python-specific format, not with XML text).
In more typical use, to pickle an object to a flat
file, we just open the file in write mode and call the dump
function:
%python
>>>table = {'a': [1, 2, 3],
'b': ['spam', 'eggs'],
'c': {'name':'bob'}}
>>> >>>import pickle
>>>mydb = open('dbase', 'w')
>>>pickle.dump(table, mydb)
Notice the nesting in the object pickled here—the pickler
handles arbitrary structures. To unpickle later in another session
or program run, simply reopen the file and call load
:
%python
>>>import pickle
>>>mydb = open('dbase', 'r')
>>>table = pickle.load(mydb)
>>>table
{'b': ['spam', 'eggs'], 'a': [1, 2, 3], 'c': {'name': 'bob'}}
The object you get back from unpickling has the same value and
reference structure as the original, but it is located at a
different address in memory. This is true whether the object is
unpickled in the same or a future process. In Python-speak, the
unpickled object is ==
but is not
is
:
%python
>>>import pickle
>>>f = open('temp', 'w')
>>>x = ['Hello', ('pickle', 'world')]
# list with nested tuple >>>pickle.dump(x, f)
>>>f.close( )
# close to flush changes >>> >>>f = open('temp', 'r')
>>>y = pickle.load(f)
>>>y
['Hello', ('pickle', 'world')] >>> >>>x == y, x is y
(True, False)
To make this process simpler still, the module in Example 19-1 wraps pickling and unpickling calls in functions that also open the files where the serialized form of the object is stored.
Example 19-1. PP3EDbasefilepickle.py
import pickle def saveDbase(filename, object): file = open(filename, 'w') pickle.dump(object, file) # pickle to file file.close( ) # any file-like object will do def loadDbase(filename): file = open(filename, 'r') object = pickle.load(file) # unpickle from file file.close( ) # re-creates object in memory return object
To store and fetch now, simply call these module functions;
here they are in action managing a fairly complex structure with
multiple references to the same nested object—the nested list called
L
at first is stored only once in
the file:
C:...PP3EDbase>python
>>>from filepickle import *
>>>L = [0]
>>>D = {'x':0, 'y':L}
>>>table = {'A':L, 'B':D}
# L appears twice >>>saveDbase('myfile', table)
# serialize to file C:...PP3EDbase>python
>>>from filepickle import *
>>>table = loadDbase('myfile')
# reload/unpickle >>>table
{'B': {'x': 0, 'y': [0]}, 'A': [0]} >>>table['A'][0] = 1
# change shared object >>>saveDbase('myfile', table)
# rewrite to the file C:...PP3EDbase>python
>>>from filepickle import *
>>>print loadDbase('myfile')
# both L's updated as expected {'B': {'x': 0, 'y': [1]}, 'A': [1]}
Besides built-in types like the lists, tuples, and dictionaries of the examples so far, class instances may also be pickled to file-like objects. This provides a natural way to associate behavior with stored data (class methods process instance attributes) and provides a simple migration path (class changes made in module files are automatically picked up by stored instances). Here’s a brief interactive demonstration:
>>>class Rec:
def _ _init_ _(self, hours):
self.hours = hours
def pay(self, rate=50):
return self.hours * rate
>>>bob = Rec(40)
>>>import pickle
>>>pickle.dump(bob, open('bobrec', 'w'))
>>> >>>rec = pickle.load(open('bobrec'))
>>>rec.hours
40 >>>rec.pay( )
2000
We’ll explore how this works in more detail in conjunction
with shelves later in this chapter—as we’ll see, although the
pickle
module can be used
directly, it is also the underlying translation engine in both
shelves and ZODB databases.
In fact, Python can pickle just about anything, except for:
Compiled code objects; functions and classes record just their names in pickles, to allow for later reimport and automatic acquisition of changes made in module files.
Instances of classes that do not follow class importability rules (more on this at the end of the section "Shelve Files,” later in this chapter).
Instances of some built-in and user-defined types that are coded in C or depend upon transient operating system states (e.g., open file objects cannot be pickled).
A PicklingError
is raised
if an object cannot be pickled.
In recent Python releases, the pickler introduced the notion of protocols—storage formats for pickled data. Specify the desired protocol by passing an extra parameter to the pickling calls (but not to unpickling calls: the protocol is automatically determined from the pickled data):
pickle.dump(object, file,protocol
)
Pickled data may be created in either text or binary protocols. By default, the storage protocol is text (also known as protocol 0). In text mode, the files used to store pickled objects may be opened in text mode as in the earlier examples, and the pickled data is printable ASCII text, which can be read (it’s essentially instructions for a stack machine).
The alternative protocols (protocols 1 and 2) store the
pickled data in binary format and require that files be opened in
binary mode (e.g., rb
, wb
). Protocol 1 is the original binary
format; protocol 2, added in Python 2.3, has improved support for
pickling of new-style classes. Binary format is slightly more
efficient, but it cannot be inspected. An older option to pickling
calls, the bin
argument, has been
subsumed by using a pickling protocol higher than 0. The pickle
module also provides a HIGHEST_PROTOCOL
variable that can be
passed in to automatically select the maximum value.
One note: if you use the default text protocol, make sure you open pickle files in text mode later. On some platforms, opening text data in binary mode may cause unpickling errors due to line-end formats on Windows:
>>>f = open('temp', 'w')
# text mode file on Windows >>>pickle.dump(('ex', 'parrot'), f)
# use default text protocol >>>f.close( )
>>> >>>pickle.load(open('temp', 'r'))
# OK in text mode ('ex', 'parrot') >>>pickle.load(open('temp', 'rb'))
# fails in binary Traceback (most recent call last): File "<pyshell#337>", line 1, in -toplevel- pickle.load(open('temp', 'rb')) ...lines deleted... ValueError: insecure string pickle
One way to sidestep this potential issue is to always use binary mode for your files, even for the text pickle protocol. Since you must open files in binary mode for the binary pickler protocols anyhow (higher than the default 0), this isn’t a bad habit to get into:
>>>f = open('temp', 'wb')
# create in binary mode >>>pickle.dump(('ex', 'parrot'), f)
# use text protocol >>>f.close( )
>>> >>>pickle.load(open('temp', 'rb'))
('ex', 'parrot') >>>pickle.load(open('temp', 'r'))
('ex', 'parrot')
Refer to Python’s library manual for more information on the
pickler. Also check out marshal
,
a module that serializes an object too, but can handle only simple
object types. pickle
is more
general than marshal
and is
normally preferred.
And while you are flipping (or clicking) through that manual,
be sure to also see the entries for the cPickle
module—a reimplementation of
pickle
coded in C for faster
performance. You can explicitly import cPickle
for a substantial speed boost; its
chief limitation is that you cannot subclass its versions of
Pickle
and Unpickle
because they are functions, not
classes (this is not required by most programs). The pickle
and cPickle
modules use compatible data
formats, so they may be used interchangeably.
If it is available in your Python, the shelve
module automatically chooses the cPickle
module for faster serialization,
instead of pickle
. I haven’t
explained shelve
yet, but I will
now.
18.117.189.228