Variables Are Not Boxes

In 1997, I took a summer course on Java at MIT. The professor, Lynn Andrea Stein—an award-winning computer science educator who currently teaches at Olin College of Engineering—made the point that the usual “variables as boxes” metaphor actually hinders the understanding of reference variables in OO languages. Python variables are like reference variables in Java, so it’s better to think of them as labels attached to objects.

Example 8-1 is a simple interaction that the “variables as boxes” idea cannot explain. Figure 8-1 illustrates why the box metaphor is wrong for Python, while sticky notes provide a helpful picture of how variables actually work.

>>> a = [1, 2, 3]
>>> b = a
>>> a.append(4)
>>> b
[1, 2, 3, 4]

Figure 8-1. If you imagine variables are like boxes, you can’t make sense of assignment in Python; instead, think of variables as sticky notes—Example 8-1 then becomes easy to explain

Prof. Stein also spoke about assignment in a very deliberate way. For example, when talking about a seesaw object in a simulation, she would say: “Variable s is assigned to the seesaw,” but never “The seesaw is assigned to variable s.” With reference variables, it makes much more sense to say that the variable is assigned to an object, and not the other way around. After all, the object is created before the assignment. Example 8-2 proves that the righthand side of an assignment happens first.

>>> class Gizmo:
...    def __init__(self):
...         print('Gizmo id: %d' % id(self))
...
>>> x = Gizmo()
Gizmo id: 4301489152  
>>> y = Gizmo() * 10  
Gizmo id: 4301489432  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for *: 'Gizmo' and 'int'
>>>
>>> dir()  
['Gizmo', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__', 'x']

The output Gizmo id: ... is a side effect of creating a Gizmo instance.

Multiplying a Gizmo instance will raise an exception.

Here is proof that a second Gizmo was actually instantiated before the multiplication was attempted.

But variable y was never created, because the exception happened while the right-hand side of the assignment was being evaluated.

Because variables are mere labels, nothing prevents an object from having several labels assigned to it. When that happens, you have aliasing, our next topic.

Identity, Equality, and Aliases

Lewis Carroll is the pen name of Prof. Charles Lutwidge Dodgson. Mr. Carroll is not only equal to Prof. Dodgson: they are one and the same. Example 8-3 expresses this idea in Python.

>>> charles = {'name': 'Charles L. Dodgson', 'born': 1832}
>>> lewis = charles  
>>> lewis is charles
True
>>> id(charles), id(lewis)  
(4300473992, 4300473992)
>>> lewis['balance'] = 950  
>>> charles
{'name': 'Charles L. Dodgson', 'balance': 950, 'born': 1832}

lewis is an alias for charles.

The is operator and the id function confirm it.

Adding an item to lewis is the same as adding an item to charles.

However, suppose an impostor—let’s call him Dr. Alexander Pedachenko—claims he is Charles L. Dodgson, born in 1832. His credentials may be the same, but Dr. Pedachenko is not Prof. Dodgson. Figure 8-2 illustrates this scenario.

Figure 8-2. charles and lewis are bound to the same object; alex is bound to a separate object of equal contents

Example 8-4 implements and tests the alex object depicted in Figure 8-2.

>>> alex = {'name': 'Charles L. Dodgson', 'born': 1832, 'balance': 950}  
>>> alex == charles  
True
>>> alex is not charles  
True

alex refers to an object that is a replica of the object assigned to charles.

The objects compare equal, because of the __eq__ implementation in the dict class.

But they are distinct objects. This is the Pythonic way of writing the negative identity comparison: a is not b.

Example 8-3 is an example of aliasing. In that code, lewis and charles are aliases: two variables bound to the same object. On the other hand, alex is not an alias for charles: these variables are bound to distinct objects. The objects bound to alex and charles have the same value—that’s what == compares—but they have different identities.

In The Python Language Reference, “3.1. Objects, values and types” states:

Every object has an identity, a type and a value. An object’s identity never changes once it has been created; you may think of it as the object’s address in memory. The is operator compares the identity of two objects; the id() function returns an integer representing its identity.

The real meaning of an object’s ID is implementation-dependent. In CPython, id() returns the memory address of the object, but it may be something else in another Python interpreter. The key point is that the ID is guaranteed to be a unique numeric label, and it will never change during the life of the object.

In practice, we rarely use the id() function while programming. Identity checks are most often done with the is operator, and not by comparing IDs. Next, we’ll talk about is versus ==.

x is None

x is not None

>>> t1 = (1, 2, [30, 40])  
>>> t2 = (1, 2, [30, 40])  
>>> t1 == t2  
True
>>> id(t1[-1])  
4302515784
>>> t1[-1].append(99)  
>>> t1
(1, 2, [30, 40, 99])
>>> id(t1[-1])  
4302515784
>>> t1 == t2  
False

Copies Are Shallow by Default

The easiest way to copy a list (or most built-in mutable collections) is to use the built-in constructor for the type itself. For example:

>>> l1 = [3, [55, 44], (7, 8, 9)]
>>> l2 = list(l1)  
>>> l2
[3, [55, 44], (7, 8, 9)]
>>> l2 == l1  
True
>>> l2 is l1  
False

list(l1) creates a copy of l1.

The copies are equal.

But refer to two different objects.

For lists and other mutable sequences, the shortcut l2 = l1[:] also makes a copy.

However, using the constructor or [:] produces a shallow copy (i.e., the outermost container is duplicated, but the copy is filled with references to the same items held by the original container). This saves memory and causes no problems if all the items are immutable. But if there are mutable items, this may lead to unpleasant surprises.

In Example 8-6, we create a shallow copy of a list containing another list and a tuple, and then make changes to see how they affect the referenced objects.

l1 = [3, [66, 55, 44], (7, 8, 9)]
l2 = list(l1)      
l1.append(100)     
l1[1].remove(55)   
print('l1:', l1)
print('l2:', l2)
l2[1] += [33, 22]  
l2[2] += (10, 11)  
print('l1:', l1)
print('l2:', l2)

l2 is a shallow copy of l1. This state is depicted in Figure 8-3.

Appending 100 to l1 has no effect on l2.

Here we remove 55 from the inner list l1[1]. This affects l2 because l2[1] is bound to the same list as l1[1].

For a mutable object like the list referred by l2[1], the operator += changes the list in place. This change is visible at l1[1], which is an alias for l2[1].

+= on a tuple creates a new tuple and rebinds the variable l2[2] here. This is the same as doing l2[2] = l2[2] + (10, 11). Now the tuples in the last position of l1 and l2 are no longer the same object. See Figure 8-4.

The output of Example 8-6 is Example 8-7, and the final state of the objects is depicted in Figure 8-4.

l1: [3, [66, 44], (7, 8, 9), 100]
l2: [3, [66, 44], (7, 8, 9)]
l1: [3, [66, 44, 33, 22], (7, 8, 9), 100]
l2: [3, [66, 44, 33, 22], (7, 8, 9, 10, 11)]

Figure 8-3. Program state immediately after the assignment l2 = list(l1) in Example 8-6. l1 and l2 refer to distinct lists, but the lists share references to the same inner list object [66, 55, 44] and tuple (7, 8, 9). (Diagram generated by the Online Python Tutor.)

Figure 8-4. Final state of l1 and l2: they still share references to the same list object, now containing [66, 44, 33, 22], but the operation l2[2] += (10, 11) created a new tuple with content (7, 8, 9, 10, 11), unrelated to the tuple (7, 8, 9) referenced by l1[2]. (Diagram generated by the Online Python Tutor.)

It should be clear now that shallow copies are easy to make, but they may or may not be what you want. How to make deep copies is our next topic.

class Bus:

    def __init__(self, passengers=None):
        if passengers is None:
            self.passengers = []
        else:
            self.passengers = list(passengers)

    def pick(self, name):
        self.passengers.append(name)

    def drop(self, name):
        self.passengers.remove(name)

>>> import copy
>>> bus1 = Bus(['Alice', 'Bill', 'Claire', 'David'])
>>> bus2 = copy.copy(bus1)
>>> bus3 = copy.deepcopy(bus1)
>>> id(bus1), id(bus2), id(bus3)
(4301498296, 4301499416, 4301499752)  
>>> bus1.drop('Bill')
>>> bus2.passengers
['Alice', 'Claire', 'David']          
>>> id(bus1.passengers), id(bus2.passengers), id(bus3.passengers)
(4302658568, 4302658568, 4302657800)  
>>> bus3.passengers
['Alice', 'Bill', 'Claire', 'David']  

>>> a = [10, 20]
>>> b = [a, 30]
>>> a.append(b)
>>> a
[10, 20, [[...], 30]]
>>> from copy import deepcopy
>>> c = deepcopy(a)
>>> c
[10, 20, [[...], 30]]

Function Parameters as References

The only mode of parameter passing in Python is call by sharing. That is the same mode used in most OO languages, including Ruby, SmallTalk, and Java (this applies to Java reference types; primitive types use call by value). Call by sharing means that each formal parameter of the function gets a copy of each reference in the arguments. In other words, the parameters inside the function become aliases of the actual arguments.

The result of this scheme is that a function may change any mutable object passed as a parameter, but it cannot change the identity of those objects (i.e., it cannot altogether replace an object with another). Example 8-11 shows a simple function using += on one of its parameters. As we pass numbers, lists, and tuples to the function, the actual arguments passed are affected in different ways.

>>> def f(a, b):
...     a += b
...     return a
...
>>> x = 1
>>> y = 2
>>> f(x, y)
3
>>> x, y  
(1, 2)
>>> a = [1, 2]
>>> b = [3, 4]
>>> f(a, b)
[1, 2, 3, 4]
>>> a, b  
([1, 2, 3, 4], [3, 4])
>>> t = (10, 20)
>>> u = (30, 40)
>>> f(t, u)  
(10, 20, 30, 40)
>>> t, u
((10, 20), (30, 40))

The number x is unchanged.

The list a is changed.

The tuple t is unchanged.

Another issue related to function parameters is the use of mutable values for defaults, as discussed next.

class HauntedBus:
    """A bus model haunted by ghost passengers"""

    def __init__(self, passengers=[]):  
        self.passengers = passengers  

    def pick(self, name):
        self.passengers.append(name)  

    def drop(self, name):
        self.passengers.remove(name)

>>> bus1 = HauntedBus(['Alice', 'Bill'])
>>> bus1.passengers
['Alice', 'Bill']
>>> bus1.pick('Charlie')
>>> bus1.drop('Alice')
>>> bus1.passengers  
['Bill', 'Charlie']
>>> bus2 = HauntedBus()  
>>> bus2.pick('Carrie')
>>> bus2.passengers
['Carrie']
>>> bus3 = HauntedBus()  
>>> bus3.passengers  
['Carrie']
>>> bus3.pick('Dave')
>>> bus2.passengers  
['Carrie', 'Dave']
>>> bus2.passengers is bus3.passengers  
True
>>> bus1.passengers  
['Bill', 'Charlie']

>>> dir(HauntedBus.__init__)  # doctest: +ELLIPSIS
['__annotations__', '__call__', ..., '__defaults__', ...]
>>> HauntedBus.__init__.__defaults__
(['Carrie', 'Dave'],)

>>> HauntedBus.__init__.__defaults__[0] is bus2.passengers
True

>>> basketball_team = ['Sue', 'Tina', 'Maya', 'Diana', 'Pat']  
>>> bus = TwilightBus(basketball_team)  
>>> bus.drop('Tina')  
>>> bus.drop('Pat')
>>> basketball_team  
['Sue', 'Maya', 'Diana']

class TwilightBus:
    """A bus model that makes passengers vanish"""

    def __init__(self, passengers=None):
        if passengers is None:
            self.passengers = []  
        else:
            self.passengers = passengers  

    def pick(self, name):
        self.passengers.append(name)

    def drop(self, name):
        self.passengers.remove(name)  

    def __init__(self, passengers=None):
        if passengers is None:
            self.passengers = []
        else:
            self.passengers = list(passengers) 

del and Garbage Collection

Objects are never explicitly destroyed; however, when they become unreachable they may be garbage-collected.

Data Model, chapter of The Python Language Reference

The del statement deletes names, not objects. An object may be garbage collected as result of a del command, but only if the variable deleted holds the last reference to the object, or if the object becomes unreachable.² Rebinding a variable may also cause the number of references to an object to reach zero, causing its destruction.

In CPython, the primary algorithm for garbage collection is reference counting. Essentially, each object keeps count of how many references point to it. As soon as that refcount reaches zero, the object is immediately destroyed: CPython calls the __del__ method on the object (if defined) and then frees the memory allocated to the object. In CPython 2.0, a generational garbage collection algorithm was added to detect groups of objects involved in reference cycles—which may be unreachable even with outstanding references to them, when all the mutual references are contained within the group. Other implementations of Python have more sophisticated garbage collectors that do not rely on reference counting, which means the __del__ method may not be called immediately when there are no more references to the object. See “PyPy, Garbage Collection, and a Deadlock” by A. Jesse Jiryu Davis for discussion of improper and proper use of __del__.

To demonstrate the end of an object’s life, Example 8-16 uses weakref.finalize to register a callback function to be called when an object is destroyed.

>>> import weakref
>>> s1 = {1, 2, 3}
>>> s2 = s1         
>>> def bye():      
...     print('Gone with the wind...')
...
>>> ender = weakref.finalize(s1, bye)  
>>> ender.alive  
True
>>> del s1
>>> ender.alive  
True
>>> s2 = 'spam'  
Gone with the wind...
>>> ender.alive
False

s1 and s2 are aliases referring to the same set, {1, 2, 3}.

This function must not be a bound method of the object about to be destroyed or otherwise hold a reference to it.

Register the bye callback on the object referred by s1.

The .alive attribute is True before the finalize object is called.

As discussed, del does not delete an object, just a reference to it.

Rebinding the last reference, s2, makes {1, 2, 3} unreachable. It is destroyed, the bye callback is invoked, and ender.alive becomes False.

The point of Example 8-16 is to make explicit that del does not delete objects, but objects may be deleted as a consequence of being unreachable after del is used.

You may be wondering why the {1, 2, 3} object was destroyed in Example 8-16. After all, the s1 reference was passed to the finalize function, which must have held on to it in order to monitor the object and invoke the callback. This works because finalize holds a weak reference to {1, 2, 3}, as explained in the next section.

Weak References

The presence of references is what keeps an object alive in memory. When the reference count of an object reaches zero, the garbage collector disposes of it. But sometimes it is useful to have a reference to an object that does not keep it around longer than necessary. A common use case is a cache.

Weak references to an object do not increase its reference count. The object that is the target of a reference is called the referent. Therefore, we say that a weak reference does not prevent the referent from being garbage collected.

Weak references are useful in caching applications because you don’t want the cached objects to be kept alive just because they are referenced by the cache.

Example 8-17 shows how a weakref.ref instance can be called to reach its referent. If the object is alive, calling the weak reference returns it, otherwise None is returned.

>>> import weakref
>>> a_set = {0, 1}
>>> wref = weakref.ref(a_set)  
>>> wref
<weakref at 0x100637598; to 'set' at 0x100636748>
>>> wref()  
{0, 1}
>>> a_set = {2, 3, 4}  
>>> wref()  
{0, 1}
>>> wref() is None  
False
>>> wref() is None  
True

The wref weak reference object is created and inspected in the next line.

Invoking wref() returns the referenced object, {0, 1}. Because this is a console session, the result {0, 1} is bound to the _ variable.

a_set no longer refers to the {0, 1} set, so its reference count is decreased. But the _ variable still refers to it.

Calling wref() still returns {0, 1}.

When this expression is evaluated, {0, 1} lives, therefore wref() is not None. But _ is then bound to the resulting value, False. Now there are no more strong references to {0, 1}.

Because the {0, 1} object is now gone, this last call to wref() returns None.

The weakref module documentation makes the point that the weakref.ref class is actually a low-level interface intended for advanced uses, and that most programs are better served by the use of the weakref collections and finalize. In other words, consider using WeakKeyDictionary, WeakValueDictionary, WeakSet, and finalize (which use weak references internally) instead of creating and handling your own weakref.ref instances by hand. We just did that in Example 8-17 in the hope that showing a single weakref.ref in action could take away some of the mystery around them. But in practice, most of the time Python programs use the weakref collections.

The next subsection briefly discusses the weakref collections.

class Cheese:

    def __init__(self, kind):
        self.kind = kind

    def __repr__(self):
        return 'Cheese(%r)' % self.kind

>>> import weakref
>>> stock = weakref.WeakValueDictionary()  
>>> catalog = [Cheese('Red Leicester'), Cheese('Tilsit'),
...                 Cheese('Brie'), Cheese('Parmesan')]
...
>>> for cheese in catalog:
...     stock[cheese.kind] = cheese  
...
>>> sorted(stock.keys())
['Brie', 'Parmesan', 'Red Leicester', 'Tilsit']  
>>> del catalog
>>> sorted(stock.keys())
['Parmesan']  
>>> del cheese
>>> sorted(stock.keys())
[]

class MyList(list):
    """list subclass whose instances may be weakly referenced"""

a_list = MyList(range(10))

# a_list can be the target of a weak reference
wref_to_a_list = weakref.ref(a_list)

Tricks Python Plays with Immutables

I was surprised to learn that, for a tuple t, t[:] does not make a copy, but returns a reference to the same object. You also get a reference to the same tuple if you write tuple(t).⁵ Example 8-20 proves it.

>>> t1 = (1, 2, 3)
>>> t2 = tuple(t1)
>>> t2 is t1  
True
>>> t3 = t1[:]
>>> t3 is t1  
True

t1 and t2 are bound to the same object.

And so is t3.

The same behavior can be observed with instances of str, bytes, and frozenset. Note that a frozenset is not a sequence, so fs[:] does not work if fs is a frozenset. But fs.copy() has the same effect: it cheats and returns a reference to the same object, and not a copy at all, as Example 8-21 shows.⁶

>>> t1 = (1, 2, 3)
>>> t3 = (1, 2, 3)  
>>> t3 is t1  
False
>>> s1 = 'ABC'
>>> s2 = 'ABC'  
>>> s2 is s1 
True

Creating a new tuple from scratch.

t1 and t3 are equal, but not the same object.

Creating a second str from scratch.

Surprise: a and b refer to the same str!

The sharing of string literals is an optimization technique called interning. CPython uses the same technique with small integers to avoid unnecessary duplication of “popular” numbers like 0, –1, and 42. Note that CPython does not intern all strings or integers, and the criteria it uses to do so is an undocumented implementation detail.

The tricks discussed in this section, including the behavior of frozenset.copy(), are “white lies”; they save memory and make the interpreter faster. Do not worry about them, they should not give you any trouble because they only apply to immutable types. Probably the best use of these bits of trivia is to win bets with fellow Pythonistas.

Chapter Summary

Every Python object has an identity, a type, and a value. Only the value of an object changes over time.⁷

If two variables refer to immutable objects that have equal values (a == b is True), in practice it rarely matters if they refer to copies or are aliases referring to the same object because the value of an immutable object does not change, with one exception. The exception is immutable collections such as tuples and frozensets: if an immutable collection holds references to mutable items, then its value may actually change when the value of a mutable item changes. In practice, this scenario is not so common. What never changes in an immutable collection are the identities of the objects within.

The fact that variables hold references has many practical consequences in Python programming:

• Simple assignment does not create copies.

• Augmented assignment with += or *= creates new objects if the lefthand variable is bound to an immutable object, but may modify a mutable object in place.

• Assigning a new value to an existing variable does not change the object previously bound to it. This is called a rebinding: the variable is now bound to a different object. If that variable was the last reference to the previous object, that object will be garbage collected.

• Function parameters are passed as aliases, which means the function may change any mutable object received as an argument. There is no way to prevent this, except making local copies or using immutable objects (e.g., passing a tuple instead of a list).

• Using mutable objects as default values for function parameters is dangerous because if the parameters are changed in place, then the default is changed, affecting every future call that relies on the default.

In CPython, objects are discarded as soon as the number of references to them reaches zero. They may also be discarded if they form groups with cyclic references but no outside references. In some situations, it may be useful to hold a reference to an object that will not—by itself—keep an object alive. One example is a class that wants to keep track of all its current instances. This can be done with weak references, a low-level mechanism underlying the more useful collections WeakValueDictionary, WeakKeyDictionary, WeakSet, and the finalize function from the weakref module.

Further Reading

The “Data Model” chapter of The Python Language Reference starts with a clear explanation of object identities and values.

Wesley Chun, author of the Core Python series of books, made a great presentation about many of the topics covered in this chapter during OSCON 2013. You can download the slides from the “Python 103: Memory Model & Best Practices” talk page. There is also a YouTube video of a longer presentation Wesley gave at EuroPython 2011, covering not only the theme of this chapter but also the use of special methods.

Doug Hellmann wrote a long series of excellent blog posts titled Python Module of the Week, which became a book, The Python Standard Library by Example. His posts “copy – Duplicate Objects” and “weakref – Garbage-Collectable References to Objects” cover some of the topics we just discussed.

More information on the CPython generational garbage collector can be found in the gc module documentation, which starts with the sentence “This module provides an interface to the optional garbage collector.” The “optional” qualifier here may be surprising, but the “Data Model” chapter also states:

An implementation is allowed to postpone garbage collection or omit it altogether—it is a matter of implementation quality how garbage collection is implemented, as long as no objects are collected that are still reachable.

Fredrik Lundh—creator of key libraries like ElementTree, Tkinter, and the PIL image library—has a short post about the Python garbage collector titled “How Does Python Manage Memory?” He emphasizes that the garbage collector is an implementation feature that behaves differently across Python interpreters. For example, Jython uses the Java garbage collector.

The CPython 3.4 garbage collector improved handling of objects with a __del__ method, as described in PEP 442 — Safe object finalization.

Wikipedia has an article about string interning, mentioning the use of this technique in several languages, including Python.