Python uses whitespace (tabs or spaces) to structure code instead of using braces as in many other languages like R, C++, Java, and Perl. Take the for loop in the above quicksort algorithm:

for x in array:
    if x < pivot:
        less.append(x)
    else:
        greater.append(x)

A colon denotes the start of an indented code block after which all of the code must be indented by the same amount until the end of the block. In another language, you might instead have something like:

for x in array {
        if x < pivot {
            less.append(x)
        } else {
            greater.append(x)
        }
    }

One major reason that whitespace matters is that it results in most Python code looking cosmetically similar, which means less cognitive dissonance when you read a piece of code that you didn’t write yourself (or wrote in a hurry a year ago!). In a language without significant whitespace, you might stumble on some differently formatted code like:

for x in array
    {
      if x < pivot
      {
        less.append(x)
      }
      else
      {
        greater.append(x)
      }
    }

Love it or hate it, significant whitespace is a fact of life for Python programmers, and in my experience it helps make Python code a lot more readable than other languages I’ve used. While it may seem foreign at first, I suspect that it will grow on you after a while.

As you can see by now, Python statements also do not need to be terminated by semicolons. Semicolons can be used, however, to separate multiple statements on a single line:

a = 5; b = 6; c = 7

Putting multiple statements on one line is generally discouraged in Python as it often makes code less readable.

An important characteristic of the Python language is the consistency of its object model. Every number, string, data structure, function, class, module, and so on exists in the Python interpreter in its own “box” which is referred to as a Python object. Each object has an associated type (for example, string or function) and internal data. In practice this makes the language very flexible, as even functions can be treated just like any other object.

Any text preceded by the hash mark (pound sign) # is ignored by the Python interpreter. This is often used to add comments to code. At times you may also want to exclude certain blocks of code without deleting them. An easy solution is to comment out the code:

results = []
for line in file_handle:
    # keep the empty lines for now
    # if len(line) == 0:
    #   continue
    results.append(line.replace('foo', 'bar'))

Functions are called using parentheses and passing zero or more arguments, optionally assigning the returned value to a variable:

result = f(x, y, z)
g()

Almost every object in Python has attached functions, known as methods, that have access to the object’s internal contents. They can be called using the syntax:

obj.some_method(x, y, z)

Functions can take both positional and keyword arguments:

result = f(a, b, c, d=5, e='foo')

More on this later.

When assigning a variable (or name) in Python, you are creating a reference to the object on the right hand side of the equals sign. In practical terms, consider a list of integers:

In [241]: a = [1, 2, 3]

Suppose we assign a to a new variable b:

In [242]: b = a

In some languages, this assignment would cause the data [1, 2, 3] to be copied. In Python, a and b actually now refer to the same object, the original list [1, 2, 3] (see Figure A-1 for a mockup). You can prove this to yourself by appending an element to a and then examining b:

In [243]: a.append(4)

In [244]: b
Out[244]: [1, 2, 3, 4]

Figure A-1. Two references for the same object

Understanding the semantics of references in Python and when, how, and why data is copied is especially critical when working with larger data sets in Python.

When you pass objects as arguments to a function, you are only passing references; no copying occurs. Thus, Python is said to pass by reference, whereas some other languages support both pass by value (creating copies) and pass by reference. This means that a function can mutate the internals of its arguments. Suppose we had the following function:

def append_element(some_list, element):
    some_list.append(element)

Then given what’s been said, this should not come as a surprise:

In [2]: data = [1, 2, 3]

In [3]: append_element(data, 4)

In [4]: data
Out[4]: [1, 2, 3, 4]

In contrast with many compiled languages, such as Java and C++, object references in Python have no type associated with them. There is no problem with the following:

In [245]: a = 5        In [246]: type(a)
                       Out[246]: int    
                                        
In [247]: a = 'foo'    In [248]: type(a)
                       Out[248]: str

Variables are names for objects within a particular namespace; the type information is stored in the object itself. Some observers might hastily conclude that Python is not a “typed language”. This is not true; consider this example:

In [249]: '5' + 5
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-249-f9dbf5f0b234> in <module>()
----> 1 '5' + 5
TypeError: cannot concatenate 'str' and 'int' objects

In some languages, such as Visual Basic, the string '5' might get implicitly converted (or casted) to an integer, thus yielding 10. Yet in other languages, such as JavaScript, the integer 5 might be casted to a string, yielding the concatenated string '55'. In this regard Python is considered a strongly-typed language, which means that every object has a specific type (or class), and implicit conversions will occur only in certain obvious circumstances, such as the following:

In [250]: a = 4.5

In [251]: b = 2

# String formatting, to be visited later
In [252]: print 'a is %s, b is %s' % (type(a), type(b))
a is <type 'float'>, b is <type 'int'>

In [253]: a / b
Out[253]: 2.25

Knowing the type of an object is important, and it’s useful to be able to write functions that can handle many different kinds of input. You can check that an object is an instance of a particular type using the isinstance function:

In [254]: a = 5        In [255]: isinstance(a, int)
                       Out[255]: True

isinstance can accept a tuple of types if you want to check that an object’s type is among those present in the tuple:

In [256]: a = 5; b = 4.5

In [257]: isinstance(a, (int, float))      In [258]: isinstance(b, (int, float))
Out[257]: True                             Out[258]: True

Objects in Python typically have both attributes, other Python objects stored “inside” the object, and methods, functions associated with an object which can have access to the object’s internal data. Both of them are accessed via the syntax obj.attribute_name:

In [1]: a = 'foo'

In [2]: a.<Tab>
a.capitalize  a.format      a.isupper     a.rindex      a.strip
a.center      a.index       a.join        a.rjust       a.swapcase
a.count       a.isalnum     a.ljust       a.rpartition  a.title
a.decode      a.isalpha     a.lower       a.rsplit      a.translate
a.encode      a.isdigit     a.lstrip      a.rstrip      a.upper
a.endswith    a.islower     a.partition   a.split       a.zfill
a.expandtabs  a.isspace     a.replace     a.splitlines
a.find        a.istitle     a.rfind       a.startswith

Attributes and methods can also be accessed by name using the getattr function:

>>> getattr(a, 'split')
<function split>

While we will not extensively use the functions getattr and related functions hasattr and setattr in this book, they can be used very effectively to write generic, reusable code.

Often you may not care about the type of an object but rather only whether it has certain methods or behavior. For example, you can verify that an object is iterable if it implemented the iterator protocol. For many objects, this means it has a __iter__ “magic method”, though an alternative and better way to check is to try using the iter function:

def isiterable(obj):
    try:
        iter(obj)
        return True
    except TypeError: # not iterable
        return False

This function would return True for strings as well as most Python collection types:

In [260]: isiterable('a string')        In [261]: isiterable([1, 2, 3])
Out[260]: True                          Out[261]: True                 
                                                                       
In [262]: isiterable(5)
Out[262]: False

A place where I use this functionality all the time is to write functions that can accept multiple kinds of input. A common case is writing a function that can accept any kind of sequence (list, tuple, ndarray) or even an iterator. You can first check if the object is a list (or a NumPy array) and, if it is not, convert it to be one:

if not isinstance(x, list) and isiterable(x):
    x = list(x)

In Python a module is simply a .py file containing function and variable definitions along with such things imported from other .py files. Suppose that we had the following module:

# some_module.py
PI = 3.14159

def f(x):
    return x + 2

def g(a, b):
    return a + b

If we wanted to access the variables and functions defined in some_module.py, from another file in the same directory we could do:

import some_module
result = some_module.f(5)
pi = some_module.PI

Or equivalently:

from some_module import f, g, PI
result = g(5, PI)

By using the as keyword you can give imports different variable names:

import some_module as sm
from some_module import PI as pi, g as gf

r1 = sm.f(pi)
r2 = gf(6, pi)

Most of the binary math operations and comparisons are as you might expect:

In [263]: 5 - 7        In [264]: 12 + 21.5
Out[263]: -2           Out[264]: 33.5     
                                          
In [265]: 5 <= 2
Out[265]: False

See Table A-1 for all of the available binary operators.

To check if two references refer to the same object, use the is keyword. is not is also perfectly valid if you want to check that two objects are not the same:

In [266]: a = [1, 2, 3]

In [267]: b = a

# Note, the list function always creates a new list
In [268]: c = list(a)

In [269]: a is b        In [270]: a is not c
Out[269]: True          Out[270]: True

Note this is not the same thing is comparing with ==, because in this case we have:

In [271]: a == c
Out[271]: True

A very common use of is and is not is to check if a variable is None, since there is only one instance of None:

In [272]: a = None

In [273]: a is None
Out[273]: True

When using any programming language, it’s important to understand when expressions are evaluated. Consider the simple expression:

a = b = c = 5
d = a + b * c

In Python, once these statements are evaluated, the calculation is immediately (or strictly) carried out, setting the value of d to 30. In another programming paradigm, such as in a pure functional programming language like Haskell, the value of d might not be evaluated until it is actually used elsewhere. The idea of deferring computations in this way is commonly known as lazy evaluation. Python, on the other hand, is a very strict (or eager) language. Nearly all of the time, computations and expressions are evaluated immediately. Even in the above simple expression, the result of b * c is computed as a separate step before adding it to a.

There are Python techniques, especially using iterators and generators, which can be used to achieve laziness. When performing very expensive computations which are only necessary some of the time, this can be an important technique in data-intensive applications.

Most objects in Python are mutable, such as lists, dicts, NumPy arrays, or most user-defined types (classes). This means that the object or values that they contain can be modified.

In [274]: a_list = ['foo', 2, [4, 5]]

In [275]: a_list[2] = (3, 4)

In [276]: a_list
Out[276]: ['foo', 2, (3, 4)]

Others, like strings and tuples, are immutable:

In [277]: a_tuple = (3, 5, (4, 5))

In [278]: a_tuple[1] = 'four'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-278-b7966a9ae0f1> in <module>()
----> 1 a_tuple[1] = 'four'
TypeError: 'tuple' object does not support item assignment

Remember that just because you can mutate an object does not mean that you always should. Such actions are known in programming as side effects. For example, when writing a function, any side effects should be explicitly communicated to the user in the function’s documentation or comments. If possible, I recommend trying to avoid side effects and favor immutability, even though there may be mutable objects involved.

The primary Python types for numbers are int and float. The size of the integer which can be stored as an int is dependent on your platform (whether 32 or 64-bit), but Python will transparently convert a very large integer to long, which can store arbitrarily large integers.

In [279]: ival = 17239871

In [280]: ival ** 6
Out[280]: 26254519291092456596965462913230729701102721L

Floating point numbers are represented with the Python float type. Under the hood each one is a double-precision (64 bits) value. They can also be expressed using scientific notation:

In [281]: fval = 7.243

In [282]: fval2 = 6.78e-5

In Python 3, integer division not resulting in a whole number will always yield a floating point number:

In [284]: 3 / 2
Out[284]: 1.5

In Python 2.7 and below (which some readers will likely be using), you can enable this behavior by default by putting the following cryptic-looking statement at the top of your module:

from __future__ import division

Without this in place, you can always explicitly convert the denominator into a floating point number:

In [285]: 3 / float(2)
Out[285]: 1.5

To get C-style integer division (which drops the fractional part if the result is not a whole number), use the floor division operator //:

In [286]: 3 // 2
Out[286]: 1

Complex numbers are written using j for the imaginary part:

In [287]: cval = 1 + 2j

In [288]: cval * (1 - 2j)
Out[288]: (5+0j)

Many people use Python for its powerful and flexible built-in string processing capabilities. You can write string literal using either single quotes ' or double quotes ":

a = 'one way of writing a string'
b = "another way"

For multiline strings with line breaks, you can use triple quotes, either ''' or """:

c = """
This is a longer string that
spans multiple lines
"""

Python strings are immutable; you cannot modify a string without creating a new string:

In [289]: a = 'this is a string'

In [290]: a[10] = 'f'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-290-5ca625d1e504> in <module>()
----> 1 a[10] = 'f'
TypeError: 'str' object does not support item assignment

In [291]: b = a.replace('string', 'longer string')

In [292]: b
Out[292]: 'this is a longer string'

Many Python objects can be converted to a string using the str function:

In [293]: a = 5.6        In [294]: s = str(a)
                                             
In [295]: s
Out[295]: '5.6'

Strings are a sequence of characters and therefore can be treated like other sequences, such as lists and tuples:

In [296]: s = 'python'        In [297]: list(s)                       
                              Out[297]: ['p', 'y', 't', 'h', 'o', 'n']
                                                                      
In [298]: s[:3]
Out[298]: 'pyt'

The backslash character is an escape character, meaning that it is used to specify special characters like newline or unicode characters. To write a string literal with backslashes, you need to escape them:

In [299]: s = '12\34'

In [300]: print s
1234

If you have a string with a lot of backslashes and no special characters, you might find this a bit annoying. Fortunately you can preface the leading quote of the string with r which means that the characters should be interpreted as is:

In [301]: s = r'thishas
ospecialcharacters'

In [302]: s
Out[302]: 'this\has\no\special\characters'

Adding two strings together concatenates them and produces a new string:

In [303]: a = 'this is the first half '

In [304]: b = 'and this is the second half'

In [305]: a + b
Out[305]: 'this is the first half and this is the second half'

String templating or formatting is another important topic. The number of ways to do so has expanded with the advent of Python 3, here I will briefly describe the mechanics of one of the main interfaces. Strings with a % followed by one or more format characters is a target for inserting a value into that string (this is quite similar to the printf function in C). As an example, consider this string:

In [306]: template = '%.2f %s are worth $%d'

In this string, %s means to format an argument as a string, %.2f a number with 2 decimal places, and %d an integer. To substitute arguments for these format parameters, use the binary operator % with a tuple of values:

In [307]: template % (4.5560, 'Argentine Pesos', 1)
Out[307]: '4.56 Argentine Pesos are worth $1'

String formatting is a broad topic; there are multiple methods and numerous options and tweaks available to control how values are formatted in the resulting string. To learn more, I recommend you seek out more information on the web.

I discuss general string processing as it relates to data analysis in more detail in Chapter 7.

The two boolean values in Python are written as True and False. Comparisons and other conditional expressions evaluate to either True or False. Boolean values are combined with the and and or keywords:

In [308]: True and True
Out[308]: True

In [309]: False or True
Out[309]: True

Almost all built-in Python tops and any class defining the __nonzero__ magic method have a True or False interpretation in an if statement:

In [310]: a = [1, 2, 3]
   .....: if a:
   .....:     print 'I found something!'
   .....:
I found something!

In [311]: b = []
   .....: if not b:
   .....:     print 'Empty!'
   .....:
Empty!

Most objects in Python have a notion of true- or falseness. For example, empty sequences (lists, dicts, tuples, etc.) are treated as False if used in control flow (as above with the empty list b). You can see exactly what boolean value an object coerces to by invoking bool on it:

In [312]: bool([]), bool([1, 2, 3])
Out[312]: (False, True)

In [313]: bool('Hello world!'), bool('')
Out[313]: (True, False)

In [314]: bool(0), bool(1)
Out[314]: (False, True)

The str, bool, int and float types are also functions which can be used to cast values to those types:

In [315]: s = '3.14159'

In [316]: fval = float(s)        In [317]: type(fval)
                                 Out[317]: float     
                                                     
In [318]: int(fval)        In [319]: bool(fval)        In [320]: bool(0)
Out[318]: 3                Out[319]: True              Out[320]: False

None is the Python null value type. If a function does not explicitly return a value, it implicitly returns None.

In [321]: a = None      In [322]: a is None
                        Out[322]: True     
                                           
In [323]: b = 5         In [324]: b is not None
                        Out[324]: True

None is also a common default value for optional function arguments:

def add_and_maybe_multiply(a, b, c=None):
	result = a + b

	if c is not None:
		result = result * c

	return result

While a technical point, it’s worth bearing in mind that None is not a reserved keyword but rather a unique instance of NoneType.

The built-in Python datetime module provides datetime, date, and time types. The datetime type as you may imagine combines the information stored in date and time and is the most commonly used:

In [325]: from datetime import datetime, date, time

In [326]: dt = datetime(2011, 10, 29, 20, 30, 21)

In [327]: dt.day    In [328]: dt.minute
Out[327]: 29        Out[328]: 30

Given a datetime instance, you can extract the equivalent date and time objects by calling methods on the datetime of the same name:

In [329]: dt.date()                      In [330]: dt.time()                
Out[329]: datetime.date(2011, 10, 29)    Out[330]: datetime.time(20, 30, 21)

The strftime method formats a datetime as a string:

In [331]: dt.strftime('%m/%d/%Y %H:%M')
Out[331]: '10/29/2011 20:30'

Strings can be converted (parsed) into datetime objects using the strptime function:

In [332]: datetime.strptime('20091031', '%Y%m%d')
Out[332]: datetime.datetime(2009, 10, 31, 0, 0)

See Table 10-2 for a full list of format specifications.

When aggregating or otherwise grouping time series data, it will occasionally be useful to replace fields of a series of datetimes, for example replacing the minute and second fields with zero, producing a new object:

In [333]: dt.replace(minute=0, second=0)
Out[333]: datetime.datetime(2011, 10, 29, 20, 0)

The difference of two datetime objects produces a datetime.timedelta type:

In [334]: dt2 = datetime(2011, 11, 15, 22, 30)

In [335]: delta = dt2 - dt

In [336]: delta                             In [337]: type(delta)       
Out[336]: datetime.timedelta(17, 7179)      Out[337]: datetime.timedelta

Adding a timedelta to a datetime produces a new shifted datetime:

In [338]: dt
Out[338]: datetime.datetime(2011, 10, 29, 20, 30, 21)

In [339]: dt + delta
Out[339]: datetime.datetime(2011, 11, 15, 22, 30)

The if statement is one of the most well-known control flow statement types. It checks a condition which, if True, evaluates the code in the block that follows:

if x < 0:
    print 'It's negative'

An if statement can be optionally followed by one or more elif blocks and a catch-all else block if all of the conditions are False:

if x < 0:
    print 'It's negative'
elif x == 0:
    print 'Equal to zero'
elif 0 < x < 5:
    print 'Positive but smaller than 5'
else:
    print 'Positive and larger than or equal to 5'

If any of the conditions is True, no further elif or else blocks will be reached. With a compound condition using and or or, conditions are evaluated left-to-right and will short circuit:

In [340]: a = 5; b = 7

In [341]: c = 8; d = 4

In [342]: if a < b or c > d:
   .....:     print 'Made it'
Made it

In this example, the comparison c > d never gets evaluated because the first comparison was True.

for loops are for iterating over a collection (like a list or tuple) or an iterater. The standard syntax for a for loop is:

for value in collection:
    # do something with value

A for loop can be advanced to the next iteration, skipping the remainder of the block, using the continue keyword. Consider this code which sums up integers in a list and skips None values:

sequence = [1, 2, None, 4, None, 5]
total = 0
for value in sequence:
    if value is None:
        continue
    total += value

A for loop can be exited altogether using the break keyword. This code sums elements of the list until a 5 is reached:

sequence = [1, 2, 0, 4, 6, 5, 2, 1]
total_until_5 = 0
for value in sequence:
    if value == 5:
        break
    total_until_5 += value

As we will see in more detail, if the elements in the collection or iterator are sequences (tuples or lists, say), they can be conveniently unpacked into variables in the for loop statement:

for a, b, c in iterator:
    # do something

A while loop specifies a condition and a block of code that is to be executed until the condition evaluates to False or the loop is explicitly ended with break:

x = 256
total = 0
while x > 0:
    if total > 500:
        break
    total += x
    x = x // 2

pass is the “no-op” statement in Python. It can be used in blocks where no action is to be taken; it is only required because Python uses whitespace to delimit blocks:

if x < 0:
    print 'negative!'
elif x == 0:
    # TODO: put something smart here
    pass
else:
    print 'positive!'

It’s common to use pass as a place-holder in code while working on a new piece of functionality:

def f(x, y, z):
    # TODO: implement this function!
    pass

Handling Python errors or exceptions gracefully is an important part of building robust programs. In data analysis applications, many functions only work on certain kinds of input. As an example, Python’s float function is capable of casting a string to a floating point number, but fails with ValueError on improper inputs:

In [343]: float('1.2345')
Out[343]: 1.2345

In [344]: float('something')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-344-439904410854> in <module>()
----> 1 float('something')
ValueError: could not convert string to float: something

Suppose we wanted a version of float that fails gracefully, returning the input argument. We can do this by writing a function that encloses the call to float in a try/except block:

def attempt_float(x):
    try:
        return float(x)
    except:
        return x

The code in the except part of the block will only be executed if float(x) raises an exception:

In [346]: attempt_float('1.2345')
Out[346]: 1.2345

In [347]: attempt_float('something')
Out[347]: 'something'

You might notice that float can raise exceptions other than ValueError:

In [348]: float((1, 2))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-348-842079ebb635> in <module>()
----> 1 float((1, 2))
TypeError: float() argument must be a string or a number

You might want to only suppress ValueError, since a TypeError (the input was not a string or numeric value) might indicate a legitimate bug in your program. To do that, write the exception type after except:

def attempt_float(x):
    try:
        return float(x)
    except ValueError:
        return x

We have then:

In [350]: attempt_float((1, 2))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-350-9bdfd730cead> in <module>()
----> 1 attempt_float((1, 2))
<ipython-input-349-3e06b8379b6b> in attempt_float(x)
      1 def attempt_float(x):
      2     try:
----> 3         return float(x)
      4     except ValueError:
      5         return x
TypeError: float() argument must be a string or a number

You can catch multiple exception types by writing a tuple of exception types instead (the parentheses are required):

def attempt_float(x):
    try:
        return float(x)
    except (TypeError, ValueError):
        return x

In some cases, you may not want to suppress an exception, but you want some code to be executed regardless of whether the code in the try block succeeds or not. To do this, use finally:

f = open(path, 'w')

try:
    write_to_file(f)
finally:
    f.close()

Here, the file handle f will always get closed. Similarly, you can have code that executes only if the try: block succeeds using else:

f = open(path, 'w')

try:
    write_to_file(f)
except:
    print 'Failed'
else:
    print 'Succeeded'
finally:
    f.close()

The range function produces a list of evenly-spaced integers:

In [352]: range(10)
Out[352]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Both a start, end, and step can be given:

In [353]: range(0, 20, 2)
Out[353]: [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

As you can see, range produces integers up to but not including the endpoint. A common use of range is for iterating through sequences by index:

seq = [1, 2, 3, 4]
for i in range(len(seq)):
    val = seq[i]

For very long ranges, it’s recommended to use xrange, which takes the same arguments as range but returns an iterator that generates integers one by one rather than generating all of them up-front and storing them in a (potentially very large) list. This snippet sums all numbers from 0 to 9999 that are multiples of 3 or 5:

sum = 0
for i in xrange(10000):
    # % is the modulo operator
    if i % 3 == 0 or i % 5 == 0:
        sum += i

A ternary expression in Python allows you combine an if-else block which produces a value into a single line or expression. The syntax for this in Python is

value = true-expr if condition else
false-expr

Here, true-expr and false-expr can be any Python expressions. It has the identical effect as the more verbose

if condition:
    value = true-expr
else:
    value = false-expr

This is a more concrete example:

In [354]: x = 5

In [355]: 'Non-negative' if x >= 0 else 'Negative'
Out[355]: 'Non-negative'

As with if-else blocks, only one of the expressions will be evaluated. While it may be tempting to always use ternary expressions to condense your code, realize that you may sacrifice readability if the condition as well and the true and false expressions are very complex.

If you try to assign to a tuple-like expression of variables, Python will attempt to unpack the value on the right-hand side of the equals sign:

In [370]: tup = (4, 5, 6)

In [371]: a, b, c = tup

In [372]: b
Out[372]: 5

Even sequences with nested tuples can be unpacked:

In [373]: tup = 4, 5, (6, 7)

In [374]: a, b, (c, d) = tup

In [375]: d
Out[375]: 7

Using this functionality it’s easy to swap variable names, a task which in many languages might look like:

tmp = a
a = b
b = tmp

b, a = a, b

One of the most common uses of variable unpacking when iterating over sequences of tuples or lists:

seq = [(1, 2, 3), (4, 5, 6), (7, 8, 9)]
for a, b, c in seq:
    pass

Another common use is for returning multiple values from a function. More on this later.

Since the size and contents of a tuple cannot be modified, it is very light on instance methods. One particularly useful one (also available on lists) is count, which counts the number of occurrences of a value:

In [376]: a = (1, 2, 2, 2, 3, 4, 2)

In [377]: a.count(2)
Out[377]: 4

Elements can be appended to the end of the list with the append method:

In [384]: b_list.append('dwarf')

In [385]: b_list
Out[385]: ['foo', 'peekaboo', 'baz', 'dwarf']

Using insert you can insert an element at a specific location in the list:

In [386]: b_list.insert(1, 'red')

In [387]: b_list
Out[387]: ['foo', 'red', 'peekaboo', 'baz', 'dwarf']

The inverse operation to insert is pop, which removes and returns an element at a particular index:

In [388]: b_list.pop(2)
Out[388]: 'peekaboo'

In [389]: b_list
Out[389]: ['foo', 'red', 'baz', 'dwarf']

Elements can be removed by value using remove, which locates the first such value and removes it from the last:

In [390]: b_list.append('foo')

In [391]: b_list.remove('foo')

In [392]: b_list
Out[392]: ['red', 'baz', 'dwarf', 'foo']

If performance is not a concern, by using append and remove, a Python list can be used as a perfectly suitable “multi-set” data structure.

You can check if a list contains a value using the in keyword:

In [393]: 'dwarf' in b_list
Out[393]: True

Note that checking whether a list contains a value is a lot slower than dicts and sets as Python makes a linear scan across the values of the list, whereas the others (based on hash tables) can make the check in constant time.

Similar to tuples, adding two lists together with + concatenates them:

In [394]: [4, None, 'foo'] + [7, 8, (2, 3)]
Out[394]: [4, None, 'foo', 7, 8, (2, 3)]

If you have a list already defined, you can append multiple elements to it using the extend method:

In [395]: x = [4, None, 'foo']

In [396]: x.extend([7, 8, (2, 3)])

In [397]: x
Out[397]: [4, None, 'foo', 7, 8, (2, 3)]

Note that list concatenation is a compartively expensive operation since a new list must be created and the objects copied over. Using extend to append elements to an existing list, especially if you are building up a large list, is usually preferable. Thus,

everything = []
for chunk in list_of_lists:
    everything.extend(chunk)

is faster than than the concatenative alternative

everything = []
for chunk in list_of_lists:
    everything = everything + chunk

A list can be sorted in-place (without creating a new object) by calling its sort function:

In [398]: a = [7, 2, 5, 1, 3]

In [399]: a.sort()

In [400]: a
Out[400]: [1, 2, 3, 5, 7]

sort has a few options that will occasionally come in handy. One is the ability to pass a secondary sort key, i.e. a function that produces a value to use to sort the objects. For example, we could sort a collection of strings by their lengths:

In [401]: b = ['saw', 'small', 'He', 'foxes', 'six']

In [402]: b.sort(key=len)

In [403]: b
Out[403]: ['He', 'saw', 'six', 'small', 'foxes']

The built-in bisect module implements binary-search and insertion into a sorted list. bisect.bisect finds the location where an element should be inserted to keep it sorted, while bisect.insort actually inserts the element into that location:

In [404]: import bisect

In [405]: c = [1, 2, 2, 2, 3, 4, 7]

In [406]: bisect.bisect(c, 2)        In [407]: bisect.bisect(c, 5)
Out[406]: 4                          Out[407]: 6                  
                                                                  
In [408]: bisect.insort(c, 6)

In [409]: c
Out[409]: [1, 2, 2, 2, 3, 4, 6, 7]

You can select sections of list-like types (arrays, tuples, NumPy arrays) by using slice notation, which in its basic form consists of start:stop passed to the indexing operator []:

In [410]: seq = [7, 2, 3, 7, 5, 6, 0, 1]

In [411]: seq[1:5]
Out[411]: [2, 3, 7, 5]

Slices can also be assigned to with a sequence:

In [412]: seq[3:4] = [6, 3]

In [413]: seq
Out[413]: [7, 2, 3, 6, 3, 5, 6, 0, 1]

While element at the start index is included, the stop index is not included, so that the number of elements in the result is stop - start.

Either the start or stop can be omitted in which case they default to the start of the sequence and the end of the sequence, respectively:

In [414]: seq[:5]                In [415]: seq[3:]           
Out[414]: [7, 2, 3, 6, 3]        Out[415]: [6, 3, 5, 6, 0, 1]

Negative indices slice the sequence relative to the end:

In [416]: seq[-4:]            In [417]: seq[-6:-2]  
Out[416]: [5, 6, 0, 1]        Out[417]: [6, 3, 5, 6]

Slicing semantics takes a bit of getting used to, especially if you’re coming from R or MATLAB. See Figure A-2 for a helpful illustrating of slicing with positive and negative integers.

A step can also be used after a second colon to, say, take every other element:

In [418]: seq[::2]
Out[418]: [7, 3, 3, 6, 1]

A clever use of this is to pass -1 which has the useful effect of reversing a list or tuple:

In [419]: seq[::-1]
Out[419]: [1, 0, 6, 5, 3, 6, 3, 2, 7]

Figure A-2. Illustration of Python slicing conventions

It’s common when iterating over a sequence to want to keep track of the index of the current item. A do-it-yourself approach would look like:

i = 0
for value in collection:
   # do something with value
   i += 1

Since this is so common, Python has a built-in function enumerate which returns a sequence of (i, value) tuples:

for i, value in enumerate(collection):
   # do something with value

When indexing data, a useful pattern that uses enumerate is computing a dict mapping the values of a sequence (which are assumed to be unique) to their locations in the sequence:

In [420]: some_list = ['foo', 'bar', 'baz']

In [421]: mapping = dict((v, i) for i, v in enumerate(some_list))

In [422]: mapping
Out[422]: {'bar': 1, 'baz': 2, 'foo': 0}

The sorted function returns a new sorted list from the elements of any sequence:

In [423]: sorted([7, 1, 2, 6, 0, 3, 2])
Out[423]: [0, 1, 2, 2, 3, 6, 7]

In [424]: sorted('horse race')
Out[424]: [' ', 'a', 'c', 'e', 'e', 'h', 'o', 'r', 'r', 's']

A common pattern for getting a sorted list of the unique elements in a sequence is to combine sorted with set:

In [425]: sorted(set('this is just some string'))
Out[425]: [' ', 'e', 'g', 'h', 'i', 'j', 'm', 'n', 'o', 'r', 's', 't', 'u']

zip “pairs” up the elements of a number of lists, tuples, or other sequences, to create a list of tuples:

In [426]: seq1 = ['foo', 'bar', 'baz']

In [427]: seq2 = ['one', 'two', 'three']

In [428]: zip(seq1, seq2)
Out[428]: [('foo', 'one'), ('bar', 'two'), ('baz', 'three')]

zip can take an arbitrary number of sequences, and the number of elements it produces is determined by the shortest sequence:

In [429]: seq3 = [False, True]

In [430]: zip(seq1, seq2, seq3)
Out[430]: [('foo', 'one', False), ('bar', 'two', True)]

A very common use of zip is for simultaneously iterating over multiple sequences, possibly also combined with enumerate:

In [431]: for i, (a, b) in enumerate(zip(seq1, seq2)):
   .....:     print('%d: %s, %s' % (i, a, b))
   .....:
0: foo, one
1: bar, two
2: baz, three

Given a “zipped” sequence, zip can be applied in a clever way to “unzip” the sequence. Another way to think about this is converting a list of rows into a list of columns. The syntax, which looks a bit magical, is:

In [432]: pitchers = [('Nolan', 'Ryan'), ('Roger', 'Clemens'),
   .....:             ('Schilling', 'Curt')]

In [433]: first_names, last_names = zip(*pitchers)

In [434]: first_names
Out[434]: ('Nolan', 'Roger', 'Schilling')

In [435]: last_names
Out[435]: ('Ryan', 'Clemens', 'Curt')

We’ll look in more detail at the use of * in a function call. It is equivalent to the following:

zip(seq[0], seq[1], ..., seq[len(seq) - 1])

reversed iterates over the elements of a sequence in reverse order:

In [436]: list(reversed(range(10)))
Out[436]: [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

It’s common to occasionally end up with two sequences that you want to pair up element-wise in a dict. As a first cut, you might write code like this:

mapping = {}
for key, value in zip(key_list, value_list):
    mapping[key] = value

Since a dict is essentially a collection of 2-tuples, it should be no shock that the dict type function accepts a list of 2-tuples:

In [453]: mapping = dict(zip(range(5), reversed(range(5))))

In [454]: mapping
Out[454]: {0: 4, 1: 3, 2: 2, 3: 1, 4: 0}

In a later section we’ll talk about dict comprehensions, another elegant way to construct dicts.

It’s very common to have logic like:

if key in some_dict:
    value = some_dict[key]
else:
    value = default_value

Thus, the dict methods get and pop can take a default value to be returned, so that the above if-else block can be written simply as:

value = some_dict.get(key, default_value)

get by default will return None if the key is not present, while pop will raise an exception. With setting values, a common case is for the values in a dict to be other collections, like lists. For example, you could imagine categorizing a list of words by their first letters as a dict of lists:

In [455]: words = ['apple', 'bat', 'bar', 'atom', 'book']

In [456]: by_letter = {}

In [457]: for word in words:
   .....:     letter = word[0]
   .....:     if letter not in by_letter:
   .....:         by_letter[letter] = [word]
   .....:     else:
   .....:         by_letter[letter].append(word)
   .....:

In [458]: by_letter
Out[458]: {'a': ['apple', 'atom'], 'b': ['bat', 'bar', 'book']}

The setdefault dict method is for precisely this purpose. The if-else block above can be rewritten as:

by_letter.setdefault(letter, []).append(word)

The built-in collections module has a useful class, defaultdict, which makes this even easier. One is created by passing a type or function for generating the default value for each slot in the dict:

from collections import defaultdict
by_letter = defaultdict(list)
for word in words:
    by_letter[word[0]].append(word)

The initializer to defaultdict only needs to be a callable object (e.g. any function), not necessarily a type. Thus, if you wanted the default value to be 4 you could pass a function returning 4

counts = defaultdict(lambda: 4)

While the values of a dict can be any Python object, the keys have to be immutable objects like scalar types (int, float, string) or tuples (all the objects in the tuple need to be immutable, too). The technical term here is hashability. You can check whether an object is hashable (can be used as a key in a dict) with the hash function:

In [459]: hash('string')
Out[459]: -9167918882415130555

In [460]: hash((1, 2, (2, 3)))
Out[460]: 1097636502276347782

In [461]: hash((1, 2, [2, 3])) # fails because lists are mutable
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-461-800cd14ba8be> in <module>()
----> 1 hash((1, 2, [2, 3])) # fails because lists are mutable
TypeError: unhashable type: 'list'

To use a list as a key, an easy fix is to convert it to a tuple:

In [462]: d = {}

In [463]: d[tuple([1, 2, 3])] = 5

In [464]: d
Out[464]: {(1, 2, 3): 5}

Suppose we have a list of lists containing some boy and girl names:

In [483]: all_data = [['Tom', 'Billy', 'Jefferson', 'Andrew', 'Wesley', 'Steven', 'Joe'],
   .....:             ['Susie', 'Casey', 'Jill', 'Ana', 'Eva', 'Jennifer', 'Stephanie']]

You might have gotten these names from a couple of files and decided to keep the boy and girl names separate. Now, suppose we wanted to get a single list containing all names with two or more e’s in them. We could certainly do this with a simple for loop:

names_of_interest = []
for names in all_data:
    enough_es = [name for name in names if name.count('e') >= 2]
    names_of_interest.extend(enough_es)

You can actually wrap this whole operation up in a single nested list comprehension, which will look like:

In [484]: result = [name for names in all_data for name in names
   .....:           if name.count('e') >= 2]

In [485]: result
Out[485]: ['Jefferson', 'Wesley', 'Steven', 'Jennifer', 'Stephanie']

At first, nested list comprehensions are a bit hard to wrap your head around. The for parts of the list comprehension are arranged according to the order of nesting, and any filter condition is put at the end as before. Here is another example where we “flatten” a list of tuples of integers into a simple list of integers:

In [486]: some_tuples = [(1, 2, 3), (4, 5, 6), (7, 8, 9)]

In [487]: flattened = [x for tup in some_tuples for x in tup]

In [488]: flattened
Out[488]: [1, 2, 3, 4, 5, 6, 7, 8, 9]

Keep in mind that the order of the for expressions would be the same if you wrote a nested for loop instead of a list comprehension:

flattened = []

for tup in some_tuples:
    for x in tup:
        flattened.append(x)

You can have arbitrarily many levels of nesting, though if you have more than two or three levels of nesting you should probably start to question your data structure design. It’s important to distinguish the above syntax from a list comprehension inside a list comprehension, which is also perfectly valid:

In [229]: [[x for x in tup] for tup in some_tuples]

A simple way to make a generator is by using a generator expression. This is a generator analogue to list, dict and set comprehensions; to create one, enclose what would otherwise be a list comprehension with parenthesis instead of brackets:

In [510]: gen = (x ** 2 for x in xrange(100))

In [511]: gen
Out[511]: <generator object <genexpr> at 0x10a0a31e0>

This is completely equivalent to the following more verbose generator:

def _make_gen():
    for x in xrange(100):
        yield x ** 2
gen = _make_gen()

Generator expressions can be used inside any Python function that will accept a generator:

In [512]: sum(x ** 2 for x in xrange(100))
Out[512]: 328350

In [513]: dict((i, i **2) for i in xrange(5))
Out[513]: {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

The standard library itertools module has a collection of generators for many common data algorithms. For example, groupby takes any sequence and a function; this groups consecutive elements in the sequence by return value of the function. Here’s an example:

In [514]: import itertools

In [515]: first_letter = lambda x: x[0]

In [516]: names = ['Alan', 'Adam', 'Wes', 'Will', 'Albert', 'Steven']

In [517]: for letter, names in itertools.groupby(names, first_letter):
   .....:     print letter, list(names) # names is a generator
A ['Alan', 'Adam']
W ['Wes', 'Will']
A ['Albert']
S ['Steven']

See Table A-4 for a list of a few other itertools functions I’ve frequently found useful.