Index

Symbols

%r, for printable strings, 203

%s, for human-readable strings, 202

* operator, liability of, 44–45

* symbol, for keyword-only arguments, 52–53

*args

optional keyword arguments and, 48

variable positional arguments and, 43–45

**kwargs, for keyword-only arguments, 53–54

A

__all__ special attribute

avoiding, 183

listing all public names, 181–183

ALL_CAPS format, 3

Allocation of memory, tracemalloc module and, 214–216

APIs (application programming interfaces)

future-proofing, 186–187

internal, allowing subclass access to, 80–82

packages providing stable, 181–184

root exceptions and, 184–186

using functions for, 61–64

append method, 36–37

Arguments

defensively iterating over, 38–42

keyword, 45–48

keyword-only, 51–54

optional positional, 43–45

as clauses, in renaming modules, 181

as targets, with statements and, 155–156

assertEqual helper method, verifying equality, 206

assertRaises helper method, verifying exceptions, 206

assertTrue helper method, for Boolean expressions, 206

asyncio built-in module, vs. blocking I/O, 125

AttributeError exception raising, 102–103

Attribute(s). See also Private attributes; Public attributes

adding missing default values, 159–160

lazily loading/saving, 100–105

metaclasses annotating, 112–115

names, conflicts over, 81–82

B

Binary mode, for reading/writing data, 7

Binary tree class, inheriting from collections.abc, 84–86

bisect module, for binary searches, 169

Blocking operations, in Queue class, 132–136

Bookkeeping

with dictionaries, 55–58

helper classes for, 58–60

bt command, of interactive debugger, 208

Buffer sizes, in Queue class, 132–136

Bytecode, interpreter state for, 122

bytes instances, for character sequences, 5–7

C

__call__ special method, with instances, 63–64

callable built-in function, 63

CapitalizedWord format, 3

Central processing unit. See CPU (central processing unit)

C-extension modules

for CPU bottlenecks, 145

problems with, 146

chain function, of itertools module, 170

Child classes, initializing parent classes from, 69–73

Child processes, subprocess managing, 118–121

Circular dependencies

dynamic imports resolving, 191–192

import reordering for, 189–190

import/configure/run steps for, 190–191

in importing modules, 187–188

refactoring code for, 189

Clarity, with keyword arguments, 51–54

Class interfaces

@property method improving, 91–94

use public attributes for defining, 87–88

class statements, metaclasses receiving, 106–107

__class__ variable

registering classes and, 108–112

super built_in function with, 73

Classes. See also Metaclasses; Subclasses

annotating properties of, 112–115

for bookkeeping, 58–60

docstrings for, 177

initializing parent, 69–73

metaclasses registering, 108–112

mix-in, 73–78

versioning, 160–161

@classmethod

in accessing private attributes, 78–79

polymorphism, for constructing objects generically, 67–69

Closures, interacting with variable scope, 31–36

collections module

defaultdict class from, 168

deque class from, 166–167

OrderedDict class from, 167–168

collections.abc module, custom containers inheriting from, 84–86

combination function, of itertools module, 170

Command-lines

correct Python version, 1, 2

starting child processes, 119–120

communicate method

reading child process output, 118–119

timeout parameter with, 121

Community-built modules, Python Package Index for, 173–174

Complex expressions, helper functions and, 8–10

Concurrency

coroutines and, 137–138

defined, 117

in pipelines, 129–132

Queue class and, 132–136

concurrent.futures built-in module, enabling parallelism, 146–148

configparser built-in module, for production configuration, 201

Containers

inheriting from collections.abc, 84–86

iterable, 41–42

contextlib built-in module, enabling with statements, 154–155

contextmanager decorator

purpose of, 154

as targets and, 155–156

continue command, of interactive debugger, 209

Conway’s Game of Life, coroutines and, 138–143

Coordinated Universal Time (UTC), in time conversions, 162–165

copyreg built-in module

adding missing attribute values, 159–160

controlling pickle behavior, 158

providing stable import paths, 161–162

versioning classes with, 160–161

Coroutines

in Conway’s Game of Life, 138–143

purpose of, 137–138

in Python 2, 143–145

count method, for custom container types, 85–86

cProfile module, for accurate profiling, 210–213

CPU (central processing unit)

bottleneck difficulties, 145–146

time, threads wasting, 131–132

usage, child processes and, 118–121

CPython interpreter, effect of GIL on, 122–123

CPython runtime

memory management with, 214

cumtime column, in profiler statistics, 211

cumtime percall column, in profiler statistics, 211

cycle function, of itertools module, 170

D

Data models, @property improving, 91–95

Data races, Lock preventing, 126–129

datetime built-in module, for time conversions, 164–166

deactivate command, disabling pyvenv tool, 195–196

Deadlocks, timeout parameter avoiding, 121

Deallocation of memory, tracemalloc managing, 214–216

Debuggers, decorator problems with, 151, 153

Debugging

interactive, with pdb module, 208–209

memory usage, 214–216

print function and, 202

repr strings for, 202–204

root exceptions for, 185–186

Decimal class, for numerical precision, 171–173

Decorators, functionality of, 151–153

functools, 151–153

Default arguments

approach to serialization, 159–160

namedtuple classes and, 59

using dynamic values for, 48–51

Default value hooks, 62–64

defaultdict using, 62–64

Default values

copyreg built-in module and, 159–160

of keyword arguments, 46–47

defaultdict class, for dictionaries, 168

Dependencies

circular, 187–192

reproducing, 196–197

transitive, 192–194

Dependency injection, 191

Deployment environments, module-scoped code for, 199–201

deque class, as double-ended queue, 166–167

Descriptors

enabling reusable property logic, 90

in modifying class properties, 112–115

for reusable @property methods, 97–100

Deserializing objects

default attribute values and, 159–160

pickle built-in module for, 157–158

stable import paths and, 161–162

Development environment, unique configurations/assumptions for, 199–201

Diamond inheritance, initializing parent classes and, 70–71

__dict__ attribute, viewing object internals, 204

Dictionaries

bookkeeping with, 55–58

comprehension expressions in, 16

default, 168

ordered, 167–168

translating related objects into, 74–75

__doc__ special attribute, retrieving docstrings, 175–176

Docstrings

class-level, 177

documenting default behavior in, 48–51

for functions, 178–179

importance/placement of, 175–176

module, 176–177

doctest built-in module, 179

Documentation

docstrings for. See Docstrings

importance of, 175

Documentation-generation tools, 176

Double-ended queues, deque classes as, 166–167

__double_leading_underscore format, 3

down command, of interactive debugger, 209

dropwhile function, of itertools module, 170

Dynamic imports

avoiding, 192

resolving circular dependencies, 191–192

Dynamic state, defined, 55

E

else blocks

after for/while loops, 23–25

during exception handling, 26–27

end indexes, in slicing sequences, 10–13

__enter__ method, in defining new classes, 154

enumerate built-in function, preferred features of, 20–21

environ dictionary, tailoring modules with, 201

eval built-in function, for re-creating original values, 203

Exceptions

raising, 29–31

root, 184–187

try/finally blocks and, 26–28

Execution time, optimization of, 209–213

__exit__ method, in defining new classes, 154

Expressions

in list comprehensions, 16–18

PEP 8 guidance for, 4

F

filter built-in function, list comprehensions vs., 15–16

filterfalse function, of itertools module, 170

finally blocks, during exception handling, 26–27

First-in-first-out queues, deque class for, 166–167

for loops

else blocks after, 23–25

iterator protocol and, 40–42

Fraction class, for numerical precision, 172

Functions

closure/variable scope interaction, 31–36

decorated, 151–153

docstrings for, 178–179

exceptions vs. return None, 29–31

as first-class objects, 32, 63–64

generator vs. returning lists, 36–38

iterating over arguments, 38–42

keyword arguments for, 45–48

keyword-only arguments for, 51–54

optional positional arguments for, 43–45

for simple interfaces, 61–64

simultaneous, coroutines for, 137–138

functools built-in module, for defining decorators, 152–153

G

Game of Life, coroutines in, 138–143

Garbage collector, cleanup by, 99

gc built-in module, debugging memory usage, 214–215

Generator(s)

coroutine extensions of, 137–138

expressions, for large comprehensions, 18–20

returning lists vs., 36–38

Generic class method, for constructing objects, 67–69

Generic functionality, with mix-in classes, 74–78

__get__ method, for descriptor protocol, 97–100

__getattr__ special method, to lazily load attributes, 100–103

__getattribute__ method, accessing instance variables in, 104–105

__getattribute__ method, descriptor protocol and, 98–100

__getattribute__ special method, for repeated access, 102–105

__getitem__ special method

custom implementation of, 84–86

in slicing sequences, 10

Getter methods

descriptor protocol for, 98–100

problems with using, 87–88

providing with @property, 88–89

GIL (global interpreter lock)

corruption of data structures and, 126–127

defined, 122

preventing parallelism in threads, 122–125, 145, 146–147

Global scope, 33

H

hasattr built-in function, determining existence of properties, 103

hashlib built-in module, 120

heappop function, for priority queues, 168–169

heappush function, for priority queues, 168–169

heapq module, for priority queues, 168–169

help function

decorator problems with, 152–153

in interactive debugger, 208

Helper classes

for bookkeeping, 58–60

providing stateful closure behavior, 62–63

Helper functions, complex expressions into, 8–10

Hooks

to access missing attributes, 100–105

default value, 62–64

functions acting as, 61–62

in modifying class properties, 113

I

IEEE 754 (IEEE Standard for Floating-Point Arithmetic), 171–172

if/else expressions, for simplification, 9–10

import * statements

avoiding, 183–184

in providing stable APIs, 182–183

Import paths, stable, copyreg providing, 161–162

Import reordering, for circular dependencies, 189–190

import statements

as dynamic imports, 191–192

with packages, 180–181

Incrementing in place, public attributes for, 88

index method, for custom container types, 85–86

Infinite recursion, super() function avoiding, 101–105

Inheritance

from collections.abc, 84–86

method resolution order (MRO) and, 71

multiple, for mix-in utility classes, 77–78

__init__ method

as single constructor per class, 67, 69

initializing parent class, 69–71

__init__.py

defining packages, 180

modifying, 182

Initializing parent classes

__init__ method for, 69–71

method resolution order (MRO) and, 71

super built-in function for, 70–73

Integration tests, 207

Interactive debugging, with pdb, 208–209

Intermediate root exceptions, future-proofing APIs, 186–187

I/O (input/output)

between child processes, 118–121

threads for blocking I/O, 124–125

IOError, except blocks and, 26–27

IronPython runtime, 1, 2

isinstance

bytes/str/unicode and, 5–6

with coroutines, 142

dynamic type inspection with, 74–75

metaclasses and, 114

pickle module and, 158

testing and, 205

islice function, of itertools module, 170

iter built-in function, 41–42

__iter__ method

as generator, 41–42

iterable container class, defined, 41–42

Iterator protocol, 40–42

Iterators

as function arguments, 39

generators returning, 37–38

zip function processing, 21–23

itertools built-in module

functions of, 169–170

izip_longest function, for iterating in parallel, 23

J

join method, of Queue class, 132–136

Jython runtime, 1, 2

K

Keyword arguments

constructing classes with, 58

dynamic default argument values, 48–51

providing optional behavior, 45–48

Keyword-only arguments

for clarity, 51–53

in Python 2, 53–54

L

lambda expression

as key hook, 61

vs. list comprehensions, 15–16

producing iterators and, 40

in profiling, 210–212

Language hooks, for missing attributes, 100–105

Lazy attributes, __getattr__/__setattr__/__getattribute__ for, 100–105

_leading_underscore format, 3

Leaky bucket quota, implementing, 92–95

len built-in function, for custom sequence types, 85

__len__ special method, for custom sequence types, 85

list built-in type, performance as FIFO queue, 166–167

List comprehensions

generator expressions for, 18–20

instead of map/filter, 15–16

number of expressions in, 16–18

list type, subclassing, 83–84

Lists, slicing, 10–13

locals built-in function, 152, 208

localtime function, from time module, 163–164

Lock class

preventing data races, 126–129

in with statements, 153–154

Logging

debug function for, 154–156

severity levels, 154–155

Loops

else blocks after, 23–25

in list comprehensions, 16–18

range/enumerate functions, 20–21

lowercase_underscore format, 3

M

map built-in function, list comprehensions vs., 15–16

Memory

coroutine use of, 137

threads requiring, 136

Memory leaks

by descriptor classes, 99–100

identifying, 214–216

Memory management, with tracemalloc module, 214–216

Meta.__new__ method

in metaclasses, 107

setting class attributes, 114

__metaclass__ attribute, in Python 2, 106–107

Metaclasses

annotating attributes with, 112–115

for class registration, 108–112

defined, 87, 106

validating subclasses, 105–108

method resolution order (MRO), for superclass initialization order, 70–73

Mix-in classes

composing from simple behaviors, 74–75

defined, 73–74

pluggable behaviors for, 75–76

utility, creating hierachies of, 77–78

mktime, for time conversion, 163, 165

Mock functions and classes

unittest.mock built-in module, 206

__module__ attribute, 106, 153

Modules

breaking circular dependencies in, 187–192

community-built, 173–174

docstrings, 176–177

packages for organizing, 179–184

providing stable APIs from, 181–184

tailoring for deployment environment, 199–201

Module-scoped code, for deployment environments, 199–201

MRO (method resolution order), for superclass initialization order, 70–73

Multiple conditions, in list comprehensions, 16–18

Multiple inheritance, for mix-in utility classes, 73–78

Multiple iterators, zip built-in function and, 21–23

Multiple loops, in list comprehensions, 16–18

multiprocessing built-in module, enabling parallelism, 146–148

Mutual-exclusion locks (mutex)

GIL as, 122

Lock class as, 126–129

in with statements, 153–154

N

__name__ attribute in defining decorators, 151, 153

in registering classes, 109–110

testing and, 206

namedtuple type

defining classes, 58

limitations of, 59

NameError exception, 33

Namespace packages, with Python 3.4, 180

Naming conflicts, private attributes to avoid, 81–82

Naming styles, 3–4

ncalls column in profiler statistics, 211

__new__ method, of metaclasses, 106–108

next built-in function, 41–42

next command, of interactive debugger, 209

__next__ special method, iterator object implementing, 41

Noise reduction, keyword arguments and, 45–48

None value

functions returning, 29–31

specifying dynamic default values, 48–51

nonlocal statement, in closures modifying variables, 34–35

nsmallest function, for priority queues, 168–169

Numerical precision, with Decimal class, 171–173

O

Objects, accessing missing attributes in, 100–105

On-the-fly calculations, using @property for, 91–95

Optimization, profiling prior to, 209–213

Optional arguments

keyword, 47–48

positional, 43–45

OrderedDict class, for dictionaries, 167–168

OverflowError exceptions, 51

P

Packages

dividing modules into namespaces, 180–181

as modules containing modules, 179–180

providing stable APIs with, 181–184

Parallelism

avoiding threads for, 122–123

child processes and, 118–121

concurrent.futures for true, 146–148

corruption of data structures and, 126–128

defined, 117

need for, 145–146

Parent classes

accessing private attributes of, 79–81

initializing, 70–73

pdb built-in module, for interactive debugging, 208–209

pdb.set_trace() statements, 208–209

PEP 8 (Python Enhancement Proposal #8) style guide

expression/statement rules, 4

naming styles in, 3–4, 80

overview of, 2–3

whitespace rules, 3

permutations function, of itertools module, 170

pickle built-in module

adding missing attribute values, 159–160

providing stable import paths for, 161–162

serializing/deserializing objects, 157–158

versioning classes for, 160–161

pip command-line tool

reproducing environments, 196–197

transitive dependencies and, 192–193

for utilizing Package Index, 173

pip freeze command, saving package dependencies, 196

Pipelines

concurrency in, 129–131

problems with, 132

Queue class building, 132–136

Polymorphism

@classmethods utilizing, 65–69

defined, 64

Popen constructor, starting child processes, 118

Positional arguments

constructing classes with, 58

keyword arguments and, 45–48

reducing visual noise, 43–45

print function, for debugging output, 202–203, 208

print_stats output, for profiling, 213

Printable representation, repr function for, 202–204

Private attributes

accessing, 78–80

allowing subclass access to, 81–83

indicating internal APIs, 80

ProcessPoolExecutor class, enabling parallelism, 147–148

product function, of itertools module, 170

Production environment, unique configurations for, 199–201

profile module, liabilities of, 210

@property method

defining special behavior with, 88–89

descriptors for reusing, 97–100

giving attributes new functionality, 91–94

improving data models with, 95

numerical attributes, into on-the-fly calculations, 91–95

problems with overusing, 95–96

unexpected side effects in, 90–91

@property.setter, modifying object state in, 91

pstats built-in module, extracting statistics, 211

Public attributes

accessing, 78

defining new class interfaces with, 87–88

giving new functionality to, 91–94

preferred features of, 80–82

Pylint tool, for Python source code, 4

PyPI (Python Package Index), for community-built modules, 173–174

PyPy runtime, 1, 2

Python 2

coroutines in, 143–145

determining use of, 2

keyword-only arguments in, 53–54

metaclass syntax in, 106–107

mutating closure variables in, 35

str and unicode in, 5–7

zip built-in function in, 22

Python 3

class decorators in, 111

determining use of, 2

closures and nonlocal statements in, 34–35

keyword-only arguments in, 51–53

metaclass syntax in, 106

str and bytes in, 5–7

Python Enhancement Proposal #8. See PEP 8 (Python Enhancement Proposal #8) style guide

Python Package Index (PyPI), for community-built modules, 173–174

Python threads. See Threads

pytz module

installing, 173

pyvenv tool and, 194

for time conversions, 165–166

pyvenv command-line tool

purpose of, 194

reproducing environments, 196–197

for virtual environments, 194–196

Q

quantize method, of Decimal class, for numerical data, 172

Queue class, coordinating work between threads, 132–136

R

range built-in function, in loops, 20

Read the Docs community-funded site, 176

Refactoring attributes, @property instead of, 91–95

Refactoring code, for circular dependencies, 189

Registering classes, metaclasses for, 108–112

Repetitive code

composing mix-ins to minimize, 74

keyword arguments eliminating, 45–48

__repr__ special method, customizing class printable representation, 203–204

repr strings, for debugging output, 202–204

requirements.txt file, for installing packages, 197

return command, of interactive debugger, 209

return statements

in generators, 140

not allowed in Python 2 generators, 144

Root exceptions

finding bugs in code with, 185–186

future-proofing APIs, 186–187

insulating callers from APIs, 184–185

Rule of least surprise, 87, 90, 91

runcall method, for profiling, 211–213

S

Scopes, variable, closure interaction with, 31–36

Scoping bug, in closures, 34

select built-in module, blocking I/O, 121, 124

Serializing, data structures, 109

Serializing objects, pickle and

default argument approach to, 159–160

default attribute values and, 159–160

pickle built-in module for, 157–158

stable import paths and, 161–162

__set__ method, for descriptor protocol, 97–100

set_trace function, pdb module running, 208–209

setattr built-in function

annotating class attributes and, 113

in bad thread interactions, 127–128

lazy attributes and, 101–102, 104

__setattr__ special method, to lazily set attributes, 103–105

__setitem__ special method, in slicing sequences, 10

Sets, comprehension expressions in, 16

setter attribute, for @property method, 88–89

Setter methods

descriptor protocol for, 98–100

liability of using, 87–88

providing with @property, 88–89

setuptools, in virtual environments, 195–197

Single constructor per class, 67, 69

Single-line expressions, difficulties with, 8–10

six tool, in adopting Python 3, 2

Slicing sequences

basic functions of, 10–13

stride syntax in, 13–15

Sort, key argument, closure functions as, 31–32

source bin/activate command, enabling pyvenv tool, 195

Speedup, concurrency vs. parallelism for, 117

Star args (*args), 43

start indexes, in slicing sequences, 10–13

Statements, PEP 8 guidance for, 4

Static type checking, lack of, 204–205

Stats object, for profiling information, 211–213

step command, of interactive debugger, 209

StopIteration exception, 39, 41

str instances, for character sequences, 5–7

stride syntax, in slicing sequences, 13–15

strptime functions, conversion to/from local time, 163–164

Subclasses

allowing access to private fields, 81–83

constructing/connecting generically, 65–69

list type, 83–84

TestCase, 206–207

validating with metaclasses, 105–108

subprocess built-in module, for child processes, 118–121

super built-in function, initializing parent classes, 71–73

super method, avoiding infinite recursion, 101–105

Superclass initialization order, MRO resolving, 70–73

Syntax

decorators, 151–153

for closures mutating variables, 34–35

for keyword-only arguments, 52–53

loops with else blocks, 23

list comprehensions, 15

metaclasses, 106

slicing, 10–13

SyntaxError exceptions, dynamic imports and, 192

sys module, guiding module definitions, 201

System calls, blocking I/O and, 124–125

T

takewhile function, of itertools module, 170

task_done call, method of the Queue class, in building pipelines, 134

tee function, of itertools module, 170

Test methods, 206–207

TestCase classes, subclassing, 206–207

threading built-in module, Lock class in, 126–129

ThreadPoolExecutor class, not enabling parallelism, 147–148

Threads

blocking I/O and, 124–125

coordinating work between, 132–136

parallelism prevented by, 122–123, 145, 146–147

preventing data races between, 126–129

problems with, 136

usefulness of multiple, 124

time built-in module, limitations of, 163–164

Time zone conversion methods, 162–166

timeout parameter, in child process I/O, 121

tottime column, in profiler statistics, 211

tottime percall column, in profiler statistics, 211

tracemalloc built-in module, for memory optimization, 214–216

Transitive dependencies, 192–194

try/except statements, root exceptions and, 185

try/except/else/finally blocks, during exception handling, 27–28

try/finally blocks

during exception handling, 26–27

with statements providing reusable, 154–155

Tuples

extending, 58

rules for comparing, 32

as values, 57

variable arguments becoming, 44

zip function producing, 21–23

TypeError

exceptions, for keyword-only arguments, 53–54

rejecting iterators, 41–42

tzinfo class, for time zone operations, 164–165

U

unicode instances, for character sequences, 5–7

Unit tests, 207

unittest built-in module, for writing tests, 205–207

UNIX timestamp, in time conversions, 163–165

Unordered dictionaries, 167

up command, of interactive debugger, 209

UTC (Coordinated Universal Time), in time conversions, 162–165

Utility classes, mix-in, creating hierarchies of, 77–78

V

Validation code, metaclasses running, 105–108

ValueError exceptions, 30–31, 184

Values

from iterators, 40–42

tuples as, 57

validating assignments to, 89

Variable positional arguments

keyword arguments and, 47–48

reducing visual noise, 43–45

Variable scopes, closure interaction with, 31–36

--version flag, determining version of Python, 1–2

Virtual environments

pyvenv tool creating, 194–196

reproducing, 196–197

virtualenv command-line tool, 194

Visual noise, positional arguments reducing, 43–45

W

WeakKeyDictionary, purpoose of, 99

weakref module, building descriptors, 113

while loops, else blocks following, 23–25

Whitespace, importance of, 3

Wildcard imports, 183

with statements

mutual-exclusion locks with, 153–154

for reusable try/finally blocks, 154–155

as target values and, 155–156

wraps helper function, from functools, for defining decorators, 152–153

Y

yield expression

in coroutines, 137–138

in generator functions, 37

use in contextlib, 155

yield from expression, unsupported in Python 2, 144

Z

ZeroDivisionError exceptions, 30–31, 51

zip built-in function

for iterators of different lengths, 170

processing iterators in parallel, 21–23

zip_longest function, for iterators of different length, 22–23, 170