When a variable is assigned a literal value in Python, what actually happens is that an object is created, the literal value is assigned to it (it becomes an attribute of the object), and then it is “bound” to a name. Objects usually have a special method called a constructor that handles the details of creating (instantiating) a new object in memory and initializing it. Conversely, an object may also have a destructor method to remove it from memory when the program is finished with it. In Python, the removal of an object is usually handled automatically in a process called garbage collection.

Here’s an example of how Python creates a new data object:

>>> some_var = 5

This statement instantiates a new object of type int with a value attribute of 5, and then binds the name some_var to it (we’ll see how name binding works shortly). One could also type the following and get the same result:

>>> some_var = int(5)

In this case, we are explicitly telling Python the object type we want (an integer) by calling the int class constructor and passing it the literal value to be assigned when the new object is instantiated. It is important to note that this is not a “cast” in a C or C++ sense; it is an instantiation of an int object that encapsulates the integer value 5.

This way of doing things may seem a bit odd at first, but one gets used to it fairly quickly. Also, most of the time you can safely ignore the fact that variables are actually objects, and just treat them as you might treat a variable in C or C++:

>>> var_one = 5
>>> var_two = 10
>>> var_one + var_two
15

You can also query an object to see what type it is:

>>> type(some_var)
<type 'int'>

Although I just stated that int() is not a cast, it can be used as something akin to that by letting the data objects do the type conversion themselves when a new object is created:

>>> float_var = 5.5
>>> int_var = int(float_var)
>>> print int_var
5

Notice that the fractional part of float_var vanished as a result of the conversion.

Octal and hexadecimal integer notation is also supported, and work as in C:

Octal and hexadecimal values don’t have their own type classes. This is because when a value written in either format is assigned to a Python variable, it is converted to its integer equivalent:

>>> foo_hex = 0x2A7
>>> print foo_hex
679

This is equivalent to writing:

>>> foo_hex = int("2A7",16)
>>> print foo_hex
679

So exactly what is a “data object”? In Python, things like variable names reside in what is called a namespace. There are various levels of namespaces, from the local namespace of a function or method to the global namespace of the Python interpreter’s execution environment. For now, we won’t worry too much about them; we’ll just work with the concept of a local namespace.

Variable names do not have any value, other than the string that makes up the name. They are more like handles or labels that we can attach to things that do have values—namely, objects. Figure 3-1 shows how this works.

Typically objects have methods, or internal functions, that operate on the data encapsulated within them. Python’s data objects are no exception. If we create an integer data object, we can ask Python to describe the object to us using the help() function, like this:

>>> int_var = 5
>>> help(5)
Help on int object:

class int(object)
 |  int(x[, base]) -> integer
 |
 |  Convert a string or number to an integer, if possible.  A floating point
 |  argument will be truncated towards zero (this does not include a string
 |  representation of a floating point number!)  When converting a string, use
 |  the optional base.  It is an error to supply a base when converting a
 |  non-string.  If base is zero, the proper base is guessed based on the
 |  string content.  If the argument is outside the integer range a
 |  long object will be returned instead.
 |
 |  Methods defined here:
 |
 |  __abs__(...)
 |      x.__abs__() <==> abs(x)
 |
 |  __add__(...)
 |      x.__add__(y) <==> x+y
 |
 |  __and__(...)
 |      x.__and__(y) <==> x&y
 |
 |  __cmp__(...)
 |      x.__cmp__(y) <==> cmp(x,y)
 |
 |  __coerce__(...)
 |      x.__coerce__(y) <==> coerce(x, y)
 |
 |  __div__(...)
 |      x.__div__(y) <==> x/y
 |
 |  __divmod__(...)
 |      x.__divmod__(y) <==> divmod(x, y)
 |
 |  __float__(...)
 |      x.__float__() <==> float(x)
 |
 |  __floordiv__(...)
 |      x.__floordiv__(y) <==> x//y
 |
 |  __format__(...)
 |
 |  __getattribute__(...)
 |      x.__getattribute__('name') <==> x.name
 |
 |  __getnewargs__(...)
 |
-- More  --

There are more internal methods, and you can peruse them if you are so inclined (just press the space bar for another screenful, Return for another line, or q to return to the prompt), but the main point here is that in Python, data objects “know” how to manipulate their internal data using the built-in methods for a particular class. In other words, the Python interpreter handles the details of converting a statement like this:

5 + 5

into the bytecode equivalent of this, internally:

int(5).__add__(int(5))

and then executing it.

Figure 3-1. Numeric data objects

The fact that variables in Python really are objects does take a little getting used to. But it is a powerful feature of the language, and because you can selectively ignore this feature it is possible to create what look like procedural programs, when in reality Python is all about objects.

Python provides three data types for ordered collections of data objects: lists (arrays), strings, and tuples (list-like objects). These are also known as sequence objects. The “sequence” part refers to the fact that each of these data objects may contain zero or more references to other data objects in an ordered sequence. All except for the string type allow their member elements to be any valid Python object. All have methods for manipulating their data; some methods are common to all sequence objects, and some are unique to a particular type. Table 3-2 lists the three sequence types and some of their properties.

Python sequence objects are either mutable (changeable) or immutable (unchangeable). A list object, for example, is mutable in that its data can be modified. A string, on the other hand, is not mutable. One cannot replace, remove, or insert characters into a string directly. A string object is an immutable collection of character values that is treated as a read-only array of byte-sized data objects.

In order to make a change to a string, one must create a new string that incorporates the changes. The original string object remains untouched, even if the same variable name is reused for the new string object (which “unbinds” the original string object; unbound objects tend to evaporate through the process of garbage collection, but that’s a low-level detail we don’t really need to worry about).

Lists

A list is Python’s closest equivalent to an array, but it has a few tricks that the arrays in C and Pascal never learned how to do. A list is an ordered sequence, and any element in the list may be replaced with something different. New elements are appended to a list using its append method (there is also a pop method, which means a list can be a queue as well), and the contents of a list can be sorted in place. Each element in a list is actually a reference to an object, just as a numeric data variable name is a reference to a numeric data object. In fact, a list can contain references to any valid Python object. Consider the following:

>>> import random
>>> alist = []
>>> alist.append(4)
>>> alist.append(55.89)
>>> alist.append('a short string')
>>> alist.append(random.random)

alist now contains four elements, which are composed of an integer, a floating-point value, a string, and a reference to a method from Python’s random module called, appropriately enough, random (we’ll discuss the import statement in more detail later). We can examine each member element of alist to verify this:

>>> alist[0]
4
>>> alist[1]
55.890000000000001
>>> alist[2]
'a short string'
>>> alist[3]
<built-in method random of Random object at 0x00A29D28>

If we want a random number, all we have to do in order to invoke random() is treat alist[3] as if it were a function by appending the expected parentheses:

>>> alist[3]()
0.87358651337544713

We can change a particular element in alist simply by assigning it a new value:

>>> alist[2]
'a short string'
>>> alist[2] = 'a better string'
>>> alist[2]
'a better string'

Figure 3-2 shows what is going on inside the alist object.

Figure 3-2. List object internal organization

We can use a list object to demonstrate Python’s underlying OO nature by entering the following at the Python prompt and observing the results:

>>> list_name = []
>>> list_name.append(0)
>>> list_name.append(1)
>>> list_name
[0, 1]
>>> var_one = list_name
>>> var_two = list_name
>>> var_one
[0, 1]
>>> var_two
[0, 1]
>>> list_name[0] = 9
>>> var_one
[9, 1]
>>> var_two
[9, 1]

Because the names var_one and var_two both refer to the list object initially bound to the name list_name, when list_name is altered the change in the list object is “seen” by both of the other variable names.

Like most every other object in Python, a list has a collection of methods. These include the indexing methods we’ve already seen, but there are more. Lists can be concatenated and are appended end-to-end in the order specified, like so:

>>> alist1 = [1,2,3,4,5]
>>> alist2 = [6,7,8,9,10]
>>> alist1 + alist2
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

To find the index offset of a particular item in a list, we can use the index() method:

>>> alist2.index(8)
2

We can also reverse the order of a list:

>>> alist1.reverse()
>>> alist1
[5, 4, 3, 2, 1]

And we can sort a list:

>>> slist = [8,22,0,5,16,99,14,-6,42,66]
>>> slist.sort()
>>> slist
[-6, 0, 5, 8, 14, 16, 22, 42, 66, 99]

Notice in the last two examples that the list itself is modified “in place.” That is, a new object is not created as a result of reversing or sorting a list. Lists are mutable.

Strings

Strings are ordered sequences of byte-value characters. Strings are immutable, meaning that (unlike in C or C++) they cannot be altered in place by using an index and treating them like arrays. In order to modify a string, one must create a new string object. The contents of a string can, however, be referenced using an index into the string.

Here are some string examples:

>>> astr1 = 'This is a short string.'
>>> astr2 = "This is another short string."
>>> astr3 = "This string has 'embedded' single-quote chracters."
>>> astr4 = """This is an example
... of a multi-line
... string.
... """
>>>

Although one cannot change the contents of a string using an index value, the data can be read using an index, and Python provides the ability to extract specific parts of a string (or “slices,” as they are called). The result is a new string object. The following line will read the first four characters of the string variable astr1, starting at the zero position and stopping before, but not at, the fourth position:

>>> print astr1[0:4]
This

We could also eliminate the 0 in the index range and just let it be assumed:

>>> print astr1[:4]
This

This form tells Python to extract everything from the start of the string up to the fourth position. We can also extract everything from the fourth position to the end of the line:

>>> print astr1[4:]
 is a short string.

Or we can get something from the middle of the string:

>>> print astr1[10:15]
short

Figure 3-3 shows how indexing works in Python.

Figure 3-3. String indexing

String objects also incorporate a set of methods that perform operations such as capitalization, centering, and counting the occurrences of particular characters, among other things, each returning a new string object.

As with lists, concatenation uses the + operator:

>>> str_cat = astr1 + " " + astr2
>>> print str_cat
This is a short string. This is another short string.

The result is, as you might expect by now, a new string object. Fortunately, Python incorporates garbage collection, and objects that are no longer bound to a name, as in the following situation, are quietly whisked away; their memory is returned to a shared pool for reuse. This is a good thing, as otherwise memory could quickly fill up with abandoned data objects:

>>> the_string = "This is the string."
>>> the_string = the_string[0:4]
>>> the_string
'This'

In this case, the name the_string is initially bound to the string object containing "This is the string.". When a section of the initial string object is pulled out, a new object is created and the name is reassigned to it. The original object, no longer bound, disappears. However, if an object is shared between two or more names, it will persist so long as one name is bound to it. This can come in handy when creating objects that need to hang around for the life of a program.

Other string methods allow you to left- or right-align a string, replace a word in a string, or convert the case of the characters in a string. Here are some examples.

The upper() method converts all alphabetic characters in a string to uppercase:

>>> print astr1.upper()
THIS IS A SHORT STRING.

find() returns the index of the first character in the search pattern string:

>>> print astr1.find('string')
16

The replace() method substitutes the new string for the search pattern:

>>> print astr1.replace('string', 'line')
This is a short line.

The rjust() method (and its counterpart, ljust()) justifies a string in a field, the width of which is the method’s argument:

>>> print astr1.rjust(30)
       This is a short string.

The default fill character is a space, but one can specify an alternative as a second argument:

>>> print astr1.rjust(30,'.')
.......This is a short string.

You can get a listing of the various string methods available by typing help(str) at the Python prompt.

>>> tuple2 = (1,2)
>>> tuple2
(1, 2)

>>> tuple4 = (9, 22.5, 0x16, 0)
>>> tuple4
(9, 22.5, 22, 0)
>>> tuple4[2]
22
>>> tuple4[0]
9

>>> tuple2
(1, 2)
>>> tuple4
(9, 22.5, 22, 0)
>>> tuple6 = tuple2 + tuple4
>>> tuple6
(1, 2, 9, 22.5, 22, 0)

>>> tpl = (0, 0, 2, 2, 6, 0, 3, 2, 1, 0)
>>> tpl.count(0)
4
>>> tpl.count(2)
3
>>> tpl.count(6)
1

Python’s dictionary is a unique data object. Instead of an ordered set of data elements, a dictionary contains data in the form of a set of unordered key/value pairs. That is, each data element has an associated key that uniquely identifies it. It is Python’s one and only mapped data object.

Like any other Python data object, a dictionary can be passed as an argument to a function or method, and returned as well. It can be a data element in a tuple or list, and its values can be any valid Python object type. The types that are usable as keys are limited to integers, strings, and tuples; in other words, keys must be immutable objects.

To create a dictionary object, we can initialize it with a set of keys and associated values:

>>> dobj = {0:"zero", 1:"one", "food":"eat", "spam":42}
>>> dobj
{0: 'zero', 1: 'one', 'food': 'eat', 'spam': 42}

To get at a particular key, we can use what looks like indexing, but is not:

>>> dobj[0]
'zero'
>>> dobj[1]
'one'

If we try a key that isn’t in the dictionary, Python complains:

>>> dobj[2]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 2

But so long as it’s a valid key, we will get a valid value back:

>>> dobj["spam"]
42

Dictionaries incorporate a set of powerful methods for manipulating their data. Table 3-3 contains a list of what’s available, and we’ll look at a few in detail.

Note that there is no append() method like the one available for lists. To add a new item to a dictionary, one simply assigns a value to a new key:

>>> dobj[99] = "agent"
>>> dobj
{0: 'zero', 1: 'one', 99: 'agent', 'food': 'eat', 'spam': 42}

Notice that the new key and its associated data are inserted in the dictionary at an arbitrary location. A dictionary is not a sequence object, and data is accessed using keys, so it really is unimportant where it is actually located amongst the other key/value pairs in the data object.

This technique can also be used to modify an existing key’s value:

>>> dobj[1] = "the big one"
>>> dobj
{0: 'zero', 1: 'the big one', 99: 'agent', 'food': 'eat', 'spam': 42}

A safer way to fetch a value from a dictionary is to use the get() method:

>>> dobj.get(99)
'agent'

If we attempt to get a value for a key that doesn’t exist, get() will by default return the special value of None. At the Python command line, this doesn’t show anything:

>>> dobj.get(256)

We can specify a default return value of our choosing, if we so desire, like this:

>>> dobj.get(256,"Nope")
'Nope'

Dictionaries are useful for keeping global data (such as parameters) in one convenient place, and the ability to return a default value allows a program to use predefined parameter values if no externally supplied values are available.

There may be times when we want to get a list of what’s in a dictionary. The items() method returns all of a dictionary’s key/value pairs as a list object of 2-tuples:

>>> dobj.items()
[(0, 'zero'), (1, 'the big one'), (99, 'agent'), ('food', 'eat'), ('spam', 42)]

If we want a list of the keys, we can get one using the keys() method:

>>> dobj.keys()
[0, 1, 99, 'food', 'spam']

Finally, if we are only interested in the values, the values() method comes in handy:

>>> dobj.values()
['zero', 'the big one', 'agent', 'eat', 42]

That should be enough on dictionaries for now. We will see other interesting ways to use dictionaries and the other Python data types later, but in the meantime, feel free to experiment with the Python command line. Trying out new things is one of the best ways to learn about them.

Python provides the usual four basic arithmetic operators: addition, subtraction, multiplication, and division. It also has two operators that are not found in some other languages: exponent and floor division. Table 3-4 lists Python’s arithmetic operators.

When dealing with a mix of numeric data types, Python will automatically “promote” all of the operands to the highest-level type, and then perform the indicated operation. The type priorities are:

complex
float
long
int

This means that if an expression contains a floating-point value but no complex values, the result will be a floating-point value. If an expression contains a long and no floating-point or complex values, the result will be a long. If an expression contains a complex value, the result will be complex. So, if one has an expression that looks like this:

5.0 * 5

the result will be a floating-point value:

25.0

As I mentioned, Python also has a unique division operator called “floor division.” This is used to return the quotient of a floating-point operation truncated down to the nearest whole value, with the result returned as a float. In Python, the behavior of // is like this:

>>> 5/2
2
>>> 5//2
2
>>> 5.0/2
2.5
>>> 5.0//2
2.0

Python’s logical operators, shown in Table 3-5, act on the truth values of any object.

Python provides the keywords True and False for use in logical expressions. Note that any of the following are also considered to be False:

All other values are considered to be True. It is also common to find 1 and 0 acting as true and false values.

Comparison operators evaluate two operands and determine the relationship between them in terms of equality, inequality, and magnitude (see Table 3-6).

Python expressions that use comparison operators always return a logical true or false.

Python’s AND, OR, and XOR operators map across bit-to-bit between the operands; they do not perform arithmetic operations. The bitwise operators are listed in Table 3-7.

The AND operation will return only those bits in each operand that are true (1), whereas the OR will “merge” the bits of both operands, as shown in Figure 3-4.

Figure 3-4. Python bitwise AND and OR operators

The bitwise operators are useful when there is a need to set a particular bit (OR) or test for a bit with a value of 1 (AND). The XOR operator returns the bitwise difference between two operands, as shown in the truth table in Figure 3-5.

Figure 3-5. Python bitwise XOR operator

The one’s complement operator changes the value of each bit to its inverse. That is, a binary value of 00101100 becomes 11010011.

The binary shift operators work by shifting the contents of a data object left or right by the number of bit positions specified by the righthand operand. The effect is the equivalent of multiplication by 2ⁿ for a left shift or division by 2ⁿ for a right shift (where n is the number of bit positions shifted). For example:

>>> 2 << 1
4
>>> 2 << 2
8
>>> 2 << 3
16
>>> 16 >> 2
4

As we’ve already seen, assignment in Python involves more than just stuffing some data into a memory location. An assignment is equivalent to instantiating a new data object. Python’s assignment operators are listed in Table 3-8.

In addition to the simple assignment operator, Python provides a set of augmented assignment operators as corollaries to each of the arithmetic operators. An augmented assignment first performs the operation and then assigns the result back to the name on the lefthand side of the operator. For example:

>>> a = 1
>>> a += 1
>>> a
2

The membership operators are used to determine whether a value or object exists (in), or doesn’t (not in), within a sequence or dictionary object (see Table 3-9). Note that when used with a dictionary only the keys are tested, not the values.

One way to use the in operator would be like this:

if x in some_list:
    DoSomething(x, some_list)

In this case, the function doSomething() will only be called if x is in some_list. Conversely, one could test to see if something is not in an object:

if x not in some_dict:
    some_dict[x] = new_value

If the key x does not already exist in the dictionary, it will be added along with a value.

Python’s identity operators (shown in Table 3-10) are used to determine if one name refers to the same object as another name (is), or if it does not (is not).

The identity operators are handy when attempting to determine if an object is available for a particular operation. An is expression will evaluate to True if the variable names on either side of the operator refer to the same object. An is not expression will evaluate to True if the variable names on either side of the operator do not refer to the same object.

Here is a (nonexecutable) example:

def GetFilePath(name):
    global pathParse

    if pathParse is None:
        pathParse = FileUtil.PathParse()

    file_path = pathParse(name)
    if len(file_path) > 1:
        return file_path
    else:
        return None

The global name pathParse would be initialized (at the start of the module) to None, but for this function it should refer to an object of the class pathParse in the FileUtil module. If it does not (i.e., it is None), it is instantiated. If the function attempts to use pathParse with a value of None, it will fail.

We already saw some of the precedence characteristics of operators in the earlier discussion of expressions, but now let’s take a closer look. Table 3-11 lists Python’s operators in order of precedence, from lowest to highest.

Parentheses are used to force the order of evaluation, as was shown earlier. If you can’t remember how the evaluation order works, or the default order isn’t what you want, use parentheses as necessary to get the desired result. Using parentheses for clarity is never a bad thing.

When talking about program structure, one often refers to blocks of statements. A block can be defined as a set of one or more statements that are logically associated. Unlike C and some other languages, Python does not use special characters or reserved words to denote how statements are logically grouped into blocks. It uses indentation. For example, in C, one could write the if-else shown above like this:

if (some_var < 10) {
    printf("Yes
");
    printf("Indeed
");
}
else {
    printf("Sorry
");
    printf("Nope
");
}

The curly braces tell the C compiler how the statements are grouped, and C does not care how much or how little each statement is indented—in C that’s considered to be “whitespace,” and the compiler ignores it. In Python, however, the indentation is essential, as it tells the interpreter how the code is structured and which statements are logically associated. The amount of indentation is not critical so long as it is consistent. The recommended amount is four spaces for each level, and no tabs (tabs are generally considered somewhat evil because they don’t always move between different editors gracefully—one editor might interpret tabs as four spaces, whereas another might translate tabs to eight spaces).

Some people have issues with Python’s use of indentation to denote blocks of code, and for those with extensive experience in C or C++ it does seem rather odd at first (although it is by no means a new idea in computer science). The advantages claimed for indentation are that it helps to enforce a consistent style across different programs by different authors and that it improves readability. Some people find that using comments such as #endif, #endfor, and #endwhile helps to make large sections of code with multiple levels of indentation easier to read, but we won’t get into that discussion here.

In Python, a comment is denoted by a # character (sometimes called a hash), and a comment can appear anywhere on a line. The interpreter ignores everything following the hash. Use comments liberally to document your programs, but make the comments worthwhile. A comment like this:

a += 1    # increment by one

isn’t very useful (although they still show up quite often), but a comment like this:

if (a + 1) > maxval:     # do not increment past limit

can help to dispel mystery.

Python utilizes 31 distinct reserved keywords, listed in Table 3-12.

We will examine some of the more commonly used keywords in the remainder of this chapter. Others will be introduced as necessary when we start developing some larger and more complex programs.

In Python, a simple statement (see Table 3-13) is one that consists of an assignment or keyword in a single line; there are no other components. The statement may have more than one expression, however.

I have intentionally skipped the del, exec, raise, and yield statements in the following subsections because they really won’t come into play for what we want to do in this book. A discussion of the import statement is deferred until later in this chapter, in the section titled Importing Modules.

if (a += 1) > maxval:

if (a + 1) > maxval:

Compound statements are composed of groups of statements that are logically related and control the execution of other statements. We will take a look at the if, while, for, and try statements, but we will skip the with statement and save the def and class statements until the next section. Table 3-14 lists Python’s compound statements.

if <expression>:
    statement
    (more statements as necessary)

if <expression>:
    statement
    (more statements as necessary)
else:
    statement
    (and yet more statement if necessary)

if <expression>:
    statement
    (more statements as necessary)
elif <expression>:
    statement
    (more statements as necessary)
else:
    statement
    (and yet more statements if necessary)

while <expression>:
    statement
    (more statements as necessary)
else:
    statement
    (and yet more statement if necessary)

>>> loop_ok = True
>>> loop_cnt = 10
>>> while loop_ok:
...     print "%d Loop is OK" % loop_cnt
...     loop_cnt -= 1
...     if loop_cnt < 0:
...         loop_ok = False
... else:
...     print "%d Loop no longer OK" % loop_cnt
...
10 Loop is OK
9 Loop is OK
8 Loop is OK
7 Loop is OK
6 Loop is OK
5 Loop is OK
4 Loop is OK
3 Loop is OK
2 Loop is OK
1 Loop is OK
0 Loop is OK
-1 Loop no longer OK

for some_var in <sequence>:
    statement
    (more statements as necessary)
else:
    statement
    (and yet more statement if necessary)

>>> for i in range(0,5):
...     print i
...
0
1
2
3
4

>>> alist = [1,2,3,4,5,6,7,8,9,10]
>>> for i in alist:
...     print i
...
1
2
3
4
5
6
7
8
9
10

>>> stuple = ("this","is","a","4-tuple")
>>> for s in stuple:
...     print s
...
this
is
a
4-tuple

try:
    statement
    (more statements as necessary)
except <exception, err_info>:
    statement
    (more statements as necessary)
else:
    statement
    (more statements as necessary)
finally:
    statement
    (and yet more statements if necessary)

try:
    f = open(fname, "r")
except Exception, err:
    print "File open failed: %s" % str(err)

A string literal is quoted using one of the following forms:

'A single-quote string.'
"A double-quote string."
'''This is a multiline string using triple single quotes.
It is a medium-length string. '''
"""This is a multiline string with triple double quotes containing many
characters along with some punctuation, and it is a very long string indeed."""

Multiline strings can span more than one line, and (newline) characters are inserted into the string automatically to preserve the original formatting.

The string type provides numerous methods, some of which we have already seen. Table 3-15 is a complete list (not including the Unicode methods) as given in the Python 2.6 documentation.

Some of these get a lot more use than others, but it’s good to have some idea of what’s available. For the methods we don’t cover here, refer to the Python documentation. Also, remember that the form:

new_sring = "string text".method()

works just as well as:

new_string = string_var.method()

Also keep in mind that there needs to be a target name for the new string object created as a result of invoking the method (strings are not mutable); otherwise, the modified string data will simply vanish.

Table 3-16 lists 14 commonly used string methods. Other less commonly used methods will be described as the need arises. In the following descriptions I will use the convention used in the Python documentation to indicate (required) and [optional] parameters.

There are basically two ways to format a string with variable data in Python. The first is to use concatenation, which we saw earlier. The second is to make use of Python’s string formatting capability. Which method is most appropriate depends on what you are trying to accomplish. While concatenation is relatively easy with simple strings, it doesn’t provide for a lot of control over things like the number of decimal places, and data in strings with lots of embedded characters can be cumbersome when using concatenation. Consider the following example:

>>> data1 = 5.05567
>>> data2 = 34.678
>>> data3 = 0.00296705087
>>> data4 = 0
>>> runid = 1
>>> outstr1 = "Run "+str(runid)+": "+str(data1)+" "+str(data2)
>>> outstr2 = " "+str(data3)+" : "+str(data4)
>>> outstr = outstr1 + outstr2
>>> outstr
'Run 1: 5.05567 34.678 0.00296705087 : 0'

There is an easier way. Python employs string formatting placeholders that are very similar to those used in the C sprintf() function. By using special formatting codes, one can specify where data is to be inserted into a string and how the data will appear in string form. Here is a string created using formatting placeholders with the same data as above:

>>> outstr = "Run %d: %2.3f %2.3f %2.3f : %d" % (runid, data1, data2, data3, data4)
>>> outstr
'Run 1: 5.056 34.678 0.003 : 0'

Notice that the variables to be used in the string are enclosed in parentheses—it is an n-tuple. If the parentheses are omitted, only the first variable name is evaluated and an error will result:

>>> "%d %d %d" % 1, 2, 3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: not enough arguments for format string

The syntax for a string format placeholder is:

%[(name)][flags][width][.precision]type_code

Each placeholder can have an optional name assigned to it as the first item after the % (the parentheses are required). Following that are optional flags for justification, leading spaces, a sign character, and 0 fill. Next is an optional width value that specifies the minimum amount of space to allow for the data. If the data contains a decimal part, the value of the .precision field specifies the number of decimal places to use. Finally, a type code specifies what kind of data to expect (string, integer, long, floating point, etc.). Table 3-17 lists the available flags. Table 3-18 summarizes the various type codes available.

String methods can be applied along with string formatting in the same statement. This may look a bit odd, but it’s perfectly valid:

>>> "%d %d".ljust(20) % (2, 5)
'2 5               '
>>> "%d %d".rjust(20) % (2, 5)
'               2 5'
>>>

Because there is no assignment of the string object to a name, Python just prints it out immediately after applying the formatting.

Lastly, Python provides a set of so-called escape characters for use with strings. These are special two-character codes composed of a backslash followed by a character, as shown in Table 3-19.

A backslash character () may also be used for line continuation if it is the last character on a line, followed immediately by a newline (LF or CRLF). This causes the newline to be ignored by the interpreter, so it treats the line and the subsequent line as a single line of code.

Earlier I obliquely referred to the notion of scope without actually defining what it is. Let’s do that now.

As I’ve already mentioned, Python utilizes the concept of namespaces as collections of names that are bound to objects. Actually, a namespace is more like a dictionary object, where the names are the keys and the objects they reference are the values. There are three levels of namespaces in Python: local, global, and built-ins. Figure 3-6 shows the namespace scopes in a Python module.

When a name is referenced in a function or method, a search is first made of the local namespace, including enclosing functions. Next, the global namespace is searched. Finally, the built-in namespace is searched. If the name cannot be found, Python will raise an exception. Figure 3-7 shows the namespace search hierarchy.

Figure 3-6. Python’s namespaces

Local scope

The local scope is the namespace of a particular function, class, or method. In other words, any variables defined within a function are local to that function and are not visible outside of it. The local scope also includes the nearest surrounding function (if any). We will look at nested functions shortly.

Figure 3-7. Namespace search hierarchy

Class objects introduce yet another namespace into the local context. In a class object, any variables defined within the namespace of the class are accessible to any method within the class by prefixing the name with self, like this:

self.some_var

The data variable attributes and methods of an object instance of a class are visible outside of the object and may be accessed using the “dot notation” we’ve already seen:

SomeObj = SomeClass()
SomeObj.var_name = value

This will assign a value to the attribute var_name in the object instance SomeObj. If var_name does not exist, it will be created in the object’s context. This leads us to an interesting observation: Python objects do not have truly private data or methods in the sense that they cannot be accessed from outside of the object. Everything is accessible, although some things are not as readily available as others. You can prefix the name of a function, class, or variable with a leading underscore to prevent it from being included in a wildcard import, but that doesn’t hide it. Using two leading underscore characters will “mangle” the object’s name, but even then it is still accessible if you know how. So, while nothing is really hidden, the onus is on the programmer to be polite and not look.

If you’re not sure exactly what this means, don’t worry about it for now. We’ll address objects in more detail later, when we start building user interfaces for our instrumentation applications.

# globals.py

var1 = 0
var2 = 1

def Function1():
    var1 = 1
    var2 = 2

    print var1, var2

def Function2():
    global var1, var2

    print var1, var2

    var1 = 3
    var2 = 4

    print var1, var2

>>> import globals

>>> help(globals)
Help on module globals:

NAME
    globals

FILE
    globals.py

FUNCTIONS
    Function1()

    Function2()

DATA
    var1 = 3
    var2 = 4

>>> globals.var1
0
>>> globals.var2
1
>>> globals.Function1()
1 2
>>> globals.var1
0
>>> globals.var2
1

>>> globals.Function2()
0 1
3 4
>>> globals.var1
3
>>> globals.var2
4

# globals2.py

var1 = 0
var2 = 1

def Function1():
    print var1, var2

def Function2():
    var1 = 1
    var2 = 2

    print var1, var2

def Function3():
    print var1, var2

    var1 = 1
    var2 = 2

    print var1, var2

>>> import globals2
>>> globals2.Function1()
0 1
>>> globals2.Function2()
1 2
>>> globals2.Function3()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "globals2.py", line 14, in Function3
    print var1, var2
UnboundLocalError: local variable 'var1' referenced before assignment

A Python source code file is called a module. It is a collection of statements composed of variable definition statements, import statements, directly executable statements, function definition statements, and class definition statements, with the variables and methods that go with them.

Modules are contained within packages. A package is, in effect, a directory that contains one or more modules. Packages may contain other packages. Figure 3-8 shows this graphically.

A module is an object, and as we’ve already seen, it has its own namespace. A module also has attributes just like any other Python object. A module’s attributes include the functions, classes, methods, and variables defined in its namespace.

The def statement is used to define both functions within modules and methods within classes:

def SomeName (parameters):
    """ docstring goes here.
    """
    local_var = value

    statement...
    statement...
    more statements...

When used to define a function, the def statement begins at the leftmost column and all of the function’s statements are indented relative to it. When used to define a method in a class, the def statement is indented relative to the class statement.

Figure 3-8. Packages and modules

Functions and methods may be nested. When this is done, the internal functions are not accessible from outside of the enclosing function. Here is a rather contrived example of nested functions called subfuncs.py:

#subfuncs.py

def MainFunc():
    def SubFunc1():
        print "SubFunc1"
    def SubFunc2():
        print "SubFunc2"
    def SubFunc3():
        def SubSubFunc1():
            print "SubSubFunc1"
        def SubSubFunc2():
            print "SubSubFunc2"
        SubSubFunc1()
        SubSubFunc2()
    SubFunc1()
    SubFunc2()
    SubFunc3()

We can only execute the function MainFunc(); none of the other functions nested within it are directly accessible from outside of the scope of MainFunc(). If you import subfuncs and try to get help on it, this is all you will see:

>>> import subfuncs
>>> help(subfuncs)
Help on module subfuncs:
NAME
    subfuncs
FILE
    subfuncs.py
FUNCTIONS
    MainFunc()

However, if we execute MainFunc() we can see that the subfunctions do get executed:

>>> import subfuncs
>>> subfuncs.MainFunc()
SubFunc1
SubFunc2
SubSubFunc1
SubSubFunc2

The class statement defines a class object, which in turn is used to create object instances of the class. The following class defines a timer object that may be used to get elapsed times during program execution:

import time

class TimeDelta:
    def __init__(self):
        self.tstart = 0
        self.tlast  = 0
        self.tcurr  = 0

        self.Reset()

    def GetDelta(self):
        """ Returns time since last call to GetDelta(). """
        self.tcurr = time.clock()
        delta = self.tcurr - self.tlast
        self.tlast = self.tcurr
        return delta

    def GetTotal(self):
        """ Returns time since object created. """
        return time.clock() - self.tstart

    def Reset(self):
        """ Initializes time attributes. """
        self.tstart = time.clock()
        self.tlast = self.tstart

Objects of this class can be instantiated in the code wherever one might want to check on elapsed times, and multiple occurrences may exist simultaneously. This would be rather awkward to do if TimeDelta was a function in a module, but as a class each instance can maintain its own data for when it was started and when it was last checked.

Docstrings are used to document modules, classes, methods, and functions. A multiline string that appears at the start of a module, function, class, or method is seen by Python as a docstring, and it is stored in the object’s internal __doc__ variable. This is what you are seeing when you type help() for a specific function at the command-line prompt.

The following example shows how docstrings are used. The pass statement has been used so that we can import this code and use help() to display the embedded documentation:

#docstrings.py

""" Module level docstring.

    This describes the overall purpose and features of the module.
    It should not go into detail about each function or class as
    each of those objects has its own docstring.
"""

def Function1():
    """ A function docstring.

        Describes the purpose of the function, its inputs (if any)
        and what it will return (if anything).
    """
    pass

class Class1:
    """ Top-level class docstring.

        Like the module docstring, this is a general high-level
        description of the class. The methods and variable
        attributes are not described here.
    """

    def Method1():
        """ A method docstring.

            Similar to a function docstring.
        """
        pass

    def Method2():
        """ A method docstring.

            Similar to a function docstring.
        """
        pass

When the help() function is used on this module, the following output is the result:

>>> import docstrings
>>> help(docstrings)
Help on module docstrings:

NAME
    docstrings - Module level docstring.

FILE
    docstrings.py

DESCRIPTION
    This describes the overall purpose and features of the module.
    It should not go into detail about each function or class as
    each of those objects has its own docstring.

CLASSES
    Class1

    class Class1
     |  Top-level class docstring.
     |
     |  Like the module docstring, this is a general high-level
     |  description of the class. The methods and variable
     |  attributes are not described here.
     |
     |  Methods defined here:
     |
     |  Method1()
     |      A method docstring.
     |
     |      Similar to a function docstring.
     |
     |  Method2()
     |      A method docstring.
     |
     |      Similar to a function docstring.

FUNCTIONS
    Function1()
        A function docstring.

        Describes the purpose of the function, its inputs (if any)
        and what it will return (if anything).

The import statement comes in several different forms. This is the most common, and safest, form:

import module

The objects in module are added to the current namespace as references of the form module.function() or module.class(). To access data attributes within a module, the notation module.variable is used.

A variation on this is the aliased import form:

import module as alias

This is identical to the import module statement, except now the alias can be used to reference objects in module. This is handy when a module has a long name. For example:

import CommonReturnCodes as RetCodes

One can also specify what to import from a module:

from module import somename

This form imports a specific function, class, or data attribute from a module. The function or attribute somename can then be used without a module prefix.

The wildcard form imports everything from the external module and adds it to the current namespace:

from module import *

The wildcard import is generally considered to be a bad idea except in special cases, such as when importing a module that has been specifically designed to be used in this fashion and contains only unique names that are unlikely to conflict with existing names. It is considered problematic because it imports everything from the imported module unless special precautions are taken. If the imported module happens to have attributes with the same names as those in the current module, the current names will be overwritten.

There is, as one might expect from Python, a way to control what is exported by using single or double leading underscore characters for attribute names. An attribute name of the form:

_some_name

will not be included in a wildcard import, but it can still be referenced using the module prefix notation. A double leading underscore of the form:

__some_name

is about as close as Python gets to data hiding. It can still be accessed from outside the parent module, but its external name is “mangled” to make it more difficult to get at.

Because Python executes any import statements that are not within the scope of a function immediately when a module is imported, it will descend through the import statements in each module in a depth-first fashion until all imports have been processed. Figure 3-9 shows graphically how this works.

Figure 3-9. Module import sequence

The import sequence in Figure 3-9 is indicated by numbers in circles. Module A imports module B, which imports Module C, which in turn imports modules G and H. Module D and the modules it imports will be next in line after module H is processed.

One drawback to Python’s import scheme is that it is possible to create situations where imports can become “hung.” This is called a cyclic import. Consider the diagram in Figure 3-10.

Figure 3-10. Cyclic import situation

Here we have a situation where Module A imports Module B, which in turn imports Modules C and D. However, Module C attempts to import Module A, which is currently waiting for Module B to finish importing Module C, so Module B can then move on to Modules D and E. Because the import of Module B cannot complete, the entire process deadlocks.

One sure way to avoid cyclic imports is to remember the rule “Never import up, only down.” This means that modules should be imported hierarchically, and also that modules should be architected such that there is no need to import from a higher-level module. A typical mistake made by many newcomers to Python is to place a set of pseudoconstants (assignments to names with values that don’t change) in a module with other related functionality, and then import the entire module solely to gain access to the pseudoconstant objects. Things like pseudoconstants that are referenced by more than one module should go into their own module, which can then be imported when needed without worrying about causing a cyclic import situation.

Getting user input from stdin (standard input) is straightforward. Python provides the raw_input() function for just this purpose.

The module getInfo.py contains a simple example of how raw_input() can be used:

# getInfo.py

def ask():
    uname = raw_input("What is your name? ")
    utype = raw_input("What kind of being are you? ")
    uhome = raw_input("What planet are you from? ")
    print ""
    print "So, %s, you are a %s from %s." % (uname, utype, uhome)
    uack = raw_input("Is that correct? ")
    if uack[0] in ('y', 'Y'):
        print "Cool. Welcome."
    else:
        print "OK, whatever."

To see how this works, we can import the module getInfo and then call its function ask():

>>> import getInfo
>>> getInfo.ask()
What is your name? zifnorg
What kind of being are you? Zeeble
What planet are you from? Arcturus III

So, zifnorg, you are a Zeeble from Arcturus III.
Is that correct? y
Cool. Welcome.

The raw_input() function accepts an optional prompt string and always returns the data from stdin as a string. If the program is looking for a numeric value, it will need to be converted. A safe way to do this is by using the try-except construct. Here is getInfo2.py with the try-except modification:

def ask2():
    uname = raw_input("What is your name? ")
    utype = raw_input("What kind of being are you? ")
    uhome = raw_input("What planet are you from? ")
    getgumps = True
    while (getgumps):
        intmp = raw_input("How many mucklegumps do you own? ")
        try:
            ugumps = int(intmp)
        except:
            print "Sorry, you need to enter an integer number."
            continue
        else:
            getgumps = False
    print ""
    print "So, %s, you are a %s from %s, with %d mucklegumps."
        % (uname, utype, uhome, ugumps)
    uack = raw_input("Is that correct? ")
    if uack[0] in ('y', 'Y'):
        print "Cool. Welcome."
    else:
        print "OK, whatever."

Before we move on, there are a few things to consider about this simple function. First, it will only accept an integer value for the number of “mucklegumps.” Strings and floats will be rejected. Secondly, there is no way for the user to gracefully abort the input process. This could be easily handled by checking for a special character (a ., for example), or just detecting null input (just pressing the Enter key with no input). Speaking of null input, if the user does press the Enter key in response to the last question, Python will raise an exception:

Is that correct? <enter>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "getInfo2.py", line 18, in ask2
    if uack[0] in ('y', 'Y'):
IndexError: string index out of range

The expression in the if statement is attempting to match whatever is in uack[0] with either of the values in the 2-tuple ('y', 'Y'), and just pressing Enter returns a zero-length string, which causes the exception. Using a try-except here will prevent this from happening:

    uack = raw_input("Is that correct? ")
    try:
        if uack[0] in ('y', 'Y'):
            print "Cool. Welcome."
        else:
            print "OK, whatever."
    except:
        print "Fine. Have a nice day."

When dealing with user input (that is, whatever a human being types in response to a prompt), one must always be aware of possible input errors or exceptions. Humans can, and often will, type in erroneous data, values that are out of range, unexpected words or phrases, or even nothing at all. Users are unpredictable, so building in safeguards to catch bad input values is always a good idea.

Program parameters entered at the command line are captured by the operating system and passed to the program via the Python interpreter as a list. The first item in the list (at index 0) is always the name of the program itself. Python’s included sys module contains methods for dealing with this data.

This simple program (argshow.py) will print out all the items from the command-line parameter list:

import sys

print "%d items in argument list
" % len(sys.argv)

i = 1
for arg in sys.argv:
    print "%d: %s" % (i, arg)
    i += 1

And here is what happens when we run it:

C:samples> python argshow.py 1 2 3 4 -h -v
7 items in argument list

1: argshow.py
2: 1
3: 2
4: 3
5: 4
6: -h
7: -v

Python also provides tools for detecting specific arguments and extracting values from command-line parameters, which we won’t cover at this point. We will see them in action in later chapters.

Python has a basic built-in object type for dealing with files that provides methods to read and write data from and to a disk file, among other actions. We’ve already seen a little bit of it with the pgmrand.py script we looked at earlier.

The open() method is used to create an instance of a file object:

>>> fname = "test1.txt"
>>> fmode = "w"
>>> f = open(fname, fmode)

Of course, you could also write:

f = open("test1.txt", "w")

and get the same result.

Once we have a file object, we can write something to it using its write() method:

>>> f.write("Test line 1
")
>>> f.write("Test line 2
")
>>> f.close()

The resulting file should now contain two lines of text:

Test line 1
Test line 2

Notice that the strings to be written to the file end with a (the code for a newline character). The file write() method does not append a newline to the end of a string like print does, so it must be explicitly included in the string.

Table 3-20 lists the most commonly encountered file modes.

Table 3-21 lists some commonly used file object methods. For a description of the other file object methods that are available, refer to the Python documentation.

We’ve already seen Python’s print function in action. Its primary purpose is to send output to whatever is currently defined as stdout (standard output). The print function is capable of handling conversions between numeric types and strings for console output in a transparent fashion. The string formatting discussed earlier works with the print statement to create nicely formatted output.

By default, the output of print is sent to whatever is currently defined as stdout. By using the “chevron” (>>) operator this behavior can be modified, and print can send output to any object that provides a write() method. Typically this would be a file object, as shown here:

>>> datastr = "This is a test."
>>> f = open("testfile.txt", "w")
>>> print >> f,datastr
>>> f.close()

It is usually a good idea to initialize module global `variables at the start of a module file. Attempting to check a global variable that does not yet exist will result in an exception, and taking care of this beforehand can save some aggravation later.

Because Python does not execute the internal statements (i.e., the body) of functions or methods when a module is imported, only the def statement, it is always possible for bugs to be lurking there that will not become apparent until the code is invoked. In such situations the try statement is a powerful ally, but it is not a cure-all. Good unit testing is key to detecting and removing such defects before they can cause problems.

Sometimes you may encounter code where the original author attempted to resolve a cyclic import by deferring the import of the problematic module by placing the import statement within a function or method, instead of at the top of the module file. While this is syntactically allowed in Python it is considered to be bad form, and it’s a sure sign that someone didn’t think the design through before sitting down at the keyboard and hammering away at it. However, when dealing with legacy code (or just poorly written code) it may not be possible to avoid using this trick. Use it sparingly, only when you really have to, and test it thoroughly.

Although Python allows any data object to be used as a parameter to a function or method, resist the temptation to use dictionary objects unless you have a compelling reason to do so. If a dictionary object is used as a parameter, document it in detail and try to avoid altering its structure dynamically as it gets passed from function to function. Code that dynamically alters the structure of a shared dictionary object can be very difficult to understand and a nightmare to debug. It could even be considered a form of obfuscation, albeit (hopefully) unintentional. The same common-sense rationale applies to lists.

Tuples are a handy way to return more than one value from a function. For example, one could return a tuple containing both a status code value and a data value by using a 2-tuple. To see if the function succeeded one would examine the status code, and if it is OK one would then get the data value.

Of course, in Python a module actually is an object (everything is, as you may recall), but the tendency seems to be to treat a module as something akin to a source code module in C or C++. One can achieve some neat and tidy data encapsulation using just a module with nothing in it but assignment statements to associate names with values. Here is part of a module that contains nothing but event ID values for use with a wxPython GUI, which we will get to in a later chapter:

# ResourceIDs.py

import  wx

# File
idFileSave                          = wx.NewId()
idFileSaveAs                        = wx.NewId()
idFileNew                           = wx.NewId()
idFileOpen                          = wx.NewId()
idFileOpenGroup                     = wx.NewId()
idFileClose                         = wx.NewId()
idFileCloseAll                      = wx.NewId()
idFilePrint                         = wx.NewId()
idFilePrintPreview                  = wx.NewId()
idFilePrintSetup                    = wx.NewId()

The wxPython package includes a function called NewID() that automatically assigns a new ID number each time it is called. When ResourceIDs is imported, every statement is evaluated and a value is assigned to each data object. To use these one simply imports the module (perhaps using an alias, as shown here):

import ResourceIDs as rID

event_id = rID.idFileSave

This comes in handy in large programs, especially those that employ a GUI with lots of event ID names. A data-only module can also be imported from any other module without worrying about creating a cyclic import, provided that it does not itself import anything else (except perhaps system-level modules). If the attribute names in a data-only module are unique (using, say, a special prefix on each name), it could also be safely imported using the wildcard import style.

Once upon a time, a physics professor told me: “Document everything you do in the lab like you were going to get struck with amnesia tomorrow.” Sage advice, to be sure, but many people are loath to spend the time necessary to include docstrings and descriptive comments in their code. This is silly, because no one can be expected to remember exactly what something does or why it’s even there 12 months or more down the road (some might say that even a couple of months is a stretch). It also says something about how the author of the software feels about those who might come along later and try to fix or maintain the code.

The document known as “PEP-8” (available from http://www.python.org) contains some suggested coding style guidelines. You may not agree with all of it, but you should at least read it and be familiar with it. There is a lot of good advice there. In any case, you should attempt to arrive at some type of consistent style for your code, if for no other reason than that it improves readability and makes things a whole lot easier when there is a need to revisit old code.

If you think you would prefer to use a standalone editor (which is what I use, by the way), there are several excellent packages to choose from. Table 3-22 lists a few of the more popular ones to consider.

This is only a partial list, and there are other editors available, including some good FOSS ones. If you don’t already have a favorite editor (or even if you do), it would probably be worthwhile to try to compare what’s available for your development platform. But, a word of caution: some people seem to become rather attached to a particular editor, even to the point of being somewhat fanatical about it. This is particularly apparent in the Emacs versus vi debate that has been going on now for well over 20 years (refer to http://en.wikipedia.org/wiki/Editor_war for details). Just keep an open mind, select the right tool for the job, and see the editor war for what it really is: free entertainment.

An IDE attempts to integrate everything a programmer might need into a single tool. The first popular and low-cost IDE for the PC was Borland’s Turbo Pascal, developed by Philippe Kahn in the mid-1980s. Most modern IDEs provide a text editor for source code, an interface to a compiler or interpreter, tools to automate the build process, perhaps some support for version control, and a debugger of some sort. In other words, it’s a one-stop shopping experience for software development. Not every IDE will provide all the functionality we’ve listed here, but at the very least you should expect a text editor and the ability to run external tools and applications such as a compiler, interpreter, and debugger. In this sense even editors such as UltraEdit and Emacs (listed in Table 3-22) could be used as IDEs (and often are, actually). Table 3-23 lists some readily available IDE tools suitable for use with Python.