It’s common to feel some helplessness when presented with a new shell prompt for the very first time. It’s not at all obvious exactly what to do. Remember the first time you logged into a Unix box and were presented with a (ba|k|c|z)sh prompt? Since you are reading this book, we assume you have attained some level of familiarity over the Unix shell. If that is the case, then gaining some level of mastery over IPython will be easy.

One of the reasons that it is unclear what you should do when you first see the IPython prompt, is that what you can do is virtually unlimited. So it is more appropriate to think in terms of what you want to do. All of the features of the Python language are available to you from within the IPython prompt. Plus, there are a number of IPython “magic” functions available to you. And you can easily execute any Unix shell command from within IPython and then store the output into a Python variable. The next few examples will demonstrate what you can expect from IPython with its default configuration.

Here are the input and output of a few simple assignment operations:

In [1]: a = 1

In [2]: b = 2

In [3]: c = 3

This doesn’t look much different from what you would see if you entered the same thing at a standard Python prompt. We simply assigned 1, 2, 3 to a, b, c, respectively. The biggest difference you’ll see between the IPython and standard Python shells is that IPython gives you a numbered prompt.

Now that we have values (1, 2, and 3) stored in a few variables (a, b, and c, respectively), we can see the values that those variables contain:

In [4]: print a
1

In [5]: print b
2

In [6]: print c
3

This is a contrived example. We just typed in the values, so we can, at worst, just scroll up to see what the values were. Each variable that we displayed took six characters more than was necessary to display its value. Here’s a slightly less verbose way of showing the value:

In [7]: a
Out[7]: 1

In [8]: b
Out[8]: 2

In [9]: c
Out[9]: 3

While the outlook values look pretty much the same, there is a difference. The print statements use the “unofficial” string representation, while the bare variable names use the “official” string representation. This distinction is typically more important when you are dealing with custom classes than when you are dealing with built-in classes. Here is an example of the different string representations in use:

In [10]: class DoubleRep(object):
....:     def __str__(self):
....:         return "Hi, I'm a __str__"
....:     def __repr__(self):
....:         return "Hi, I'm a __repr__"
....:
....:

In [11]: dr = DoubleRep()

In [12]: print dr
Hi, I'm a __str__

In [13]: dr
Out[13]: Hi, I'm a __repr__

We created the class DoubleRep with two methods, __str__ and __repr__, to demonstrate the difference between printing an object and showing the “official” string representation of it. The special method __str__ on an object will be called when the “unofficial” string representation is needed. The special method __repr__ on an object will be called when the “official” representation is needed. After instantiating our DoubleRep object and assigning the variable dr as its value, we printed out the value of dr. The __str__ method was called. Next, we simply typed in the variable name and the __repr__ method was called. The point of all this is that when we simply type in the name of a variable, the result that IPython displays is the “official” string representation. When we tell IPython to print the variable, we see the “unofficial” string representation. In Python in general, __str__ is what gets called when you call str(obj) on an object or when it is used in a formatted string like this: "%s" % obj. When repr(obj) gets called, or when it is used in a formatting string like this: "%r" % obj, __repr__ is what gets called.

This behavior isn’t particular to IPython, however. This is exactly how the standard Python shell works. Here is the same DoubleRep example using the standard Python shell:

>>> class DoubleRep(object):
...    def __str__(self):
...        return "Hi, I'm a __str__"
...    def __repr__(self):
...        return "Hi, I'm a __repr__"
... 
>>> 
>>> dr = DoubleRep()
>>> print dr
Hi, I'm a __str__
>>> dr
Hi, I'm a __repr__

You may have noticed that the standard Python prompt and the IPython prompt look different. The standard Python prompt consists of three greater-than signs (>>>), whereas the IPython prompt consists of the word “In,” followed by a number in brackets, followed by a colon (In [1]:). A likely reason for this is that IPython keeps track of the commands that you input and stores them in a list named In. After you’ve assigned 1, 2, 3 to a, b, c, in the previous example, here is what the In list would look like:

In [4]: print In
['
', u'a = 1
', u'b = 2
', u'c = 3
', u'print In
']

The output from the IPython prompt is different from the output from the standard Python prompt as well. The IPython prompt seems to distinguish between two types of output: written output and evaluated output. In reality, IPython really doesn’t distinguish between the two types. What happens is that print calls are a side effect of computation; so IPython doesn’t see them, and so it can’t trap them. These print side effects just wind up on stdout, which is where the calling process sent them. However, as IPython executes user’s code, it checks return values. If the return value is not None, it prints it at an Out [number]: prompt.

The standard Python prompt does not even appear to distinguish between these two types of output. If a statement that you typed into an IPython prompt evaluates to some value other than None, IPython will write it to a line that starts with Out, followed by a bracketed number, followed by a colon, and, finally, followed by the value that the statement evaluated to (i.e., Out[1]: 1). Here is an example of what happens when IPython assigns an integer to a variable, evaluates the variable, displays its value, and then prints out that value. Note the differences among the tasks of assigning to the variable, showing what the variable value evaluates to, and printing the value of the variable. First, the IPython prompt:

In [1]: a = 1

In [2]: a
Out[2]: 1

In [3]: print a
1

In [4]:

Next, the standard Python prompt:

>>> a = 1
>>> a
1
>>> print a
1
>>>

There is really no difference between the way IPython and Python assign the integer, IPython prompt, and the standard Python prompt. Both immediately returned an input prompt to the user. But in showing the “official” string representation of a variable, IPython and standard Python are different. IPython shows only an Out prompt, while Python showed the output. For printing, there was no difference; both showed the output with no prompt.

This In [some number]: and Out [some number]: business may have you wondering if there is a deeper difference between IPython and standard Python, or if the difference is purely cosmetic. The difference is definitely deeper. In fact, the difference represents an area of functionality that places IPython into what seems to be a different category of interactive shell from the standard Python shell.

There are two built-in variables that you will want to be aware of. They are In and Out. The former is an IPython input list object and the latter is a dict object. Here is what type says about the ins and outs of In and Out:

In [1]: type(In)
Out[1]: <class 'IPython.iplib.InputList'>

In [2]: type(Out)
Out[2]: <type 'dict'>

After you start using In and Out, this will make sense.

So, what do these datatypes hold?

In [3]: print In
['
', u'type(In)
', u'type(Out)
', u'print In
']

In [4]: print Out
{1: <class 'IPython.iplib.InputList'>, 2: <type 'dict'>}

As you may expect, In and Out, respectively, hold the input that you typed in and the output that non-None statements and expressions evaluated to. Since each line must necessarily have input, it would seem to make sense to keep track of input in some sort of list-like structure. But keeping track of the output in a list-like structure would result in a number of empty fields or fields containing only None. So, since not every line will have evaluatable non-None output, it makes sense to keep track of output in a dictionary-like data structure or even a pure dict object.

Another of the incredibly useful data-entry features of IPython is tab completion by default. The standard Python shell has tab-completion capability if it is compiled with readline support, but you have to do something like the following:

>>> import rlcompleter, readline
>>> readline.parse_and_bind('tab: complete')

This will give us functionality as follows:

>>> import os
>>> os.lis<TAB>
>>> os.listdir
>>> os.li<TAB><TAB>
os.linesep  os.link     os.listdir

After importing rlcompleter and readline and setting the readline tab complete option, we were able to import os, type in os.lis, hit the Tab key once, and have it complete to os.listdir. We were also able to enter os.li, hit the Tab key twice, and get a list of possible completions.

We get this same behavior with IPython for without any extra configuration necessary. Well, it’s free with the standard Python shell as well, but with IPython, it’s the default behavior. Here is the previous example run with IPython:

In [1]: import os

In [2]: os.lis<TAB>
In [2]: os.listdir
In [2]: os.li<TAB>
os.linesep  os.link     os.listdir

Notice that we had to hit the Tab key only once on the last part of the example.

The os.TAB example really only shows off the attribute lookup and completion functionality of IPython, but another cool thing that IPython will complete on is module imports. Open a new IPython shell so that you can see IPython help us find a module to import:

In [1]: import o
opcode       operator     optparse     os           os2emxpath   ossaudiodev  

In [1]: import xm
xml        xmllib     xmlrpclib

Notice that all of the items that import completed on were modules, so this wasn’t accidental behavior. This is a feature.

IPython exposes two types of completion: “complete” and “menu-complete.” The difference between the two is that “complete” expands the current “word” as far as it can and then provides a list of alternatives, while “menu-complete” expands the word fully to match one of the alternatives, and then each subsequent press of the Tab key morphs the word into the next alternative. IPython’s default completion option is “complete.” We’ll get into configuring your IPython in just a bit.

The last basic input and output topic we will cover is the “magic” edit function. (We will go over magic functions in the next section.) Strictly line-oriented user interaction with a shell has tremendous, but limited, usefulness. Since that statement sounds like a contradiction, we’ll unpack it. Typing commands into a shell one line at a time is very useful. You type in a command; the shell goes off and does its thing; you sometimes sit and wait for it to return; you type in your next command. This is not a bad cycle. In fact, it’s quite effective. But sometimes it would be nice to work with a block of lines all at the same time. And it would be nice to work with them in your text editor of choice, although readline support in IPython does improve its usefulness, in this respect. We are aware of using a text editor to create Python modules, but that isn’t what we’re talking about here. We’re talking about more of a compromise between line-oriented input and text editor input to feed commands to the shell. If we can say that adding support for working with blocks of lines of commands would be better, then we can say that a strictly line-oriented interface is limited. So, we can say that a strictly line-oriented interface is exceptionally useful but limited at the same time.

The magic edit function acts as the compromise we just mentioned between pure command-line interaction with the Python shell and interaction using a text editor. The benefit of the compromise is that you have the full power of both environments at your fingertips. You have the benefit of the full-featured power of your text editor of choice. You can easily edit blocks of code and change lines of code around within a loop or a method or function. Plus, you have the nimbleness and agility that comes from directly interacting with the shell. When you combine these two approaches to working with code, a synergistic benefit emerges. You are able to maintain the environment you were working in directly from within your shell, and you can pause, edit, and execute code from within an editor. When you resume working within your shell, you will see the changes you just made in your editor.

The final “basic” information you need to know in order to begin is how to configure IPython. If you didn’t assign a different location when you ran IPython for the first time, it created an .ipython directory in your home directory. Inside the .ipython directory is a file called ipy_user_conf.py. This user file is simply a configuration file that uses Python syntax. In order to help you give IPython the look and feel that you want it to have, the config file contains a wide variety of elements that you can customize. For example, you can choose the colors used in the shell, the components of the shell prompt, and the text editor that will automatically be used use when you %edit text. We won’t go into any more detail than that here. Just know that the config file exists, and it is worth looking through to see if there are some elements you need to or want to configure.

The first feature of a Python/Unix shell bridge that we will look at is the alias magic function. With alias you can create an IPython shortcut to execute system commands. To define an alias, simply type alias followed by the system command (and any arguments for that command). For example:

In [1]: alias nss netstat -lptn

In [2]: nss
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN                        
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN     

There are a few ways to get different input into an alias. One option is the do-nothing approach. If all the extras you wanted to pass into your command can be lumped together, the do-nothing approach may be for you. For example, if you wanted to grep the results of the netstat command above for 80, you could do this:

In [3]: nss | grep 80
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     -

This isn’t passing in extra options, but for the sake of how things are happening, it winds up being the same thing.

Next, there is the do-everything approach. It’s pretty similar to the do-nothing approach except that, by implicitly handling all arguments, you’re explicitly handling all subsequent arguments. Here is an example that shows how to treat the subsequent arguments as a single group:

In [1]: alias achoo echo "|%l|"

In [2]: achoo
||

In [3]: achoo these are args
|these are args|

This demonstrates the %l (percent sign followed by the letter “l”) syntax that is used to insert the rest of the line into an alias. In real life, you would be most likely to use this to insert everything after the alias somewhere in the middle of the implemented command that the alias is standing in for.

And here is the do-nothing example retooled to handle all arguments explicitly:

In [1]: alias nss netstat -lptn %l

In [2]: nss
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN                        
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN                        

In [3]: nss | grep 80
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     

In this example, we really didn’t need to put the %l in there at all. If we had just left it out, we would have gotten up with the same result.

To insert different parameters throughout a command string, we would use the %s substitution string. This example shows how to run the parameters:

In [1]: alias achoo echo first: "|%s|", second: "|%s|"

In [2]: achoo foo bar
first: |foo|, second: |bar|

This can be a bit problematic, however. If you supply only one parameter and two were expected, you can expect an error:

In [3]: achoo foo
ERROR: Alias <achoo> requires 2 arguments, 1 given.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)

On the other hand, providing more parameters than expected is safe:

In [4]: achoo foo bar bam
first: |foo|, second: |bar| bam

foo and bar are properly inserted into their respective positions, while bam is appended to the end, which is where you would expect it to be placed.

You can also persist your aliases with the %store magic function, and we will cover how to do that later in this chapter. Continuing with the previous example, we can persist the achoo alias so that the next time we open IPython, we’ll be able to use it:

In [5]: store achoo
Alias stored: achoo (2, 'echo first: "|%s|", second: "|%s|"')

In [6]:
Do you really want to exit ([y]/n)?
(psa)jmjones@dinkgutsy:code$ ipython -nobanner

In [1]: achoo one two
first: |one|, second: |two|

Another, and possibly easier, way of executing a shell command is to place an exclamation point (!) in front of it:

In [1]: !netstat -lptn
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN                        
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN     

You can pass in variables to your shell commands by prefixing them with a dollar sign ($). For example:

In [1]: user = 'jmjones'

In [2]: process = 'bash'

In [3]: !ps aux | grep $user | grep $process
jmjones   5967  0.0  0.4  21368  4344 pts/0    Ss+  Apr11   0:01 bash
jmjones   6008  0.0  0.4  21340  4304 pts/1    Ss   Apr11   0:02 bash
jmjones   8298  0.0  0.4  21296  4280 pts/2    Ss+  Apr11   0:04 bash
jmjones  10184  0.0  0.5  22644  5608 pts/3    Ss+  Apr11   0:01 bash
jmjones  12035  0.0  0.4  21260  4168 pts/15   Ss   Apr15   0:00 bash
jmjones  12943  0.0  0.4  21288  4268 pts/5    Ss   Apr11   0:01 bash
jmjones  15720  0.0  0.4  21360  4268 pts/17   Ss   02:37   0:00 bash
jmjones  18589  0.1  0.4  21356  4260 pts/4    Ss+  07:04   0:00 bash
jmjones  18661  0.0  0.0    320    16 pts/15   R+   07:06   0:00 grep bash
jmjones  27705  0.0  0.4  21384  4312 pts/7    Ss+  Apr12   0:01 bash
jmjones  32010  0.0  0.4  21252  4172 pts/6    Ss+  Apr12   0:00 bash

This listed all bash sessions belonging to jmjones.

Here’s an example of the way to store the result of a ! command:

In [4]: l = !ps aux | grep $user | grep $process

In [5]: l

Out[5]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: jmjones   5967  0.0  0.4  21368  4344 pts/0    Ss+  Apr11   0:01 bash
1: jmjones   6008  0.0  0.4  21340  4304 pts/1    Ss   Apr11   0:02 bash
2: jmjones   8298  0.0  0.4  21296  4280 pts/2    Ss+  Apr11   0:04 bash
3: jmjones  10184  0.0  0.5  22644  5608 pts/3    Ss+  Apr11   0:01 bash
4: jmjones  12035  0.0  0.4  21260  4168 pts/15   Ss   Apr15   0:00 bash
5: jmjones  12943  0.0  0.4  21288  4268 pts/5    Ss   Apr11   0:01 bash
6: jmjones  15720  0.0  0.4  21360  4268 pts/17   Ss   02:37   0:00 bash
7: jmjones  18589  0.0  0.4  21356  4260 pts/4    Ss+  07:04   0:00 bash
8: jmjones  27705  0.0  0.4  21384  4312 pts/7    Ss+  Apr12   0:01 bash
9: jmjones  32010  0.0  0.4  21252  4172 pts/6    Ss+  Apr12   0:00 bash

You may notice that the output stored in the variable l is different from the output in the previous example. That’s because the variable l contains a list-like object, while the previous example showed the raw output from the command. We’ll discuss that list-like object later in String Processing.”

An alternative to ! is !!, except that you can’t store the result in a variable as you are running it, !! does the same thing that ! does. But you can access it with the _ or _[0-9]* notation that we’ll discuss later in History results.”

Programming a quick ! or !! before a shell command is definitely less work than creating an alias, but you may be better off creating aliases in some cases and using the ! or !! in others. For example, if you are typing in a command you expect to execute all the time, create an alias or macro. If this is a one time or infrequent occurrence, then just use ! or !!.

There is another option for aliasing and/or executing shell commands from IPython: rehashing. Technically, this is creating an alias for shell commands, but it doesn’t really feel like that is what you’re doing. The rehash “magic” function updates the “alias table” with everything that is on your PATH. You may be asking, “What is the alias table?” When you create an alias, IPython has to map the alias name to the shell command with which you wanted it to be associated. The alias table is where that mapping occurs.

IPython exposes a number of variables that you have access to when running IPython, such as In and Out, which we saw earlier. One of the variables that IPython exposes is __IP, which is actually the interactive shell object. An attribute named alias_table hangs on that object. This is where the mapping of alias names to shell commands takes place. We can look at this mapping in the same way we would look at any other variable:

In [1]: __IP.alias_table

Out[1]: 
{'cat': (0, 'cat'),
 'clear': (0, 'clear'),
 'cp': (0, 'cp -i'),
 'lc': (0, 'ls -F -o --color'),
 'ldir': (0, 'ls -F -o --color %l | grep /$'),
 'less': (0, 'less'),
 'lf': (0, 'ls -F -o --color %l | grep ^-'),
 'lk': (0, 'ls -F -o --color %l | grep ^l'),
 'll': (0, 'ls -lF'),
 'lrt': (0, 'ls -lart'),
 'ls': (0, 'ls -F'),
 'lx': (0, 'ls -F -o --color %l | grep ^-..x'),
 'mkdir': (0, 'mkdir'),
 'mv': (0, 'mv -i'),
 'rm': (0, 'rm -i'),
 'rmdir': (0, 'rmdir')}

It looks like a dictionary:

In [2]: type(__IP.alias_table)

Out[2]: <type 'dict'>

Looks can be deceiving, but they’re not this time.

Right now, this dictionary has 16 entries:

In [3]: len(__IP.alias_table)

Out[3]: 16

After we rehash, this mapping gets much larger:

In [4]: rehash

In [5]: len(__IP.alias_table)

Out[5]: 2314

Let’s look for something that wasn’t there before, but should be there now—the transcode utility should be in the alias table now:

In [6]: __IP.alias_table['transcode']

Out[6]: (0, 'transcode')

Excepting that rehashx looks for things on your PATH that it thinks are executable to add to the alias table, rehashx is similar to rehash. So, when we start a new IPython shell and rehashx, we would expect the alias table to be the same size as or smaller than the result of rehash:

In [1]: rehashx

In [2]: len(__IP.alias_table)

Out[2]: 2307

Interesting; rehashx produces an alias table with seven fewer items than rehash. Here are the seven differences:

In [3]: from sets import Set

In [4]: rehashx_set = Set(__IP.alias_table.keys())

In [5]: rehash

In [6]: rehash_set = Set(__IP.alias_table.keys())

In [7]: rehash_set - rehashx_set

Out[7]: Set(['fusermount', 'rmmod.modutils', 'modprobe.modutils', 'kallsyms', 'ksyms', /
       'lsmod.modutils', 'X11'])

And if we look to see why rmmod.modutils didn’t show up when we ran rehashx but did show up when we ran for rehash, here is what we find:

jmjones@dinkgutsy:Music$ slocate rmmod.modutils
/sbin/rmmod.modutils
jmjones@dinkgutsy:Music$ ls -l /sbin/rmmod.modutils 
lrwxrwxrwx 1 root root 15 2007-12-07 10:34 /sbin/rmmod.modutils -> insmod.modutils
jmjones@dinkgutsy:Music$ ls -l /sbin/insmod.modutils
ls: /sbin/insmod.modutils: No such file or directory

So, you can see that rmmod.modutils is a link to insmod.modutils, and insmod.modutils doesn’t exist.

If you have the standard Python shell, you may have noticed that it can be hard to determine which directory you’re in. You can use os.chdir() to change the directory, but that isn’t very convenient. You could also get the current directory via os.getcwd(), but that’s not terribly convenient either. Since you are executing Python commands rather than shell commands with the standard Python shell, maybe it isn’t that big of a problem, but when you are using IPython and have easy access to the system shell, having comparably easy access to directory navigation is critical.

Enter cd magic. It seems like we’re making a bigger deal out of this than it warrants: this isn’t a revolutionary concept; it’s not all that difficult. But just imagine if it were missing. That would be painful.

In IPython, cd works mostly as it does in Bash. The primary usage is cd directory_name. That works as you would expect it to from your Bash experience. With no arguments, cd takes you to your home directory. With a space and hyphen as an argument, cd - takes you to your previous directory. There are three additional options that Bash cd doesn’t give you.

The first is the -q, or quiet, option. Without this option, IPython will output the directory into which you just changed. The following example shows the ways to change a directory both with and without the -q option:

In [1]: cd /tmp
/tmp

In [2]: pwd

Out[2]: '/tmp'

In [3]: cd -
/home/jmjones

In [4]: cd -q /tmp

In [5]: pwd

Out[5]: '/tmp'

Using the -q prevented IPython from outputting the /tmp directory we had gone into.

Another feature that IPython’s cd includes is the ability to go to defined bookmarks. (We’ll explain how to create bookmarks soon.) Here is an example of how to change a directory for which you have created a bookmark:

In [1]: cd -b t
(bookmark:t) -> /tmp
/tmp

This example assumes that we have bookmarked /tmp the name t. The formal syntax to change to a bookmarked directory is cd -b bookmark_name, but, if a bookmark of bookmark_name is defined and there is not a directory called bookmark_name in the current directory, the -b flag is optional; IPython can figure out that you are intending to go into a bookmarked directory.

The final extra feature that cd offers in IPython is the ability to change into a specific directory given a history of directories that have been visited. The following is an example that makes use of this directory history:

0: /home/jmjones
1: /home/jmjones/local/Videos
2: /home/jmjones/local/Music
3: /home/jmjones/local/downloads
4: /home/jmjones/local/Pictures
5: /home/jmjones/local/Projects
6: /home/jmjones/local/tmp
7: /tmp
8: /home/jmjones

In [2]: cd -6
/home/jmjones/local/tmp

First, you see there is a list of all the directories in our directory history. We’ll get to where it came from in a moment. Next, we pass the numerical argument –6. This tells IPython that we want to go to the item in our history marked “6”, or /home/jmjones/local/tmp. Finally, you can see that these are now in /home/jmjones/local/tmp.

We just showed you how to use a cd option to move into a bookmarked directory. Now we’ll show you how to create and manage your bookmarks. It deserves mentioning that bookmarks persist across IPython sessions. If you exit IPython and start it back up, your bookmarks will still be there. There are two ways to create bookmarks. Here is the first way:

In [1]: cd /tmp
/tmp

In [2]: bookmark t

By typing in bookmark t while we’re in /tmp, a bookmark named t is created and pointing at /tmp. The next way to create a bookmark requires typing one more word:

In [3]: bookmark muzak /home/jmjones/local/Music

Here, we created a bookmark named muzak that points to a local music directory. The first argument is the bookmark’s name, while the second is the directory the bookmark points to.

The -l option tells IPython to get the list of bookmarks, of which we have only two. Now, let’s see a list of all our bookmarks:

In [4]: bookmark -l
Current bookmarks:
muzak -> /home/jmjones/local/Music
t     -> /tmp

There are two options for removing bookmarks: remove them all, or remove one at a time. In this example, we’ll create a new bookmark, remove it, and then remove all in the following example:

In [5]: bookmark ulb /usr/local/bin

In [6]: bookmark -l
Current bookmarks:
muzak -> /home/jmjones/local/Music
t     -> /tmp
ulb   -> /usr/local/bin

In [7]: bookmark -d ulb

In [8]: bookmark -l
Current bookmarks:
muzak -> /home/jmjones/local/Music
t     -> /tmp

An alternative to using bookmark -l is to use cd -b:

In [9]: cd -b<TAB>
muzak  t      txt

And after a few backspaces, we’ll continue where we left off:

In [9]: bookmark -r

In [10]: bookmark -l
Current bookmarks:

We created a bookmark named ulb pointing to /usr/local/bin. Then, we deleted it with the -d bookmark_name option for bookmark. Finally, we deleted all bookmarks with the -r option.

In the cd example above, we show a list of the directories we had visited. Now we’ll show you how to view that list. The magic command is dhist, which not only saves the session list, but also saves the list of directories across IPython sessions. Here is what happens when you run dhist with no arguments:

In [1]: dhist
Directory history (kept in _dh)
0: /home/jmjones
1: /home/jmjones/local/Videos
2: /home/jmjones/local/Music
3: /home/jmjones/local/downloads
4: /home/jmjones/local/Pictures
5: /home/jmjones/local/Projects
6: /home/jmjones/local/tmp
7: /tmp
8: /home/jmjones
9: /home/jmjones/local/tmp
10: /tmp

A quick way to access directory history is to use cd -<TAB> like this:

In [1]: cd -
-00 [/home/jmjones]                  -06 [/home/jmjones/local/tmp]
-01 [/home/jmjones/local/Videos]     -07 [/tmp]
-02 [/home/jmjones/local/Music]      -08 [/home/jmjones]
-03 [/home/jmjones/local/downloads]  -09 [/home/jmjones/local/tmp]
-04 [/home/jmjones/local/Pictures]   -10 [/tmp]
-05 [/home/jmjones/local/Projects]

There are two options that make dhist more flexible than cd -<TAB>. The first is that you can provide a number to specify how many directories should be displayed. To specify that we want to see only the last five directories that were visited, we would input the following:

In [2]: dhist 5
Directory history (kept in _dh)
6: /home/jmjones/local/tmp
7: /tmp
8: /home/jmjones
9: /home/jmjones/local/tmp
10: /tmp

The second option is that you can specify a range of directories that were visited. For example, to view from the third through the sixth directories visited, we would enter the following:

In [3]: dhist 3 7
Directory history (kept in _dh)
3: /home/jmjones/local/downloads
4: /home/jmjones/local/Pictures
5: /home/jmjones/local/Projects
6: /home/jmjones/local/tmp

Notice that the ending range entry is noninclusive, so you have to indicate the directory immediately following the final directory you want to see.

A simple but nearly necessary function for directory navigation, pwd simply tells you what your current directory is. Here is an example:

In [1]: cd /tmp
/tmp

In [2]: pwd

Out[2]: '/tmp'

The previous eight or so IPython features are definitely helpful and necessary, but the next three features will give great joy to power users. The first of these is variable expansion. Up to this point, we’ve mostly kept shell stuff with shell stuff and Python stuff with Python stuff. But now, we’re going to cross the line and mingle the two of them. That is, we’re going to take a value that we get from Python and hand it to the shell:

In [1]: for i in range(10):
   ...:     !date > ${i}.txt
   ...:     
   ...:     

In [2]: ls
0.txt  1.txt  2.txt  3.txt  4.txt  5.txt  6.txt  7.txt  8.txt  9.txt

In [3]: !cat 0.txt
Sat Mar  8 07:40:05 EST 2008

This example isn’t all that realistic. It is unlikely that you will want to create 10 text files that all contain the date. But the example does show how to mingle Python code and shell code. We iterated over a list created by the range() function and stored the current item in the list in variable i. For each time through the iteration, we use the shell execution ! notation to call out to the date command-line system utility. Notice that the syntax we use for calling date is identical to the way we would call it if we had defined a shell variable i. So, the date utility is called, and the output is redirected to the file {current list item}.txt. We list the files after creating them and even cat one out to see that it contains something that looks like a date.

You can pass any kind of value you can come up with in Python into your system shell. Whether it is in a database or in a pickle file, generated by computation, an XMLRPC service, or data you extract from a text file, you can pull it in with Python and then pass it to the system shell with the ! execution trick.

Another incredibly powerful feature that IPython offers is the ability to string process the system shell command output. Suppose we want to see the PIDs of all the processes belonging to the user jmjones. We could do that by inputting the following:

ps aux | awk '{if ($1 == "jmjones") print $2}'

This is pretty tight, succinct, and legible. But let’s tackle the same task using IPython. First, let’s grab the output from an unfiltered ps aux:

In [1]: ps = !ps aux

In [2]:

The result of calling ps aux, which is stored in the variable ps, is a list-like structure whose elements are the lines that were returned from the shell system call. It is list-like, in this case, because we mean that it inherits from the built-in list type, so it supports all the methods of that type. So, if you have a function or method that expects a list, you can pass one of these result objects to it as well. In addition to supporting the standard list methods, it also supports a couple of very interesting methods and one attribute that will come in handy. Just to show what the “interesting methods” do, we’ll divert from our end goal of finding all processes owned by jmjones for just a moment. The first “interesting method” we’ll look at is the grep() method. This is basically a simple filter that determines which lines of the output to keep and which to leave off. To see if any of the lines in the output match lighttpd, we would input the following:

In [2]: ps.grep('lighttpd')

Out[2]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: www-data  4905  0.0  0.1........0:00 /usr/sbin/lighttpd -f /etc/lighttpd/l

We called the grep() method and passed it the regular expression 'lighttpd'. Remember, regular expressions passed to grep() are case-insensitive. The result of this grep() call was a line of output that showed that there was a positive match for the 'lighttpd' regular expression. To see all records except those that match a certain regular expression, we would do something more like this:

In [3]: ps.grep('Mar07', prune=True)

Out[3]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1: jmjones  19301  0.0  0.4  21364  4272 pts/2    Ss+  03:58   0:00 bash
2: jmjones  21340  0.0  0.9 202484 10184 pts/3    Sl+  07:00   0:06 vim ipytho
3: jmjones  23024  0.0  1.1  81480 11600 pts/4    S+   08:58   0:00 /home/jmjo
4: jmjones  23025  0.0  0.0      0     0 pts/4    Z+   08:59   0:00 [sh] <defu
5: jmjones  23373  5.4  1.0  81160 11196 pts/0    R+   09:20   0:00 /home/jmjo
6: jmjones  23374  0.0  0.0   3908   532 pts/0    R+   09:20   0:00 /bin/sh -c
7: jmjones  23375  0.0  0.1  15024  1056 pts/0    R+   09:20   0:00 ps aux

We passed in the regular expression 'Mar07' to the grep() method and found that most of the processes on this system were started on March 7, so we decided that we wanted to see all processes not started on March 7. In order to exclude all 'Mar07' entries, we had to pass in another argument to grep(), this time a keyword argument: prune=True. This keyword argument tells IPython, “Any records you find that match the stated regular expression—throw them out.” And as you can see, there are no records that match the 'Mar07' regex.

Callbacks can also be used with grep(). This just means that grep() will take a function as an argument and call that function. It will pass the function to the item in the list that it is working on. If the function returns True on that item, the item is included in the filter set. For example, we could do a directory listing and filter out only files or only directories:

In [1]: import os

In [2]: file_list = !ls

In [3]: file_list

Out[3]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: ch01.xml
1: code
2: ipython.pdf
3: ipython.xml

This directory listing shows four “files.” We can’t tell from this list which are files and which are directories, but if we filter using the os.path.isfile() test, we can see which ones are files:

In [4]: file_list.grep(os.path.isfile)

Out[4]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: ch01.xml
1: ipython.pdf
2: ipython.xml

This left out the “file” named code, so code must not be a file at all. Let’s filter for directories:

In [5]: file_list.grep(os.path.isdir)

Out[5]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: code

Now that we see that code is, in fact, a directory, another interesting method is fields(). After (or, we guess, even before) you filter your result set down to the desired level of specificity, you can display exactly the fields that you want to display. Let’s take the non-Mar07 example that we just walked through and output the user, pid, and start columns:

In [4]: ps.grep('Mar07', prune=True).fields(0, 1, 8)

Out[4]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: USER PID START
1: jmjones 19301 03:58
2: jmjones 21340 07:00
3: jmjones 23024 08:58
4: jmjones 23025 08:59
5: jmjones 23373 09:20
6: jmjones 23374 09:20
7: jmjones 23375 09:20

First, notice that whatever it is that fields() does, we’re doing it to the result of the grep() method call. We are able to do this because grep() returns an object of the same type as the ps object that we started with. And fields() itself returns the same object type as grep(). Since that is the case, you can chain grep() and fields() calls together. Now, on to what is going on here. The fields() method takes an indefinite number of arguments, and these arguments are expected to be the “columns” from the output, if the output lines were split on whitespace. You can think of this very much like the default splitting that awk does to lines of text. In this case, we called fields() to view columns 0, 1, and 8. These are, respectively, USERNAME, PID, and STARTTIME.

Now, back to showing the PIDs of all processes belonging to jmjones:

In [5]: ps.fields(0, 1).grep('jmjones').fields(1)

Out[5]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: 5385
1: 5388
2: 5423
3: 5425
4: 5429
5: 5431
6: 5437
7: 5440
8: 5444
<continues on...>

This example first trims the result set back to only two columns, 0 and 1, which are the username and PID, respectively. Then, we take that narrower result set and grep() for 'jmjones'. Finally, we take that filtered result set and request the second field by calling fields(1). (Remember, lists start at zero.)

The final piece of string processing that we want to showcase is the s attribute of the object trying to directly access your process list. This object is probably not going to give you the results you were looking for. In order to get the system shell to work with your output, use the s attribute on your process list object:

In [6]: ps.fields(0, 1).grep('jmjones').fields(1).s

Out[6]: '5385 5388 5423 5425 5429 5431 5437 5440 5444 5452 5454 5457 5458 5468 
5470 5478 5480 5483 5489 5562 5568 5593 5595 5597 5598 5618 5621 5623 5628 5632 
5640 5740 5742 5808 5838 12707 12913 14391 14785 19301 21340 23024 23025 23373 
23374 23375'

The s attribute gives us a nice space-separated string of PIDs that we can work with in a system shell. We wanted to, we could store that stringified list in a variable called pids and do something like kill $pids from within IPython. But that would send a SIGTERM to every process owned by jmjones, and it would kill his text editor and his IPython sessions.

Earlier, we demonstrated that we could accomplish the stated goals for our IPython script with the following awk one-liner:

ps aux | awk '{if ($1 == "jmjones") print $2}'

We will be ready to accomplish this goal after we’ve introduced one more concept. The grep() method takes a final optional parameter called field. If we specify a field parameter, the search criteria has to match that field in order for that item to be included in the result set:

In [1]: ps = !ps aux

In [2]: ps.grep('jmjones', field=0)

Out[2]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: jmjones   5361  0.0  0.1  46412  1828 ?        SL   Apr11   
  0:00 /usr/bin/gnome-keyring-daemon -d
1: jmjones   5364  0.0  1.4 214948 14552 ?        Ssl  Apr11   
  0:03 x-session-manager
....
53: jmjones  32425  0.0  0.0   3908   584 ?        S    Apr15  
   0:00 /bin/sh /usr/lib/firefox/run-mozilla.
54: jmjones  32429  0.1  8.6 603780 88656 ?        Sl   Apr15   
  2:38 /usr/lib/firefox/firefox-bin

This matched the exact rows that we wanted, but printed out the whole row. To get at just the PID, we’ll have to do something like this:

In [3]: ps.grep('jmjones', field=0).fields(1)

Out[3]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: 5361
1: 5364
....
53: 32425
54: 32429

And with that, we are able to meet the goal of performing that specific awk filter.

One IPython concept that we haven’t described yet is a profile. A profile is simply a set of configuration data that is loaded when you start IPython. You can customize a number of profiles to make IPython perform in different ways depending on a session’s needs. To invoke a specific profile, use the -p command-line option and specify the profile you’d like to use.

The sh, or shell, profile is one of the built-in profiles for IPython. The sh profile sets some configuration items so that IPython becomes a more friendly system shell. Two examples of configuration values that are different from the standard IPython profile are that sh displays the current directory and it rehashes your PATH so that you have instant access to all of the same executables that you would have in, say, Bash.

In addition to setting certain configuration values, the sh profile also enables a few shell-helpful extensions. For example, it enables the envpersist extension. The envpersist extension allows you to modify various environment variables easily and persistently for your IPython sh profile, and you don’t have to update a .bash_profile or .bashrc.

Here, is what our PATH looks like:

jmjones@dinkgutsy:tmp$ ipython -p sh
IPython 0.8.3.bzr.r96   [on Py 2.5.1]
[~/tmp]|2> import os
[~/tmp]|3> os.environ['PATH']
       <3> '/home/jmjones/local/python/psa/bin:
       /home/jmjones/apps/lb/bin:/home/jmjones/bin:
       /usr/local/sbin:/usr/local/bin:/usr/sbin:
       /usr/bin:/sbin:/bin:/usr/games'

Now we add :/appended to the end of our current PATH:

[~/tmp]|4> env PATH+=:/appended
PATH after append = /home/jmjones/local/python/psa/bin:
/home/jmjones/apps/lb/bin:/home/jmjones/bin:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:
/sbin:/bin:/usr/games:/appended

and /prepended: to the beginning of our current PATH:

[~/tmp]|5> env PATH-=/prepended:
PATH after prepend = /prepended:/home/jmjones/local/python/psa/bin:
/home/jmjones/apps/lb/bin:/home/jmjones/bin:/usr/local/sbin:
/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/appended

This shows the PATH environment variable using os.environ:

[~/tmp]|6> os.environ['PATH']
       <6> '/prepended:/home/jmjones/local/python/psa/bin:
       /home/jmjones/apps/lb/bin:/home/jmjones/bin:
       /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:
       /bin:/usr/games:/appended'

Now we’ll exit our IPython shell:

[~/tmp]|7>
Do you really want to exit ([y]/n)?
jmjones@dinkgutsy:tmp$

Finally, we’ll open a new IPython shell to see what the PATH environment variable shows:

jmjones@dinkgutsy:tmp$ ipython -p sh
IPython 0.8.3.bzr.r96   [on Py 2.5.1]
[~/tmp]|2> import os
[~/tmp]|3> os.environ['PATH']
       <3> '/prepended:/home/jmjones/local/python/psa/bin:
       /home/jmjones/apps/lb/bin:/home/jmjones/bin:/usr/local/sbin:
       /usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/appended'

Interestingly, it shows our prepended and appended values, even though we didn’t update any profile scripts. It persisted the change to PATH without any additional work on our part. Now let’s display all persistent changes to environment variables:

[~/tmp]|4> env -p
       <4> {'add': [('PATH', ':/appended')], 'pre': [('PATH', '/prepended:')], 'set': {}}

We can delete any persistent changes to PATH:

[~/tmp]|5> env -d PATH
Forgot 'PATH' (for next session)

and we can check to see the value of PATH:

[~/tmp]|6> os.environ['PATH']
       <6> '/prepended:/home/jmjones/local/python/psa/bin:/home/jmjones/apps/lb/bin:
       /home/jmjones/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:
       /sbin:/bin:/usr/games:/appended'

Even after we’ve told IPython to remove the persistent entries for PATH, they are still there. But that makes sense. That just means that IPython should remove the directive to persist those entries. Note that the process started with certain values in an environment variable will retain those values unless something changes them. The next time the IPython shell starts, things should be different:

[~/tmp]|7>
Do you really want to exit ([y]/n)?
jmjones@dinkgutsy:tmp$ ipython -p sh
IPython 0.8.3.bzr.r96   [on Py 2.5.1]
[~/tmp]|2> import os
[~/tmp]|3> os.environ['PATH']
       <3> '/home/jmjones/local/python/psa/bin:/home/jmjones/apps/lb/bin:
       /home/jmjones/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:
       /sbin:/bin:/usr/games'

And, just as we would expect, this is back to what it was before we started making changes to our PATH.

One other useful feature in the sh profile is mglob. mglob has a simpler syntax for a lot of common uses. For example, to find all of the .py files in the Django project, we could just do this:

[django/trunk]|3> mglob rec:*py
              <3> SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: ./setup.py
1: ./examples/urls.py
2: ./examples/manage.py
3: ./examples/settings.py
4: ./examples/views.py
...
1103: ./django/conf/project_template/urls.py
1104: ./django/conf/project_template/manage.py
1105: ./django/conf/project_template/settings.py
1106: ./django/conf/project_template/__init__.py
1107: ./docs/conf.py
[django/trunk]|4>

The rec directive simply says to look recursively for the following pattern. In this case, the pattern is *py. To show all directories in the Django root directory, we would issue a command like this:

[django/trunk]|3> mglob dir:*
              <3> SList (.p, .n, .l, .s, .grep(), .fields() available).
              Value:
              0: examples
              1: tests
              2: extras
              3: build
              4: django
              5: docs
              6: scripts
              </3>

The mglob command returns a Python list-like object, so anything we can do in Python, we can do to this list of returned files or folders.

This was just a taste of how the sh behaves. There are some sh profile features and feature options that we didn’t cover.

If an object you are dealing with has too much of a representation to fit on one screen, you may want to try the magic page function. You can use page to pretty print your object and run it through a pager. The default pager on many systems is less, but yours might use something different. Standard usage is as follows:

In [1]: p = !ps aux
 ==
['USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND',
 'root         1  0.0  0.1   5116  1964 ?        Ss   Mar07   0:00 /sbin/init',
< ... trimmed result ... >
In [2]: page p
['USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND',
 'root         1  0.0  0.1   5116  1964 ?        Ss   Mar07   0:00 /sbin/init',
< ... trimmed result ... >

Here, we stored the result of the system shell command ps aux in the variable p. Then, we called page and passed the process result object to it. The page function then opened less.

There is one option for page: -r. This option tells page not to pretty print the object, but to run its string representation (result of str()) through the pager instead. For our process output object, that would look like this:

In [3]: page -r p
ilus-cd-burner/mapping-d', 'jmjones   5568  0.0  1.0 232004 10608 ?        S    
Mar07   0:00 /usr/lib/gnome-applets/trashapplet --', 'jmjones   5593  0.0  0.9 
188996 10076 ?        S    Mar07   0:00 /usr/lib/gnome-applets/battstat-apple', 
'jmjones   5595  0.0  2.8 402148 29412 ?        S    Mar07   0:01 p
< ... trimmed result ... >

This non-pretty-printed result is not pretty, indeed. We recommend starting out with the pretty printer and then working from there.

The magic pdef function prints the definition headers or the function signature of any callable object. In this example, we create our own function with a docstring and return statement:

In [1]: def myfunc(a, b, c, d):
   ...:     '''return something by using a, b, c, d to do something'''
   ...:     return a, b, c, d
   ...: 

In [2]: pdef myfunc
 myfunc(a, b, c, d)

The pdef function ignored our docstring and return statement, but printed out the function signature portion of the function. You can use this on any callable function. This function even works if the source code is not available as long as it has access to either the .pyc file or the egg.

The pdoc function prints the docstring of the function you pass to it. Here, we run the same myfunc() function that we used in the pdef example through pdoc:

In [3]: pdoc myfunc
Class Docstring:
    return something by using a, b, c, d to do something
Calling Docstring:
    x.__call__(...) <==> x(...)

This one is pretty self-explanatory.

The pfile function will run the file that contains an object through the pager if it can find the containing file:

In [1]: import os

In [2]: pfile os

r"""OS routines for Mac, NT, or Posix depending on what system we're on.

This exports:
  - all functions from posix, nt, os2, mac, or ce, e.g. unlink, stat, etc.

< ... trimmed result ... >

This opened the os module and ran it through less. This can definitely be handy if you are trying to understand the reason a piece of code is behaving in a particular way. It will not work if the only access to the file is an egg or a .pyc file.

The pinfo function and related utilities have become such a convenience for us that it’s hard to imagine not having them. The pinfo function provides information such as type, base class, namespace, and docstring. If we have a module that contains:

#!/usr/bin/env python

class Foo:
    """my Foo class"""
    def __init__(self):
        pass

class Bar:
    """my Bar class"""
    def __init__(self):
        pass

class Bam:
    """my Bam class"""
    def __init__(self):
        pass

then we can request information from the module itself:

In [1]: import some_module

In [2]: pinfo some_module
Type:           module
Base Class:     <type 'module'>
String Form:    <module 'some_module' from 'some_module.py'>
Namespace:      Interactive
File:           /home/jmjones/code/some_module.py
Docstring:
    <no docstring>

We can request information from a class in the module:

In [3]: pinfo some_module.Foo
Type:           classobj
String Form:    some_module.Foo
Namespace:      Interactive
File:           /home/jmjones/code/some_module.py
Docstring:
    my Foo class

Constructor information:
Definition:     some_module.Foo(self)

We can request information from an instance of one of those classes:

In [4]: f = some_module.Foo()

In [5]: pinfo f
Type:           instance
Base Class:     some_module.Foo
String Form:    <some_module.Foo instance at 0x86e9e0>
Namespace:      Interactive
Docstring:
    my Foo class

A question mark (?) preceeding or following an object name provides the same functionality as pinfo:

In [6]: ? f
Type:           instance
Base Class:     some_module.Foo
String Form:    <some_module.Foo instance at 0x86e9e0>
Namespace:      Interactive
Docstring:
    my Foo class


In [7]: f ?
Type:           instance
Base Class:     some_module.Foo
String Form:    <some_module.Foo instance at 0x86e9e0>
Namespace:      Interactive
Docstring:
    my Foo class

But two question marks (??) preceeding or following an object name provides us with even more information:

In [8]: some_module.Foo ??
Type:           classobj
String Form:    some_module.Foo
Namespace:      Interactive
File:           /home/jmjones/code/some_module.py
Source:
class Foo:
    """my Foo class"""
    def __init__(self):
        pass
Constructor information:
Definition:     some_module.Foo(self)

The ?? notation provides us with all the information that pinfo provided us plus the source code for the requested object. Because we only asked for the class, ?? gave us the source code for this class rather than for the whole file. This is one of the features of IPython that we find ourselves using more than nearly any other.

The psource function shows the source code for the element you define, whether that’s a module or something in a module, like a class or function. It runs the source code through a pager in order to display it. Here is an example of psource for a module:

In [1]: import some_other_module

In [2]: psource some_other_module
#!/usr/bin/env python

class Foo:
    """my Foo class"""
    def __init__(self):
        pass

class Bar:
    """my Bar class"""
    def __init__(self):
        pass

class Bam:
    """my Bam class"""
    def __init__(self):
        pass

def baz():
    """my baz function"""
    return None

Here is an example of psource for a class in a module:

In [3]: psource some_other_module.Foo
class Foo:
    """my Foo class"""
    def __init__(self):
        pass

and here is an example of psource for a function in a module:

In [4]: psource some_other_module.baz
def baz():
    """my baz function"""
    return None

The psearch magic function will look for Python objects by name, with the aid of wildcards. We’ll just briefly describe the psearch function here and if you want to know more, you can find documentation on the magic functions by typing magic at an IPython prompt, and then searching within the alphabetical list for psearch.

Let’s start by declaring the following objects:

In [1]: a = 1

In [2]: aa = "one"

In [3]: b = 2

In [4]: bb = "two"

In [5]: c = 3

In [6]: cc = "three"

We can search for all of the objects starting with a, b, or c as follows:

In [7]: psearch a*
a
aa
abs
all
any
apply

In [8]: psearch b*
b
basestring
bb
bool
buffer

In [9]: psearch c*
c
callable
cc
chr
classmethod
cmp
coerce
compile
complex
copyright
credits

Notice all the objects that were found in addition to a, aa, b, bb, c, cc; those are built-ins.

There is a quick and dirty alternative to using psearch: the ? operator. Here’s an example:

In [2]: import os

In [3]: psearch os.li*
os.linesep
os.link
os.listdir

In [4]: os.li*?
os.linesep
os.link
os.listdir

Instead of psearch, we were able to use *?.

There is an option to search -s or exclude searching -e a given namespace built-in to psearch. Namespaces include builtin, user, user_global, internal, and alias. By default, psearch searches builtin and user. To explicitly search user only, we would pass a -e builtin psearch option to exclude searching the builtin namespace. This is a little counterintuitive, but it makes an odd sort of sense. The default search path for psearch is builtin and user, so if we specify a -s user, searching builtin and user would still be what we’re asking it to do. In this example, the search is run again; notice that these results exclude the built-ins:

In [10]: psearch -e builtin a*
a
aa

In [11]: psearch -e builtin b*
b
bb

In [12]: psearch -e builtin c*
c
cc

The psearch function also allows searching for specific types of objects. Here, we search the user namespace for integers:

In [13]: psearch -e builtin * int
a
b
c

and here we search for strings:

In [14]: psearch -e builtin * string
__
___
__name__
aa
bb
cc

The __ and ___ objects that were found are IPython shorthand for previous return results. The __name__ object is a special variable that denotes the name of the module. If __name__ is '__main__', it means that the module is being run from the interpreter rather than being imported from another module.

IPython provides a number of facilities for listing all interactive objects. The first of these is the who function. Here is the previous example, including the a, aa, b, bb, c, cc variables, with the addition of the magic who function:

In [15]: who
a       aa      b       bb      c       cc

That’s pretty straightforward; it returns a simple listing of all interactively defined objects. You can also use who to filter on types. For example:

In [16]: who int
a       b       c

In [17]: who str
aa      bb      cc

Except that it returns a list rather than printing the names of the matching variables, who_ls is similar to who. Here is an example of the who_ls function with no arguments:

In [18]: who_ls

Out[18]: ['a', 'aa', 'b', 'bb', 'c', 'cc']

and here is an example of filtering based on the types of objects:

In [19]: who_ls int

Out[19]: ['a', 'b', 'c']

In [20]: who_ls str

Out[20]: ['aa', 'bb', 'cc']

Since who_ls returns a list of the names, you can access the list of names using the _ variable, which just means “the last output.” Here is the way to iterate the last returned list of matching variable names:

In [21]: for n in _:
   ....:     print n
   ....:     
   ....:     
aa
bb
cc

The whos function is similar to the who function except that whos prints out information that who doesn’t. Here is an example of the whos function used with no command-line arguments:

In [22]: whos
Variable   Type    Data/Info
----------------------------
a          int     1
aa         str     one
b          int     2
bb         str     two
c          int     3
cc         str     three
n          str     cc

And as we can with who, we can filter on type:

In [23]: whos int
Variable   Type    Data/Info
----------------------------
a          int     1
b          int     2
c          int     3

In [24]: whos str
Variable   Type    Data/Info
----------------------------
aa         str     one
bb         str     two
cc         str     three
n          str     cc

There are two ways to gain access to your history of typed-in commands in IPython. The first is readline-based; the second is the hist magic function.

In [1]: foo = 1

In [2]: bar = 2

In [3]: bam = 3

In [4]: d = dict(foo=foo, bar=bar, bam=bam)

In [5]: dict2 = dict(d=d, foo=foo)

In [6]: <CTRL-s>

(reverse-i-search)`fo': dict2 = dict(d=d, foo=foo)

<CTRL-r>

(reverse-i-search)`fo': d = dict(foo=foo, bar=bar, bam=bam)

In [1]: foo = 1

In [2]: bar = 2

In [3]: bam = 3

In [4]: cd /tmp
/tmp

In [5]: hist
1: foo = 1
2: bar = 2
3: bam = 3
4: _ip.magic("cd /tmp")
5: _ip.magic("hist ")

kIn [6]: hist -n
foo = 1
bar = 2
bam = 3
_ip.magic("cd /tmp")
_ip.magic("hist ")
_ip.magic("hist -n")

In [7]: hist -t
1: foo = 1
2: bar = 2
3: bam = 3
4: _ip.magic("cd /tmp")
5: _ip.magic("hist ")
6: _ip.magic("hist -n")
7: _ip.magic("hist -t")

In [8]: hist -r
1: foo = 1
2: bar = 2
3: bam = 3
4: cd /tmp
5: hist
6: hist -n
7: hist -t
8: hist -r

In [9]: hist -g hist
0187: hist
0188: hist -n
0189: hist -g import
0190: hist -h
0191: hist -t
0192: hist -r
0193: hist -d
0213: hist -g foo
0219: hist -g hist
===
^shadow history ends, fetch by %rep <number> (must start with 0)
=== start of normal history ===
5 : _ip.magic("hist ")
6 : _ip.magic("hist -n")
7 : _ip.magic("hist -t")
8 : _ip.magic("hist -r")
9 : _ip.magic("hist -g hist")

In [1]: foo = "foo_string"

In [2]: _

Out[2]: ''

In [3]: foo

Out[3]: 'foo_string'

In [4]: _

Out[4]: 'foo_string'

In [5]: a = _

In [6]: a

Out[6]: 'foo_string'

>>> foo = "foo_string"
>>> _
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name '_' is not defined
>>> foo
'foo_string'
>>> _
'foo_string'
>>> a = _
>>> a
'foo_string'

In [1]: !!ls apa*py

Out[1]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: apache_conf_docroot_replace.py
1: apache_log_parser_regex.py
2: apache_log_parser_split.py

In [2]: !!ls e*py

Out[2]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: elementtree_system_profile.py
1: elementtree_tomcat_users.py

In [3]: !!ls t*py

Out[3]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: test_apache_log_parser_regex.py
1: test_apache_log_parser_split.py

In [4]: apache_list = _1

In [5]: element_tree_list = _2

In [6]: tests = _3

In [7]: apache_list

Out[7]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: apache_conf_docroot_replace.py
1: apache_log_parser_regex.py
2: apache_log_parser_split.py

In [8]: element_tree_list

Out[8]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: elementtree_system_profile.py
1: elementtree_tomcat_users.py

In [9]: tests

Out[9]: SList (.p, .n, .l, .s, .grep(), .fields() available). Value:
0: test_apache_log_parser_regex.py
1: test_apache_log_parser_split.py

We’ll first mention the alias “magic” command. We already covered this earlier in this chapter, so we won’t rehash usage of it again. But we wanted to just point out here that alias cannot only help you use *nix shell commands directly from within IPython, it can help you automate tasks as well.

The macro function lets you define a block of code that can be executed later inline with whatever code you are working on. This is different from creating functions or methods. The macro, in a sense, becomes aware of the current context of your code. If you have a common set of processing steps you frequently execute on all your files, you can create a macro to work on the files. To get a feel for the way a macro will work on a list of files, look at the following example:

In [1]: dirlist = []

In [2]: for f in dirlist:
   ...:     print "working on", f
   ...:     print "done with", f
   ...:     print "moving %s to %s.done" % (f, f)
   ...:     print "*" * 40
   ...:     
   ...:     

In [3]: macro procdir 2
Macro `procdir` created. To execute, type its name (without quotes).
Macro contents:
for f in dirlist:
    print "working on", f
    print "done with", f
    print "moving %s to %s.done" % (f, f)
    print "*" * 40

At the time that we created the loop in In [2], there were no items in dirlist for the loop to walk over, but because we anticipated that future iterations would include items in dirlist, we created a macro named procdir to walk over the list. The syntax for creating a macro is macro macro_name range_of_lines, where the range of lines is a list of the lines from your history that you want incorporated into the macro. The lines for your macro list should be designated by a space-separated list of either numbers or ranges of numbers (such as 1-4).

In this example, we create a list of filenames and store them in dirlist, then execute the macro procdir. The macro will walk over the list of files in dirlist:

In [4]: dirlist = ['a.txt', 'b.txt', 'c.txt']

In [5]: procdir
------> procdir()
working on a.txt
done with a.txt
moving a.txt to a.txt.done
****************************************
working on b.txt
done with b.txt
moving b.txt to b.txt.done
****************************************
working on c.txt
done with c.txt
moving c.txt to c.txt.done
****************************************

Once you have a macro defined, you can edit it. This will open in your defined text editor. This can be very helpful when you are tweaking a macro to make sure it is right before you persist it.

You can persist your macros and plain Python variables with the store magic function. The simple standard use of store is store variable. However, store also takes a number of parameters that you may find useful: the -d variable function deletes the specified variable from the persistence store; -z function deletes all stored variables; and the -r function reloads all variables from the persistence store.

The reset function deletes all variables from the interactive namespace. In the following example, we define three variables, use whos to verify they are set, reset the namespace, and use whos again to verify that they are gone:

In [1]: a = 1

In [2]: b = 2

In [3]: c = 3

In [4]: whos
Variable   Type    Data/Info
----------------------------
a          int     1
b          int     2
c          int     3

In [5]: reset
Once deleted, variables cannot be recovered. Proceed (y/[n])?  y

In [6]: whos
Interactive namespace is empty.

The run function executes the specified file in IPython. Among other things, this allows you to work on a Python module in an external text editor and interactively test changes you are making in it from within IPython. After executing the specified program, you are returned back to the IPython shell. The syntax for using run is run options specified_file args.

The -n option causes the module’s __name__ variable to be set not to '__main__', but to its own name. This causes the module to be run much as it would be run if it were simply imported.

The -i option runs the module in IPython’s current namespace and, thereby, gives the running module access to all defined variables.

The -e option causes IPython to ignore calls to sys.exit() and SystemExit exceptions. If either of these occur, IPython will just continue.

The -t option causes IPython to print out information about the length of time it took the module to run.

The -d option causes the specified module to be run under the Python debugger (pdb).

The -p option runs the specified module under the Python profiler.

The save function will save the specified input lines to the specified output file. Syntax for using save is save options filename lines. The lines may be specified in the same range format as is used for macro. The only save option is -r, which designates that raw input rather than translated should be saved. Translated input, which is standard Python, is the default.

The final automation-enabling function is rep. The rep function takes a number of parameters that you might find useful. Using rep without parameters takes the last result that was processed and places a string representation of it on the next input line. For example:

In [1]: def format_str(s):
   ...:     return "str(%s)" % s
   ...: 

In [2]: format_str(1)

Out[2]: 'str(1)'

In [3]: rep

In [4]: str(1)

The rep call at In [3] causes the text you see to be placed on In [4]. This allows you to programatically generate input for IPython to process. This comes in handy, particularly when you are using a combination of generators and macros.

A fairly common use case for rep without arguments is lazy, mouseless editing. If you have a variable containing some value, you can edit that value directly. As an example, assume that we are using a function that returns to the bin directory for specific installed packages. We’ll store the bin directory in a variable called a:

In [2]: a = some_blackbox_function('squiggly')

In [3]: a

Out[3]: '/opt/local/squiggly/bin'

If we type rep right here, we’ll see /opt/local/squiggly/bin on a new input line with a blinking cursor expecting us to edit it:

In [4]: rep

In [5]: /opt/local/squiggly/bin<blinking cursor>

If we wanted to store the base directory of the package rather than the bin directory, we can just delete the bin from the end of the path, prefix the path with a new variable name, follow that with an equal sign and quotation marks, and suffix it with just a quotation mark:

In [5]: new_a = '/opt/local/squiggly'

Now we have a new variable containing a string that is the base directory for this package.

Sure, we could have just copied and pasted, but that would have been more work. Why should you leave the comfort of your cozy keyboard to reach for the mouse? You can now use new_a as a base directory for anything that you need to do regarding the squiggly package.

When one number is given as an argument to rep, IPython brings up the input from that particular line of history and places it on the next line, and then places the cursor at the end of that line. This is helpful for executing, editing, and re-executing single lines or even small blocks of code. For example:

In [1]: map = (('a', '1'), ('b', '2'), ('c', '3'))

In [2]: for alph, num in map:
   ...:     print alph, num
   ...:     
   ...:     
a 1
b 2
c 3

Here, we edit In [2] and print the number value times 2 rather than a noncomputed value. We could either type the for loop in again, or we can use rep:

In [3]: rep 2

In [4]: for alph, num in map:
    print alph, int(num) * 2
   ...:     
   ...:     
a 2
b 4
c 6

The rep function also takes ranges of numbers for arguments. The numeric range syntax is identical to the macro numeric range syntax that we discussed elsewhere in this chapter. When you specify a range for rep, the lines are executed immediately. Here is an example of rep:

In [1]: i = 1

In [2]: i += 1

In [3]: print i
2

In [4]: rep 2-3
lines [u'i += 1
print i
']
3

In [7]: rep 2-3
lines [u'i += 1
print i
']
4

We defined a counter incrementer and code that prints out the current count in In [1] through In [3]. In In [4] and In [7], we told rep to repeat lines 2 and 3. Notice that 2 lines (5 and 6) are missing since they were executed after In [4].

The last option for rep that we’ll go over is passing in a string. This is more like “passing in a word to rep” or even “passing in a nonquoted search string to rep.” Here is an example:

In [1]: a = 1

In [2]: b = 2

In [3]: c = 3

In [4]: rep a

In [5]: a = 1

We defined a few variables and told rep to repeat the last line that has an “a” in it. It brought In [1] back to us to edit and re-execute.