3.1 Native Console

Launching Python without any arguments opens the interactive console. It is a good idea to use pyenv or a virtual environment to make sure that the correct Python version is up to date. Some operating systems have a default version of Python running a legacy version of Python.

The availability of an interactive console immediately, without installing anything else, is one reason why Python is suited for exploratory programming. The interpreter can immediately answer questions.

This checks whether the user has .bashrc in their home directory; it is true for some users on some systems and false for others.

Pythons native console using the GNU readline library to afford editing. If python -c 'import readline' fails with an ImportError, Python was not built with readline support, leading to a degraded experience at the console. If rebuilding Python with readline support is not an option (for example, using a locally built Python by a different team), it is highly recommended to use one of the alternative consoles described in this chapter.

This can be useful after having used the console for a while. The history file can serve as a reference for what has been done or copy whatever ideas worked into a more permanent form.

After using the .get function, you realize that what you wanted was the text. Luckily, the Response object is saved in the _ variable. The value of the variable is put in a immediately because: _ is replaced quickly. As soon as you evaluate a.text[:50], _ is a 50-character string. If you had not saved _ in a variable, all but the first 50 characters would have been lost.

Notice that every good Python REPL keeps this _ convention, and so the trick of keeping returned values in one-letter variables is often useful when doing explorations.

3.2 The Code Module

The code module allows you to run your own interactive loop. An example of when this can be useful is when running commands with a special flag. You can drop into a prompt at a specific point, allowing you to have a REPL environment after setting things up in a certain way. This holds for both inside the interpreter, setting up the namespace with useful things, and in the external environment, perhaps initializing files or setting up external services.

This example is running a REPL loop with a special variable, which is set to a short list. To get out of the inner-interpreter, press Control+D (indicated as ^D).

For the lowest-level use of code, allowing complete ownership of the UI, code.compile_command(source, [filename="<input>"], symbol="single") returns a code object (that can be passed to exec), None if the command is incomplete, or raises SyntaxError, OverflowError, or ValueError if there is a problem with the command.

The symbol argument should almost always be "single". The exception is if the user is prompted to enter code that evaluates an expression (for example, if the value is to be used by the underlying system). In that case, symbol should be set to "eval".

This allows you to manage interaction with users. It can be integrated with a UI, or a remote network interface, to allow interactivity in any environment.

3.3 ptpython

The ptpython tool, short for prompt toolkit Python, is an alternative to the built-in REPL. It uses the prompt toolkit for console interaction, instead of readline.

Its main advantage is the simplicity of installation. A simple pip install ptpython in a virtual environment, and regardless of readline build problems, a high-quality Python REPL appears.

ptpython supports completion suggestions, multiline editing, and syntax highlighting.

On start-up, it reads ~/.ptpython/config.py, which means it is possible to locally customize ptpython in arbitrary ways. To configure, implement a configure function, which accepts an object (of type PythonRepl) and mutates it.

There are a lot of possibilities, and sadly the only real documentation is the source code. The relevant reference __init__ is ptpython.python_input.PythonInput. Note that config.py is an arbitrary Python file. Therefore, if you want to distribute modifications internally, it is possible to distribute a local PyPI package and have people import a configure function from it.

3.4 IPython

IPython, the foundation of Jupyter, is an interactive environment whose roots are in the scientific computing community. IPython is an interactive command prompt, similar to the ptpython utility or Python’s native REPL.

It also supports tab completion out of the box. IPython uses both its own completion and the jedi library for static completion.

IPython also supports Magics commands, where prefixing a line with % executes a magic function; for example, %run runs a Python script inside the current namespace. As another example, %edit launches an editor. This is useful if a statement needs more sophisticated editing.

In addition, prefixing a line with ! runs a system command. One useful way to take advantage of this is !pip install something. This is why installing IPython inside virtual environments used for interactive development is useful.

IPython can be customized in a number of ways. While in an interactive session, the %config magic command can be used to change any option. For example, %config InteractiveShell.autocall = True sets the autocall option, which means callable expressions are called, even without parentheses. This is moot for any options that only affect startup. You can change these options, as well as any others, using the command line. For example, ipython --InteractiveShell.autocall=True, launches into an autocalling interpreter.

If this is in a dedicated Python package, you can distribute it to a team using PyPI or a private package repository. This allows a homogenous custom IPython configuration for a development team.

Finally, a configuration can also be encoded in profiles, which are Python snippets located under ~/.ipython by default. The profile directory can be modified by an explicit --ipython-dir command-line parameter or an IPYTHONDIR environment variable.

3.5 JupyterLab

Jupyter is a project that uses web-based interaction to allow for sophisticated exploratory programming. It is not limited to Python, though it does originate in Python. The name is short for Julia/Python/R, the three languages most popular for exploratory programming, especially data science.

JupyterLab, the latest evolution of Jupyter, was originally based on IPython. It now sports a full-featured web interface and a way to edit files remotely. The main users of Jupyter tend to be scientists. They take advantage of seeing how results were derived to add reproducibility and peer review.

Reproducibility and peer-review are also important for DevOps work. The ability to show the steps that led to deciding which list of hosts to restart so that it can be regenerated if circumstances change, for example, is highly useful. The ability to attach a notebook detailing the steps taken during an outage, together with the output from the steps, to a post-mortem analysis can aid in understanding what happened and how to avoid a problem in the future or recover from it more effectively.

It is important to note here that notebooks are not an auditability tool. They can be executed out of order and have blocks modified and re-executed. However, properly used, they allow you to record what has been done.

Jupyter allows true exploratory programming, which is useful for scientists who might not understand the true scope of a problem beforehand.

This is also useful for systems integrators faced with complex systems where it is hard to predict where the problem lies before exploration.

Installing JupyterLab in a virtual environment is a simple matter of typing pip install jupyterlab. By default, it starts a web server on an open port starting at 8888 and attempts to launch a web browser to watch it. If working on an environment that is too interesting (for example, the default web browser is not configured properly), the standard output contains a preauthorized URL to access the server. If all else fails, it is possible to copy-paste the token printed to standard output into the browser after manually entering the URL in a web browser. It is also possible to access the token with jupyter notebook list, which lists all currently running servers.

The console is a web-based interface to IPython. All that was said about IPython previously (for example, the In and Out arrays). The terminal is a full-fledged terminal emulator in the browser. This is useful for a remote terminal inside a VPN. All it needs as far as connectivity needs is an open web port. It can also be protected in the regular ways web ports are protected: TLS, client-side certificates, and more. The text editor is useful for similar reasons; this avoids the need to run vi through a shell to a remote location, resulting in lag, while full file editing capabilities still exist.

The most interesting thing to launch is a notebook; indeed, many sessions use nothing but notebooks. A notebook is a JSON file that records a session. As the session unfolds, Jupyter saves snapshots of the notebook and the latest version. A notebook is made of a sequence of cells. The two most popular cell types are Code and Markdown. A code cell type contains a Python code snippet. It executes it in the context of the session’s namespace. The namespace is persistent from one cell execution to the other, corresponding to a kernel running. The kernel accepts cell content using a custom protocol, interprets them as Python, executes them, and returns whatever was returned by the snippet and the output.

This causes the Jupyter server in this environment to support the kernel from the special environment. The --sys-prefix installs it inside the virtual environment where Jupyter is installed.

This allows running one semi-permanent Jupyter server and connecting kernels from any interesting environment. Environments can be interesting because they have specific modules installed or run a specific Python version.

Alternative languages are another usage of alternative kernels. Julia and R kernels are supported upstream, but third-party kernels exist for many languages—even bash!

Jupyter supports all magic commands from IPython. Especially useful, again, is the !pip install ... command to install new packages in the virtual environment. Especially if being careful and installing precise dependencies, this makes a notebook be high-quality documentation of how to achieve a result in a replayable way.

Since Jupyter is one level of indirection away from the kernel, you can restart the kernel directly from Jupyter. This means the whole Python process gets restarted, and any in-memory results are gone. You can re-execute cells in any order, but there is a single-button way to execute all cells in order. Restarting the kernel, and executing all cells in order, is a nice way of testing a notebook for working conditions. Although, naturally, any effects on the external world are not reset.

Jupyter notebooks are useful as attachments to tickets and post-mortems, both as a way of documenting specific remediations and documenting the state of things by running query APIs and collecting the results in the notebook. Usually, when attaching a notebook in such a way, it is useful to export it to a more easily readable format, such as HTML or PDF, and attach that. However, more and more tools integrate direct notebook viewing, making this step redundant. For example, GitHub projects and Gists already render notebooks directly.

JupyterLab sports a rudimentary but functional, browser-based remote development environment. The first part is a remote file manager. Among other things, this allows uploading and downloading files. One use for it, among many, is the ability to upload notebooks from the local computer and download them back again. There are better ways to manage notebooks, but in a pinch, being able to retrieve a notebook is extremely useful. Similarly, any persistent outputs from Jupyter, such as processed data files, images, or charts, can also be downloaded.

Next, alongside the notebooks, is a remote IPython console. Though of limited use next to the notebook, there are still some cases where using the console is easier. A session that requires a lot of short commands can be more keyboard-centric by using the IPython console and thus more efficient.

There is also a file editor. Although it is a far cry from being a full-fledged developer editor, lacking thorough code understanding and completion, it is often useful in a pinch. It allows you to edit files on the remote Jupyter host directly. One use case is directly fixing the library code that the notebook uses and then restarting the kernel. While integrating it into a development flow takes some care, this is invaluable as an emergency measure to fix and continue.

Last, there is a remote browser-based terminal. Between the terminal, the file editor, and the file manager, a running Jupyter server allows complete browser-based remote access and management, even before thinking of the notebooks. This is important for security implications and a powerful tool whose various uses are explored later. For now, suffice it to say, the power that using a Jupyter notebook brings to remote system administration tasks is hard to overestimate.

3.6 Summary

The faster the feedback cycle, the faster you can deploy new, tested solutions. Using Python interactively provides immediate feedback. This is often useful to clarify a library’s documentation, a hypothesis about a running system, or your understanding of Python.

The interactive console is also a powerful control panel from which to launch computations when the result is not well understood, for example, when debugging the state of software systems.