6

Files and the Internet

Python makes it easy for your programs to use files and connect to the Internet. You can read data from files, write data to files, and fetch content from the Internet. You can even check for new mail and tweet—all from your program.

Files

When you run a Python program, any values you have in variables will be lost. Files provide a means of making data more permanent.

Reading Files

Python makes reading the contents of a file extremely easy. As an example, we can convert the Hangman program from Chapter 4 to read the list of words from a file rather than have them fixed in the program.

First of all, start a new file in Mu and put some words in it, one per line. Then save the file with the name hangman_words.txt in the directory /home/pi/mu_code/prog_pi_ed3. Note that in the Save dialog you will have to change the file type to .txt (see Figure 6-1).

Figure 6-1 Creating a text file in Mu.

Before we modify the Hangman program itself, we can just experiment with reading the file in the Python console. Enter the following into the REPL:

>>> f = open('prog_pi_ed3/hangman_words.txt')

Note that REPL in Mu has a current directory of /home/mu/mu_code, so the directory of wherever you saved the file must be included in the 'open' command.

Next enter the following into the Python console:

I told you it was easy! All we need to do to add this file to the Hangman program is replace the line

with the following lines:

The line f.close() has been added. You should always call the close command when you are done with a file to free up operating system resources. Leaving a file open can lead to problems.

The full program is contained in the file 06_01_hangman_file.py, and a suitable list of animal names can be found in the file hangman_words.txt. This program does nothing to check that the file exists before trying to read it. So, if the file isn’t there, we get an error that looks something like this:

To make this a bit more user friendly, the file-reading code needs to be inside a try command, like this:

Python will try to open the file, but because the file is missing it will not be able to. Therefore, the except part of the program will apply, and the more friendly message will be displayed. Because we cannot do anything without a list of words to guess, there is no point in continuing, so the exit command is used to quit.

In writing the error message, we have repeated the name of the file. Sticking strictly to the Don’t Repeat Yourself (DRY) principle, the filename should be put in a variable, as shown next. That way, if we decide to use a different file, we only have to change the code in one place.

A modified version of Hangman with this code in it can be found in the file 06_02_hangman_file_try.py.

Reading Big Files

The way we did things in the previous section is fine for a small file containing some words. However, if we were reading a really huge file (say, several megabytes), then two things would happen. First, it would take a significant amount of time for Python to read all the data. Second, because all the data is read at once, at least as much memory as the file size would be used, and for truly enormous files, that might result in Python running out of memory.

If you find yourself in the situation where you are reading a big file, you need to think about how you are going to handle it. For example, if you were searching a file for a particular string, you could just read one line of the file at a time, like this:

When the function readline gets to the last line of the file, it returns an empty string (''). Otherwise, it returns the contents of the line, including the end-of-line character ( ). If it reads a blank line that is actually just a gap between lines and not the end of the file, it will return just the end-of-line character ( ). By the program only reading one line at a time, the memory being used is only ever equivalent to one full line.

If the file is not broken into convenient lines, you can specify an argument in read that limits the number of characters read. For example, the following will just read the first 20 characters of a file:

Writing Files

Writing files is almost as simple. When a file is opened, as well as specifying the name of the file to open, you can also specify the mode in which to open the file. The mode is represented by a character, and if no mode is specified it is assumed to be r for read. The modes are as follows:

   r (read).

   w (write)  Replaces the contents of any existing file with that name.

   a (append)  Appends anything to be written onto the end of an existing file.

   r+  Opens the file for both reading and writing (not often used).

You can also add 'b' after 'r', 'w', or 'a' to indicate that the file contains binary data rather than readable text.

To write a file, you open it with a second parameter of 'w', 'a', or 'r+'. Here’s an example:

Try finding the file using the file manager just to check it's there. You will find it in /home/pi/mu_code.

The File System

Occasionally, you will need to do some file-system-type operations on files (moving them, copying them, and so on). Python uses Linux to perform these actions, but provides a nice Python-style way of doing them. Many of these functions are in the shutil (shell utility) package. There’s a number of subtle variations on the basic copy and move features that deal with file permissions and metadata. In this section, we just deal with the basic operations. You can refer to the official Python documentation for any other functions (http://docs.python.org/release/3.1.5/library).

Here’s how to copy a file:

To move a file, either to change its name or move it to a different directory:

shutil.move('test_copy.txt', 'test_dup.txt')

This works on directories as well as files. If you want to copy an entire folder—including all its contents and its content’s contents—you can use the function copytree. The rather dangerous function rmtree, on the other hand, will recursively remove a directory and all its contents—exercise extreme caution with this one!

The nicest way of finding out what is in a directory is via globbing. The package glob allows you to create a list of files in a directory by specifying a wildcard (*). Here’s an example:

If you just want all the files in the folder, you could use this:

glob.glob('*')

Pickling

Pickling involves saving the contents of a variable to a file in such a way that the file can be later loaded to get the original value back. The most common reason for wanting to do this is to save data between runs of a program. As an example, we can create a complex list containing another list and various other data objects and then pickle it into a file called mylist.pickle, like so:

The pickle file is binary (that’s why you open the file in 'wb' mode) and unfortunately you can’t view it in a text editor. To reconstruct a pickle file into an object, here is what you do:

JSON

JSON is a text format for representing data. It is used both in files and in web services using JSON as a common interchange format for data. Python makes it really easy to use JSON by creating lists and dictionaries from JSON text retrieved from a file or from the Internet.

As an example, use Mu to create a file called “books.json” in /home/pi/mu_files and put the following JSON text into it.

When it comes to JSON, think of a { and } as enclosing a Python dictionary. In this case the dictionary has a key of the string “books” and a value that is a list contained between [ and ]. There are two values in that list, both of which are themselves dictionaries containing information about books.

Each book dictionary has keys of “title” and “price.” The title is a string enclosed within quotes and the price is a number (and so does not need quotes around it).

The program file 06_04_json_file.py reads the file books.json and converts it into Python dictionaries and lists for us. Load 06_04_json_file.py into Mu and run it. You should see something like Figure 6-2.

Figure 6-2 Reading the contents of a JSON file.

Let’s have a look at the code for this.

The first thing we do is import the json library. We then open the file books.json, and because json is in a text format, unlike a pickle file, we do not have to open it as a binary file.

The json method 'load' reads the contents of the files and converts it into a dictionary. This is then assigned to the variable 'j'. The files are then closed, as we have read what we need from it.

The final line of the program shows how you can use the [] notation to navigate into the structure of dictionaries and lists that have been created. This first uses ['books'] to retrieve the list of books and then uses [1] to select item 1 in the list (the second item) and then ['title'] selects the title of the second book.

In the next section you will learn how to use a web service to retrieve information about the weather from the Internet into your Python program.

Internet

As well as being full of web pages, the Internet is also a source of information for your programs rather than for viewing in a browser. Such web services are available for pretty much everything from currency rates and stock prices to weather forecasts. The software on the server for such web service is called an API (Application Programming Interface) and APIs generally use JSON as a way of communicating with your programs.

As an example of this, we are going to use a weather web service called weatherstack.com. In line with many such web services, you can use the web service for free as long as you don’t make too many demands on it. However, you do normally have to register and make an account. weatherstack is no exception, so, to try out this example, you will need to register for an account at https://weatherstack.com/.

Not unreasonably, weatherstack likes to know who’s using its API and so to access it from your programs you need to include your own personal key with each web request. Once you have registered, you can find your weatherstack key by clicking on the Dashboard button (Figure 6-3).

Figure 6-3 Finding your API key in weatherstack.

You will need to copy this key and paste it into the 'key = ' line of program 06_05_weather.py listed below and run the program.

You should see output like that shown in Figure 6-4. You can change the city to your city.

Figure 6-4 Calling the weatherstack API.

As you can see we have received a whole load of information back from the API that we can make use of in our programs.

The program uses the urllib library to perform the web request, contacting weatherstack’s API, so this library has to be imported. Three variables are used, which will be assembled into the full URL to be sent to the API. They are the base URL, the API key (paste yours here), and the city. URLs cannot have spaces in them, but some city names (such as San Francisco) do. So, the method urllib.parse.quote is used to convert spaces into the %20 escape characters expected in a URL.

The method urllib.request.urlopen opens a connection to the API as if you were opening a file on your machine, and then json.load converts this to the JSON structure shown in Figure 6-4.

We probably don’t want to see all this information, so we can navigate into the JSON and just pull out a summary by replacing the last line of the program with:

print(j['current']['weather_descriptions'][0])

Now, when you run the program, it will just display a message something like:

When you find a web service like this, that you want to use, find the documentation for the API so that you can see what kind of JSON you are going to need to deal with.

Summary

This chapter has given you the basics of how to use files and access web pages from Python. There is actually a lot more to Python and the Internet, including accessing e-mail and other Internet protocols. For more information on this, have a look at the Python documentation at http://docs.python.org/release/3.1.5/library/internet.html.