Functions that take functions as arguments or return functions are called higher order functions. Functions that work with functions can be extremely useful. They allow you to separate out parts of a program more easily; for example, a higher order function might provide a traversal of a data structure, where the function you pass in decides what to do for each item.

This technique is highly reminiscent of the strategy pattern from Design Patterns: Elements of Reusable Object-Oriented Software:^[1]

You can implement this pattern very simply with higher order functions. Imagine you’re writing a banking application that handles many different transactions. You need to apply these transactions to accounts, rolling back the transactions in the event of an error (such as insufficient funds in one of the accounts). Transactions of many different types can be created, but the code that consumes the transactions should be able to apply them without knowing anything about how they’re implemented.

A function to create a transaction may look something like this:

def get_transfer(account1, account2, amount):
   def do_transfer():
      account1.withdraw(amount)
      account2.deposit(amount)

   return do_transfer

The do_transfer function is for transactions that transfer money from one account to another. do_transfer closes over the parameters to get_transfer and encapsulates them. The parameters will be used when do_transfer is applied.

class Callable(object):
   def __call__(self, x):
      print x

>>> instance = Callable()
>>> instance('hello')
hello

The transaction is applied by apply_transaction.

def apply_transaction(transaction, connection):
   connection.begin_transaction()
   try:
      transaction()
   except TransactionError:
      connection.abort_transaction()
      return False
   else:
      connection.commit_transaction()
      return True

apply_transaction can apply any transaction. It handles errors, aborting the transaction in case of error. It returns True or False depending on whether the transaction succeeds or not. The advantage of this kind of structure is that you can change how the transactions are applied in a single place, and individual transactions can be modified without affecting how they’re used.

Another place where higher order functions commonly turn up in Python is in the form of decorators.

It turns out that writing a function that takes a function, and then wraps it, is a common idiom in Python. Python has syntax to make this easier, syntactic sugar called decorators.

First, we show you an example that doesn’t use the decorator syntax. We wrap a function so that it prints whenever it is called. In practice, you might put logging or timing code inside the wrapper function.

The decorator syntax allows a nicer way of expressing the line wrapped = wrapper (somefunction). You’ll especially appreciate it if you want to decorate all the methods in a class. Decorators use the @ symbol, along with the name of the decorator function, above the function definition.

When you decorate a function, the function name is automatically rebound to the wrapped function rather than the original one. So is this actually useful? Let’s look at how you can use this to automate a repetitive task like checking method arguments for null values.

Even with static typing, method parameters can still be None (or null, using .NET speak) in .NET languages such as C#. If your code requires that the argument is valid and can’t be null, then it’s common to have code like the following:

void Present(Region present, Region selection)
{
  if (present == null)
  { throw new ArgumentNullException("present"); }
  if (selection == null)
  { throw new ArgumentNullException("selection"); }

  // Actual code that we care about
}

In Python, you can write a decorator that checks function arguments and raises a TypeError if any of them are None. Any methods that you want to check in this way can then be decorated.

To do this, you need a decorator that returns a wrapped function. The wrapper should check all the arguments before calling the function or method that’s wrapping. The wrapper function will need to call the wrappee with the same arguments it’s called with, and return the value that the wrapped function returns. This should do the trick:

def checkarguments(function):
   def decorated(*args):
      if None in args:
         raise TypeError("Invalid Argument")
      return function(*args)
   return decorated

Any arguments a wrapped function is called with are collected as a tuple (args). If any of these arguments are None (if None is in args), then a TypeError is raised. You use it like this:

class MyClass(object):

   @checkarguments
   def method(self, arg1, arg2):
      return arg1 + arg2
>>> instance = MyClass()
>>> instance.method(1, 2)
3
>>> instance.method(2, None)
Traceback (most recent call last):
TypeError: Invalid Argument

You should pass the parameter name when you raise the exception. You could do this by using introspection on the function object, but that’s another subject altogether.

OK, we’ve had some fun with functions in Python. Now it’s time to put what you’ve learned to practical use in working with XML.

The .NET XML support is enshrined in the System.Xml assembly. The System.Xml namespace has classes for reading and writing XML. There are further namespaces for working with XML schema, XSLT transformations, and XPath queries. Table 5.2 shows the major XML namespaces in .NET 2.0.

The basic classes for reading and writing XML documents are the XmlReader and XmlWriter. Because you’ll be modifying the SaveCommand, we start with the XmlWriter.

XmlWriter is designed for creating conformant documents. The documents it creates will be valid XML, capable of being read by any standards-based reader. Along with the XmlWriter class, you use XmlWriterSettings. This is a class used for configuring an XmlWriter instance; you set attributes on the XmlWriterSettings instance to configure how the XML is written out.

Table 5.3 shows the different settings (properties) on XmlWriterSettings. The defaults are largely sensible, but we do like to change a couple. We like the XML tags to be indented with each level of nesting. This gives you a visual indication of the structure of the document (and we all know that indentation to indicate structure is a brilliant idea). The following segment creates an XmlWriterSettings instance, and sets the two properties required for indentation with four spaces. Because you haven’t yet used the System.Xml assembly, you first need to add a reference to it.

>>> import clr
>>> clr.AddReference('System.Xml')
>>> from System.Xml import XmlWriter, XmlWriterSettings
>>> settings = XmlWriterSettings()
>>> settings.Indent = True
>>> settings.IndentChars = '  ' # four spaces

You don’t instantiate a new XmlWriter instance directly; instead, you call the static method Create, which returns a new instance of the correct type of writer for the settings passed in. There are various call signatures for Create; for example, you could pass in a filename and your settings object, and the writer would create the file for you. If you don’t want the writer to be responsible for creating the file, you can pass in an opened FileStream instead. In the segments that follow, you’ll pass in a StringBuilder instance. StringBuilder is in the System.Text namespace, and is a mutable string type—it allows strings to be built up incrementally.

An odd side effect of passing a StringBuilder to the XmlWriter is that it will refuse to write any encoding in the XML declaration other than UTF-16.^[2] Because you’re likely to be happy with the default encoding of UTF-8, you set OmitXmlDeclaration to True.

>>> settings.OmitXmlDeclaration = True
>>> from System.Text import StringBuilder
>>> document = StringBuilder()
>>> writer = XmlWriter.Create(document, settings)
>>> writer.WriteStartDocument()
>>> writer.WriteStartElement("document")
>>> writer.WriteStartElement("page")
>>> writer.WriteAttributeString("title", "A page title")
>>> writer.WriteString("This is a page contents")
>>> writer.WriteEndElement()
>>> writer.WriteEndElement()
>>> writer.WriteEndDocument()
>>> writer.Flush()
>>> writer.Close()
>>> print document.ToString()
   <document>
    <page title="A page title">This is a page contents</page>
   </document>

This is great because (by happy coincidence) it’s exactly the kind of structure that we want for our documents. Having to remember to close all the elements in the right order is a nuisance, though. This is what happens if you get it wrong:

>>> document = StringBuilder()
>>> writer = XmlWriter.Create(document, settings)
>>> writer.WriteEndElement()
Traceback (most recent call last):
 File System.Xml, line unknown, in WriteEndElement
 File System.Xml, line unknown, in AdvanceState
 File System.Xml, line unknown, in ThrowInvalidStateTransition
SystemError: Token EndElement in state Start would result in an invalid XML document.
Make sure that the ConformanceLevel setting is set to ConformanceLevel.Fragment or
ConformanceLevel.Auto if you want to write an XML fragment.

Oops. A sensible way to avoid this is to make sure that your XML document structure and your program structure are as similar as possible. Ideally, a top-level method should create (and close) the root node, calling down to submethods to write out its child elements. Each method should only be responsible for creating and closing a single element, again calling submethods for their child elements. This way you know that, as long as the code runs to completion, every element will be closed and the result will be valid XML. A nice side effect is that this is also a good way to write modular and readable code—which is important because you’re writing Python. In the next section, we apply this strategy to MultiDoc.

Writing out XML from a MultiDoc document is an important enough job that we can encapsulate it in its own class. This keeps the logic separate from the other machinations of the save commands, and easier to understand.

Listing 5.1 shows the DocumentWriter for MultiDoc; it uses XmlWriter and is instantiated with a filename. To write out the document, you must call write, passing in an instance of a MultiDoc document. The write method is responsible for creating the XmlWriter instance and opening and closing the root element of the XML. It calls down to writePage for each page in the document, creating the page element with title attribute.

Example 5.1. A DocumentWriter class that writes out MultiDoc documents as XML

This class needs to be saved as the file documentwriter.py. To plug this into MultiDoc, you need to modify the commands to use it. First, DocumentWriter needs to be imported inside the savecommands module.

from documentwriter import DocumentWriter

The SaveAsCommand inherits from SaveCommand, and the file writing is done in the saveFile method. You can get most of the way toward the changes you want by modifying saveFile as follows:

def saveFile(self, fileName, document):
      try:
         writer = DocumentWriter(fileName)
         writer.write(document)
         return True
      except IOError, e:
         ...

Previously the execute and promptAndSave methods of the commands only needed to pass some text to saveFile; now they need to pass the document instead. The document still needs to be updated before saving, so getText becomes getUpdatedDocument.

def getUpdatedDocument(self):
   self.tabController.updateDocument()
   return self.document

The execute method of both SaveCommand and SaveAsCommand must be modified to call getUpdatedDocument, and to pass the document through to saveFile and save-AndPrompt. These changes are simple; and, rather than using space here we leave you to figure them out. If you want to see the changes, they’re in the 5.3 folder of the sources that go with this book.

The DocumentWriter you’ve created follows the structure we suggested earlier, with a top-level method that writes out the top-level node (write). This calls down to writePage to write out the child page nodes. To do this, the XmlWriter has to be stored as state on the DocumentWriter, as the self.writer instance variable. Because you only have one root node, the structure is simple. You could avoid having to store state by refactoring to use an inner function.

The writePage method is simple. You can refactor this into an inner function that takes a page as its argument. It needs access to the writer, which can be a local variable in its enclosing scope, which is the body of the write method.

The refactored write method looks like listing 5.2.

def write(self, document):
   settings = XmlWriterSettings()
   settings.Indent = True
   settings.IndentChars = '   '
   settings.OmitXmlDeclaration = True
   writer = XmlWriter.Create(self.fileName, settings)

   writer.WriteStartDocument()
   writer.WriteStartElement("document")

   def WritePage(page):
      writer.WriteStartElement("page")
      writer.WriteAttributeString("title", page.title)
      writer.WriteString(page.text)
      writer.WriteEndElement()
      for page in document:
         WritePage(page)

      writer.WriteEndElement()
      writer.WriteEndDocument()
      writer.Flush()
      writer.Close()

This version of write is still an acceptable length, and the number of occurrences of self have been dramatically reduced, making the code more readable!

Now that we’ve created a way of writing MultiDoc documents in XML formats, we ought to provide a way of reading them back in again.

XmlReader is usually instantiated in the same way as XmlWriter—through the static Create method. There’s a plethora of different overloads for creating new instances; you can supply a stream, TextReader, or a resource locator as a string. You can also optionally pass in XmlReaderSettings and an XmlParserContext, and just about any combination of these items.

The resource locator doesn’t need to be a filename; it can also be a URI^[3] so that the XML document can be fetched from the internet. The default XmlResolver used by XmlReader is an XmlUrlResolver, which supports URIs that use the http:// and file:// protocols. You can supply authentication credentials, or use a different resolver by setting it on the instance of XmlReaderSettings that you use to Create your XmlReader.

The most convenient way, especially for this use case, is to supply a filename and an XmlReaderSettings instance. Table 5.4 shows the configurable properties on XmlReaderSettings, along with their default values. If you were reading only a fragment of XML, you might want to set ConformanceLevel to ConformanceLevel.Fragment. If the XML could have processing instructions that you don’t want to handle, then you can set IgnoreProcessingInstructions to True.

Curiously, the default is for the XmlReader to not ignore insignificant whitespace (for example, whitespace that indents elements rather than being part of an attribute value or element content). Because you probably won’t want to handle insignificant whitespace, the following code segment shows the pattern you’ll be using for creating and configuring an XmlReader:

from System.Xml import XmlReader, XmlReaderSettings
settings = XmlReaderSettings()
settings.IgnoreWhitespace = True
reader = XmlReader.Create(filename, settings)

Once you’ve opened the XML file with XmlReader, the most straightforward way to use your instance is to repeatedly call the Read method. This consumes the document, one node at a time, exposing information about each node.

On completion, you should call reader.Close(), which frees the file. Forgetting to do this will cause the file to be held open until the reader is garbage-collected.

If you take the straightforward approach to reading MultiDoc files, the code would be simple, but also tedious and trivial. To make this section more interesting (and possibly even useful), we look at a more general approach to reading XML documents.

A useful pattern for implementing a Read loop is to establish handlers for the types of node that you expect. Read can be called in a simple loop, delegating to the handlers for each node you encounter.

Our documents have elements, with or without attributes and with or without contents. This doesn’t use the whole XML spec, not by a long stretch of the imagination. XML has many additional aspects to it, such as CDATA, namespaces, and processing instructions. You could write more handlers for these components we haven’t yet dealt with, but a significant proportion of XML documents are made up of nothing more than elements, attributes, and text.

The following code is a general XmlDocumentReader class. It’s adapted from a similar class used in Resolver One; many thanks to the kind Resolver Systems folk for letting us use and abuse the code. We’ve mangled it quite a bit from the original, so any bugs are entirely of our own devising. As well as being a general (and easily extensible) XML reading class, it’s another example of using Python functions as first-class objects. The node handlers are packaged as a dictionary of node types mapping to functions (or methods) that know how to process them.

Listing 5.3 shows the imports and the constructor for XmlDocumentReader. It’s instantiated with the node handling functions. These will be explained when we get to using them, and we’ll put together a concrete example for reading MultiDoc documents.

Example 5.3. Importing code and constructor for XmlDocumentReader

Listing 5.4 is the read method of XmlDocumentReader. The element handlers passed into the constructor will be called when the reader encounters different elements in the document. This happens within the onStartElement method, which is called whenever an element start tag is encountered. Because you’re just handling elements, attributes, and contents, you need only three general node handlers: element start tags, element end tags, and the text contents of elements. Element attributes will be dealt with inside the element start tag node handler (listing 5.4).

Example 5.4. XmlDocumentReader read method

nodeTypeHandlers is a dictionary mapping the different XmlNodeTypes to handler functions. The call to reader.Read() advances the reader to the next node. It returns True if the reader finds a node, or False when the end of the document is read—which ends the reading loop.

nodeTypeHandlers.get(reader.NodeType) looks up the current node in the dictionary of handlers. If the node type isn’t recognized, then get returns None and an XmlException^[4] is raised. If the node type is recognized, then the handler is called and the reader passed in. Figure 5.2 shows how the read loop maps XML components it encounters into the handler method calls.

Figure 5.2. The mapping of an XML document to node handler calls

The finally block around the read loop ensures that the reader is closed, whatever error might occur.

The first node handler is onStartElement. This will be called when the reader encounters a start element tag such as <document> or <page title="Page title">. onStartElement is shown in listing 5.5.

Example 5.5. Node handler for start element tags

onStartElement needs to extract the attributes from the element. These are collected in the attributes dictionary in another loop that calls reader.MoveToNextAttribute(). Next you check in the _elementHandlers to see if you have a handler for this type of element. If you do, the handler is called with the line number and attributes as arguments. Otherwise, an XmlException is raised.

_elementHandlers is a dictionary, and get returns a default value if the key (the element name) is missing. You use MISSING_HANDLERS as the default value (as defined in listing 5.1). Every element needs a handler for the start tag, end tag, and contents.

Element handlers are passed into the constructor as a dictionary that maps element names to the three handler functions. Elements can be self-closing (like <element />), which means that they won’t have any contents or a separate end tag. The only required handler is the start tag.^[5] The three handlers for each element type should be provided in a tuple (which will become clearer later when you use XmlDocumentReader to read MultiDoc documents). For handling the page element, you have these handlers:

elementHandlers = {
   'page': (handleStartPage, handlePageText, handleEndPage)
}

If you have elements that don’t have text contents or don’t need their end tag handling, then you can replace these handlers with None.

elementHandlers = {
   'someElement': (handleStartPage, None, None)
}

MISSING_HANDLERS is a tuple of three Nones. self._elementHandlers.get(name, MISSING_HANDLERS) will always return a tuple of three values. The start tag handler is the first value; if the element isn’t contained in the elementHandlers dictionary, then this value will be None and an exception will be raised.

The middle handler in the tuple, index position one, is the handler for the text contents of elements. Listing 5.6 shows the code for the onText method. It’s similar to onStartElement, but simpler, because it doesn’t need to collect attributes. Because the text handler is optional, it doesn’t throw an exception if the text handler is None. The text handler gets called with the same arguments as the start tag handler.

Example 5.6. Node handler for element text values

Tuples of element handlers have three members. The third member is the handler for end element tags. Listing 5.7 shows onEndElement.

def onEndElement(self, reader):
   endHandler = self._elementHandlers.get(reader.Name,
                  MISSING_HANDLERS)[2]
   if endHandler:
    endHandler(reader.LineNumber)

The end tag handler is called with the line number.

In the first part of the chapter, we looked at first-class functions in Python. In this part, we’ve put together a general-purpose XML reader class that uses handler functions stored in a dictionary to process different nodes. As long as the handler functions all have the same signature (by taking the same arguments), you’re free to implement your handler functions how you want.

To illustrate our general purpose reader at work, let’s implement the handler functions needed for the MultiDoc XML save format.