Methods that implement protocols in Python usually have names that start and end with double underscores. They’re known as the magic methods because, instead of you calling them directly, they’re usually called for you by the interpreter.

When you compare objects, the comparison operation will cause the method corresponding to the operation to be called magically (as long as the object provides the appropriate protocol method).

We’ve already encountered several of the protocols in our journey so far: the __init__ constructor and the __getitem__ and __setitem__ methods that implement the mapping and sequence protocols. As you’d expect, there are many more that we haven’t yet encountered. A full list of all the common magic methods is in one of the appendixes, but some of them are so central to Python that we can’t do justice to the language without looking at them.

As the heading implies, Python has a lot of magic methods. You implement these to customize the behavior of your objects. Some of the protocols central to Python are also simple to implement. We start with one of these, the __len__ protocol method.

>>> class Container(object):
...      def __len__(self):
...         return 3
...
>>> container = Container()
>>> len(container)
3
>>>

>>> x = 'string with 'escaped' quotes'
>>> str(x)
"string with 'escaped' quotes"
>>> repr(x)
'"string with 'escaped' quotes"'
>>>

>>> class Something(object):
...     def __str__(self):
...         return 'Something stringified'
...     def __repr__(self):
...         return "Something repr'd"
...
>>> something = Something()
>>> str(something)
'Something stringified'
>>> repr(something)
"Something repr'd"
>>>

>>> print something
Something stringified
>>> something
"Something repr'd"

>>> '%s and %r' % (something, something)
"Something stringified and Something repr'd"

>>> class Something(object):
...     def __nonzero__(self):
...         return False
...
>>> something = Something()
>>> bool(something)
False

Operator overloading allows you to control how objects take part in operations such as addition, subtraction, multiplication, and so on. You can create custom datatypes that represent values and can be used in calculations. You might want to define types that represent different currencies, and use a converter when different currencies are added together.

Each of the built-in operators (+, -, /, *, and friends) has an equivalent magic method that you can implement to support that operation. The method for addition is __add__. This method takes one argument: the object being added to. Traditionally, this argument is called other. Here’s a simple class that behaves like the number three in addition:

>>> class Three(object):
...     def __add__(self, other):
...         return other + 3
...
>>> t = Three()
>>> t + 2
5

That’s nice and easy. What happens if you try a slightly different addition with this class?

>>> 2 + t
Traceback (most recent call last):
  File , line 0, in ##29
TypeError: unsupported operand type(s) for +: 'int' and 'Three'

Because the Three instance is on the right-hand side of the addition, the integer add method is called—and integers don’t know how to add themselves to the custom class. You can solve this problem with the __radd__ method, which is the right-handed companion of __add__ and the method called when a custom type is on the right-hand side of an addition operation.

>>> class Three(object):
...     def __add__(self, other):
...         return other + 3
...     def __radd__(self, other):
...         return 3 + other
...
>>> t = Three()
>>> t + 2
5
>>> 2 + t
5

The good news is that fixing the problem is this simple; the bad news is that you need to implement two methods for each operation. The next listing shows a simple way around this two-methods problem.

i += 1

i = i + 1

Table 8.1 shows the common^[3] operator magic methods and the operations they provide. All the methods listed in this table also have right-hand and in-place equivalents as well.

A practical application of operator overloading is creating custom datatypes. The next listing is a datatype that allows you to store values with a known amount of uncertainty. When you add two uncertain values, the uncertainty is added. When you multiply them, the uncertainty increases dramatically! Listing 8.3 shows the code for an Uncertainty class that supports addition and multiplication.

Example 8.3. Specifying length of custom containers

There are a few things of note in listing 8.3. Like integers and floating point numbers, Uncertainty instances are immutable, so addition and multiplication return new instances rather than modifying themselves. It would be an odd side effect if taking part in an addition modified a number that you held a reference to elsewhere! Inside the numeric protocol methods, Uncertainty has to handle operations involving another Uncertainty differently. The first thing it does is a type check. Because __radd__ needs to do exactly the same as __add__, you make them the same method with an assignment.

There are also different implementations for __repr__ and __str__. The repr of an Uncertainty looks like Uncertainty(6, 3); the str (which is more of a pretty print) looks like 6±3. __str__ returns a Unicode string because it uses a Unicode character. This would cause problems in CPython, unless your default encoding can handle this character, but is fine under IronPython because strings are always Unicode. You’ll still need a terminal (or GUI control) capable of displaying Unicode characters to see the fancy string representation.

Operator overloading isn’t limited strictly to emulating numeric types. You can also implement these methods for objects that don’t represent values, using ordinary Python syntax to perform operations on them. For example, the popular path module by Jason Orendorff^[4] overloads division so that you can concatenate paths to strings by using the division operator, which is the normal path separator on Unix systems.

newPath = path('some path') / 'subdirectory' / 'filename'

This kind of overloading operators with new meanings is a dubious practice in our opinion, but some programmers are fond of this kind of trick. Something that isn’t dubious, and in fact is central to Python, is working with iterators—the subject of the next section.

Iteration means repeatedly performing an action on members of an object. It’s one of the fundamental concepts of programming, so being able to make your own objects iterable (or enumerable) is important.

You saw in the Document class for MultiDoc that providing a sequence-like API (a zero-indexed __getitem__ method) gets you iteration for free. Life is rarely that simple, so you need to know how to support the iteration protocol.

When Python encounters iteration over an object—as a result of a for loop, a list comprehension, or an explicit call to the iter function—it will attempt to call __iter__ on the object. If the object supports iteration, then this method should return an iterator. An iterator is an object that obeys the following three rules:

That’s all nice and easy, but you probably immediately noticed that, unlike the other protocol methods that we’ve used, next isn’t held in the double embrace of underscores. Most magic methods are rarely called directly. The double underscores are a warning sign that, if you’re calling them directly, then you should be sure you know what you’re doing. Calling next on an iterator is perfectly normal. One place where we use this is for incrementing a counter, often on an iterator, like the following, returned by the built-in function xrange:

>>> counter = xrange(100)
>>> counter.next()
0

So without further ado, listing 8.4 demonstrates iteration with a simple class that returns every number between a start and a stop value.

Example 8.4. An example Iterator class

This example is a simple one, but your classes can support iteration by returning an object like this from their __iter__ methods. Your iterator is free to do as much dynamic magic as it wants in next, such as reading from a file or a socket. A simpler, and more powerful, way of implementing an iterator is through generators.

In the iterator created in listing 8.4, you needed to store state, in the form of the count instance variable, so that successive calls to next could calculate the next return value. A simpler pattern is available using the Python yield keyword, which is similar to the C# Yield Return statement.

When a function (or method) uses yield, it becomes a generator. Generators are a lightweight form of coroutines.

Where a function uses yield, the value is returned and execution of the generator is suspended. On the next iteration, execution continues at the point it was suspended. Complex iteration can be implemented with generators, without having to explicitly store state.

Listing 8.5 is an example with a Directory object that stores all the files in a specified path from the filesystem. It recurses into subdirectories, storing them as Directory objects. The __iter__ method is implemented as a generator and also recurses through subdirectories.

Example 8.5. Directory class that supports iteration with a generator

Because the __iter__ method yields, it returns a generator, which is a specific kind of iterator. As an iterator, it has the next method available.

>>> generator = iter(Directory(some_path))
>>> generator.next()
'C:\some_path\some_file.txt'

You can easily extract all members from objects that support iteration, by calling list (or tuple) on them.

>>> paths = list(Directory(some_path))

We can’t leave Python protocols without looking at one of the most common programming concepts implemented with magic methods: equality and inequality.

Comparing objects is a basic part of programming and one of the first things a new programmer will learn; you need to know how to support equality and inequality operations on your own classes. You do this with the __eq__ (equals) and __ne__ (not equals) protocol methods. For equality to work correctly, you need to implement both these methods. If you don’t provide these methods, then Python will use object identity as the test for equality. An object will only compare equal to itself and not equal to everything else.

Listing 8.6 is a class with custom equality and inequality methods defined.

Example 8.6. Object equality and inequality methods

The __ne__ method returns the not of the equality method. This is the simplest possible implementation, and it would be nice if Python did this for you! The __eq__ method always returns False unless it’s compared against another Value instance. If it’s compared to a Value object, then it compares value attributes. Without the isinstance check, equality testing would fail with an attribute error for objects that don’t have a value attribute.

def __hash__(self):
    return hash(self.value)

def __hash__(self):
    raise TypeError('Value objects are unhashable')

As well as protocols to control equality and inequality comparisons, there are protocol methods for the other comparison operators.^[8] Table 8.2 shows the rich comparison operators and corresponding protocol methods.

To support the full range of rich comparison operators, you need to implement all these methods.^[9]

We’ve now looked at quite a range of Python’s protocol methods. We haven’t used them all (by any stretch of the imagination), but we’ve covered all the most common ones. The important thing is to know the ones that you’ll use most often and where to look up the rest!

The next section will look at two protocol methods that are a little different. Instead of enabling a specific operation, __getattr__ and friends allow you to customize attribute access.

The following are four built-in functions available for working with attributes:

So what are these functions useful for?

hasattr in particular is useful as a duck typing mechanism. If you want to support a particular interface, one possible pattern is it’s better to ask forgiveness than ask permission.

try:
    instance.someMethod()
except AttributeError:
    # different kind of object

Sometimes you want to ask permission, though—perhaps particular objects need different treatment. You could do type checking with isinstance, but this defeats duck typing and requires users of your code to use specific types instead of passing in objects with a compatible API. Instead, you can use hasattr, as follows:

if hasattr(instance, 'someMethod'):
    instance.someMethod()
    # and so on

getattr, setattr, and delattr are particularly useful when you have a list of attributes as strings (potentially as the result of a call to dir) and need to loop over the list performing operations. In a brief while, we’ll use these functions in a proxy class that illustrates the attribute-access protocol methods. Before we can do that, you need to learn about the attribute-access protocol methods themselves.

As with built-in functions like len and str, the attribute-access functions have corresponding protocol methods that you can implement. With these functions, the relationship between the protocol method and the corresponding function is a little more complex.

The attribute-access functions invoke the full attribute lookup mechanism for Python objects. Included in this mechanism are three protocol methods that you can implement to govern how some attributes are fetched, set, and deleted.

The three protocol methods (hasattr has no direct equivalent as a protocol method) are as follows:

As you can see, these methods are asymmetrical. __setattr__ and __delattr__ are invoked for all set and delete operations on an instance, whereas __getattr__ is only invoked if the attribute doesn’t exist. According to the Python documentation,^[11] this is for efficiency and to allow __setattr__ to access instance variables.

At Resolver Systems, we’ve created a spreadsheet application with IronPython.^[12] This is a spreadsheet for creating complex models or business applications using a spreadsheet interface. IronPython objects that represent the spreadsheet are exposed to the user, and can be manipulated from user code sections. One of our core classes is the Worksheet, representing different sheets in the spreadsheet. The user accesses values in a worksheet by indexing it with the column and the row number. Most spreadsheet users are more used to referring to locations by the A1 style name, so we use __getattr__ to provide a convenient way of accessing values using a syntax like this: worksheet.A1 = 3. We implemented this with code that looks like the following:

def __getattr__(self, name):
    location = CoordinatesFromCellName(name)
    if location is None:
        raise AttributeError(name)
    col, row = location
    return self[col, row]

We also need to implemented __setattr__ to allow users to set values in the worksheet. __setattr__ is called for all attribute-setting operations, so we needed to delegate to object.__setattr__ for all attributes except cells.

def __setattr__(self, name, value):
    location = CoordinatesFromCellName(name)
    if location is not None:
        col, row = location
        self[col, row] = value
    else:
        object.__setattr__(self, name, value)

You can put these methods to work in a proxy class that proxies attribute access from one object to another.

You’ve seen how Python has no true concept of private or protected members. If you come to Python from a language like C#, you’ll probably be surprised at how infrequently you need these features. You can achieve the same effect as private members through a factory function and a proxy class like the one in listing 8.7.

Example 8.7. Attribute protection with a factory function and proxy class

You pass in the object you want proxied as the thing argument to GetProxy. You also pass in three lists of attribute names (all optional). These are attributes that you do want to allow access to—for read access, write access, and delete access.

When you call GetProxy, it returns an instance of the Proxy class. This instance has access to thing, and the attribute lists, through the closure; but, from the instance, there’s no way to get back to the original object. Accessing attributes on the Proxy instance triggers __getattr__ (or __setattr__ or __delattr__). If the attribute name is in the corresponding attribute list, then the access is allowed. If the attribute isn’t in the list, then the access is disallowed, and an AttributeError is raised.

This pattern is useful for more than protecting attribute access; it’s a general example of the delegation pattern,^[13] but it does have a couple of restrictions. Although it works fine for fetching normal methods, magic methods are looked up on the class rather than the instance. Indexing the object, or any other operation that uses a protocol method, will look for the method on the Proxy class instead of the original instance. Two possible approaches to solving this are as follows:

Another restriction is that, although attribute access is proxied, the proxy instance has a different type to the object it’s proxying and so can’t necessarily be used in all the same circumstances as the original.

Using the __getattr__ and __setattr__ protocol methods to customize attribute access isn’t something you do every day; but, in the right places, they can be used to create elegant and intuitive APIs.

Something that takes us even deeper into the dynamic aspects of Python is metaprogramming with Python metaclasses. Understanding metaclasses will further deepen your understanding of Python and open up some fun possibilities.

In Python, everything is an object. Functions and classes are all first-class objects that can be created at runtime and passed around your code. Every object has a type. For most objects, their type is their class, so what’s the type of a class? The answer is that classes are instances of their metaclass.

Just as objects are created by calling their class with the correct parameters, classes are created by calling their metaclass with certain parameters. The default metaclass (the type of classes) is type. This leads to the following wonderful expression:

type(type) is type

type is itself a class, so its type is type!^[14]

What does this have to do with metaprogramming? Python is an interpreted language—classes are created at runtime; and, by providing a custom metaclass, you can control what happens when a class is created, including modifying the class.

As usual, the easiest way to explain this is to show it in action. Listing 8.8 shows the simplest possible metaclass.

Example 8.8. The simplest example of a metaclass

This metaclass doesn’t do anything, but illustrates the basics of the metaclass. You set the metaclass on a class by assigning the __metaclass__ attribute inside the class definition. Subclasses automatically inherit the metaclass of their superclass.^[15] When the class is created, the metaclass is called with a set of arguments. These are the class name, a tuple of base classes, and a dictionary of all the attributes (including methods) defined in the class. To customize class creation with a metaclass, you need to inherit from type and override the __new__ method.

You can experiment here by defining methods and attributes on SomeClass, and putting print statements in the body of PointlessMetaclass. You’ll see that methods appear as functions in the classDict, keyed by their name. Inside the metaclass, you can modify this dictionary (plus the name and the bases tuple if you want) to change how the class is created. But what can you use metaclasses for?

Despite their reputation for being deep magic, sometimes only a metaclass can achieve something that’s difficult or impossible to do via other means. They’re invoked at class-creation time, so they can be used to perform operations that would otherwise require a manual step. These operations include^[16] the following:

We can show a practical use of metaclasses with a profiling metaclass. This is an example of the fifth use of metaclasses listed—wrapping every method in a class with a decorator function.

One of the use cases mentioned in the bulleted list is wrapping all methods with a decorator. You can use this for profiling by recording method calls and how long they take. This approach is useful if you’re looking to optimize the performance of your application. With IronPython you always have the option of moving parts of your code into C# to improve performance; but, before you consider this, it’s important to profile so that you know exactly which parts of your code are the bottlenecks—the results are often not what you’d expect. At Resolver, we’ve been through this process many times, and have always managed to improve performance by optimizing our Python code; we haven’t had to drop down into C# to improve speed so far.

For profiling IronPython code you can use the .NET 'DateTime'^[18] class.

from System import DateTime
start = DateTime.Now

someFunction()

timeTaken = (DateTime.Now - start).TotalMilliseconds

There’s a drawback to this code. DateTime has a granularity of about 15 milliseconds. For timing individual calls to fast-running code, this can be way too coarse. An alternative is to use a high-performance timer class from System.Diagnostics: the Stopwatch^[19] class. Listing 8.9 is the code to time a function call using a Stopwatch.^[20]

from System.Diagnostics import Stopwatch
s = Stopwatch()
s.Start()

someFunction()

s.Stop()
timeTaken = s.ElapsedMilliseconds

You can wrap this code in a decorator that tracks the number of calls to functions, and how long each call takes (listing 8.10).

Example 8.10. A function decorator that times calls

The profiler decorator takes a function and returns a wrapped function that times how long the call takes. The times are stored in a cache (the times dictionary), keyed by the function name.

Now you need a metaclass that can apply this decorator to all the methods in a class. You’ll be able to recognize methods by using FunctionType from the Python standard library types module. Listing 8.11 shows a metaclass that does this.

Example 8.11. A profiling metaclass that wraps methods with profiler

Of course, having created a profiling metaclass, we need to use it. Listing 8.12 shows how to use the ProfilingMetaclass.

from System.Threading import Thread

class Test(object):

    __metaclass__ = ProfilingMetaclass

    def __init__(self):
        counter = 0
        while counter < 100:
            counter += 1
            self.method()

    def method(self):
        Thread.CurrentThread.Join(20)

t = Test()

for name, calls in times.items():
    print 'Function: %s' % name
    print 'Called: %s times' % len(calls)
    print ('Total time taken: %s seconds' %
               (sum(calls) / 1000.0))
    avg = (sum(calls) / float(len(calls)))
    print 'Max: %sms, Min: %sms, Avg: %sms' % (max(calls), min(calls), avg)

When the Test class is created, all the methods are wrapped with the profiler. When it’s constructed, it calls method 100 hundred times. When the code has run, you print out the entries in the times cache to analyze the results. It should print something like this:

Function: method
Called: 100 times
Total time taken: 2.07 seconds
Max: 39ms, Min: 16ms, Avg: 20.7ms

Function: __init__
Called: 1 times
Total time taken: 2.093 seconds
Max: 2093ms, Min: 2093ms, Avg: 2093.0ms

If this were real code, you’d know how many calls were being made to each method and how long they were taking. With this simple technique, the times cache includes calls to other methods, as well as overhead for the profiling code itself. It’s possible to write more sophisticated profiling code that tracks trees of calls so that you can see how long different branches of your code take.

Metaclasses are magic, but perhaps not as black as they’re painted. This is true of all the Python magic that you’ve been learning about. Python’s magic methods are magic because they’re invoked by the interpreter on your behalf, not because they’re difficult to understand. Using these protocol methods is a normal part of programming in Python. You should now be confident in navigating which methods correspond to language features and how to implement them.

We’ve plunged into the depths of Python, so now it’s time to take a journey around the edges—in particular, the edges where IronPython meets the Common Language Runtime (CLR). Not all the key concepts in the CLR match directly to Python semantics, particularly because it’s a dynamic language. To make full use of framework classes, you need to know a few details.

Arrays are one of the standard .NET container types, roughly equivalent to the Python tuple.^[23] They’re also statically typed, of course. When you create an array, you specify type and the size. The array can then be populated only with members of the specified type.

The tuple is a built-in Python type; and, being dynamically typed, it’s generally more convenient to use than arrays when you have a choice. When you’re working with .NET APIs, they will sometimes require an array, and you need to know how to create them.

In C#, you can create an array of integers using the syntax in figure 8.1.

Figure 8.1. Creating an integer array in C#

In IronPython, you use the following syntax to achieve the same thing:

>>> from System import Array
>>> intArray = Array[int]((1, 2, 3, 4, 5))

This syntax looks like you’re indexing the Array type, so it’s valid Python syntax. This tells IronPython that you want an array of integers. It’s populated from a tuple of the initial members.

List<string> dinosaurs = new List<string>();

dinosaurs = List[str]()

This technique isn’t the only way to create arrays. Another way is by using the CreateInstance class method, which can be used to create multidimensional arrays.

>>> multiArray = Array.CreateInstance(str, 5)

>>> multiArray = Array.CreateInstance(str, 5, 5)

>>> aMember = multiArray[2, 2]

>>> dimensions = Array[int]((5, 5, 5, 5, 5))
>>> multiArray = Array.CreateInstance(object, dimensions)
>>> multiArray[0, 0, 0, 0, 0] = SomeObject()
>>> print multiArray[0, 0, 0, 0, 0]
<SomeObject object at 0x....>

>>> arrays = tuple(Array[int]((0, 0, 0, 0, 0)) for i in range(5))
>>> multiArray = Array[Array[int]](arrays)
>>> multiArray[0][0]
0

C# allows overloaded methods—the same method being defined several times, but with different types in the parameter list. The compiler can tell at compile time which of the overloads you’re calling by the types of the arguments.

This largely works seamlessly in IronPython. The IronPython runtime uses reflection (the .NET equivalent of introspection using the System.Reflection namespace) to deduce which overload you’re calling. If IronPython gets it wrong, it’s a bug and you should report it on the IronPython mailing list or the issue tracker on the IronPython CodePlex site!

On occasion, you may want direct access to a specific overload. IronPython provides an Overloads collection that you can index by type (or a tuple of types). The IronPython documentation provides the following illustration, choosing the WriteLine method that takes an object and then passing None to it:

from System import Console
Console.WriteLine.Overloads[object](None)

Three more .NET concepts that don’t map easily to Python are the out, ref, and pointer parameters.

Occasionally you’ll find a .NET method that requires an out or ref type parameter, an argument that takes a pointer, or one that uses the params keywords. These concepts don’t exist directly in IronPython, but you can handle them all straightforwardly.

In C#, the body of the method modifies out (output) parameters. In Python, if you passed in an immutable value for this parameter, such as a string or an integer, then in Python the original reference wouldn’t be modified—which would kind of defeat the purpose of having the parameter. This difficulty is easily solved in IronPython. Because the original value of an out parameter isn’t used, the IronPython runtime does some magic, and the updated value becomes an extra return value. Let’s observe this in action with the Dictionary.TryGetValue method.^[26] This method takes two parameters: the key you’re searching for and an out parameter modified to contain the corresponding value if it’s found. Normally, it returns a Boolean that indicates whether or not the key was found. The out parameter is modified to contain the value. In IronPython, you get two return values.

>>> from System.Collections.Generic import Dictionary
>>> d = Dictionary[str, str]()
>>> d['key1'] = 'Value 1'
>>> d['key2'] = 'Value 2'
>>> d.TryGetValue('key1')
(True, 'Value 1')
>>> d.TryGetValue('key3')
(False, None)

Passing by reference is similar; in fact, you can often pass by reference instead of using an out parameter. To pass by reference, you need to create a reference using a type provided by the clr module. One example is the class method Array.Resize for resizing arrays. It’s a generic method, so you need to provide type information—making it a great example of how dynamic typing makes life easier! This method takes a reference to an array and the new size as an integer.

First you construct an array of integers to resize, and then create a reference to that array. clr.Reference is a generic, so you need to tell it the type of the reference you require.

>>> from System import Array
>>> import clr
>>> array = Array[int]((0, 1, 2, 3, 4))
>>> array
Array[int](0, 1, 2, 3, 4)
>>> ref = clr.Reference[Array[int]](array)

Next you call Array.Resize. You need to specify the type of array (again); and, because Resize is a generic method, you need to specify the type of array here as well (otherwise the IronPython runtime won’t be able to find the right target and will report TypeError: no callable targets). The resized array is stored as the Value attribute of the reference.

>>> Array[int].Resize[int](ref, 3)
>>> ref.Value
Array[int]((0, 1, 2))

A pointer is a good old-fashioned kind of reference. You usually come across them when dealing with unmanaged resources, and often they’ll be pointers to integers. Integer pointers are represented with the System.IntPtr structure.

>>> from System import IntPtr
>>> ptr = IntPtr(3478)
>>> ptr.ToInt32()
3478
>>> IntPtr.Zero
<System.IntPtr object at 0x... [0]>

We’ve not encountered this in the wild; but, for completeness, we need to mention another way of passing arguments: the C# params keyword. This is used to pass a variable number of arguments (like the Python *args syntax), and the method receives a collection of the arguments you pass. In IronPython, .NET APIs that use the params keyword are handled seamlessly—you pass arguments as normal.

>>> something.Method(1, 2, 3, 4)

Another area of .NET that can potentially cause confusion is the peculiarity of value types.

The .NET framework makes a distinction between value types and reference types. The technical distinction is that value types are allocated on the stack or inline within an object, whereas reference types are allocated on the heap. Value types are passed by value (rather than by reference—hence, the names) making them efficient to use.^[27] Table 8.3 lists the value and reference types.

In .NET land, you can pass a value type to methods expecting a reference type. Take, for example, this overload of Console.WriteLine:

Console.WriteLine(String, Object)

This takes a format string and an object, and it writes the string representation of the object to the standard output stream. The declaration of Object as the second argument means that this method can take any arbitrary object, but also means that it’s expecting a reference type (because Object is a reference type). If you call this method with a value type, like an integer, then the CLR will automatically box and unbox the value so that it can be used in place of a reference type.

>>> from System.Net.Sockets import SocketOptionName
>>> int(SocketOptionName.UnblockSource)
18

>>> str(SocketOptionName.UnblockSource)
'UnblockSource'

The boxing and unboxing can lead to certain difficulties, particularly because Python doesn’t have value types and the programmer will expect everything to behave as a reference.

Take this code that uses Point, which, as a structure, is a value type:

>>> from System import Array
>>> from System.Drawing import Point
>>> point = Point(0, 0)
>>> array = Array[Point]((point,))
>>> array[0].X = 30
>>> array[0].X
0

This code creates a new Point object (point) and stores it in an array (array). When you fetch the point by indexing the array, you end up setting the X member on an unboxed value—probably not on the object you expected. This problem isn’t restricted to Python; if you re-create the code in C#, the same thing happens.

The situation is worse if you attempt to modify a value type exposed as a field (attribute, in Python speak) on an object. Referencing the value type will return a copy, and the update would be lost. In these cases, the IronPython designers have decided^[28] that the best thing to do is to raise a ValueError rather than letting the unexpected behavior slip through. As a result, value types are mostly immutable,^[29] although updates will still be possible by methods exposed on the value types themselves.

Interfaces are a way of specifying that certain types have known behavior. An interface defines functionality, and the methods and properties for that functionality.

Where a class implements an interface, that interface will be added to the method resolution order^[30] when you use it from Python (effectively making the interface behave like a base class—coincidentally you can also implement an interface in a Python class by inheriting from it). Even if an explicitly implemented interface method is marked as private on a class, you’ll still be able to call it.

As well as calling the method directly, you can call the method on the interface—passing in the instance as the first argument in the same way you pass the instance (self) as the first argument when calling an unbound method on a class.^[31]

The following snippet shows an example of calling BeginInit on a DataGridView, through the ISupportInitialize interface that it implements:

>>> import clr
>>> clr.AddReference('System.Windows.Forms')
>>> from System.Windows.Forms import DataGridView
>>> from System.ComponentModel import ISupportInitialize
>>> grid = DataGridView()
>>> ISupportInitialize.BeginInit(grid)

How is this useful? Well, it’s possible for a class to implement two interfaces with conflicting member names. This technique allows you to call a specific interface method.

This workaround is fine for methods, but you don’t normally pass in arguments to invoke properties. Instead, you can use GetValue and SetValue on the interface property descriptor.

ISomeInterface.PropertyName.GetValue(instance)
ISomeInterface.PropertyName.SetValue(instance, value)

Working with interfaces is mostly straightforward. Unfortunately, when it comes to .NET attributes, it’s a much sorrier tale.

Attributes in the .NET framework are a bit like decorators in Python and annotations in Java. They’re metadata applied to objects that provide information to the CLR. They can be applied to classes, methods, properties, and even method parameters or return values (that is, pretty much everywhere).

Attributes are used to provide documentation, specify runtime information, and even specify behavior. The DllImport attribute provides access to unmanaged code; and, in Silverlight, you use the Scriptable attribute to expose objects to JavaScript. The following code segment uses the DllImport attribute to expose functions user32.dll as static methods on a class in C#:

[DllImport("user32.dll")]
public static extern IntPtr GetDesktopWindow();

[DllImport("user32.dll")]
public static extern IntPtr GetTopWindow(IntPtr hWnd);

Because attributes are for the compiler, they’re used at compile time—a problem for IronPython. First, there’s no syntax in Python that maps to attributes. The IronPython team is keen to avoid introducing syntax to IronPython that isn’t backwards compatible with Python.

A worse problem is that IronPython classes aren’t true .NET classes. There are various reasons for this, among them that .NET classes aren’t garbage collected and that you can swap out the type of Python classes at runtime! The upshot is that you can’t apply attributes to Python classes.

The most common solution is to write stub C# classes and then subclass from IronPython. You can even generate and compile these classes at runtime. We use both of these techniques later in the book. This isn’t the end of the story, though—the IronPython team is still keen to find a solution to this problem that will allow you to use attributes from IronPython.

A feature of IronPython 1 that nearly didn’t make it into IronPython 2 is the ability to compile Python code into binary assemblies. Fortunately, this feature made a comeback in IronPython 2 beta 4.

IronPython works by compiling Python code to assemblies in memory. If you execute a Python script using the IronPython ipy.exe executable, then you can dump these assemblies to disk by launching it with the -X:SaveAssemblies command-line arguments. They’re not executable on their own though; they’re useful for debugging, but they make calls into dynamic call sites^[32] generated at runtime when running IronPython scripts.

The clr module includes a function that can save executable assemblies: CompileModules (IronPython 2 only). Its call signature is as follows:

clr.CompileModules(assemblyName, *filenames, **kwArgs)

This function compiles Python files into assemblies that you can add references to and import from in the normal way with IronPython. This feature allows you to package several Python modules as a single assembly. The following snippet of code takes two modules and outputs a single assembly:

import clr
clr.CompileModules("modules.dll", "module.py", "module2.py")

Having created this assembly, you can add a reference to it and then import the module and module2 namespaces from it.

import clr
clr.AddReference('modules.dll')
import module, module2

CompileModules also takes a keyword argument, mainModule, that allows you to specify the Python file that acts as the entry point for your application. This still outputs to a dll rather than an exe file, but it can be combined with a stub executable to create binary distributions of Python applications. Creating a stub executable can be automated with some fiddly use of the .NET Reflection.Emit API, but it’s far simpler to use the IronPython Pyc^[33] sample. Pyc is a command-line tool that creates console or Windows executables from Python files. It comes with good documentation, plus a command-line help switch. The basic usage pattern is as follows:

ipy.exe pyc.py /out:program.exe /main:main.py /target:winexe module.py module2.py

In this chapter, we’ve climbed under the hood of both the Python language and how IronPython interacts with the .NET framework. It’s time to summarize and move on to the next chapter.