You want to code and build a C extension type for Python with a minimal amount of hard work.

First of all, we need to create a setup.py file to use the distutils package to build and install our module:

from distutils.core import setup, Extension
setup(name = "elemlist",
      version = "1.0",
      maintainer = "Alex Martelli",
      maintainer_email = "[email protected]",
      description = "Sample, simple Python extension module",
      ext_modules = [Extension('elemlist',sources=['elemlist.c'])]
)

Then, we need a file elemlist.c with our module’s source code:

#include "Python.h"
/* type-definition and utility-macros */
typedef struct {
    PyObject_HEAD
    PyObject *car, *cdr;
} cons_cell;
staticforward PyTypeObject cons_type;
/* a type-testing macro (we don't actually use it here) */
#define is_cons(v) ((v)->ob_type == &cons_type)
/* utility macros to access car and cdr, as either lvalues or rvalues */
#define carof(v) (((cons_cell*)(v))->car)
#define cdrof(v) (((cons_cell*)(v))->cdr)
/* ctor ("internal" factory-function) and dtor */
static cons_cell*
cons_new(PyObject *car, PyObject *cdr)
{
    cons_cell *cons = PyObject_New(cons_cell, &cons_type);
    if(cons) {
        cons->car = car; Py_INCREF(car); /* INCREF when holding a PyObject */
        cons->cdr = cdr; Py_INCREF(cdr); /* ditto */
    }
    return cons;
}
static void
cons_dealloc(cons_cell* cons)
{
    /* DECREF when releasing previously-held PyObject*'s */
    Py_DECREF(cons->car); Py_DECREF(cons->cdr);
    PyObject_Del(cons);
}
/* A minimal Python type-object */
statichere PyTypeObject cons_type = {
    PyObject_HEAD_INIT(0)     /* initialize to 0 to ensure Win32 portability  */
    0,                        /* ob_size */
    "cons",                   /* tp_name */
    sizeof(cons_cell),        /* tp_basicsize */
    0,                        /* tp_itemsize */
    /* methods */
    (destructor)cons_dealloc, /* tp_dealloc */
    /* implied by ISO C: all zeros thereafter, i.e., no other method */
};
/* module-functions */
static PyObject*
cons(PyObject *self, PyObject *args)    /* the exposed factory-function */
{
    PyObject *car, *cdr;
    if(!PyArg_ParseTuple(args, "OO", &car, &cdr))
        return 0;
    return (PyObject*)cons_new(car, cdr);
}
static PyObject*
car(PyObject *self, PyObject *args)     /* car-accessor */
{
    PyObject *cons;
    if(!PyArg_ParseTuple(args, "O!", &cons_type, &cons)) /* type-checked */
        return 0;
    return Py_BuildValue("O", carof(cons));
}
static PyObject*
cdr(PyObject *self, PyObject *args)     /* cdr-accessor */
{
    PyObject *cons;
    if(!PyArg_ParseTuple(args, "O!", &cons_type, &cons)) /* type-checked */
        return 0;
    return Py_BuildValue("O", cdrof(cons));
}
static PyObject*
setcar(PyObject *self, PyObject *args)  /* car-setter */
{
    PyObject *cons;
    PyObject *value;
    if(!PyArg_ParseTuple(args, "O!O", &cons_type, &cons, &value))
        return 0;
    Py_INCREF(value);
    Py_DECREF(carof(cons));
    carof(cons) = value;
    return Py_BuildValue("");
}
static PyObject*
setcdr(PyObject *self, PyObject *args)  /* cdr-setter */
{
    PyObject *cons;
    PyObject *value;
    if(!PyArg_ParseTuple(args, "O!O", &cons_type, &cons, &value))
        return 0;
    Py_INCREF(value);
    Py_DECREF(cdrof(cons));
    cdrof(cons) = value;
    return Py_BuildValue("");
}
static PyMethodDef elemlist_module_functions[  ] = {
    {"cons",   cons,   METH_VARARGS},
    {"car",    car,    METH_VARARGS},
    {"cdr",    cdr,    METH_VARARGS},
    {"setcar", setcar, METH_VARARGS},
    {"setcdr", setcdr, METH_VARARGS},
    {0, 0}
};
/* module entry-point (module-initialization) function */
void
initelemlist(void)
{
    /* Create the module, with its functions */
    PyObject *m = Py_InitModule("elemlist", elemlist_module_functions);
    /* Finish initializing the type-objects */
    cons_type.ob_type = &PyType_Type;
}

C-coded Python extension types have an undeserved aura of mystery and difficulty. Sure, it’s a lot of work to implement every possible feature, but a minimal yet useful type doesn’t necessarily take all that much effort.

This module is roughly equivalent to the Python-coded module:

def cons(car, cdr): return car, cdr
def car(conscell): return conscell[0]
def cdr(conscell): return conscell[1]
def setcar(conscell, value): conscell[0] = value
def setcdr(conscell, value): conscell[1] = value

except that the C source is about 25 times larger, even excluding comments and empty lines (and it is not much faster than the Python-coded version, either).

However, the point of this recipe is to demonstrate a minimal C-coded extension type. I’m not even supplying object methods (except the indispensable destructor) but, rather, I am providing module-level functions to build cons cells and to read and write their car and cdr fields. This recipe also shows the utter simplicity of building a C-coded extension module on any platform, thanks to the distutils package, which does all of the hard work.

Lisp-savvy readers will have recognized from the names involved that this little extension offers the core functionality to implement a Lisp-like linked list type—using some NIL marker (e.g. None), by convention, as the cdr of the last cons-cell of a list, and otherwise "consing up a list” by having every cdr be another cons-cell. You might easily constrain the cdr to be either None or another cons-cell, giving up on generality for a bit of extra error checking.

Because this recipe is meant as an introduction to writing extension modules in C for Python, here are the instructions for building this extension module, assuming you have a Windows machine with Python 2.3 and Microsoft Visual C++ 6 (or the free command-line equivalent that you can download from Microsoft’s site as a part of their .NET Framework SDK). You can presumably translate mentally to other platforms such as Linux with gcc, Mac OS X with gcc, and so on. On the other hand, using different C compilers on Windows involves more work, and I’m not going to cover that here (see http://sebsauvage.net/python/mingw.html).

Here are the steps you should follow to build this recipe’s extension:

There—your new extension module is installed and ready!

The Extending and Embedding manual is available as part of the standard Python documentation set at http://www.python.org/doc/current/ext/ext.html; the section “Distributing Python Modules” of the standard Python documentation set is still incomplete, but it’s a reliable source of information on the distutils package. Python in a Nutshell covers the essentials of extending and embedding and of the distutils package.

You want to code and build an extension type for Python with a minimal amount of hard work.

The Pyrex language is the simplest and fastest way to build Python extensions. Once we have Pyrex installed, the next step is to create a setup.py file to use the distutils package to build and install our module:

from distutils.core import setup, Extension
from Pyrex.Distutils import build_ext
setup(name = "elemlist",
      version = "1.0",
      maintainer = "Alex Martelli",
      maintainer_email = "[email protected]",
      description = "Simple Python extension module in Pyrex",
      ext_modules = [Extension('elemlist',sources=['elemlist.pyx'])],
      cmdclass = {'build_ext': build_ext},
)

Then, we need a file elemlist.pyx with our module’s source code:

cdef class cons:
    cdef public object car, cdr
    def _ _init_ _(self, car, cdr):
        self.car = car
        self.cdr = cdr
    def _ _repr_ _(self):
        return 'cons(%r, %r)' % (self.car, self.cdr)

Pyrex is a language for writing Python extension modules. It was developed by Greg Ewing and is freely downloadable and installable. The Pyrex language is a large subset of Python, with the addition of a few language constructs to allow easy generation of fast C code. In this recipe, the only special Pyrex construct we use is the cdef statement, which lets us express C-level declarations.

This module is roughly equivalent to the Python-coded module:

class cons(object):
    _ _slots_ _ = ('car', 'cdr')
    def _ _init_ _(self, car, cdr):
        self.car = car
        self.cdr = cdr
    def _ _repr_ _(self):
        return 'cons(%r, %r)' % (self.car, self.cdr)

As you see, Pyrex code is very similar to Python code. Pyrex code gets compiled into C, which in turn gets compiled to machine code, while Python code is compiled into bytecode instead. For such a simple type as cons, the Pyrex version is not much faster than the Python version, but a pure C version, such as the one that I demonstrated previously in Recipe 17.1, despite having 25 times as much code, wouldn’t be any faster either.

Building a compiled extension module is just as simple when it’s coded in Pyrex as when it’s coded directly in C, thanks to the distutils package, which does all of the hard work. (You need to have Pyrex installed.) When you build a Pyrex-coded module, you may get warnings from your C compiler about symbols with names starting with _ _pyx or _ _pyx that are defined but not used, or declared but not defined. Do not let these warning messages worry you: your C compiler is running with the highest possible level of warnings, and the little anomalies it’s warning about are just perfectly normal and innocuous artifacts in the C sources that Pyrex automatically generates. Pyrex is not quite finished yet (the Pyrex version at the time of writing is 0.9.3), so no attention has yet been spent on purely cosmetic warts. (By the time you read this, a mature version of Pyrex may have been released, with all i’s dotted and all t’s crossed. Nevertheless, I would recommend Pyrex even if the latest version still causes numerous warnings.)

Abundant documentation on Pyrex, as well as examples, can be found in the directory (and particularly in subdirectories Doc and Demos) where you unpacked Pyrex’s .tar.gz file; essentially the same documentation can also be read online, starting from the Pyrex web site at http://nz.cosc.canterbury.ac.nz/~greg/python/Pyrex/.

You want to use a C++ library in Python. For example, you might have a fast rational-numbers library, coded in C++, that you wish to wrap for use from Python.

Boost, http://www.boost.org, is a large free package with more than 50 fast and solid C++ libraries. Among those libraries, we find both Boost.Rational, a rational number library, and Boost.Python, which makes it easy to turn any other C++ library into a Python extension. So, we simply use the latter to wrap the former:

#include <boost/python.hpp>
#include <boost/rational.hpp>
/* two specific conversion functions: rational to float and to str */
static double
as_float(boost::rational<int> const& self)
{
  return double(self.numerator( )) / self.denominator( );
}
static boost::python::object
as_str(boost::rational<int> const& self)
{
  using boost::python::str;
  if (self.denominator( ) == 1) return str(self.numerator( ));
  return str(self.numerator( )) + "/" + str(self.denominator( ));
}
/* the 'rational' Python extension module, with just one class in it: */
BOOST_PYTHON_MODULE(rational)
{
  boost::python::class_<boost::rational<int> >("int")
    .def(boost::python::init<int, optional<int> >( ))
    .def("numerator", &boost::rational<int>::numerator)
    .def("denominator", &boost::rational<int>::denominator)
    .def("_ _float_ _", as_float)
    .def("_ _str_ _", as_str)
    .def(-self)
    .def(self + self)
    .def(self - self)
    .def(self * self)
    .def(self / self)
    .def(self + int( ))
    .def(self - int( ))
    .def(self * int( ))
    .def(self / int( ))
    .def(int( ) + self)
    .def(int( ) - self)
    .def(int( ) * self)
    .def(int( ) / self)
  ;
}

Once you have built and installed the rational extension shown in this recipe’s Solution, you can use it to perform simple, natural tasks, such as:

>>> import rational
>>> x = rational.int(1, 3)
>>> y = rational.int(-2, 4)
>>> print "x =", xx = 1/3
>>> print "y =", y
y = -1/2
>>> print "x+y =", x+y
x+y = -1/6
>>> print "x*2 =", x * 2
x*2 = 2/3
>>> print "3/y =", 3 / y
3/y = -6

Compiling and linking Boost.Python extensions is supported by the Boost.Build tool; we do not cover that topic here. Extensive documentation is available online at the Boost site. Such tools as make and SCons are also popular for software compilation and linking tasks, including tasks that involve Boost.

The Solution’s code shows off a few of Boost.Python’s powerful features. Consider the snippet:

BOOST_PYTHON_MODULE(rational)
{
  class_<boost::rational<int> >("int")
    ...

The BOOST_PYTHON_MODULE macro takes a module name as a parameter, and a module body immediately afterwards within braces, and does all that’s needed to make a module we can import from Python.

The class_ template, instantiated with the boost::rational type as a parameter and “called” with the string argument "int“, does all we need to have as part of our module a Python-usable class, named rational.int, each of whose instances wraps an instance of the boost::rational class. The type boost::rational is itself a template, and we instantiate that template with int as a parameter, to use int as the type of each rational number’s numerator and denominator.

If we stopped here, wrapping a C++ class in the class_ template, and exposing the wrapper without adding any more to it, we’d have a rather empty type available on the Python side. It would have no constructor (save for the default argument-less one), no methods, and no attributes. To remedy this, the Solution code continues with several .def(...) calls, which are chained: each call enriches the object, and also returns it, so you can just string such calls one after the other. The methods we add with all those def calls include a constructor (which uses the init template), then a couple of ordinary methods that delegate to the methods of the same names in the wrapped class (accessors to the numerator and denominator parts of a rational number), and then a couple of type-conversion special methods for which we’ve previously coded corresponding functions (just before the BOOST_PYTHON_MODULE macro). Note, in particular, that the implementation of the as_str function is so concise because it makes full use of Boost.Python’s object interface—it’s almost like writing Python in C++.

The baker’s dozen of .def(...) calls that begins with:

    .def(-self)

and proceeds all the way to:

    .def(int( ) / self)

exposes all the arithmetic special methods for our new rational.int class—unary minus (_ _neg_ _), and the four operations, each in three versions—between two instances of our class, and between such instances and ints on either side (_ _add_ _, _ _radd_ _, etc.). The magic is performed using expression templates, a technique originally developed for optimizing high-performance matrix algebra expressions. Boost.Python’s use of expression templates has a different purpose, but it certainly comes in handy anyway!

A comprehensive rational number extension would require more functionality—comparison operators, _ _repr_ _, _ _hash_ _, support for pickling, and so on. A more complete implementation, one that is actively used in applications, can be found at http://cvs.sourceforge.net/viewcvs.py/cctbx/boost_adaptbx/, in the file rational_ext.cpp.

Boost’s site is http://www.boost.org; the rational number library Boost.Rational, is at http://www.boost.org/libs/rational; Boost.Python is at http://www.boost.org/libs/python.

You want to avoid writing a Python extension in C, by directly calling from Python functions that already exist in a Windows DLL.

The third-party ctypes extension makes this task pretty easy:

from ctypes import windll, c_int, c_string, byref
# load 'Ehllapi.dll' (from current dir), and function 'hllapi' from the DLL
Ehllap32 = windll.ehllapi
hllapi = Ehllap32.hllapi
# prepare the arguments with types and initial values
h_func = c_int(1)
h_text = c_string('A')
h_len = c_int(1)
h_ret = c_int(999)
# call the function
hllapi(byref(h_func), h_text, byref(h_len), byref(h_ret))
# print the resulting values of all arguments after the call
print h_func.value, h_text.value, h_len.value, h_ret.value

I needed the code in this recipe specifically to call a C function whose prototype is:

void FAR PASCAL hllapi(int FAR *, char FAR *, int FAR *, int FAR *);

from a DLL named Ehllapi.DLL (an implementation of the IBM 3270 HLLAPI for an Italian 3270 terminal emulator, as it happens). Thomas Heller’s ctypes extension, found at http://sourceforge.net/projects/ctypes, made the task very easy. In particular, ctypes makes mincemeat of problems related to representing function arguments that must belong to a certain C type and possibly get passed “by reference” (i.e., via a pointer).

In the past, I used another extension, known as calldll, which was (and still is) available from http://www.nightmare.com/software.html. While once very useful, calldll cannot rely on some of the modern techniques that ctypes uses internally, because these possibilities were introduced only in relatively recent versions of Python. calldll, using a single membuf Python type to represent all possible C types, tends to be much more cumbersome than ctypes when they are both used to perform the same tasks.

Judge for yourself: here is a working calldll version of the same script that I just showed how to code with ctypes:

import calldll, struct
# some helpful auxiliary functions
def myPrintLong(vVar):
    ''' print a long contained in a membuf '''
    print calldll.read_long(vVar.address( ))
def myPrintString(vVar):
    ''' print a string contained in a membuf '''
    a = calldll.read_string(vVar.address( ))
    print a, len(a)
def mySetLong(vMemBuf, vValueToSet):
    ''' set to an unsigned long the value of a membuf with len == 4 '''
    vMemBuf.write(struct.pack('L', vValueToSet))
def mySetString(vMemBuf, vValueToSet):
    ''' set to a string (with  terminator) the value of a membuf '''
    pack_format = "%ds" % 1+len(vValueToSet)              # +1 for the 
    string_packed = struct.pack(pack_format, vValueToSet) # pack( ) adds the 
    vMemBuf.write(string_packed)
# load 'Ehllapi.dll' (from current dir), and function 'hllapi' from the DLL
dll_handle = calldll.load_library ('.\Ehllapi')
function_address = calldll.get_proc_address (dll_handle, 'HLLAPI')
# allocate and init three membufs with the size to hold an unsigned long
Lsize = struct.calcsize('L')
vFunction = calldll.membuf(Lsize)
mySetLong(vFunction, 1)
vTextLen = calldll.membuf(Lsize)
vResult = calldll.membuf(Lsize)
mySetLong(vResult, 1)
# allocate a membuf as large as the DLL requires; in this case, space
# for 24 x 80 characters + 1 for a  terminator
vText = calldll.membuf(1921)
# init the text and text-length variables based on string of interest
string_value_to_write = 'A'
mySetString(vText, string_value_to_write)
mySetLong(vTextLen, len(string_value_to_write))
# call the function, print the results, and clean up
calldll.call_foreign_function(function_address, 'llll', 'l',
  (vFunction.address( ), vText.address( ), vTextLen.address( ), vResult.address( )))
myPrintLong(vResult)
myPrintString(vText)
calldll.free_library(dll_handle)

To be honest, I can’t quite be sure whether all of these gyrations are truly indispensable to making this calldll-based version work. Whenever I try to simplify this version a bit, something or other always breaks noisily, so I’ve stopped messing with it. One reason the ctypes-based version is cleaner and simpler is that ctypes has never given me trouble, so I’ve been encouraged to continue working on that version to improve it.

ctypes is at http://sourceforge.net/projects/ctypes; calldll is at http://www.nightmare.com/software.html.

You want to use SWIG-generated modules in a multithreaded environment; therefore, the C code in those modules must release the Python global interpreter lock (see the Introduction to Chapter 9 for more information about the global interpreter lock).

Use a typemap for SWIG, written by Mark Hammond, that was posted on comp.lang.python. It maps Win32 API functions that return BOOL to Python functions that return None and raise exceptions to diagnose errors. The wrapped function must set the standard Windows global LastError if it returns FALSE (indicating that it has detected an error). The wrapping function also automatically releases the Python global interpreter lock (GIL) for the duration of the wrapped function’s execution, to allow free multithreading.

%typedef BOOL BOOLAPI
%typemap(python,except) BOOLAPI {Py_BEGIN_ALLOW_THREADS
      $function
        Py_END_ALLOW_THREADS
        if (!$source)  {
              $cleanup
               return PyWin_SetAPIError("$name");
        }
}

To use multiple threads effectively, you must release the Python GIL from your C-coded extension whenever it’s safe to do so. The simplest way to do this with SWIG is to use an except directive, as shown in the recipe’s typemap. Within the typemap, you can then use the normal Python C API’s macros Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS (around the call to the wrapped function, indicated by the special SWIG directive $function) to release the GIL and acquire it again.

Another interesting effect of this simple typemap is that it turns the C-oriented error-return convention (returning FALSE and setting a global error indicator code) into a highly Pythonic convention (raising an exception).

SWIG and its typemaps are documented at http://www.swig.org; Windows API documentation on LastError is available from the Microsoft MSDN site at http://msdn.microsoft.com; Chapter 9 for general information on threads and particularly its Introduction for information on the GIL.

You have an existing C function that takes as an argument a C array of C-level values (e.g., doubles), and you want to wrap it into a Python-callable C extension that takes as an argument a Python sequence or iterator.

The easiest way to accept an arbitrary Python sequence (or any other iterable object) in the Python C API is with the PySequence_Fast function. It builds and returns a tuple when needed but returns only its argument (with the reference count incremented) when the argument is already a list or tuple:

#include <Python.h>
/* a preexisting C-level function you want to expose, e.g: */
static double total(double* data, int len)
{
    double total = 0.0;
    int i;
    for(i=0; i<len; ++i)
        total += data[i];
    return total;
}
/* here is how you expose it to Python code: */
static PyObject *totalDoubles(PyObject *self, PyObject *args)
{
    PyObject* seq;
    double *dbar;
    double result;
    int seqlen;
    int i;
    /* get one argument as a sequence */
    if(!PyArg_ParseTuple(args, "O", &seq))
        return 0;seq = PySequence_Fast(seq, "argument must be iterable");
    if(!seq)
        return 0;
    /* prepare data as an array of doubles */
    seqlen = PySequence_Fast_GET_SIZE(seq);
    dbar = malloc(seqlen*sizeof(double));
    if(!dbar) {
        Py_DECREF(seq);
        return PyErr_NoMemory( );
    }
    for(i=0; i < seqlen; i++) {
        PyObject *fitem;
        PyObject *item = PySequence_Fast_GET_ITEM(seq, i);
        if(!item) {
            Py_DECREF(seq);
            free(dbar);
            return 0;
        }
        fitem = PyNumber_Float(item);
        if(!fitem) {
            Py_DECREF(seq);
            free(dbar);
            PyErr_SetString(PyExc_TypeError, "all items must be numbers");
            return 0;
        }
        dbar[i] = PyFloat_AS_DOUBLE(fitem);
        Py_DECREF(fitem);
    }
    /* clean up, compute, and return result */
    Py_DECREF(seq);
    result = total(dbar, seqlen);
    free(dbar);
    return Py_BuildValue("d", result);
}
static PyMethodDef totalMethods[  ] = {
    {"total", totalDoubles, METH_VARARGS, "Sum a sequence of numbers."},
    {0} /* sentinel */
};
void
inittotal(void)
{
    (void) Py_InitModule("total", totalMethods);
}

The two best ways for your C-coded, Python-callable extension functions to accept generic Python sequences as arguments are PySequence_Fast and PyObject_GetIter. The latter, which I cover in the next recipe, can often save some memory, but it is appropriate only when it’s OK for the rest of your C code to get the items one at a time, without knowing beforehand how many items there will be in total. You often have preexisting C functions from an existing library that you want to expose to Python code, and such functions may require C arrays as their input arguments. Thus, this recipe shows how to build a C array (in this case, an array of double) from a generic Python sequence (or other iterable) argument, so that you can pass the array (and the integer that gives the array’s length) to your existing C function (represented here, purely as an example, by the total function at the start of the recipe). (In the real world, you would use Python’s built-in function sum for this specific functionality, rather than exposing any existing C function (but this is meant to be just an example!)

PySequence_Fast takes two arguments: a Python iterable object to be presented as a sequence, and a string to use as the error message in case the Python object cannot be presented as a sequence, in which case PySequence_Fast returns 0 (the C null pointer, NULL, an error indicator). If the Python object is already a list or tuple, PySequence_Fast returns the same object with the reference count increased by one. If the Python object is any other kind of sequence (or any iterator, or other iterable), PySequence_Fast builds and returns a new tuple with all items already in place. In any case, PySequence_Fast returns an object on which you can call PySequence_Fast_GET_SIZE to obtain the sequence length (as we do in the recipe, in order to malloc the appropriate amount of storage for the C array) and PySequence_Fast_GET_ITEM to get an item given a valid index (an int between 0, included, and the sequence length, excluded).

The recipe requires quite a bit of care (as is typical of all C-coded Python extensions, and more generally of any C code) to deal properly with memory issues and error conditions. For C-coded Python extensions, in particular, it’s imperative that you know which Python C API functions return new references (which you must Py_DECREF when you are done with them) and which ones return borrowed references (which you must not Py_DECREF when you’re done with them; on the contrary, you must Py_INCREF such a reference if you want to keep a copy for a longer time). In this specific case, you have to know the following (by reading the Python documentation):

There is method to this madness, even though, as you start your career as a coder of C API Python extensions, you’ll no doubt have to double-check each case carefully. Python’s C API strives to return borrowed references (for the sake of the modest performance increase that they afford, by avoiding needless incrementing and decrementing of reference counts), when it knows it can always do so safely (i.e., it knows that the reference it is returning necessarily refers to an already existing object). However, Python’s C API has to return a new reference when it’s possible (or certain) that a new object may have to be created.

For example, in the preceding list, PyNumber_Float and PySequence_Fast may be able to return the same object they were given as an argument, but it’s also quite possible that they may have to create a new object for this purpose, to ensure that the returned object has the correct type. Therefore, these two functions are specified as always returning new references. PyArg_ParseTuple and PySequence_Fast_GET_ITEM, on the other hand, always return references to objects that already exist elsewhere (as items in the arguments’ tuple, or as items in the fast-sequence container, respectively). Therefore, these two functions can afford to return borrowed references and are thus specified as doing so.

One last note: in this recipe, as soon as we obtain an item from the fast-sequence container, we immediately try to transform it into a Python float object, and thus we have to deal with the possibility that the transformation will fail (e.g., if we’re passed a sequence containing a string, a complex number, etc.). It is most often futile to first attempt a check (with PyNumber_Check) because the check might succeed, and the later transformation attempt might fail anyway (e.g., with a complex-number item). Therefore, it’s better to attempt the transformation and deal with the resulting error, if any. This approach is yet another case of the common situation in which it’s easier to get forgiveness than permission!

As usual, the best way to build this extension (assuming e.g., that you’ve saved the extension’s source code as a file named total.c) is with the distutils package. Place a file named setup.py in the same directory as the C source:

from distutils.core import setup, Extension
setup(name="total", maintainer="Luther Blissett", maintainer_email=
    "[email protected]", ext_modules=[Extension('total', sources=['total.c'])]
)

then build and install by running:

$ python setup.py install

An appealing aspect of this approach is that it works on any platform, assuming that you have access to the same C compiler used to build your version of Python, and permission to write on the site-packages directory where the resulting dynamically loaded library gets installed.

The Extending and Embedding manual is available as part of the standard Python documentation set at http://www.python.org/doc/current/ext/ext.html; documentation on the Python C API is at http://www.python.org/doc/current/api/api.html; the section “Distributing Python Modules” in the standard Python documentation set is still incomplete, but it’s a good source of information on the distutils package; Python in a Nutshell covers the essentials of extending and embedding, of the Python C API, and of the distutils package.

You want to write a Python-callable C extension that takes as an argument a Python sequence (or other iterable) and accesses it sequentially, one item at a time, requiring no extra storage.

If you can afford to access the sequence item-by-item, without knowing in advance the number of items it has, you can often save memory by using PyObject_GetIter instead of PySequence_Fast:

#include <Python.h>
static PyObject *totalIter(PyObject *self, PyObject *args)
{
    PyObject* seq;
    PyObject* item;
    double result;
    /* get one argument as an iterator */
    if(!PyArg_ParseTuple(args, "O", &seq))
        return 0;seq = PyObject_GetIter(seq);
    if(!seq)
        return 0;
    /* process data sequentially */
    result = 0.0;
    while((item=PyIter_Next(seq))) {
        PyObject *fitem;
        fitem = PyNumber_Float(item);
        if(!fitem) {
            Py_DECREF(seq);
            Py_DECREF(item);
            PyErr_SetString(PyExc_TypeError, "all items must be numbers");
            return 0;
        }
        result += PyFloat_AS_DOUBLE(fitem);
        Py_DECREF(fitem);
        Py_DECREF(item);
    }
    /* clean up and return result */
    Py_DECREF(seq);
    return Py_BuildValue("d", result);
}
static PyMethodDef totitMethods[  ] = {
    {"totit", totalIter, METH_VARARGS, "Sum a sequence of numbers."},
    {0} /* sentinel */
};
void
inittotit(void)
{
    (void) Py_InitModule("totit", totitMethods);
}

PyObject_GetIter is appropriate only when it’s OK for the rest of your C code to get the items one at a time, without knowing in advance the number of items in total. When this condition is met, PyObject_GetIter gives you roughly the same performance as PySequence_Fast (if the input argument is a list or tuple), but it can save memory allocation, and therefore can run faster, if the input argument is an iterator or another kind of sequence or iterable. In this recipe’s function, since we are just summing the items, it is indeed perfectly OK to get them one at a time, and we don’t need to know in advance the total number; therefore, using PyObject_GetIter is preferable. (In the real world, you would use Python’s built-in function sum for this specific functionality, rather than coding a dedicated C function, but then, this is meant to be just an example!)

PyObject_GetIter takes one argument: a Python object from which an iterator is desired (much like Python’s iter built-in function). It either returns 0, indicating an error, or an iterator object, on which you can repeatedly call PyIter_Next to get the next item (or 0, NULL, which does not indicate an error, but rather indicates the end of the iteration). Both PyObject_GetIter and PyIter_Next return new references, so we must use Py_DECREF when we’re done with the respective objects.

As usual, the best way to build this extension (assuming that you’ve saved it as a file named totit.c) is with the distutils package. Place in the same directory as the C source a file named setup.py such as:

from distutils.core import setup, Extension
setup(name="totit", maintainer="Luther Blissett", maintainer_email=
    "[email protected]", ext_modules=[Extension('totit', sources=['totit.c'])]
)

then build and install by running:

<m>$ python setup.py install</m>

Part of the appeal of this approach is that it works on any platform, assuming that you have access to the same C compiler used to build your version of Python, and permission to write on the site-packages directory where the resulting dynamically loaded library gets installed.

Since Python extensions are often coded in C to maximize performance, it’s interesting to measure performance compared to pure Python code dealing with the same task. A typical measurement setup might be a script such as the following timon.py:

import timeit, operator
from total import total
from totit import totit
def timo(fn, sq, init):
    T = timeit.Timer('timon.%s(%s)'%(fn,sq), 'import timon
'+init)
    print ' %5.5s: %5.2f' % (fn, T.timeit(40000))
def totpy(x):
    result = 0.0
    for item in x: result += item
    return result
def totre(x):
    return reduce(operator.add, x, 0.0)
def totsu(x):
    return sum(x, 0.0)
if _ _name_ _ == '_ _main_ _':
    print 'on lists:'
    for f in 'totre totpy total totit totsu'.split( ):
        timo(f, 'seq', 'seq=range(2000)')
    print 'on iters:'
    for f in 'totre totpy total totit totsu'.split( ):
        timo(f, 'g( )', 'def g( ):
  for x in range(2000): yield x')

This script uses the timeit module of the Python Standard Library to measure accurately 40,000 calls to each function on 2,000-item lists and 2,000-item generators. The timeit.Timer constructor takes two string arguments: first the statement we’re timing, then the setup statements that run before timing begins. Here, the statement we’re timing calls functions in this module; therefore, the setup statements must import this module—which is why we add the import timon at the beginning of the setup statement string. I have also taken care to make all these functions strictly comparable, by having them all sum floats (not just ints). This purpose is the reason that I provide the explicit 0.0 initial arguments to built-in functions reduce and sum.

On my machine, running with the command-line switch -O so that Python can optimize operations a little bit, the timing results on Python 2.3 are:

<m>$ python -O timon.py</m>on lists:
                totre: 136.04
                totpy: 118.18
                total: 56.61
                totit: 59.66
                totsu: 74.11
               on iters:
                totre: 220.86
                totpy: 198.98
                total: 199.72
                totit: 201.70
                totsu: 157.44

As you can see, the most important optimization is to avoid the “attractive nuisance” of the reduce built-in function: even a pure Python loop is faster! When we’re dealing with lists, the special-purpose C-coded extensions presented in the last two recipes are fastest; but when we’re dealing with generators, the fastest solution is provided by the built-in function sum. In practice, one would always use sum for this functionality, rather than bothering to code or expose special-purpose C functions.

The Extending and Embedding manual is available as part of the standard Python documentation set at http://www.python.org/doc/current/ext/ext.html; documentation on the Python C API is at http://www.python.org/doc/current/api/api.html; the section “Distributing Python Modules” in the standard Python documentation set is still incomplete but is a good source of information on the distutils package: Chapter 19 of this book covers iterators and generators in pure Python terms; Python in a Nutshell covers the essentials of extending and embedding, of the Python C API, of the distutils package, and of iterators; Python’s Library Reference covers the timeit module.

Your C-coded, Python-callable function in an extension module needs to return nothing in particular (i.e., a Python None), but it must, of course, do so without messing up reference counts.

Suppose we need an empty C-coded function, equivalent to Python:

def empty1(*args):
    pass

or, identically:

def empty2(*args):
    return None

Despite the simplicity of the task, there are right and wrong ways to perform it. The canonical solution is:

static PyObject*
empty3(PyObject* self, PyObject* args)
{
    Py_INCREF(Py_None);
    return Py_None;
}

and the simplest, but still correct way, is:

static PyObject*
empty4(PyObject* self, PyObject* args)
{return Py_BuildValue("");
}

A function written in C for Python often needs to return nothing in particular. In Python terms, it must return None. Don’t just code return Py_None; from C: that messes up reference counts! None—the Python object we must explicitly return from a Python-callable, C-coded function—is a normal Python object, subject to all normal reference count rules. One of these rules is that each function must Py_INCREF the Python object it returns.

A bare return Py_None; is a nasty lurking bug—a frequent beginner’s error that messes up reference counts:

static PyObject*
empty5(PyObject* self, PyObject* args)
{
    return Py_None;          /* ***WRONG*** */
}

Either explicitly Py_INCREF the None object you’re returning, or (a simpler approach, but one that costs a few machine cycles) delegate the work to the handy function Py_BuildValue, which can be used to handle just about all cases of returning values from C to Python, offering potential uniformity advantages. To have Py_BuildValue build a properly incref’d None on your behalf, call it with just one argument, an empty format string.

In Python 2.4, the C API has gained a new macro just for this purpose. If you’re coding a C extension that supports only Python 2.4, you can write Py_RETURN_NONE; instead of the return statement, and the macro takes care of everything for you.

The Extending and Embedding manual is available as part of the standard Python documentation set at http://www.python.org/doc/current/ext/ext.html; documentation on the Python C API is at http://www.python.org/doc/current/api/api.html.

A dynamically loaded C/C++ Python extension is giving you trouble on a Unix or Unix-like platform, and you want to use the interactive debugger gdb to determine what’s wrong.

One way to determine the cause of core dumps or other serious trouble with a C Python extension is to compile the extension source with -g and then follow these steps. (You may also want to recompile any other extensions you use, such as Numeric, with -g, if you hadn’t built them that way in the first place.)

<m>% gdb /usr/bin/python2.1</m>
(gdb) br _PyImport_LoadDynamicModule
(gdb) run    # start python
(gdb) cont   # repeat until your extension is loaded
(gdb) # you may need an import statement at python's >>> prompt
(gdb) finish # finish loading your extension module
(gdb) br wrap_myfunction  # break at the entry point in your code
(gdb) disable 1   # don't break for any more modules being loaded
(gdb) cont   # back to Python, run things normally from here

If a dynamically loaded C/C++ extension is causing Python to core dump, or causing some other kind of serious trouble, this recipe can help you determine the root cause, by demonstrating a technique for debugging your extension using gdb (if you use Unix or some Unix-like platform, and gdb is your debugger of choice). The overall concept generalizes to other debuggers with abilities similar to gdb’s.

The main point of this recipe is that you cannot set a break on your function at the start, because your function lives in a dynamic library (shared object) that isn’t initially loaded. However, you can break in the PyImport_LoadDynamicModule function, and eventually (when your module is finally being loaded) get control at the debugger prompt right after your module is in memory. You are then able, at last, to set the breakpoints you need.

This technique works. However, if you do this kind of thing often, the process of stepping through all the modules, as Python loads them at startup, can easily become tedious. A handier alternative, although more invasive, requires you to modify your Python sources and rebuild Python from them.

The key idea of this handier alternative is to add a do-nothing function somewhere in the body of code that Python loads immediately. Specifically, you can edit the Modules/main.c file, adding one new function:

void Py_DebugTrap(void) { }

In the extension you’re debugging, you can now add a call to Py_DebugTrap( ) right where you want to break into the code. The Py_DebugTrap( ) symbol is immediately available when you start gdb, because the symbol lives in main.c. So you can immediately set a breakpoint there, as soon as you are at the gdb prompt, then continue. This approach even works in parallel under MPI (message passing interface).

The gdb online documentation (just type help at the gdb interactive prompt), manual pages, and online manual (http://www.gnu.org/manual/gdb-4.17/gdb.html).

You’re developing C extensions, and you experience memory problems. You suspect mismanagement of reference counts and want to check whether your C extension code is correctly managing reference counts.

To chase these problems in an optimal way, you need to alter Python’s sources and rebuild Python. Specifically, add the following function in Objects/object.c, immediately before the _Py_PrintReferences function:

void
_Py_CountReferences(FILE *fp)
{
    int nr, no;
    PyObject *op;
    for (nr = no = 0, op = refchain._ob_next;
         op != &refchain;
         op = op->_ob_next, nr += op->ob_refcnt, no += 1)
    { }
    fprintf(fp, "%d refs (%d), %d objs
", nr, _Py_RefTotal, no);
}

I place the following macros in my C extensions:

#if defined(Py_DEBUG) || defined(DEBUG)
extern void _Py_CountReferences(FILE*);
#define CURIOUS(x) { fprintf(stderr, _ _FILE_ _ ":%d ", _ _LINE_ _); x; }
#else
#define CURIOUS(x)
#endif
#define MARKER( )        CURIOUS(fprintf(stderr, "
"))
#define DESCRIBE(x)     CURIOUS(fprintf(stderr, "  " #x "=%d
", x))
#define DESCRIBE_HEX(x) CURIOUS(fprintf(stderr, "  " #x "=%08x
", x))
#define COUNTREFS( )     CURIOUS(_Py_CountReferences(stderr))

To debug, I rebuild Python using make OPT="-DPy_DEBUG", which causes the code under Py_TRACE_REFS to be built. My own makefile for my extensions uses the same trick by including these lines:

debug:
        make clean; make OPT="-g -DPy_DEBUG" all
CFLAGS = $(OPT) -fpic -O2 -I/usr/local/include -I/usr/include/python2.3

When I’m developing C extensions and running into memory problems, I find that the typical cause is mismanagement of reference counts, particularly misuse of Py_INCREF and Py_DECREF, as well as forgetfulness of the reference-count effects of functions like Py_BuildValue, PyArg_ParseTuple, and PyTuple/List_SetItem/GetItem. The Python sources offer help with this problem (search for Py_TRACE_REFS), and function sys.getrefcounts in the Python Standard Library is also helpful. Nevertheless, it’s useful to add this recipe’s function in Objects/object.c just before _Py_PrintReferences.

Unlike _Py_PrintReferences, this recipe’s _Py_CountReferences function prints only the totals of all the refcounts and number of objects in the system, so it can be sensibly called, even in loops that repeat millions of times, while _Py_PrintReferences would print out way too much stuff to be useful. The information printed by _Py_CountReferences can help you identify errantly wandering Py_INCREFs and Py_DECREFs. _Py_CountReferences plays it safe by performing its own counts of objects references, which it prints side by side with the “official” count of references that Python itself maintains (when compiled for debugging) as global variable _Py_RefTotal. Should any discrepancy arise, you know something deeply wrong is going on.

When I suspect that one of my C-coded functions is responsible for memory problems, I liberally sprinkle the suspect function with calls to the COUNTREFS macro. Doing so allows me to keep track of exactly how many references are being created or destroyed as I go through my function. This information is particularly useful in tight loops, in which dumb mistakes can cause reference counts to grow ridiculously fast. Also, reference counts that shrink too fast (because of overzealous use of Py_DECREF) can cause core dumps because the memory for objects that should still exist has been reallocated for new objects.

The only documentation in this case is Python’s own source code. Use the source, Luke!