NumPy is a general-purpose library, designed to address most of the needs of a developer of scientific applications. However, as the code base and coverage of an application increase, so does the computation, and sometimes users demand more specific operations and optimized code segments. We have shown how NumPy and Python have tools, such as f2py and Cython, to address these demands. These tools may be an excellent choice for rewriting your functions to a natively compiled code in order to provide extra speed. But there may be some cases (leveraging a C library, such as NAG, to write some analytics) where you may want to do something more radical such as create a new data structure specifically for your own library. This would require you to have access to low-level controls in the Python interpreter. In this chapter, we will be looking into how to do this using the C-API provided by Python and its extension, the NumPy C-API. The C-API itself is a very vast topic and may require a book on its own to fully cover it. Here, we will provide a brief introduction and examples to get you started with the NumPy C-API.
Topics that will be covered in this chapter are:
- The Python C-API and the NumPy C-API
- The basic structure of an extension module
- An introduction to some NumP-specific C-API functions
- Creating functions using the C-API
- Creating a callable module
- Using a module through the Python interpreter and another module
The Python implementation that we are using is a C-based implementation of the Python interpreter. NumPy is specifically for this C-based Python implementation. This implementation of Python comes with a C-API, which is the backbone of the interpreter and provides low-level control to its user. NumPy has further augmented this by providing a rich C-API.
Writing functions in C/C++ can provide developers with the flexibility to leverage some of the advanced libraries available in these languages. However, the cost is apparent in terms of having to write too much boilerplate code around parsing input in order to construct return values. Additionally, developers have to take care while referencing/dereferencing objects since this could eventually create nasty bugs and memory leaks. There is also the problem of future compatibility of the code as the C-API keeps on evolving; hence, if a developer wants to migrate to a later version of Python, they may be up for a lot of maintenance work on these C-API-based extensions. Because of these difficulties, most developers choose to try other optimization techniques. such as Cython or F2PY, before exploring this path. However, there are cases where you would want to reuse some other existing libraries in C/C++, which could suit your specific purpose. In these cases, it might be a good idea to write wrappers over existing functions and expose your Python project.
We will next look at some example code and explain the key functions and macros as we move further along in this chapter. The code given here is compatible with the Python 2.X version and may not work for Python 3.X; however, the process of conversion should be similar.
Tip
Developers can try using a tool called cpychecker to check for common mistakes while reference counting in a module. Visit http://gcc-python-plugin.readthedocs.org/en/latest/cpychecker.html for more details.