Ctypes

Cython takes Python code, converts it to C, and compiles it for you, but what if you have existing C code that you want to use in Python like a Python module? This is where ctypes [14] can help.

The ctypes foreign function library provides C-compatible data types and lets you call functions in dynamic link libraries (DLLs) or shared libraries from within Python. In essence, it "wraps" these libraries so they can be called from Python. You can find ctypes with virtually any Python distribution.

To use ctypes, you typically start with your C/C++ code and build the shareable object as usual. However, be sure to use the position-independent code (PIC) flag and the shared flag (you'll be building a library). For example, with gcc, you use the -fPIC and -shared options:

$ gcc -fPIC -shared -o libsource.so source.c

It is up to you to compile the C code and create a library using any method you like, as long as you use the -fPIC and -shared options.

Ctypes Example – sum

In the previous example, most of the work is done in the summation, so now I'll rewrite that routine in C to get better performance. According to an online tutorial [15], an example in C for computing the sum and building it into a library is shown in Listing 5.

01 int sum_function(int num_numbers, int *numbers) {
02    int i;
03    int sum = 0;
04    for (i = 0; i < num_numbers; i++) {
05       sum += numbers[i];
06    }
07    return sum;
08 }

The function is named sum_function and the code sum.c. This code can be compiled into a shared object (library) with gcc:

$ gcc -fPIC -shared -o libsum.so sum.c

The compiler creates the shared object libsum.so, a library.

To use the library in Python, a few specific ctypes functions and variables are needed within Python. Because it can make the Python code a bit complex, I write a "wrapper function" for the library in Python (Listing 6). Notice that the specific function sum_function is defined.

01 import ctypes
02
03 _sum = ctypes.CDLL('libsum.so')
04 _sum.sum_function.argtypes = (ctypes.c_int, ctypes.POINTER(ctypes.c_int))
05
06 def sum_function(numbers):
07    global _sum
08    num_numbers = len(numbers)
09    array_type = ctypes.c_int * num_numbers
10    result = _sum.sum_function(ctypes.c_int(num_numbers), array_type(*numbers))
11    return int(result)

If you have a library with more than one function, you will have to create the interface for each function in this file. Now to test the C function in Python (Listing 7). The eagle has landed! It works! However, you might notice that it is slower than even the pure Python code. Most likely the arithmetic intensity is not great enough to show an improvement. Do not let this deter you. It's always worth trying ctypes if you want or need more performance.

import sum
import numpy
x = numpy.arange(10000000)
%time sum.sum_function(x)
CPU times: user 2.15 s, sys: 68.4 ms, total: 2.22 s
Wall time: 2.22 s

Summary

Python is amazingly popular right now, with thousands of modules that can be used to extend its capability. However, in the high-performance world, Python is not known for being fast. A number of people have written tools and extensions to make Python faster.

In this article, I presented Numba, which compiles Python code with a JIT compiler invoked with a decorator. A simple summation example was found to be much faster than the original Python code.

Cython, also discussed briefly, is a tool for "translating" Python code into C code. A simple Python "makefile" translates and compiles the code for you. It's actually pretty simple to use and allows you to create a large Python module with lots of functions.

Finally, I presented ctypes, which uses an opposite approach by taking existing C code that is compiled into a library, coupled with some ctypes functions and variables, to create a module that can be used in Python. The ctypes library has a great deal of flexibility, so you can incorporate code written in C into Python.