High-performance Python – compiled code and C interface

Step Lively

Python is one of the fastest growing languages for computing, the number one language for deep learning, and in the top three for machine learning. Literally thousands of Python add-on modules can be used for everything from plotting data to communicating with embedded hardware.

One of the common complaints about Python is that it is too slow, partly because it is interpreted and partly because of the Global Interpreter Lock (GIL), a mutex that prevents multiple threads from executing Python bytecodes at once. People started coming up with tools to improve the performance of Python, usually taking the form of compiling Python or interfacing compiled languages with Python.

In this article, I investigate compiling Python code with a just-in-time (JIT) compiler, a tool for compiling Python code into compiled C code that can be used as a module within Python, and a tool to compile existing C code into Python modules. The goal of all three tools is to make Python code faster. I use Python's Anaconda distribution, which has some of the more current tools, but it doesn't have everything, so some tools that aren't available in Anaconda won't be presented.

Compiling Python with Numba

Numba [1] is an open source JIT compiler that translates a subset of Python and NumPy [2] code into fast machine code at run time; hence, the "JIT" designation. Numba uses the LLVM [3] compiler library for ultimately compiling the code. You can also write CUDA kernels [4] with Numba. Numba has support for automatic parallelization of loops, generation of GPU-accelerated code (both Nvidia and AMD), and the creation of universal functions (ufuncs)

... comments powered by Disqus