Multiprocessing in Python with Fortran and OpenMP

Den of Snakes

Python is arguably one of the most popular languages in use today and is being used more and more in HPC. However, Python code by itself is limited to a single thread because of the almighty global interpreter lock (GIL) [1], "a mutex that protects access to Python objects, preventing multiple threads from executing Python bytecodes at once." Given servers with 32 cores per socket, it seems somewhat wasteful to use just one core when programming in Python.

A number of tools and modules allow you to write multiprocessing or multithreaded code, including the multiprocessing [2] module that comes in the standard library, Parallel Python [3], variations on queuing systems such as 0MQ [4] (zeromq ), and the mpi4py [5] bindings of the Message Passing Interface (MPI) standard for writing MPI code in Python.

Another cool aspect of Python is that it's possible to write extensions in other languages that can be loaded as modules using "wrapper" tools. For example, you can write extensions in Fortran and C as Python modules with tools such as SWIG [6], Pyfort [7], and F2PY [8]. Writing parallel functions in Python is very difficult, but it's fairly straightforward in C and Fortran with the use of a variety of abstractions, including OpenMP, which provides a path for Python functions to utilize all of the available cores.

In this article, I take a look at what's possible by writing Python modules in Fortran that use all of the cores on a node or a subset of nodes (user controllable). To achieve this, I use

... comments powered by Disqus