Creating the fastest chicken alive!
Wrapping cpp in cython and python
Python is my preferred language due to its ease of use, general application, interactivity, and availibility of libraries. There is however one big downside: python can be painfully slow for numerical applications. One possible solution is to write extensions in lower-level languages and provide bindings that one can use from python. This is often referred to as the two-language-problem as it requires a programmer to know two languages. In this post I will explore cython
which is “half” language that allows to write both native python and c/cpp in one file. In particular, I will focus on binding cpp classes to python to extend python libraries to python.
Cython
In a nutshell, cython allows a programmer to write python-like code that is then compiled into c/cpp. This can be used to write high performant extensions. The goal of this post is not to provide a novice tutorial; for those interested cython’s documentation provides some great material for that. This post will focus mainly on providing bindings of cpp to python.
CPP class
Imagine we want to write an extremely fast chicken, and we have decided python is not suitable for this. We turn to cpp and write something like
// file: chicken.hpp
#ifndef chicken_hpp
#define chicken_hpp
#include <iostream>
class Chicken {
public:
Chicken();
Chicken(std::string name);
std::string name;
void peck();
};
#endif
We start implementing straight away and end up with something like:
// file: chicken.cpp
#include "chicken.hpp"
Chicken::Chicken() { this->name = "HELP I HAVE NO NAME"; }
Chicken::Chicken(std::string name) { this->name = name; }
void Chicken::peck() { std::cout << "Peck peck!" << std::endl; }
Our chicken is alive, but it is stuck in cpp world. We have to get it running in python. How do we start doing this? In order to order to create a python chicken class, we have to provide a wrapper. In cython, we achieve this in two steps. First, we need to tell cython what cpp object we are looking at. Second, we need to create a cython extenions that provides bindings for python.
Cython header
Just like in cpp, cython has a header/implementation style system. The headers are kept in pxd
files and the implementation are in pyx
files. For our headers, we merely have to provide implementations on our cpp class methods. Cython offers three kinds; standard python def
implementations, cpdef
, and cdef
methods. The last two are special kinds; cpdef
methods allow for seamless interaction between “python-world” and “cpp-world”. In contrast, cdef
methods only allow for interaction with cpp objects. For this current tutorial, this distinction is not necessary, and we can go straight ahead with implementing our chicken.
We need to tell cython what our cpp class looks like. We create a header file cychicken.pxd
with the following content
#file cychicken.pxd
from libcpp.string cimport string
cdef extern from "chicken.hpp":
cdef cppclass Chicken:
Chicken() except+
Chicken(string name) except+
string name
void peck()
The first line imports the cython implementation of string (which is merely a wrapper for std::string
), cdef extern
is similar to extern
in c/cpp: it tells cython that there is an external header. We then “color in” the header and tell cython what objects life in this header. In our case, we have a chicken! And importantly, this chicken has a name
and it can peck
(be careful!).
Cython implementation
We now have implemented our class in cpp and defined headers for the class in cpp and cython. At this point, we could cimport
the Chicken
class and within cdef
and cpdef
classes can interact with this cpp class. However, we cannot yet import our chicken class into python directly. In order to do this, we have to provide an implementation (wrapper) for the python side to interact with.
#file: cychicken.pyx
from cychicken cimport Chicken, string
cdef class PyChicken:
# defines class property
# can be put into pxd files
cdef Chicken *cpp_chicken
# normal class init
def __init__(self, name: str):
# need to convert string to binary
# for cpp strings
self.cpp_chicken = new Chicken(f"{name}".encode('utf8'))
# ensures that point is deleted when object is
# destroyed
def __dealloc__(self):
del self.cpp_chicken
# wrap the peck function
def peck(self):
self.cpp_chicken.peck()
# wrap the name property
@property
def name(self):
return self.cpp_chicken.name.decode('utf8')
Our chicken can now freely be called from python!
Compiling
Before we can test our chicken, we have to write some code to compile it into a shared object. Setting up a proper extension is some black magic in and of itself. I won’t bore you with the details today and for now you can merely create a setup file as such
#file: setup.py
from setuptools import setup
from Cython.Build import cythonize
from setuptools.extension import Extension
exts = [
Extension(
"chicken",
sources=["cychicken.pyx", "chicken.cpp"],
include_dirs=["."],
language="c++",
)
]
setup(ext_modules=cythonize(exts))
Running python setup.py build_ext --inplace
will compile and create a shared object file inplace with the name chicken.cpython-39-x86_64-linux-gnu.so
. We can now finally test our chicken:
# file: test_chicken.py
from chicken import PyChicken
chicken = PyChicken("Carl")
# cdef Chicken chicken = Chicken(string("CARL"))
print(f"My name is {chicken.name}")
for i in range(10):
chicken.peck()
Which outputs:
My name is Carl
Peck peck!
Peck peck!
Peck peck!
Peck peck!
Peck peck!
Peck peck!
Peck peck!
Peck peck!
Peck peck!
Peck peck!
Carl is now the fastest chicken that can be called from python.
Outlook
I hope you find this little tutorial helpful. I hope to continue some explorations in the future. Cython can produce highly performant code. However, providing the bindings requires a lot of boiler-plate code. In the future I will look at another great alternative to provide python wrappers. In particular, I will focus on nimpy
and pybind11
. See you in the next post!