Python function invocation from C++

Hi there,

today ‘s mission is to load a Python module & call a module function from C++ by using Python’s C API. OK, that is cool, but why should we do that? Why not just rewrite our Python modules on C/C++ & perform the job in a native way?de

  • well because you are going to lose a significant amount of time in order to rewrite the module(s), test them & verify that everything is fine
  • to rewrite a small python module in C/C++ is fine, but what if you just want to use an existing functionality already implemented in Python? Let’s not reinvent the wheel!
  • you really want to just use the module & treat it as a black box. Implementation details about the module’s functionality may be way too out of subject. Let’s keep the requirements simple!

Let’s put some nice music (all tracks/artists that were listened & contributed to this production are provided in the Musix session) & start our experiment!

Setup

The code has been developed on Ubuntu Linux, and the dependencies are the following

  • C++ 11
  • Python 2.7
  • Python-dev :
    gclkaze@tzertzelos$: sudo apt-get install python-dev

Assumptions

Just to be clear & precise:

  • we are not taking into account multi-threading; consider a case where our program loads a module & attempts to call some Python functions in an multi-threaded manner, we do not guarantee that the parallel function calls will succeed.
  • for now, all test Python functions that accept an argument or arguments, all arguments will hold integer values.
  • didn’t check at all exception handling (a nice TODO)

Our test module

spam.py

def foo():
    print ‘Foo’
    return 6
def foo2(x,y):
    print ‘Foo2’
    return x*y

 

Steps

0. Our class, the PythonInterpreter class

The following snippet depicts the PythonInterpreter class, responsible for loading & calling a requested Python function from a given module.

#include <Python.h> // The Python (Interpreter) C API
#include <iostream> // for std::cout printing utility
#include <vector>
#include <string>
#include <utility> // std::pair
#include <map>

namespace Interpret{
    //Some enums for error checking
    enum InterpreterStatus{
        OK = 0,
        FUNCTION_NOT_FOUND = -1,
        MODULE_NOT_FOUND = -2,
        RETURNED_NULL = -3,
        NOT_CALLABLE_OBJECT = -4,
        UNKNOWN_ERROR = -5,
        ARGUMENT_ERROR = -6,
        UNKNOWN_INTEGER_ERROR = -7
    };

    class PythonInterpreter 
    {

    private:
        // The sys.path Python module need to be imported, and add the current
        // directory in the path in order to run the Python Interpreter 
   	std::vector<PyObject*> m_Globals; 
        PyObject * m_CurrentModule = nullptr; // The current loaded Python module
        // The dictionary object of the current loaded Python module, 
        //contains module namespace information, such as module functions,
        // module globals etc 
        PyObject * m_CurrentTable = nullptr; 
        // We need to create this from a string containing the 
        //module's name, in order to import the module later.
        PyObject * m_CurrentModuleName = nullptr; 
        bool m_Initialized = false;
        std::map<std::string, std::pair<PyObject*, PyObject*>> m_LoadedModules;
    public:
        PythonInterpreter(){}
    };
};

After we have instantiated our PythonInterpreter, we would like to be able first load our Python module, and if everything goes fine (that particular module exists for instance), we would like to specify a particular module function, load the module, pass the arguments (if it accepts any) and call the function.

1. Load Module

InterpreterStatus PythonInterpreter::loadModule(const std::string& module)
{
    if(!m_Initialized) {
        Py_Initialize();
        addCwdInPythonPath();
	m_Initialized = true;
        //std::cout << "PythonInterpreter initialized!" << std::endl;
    }
    //If the module isnt here yet, or the same module has being loaded earlier,
   // load it
    if(m_LoadedModules.find(module.c_str()) == m_LoadedModules.end() ) {
        //std::cout << "Module init" << std::endl;
        m_CurrentModuleName = PyString_FromString(module.c_str());

        // Load the module object
        m_CurrentModule = PyImport_Import(m_CurrentModuleName);

        if(!m_CurrentModule) return MODULE_NOT_FOUND;
        // m_CurrentTable is a borrowed reference 

        m_CurrentTable = PyModule_GetDict(m_CurrentModule);
        m_LoadedModules[module] = std::make_pair(m_CurrentModule,
                                              m_CurrentModuleName);
    }
    else {

    	m_CurrentModuleName = m_LoadedModules[module.c_str()].second;
    	m_CurrentModule = m_LoadedModules[module.c_str()].first; 
        m_CurrentTable = PyModule_GetDict(m_CurrentModule);

    }
    return OK;
}

1.1 Add current working directory in the Python path lazily

The first call to loadModule will call Py_Initialize in order to prepare the Python interpreter’s runtime in order to accept the subsequent Python C API calls and add the current working directory to the Python’s Path, by calling the class method addCwdInPythonPath that is depicted here.

void PythonInterpreter::addCwdInPythonPath()
{
    //The following is equivalent to:
    //from sys import path
    //path.append('.')
    PyObject *sys = PyImport_ImportModule("sys");
    PyObject *path = PyObject_GetAttrString(sys, "path");
    PyList_Append(path, PyUnicode_FromString("."));
    m_Globals.push_back(sys);
    m_Globals.push_back(path);
}

This step is of great importance, if it is omitted, the Python runtime won’t be able to know where to find our Python module (spam.py). If the module lives in a different directory, well, we will need to add it in our Python path by using a variation of the previous method.

 

1.2 Store loaded module pointers in the map

In the Python C API, every Python object (module, function, string, integer, float, list, dictionary) is treated as a PyObject, then based on type checking, we treat each of them differently (as a module, string, integer, float, dictionary & so on).

In our class, we are using a map of strings<=>PyObject* in order to know which modules have been already loaded in the past, allowing us to omit loading a module that has already being loaded in the past.

Thus, we first need to check if the requested module exists in our map, if not we should try to load it and then add it in our map. Otherwise, we load its PyObject* from our map.

2. Call Module Function

2.1 Pass the argument to the Python function

We will need to pass the integer arguments (if any) to the Python function. In order to do that, we will need :

  • first to create a Python tuple list with size equal to the amount of marshaling arguments by using PyTuple_New
  • iterate our string argument vector,
  • for each argument, extract its integer value and set it to our argument tuple list by calling PyTuple_SetItem

The PythonInterpreter::setFunctionArgs performs the previous operations & returns a filled PyObject* tuple list that will be used in our function invocation.

PyObject* PythonInterpreter::setFunctionArgs(const std::vector<std::string>& args)
{
    int argsN = (int)args.size();
    auto pArgs = PyTuple_New(argsN );
    PyObject *pValue = nullptr;

    for (auto i = 0; i < argsN ; i++) {
        pValue = PyInt_FromLong(atoi(args[i].c_str()));
        if (!pValue) {
            PyErr_Print();
            return nullptr;//UNKNOWN_INTEGER_ERROR;
        }
        PyTuple_SetItem(pArgs, i, pValue);  
    }
    return pArgs;
}

 

2.2 Call Module Function with arguments

The PythonInterpreter::callFunction takes as arguments the name of the function that we want to call and a list of string arguments that will be passed in our call.

InterpreterStatus PythonInterpreter::callFunction(const std::string& function, 
                               const std::vector<std::string>& args)
{

    PyObject* currentFunction = PyDict_GetItemString(m_CurrentTable, 
                                                    function.c_str());
    if(!currentFunction) return FUNCTION_NOT_FOUND;

    if (PyCallable_Check(currentFunction)) {
        PyObject *pValue = nullptr;
        // Prepare the argument list for the call
        int argsN = (int)args.size();
        if( argsN > 0 ) {
            auto pArgs = setFunctionArgs(args);
            if(!pArgs) return UNKNOWN_INTEGER_ERROR;   
            pValue = PyObject_CallObject(currentFunction, pArgs);
            if (pArgs != nullptr) {
                Py_DECREF(pArgs);
            }
        } 
        else {
            pValue = PyObject_CallObject(currentFunction, nullptr);
        }

        if (pValue != nullptr)  {
            printf("Return of call : %d\n", (int)PyInt_AsLong(pValue));
            Py_DECREF(pValue);
        }
        else {
            PyErr_Print();
            return ARGUMENT_ERROR;
        }
    } 
    else {
        //std::cout << "Cant call it" << std::endl;
        PyErr_Print();
        return NOT_CALLABLE_OBJECT;
    }
    return OK;
}

We first need to check if the requested function is included in our module’s table, which we saved in the previous step, while calling PythonInterpreter::loadModule.

If the function is not included in the module, ofc we won’t be able to call it. Thus, we return a nice & precise FUNCTION_NOT_FOUND.

Otherwise, we will check if there are any arguments & if any we will set up the tuple list holding our integer arguments (by calling PythonInterpreter::setFunctionArgs) & finally call PyObject_CallObject by passing the argument tuple list. If there are no arguments to be passed, we will just call PyObject_CallObject with a nullptr as tuple list.

After the call to the Python function, the variable pValue will hold a reference to the return results of the Python function. In our case, after calling “foo()“, pValue will store the return value “6”. After calling “foo2(2,4)”, pValue will store the return value “8”. Because pValue is a PyObject*, we will need to cast it to int by using PyInt_AsLong in order to retrieve the integer value, that is the result of our function call.

After executing the function call, pValue along with the tuple list that we created earlier (if there were arguments) will need to be  cleaned by calling Py_DECREF, marking them as not needed anymore, allowing the Python run-time to garbage collect them.

3. Cleaning stuff up

Before destroying our PythonInterpreter instance, we would like first to clean up our stored PyObjects such as the ones storing the “sys” module & the “path” modules (filled up after the call to pythonInterpreter::addCwdInPythonPath in step 1.1 ) & the module pointers stored in our module map.

void PythonInterpreter::cleanUp()
{
    // Clean up
    //std::cout << "Destroying " << m_Locals.size() << " locals" << std::endl;
    for(auto var : m_LoadedModules) {
        Py_DECREF((var).second.first);
        Py_DECREF((var).second.second);
    }

    m_LoadedModules.clear();
    //std::cout << "Destroying " << m_Globals.size() << " globals" << std::endl;
    for(auto &var : m_Globals) {
        Py_DECREF(var);
    }
}

PythonInterpreter::~PythonInterpreter()
{
    cleanUp();
    // Finish the Python Interpreter
    if(m_Initialized) {
        //std::cout << "PythonInterpreter destroyed!" << std::endl;
        Py_Finalize();
    }
}

After the previous step, we can stop the Python run-time safely, by calling Py_Finalize, which deallocates all the memory of the Python interpreter in the C API side effectively.

4. Call them!

4.1 demoMain.cpp

#include "demo.h" // Contains the version of PythonInterpreter, that was used for this demo
#include <assert.h> // Our favorite buddy

int main(int argc, char** argv)
{
    auto py = new Interpret::PythonInterpreter();

    //import spam
    auto res = py->loadModule("spam");
    assert (res == 0 );

    //spam.foo2(1,3)
    res = py->callFunction("foo2", {"1","3"});
    assert (res == 0);

    //spam.foo()
    res = py->callFunction("foo", {});

    assert (res == 0);
    return 1;
}

 

4.2 Compile them

gclkaze@tzertzelos:~/C++/CallPythonFromC$ make demo
g++ -std=c++11 -Wall -o demo -g -I/usr/include/python2.7 demoMain.cpp demo.h -lpython2.7

4.3 Run

gclkaze@tzertzelos:~/C++/CallPythonFromC$ ./demo
Foo2
Return of call : 3
Foo
Return of call : 6

You just called foo & foo2 functions of the Python module spam from C++ using the Python C API. Congratz!

I hope you enjoyed this tutorial, i am providing the demo’s code & the full version of my class in github (i am providing the link below). The canonical version of the class contains all showed code plus some extensions for verifying your data returned by the Python function call & not done functionality for handling the results in a threaded manner (many calls, many threads, many results).

Now go extend your C++ base with your existing Python utilities,

cheers

kazeone

Linx

Musix

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s