How to define a tuple that has a Python object? - cython

Question

The language docs how to define a ctuple of regular C types, but would it be possible to mix a Python object in a ctuple?

Answer 1

A PyObject * is a quite cumbersome thing in C: every time its value (i.e. the address of a Python-object) is copied the reference counter must be increased and every time a PyObject * gets a new value (or goes out of scope) the reference counter must be decreased.
Very similar to C++ std::shared_ptr, only without (copy)constructor, destructor or assignment-operator being supported by C.
For a variable of type object, Cython manages reference count - but this doesn't work with C-structs out of the box.
So one has to fall back to PyObject * in a ctuple- the main difference between PyObject * and object, is that Cython no longer manages reference counting and thus it can be used in a ctuple.
How it should be done depends on the usage of ctuple.
If we have a guarantee, that Python objects live longer than our ctuple, we don't have to care about in-/decreasing reference counter (i.e. weak references are enough), e.g.:
%%cython
from cpython cimport PyObject
cdef (PyObject *, PyObject *) create_weak(object a, object b):
return (<PyObject *>a, <PyObject *>b) # Cython no longer manages ref-counting
def use_weak(a, b):
cdef (PyObject *, PyObject *) p = create_weak(a,b)
return <object>p[0], <object>p[1] # casting to object => Cython manages ref-counting
However, if we must ensure that the objects live long enough, we must perform reference counting (and that can be quite error prone):
%%cython
from cpython cimport PyObject, Py_XINCREF, Py_XDECREF
cdef (PyObject *, PyObject *) create(object a, object b):
cdef PyObject *a_ptr = <PyObject *>a
cdef PyObject *b_ptr = <PyObject *>b
Py_XINCREF(a_ptr) # need to ensure that objects
Py_XINCREF(b_ptr) # stay alive as long as ctuple lives
return (a_ptr, b_ptr)
cdef void free((PyObject *, PyObject *) p):
Py_XDECREF(p[0]) # p will go out of scope soon
Py_XDECREF(p[1]) # no need to keep objects alive
def use(a, b):
cdef (PyObject *, PyObject *) p = create(a,b)
# as long as object of p alive use them:
res0 = <object>p[0]
res1 = <object>p[1]
# before p goes out of scope decrease ref count of objects
free(p)
# res0, res1 are still alive, because Cython ensured
# it when casting to <object>
return res0, res1
Another alternative would be to use C++ and to wrap PyObject * into a C++ which would handle the reference counting, here is a small prototype:
%%cython -+
from cpython cimport PyObject
cdef extern from *:
"""
#include <Python.h>
class PyObjectHolder{
public:
PyObject *ptr;
PyObjectHolder():ptr(nullptr){}
PyObjectHolder(PyObject *o):ptr(o){
Py_XINCREF(ptr);
}
//rule of 3
~PyObjectHolder(){
Py_XDECREF(ptr);
}
PyObjectHolder(const PyObjectHolder &h):
PyObjectHolder(h.ptr){}
PyObjectHolder& operator=(const PyObjectHolder &other){
Py_XDECREF(ptr);
ptr=other.ptr;
Py_XINCREF(ptr);
return *this;
}
};
"""
cdef cppclass PyObjectHolder:
PyObjectHolder(object o)
PyObject *ptr
cdef (PyObjectHolder, PyObjectHolder) create_cpp(object a, object b):
return (PyObjectHolder(a), PyObjectHolder(b))
def use_cpp(a, b):
cdef (PyObjectHolder, PyObjectHolder) p = create_cpp(a,b)
return <object>(p[0].ptr), <object>(p[1].ptr)
If using c++ is possible, then using a wrapper for PyObject seems to me the saner alternative.