Hello i've been trying to call a python user-defined callback from c++ using cython for a while. But it looks like it's impossible without changes on the c++ side or a static function buffer.
So, is there only one option for binding a propper callback (ctypes with CFUNCTYPE)?
Cython 0.29.23
A.hpp:
typedef void (*Callback) ();
class A{
Callback callback;
public:
A(){
this->callback = nullptr;
}
void set_callback(Callback callback){
this->callback = callback;
}
void call_callback(){
this->callback();
}
};
A.pxd:
cdef extern from "A.hpp":
ctypedef void (*Callback) ()
cdef cppclass A:
A() except +
void set_callback(Callback callback)
void call_callback()
B.pyx
from A cimport A, Callback
cdef class B:
cdef A *c_self
cdef object callback
def __cinit__(self):
self.c_self = new A()
def __dealloc__(self):
del self.c_self
cdef void callback_func(self) with gil:
print("I'm here")
self.callback()
def set_callback(self, callback):
self.callback = callback
self.c_self.set_callback(<Callback>self.callback_func)
def call_callback(self):
self.c_self.call_callback()
def print_():
print("hello")
b = B()
b.set_callback(print)
b.call_callback()
Output:
I'm here
[segmentation fault]
Looks like ctypes: get the actual address of a c function is a good one work-around, but it uses ctypes.
It scares me, but works:
B.pyx
from A cimport A, Callback
import ctypes
from libc.stdint cimport uintptr_t
cdef class B:
cdef A *c_self
cdef object callback
def __cinit__(self):
self.c_self = new A()
def __dealloc__(self):
del self.c_self
def set_callback(self, callback):
f = ctypes.CFUNCTYPE(None)(callback)
self.callback = f
cdef Callback c_callback = (<Callback*><uintptr_t>ctypes.addressof(f))[0]
self.c_self.set_callback(c_callback)
def call_callback(self):
self.c_self.call_callback()
def hello():
print("hello")
b = B()
b.set_callback(hello)
b.call_callback()
A function pointer does not have any space to store extra information. Therefore it is not possible to convert a Python callable to a function pointer in pure C. Similarly a cdef function of a cdef class must store the address of the instance to be usable and that is impossible too.
You have three options:
Use ctypes as in https://stackoverflow.com/a/34900829/4657412 (the bottom half of the answer shows how to do it). ctypes accomplishes this with runtime code generation (and thus only works on processors that it explicitly supports).
Use a std::function in C++ instead of a function pointer. These can store arbitrary information. You need to write an object wrapper (or re-use one from elsewhere) so it isn't completely pure Cython. See Pass a closure from Cython to C++. What you're trying to do is probably better covered by How to use a Cython cdef class member method in a native callback.
Use the class C scheme where the callback is of type
void (CallbackT)(/* any args */, void* user_data)
and is registered with:
void register_callback(CallbackT func, void* user_data)
In this case user_data would be the address of your B instances (and you'd need to make sure it was Py_INCREFed before setting the address and Py_DECREFed before unsetting the address). Cython callback with class method provides an example.
I have the following code which works:
%%cython
cdef int add(int a, int b):
return a+b
cdef int mult(int a, int b):
return a*b
ctypedef int (*function_type)(int a, int b)
cdef int lambda_c(function_type func, int c, int d):
return func(c,d)
print(lambda_c(add, 5, 6))
print(lambda_c(mult, 5, 6))
So I have lambda_c function that takes c function as an argument and I am not able to change it (as it it a wrapper over c library that is supported by another team).
What I want to do is to write a wrapper:
cdef class PyClass:
def py_wrap(func, e, f):
return lambda_c(func, e, f)
print(PyClass().py_wrap(add, 5, 5))
print(PyClass().py_wrap(mult, 6, 6))
But this throws an error:
Cannot convert Python object to 'function_type'
I also tried to cast func(return lambda_c(<function_type>func, e, f)) but got an error:
Python objects cannot be cast to pointers of primitive types
The idea behind this is following: any user will be able to write his own function in cython, compile it and then import and pass his function to PyClass().py_wrap method.
Is it even possible to import pure c function and pass it as a parameter with cython?
I also saw Pass cython functions via python interface but unlike the solution there I am not able to change lambda_c functon and turn it into a class. Moreover lambda_c takes only functions of certain type (function_type in our example)
If you want to pass an arbitrary Python callable as a C function pointer then it doesn't work - there's no way to do this in standard C (and thus it's impossible to for Cython to generate the code). There's a very hacky workaround involving ctypes (which can do runtime code generation which I'll find links to if needed. However I don't really recommend it.
If you're happy for your users to write cdef functions in Cython (the question implies you are) then you can just build on my answer to your question yesterday.
Write a suitable wrapper class (you just need to change the function pointer type) - this gets split between your .pxd and .pyx files that you write.
Have your users cimport it and then use it to expose their cdef classes to Python:
from your_module cimport FuncWrapper
cdef int add_c_implementation(int a, int b):
return a+b
# `add` accessible from Python
add = WrapperFunc.make_from_ptr(add_c_implementation)
Change PyClass to take FuncWrapper as an argument:
# this is in your_module.pyx
cdef class PyClass:
def py_wrap(FuncWrapper func, e, f):
return lambda_c(func.func, e, f)
Your users can then use their compiled functions from Python:
from your_module import PyClass
from users_module import add
PyClass().py_wrap(add,e,f)
Really all this is doing is using a small Python wrapper to allow you to pass around a type that Python normally can't deal with. You're pretty limited in what it's possible to do with these wrapped function pointers (for example they must be set up in Cython) but it does give a handle to select and pass them.
I am not sure if you are allowed to change the function pointer type from
ctypedef int (*function_type)(int a, int b)
to
ctypedef int (*function_type)(int a, int b, void *func_d)
but this is usually the way callback functions are implemented in C. void * parameter func_d to the function contains the user-provided data in any form. If the answer is yes, then you can have the following solution.
First, create the following definition file in Cython to reveal your C API to other Cython users:
# binary_op.pxd
ctypedef int (*func_t)(int a, int b, void *func_d) except? -1
cdef int func(int a, int b, void *func_d) except? -1
cdef class BinaryOp:
cpdef int eval(self, int a, int b) except? -1
cdef class Add(BinaryOp):
cpdef int eval(self, int a, int b) except? -1
cdef class Multiply(BinaryOp):
cpdef int eval(self, int a, int b) except? -1
This basically allows any Cython user to cimport these definitions directly into their Cython code and bypass any Python-related function calls. Then, you implement the module in the following pyx file:
# binary_op.pyx
cdef int func(int a, int b, void *func_d) except? -1:
return (<BinaryOp>func_d).eval(a, b)
cdef class BinaryOp:
cpdef int eval(self, int a, int b) except? -1:
raise NotImplementedError()
cdef class Add(BinaryOp):
cpdef int eval(self, int a, int b) except? -1:
return a + b
cdef class Multiply(BinaryOp):
cpdef int eval(self, int a, int b) except? -1:
return a * b
def call_me(BinaryOp oper not None, c, d):
return func(c, d, <void *>oper)
As you can see, BinaryOp serves as the base class which raises NotImplementedError for its users who do not implement eval properly. cpdef functions can be overridden by both Cython and Python users, and if they are called from Cython, efficient C calling mechanisms are involved. Otherwise, there is a small overhead when called from Python (well, of course, these functions work on scalars, and hence, the overhead might not be that small).
Then, a Python user might have the following application code:
# app_1.py
import pyximport
pyximport.install()
from binary_op import BinaryOp, Add, Multiply, call_me
print(call_me(Add(), 5, 6))
print(call_me(Multiply(), 5, 6))
class LinearOper(BinaryOp):
def __init__(self, p1, p2):
self.p1 = p1
self.p2 = p2
def eval(self, a, b):
return self.p1 * a + self.p2 * b
print(call_me(LinearOper(3, 4), 5, 6))
As you can see, they can not only create objects from efficient Cython (concrete) classes (i.e., Add and Multiply) but also implement their own classes based on BinaryOp (hopefully by providing the implementation to eval). When you run python app_1.py, you will see (after the compilations):
11
30
39
Then, your Cython users can implement their favorite functions as follows:
# sub.pyx
from binary_op cimport BinaryOp
cdef class Sub(BinaryOp):
cpdef int eval(self, int a, int b) except? -1:
return a - b
Well, of course, any application code that uses sub.pyx can use both libraries as follows:
import pyximport
pyximport.install()
from sub import Sub
from binary_op import call_me
print(call_me(Sub(), 5, 6))
When you run python app_2.py, you get the expected result: -1.
EDIT. By the way, provided that you are allowed to have the aforementioned function_type signature (i.e., the one that has a void * parameter as the third argument), you can in fact pass an arbitrary Python callable object as a C pointer. For this to happen, you need to have the following changes:
# binary_op.pyx
cdef int func(int a, int b, void *func_d) except? -1:
return (<object>func_d)(a, b)
def call_me(oper not None, c, d):
return func(c, d, <void *>oper)
Note, however, that Python now needs to figure out which object oper is. In the former solution, we were constraining oper to be a valid BinaryOp object. Note also that __call__ and similar special functions can only be declared def, which limits your use case. Nevertheless, with these last changes, we can have the following code run without any problems:
print(call_me(lambda x, y: x - y, 5, 6))
Thanks to #ead , I have changed code a bit and the result satisfies me:
cdef class PyClass:
cdef void py_wrap(self, function_type func, e, f):
print(lambda_c(func, e, f))
PyClass().py_wrap(mult, 5, 5)
For my purposes it is ok to have void function, but I do not know, how to make it all work with method, that should return some value. Any ideas for this case will be useful
UPD: cdef methods are not visible from python, so it looks like there is no way to make things work
I have a pxd file that declares some types with ctypedef, and some functions with cdef. To call one particular function, I have to duplicate the typedefs in my pyx file.
If I move the cdef of the function into the pyx file it fails to compile with "Cannot convert 'state_t *' to Python object". The error refers to this line of code
retval = c_explanation.table_of_states_is_bad(<state_t*>test_array, b)
The test_array parameter is created locally with malloc so is not a python object at all.
If remove the typedefs from the pyx file and import them and the cdef from the pxd file with a from .. import *, I get the same compile error.
The only way that I can get this simple bit of code to work is with the rather awkward duplication of typedefs in both pyx and pxd files.
Any thoughts on what is going on here and how to resolve it?
The full code is pasted below
pyx file
from libc.stdlib cimport malloc, free
cimport c_explanation
cdef extern from "explanation.h":
ctypedef int state_id;
ctypedef struct event_t:
pass
ctypedef void (*event_handler)(event_t event)
ctypedef struct state_t:
state_id id
event_handler handle
char *name
ctypedef enum bool:
pass
def table_of_states_is_bad(a, b):
cdef state_t *test_array
cdef bool retval
test_array = <state_t*>malloc(len(a) * sizeof(state_t))
if not test_array:
raise MemoryError()
try:
for i, s in enumerate(a):
test_array[i].id = s[0]
retval = c_explanation.table_of_states_is_bad(test_array, b)
return retval
finally:
free(test_array)
pxd file
cdef extern from "explanation.h":
cdef int mock_get_temp(int)
ctypedef enum bool:
pass
ctypedef struct event_t:
pass
ctypedef void (*event_handler)(event_t event);
ctypedef int state_id;
ctypedef struct state_t:
state_id id
event_handler handle
char *name
cdef bool table_of_states_is_bad(state_t table[], size_t size)
finally the h file
int mock_get_temp(int);
typedef int event_id;
/* an event */
typedef struct {
event_id id; /* numeric id */
void *data; /* points to anything */
char *name; /* points to string for debug or log */
} event_t;
typedef void (*event_handler)(event_t event);
typedef enum { false, true } bool;
typedef int state_id;
/* a state */
typedef struct {
state_id id; /* numeric id */
event_handler handle; /* points to handler */
char *name; /* points to string for debug or log */
} state_t;
bool table_of_states_is_bad(state_t table[], size_t size);
After some reading and experimentation I worked out what I had done wrong.
The problem is one of namespaces. When using the import * format, the local definition of table_of_states_is_bad() was replacing the imported one. Thus the function call was to the cython version of the function rather than to the c version and hence the error was correctly reported.
I solved the problem by renaming the local function to wrap_table_of_states_is_bad() hence resolving the name conflict.
when I declare a cdef function that returns a double I write cdef double method_name(...). If it does not return something and I just omit it to cdef method_name(...) then cython --annotate marks it yellow. How to declare that the method/function does not return anything?
cdef void method_name(...) crashes with a segmentation fault
cdef None method_name(...) -> 'None' is not a type identifier
--annotate marks it as yellow as cython assumes the return type to be a python object if you omit the return type annotation (Cython Language Basics).
Specifying void as return type works for me. It's also used in quite some of the official examples, Just make sure not to return anything.
For me (cython 0.21.1) defining the c function with void works:
# mymod.pyx file
cdef void mycfunc(int* a):
a[0] = 2
def myfunc():
cdef int a = 1
mycfunc(&a)
print(a)
The c function is not yellow in the annotated html file and
python -c 'from mymod import myfunc; myfunc()'
prints 2 as expected.
A bug in Cython version 0.22. Update to 0.23.4 solved it.
So, if you've got a header file.
%%file test.h
struct mystruct{
int i;
int j;
};
And then you wrap it in Cython:
cdef extern from "test.h" nogil:
struct mystruct:
int i
int j
And some function that returns back out to Py:
def spit_out_dict():
return mystruct(5,10)
Cython correctly automatically generates a dict wrapper. However, when I wrap the original C header in a namespace, I haven't been able to get get Cython to still generate the dict wrapper correctly, something along these lines:
%%file test2.h
namespace outerspace{
struct mystruct{
int i;
int j;
};
}
And Cython/Python:
cdef extern from "test2.h" namespace "outerspace" nogil:
struct mynewstruct:
int i
int j
def spit_out_dict():
return mynewstruct(5,10)
This won't compile -- lots of namespace complaint errors -- anyone experienced this before?
Your problem is that Cython seems to only expect namespaces to be used with cppclass. For structs, it generates some functions but just copies the full namespaced name in, causing errors:
static PyObject* __pyx_convert__to_py_outerspace::mystruct(struct outerspace::mystruct s);
^
py_bit.cpp: In function ‘PyObject* __pyx_pf_6py_bit_spit_out_dict(PyObject*)’:
py_bit.cpp:721:15: error: ‘__pyx_convert__to_py_outerspace’ has not been declared
where it's trying to create a function called __pyx_convert__to_py_<classname>. (I think this might be worth submitting a bug report for.)
The trick in such circumstances is usually to lie to Cython. I create three files:
// test2.hpp
namespace outerspace{
struct mystruct{
int i;
int j;
};
}
,
// test2_cy.hpp - a wrapper file purely for Cython's benefit
#include "test2.hpp"
using outerpsace::mystruct;
and the cython file
cdef extern from "test2_cy.hpp": # (I didn't test with "nogil", but it's probably fine...)
struct mynewstruct:
int i
int j
def spit_out_dict():
# for some reason using "return mystruct(5,10)" doesn't work, but this does...
cdef mystruct a = mystruct(5,10)
return a
This is a bug in Cython, fixed at https://github.com/cython/cython/commit/fa946e8435a4dcc3497fc7b0f4e87256d40844ba