Hello i've been trying to call a python user-defined callback from c++ using cython for a while. But it looks like it's impossible without changes on the c++ side or a static function buffer.
So, is there only one option for binding a propper callback (ctypes with CFUNCTYPE)?
Cython 0.29.23
A.hpp:
typedef void (*Callback) ();
class A{
Callback callback;
public:
A(){
this->callback = nullptr;
}
void set_callback(Callback callback){
this->callback = callback;
}
void call_callback(){
this->callback();
}
};
A.pxd:
cdef extern from "A.hpp":
ctypedef void (*Callback) ()
cdef cppclass A:
A() except +
void set_callback(Callback callback)
void call_callback()
B.pyx
from A cimport A, Callback
cdef class B:
cdef A *c_self
cdef object callback
def __cinit__(self):
self.c_self = new A()
def __dealloc__(self):
del self.c_self
cdef void callback_func(self) with gil:
print("I'm here")
self.callback()
def set_callback(self, callback):
self.callback = callback
self.c_self.set_callback(<Callback>self.callback_func)
def call_callback(self):
self.c_self.call_callback()
def print_():
print("hello")
b = B()
b.set_callback(print)
b.call_callback()
Output:
I'm here
[segmentation fault]
Looks like ctypes: get the actual address of a c function is a good one work-around, but it uses ctypes.
It scares me, but works:
B.pyx
from A cimport A, Callback
import ctypes
from libc.stdint cimport uintptr_t
cdef class B:
cdef A *c_self
cdef object callback
def __cinit__(self):
self.c_self = new A()
def __dealloc__(self):
del self.c_self
def set_callback(self, callback):
f = ctypes.CFUNCTYPE(None)(callback)
self.callback = f
cdef Callback c_callback = (<Callback*><uintptr_t>ctypes.addressof(f))[0]
self.c_self.set_callback(c_callback)
def call_callback(self):
self.c_self.call_callback()
def hello():
print("hello")
b = B()
b.set_callback(hello)
b.call_callback()
A function pointer does not have any space to store extra information. Therefore it is not possible to convert a Python callable to a function pointer in pure C. Similarly a cdef function of a cdef class must store the address of the instance to be usable and that is impossible too.
You have three options:
Use ctypes as in https://stackoverflow.com/a/34900829/4657412 (the bottom half of the answer shows how to do it). ctypes accomplishes this with runtime code generation (and thus only works on processors that it explicitly supports).
Use a std::function in C++ instead of a function pointer. These can store arbitrary information. You need to write an object wrapper (or re-use one from elsewhere) so it isn't completely pure Cython. See Pass a closure from Cython to C++. What you're trying to do is probably better covered by How to use a Cython cdef class member method in a native callback.
Use the class C scheme where the callback is of type
void (CallbackT)(/* any args */, void* user_data)
and is registered with:
void register_callback(CallbackT func, void* user_data)
In this case user_data would be the address of your B instances (and you'd need to make sure it was Py_INCREFed before setting the address and Py_DECREFed before unsetting the address). Cython callback with class method provides an example.
I have the following code which works:
%%cython
cdef int add(int a, int b):
return a+b
cdef int mult(int a, int b):
return a*b
ctypedef int (*function_type)(int a, int b)
cdef int lambda_c(function_type func, int c, int d):
return func(c,d)
print(lambda_c(add, 5, 6))
print(lambda_c(mult, 5, 6))
So I have lambda_c function that takes c function as an argument and I am not able to change it (as it it a wrapper over c library that is supported by another team).
What I want to do is to write a wrapper:
cdef class PyClass:
def py_wrap(func, e, f):
return lambda_c(func, e, f)
print(PyClass().py_wrap(add, 5, 5))
print(PyClass().py_wrap(mult, 6, 6))
But this throws an error:
Cannot convert Python object to 'function_type'
I also tried to cast func(return lambda_c(<function_type>func, e, f)) but got an error:
Python objects cannot be cast to pointers of primitive types
The idea behind this is following: any user will be able to write his own function in cython, compile it and then import and pass his function to PyClass().py_wrap method.
Is it even possible to import pure c function and pass it as a parameter with cython?
I also saw Pass cython functions via python interface but unlike the solution there I am not able to change lambda_c functon and turn it into a class. Moreover lambda_c takes only functions of certain type (function_type in our example)
If you want to pass an arbitrary Python callable as a C function pointer then it doesn't work - there's no way to do this in standard C (and thus it's impossible to for Cython to generate the code). There's a very hacky workaround involving ctypes (which can do runtime code generation which I'll find links to if needed. However I don't really recommend it.
If you're happy for your users to write cdef functions in Cython (the question implies you are) then you can just build on my answer to your question yesterday.
Write a suitable wrapper class (you just need to change the function pointer type) - this gets split between your .pxd and .pyx files that you write.
Have your users cimport it and then use it to expose their cdef classes to Python:
from your_module cimport FuncWrapper
cdef int add_c_implementation(int a, int b):
return a+b
# `add` accessible from Python
add = WrapperFunc.make_from_ptr(add_c_implementation)
Change PyClass to take FuncWrapper as an argument:
# this is in your_module.pyx
cdef class PyClass:
def py_wrap(FuncWrapper func, e, f):
return lambda_c(func.func, e, f)
Your users can then use their compiled functions from Python:
from your_module import PyClass
from users_module import add
PyClass().py_wrap(add,e,f)
Really all this is doing is using a small Python wrapper to allow you to pass around a type that Python normally can't deal with. You're pretty limited in what it's possible to do with these wrapped function pointers (for example they must be set up in Cython) but it does give a handle to select and pass them.
I am not sure if you are allowed to change the function pointer type from
ctypedef int (*function_type)(int a, int b)
to
ctypedef int (*function_type)(int a, int b, void *func_d)
but this is usually the way callback functions are implemented in C. void * parameter func_d to the function contains the user-provided data in any form. If the answer is yes, then you can have the following solution.
First, create the following definition file in Cython to reveal your C API to other Cython users:
# binary_op.pxd
ctypedef int (*func_t)(int a, int b, void *func_d) except? -1
cdef int func(int a, int b, void *func_d) except? -1
cdef class BinaryOp:
cpdef int eval(self, int a, int b) except? -1
cdef class Add(BinaryOp):
cpdef int eval(self, int a, int b) except? -1
cdef class Multiply(BinaryOp):
cpdef int eval(self, int a, int b) except? -1
This basically allows any Cython user to cimport these definitions directly into their Cython code and bypass any Python-related function calls. Then, you implement the module in the following pyx file:
# binary_op.pyx
cdef int func(int a, int b, void *func_d) except? -1:
return (<BinaryOp>func_d).eval(a, b)
cdef class BinaryOp:
cpdef int eval(self, int a, int b) except? -1:
raise NotImplementedError()
cdef class Add(BinaryOp):
cpdef int eval(self, int a, int b) except? -1:
return a + b
cdef class Multiply(BinaryOp):
cpdef int eval(self, int a, int b) except? -1:
return a * b
def call_me(BinaryOp oper not None, c, d):
return func(c, d, <void *>oper)
As you can see, BinaryOp serves as the base class which raises NotImplementedError for its users who do not implement eval properly. cpdef functions can be overridden by both Cython and Python users, and if they are called from Cython, efficient C calling mechanisms are involved. Otherwise, there is a small overhead when called from Python (well, of course, these functions work on scalars, and hence, the overhead might not be that small).
Then, a Python user might have the following application code:
# app_1.py
import pyximport
pyximport.install()
from binary_op import BinaryOp, Add, Multiply, call_me
print(call_me(Add(), 5, 6))
print(call_me(Multiply(), 5, 6))
class LinearOper(BinaryOp):
def __init__(self, p1, p2):
self.p1 = p1
self.p2 = p2
def eval(self, a, b):
return self.p1 * a + self.p2 * b
print(call_me(LinearOper(3, 4), 5, 6))
As you can see, they can not only create objects from efficient Cython (concrete) classes (i.e., Add and Multiply) but also implement their own classes based on BinaryOp (hopefully by providing the implementation to eval). When you run python app_1.py, you will see (after the compilations):
11
30
39
Then, your Cython users can implement their favorite functions as follows:
# sub.pyx
from binary_op cimport BinaryOp
cdef class Sub(BinaryOp):
cpdef int eval(self, int a, int b) except? -1:
return a - b
Well, of course, any application code that uses sub.pyx can use both libraries as follows:
import pyximport
pyximport.install()
from sub import Sub
from binary_op import call_me
print(call_me(Sub(), 5, 6))
When you run python app_2.py, you get the expected result: -1.
EDIT. By the way, provided that you are allowed to have the aforementioned function_type signature (i.e., the one that has a void * parameter as the third argument), you can in fact pass an arbitrary Python callable object as a C pointer. For this to happen, you need to have the following changes:
# binary_op.pyx
cdef int func(int a, int b, void *func_d) except? -1:
return (<object>func_d)(a, b)
def call_me(oper not None, c, d):
return func(c, d, <void *>oper)
Note, however, that Python now needs to figure out which object oper is. In the former solution, we were constraining oper to be a valid BinaryOp object. Note also that __call__ and similar special functions can only be declared def, which limits your use case. Nevertheless, with these last changes, we can have the following code run without any problems:
print(call_me(lambda x, y: x - y, 5, 6))
Thanks to #ead , I have changed code a bit and the result satisfies me:
cdef class PyClass:
cdef void py_wrap(self, function_type func, e, f):
print(lambda_c(func, e, f))
PyClass().py_wrap(mult, 5, 5)
For my purposes it is ok to have void function, but I do not know, how to make it all work with method, that should return some value. Any ideas for this case will be useful
UPD: cdef methods are not visible from python, so it looks like there is no way to make things work
How can I tell Cython that my C++ class has overloaded operator=? I tried:
cdef extern from "my_source.H":
cdef cppclass MyStatus:
void operator=(const char* status)
cdef public void setStatus(MyStatus& status):
status = "FOO"
but Cython either complains "Assignment to reference status" or (if I make status a non-reference) constructs a python object out of the string "FOO" and then tries to assign the python object to status.
The problem in your code is, that for Cython "FOO" is a Python-object. For expressions like
char *s = "FOO"
Cython is clever enough to understand, what you want and automatically interprets "FOO" as char *.
However, Cython doesn't really "understand"/interpret the signatures of wrapped c++-functions (for that it must be a c++-compiler) and thus cannot know, that you want "FOO" be a char *.
Thus you have to help Cython, for example:
status = <const char *>"FOO"
You also have to work around the problem with reference, for example via:
cdef public void setStatus(MyStatus *status):
status[0] = <const char *>"FOO"
or if you want to have keep the signature of the function intact:
cdef public void setStatus(MyStatus& status):
cdef MyStatus * as_ptr = &status
as_ptr[0] = <const char *>"FOO"
I'm not completely sure the problem with the assigment to reference isn't a bug.
Another observation: the assigment operators aren't part of the "official" wrap of the standard containers, see here or here.
I am a beginner and I am sure this question is too simple. I am trying to test memory views in cython to get to know them much better.In my code I pass each memory view element (like [1,2]) as the cy class element move.
cdef class cy:
cdef public long[:] move
def __init__(self, move):
self.move = move
lst = []
for i in range(100):
lst.append([i, i+1])
cdef long[:, :] memview = np.asarray(lst)
b0 = cy(memview[0])
print(b0.move)
When I print the results. I get this:
<MemoryView of 'ndarray' object> # I expect for sth like [12, 13]
I need cy class prints out a list. How can I fix it?
there is another problem which occurs to me when I use this code:
cdef class parent:
cdef public:
list children
list moves
def __init__(self):
self.children = []
def add_children(self, moves):
cdef int i = 0
cdef int N = len(moves)
for i in range(N):
self.children.append(cy(moves[i]))
cdef int[:, :] moves = np.asarray(lst, dtype=np.int32)
obj = parent()
for move in moves:
obj.add_children(move)
After running this code I always get this error:
TypeError: a bytes-like object is required, not 'int'.
What causes this error and how can I fix this one?
Your first issue is just that a memoryview doesn't have a useful __str__ function for print to use. You can either convert it to an object that does print nicely
print(list(b0.moves))
print(np.asarray(b0.moves))
Or you can iterate through it yourself:
for i in range(b0.moves.shape[0]):
print(b0.moves[i], end=' ') # need to have Cython set to use Python 3 syntax for this line
print()
Your second problem is harder to solve since you don't tell us what line the error comes from. I think it's the constructor of cy which expects a memoryview but you pass an integer to. (I get a slightly different error message though).
when I declare a cdef function that returns a double I write cdef double method_name(...). If it does not return something and I just omit it to cdef method_name(...) then cython --annotate marks it yellow. How to declare that the method/function does not return anything?
cdef void method_name(...) crashes with a segmentation fault
cdef None method_name(...) -> 'None' is not a type identifier
--annotate marks it as yellow as cython assumes the return type to be a python object if you omit the return type annotation (Cython Language Basics).
Specifying void as return type works for me. It's also used in quite some of the official examples, Just make sure not to return anything.
For me (cython 0.21.1) defining the c function with void works:
# mymod.pyx file
cdef void mycfunc(int* a):
a[0] = 2
def myfunc():
cdef int a = 1
mycfunc(&a)
print(a)
The c function is not yellow in the annotated html file and
python -c 'from mymod import myfunc; myfunc()'
prints 2 as expected.
A bug in Cython version 0.22. Update to 0.23.4 solved it.