In Cython, a class, or a extension type is a Python class, which means it can be initialized by Python. On the other hand, the parameters of its __init__ or __cinit__ have to be Python Object.
Is it possible to write a class in Cython, which can only be initilized by cdef functions, and thus can be initilized by C types and C++ objects?
I want to this because it would be easier to translate my existing Python codes to Cython code than C/C++ code.
You can quite easily create a class that can't (easily) be initialised from Python, but can only be created from a cdef factory function
cdef class ExampleNoPyInit:
cdef int value
def __init__(self):
raise RuntimeError("Cannot be initialise from python")
cdef ExampleNoPyInit_factory(int v):
cdef ExampleNoPyInit a
# bypass __init__
a = ExampleNoPyInit.__new__(ExampleNoPyInit)
a.value = v
return a
inst = ExampleNoPyInit_factory(5)
(I suspect the really committed could use the same method of initialising it in Python if they wanted. There are other ways to prevent initialisation if you want to be more thorough - for example you could use a cdef global variable in your Cython module as a flag, which would not be accessed from Python).
This class still has the Python reference counting mechanism built-in so is still a "Python class". If you want to avoid that then you could use a cdef struct, (although that can't have member functions).
Related
I have a C++ method declared as follow:
std::tuple<std::vector<int>, std::size_t, std::size_t> get_state(); // Should I use a struct and expose the struct instead?
And I would like to declare a cython extension to interface it in Python
cdef extern from "cpp_sources/CClass.hpp":
cdef cppclass CClass nogil:
tuple[vector[int], size_t, size_t] get_state()
Unfortunately, I don't see an easy import to make to have access to a C++ tuple.
I also checked here with no success.
My question is, is there an easy way to have access to a c++ tuple? Or maybe there is a better way to have access to some elements?
(We don't care about performances for this exact method)
Unfortunately this is not supported. More generally variadic templates are not supported - that's why you have pair for example, but not a generic tuple.
In the github issue I linked they have their own version of a workaround, which is what I would come up with first - for every amount of N arguments I will actually use,
template<typename T_1, ... typename T_N>
using tupleN = std::tuple<T_1, ..., TN>;
and exporting each tupleN individually. There is no magic I'm aware of to make a general variadic template here.
Effectively, I am cleaning up a cython module that has many globally scoped functions and variables. I thought cdef classes would be a great way to package some of these functions. But I ran into an issue when trying to call some of the methods of these classes. I boilded it down to these two examples that show my issue. The functionality of the code is unimportant, I am just trying to demonstrate the problem I am facing.
cdef class Foo:
def __init__(self):
pass
cdef int deref(self, int *bar):
return bar[0]
cdef int bar = 5
foo = Foo()
foo.deref(&bar)
If I run this code I get an error
Cannot convert 'int *' to Python object
But If I define everything in global scope it works just fine:
cdef int deref(int *bar):
return bar[0]
cdef int bar = 5
deref(&bar)
My question is: Is it possible to call a cdef method in this mannor?
My thought was that it would work since it is all being done within cython, but for some reason cython wants to convert the pointer into a python object and then back into a cython object? Is this always the case? I thought cdef classes were an effective tool to use when using cython.
Anyways, I have exhausted my attempts at solving this issue myself and wanted to ask here before abandoning cdef classes and going back to functional programming.
Normally Cython would detect the cdef type of a variable, e.g. for
def doit():
cdef int bar = 5
foo = Foo()
return foo.deref(&bar)
Cython would see, that foo is of type cdef Foo and treat it as such (the above code builds).
Not so for the global variables:
foo = Foo()
this is implicitly a def variable, which can be accessed (or set) via modulename.foo. That means, Cython cannot be sure, that foo is of type Foo - it could be set to int, float and what not.
Thus Cython must assume that foo is a pure Python-object with a (def-) method called deref which has Python arguments, and a pointer cannot be cast to a Python object automatically - which leads to the error you are observing.
Declaring foo as
cdef Foo foo=Foo()
hides the variable from pure-python, so it is no longer possible to access it via modulename.foo and to assign some arbitrary object to it. Thus it possible for Cython to assume, that foo is really of type Foo, thus granting access to cdef-functions of the class Foo.
foo must be declared a Foo object
cdef Foo foo = Foo()
Then all is well
I don't know python and trying to wrap an existing C library that provides 200 init functions for some objects and 200 destructors with help of PyCapsule. So my idea is to return a PyCapsule from init functions` wrappers and forget about destructors that shall be called automatically.
According to documentation PyCapsule_New() accepts:
typedef void (*PyCapsule_Destructor)(PyObject *);
while C-library has destructors in a form of:
int foo(void*);
I'm trying to generate a C function in .pyx file with help of cdef that would generate a C-function that will wrap library destructor, hide its return type and pass a pointer taken with PyCapsule_GetPointer to destructor. (pyx file is programmatically generated for 200 functions).
After a few experiments I end up with following .pyx file:
from cpython.ref cimport PyObject
from cpython.pycapsule cimport PyCapsule_New, PyCapsule_IsValid, PyCapsule_GetPointer
cdef void stateFree( PyObject *capsule ):
cdef:
void * _state
# some code with PyCapsule_GetPointer
def stateInit():
cdef:
void * _state
return PyCapsule_New(_state, "T", stateFree)
And when I'm trying to compile it with cython I'm getting:
Cannot assign type 'void (PyObject *)' to 'PyCapsule_Destructor'
using PyCapsule_New(_state, "T", &stateFree) doesn't help.
Any idea what is wrong?
UPD:
Ok, I think I found a solution. At least it compiles. Will see if it works. I'll bold the places I think I made a mistake:
from cpython.ref cimport PyObject
from cpython.pycapsule cimport PyCapsule_New, PyCapsule_IsValid, PyCapsule_GetPointer, PyCapsule_Destructor
cpdef void stateFree( object capsule ):
cdef:
void* _state
_state = PyCapsule_GetPointer(capsule, "T")
print('destroyed')
def stateInit():
cdef:
int _state = 1
print ("initialized")
return PyCapsule_New(_state, "T", < PyCapsule_Destructor >stateFree)
The issue is that Cython distinguishes between
object - a Python object that it knows about and handles the reference-counting for, and
PyObject*, which as far as it's concerned is a mystery type that it basically nothing about except that it's a pointer to a struct.
This is despite the fact that the C code generated for Cython's object ends up written in terms of PyObject*.
The signature used by the Cython cimport is ctypedef void (*PyCapsule_Destructor)(object o) (which isn't quite the same as the C definition. Therefore, define the destructor as
cdef void stateFree( object capsule ):
Practically in this case the distinction makes no difference. It matters more in cases where a function steals a reference or returns a borrowed reference. Here capsule has the same reference count on both the input and output of the function whether Cython manages it or not.
In terms of your edited-in solution:
cpdef is wrong for stateFree. Use cdef since it is not a function that should be exposed in a Python interface (and if you use cpdef it isn't obvious whether the Python or C version is passed as a function pointer).
You shouldn't need the cast to PyCapsule_Destructor and should avoid it because casts can easily hide bugs.
Can I just take a moment to express my general dislike for PyCapsule (it's occasionally useful for passing an opaque type through Python code without touching it, but for anything more I think it's usually better to wrap it properly in a cdef class). It's possible you've thought about it and it is the right tool for the job, but I'm putting this warning in to try to discourage people in the future who might be trying to use it on a more "copy-and-paste" basis.
I have a cython function:
def test(int a, int b):
return a+b
If I call it with:
test(0.5, 1)
I get the value 1.
Why doesn't it give a type error?
This is because float defines the special function __int__, which is called by Cython along the way (or more precise PyNumber_Long, at least this is my guess, because it is not easy to track the call through all these defines and ifdefs).
That is the deal: If your object defines __int__ so it can be used as an integer by Cython. Using Cython for implicit type-checking is not very robust.
If you want, you can check, whether the input is an int-object like in the following example (for Python3, for Python2 it is a little bit more complex, because there are different int-classes):
%%cython
from cpython cimport PyLong_Check
def print_me(i):
if not PyLong_Check(i):
print("Not an integer!")
else:
print(i)
I'm currently doing a Cython wrapper for an existing C++ library. I have an overloaded non-member operator in C++ like
Data operator+(Data const& a, Data const& b)
And in the pxd file describing the header, I wrote
cdef extern from 'blabla.h':
Data operator+(const Data&, const Data&)
Now how can I use this operator+ in another pyx file?
For very simple cases, like in your example you can lie to Cython and tell it that the operator is a member function:
cdef extern from 'blabla.h':
cdef cppclass Data:
# the rest of the data definitions
Data operator+(const Data&)
It only uses this information to know that it can translate the code a+b (where a and b are data objects) to __pyx_v_a + __pyx_v_b and let the c++ compiler do the rest (which it knows how to because of the import from "blabla.h"). Therefore, the distinction between member and non-member is irrelevant.
However: one of the main reasons to use nonmember operators is to allow things like
Data operator+(int a, const Data& b);
You can make this work, but it's very slightly messy. In your pxd file do
cdef extern from 'blabla.h':
Data operator+(int, const Data&) # i.e. do nothing special
In your pyx file
from my_pxd_file import * # works fine
## but this isn't accepted unfortunately:
from my_pxd_file import operator+
If you wanted to avoid too much namespace pollution from doing import *, you could probably create a pxd file that only contains operators and not the class definitions (I haven't tested this though)
In conclusion - two methods depending on how complicated your use-case is...