I am trying to use a pointer inside a cython class.
the outside_class ctypedef works like a charm but i am unable to get the inside_class to work. a "ctypedef statement not allowed here" error is thrown and i don't understand what is wrong.
Why should this work
the outside_class typdef works so i assumed it should also work inside. I was unable to get it to work so i tried to find some more information on it, unfortunately all information is about the outside_class example so i do not know whether the other is allowed or even possible. to me it seems the only difference is the self argument.
Why do i want this to work
This class is going to contain 35+ functions with the same arguments, when used only a part of those functions is called in a specific order. When initializing i want to create an array with all functions in the correct order. Of course a different way of doing so is also welcome.
updated code sample 14-02
test A & B work but C & D do not, error message is given below.
My code:
ctypedef int (*outside_class)()
ctypedef int (*inside_class)(Preprocess)
cdef int outside_foo():
return 12
cdef int outside_bar(Preprocess self):
return 20
cdef class Preprocess:
cdef int inside_foo(self):
return 18
cdef int inside_bar(self):
return 14
cdef int inside_sek(self):
return 16
def __init__(self):
cdef outside_class test_A
test_A = &outside_foo
print( test_A() )
cdef inside_class test_B
test_B = &outside_bar
print( test_B(self) )
cdef inside_class test_C
test_C = &self.inside_foo
#print( test_C(self) )
print( "no error, yet.." )
cdef inside_class test_D
test_D = &self.inside_foo
print( test_D(self) )
error
/home/boss/.pyxbld/temp.linux-x86_64-2.7/pyrex/aa/preprocessing/preprocessing.c: In function ‘__pyx_pf_7aa_13preprocessing_13preprocessing_10Preprocess___init__’:
/home/boss/.pyxbld/temp.linux-x86_64-2.7/pyrex/aa/preprocessing/preprocessing.c:938:18: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types]
__pyx_v_test_C = (&((struct __pyx_vtabstruct_7aa_13preprocessing_13preprocessing_Preprocess *)__pyx_v_se
^
/home/boss/.pyxbld/temp.linux-x86_64-2.7/pyrex/aa/preprocessing/preprocessing.c:955:18: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types]
__pyx_v_test_D = (&((struct __pyx_vtabstruct_7aa_13preprocessing_13preprocessing_Preprocess *)__pyx_v_se
^
12
20
no error, yet..
Segmentation fault (core dumped)
cython raises the error as soon as it sees the cdeftype within the class definition. It hasn't even looked at, or run, the &self.inside_foo assignment:
0000:~/mypy/cython3$ cython stack42214943.pyx -a
Error compiling Cython file:
------------------------------------------------------------
...
cdef int outside_foo():
return 12
cdef class Preprocess:
ctypedef int (*inside_class)(Preprocess)
^
------------------------------------------------------------
stack42214943.pyx:8:4: ctypedef statement not allowed here
If I try cdef int(*)(Preprocess) inside_test, I get a Syntax error in C variable declaration. Again before the self line.
(edit)
With the following code I can create and run both a python list of 3 functions and a C array of the same.
def __init__(self):
cdef outside_class test_A
test_A = &outside_foo
print( test_A() )
cdef inside_class test_B
test_B = &outside_bar
print( test_B(self) )
print(self.inside_foo())
cpdef evalc(self):
# cdef int (*inside_array[3]) (Preprocess)
cdef inside_class inside_array[3]
inside_array[0] = self.inside_foo
inside_array[1] = self.inside_bar
inside_array[2] = self.inside_sek
print('eval inside_array')
for fn in inside_array:
print(fn(self))
def evals(self):
alist = [self.inside_foo, self.inside_bar, self.inside_sek]
alist = [fn(self) for fn in alist]
print(alist)
self.evalc()
In an Ipython session I can compile and import this, and run it with:
In [3]: p=stack42214943.Preprocess()
12
20
18
In [4]: p.evals()
[18, 14, 16]
eval inside_array
18
14
16
In [5]: p.evalc()
eval inside_array
18
14
16
I haven't figured out how to define and access inside_array outside of the evalc function. But maybe I don't need to. And instead of printing, that function could return the 3 values as some sort of int array or list.
Related
I am trying to use Cython in Jupyterlab:
%load_ext Cython
%%cython -a
def add (int x, int y):
cdef int result
result = x + y
return result
add(3,67)
error:
File "<ipython-input-1-e496e2665826>", line 9
def add (int x, int y):
^
SyntaxError: invalid syntax
What am I missing?
Update:
I just measured cpdef vs def and the difference between score was quite a low one (45(cpdef) vs 52(def), smaller = better/faster), so for your function it might not matter if called just a few times, but having that chew through a large amount of data might do some real difference.
If that's not applicable for you, just call that %load_ext in a separate cell, keep def and that should be enough.
(Cython 0.29.24, GCC 9.3.0, x86_64)
Use cpdef to make it C-like function, but also to expose it to Python, so you can call it from Jupyter (because Jupyter is using Python, unless specified by the %%cython magic func). Also, check the Early Binding for Speed section.
cpdef add (int x, int y):
cdef int result
result = x + y
return result
Also make sure to check Using the Jupyter notebook which explains that the % needs to be in a separate cell as ead mentioned in the comments.
add has to be defined with cdef, not def.
cdef add (int x, int y):
cdef int result
result = x + y
return result
add(3,67)
I have a some Cython code where if a variable equals a value from a list then values from another list are copied into a testing array.
double [:] signals
cdef int total_days=signals.shape[0]
cdef size_t epoch=0
cdef int total_animals
cdef int n
cdef double[:] animal_signals
for animal in range(total_animals):
individual_animal = uniq_instr[animal]
for element in range(total_days):
if list(animal_ids[n]) == individual_animal:
animal_signals.append(signals[n])
I am getting an error:
UnboundLocalError: local variable 'animal_signals' referenced before assignment
I have thought having the line
cdef double[:] animal_signals
would have meant the array was assigned.
Update
As suggested I have also tried declaring the array animal_signals (and removing the append):
cdef int total_days=signals.shape[0]
cdef size_t epoch=0
cdef int total_animals
cdef int n
cdef int count=0
for animal in range(total_animals):
count=0
individual_animal = uniq_instr[animal]
for element in range(total_days):
if list(animal_ids[element]) == individual_animal:
cdef double[:] animal_signals[count] = signals[n]
count=count+1
however when I compile the code I get the error:
Error compiling Cython file:
------------------------------------------------------------
...
for element in range(total_days):
if list(animal_ids[element]) == individual_animal:
cdef double[:] animal_signals[count] = signals[n]
^
------------------------------------------------------------
project/temps.pyx:288:21: cdef statement not allowed here
Where am I going wrong?
Indeed, your line cdef double[:] animal_signals
declares animal_signals as a variable, but you never assign anything to it before using it (in Python assignement is done with the = operator).
In Cython, using the slice ([:]) notation when defining a variable is usually done to get the memory view of an other object (see the reference documentation).
For example :
some_1d_numpy_array = np.zeros((10,10)).reshape(-1)
cdef double[:] animal_signals = some_1d_numpy_array
If you want to create a C array, you have to allocate the memory for it (here for a size of number entries containing double) :
cdef double *my_array = <double *> malloc(number * sizeof(double))
Also, regarding to your original code, note that in both case you won't be able to use the append method on this object because it will not be a Python list, you will have to access its member by their indexes.
I'd like to call my cdef methods and improve the speed of my program as much as possible. I do not want to use cpdef (I explain why below). Ultimately, I'd like to access cdef methods (some of which return void) that are members of my Cython extensions.
I tried following this example, which gives me the impression that I can call a cdef function by making a Python (def) wrapper for it.
I can't reproduce these results, so I tried a different problem for myself (summing all the numbers from 0 to n).
Of course, I'm looking at the documentation, which says
The directive cpdef makes two versions of the method available; one fast for use from Cython and one slower for use from Python.
and later (emphasis mine),
This does slightly more than providing a python wrapper for a cdef method: unlike a cdef method, a cpdef method is fully overridable by methods and instance attributes in Python subclasses. It adds a little calling overhead compared to a cdef method.
So how does one use a cdef function without the extra calling overhead of a cpdef function?
With the code at the end of this question, I get the following results:
def/cdef:
273.04207632583245
def/cpdef:
304.4114626176919
cpdef/cdef:
0.8969507060538783
Somehow, cpdef is faster than cdef. For n < 100, I can occasionally get cpdef/cdef > 1, but it's rare. I think it has to do with wrapping the cdef function in a def function. This is what the example I link to does, but they claim better performance from using cdef than from using cpdef.
I'm pretty sure this is not how you wrap a cdef function while avoiding the additional overhead (the source of which is not clearly documented) of a cpdef.
And now, the code:
setup.py
from setuptools import setup, Extension
from Cython.Build import cythonize
pkg_name = "tmp"
compile_args=['-std=c++17']
cy_foo = Extension(
name=pkg_name + '.core.cy_foo',
sources=[
pkg_name + '/core/cy_foo.pyx',
],
language='c++',
extra_compile_args=compile_args,
)
setup(
name=pkg_name,
ext_modules=cythonize(cy_foo,
annotate=True,
build_dir='build'),
packages=[
pkg_name,
pkg_name + '.core',
],
)
foo.py
def foo_def(n):
sum = 0
for i in range(n):
sum += i
return sum
cy_foo.pyx
def foo_cdef(n):
return foo_cy(n)
cdef int foo_cy(int n):
cdef int sum = 0
cdef int i = 0
for i in range(n):
sum += i
return sum
cpdef int foo_cpdef(int n):
cdef int sum = 0
cdef int i = 0
for i in range(n):
sum += i
return sum
test.py
import timeit
from tmp.core.foo import foo_def
from tmp.core.cy_foo import foo_cdef
from tmp.core.cy_foo import foo_cpdef
n = 10000
# Python call
start_time = timeit.default_timer()
a = foo_def(n)
pyTime = timeit.default_timer() - start_time
# Call Python wrapper for C function
start_time = timeit.default_timer()
b = foo_cdef(n)
cTime = timeit.default_timer() - start_time
# Call cpdef function, which does more than wrap a cdef function (whatever that means)
start_time = timeit.default_timer()
c = foo_cpdef(n)
cpTime = timeit.default_timer() - start_time
print("def/cdef:")
print(pyTime/cTime)
print("def/cpdef:")
print(pyTime/cpTime)
print("cpdef/cdef:")
print(cpTime/cTime)
The reason for your seemingly anomalous result is that you aren't calling the cdef function foo_cy directly, but instead the def function foo_cdef wrapping it.
when you are wrapping inside another def indeed you are again calling the python function. However you should be able to reach similar results as the cpdef.
Here is what you could do:
like the python def, give the type for both input and output
def foo_cdef(int n):
cdef int val = 0
val = foo_cy(n)
return val
this should have similar results as cpdef, however again you are calling a python function. If you want to directly call the c function, you should use the ctypes and call from there.
and for the benchmarking, the way that you have written, it only considers one run and could fluctuate a lot due OS other task and as well the timer.
better to use the timeit builtin method to calculate for some iteration:
# Python call
pyTime = timeit.timeit('foo_def(n)',globals=globals(), number=10000)
# Call Python wrapper for C function
cTime = timeit.timeit('foo_cdef(n)',globals=globals(), number=10000)
# Call cpdef function, which does more than wrap a cdef function (whatever that means)
cpTime = timeit.timeit('foo_cpdef(n)',globals=globals(), number=10000)
output:
def/cdef:
154.0166154428522
def/cpdef:
154.22669848136132
cpdef/cdef:
0.9986378296327566
like this, you get consistent results and as well you see always close to 1 for both either cython itself wraps or we explicitly wrap around a python function.
This seems like a question that should have an obvious answer, but for some reason I can't find any examples online.
I am wrapping a vector of C++ objects in a Python class using Cython. I also have a Cython wrapper for the C++ class already coded. I can get several methods such as __len__(), __getitem__(), and resize() to work properly, but the __setitem__() method is giving me problems.
For simplicity, I coded a small example using a vector of ints. I figure if I can get this code to work, then I can build on that to get the solution for my C++ class as well.
MyPyModule.pyx
# distutils: language = c++
from libcpp.vector cimport vector
from cython.operator cimport dereference as deref
cdef class MyArray:
cdef vector[int]* thisptr
def __cinit__(self):
self.thisptr = new vector[int]()
def __dealloc__(self):
del self.thisptr
def __len__(self):
return self.thisptr.size()
def __getitem__(self, size_t key):
return self.thisptr.at(key)
def resize(self, size_t newsize):
self.thisptr.resize(newsize)
def __setitem__(self, size_t key, int value):
# Attempt 1:
# self.thisptr.at(key) = value
# Attempt 2:
# cdef int* itemptr = &(self.thisptr.at(key))
# itemptr[0] = value
# Attempt 3:
# (self.thisptr)[key] = value
# Attempt 4:
self[key] = value
When I tried to cythonize using Attempt 1, I got the error Cannot assign to or delete this. When I tried Attempt 2, the .cpp file was created, but the compiler complained that:
error: cannot convert ‘__Pyx_FakeReference<int>*’ to ‘int*’ in assignment
__pyx_v_itemptr = (&__pyx_t_1);
On Attempt 3, Cython would not build the file because Cannot assign type 'int' to 'vector[int]'. (When I tried this style with the C++ object instead of int, it complained because I had a reference as a left-value.) Attempt 4 compiles, but when I try to use it, I get a segfault.
Cython docs say that returning a reference as a left-value is not supported, which is fine -- but how do I get around it so that I can assign a new value to one of my vector elements?
There are two ways to access the vector through a pointer,
def __setitem__(self, size_t key, int value):
deref(self.thisptr)[key] = value
# or
# self.thisptr[0][key] = value
Cython translates those two cases as follows:
Python: deref(self.thisptr)[key] = value
C++: ((*__pyx_v_self->thisptr)[__pyx_v_key]) = __pyx_v_value;
Python: self.thisptr[0][key] = value
C++: ((__pyx_v_self->thisptr[0])[__pyx_v_key]) = __pyx_v_value;
which are equivalent i.e. access the same vector object.
Instead of trying to handle a pointer from Cython code, you can let Cython itself do it for you:
cdef class MyArray:
cdef vector[int] thisptr
def __len__(self):
return self.thisptr.size()
def __getitem__(self, size_t key):
return self.thisptr[key]
def __setitem__(self, size_t key, int value):
self.thisptr[key] = value
def resize(self, size_t newsize):
self.thisptr.resize(newsize)
Is there any problem with this approach?
I have already accepted J.J. Hakala's answer (many thanks!). I tweaked that method to include an out-of-bounds check, since it uses the [] operator instead of the at() method:
cdef class MyArray:
(....)
def __setitem__(self, size_t key, int value):
if key < self.thisptr.size():
deref(self.thisptr)[key] = value
else:
raise IndexError("Index is out of range.")
cdef double testB(double[:] x) nogil:
return x[0]
def test():
cdef double xx[2]
with nogil:
testB(xx)
# compiler error: Operation not allowed without gil
If with gil, it works fine.
Is it because that when pass in an c array, it creates a memory view and such creation action actually requires gil? So the memory view is not completely a c object?
Update
%%cython --annotate
cimport cython
cdef double testA(double[:] x) nogil:
return x[0]
cpdef myf():
cdef double pd[8]
cdef double[:] x = pd
testA(x)
cdef double[:] x = pd is compiled to:
__pyx_t_3 = __pyx_format_from_typeinfo(&__Pyx_TypeInfo_double);
__pyx_t_2 = Py_BuildValue((char*) "(" __PYX_BUILD_PY_SSIZE_T ")", ((Py_ssize_t)8));
if (unlikely(!__pyx_t_3 || !__pyx_t_2 || !PyBytes_AsString(__pyx_t_3))) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_3);
__Pyx_GOTREF(__pyx_t_2);
__pyx_t_1 = __pyx_array_new(__pyx_t_2, sizeof(double), PyBytes_AS_STRING(__pyx_t_3), (char *) "fortran", (char *) __pyx_v_pd);
if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_1);
__Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
__Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
__pyx_t_4 = __Pyx_PyObject_to_MemoryviewSlice_ds_double(((PyObject *)__pyx_t_1));
if (unlikely(!__pyx_t_4.memview)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_DECREF(((PyObject *)__pyx_t_1)); __pyx_t_1 = 0;
__pyx_v_x = __pyx_t_4;
__pyx_t_4.memview = NULL;
__pyx_t_4.data = NULL;
There exists __Pyx_PyObject_to_MemoryviewSlice_ds_double. So it seems when binding a memory view it does require gil.
You should use a numpy array, as your cdef double[:] declaration gets wrapped by a Python object, and its use is restricted without gil. You can see it by trying to slice a double[:]
def test()
cdef double[:] asd
with nogil:
asd[:1]
Your output will be:
with nogil:
asd[:1]
^
------------------------------------------------------------
prueba.pyx:16:11: Slicing Python object not allowed without gil
Using a numpy array would compile; numpy uses Python buffer protocole, and is smoothly integrated with Cython (a Google Summercamp project was financed for this). So no wrapping conflict arises inside the def:
import numpy as np
cdef double testA(double[:] x) nogil:
return x[0]
cpdef test():
xx = np.zeros(2, dtype = 'double')
with nogil:
a = testB(xx)
print(a)
This will build your module with test() on it. But it crashes, and in an ugly way (at least with mi PC):
Process Python segmentation fault (core dumped)
If I may insist with my (now deleted) previous answer, in my own experience, when dealing with Cython memoryviews and C arrays, passing pointers works just like one would expect in C. And most wrapping is avoided (actually, you are writing the code passing exactly the directions you want, thus making unnecesary wrapping). This compiles and functions as expected:
cdef double testB(double* x) nogil:
return x[0]
def test():
cdef double asd[2]
asd[0] = 1
asd[1] = 2
with nogil:
a = testB(asd)
print(a)
And, after compilig:
In [5]: import prueba
In [6]: prueba.test()
1.0
Memoryviews are not, by themselves, Python objects, but they can be wrapped in one. I am not a proficient Cython programmer, so sometimes I get unexpected wrappings or code that remains at Python level when I supposed it would be at C. Trial and error got me to the pointer strategy.