I have a some Cython code where if a variable equals a value from a list then values from another list are copied into a testing array.
double [:] signals
cdef int total_days=signals.shape[0]
cdef size_t epoch=0
cdef int total_animals
cdef int n
cdef double[:] animal_signals
for animal in range(total_animals):
individual_animal = uniq_instr[animal]
for element in range(total_days):
if list(animal_ids[n]) == individual_animal:
animal_signals.append(signals[n])
I am getting an error:
UnboundLocalError: local variable 'animal_signals' referenced before assignment
I have thought having the line
cdef double[:] animal_signals
would have meant the array was assigned.
Update
As suggested I have also tried declaring the array animal_signals (and removing the append):
cdef int total_days=signals.shape[0]
cdef size_t epoch=0
cdef int total_animals
cdef int n
cdef int count=0
for animal in range(total_animals):
count=0
individual_animal = uniq_instr[animal]
for element in range(total_days):
if list(animal_ids[element]) == individual_animal:
cdef double[:] animal_signals[count] = signals[n]
count=count+1
however when I compile the code I get the error:
Error compiling Cython file:
------------------------------------------------------------
...
for element in range(total_days):
if list(animal_ids[element]) == individual_animal:
cdef double[:] animal_signals[count] = signals[n]
^
------------------------------------------------------------
project/temps.pyx:288:21: cdef statement not allowed here
Where am I going wrong?
Indeed, your line cdef double[:] animal_signals
declares animal_signals as a variable, but you never assign anything to it before using it (in Python assignement is done with the = operator).
In Cython, using the slice ([:]) notation when defining a variable is usually done to get the memory view of an other object (see the reference documentation).
For example :
some_1d_numpy_array = np.zeros((10,10)).reshape(-1)
cdef double[:] animal_signals = some_1d_numpy_array
If you want to create a C array, you have to allocate the memory for it (here for a size of number entries containing double) :
cdef double *my_array = <double *> malloc(number * sizeof(double))
Also, regarding to your original code, note that in both case you won't be able to use the append method on this object because it will not be a Python list, you will have to access its member by their indexes.
Related
The language docs how to define a ctuple of regular C types, but would it be possible to mix a Python object in a ctuple?
A PyObject * is a quite cumbersome thing in C: every time its value (i.e. the address of a Python-object) is copied the reference counter must be increased and every time a PyObject * gets a new value (or goes out of scope) the reference counter must be decreased.
Very similar to C++ std::shared_ptr, only without (copy)constructor, destructor or assignment-operator being supported by C.
For a variable of type object, Cython manages reference count - but this doesn't work with C-structs out of the box.
So one has to fall back to PyObject * in a ctuple- the main difference between PyObject * and object, is that Cython no longer manages reference counting and thus it can be used in a ctuple.
How it should be done depends on the usage of ctuple.
If we have a guarantee, that Python objects live longer than our ctuple, we don't have to care about in-/decreasing reference counter (i.e. weak references are enough), e.g.:
%%cython
from cpython cimport PyObject
cdef (PyObject *, PyObject *) create_weak(object a, object b):
return (<PyObject *>a, <PyObject *>b) # Cython no longer manages ref-counting
def use_weak(a, b):
cdef (PyObject *, PyObject *) p = create_weak(a,b)
return <object>p[0], <object>p[1] # casting to object => Cython manages ref-counting
However, if we must ensure that the objects live long enough, we must perform reference counting (and that can be quite error prone):
%%cython
from cpython cimport PyObject, Py_XINCREF, Py_XDECREF
cdef (PyObject *, PyObject *) create(object a, object b):
cdef PyObject *a_ptr = <PyObject *>a
cdef PyObject *b_ptr = <PyObject *>b
Py_XINCREF(a_ptr) # need to ensure that objects
Py_XINCREF(b_ptr) # stay alive as long as ctuple lives
return (a_ptr, b_ptr)
cdef void free((PyObject *, PyObject *) p):
Py_XDECREF(p[0]) # p will go out of scope soon
Py_XDECREF(p[1]) # no need to keep objects alive
def use(a, b):
cdef (PyObject *, PyObject *) p = create(a,b)
# as long as object of p alive use them:
res0 = <object>p[0]
res1 = <object>p[1]
# before p goes out of scope decrease ref count of objects
free(p)
# res0, res1 are still alive, because Cython ensured
# it when casting to <object>
return res0, res1
Another alternative would be to use C++ and to wrap PyObject * into a C++ which would handle the reference counting, here is a small prototype:
%%cython -+
from cpython cimport PyObject
cdef extern from *:
"""
#include <Python.h>
class PyObjectHolder{
public:
PyObject *ptr;
PyObjectHolder():ptr(nullptr){}
PyObjectHolder(PyObject *o):ptr(o){
Py_XINCREF(ptr);
}
//rule of 3
~PyObjectHolder(){
Py_XDECREF(ptr);
}
PyObjectHolder(const PyObjectHolder &h):
PyObjectHolder(h.ptr){}
PyObjectHolder& operator=(const PyObjectHolder &other){
Py_XDECREF(ptr);
ptr=other.ptr;
Py_XINCREF(ptr);
return *this;
}
};
"""
cdef cppclass PyObjectHolder:
PyObjectHolder(object o)
PyObject *ptr
cdef (PyObjectHolder, PyObjectHolder) create_cpp(object a, object b):
return (PyObjectHolder(a), PyObjectHolder(b))
def use_cpp(a, b):
cdef (PyObjectHolder, PyObjectHolder) p = create_cpp(a,b)
return <object>(p[0].ptr), <object>(p[1].ptr)
If using c++ is possible, then using a wrapper for PyObject seems to me the saner alternative.
I want to use a library that gives me a dynamic array. The dynamic array struct has a property void* _heap_ptr which gives the start of the array.
After having built the list, I want to access this pointer in cython (to make a copy of the array). But I cannot seem to get the pointer element from the struct.
Here is my pyx:
cimport src.clist as l
def main():
cdef l.ptr_list basic_list
cdef int i = 42
basic_list = l.create_list_size(sizeof(i), 100)
l.list_add_ptr(basic_list, &i)
cdef int* arr;
arr = basic_list._heap_ptr
for i in range(1):
print(arr[i])
This is the error message:
Error compiling Cython file:
------------------------------------------------------------
...
l.list_add_ptr(basic_list, &i)
cdef int* arr;
arr = basic_list._heap_ptr
^
------------------------------------------------------------
src/test.pyx:14:20: Cannot convert Python object to 'int *'
Error compiling Cython file:
------------------------------------------------------------
...
l.list_add_ptr(basic_list, &i)
cdef int* arr;
arr = basic_list._heap_ptr
^
------------------------------------------------------------
src/test.pyx:14:20: Storing unsafe C derivative of temporary Python reference
And my pxd:
cdef extern from "src/list.h":
ctypedef struct _list:
void* _heap_ptr
ctypedef struct ptr_list:
pass
ptr_list create_list_size(size_t size, int length)
list_destroy(ptr_list this_list)
void* list_at_ptr(ptr_list this_list, int index)
list_add_ptr(ptr_list this_list, void* value)
How can I fix my code? Why is this happening? From my investigations that error message pops up if you have forgotten to declare something as C (ie. use malloc not libc.stdlib.malloc, but I cannot see that anything similar is happening here.)
There are two issues in your code.
First: struct ptr_list has no members and thus no member _heap_ptr. It probably should have been
ctypedef struct ptr_list:
void* _heap_ptr
Cython's error message is not really helpful here, but as you said it pops up usually when a C-declaration is forgotten.
Second: you need to cast from void * to int * explicitly:
arr = <int*>basic_list._heap_ptr
Hi I am trying to convert a python code into cython in order to speed up its calculation. I am trying to return multiple arrays within the cython code from a cdef to cpdef. Based on classical C, I could either use a pointer or a tuple. I decide to use tuple because the size varies. I know the following code doesn't work, any help? Thank you!
import numpy as np
cimport numpy as np
cdef tuple funA(double[:] X, double[:] Y):
cdef int nX, nY, i
nX = len(X)
nY = len(Y)
for i in range(nX):
X[i] = X[i]*X[i]
for i in range(nY):
Y[i] = Y[i]*Y[i]
return X,Y
cpdef Run(double[:] X, double[:] Y)
cdef Tuple1, Tuple2 = funA(X,Y)
# Do some calculation with Tuple1 and Tuple2
# Example
cdef int i, nTuple1, nTuple2
nTuple1 = len(Tuple1)
for i in range(nTuple1):
Tuple1[i] = Tuple1[i]**2
nTuple2 = len(Tuple2)
for i in range(nTuple2):
Tuple2[i] = Tuple2[i]/2
return Tuple1, Tuple2
You've got a few indentation errors and missing colons. But your real issue is:
cdef Tuple1, Tuple2 = funA(X,Y)
Remove the cdef and it's fine. It doesn't look like cdef and tuple unpacking quite mix, and since you're treating them as Python objects it should be OK.
However, note that you don't really need to return anything from funA since you modify X and Y them in place there.
This seems like a question that should have an obvious answer, but for some reason I can't find any examples online.
I am wrapping a vector of C++ objects in a Python class using Cython. I also have a Cython wrapper for the C++ class already coded. I can get several methods such as __len__(), __getitem__(), and resize() to work properly, but the __setitem__() method is giving me problems.
For simplicity, I coded a small example using a vector of ints. I figure if I can get this code to work, then I can build on that to get the solution for my C++ class as well.
MyPyModule.pyx
# distutils: language = c++
from libcpp.vector cimport vector
from cython.operator cimport dereference as deref
cdef class MyArray:
cdef vector[int]* thisptr
def __cinit__(self):
self.thisptr = new vector[int]()
def __dealloc__(self):
del self.thisptr
def __len__(self):
return self.thisptr.size()
def __getitem__(self, size_t key):
return self.thisptr.at(key)
def resize(self, size_t newsize):
self.thisptr.resize(newsize)
def __setitem__(self, size_t key, int value):
# Attempt 1:
# self.thisptr.at(key) = value
# Attempt 2:
# cdef int* itemptr = &(self.thisptr.at(key))
# itemptr[0] = value
# Attempt 3:
# (self.thisptr)[key] = value
# Attempt 4:
self[key] = value
When I tried to cythonize using Attempt 1, I got the error Cannot assign to or delete this. When I tried Attempt 2, the .cpp file was created, but the compiler complained that:
error: cannot convert ‘__Pyx_FakeReference<int>*’ to ‘int*’ in assignment
__pyx_v_itemptr = (&__pyx_t_1);
On Attempt 3, Cython would not build the file because Cannot assign type 'int' to 'vector[int]'. (When I tried this style with the C++ object instead of int, it complained because I had a reference as a left-value.) Attempt 4 compiles, but when I try to use it, I get a segfault.
Cython docs say that returning a reference as a left-value is not supported, which is fine -- but how do I get around it so that I can assign a new value to one of my vector elements?
There are two ways to access the vector through a pointer,
def __setitem__(self, size_t key, int value):
deref(self.thisptr)[key] = value
# or
# self.thisptr[0][key] = value
Cython translates those two cases as follows:
Python: deref(self.thisptr)[key] = value
C++: ((*__pyx_v_self->thisptr)[__pyx_v_key]) = __pyx_v_value;
Python: self.thisptr[0][key] = value
C++: ((__pyx_v_self->thisptr[0])[__pyx_v_key]) = __pyx_v_value;
which are equivalent i.e. access the same vector object.
Instead of trying to handle a pointer from Cython code, you can let Cython itself do it for you:
cdef class MyArray:
cdef vector[int] thisptr
def __len__(self):
return self.thisptr.size()
def __getitem__(self, size_t key):
return self.thisptr[key]
def __setitem__(self, size_t key, int value):
self.thisptr[key] = value
def resize(self, size_t newsize):
self.thisptr.resize(newsize)
Is there any problem with this approach?
I have already accepted J.J. Hakala's answer (many thanks!). I tweaked that method to include an out-of-bounds check, since it uses the [] operator instead of the at() method:
cdef class MyArray:
(....)
def __setitem__(self, size_t key, int value):
if key < self.thisptr.size():
deref(self.thisptr)[key] = value
else:
raise IndexError("Index is out of range.")
cdef double testB(double[:] x) nogil:
return x[0]
def test():
cdef double xx[2]
with nogil:
testB(xx)
# compiler error: Operation not allowed without gil
If with gil, it works fine.
Is it because that when pass in an c array, it creates a memory view and such creation action actually requires gil? So the memory view is not completely a c object?
Update
%%cython --annotate
cimport cython
cdef double testA(double[:] x) nogil:
return x[0]
cpdef myf():
cdef double pd[8]
cdef double[:] x = pd
testA(x)
cdef double[:] x = pd is compiled to:
__pyx_t_3 = __pyx_format_from_typeinfo(&__Pyx_TypeInfo_double);
__pyx_t_2 = Py_BuildValue((char*) "(" __PYX_BUILD_PY_SSIZE_T ")", ((Py_ssize_t)8));
if (unlikely(!__pyx_t_3 || !__pyx_t_2 || !PyBytes_AsString(__pyx_t_3))) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_3);
__Pyx_GOTREF(__pyx_t_2);
__pyx_t_1 = __pyx_array_new(__pyx_t_2, sizeof(double), PyBytes_AS_STRING(__pyx_t_3), (char *) "fortran", (char *) __pyx_v_pd);
if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_1);
__Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
__Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
__pyx_t_4 = __Pyx_PyObject_to_MemoryviewSlice_ds_double(((PyObject *)__pyx_t_1));
if (unlikely(!__pyx_t_4.memview)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_DECREF(((PyObject *)__pyx_t_1)); __pyx_t_1 = 0;
__pyx_v_x = __pyx_t_4;
__pyx_t_4.memview = NULL;
__pyx_t_4.data = NULL;
There exists __Pyx_PyObject_to_MemoryviewSlice_ds_double. So it seems when binding a memory view it does require gil.
You should use a numpy array, as your cdef double[:] declaration gets wrapped by a Python object, and its use is restricted without gil. You can see it by trying to slice a double[:]
def test()
cdef double[:] asd
with nogil:
asd[:1]
Your output will be:
with nogil:
asd[:1]
^
------------------------------------------------------------
prueba.pyx:16:11: Slicing Python object not allowed without gil
Using a numpy array would compile; numpy uses Python buffer protocole, and is smoothly integrated with Cython (a Google Summercamp project was financed for this). So no wrapping conflict arises inside the def:
import numpy as np
cdef double testA(double[:] x) nogil:
return x[0]
cpdef test():
xx = np.zeros(2, dtype = 'double')
with nogil:
a = testB(xx)
print(a)
This will build your module with test() on it. But it crashes, and in an ugly way (at least with mi PC):
Process Python segmentation fault (core dumped)
If I may insist with my (now deleted) previous answer, in my own experience, when dealing with Cython memoryviews and C arrays, passing pointers works just like one would expect in C. And most wrapping is avoided (actually, you are writing the code passing exactly the directions you want, thus making unnecesary wrapping). This compiles and functions as expected:
cdef double testB(double* x) nogil:
return x[0]
def test():
cdef double asd[2]
asd[0] = 1
asd[1] = 2
with nogil:
a = testB(asd)
print(a)
And, after compilig:
In [5]: import prueba
In [6]: prueba.test()
1.0
Memoryviews are not, by themselves, Python objects, but they can be wrapped in one. I am not a proficient Cython programmer, so sometimes I get unexpected wrappings or code that remains at Python level when I supposed it would be at C. Trial and error got me to the pointer strategy.