I'd like to initialize a C++ STL vector by using the
template<class InputIt>
std::vector(InputIt first, InputIt last);
constructor in Cython. Here's a simple example:
from libcpp.vector cimport vector
import numpy as np
cimport numpy as cnp
cnp.import_array()
def foo(double[::1] a):
cdef vector[double] vec = vector[double](&a[0], &a[0] + a.size)
# do something with vec..
However, this yields the error message
from libcpp.vector cimport vector
def foo(double[::1] a):
cdef vector[double] vec = vector[double](&a[0], &a[0] + a.size)
^
------------------------------------------------------------
.ipython/cython/_cython_magic_c522d88d9553084771dd501620335f85a93b04ae.pyx:5:52: Cannot convert 'double *' to Python object
Error compiling Cython file:
------------------------------------------------------------
...
from libcpp.vector cimport vector
def foo(double[::1] a):
cdef vector[double] vec = vector[double](&a[0], &a[0] + a.size)
^
------------------------------------------------------------
.ipython/cython/_cython_magic_c522d88d9553084771dd501620335f85a93b04ae.pyx:5:45: Cannot assign type 'double *' to 'size_type
Any idea what I am doing wrong?
As mentioned in the comments, your desired constructor is not available inside Cython. AFAIK, Cython doesn't support templated constructors yet. As a workaround, you could use .assign(), i.e.:
from libcpp.vector cimport vector
def foo(double[::1] a):
cdef vector[double] vec
vec.assign(&a[0], &a[0] + <size_t>a.size)
# do something with vec..
Related
I would like to split a pyx file which contains the declaration and definition of a class hierarchy into a pxd and pyx file for further use in another cython module.
Here is my pxd file (instrument.pxd):
cdef struct s_detector:
int size_x
int size_y
int offset
int board_id_max
ctypedef s_detector Detector
cdef class Instrument:
cdef int nbdet
cdef Detector* detectors
cdef class D11Instrument(Instrument):
pass
and here is my pyx file (instrument.pyx):
from libc.stdlib cimport free, malloc
from instrument cimport Detector, DetectorDescr, PixelInfo
cdef class Instrument:
def __dealloc__(self):
free(self.detectors)
cdef class D11Instrument(Instrument):
def __cinit__(self):
self.nbdet = 3
self.detectors = <Detector*>malloc(self.nbdet*sizeof(Detector))
self.detectors[0] = [256,192,0,6]
self.detectors[1] = [32,256,49152,7]
self.detectors[2] = [32,256,57344,8]
When compiling those files, I get the following kind of errors:
Error compiling Cython file:
------------------------------------------------------------
...
cdef class Instrument:
def __dealloc__(self):
free(self.detectors)
^
------------------------------------------------------------
extensions/instrument.pyx:11:17: Cannot convert Python object to 'void *'
Error compiling Cython file:
------------------------------------------------------------
...
self.detectors = <Detector*>malloc(self.nbdet*sizeof(Detector))
^
------------------------------------------------------------
extensions/instrument.pyx:48:25: Cannot convert 'Detector *' to Python object
Would you know what I am doing wrong with my pxd/pyx files ?
I have the following typedef
ctypedef fused sa_t:
np.int32_t
np.int64_t
In a function with a sa_t argument, I have the following declaration
cdef vector[pair[sa_t, sa_t]] count
I would like to make pair[sa_t, sa_t] more accessible.
However, I cannot do a ctypedef in the function and my attempts to do one outside don't work.
The following demo works fine(Cython version 0.29.13), does it give you some hints?
from libcpp.vector cimport vector
from libcpp.pair cimport pair
cimport numpy as np
ctypedef fused sa_t:
np.int32_t
np.int64_t
ctypedef pair[sa_t, sa_t] sa_pair_t
ctypedef vector[sa_pair_t] sa_pairs_t
cdef void func(sa_t value):
cdef sa_pairs_t sa_pairs
sa_pairs.push_back(pair[sa_t, sa_t](value, value))
print(sa_pairs)
func[np.int32_t](1) # print [(1, 1)]
I have a cython extension type that I want to make more general. One of the attributes of this extension type is a double and I want it to be a memoryview (double[::1]) when needed.
Here is a simple example :
import numpy as np
cimport numpy as np
cimport cython
cdef class Test:
cdef bint numeric
cdef double du
def __init__(self, bint numeric):
self.numeric = numeric
if self.numeric:
self.du = 1
else:
self.du = np.ones(10)
def disp(self)
print(self.du)
Test(True).disp() # returns 1
Test(False).disp() # gives of course an error
I tried to subclass Test changing du type to double[::1] and implementing a new __init__ but it seems that we can't override class attributes of extension types. Even if it worked, it wouldn't be satisfactory because I don't really want to have one extension type for each case.
The best would be that my extension type directly handle both cases (scalar du and memoryview du).
Is there a way to do this with Cython ?
Unfortunately, you cannot use fused_type as attributes type. You can have two options here:
You could try to use the memory adress of the variable you want to call, and cast it when needed (everything is explained here.) Unfortunately, I did not succeed at making it work with typed memory views.
Or you can use your defined attribute numeric to call the appropriate method:
import numpy as np
cimport numpy as np
cimport cython
cdef class Test:
cdef bint numeric
cdef double du_numeric
cdef double[:] du_mem_view
def __init__(self, bint numeric):
self.numeric = numeric
if self.numeric:
self.du_numeric = 1
else:
self.du_mem_view = np.ones(10)
def disp(self):
if self.numeric:
print(self.du_numeric)
else:
print(self.du_numeric_mem_view)
Test(True).disp() # returns 1
Test(False).disp() # Does not give an error anymore !
I have implement myself defined a chainer Link, but because it is too slow.
I have implemented cython CPU version of my code. But I want to further boost speed via GPU. So I test the following code , but it failed:
%%cython
import numpy as np
cimport numpy as np
import cupy as cp
cimport cupy as cp
cdef class A:
def __init__(self):
pass
cdef cp_test(self, cp.ndarray[cp.float_t, ndim=2] arr):
return cp.sum(arr)
a = A()
arr = cp.arange(100).reshape(20,50)
print(a.cp_test(arr))
reporting:
cdef cp_test(self, cp.ndarray[cp.float_t, ndim=2] arr):
^
------------------------------------------------------------
C:\Users\.ipython\cython\_cython_magic_d4940a274af88f0257c368b8a5d0e3f5.pyx:13:23: 'ndarray' is not a type identifier
Sorry, but CuPy does not provide cython interface currently (I am one of CuPy developers).
cdef double testB(double[:] x) nogil:
return x[0]
def test():
cdef double xx[2]
with nogil:
testB(xx)
# compiler error: Operation not allowed without gil
If with gil, it works fine.
Is it because that when pass in an c array, it creates a memory view and such creation action actually requires gil? So the memory view is not completely a c object?
Update
%%cython --annotate
cimport cython
cdef double testA(double[:] x) nogil:
return x[0]
cpdef myf():
cdef double pd[8]
cdef double[:] x = pd
testA(x)
cdef double[:] x = pd is compiled to:
__pyx_t_3 = __pyx_format_from_typeinfo(&__Pyx_TypeInfo_double);
__pyx_t_2 = Py_BuildValue((char*) "(" __PYX_BUILD_PY_SSIZE_T ")", ((Py_ssize_t)8));
if (unlikely(!__pyx_t_3 || !__pyx_t_2 || !PyBytes_AsString(__pyx_t_3))) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_3);
__Pyx_GOTREF(__pyx_t_2);
__pyx_t_1 = __pyx_array_new(__pyx_t_2, sizeof(double), PyBytes_AS_STRING(__pyx_t_3), (char *) "fortran", (char *) __pyx_v_pd);
if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_1);
__Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
__Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
__pyx_t_4 = __Pyx_PyObject_to_MemoryviewSlice_ds_double(((PyObject *)__pyx_t_1));
if (unlikely(!__pyx_t_4.memview)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_DECREF(((PyObject *)__pyx_t_1)); __pyx_t_1 = 0;
__pyx_v_x = __pyx_t_4;
__pyx_t_4.memview = NULL;
__pyx_t_4.data = NULL;
There exists __Pyx_PyObject_to_MemoryviewSlice_ds_double. So it seems when binding a memory view it does require gil.
You should use a numpy array, as your cdef double[:] declaration gets wrapped by a Python object, and its use is restricted without gil. You can see it by trying to slice a double[:]
def test()
cdef double[:] asd
with nogil:
asd[:1]
Your output will be:
with nogil:
asd[:1]
^
------------------------------------------------------------
prueba.pyx:16:11: Slicing Python object not allowed without gil
Using a numpy array would compile; numpy uses Python buffer protocole, and is smoothly integrated with Cython (a Google Summercamp project was financed for this). So no wrapping conflict arises inside the def:
import numpy as np
cdef double testA(double[:] x) nogil:
return x[0]
cpdef test():
xx = np.zeros(2, dtype = 'double')
with nogil:
a = testB(xx)
print(a)
This will build your module with test() on it. But it crashes, and in an ugly way (at least with mi PC):
Process Python segmentation fault (core dumped)
If I may insist with my (now deleted) previous answer, in my own experience, when dealing with Cython memoryviews and C arrays, passing pointers works just like one would expect in C. And most wrapping is avoided (actually, you are writing the code passing exactly the directions you want, thus making unnecesary wrapping). This compiles and functions as expected:
cdef double testB(double* x) nogil:
return x[0]
def test():
cdef double asd[2]
asd[0] = 1
asd[1] = 2
with nogil:
a = testB(asd)
print(a)
And, after compilig:
In [5]: import prueba
In [6]: prueba.test()
1.0
Memoryviews are not, by themselves, Python objects, but they can be wrapped in one. I am not a proficient Cython programmer, so sometimes I get unexpected wrappings or code that remains at Python level when I supposed it would be at C. Trial and error got me to the pointer strategy.