Polymorphism with cython extesion types - cython

I have a cython extension type that I want to make more general. One of the attributes of this extension type is a double and I want it to be a memoryview (double[::1]) when needed.
Here is a simple example :
import numpy as np
cimport numpy as np
cimport cython
cdef class Test:
cdef bint numeric
cdef double du
def __init__(self, bint numeric):
self.numeric = numeric
if self.numeric:
self.du = 1
else:
self.du = np.ones(10)
def disp(self)
print(self.du)
Test(True).disp() # returns 1
Test(False).disp() # gives of course an error
I tried to subclass Test changing du type to double[::1] and implementing a new __init__ but it seems that we can't override class attributes of extension types. Even if it worked, it wouldn't be satisfactory because I don't really want to have one extension type for each case.
The best would be that my extension type directly handle both cases (scalar du and memoryview du).
Is there a way to do this with Cython ?

Unfortunately, you cannot use fused_type as attributes type. You can have two options here:
You could try to use the memory adress of the variable you want to call, and cast it when needed (everything is explained here.) Unfortunately, I did not succeed at making it work with typed memory views.
Or you can use your defined attribute numeric to call the appropriate method:
import numpy as np
cimport numpy as np
cimport cython
cdef class Test:
cdef bint numeric
cdef double du_numeric
cdef double[:] du_mem_view
def __init__(self, bint numeric):
self.numeric = numeric
if self.numeric:
self.du_numeric = 1
else:
self.du_mem_view = np.ones(10)
def disp(self):
if self.numeric:
print(self.du_numeric)
else:
print(self.du_numeric_mem_view)
Test(True).disp() # returns 1
Test(False).disp() # Does not give an error anymore !

Related

#cython.locals(arr=np.ndarray[np.int_t, ndim=1]) not supported?

It seems that one can't declare np.ndarray in cython.locals in .pxd files. It works with memoryviews but not with np.ndarray. However, there are cases where we need np.ndarray.
In notsupported.py
import numpy as np
def func():
arr = np.ones(2)
return arr**2
In notsupported.pxd
import cython
import numpy as np
cimport numpy as np
#cython.locals(arr=np.ndarray[np.int_t, ndim=1])
cpdef func()
Error log:
Error compiling Cython file:
------------------------------------------------------------
...
import cython
import numpy as np
cimport numpy as np
#cython.locals(arr=np.ndarray[np.int_t, ndim=1])
^
------------------------------------------------------------
notsupported.pxd:6:44: Expected ']', found '='
Is there something wrong with this code? What is the alternative?
Since it looks like this isn't supported I assume you're really interested in workarounds. For the purpose of this question I'm assuming you want your code to also be valid in pure Python. I'm also assuming that your code is of the form:
def func():
arr = np.ones(2)
for n in range(arr.shape[0]):
arr[n] = # some operation element-by-element
return arr**2
If your code doesn't have the element-by-element section then there's really no benefit to setting the type at all - I don't believe Cython uses the type for Numpy "whole array" operations like the power operator you show here.
My first choice would be to have two variables: arr and arr_view. arr should be untyped, and arr_view a memoryview. You only use the memoryview in the element-by-element section. Provided you stick to in-place operations the two share the same memory so modifying one modifies the other:
def func():
arr = np.ones(2)
arr_view = arr
for n in range(arr_view.shape[0]):
arr_view[n] = ...
return arr**2
The pxd file is then:
#cython.locals(arr_view=np.int_t[:])
cpdef func()
My second choice would be to type arr as a memoryview, and use np.asarray when you want to do "whole array" operations
def func():
arr = np.ones(2)
for n in range(arr.shape[0]):
arr[n] = # some operation element-by-element
return np.asarray(arr)**2
with pxd:
#cython.locals(arr=nnp.int_t[:])
cpdef func()
np.asarray is essentially a no-op if it's passed an array, and can usuaully avoid a copy if passed a memoryview, so it won't slow things down too much.
A third option is to use the arr.base object of a memoryview to get the underlying Numpy array. This loses pure Python compatibility though since arr.base is often None when arr is a Numpy array. Therefore I don't really recommend it here.

Will cupy support cython( eg. buffered index)?

I have implement myself defined a chainer Link, but because it is too slow.
I have implemented cython CPU version of my code. But I want to further boost speed via GPU. So I test the following code , but it failed:
%%cython
import numpy as np
cimport numpy as np
import cupy as cp
cimport cupy as cp
cdef class A:
def __init__(self):
pass
cdef cp_test(self, cp.ndarray[cp.float_t, ndim=2] arr):
return cp.sum(arr)
a = A()
arr = cp.arange(100).reshape(20,50)
print(a.cp_test(arr))
reporting:
cdef cp_test(self, cp.ndarray[cp.float_t, ndim=2] arr):
^
------------------------------------------------------------
C:\Users\.ipython\cython\_cython_magic_d4940a274af88f0257c368b8a5d0e3f5.pyx:13:23: 'ndarray' is not a type identifier
Sorry, but CuPy does not provide cython interface currently (I am one of CuPy developers).

Cython return tuple within cdef?

Hi I am trying to convert a python code into cython in order to speed up its calculation. I am trying to return multiple arrays within the cython code from a cdef to cpdef. Based on classical C, I could either use a pointer or a tuple. I decide to use tuple because the size varies. I know the following code doesn't work, any help? Thank you!
import numpy as np
cimport numpy as np
cdef tuple funA(double[:] X, double[:] Y):
cdef int nX, nY, i
nX = len(X)
nY = len(Y)
for i in range(nX):
X[i] = X[i]*X[i]
for i in range(nY):
Y[i] = Y[i]*Y[i]
return X,Y
cpdef Run(double[:] X, double[:] Y)
cdef Tuple1, Tuple2 = funA(X,Y)
# Do some calculation with Tuple1 and Tuple2
# Example
cdef int i, nTuple1, nTuple2
nTuple1 = len(Tuple1)
for i in range(nTuple1):
Tuple1[i] = Tuple1[i]**2
nTuple2 = len(Tuple2)
for i in range(nTuple2):
Tuple2[i] = Tuple2[i]/2
return Tuple1, Tuple2
You've got a few indentation errors and missing colons. But your real issue is:
cdef Tuple1, Tuple2 = funA(X,Y)
Remove the cdef and it's fine. It doesn't look like cdef and tuple unpacking quite mix, and since you're treating them as Python objects it should be OK.
However, note that you don't really need to return anything from funA since you modify X and Y them in place there.

Cython: dimensions is not a member of 'tagPyArrayObject'

I implemented a pure Python code in object-oriented style. In some of the methods there are time intensive loops, which I hope to speed up by cythonizing the code.
I am using a lot of numpy arrays and struggle with converting classes into Cython extension types.
Here I declare two numpy arrays 'verteces' and 'norms' as attributes:
import numpy as np
cimport numpy as np
cdef class Geometry(object):
cdef:
np.ndarray verteces
np.ndarray norms
def __init__(self, config):
""" Initialization"""
self.config = config
self.verteces = np.empty([1,3,3],dtype=np.float32)
self.norms = np.empty(3,dtype=np.float32)
During runtime the actual size of the arrays will be defined. This happens when calling the Geometry.load() method of the same class. The method opens an STL-file and loops over the triangle entries.
Finally I want to determine the intersection points of the triangles and a ray. In the respective method I use the following declarations.
cdef void hit(self, object photon):
""" Ray-triangle intersection according to Moeller and Trumbore algorithm """
cdef:
np.ndarray[DTYPE_t, ndim=3] verteces = self.verteces # nx3x3
np.ndarray[DTYPE_t, ndim=2] norms = self.norms
np.ndarray[DTYPE_t, ndim=1] ph_dir = photon.direction
np.ndarray[DTYPE_t, ndim=1] ph_origin = photon.origin
np.ndarray[DTYPE_t, ndim=1] v0, v1, v2, vec1, vec2, trsc, norm, v, p_inter
float a, b, par, q, q0, q1, s0, s1
int i_tri
When I try to compile this code I get the following error message:
'dimensions' is not a member of 'tagPyArrayObject'
I am not very familiar cython programming, but maybe the error is do to the fact that I have to initialize an array of fixed size in a C-extension type? The size of the array is, however, unkown until the STL-file is read.
Not sure if this is related to your problem, but I've got the same identical error message when specifying the "NPY_1_7_API_VERSION" macro in my setup.py file.
extension_module = Extension(
'yourfilename',
sources=["yourfilename.pyx],
include_dirs=[numpy.get_include()],
define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],
)
With this macro, a simple npmatrix.shape[0] numpy function is compiled as:
/* "yourfilename.pyx":35
*
* cpdef int vcount(self):
* return self.npmatrix.shape[0]
*
*/
__pyx_r = (__pyx_v_self->npmatrix->dimensions[0]);
which causes the error. Just removing the macro resolved this error to me.

Using Cython extension module to wrap std::vector - How do I program __setitem__() method?

This seems like a question that should have an obvious answer, but for some reason I can't find any examples online.
I am wrapping a vector of C++ objects in a Python class using Cython. I also have a Cython wrapper for the C++ class already coded. I can get several methods such as __len__(), __getitem__(), and resize() to work properly, but the __setitem__() method is giving me problems.
For simplicity, I coded a small example using a vector of ints. I figure if I can get this code to work, then I can build on that to get the solution for my C++ class as well.
MyPyModule.pyx
# distutils: language = c++
from libcpp.vector cimport vector
from cython.operator cimport dereference as deref
cdef class MyArray:
cdef vector[int]* thisptr
def __cinit__(self):
self.thisptr = new vector[int]()
def __dealloc__(self):
del self.thisptr
def __len__(self):
return self.thisptr.size()
def __getitem__(self, size_t key):
return self.thisptr.at(key)
def resize(self, size_t newsize):
self.thisptr.resize(newsize)
def __setitem__(self, size_t key, int value):
# Attempt 1:
# self.thisptr.at(key) = value
# Attempt 2:
# cdef int* itemptr = &(self.thisptr.at(key))
# itemptr[0] = value
# Attempt 3:
# (self.thisptr)[key] = value
# Attempt 4:
self[key] = value
When I tried to cythonize using Attempt 1, I got the error Cannot assign to or delete this. When I tried Attempt 2, the .cpp file was created, but the compiler complained that:
error: cannot convert β€˜__Pyx_FakeReference<int>*’ to β€˜int*’ in assignment
__pyx_v_itemptr = (&__pyx_t_1);
On Attempt 3, Cython would not build the file because Cannot assign type 'int' to 'vector[int]'. (When I tried this style with the C++ object instead of int, it complained because I had a reference as a left-value.) Attempt 4 compiles, but when I try to use it, I get a segfault.
Cython docs say that returning a reference as a left-value is not supported, which is fine -- but how do I get around it so that I can assign a new value to one of my vector elements?
There are two ways to access the vector through a pointer,
def __setitem__(self, size_t key, int value):
deref(self.thisptr)[key] = value
# or
# self.thisptr[0][key] = value
Cython translates those two cases as follows:
Python: deref(self.thisptr)[key] = value
C++: ((*__pyx_v_self->thisptr)[__pyx_v_key]) = __pyx_v_value;
Python: self.thisptr[0][key] = value
C++: ((__pyx_v_self->thisptr[0])[__pyx_v_key]) = __pyx_v_value;
which are equivalent i.e. access the same vector object.
Instead of trying to handle a pointer from Cython code, you can let Cython itself do it for you:
cdef class MyArray:
cdef vector[int] thisptr
def __len__(self):
return self.thisptr.size()
def __getitem__(self, size_t key):
return self.thisptr[key]
def __setitem__(self, size_t key, int value):
self.thisptr[key] = value
def resize(self, size_t newsize):
self.thisptr.resize(newsize)
Is there any problem with this approach?
I have already accepted J.J. Hakala's answer (many thanks!). I tweaked that method to include an out-of-bounds check, since it uses the [] operator instead of the at() method:
cdef class MyArray:
(....)
def __setitem__(self, size_t key, int value):
if key < self.thisptr.size():
deref(self.thisptr)[key] = value
else:
raise IndexError("Index is out of range.")