Segmentation fault in Cython - cython

I am new to Cython. I have written a pyx file that returns a 2D Numpy or a memoryview array.
Here is the pyx code:
import numpy as np
import cython
cimport numpy as np
from libc.math cimport int
#cython.wraparound(False)
#cython.boundscheck(False)
#cython.cdivision(True)
#cython.nonecheck(False)
cdef class TwoDMatrix():
''' make a two dimensional matrix '''
cdef int N
def __init__(self,N):
self.N = N
cpdef twoD(self, int [:,:] vector,str array):
if array == 'numpy':
state = 2*np.random.randint(2,size=(self.N,self.N))-1
return state
else:
vec=self.initialise(vector,self.N)
return vec
cdef int [:,:] initialise(self, int [:,:] s,k) :
cdef int i,j
for i in range(k+1):
for j in range(k+1):
if np.random.rand() <0.5:
s[i,j] = -1
else:
s[i,j] = 1
return s
And here is the main py file where I give input:
import numpy as np
import main
def matrix():
N = 10
vector = main.TwoDMatrix(N)
spin = np.zeros((N,N),dtype=np.int32)
print(f"from numpy: {vector.twoD(spin,'numpy')}")
print(f"from memoryview: {vector.twoD(spin,'memoryview')}")
print("program successfully exited")
if __name__ == "__main__":
matrix()
The problem is for N >5 , when I run in terminal it always shows a segmentation fault(core dumped) message at the bottom. And for N>1, a similar following message is shown:
"corrupted size vs. prev_size while consolidating
Aborted (core dumped)"
Why is this message popping up? Is there anything that I should consider about memory allocation?

The segmentation fault is due to out-of-bounds accesses. Indeed, you create matrices of size (N,N) while initialise iterate over range range(0, k+1) where k=N. Thus you need to either use bigger matrices or to fix the two nested loops so to iterate over range(0, k) / range(0, N).

Related

Cython: declaring a variable window during looping over an array

I'm trying to loop over a 3D array with a window. At each iteration, the window is moved 1 pixel and the variance for the (3D)window is calculated.
I'm trying to do this in Cython for performance reasons, in Jupyter notebook.
My (working, but slow) code in python looks approximately like this:
## PYTHON
#code adapted from https://stackoverflow.com/questions/36353262/i-need-a-fast-way-to-loop-through-pixels-of-an-image-stack-in-python
def Variance_Filter_3D_python(image, kernel = 30):
min_var = 10000
min_var_coord = [0,0,0]
window = np.zeros(shape=(kernel,kernel,kernel), dtype = np.uint8)
z,y,x = image.shape
for i in np.arange(0,(z-kernel),1):
for j in np.arange(0,(y-kernel),1):
for k in np.arange(0,(x-kernel),1):
window[:,:,:] = image[i:i+kernel,j:j+kernel,k:k+kernel]
var = np.var(window)
if var < min_var:
min_var = var
min_var_coord = [i,j,k]
print(min_var_coord)
return min_var,min_var_coord
When I try to declare the variables in the cython code:
%%cython
#cython.boundscheck(False) # Deactivate bounds checking
#cython.wraparound(False)
def Variance_Filter_3D(image, kernel = 30):
cdef double min_var = 10000
cdef list min_var_coord = [0,0,0]
cdef unsigned int z,y,x = image.shape
cdef np.ndarray[float, ndim=3] window = np.zeros(shape=(kernel,kernel,kernel),
dtype=FTYPE)
....etc
I get a error saying that "'np' is not declared" in the following line:
cdef np.ndarray[float, ndim=3] window = np.zeros(shape=(kernel,kernel,kernel),
dtype=FTYPE)
and that cython isn't declared in these lines:
#cython.boundscheck(False) # Deactivate bounds checking
#cython.wraparound(False)
However, I have used cimport previously:
%%cython
cimport numpy as np
cimport cython
What's going wrong?
You probably need to put the Numpy and Cython cimports in the exact notebook cell you need them in. Cython doesn't have a lot of "global scope" in Jupiter.
However,
window[:,:,:] = image[i:i+kernel,j:j+kernel,k:k+kernel]
will work a lot better if:
you set the type of image to be a memoryview. Slicing a memoryview is fairly quick while viewing an arbitrary Python object as a memoryview is slower.
You made the left-hand side window instead of window[:,:,:] (a view rather than a copy)

Why is the time identical for Cython and Python versions of function?

I am just starting out learning about Cython. I have some code written in pure Python, and started to convert it to Cython. The first function is below (both versions). I expected Cython to be faster, but there is no difference in speed. Why is this? Am I doing something wrong, or is it expected?
import numpy as np
cimport numpy as np
import time
ctypedef np.int8_t DTYPE_int
cpdef np.ndarray[DTYPE_int, ndim=2] init_population(int N, int pop_size):
cdef np.ndarray[DTYPE_int, ndim=2] p
cdef np.ndarray[DTYPE_int, ndim=2] idx
cdef double t1
t1=time.time()
p = np.full((pop_size, N), np.arange(N, dtype=np.int8))
idx = np.random.rand(p.shape[0], p.shape[1]).argsort(axis=1).astype(np.int8)
print("%.20f" % (time.time() - t1))
return np.take_along_axis(p, idx, axis=1)
def py_init_population(N, pop_size):
t1=time.time()
p = np.full((pop_size, N), np.arange(N, dtype=np.int8))
idx = np.random.rand(p.shape[0], p.shape[1]).argsort(axis=1).astype(np.int8)
print("%.20f" % (time.time() - t1))
return np.take_along_axis(p, idx, axis=1)
init_population(1000, 1000)
py_init_population(1000, 1000)
Output:
7.24499845504760742188
7.26293945312500000000

How to create an array of structs of arbitrary size cython

I would like to know how can I create an array of structs in Cython that I can populate and make computations afterwards.
Example
Here I have the Cython code
%%cython
cimport numpy as cnp
cimport cython
from collections import namedtuple
Couple = namedtuple('Couple', ['female', 'male'], verbose=False)
cdef struct CyCouple:
int female
int male
cpdef int np_cy_count_women_earning_more2(list py_couples):
cdef:
int count = 0, r, N
CyCouple cy_couples[100_0000] # THIS IS HARDCODED
N = len(py_couples)
make_CyCouple_array(py_couples, cy_couples, N)
for n in range(N):
r = cy_couples[n].female > cy_couples[n].male
count += r
return count
I would like to have a general versions instead of the definition in # THIS IS HARDCODED.
What could I do?

Cython return tuple within cdef?

Hi I am trying to convert a python code into cython in order to speed up its calculation. I am trying to return multiple arrays within the cython code from a cdef to cpdef. Based on classical C, I could either use a pointer or a tuple. I decide to use tuple because the size varies. I know the following code doesn't work, any help? Thank you!
import numpy as np
cimport numpy as np
cdef tuple funA(double[:] X, double[:] Y):
cdef int nX, nY, i
nX = len(X)
nY = len(Y)
for i in range(nX):
X[i] = X[i]*X[i]
for i in range(nY):
Y[i] = Y[i]*Y[i]
return X,Y
cpdef Run(double[:] X, double[:] Y)
cdef Tuple1, Tuple2 = funA(X,Y)
# Do some calculation with Tuple1 and Tuple2
# Example
cdef int i, nTuple1, nTuple2
nTuple1 = len(Tuple1)
for i in range(nTuple1):
Tuple1[i] = Tuple1[i]**2
nTuple2 = len(Tuple2)
for i in range(nTuple2):
Tuple2[i] = Tuple2[i]/2
return Tuple1, Tuple2
You've got a few indentation errors and missing colons. But your real issue is:
cdef Tuple1, Tuple2 = funA(X,Y)
Remove the cdef and it's fine. It doesn't look like cdef and tuple unpacking quite mix, and since you're treating them as Python objects it should be OK.
However, note that you don't really need to return anything from funA since you modify X and Y them in place there.

Why cannot I pass a c array to a function which expects memory view in nogil content?

cdef double testB(double[:] x) nogil:
return x[0]
def test():
cdef double xx[2]
with nogil:
testB(xx)
# compiler error: Operation not allowed without gil
If with gil, it works fine.
Is it because that when pass in an c array, it creates a memory view and such creation action actually requires gil? So the memory view is not completely a c object?
Update
%%cython --annotate
cimport cython
cdef double testA(double[:] x) nogil:
return x[0]
cpdef myf():
cdef double pd[8]
cdef double[:] x = pd
testA(x)
cdef double[:] x = pd is compiled to:
__pyx_t_3 = __pyx_format_from_typeinfo(&__Pyx_TypeInfo_double);
__pyx_t_2 = Py_BuildValue((char*) "(" __PYX_BUILD_PY_SSIZE_T ")", ((Py_ssize_t)8));
if (unlikely(!__pyx_t_3 || !__pyx_t_2 || !PyBytes_AsString(__pyx_t_3))) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_3);
__Pyx_GOTREF(__pyx_t_2);
__pyx_t_1 = __pyx_array_new(__pyx_t_2, sizeof(double), PyBytes_AS_STRING(__pyx_t_3), (char *) "fortran", (char *) __pyx_v_pd);
if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_1);
__Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
__Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
__pyx_t_4 = __Pyx_PyObject_to_MemoryviewSlice_ds_double(((PyObject *)__pyx_t_1));
if (unlikely(!__pyx_t_4.memview)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_DECREF(((PyObject *)__pyx_t_1)); __pyx_t_1 = 0;
__pyx_v_x = __pyx_t_4;
__pyx_t_4.memview = NULL;
__pyx_t_4.data = NULL;
There exists __Pyx_PyObject_to_MemoryviewSlice_ds_double. So it seems when binding a memory view it does require gil.
You should use a numpy array, as your cdef double[:] declaration gets wrapped by a Python object, and its use is restricted without gil. You can see it by trying to slice a double[:]
def test()
cdef double[:] asd
with nogil:
asd[:1]
Your output will be:
with nogil:
asd[:1]
^
------------------------------------------------------------
prueba.pyx:16:11: Slicing Python object not allowed without gil
Using a numpy array would compile; numpy uses Python buffer protocole, and is smoothly integrated with Cython (a Google Summercamp project was financed for this). So no wrapping conflict arises inside the def:
import numpy as np
cdef double testA(double[:] x) nogil:
return x[0]
cpdef test():
xx = np.zeros(2, dtype = 'double')
with nogil:
a = testB(xx)
print(a)
This will build your module with test() on it. But it crashes, and in an ugly way (at least with mi PC):
Process Python segmentation fault (core dumped)
If I may insist with my (now deleted) previous answer, in my own experience, when dealing with Cython memoryviews and C arrays, passing pointers works just like one would expect in C. And most wrapping is avoided (actually, you are writing the code passing exactly the directions you want, thus making unnecesary wrapping). This compiles and functions as expected:
cdef double testB(double* x) nogil:
return x[0]
def test():
cdef double asd[2]
asd[0] = 1
asd[1] = 2
with nogil:
a = testB(asd)
print(a)
And, after compilig:
In [5]: import prueba
In [6]: prueba.test()
1.0
Memoryviews are not, by themselves, Python objects, but they can be wrapped in one. I am not a proficient Cython programmer, so sometimes I get unexpected wrappings or code that remains at Python level when I supposed it would be at C. Trial and error got me to the pointer strategy.