I'm trying to loop over a 3D array with a window. At each iteration, the window is moved 1 pixel and the variance for the (3D)window is calculated.
I'm trying to do this in Cython for performance reasons, in Jupyter notebook.
My (working, but slow) code in python looks approximately like this:
## PYTHON
#code adapted from https://stackoverflow.com/questions/36353262/i-need-a-fast-way-to-loop-through-pixels-of-an-image-stack-in-python
def Variance_Filter_3D_python(image, kernel = 30):
min_var = 10000
min_var_coord = [0,0,0]
window = np.zeros(shape=(kernel,kernel,kernel), dtype = np.uint8)
z,y,x = image.shape
for i in np.arange(0,(z-kernel),1):
for j in np.arange(0,(y-kernel),1):
for k in np.arange(0,(x-kernel),1):
window[:,:,:] = image[i:i+kernel,j:j+kernel,k:k+kernel]
var = np.var(window)
if var < min_var:
min_var = var
min_var_coord = [i,j,k]
print(min_var_coord)
return min_var,min_var_coord
When I try to declare the variables in the cython code:
%%cython
#cython.boundscheck(False) # Deactivate bounds checking
#cython.wraparound(False)
def Variance_Filter_3D(image, kernel = 30):
cdef double min_var = 10000
cdef list min_var_coord = [0,0,0]
cdef unsigned int z,y,x = image.shape
cdef np.ndarray[float, ndim=3] window = np.zeros(shape=(kernel,kernel,kernel),
dtype=FTYPE)
....etc
I get a error saying that "'np' is not declared" in the following line:
cdef np.ndarray[float, ndim=3] window = np.zeros(shape=(kernel,kernel,kernel),
dtype=FTYPE)
and that cython isn't declared in these lines:
#cython.boundscheck(False) # Deactivate bounds checking
#cython.wraparound(False)
However, I have used cimport previously:
%%cython
cimport numpy as np
cimport cython
What's going wrong?
You probably need to put the Numpy and Cython cimports in the exact notebook cell you need them in. Cython doesn't have a lot of "global scope" in Jupiter.
However,
window[:,:,:] = image[i:i+kernel,j:j+kernel,k:k+kernel]
will work a lot better if:
you set the type of image to be a memoryview. Slicing a memoryview is fairly quick while viewing an arbitrary Python object as a memoryview is slower.
You made the left-hand side window instead of window[:,:,:] (a view rather than a copy)
Related
Actually I have a working version of what I need, but it is awfully slow:
%%cython -a
cimport numpy as cnp
import numpy as np # for np.empty
from time import time
cdef packed struct Bar:
cnp.int64_t dt
double open
double high
double low
double close
cnp.int64_t volume
bar_dtype = np.dtype([('dt', 'i8'), ('open', 'f8'), ('high', 'f8'), ('low', 'f8'), ('close', 'f8'), ('volume', 'i8')])
cpdef f(Bar bar):
cdef cnp.ndarray bars_sar
cdef Bar[:] buffer
bars_sar = np.empty((1000000,), dtype=bar_dtype)
start = time()
for n in range(1000000):
buffer = bars_sar
buffer[0] = bar
print (f'Elapsed: {time() - start}')
return buffer
def test():
sar = f(Bar(1,2,3,4,5,6))
print(sar[0])
return sar
This 1M iterations loop takes about 3 seconds - because of the line takes memory view from numpy structured array:
buffer = bars_sar
The idea is that I need bars_sar untyped. Only when I want to store or read something from it, I want to reinterpret it as a particular type of memory view, and I don't see any problem why it cannot be done fast, but don't know how to do it. I hope, there is something similar to C reinterpret_cast in Cython.
I tried to declare bars_sar as void *, but I'm unable to store memory view address there like:
cdef Bar[:] view = np.empty((5,), dtype=bar_dtype)
bars_sar = <void*>view
or even
cdef Bar[:] view = np.empty((5,), dtype=bar_dtype)
bars_sar = <void*>&view
The first one results in error C2440 that it cannot convert "__Pyx_memviewslice" to "void *"
The second one results in error: "Cannot take address of memoryview slice"
Please suggest
I'd like to call my cdef methods and improve the speed of my program as much as possible. I do not want to use cpdef (I explain why below). Ultimately, I'd like to access cdef methods (some of which return void) that are members of my Cython extensions.
I tried following this example, which gives me the impression that I can call a cdef function by making a Python (def) wrapper for it.
I can't reproduce these results, so I tried a different problem for myself (summing all the numbers from 0 to n).
Of course, I'm looking at the documentation, which says
The directive cpdef makes two versions of the method available; one fast for use from Cython and one slower for use from Python.
and later (emphasis mine),
This does slightly more than providing a python wrapper for a cdef method: unlike a cdef method, a cpdef method is fully overridable by methods and instance attributes in Python subclasses. It adds a little calling overhead compared to a cdef method.
So how does one use a cdef function without the extra calling overhead of a cpdef function?
With the code at the end of this question, I get the following results:
def/cdef:
273.04207632583245
def/cpdef:
304.4114626176919
cpdef/cdef:
0.8969507060538783
Somehow, cpdef is faster than cdef. For n < 100, I can occasionally get cpdef/cdef > 1, but it's rare. I think it has to do with wrapping the cdef function in a def function. This is what the example I link to does, but they claim better performance from using cdef than from using cpdef.
I'm pretty sure this is not how you wrap a cdef function while avoiding the additional overhead (the source of which is not clearly documented) of a cpdef.
And now, the code:
setup.py
from setuptools import setup, Extension
from Cython.Build import cythonize
pkg_name = "tmp"
compile_args=['-std=c++17']
cy_foo = Extension(
name=pkg_name + '.core.cy_foo',
sources=[
pkg_name + '/core/cy_foo.pyx',
],
language='c++',
extra_compile_args=compile_args,
)
setup(
name=pkg_name,
ext_modules=cythonize(cy_foo,
annotate=True,
build_dir='build'),
packages=[
pkg_name,
pkg_name + '.core',
],
)
foo.py
def foo_def(n):
sum = 0
for i in range(n):
sum += i
return sum
cy_foo.pyx
def foo_cdef(n):
return foo_cy(n)
cdef int foo_cy(int n):
cdef int sum = 0
cdef int i = 0
for i in range(n):
sum += i
return sum
cpdef int foo_cpdef(int n):
cdef int sum = 0
cdef int i = 0
for i in range(n):
sum += i
return sum
test.py
import timeit
from tmp.core.foo import foo_def
from tmp.core.cy_foo import foo_cdef
from tmp.core.cy_foo import foo_cpdef
n = 10000
# Python call
start_time = timeit.default_timer()
a = foo_def(n)
pyTime = timeit.default_timer() - start_time
# Call Python wrapper for C function
start_time = timeit.default_timer()
b = foo_cdef(n)
cTime = timeit.default_timer() - start_time
# Call cpdef function, which does more than wrap a cdef function (whatever that means)
start_time = timeit.default_timer()
c = foo_cpdef(n)
cpTime = timeit.default_timer() - start_time
print("def/cdef:")
print(pyTime/cTime)
print("def/cpdef:")
print(pyTime/cpTime)
print("cpdef/cdef:")
print(cpTime/cTime)
The reason for your seemingly anomalous result is that you aren't calling the cdef function foo_cy directly, but instead the def function foo_cdef wrapping it.
when you are wrapping inside another def indeed you are again calling the python function. However you should be able to reach similar results as the cpdef.
Here is what you could do:
like the python def, give the type for both input and output
def foo_cdef(int n):
cdef int val = 0
val = foo_cy(n)
return val
this should have similar results as cpdef, however again you are calling a python function. If you want to directly call the c function, you should use the ctypes and call from there.
and for the benchmarking, the way that you have written, it only considers one run and could fluctuate a lot due OS other task and as well the timer.
better to use the timeit builtin method to calculate for some iteration:
# Python call
pyTime = timeit.timeit('foo_def(n)',globals=globals(), number=10000)
# Call Python wrapper for C function
cTime = timeit.timeit('foo_cdef(n)',globals=globals(), number=10000)
# Call cpdef function, which does more than wrap a cdef function (whatever that means)
cpTime = timeit.timeit('foo_cpdef(n)',globals=globals(), number=10000)
output:
def/cdef:
154.0166154428522
def/cpdef:
154.22669848136132
cpdef/cdef:
0.9986378296327566
like this, you get consistent results and as well you see always close to 1 for both either cython itself wraps or we explicitly wrap around a python function.
Hi I am trying to convert a python code into cython in order to speed up its calculation. I am trying to return multiple arrays within the cython code from a cdef to cpdef. Based on classical C, I could either use a pointer or a tuple. I decide to use tuple because the size varies. I know the following code doesn't work, any help? Thank you!
import numpy as np
cimport numpy as np
cdef tuple funA(double[:] X, double[:] Y):
cdef int nX, nY, i
nX = len(X)
nY = len(Y)
for i in range(nX):
X[i] = X[i]*X[i]
for i in range(nY):
Y[i] = Y[i]*Y[i]
return X,Y
cpdef Run(double[:] X, double[:] Y)
cdef Tuple1, Tuple2 = funA(X,Y)
# Do some calculation with Tuple1 and Tuple2
# Example
cdef int i, nTuple1, nTuple2
nTuple1 = len(Tuple1)
for i in range(nTuple1):
Tuple1[i] = Tuple1[i]**2
nTuple2 = len(Tuple2)
for i in range(nTuple2):
Tuple2[i] = Tuple2[i]/2
return Tuple1, Tuple2
You've got a few indentation errors and missing colons. But your real issue is:
cdef Tuple1, Tuple2 = funA(X,Y)
Remove the cdef and it's fine. It doesn't look like cdef and tuple unpacking quite mix, and since you're treating them as Python objects it should be OK.
However, note that you don't really need to return anything from funA since you modify X and Y them in place there.
I implemented a pure Python code in object-oriented style. In some of the methods there are time intensive loops, which I hope to speed up by cythonizing the code.
I am using a lot of numpy arrays and struggle with converting classes into Cython extension types.
Here I declare two numpy arrays 'verteces' and 'norms' as attributes:
import numpy as np
cimport numpy as np
cdef class Geometry(object):
cdef:
np.ndarray verteces
np.ndarray norms
def __init__(self, config):
""" Initialization"""
self.config = config
self.verteces = np.empty([1,3,3],dtype=np.float32)
self.norms = np.empty(3,dtype=np.float32)
During runtime the actual size of the arrays will be defined. This happens when calling the Geometry.load() method of the same class. The method opens an STL-file and loops over the triangle entries.
Finally I want to determine the intersection points of the triangles and a ray. In the respective method I use the following declarations.
cdef void hit(self, object photon):
""" Ray-triangle intersection according to Moeller and Trumbore algorithm """
cdef:
np.ndarray[DTYPE_t, ndim=3] verteces = self.verteces # nx3x3
np.ndarray[DTYPE_t, ndim=2] norms = self.norms
np.ndarray[DTYPE_t, ndim=1] ph_dir = photon.direction
np.ndarray[DTYPE_t, ndim=1] ph_origin = photon.origin
np.ndarray[DTYPE_t, ndim=1] v0, v1, v2, vec1, vec2, trsc, norm, v, p_inter
float a, b, par, q, q0, q1, s0, s1
int i_tri
When I try to compile this code I get the following error message:
'dimensions' is not a member of 'tagPyArrayObject'
I am not very familiar cython programming, but maybe the error is do to the fact that I have to initialize an array of fixed size in a C-extension type? The size of the array is, however, unkown until the STL-file is read.
Not sure if this is related to your problem, but I've got the same identical error message when specifying the "NPY_1_7_API_VERSION" macro in my setup.py file.
extension_module = Extension(
'yourfilename',
sources=["yourfilename.pyx],
include_dirs=[numpy.get_include()],
define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],
)
With this macro, a simple npmatrix.shape[0] numpy function is compiled as:
/* "yourfilename.pyx":35
*
* cpdef int vcount(self):
* return self.npmatrix.shape[0]
*
*/
__pyx_r = (__pyx_v_self->npmatrix->dimensions[0]);
which causes the error. Just removing the macro resolved this error to me.
cdef double testB(double[:] x) nogil:
return x[0]
def test():
cdef double xx[2]
with nogil:
testB(xx)
# compiler error: Operation not allowed without gil
If with gil, it works fine.
Is it because that when pass in an c array, it creates a memory view and such creation action actually requires gil? So the memory view is not completely a c object?
Update
%%cython --annotate
cimport cython
cdef double testA(double[:] x) nogil:
return x[0]
cpdef myf():
cdef double pd[8]
cdef double[:] x = pd
testA(x)
cdef double[:] x = pd is compiled to:
__pyx_t_3 = __pyx_format_from_typeinfo(&__Pyx_TypeInfo_double);
__pyx_t_2 = Py_BuildValue((char*) "(" __PYX_BUILD_PY_SSIZE_T ")", ((Py_ssize_t)8));
if (unlikely(!__pyx_t_3 || !__pyx_t_2 || !PyBytes_AsString(__pyx_t_3))) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_3);
__Pyx_GOTREF(__pyx_t_2);
__pyx_t_1 = __pyx_array_new(__pyx_t_2, sizeof(double), PyBytes_AS_STRING(__pyx_t_3), (char *) "fortran", (char *) __pyx_v_pd);
if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_1);
__Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
__Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
__pyx_t_4 = __Pyx_PyObject_to_MemoryviewSlice_ds_double(((PyObject *)__pyx_t_1));
if (unlikely(!__pyx_t_4.memview)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_DECREF(((PyObject *)__pyx_t_1)); __pyx_t_1 = 0;
__pyx_v_x = __pyx_t_4;
__pyx_t_4.memview = NULL;
__pyx_t_4.data = NULL;
There exists __Pyx_PyObject_to_MemoryviewSlice_ds_double. So it seems when binding a memory view it does require gil.
You should use a numpy array, as your cdef double[:] declaration gets wrapped by a Python object, and its use is restricted without gil. You can see it by trying to slice a double[:]
def test()
cdef double[:] asd
with nogil:
asd[:1]
Your output will be:
with nogil:
asd[:1]
^
------------------------------------------------------------
prueba.pyx:16:11: Slicing Python object not allowed without gil
Using a numpy array would compile; numpy uses Python buffer protocole, and is smoothly integrated with Cython (a Google Summercamp project was financed for this). So no wrapping conflict arises inside the def:
import numpy as np
cdef double testA(double[:] x) nogil:
return x[0]
cpdef test():
xx = np.zeros(2, dtype = 'double')
with nogil:
a = testB(xx)
print(a)
This will build your module with test() on it. But it crashes, and in an ugly way (at least with mi PC):
Process Python segmentation fault (core dumped)
If I may insist with my (now deleted) previous answer, in my own experience, when dealing with Cython memoryviews and C arrays, passing pointers works just like one would expect in C. And most wrapping is avoided (actually, you are writing the code passing exactly the directions you want, thus making unnecesary wrapping). This compiles and functions as expected:
cdef double testB(double* x) nogil:
return x[0]
def test():
cdef double asd[2]
asd[0] = 1
asd[1] = 2
with nogil:
a = testB(asd)
print(a)
And, after compilig:
In [5]: import prueba
In [6]: prueba.test()
1.0
Memoryviews are not, by themselves, Python objects, but they can be wrapped in one. I am not a proficient Cython programmer, so sometimes I get unexpected wrappings or code that remains at Python level when I supposed it would be at C. Trial and error got me to the pointer strategy.