Actually I have a working version of what I need, but it is awfully slow:
%%cython -a
cimport numpy as cnp
import numpy as np # for np.empty
from time import time
cdef packed struct Bar:
cnp.int64_t dt
double open
double high
double low
double close
cnp.int64_t volume
bar_dtype = np.dtype([('dt', 'i8'), ('open', 'f8'), ('high', 'f8'), ('low', 'f8'), ('close', 'f8'), ('volume', 'i8')])
cpdef f(Bar bar):
cdef cnp.ndarray bars_sar
cdef Bar[:] buffer
bars_sar = np.empty((1000000,), dtype=bar_dtype)
start = time()
for n in range(1000000):
buffer = bars_sar
buffer[0] = bar
print (f'Elapsed: {time() - start}')
return buffer
def test():
sar = f(Bar(1,2,3,4,5,6))
print(sar[0])
return sar
This 1M iterations loop takes about 3 seconds - because of the line takes memory view from numpy structured array:
buffer = bars_sar
The idea is that I need bars_sar untyped. Only when I want to store or read something from it, I want to reinterpret it as a particular type of memory view, and I don't see any problem why it cannot be done fast, but don't know how to do it. I hope, there is something similar to C reinterpret_cast in Cython.
I tried to declare bars_sar as void *, but I'm unable to store memory view address there like:
cdef Bar[:] view = np.empty((5,), dtype=bar_dtype)
bars_sar = <void*>view
or even
cdef Bar[:] view = np.empty((5,), dtype=bar_dtype)
bars_sar = <void*>&view
The first one results in error C2440 that it cannot convert "__Pyx_memviewslice" to "void *"
The second one results in error: "Cannot take address of memoryview slice"
Please suggest
Related
I'm trying to loop over a 3D array with a window. At each iteration, the window is moved 1 pixel and the variance for the (3D)window is calculated.
I'm trying to do this in Cython for performance reasons, in Jupyter notebook.
My (working, but slow) code in python looks approximately like this:
## PYTHON
#code adapted from https://stackoverflow.com/questions/36353262/i-need-a-fast-way-to-loop-through-pixels-of-an-image-stack-in-python
def Variance_Filter_3D_python(image, kernel = 30):
min_var = 10000
min_var_coord = [0,0,0]
window = np.zeros(shape=(kernel,kernel,kernel), dtype = np.uint8)
z,y,x = image.shape
for i in np.arange(0,(z-kernel),1):
for j in np.arange(0,(y-kernel),1):
for k in np.arange(0,(x-kernel),1):
window[:,:,:] = image[i:i+kernel,j:j+kernel,k:k+kernel]
var = np.var(window)
if var < min_var:
min_var = var
min_var_coord = [i,j,k]
print(min_var_coord)
return min_var,min_var_coord
When I try to declare the variables in the cython code:
%%cython
#cython.boundscheck(False) # Deactivate bounds checking
#cython.wraparound(False)
def Variance_Filter_3D(image, kernel = 30):
cdef double min_var = 10000
cdef list min_var_coord = [0,0,0]
cdef unsigned int z,y,x = image.shape
cdef np.ndarray[float, ndim=3] window = np.zeros(shape=(kernel,kernel,kernel),
dtype=FTYPE)
....etc
I get a error saying that "'np' is not declared" in the following line:
cdef np.ndarray[float, ndim=3] window = np.zeros(shape=(kernel,kernel,kernel),
dtype=FTYPE)
and that cython isn't declared in these lines:
#cython.boundscheck(False) # Deactivate bounds checking
#cython.wraparound(False)
However, I have used cimport previously:
%%cython
cimport numpy as np
cimport cython
What's going wrong?
You probably need to put the Numpy and Cython cimports in the exact notebook cell you need them in. Cython doesn't have a lot of "global scope" in Jupiter.
However,
window[:,:,:] = image[i:i+kernel,j:j+kernel,k:k+kernel]
will work a lot better if:
you set the type of image to be a memoryview. Slicing a memoryview is fairly quick while viewing an arbitrary Python object as a memoryview is slower.
You made the left-hand side window instead of window[:,:,:] (a view rather than a copy)
I'm trying to create a memoryview to store several vectors as rows, but when I try to change the value of any I got an error, like it is expecting a scalar.
%%cython
import numpy as np
cimport numpy as np
DTYPE = np.float
ctypedef np.float_t DTYPE_t
cdef DTYPE_t[:, ::1] results = np.zeros(shape=(10, 10), dtype=DTYPE)
results[:, 0] = np.random.rand(10)
This trows me the following error:
TypeError: only size-1 arrays can be converted to Python scalars
Which I don't understand given that I want to overwrite the first row with that vector... Any idea about what I am doing wrong?
The operation you would like to use is possible between numpy arrays (Python functionality) or Cython's memory views (C functionality, i.e. Cython generates right for-loops in the C-code), but not when you mix a memory view (on the left-hand side) and a numpy array (on the right-hand side).
So you have either to use Cython's memory-views:
...
cdef DTYPE_t[::1] r = np.random.rand(10)
results[:, 0] = r
#check it worked:
print(results.base)
...
or numpy-arrays (we know .base is a numpy-array):
results.base[:, 0] = np.random.rand(10)
#check it worked:
print(results.base)
Cython's version has less overhead, but for large matrices there won't be much difference.
I have a number of objects that I am trying to pickle which all share the same (large) cython memoryview as an attribute. Since the memoryviews are passed by reference, they all share the same memory and the implementation is memory efficient.
Now I need to pickle these objects and reload them while keeping the shared data shared (if the shared data becomes not shared then the file size blows up and it is impossible to read into memory). Normally I think pickle recognizes shared data and just pickles/unpickles it once, but because memory views can't be pickled directly, they need to be converted to a numpy array in the reduce method for each object and pickle no longer recognizes that the data is shared.
Is there some way that I can maintain the shared data through the pickle/unpickle process?
A MWE follows:
import numpy as np
import pickle
cdef class SharedMemory:
cdef public double[:, :] data
def __init__(self, data):
self.data = data
def duplicate(self):
return SharedMemory(self.data)
def __reduce__(self):
return self.__class__, (np.asarray(self.data),)
def main():
x = SharedMemory(np.random.randn(100, 100))
duplicates = [x.duplicate() for _ in range(5)]
cdef double* pointerx = &x.data[0, 0]
cdef double* pointerd
cdef double[:, :] ddata
for d in duplicates:
ddata = d.data
pointerd = &ddata[0, 0]
if pointerd != pointerx:
print('Memory is not shared')
else:
print('Memory is shared')
print('pickling')
with open('./temp.pickle', 'wb') as pfile:
pickle.dump(x, pfile, protocol=pickle.HIGHEST_PROTOCOL)
for d in duplicates:
pickle.dump(d, pfile, protocol=pickle.HIGHEST_PROTOCOL)
with open('./temp.pickle', 'rb') as pfile:
nx = pickle.load(pfile)
nd = []
for d in duplicates:
nd.append(pickle.load(pfile))
ddata = nx.data
cdef double* pointernx = &ddata[0, 0]
for d in nd:
ddata = d.data
pointerd = &ddata[0, 0]
if pointerd != pointernx:
print('Memory is not shared')
else:
print('Memory is shared')
Put the above in a file test.pyx an cythonize with "cythonize -a -i test.pyx". Then "export PYTHONPATH="$PYTHONPATH":." and run
from test import main
main()
from python.
There are actually two problems:
First: The shared objects are also shared after dump/load only if they where pickled in one go (see also this answer).
That means you need to do the following (or similar):
...
with open('./temp.pickle', 'wb') as pfile:
pickle.dump((x,duplicates), pfile, protocol=pickle.HIGHEST_PROTOCOL)
...
with open('./temp.pickle', 'rb') as pfile:
nx, nd = pickle.load(pfile)
...
When you dump single objects, pickle isn't able to track identical objects - doing so would be an issue: objects with the same id between two dump-calls could be completely different objects or the same objects with different content!
Second: You should not create new objects, but pass the shared numpy-object in __reduce__ (pickle doesn't look inside of a numpy-array to see, whether buffer is shared or not, but only at the id of the array):
def __reduce__(self):
return self.__class__, (self.data.base,)
which will give you the desired result. data.base is a reference to the underlying original numpy-array (or whatever type, which must support pickling/unpickling, obviously).
Warning: As #DavidW has rightly pointed out, additional considerations must be taken into account, when working with sliced memory-views - because in this case base might not be "the same" as the actual memory view.
I'd like to call my cdef methods and improve the speed of my program as much as possible. I do not want to use cpdef (I explain why below). Ultimately, I'd like to access cdef methods (some of which return void) that are members of my Cython extensions.
I tried following this example, which gives me the impression that I can call a cdef function by making a Python (def) wrapper for it.
I can't reproduce these results, so I tried a different problem for myself (summing all the numbers from 0 to n).
Of course, I'm looking at the documentation, which says
The directive cpdef makes two versions of the method available; one fast for use from Cython and one slower for use from Python.
and later (emphasis mine),
This does slightly more than providing a python wrapper for a cdef method: unlike a cdef method, a cpdef method is fully overridable by methods and instance attributes in Python subclasses. It adds a little calling overhead compared to a cdef method.
So how does one use a cdef function without the extra calling overhead of a cpdef function?
With the code at the end of this question, I get the following results:
def/cdef:
273.04207632583245
def/cpdef:
304.4114626176919
cpdef/cdef:
0.8969507060538783
Somehow, cpdef is faster than cdef. For n < 100, I can occasionally get cpdef/cdef > 1, but it's rare. I think it has to do with wrapping the cdef function in a def function. This is what the example I link to does, but they claim better performance from using cdef than from using cpdef.
I'm pretty sure this is not how you wrap a cdef function while avoiding the additional overhead (the source of which is not clearly documented) of a cpdef.
And now, the code:
setup.py
from setuptools import setup, Extension
from Cython.Build import cythonize
pkg_name = "tmp"
compile_args=['-std=c++17']
cy_foo = Extension(
name=pkg_name + '.core.cy_foo',
sources=[
pkg_name + '/core/cy_foo.pyx',
],
language='c++',
extra_compile_args=compile_args,
)
setup(
name=pkg_name,
ext_modules=cythonize(cy_foo,
annotate=True,
build_dir='build'),
packages=[
pkg_name,
pkg_name + '.core',
],
)
foo.py
def foo_def(n):
sum = 0
for i in range(n):
sum += i
return sum
cy_foo.pyx
def foo_cdef(n):
return foo_cy(n)
cdef int foo_cy(int n):
cdef int sum = 0
cdef int i = 0
for i in range(n):
sum += i
return sum
cpdef int foo_cpdef(int n):
cdef int sum = 0
cdef int i = 0
for i in range(n):
sum += i
return sum
test.py
import timeit
from tmp.core.foo import foo_def
from tmp.core.cy_foo import foo_cdef
from tmp.core.cy_foo import foo_cpdef
n = 10000
# Python call
start_time = timeit.default_timer()
a = foo_def(n)
pyTime = timeit.default_timer() - start_time
# Call Python wrapper for C function
start_time = timeit.default_timer()
b = foo_cdef(n)
cTime = timeit.default_timer() - start_time
# Call cpdef function, which does more than wrap a cdef function (whatever that means)
start_time = timeit.default_timer()
c = foo_cpdef(n)
cpTime = timeit.default_timer() - start_time
print("def/cdef:")
print(pyTime/cTime)
print("def/cpdef:")
print(pyTime/cpTime)
print("cpdef/cdef:")
print(cpTime/cTime)
The reason for your seemingly anomalous result is that you aren't calling the cdef function foo_cy directly, but instead the def function foo_cdef wrapping it.
when you are wrapping inside another def indeed you are again calling the python function. However you should be able to reach similar results as the cpdef.
Here is what you could do:
like the python def, give the type for both input and output
def foo_cdef(int n):
cdef int val = 0
val = foo_cy(n)
return val
this should have similar results as cpdef, however again you are calling a python function. If you want to directly call the c function, you should use the ctypes and call from there.
and for the benchmarking, the way that you have written, it only considers one run and could fluctuate a lot due OS other task and as well the timer.
better to use the timeit builtin method to calculate for some iteration:
# Python call
pyTime = timeit.timeit('foo_def(n)',globals=globals(), number=10000)
# Call Python wrapper for C function
cTime = timeit.timeit('foo_cdef(n)',globals=globals(), number=10000)
# Call cpdef function, which does more than wrap a cdef function (whatever that means)
cpTime = timeit.timeit('foo_cpdef(n)',globals=globals(), number=10000)
output:
def/cdef:
154.0166154428522
def/cpdef:
154.22669848136132
cpdef/cdef:
0.9986378296327566
like this, you get consistent results and as well you see always close to 1 for both either cython itself wraps or we explicitly wrap around a python function.
I'm making a few experimentations using Cython to determine whether I should use it to speed up some parts of my Django modules.
Since I'm using a lot of strings and lists of strings, I tried this:
from libcpp.string cimport string
from libcpp.vector cimport vector
def cython_test_string():
cdef unsigned short int i = 0
cdef string s
# cdef vector[string] ls
for i in range(10000):
s.append('a')
# ls.push_back(s)
def python_test_string():
s = ''
# ls = []
for i in range(10000):
s += 'a'
# ls.append(s)
Of course, C++ strings are faster than Python objects, so we get these results (for 100 iterations):
Cython : 0.0110609531403 seconds
Python : 0.193608045578 seconds
But when we uncomment the four lines that deal with vector and list, we get this:
Cython : 2.80126094818 seconds
Python : 2.13021802902 seconds
Am I missing something, or are vectors of strings extremely slow?