I'm chasing my tail with what I suspect is a simple problem, but I can't seem to find any explanation for the observed behavior. Assume I have a constant in a C header file defined by:
#define FOOBAR 128
typedef uint32_t mytype_t;
I convert this in Cython by putting the following in the .pxd file:
cdef int _FOOBAR "FOOBAR"
ctypedef uint32_t mytype_t
In my .pyx file, I have a declaration:
FOOBAR = _FOOBAR
followed later in a class definition:
cdef class MyClass:
cdef mytype_t myvar
def __init__(self):
try:
self.myvar = FOOBAR
print("GOOD")
except:
print("BAD")
I then try to execute this with a simple program:
try:
foo = MyClass()
except:
print("FAILED TO CREATE CLASS")
Sadly, this errors out, but I don't get an error message - I just get the exception print output:
BAD
Any suggestions on root cause would be greatly appreciated.
I believe I have finally tracked it down. The root cause issue is that FOOBAR in my code was actually set to UINT32MAX. Apparently, Cython/Python interprets that as a -1 and Python then rejects setting a uint32_t variable equal to it. The solution is to define FOOBAR to be 0xffffffff - apparently Python thinks that is a non-negative value and accepts it.
Related
I am trying to use a c DLL in cython and during the compilation phase I get a warning from the C compiler :
warning C4133: "=" : incompatible types - from 'foobar *' to 'foobar *'.
My pxd looks like this :
#!/usr/bin/env python3
#cython: language_level=3
cdef extern from "typedef.h"
struct foobar:
long *index
double *my_array
int value
cdef extern from "functions.h"
foobar *get_foobar(char *name);
And my pyx like that :
cimport pxd_file_name
cdef class Handler:
cdef pxd_file_name.foobar *__foobar
def load_foobar(self, char *name):
self.__foobar = pxd_file_name.get_foobar(name) <==
def another_method(self):
pass
I got the warning because of the line marked by an arrow and I don't understand why.
Is there a way to fix this ?
I manage to found my mistake.
Because in my .h file, my struct was declared using typedef, I had to write ctypedef struct foobar instead of struct foobar in my pxd file
I have a some Cython code where if a variable equals a value from a list then values from another list are copied into a testing array.
double [:] signals
cdef int total_days=signals.shape[0]
cdef size_t epoch=0
cdef int total_animals
cdef int n
cdef double[:] animal_signals
for animal in range(total_animals):
individual_animal = uniq_instr[animal]
for element in range(total_days):
if list(animal_ids[n]) == individual_animal:
animal_signals.append(signals[n])
I am getting an error:
UnboundLocalError: local variable 'animal_signals' referenced before assignment
I have thought having the line
cdef double[:] animal_signals
would have meant the array was assigned.
Update
As suggested I have also tried declaring the array animal_signals (and removing the append):
cdef int total_days=signals.shape[0]
cdef size_t epoch=0
cdef int total_animals
cdef int n
cdef int count=0
for animal in range(total_animals):
count=0
individual_animal = uniq_instr[animal]
for element in range(total_days):
if list(animal_ids[element]) == individual_animal:
cdef double[:] animal_signals[count] = signals[n]
count=count+1
however when I compile the code I get the error:
Error compiling Cython file:
------------------------------------------------------------
...
for element in range(total_days):
if list(animal_ids[element]) == individual_animal:
cdef double[:] animal_signals[count] = signals[n]
^
------------------------------------------------------------
project/temps.pyx:288:21: cdef statement not allowed here
Where am I going wrong?
Indeed, your line cdef double[:] animal_signals
declares animal_signals as a variable, but you never assign anything to it before using it (in Python assignement is done with the = operator).
In Cython, using the slice ([:]) notation when defining a variable is usually done to get the memory view of an other object (see the reference documentation).
For example :
some_1d_numpy_array = np.zeros((10,10)).reshape(-1)
cdef double[:] animal_signals = some_1d_numpy_array
If you want to create a C array, you have to allocate the memory for it (here for a size of number entries containing double) :
cdef double *my_array = <double *> malloc(number * sizeof(double))
Also, regarding to your original code, note that in both case you won't be able to use the append method on this object because it will not be a Python list, you will have to access its member by their indexes.
I have a C-header file (let's call it myheader.h) that contains some character string definitions such as:
#define MYSTRING "mystring-constant"
In Cython, I create a cmy.pxd file that contains:
cdef extern from "myheader.h":
cdef const char* MYSTRING "MYSTRING"
and a corresponding my.pyx file that contains some class definitions, all headed by:
from cmy cimport *
I then try to reference that string in a Python script:
from my import *
def main():
print("CONSTANT ", MYSTRING)
if __name__ == '__main__':
main()
Problem is that I keep getting an error:
NameError: name 'MYSTRING' is not defined
I've searched the documentation and can't identify the problem. Any suggestions would be welcomed - I confess it is likely something truly silly.
You cannot access cdef-variables from Python. So you have to create a Python object which would correspond to your define, something like this (it uses Cython>=0.28-feature verbatim-C-code, so you need a recent Cython version to run the snippet):
%%cython
cdef extern from *:
"""
#define MYSTRING "mystring-constant"
"""
# avoid name clash with Python-variable
# in cdef-code the value can be accessed as MYSTRING_DEFINE
cdef const char* MYSTRING_DEFINE "MYSTRING"
#python variable, can be accessed from Python
#the data is copied from MYSTRING_DEFINE
MYSTRING = MYSTRING_DEFINE
and now MYSTRING is a bytes-object:
>>> print(MYSTRING)
b'mystring-constant'
I am trying to convert a list of python strings to a 2D character array, and then pass it into a C function.
Python version: 3.6.4, Cython version: 0.28.3, OS Ubuntu 16.04
My first try looks like this:
def my_function(name_list):
cdef char name_array[50][30]
for i in range(len(name_list)):
name_array[i] = name_list[i]
The code builds, but during runtime I receive the following response:
Traceback (most recent call last):
File "test.py", line 532, in test_my_function
my_function(name_list)
File "my_module.pyx", line 817, in my_module.my_function
File "stringsource", line 93, in
carray.from_py.__Pyx_carray_from_py_char
IndexError: not enough values found during array assignment, expected 25, got 2
I then tried to make sure that the string on the right-hand side of the assignment is exactly 30 characters by doing the following:
def my_function(name_list):
cdef char name_array[50][30]
for i in range(len(name_list)):
name_array[i] = (name_list[i] + ' '*30)[:30]
This caused another error, as follows:
Traceback (most recent call last):
File "test.py", line 532, in test_my_function
my_function(name_list)
File "my_module.pyx", line 818, in my_module.my_function
File "stringsource", line 87, in carray.from_py.__Pyx_carray_from_py_char
TypeError: an integer is required
I will appreciate any help. Thanks.
I don't like this functionality of Cython and seems to be at least not very well thought trough:
It is convenient to use char-array and thus to avoid the hustle with allocating/freeing of dynamically allocated memory. However, it is only natural that the allocated buffer is larger than the strings for which it is used. Enforcing equal lengths doesn't make sense.
C-strings are null-terminated. Not always is \0 at the end needed, but often it is necessary, so some additional steps are needed to ensure this.
Thus, I would roll out my own solution:
%%cython
from libc.string cimport memcpy
cdef int from_str_to_chararray(source, char *dest, size_t N, bint ensure_nullterm) except -1:
cdef size_t source_len = len(source)
cdef bytes as_bytes = source.encode('ascii') #hold reference to the underlying byte-object
cdef const char *as_ptr = <const char *>(as_bytes)
if ensure_nullterm:
source_len+=1
if source_len > N:
raise IndexError("destination array too small")
memcpy(dest, as_ptr, source_len)
return 0
and then use it as following:
%%cython
def test(name):
cdef char name_array[30]
from_str_to_chararray(name, name_array, 30, 1)
print("In array: ", name_array)
A quick test yields:
>>> tests("A")
In array: A
>>> test("A"*29)
In array: AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
>>> test("A"*30)
IndexError: destination array too small
Some additional remarks to the implementation:
it is necessary to hold the reference of the underlying bytes object, to keep it alive, otherwise as_ptr will become dangling as soon as it is created.
internal representation of bytes-objects has a trailing \0, so memcpy(dest, as_ptr, source_len) is safe even if source_len=len(source)+1.
except -1 in the signature is needed, so the exception is really passed to/checked in Python code.
Obviously, not everything is perfect: one has to pass the size of the array manually and this will leads to errors in the long run - something Cython's version does automatically right. But given the lacking functionality in Cython's version right now, the roll-out version is the better option in my opinion.
Thanks to #ead for responding. It got me to something that works. I am not convinced that it is the best way, but for now it is OK.
I addressed null termination, as #ead suggested, by appending null characters.
I received a TypeError: string argument without an encoding error, and had to encode the string before converting it to a bytearray. That is what the added .encode('ASCII') bit is for.
Here is the working code:
def my_function(name_list):
cdef char name_array[50][30]
for i in range(len(name_list)):
name_array[i] = bytearray((name_list[i] + '\0'*30)[:30].encode('ASCII'))
This seems like a question that should have an obvious answer, but for some reason I can't find any examples online.
I am wrapping a vector of C++ objects in a Python class using Cython. I also have a Cython wrapper for the C++ class already coded. I can get several methods such as __len__(), __getitem__(), and resize() to work properly, but the __setitem__() method is giving me problems.
For simplicity, I coded a small example using a vector of ints. I figure if I can get this code to work, then I can build on that to get the solution for my C++ class as well.
MyPyModule.pyx
# distutils: language = c++
from libcpp.vector cimport vector
from cython.operator cimport dereference as deref
cdef class MyArray:
cdef vector[int]* thisptr
def __cinit__(self):
self.thisptr = new vector[int]()
def __dealloc__(self):
del self.thisptr
def __len__(self):
return self.thisptr.size()
def __getitem__(self, size_t key):
return self.thisptr.at(key)
def resize(self, size_t newsize):
self.thisptr.resize(newsize)
def __setitem__(self, size_t key, int value):
# Attempt 1:
# self.thisptr.at(key) = value
# Attempt 2:
# cdef int* itemptr = &(self.thisptr.at(key))
# itemptr[0] = value
# Attempt 3:
# (self.thisptr)[key] = value
# Attempt 4:
self[key] = value
When I tried to cythonize using Attempt 1, I got the error Cannot assign to or delete this. When I tried Attempt 2, the .cpp file was created, but the compiler complained that:
error: cannot convert ‘__Pyx_FakeReference<int>*’ to ‘int*’ in assignment
__pyx_v_itemptr = (&__pyx_t_1);
On Attempt 3, Cython would not build the file because Cannot assign type 'int' to 'vector[int]'. (When I tried this style with the C++ object instead of int, it complained because I had a reference as a left-value.) Attempt 4 compiles, but when I try to use it, I get a segfault.
Cython docs say that returning a reference as a left-value is not supported, which is fine -- but how do I get around it so that I can assign a new value to one of my vector elements?
There are two ways to access the vector through a pointer,
def __setitem__(self, size_t key, int value):
deref(self.thisptr)[key] = value
# or
# self.thisptr[0][key] = value
Cython translates those two cases as follows:
Python: deref(self.thisptr)[key] = value
C++: ((*__pyx_v_self->thisptr)[__pyx_v_key]) = __pyx_v_value;
Python: self.thisptr[0][key] = value
C++: ((__pyx_v_self->thisptr[0])[__pyx_v_key]) = __pyx_v_value;
which are equivalent i.e. access the same vector object.
Instead of trying to handle a pointer from Cython code, you can let Cython itself do it for you:
cdef class MyArray:
cdef vector[int] thisptr
def __len__(self):
return self.thisptr.size()
def __getitem__(self, size_t key):
return self.thisptr[key]
def __setitem__(self, size_t key, int value):
self.thisptr[key] = value
def resize(self, size_t newsize):
self.thisptr.resize(newsize)
Is there any problem with this approach?
I have already accepted J.J. Hakala's answer (many thanks!). I tweaked that method to include an out-of-bounds check, since it uses the [] operator instead of the at() method:
cdef class MyArray:
(....)
def __setitem__(self, size_t key, int value):
if key < self.thisptr.size():
deref(self.thisptr)[key] = value
else:
raise IndexError("Index is out of range.")