Auto Conversion of Structs to Dicts in Cython - cython

So, if you've got a header file.
%%file test.h
struct mystruct{
int i;
int j;
};
And then you wrap it in Cython:
cdef extern from "test.h" nogil:
struct mystruct:
int i
int j
And some function that returns back out to Py:
def spit_out_dict():
return mystruct(5,10)
Cython correctly automatically generates a dict wrapper. However, when I wrap the original C header in a namespace, I haven't been able to get get Cython to still generate the dict wrapper correctly, something along these lines:
%%file test2.h
namespace outerspace{
struct mystruct{
int i;
int j;
};
}
And Cython/Python:
cdef extern from "test2.h" namespace "outerspace" nogil:
struct mynewstruct:
int i
int j
def spit_out_dict():
return mynewstruct(5,10)
This won't compile -- lots of namespace complaint errors -- anyone experienced this before?

Your problem is that Cython seems to only expect namespaces to be used with cppclass. For structs, it generates some functions but just copies the full namespaced name in, causing errors:
static PyObject* __pyx_convert__to_py_outerspace::mystruct(struct outerspace::mystruct s);
^
py_bit.cpp: In function ‘PyObject* __pyx_pf_6py_bit_spit_out_dict(PyObject*)’:
py_bit.cpp:721:15: error: ‘__pyx_convert__to_py_outerspace’ has not been declared
where it's trying to create a function called __pyx_convert__to_py_<classname>. (I think this might be worth submitting a bug report for.)
The trick in such circumstances is usually to lie to Cython. I create three files:
// test2.hpp
namespace outerspace{
struct mystruct{
int i;
int j;
};
}
,
// test2_cy.hpp - a wrapper file purely for Cython's benefit
#include "test2.hpp"
using outerpsace::mystruct;
and the cython file
cdef extern from "test2_cy.hpp": # (I didn't test with "nogil", but it's probably fine...)
struct mynewstruct:
int i
int j
def spit_out_dict():
# for some reason using "return mystruct(5,10)" doesn't work, but this does...
cdef mystruct a = mystruct(5,10)
return a

This is a bug in Cython, fixed at https://github.com/cython/cython/commit/fa946e8435a4dcc3497fc7b0f4e87256d40844ba

Related

difference in behaviour between declaration in .pyd and .pyx files

I have a pxd file that declares some types with ctypedef, and some functions with cdef. To call one particular function, I have to duplicate the typedefs in my pyx file.
If I move the cdef of the function into the pyx file it fails to compile with "Cannot convert 'state_t *' to Python object". The error refers to this line of code
retval = c_explanation.table_of_states_is_bad(<state_t*>test_array, b)
The test_array parameter is created locally with malloc so is not a python object at all.
If remove the typedefs from the pyx file and import them and the cdef from the pxd file with a from .. import *, I get the same compile error.
The only way that I can get this simple bit of code to work is with the rather awkward duplication of typedefs in both pyx and pxd files.
Any thoughts on what is going on here and how to resolve it?
The full code is pasted below
pyx file
from libc.stdlib cimport malloc, free
cimport c_explanation
cdef extern from "explanation.h":
ctypedef int state_id;
ctypedef struct event_t:
pass
ctypedef void (*event_handler)(event_t event)
ctypedef struct state_t:
state_id id
event_handler handle
char *name
ctypedef enum bool:
pass
def table_of_states_is_bad(a, b):
cdef state_t *test_array
cdef bool retval
test_array = <state_t*>malloc(len(a) * sizeof(state_t))
if not test_array:
raise MemoryError()
try:
for i, s in enumerate(a):
test_array[i].id = s[0]
retval = c_explanation.table_of_states_is_bad(test_array, b)
return retval
finally:
free(test_array)
pxd file
cdef extern from "explanation.h":
cdef int mock_get_temp(int)
ctypedef enum bool:
pass
ctypedef struct event_t:
pass
ctypedef void (*event_handler)(event_t event);
ctypedef int state_id;
ctypedef struct state_t:
state_id id
event_handler handle
char *name
cdef bool table_of_states_is_bad(state_t table[], size_t size)
finally the h file
int mock_get_temp(int);
typedef int event_id;
/* an event */
typedef struct {
event_id id; /* numeric id */
void *data; /* points to anything */
char *name; /* points to string for debug or log */
} event_t;
typedef void (*event_handler)(event_t event);
typedef enum { false, true } bool;
typedef int state_id;
/* a state */
typedef struct {
state_id id; /* numeric id */
event_handler handle; /* points to handler */
char *name; /* points to string for debug or log */
} state_t;
bool table_of_states_is_bad(state_t table[], size_t size);
After some reading and experimentation I worked out what I had done wrong.
The problem is one of namespaces. When using the import * format, the local definition of table_of_states_is_bad() was replacing the imported one. Thus the function call was to the cython version of the function rather than to the c version and hence the error was correctly reported.
I solved the problem by renaming the local function to wrap_table_of_states_is_bad() hence resolving the name conflict.

How to change the default code generated by SWIG for the allocation of memory for a C structure?

I am using a flexible array in the structure. So I want to change the memory allocated for that structure with some of my own code. Basically I want to change the new_structname() and structname_variable_set() functions.
typedef struct vector{
int x;
char y;
int arr[0];
} vector;
here, SWIG generated new_vector() function to allocate memory by calling calloc(1,sizeof(struct vector)) where swig will not handle these type of structure in a special manner. So we need to modify the swig generated new_vector() in order to allocate memory for the flexible array. So is there any way to handle this?
There are a few ways you can do this. What you're looking for though is %extend. That lets us define new constructors and implement them as we see fit. (It even works with a C compiler, they're only constructors from the perspective of the target language).
Using your vector as a starting point we can illustrate this:
%module test
%include <stdint.i>
%inline %{
typedef struct vector{ int x; char y; int arr[0]; }vector;
%}
%extend vector {
vector(const size_t len) {
vector *v = calloc(1, sizeof *v + len);
v->x = len;
return v;
}
}
With this SWIG synthesises a new_vector function in the generated module code as you'd hoped.
I also assumed that you want to record the length inside the struct as one of its members. If that's not the case you can simply delete the assignment I made.

Accessing C arrays in cython wrapped C code from Python

I have successfully wrapped some of the api functions of a C library with Cython. I would now like to expose some of the global C arrays used within the library but I am having a tough time figuring out how to do it.
The array pointer is initialized and allocated inside epanet.c:
void initpointers()
{
H = NULL;
}
int allocdata()
{
n = MaxNodes + 1;
H = (double *) calloc(n, sizeof(double));
}
It is made global in a header file (vars.h):
extern double *H;
I can access the values in the array within the C code of course by doing something like:
for (i=1; i<=Nnodes; i++)
{
H[i] = some_value;
}
I would like to write a function in my pyepanet.pyx file like:
def printH(int Nnodes, double *H):
for i in range(1, Nnodes+1):
print H[i]
but, I am missing the necessary connection between C and cython/python. I have searched quite a bit for an answer but with no luck. Please help.
I finally found something that works. It seems that I can declare the pointer without referring to the header file. When I was trying to refer to the header file before, I think I was getting errors due to redefining items. Here is the code from my .pyx file that works now:
cdef:
extern double *H
extern int Nnodes
def printH():
cdef int i
for i in range(1,Nnodes+1):
print round(H[i], 2)

getting struct elements in Cython

Amazingly I can't seem to find a single example of getting elements of a struct, by name (both on the web and in the cython examples).
So I'm receiving a pointer to a struct out of a C function, and would want to access those elements one by one and repackage them into a python list/dict.
maybe:
structPointer['propertyName']
or
structPointer.propertyName
I want to get the effect of structName->propertyName.
Your second syntax is the correct one, but you have to have an extern declaration for the struct type:
cdef extern from "someheader.h":
struct properties_t:
int value1
int value2
properties_t* getthem()
cdef void foo():
cdef properties_t* prop
prop = getthem()
i = prop.value1

C++: Explicit DLL Loading: First-chance Exception on non "extern C" functions

I am having trouble importing my C++ functions. If I declare them as C functions I can successfully import them. When explicit loading, if any of the functions are missing the extern as C decoration I get a the following exception:
First-chance exception at 0x00000000 in cpp.exe: 0xC0000005: Access violation.
DLL.h:
extern "C" __declspec(dllimport) int addC(int a, int b);
__declspec(dllimport) int addCpp(int a, int b);
DLL.cpp:
#include "DLL.h"
int addC(int a, int b) {
return a + b;
}
int addCpp(int a, int b) {
return a + b;
}
main.cpp:
#include "..DLL/DLL.h"
#include <stdio.h>
#include <windows.h>
int main() {
int a = 2;
int b = 1;
typedef int (*PFNaddC)(int,int);
typedef int (*PFNaddCpp)(int,int);
HMODULE hDLL = LoadLibrary(TEXT("../Debug/DLL.dll"));
if (hDLL != NULL)
{
PFNaddC pfnAddC = (PFNaddC)GetProcAddress(hDLL, "addC");
PFNaddCpp pfnAddCpp = (PFNaddCpp)GetProcAddress(hDLL, "addCpp");
printf("a=%d, b=%d\n", a,b);
printf("pfnAddC: %d\n", pfnAddC(a,b));
printf("pfnAddCpp: %d\n", pfnAddCpp(a,b)); //EXCEPTION ON THIS LINE
}
getchar();
return 0;
}
How can I import c++ functions for dynamic loading? I have found that the following code works with implicit loading by referencing the *.lib, but I would like to learn about dynamic loading.
Thank you to all in advance.
Update:
bindump /exports
1 00011109 ?addCpp##YAHHH#Z = #ILT+260(?addCpp##YAHHH#Z)
2 00011136 addC = #ILT+305(_addC)
Solution:
Create a conversion struct as
found here
Take a look at the
file exports and copy explicitly the
c++ mangle naming convention.
PFNaddCpp pfnAddCpp = (PFNaddCpp)GetProcAddress(hDLL, "?addCpp##YAHHH#Z");
Inevitably, the access violation on the null pointer is because GetProcAddress() returns null on error.
The problem is that C++ names are mangled by the compiler to accommodate a variety of C++ features (namespaces, classes, and overloading, among other things). So, your function addCpp() is not really named addCpp() in the resulting library. When you declare the function with extern "C", you give up overloading and the option of putting the function in a namespace, but in return you get a function whose name is not mangled, and which you can call from C code (which doesn't know anything about name mangling.)
One option to get around this is to export the functions using a .def file to rename the exported functions. There's an article, Explicitly Linking to Classes in DLLs, that describes what is necessary to do this.
It's possible to just wrap a whole header file in extern "C" as follows. Then you don't need to worry about forgetting an extern "C" on one of your declarations.
#ifdef __cplusplus
extern "C" {
#endif
__declspec(dllimport) int addC(int a, int b);
__declspec(dllimport) int addCpp(int a, int b);
#ifdef __cplusplus
} /* extern "C" */
#endif
You can still use all of the C++ features that you're used to in the function bodies -- these functions are still C++ functions -- they just have restrictions on the prototypes to make them compatible with C code.