difference in behaviour between declaration in .pyd and .pyx files - cython

I have a pxd file that declares some types with ctypedef, and some functions with cdef. To call one particular function, I have to duplicate the typedefs in my pyx file.
If I move the cdef of the function into the pyx file it fails to compile with "Cannot convert 'state_t *' to Python object". The error refers to this line of code
retval = c_explanation.table_of_states_is_bad(<state_t*>test_array, b)
The test_array parameter is created locally with malloc so is not a python object at all.
If remove the typedefs from the pyx file and import them and the cdef from the pxd file with a from .. import *, I get the same compile error.
The only way that I can get this simple bit of code to work is with the rather awkward duplication of typedefs in both pyx and pxd files.
Any thoughts on what is going on here and how to resolve it?
The full code is pasted below
pyx file
from libc.stdlib cimport malloc, free
cimport c_explanation
cdef extern from "explanation.h":
ctypedef int state_id;
ctypedef struct event_t:
pass
ctypedef void (*event_handler)(event_t event)
ctypedef struct state_t:
state_id id
event_handler handle
char *name
ctypedef enum bool:
pass
def table_of_states_is_bad(a, b):
cdef state_t *test_array
cdef bool retval
test_array = <state_t*>malloc(len(a) * sizeof(state_t))
if not test_array:
raise MemoryError()
try:
for i, s in enumerate(a):
test_array[i].id = s[0]
retval = c_explanation.table_of_states_is_bad(test_array, b)
return retval
finally:
free(test_array)
pxd file
cdef extern from "explanation.h":
cdef int mock_get_temp(int)
ctypedef enum bool:
pass
ctypedef struct event_t:
pass
ctypedef void (*event_handler)(event_t event);
ctypedef int state_id;
ctypedef struct state_t:
state_id id
event_handler handle
char *name
cdef bool table_of_states_is_bad(state_t table[], size_t size)
finally the h file
int mock_get_temp(int);
typedef int event_id;
/* an event */
typedef struct {
event_id id; /* numeric id */
void *data; /* points to anything */
char *name; /* points to string for debug or log */
} event_t;
typedef void (*event_handler)(event_t event);
typedef enum { false, true } bool;
typedef int state_id;
/* a state */
typedef struct {
state_id id; /* numeric id */
event_handler handle; /* points to handler */
char *name; /* points to string for debug or log */
} state_t;
bool table_of_states_is_bad(state_t table[], size_t size);

After some reading and experimentation I worked out what I had done wrong.
The problem is one of namespaces. When using the import * format, the local definition of table_of_states_is_bad() was replacing the imported one. Thus the function call was to the cython version of the function rather than to the c version and hence the error was correctly reported.
I solved the problem by renaming the local function to wrap_table_of_states_is_bad() hence resolving the name conflict.

Related

ctypedef a cython function with default arguments

I have a cython function like:
cdef void foo(int a, int b=1, int c=2):
pass
and a getter function to return its address:
def get_foo():
return <size_t>foo
so I can get function foo's address somewhere and cast it back to the real function in cython(eg: a callback one can use in python).
The problem is how to write such a type?
I tried:
cdef void foo(int a, int b=1, int c=2):
pass
ctypedef void (*foo_type)(int a, int b, int c)
cdef foo_type f = foo
this won't compile, cython complains: Cannot assign type 'void (int, struct __pyx_opt_args_46_cython_magic_49f265438c694830523a60bef4fe2ee8_foo *__pyx_optional_args)' to 'foo_type '
From the error mesage , one can note that, the option(default) arguments are wrapped in a struct by cython.
Is there a way to do such ctypedef in cython? If not, I think I'd better leave out the default value.:-(
You can create small wrappers without the default values:
cdef void foo_abc(int a, int b, int c):
foo(a,b,c)
# if you need it
cdef void foo_ab(int a, int b):
foo(a,b)
# etc
These will be convertable to function pointers:
cdef foo_type f = foo_abc
The speed loss from the wrapper functions will be small, so the main cost is just that you need to write a bit more code.

Auto Conversion of Structs to Dicts in Cython

So, if you've got a header file.
%%file test.h
struct mystruct{
int i;
int j;
};
And then you wrap it in Cython:
cdef extern from "test.h" nogil:
struct mystruct:
int i
int j
And some function that returns back out to Py:
def spit_out_dict():
return mystruct(5,10)
Cython correctly automatically generates a dict wrapper. However, when I wrap the original C header in a namespace, I haven't been able to get get Cython to still generate the dict wrapper correctly, something along these lines:
%%file test2.h
namespace outerspace{
struct mystruct{
int i;
int j;
};
}
And Cython/Python:
cdef extern from "test2.h" namespace "outerspace" nogil:
struct mynewstruct:
int i
int j
def spit_out_dict():
return mynewstruct(5,10)
This won't compile -- lots of namespace complaint errors -- anyone experienced this before?
Your problem is that Cython seems to only expect namespaces to be used with cppclass. For structs, it generates some functions but just copies the full namespaced name in, causing errors:
static PyObject* __pyx_convert__to_py_outerspace::mystruct(struct outerspace::mystruct s);
^
py_bit.cpp: In function ‘PyObject* __pyx_pf_6py_bit_spit_out_dict(PyObject*)’:
py_bit.cpp:721:15: error: ‘__pyx_convert__to_py_outerspace’ has not been declared
where it's trying to create a function called __pyx_convert__to_py_<classname>. (I think this might be worth submitting a bug report for.)
The trick in such circumstances is usually to lie to Cython. I create three files:
// test2.hpp
namespace outerspace{
struct mystruct{
int i;
int j;
};
}
,
// test2_cy.hpp - a wrapper file purely for Cython's benefit
#include "test2.hpp"
using outerpsace::mystruct;
and the cython file
cdef extern from "test2_cy.hpp": # (I didn't test with "nogil", but it's probably fine...)
struct mynewstruct:
int i
int j
def spit_out_dict():
# for some reason using "return mystruct(5,10)" doesn't work, but this does...
cdef mystruct a = mystruct(5,10)
return a
This is a bug in Cython, fixed at https://github.com/cython/cython/commit/fa946e8435a4dcc3497fc7b0f4e87256d40844ba

with emacs,I can't export source code in org file to html in highlight mode

I have taken htmlize in use, here is the configuration
(add-to-list 'load-path
"~/.emacs.d/plugins/")
(require 'htmlize)
(setq org-src-fontify-natively t)
(setq org-export-with-sub-superscripts nil)
with this configuation, After I export the org file below: I get the output like the picture at the bottom of this page.
#+STYLE: <style> pre.src { background-color:#1e3436; color: #eeeeec;}</style>
*tst code
#+BEGIN_SRC c
extern struct list_head llc_sap_list;
int a = 0;
#+END_SRC
#+begin_src c
#define LLC_DEST_INVALID 0 /* Invalid LLC PDU type */
#define LLC_DEST_SAP 1 /* Type 1 goes here */
#define LLC_DEST_CONN 2 /* Type 2 goes here */
extern spinlock_t llc_sap_list_lock;
extern int llc_rcv(struct sk_buff *skb, struct net_device *dev,
struct packet_type *pt, struct net_device *orig_dev);
extern int llc_mac_hdr_init(struct sk_buff *skb,
const unsigned char *sa, const unsigned char *da);
extern void llc_add_pack(int type, void (*handler)(struct llc_sap *sap,
struct sk_buff *skb));
extern void llc_remove_pack(int type);
extern void llc_set_station_handler(void (*handler)(struct sk_buff *skb));
extern struct llc_sap *llc_sap_open(unsigned char lsap,
int (*rcv)(struct sk_buff *skb,
struct net_device *dev,
struct packet_type *pt,
struct net_device *orig_dev));
static inline void llc_sap_hold(struct llc_sap *sap)
{
atomic_inc(&sap->refcnt);
}
extern void llc_sap_close(struct llc_sap *sap);
#+end_src
the output html, the color of the code is all in gray.
I can't upload it, but this problem really blocked me, in this whole night. thanks for your help on this.
Why are you setting this?
(setq org-src-fontify-natively t)
I think you want the opposite (which is the default). Try setting it to nil and htmlize should kick in.
In fact, I don't think you even need the
(require 'htmlize)
I have it installed from ELPA, so maybe that set up the autoload, but I don't have any reference to htmlize in my .emacs and it "just works."

SWIG and triggering a Python callback from C code

Apologies for not being familiar with formatting on here...I've made
some progress thanks to helpful replies and edited and removed my original
question to be replaced by the current one.
My problem lies with converting a C struct or struct pointer to PyObject. There
is no alternative to this because I am wrapping an existing C library whose
callback requires a C struct pointer.
Following works but with limitations:
%module cain1
%{
typedef struct {
double price;
int volume;
} book_entry_t;
typedef struct {
char symbol[10];
book_entry_t *book;
} trade_t;
typedef void (*CALLBACK)(trade_t trade);
CALLBACK my_callback = 0;
static PyObject *my_pycallback = NULL;
static void bigSnake(trade_t trade)
{
PyObject *result;
PyObject *d1;
result = PyEval_CallObject(my_pycallback,
Py_BuildValue("(y#)",
(char*)&trade,
sizeof(trade_t)
)
);
Py_XDECREF(result);
return /*void*/;
}
void test_cb (PyObject *callMe1) {
trade_t d1;
book_entry_t b1;
b1.price = 123.45;
b1.volume = 99;
Py_XINCREF(callMe1); /* Add a reference to new callback */
my_pycallback = callMe1; /* Remember new callback */
strcpy (d1.symbol,"Gupta Ltd");
d1.book = &b1;
bigSnake(d1);
}
%}
// Expose in python module..
typedef struct {
double price;
int volume;
} book_entry_t;
typedef struct {
char symbol[10];
book_entry_t *book;
} trade_t;
void test_cb(PyObject *callMe1);
and then triggering the callback from Python:
import cain1
import struct
def dave(d1):
N1,N2 = struct.unpack('10sP', d1)
print ('\n %s: %x' % (N1.decode() ,N2))
price,volume = struct.unpack('di',N2)
print (price,volume)
def main():
cain1.test_cb(dave)
main()
but I am unable to recover the book_entry_t strcut contents pointed to by trade_t....
I just feel this is all too convoluted since I have the pointer to structs and there
must be a straightforward way for Python to use that without any fuss.
Py_BuildValue("(N)",details) expects a PyObject* (your "N" says so), and you pass it something very different. Try Py_BuildValue("(i)", details.index) instead, and change it to accomodate any changes in details_t.
You're attempting to build a PyObject from a details_t struct. This isn't valid. Either pass the callback an integer (seems easier since details_t only has the one field) OR create a proper PyObject type. You can't blindly cast one type to another and expect it to work (a PyObject is more than just a pointer).

C++: Explicit DLL Loading: First-chance Exception on non "extern C" functions

I am having trouble importing my C++ functions. If I declare them as C functions I can successfully import them. When explicit loading, if any of the functions are missing the extern as C decoration I get a the following exception:
First-chance exception at 0x00000000 in cpp.exe: 0xC0000005: Access violation.
DLL.h:
extern "C" __declspec(dllimport) int addC(int a, int b);
__declspec(dllimport) int addCpp(int a, int b);
DLL.cpp:
#include "DLL.h"
int addC(int a, int b) {
return a + b;
}
int addCpp(int a, int b) {
return a + b;
}
main.cpp:
#include "..DLL/DLL.h"
#include <stdio.h>
#include <windows.h>
int main() {
int a = 2;
int b = 1;
typedef int (*PFNaddC)(int,int);
typedef int (*PFNaddCpp)(int,int);
HMODULE hDLL = LoadLibrary(TEXT("../Debug/DLL.dll"));
if (hDLL != NULL)
{
PFNaddC pfnAddC = (PFNaddC)GetProcAddress(hDLL, "addC");
PFNaddCpp pfnAddCpp = (PFNaddCpp)GetProcAddress(hDLL, "addCpp");
printf("a=%d, b=%d\n", a,b);
printf("pfnAddC: %d\n", pfnAddC(a,b));
printf("pfnAddCpp: %d\n", pfnAddCpp(a,b)); //EXCEPTION ON THIS LINE
}
getchar();
return 0;
}
How can I import c++ functions for dynamic loading? I have found that the following code works with implicit loading by referencing the *.lib, but I would like to learn about dynamic loading.
Thank you to all in advance.
Update:
bindump /exports
1 00011109 ?addCpp##YAHHH#Z = #ILT+260(?addCpp##YAHHH#Z)
2 00011136 addC = #ILT+305(_addC)
Solution:
Create a conversion struct as
found here
Take a look at the
file exports and copy explicitly the
c++ mangle naming convention.
PFNaddCpp pfnAddCpp = (PFNaddCpp)GetProcAddress(hDLL, "?addCpp##YAHHH#Z");
Inevitably, the access violation on the null pointer is because GetProcAddress() returns null on error.
The problem is that C++ names are mangled by the compiler to accommodate a variety of C++ features (namespaces, classes, and overloading, among other things). So, your function addCpp() is not really named addCpp() in the resulting library. When you declare the function with extern "C", you give up overloading and the option of putting the function in a namespace, but in return you get a function whose name is not mangled, and which you can call from C code (which doesn't know anything about name mangling.)
One option to get around this is to export the functions using a .def file to rename the exported functions. There's an article, Explicitly Linking to Classes in DLLs, that describes what is necessary to do this.
It's possible to just wrap a whole header file in extern "C" as follows. Then you don't need to worry about forgetting an extern "C" on one of your declarations.
#ifdef __cplusplus
extern "C" {
#endif
__declspec(dllimport) int addC(int a, int b);
__declspec(dllimport) int addCpp(int a, int b);
#ifdef __cplusplus
} /* extern "C" */
#endif
You can still use all of the C++ features that you're used to in the function bodies -- these functions are still C++ functions -- they just have restrictions on the prototypes to make them compatible with C code.