Argument passing in Python module written in C - python-c-extension

I have been getting my feet wet with writing Python modules in C. I've started with just a simple example of computing the norm of two points. The code looks like this,
_norm.c
#include <Python.h>
#include "norm.h"
static char module_docstring[] =
"This module provides an interface for computing the norm using C.";
static char norm_docstring[] =
"Calculate the norm between two points";
static char norm2_docstring[] =
"Calculate the square norm between two points. For efficiency.";
static PyObject *norm_norm(PyObject *self, PyObject *args);
static PyObject *norm_norm2(PyObject *self, PyObject *args);
static PyMethodDef module_methods[] = {
{"norm", norm_norm, METH_VARARGS, norm_docstring},
{"norm2", norm_norm2, METH_VARARGS, norm2_docstring},
{NULL, NULL, 0, NULL}
};
PyMODINIT_FUNC init_norm(void) {
PyObject *m = Py_InitModule3("_norm", module_methods, module_docstring);
if (m == NULL)
return;
}
static PyObject *norm_norm(PyObject *self, PyObject *args) {
double x1, y1, x2, y2;
/* Parse the input tuple */
if (!PyArg_ParseTuple(args, "ddOOO", &x1, &y1, &x2, &y2))
return NULL;
/* Call the external C function to compute the norm. */
double value = norm(x1, y1, x2, y2);
if (value < 0.0) {
PyErr_SetString(PyExc_RuntimeError,
"Norm returned an impossible value.");
}
PyObject *ret = Py_BuildValue("d", value);
return ret;
}
norm.c
#include <math.h>
long double norm2(long double x1, long double y1, long double x2, long double y2) {
long double xd = x2 - x1;
long double yd = y2 - y1;
return xd * xd + yd * yd;
}
long double norm(long double x1, long double y1, long double x2, long double y2) {
return sqrt(norm2(x1, y1, x2, y2));
}
setup.py
from distutils.core import setup, Extension
setup(
ext_modules=[Extension("_norm", ["_norm.c", "norm.c"])]
)
I build the package like this,
python setup.py build_ext --inplace
And it compiles with no problems. However, when I try to use it I get an error about the number of arguments.
>>> import _norm
>>> _norm.norm(1,2,5,6)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: function takes exactly 5 arguments (4 given)
I guess it might have something to do with the declaration of PyObject *norm_norm() because it gets passed *self, but I'm not sure if that should effect the arguments that I pass to the module on the Python side of things. I'd appreciate any help/suggestions.

The problem in the format string of PyArg_ParseTuple. You are trying to extract four double arguments, but the format string is for two double arguments and three arbitrary Python objects ("ddOOO").
The correct format string should be "dddd" for what you are trying to do.
Change this
if (!PyArg_ParseTuple(args, "ddOOO", &x1, &y1, &x2, &y2))
with
if (!PyArg_ParseTuple(args, "dddd", &x1, &y1, &x2, &y2))

Related

Cython: Workaround scipy.optimize.cython_optimize.brentq argument ctype to work with class methods ctype

I'm trying to use the root finder scipy.optimize.cython_optimize.brentq inside a class, but the first argument of this function accepts only the type ctypedef double (*callback_type)(double, void*) and my class method is the type of ctypedef double (*w_func)(test,double, void*).
How could I get this working?
The following code is an exemple of the problem.
%%cython
from scipy.optimize.cython_optimize cimport brentq
ctypedef double (*f_func)(test,double, double)
ctypedef double (*w_func)(test,double, void*)
ctypedef double (*callback_type)(double, void*)
ctypedef struct test_params:
double y0
double x1
f_func f
w_func w
cdef class test():
def __init__(self):
cdef test_params myargs
myargs.y0 = 1.0
myargs.x1 = 0.7
myargs.f = self.sum2
myargs.w = self.w
print(self.brentqWrapper(myargs, -10, 10, 1e-3, 1e-3, 10))
cdef double sum2(self, double x1,double y1):
return x1+y1
cdef double w(self, double y1, void *args):
cdef test_params *myargs = <test_params *> args
return y1 - myargs.y0 - myargs.f(self,myargs.x1,y1)
# Cython wrapper function
cdef double brentqWrapper(self,test_params args, double xa, double xb,
double xtol, double rtol, int mitr):
return brentq(args.w, xa, xb, <test_params *> &args, xtol, rtol, mitr, NULL)
test()
ISTM it's easiest and most clean to restructure your code to have a module-level function which receives the class instance in its last argument.

ctypes How to get address of NULL c_void_p field?

I need to get the address of a NULL void pointer. If I make a NULL c_void_p in Python I have no problem getting its address:
ptr = c_void_p(None)
print(ptr)
print(ptr.value)
print(addressof(ptr))
gives
c_void_p(None)
None
4676189120
But I have a
class Effect(structure):
_fields_ = [("ptr", c_void_p)]
where ptr gets initialized to NULL in C. When I access it in python
myclib.get_effect.restype = POINTER(Effect)
effect = myclib.get_effect().contents
print(effect.ptr)
gives None, so I can't take addressof(effect.ptr).
If I change my field type to a pointer to any ctype type
class Effect(structure):
_fields_ = [("ptr", POINTER(c_double)]
# get effect instance from C shared library
print(addressof(effect.ptr))
I have checked that I get the right address on the heap on the C side
140530973811664
Unfortunately, changing the field type from c_void_p is not an option. How can I do this?
Clarification
Here's C code following #CristiFati for my specific situation. struct is allocated in C, I get a ptr back to it in Python, and now I need to pass a reference to a ptr in the struct. First if I make the ptr a double, there's no problem!
#include <stdio.h>
#include <stdlib.h>
#define PRINT_MSG_2SX(ARG0, ARG1) printf("From C - [%s] (%d) - [%s]: ARG0: [%s], ARG1: 0x%016llX\n", __FILE__, __LINE__, __FUNCTION__, ARG0, (unsigned long long)ARG1)
typedef struct Effect {
double* ptr;
} Effect;
void print_ptraddress(double** ptraddress){
PRINT_MSG_2SX("Address of Pointer:", ptraddress);
}
Effect* get_effect(){
Effect* pEffect = malloc(sizeof(*pEffect));
pEffect->ptr = NULL;
print_ptraddress(&pEffect->ptr);
return pEffect;
}
And in Python
from ctypes import cdll, Structure, c_int, c_void_p, addressof, pointer, POINTER, c_double, byref
clibptr = cdll.LoadLibrary("libpointers.so")
class Effect(Structure):
_fields_ = [("ptr", POINTER(c_double))]
clibptr.get_effect.restype = POINTER(Effect)
pEffect = clibptr.get_effect()
effect = pEffect.contents
clibptr.print_ptraddress(byref(effect.ptr))
gives matching addresses:
From C - [pointers.c] (11) - [print_ptraddress]: ARG0: [Address of Pointer:], ARG1: 0x00007FC2E1AD3770
From C - [pointers.c] (11) - [print_ptraddress]: ARG0: [Address of Pointer:], ARG1: 0x00007FC2E1AD3770
But if I change the double* to void* and c_void_p, I get an error, because the c_void_p in python is set to None
ctypes ([Python 3]: ctypes - A foreign function library for Python) is meant to be able to "talk to" C from Python, which makes it Python friendly, and that means no pointers, memory addresses, ... whatsoever (well at least as possible, to be more precise).
So, under the hood, it does some "magic", which in this case stands between you and your goal.
#EDIT0: Updated the answer to better fit the (clarified) question.
Example:
>>> import ctypes
>>> s0 = ctypes.c_char_p(b"Some dummy text")
>>> s0, type(s0)
(c_char_p(2180506798080), <class 'ctypes.c_char_p'>)
>>> s0.value, "0x{:016X}".format(ctypes.addressof(s0))
(b'Some dummy text', '0x000001FBB021CF90')
>>>
>>> class Stru0(ctypes.Structure):
... _fields_ = [("s", ctypes.c_char_p)]
...
>>> stru0 = Stru0(s0)
>>> type(stru0)
<class '__main__.Stru0'>
>>> "0x{:016X}".format(ctypes.addressof(stru0))
'0x000001FBB050E310'
>>> stru0.s, type(stru0.s)
(b'Dummy text', <class 'bytes'>)
>>>
>>>
>>> b = b"Other dummy text"
>>> char_p = ctypes.POINTER(ctypes.c_char)
>>> s1 = ctypes.cast((ctypes.c_char * len(b))(*b), char_p)
>>> s1, type(s1)
(<ctypes.LP_c_char object at 0x000001FBB050E348>, <class 'ctypes.LP_c_char'>)
>>> s1.contents, "0x{:016X}".format(ctypes.addressof(s1))
(c_char(b'O'), '0x000001FBB050E390')
>>>
>>> class Stru1(ctypes.Structure):
... _fields_ = [("s", ctypes.POINTER(ctypes.c_char))]
...
>>> stru1 = Stru1(s1)
>>> type(stru1)
<class '__main__.Stru1'>
>>> "0x{:016X}".format(ctypes.addressof(stru1))
'0x000001FBB050E810'
>>> stru1.s, type(stru1.s)
(<ctypes.LP_c_char object at 0x000001FBB050E6C8>, <class 'ctypes.LP_c_char'>)
>>> "0x{:016X}".format(ctypes.addressof(stru1.s))
'0x000001FBB050E810'
This is a parallel between 2 types which in theory are the same thing:
ctypes.c_char_p: as you can see, s0 was automatically converted to bytes. This makes sense, since it's Python, and there's no need to work with pointers here; also it would be very annoying to have to convert each member from ctypes to plain Python (and viceversa), every time when working with it.
Current scenario is not part of the "happy flow", it's rather a corner case and there's no functionality for it (or at least I'm not aware of any)
ctypes.POINTER(ctypes.c_char) (named it char_p): This is closer to C, and offers the functionality you needed, but as seen it's also much harder (from Python perspective) to work with it
The problem is that ctypes.c_void_p is similar to #1., so there's no OOTB functionality for what you want, and also there's no ctypes.c_void to go with #2.. However, it is possible to do it, but additional work is required.
The well known (C) rule is: AddressOf(Structure.Member) = AddressOf(Structure) + OffsetOf(Structure, Member) (beware of memory alignment who can "play dirty tricks on your mind").
For this particular case, things couldn't be simpler. Here's an example:
dll.c:
#include <stdio.h>
#include <stdlib.h>
#if defined(_WIN32)
# define DLL_EXPORT __declspec(dllexport)
#else
# define DLL_EXPORT
#endif
#define PRINT_MSG_2SX(ARG0, ARG1) printf("From C - [%s] (%d) - [%s]: ARG0: [%s], ARG1: 0x%016llX\n", __FILE__, __LINE__, __FUNCTION__, ARG0, (unsigned long long)ARG1)
static float f = 1.618033;
typedef struct Effect {
void *ptr;
} Effect;
DLL_EXPORT void test(Effect *pEffect, int null) {
PRINT_MSG_2SX("pEffect", pEffect);
PRINT_MSG_2SX("pEffect->ptr", pEffect->ptr);
PRINT_MSG_2SX("&pEffect->ptr", &pEffect->ptr);
pEffect->ptr = !null ? NULL : &f;
PRINT_MSG_2SX("new pEffect->ptr", pEffect->ptr);
}
code.py:
#!/usr/bin/env python3
import sys
from ctypes import CDLL, POINTER, \
Structure, \
c_int, c_void_p, \
addressof, pointer
DLL = "./dll.dll"
class Effect(Structure):
_fields_ = [("ptr", c_void_p)]
def hex64_str(item):
return "0x{:016X}".format(item)
def print_addr(ctypes_inst, inst_name, heading=""):
print("{:s}{:s} addr: {:s} (type: {:})".format(heading, "{:s}".format(inst_name) if inst_name else "", hex64_str(addressof(ctypes_inst)), type(ctypes_inst)))
def main():
dll_dll = CDLL(DLL)
test_func = dll_dll.test
test_func.argtypes = [POINTER(Effect), c_int]
effect = Effect()
print_addr(effect, "effect")
test_func(pointer(effect), 1)
print(effect.ptr, type(effect.ptr)) # Not helping, it's Python int for c_void_p
try:
print_addr(effect.ptr, "effect.ptr")
except:
print("effect.ptr: - wrong type")
print_addr(effect, "effect", "\nSecond time...\n ")
print("Python addrs (irrelevant): effect: {:s}, effect.ptr: {:s}".format(hex64_str(id(effect)), hex64_str(id(effect.ptr))))
if __name__ == "__main__":
print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
main()
Output:
(py35x64_test) e:\Work\Dev\StackOverflow\q053531795>call "c:\Install\x86\Microsoft\Visual Studio Community\2015\vc\vcvarsall.bat" x64
(py35x64_test) e:\Work\Dev\StackOverflow\q053531795>dir /b
code.py
dll.c
(py35x64_test) e:\Work\Dev\StackOverflow\q053531795>cl /nologo /DDLL /MD dll.c /link /NOLOGO /DLL /OUT:dll.dll
dll.c
Creating library dll.lib and object dll.exp
(py35x64_test) e:\Work\Dev\StackOverflow\q053531795>dir /b
code.py
dll.c
dll.dll
dll.exp
dll.lib
dll.obj
(py35x64_test) e:\Work\Dev\StackOverflow\q053531795>"e:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" code.py
Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32
effect addr: 0x000001FB25B8CB10 (type: <class '__main__.Effect'>)
From C - [dll.c] (21) - [test]: ARG0: [pEffect], ARG1: 0x000001FB25B8CB10
From C - [dll.c] (22) - [test]: ARG0: [pEffect->ptr], ARG1: 0x0000000000000000
From C - [dll.c] (23) - [test]: ARG0: [&pEffect->ptr], ARG1: 0x000001FB25B8CB10
From C - [dll.c] (25) - [test]: ARG0: [new pEffect->ptr], ARG1: 0x00007FFFAFB13000
140736141012992 <class 'int'>
effect.ptr: - wrong type
Second time...
effect addr: 0x000001FB25B8CB10 (type: <class '__main__.Effect'>)
Python addrs (irrelevant): effect: 0x000001FB25B8CAC8, effect.ptr: 0x000001FB25BCC9F0
As seen, the address of effect is the same as the address of effect's ptr. But again, this is the simplest possible scenario. But, as explained a general solution, is preferred. However that's not possible, but it can be worked around:
Use the above formula and get the field offset using [SO]: Getting elements from ctype structure with introspection? (it's long, I had a hard time coming to the current solution - especially because of the 2 container types (Structure and Array) nesting possibilities; hopefully, it's bug free (or as close as possible) :) )
Modify the C interface to something like: Effect *get_effect(void **ptr), and store the address in the parameter
Modify the (Python) Effect structure, and instead of ctypes.c_void_p field have something that involves POINTER (e.g.: ("ptr", POINTER(c_ubyte))). The definition will differ from C, and semantically things are not OK, but at the end they're both pointers
Note: don't forget to have a function that destroys a pointer returned by get_effect (to avoid memory leaks)
So after raising this in the python bug tracker, Martin Panter and Eryk Sun provided a better solution.
There is indeed an undocumented offset attribute, which allows us to access the right location in memory without having to do any introspection. We can get back our pointer using
offset = type(Effect).ptr.offset
ptr = (c_void_p).from_buffer(effect, offset)
We can more elegantly wrap this into our class by using a private field and adding a property:
class Effect(Structure):
_fields_ = [("j", c_int),
("_ptr", c_void_p)]
#property
def ptr(self):
offset = type(self)._ptr.offset
return (c_void_p).from_buffer(self, offset)
I have added an integer field before our pointer so the offset isn't just zero. For completeness, here is the code above adapted with this solution showing that it works. In C:
#include <stdio.h>
#include <stdlib.h>
#define PRINT_MSG_2SX(ARG0, ARG1) printf("%s : 0x%016llX\n", ARG0, (unsigned long long)ARG1)
typedef struct Effect {
int j;
void* ptr;
} Effect;
void print_ptraddress(double** ptraddress){
PRINT_MSG_2SX("Address of Pointer:", ptraddress);
}
Effect* get_effect(){
Effect* pEffect = malloc(sizeof(*pEffect));
pEffect->ptr = NULL;
print_ptraddress(&pEffect->ptr);
return pEffect;
}
In Python (omitting the above Effect definition):
from ctypes import cdll, Structure, c_int, c_void_p, POINTER, byref
clibptr = cdll.LoadLibrary("libpointers.so")
clibptr.get_effect.restype = POINTER(Effect)
effect = clibptr.get_effect().contents
clibptr.print_ptraddress(byref(effect.ptr))
yields
Address of Pointer: : 0x00007F9EB248FB28
Address of Pointer: : 0x00007F9EB248FB28
Thanks again to everyone for quick suggestions. For more, see here:

Cython return tuple within cdef?

Hi I am trying to convert a python code into cython in order to speed up its calculation. I am trying to return multiple arrays within the cython code from a cdef to cpdef. Based on classical C, I could either use a pointer or a tuple. I decide to use tuple because the size varies. I know the following code doesn't work, any help? Thank you!
import numpy as np
cimport numpy as np
cdef tuple funA(double[:] X, double[:] Y):
cdef int nX, nY, i
nX = len(X)
nY = len(Y)
for i in range(nX):
X[i] = X[i]*X[i]
for i in range(nY):
Y[i] = Y[i]*Y[i]
return X,Y
cpdef Run(double[:] X, double[:] Y)
cdef Tuple1, Tuple2 = funA(X,Y)
# Do some calculation with Tuple1 and Tuple2
# Example
cdef int i, nTuple1, nTuple2
nTuple1 = len(Tuple1)
for i in range(nTuple1):
Tuple1[i] = Tuple1[i]**2
nTuple2 = len(Tuple2)
for i in range(nTuple2):
Tuple2[i] = Tuple2[i]/2
return Tuple1, Tuple2
You've got a few indentation errors and missing colons. But your real issue is:
cdef Tuple1, Tuple2 = funA(X,Y)
Remove the cdef and it's fine. It doesn't look like cdef and tuple unpacking quite mix, and since you're treating them as Python objects it should be OK.
However, note that you don't really need to return anything from funA since you modify X and Y them in place there.

Extending Python by compiling C code in Cython

I'm having trouble compiling a particular source file in C from an R package (fastclime) so that I can use it as a Python module. This is my code to extract the R package online:
from __future__ import division
import os
import sys
import glob
if not os.path.exists('fastlp.h'):
! wget http://cran.r-project.org/src/contrib/fastclime_1.2.4.tar.gz
! tar -xzvf fastclime_1.2.4.tar.gz
Within the R package (in the [src] folder) is the source file [parametric.c]. I'd like to compile this in Python as a module with numpy inputs. Since I can't immediately test the the functionality of this code, I first tried to compile the [fastlp.c] source file. With fastlp, I can create a toy example to test if I successfully compiled a version of fastlp in Python. Then I can probably extend the steps to compile [parametric.c]
Here is my attempt:
1) First I create the header file since it does not exist
%%file fastclime/src/fastlp.h
int ratio_test0(double *dy, int *idy,int ndy, double *y, double *ybar, double mu);
void fastlp(double *obj, double *mat, double *rhs, int *m0 , int *n0, double *opt, int *status, double *lam)
void solver20(int m,int n,int nz,int *ia, int *ka, double *a,double *b, double *c)
int ratio_test0(double *dy, int *idy,int ndy,double *y, double *ybar, double mu)
2) Next, I write the files for wrapping the C code in Cython:
%%file fastlp.pxd
cdef extern from "fastclime/src/fastlp.h":
void fastlp(double *obj, double *mat, double *rhs, int *m0 , int *n0, double *opt, int *status, double *lam)
%%file fastlp.pyx
cimport fastlp
def fastlp(double *obj, double *mat, double *rhs, int *m0 , int *n0, double *opt, int *status, double *lam):
fastlp.fastlp(*obj, *mat, *rhs, *m0, *n0, *opt, *status, *lam)
3) Next I use the distutils build system to compile.
%%file setup.py
from distutils.core import setup, Extension
from Cython.Build import cythonize
ext = Extension("fastlp",
sources=["fastlp.pyx", "fastclime/src/fastlp.c"])
setup(name="fastlp",
ext_modules = cythonize([ext]))
4) Finally I compile
! python setup.py build_ext -i
#4 is where things fail big time. Not sure how to resolve it in steps 1-3.
5) If all is successful (which it's not), I should be able to run this code (taken from the vignette)
import fastlp as flp
import numpy as np
A=np.array([-1.0,-1,0,1,-2,1]).reshape((3,2))
b=np.array([-1.0,-2,1])
c=np.array([-2.0,3])
flp.fastlp(c,A,b)
This should output the vector [2., 1.].
I'm not very experienced with Cython or Python for that matter, but getting this functionality would help out a lot with future projects. Could someone please let me know the proper way of taking a source file in C and getting through the steps in interfacing with numpy to create a Python module?
Thanks!
UPDATE:
I'm trying to follow the Numpy+Cython example on this site: https://scipy-lectures.github.io/advanced/interfacing_with_c/interfacing_with_c.html
I've since updated my code
%%cython
# import both numpy and the Cython declarations for numpy
import numpy as np
cimport numpy as np
# if you want to use the Numpy-C-API from Cython
# (not strictly necessary for this example)
np.import_array()
# cdefine the signature of our c function
cdef extern from "/home/bitnami/STA663-pura-project/fastclime/src/fastlp.h":
void fastlp(double *obj, double *mat, double *rhs, int *m0 , int *n0, double *opt, int *status, double *lam)
# create the wrapper code, with numpy type annotations
def fastlp_func(np.ndarray[double, ndim=1, mode="c"] obj not None,
np.ndarray[double, ndim=2, mode="c"] mat not None,
np.ndarray[double, ndim=1, mode="c"] rhs not None,
double lam):
cdef int m1 = mat.shape[0]
cdef int m2 = mat.shape[1]
cdef np.ndarray[double, ndim=1, mode='c'] z = np.zeros(obj.shape[0])
cdef double [:] zp = z #convert from Python to C object
cdef int stat = 0
cdef double lambd = 0
fastlp(<double*> np.PyArray_DATA(obj),
<double*> np.PyArray_DATA(mat.T),
<double*> np.PyArray_DATA(rhs),
&m1,
&m2,
&zp[0],
&stat,
&lambd
)
However, I get the following error (which I don't understand at all):
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-85-29981393af41> in <module>()
----> 1 get_ipython().run_cell_magic(u'cython', u'', u'# import both numpy and the Cython declarations for numpy\nimport numpy as np\ncimport numpy as np\n\n# if you want to use the Numpy-C-API from Cython\n# (not strictly necessary for this example)\nnp.import_array()\n\n# cdefine the signature of our c function\ncdef extern from "/home/bitnami/STA663-pura-project/fastclime/src/fastlp.h":\n void fastlp(double *obj, double *mat, double *rhs, int *m0 , int *n0, double *opt, int *status, double *lam)\n \n# create the wrapper code, with numpy type annotations\ndef fastlp_func(np.ndarray[double, ndim=1, mode="c"] obj not None,\n np.ndarray[double, ndim=2, mode="c"] mat not None,\n np.ndarray[double, ndim=1, mode="c"] rhs not None,\n double lam):\n cdef int m1 = mat.shape[0]\n cdef int m2 = mat.shape[1]\n cdef np.ndarray[double, ndim=1, mode=\'c\'] z = np.zeros(obj.shape[0])\n cdef double [:] zp = z #convert from Python to C object\n cdef int stat = 0\n cdef double lambd = 0\n fastlp(<double*> np.PyArray_DATA(obj),\n <double*> np.PyArray_DATA(mat.T),\n <double*> np.PyArray_DATA(rhs),\n &m1,\n &m2,\n &zp[0], \n &stat, \n &lambd \n )')
/home/bitnami/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in run_cell_magic(self, magic_name, line, cell)
2160 magic_arg_s = self.var_expand(line, stack_depth)
2161 with self.builtin_trap:
-> 2162 result = fn(magic_arg_s, cell)
2163 return result
2164
/home/bitnami/anaconda/lib/python2.7/site-packages/IPython/extensions/cythonmagic.pyc in cython(self, line, cell)
/home/bitnami/anaconda/lib/python2.7/site-packages/IPython/core/magic.pyc in <lambda>(f, *a, **k)
191 # but it's overkill for just that one bit of state.
192 def magic_deco(arg):
--> 193 call = lambda f, *a, **k: f(*a, **k)
194
195 if callable(arg):
/home/bitnami/anaconda/lib/python2.7/site-packages/IPython/extensions/cythonmagic.pyc in cython(self, line, cell)
269 self._code_cache[key] = module_name
270
--> 271 module = imp.load_dynamic(module_name, module_path)
272 self._import_all(module)
273
ImportError: /home/bitnami/.cache/ipython/cython/_cython_magic_16feaf686ed172960f59fa6333ae74b5.so: undefined symbol: fastlp

Why cannot I pass a c array to a function which expects memory view in nogil content?

cdef double testB(double[:] x) nogil:
return x[0]
def test():
cdef double xx[2]
with nogil:
testB(xx)
# compiler error: Operation not allowed without gil
If with gil, it works fine.
Is it because that when pass in an c array, it creates a memory view and such creation action actually requires gil? So the memory view is not completely a c object?
Update
%%cython --annotate
cimport cython
cdef double testA(double[:] x) nogil:
return x[0]
cpdef myf():
cdef double pd[8]
cdef double[:] x = pd
testA(x)
cdef double[:] x = pd is compiled to:
__pyx_t_3 = __pyx_format_from_typeinfo(&__Pyx_TypeInfo_double);
__pyx_t_2 = Py_BuildValue((char*) "(" __PYX_BUILD_PY_SSIZE_T ")", ((Py_ssize_t)8));
if (unlikely(!__pyx_t_3 || !__pyx_t_2 || !PyBytes_AsString(__pyx_t_3))) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_3);
__Pyx_GOTREF(__pyx_t_2);
__pyx_t_1 = __pyx_array_new(__pyx_t_2, sizeof(double), PyBytes_AS_STRING(__pyx_t_3), (char *) "fortran", (char *) __pyx_v_pd);
if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_1);
__Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
__Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
__pyx_t_4 = __Pyx_PyObject_to_MemoryviewSlice_ds_double(((PyObject *)__pyx_t_1));
if (unlikely(!__pyx_t_4.memview)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_DECREF(((PyObject *)__pyx_t_1)); __pyx_t_1 = 0;
__pyx_v_x = __pyx_t_4;
__pyx_t_4.memview = NULL;
__pyx_t_4.data = NULL;
There exists __Pyx_PyObject_to_MemoryviewSlice_ds_double. So it seems when binding a memory view it does require gil.
You should use a numpy array, as your cdef double[:] declaration gets wrapped by a Python object, and its use is restricted without gil. You can see it by trying to slice a double[:]
def test()
cdef double[:] asd
with nogil:
asd[:1]
Your output will be:
with nogil:
asd[:1]
^
------------------------------------------------------------
prueba.pyx:16:11: Slicing Python object not allowed without gil
Using a numpy array would compile; numpy uses Python buffer protocole, and is smoothly integrated with Cython (a Google Summercamp project was financed for this). So no wrapping conflict arises inside the def:
import numpy as np
cdef double testA(double[:] x) nogil:
return x[0]
cpdef test():
xx = np.zeros(2, dtype = 'double')
with nogil:
a = testB(xx)
print(a)
This will build your module with test() on it. But it crashes, and in an ugly way (at least with mi PC):
Process Python segmentation fault (core dumped)
If I may insist with my (now deleted) previous answer, in my own experience, when dealing with Cython memoryviews and C arrays, passing pointers works just like one would expect in C. And most wrapping is avoided (actually, you are writing the code passing exactly the directions you want, thus making unnecesary wrapping). This compiles and functions as expected:
cdef double testB(double* x) nogil:
return x[0]
def test():
cdef double asd[2]
asd[0] = 1
asd[1] = 2
with nogil:
a = testB(asd)
print(a)
And, after compilig:
In [5]: import prueba
In [6]: prueba.test()
1.0
Memoryviews are not, by themselves, Python objects, but they can be wrapped in one. I am not a proficient Cython programmer, so sometimes I get unexpected wrappings or code that remains at Python level when I supposed it would be at C. Trial and error got me to the pointer strategy.