Extending Python by compiling C code in Cython - cython
I'm having trouble compiling a particular source file in C from an R package (fastclime) so that I can use it as a Python module. This is my code to extract the R package online:
from __future__ import division
import os
import sys
import glob
if not os.path.exists('fastlp.h'):
! wget http://cran.r-project.org/src/contrib/fastclime_1.2.4.tar.gz
! tar -xzvf fastclime_1.2.4.tar.gz
Within the R package (in the [src] folder) is the source file [parametric.c]. I'd like to compile this in Python as a module with numpy inputs. Since I can't immediately test the the functionality of this code, I first tried to compile the [fastlp.c] source file. With fastlp, I can create a toy example to test if I successfully compiled a version of fastlp in Python. Then I can probably extend the steps to compile [parametric.c]
Here is my attempt:
1) First I create the header file since it does not exist
%%file fastclime/src/fastlp.h
int ratio_test0(double *dy, int *idy,int ndy, double *y, double *ybar, double mu);
void fastlp(double *obj, double *mat, double *rhs, int *m0 , int *n0, double *opt, int *status, double *lam)
void solver20(int m,int n,int nz,int *ia, int *ka, double *a,double *b, double *c)
int ratio_test0(double *dy, int *idy,int ndy,double *y, double *ybar, double mu)
2) Next, I write the files for wrapping the C code in Cython:
%%file fastlp.pxd
cdef extern from "fastclime/src/fastlp.h":
void fastlp(double *obj, double *mat, double *rhs, int *m0 , int *n0, double *opt, int *status, double *lam)
%%file fastlp.pyx
cimport fastlp
def fastlp(double *obj, double *mat, double *rhs, int *m0 , int *n0, double *opt, int *status, double *lam):
fastlp.fastlp(*obj, *mat, *rhs, *m0, *n0, *opt, *status, *lam)
3) Next I use the distutils build system to compile.
%%file setup.py
from distutils.core import setup, Extension
from Cython.Build import cythonize
ext = Extension("fastlp",
sources=["fastlp.pyx", "fastclime/src/fastlp.c"])
setup(name="fastlp",
ext_modules = cythonize([ext]))
4) Finally I compile
! python setup.py build_ext -i
#4 is where things fail big time. Not sure how to resolve it in steps 1-3.
5) If all is successful (which it's not), I should be able to run this code (taken from the vignette)
import fastlp as flp
import numpy as np
A=np.array([-1.0,-1,0,1,-2,1]).reshape((3,2))
b=np.array([-1.0,-2,1])
c=np.array([-2.0,3])
flp.fastlp(c,A,b)
This should output the vector [2., 1.].
I'm not very experienced with Cython or Python for that matter, but getting this functionality would help out a lot with future projects. Could someone please let me know the proper way of taking a source file in C and getting through the steps in interfacing with numpy to create a Python module?
Thanks!
UPDATE:
I'm trying to follow the Numpy+Cython example on this site: https://scipy-lectures.github.io/advanced/interfacing_with_c/interfacing_with_c.html
I've since updated my code
%%cython
# import both numpy and the Cython declarations for numpy
import numpy as np
cimport numpy as np
# if you want to use the Numpy-C-API from Cython
# (not strictly necessary for this example)
np.import_array()
# cdefine the signature of our c function
cdef extern from "/home/bitnami/STA663-pura-project/fastclime/src/fastlp.h":
void fastlp(double *obj, double *mat, double *rhs, int *m0 , int *n0, double *opt, int *status, double *lam)
# create the wrapper code, with numpy type annotations
def fastlp_func(np.ndarray[double, ndim=1, mode="c"] obj not None,
np.ndarray[double, ndim=2, mode="c"] mat not None,
np.ndarray[double, ndim=1, mode="c"] rhs not None,
double lam):
cdef int m1 = mat.shape[0]
cdef int m2 = mat.shape[1]
cdef np.ndarray[double, ndim=1, mode='c'] z = np.zeros(obj.shape[0])
cdef double [:] zp = z #convert from Python to C object
cdef int stat = 0
cdef double lambd = 0
fastlp(<double*> np.PyArray_DATA(obj),
<double*> np.PyArray_DATA(mat.T),
<double*> np.PyArray_DATA(rhs),
&m1,
&m2,
&zp[0],
&stat,
&lambd
)
However, I get the following error (which I don't understand at all):
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-85-29981393af41> in <module>()
----> 1 get_ipython().run_cell_magic(u'cython', u'', u'# import both numpy and the Cython declarations for numpy\nimport numpy as np\ncimport numpy as np\n\n# if you want to use the Numpy-C-API from Cython\n# (not strictly necessary for this example)\nnp.import_array()\n\n# cdefine the signature of our c function\ncdef extern from "/home/bitnami/STA663-pura-project/fastclime/src/fastlp.h":\n void fastlp(double *obj, double *mat, double *rhs, int *m0 , int *n0, double *opt, int *status, double *lam)\n \n# create the wrapper code, with numpy type annotations\ndef fastlp_func(np.ndarray[double, ndim=1, mode="c"] obj not None,\n np.ndarray[double, ndim=2, mode="c"] mat not None,\n np.ndarray[double, ndim=1, mode="c"] rhs not None,\n double lam):\n cdef int m1 = mat.shape[0]\n cdef int m2 = mat.shape[1]\n cdef np.ndarray[double, ndim=1, mode=\'c\'] z = np.zeros(obj.shape[0])\n cdef double [:] zp = z #convert from Python to C object\n cdef int stat = 0\n cdef double lambd = 0\n fastlp(<double*> np.PyArray_DATA(obj),\n <double*> np.PyArray_DATA(mat.T),\n <double*> np.PyArray_DATA(rhs),\n &m1,\n &m2,\n &zp[0], \n &stat, \n &lambd \n )')
/home/bitnami/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in run_cell_magic(self, magic_name, line, cell)
2160 magic_arg_s = self.var_expand(line, stack_depth)
2161 with self.builtin_trap:
-> 2162 result = fn(magic_arg_s, cell)
2163 return result
2164
/home/bitnami/anaconda/lib/python2.7/site-packages/IPython/extensions/cythonmagic.pyc in cython(self, line, cell)
/home/bitnami/anaconda/lib/python2.7/site-packages/IPython/core/magic.pyc in <lambda>(f, *a, **k)
191 # but it's overkill for just that one bit of state.
192 def magic_deco(arg):
--> 193 call = lambda f, *a, **k: f(*a, **k)
194
195 if callable(arg):
/home/bitnami/anaconda/lib/python2.7/site-packages/IPython/extensions/cythonmagic.pyc in cython(self, line, cell)
269 self._code_cache[key] = module_name
270
--> 271 module = imp.load_dynamic(module_name, module_path)
272 self._import_all(module)
273
ImportError: /home/bitnami/.cache/ipython/cython/_cython_magic_16feaf686ed172960f59fa6333ae74b5.so: undefined symbol: fastlp
Related
Cython: Workaround scipy.optimize.cython_optimize.brentq argument ctype to work with class methods ctype
I'm trying to use the root finder scipy.optimize.cython_optimize.brentq inside a class, but the first argument of this function accepts only the type ctypedef double (*callback_type)(double, void*) and my class method is the type of ctypedef double (*w_func)(test,double, void*). How could I get this working? The following code is an exemple of the problem. %%cython from scipy.optimize.cython_optimize cimport brentq ctypedef double (*f_func)(test,double, double) ctypedef double (*w_func)(test,double, void*) ctypedef double (*callback_type)(double, void*) ctypedef struct test_params: double y0 double x1 f_func f w_func w cdef class test(): def __init__(self): cdef test_params myargs myargs.y0 = 1.0 myargs.x1 = 0.7 myargs.f = self.sum2 myargs.w = self.w print(self.brentqWrapper(myargs, -10, 10, 1e-3, 1e-3, 10)) cdef double sum2(self, double x1,double y1): return x1+y1 cdef double w(self, double y1, void *args): cdef test_params *myargs = <test_params *> args return y1 - myargs.y0 - myargs.f(self,myargs.x1,y1) # Cython wrapper function cdef double brentqWrapper(self,test_params args, double xa, double xb, double xtol, double rtol, int mitr): return brentq(args.w, xa, xb, <test_params *> &args, xtol, rtol, mitr, NULL) test()
ISTM it's easiest and most clean to restructure your code to have a module-level function which receives the class instance in its last argument.
How to create an array of structs of arbitrary size cython
I would like to know how can I create an array of structs in Cython that I can populate and make computations afterwards.
Example
Here I have the Cython code
%%cython
cimport numpy as cnp
cimport cython
from collections import namedtuple
Couple = namedtuple('Couple', ['female', 'male'], verbose=False)
cdef struct CyCouple:
int female
int male
cpdef int np_cy_count_women_earning_more2(list py_couples):
cdef:
int count = 0, r, N
CyCouple cy_couples[100_0000] # THIS IS HARDCODED
N = len(py_couples)
make_CyCouple_array(py_couples, cy_couples, N)
for n in range(N):
r = cy_couples[n].female > cy_couples[n].male
count += r
return count
I would like to have a general versions instead of the definition in # THIS IS HARDCODED.
What could I do?
Define array of strings in Cython
Stuck on some basic Cython here - what's a canonical and efficient way to define an an array of strings in Cython? Specifically, I want to define a fixed-length constant array of char. (Please note that I would prefer not to bring in NumPy at this point.)
In C this would be:
/* cletters.c */
#include <stdio.h>
int main(void)
{
const char *headers[3] = {"to", "from", "sender"};
int i;
for (i = 0; i < 3; i++)
printf("%s\n", headers[i]);
}
Attempt in Cython:
# cython: language_level=3
# letters.pyx
cpdef main():
cdef const char *headers[3] = {"to", "from", "sender"}
print(headers)
However, this gives:
(cy) $ python3 ./setup.py build_ext --inplace --quiet
cpdef main():
cdef const char *headers[3] = {"to", "from", "sender"}
^
------------------------------------------------------------
letters.pyx:5:32: Syntax error in C variable declaration
You need two lines: %%cython cpdef main(): cdef const char *headers[3] headers[:] = ['to','from','sender`] print(headers) Somewhat counterintuitive is than one assigns unicode-strings (Python3!) to char*. That is one of Cython's quirks. On the other hand, while initializing everything with only one value, bytes-object is needed: %%cython cpdef main(): cdef const char *headers[3] headers[:] = b'init_value` ## unicode-string 'init_value' doesn't work. print(headers) Another alternative is the following oneliner: %%cython cpdef main(): cdef const char **headers=['to','from','sender`] print(headers[0], headers[1], headers[2]) which is not exactly the same as above and leads to the following C-code: char const **__pyx_v_headers; ... char const *__pyx_t_1[3]; ... __pyx_t_1[0] = ((char const *)"to"); __pyx_t_1[1] = ((char const *)"from"); __pyx_t_1[2] = ((char const *)"sender"); __pyx_v_headers = __pyx_t_1; __pyx_v_headers is of type char ** and downside is, that print(headers)no longer works out of the box.
For python3 Unicode strings, this is possible- cdef Py_UNICODE* x[2] x = ["hello", "worlᏪd"] or cdef Py_UNICODE** x x = ["hello", "worlᏪd"]
Cython - Wrapping pointer to structure from C to python
I have a C function which take pointer to struct and i want to use it in python by C-Extensions by Cython way but when i want to pass pointer to struct from python give me an error: "Cannot convert Python object to 'Foo *'"
In the below example i make object to call the C function but what passed to C function is NULL pointer.
My Trial:
hello.h
#include <stdio.h>
typedef struct
{
int x;
} Foo;
int hello(Foo* what);
hello.c
#include "hello.h"
int hello(Foo* what)
{
printf("Hello Wrapper\n");
printf("what: %p\n", what);
what->x = 5;
return what->x;
}
phello.pxd
cdef extern from "hello.h":
ctypedef struct Foo:
int x
cdef int hello(Foo* what)
phello.pyx
cimport phello
cdef class Foo_c:
cdef phello.Foo* s
def hello_fn(self):
return phello.hello(self.s)
setup.py
from distutils.core import setup, Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules=[ Extension("hellomodule",
sources=["phello.pyx", "hello.c"],
) ]
test.py
import hellomodule
print "Hello test.py"
tobject = hellomodule.Foo_c()
print "Object:", tobject
tobject.hello_fn()
So i want create "Foo" struct in "test.py" and pass it to "hello_fn()" function to call the C function "hello()" after passing this struct, so i can read or write on this structure from both sides python & C.
Can Anyone help me in this, please?
Your code does not allocate memory for phello.Foo. Allocation can be done in __cinit__ with calloc (or malloc) and deallocation in __dealloc__ with free. cimport phello from libc.stdlib cimport calloc, free cdef class Foo_c: cdef phello.Foo* s def __cinit__(self, int n): self.s = <phello.Foo *>calloc(1, sizeof(phello.Foo)) def __dealloc__(self): free(<void *>self.s) def __init__(self, int n): self.s.x = n
Why cannot I pass a c array to a function which expects memory view in nogil content?
cdef double testB(double[:] x) nogil:
return x[0]
def test():
cdef double xx[2]
with nogil:
testB(xx)
# compiler error: Operation not allowed without gil
If with gil, it works fine.
Is it because that when pass in an c array, it creates a memory view and such creation action actually requires gil? So the memory view is not completely a c object?
Update
%%cython --annotate
cimport cython
cdef double testA(double[:] x) nogil:
return x[0]
cpdef myf():
cdef double pd[8]
cdef double[:] x = pd
testA(x)
cdef double[:] x = pd is compiled to:
__pyx_t_3 = __pyx_format_from_typeinfo(&__Pyx_TypeInfo_double);
__pyx_t_2 = Py_BuildValue((char*) "(" __PYX_BUILD_PY_SSIZE_T ")", ((Py_ssize_t)8));
if (unlikely(!__pyx_t_3 || !__pyx_t_2 || !PyBytes_AsString(__pyx_t_3))) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_3);
__Pyx_GOTREF(__pyx_t_2);
__pyx_t_1 = __pyx_array_new(__pyx_t_2, sizeof(double), PyBytes_AS_STRING(__pyx_t_3), (char *) "fortran", (char *) __pyx_v_pd);
if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_1);
__Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
__Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
__pyx_t_4 = __Pyx_PyObject_to_MemoryviewSlice_ds_double(((PyObject *)__pyx_t_1));
if (unlikely(!__pyx_t_4.memview)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 8; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_DECREF(((PyObject *)__pyx_t_1)); __pyx_t_1 = 0;
__pyx_v_x = __pyx_t_4;
__pyx_t_4.memview = NULL;
__pyx_t_4.data = NULL;
There exists __Pyx_PyObject_to_MemoryviewSlice_ds_double. So it seems when binding a memory view it does require gil.
You should use a numpy array, as your cdef double[:] declaration gets wrapped by a Python object, and its use is restricted without gil. You can see it by trying to slice a double[:] def test() cdef double[:] asd with nogil: asd[:1] Your output will be: with nogil: asd[:1] ^ ------------------------------------------------------------ prueba.pyx:16:11: Slicing Python object not allowed without gil Using a numpy array would compile; numpy uses Python buffer protocole, and is smoothly integrated with Cython (a Google Summercamp project was financed for this). So no wrapping conflict arises inside the def: import numpy as np cdef double testA(double[:] x) nogil: return x[0] cpdef test(): xx = np.zeros(2, dtype = 'double') with nogil: a = testB(xx) print(a) This will build your module with test() on it. But it crashes, and in an ugly way (at least with mi PC): Process Python segmentation fault (core dumped) If I may insist with my (now deleted) previous answer, in my own experience, when dealing with Cython memoryviews and C arrays, passing pointers works just like one would expect in C. And most wrapping is avoided (actually, you are writing the code passing exactly the directions you want, thus making unnecesary wrapping). This compiles and functions as expected: cdef double testB(double* x) nogil: return x[0] def test(): cdef double asd[2] asd[0] = 1 asd[1] = 2 with nogil: a = testB(asd) print(a) And, after compilig: In [5]: import prueba In [6]: prueba.test() 1.0 Memoryviews are not, by themselves, Python objects, but they can be wrapped in one. I am not a proficient Cython programmer, so sometimes I get unexpected wrappings or code that remains at Python level when I supposed it would be at C. Trial and error got me to the pointer strategy.