How do I compile using Cython, a C function from file simulate_fast.c, where this also depends on further C files matrix.c and random_generator.c. This seems like it must be a common use of Cython but after reading through the documentation, I still cannot figure out how to do it. My directory contains the following files
matrix.c
matrix.h
random_generator.c
random_generator.h
simulate_fast.c
simulate_fast.h
test.c
simulate_fast_c.pyx
setup.py
The matrix.c and random_generator.c files contain standalone functionality. The simulate_fast.c file uses both of these and contains the function I want to be exposed to Python, simulate().
The test.c file tests that all the C functionality runs correctly i.e. I can execute
$ gcc test.c simulate_fast.c matrix.c random_generator.c -o test
to compile into a test executable that works.
My issue now is trying to compile this with Cython. My .pyx file is
cimport cython
cdef extern from "simulate_fast.h":
int simulate()
def simulate_cp():
return simulate()
I then use the basic setup.py
from distutils.core import setup
from Cython.Build import cythonize
from Cython.Distutils import build_ext
import numpy as np
setup(
name='simulate_fast',
ext_modules = cythonize(["simulate_fast_c.pyx"]),
include_dirs=[]
)
However if I try and compile this using
python3 setup.py build_ext --inplace
I get an error
In file included from simulate_fast_c.c:492:0:
simulate_fast.h:89:28: error: field ‘RG’ has incomplete type
struct RandomGenerator RG;
where structure RandomGenerator is declared in random_generator.h.
How do I tell the compiler we must also consider the matrix and random_generator files when compiling.
Update
If doing as ead said in the comments, I include random_generator.h and matrix.h in simulate_fast.h then the program now compiles. However when I try to import the simulate_fast_c module in Python I get an ImportError:
undefined symbol: simulate
Further, if I change the extern declaration line in simulate_fast.pyx to
cdef extern from "simulate_fast.c":
int simulate()
I then get an import error
undefined symbol: get_random_number
which is a function in random_generator.h
The cythonized module must either be linked to a shared library containing the compiled C code or embed it. One way to do the latter is to list the C sources as being an "Extension", then pass this extension to the cythonize command, as mentioned in Cython's documentation
The example setup.py file in the link can be summarized by (modulo the imports):
setup(
ext_modules = cythonize(Extension("simulate_fast", ["matrix.c", "random_generator.c", "simulate_fast_c.pyx"]))
)
Related
If I made a python file named hello.py that has a script made like this.
msg = input("insert your message here: ")
script = '''
def say_something():
print("{msg}")
'''
exec(script)
say_something()
And then I tried to use Cython
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules=cythonize("Hello.py")
)
It will show an error like this: undeclared name not builtin: say_something
I do understand why this happens but I'm not really an expert with python and C just yet. This is just an example, but it's similar to what I'm trying to do with one of my projects. Is there any way I could resolve this? I want to find a way to convert the script string into C as well.
I was trying to build an editable python script.
Cython compiles the Python functions to a native binary that does what the CPython interpreter should do. exec is a function that execute arbitrary code at runtime (which is generally a very bad idea for speed, maintainability/readability and security). Cython does not support exec because it would mean that the could would be compiled at runtime. Thus, the code executed by exec cannot be a Cython code. However, the exec function can still be used to execute a pure-Python code. The error can be removed by turning off the Cython.Compiler.Options.error_on_unknown_names in the setup script (just before calling setup) as pointed out by #DavidW. With this Cython will not complain when it does not find a function defined by exec (or similar methods). Please keep in mind that CPython can only be used in this case instead of Cython (which partially defeat the purpose of using Cython in the first place).
Is there a way to register a function within an Cython extenstion module such that this function is called when the Python garbage collector destroys the module object ?
Something like
def __dealloc__(...)
on module level ?
I'm not sure there is an explicit/official way to achieve that. However, you could add a global variable which will be destroyed, when the cython module is deallocated:
# foo.pyx:
class Observer:
def __init__(self):
print("module initialized")
def __del__(self):
print ("module deleted")
_o=Observer()
However, cythonizing it via cythonize foo.pyx -i will not lead to the desired effect:
[$] python -c `import foo`
module initialized
there is no "module deleted" printed to the console!
The problem: one cannot reload a cython-extension anyway (for example via importlib.reload()), so it gets only deallocated when the Python interpreter is shut down. But then there is no need to deallocate the resources ...
Cython keeps all global Python-variables in a global C-variable which is called static PyObject *__pyx_d; in the cythonized C-source; this is just a Python-dictionary. However, because there is a reference to the global _o in this not destroyed dictionary, the reference count of _o will never become 0 an thus the object doesn't get destroyed.
By setting generate_cleanup_code=True one can force Cython to generate a clean-up function which will be put into the m_free-slot of the PyModuleDef definition struct. For example with the following setup-file:
from distutils.core import setup
from Cython.Build import cythonize
from Cython.Compiler import Options
Options.generate_cleanup_code = True # this is important!
setup(
name = "foo",
ext_modules = cythonize("foo.pyx"),
)
And now after python setup.py build_ext -i:
[$] python -c "import foo"
module initialized
module deleted
it works as expected.
Because there is no reload for Cython-extensions, "module deleted" will be only seen when module_dealloc is called.
However, when the first listing were pure-Python, we would also see module deleted if importlib.reload(foo) were called. In this case "module_dealloc" isn't used, but all references to global _o would be cleared and object would be destroyed.
Depending on what one want, it might be either the expected or unexpected behavior.
I am attempting to use the LMDB C API with Cython.
I want to import the following definitions from the header file:
typedef struct MDB_env MDB_env;
int mdb_env_create(MDB_env **env);
So I created a .pxd file:
cdef extern from 'lmdb.h':
struct MDB_env:
pass
int mdb_env_create(MDB_env **env)
And I am using it in a Cython script:
cdef MDB_env *e
x = mdb_env_create(&e)
This code compiles fine, but If I run it, I get:
ImportError: /home/me/.cache/ipython/cython/_cython_magic_15705c11c6f56670efe6282cbabe4abc.cpython-36m-x86_64-linux-gnu.so: undefined symbol: mdb_env_create
This happens both in a Cython .pyx + .pxd setup and in a prototype typed in IPython.
If I import another symbol, say a constant, I can access it. So I seem to be looking at the right header file.
I don't see any discrepancy between my syntax and the documentation, but I am clearly doing something wrong. Can somebody give me a hint?
Thanks.
To compile it with IPythons-magic (would be nice if you would mention this explicitly in your question) you have to provide library-path (via -L-option) and library name (via -l-option) of the built c-library you want to wrap, see also the documentation:
%%cython -L=<path to your library> -l=<your_library>
The library you are trying to wrap is not a header-only library. That means that some symbols (e.g. mdb_env_create) are only declared but not defined in the header. When you build the library, the definitions of those symbols can be found in the resulting artifact, which should be provided to the linker when your extension is built. These definitions is what is needed when the program runs.
If you don't do it, the following happens on Linux: When the extension (the *.so-file) is built,the linker allows undefined symbols per default - so this step is "successful" - but the failure is only postponed. When the extension is loaded via import, Python loads the corresponding *.so with help of ldopen and in this step loader checks that the definitions of all symbols are known. But we didn't provide a definition of mdb_env_create so, the loader fails with
undefined symbol: mdb_env_create
It is differently for symbols which are defined in the header-file, for example enums MDB_FIRST&Co - the compiled library isn't necessary and thus the extension can be loaded, as there are no undefined symbols.
I am trying to use the GSL library in a Cython program but don't seem to have the paths correctly specified; I encounter the following error when I try writing a simple example:
%load_ext cythonmagic
%%cython -lgsl -lgslcblas
cdef extern from "gsl/gsl_ran_poisson_pdf.h":
double gsl_ran_poisson_pdf(int x, double mu)
def poison(int x, double mu):
return gsl_ran_poisson_pdf(x,mu)
/Users/name/.ipython/cython/_cython_magic_189673701925d12059c18b75663da8bd.c:317:10: fatal error:
'gsl/gsl_mode.h' file not found
#include "gsl/gsl_mode.h"
I get the same error using CythonGSL and the demo program here: http://nbviewer.ipython.org/github/twiecki/CythonGSL/blob/master/examples/cython_gsl_ipythonnb.ipynb
The GSL libraries are located in the following directories:
-I/usr/local/include
-L/usr/local/lib -lgsl
I know that similar questions have been asked on SO before, but I couldn't find one relevant to my situation and system (I'm using OS-X). Any help would be appreciated.
Thanks!
Since I made some progress, I changed the title and made a second edit describing my new problem. You may choose to ignore Edit1
I have been trying to run python code from C code. And for this purpose I have been using Cython.
The semantics of my system is such that there is a binary (whos source I can not access) that calls a C function defined in a file (source is accessible) and within this function I need to call python functions, do some processing and return the result to binary.
To achieve this purpose, there are two approaches that I came across:
http://docs.python.org/release/2.5.2/ext/callingPython.html ===> This approach suggests to have the python callback function passed to the C side, so that the callback is called as necessary, but this doesn't work for me as I don't have access to the binary's source (which is used to run the entire system)
https://stackoverflow.com/a/5721123/1126425 ==> I have tried this approach and I get this error when the cython function is called:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb47deb70 (LWP 2065)]
0x007fd38a in PySys_GetObject () from /usr/lib/libpython2.6.so.1.0
http://www.linuxjournal.com/article/8497?page=0,0 ==> This is in fact the basis for cython's functionality but again when I use the examples described there, I get errors similar to 2.
I have no idea how to resolve these errors. Any help would be much appreciated.
Thanks!!
Edit1:
here is a simple scenario that reflects situation:
external.c
#include <external.h>
int callback(int param1,int param2)//Function that the binary calls
{
/*SomeTasks*/
cython_func();//Function defined in the following .pyx file
/*SomeTasks*/
}
cython_file.pyx
cdef void cython_function():
print "Do Nothing!"
I am linking the shared library file created by cython with the library generated by compiling the above C code and then that library is used by the binary...
Edit2:
The segmentation fault goes away when I added Py_Initialize(); before calling cython_function(). But now I am getting the undefined symbol error as : symbol lookup error: lib_c_code.so: undefined symbol: cython_function
Here lib_c_code.so is the shared library created out of the external.c file above. I have tried including the .h file created by the cython compiler in external.c but it still didn't work out.. Here is how I am compiling lib_c_code.so:
gcc -shared -dynlib -lm -W1 -o lib_c_code.so $(OBJDIR)/*.o -lc -lm -lpy_code
and the libpy_code.so is the shared object file that was created out of the cython_file.pyx file as:
cython cython_file.pyx -o cython_file.c
gcc $(IFLAGS) -I/usr/include/python2.6 -fPIC -shared cython_file.c -lpython2.6 -lm -o libpy_code.so
Also, I can see the symbol cython_function in the lib_c_code.so file when I do : nm -g lib_c_code.so..
Any ideas please?
I have to guess here that there's a callback registration function to which you can pass the function pointer, in which case you can simply forego the C file and define a cdef function directly in your Cython code, and pass that with the callback registration function. Use with gil in case you manipulate any Python objects in it.
cdef extern from "external.h":
ctypedef int (*Cb_Func)(int param1, int param2)
void register_callback(Cb_Func func)
cdef int my_callback(int param1,int param2) with gil:
<implementation>
register_callback(my_callback)
This is also explained in the Cython user manual here: http://docs.cython.org/src/userguide/external_C_code.html