cython cimport from another pyx file - cython

I have a project with the following structure:
lst_viewer/
setup.py
extensions/
instrument.pxd
instrument.pyx
read_events.pyx
src/
lst_viewer/
extensions/
__init__.py
Here is my setup.py file:
import os
from Cython.Distutils import build_ext as cython_build_ext
from setuptools import find_packages, setup
from distutils.core import setup, Extension
EXTENSIONS = [
Extension('lst_viewer.extensions.instrument',sources = [os.path.join('extensions','instrument.pyx')]),
Extension('lst_viewer.extensions.read_events',sources = [os.path.join('extensions','read_events.pyx')]),
]
CMDCLASS = {'build_ext' : cython_build_ext}
setup(name="lst_viewer",
packages=find_packages('src'),
include_package_data=True,
package_dir={'': 'src'},
ext_modules=EXTENSIONS,
cmdclass=CMDCLASS)
Here is my instrument.pxd file:
cdef class Instrument:
cdef int nbdet
cdef class D11Instrument:
pass
cdef Instrument get_instrument(str name)
Here is my instrument.pyx file:
cdef class Instrument:
pass
cdef class D11Instrument(Instrument):
def __cinit__(self):
self.nbdet = 3
cdef Instrument get_instrument(str name):
if name == "D11":
return D11Instrument()
And here is my read_events.pyx file:
from instrument cimport Instrument, get_instrument
def read_events(str instrument_name):
cdef Instrument instr = get_instrument(instrument_name)
I have problems with importing the Instrument class and get_instrument function from the other pyx file of the project (read_events.pyx). Indeed, the code compiles but, when running it, I get a ModuleNotFoundError: No module named 'instrument' error. I tried to add an (empty) __init__.pxd to the lst_viewer/extensions directory but it did not help. Would you have any idea about what I am doing wrong ?

Related

cython: how to package a python module?

For example, I have a python module, and the structure is
moduleA
__init__.py
functionA.py
moduleB
__init__.py
functionB.py
setup.py
How can I package the moduleA using cython?
What I have tested:
test
__init__.py
foo.py
setup.py
#----------------------------------------------------
## __init__.py
from foo import *
#---------------------------------------------------
## foo.py
def hello():
print("hello")
# ----------------------------------------------------------
## setup.py
from setuptools import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [
Extension('test.foo', ['test/foo.py']),
]
setup(
name='Test',
ext_modules=ext_modules,
include_dirs=["test"],
cmdclass={'build_ext': build_ext},
)
After python setup.py build_ext, a new fold appear:
build
lib.win-amd64-3.7
test
foo.cp37-win_amd64.pyd
temp.win-amd64-3.7
Release
test
foo.cp37-win_amd64.exp
foo.cp37-win_amd64.lib
foo.obj
Then, cd ./build/lib.win-amd64-3.7, and open python interpreter. import test is OK, but I can use hello() by test.hello(), test.foo.hello(), test.foo().
What's wrong with my process?
The environment is:
win 10
python 3.7
cython 0.29.26
Update----------------------------------------------------
The above problem can be solved by `python setup.py build_ext --inplace'. Thanks for DavidW's suggestion.
My next question is how to pack a python module?
For example, I have a module as:
App
ModuleA
__init__.py
funcA.py
ModuleB
__init__.py
funcB.py
__init__.py
setup.py
I can pack the aoove module by the following setup.py:
from setuptools import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [
Extension('App.ModuleA.funcA', ['App/ModuleA/funcA.py']),
Extension('App.ModuleB.funcB', ['App/ModuleB/funcB.py']),
]
setup(
name='App',
ext_modules=ext_modules,
include_dirs=["App"],
cmdclass={'build_ext': build_ext},
)
I manually collect the ext_modules. But, if the module has many file, how can I automatically collect the ext_modules?
I can only write the following stranger code:
name = 'App.'
moduleName = os.listdir('./App')
for mName in moduleName:
name += mName
...
Is there any other elegant methods?

How to use atexit in Cython?

I am trying to make the following code in cython to compile
from cython_gsl cimport *
import atexit
cdef gsl_rng_type * rng_T = gsl_rng_default
cdef gsl_rng * rng_r
gsl_rng_env_setup()
rng_r = gsl_rng_alloc(rng_T)
#atexit.register
def free_gsl_rng():
gsl_rng_free(rng_r)
But I always get the error
Assignment to non-lvalue 'atexit'
Corresponding .pxd file
from cython_gsl cimport *
cdef gsl_rng * rng_r
I am actually compiling this with SageMath 8.7 by
sage setup.py build_ext --inplace
Here is my setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
from Cython.Build import cythonize
import cython_gsl
extensions = [
Extension("gsl_rand", ["gsl_rand.pyx"],
libraries=cython_gsl.get_libraries(),
library_dirs=[cython_gsl.get_library_dir()],
include_dirs=[cython_gsl.get_cython_include_dir()]
),
]
setup(
name='Simulation of k-cut on conditional Galton-Watson trees',
cmdclass={'build_ext': build_ext},
include_dirs = [cython_gsl.get_include()],
ext_modules=cythonize(extensions),
)
Full error log here
sage setup.py build_ext --inplace
Compiling gsl_rand.pyx because it changed.
[1/1] Cythonizing gsl_rand.pyx
/home/xing/Downloads/software/sage/8.7/local/lib/python2.7/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/xing/Dropbox/Research/2017/k-cut/GW/sage/moments/cython-v5/gsl_rand.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
warning: /home/xing/Downloads/software/sage/8.7/local/lib/python2.7/site-packages/cython_gsl/gsl_integration.pxd:65:9: 'GSL_EMAXITER' redeclared
warning: /home/xing/Downloads/software/sage/8.7/local/lib/python2.7/site-packages/cython_gsl/gsl_integration.pxd:67:9: 'GSL_EROUND' redeclared
warning: /home/xing/Downloads/software/sage/8.7/local/lib/python2.7/site-packages/cython_gsl/gsl_integration.pxd:69:9: 'GSL_ESING' redeclared
warning: /home/xing/Downloads/software/sage/8.7/local/lib/python2.7/site-packages/cython_gsl/gsl_integration.pxd:71:9: 'GSL_EDIVERGE' redeclared
warning: gsl_rand.pyx:14:0: Overriding cdef method with def method.
Error compiling Cython file:
------------------------------------------------------------
...
# cython: profile=False
from cython_gsl cimport *
import atexit
^
------------------------------------------------------------
gsl_rand.pyx:5:7: Assignment to non-lvalue 'atexit'
Error compiling Cython file:
------------------------------------------------------------
...
cdef gsl_rng * rng_r
gsl_rng_env_setup()
rng_r = gsl_rng_alloc(rng_T)
#atexit.register
^
------------------------------------------------------------
gsl_rand.pyx:13:0: Object of type 'int (void (*)(void) nogil) nogil' has no attribute 'register'
Traceback (most recent call last):
File "setup.py", line 29, in <module>
ext_modules=cythonize(extensions),
File "/home/xing/Downloads/software/sage/8.7/local/lib/python2.7/site-packages/Cython/Build/Dependencies.py", line 1097, in cythonize
cythonize_one(*args)
File "/home/xing/Downloads/software/sage/8.7/local/lib/python2.7/site-packages/Cython/Build/Dependencies.py", line 1220, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: gsl_rand.pyx
atexit is getting cimported with from cython_gsl cimport *. This appears to happen with the line from libc.stdlib cimport *. This name therefore conflicts with the Python atexit module. I think this is a good example of why from something [c]import * is not recommended, both in your code and in cython_gsl's code. It gets slightly more confusing because Cython has two types of names (for Python and C) so you get odd error messages.
The best solution is to either do cimport cython_gsl or to cimport the specific symbols you need: from cython_gsl cimport gsl_rng, etc. Remember to change both the pyx and pxd file.
The worse workround is to rename the atexit module when you import it: import atexit as ae.

How to create python executable with Cython? (Segmentation fault)

I'm trying to set up a cython setup that will compile
a python source code to an executable (It should embed the main method within) - currently I have managed to set it up as an importable module but not as a standalone executable.
I saw that there is a compiler option Options.embed that should handle this. (In this post it said that it should be set to the function that the interpreter should call - main)
This is the module code:
def main():
print('Cython Demo')
if __name__ == '__main__':
main()
This is the setup "compile.py" code
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
from Cython.Build import cythonize
from Cython.Compiler import Options
Options.docstrings = False
Options.emit_code_comments = False
Options.embed = "main"
ext_modules = cythonize([
Extension("cython_demo.mymod.moduleA",["/home/myuser/PycharmProjects/cython_demo/mymod/moduleA.py"])],
compiler_directives=dict(always_allow_keywords=True,language_level = '3'))
setup(
name = 'My Program Name',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
Unfortunately after compiling the python code and trying to run the executable by calling:
./moduleA.cpython-36m-x86_64-linux-gnu.so
i get the segmentation error.
Segmentation fault (core dumped)
I saw that the main function is there by running grep "int main" on the file. What may be the problem?
When I'm importing the module from somewhere else and running main directly - it works:
import moduleA
moduleA.main()
Thanks!

extra_compile_args in Cython

I want to pass some extra options to the Cython compiler by using extra_compile_args.
My setup.py:
from distutils.core import setup
from Cython.Build import cythonize
setup(
name = 'Test app',
ext_modules = cythonize("test.pyx", language="c++", extra_compile_args=["-O3"]),
)
However, when I run python setup.py build_ext --inplace, I get the following warning:
UserWarning: got unknown compilation option, please remove: extra_compile_args
Question: How does one use extra_compile_args correctly?
I use Cython 0.23.4 under Ubuntu 14.04.3.
Use the more traditional way without cythonize to supply extra compiler options:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
name = 'Test app',
ext_modules=[
Extension('test',
sources=['test.pyx'],
extra_compile_args=['-O3'],
language='c++')
],
cmdclass = {'build_ext': build_ext}
)
Mike Muller's answer works, but builds extensions in the current directory, not next to the .pyx file when --inplace is given as in:
python3 setup.py build_ext --inplace
So my workaround is to compose a CFLAGS string and override the env variable:
os.environ['CFLAGS'] = '-O3 -Wall -std=c++11 -I"some/custom/paths"'
setup(ext_modules = cythonize(src_list_pyx, language = 'c++'))
There's another way to do this, I found it to be the best one from other two presented because with this you still can use all regular cythonize arguments in usual way:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
name="Test app",
ext_modules=cythonize(
Extension(
"test_ext", ["test.pyx"],
extra_compile_args=["-O3"],
language="c++",
),
),
)

cython works fine on winpython but not on anaconda - linking issue

I have a Cython code that runs and works fine under WinPython but giving me issues when I switch from WinPython to Anaconda3. I am using Python 3.4
The test code is :
# cython: boundscheck=False
# cython: wraparound=False
# cython: cdivision=True
cimport cython
cimport numpy as np
import numpy as np
from numpy cimport ndarray as ar
from libc.math cimport *
cpdef ar[double, ndim=1, mode='c'] test(ar[double, ndim=1, mode='c'] x):
cdef:
int n = x.shape[0]
Py_ssize_t i
ar[double, ndim=1, mode='c'] y = np.zeros(n)*np.nan
with nogil:
for i in range(0, n):
y[i] = x[i]+1
return y
The compilation code is:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy as np
ext_modules = [Extension('test', ['tech/test.pyx'], include_dirs=[np.get_include()],
define_macros=[('NPY_NO_DEPRECATED_API', None)],
extra_compile_args=['-O3', '-march=native', '-ffast-math'],
libraries=['m']
)]
setup(
name="Test Function",
cmdclass={'build_ext': build_ext},
ext_modules=ext_modules
)
I am using MinGw to compile, so under the ...\Anaconda3\Lib\distutils folder I have a file that has
[build]
compiler = mingw32
In addition in the PATH environment variables, I have:
default:
...\Anaconda3
...\Anaconda3\Scripts
also added:
...\Anaconda3\libs (this contains python34)
also added: mingw32 files copied from WinPython that contain gcc etc:
...\Anaconda3\Tools\tools\mingw32\bin (this contains gcc)
...\Anaconda3\Tools\tools\mingw32\x86_64-w64-mingw32\bin
when I try:
python setup.py build_ext --inplace
the code runs fine until the test.c, test.o and test.def have been generated. thereafter this is what I get:
C:\Anaconda3\Tools\tools\mingw32\bin\gcc.exe -shared -s build\temp.win-amd64-3.4\Release\tech\test.o build\temp.win-amd64-3.4\Release\tech\test.def -LC:\Anaconda3\libs -LC:\Anaconda3\PCbuild\amd64 -lm -lpython34 -lmsvcr100 -
o P:\Documents\Temp\python-master\python-master\common\test.pyd
build\temp.win-amd64-3.4\Release\tech\test.o:test.c:(.text+0x8e): undefined reference to `__imp_PyExc_TypeError'
build\temp.win-amd64-3.4\Release\tech\test.o:test.c:(.text+0x10e): undefined reference to `__imp_PyExc_ValueError'
build\temp.win-amd64-3.4\Release\tech\test.o:test.c:(.text+0x259): undefined reference to `__imp__PyThreadState_Current'
....
....
C:/Anaconda3/Tools/tools/mingw32/bin/../lib/gcc/x86_64-w64-mingw32/4.9.2/../../../../x86_64-w64-mingw3 ild\temp.win-amd64-3.4\Release\tech\test.o: bad reloc address 0x0 in section `.data'
collect2.exe: error: ld returned 1 exit status
error: command 'C:\\Anaconda3\\Tools\\tools\\mingw32\\bin\\gcc.exe' failed with exit status 1
First of all, I couldn't find a Anacond3\PCBuild\amd64 directory/file anywhere.
I tried to look all over the internet, but I couldn't find any reference to __imp_PyExc_TypeError, __imp__PyThreadState_Current, __imp_PyExc_ValueError.
What could be going wrong with the final conversion to *.pyd file?
I suspect Anaconda doesn't support mingw-64 yet.
Historically mingw-64 was indeed a hopeless idea until Cark Kleffner came with his static version.
Things may (should ?) change when Carl's updated version will be available as mingwpy wheel.
You can run
conda remove libpython
to make Anaconda use Visual Studio instead of mingw, which may work better.