I'm generating different .pyx files and turning them to C++ .so libraries. So I got an example generatedLib.pyx like this:
from libcpp.vector cimport vector
def cinit(self): pass
cdef public void someFunction(vector[...]& inputs):
...
The public function always has same signature, except for the name. I somehow wanna call this public function using python interpreter because I prefer to write tests in Python rather than C++. I cannot do it directly of course. I cannot define this function as def as well since its parameter is of C++ vector type.
I have came up with an idea to have a wrapper function
def wrapperSomeFunction(input: list)
{
# convert Python list to c++ vector
someFunction(v)
# convert c++ vector back to python list and return
}
I can import the generatedLib as python module and call this wrapper just fine, it requires however, to append it to my generatedFile.pyx file contents every time I wanna test it before I build it. I would rather look for alternative solutions.
Is there some way I can build a separate .pyx which will have this wrapper inside and will be able to call my generatedLib.so? Or perhaps something else I didn't think of?
Related
If I made a python file named hello.py that has a script made like this.
msg = input("insert your message here: ")
script = '''
def say_something():
print("{msg}")
'''
exec(script)
say_something()
And then I tried to use Cython
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules=cythonize("Hello.py")
)
It will show an error like this: undeclared name not builtin: say_something
I do understand why this happens but I'm not really an expert with python and C just yet. This is just an example, but it's similar to what I'm trying to do with one of my projects. Is there any way I could resolve this? I want to find a way to convert the script string into C as well.
I was trying to build an editable python script.
Cython compiles the Python functions to a native binary that does what the CPython interpreter should do. exec is a function that execute arbitrary code at runtime (which is generally a very bad idea for speed, maintainability/readability and security). Cython does not support exec because it would mean that the could would be compiled at runtime. Thus, the code executed by exec cannot be a Cython code. However, the exec function can still be used to execute a pure-Python code. The error can be removed by turning off the Cython.Compiler.Options.error_on_unknown_names in the setup script (just before calling setup) as pointed out by #DavidW. With this Cython will not complain when it does not find a function defined by exec (or similar methods). Please keep in mind that CPython can only be used in this case instead of Cython (which partially defeat the purpose of using Cython in the first place).
How do I modify the C++ class constructor that cython will generate from a Cython extension type that I've defined in a .pyx file wo that I don't incur the penalty of using def __cinit__(self)? Assume I am compiling to C++, not C.
What I would like to do is set the default values as I would in an initialization list in C++, and maybe run some code in the constructor without interacting with Python. It looks like the __cinit__ method runs after the C++ class members are initialized and the C++ constructor finishes running.
In this example, it says
Cython initializes C++ class attributes of a cdef class using the nullary constructor.
I assume this refers to the default constructor since the C++ constructor is a nullary constructor. I also assumes this happens with or without a __cinit__ method.
If the class you’re wrapping does not have a nullary constructor, you must store a pointer to the wrapped class and manually allocate and deallocate it. A convenient and safe place to do so is in the cinit and dealloc methods which are guaranteed to be called exactly once upon creation and deletion of the Python instance.
I have an existing class where I call Py_XDECREF on a member object in a loop inside of __dealloc__. This object holds pointers to Python objects. When I convert to C++ code, the annotated HTML shows no highlighted yellow text in the lines where the __dealloc__ method is defined. This is interesting because __dealloc__ is also defined using def.
When I try to initialize a member that is an instance of a Cython Extension that I defined (and did not create from wrapping C++ code), however, the __cinit__ method is highlighted in yellow.
def __cinit__(self):
self._member = MyCyExt()
I'm wondering if it's because I can't initialize a Cython Extension Type in pure C++. I don't know why that would be the case since Cython calls the nullary constructor by default.
This answer solves the problem, but it incurs the overhead I am trying to avoid. Is there a way to do this without incurring that overhead and actually writing the constructor itself in a .pyx file?
I am attempting to use the LMDB C API with Cython.
I want to import the following definitions from the header file:
typedef struct MDB_env MDB_env;
int mdb_env_create(MDB_env **env);
So I created a .pxd file:
cdef extern from 'lmdb.h':
struct MDB_env:
pass
int mdb_env_create(MDB_env **env)
And I am using it in a Cython script:
cdef MDB_env *e
x = mdb_env_create(&e)
This code compiles fine, but If I run it, I get:
ImportError: /home/me/.cache/ipython/cython/_cython_magic_15705c11c6f56670efe6282cbabe4abc.cpython-36m-x86_64-linux-gnu.so: undefined symbol: mdb_env_create
This happens both in a Cython .pyx + .pxd setup and in a prototype typed in IPython.
If I import another symbol, say a constant, I can access it. So I seem to be looking at the right header file.
I don't see any discrepancy between my syntax and the documentation, but I am clearly doing something wrong. Can somebody give me a hint?
Thanks.
To compile it with IPythons-magic (would be nice if you would mention this explicitly in your question) you have to provide library-path (via -L-option) and library name (via -l-option) of the built c-library you want to wrap, see also the documentation:
%%cython -L=<path to your library> -l=<your_library>
The library you are trying to wrap is not a header-only library. That means that some symbols (e.g. mdb_env_create) are only declared but not defined in the header. When you build the library, the definitions of those symbols can be found in the resulting artifact, which should be provided to the linker when your extension is built. These definitions is what is needed when the program runs.
If you don't do it, the following happens on Linux: When the extension (the *.so-file) is built,the linker allows undefined symbols per default - so this step is "successful" - but the failure is only postponed. When the extension is loaded via import, Python loads the corresponding *.so with help of ldopen and in this step loader checks that the definitions of all symbols are known. But we didn't provide a definition of mdb_env_create so, the loader fails with
undefined symbol: mdb_env_create
It is differently for symbols which are defined in the header-file, for example enums MDB_FIRST&Co - the compiled library isn't necessary and thus the extension can be loaded, as there are no undefined symbols.
The following code defines a simple Cython function (using Ipython magic, for convenience).
%load_ext cython
%%cython
def f(float x, float y=2):
return x+y
Then, calling help(f) gives this message:
Help on built-in function f in module _cython_magic_e37eeabbc63d5167217465ba978239fc:
f(...)
Note that the arguments of f are not shown. Also, the tab-completion does not work either for the argument names in ipython (e.g. typing f(x then tab).
If I define this function without using Cython:
def g(x, y=2):
return x+y
Calling help(g) gives this and the tab-completion works as expected:
Help on function g in module __main__:
g(x, y=2)
Is there a way to get this behavior with the Cython function? I tried with def, cdef, cpdef, with and without ipython magic, with no success.
Disagree with the agreed answer.
While enabling binding does have the side-effect of documentation strings showing up in code, it also binds all other Python class attributes to Cython extension classes which makes for lower performance, more memory used for each extension object and so on.
The correct flag to enable docstrings only without side effects is embedsignature=True.
It can either be used as decorator - #cython.embedsignature(True) on top of all functions, or part of cython directives in setup.py Extension to apply to all Cython functions - {'embedsignature': True}
From docs:
embedsignature (True / False)
If set to True, Cython will embed a textual copy of the call signature in the docstring of all Python visible functions and
classes. Tools like IPython and epydoc can thus display the signature,
which cannot otherwise be retrieved after compilation. Default is
False.
import cython
#cython.binding(True)
def f(float x, float y=2):
# ...
now help(f) gives
Help on cython_function_or_method in module cy_exc:
f(x, y=2.0)
The documentation says
When enabled, functions will bind to an instance when looked up as a class attribute (hence the name) and will emulate the attributes of Python functions, including introspections like argument names and annotations. Default is False.
You can enable the compilation option in other ways (for example, if you want it enabled everywhere).
You might also want to look at this related question
My C++ class:
class Base {
public:
virtual void foo(int);
};
In the python module, I have a class that derives from above and overrides foo.
class PyDerived(Base):
...
def foo(self):
...
Then I create an object of this derived class using a factory method that is defined in python module like this:
def createObject():
m = PyDerived()
return m
With this PyObject in C++ code, I want to call foo and I want foo in python module to be executed. Is this possible? If so how?
(I already tried calling virtual methods from python which dispatch the actual call to a C++ method, but that does not match my requirements)
Yes, you need to enable the SWIG "directors" feature for the class Base containing the virtual method. You can read about it in the documentation here.
The documentation will tell you that you need to add two things to the SWIG interface file:
At the very beginning, edit the %module directive to enable the use of directors at all:
%module(directors="1") your_modulename
Before the declaration of the class Base or the corresponding %include directive, put the following to enable the directors feature for that class:
%feature("director") Base;