Compile file with two separate libraries in Cython - cython

I wrote a library in Cython that has two different "modes":
If rendering, I compile using GLFW.
If not rendering, I compile using EGL, which is faster, but I have not figured out how to render with it.
What is the recommended way to handle this situation?
Right now, I have the following directory structure:
mujoco
├── __init__.py
├── simEgl.pyx
├── simGlfw.pyx
├── sim.pxd
└── sim.pyx
simEgl.pyx contains EGL code and simGlfw.pyx contains GLFW code. setup.py uses an environment variable to choose one or the other for the build.
This works ok, except that I need to recompile the code every time I want to switch between modes. There must be a better way.
Update
I agree that the best approach is to simultaneously compile two different libraries and use a toggle to choose which one to import. I already do have a base class in sim.pyx with shared functionality. However this base class must itself be compiled with the separate libraries. Specifically, sim.pyx depends on libmujoco.so which depends on either GLFW or EGL.
Here is my exhaustive search of possible approaches:
If I do not compile an extension for sim.pyx, I get ImportError: No module named 'mujoco.sim'
If I compile an extension for sim.pyx without including graphics libraries in the extension, I get ImportError: /home/ethanbro/.mujoco/mjpro150/bin/libmujoco150.so: undefined symbol: __glewBlitFramebuffer
If I compile an extension for sim.pyx and choose one set of graphics libraries (GLFW), then when I try to use the other set of graphics libraries (EGL) this does not work either unsurprisingly:
ERROR: GLEW initalization error: Missing GL version
If I compile two different versions of the sim.pyx library, one with one set of libraries, one with the other, I get: TypeError: unorderable types: dict() < dict() which is not a very helpful error message, but appears to result from trying to share a source file between two different extensions.
Something like option 4 should be possible. In fact, if I were working in raw C, I would simply build two shared objects side by side using the different libraries. Any advice on how to get around this Cython limitation would be very welcome.

(This answer is just a summary of the comments with a bit more explanation.)
My initial suggestion was to create two extension modules defining a common interface. That way you pick which to import in Python but be able to use them in the same way once imported:
if rendering:
import simGlfw as s
else:
import simEgl as s
s.do_something() # doesn't matter which you imported
It appears from the comments that the two modules also share a large chunk of their code and its really just the library that they're linked with that defines how they behave. Trying to re-use the same sources with
Extension(name='sim1', sources=["sim.pyx",...)
Extension(name='sim2', sources=["sim.pyx",...)
fails. This is because Cython assumes that the module name will be the same as the filename, and so creates a function PyInit_sim (on Python 3 - Python 2 is named slightly differently but the idea is the same). However, when you import sim1.so it looks for the function PyInit_sim1, fails to find it, and gives an error.
An easy way round it is to put the common code in "sim.pxi" and use Cython's largely obsolete include mechanism to textually include that code in sim1.pyx and sim2.pyx
include "sim.pxi"
Although include is generally no longer recommended and cimport is preferred since it provides more "Python-like" behaviour, include is a simple solution to this particular problem.

Related

Docstrings are not generated on Read the Docs with Sphinx autodoc and napoleon extensions

I am using the Sphinx autodoc and napoleon extensions to generate the documentation for my project (Qtools). This works well on my local machines. I am using Sphinx 3.1.2 (or higher). However, when I build the documentation on Read the Docs (RTD), only text added directly to the reStructuredText files that form the source of the documentation is processed. The docstrings that are supposed to be pulled in by autodoc do not appear in the HTML documentation generated by RTD. So for example in docs\source\section2_rsdoc.rst I have:
Response spectra
================
The response spectrum class
---------------------------
.. autoclass:: qtools.ResponseSpectrum
:members:
Response spectrum creation
--------------------------
.. autofunction:: qtools.calcrs
.. autofunction:: qtools.calcrs_cmp
.. autofunction:: qtools.loadrs
See also :func:`qtools.convert2rs` (converts a power spectrum into a response spectrum).
This results in:
Response spectra
The response spectrum class
Response spectrum creation
See also qtools.convert2rs (converts a power spectrum into a response spectrum).
In other words, all directives are apparently ignored, and hyperlinks to other functions are not added. I have examined several basic guidance documents such as this one, but I cannot figure out what I am doing wrong. RTD builds the documentation without any errors or warnings. In RTD advanced settings I have:
Documentation type: Sphinx HTML
Requirements file: requirements.txt
Python interpreter: CPython 3.x
Install Project: no
Use system packages: no
Python configuration file: blank
Enable PDF build: no
Enable EPUB build: no
I haven't touched any other settings.
In conf.py I have tried the following variations of line 15: sys.path.insert(0, os.path.abspath('.')), sys.path.insert(0, os.path.abspath('../..')) and the current sys.path.insert(0, os.path.abspath('../../..')). None of those made any difference.
I would be grateful for any help!
RTD builds the documentation without any errors or warnings
This is slightly incorrect. As you can see in the build logs, autodoc is emitting numerous warnings like this one:
WARNING: autodoc: failed to import class 'ResponseSpectrum' from module 'qtools'; the following exception was raised:
No module named 'qtools'
This has happened for all your variations of sys.path.insert, as you can see in some past builds.
Trying to make it work this way is tricky, since Read the Docs does some magic to guess the directory where your documentation is located, and also the working directory changes between commands.
Instead, there are two options:
Locate where the conf.py is located (see How do you properly determine the current script directory?) and work out a relative package from there.
Invest some time into making your code installable using up-to-date Python packaging standards, for example putting all your sources inside a qtools directory, and creating an appropriate pyproject.toml file using flit.

Load ordinary JavaScripts in Polymer 3 elements or lit-elements

I am in the middle of converting a Polymer 2 app to Polymer 3. Modulizer did not work for me so I converted it manually. Thanks to the great upgrade guide it has been mostly straight forward so far.
One task is left though:
in my Polymer 2 app I had a special html import (d3-import.html) that brought in the d3.js lib version 3 which comes as a plain JavaScript file (no ES6 module!). This import was dynamically loaded in only two out of overall 20 pages because the other 18 pages did not need it.
In Polymer 3 I can not import it as an ES6 module because it is not a module. Loading it in my main start.html would mean it gets loaded even if the user only uses the other 18 pages that don't need it.
I tried writing script-tags in my web component templates but that doesn't seem to work. Unfortunately I don't see any error in the browser tools. The template simply stops to load at the line of the script-tags.
Any idea how to do this?
Additional question:
since I start using lit-element in the same application. How to solve the same problem with lit-element?
Edit: note that I currently don't use any build steps/tools except for polymer-build to replace the module paths with actual file paths.
Note that this challenge has nothing to do with Polymer or LitElement, this is only an issue with how to load non-module resources from a module.
The most straightforward way that I know of is to use a bundler like Rollup that can support CommonJS or UMD. Rollup has the commonjs plugin for this: https://github.com/rollup/plugins/tree/master/packages/commonjs
The other option is to upgrade to D3 5.x, which appears to be published as standard modules itself. Given the number of files involved, you'll still likely want a bundler to reduce network roundtrips.

Failed to create shared library with wx and STL, "multiple definition" error?

I tried to build a shared library using wx and STL, and failed in an error of "multiple definition of". Please refer to:
https://code.google.com/p/gppanel/issues/detail?id=7
The declaration of wxPointListNode is not found in the sources. The suspicious lines are like these:
include/mathplot.h:85:WX_DECLARE_LIST(wxPoint, PointList);
include/mathplot.h:87:WX_DEFINE_LIST(PointList);
include/gpLineLayer.h:16:typedef std::deque<mpPointLayer*> mpPointList_t;
What the problem is?
Without the actual code this is just a guess, but I suspect that
include/mathplot.h:87:WX_DEFINE_LIST(PointList);
generates the full definition of PointList, including a non-templated method wxPointListNode::DeleteData. mathplot.h is included by all of the .cpp files (gpPanel.cpp, gpSeries.cpp, and baseData.cpp). Each cpp file is compiled into a .o file, so each has its own definition of DeleteData, and when you try to link the .o files together into lib/libgpPanel.so the linker issues the errors you're reporting.
The definition of the method needs to be in its own cpp file that's compiled and linked in.
All wxWidgets methods with DEFINE in their name expand into a definition of something and a definition can only be used once in a module, so it typically can't appear in a header file (unless you can guarantee that it's included by only a single source file). So just don't put it there.
Moreover, if this is your code, you should avoid using the legacy WX_DECLARE_LIST macro at all and just use std::list<> or std::vector<> instead. Or, if you really want to use only wx (which can only be important if you are targeting some embedded platform without good STL implementation), then use wxVector<>.

SimpleCV 1.3 update & pygame

I recently updated to SimpleCV 1.3. When I try to run examples I get the following:
objc[92210]: Class SDLTranslatorResponder is implemented in both /Library/Frameworks/SDL.framework/Versions/A/SDL and /usr/local/lib/libSDL-1.2.0.dylib. One of the two will be used. Which one is undefined.
objc[92210]: Class SDL_QuartzWindow is implemented in both /Library/Frameworks/SDL.framework/Versions/A/SDL and /usr/local/lib/libSDL-1.2.0.dylib. One of the two will be used. Which one is undefined.
objc[92210]: Class SDL_QuartzWindowDelegate is implemented in both /Library/Frameworks/SDL.framework/Versions/A/SDL and /usr/local/lib/libSDL-1.2.0.dylib. One of the two will be used. Which one is undefined.
objc[92210]: Class SDL_QuartzView is implemented in both /Library/Frameworks/SDL.framework/Versions/A/SDL and /usr/local/lib/libSDL-1.2.0.dylib. One of the two will be used. Which one is undefined.
WARNING: You need the python image library to save by filehandle
WARNING: You need the python image library to save by filehandle
WARNING: You need the python image library to save by filehandle
Considering this is related to the pygame installations before, when I remove the old pygame folders (There are few of them) I get the error: "ImportError: No module named pygame".
How can I get rid of this? This is something I must take care of until Tomorrow night.
ImportError means the correct folder isn't in your PYTHONPATH environment variable.

AS3 alternatives to *_exclude.xml files, loading class definitions dynamically

I've been banging my head against a wall for this for almost a couple days now and hoping that someone can point me in the right direction.
Working in a very large Flash application, previously in AS2/CS3 I would have a setup like the following:
root.swf
-- modules
---- code_a.swf
---- code_b.swf
-- views
---- view_a.swf
---- view_b.swf
Using _exclude.xml files, I could exclude the classes defined in code_a and code_b from the ouptut .swf of view_a and view_b. root.swf would be responsible for loading the code modules before view_a or view_b, ensuring that class definitions that view_a and view_b depended on existed.
The Problem
We've recently migrated to using Actionscript 3/CS5. *_exclude.xml files no longer exist. To get the same functionality as above, I've tried the following:
My setup now looks something like:
root.swf
-- modules
---- class_a.as
---- class_b.as
-- views
---- view_a.swf
---- view_b.swf
Use mxmlc to compile root.swf, view_a.swf and view_b.swf, passing it -externs option to specify classes that will be loaded externally (the two classes in modules). This ensures that the class is excluded from the compiled swf.
Use compc to compile class_a.as and class_b.as into classes.swf, using -directory=true to access library.swf for external loading.
However, when I try running one of the two view files which depend on classes.swf, I get runtime errors telling me that a class definition is not present.
Current Workaround
I've devised a workaround which I'm currently not happy with as it's backwards to the modular approach that I was previously using:
Rather than loading the code modules, I statically link all class definitions required by child movies into root.swf. When building root.swf, I use the -link-report option of mxmlc to provide a list of included classes. When building child swfs, I can the use -load-externs to ensure that class definitions that already exist will not be included in the compile output.
Is there a way that anyone is aware of to replicate the AS2/_exclude.xml solution that I had using AS3/CS5?
I'd recommend compiling shared libraries to SWCs.
There are other options such as RSLs:
http://help.adobe.com/en_US/flex/using/WS2db454920e96a9e51e63e3d11c0bf674ba-7fff.html#WS2db454920e96a9e51e63e3d11c0bf69084-7f1e
Hope that helps.