I want to compile and link some cython (and cython_freeze) generated c files into an executable (on Windows and Linux)
I can do this by setting up (a machine specifc) environment (eg what C compiler, where is it installed (path), what include directories, what lib directories...) and calling cl directly (with some os specific options).
But cythonize is is able to compile cython created c files into dynamic libraries without me explicitly setting any machine or os specific settings (just using distutils).
Is there a simple way of using distutils / setuptools to do that (ie compile a c file into a object and link multiple objects into a executable)?
NB: I don't want any of the complex setup logic normally in setup.py. Just the plain compile and link primitives for my development machines.
There is a quite simple solution (that is not really documented):
import sys
import sysconfig
#from distutils.ccompiler import new_compiler # distutils is currently deprecated
from setuptools._distutils.ccompiler import new_compiler
compiler = new_compiler()
compiler.add_include_dir(sysconfig.get_path("include"))
compiler.add_library_dir(sysconfig.get_config_var("installed_platbase") + "/libs")
if sys.platform == "linux":
compiler.add_library_dir(sysconfig.get_config_var("LIBPL"))
compiler.add_library_dir(sysconfig.get_config_var("LIBDIR"))
compiler.add_library(sysconfig.get_config_var("LIBRARY")[3:-2])
for lib in sysconfig.get_config_var("LIBS").split():
compiler.add_library(lib[2:])
objectFiles = compiler.compile(["source1.c", "source2.c", "test.rc"])
compiler.link_executable(objectFiles, "test")
Related
I have a very simple question:
I failed to find any tutorial on using the Spyder cython console. How should be the cython code used?
On the official site only states that "A Cython console allows you to use the Cython language to speed up your code and call C functions directly from Python." But how?
Following this tutorial I tried to run the setup.py, the hello.pyx, and got errors when trying %cython, %%cython, or cdef in .py or .ipy files.
.c or .cpp files cannot be run at all.
Moreover, when I try to run a file containing simply print("bum") or a = 1 in cython console, nothing happens (it works when written directely to the console).
Many thanks for a little push.
In my notebook I get some data from URL, perform some analysis and do some plotting.
I want also create a html animation using FuncAnimation of matplotlib.animation.
So in the preamble I do
import matplotlib.animation as manim
plt.rcParams["animation.html"] = "html5"
%matplotlib inline
(something else... def init()..., def animate(i)...) then
anima = manim.FuncAnimation(fig,
animate,
init_func=init,
frames=len(ypos)-d0,
interval=200,
repeat=False,
blit=True)
To visualise, I then call
FFMpegWriter = manim.writers['ffmpeg']
writer = FFMpegWriter(fps=15)
link = anima.to_html5_video()
from IPython.core.display import display, HTML
display(HTML(link))
because I want the clip to show up as a neat html video in the notebook
Whereas this works well on my machine, on Watson-Studio I get the following error:
RuntimeError: Requested MovieWriter (ffmpeg) not available
I've checked that ffmpeg is available in the form of a Python package
(!pip freeze --isolated | grep ffmpeg gives ffmpeg-python==0.2.0)
The question is: how can I tell matplotlib.animation.writers to use the codec in ffmpeg-python?
Many thanks to all responders and supporters
We currently don't have ffmpeg pre-installed in Watson Studio on Cloud. The package ffmpeg-python that you mention is just a Python wrapper, but it won't work without the actual ffmpeg.
You can install ffmpeg from conda:
!conda install ffmpeg
Once you have the full list of additional packages that your notebook needs, I recommend to create a custom environment. Then you don't have to put install commands into the actual notebook.
The customization might look like this:
dependencies:
- ffmpeg=4.2.2
- pip
- pip:
- ffmpeg-python==0.2.0
I wrote a library in Cython that has two different "modes":
If rendering, I compile using GLFW.
If not rendering, I compile using EGL, which is faster, but I have not figured out how to render with it.
What is the recommended way to handle this situation?
Right now, I have the following directory structure:
mujoco
├── __init__.py
├── simEgl.pyx
├── simGlfw.pyx
├── sim.pxd
└── sim.pyx
simEgl.pyx contains EGL code and simGlfw.pyx contains GLFW code. setup.py uses an environment variable to choose one or the other for the build.
This works ok, except that I need to recompile the code every time I want to switch between modes. There must be a better way.
Update
I agree that the best approach is to simultaneously compile two different libraries and use a toggle to choose which one to import. I already do have a base class in sim.pyx with shared functionality. However this base class must itself be compiled with the separate libraries. Specifically, sim.pyx depends on libmujoco.so which depends on either GLFW or EGL.
Here is my exhaustive search of possible approaches:
If I do not compile an extension for sim.pyx, I get ImportError: No module named 'mujoco.sim'
If I compile an extension for sim.pyx without including graphics libraries in the extension, I get ImportError: /home/ethanbro/.mujoco/mjpro150/bin/libmujoco150.so: undefined symbol: __glewBlitFramebuffer
If I compile an extension for sim.pyx and choose one set of graphics libraries (GLFW), then when I try to use the other set of graphics libraries (EGL) this does not work either unsurprisingly:
ERROR: GLEW initalization error: Missing GL version
If I compile two different versions of the sim.pyx library, one with one set of libraries, one with the other, I get: TypeError: unorderable types: dict() < dict() which is not a very helpful error message, but appears to result from trying to share a source file between two different extensions.
Something like option 4 should be possible. In fact, if I were working in raw C, I would simply build two shared objects side by side using the different libraries. Any advice on how to get around this Cython limitation would be very welcome.
(This answer is just a summary of the comments with a bit more explanation.)
My initial suggestion was to create two extension modules defining a common interface. That way you pick which to import in Python but be able to use them in the same way once imported:
if rendering:
import simGlfw as s
else:
import simEgl as s
s.do_something() # doesn't matter which you imported
It appears from the comments that the two modules also share a large chunk of their code and its really just the library that they're linked with that defines how they behave. Trying to re-use the same sources with
Extension(name='sim1', sources=["sim.pyx",...)
Extension(name='sim2', sources=["sim.pyx",...)
fails. This is because Cython assumes that the module name will be the same as the filename, and so creates a function PyInit_sim (on Python 3 - Python 2 is named slightly differently but the idea is the same). However, when you import sim1.so it looks for the function PyInit_sim1, fails to find it, and gives an error.
An easy way round it is to put the common code in "sim.pxi" and use Cython's largely obsolete include mechanism to textually include that code in sim1.pyx and sim2.pyx
include "sim.pxi"
Although include is generally no longer recommended and cimport is preferred since it provides more "Python-like" behaviour, include is a simple solution to this particular problem.
i have many package requires which is used inside the tcl code as below:
package require tlautils 1.0
package require profilemanager
package require tla::log 1.2
package require res_mgr 1.1
package require resource 1.2
package require sysexits
package require utils 1.5
package require testutil 1.0
package require profilemgr
package require leakChecker
package require testutil
what is the alternative to use instead of using so many package requires? this is taking time and i am in search of any other alternatives for package require whcih increases the time in seconds/miliseconds
The package require lines don't really take much longer than the load and source calls that they delegate to (all the packages are doing is stopping you from having to hard-code paths to everything, taking care of all the details of versions and so on). However, when you do package require of a package whose name is not already known, Tcl has to actually search for the pkgIndex.tcl files that describe how to actually load the packages. It does this by calling the code that you can look up (or replace if necessary) using package unknown, and that's actually really quite slow. Depending on the TCLLIBPATH environment variable's contents, it could be extremely slow.
But we can “compile” that so that we will be able to source a single file and be able to load these specific packages on this machine quickly.
To do that, we need the above package requires and a bit of extra wrapping code:
package require tlautils 1.0
package require profilemanager
package require tla::log 1.2
package require res_mgr 1.1
package require resource 1.2
package require sysexits
package require utils 1.5
package require testutil 1.0
package require profilemgr
package require leakChecker
package require testutil
set f [open "pkgdefs.tcl" w]
foreach pkg [package names] {
# Skip the packages built directly into Tcl 8.6
if {$pkg in {zlib TclOO tcl::tommath Tcl}} continue
# Get the actual loaded version; known but absent would be an error
if {[catch {package present $pkg} version]} continue
# Get the actual script that we'd run to load the package.
set script [package ifneeded $pkg $version]
# Add to the file
puts $f [list package ifneeded $pkg $version $script]
}
close $f
Once you've run that, you'll have a script in pkgdefs.tcl that you can source. If in future runs you source it before doing any of those package require calls that you listed, those package require calls will be fast. (This also includes any packages that are dependencies of the ones you list.) However, if you ever install new packages to use in the list, or update the versions of the packages, or move them around, you'll need to rebuild the list: it makes your code quite a lot more inflexible, which is why we don't do it by default.
Just a disclaimer - this is a hypothetical scenario as I'm trying to figure out the best way to handle web component direct and transitive dependencies using something like JSPM and SystemJS.
Scenario 1
Suppose I have 2 web components - component-a and component-b. One is built with momentjs#1.2.4 and another is built with momentjs#1.6.4.
Each component author listed Polymer as a peer dependency and momentjs as a direct dependency in their package.json file like this:
"dependencies": {
"moment.js": ">= 1.0.1 <= 1.8.0"
}
"peerDependencies": {
"Polymer.js": "0.5.0^",
}
So in this case when the developer declares the package.json dependencies on both of these components the package manager could figure out the best version of moment.js to install and make it available. Lets assume that's version 1.8.0.
Great - just one version of moment.js to deal with for both components. Now how do the components load / inject the dependency? Does JSPM and SystemJS currently have the ability to support this?
For example in this article Taming Polymer with SystemJS and Typescript the author performs imports that delegate to SystemJS like this:
import 'elements/app-frontend';
So I suppose momentjs would follow a similar approach and import like this (Within the typescript files of component-a and component-b:
import 'js/momentjs';
In the above case the momentjs dependency does not know what version will be fetched since the import declaration is unaware of the version (since it written at design time).
Later on when the component is used in and component is used in an application JSPM figures out the best for the momentjs depedency to install.
In this case lets assume that it install it in a layout like this:
`jspm_packages/momentjs/momentjs#1.8.0`
So how does JSPM become aware the import statement import 'js/momentjs' translates to the import of jspm_packages/momentjs/momentjs#1.8.0?
In this case it's fairly trivial, but it get more tricky in scenario 2 below ... I think.
Scenario 2
Same as scenario one except the version requirements on momentjs are mutually exclusive. For example component-a requires version 1.2.4 and component-b requires version 2.4.4.
So both components perform the import like this:
import 'js/momentjs';
But jspm_packages installs both versions like this:
`jspm_packages/momentjs/momentjs#1.4.4`
`jspm_packages/momentjs/momentjs#2.4.4`
So now how does SystemJS know to that component-a needs version 1.4.4 and component-b needs version 2.4.4?
To summarize:
1) Is JSPM ok with having multiple versions of the same dependency?
According to this article Introduction to the Jspm package manager and the SystemJs module loader and an answer from guybeford it does, but how then does each component get the right version?
2) Does SystemJS / JSPM have a way of configuring version meta data for each component?
3) Does SystemJS have a way of understanding and injecting the right version of a dependency into the web component?
TIA,
Ole
Also what happens if the component requires several other modules, like CSS etc?
I'm not sure what you mean. You can import multiple modules and other contents (likely via plugins) usting SystemJS.
1) Is JSPM ok with having multiple versions of the same dependency?
I think you partially answered this question. JSPM will always select and install only one version for each component, which best matches the requirements from other components. If there is a conflict, JSPM will ask to choose manually.
So how does JSPM become aware the import statement import 'js/momentjs' translates to the import of jspm_packages/momentjs/momentjs#1.8.0?
The config.js file assigns import names (not sure the nomenclature) for installed packages. It is actually possible to change those names using jspm install x=npm:package syntax. This will result in config like
"x": "npm:package#1.2.3"
You can then both import x and import npm:package#1.2.3.
2) Does SystemJS / JSPM have a way of configuring version meta data for each component?
I think that the structure of config.js has all you need. It contains dependencies with their versions.
"component-a": {
"momentjs": "momentjs#1.2.4"
},
"component-b": {
"momentjs": "momentjs#1.6.4"
}
3) Does SystemJS have a way of understanding and injecting the right version of a dependency into the web component?
I would expect SystemJS to use this information to import the correct version when either component-a or component-b is requested.
I am not sure though how to have JSPM install packages this way, retaining multiple versions of a package. I think it deserves a separate, specialized question.