Is there a way to register a function within an Cython extenstion module such that this function is called when the Python garbage collector destroys the module object ?
Something like
def __dealloc__(...)
on module level ?
I'm not sure there is an explicit/official way to achieve that. However, you could add a global variable which will be destroyed, when the cython module is deallocated:
# foo.pyx:
class Observer:
def __init__(self):
print("module initialized")
def __del__(self):
print ("module deleted")
_o=Observer()
However, cythonizing it via cythonize foo.pyx -i will not lead to the desired effect:
[$] python -c `import foo`
module initialized
there is no "module deleted" printed to the console!
The problem: one cannot reload a cython-extension anyway (for example via importlib.reload()), so it gets only deallocated when the Python interpreter is shut down. But then there is no need to deallocate the resources ...
Cython keeps all global Python-variables in a global C-variable which is called static PyObject *__pyx_d; in the cythonized C-source; this is just a Python-dictionary. However, because there is a reference to the global _o in this not destroyed dictionary, the reference count of _o will never become 0 an thus the object doesn't get destroyed.
By setting generate_cleanup_code=True one can force Cython to generate a clean-up function which will be put into the m_free-slot of the PyModuleDef definition struct. For example with the following setup-file:
from distutils.core import setup
from Cython.Build import cythonize
from Cython.Compiler import Options
Options.generate_cleanup_code = True # this is important!
setup(
name = "foo",
ext_modules = cythonize("foo.pyx"),
)
And now after python setup.py build_ext -i:
[$] python -c "import foo"
module initialized
module deleted
it works as expected.
Because there is no reload for Cython-extensions, "module deleted" will be only seen when module_dealloc is called.
However, when the first listing were pure-Python, we would also see module deleted if importlib.reload(foo) were called. In this case "module_dealloc" isn't used, but all references to global _o would be cleared and object would be destroyed.
Depending on what one want, it might be either the expected or unexpected behavior.
Related
I'm generating different .pyx files and turning them to C++ .so libraries. So I got an example generatedLib.pyx like this:
from libcpp.vector cimport vector
def cinit(self): pass
cdef public void someFunction(vector[...]& inputs):
...
The public function always has same signature, except for the name. I somehow wanna call this public function using python interpreter because I prefer to write tests in Python rather than C++. I cannot do it directly of course. I cannot define this function as def as well since its parameter is of C++ vector type.
I have came up with an idea to have a wrapper function
def wrapperSomeFunction(input: list)
{
# convert Python list to c++ vector
someFunction(v)
# convert c++ vector back to python list and return
}
I can import the generatedLib as python module and call this wrapper just fine, it requires however, to append it to my generatedFile.pyx file contents every time I wanna test it before I build it. I would rather look for alternative solutions.
Is there some way I can build a separate .pyx which will have this wrapper inside and will be able to call my generatedLib.so? Or perhaps something else I didn't think of?
If I made a python file named hello.py that has a script made like this.
msg = input("insert your message here: ")
script = '''
def say_something():
print("{msg}")
'''
exec(script)
say_something()
And then I tried to use Cython
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules=cythonize("Hello.py")
)
It will show an error like this: undeclared name not builtin: say_something
I do understand why this happens but I'm not really an expert with python and C just yet. This is just an example, but it's similar to what I'm trying to do with one of my projects. Is there any way I could resolve this? I want to find a way to convert the script string into C as well.
I was trying to build an editable python script.
Cython compiles the Python functions to a native binary that does what the CPython interpreter should do. exec is a function that execute arbitrary code at runtime (which is generally a very bad idea for speed, maintainability/readability and security). Cython does not support exec because it would mean that the could would be compiled at runtime. Thus, the code executed by exec cannot be a Cython code. However, the exec function can still be used to execute a pure-Python code. The error can be removed by turning off the Cython.Compiler.Options.error_on_unknown_names in the setup script (just before calling setup) as pointed out by #DavidW. With this Cython will not complain when it does not find a function defined by exec (or similar methods). Please keep in mind that CPython can only be used in this case instead of Cython (which partially defeat the purpose of using Cython in the first place).
How do I compile using Cython, a C function from file simulate_fast.c, where this also depends on further C files matrix.c and random_generator.c. This seems like it must be a common use of Cython but after reading through the documentation, I still cannot figure out how to do it. My directory contains the following files
matrix.c
matrix.h
random_generator.c
random_generator.h
simulate_fast.c
simulate_fast.h
test.c
simulate_fast_c.pyx
setup.py
The matrix.c and random_generator.c files contain standalone functionality. The simulate_fast.c file uses both of these and contains the function I want to be exposed to Python, simulate().
The test.c file tests that all the C functionality runs correctly i.e. I can execute
$ gcc test.c simulate_fast.c matrix.c random_generator.c -o test
to compile into a test executable that works.
My issue now is trying to compile this with Cython. My .pyx file is
cimport cython
cdef extern from "simulate_fast.h":
int simulate()
def simulate_cp():
return simulate()
I then use the basic setup.py
from distutils.core import setup
from Cython.Build import cythonize
from Cython.Distutils import build_ext
import numpy as np
setup(
name='simulate_fast',
ext_modules = cythonize(["simulate_fast_c.pyx"]),
include_dirs=[]
)
However if I try and compile this using
python3 setup.py build_ext --inplace
I get an error
In file included from simulate_fast_c.c:492:0:
simulate_fast.h:89:28: error: field ‘RG’ has incomplete type
struct RandomGenerator RG;
where structure RandomGenerator is declared in random_generator.h.
How do I tell the compiler we must also consider the matrix and random_generator files when compiling.
Update
If doing as ead said in the comments, I include random_generator.h and matrix.h in simulate_fast.h then the program now compiles. However when I try to import the simulate_fast_c module in Python I get an ImportError:
undefined symbol: simulate
Further, if I change the extern declaration line in simulate_fast.pyx to
cdef extern from "simulate_fast.c":
int simulate()
I then get an import error
undefined symbol: get_random_number
which is a function in random_generator.h
The cythonized module must either be linked to a shared library containing the compiled C code or embed it. One way to do the latter is to list the C sources as being an "Extension", then pass this extension to the cythonize command, as mentioned in Cython's documentation
The example setup.py file in the link can be summarized by (modulo the imports):
setup(
ext_modules = cythonize(Extension("simulate_fast", ["matrix.c", "random_generator.c", "simulate_fast_c.pyx"]))
)
I want to make a parity_check function which can be accessed by three different modules. Is this possible in SV? If yes, where do I declare this function and how do I import it into my module?
You can put a function into a separate file and include it using `include:
`include "some_file_name.sv"
However, a much better way is to use a package:
package some_package_name;
function string some_function_name;
return "some_function_name called";
endfunction
endpackage
You would put that into a separate file and you must compile that before compiling any module that uses it. You then import the package into each module:
module some_module_name;
import some_package_name::*; // or import some_package_name::some_function_name;
initial
$display(some_function_name);
endmodule
Putting a function in a package is better than just putting it into a file and using include, because a package is a named scope. Because a package is a named scope, any issues with some clash of names can be resolved by, instead of using import, referring to the full name of the function in its package, eg:
module some_module_name;
initial
$display(some_package_name::some_function_name);
endmodule
I'm new in python, my native language is C. I'm doing a code in python for a surveillance system triggered by motion using OpenCV. I based my code in the one made by Adrian Rosebrock in his blog pyimagesearch.com. Originally the code was developed for a Raspiberry Pi with a Pi Camera module attached to it, now I'm trying to adapt to my notebook's webcam. He made a easier tutorial about a simple code for motion detection and it worked very nicely in my PC. But I'm having a hardtime with this other code. Probably it's a silly mistake, but as begginer I couldn't found a specific answer to this issue.
This image have the part of the code that is causing the error (line 15) and the structure of the project on the left side of the screen. Image of python project for surveillance.
Similar part, originall code:
# import the necessary packages
from pyimagesearch.tempimage import TempImage
from dropbox.client import DropboxOAuth2FlowNoRedirect
from dropbox.client import DropboxClient
from picamera.array import PiRGBArray
from picamera import PiCamera
import argparse
import warnings
import datetime
import imutils
import json
import time
import cv2
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-c", "--conf", required=True,
help="path to the JSON configuration file")
args = vars(ap.parse_args())
# filter warnings, load the configuration and initialize the Dropbox
# client
warnings.filterwarnings("ignore")
conf = json.load(open(args["conf"]))
client = None
Until now I only change these things:
Exclude the imports relatives to pi camera.
Change camera = PiCamera() by camera = cv2.VideoCapture(0). This way I use notebook's webcam.
Exclude:
camera.resolution = tuple(conf["resolution"])
camera.framerate = conf["fps"]
rawCapture = PiRGBArray(camera, size=tuple(conf["resolution"]))
Substitute the line for f in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True): by while True:.
Exclude two lines in program that was rawCapture.truncate(0).
Probably there is more things to repair, if you now please tell me, but first I'd like to understand how solve that mensage error. I use PyCharm in Windows 7 with Python 2.7 and OpenCV 3.1. Sorry for not post the entire code, but once that this is my first question in the site and I have 0 reputation, apparently I can just post 2 links. The entire originall code is in the pyimagesearch.com. Thank you for your time!
I think you probably not running it properly. Error message is clear. You are adding argument that means you need to provide them while running which you are not doing.
Check this how he ran this in tutorial link you provided
http://www.pyimagesearch.com/2015/06/01/home-surveillance-and-motion-detection-with-the-raspberry-pi-python-and-opencv#crayon-56d3c5551ac59089479643
Notice on the Figure 6 screen capture in #Rhoit's link.
python pi_surveillance.py --conf conf.json
The program was initialized with the name and these --conf conf.json words.
In your code:
ap = argparse.ArgumentParser()
ap.add_argument("-c", "--conf", required=True,
help="path to the JSON configuration file")
ap is a piece of code that reads these inputs from the commandline, and parses the information. This definition specifies that a --conf argument is required, as demonstrated in Figure 6.
The error indicates that you omitted this information:
argument -c/--conf is required