I've successfully pickled persistent instances freshly loaded from the DB, but I can't seem to do the same for an instance that I've just created, and is in session.new.
Getting the following error (the python pickle module had the more helpful version of the message):
*** PicklingError: Can't pickle <function <lambda> at 0xb08d3ac>:
it's not found as sqlalchemy.ext.associationproxy.<lambda>
If I clear the association_proxy by doing the following:
my_new_obj.my_proxy = []
del my_new_obj.my_proxy
my_new_obj pickles fine.
Any ideas how I can have my association_proxy and eat it too?
Might have found a solution here:
http://www.sqlalchemy.org/trac/ticket/1446
(solution is to upgrade)
Related
I am working on topic inference that will require to load a previously saved model.
However, I got a pickle error that says
Traceback (most recent call last):
File "topic_inference.py", line 35, in <module>
model_for_inference = gensim.models.LdaModel.load(model_name, mmap = 'r')
File "topic_modeling/env/lib/python3.8/site-packages/gensim/models/ldamodel.py", line 1663, in load
result = super(LdaModel, cls).load(fname, *args, **kwargs)
File "topic_modeling/env/lib/python3.8/site-packages/gensim/utils.py", line 486, in load
obj = unpickle(fname)
File "topic_modeling/env/lib/python3.8/site-packages/gensim/utils.py", line 1461, in unpickle
return _pickle.load(f, encoding='latin1') # needed because loading from S3 doesn't support readline()
TypeError: __randomstate_ctor() takes from 0 to 1 positional arguments but 2 were given
The code I use to load the model is simply
gensim.models.LdaModel.load(model_name, mmap = 'r')
Here is the code that I use to create and save the model
model = gensim.models.ldamulticore.LdaMulticore(
corpus=comment_corpus,
id2word=key_word_dict, ## This is now a gensim.corpora.Dictionary Object, previously it was the .id2token attribute
chunksize=chunksize,
alpha='symmetric',
eta='auto',
iterations=iterations,
num_topics=num_topics,
passes=epochs,
eval_every=eval_every,
workers = 15,
minimum_probability= 0.0)
model.save(output_model)
where output_model doesn't have an extension like .model or .pkl
In the past, I tried the similar approach with the exception that I passed in a .id2token attribute under the gensim.corpora.Dictionary object instead of the full gensim.corpora.Dictionary to the id2word parameter when I created the model, and the method loads the model fine back then. I wonder if passing in a corpora.Dictionary will make a difference in the loading output...? Back that time, I was using regular python, but now I am using anaconda. However, all the versions of the packages are the same.
Another report of an error about __randomstate_ctor (at https://github.com/numpy/numpy/issues/14210) suggests the problem may be related to numpy object pickling.
Is there a chance that the configuration where your load is failing is using a later version of numpy than when the save occurred? Could you try, at least temporarily, rolling back to some older numpy (that's still sufficient for whatever Gensim you're using) to see if it helps?
If you find any load that works, even in a suboptimal config, you might be able to null-out whatever random-related objects are causing the problem and re-save, then having a saved version that loads better in your truly-desired configuration. Then, if the random-related objects truly needed after reload, it may be possible to manually re-constitute them. (I haven't looked into this yet, but if you find any workaround allowing a load, but then aren't sure what to manually null/rebuild, I could take a closer look.)
I am trying to run some example python3 code
https://docs.databricks.com/applications/deep-learning/distributed-training/horovod-runner.html
on databricks GPU cluster (with 1 driver and 2 workers).
Databricks environment:
ML 6.6, scala 2.11, Spark 2.4.5, GPU
It is for distributed deep learning model training.
I just tried a very simple example at first:
from sparkdl import HorovodRunner
hr = HorovodRunner(np=2)
def train():
print('in train')
import tensorflow as tf
print('after import tf')
hvd.init()
print('done')
hr.run(train)
But, the command is alway running without any progress.
HorovodRunner will stream all training logs to notebook cell output. If there are too many
logs, you
can adjust the log level in your train method. Or you can set driver_log_verbosity to
'log_callback_only' and use a HorovodRunner log callback on the first worker to get concise
progress updates.
The global names read or written to by the pickled function are {'print', 'hvd'}.
The pickled object size is 1444 bytes.
### How to enable Horovod Timeline? ###
HorovodRunner has the ability to record the timeline of its activity with Horovod Timeline.
To
record a Horovod Timeline, set the `HOROVOD_TIMELINE` environment variable to the location
of the
timeline file to be created. You can then open the timeline file using the chrome://tracing
facility of the Chrome browser.
Do I miss something or need to set up something to make it work ?
Thanks
your code does no actual training within it.. you might have better luck with the better example code
https://docs.databricks.com/applications/machine-learning/train-model/distributed-training/mnist-pytorch.html
I'm having issues loading a pretrained xgboost model using the following code:
xgb_model = pickle.load(open('churnfinalunscaled.pickle.dat', 'rb'))
And when I do that, I get the following error:
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-29-31e7f426e19e> in <module>()
----> 1 xgb_model = pickle.load(open('churnfinalunscaled.pickle.dat', 'rb'))
ModuleNotFoundError: No module named 'sklearn.preprocessing._label'
I haven't seen anything online so any help would be much appreciated.
I was able to solve my issue. Simply update scikit-learn from 0.21.3 to 0.22.0 seems to solve the issue. Along the way I have to update my pandas version to 0.25.2 as well.
The cue is provided in this link: https://www.gitmemory.com/vruusmann, where it states:
During Scikit-Learn version upgrade from 0.21.X to 0.22.X many modules were renamed (typically, by prepending an underscore character to the module name). For example, sklearn.preprocessing.label.LabelEncoder became sklearn.preprocessing._label.LabelEncoder.
I'm new in python, my native language is C. I'm doing a code in python for a surveillance system triggered by motion using OpenCV. I based my code in the one made by Adrian Rosebrock in his blog pyimagesearch.com. Originally the code was developed for a Raspiberry Pi with a Pi Camera module attached to it, now I'm trying to adapt to my notebook's webcam. He made a easier tutorial about a simple code for motion detection and it worked very nicely in my PC. But I'm having a hardtime with this other code. Probably it's a silly mistake, but as begginer I couldn't found a specific answer to this issue.
This image have the part of the code that is causing the error (line 15) and the structure of the project on the left side of the screen. Image of python project for surveillance.
Similar part, originall code:
# import the necessary packages
from pyimagesearch.tempimage import TempImage
from dropbox.client import DropboxOAuth2FlowNoRedirect
from dropbox.client import DropboxClient
from picamera.array import PiRGBArray
from picamera import PiCamera
import argparse
import warnings
import datetime
import imutils
import json
import time
import cv2
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-c", "--conf", required=True,
help="path to the JSON configuration file")
args = vars(ap.parse_args())
# filter warnings, load the configuration and initialize the Dropbox
# client
warnings.filterwarnings("ignore")
conf = json.load(open(args["conf"]))
client = None
Until now I only change these things:
Exclude the imports relatives to pi camera.
Change camera = PiCamera() by camera = cv2.VideoCapture(0). This way I use notebook's webcam.
Exclude:
camera.resolution = tuple(conf["resolution"])
camera.framerate = conf["fps"]
rawCapture = PiRGBArray(camera, size=tuple(conf["resolution"]))
Substitute the line for f in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True): by while True:.
Exclude two lines in program that was rawCapture.truncate(0).
Probably there is more things to repair, if you now please tell me, but first I'd like to understand how solve that mensage error. I use PyCharm in Windows 7 with Python 2.7 and OpenCV 3.1. Sorry for not post the entire code, but once that this is my first question in the site and I have 0 reputation, apparently I can just post 2 links. The entire originall code is in the pyimagesearch.com. Thank you for your time!
I think you probably not running it properly. Error message is clear. You are adding argument that means you need to provide them while running which you are not doing.
Check this how he ran this in tutorial link you provided
http://www.pyimagesearch.com/2015/06/01/home-surveillance-and-motion-detection-with-the-raspberry-pi-python-and-opencv#crayon-56d3c5551ac59089479643
Notice on the Figure 6 screen capture in #Rhoit's link.
python pi_surveillance.py --conf conf.json
The program was initialized with the name and these --conf conf.json words.
In your code:
ap = argparse.ArgumentParser()
ap.add_argument("-c", "--conf", required=True,
help="path to the JSON configuration file")
ap is a piece of code that reads these inputs from the commandline, and parses the information. This definition specifies that a --conf argument is required, as demonstrated in Figure 6.
The error indicates that you omitted this information:
argument -c/--conf is required
I'm doing some very simple PySide (and PyQt) tutorials in IPython. One tutorial just creates a window with some sliders to demonstrate slots and signals.
When I close the window of the running demo application, I see this error:
An exception has occurred, use %tb to see the full traceback.
SystemExit: 0
To exit: use 'exit', 'quit', or Ctrl-D.
So I run %tb and get this:
SystemExit Traceback (most recent call last)
/Workspaces/scratch/<ipython-input-1-88966dcfb499> in <module>()
33
34 if __name__ == "__main__":
---> 35 main()
/Workspaces/scratch/<ipython-input-1-88966dcfb499> in main()
29 w.show()
30 app.exec_()
---> 31 sys.exit(0)
32
33
SystemExit: 0
If I try to execute my code again, I get this:
RuntimeError: A QApplication instance already exists.
In case it helps, here my code:
from PySide.QtCore import *
from PySide.QtGui import *
import sys
class MyWindow(QWidget):
def __init__(self):
QWidget.__init__(self, None)
vbox = QVBoxLayout(self)
self.slider1 = QSlider(Qt.Horizontal)
self.slider1.setRange(0, 99)
self.slider1.setValue(0)
vbox.addWidget(self.slider1)
self.slider2 = QSlider(Qt.Horizontal)
self.slider2.setRange(0, 99)
self.slider2.setValue(99)
vbox.addWidget(self.slider2)
self.slider1.valueChanged.connect(self.slider2Changed)
def slider2Changed(self, position):
self.slider2.setValue(self.slider2.maximum() - position)
def main():
app = QApplication(sys.argv)
w = MyWindow()
w.show()
app.exec_()
sys.exit(0)
if __name__ == "__main__":
main()
I do not have errors when running the code using python:
python myexample.py
This error only happens when I run the code in an IPython (including a notebook or the qtconsole or the regular ipython terminal).
UPDATE: My main problem is that I cannot run the application again quickly and easily. If I try to run my code again, I get this:
RuntimeError: A QApplication instance already exists.
That kills the fast, interactive nature of IPython :(
This answer is thanks to Matthias BUSSONNIER from the ipython-users mailing list.
When I close the window of the running demo application, I see this error:
An exception has occurred, use %tb to see the full traceback.
SystemExit: 0
Just don't use sys.exit(0) as you are not exiting python, but still running IPython.
Add it back if you wish to run your app from a (real) command line and have a return status.
If I try to execute my code again, I get this:
RuntimeError: A QApplication instance already exists.
This is a PySide Bug that they "won't fix" as they don't consider it a bug.
See https://github.com/ipython/ipython/issues/1124)
and http://bugs.pyside.org/show_bug.cgi?id=855
QApplication can only have one instance and quitting an app is
apparently not considered a reason sufficient enough do delete the
object...
You can use this code from above issues :
app=QtGui.QApplication.instance() # checks if QApplication already exists
if not app: # create QApplication if it doesnt exist
app = QtGui.QApplication(sys.argv)
This was a sufficient solution for my present needs.
What you need to do is to cause the QApplication to be deleted later as in:
app = QApplication(sys.argv)
app.aboutToQuit.connect(app.deleteLater)
Using this code you can rerun the application as many times as you want in IPython, or anywhere else and every time you close your qt application, the object will be deleted in python.
sys.exit just raises SystemExit to terminate the interperter.
ipython catches SysExit when it executes a script in interactive mode, so this isn't acutally an error but a feature of ipython do avoid the interactive interpreter from being shutdown when a script is executed, as that's not what you usually want in an interactive session.
According to https://github.com/spyder-ide/spyder/issues/4639,
First, check if the instance is already present, and either assign that, or make a new one:
if not QApplication.instance():
app = QApplication(sys.argv)
else:
app = QApplication.instance()
(This snippet is taken from the answer by #NotSoBrainy.)
Then, after the show() method:
QtWidgets.QApplication.setQuitOnLastWindowClosed(True)
app.exec_()
app.quit()
There’s no need for sys.exit().
This solved my issue, which was exactly the same.
Check if already an instance of QApplication is present or not as the error occurs when an instance is already running and you are trying to create a new one.
if not QtWidgets.QApplication.instance():
app = QtWidgets.QApplication(sys.argv)
else:
app = QtWidgets.QApplication.instance()