TypeError("can't pickle re.Match objects") error when pickling using dill / pickle - pickle

I can't seem to figure out a way to pickle this, can anyone help?
It's because of the way reduce function is written for re.match.
Code:
import re
x = re.match('abcd', 'abcd')
print(type(x))
print(x.__reduce_ex__(3))
Output:
<class 're.Match'>
Traceback (most recent call last):
File "an.py", line 4, in <module>
print(x.__reduce_ex__(3))
TypeError: can't pickle re.Match objects
My exact issue is that I am trying to pickle an object of a lex / yacc parser implementation class after submitting a string to it to parse.
If I try to pickle the class object without parsing any string via it, it is able to pickle. Problem arises only after I parse a string using it and then try to pickle the class object.

Match objects does not have a __getstate__ and __setstate__ thus cannot be pickled, the entire iterator could not be pickled.
More about this subject can be found here:
https://docs.python.org/3/library/pickle.html#pickle-picklable
here is a further explanation on the desired objects:
https://docs.python.org/3/library/re.html#match-objects
An alternative solution is to implement __getstate__ and __setstate__ to help the pickling process, this will require you to create a custom class and implement this function, which seem to overcomplicated for this situation
Hope that helped

Related

Gensim Pickle Error: Enable to Load the Saved Topic Model

I am working on topic inference that will require to load a previously saved model.
However, I got a pickle error that says
Traceback (most recent call last):
File "topic_inference.py", line 35, in <module>
model_for_inference = gensim.models.LdaModel.load(model_name, mmap = 'r')
File "topic_modeling/env/lib/python3.8/site-packages/gensim/models/ldamodel.py", line 1663, in load
result = super(LdaModel, cls).load(fname, *args, **kwargs)
File "topic_modeling/env/lib/python3.8/site-packages/gensim/utils.py", line 486, in load
obj = unpickle(fname)
File "topic_modeling/env/lib/python3.8/site-packages/gensim/utils.py", line 1461, in unpickle
return _pickle.load(f, encoding='latin1') # needed because loading from S3 doesn't support readline()
TypeError: __randomstate_ctor() takes from 0 to 1 positional arguments but 2 were given
The code I use to load the model is simply
gensim.models.LdaModel.load(model_name, mmap = 'r')
Here is the code that I use to create and save the model
model = gensim.models.ldamulticore.LdaMulticore(
corpus=comment_corpus,
id2word=key_word_dict, ## This is now a gensim.corpora.Dictionary Object, previously it was the .id2token attribute
chunksize=chunksize,
alpha='symmetric',
eta='auto',
iterations=iterations,
num_topics=num_topics,
passes=epochs,
eval_every=eval_every,
workers = 15,
minimum_probability= 0.0)
model.save(output_model)
where output_model doesn't have an extension like .model or .pkl
In the past, I tried the similar approach with the exception that I passed in a .id2token attribute under the gensim.corpora.Dictionary object instead of the full gensim.corpora.Dictionary to the id2word parameter when I created the model, and the method loads the model fine back then. I wonder if passing in a corpora.Dictionary will make a difference in the loading output...? Back that time, I was using regular python, but now I am using anaconda. However, all the versions of the packages are the same.
Another report of an error about __randomstate_ctor (at https://github.com/numpy/numpy/issues/14210) suggests the problem may be related to numpy object pickling.
Is there a chance that the configuration where your load is failing is using a later version of numpy than when the save occurred? Could you try, at least temporarily, rolling back to some older numpy (that's still sufficient for whatever Gensim you're using) to see if it helps?
If you find any load that works, even in a suboptimal config, you might be able to null-out whatever random-related objects are causing the problem and re-save, then having a saved version that loads better in your truly-desired configuration. Then, if the random-related objects truly needed after reload, it may be possible to manually re-constitute them. (I haven't looked into this yet, but if you find any workaround allowing a load, but then aren't sure what to manually null/rebuild, I could take a closer look.)

Transforming shapefiles to dataframes with shapefile_to_dataframe() helper function - fiona related error

I am trying to use the Palantir Foundry helper function shapefile_to_dataframe() in order to ingest shapefiles for later usage in geolocation features.
I have manually imported the shapefiles (.shp, .shx & .dbf) in a single dataset (no access issues through the filesystem API).
As per documentation, I have imported geospatial-tools and the GEOSPARK profiles + included dependencies in the transforms-python build.gradle.
Here is my transform code, which is mostly extracted from the documentation:
from transforms.api import transform, Input, Output, configure
from geospatial_tools import geospatial
from geospatial_tools.parsers import shapefile_to_dataframe
#geospatial()
#transform(
raw = Input("ri.foundry.main.dataset.0d984138-23da-4bcf-ad86-39686a14ef21"),
output = Output("/Indhu/InDhu/Vincent/geo_energy/datasets/extract_coord/raw_df")
)
def compute(raw, output):
return output.write_dataframe(shapefile_to_dataframe(raw))
Code assist then become extremely slow to load, and then I am finally getting following error:
AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)
Traceback (most recent call last):
File "/myproject/datasets/shp_to_df.py", line 3, in <module>
from geospatial_tools.parsers import shapefile_to_dataframe
File "/scratch/standalone/3a553998-623b-48f5-9c3f-03de7e64f328/code-assist/contents/transforms-python/build/conda/env/lib/python3.8/site-packages/geospatial_tools/parsers.py", line 11, in <module>
from fiona.drvsupport import supported_drivers
File "/scratch/standalone/3a553998-623b-48f5-9c3f-03de7e64f328/code-assist/contents/transforms-python/build/conda/env/lib/python3.8/site-packages/fiona/__init__.py", line 85, in <module>
with fiona._loading.add_gdal_dll_directories():
AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)
Thanks a lot for your help,
Vincent
I was able to reproduce this error and it seems like it happens only in previews - running the full build seems to be working fine. The simplest way to get around it is to move the import inside the function:
from transforms.api import transform, Input, Output, configure
from geospatial_tools import geospatial
#geospatial()
#transform(
raw = Input("ri.foundry.main.dataset.0d984138-23da-4bcf-ad86-39686a14ef21"),
output = Output("/Indhu/InDhu/Vincent/geo_energy/datasets/extract_coord/raw_df")
)
def compute(raw, output):
from geospatial_tools.parsers import shapefile_to_dataframe
return output.write_dataframe(shapefile_to_dataframe(raw))
However, at the moment, the function shapefile_to_dataframe isn't going to work in the Preview anyway because the full transforms.api.FileSystem API isn't implemented - specifically, the functions ls doesn't implement the parameter glob which the full transforms API does.

MySQLConnection' object has no attribute 'my_cursor'

Im new to mysql. installed gitbash, sublime text and mysql. pip3 working. able to import mysql.connector.
Im trying to create a database called mydb
Problem is, Im getting attribute error: 'MySQLConnection' object has no attribute 'my_cursor'
python database.py
Traceback (most recent call last):
File "database.py", line 10, in
my_cursor = mydb.my_cursor()
AttributeError: 'MySQLConnection' object has no attribute 'my_cursor''
I don't know which import you are using and did you used some at all but you should try to use this:
import MySQLdb
Also i think you need to use cursor() instead of my_cursor(). Question is not explained well, i don't have source code so i cannot be precise.

'Word2Vec' object has no attribute 'generate_training_data'

Code :
from gensim.models.word2vec import Word2Vec
w2v = Word2Vec()
training_data = w2v.generate_training_data(settings, corpus)
Error :
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-45-bae554564046> in <module>
1 w2v = Word2Vec()
2 # Numpy ndarray with one-hot representation for [target_word, context_words]
----> 3 training_data = w2v.generate_training_data(settings, corpus)
AttributeError: 'Word2Vec' object has no attribute 'generate_training_data'
I even tried importing gensim.models.word2vec and tried every possibility but couldn't get it done.
Can someone help me with it?
Thanks in advance !
Yes, the gensim Word2Vec class doesn't have that method – and as far as I know, it never has.
And from your example usage, I'm not even sure what it might purport to do: a Word2Vec model needs to be provided data in the right format – it doesn't "generate" it (even as a translation from some other corpus).
I suspect you are looking at docs or a code example from some other unrelated library.
For using gensim's Word2Vec, you should rely on the gensim documentation & examples. The class docs include some basic details of proper usage, and there's a Jupyter notebook word2vec.ipynb included with the library, in its docs/notebooks directory (and also viewable online).

import issue of "serialization.json.JSON" class in actionscript:

I'm working on a project using actionscript and Flex. For some reason I have a problem importing the com.adobe.serialization.json.JSON class.
When I'm working only with the FlexSDK files and try to use it I'm getting
the following error:
Error:(142, 70) [..]: Error code: 1120: Access of undefined property
JSON.
And of course IntelliJ marks this file and the import in red.
On the other hand when I import the corelib.swc that includes this file I get the following error:
Error:[..]: Can not resolve a multiname reference unambiguously. JSON
(from /Volumes/backup/FlexSDK/frameworks/libs/air/airglobal.swc(JSON,
Walker)) and com.adobe.serialization.json:JSON (from
/Volumes/backup/.../libs/corelib.swc(com.adobe.serialization.json:JSON))
are available.
What is going on here? How can I solve this?
JSON is a top level class available in all the scopes since FP11. Trying to import any class with name JSON will result in an error. If (for some reason) you really do not want to use the already available JSON class and instead import a custom one you'll have to rename it.
Using Intellij, the best you can do is use the JSON class from the current SDK that you have, it has the methods parse() and stringify(). those two methods do the same as the corelib methods for the json.
In case you wanted to use the com.adobe.serialization.json.JSON it will enter in conflict with the one declared in the SDK.
Hope the information is useful