TypeError while loading the pickle file using pickle.load() in python3:jupyter notebook
with open("/home/amit/Downloads/may2_18_company_data_with_cluster.pickle","r",encoding='utf8') as f:
c = pickle.load(f) //this line shows an error
TypeError: a bytes-like object is required, not 'str'
Please help me to fix this on jupyter notebook with python3.
This is because pickle.load expects a binary type file as an argument, you need to specify this in the use of open by specifying rb for read binary instead of 'r' which reads as a string. Changing the code to:
with open("/home/amit/Downloads/may2_18_company_data_with_cluster.pickle","rb",encoding='utf8') as f:
c = pickle.load(f)
should solve this.
Related
So, I'm using the following code to get pandas to read my JSON text file-
f = open('C:/Users/stans/WFH Project/data.json')
data = json.load(f)
df = pd.DataFrame(data, index=[0])
f.close()
Once I execute the cell, I get
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position
1535: character maps to
I used the above coding for a smaller sample of JSON data and it worked. But, since I updated the file to include a much larger sample, I get that error.
I verified that the JSON format is correct, and I also tried in the open statement-
encoding='utf-8'
and
errors='ignore'
Both produced value errors. Any ideas? Thanks in advance for your help!
I am getting an error while reading JSON file using python's json module and I'm not able to understand what is wrong. Below are my files and code for your reference:
json.load(files_lst[1])
error:
AttributeError: 'str' object has no attribute 'read'
After reading several answers I also tried:
json.loads(files_lst[1])
but I get the following error:
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
What is wrong here? Thanks a lot for your help
>>> files_lst
['All_Transcripts/x3021_10.50.48.111_04-04-2019.json',
'All_Transcripts/x5363_09.33.36.955_08-27-2019.mp3.json',
'All_Transcripts/x3580_11.35.53.462_05-13-2019.json',
'All_Transcripts/x4342_08.55.01.523_08-01-2019.json',
'All_Transcripts/x9496_15.26.32.382_05-21-2019.json',
'All_Transcripts/x5374_08.38.15.692_06-17-2019.json',
'All_Transcripts/x4342_13.43.57.128_03-21-2019.json']
json.load accepts an open file descriptor. So you should call it like this:
json.load(open(files_lst[1]))
json.loads accepts a json as a string. So you should call it like this:
json.loads(open(files_lst[1]).read())
I am trying to persist some data to disk, I am attempting to use Python's JSON module but I can't access the data on simple json.load and I can't figure out why. Here's my code:
jsondata=json.dumps({'a':1,
'b':'string',
'c':{'k1':(1,3),'k2':(12,3)}})
f= open('jsonfile.json', 'w')
json.dump(jsondata,f)
f.close()
g=open('jsonfile.json', 'r')
result=json.load(g)
g.close()
print(result['b'])
This gives me the error "TypeError: string indicies must be integers"
However if I replace the access block with
g=open('jsonfile.json', 'r')
result=json.loads(json.load(g))
g.close()
print(result['b'])
It gives me the result I expect. I have read through the documentation a number of times and it seems like the simple json.load by itself should be sufficient. I can't figure out why I would have to use json.loads as well. I feel like I'm missing something. Any insight would be welcome.
Thanks петр костюкевич
Problem was I was converting it to a string before the dump so needed to convert it back. This code worked.
jsondata=({'a':1,
'b':'string',
'c':{'k1':(1,3),'k2':(12,3)}})
f= open('jsonfile.json', 'w')
json.dump(jsondata,f)
f.close()
g=open('jsonfile.json', 'r')
result=json.load(g)
g.close()
print(result['b'])
I am trying to read twitter data from json file using python 2.7.12.
Code I used is such:
import json
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
def get_tweets_from_file(file_name):
tweets = []
with open(file_name, 'rw') as twitter_file:
for line in twitter_file:
if line != '\r\n':
line = line.encode('ascii', 'ignore')
tweet = json.loads(line)
if u'info' not in tweet.keys():
tweets.append(tweet)
return tweets
Result I got:
Traceback (most recent call last):
File "twitter_project.py", line 100, in <module>
main()
File "twitter_project.py", line 95, in main
tweets = get_tweets_from_dir(src_dir, dest_dir)
File "twitter_project.py", line 59, in get_tweets_from_dir
new_tweets = get_tweets_from_file(file_name)
File "twitter_project.py", line 71, in get_tweets_from_file
line = line.encode('ascii', 'ignore')
UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 3131: invalid start byte
I went through all the answers from similar issues and came up with this code and it worked last time. I have no clue why it isn't working now.
In my case(mac os), there was .DS_store file in my data folder which was a hidden and auto generated file and it caused the issue. I was able to fix the problem after removing it.
It doesn't help that you have sys.setdefaultencoding('utf-8'), which is confusing things further - It's a nasty hack and you need to remove it from your code.
See https://stackoverflow.com/a/34378962/1554386 for more information
The error is happening because line is a string and you're calling encode(). encode() only makes sense if the string is a Unicode, so Python tries to convert it Unicode first using the default encoding, which in your case is UTF-8, but should be ASCII. Either way, 0x80 is not valid ASCII or UTF-8 so fails.
0x80 is valid in some characters sets. In windows-1252/cp1252 it's €.
The trick here is to understand the encoding of your data all the way through your code. At the moment, you're leaving too much up to chance. Unicode String types are a handy Python feature that allows you to decode encoded Strings and forget about the encoding until you need to write or transmit the data.
Use the io module to open the file in text mode and decode the file as it goes - no more .decode()! You need to make sure the encoding of your incoming data is consistent. You can either re-encode it externally or change the encoding in your script. Here's I've set the encoding to windows-1252.
with io.open(file_name, 'r', encoding='windows-1252') as twitter_file:
for line in twitter_file:
# line is now a <type 'unicode'>
tweet = json.loads(line)
The io module also provide Universal Newlines. This means \r\n are detected as newlines, so you don't have to watch for them.
For others who come across this question due to the error message, I ran into this error trying to open a pickle file when I opened the file in text mode instead of binary mode.
This was the original code:
import pickle as pkl
with open(pkl_path, 'r') as f:
obj = pkl.load(f)
And this fixed the error:
import pickle as pkl
with open(pkl_path, 'rb') as f:
obj = pkl.load(f)
I got a similar error by accidentally trying to read a parquet file as a csv
pd.read_csv(file.parquet)
pd.read_parquet(file.parquet)
The error occurs when you are trying to read a tweet containing sentence like
"#Mike http:\www.google.com \A8&^)((&() how are&^%()( you ". Which cannot be read as a String instead you are suppose to read it as raw String .
but Converting to raw String Still gives error so i better i suggest you to
read a json file something like this:
import codecs
import json
with codecs.open('tweetfile','rU','utf-8') as f:
for line in f:
data=json.loads(line)
print data["tweet"]
keys.append(data["id"])
fulldata.append(data["tweet"])
which will get you the data load from json file .
You can also write it to a csv using Pandas.
import pandas as pd
output = pd.DataFrame( data={ "tweet":fulldata,"id":keys} )
output.to_csv( "tweets.csv", index=False, quoting=1 )
Then read from csv to avoid the encoding and decoding problem
hope this will help you solving you problem.
Midhun
I'm trying to parse a json-stat file using jsonstat.py (v 0.1.7) but am getting an error.
The code below is copied from the examples on github (https://github.com/26fe/jsonstat.py/tree/master/examples-notebooks):
from __future__ import print_function
import os
import jsonstat
os.chdir(r'D:\Desktop\JSON_Stat')
url = 'http://www.cso.ie/StatbankServices/StatbankServices.svc/jsonservice/responseinstance/NQQ25'
file_name = "test02.json"
file_path = os.path.abspath(os.path.join("..","JSON_Stat", "CSO", file_name))
I added this line to deal with non ascii characters in the file:
# -*- coding: utf-8 -*-
this succesfully downloads the json file to my desktop:
if os.path.exists(file_path):
print("using already downloaded file {}".format(file_path))
else:
print("download file and storing on disk")
jsonstat.download(url, file_path)
From here, I can load and pprint the data using the json module:
import json
import pprint as pp
with open(r"CSO\test02.json") as data_file:
data = json.load(data_file)
pp.pprint(data)
... but when I try and use the jsonstat module (as specified in the examples) I get the error mentioned in the subject:
collection = jsonstat.from_file(r"D:\Desktop\JSON_Stat\CSO\test02.json")
collection
# abbreviated error message
--> 384 self.__pos2cat = self.__size * [None]
TypeError: can't multiply sequence by non-int of type 'list'
I understand what the error message itself means but, having studied the the dimensions.py module where it occurs, am stuck trying to understand why. I was able to run the sample OECD code without issue so perhaps the data itself is not formatted in the expected way, though the source site (http://www.cso.ie/webserviceclient/) states that the json-stat format is being used.
So, finally, my questions are: has anyone run into this error and resolved it? Has anyone succesfully used the jsonstat module to parse this specific data? Alternatively, any general advice towards troubleshooting this issue is welcome.
Thanks