There are couple of questions related to this error:
JSON object must be str, not 'bytes'
However, except obvious solution to read and decode response, I didn't learn anything special.
Here is example to the problem:
>>> import json
>>> from urllib.request import urlopen
>>> url = 'http://echo.jsontest.com/key/value/one/two'
>>> with urlopen(url) as request:
... json.load(request)
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "C:\Python35\lib\json\__init__.py", line 268, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "C:\Python35\lib\json\__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
So my question is why Python's JSON deserializer (that accepts file-like objects with .read() method), does not try to handle this request, as response headers hint all there is needed to know:
Content-Type: application/json; charset=ISO-8859-1
Headers hint, they do not guarantee, but that can not be a reason not to try the obvious IMHO.
Use json.loads instead of json.load
Related
I am new to Python (and JSON) so apologies of this is obvious to you.
I pull some data from an API using the following code
import requests
import json
headers = {'Content-Type': 'application/json', 'accept-encoding':'identity'}
api_url = api_url_base+api_token+api_request #variables removed for security
response = requests.get(api_url, headers=headers)
data=response.json()
keys=data.keys
if response.status_code == 200:
print(data["message"], "saving to file...")
print("Found the following keys:")
print(keys)
with open('vulns.json', 'w') as outfile:
json.dump(response.content.decode('utf-8'),outfile)
print("File Saved.")
else:
print('The site returned a', response.status_code, 'error')
this works, I get some data returned and I am able to write the file.
I am trying to change what's returned form a short format to a long format and to check its working I need to see the keys, I was trying to do this offline using the written file (as practice for reading JSON from files).
I wrote these few lines (taken from this site https://www.kite.com/python/answers/how-to-print-the-keys-of-a-dictionary-in-python)
import json
with open('vulns.json') as json_file:
data=json.load(json_file)
print(data)
keys=list(data.keys())
print(keys)
Unfortunately, whenever I run this it returns this error
Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec 7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> print(keys)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'keys' is not defined
>>> & C:/Users/xxxx/AppData/Local/Microsoft/WindowsApps/python.exe c:/Temp/read-vulnfile.py
File "<stdin>", line 1
& C:/Users/xxxx/AppData/Local/Microsoft/WindowsApps/python.exe c:/Temp/read-vulnfile.py
^
SyntaxError: invalid syntax
>>> exit()
PS C:\Users\xxxx\Documents\scripts\Python> & C:/Users/xxx/AppData/Local/Microsoft/WindowsApps/python.exe c:/Temp/read-vulnfile.py
Traceback (most recent call last):
File "c:\Temp\read-vulnfile.py", line 6, in <module>
keys=list(data.keys)
AttributeError: 'str' object has no attribute 'keys'
The Print(data) command returns what looks like JSON, this is the opening line:
{"count": 1000, "message": "Vulnerabilities found: 1000", "data":
[{"...
I cant show the content it's sensitive.
why is this looking at a str object rather than a dictionary?
how do I read JSON back into a dictionary please?
You just have that content stored in file as a string. Just open the vulns.json in some editor and there most likely is something like "{'count': 1000, ... instead of {"count": 1000, ....
It's opened by json.load, but translated to string (see this table).
So you should take one step back and take a look what happens during saving to file. You take some content from your response, but dump the string decoded value into a file. Take instead a try with
json.dump(response.json(), outfile)
(or just use data variable you already have provided).
This should allow you to succesfully dump and load data as a dict.
json.dumps gives an error if both json and jsondatetime are imported. The error is the following:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/json/__init__.py", line 230, in dumps
return _default_encoder.encode(obj)
TypeError: encode() missing 1 required positional argument: 'o'
But I just import JSON, then json.dumps work fine. I don't know how to deal with this. I need jsondatetime as well
This works::
import json
json.dumps({'DbName': 'DB','Hostname': '10.0.0.6','DbUsername':'SYSTEM'})
'{"Hostname": "10.0.0.6","DbName": "DB", "DbUsername": "SYSTEM"}'
This does not work::
import jsondatetime
import json
json.dumps({'DbName': 'DB', 'Hostname': '10.0.0.6', 'DbUsername': 'SYSTEM'})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/json/__init__.py", line 230, in dumps
return _default_encoder.encode(obj)
TypeError: encode() missing 1 required positional argument: 'o'
jsondatetime is a drop-in replacement for json. You should only have
import jsondatetime as json
From the documentation:
JSON-datetime is a very simple wrapper around Python simplejson loads method. It decodes datetime values contained in JSON strings.
I'm using a NAO robot with naoqi version 2.1 and Choregraphe on Windows. I want to parse json from an attached file to the behavior. I attached the file like in that link.
Code:
def onLoad(self):
self.filepath = os.path.join(os.path.dirname(ALFrameManager.getBehaviorPath(self.behaviorId)), "fileName.json")
def onInput_onStart(self):
with open(self.filepath, "r") as f:
self.data = self.json.load(f.get_Response())
self.dataFromFile = self.data['value']
self.log("Data from file: " + str(self.dataFromFile))
But when I run this code on the robot (connected with a router) I'll get an error:
[ERROR] behavior.box :_safeCallOfUserMethod:281 _Behavior__lastUploadedChoregrapheBehaviorbehavior_1136151280__root__AbfrageKontostand_3__AuslesenJSONDatei_1: Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/naoqi.py", line 271, in _safeCallOfUserMethod
func()
File "<string>", line 20, in onInput_onStart
File "/usr/lib/python2.7/site-packages/inaoqi.py", line 265, in <lambda>
__getattr__ = lambda self, name: _swig_getattr(self, behavior, name)
File "/usr/lib/python2.7/site-packages/inaoqi.py", line 55, in _swig_getattr
raise AttributeError(name)
AttributeError: json
I already tried to understand the code from the correspondending lines but I couldn't fixed the error. But I know that the type of my object f is 'file'. How can I open the json file as a json file?
Your problem comes from this:
self.json.load(f.get_Response())
... there is no such thing as "self.json" on a Choregraphe box, import json and then do json.load. And what is get_Response ? That method doesn't exist on anything in Python that I know of.
You might want to first try making a standalone python script (that doesn't use the robot) that can read your json file before you try it with choregraphe. It will be easier.
I am trying to send a http request to any url and get the response using urllib library. Following is the code that I have used :
>>> import requests
>>> r = requests.get("http://www.youtube.com/results?bad+blood")
>>> r.status_code
200
when I try to do this I get following error.
>>> r.json()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Python/2.7/site-packages/requests/models.py", line 808, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Python/2.7/site-packages/simplejson/__init__.py", line 516, in loads
return _default_decoder.decode(s)
File "/Library/Python/2.7/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/Library/Python/2.7/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 3 (char 2)
can someone tell me whats wrong with the code.
PS: I am using python 2.7.10
The response isn't JSON, it is 'text/html; charset=utf-8'. If you want to parse it, use something like BeautifulSoup.
>>> import requests, bs4
>>> rsp = requests.get('http://www.youtube.com/results?bad+blood')
>>> rsp.headers['Content-Type']
'text/html; charset=utf-8'
>>> soup = bs4.BeautifulSoup(rsp.content, 'html.parser')
I'd recommend using the YouTube Search API instead. Log in to Google Developers Console, set up a API key following the API Key Setup instructions, then you can make the request using the YouTube Search API:
>>> from urllib import parse
>>> import requests
>>> query = parse.urlencode({'q': 'bad blood',
... 'part': 'snippet',
... 'key': 'OKdE7HRNPP_CzHiuuv8FqkaJhPI2MlO8Nns9vuM'})
>>> url = parse.urlunsplit(('https', 'www.googleapis.com',
... '/youtube/v3/search', query, None))
>>> rsp = requests.get(url, headers={'Accept': 'application/json'})
>>> rsp.raise_for_status()
>>> response = rsp.json()
>>> response.keys()
dict_keys(['pageInfo', 'nextPageToken', 'regionCode', 'etag', 'items', 'kind'])
Note that the example is using Python 3. If you want to use Python 2, then you will have to import urlencode from urllib and urlunsplit from urlparse.
That URL returns HTML, not JSON, so there's no point calling .json() on the response.
I am using Python 2.6.6, and trying to generate a ordered_dict from json string. I could understand that I could use object_pairs_hook of json Decoder/loads, but unfortunately it's not supported in 2.6.6. Is there any way out?
e.g.
template_s = '{ "aa": {"_type": "T1"}, "bb": {"_type": "T11"}}'
json.loads(template_s, object_pairs_hook=OrderedDict)
>>> json.loads(json_str, object_pairs_hook=OrderedDict)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.6/json/__init__.py", line 318, in loads
return cls(encoding=encoding, **kw).decode(s)
TypeError: __init__() got an unexpected keyword argument 'object_pairs_hook'
Thanks
I was able to do the same with simplejson
import simplejson as json
json.loads(config_str, object_pairs_hook=json.OrderedDict)