I'm trying to parse the messages sent on an MQTT topic as JSONs, but when i try to call json.loads() on the decoded string, the code goes into exception:
File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 153 (char 152)
The code is the following:
def on_message_mqtt(self, client, userdata, message):
m_decode = message.payload.decode('utf-8')
logger.info(m_decode)
m_in = json.loads(m_decode)
logger.info(m_in)
While the payload of the messages has the following format:
{"Timestamp":"2021-05-24-13:27:13.450","AI":[1204,7,0,4,18,2,1176,802,11,0,381,2496,0,0,810,1282],"DI":[false,false,false,false,false,false,false,false]}
I tried to put the text into a string and loads() is able to parse it, so I bet that the problem is on the initial string decoding.
Here you can see a print of the decoded string, and then the error on the call to json.loads():
2021-05-24 16:31:47 - INFO - {"Timestamp":"2021-05-24-14:31:44.790","AI":[912,9,0,2,16,2,886,605,11,0,321,1924,1,963,620,9],"DI":[false,false,false,false,false,false,false,false]}
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "router.py", line 136, in run
self.mqtt.loop_forever()
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 1779, in loop_forever
rc = self.loop(timeout, max_packets)
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 1181, in loop
rc = self.loop_read(max_packets)
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 1572, in loop_read
rc = self._packet_read()
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 2310, in _packet_read
rc = self._packet_handle()
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 2940, in _packet_handle
return self._handle_pubrel()
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 3246, in _handle_pubrel
self._handle_on_message(message)
File "/opt/control/env/lib/python3.7/site-packages/paho/mqtt/client.py", line 3444, in _handle_on_message
self.on_message(self, self._userdata, message)
File "router.py", line 87, in on_message_mqtt
logger.info(json.loads(m_decode), exc_info=True)
File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 151 (char 150)
Judging from the error, character 152 is the square bracket closing the arroy of the value for the key of "DI" ] which would imply incorrect parsing using the payload.decode function
Can you check if this works
def on_message_mqtt(self, client, userdata, message):
m_in = json.loads(message.decode("utf-8"))
logger.info(m_in)
Related
I'm trying to deploy Telegram bot on pythonanywhere. Here is my code. All was working until I updated repository. Nothing was change in this piece of code. But it happed not working now. The get_message function not even calls. May be it is some problems with JSON or so, i don't understand exactly. How to get rid of this error?
URL = 'name.pythonanywhere.com'
app = Flask(__name__)
sslify = SSLify(app)
bot.remove_webhook()
bot.set_webhook(url=URL)
bot.send_message(userID, 'helo')
#app.route('/', methods=['POST', 'GET'])
def get_message():
update = types.Update.de_json(request.stream.read().decode('utf-8'))
bot.process_new_updates([update])
return 'ok', 200
2023-01-15 22:15:05,776: Exception on / [GET]
Traceback (most recent call last):
File "/home/kentus/.local/lib/python3.9/site-packages/flask/app.py", line 2077, in wsgi_app
response = self.full_dispatch_request()
File "/home/kentus/.local/lib/python3.9/site-packages/flask/app.py", line 1526, in full_dispatch_request
return self.finalize_request(rv)
File "/home/kentus/.local/lib/python3.9/site-packages/flask/app.py", line 1545, in finalize_request
response = self.make_response(rv)
File "/home/kentus/.local/lib/python3.9/site-packages/flask/app.py", line 1701, in make_response
raise TypeError(
TypeError: The view function for 'get_message' did not return a valid response. The function either returned None or ended without a return statement.
2023-01-15 22:52:17,579: Exception on / [GET]
Traceback (most recent call last):
File "/home/kentus/.local/lib/python3.9/site-packages/flask/app.py", line 2077, in wsgi_app
response = self.full_dispatch_request()
File "/home/kentus/.local/lib/python3.9/site-packages/flask/app.py", line 1525, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/kentus/.local/lib/python3.9/site-packages/flask/app.py", line 1523, in full_dispatch_request
rv = self.dispatch_request()
File "/home/kentus/.local/lib/python3.9/site-packages/flask/app.py", line 1509, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/home/kentus/english_bot/app/main.py", line 177, in get_message
update = types.Update.de_json(request.stream.read().decode('utf-8'))
File "/home/kentus/.local/lib/python3.9/site-packages/telebot/types.py", line 96, in de_json
obj = cls.check_json(json_string, dict_copy=False)
File "/home/kentus/.local/lib/python3.9/site-packages/telebot/types.py", line 80, in check_json
return json.loads(json_type)
File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I tried to manipulate the webhook but it ok with it. I guess. getWebhookInfo method shows correct url.
i tried to run locally flask app with public URL using ngrok to debug it locally, but i had errors with setting webhook on ngrok URL.
I even asked ChatGPT and he didn't help.
I am trying to get some order details using the python requests library. The API is working fine with the postman, but when I try to run it with python the following error occurs.
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I am using the following python code to call the API in the screenshots:
import requests
url = "https://xxxxxxxxxxxxxx/purchases"
header = {"Content-Type":"application/json", "Authorization":'token {}'.format('eyXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')}
data = {"from_date":"2021-06-15T07:19:32", "to_date":"2021-09-16 00:01:09"}
response = requests.get(url, params=data, headers=header)
print(response.json())
The result I am getting is:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3/dist-packages/requests/models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 518, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The issue was I need to add a text "Bearer " in front of the token passed in the Autorization parameter of header section as the following because the token is to be passed as a Bearer Token:
header = {"Content-Type":"application/json", "Authorization":'Bearer eyXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'}
I have a json object file and at the beginning of this file a header of text, date, time, IP address.
I've tried f.readlines()[5:] to no avail.
I've tried next(f)
I wish to skip 5 or 6 lines of text and go directly into the json data.
Here is an example.
import jsonlines
import json
data_file = input("Enter a file to parse: ")
with jsonlines.open(data_file) as file:
for obj in file:
try:
jsonparse = json.loads(obj)
except Exception as e:
pass
print(obj)
Error:
jsonlines.jsonlines.InvalidLineError: line contains invalid json: Expecting value: line 1 column 1 (char 0) (line 1)
Top of json file:
Start: 07/02/2019 14:59:40.686
Connected To:
192.168.11.203
Here is the full long error:
Enter a file to parse: Play.raw
Traceback (most recent call last):
File "C:\Users\sdickey\AppData\Local\Programs\Python\Python37\lib\site-packages\jsonlines\jsonlines.py", line 159, in read
value = self._loads(line)
File "C:\Users\sdickey\AppData\Local\Programs\Python\Python37\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\sdickey\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\sdickey\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:/Users/sdickey/PycharmProjects/Python Testing/Actall Data/testing.py", line 6, in <module>
for obj in file:
File "C:\Users\sdickey\AppData\Local\Programs\Python\Python37\lib\site-packages\jsonlines\jsonlines.py", line 204, in iter
skip_empty=skip_empty)
File "C:\Users\sdickey\AppData\Local\Programs\Python\Python37\lib\site-packages\jsonlines\jsonlines.py", line 164, in read
six.raise_from(exc, orig_exc)
File "<string>", line 3, in raise_from
jsonlines.jsonlines.InvalidLineError: line contains invalid json: Expecting value: line 1 column 1 (char 0) (line 1)
Process finished with exit code 1
It is always preferable not to read the whole file in memory but one line at a time.
Assuming your input file contains:
first line
second line
third line
fourth line
fifth line
{ "k1": "val1", "k2": "val2" }
{ "k3": "val3", "k4": "val4" }
if you just want to skip 5 lines, you could do it brutally as:
import json
with open("test.txt") as f:
for _ in range(5):
next(f)
for line in f:
obj = json.loads(line)
print(obj)
or using enumerate:
import json
with open("test.txt") as f:
for i, line in enumerate(f):
if i<5:
continue
obj = json.loads(line)
print(obj)
or use itertools' dropwhile:
import itertools as it
import json
with open("test.txt") as f:
for i, line in it.dropwhile(lambda i: i[0]<5, enumerate(f)):
obj = json.loads(line)
print(obj)
I think you are trying to convert to json line by line. You need to join all lines by \n igoring the first 5 lines and then load it:
import json
with open("test.txt") as f:
json_obj = "\n".join(f.readlines()[5:])
jsonparse = json.loads(json_obj)
print(jsonparse)
I want to create a temporary JSON file to store the google calendar credentials, that were stored in a "job" object.
I am using the ServiceAccountCredentials to then get the credentials from the file.
Client = {
"clientID": job.getClientID(),
"clientSecret": job.getClientSecret()
}
temp = tempfile.NamedTemporaryFile(mode="w+b", suffix=".json")
complex_data = open(temp.name, "w", encoding="UTF-8")
complex_data.write(json.dumps(Client))
# data = complex_data.write(json.dumps(Client))
# z = json.loads(data)
credentials = ServiceAccountCredentials.from_json(
temp.name
)
```
I get the following error:
Traceback (most recent call last):
File "/var/www/library-offers-google-calendar/main.py", line 209, in <module>
temp.name
File "/var/www/library-offers-google-calendar/venv/lib/python3.7/site-packages/oauth2client/service_account.py", line 436, in from_json
json_data = json.loads(_helpers._from_bytes(json_data))
File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
From the docs, the method ServiceAccountCredentials.from_json accepts a json data rather than a json file name.
If you want to use a json file name, you need to use the method ServiceAccountCredentials.from_json_keyfile_name
Or else, you can directly use Client dictionary as a parameter to ServiceAccountCredentials.from_json without creating and reading from temporary file.
When I try to use app.current_request.json_body in chalice I get a decode error:
Traceback (most recent call last): File "/var/task/chalice/app.py",
line 659, in _get_view_function_response
response = view_function(**function_args) File "/var/task/app.py", line 34, in post_item
data = app.current_request.json_body File "/var/task/chalice/app.py", line 303, in json_body
self._json_body = json.loads(self.raw_body) File "/var/lang/lib/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s) File "/var/lang/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/var/lang/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char
0)
It doesn't matter how simple the data is. Example: {"Company":"ABC"} or {}.
As can be seen in the following code in the API Gateway all I try to do is return the data that has been sent so I don't think this is the problem:
#app.route('/test', methods=['POST'], content_types=['application/json'], cors=cors_config)
def post_item(data):
data = app.current_request.json_body
return data
Does anyone know what I might have done wrong?
You must remove the data from the parameters of the function.
That is used for pass url parameters.
#app.route('/test', methods=['POST'], content_types=['application/json'], cors=cors_config)
def post_item():
data = app.current_request.json_body
return data