import base64
import logging
def hello_pubsub(event, context):
"""Background Cloud Function to be triggered by Pub/Sub.
Args:
event (dict): The dictionary with data specific to this type of
event. The `data` field contains the PubsubMessage message. The
`attributes` field will contain custom attributes if there are any.
context (google.cloud.functions.Context): The Cloud Functions event
metadata. The `event_id` field contains the Pub/Sub message ID. The
`timestamp` field contains the publish time.
"""
import base64
import time
import json
import requests
print(event)
print("""This Function was triggered by messageId {} published at {}
""".format(context.event_id, context.timestamp))
if 'data' in event:
name = base64.b64decode(event['data']).decode('utf-8')
else:
name = 'World'
print(name)
print(type(name))
print('Hello {}!'.format(name))
payload = json.loads(name)
logging.debug(payload)
I was trying to execute this is cloud function but I am getting an error, which is probably because of json.loads(). How can I get the payload as Json
Error:
line 32, in hello_pubsub payload = json.loads(name) File "/opt/python3.8/lib/python3.8/json/__init__.py", line 357, in loads return _default_decoder.decode(s) File "/opt/python3.8/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/opt/python3.8/lib/python3.8/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Solved the Issue by Changing encoding.
I was encoding string data, changed it to encode json data and it worked
This is working fine.
Please try to publish the message directly to topic on UI.
I tried with following message and it gave the output
{"abc":"123456","Def":"udebj"}
Related
Good day.
I have a small password generator program and I want to save the created passwords into a json file (append each time) so I can add them to an SQLITE3 database.
Just trying to do the append functionality I receive several errors that I don't understand.
Here are the errors I receive and below that is the code itself.
I'm quite new to Python so additional details are welcomed.
Traceback (most recent call last):
File "C:\Users\whitmech\OneDrive - Six Continents Hotels, Inc\04 - Python\02_Mosh_Python_Course\Py_Projects\PWGenerator.py", line 32, in
data = json.load(file)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1264.0_x64__qbz5n2kfra8p0\lib\json_init_.py", line 293, in load
return loads(fp.read(),
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1264.0_x64__qbz5n2kfra8p0\lib\json_init_.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1264.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1264.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
import random
import string
import sqlite3
import json
from pathlib import Path
print('hello, Welcome to Password generator!')
# input the length of password
length = int(input('\nEnter the length of password: '))
# define data
lower = string.ascii_lowercase
upper = string.ascii_uppercase
num = string.digits
symbols = string.punctuation
# string.ascii_letters
# combine the data
all = lower + upper + num + symbols
# use random
temp = random.sample(all, length)
# create the password
password = "".join(temp)
filename = 'saved.json'
entry = {password}
with open(filename, "r+") as file:
data = json.load(file)
data.append(entry)
file.seek(0)
json.dump(data, file)
# print the password
print(password)
Update: I've changed the JSON code as directed and it works but when trying to do the SQLite3 code I'm knowing receiving a typeerror
Code:
with open(filename, "r+") as file:
try:
data = json.load(file)
data.append(entry)
except json.decoder.JSONDecodeError as e:
data = entry
file.seek(0)
json.dump(data, file)
# print the password
print(password)
store = input('Would you like to store the password? ')
if store == "Yes":
pwStored = json.loads(Path("saved.json").read_text())
with sqlite3.connect("db.pws") as conn:
command = "INSERT INTO Passwords VALUES (?)"
for i in pwStored:
conn.execute(command, tuple(i.values)) # Error with this code
conn.commit()
else:
exit()
Error:
AttributeError: 'str' object has no attribute 'values'
The error is because
Your json file is empty, you need to update the following block
entry = [password]
with open(filename, "r+") as file:
try:
data = json.load(file)
data.extend(entry)
except json.decoder.JSONDecodeError as e:
data = entry
file.seek(0)
json.dump(data, file)
Also you are adding password in a set ie., entry, and it will again throw you an error TypeError: Object of type set is not JSON serializable
So you need to convert that to either a list or dict
Note: Here I have used entry as a list
I am trying to write a aws lambda function which will push the SQS queue output in a s3 bucket.
But the lambda function is failing to push the message , the cloudwatch log is showing
JSONDecodeError: Expecting value: line 1 column 1
i am posting the lambda function which i am using
import json
import boto3
def lambda_handler(event, context):
s3 = boto3.client("s3")
data = json.loads(event["Records"][0]["body"]) --getting error in this line
print(data)
s3.put_object(Bucket="sqsmybucket",key="data.json", Body=json.dumps(data))
#print(event)
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
The cloud watch log is showing
2020-05-30T23:51:45.276+05:30
[ERROR] JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 6, in lambda_handler
data = json.loads(event["Records"][0]["body"])
File "/var/lang/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/var/lang/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/var/lang/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
I have formatted the message and saved it to the cloud below is the link
[formatted JSON code][1]
formatted JSON
Please help , thanks in advance
Your event["Records"][0]["body"] is a plain string, not json:
"body": "A difficult message."
Therefore, json.loads(event["Records"][0]["body"]) is equivalent to json.loads("A difficult message.") which obviously fails.
To get body's value you can do the following instead:
data = event["Records"][0]["body"]
However, since later you have the following statment:
Body=json.dumps(data)
The Body will be:
Body='"A difficult message."'
which may or may not be what you desire.
I'm trying to grab a response from an api but i cannot load it in a dictionary.
Here's my code:
r = session.post(url, headers=headers, json=payload)
The returned status code is 200 so i get a response.
By trying to do: reply = r.json() i get:
Traceback (most recent call last):
File "C:/Users/PycharmProjects/honeymonitor/main.py", line 46, in <module>
data = get_data(session)
File "C:/Users/PycharmProjects/honeymonitor/main.py", line 35, in get_data
reply = r.json()
File "C:\Users\PycharmProjects\honeymonitor\venv\lib\site-packages\requests\models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\AppData\Local\Programs\Python\Python37\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
And by using reply = r.text i get a lot of gibberish:
� d�M���)b�b��$�j�np������X2�Al�w{g��wO�n-���w��g�����}�1�Y5#�R��/�
� X���o]>6,�N�_#�#Ң����K�#���s
��Ɩ�ː�����2��N�:D��p�����7{K��������V��
-")�p0n#I�f!���Q�+| /We�.�p�=����u���-�ʌ�cs�"jfH����}qkː��q�,q�bږq�ٟ��Z���>��U#�����ET�V/��+cU����ʁ�LLe��(���V�2���å}�UC���#�8y��B�7Λ��c���Y?��q��TOMTv{����ߎ������,�[�|��wf_�˰�g������?
The content type in the headers of the post request is:
'Content-Type': 'application/json;charset=UTF-8',
Also by checking the r.encoding i get None
At the end i've tried json.loads(r.text) but i get the same error.
Also tried encoding the response with r.content.decode('utf-8')
and got UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 1: invalid continuation byte
What i'm doing wrong?
EDIT:
I've tried replicating the request with Fiddler and i can clearly see the response json there
I've compared the response headers between Fiddler and requests, the only difference is that with requests i ge 'Transfer-Encoding': 'chunked' in the headers
EDIT2:
Here's a portion of the r.content:
b'\x03\xd5\x01\x00d\xca9\x9cvwC\x7f \x1b]j\xfb\xc5\xa3tA\xa1k.N9\x01\x08\x03\r\xcc\x02i\x01\xb7\x0b\xc2=\xc6\x92\xb1\rb\xe3\xbf\xdb;\x83\xb7\xbf{\xe2vk\xd1\xed\xef\xbf[d<{.n\x7fo\xd5WC\xab\x84\xeb\xecN\xd3#\xcc\xed\x0cAks\xa5\xe6\x87?\x15\xb3:\x8d\xfdS}\xb5O\x9a\xb8\x9d\xddK\x98\xf5+HSLq\x14\x84\xab:Fd!\xa8\xc0\x0c\xd2\xd3\x0eL\xd4\xec\x9b\xcb\xf2\xcf\xae\x18\xd9\x15&fg\xbf\xd6.```
There can be a couple of reasons for this error
I faced the same error. In my case, I am calling a put URL. In your case, it is post
I implemented my put API in such a way that, it won't return anything except status code. So after receiving the Response body contains no data. And we are trying to parse that to JSON. But actually nothing is there to parse.
I'm trying to get a unit test working that validates a function that reads credentials from a JSON-encoded file. Since the credentials themselves aren't fixed, the unit test needs to provide some and then test that they are correctly retrieved.
Here is the credentials function:
def read_credentials():
basedir = os.path.dirname(__file__)
with open(os.path.join(basedir, "authentication.json")) as f:
data = json.load(f)
return data["bot_name"], data["bot_password"]
and here is the test:
def test_credentials(self):
with patch("builtins.open", mock_open(
read_data='{"bot_name": "name", "bot_password": "password"}\n'
)):
name, password = shared.read_credentials()
self.assertEqual(name, "name")
self.assertEqual(password, "password")
However, when I run the test, the json code blows up with a decode error. Looking at the json code itself, I'm struggling to see why the mock test is failing because json.load(f) simply calls f.read() then calls json.loads().
Indeed, if I change my authentication function to the following, the unit test works:
def read_credentials():
# Read the authentication file from the current directory and create a
# HTTPBasicAuth object that can then be used for future calls.
basedir = os.path.dirname(__file__)
with open(os.path.join(basedir, "authentication.json")) as f:
content = f.read()
data = json.loads(content)
return data["bot_name"], data["bot_password"]
I don't necessarily mind leaving my code in this form, but I'd like to understand if I've got something wrong in my test that would allow me to keep my function in its original form.
Stack trace:
Traceback (most recent call last):
File "test_shared.py", line 56, in test_credentials
shared.read_credentials()
File "shared.py", line 60, in read_credentials
data = json.loads(content)
File "/home/philip/.local/share/virtualenvs/atlassian-webhook-basic-3gOncDp4/lib/python3.6/site-packages/flask/json/__init__.py", line 205, in loads
return _json.loads(s, **kwargs)
File "/usr/lib/python3.6/json/__init__.py", line 367, in loads
return cls(**kw).decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I had the same issue and got around it by mocking json.load and builtins.open:
import json
from unittest.mock import patch, MagicMock
# I don't care about the actual open
p1 = patch( "builtins.open", MagicMock() )
m = MagicMock( side_effect = [ { "foo": "bar" } ] )
p2 = patch( "json.load", m )
with p1 as p_open:
with p2 as p_json_load:
f = open( "filename" )
print( json.load( f ) )
Result:
{'foo': 'bar'}
I had the exact same issue and solved it. Full code below, first the function to test, then the test itself.
The original function I want to test loads a json file that is structured like a dictionary, and checks to see if there's a specific key-value pair in it:
def check_if_file_has_real_data(filepath):
with open(filepath, "r") as f:
data = json.load(f)
if "fake" in data["the_data"]:
return False
else:
return True
But I want to test this without loading any actual file, exactly as you describe. Here's how I solved it:
from my_module import check_if_file_has_real_data
import mock
#mock.patch("my_module.json.load")
#mock.patch("my_module.open")
def test_check_if_file_has_real_data(mock_open, mock_json_load):
mock_json_load.return_value = dict({"the_data": "This is fake data"})
assert check_if_file_has_real_data("filepath") == False
mock_json_load.return_value = dict({"the_data": "This is real data"})
assert check_if_file_has_real_data("filepath") == True
The mock_open object isn't called explicitly in the test function, but if you don't include that decorator and argument you get a filepath error when the with open part of the check_if_file_has_real_data function tries to run using the actual open function rather than the MagicMock object that's been passed into it.
Then you overwrite the response provided by the json.load mock with whatever you want to test.
i am using Flask as framework for my server, and while returning a response i get the following error:
> Traceback (most recent call last):
File "C:\Python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "C:\Python27\lib\site-packages\flask_restful\__init__.py", line 480, in wrapper
resp = resource(*args, **kwargs)
File "C:\Python27\lib\site-packages\flask\views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "C:\Python27\lib\site-packages\flask_restful\__init__.py", line 595, in dispatch_request
resp = meth(*args, **kwargs)
File "rest.py", line 27, in get
return jsonify(**solution)
File "C:\Python27\lib\site-packages\flask\json.py", line 263, in jsonify
(dumps(data, indent=indent, separators=separators), '\n'),
File "C:\Python27\lib\site-packages\flask\json.py", line 123, in dumps
rv = _json.dumps(obj, **kwargs)
File "C:\Python27\lib\json\__init__.py", line 251, in dumps
sort_keys=sort_keys, **kw).encode(obj)
File "C:\Python27\lib\json\encoder.py", line 209, in encode
chunks = list(chunks)
File "C:\Python27\lib\json\encoder.py", line 434, in _iterencode
for chunk in _iterencode_dict(o, _current_indent_level):
File "C:\Python27\lib\json\encoder.py", line 408, in _iterencode_dict
for chunk in chunks:
File "C:\Python27\lib\json\encoder.py", line 332, in _iterencode_list
for chunk in chunks:
File "C:\Python27\lib\json\encoder.py", line 332, in _iterencode_list
for chunk in chunks:
File "C:\Python27\lib\json\encoder.py", line 442, in _iterencode
o = _default(o)
File "C:\Python27\lib\site-packages\flask\json.py", line 80, in default
return _json.JSONEncoder.default(self, o)
File "C:\Python27\lib\json\encoder.py", line 184, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: {'origin': u'porto', 'dest': u'lisboa', 'price': '31', 'date': '2017-12-23', 'url': u'https://www.google.pt/flights/#search;f=opo;t=lis;d=2017-12-23;r=2017-12-24'} is not JSON serializable
i have the following function:
from flask import Flask, request, jsonify
from flask_restful import Resource, Api
from flask_cors import CORS, cross_origin
from json import dumps
import flights
import solveProblem
app = Flask(__name__)
api = Api(app)
CORS(app)
class Flights(Resource):
def get(self, data):
print 'received data from client: ' + data
solution = solveProblem.solve(data)
print 'got the solution from the script! \nSOLUTION: \n'
print solution
return jsonify(solution)
api.add_resource(Flights, '/flights/<string:data>')
if __name__ == '__main__':
app.run()
while debugging the problem, i found the following solutions which did not work:
1) return solution instead of {'solution': solution}
2) do jsonify(solution)
3) do jsonify(**solution)
none of the above worked for me;
i wonder why this happens, when i am trying to return a valid dictionary:
{'flights': [[{'origin': u'porto', 'dest': u'lisboa', 'price': '31', 'date': '2017-12-23', 'url': u'https://www.google.pt/flights/#search;f=opo;t=lis;d=2017-12-23;r=2017-12-24'}]], 'cost': '31'}
any help is appreciated.
Thanks
My guess is when you were creating 'solution', the data that got assigned to it was an incorrectly formatted dictionary
{'item', 'value'}
Instead of:
{'item': 'value'}
Thus creating a set instead of a dict
we cannot directly use the jsonify when your trying to converting list of data into json.
there is two approaches are there you can convert list into dictionary for that we need to write function that convert your list data into dictionary which is complicated task .
there is one smart work you can use Marshmallow library . it serialized you list data after that you can use jsonify.
In flask-restful, Resource class get method will just need to return python data structure. So just remove jsonify. For User Defined Object, you can use marshal_with() decorator.
See more: https://flask-restful.readthedocs.io/en/latest/quickstart.html#a-minimal-api
Since most of your functions are declared elsewhere, I worked a toy Flask program just to pass the dictionary you got stuck with.
[Edit] Before I was using the standard python json module. I edited it to use flask's own jsonify, and it works with the direct dictionary still. So the error is not where the OP is looking for.
{'flights': [[{'origin': u'porto', 'dest': u'lisboa', 'price': '31', 'date': '2017-12-23', 'url': u'https://www.google.pt/flights/#search;f=opo;t=lis;d=2017-12-23;r=2017-12-24'}]], 'cost': '31'}
The following program runs and returns the dictionary as a JSON object:
import flask
app = flask.Flask(__name__)
#app.route('/')
def hello():
jdic = flask.jsonify( {'origin': u'porto', 'dest': u'lisboa', 'price': '31', 'date': '2017-12-23', 'url': u'https://www.google.pt/flights/#search;f=opo;t=lis;d=2017-12-23;r=2017-12-24'} )
return jdic
if __name__ == '__main__':
app.run()
As I found out, this error generally occurs when the response is not a pure python dictionary. This happened to me because I was trying to pass a class object. So, to solve the problem, i created a class method which returns a dictionary describing the object, and use this to create the json response.
Conclusion: Use Pure python objects, which are easily translated to JSON.
I had the same problem with a 3 level Nested Dictionary; it was valid, json serializable and via command line json.dumps had no issue. However, Flask did not want to output it: "TypeError", not json serializable. The only difference is that I am using Python 3.5.
So I made a copy of it as a string (that on command line was json serializable!) and passed to Flask output, it worked.
Try to pass the nested json as
eval(str(solution))
and see the error. It's not a definitive solution but more a workaround.
Hope it helps.