Python 3.6: get JSON from aiohttp request - json

I have some application which uses aiohttp.
I sent POST request into approptiate endpoint, e.g.:
POST mysite.com/someendpoind/
with data similar to:
{"param1": "value1", "param2": "value2", ..., "paramn": None}
Then on backend side, I want to add some additional conditional into this request:
data = await request.json()
data["additional_conditional"] = True
But request.json() fails with an error:
[ERROR] Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/aiohttp/web_protocol.py", line 422, in start
resp = yield from self._request_handler(request)
File "/usr/local/lib/python3.5/dist-packages/aiohttp/web.py", line 306, in _handle
resp = yield from handler(request)
File "/usr/local/lib/python3.5/dist-packages/aiohttp_session/__init__.py", line 129, in middleware
response = yield from handler(request)
File "/opt/bikeamp/auth/__init__.py", line 57, in wrapped
return (yield from f(request, user))
File "<my_module>.py", line 185, in <my_func>
data_json = await request.json()
File "/usr/local/lib/python3.5/dist-packages/aiohttp/web_request.py", line 469, in json
return loads(body)
File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Then I decided to check somehow what is the content of my request:
await request.read()
b'field1=value1&field2=value2&field3=value3&field4=&field5=&field6='
So, I'm not sure, but the problem may be with empty parameters.
Also, I was trying to get this data via:
data = await request.post()
data["additional_condition"] = True
But this returns MultiDictProxy. Python can't pickle these objects.
Is there any known solutions?

I had the same issue, if post was something like {"email": "some#email.com"} check it with:
#router('/', methods=['POST', ])
async def post_request(request):
post = await request.post()
email = post.get('email') # because it's MultiDict
logging.warning(post) # see post details
logging.warning(email) # shows value "some#email.com"
json = await request.text() #
logging.warning(json) # shows json if it was ajax post request

Related

How to use marshmallow-sqlalchemy with async code?

I'm trying to use marshmallow-sqlalchemy with aiohttp and I have followed their docs with the basic example and I'm getting an error.
I have this schema:
from marshmallow_sqlalchemy import SQLAlchemyAutoSchema
from db.customer import Customer
class CustomerSchema(SQLAlchemyAutoSchema):
class Meta:
model = Customer
include_relationships = True
load_instance = True
And then the following code for the query:
from sqlalchemy import select
from db import db_conn
from db.customer import Customer
from queries.schema import CustomerSchema
customer_schema = CustomerSchema()
async def get_all_users():
async with db_conn.get_async_sa_session() as session:
statement = select(Customer)
results = await session.execute(statement)
_ = (results.scalars().all())
print(_)
response = customer_schema.dump(_, many=True)
print(response)
For the first print statement I'm getting
[<db.customer.Customer object at 0x10a183340>, <db.customer.Customer object at 0x10a183940>, <db.customer.Customer object at 0x10b0cd9d0>]
But then it fails with
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 60, in await_only
raise exc.MissingGreenlet(
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_() here. Was IO attempted in an unexpected place? (Background on this error at: http://sqlalche.me/e/14/xd2s)
So how can I use marshmallow-sqlalchemy to serialize the SqlAlchemy reponse?
Another options (packages, etc) or a generic custom solutions are OK too.
For the time being I'm using this:
statement = select(Customer)
results = await session.execute(statement)
_ = (results.scalars().all())
response = {}
for result in _:
value = {k: (v if not isinstance(v, sqlalchemy.orm.state.InstanceState) else '_') for k, v in result.__dict__.items()}
response[f'customer {value["id"]}'] = value
return response
Full traceback:
Traceback (most recent call last):
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
resp = await self._request_handler(request)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/aiohttp/web_app.py", line 499, in _handle
resp = await handler(request)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/aiohttp/web_urldispatcher.py", line 948, in _iter
resp = await method()
File "/Users/ruslan/OneDrive/Home/Dev/projects/code/education/other/cft/views/user.py", line 24, in get
await get_all_users()
File "/Users/ruslan/OneDrive/Home/Dev/projects/code/education/other/cft/queries/user.py", line 18, in get_all_users
response = customer_schema.dump(_, many=True)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/schema.py", line 547, in dump
result = self._serialize(processed_obj, many=many)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/schema.py", line 509, in _serialize
return [
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/schema.py", line 510, in <listcomp>
self._serialize(d, many=False)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/schema.py", line 515, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/fields.py", line 310, in serialize
value = self.get_value(obj, attr, accessor=accessor)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow_sqlalchemy/fields.py", line 27, in get_value
return super(fields.List, self).get_value(obj, attr, accessor=accessor)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/fields.py", line 239, in get_value
return accessor_func(obj, check_key, default)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/schema.py", line 472, in get_attribute
return get_value(obj, attr, default)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/utils.py", line 239, in get_value
return _get_value_for_key(obj, key, default)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/utils.py", line 253, in _get_value_for_key
return getattr(obj, key, default)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 480, in __get__
return self.impl.get(state, dict_)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 931, in get
value = self.callable_(state, passive)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 879, in _load_for_state
return self._emit_lazyload(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 1036, in _emit_lazyload
result = session.execute(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1689, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1582, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/sql/lambdas.py", line 481, in _execute_on_connection
return connection._execute_clauseelement(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1451, in _execute_clauseelement
ret = self._execute_context(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
self._handle_dbapi_exception(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1998, in _handle_dbapi_exception
util.raise_(exc_info[1], with_traceback=exc_info[2])
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1770, in _execute_context
self.dialect.do_execute(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
cursor.execute(statement, parameters)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 449, in execute
self._adapt_connection.await_(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 60, in await_only
raise exc.MissingGreenlet(
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_() here. Was IO attempted in an unexpected place? (Background on this error at: http://sqlalche.me/e/14/xd2s)
The problem in this case is that the Marshmallow schema is configured to load related models (include_relationships=True). Since the initial query doesn't load them automatically, the schema triggers a query to fetch them, and this causes the error.
The simplest solution, demonstrated in the docs, is to eagerly load the related objects with their "parent":
async def get_all_users():
async with db_conn.get_async_sa_session() as session:
# Let's assume a Customer has a 1 to many relationship with an Order model
statement = select(Customer).options(orm.selectinload(Customer.orders))
results = await session.execute(statement)
_ = (results.scalars().all())
print(_)
response = customer_schema.dump(_, many=True)
print(response)
There is more discussion in the Preventing Implicit IO when Using AsyncSession section of the docs.

Receive API body POST by Python

I have a problem with API POST, can you help me?
import requests
import json
url = 'http://xxx/api/getTotalPrice'
param = dict(itineraryType=1,
departureAirportCode='HAN',
destinationAirportCode='DLI',
departureDate='2020-12-30T14:00',
returnDate='2020-12-30T09:00',
adult=1,
children=1,
infant=1
)
resp = requests.post(url=url, params=param)
u_data = resp.json()
print(u_data)
I want to receive data from API POST. This is body API and this made error.
Here, this is error
C:\Users\Admin\PycharmProjects\untitled\venv\Scripts\python.exe C:/Users/Admin/PycharmProjects/untitled/venv/test.py
Traceback (most recent call last):
File "C:\Users\Admin\PycharmProjects\untitled\venv\test.py", line 17, in <module>
u_data = resp.json()
File "C:\Users\Admin\PycharmProjects\untitled\venv\lib\site-packages\requests\models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Process finished with exit code 1
I can get data from POST API without body, but with body, I cannot.
Can you help to fix code?
Many thanks!
P/S:
Here value which I want to get
{
"departureFlight": {
"totalPrice": 1896000.0,
"airlineCode": "VN"
},
"returnFlight": {
"totalPrice": 3263800.0,
"airlineCode": "VJ"
}
}
OK, I have an answer.
data = {'itineraryType':1,'departureAirportCode':'HAN','destinationAirportCode':'DLI','departureDate':'2020-12-30T15:00','returnDate':'2020-12-30T09:00','adult':1,'children':1,'infant':1}
headers={'Content-type':'application/json', 'Accept':'application/json'}
resp = requests.post(url=url, json=data, headers=headers)
u_data = resp.json()
print(u_data)
change dict(...) to {...} and set headers.

Python requests - JSONDecodeError("Expecting value", s, err.value) from None

I'm trying to grab a response from an api but i cannot load it in a dictionary.
Here's my code:
r = session.post(url, headers=headers, json=payload)
The returned status code is 200 so i get a response.
By trying to do: reply = r.json() i get:
Traceback (most recent call last):
File "C:/Users/PycharmProjects/honeymonitor/main.py", line 46, in <module>
data = get_data(session)
File "C:/Users/PycharmProjects/honeymonitor/main.py", line 35, in get_data
reply = r.json()
File "C:\Users\PycharmProjects\honeymonitor\venv\lib\site-packages\requests\models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\AppData\Local\Programs\Python\Python37\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
And by using reply = r.text i get a lot of gibberish:
� d�M���)b�b��$�j�np������X2�Al�w{g��wO�n-���w��g�����}�1�Y5#�R��/�
� X���o]>6,�N�_#�#Ң����K�#���s
��Ɩ�ː�����2��N�:D��p�����7{K��������V��
-")�p0n#I�f!���Q�+| /We�.�p�=����u���-�ʌ�cs�"jfH����}qkː��q�,q�bږq�ٟ��Z���>��U#�����ET�V/��+cU����ʁ�LLe��(���V�2���å}�UC���#�8y��B�7Λ��c���Y?��q��TOMTv{����ߎ������,�[�|��wf_�˰�g������?
The content type in the headers of the post request is:
'Content-Type': 'application/json;charset=UTF-8',
Also by checking the r.encoding i get None
At the end i've tried json.loads(r.text) but i get the same error.
Also tried encoding the response with r.content.decode('utf-8')
and got UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 1: invalid continuation byte
What i'm doing wrong?
EDIT:
I've tried replicating the request with Fiddler and i can clearly see the response json there
I've compared the response headers between Fiddler and requests, the only difference is that with requests i ge 'Transfer-Encoding': 'chunked' in the headers
EDIT2:
Here's a portion of the r.content:
b'\x03\xd5\x01\x00d\xca9\x9cvwC\x7f \x1b]j\xfb\xc5\xa3tA\xa1k.N9\x01\x08\x03\r\xcc\x02i\x01\xb7\x0b\xc2=\xc6\x92\xb1\rb\xe3\xbf\xdb;\x83\xb7\xbf{\xe2vk\xd1\xed\xef\xbf[d<{.n\x7fo\xd5WC\xab\x84\xeb\xecN\xd3#\xcc\xed\x0cAks\xa5\xe6\x87?\x15\xb3:\x8d\xfdS}\xb5O\x9a\xb8\x9d\xddK\x98\xf5+HSLq\x14\x84\xab:Fd!\xa8\xc0\x0c\xd2\xd3\x0eL\xd4\xec\x9b\xcb\xf2\xcf\xae\x18\xd9\x15&fg\xbf\xd6.```
There can be a couple of reasons for this error
I faced the same error. In my case, I am calling a put URL. In your case, it is post
I implemented my put API in such a way that, it won't return anything except status code. So after receiving the Response body contains no data. And we are trying to parse that to JSON. But actually nothing is there to parse.

Error json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I want to create a temporary JSON file to store the google calendar credentials, that were stored in a "job" object.
I am using the ServiceAccountCredentials to then get the credentials from the file.
Client = {
"clientID": job.getClientID(),
"clientSecret": job.getClientSecret()
}
temp = tempfile.NamedTemporaryFile(mode="w+b", suffix=".json")
complex_data = open(temp.name, "w", encoding="UTF-8")
complex_data.write(json.dumps(Client))
# data = complex_data.write(json.dumps(Client))
# z = json.loads(data)
credentials = ServiceAccountCredentials.from_json(
temp.name
)
```
I get the following error:
Traceback (most recent call last):
File "/var/www/library-offers-google-calendar/main.py", line 209, in <module>
temp.name
File "/var/www/library-offers-google-calendar/venv/lib/python3.7/site-packages/oauth2client/service_account.py", line 436, in from_json
json_data = json.loads(_helpers._from_bytes(json_data))
File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
From the docs, the method ServiceAccountCredentials.from_json accepts a json data rather than a json file name.
If you want to use a json file name, you need to use the method ServiceAccountCredentials.from_json_keyfile_name
Or else, you can directly use Client dictionary as a parameter to ServiceAccountCredentials.from_json without creating and reading from temporary file.

json decod error when using POST in chalice

When I try to use app.current_request.json_body in chalice I get a decode error:
Traceback (most recent call last): File "/var/task/chalice/app.py",
line 659, in _get_view_function_response
response = view_function(**function_args) File "/var/task/app.py", line 34, in post_item
data = app.current_request.json_body File "/var/task/chalice/app.py", line 303, in json_body
self._json_body = json.loads(self.raw_body) File "/var/lang/lib/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s) File "/var/lang/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/var/lang/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char
0)
It doesn't matter how simple the data is. Example: {"Company":"ABC"} or {}.
As can be seen in the following code in the API Gateway all I try to do is return the data that has been sent so I don't think this is the problem:
#app.route('/test', methods=['POST'], content_types=['application/json'], cors=cors_config)
def post_item(data):
data = app.current_request.json_body
return data
Does anyone know what I might have done wrong?
You must remove the data from the parameters of the function.
That is used for pass url parameters.
#app.route('/test', methods=['POST'], content_types=['application/json'], cors=cors_config)
def post_item():
data = app.current_request.json_body
return data