Different ways of reading data from s3 bucket using boto3 - json

I'm trying to read some json file from s3 bucket and then trying to post the data into rds and redshift(using a post api developed/intended for that).
I approached to do this in 2 ways using boto3. Below are those:
1st way:
import boto3
import json
import requests
url = 'xxxx.us-east-1.elb.amazonaws.com/v1'
headers = {'content-type': 'application/json', 'Host': 'development.my.dns.com'}
endpoint = url+'/my/post/endpoint'
s3_client = boto3.client("s3")
fileObj = s3_client.get_object(Bucket='my-bucket-name', Key='my-key-name'])
data = fileObj['Body'].read().decode('utf-8')
with requests.request('POST', endpoint, data=data, headers=headers, auth=(username, pwd), verify=False, stream=True) as r:
print("Status Code:",r.status_code)
2nd way:
import boto3
import json
import requests
url = 'xxxx.us-east-1.elb.amazonaws.com/v1'
headers = {'content-type': 'application/json', 'Host': 'development.my.dns.com'}
endpoint = url+'/my/post/endpoint'
s3_res = boto3.resource('s3')
contentObj = s3_res.Object('my-bucket-name', 'my-key-name')
fileContent = contentObj.get()['Body'].read().decode('utf-8')
data = json.dumps(json.loads(fileContent))
with requests.request('POST', endpoint, data=data, headers=headers, auth=(username, pwd), verify=False, stream=True) as r:
print("Status Code:",r.status_code)
Basically everything is same(url, endpoint, headers, requests.request) and type(data) is <class 'str'> in both the approaches. But the status codes are always different.
The 2nd way gives Status Code: 200. But the 1st way gives Status Code: 413 or Status Code: 502 randomly
Can any one please explain why is it this way? what's different in the above two approaches? I'm trying to understand what's going on when boto3 read() the data differently.

Related

Python: request json gets error - If using all scalar values, you must pass an index

When attempting to request a json file from API, i have an error after get the first result.
Does anyone have any idea why is requested an index from a list ?
Best regards :)
import requests
import json
import pandas as pd
import time
import datetime
### OCs List ids
OCs = ['1003473-1116-SE21','1003473-1128-AG21','1031866-12-CC21','1057440-3184-AG21','1070620-1832-CM21', '1070620-2219-SE21', '1070620-2499-CM21']
for i in OCs:
link ="http://api.mercadopublico.cl/servicios/v1/publico/ordenesdecompra.json?codigo="+ str(i) +"&ticket=F8537A18-6766-4DEF-9E59-426B4FEE2844"
response = requests.get(link, [])
data = response.json()
df = pd.DataFrame.from_dict(data)
### remove unnecessary columns
df.drop(df.columns[[0,1,2]],axis=1, inplace=True)
### flat json to pandas dataframe
df_detail = pd.json_normalize(df['Listado'])
ValueError: If using all scalar values, you must pass an index
The server detects too many requests and sends error response (and then your script throws an error). Solution is to wait for correct response, for example:
import requests
import json
import pandas as pd
import time
import datetime
### OCs List ids
OCs = [
"1003473-1116-SE21",
"1003473-1128-AG21",
"1031866-12-CC21",
"1057440-3184-AG21",
"1070620-1832-CM21",
"1070620-2219-SE21",
"1070620-2499-CM21",
]
for i in OCs:
link = (
"http://api.mercadopublico.cl/servicios/v1/publico/ordenesdecompra.json?codigo="
+ str(i)
+ "&ticket=F8537A18-6766-4DEF-9E59-426B4FEE2844"
)
while True: # <--- repeat until we get correct response
print(link)
response = requests.get(link, [])
data = response.json()
if "Listado" in data:
break
time.sleep(3) # <--- wait 3 seconds and try again
df = pd.DataFrame.from_dict(data)
### remove unnecessary columns
df.drop(df.columns[[0, 1, 2]], axis=1, inplace=True)
### flat json to pandas dataframe
df_detail = pd.json_normalize(df["Listado"])
# ...

Using the reults of multiple for loops to post a single json response

Okay, so this is a loaded question but and I'm sure theres an easy method to use here, but I'm stuck.
Long story short, I am tasked with creating a function in python (to be run an AWS lambda) which can perform acceptance tests on a series of URL's using python-requests. These requests will be used to assert the HTTP response codes and a custom HTTP header identifying if an haproxy backend is correct.
The URL's themselves will be maintained in a yaml document which will be converted to a dict in python and passed to a for loop which will use python requests to HTTP GET the response code and header of the URL.
The issue I am having is getting a single body object to return the results of multiple for loops.
I have tried to find similar use cases but cannot
import requests
import json
import yaml
def acc_tests():
with open("test.yaml", 'r') as stream:
testurls = yaml.safe_load(stream)
results = {}
# endpoint/path 1
for url in testurls["health endpoints"]:
r = requests.get(url, params="none")
stat = r.status_code
result = json.dumps(print(url, stat))
results = json.dumps(result)
# endpoint path with headers
for url in testurls["xtvapi"]:
headers = {'H': 'xtvapi.cloudtv.comcast.net'}
r = requests.get(url, headers=headers, params="none")
stat = r.status_code
head = r.headers["X-FINITY-TANGO-BACKEND"]
result = json.dumps((url, stat, head))
results = json.dumps(result)
return {
'statusCode': 200,
'body': json.dumps(results)
}
acc_tests()
YAML file:
health endpoints:
- https://xfinityapi-tango-production-aws-us-east-1-active.r53.aae.comcast.net/tango-health/
- https://xfinityapi-tango-production-aws-us-east-1-active.r53.aae.comcast.net/
- https://xfinityapi-tango-production-aws-us-east-2-active.r53.aae.comcast.net/tango-health/
- https://xfinityapi-tango-production-aws-us-east-2-active.r53.aae.comcast.net/
- https://xfinityapi-tango-production-aws-us-west-2-active.r53.aae.comcast.net/tango-health/
- https://xfinityapi-tango-production-aws-us-west-2-active.r53.aae.comcast.net/
xtvapi:
- https://xfinityapi-tango-production-aws-us-east-1-active.r53.aae.comcast.net/
- https://xfinityapi-tango-production-aws-us-east-2-active.r53.aae.comcast.net/
- https://xfinityapi-tango-production-aws-us-west-2-active.r53.aae.comcast.net/
What I think is happening is that both for loops are running one after another, but the value of results is empty, but I'm not sure what to do in order to update/append the results dict with the results of each loop.
Thanks folks. I ended up solving this by creating a dict with immutable keys for each test type and then using append to add the results to a nested list within the dict.
Here is the "working" code as it is in the AWS Lambda function:
from botocore.vendored import requests
import json
import yaml
def acc_tests(event, context):
with open("test.yaml", 'r') as stream:
testurls = yaml.safe_load(stream)
results = {'tango-health': [], 'xtvapi': []}
# Tango Health
for url in testurls["health endpoints"]:
r = requests.get(url, params="none")
result = url, r.status_code
assert r.status_code == 200
results["tango-health"].append(result)
# xtvapi default/cloudtv
for url in testurls["xtvapi"]:
headers = {'H': 'xtvapi.cloudtv.comcast.net'}
r = requests.get(url, headers=headers, params="none")
result = url, r.status_code, r.headers["X-FINITY-TANGO-BACKEND"]
assert r.status_code == 200
assert r.headers["X-FINITY-TANGO-BACKEND"] == "tango-big"
results["xtvapi"].append(result)
resbody = json.dumps(results)
return {
'statusCode': 200,
'body': resbody
}

json posts in micropython

I use the code below in python on a rpi to do json posts
import requests
payload = '{"msg_type":"observation_payload","msg":
{"serial":"FZtank","observations":{"8":{"d":0,"m":"JSON upload
test","t":"1543540780"}}}}'
r = requests.post('https://my.website/api/observations', data =
{'post_wrapper':payload})
print r.text
it produces
{"msg": {"remote_pi": "FZtank", "stored_ids": ["8"]}, "msg_type": "receipt"}
I figured it wouldn't be too hard to do the same thing in micropython on a pycom device
import urequests as requests
payload = '{"msg_type":"observation_payload","msg":
{"serial":"FZtank","observations":{"8":{"d":0,"m":"JSON upload
test","t":"1543540780"}}}}'
r = requests.post('https://my.website/api/observations', data =
{"post_wrapper":payload})
print(r.text)
r.close()
but it gives
Traceback (most recent call last):
File "<stdin>", line 12, in <module>
File "/flash/lib/urequests.py", line 115, in post
File "/flash/lib/urequests.py", line 100, in request
File "/flash/lib/urequests.py", line 79, in request
TypeError: object with buffer protocol required
The TypeError msg means it wants a string instead of a dictionary for the data
so I tried
import urequests as requests
payload = '{"post_wrapper":{"msg_type":"observation_payload","msg":
{"serial":"FZtank","observations":{"8":{"d":0,"m":"JSON upload
test","t":"1543540780"}}}}}'
r = requests.post('https://my.website/api/observations', data =payload)
print(r.text)
r.close()
but that gives
{"msg": {"errorText": "Malformed POST - unable to decode JSON"}, "msg_type":
"error"}
So now I'm confused. Is the problem that the website expects a dictionary & micropython can only send a string?
.
Some further detail:
I'm using https://github.com/micropython/micropython-lib/blob/master/urequests/urequests.py but with line 52 changed from ai = usocket.getaddrinfo(host, port, 0, usocket.SOCK_STREAM) to ai = usocket.getaddrinfo(host, port) because it used to throw an error msg about 4 arguments Vs 2. If I run
import urequests as requests
payload = '{"post_wrapper":{"msg_type":"observation_payload","msg":
{"serial":"FZtank","observations":{"8":{"d":0,"m":"JSON upload
test","t":"1543540780"}}}}}'
r = requests.post('https://my.website/api/observations', data=payload)
or
import urequests as requests
payload = {"post_wrapper":{"msg_type":"observation_payload","msg":
{"serial":"FZtank","observations":{"8":{"d":0,"m":"JSON upload
test","t":"1543540780"}}}}}
r = requests.post('https://my.website/api/observations', json=payload)
or
import urequests as requests
payload = {"msg_type":"observation_payload","msg":
{"serial":"FZtank","observations":{"8":{"d":0,"m":"JSON upload
test","t":"1543540780"}}}}
r = requests.post('https://my.website/api/observations', json=
{"post_wrapper":payload})
I get the same result
{"msg": {"errorText": "Malformed POST - unable to decode JSON"}, "msg_type":
"error"}
I was stuck with the same issue here while trying to send push notifications through pushed.co.
I was using data=payload on urequests and getting this TypeError: object with buffer protocol required error. All worked well when I changed to json=payload following #stijn suggestion on his answer.
Steps I did:
on micropython REPL install urequests
import upip
upip.install('micropython-urequests')
for my example the code is
import urequests as requests
payload = {
"app_key": "your_app_key",
"app_secret": "your_app_secret",
"target_type": "your_target_type",
"content": "Hello from micropython",
}
r = requests.post("https://api.pushed.co/1/push", json=payload)
Probably any other url will do fine too.
I was able to make posts request on micropython dumping data into a json:
import json
import urequests
data = {"mykey": 'myvalue'}
urequests.post(url, data=json.dumps(data))

JWT is being converted from str to bytes on requests.post()

I am writing a Flask application in which I have a service that generates a JWT and passes this onto another service using requests.post(), after decoding it to 'UTF-8'.
While sending the JWT, I can see that the type is 'str'. However, on performing a json.loads() on the other service, I get an error that says
TypeError: the JSON object must be str, not 'bytes'
Here is my code:
Service 1:
#app.route('/')
def index():
token = jwt.encode({'message': 'Hello'}, app.config['SECRET_KEY'])
# After this statement I am able to verify the type is str and not bytes
token = token.decode('UTF-8')
headers = {'content-type': 'application/json'}
url = 'someUrl'
data = {"token": token}
data = json.dumps(data)
requests.post(url, data=data, headers=headers)
return 'Success'
Service 2:
#app.route('/', methods=['POST'])
def index():
data = json.loads(request.data)
return 'Success'
Why do I get this error even though the type was converted to string ?
EDIT: I was able to successfully retrieve the token by passing it through header. But I would still like to know what caused this error.
You could post it as JSON instead of data, and let the underlying library take care of it for you.
Service 1
#app.route('/')
def index():
token = jwt.encode({'message': 'Hello'}, app.config['SECRET_KEY']).decode('UTF-8')
url = 'someUrl'
data = {"token": token}
requests.post(url, json=data)
return 'Success'
Service 2
data = request.get_json()

How to get JSON data in an Odoo controller using type='json'?

A few days ago I did a similar question here: How to get JSON data in an Odoo controller?
But now, I need to create a controller which receives only JSON data. So, I am doing the request from a Python console, this way:
import requests
import json
url = 'http://localhost:8069/odoo/test'
headers = {'Content-Type': 'application/json'}
data = {
'name': 'Jane',
'email': 'jane.doe#gmail.com',
}
data_json = json.dumps(data)
r = requests.post(url=url, data=data_json, headers=headers)
I have created a controller which listens to http://localhost:8069/odoo/test, this way:
import openerp.http as http
from openerp.http import Response
import logging
_logger = logging.getLogger(__name__)
class WebFormController(http.Controller):
#http.route('/odoo/test', type='json',
auth='public', methods=['POST'], website=True)
def index(self, **args):
_logger.info('CONNECTION SUCCESSFUL')
_logger.info(args)
name = args.get('name', False)
email = args.get('email', False)
_logger.info(name)
_logger.info(email)
if not name:
Response.status = '400 Bad Request'
return '{"response": "OK"}'
The problem is that I am receiving an empty JSON in the controller. I can read CONNECTION SUCCESFUL in the log, with no error, but when I show args, I get {}, and obviously due to that, False when writing name and email.
If I pass the data as a Python dictionary or as a string, I get the following error:
Invalid JSON data: 'name=Jane&email=jane.doe%40gmail.com' or
Invalid JSON data: "{'name': 'Jane', 'email': 'jane.doe#gmail.com'}" respectively.
If I modify type='json' and I write type='http' instead, I get the following error:
Function declared as capable of handling request of type 'http' but called with a request of type 'json'.
I have read that may be this could be solved if the request is sent using the parameter json instead of data, this way:
r = requests.post(url=url, json=data_json, headers=headers)
Unfortunately, the server which is going to make the request has an old operating system which cannot update the python-requests package, so I cannot use json parameter since it did not exist at the version installed in that server.
Please, can anyone help me? I need get JSON data in the controller, not a string neither Python dictionaries.
You have just forgotten to put your data inside the params keywords:
Use this correct syntax :
data = {"params": dict(key="value")}
data = {
"params": {
"name":"prakashsharma",
"email":"prakashsharmacs24#gmail.com",
"phone":"+917859884833"
}
}
Please don't forget to use json.dumps(data) and 'Content-Type': 'application/json' while requesting a resource in json format.
I am damn sure your issue will be solved after using this one my friend... cheers :)!!
You can use below format for a POST request
{
"params" : {
"name" : "Order/1/18",
"session_id" : 1,
"customer_count" : 2,
"partner_id" : 9,
"lines": [
{
"product_id": 37,
"qty" : 2,
"price_unit" : 2,
"discount" : 10
}
],
"pos_reference" : 2,
"note" : "This is a test note"
}
}
Content type must be application/json
How odoo route will handle request ?
Route will help creating a POS order in odoo [POST]
#http.route(['/api/v1/resources/<string:api_key>/pos_order'],
auth="public",
website=False,
type="json",
csrf=False,
methods = ['POST'])
def create_update_pos_order(self, api_key=None, **kwargs):
print(kwargs.get('name')) -> Order/1/18
God Bless Forvas::
But for more clearity:
if you want to test through cURL:
curl -i -X POST -H "Content-Type: application/json" -d '{"params": {"name":"prakashsharma","email":"prakashsharmacs24#gmail.com","phone":"+917859884833"}}' 'http://localhost:8069/web/yourlistoner/'
if you want to test through python request:
import requests
headers = {
'Content-Type': 'application/json',
}
data = '{"params": {"name":"prakashsharma","email":"prakashsharmacs24#gmail.com","phone":"+917859884833"}}'
requests.post('http://localhost:8069/web/yourlistoner/', headers=headers, data=data)
the function in odoo will be something like
from odoo import http
import json
class YourClass(http.Controller):
#http.route('/web/yourlistoner/', type='json', auth="none", methods=['POST'],cors="*", csrf=False)
def listoner(self, **kw):
print http.request.params
print "lllllllllllllllllllll"
return json.dumps({"result":"Success"})