How do I access single JSON API item? (Python 3.6) - json

I am trying to access the values from an API json item called DataClasses that looks like this.
"DataClasses":["Dates of birth","Email addresses","IP addresses","Passwords","Usernames","Website activity"]
With this code:
import requests
import json
with open('dataclasstest.txt', 'r') as f:
emails = [line.strip() for line in f]
print("Email" + ":" + "Data Classes")
def main():
for name in emails:
url = 'https://haveibeenpwned.com/api/v2/breachedaccount/' + name
headers = {
'User-Agent': 'dataclass.py'
'Accept: application/vnd.haveibeenpwned.v2+json'
}
r = requests.get(url, headers=headers)
content = r.json()
data = json.loads(content)
if r.status_code == 200:
for item in data:
print(unicode(item['DataClasses']))
time.sleep(2)
elif r.status_code == 404:
print("No Data")
time.sleep(2)
else:
time.sleep(2)
if __name__ = "__main__":
main()
Can anyone help me get on the right track? I've been looking online for weeks and still can't figure it out. I've found similar answers on here but when I try them it still doesn't work. Thanks in advance!

Here is how I did it. I will post only the relevant lines of code:
r = requests.get(url, headers=headers)
print(json.loads(r.content)[0].get('DataClasses'))
After reading the docs for requests the code above can be rewritten more simpler.
r = requests.get(url, headers=headers)
print(r.json()[0].get('DataClasses'))
Because we know that the response represents JSON data, requests has a built-in JSON decoder. r.json() returns a list of objects, or a list of dict. We then extract the first element with r.json()[0]. The .get method is exactly the same as calling r.json()[0]['DataClasses']. You can read more about it here.

Related

Using the reults of multiple for loops to post a single json response

Okay, so this is a loaded question but and I'm sure theres an easy method to use here, but I'm stuck.
Long story short, I am tasked with creating a function in python (to be run an AWS lambda) which can perform acceptance tests on a series of URL's using python-requests. These requests will be used to assert the HTTP response codes and a custom HTTP header identifying if an haproxy backend is correct.
The URL's themselves will be maintained in a yaml document which will be converted to a dict in python and passed to a for loop which will use python requests to HTTP GET the response code and header of the URL.
The issue I am having is getting a single body object to return the results of multiple for loops.
I have tried to find similar use cases but cannot
import requests
import json
import yaml
def acc_tests():
with open("test.yaml", 'r') as stream:
testurls = yaml.safe_load(stream)
results = {}
# endpoint/path 1
for url in testurls["health endpoints"]:
r = requests.get(url, params="none")
stat = r.status_code
result = json.dumps(print(url, stat))
results = json.dumps(result)
# endpoint path with headers
for url in testurls["xtvapi"]:
headers = {'H': 'xtvapi.cloudtv.comcast.net'}
r = requests.get(url, headers=headers, params="none")
stat = r.status_code
head = r.headers["X-FINITY-TANGO-BACKEND"]
result = json.dumps((url, stat, head))
results = json.dumps(result)
return {
'statusCode': 200,
'body': json.dumps(results)
}
acc_tests()
YAML file:
health endpoints:
- https://xfinityapi-tango-production-aws-us-east-1-active.r53.aae.comcast.net/tango-health/
- https://xfinityapi-tango-production-aws-us-east-1-active.r53.aae.comcast.net/
- https://xfinityapi-tango-production-aws-us-east-2-active.r53.aae.comcast.net/tango-health/
- https://xfinityapi-tango-production-aws-us-east-2-active.r53.aae.comcast.net/
- https://xfinityapi-tango-production-aws-us-west-2-active.r53.aae.comcast.net/tango-health/
- https://xfinityapi-tango-production-aws-us-west-2-active.r53.aae.comcast.net/
xtvapi:
- https://xfinityapi-tango-production-aws-us-east-1-active.r53.aae.comcast.net/
- https://xfinityapi-tango-production-aws-us-east-2-active.r53.aae.comcast.net/
- https://xfinityapi-tango-production-aws-us-west-2-active.r53.aae.comcast.net/
What I think is happening is that both for loops are running one after another, but the value of results is empty, but I'm not sure what to do in order to update/append the results dict with the results of each loop.
Thanks folks. I ended up solving this by creating a dict with immutable keys for each test type and then using append to add the results to a nested list within the dict.
Here is the "working" code as it is in the AWS Lambda function:
from botocore.vendored import requests
import json
import yaml
def acc_tests(event, context):
with open("test.yaml", 'r') as stream:
testurls = yaml.safe_load(stream)
results = {'tango-health': [], 'xtvapi': []}
# Tango Health
for url in testurls["health endpoints"]:
r = requests.get(url, params="none")
result = url, r.status_code
assert r.status_code == 200
results["tango-health"].append(result)
# xtvapi default/cloudtv
for url in testurls["xtvapi"]:
headers = {'H': 'xtvapi.cloudtv.comcast.net'}
r = requests.get(url, headers=headers, params="none")
result = url, r.status_code, r.headers["X-FINITY-TANGO-BACKEND"]
assert r.status_code == 200
assert r.headers["X-FINITY-TANGO-BACKEND"] == "tango-big"
results["xtvapi"].append(result)
resbody = json.dumps(results)
return {
'statusCode': 200,
'body': resbody
}

How to pass a soup object in a JSON payload prepared for requests.put method in python?

I'm trying to read a web page using requests module in python, scrape the read data using beautifulsoup, modify the data in the soup object and write it back to the same web page using requests.put() method.
Is there anything that I got to do additionally in the soup object to get it assigned to the JSON payload?
Modules used for this purpose:
1. JSON
2. requests
3. BeautifulSoup
If the JSON payload that I'm preparing is without soup object, then the requests.put() writes the data properly
import requests
from bs4 import BeautifulSoup
response = requests.get(confluenceServerURL+getPageURL, verify=False, headers=headers)
# 200 OK is the response for the web page is accessible
if response.status_code == 200:
jsonD = response.json()
pageId = jsonD["results"][0]['id']
versionNum = jsonD["results"][0]['version']['number']
body = jsonD["results"][0]['body']['storage']['value']
soup = BeautifulSoup(response.content, 'html.parser')
input_str = "test string"
testMan = "found string"
for row in soup.find_all('tr'):
for node in row.find_all('td'):
if node.text == testMan:
node.string = input_str
newBody = soup
else: #response.status_code
logger.error("Error while obtaining page with given page title %s ", response.status_code)
exit(1);
## Space to write confluence page
logger.info('--------------Updating page--------------')
updatePageURL = "/rest/api/content/"+pageId
jsonPayload = {"id":pageId,"type":"page","title":pageTitle,"space":{"key":spaceKey},
"body":{"storage":{"value":"{0}".format(newBody),"representation":"storage"}},
"version":{"number":int(versionNum)+1}}
#print(jsonPayload)
# update page
response = requests.put(confluenceServerURL+updatePageURL,verify=False,data=json.dumps(jsonPayload),headers=headers)
logger.debug('URL for updating page is %s', response.url)
if response.status_code == 200:
logger.info("Page with id %s updated successfully",pageId)
#break;
else:
logger.error("Unable to update page with id %s",pageId)
logger.error(response.text)
I get the response code as 200 for request.put() but the page remains empty if beautifulsoup is being used. If it is directly in JSON format, the page is getting updated successfully
INFO:conf_rest_get_page:--------------Getting page -------------------
DEBUG:conf_rest_get_page:Getting SMSESSION from URL
Accessible
DEBUG:conf_rest_get_page:URL for getting confluence page is URL
INFO:conf_rest_get_page:--------------Updating page ----------------
INFO:conf_rest_get_page:Page with id 261042277 updated successfully
2019-06-09 10.26.58 ===== table procedure Ended, Status : True =====

How to correctly produce and consume UTF-8 json in Python3? (with flask and requests)

I've made some basic code to explain my question:
produceUTF8.py (replies 'ít wórked!' with unicode characters) - you run this first
# -*- coding: utf-8 -*-
import os
from sys import argv
from flask import Flask, request, Response, jsonify
import json
app = Flask(__name__)
app.config['JSON_AS_ASCII'] = False # contribution from Erdem
#app.route('/reply', methods=['POST'])
def reply():
"""Fetch a reply
"""
print("DEBUG entered")
params = request.json
print("DEBUG entered2")
if not params:
return jsonify({
'status': 'error',
'error': 'Request must be of the application/json type!',
})
reply = "ít wórked!"
# Send the response.
return jsonify({
'status': 'ok',
'reply': reply,
})
if __name__ == '__main__':
app.run(host='0.0.0.0', debug=True)
consumeUTF8.py (posts the message 'óíá' to get answer from producer)
# -*- coding: utf-8 -*-
import requests
HEADERS = {'Content-Type': 'application/json; charset=utf-8',}
DATA = '{"message": "óíá"}'
my_request = requests.post('http://localhost:5000/reply', headers=HEADERS, data=DATA)
response = my_request.json()['reply']
In my producer I am getting "Bad Request (400)" and in the consumer "json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)."
It seems to be a problem in params = request.json as seen in the debug prints. What is the recommended approach here?
Thanks!
You can fix the way you make the request by encoding the data object :
my_request = requests.post('http://localhost:5000/reply',
headers=HEADERS,
data=DATA.encode('utf-8'))
#>>> ít wórked with óíá!
If you add a try/except statement in the app, it returns :
try:
params = request.json
except Exception as e:
params = None
print(e)
400 Bad Request: Failed to decode JSON object: 'utf-8' codec can't decode byte 0xf3 in position 14: invalid continuation byte
You could use this pattern to assign a default value for param
I have not tried the code but may be setting
app.config['JSON_AS_ASCII'] = False
may help.

JWT is being converted from str to bytes on requests.post()

I am writing a Flask application in which I have a service that generates a JWT and passes this onto another service using requests.post(), after decoding it to 'UTF-8'.
While sending the JWT, I can see that the type is 'str'. However, on performing a json.loads() on the other service, I get an error that says
TypeError: the JSON object must be str, not 'bytes'
Here is my code:
Service 1:
#app.route('/')
def index():
token = jwt.encode({'message': 'Hello'}, app.config['SECRET_KEY'])
# After this statement I am able to verify the type is str and not bytes
token = token.decode('UTF-8')
headers = {'content-type': 'application/json'}
url = 'someUrl'
data = {"token": token}
data = json.dumps(data)
requests.post(url, data=data, headers=headers)
return 'Success'
Service 2:
#app.route('/', methods=['POST'])
def index():
data = json.loads(request.data)
return 'Success'
Why do I get this error even though the type was converted to string ?
EDIT: I was able to successfully retrieve the token by passing it through header. But I would still like to know what caused this error.
You could post it as JSON instead of data, and let the underlying library take care of it for you.
Service 1
#app.route('/')
def index():
token = jwt.encode({'message': 'Hello'}, app.config['SECRET_KEY']).decode('UTF-8')
url = 'someUrl'
data = {"token": token}
requests.post(url, json=data)
return 'Success'
Service 2
data = request.get_json()

Working with JSON and Django

I am new to Python and Django. I am an IT professional that deploys software that monitors computers. The api outputs to JSON. I want to create a Django app that reads the api and outputs the data to an html page. Where do I get started? I think the idea is to write the JSON feed to a Django model. Any help/advice is greatly appreciated.
Here's a simple single file to extract the JSON data:
import urllib2
import json
def printResults(data):
theJSON = json.loads(data)
for i in theJSON[""]
def main():
urlData = ""
webUrl = urllib2.urlopen(urlData)
if (webUrl.getcode() == 200):
data = webUrl.read()
printResults(data)
else:
print "Received error"
if __name__ == '__main__':
main()
If you have an URL returning a json as response, you could try this:
import requests
import json
url = 'http://....' # Your api url
response = requests.get(url)
json_response = response.json()
Now json_response is a list containing dicts. Let's suppose you have this structure:
[
{
'code': ABC,
'avg': 14.5,
'max': 30
},
{
'code': XYZ,
'avg': 11.6,
'max': 21
},
...
]
You can iterate over the list and take every dict into a model.
from yourmodels import CurrentModel
...
for obj in json_response:
cm = CurrentModel()
cm.avg = obj['avg']
cm.max = obj['max']
cm.code = obj['code']
cm.save()
Or you could use a bulk method, but keep in mind that bulk_create does not trigger save method.