I am very new in python. I have an API link, username, password to read the data. I have to read the data from the API request and then store it into a JSON file. Later I will read it in pandas or R dataframe to do more analysis. So I did:
First approach:
import requests
import json
import urllib, base64
url = 'https://abcXXXXXXXX.com/'
username = 'kebXXXXXXXXX'
password = 'XXXb8thXXXXpb8thXX'
The following code gives me a byte string which I am not able to export as JSON probably.
print(requests.get(url, auth=(username, password)).content)
so I decode it as:
print(requests.get(url, auth=(username, password)).content.decode("utf-8"))
which give me a list but the child text are in a European language and it does not read the proper format. In bot file the byte and decode file the text comes like that format Dobr\u00fd den, where \u00fd should be a Latin word.
The Second Approach:
I try to use urllib library.
URL = 'https://abcXXXXXXXX.com/'
username = b'kebXXXXXXXXX'
password = b'XXXb8thXXXXpb8thXX'
base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
but the base64 string gives me an error: expected bytes-like object, not str. I tried different sources could not find any solution. If someone can tell me What will be the best approach to do that. My final goal is to read the JSON data, store it in a file and read by pandas or data frame and do some text analysis. Python or R solution will be ok.
The data looks like(sample):
[
"messages": [
{
"type": "agent",
"name": "",
"from": null,
"content": "Hezký den, potřebujete poradit s výběrem eliptického trenažeru?",
"sentAt": 1509526170320,
"triggeredBy": "Eliptický trenažér",
"email": null
}
]
]
If you are not aware, requests result objects have a .json() method. This should automatically decode the content for you, assuming the results have returned in a json format. Some APIs require an extra header. The line that you wrote,
print(requests.get(url, auth=(username, password)).content)
would change to something like
print(requests.get(url, auth=(username, password)).json())
if headers are not necessary or
print(requests.get(url, auth=(username, password), headers={'Accept': 'application/json'}).json())
if they are.
Related
So I've been beating my head against a wall for days now and have been diving down the google/SO rabbit hole in search of answers. I've been debating on how to phrase this question as the API that I am pulling from, may or may not contain some sensitive information that gets uncomfortably close to HIPPA laws for my liking. For that reason I will not be providing the direct link/auth for the my code. That being said I will be providing a made up JSON script to help with the explaining.
import requests
import json
import urllib3
r = requests.get('https://madeup.url.com/api/vi/information here', auth=('123456789', '1111111111222222222223333333333444444455555555'))
payload = {'query': 'firstName'}
response = requests.get(r, params=payload)
json_response = response.json()
print(json.dumps(json_response))
The JSON file that I'm trying to parse looks in part like this:
"{\"id\": 123456789, \"firstName\": \"NAME\", \"lastName\": \"NAME\", \"phone\": \"NUMBER\", \"email\": \"EMAIL#gmail.com\", \"date\": \"December 16, 2021\", \"time\": \"9:50am\", \"endTime\": \"10:00am\",.....
When I run the code I am getting a "urllib3.exceptions.LocationParseError: Failed to parse: <Response [200]>" traceback and I can not for the life of me figure out what is going on. urllib3 is installed and updated according to the console.
Any help would be much appreciated. TIA
That is not a JSON file. It is a string containing escaped characters. It needs to be unescaped before parsing can work.
youre passing r to requests.get() (line 9) , but r is a response to another requests.get() (line 5)... shouldn't you be passing params=payload in line 5? then getting de response from there, in one single request
import requests
import json
import urllib3
payload = {'query': 'firstName'}
response = requests.get('{YOUR_URL}', auth=('{USER}', '{PASS}'), params=payload)
json_response = response.json()
print(json.dumps(json_response))
That is not a JSON file. It is a string containing escaped characters. It needs to be unescaped before parsing can work.
Well now I'm even more confused. I'm trying to self teach myself python and clearly struggling. To get the "JSON" I posted I used the following code:
r = requests.get('URL', 'auth = ('user', 'pass'))
Data = r.json()
packages_str = json.dumps(Data[0])
with open('Data.json', 'w') as f:
json.dump(packages_str, f)
So basically I'm even more lost now...
Okay, update: Good news! kinda... so my code now reads as follows;
import requests
import json
import urllib3
payload = {
'query1'= 'firstName',
'query2'= 'lastName'
}
response = requests.get("url", auth= ("user","pass"), params=payload)
Data = response.json()
packages_str = json.dumps(Data, ensure_ascii=False, indent=2)
with open('Data.json), 'w') as f:
json.dump(packages_str,f)
f.write(packages_str)
And when I then open the JOSN file, the first line of is the entire API in a string but below that, is a properly formatted JSON file. Unfortunately its the entire API and not a parsed JSON file looking for the the information That I need...
Continuing down the google/youtube/SO rabbit hole and will update at a later date if i find a work around.
I need to create some JSON files for exporting data from a Django system to Google Big Query.
The problem is that Google BQ imposes some characteristics in the JSON file, for example, that each object must be in a different line.
json.dumps writes a stringified version of the JSON, so it is not useful for me.
Django serializes writes better JSON, but it put all in one line. All the information I found about pretty-printing is about json.dumps, which I cannot use.
I will like to know if anyone knows a way to create a JSON file in the format required by Big Query.
Example:
JSONSerializer = serializers.get_serializer("json")
json_serializer = JSONSerializer()
data_objects = DataObject.objects.all()
with open("dataobjects.json", "w") as out:
json_serializer.serialize(data_objects, stream=out)
json.dumps is OK. You have to use indent like this.
import json
myjson = '{"latitude":48.858093,"longitude":2.294694}'
mydata = json.loads(myjson)
print(json.dumps(mydata, indent=4, sort_keys=True))
Output:
{
"latitude": 48.858093,
"longitude": 2.294694
}
I'm having some difficulties with extracting a single value from a JSON dump. I'm trying to extract the single value of a stock from JSON output generated by using the GoogleFinance package, but I keep getting this error message:
expected string or buffer.
I've tried to find a solution on the forum but nothing seems to be working.
One thing I've tried is loading the JSON in to a string by using json.loads, but I keep running in to the same wall.
from googlefinance import getQuotes
import json
import sys
Apple = json.dumps(getQuotes('AAP'), indent=2) #gets the quote from Apple stock. I know I should use the json.loads but that doesn't seem to be working for the getQuotes.
#JSON output
[
{
"Index": "NYSE",
"LastTradeWithCurrency": "137.24",
"LastTradeDateTime": "2016-11-07T13:09:43Z",
"LastTradePrice": "137.24",
"LastTradeTime": "1:09PM EST",
"LastTradeDateTimeLong": "Nov 7, 1:09PM EST",
"StockSymbol": "AAP",
"ID": "668575"
}
]
#Trying to solve the issue by loading the json to a string
resp = json.loads(Apple)
#print the resp
print (resp)
#extract an element in the response
print (resp["LastTradeWithCurrency"])
change resp = json.loads(Apple) to resp = json.loads(Apple)[0]
I am trying to get Python3 to PUT json info in string format to an API. And I want to do it without
import requests
Thus far I am stuck with this code:
import urllib.request
import urllib.parse
import json
url = "http://www.example.com"
DATA = json.dumps({'grades': {"math": "92", "chem": "39"}})
req = urllib.request.Request(url, data=DATA, method='PUT')
response = urllib.request.urlopen(req)
Naturally this raises the error:
raise TypeError(msg)
TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str.
To get rid of the error I can do:
DATA= str.encode(DATA)
But this turns my data into bytes format, instead of string that I want to put up. Is there a way to PUT up strings without importing "requests"?(importing anything that comes with python install is OK). Or can I PUT up a *.json file?
Essentially I am trying to do the opposite of this.
I try to test a view, I receive a json request from the IPad, the format is:
req = {"custom_decks": [
{
"deck_name": "deck_test",
"updates_last_applied": "1406217357",
"created_date": 1406217380,
"slide_section_ids": [
1
],
"deck_id": 1
}
],
"custom_decks_to_delete": []
}
I checked this in jsonlint and it passed.
I post the req via:
response = self.client.post('/library/api/6.0/user/'+ uuid +
'/store_custom_dec/',content_type='application/json', data=req)
The view return "creation_success": false
The problem is the post method in view doesn't find the key custom_decks.
QueryDict: {u'{"custom_decks": [{"deck_id": 1, "slide_section_ids": [1],
"created_date":1406217380, "deck_name": "deck_test"}],
"custom_decks_to_delete": []}': [u'']}>
The problem is the post method in view doesn't find the key custom_decks.
Because it is converting my dict to QueryDict with one key.
I appreciate all helps.
Thanks
You're posting JSON, which is not the same as form-encoded data. You need to get the value of request.body and deserialize it:
data = json.loads(request.body)
custom_decks = data['custom_decks']
As I was having problems with getting JSON data from HttpRequest directly with the code of the other answer:
data = json.loads(request.body)
custom_decks = data['custom_decks']
error:
the JSON object must be str, not 'bytes'
Here is an update of the other answer for Python version >3:
json_str=((request.body).decode('utf-8'))
json_obj=json.loads(json_str)
Regarding decode('utf-8'), as mention in:
RFC 4627:
"JSON text shall be encoded in Unicode. The default encoding is
UTF-8."
I attached the Python link referred to this specific problem for version >3.
http://bugs.python.org/issue10976
python 3.6 and django 2.0 :
post_json = json.loads(request.body)
custom_decks = post_json.get("custom_decks")
json.loads(s, *, encoding=None,...)
Changed in version 3.6: s can now be of type bytes or bytearray. The input encoding should be UTF-8, UTF-16 or UTF-32.
From python 3.6 NO need request.body.decode('utf-8') .
Since HttpRequest has a read() method loading JSON from request is actually as simple as:
def post(self, request, *args, **kwargs):
import json
data = json.load(request)
return JsonResponse(data=data)
If you put this up as a view, you can test it and it'll echo any JSON you send back to you.