DOWNVOTERS: leave a comment with some constructive critisism! I have no fkn clue what is wrong with this question.
How do I make a POST-request in python3?
I'm trying to fetch google plus ones for some url's the common but unofficial way through json-rpc.
I found this code but I can't get it to work:
import urllib2, json
data = '[{"method":"pos.plusones.get","id":"p","params":{"nolog":true,"id":"%s","source":"widget","userId":"#viewer","groupId":"#self"},"jsonrpc":"2.0","key":"p","apiVersion":"v1"}]' % "http://stackoverflow.com"
url = "https://clients6.google.com/rpc?key=AIzaSyCKSbrvQasunBoV16zDH9R33D88CeLr9gQ"
req = urllib2.Request(url, data, {'Content-Type': 'application/json'})
f = urllib2.urlopen(req)
response = f.read()
f.close()
result = json.loads(response)
print int(result[0]['result']['metadata']['globalCounts']['count'])
The docs aren't really that helpfull either. I can do the request without problems with other tools but not in python3. Any hints, please?
If you want to do something like making a POST or other HTTP request, may I suggest the excellently written Requests library.
Try something like this:
payload = {'key1': 'value1', 'key2': 'value2'}
r = requests.post("http://URL.org", params=payload)
if r.status_code == 200:
json = r.json()
else:
json = None
Related
So I've been beating my head against a wall for days now and have been diving down the google/SO rabbit hole in search of answers. I've been debating on how to phrase this question as the API that I am pulling from, may or may not contain some sensitive information that gets uncomfortably close to HIPPA laws for my liking. For that reason I will not be providing the direct link/auth for the my code. That being said I will be providing a made up JSON script to help with the explaining.
import requests
import json
import urllib3
r = requests.get('https://madeup.url.com/api/vi/information here', auth=('123456789', '1111111111222222222223333333333444444455555555'))
payload = {'query': 'firstName'}
response = requests.get(r, params=payload)
json_response = response.json()
print(json.dumps(json_response))
The JSON file that I'm trying to parse looks in part like this:
"{\"id\": 123456789, \"firstName\": \"NAME\", \"lastName\": \"NAME\", \"phone\": \"NUMBER\", \"email\": \"EMAIL#gmail.com\", \"date\": \"December 16, 2021\", \"time\": \"9:50am\", \"endTime\": \"10:00am\",.....
When I run the code I am getting a "urllib3.exceptions.LocationParseError: Failed to parse: <Response [200]>" traceback and I can not for the life of me figure out what is going on. urllib3 is installed and updated according to the console.
Any help would be much appreciated. TIA
That is not a JSON file. It is a string containing escaped characters. It needs to be unescaped before parsing can work.
youre passing r to requests.get() (line 9) , but r is a response to another requests.get() (line 5)... shouldn't you be passing params=payload in line 5? then getting de response from there, in one single request
import requests
import json
import urllib3
payload = {'query': 'firstName'}
response = requests.get('{YOUR_URL}', auth=('{USER}', '{PASS}'), params=payload)
json_response = response.json()
print(json.dumps(json_response))
That is not a JSON file. It is a string containing escaped characters. It needs to be unescaped before parsing can work.
Well now I'm even more confused. I'm trying to self teach myself python and clearly struggling. To get the "JSON" I posted I used the following code:
r = requests.get('URL', 'auth = ('user', 'pass'))
Data = r.json()
packages_str = json.dumps(Data[0])
with open('Data.json', 'w') as f:
json.dump(packages_str, f)
So basically I'm even more lost now...
Okay, update: Good news! kinda... so my code now reads as follows;
import requests
import json
import urllib3
payload = {
'query1'= 'firstName',
'query2'= 'lastName'
}
response = requests.get("url", auth= ("user","pass"), params=payload)
Data = response.json()
packages_str = json.dumps(Data, ensure_ascii=False, indent=2)
with open('Data.json), 'w') as f:
json.dump(packages_str,f)
f.write(packages_str)
And when I then open the JOSN file, the first line of is the entire API in a string but below that, is a properly formatted JSON file. Unfortunately its the entire API and not a parsed JSON file looking for the the information That I need...
Continuing down the google/youtube/SO rabbit hole and will update at a later date if i find a work around.
I would like to crawl several urls, while using the requests library in python. I am scrutinizing the GET requests as well as the response headers. However, when crawling and getting the data from different urls I am facing the problem, that I don't know all 'key:values', which are coming in. Thus writing those data to a valid csv file is not really possible, in my point of view. Therefore I want to write the data into a json file.
The problem is similar to the following thread from 2014, but not the same:
Get a header with Python and convert in JSON (requests - urllib2 - json)
import requests, json
urls = ['http://www.example.com/', 'http://github.com']
with open('test.json', 'w') as f:
for url in urls:
r = requests.get(url)
rh = r.headers
f.write(json.dumps(dict(rh), sort_keys=True, separators=(',', ':'), indent=4))
I expect a json file, with the headers for each URL. I get a Json file with those data, but my IDE (PyCHarm) is showing an Error, which states out that
JSON standard allows only one top-level value. I have read the documentation:https://docs.python.org/3/library/json.html#repeated-names-within-an-object; but did not get it. Any hint would be appreciated.
EDIT: The only thing which is missing in the outcome is another comma. But where do I enter it and what command do I need for this?
You need to add it to an array and then finally do the json dump to a file. This will work.
urls = ['http://www.example.com/', 'http://github.com']
headers = []
for url in urls:
r = requests.get(url)
header_dict = dict(r.headers)
header_dict['source_url'] = url
headers.append(header_dict)
with open('test.json', 'w', encoding='utf-8') as f:
json.dump(headers, f, sort_keys=True, separators=(',', ':'), indent=4)
You still can write it to a csv:
import pandas as pd
df = pd.DataFrame(headers)
df.to_csv('test.csv')
Is there any way to get the JSON Code from a HTML Website? If I use a code like those:
r = requests.get(url)
if r.status_code == 200:
r.json()
result = json.loads(r)
I will always have an error at HTML pages. What modules should I use for getting HTML pages to an Python-dictionary?
You only have one error in your code.
Once you did
r.json()
You didn't assign it to anything. To correct this problem just change your previous line with the line below and you should be good :).
r = r.json()
Not all webpages responds with JSON data. But you can use json.loads to print data in json string. You can also use r.contents or r.text to know the type of data coming from webpage. Most of the time it will be just HTML Content
import requests
import json
r = requests.get('http://www.google.com')
# you can use r.content to print the webpage data
print r.content
# json.loads(data) `json_loads` is to convert data into `json string`
print json.loads(r.content)
json.loads will go into ValueError if the data cannot be decoded into JSON Object
I recieve a POSTed JSON with mod_wsgi on Apache. I have to forward the JSON to some API (using POST), take API's response and respond back to where the initial POST came from.
Here goes the python code
import requests
import urllib.parse
def application(environ, start_response):
url = "http://texchange.nowtaxi.ru/api/secret_api_key/"
query = environ['QUERY_STRING']
if query == "get":
url += "tariff/list"
r = requests.get(url)
response_headers = [('Content-type', 'application/json')]
else:
url += "order/put"
input_len = int(environ.get('CONTENT_LENGTH', '0'))
data = environ['wsgi.input'].read(input_len)
decoded = data.decode('utf-8')
unquoted = urllib.parse.unquote(decoded)
print(decoded) # 'from%5Baddress%5D=%D0%'
print(unquoted) # 'from[address]=\xd0\xa0'
r = requests.post(url,data)
output_len = sum(len(line) for line in r.text)
response_headers = [('Content-type', 'application/json'),
('Content-Length', str(output_len))]
status = "200 OK"
start_response(status, response_headers)
return [r.text.encode('utf-8')]
The actual JSON starts "{"from":{"address":"Россия
I thought those \x's are called escaped symbols, so I tried ast.literal_eval and codecs.getdecoder("unicode_escape"), but it didn't help. I can't properly google the case, because I feel like I misunderstood wtf is happening here. Maybe I have to somehow change the $.post() call in the .js file that sends POST to the wsgi script?
UPD: my bro said that it's totally unclear what I need. I'll clarify. I need to get the string that represents the recieved JSON in it's initial form. With cyrillic letters, "s, {}s, etc. What I DO get after decoding recieved byte-sequence is 'from%5Baddress%5D=%D0%'. If I unquote it, it converts into 'from[address]=\xd0\xa0', but that's still not what I want
I'm trying to write a simple server frontend to a python3 application, using a restful JSON-based protocol. So far, bottle seems the best suited framework for the task (it supports python3, handles method dispatching in a nice way, and easily returns JSON.) The problem is parsing the JSON in the input request.
The documentation only mention request.fields and request.files, both I assume refer to multipart/form-data data. No mention of accessing the request data directly.
Peeking at the source code, I can see a request.body object of type BytesIO. json.load refuses to act on it directly, dying in the json lib with can't use a string pattern on a bytes-like object. The proper way to do it may be to first decode the bytes to unicode characters, according to whichever charset was specified in the Content-Type HTTP header. I don't know how to do that; I can see a StringIO class and assume it may hold a buffer of characters instead of bytes, but see no way of decoding a BytesIO to a StringIO, if this is even possible at all.
Of course, it may also be possible to read the BytesIO object into a bytestring, then decode it into a string before passing it to the JSON decoder, but if I understand correctly, that breaks the nice buffering behavior of the whole thing.
Or is there any better way to do it ?
It seems that io.TextIOWrapper from the standard library does the trick !
def parse(request):
encoding = ... #get encoding from headers
return json.load(TextIOWrapper(request.body, encoding=encoding))
Here's what I do to read in json on a RESTful service with Python3 and Bottle:
import bson.json_util as bson_json
#app.post('/location/API')
def post_json_example():
"""
param: _id, value
return: I usually return something like {"status": "successful", "message": "discription"}
"""
query_string = bottle.request.query.json
query_dict = bson_json.loads(query_string)
_id = query_dict['_id']
value = query_dict['value']
Then to Test
from python3 interpreter, import requests
s = request.Session()
r = s.post('http://youserver.com:8080/location/API?json
{"_id":"540a16663dafb492a0a7626c","value":"test"}')
use r.text to verify what was returned.
I wrote an helper to use the good idea of b0fh.
After 2 weeks on response.json analyzing, I connect to StackOver Flow and understand that we need a work around
Here is:
def json_app_rqt():
# about request
request.accept = 'application/json, text/plain; charset=utf-8'
def json_app_resp():
# about response
response.headers['Access-Control-Allow-Origin'] = _allow_origin
response.headers['Access-Control-Allow-Methods'] = _allow_methods
# response.headers['Access-Control-Allow-Headers'] = _allow_headers
response.headers['Content-Type'] = 'application/json; charset=utf-8'
def json_app():
json_app_rqt()
json_app_resp()
def get_json_request(rqt):
with TextIOWrapper(rqt.body, encoding = "UTF-8") as json_wrap:
json_text = ''.join(json_wrap.readlines())
json_data = json.loads(json_text)
return json_data
For the using, we cand do:
if __name__ == "__main__":
json_app()
#post("/train_control/:control")
def do_train_control(control):
json_app_resp()
data = get_json_request(request)
print(json.dumps(data))
return data
Thanks to all