I have a URL on my Raspberry Pi device server. How can I parse it?
I have used the urllib2 module to get the contents of the URL from server.
I want to use JSON for parsing the url.
so if i understad properly , you have a json file with a URL that you want to pass to your python module
used this a while back
import json
with open('yourjasonfilewithurl.jason') as json_data_file:
data = json.load(json_data_file)
The use the data function like this, the below is dependant on the number of elements you have set up in your json config file
myurl = (data["details"][0]["url"])
hope this helps
Related
I am trying to access a json object which is stored as a zipped gz on an html website. I would like to do this directly with urllib if possible.
This is what I have tried:
import urllib
import json
#get the zip file
test = urllib.request.Request('http://files.tmdb.org/p/exports/movie_ids_01_27_2021.json.gz')
#unzip and read
with gzip.open(test, 'rt', encoding='UTF-8') as zipfile:
my_object = json.loads(zipfile)
but this fails with:
TypeError: filename must be a str or bytes object, or a file
Is it possible to read the json directly like this (e.g. I don't want to download locally).
Thank you.
Use requests library. pip install requests if you don't have it.
Then use the following code:
import requests
r = requests.get('http://files.tmdb.org/p/exports/movie_ids_01_27_2021.json.gz')
print(r.content)
r.content will be the binary content of the gzip file, but it will consume 11352985 bytes of memory (10.8 MB) because the data need to be kept somewhere.
then you can use
gzip.decompress(r.content)
to decompress the gzip binary and get the data. that will consume much bigger memory after decompression.
I have a json file that I provide via flask, and I'd like to parse it from a requests object, but I can't get the full data back. I get a broken json file. What I get is about one third of the way through the file to the end.
I get different answers whether I use r.json() or json.loads(r.text)from the simple get request:
r = requests.get("url/api")
If I stream it manually, I can see the start of the file
r= requests.get("url/api", stream=True)
r.raw.read(1000).decode('utf-8')
And I successfully save down the file as here how to parse big requests object, but the problem persists when I try to then upload the file using json.load(local_filename)
The shape of my json file is:
[{"key1":"value1_1", "key2":"value2_1"},{"key1":"value1_2", "key2":"value2_2"}.......]
This is how I provide the json from flask
#app.route('/api', methods=['GET'])
def provide_json():
with open('record_file.json') as f:
record_file = json.load(f)
return json.dumps(record_file)
I've checked that my json file is valid, and I'm also assuming that my flask code with throw an exception is I wasn't delivering a valid json file because I've used json.dumps. Is that right? I've also tried wrapping the json file in flask with return json.dumps({"data_file":record_file}) but it doesn't change the problem.
You are returning list which is not a valid response for flask. Try this
from flask import jsonify
#app.route('/api', methods=['GET'])
def provide_json():
with open('record_file.json') as f:
record_file = json.load(f)
return jsonify({"data": record_file})
I am trying to send to my flask app json data and having it return a CSV file. My ajax request is sending JSON data to the view via POST request and then the view is supposed to return back a csv file. However, it fails to return the csv file in the browser as a download. I'm not sure how to make it work or if its even possible. Thanks!
// AJAX - Send data over to python and return csv
$("#export").click(
function(){
$.ajax({
url: "/dbCSV",
type: "POST",
contentType: 'application/json;charset=UTF-8',
dataType:"json",
data: JSON.stringify(datam)
});
event.preventDefault();
}
);
#analyzers.route("/dbCSV", methods=["GET","POST"])
def dbCSV():
if request.method=="POST":
data = pd.DataFrame(request.get_json())
resp = make_response(data.to_csv())
resp.headers["Content-Disposition"] = "attachment; filename=export.csv"
resp.headers["Content-Type"] = "text/csv"
return resp
return jsonify({"msg":"Could not generate CSV File"})
I'd recommend using send_file(...) with a BytesIO (file stream) object:
from io import BytesIO
from flask import send_file
...
response_stream = BytesIO(data.to_csv().encode())
return send_file(
response_stream,
mimetype="text/csv",
attachment_filename="export.csv",
)
Keep in mind that you will not be able to open the download prompt when sending a POST request using AJAX. Instead, you will simply receive the file as an AJAX response. To solve this issue, you will have to take a look at this question:
download file using an ajax request
Maybe your code was already working and this was your problem – I can not tell from looking at it.
I finally figure it out. Basically I can store the user input using the session object available from the flask library. This allows different functions to access this rather then having to worry about creating global variables or passing them around via functions or objects.
Note 1- If the amount of user data that has to be saved is extensive then using Redis or some other type of in memory data storage would be a better choice in this case.
Save the csv file in static path and then use the that static path csv URL to get csv file in download form from browser.
It's not under the supported libraries here:
https://developers.google.com/api-client-library/python/reference/supported_apis
Is it just not available with Python? If not, what language is it available for?
Andre's answer points you at a correct place to reference the API. Since your question was python specific, allow me to show you a basic approach to building your submitted search URL in python. This example will get you all the way to search content in just a few minutes after you sign up for Google's free API key.
ACCESS_TOKEN = <Get one of these following the directions on the places page>
import urllib
def build_URL(search_text='',types_text=''):
base_url = 'https://maps.googleapis.com/maps/api/place/textsearch/json' # Can change json to xml to change output type
key_string = '?key='+ACCESS_TOKEN # First think after the base_url starts with ? instead of &
query_string = '&query='+urllib.quote(search_text)
sensor_string = '&sensor=false' # Presumably you are not getting location from device GPS
type_string = ''
if types_text!='':
type_string = '&types='+urllib.quote(types_text) # More on types: https://developers.google.com/places/documentation/supported_types
url = base_url+key_string+query_string+sensor_string+type_string
return url
print(build_URL(search_text='Your search string here'))
This code will build and print a URL searching for whatever you put in the last line replacing "Your search string here". You need to build one of those URLs for each search. In this case I've printed it so you can copy and paste it into your browser address bar, which will give you a return (in the browser) of a JSON text object the same as you will get when your program submits that URL. I recommend using the python requests library to get that within your program and you can do that simply by taking the returned URL and doing this:
response = requests.get(url)
Next up you need to parse the returned response JSON, which you can do by converting it with the json library (look for json.loads for example). After running that response through json.loads you will have a nice python dictionary with all your results. You can also paste that return (e.g. from the browser or a saved file) into an online JSON viewer to understand the structure while you write code to access the dictionary that comes out of json.loads.
Please feel free to post more questions if part of this isn't clear.
Somebody has written a wrapper for the API: https://github.com/slimkrazy/python-google-places
Basically it's just HTTP with JSON responses. It's easier to access through JavaScript but it's just as easy to use urllib and the json library to connect to the API.
Ezekiel's answer worked great for me and all of the credit goes to him. I had to change his code in order for it to work with python3. Below is the code I used:
def build_URL(search_text='',types_text=''):
base_url = 'https://maps.googleapis.com/maps/api/place/textsearch/json'
key_string = '?key=' + ACCESS_TOKEN
query_string = '&query=' + urllib.parse.quote(search_text)
type_string = ''
if types_text != '':
type_string = '&types='+urllib.parse.quote(types_text)
url = base_url+key_string+query_string+type_string
return url
The changes were urllib.quote was changed to urllib.parse.quote and sensor was removed because google is deprecating it.
I'm trying to make an app where I will gather information from a json api http://pool-x.eu/api, and print information easly bo choosing parameter.
What is the easiest way to print each of the informations?
Was thinking something in the way of making the information a string, and then request each of the parameters that way, but I don't know if that's the way to do it.
here's a sample code to decode the json data i just happen to make a json text file out if the link you gave and decode it hope it helps
local json = require "json"
local txt
local path = system.pathForFile( "json.txt", system.ResourceDirectory )
local file = io.open( path, "r" )
for line in file:lines() do
txt = line
end
print(txt)
local t = json.decode( txt )
print(t["pool_name"])
print(t["hashrate"])
print(t["workers"])
print(t["share_this_round"])
print(t["last_block"])
print(t["network_hashrate"])