I am new in python. Now I am studying web scraping using request but I am stuck in get data table in json format on link below
http://hyd-app.rid.go.th/hydro3d.html
Anyone can help
Thanks
If you right click then inspect in Chrome or Firefox you can see under the network tab a XHR request is made to a different URL. You can access the URL directly like this (just update the "data" payload to what you want)
import requests
import json
url = "http://hyd-app.rid.go.th/webservice/getDailyWaterLevelListReport.ashx?option=2"
headers = {'Accept': 'application/json'}
data = {"DW[UtokID]": "3",
"DW[TimeCurrent]": "23/10/2561",
"_search": "false",
"nd": "1540287188732",
"rows": "1000",
"page": "1",
"sidx": "indexcount",
"sord": "asc"}
r = requests.post(url, data = data, headers = headers)
with open('data.json', 'w') as outfile:
json.dump(r.json(), outfile)
I have dumped it to a file as my terminal will not display the json with the encoding it uses.
Related
I am extracting JSON data from a API and trying to write on Azure container path. I am able to display data correctly in notebook, but when i write JSON most of the values are NULL. Any help on where i am going wrong?
headers = {
"accept" : "application/json",
"Content-Type": "application/json",
"Authorization": "Bearer " + str(token)
}
response_get= requests.get(getURL, headers=headers)
response_final=response_get.json()
print("Type:", type(response_final))
data = json_normalize(response_final)
df = spark.createDataFrame(data)
##df.coalesce(1).write.parquet(stagingpath,mode='overwrite')
df.coalesce(1).write.json(stagingpath,mode='overwrite')
I have reproduced in my environment and followed below process and got expected results as below and followed Microsoft-Document and SO-Thread:
import requests
response = requests.get('https://reqres.in/api/users?page=3')
rdd = spark.sparkContext.parallelize([response.text])
df = spark.read.json(rdd)
df.show()
dbutils.fs.mount( source = "wasbs://mycontainer#myblobstorageaccount.blob.core.windows.net", mount_point = "/mnt/mymountpoint", extra_configs = {"fs.azure.sas.mycontainer.myblobstorageaccount.blob.core.windows.net": "SAS"})
The run below script to write json:
df.coalesce(1).write.json( "/mnt/mymountpoint/vamo.json")
Output:
Click on folder Vammo.json:
Click on part-00xxx:
Then Click on View/Edit:
I'm working with a raspberry pi zero and Python to send and recieve sensor data with Azure IoT. I've already created an endpoint and message routing to the storage container. But when I check the JSON-Files in the container, I've got two problems:
The file include various general data which i don't need
My message body is in Base24-format
My message looks like this:
{"EnqueuedTimeUtc":"2021-06-25T13:03:25.7110000Z","Properties":{},"SystemProperties":{"connectionDeviceId":"RaspberryPi","connectionAuthMethod":"{"scope":"device","type":"sas","issuer":"iothub","acceptingIpFilterRule":null}","connectionDeviceGenerationId":"637555519600003402","enqueuedTime":"2021-06-25T13:03:25.7110000Z"},"Body":"eyJ0ZW1wZXJhdHVyZSI6IDI4Ljk1LCAicHJlc3N1cmUiOiA5ODEuMDg2Njk1NDU5MzMyNiwgImh1bWlkaXR5IjogNDYuMjE0ODE3NjkyOTEyODgsICJ0aW1lIjogIjIwMjEtMDYtMjUgMTQ6MDM6MjUuNjMxNzk1In0="}
The body included my sensor data in Base64-format. I've already read about contentType = application/JSON and contentEncoding = UTF-8 so that Azure can work with correct JSON files. But where do i apply these settings? When I apply it to the routing query, I get the following error:
Routing Query Error (The server didn't understand your query. Check your query syntax and try again)
I just want to get the body-message in correct JSON Format.
Thank you all for any kind of help! Since it's my first experience with this kind of stuff, I'm a little helpless.
Zero clue if this helps, but here is my code for sending data from Raspberry Pi Python to AWS - Parse Server using base64/JSON. The only reason I use base64 is to send pictures. You should only have to use JSON to send your other data.
import requests
import random, time
import math
import json
import Adafruit_DHT
import base64
from Adafruit_CCS811 import Adafruit_CCS811
from picamera import PiCamera
from time import sleep
DHT_SENSOR = Adafruit_DHT.DHT22
DHT_PIN =4
ccs = Adafruit_CCS811()
camera = PiCamera()
while True:
time.sleep(5)
camera.start_preview()
sleep(5)
camera.capture('/home/pi/Desktop/image.jpg')
camera.stop_preview()
with open('/home/pi/Desktop/image.jpg', 'rb') as binary_file:
binary_file_data = binary_file.read()
base64_encoded_data = base64.b64encode(binary_file_data)
base64_message = base64_encoded_data.decode('utf-8')
humidity, temperature = Adafruit_DHT.read_retry(DHT_SENSOR, DHT_PIN)
ccs.readData()
parseServer = {
"temp": temperature,
"humid": humidity,
"co2": ccs.geteCO2(),
"pic": base64_message
}
resultJSON = json.dumps(parseServer)
headers = {
'X-Parse-Application-Id': 'myappID',
'Content-Type': 'application/json',
}
data = resultJSON
response =
requests.put('http://1.11.111.1111/parse/classes/Gamefuck/TIuRnws3Ag',
headers=headers, data=data)
print(data)
If you're using the Python SDK for Azure IoT, sending the message as UTF-8 encoded JSON is as easy as setting two properties on your message object. There is a good example here
msg.content_encoding = "utf-8"
msg.content_type = "application/json"
Furthermore, you don't need to change anything in IoT Hub for this. This message setting is a prerequisite to be able to do message routing based on the body of the message.
I want send json data from odoo controller.For that i have created below controller
from odoo.http import Response
import json
#http.route('/api/json_get_request',auth='public',type='json',methods=["GET"],csrf=False)
def printjson(self,**kw):
headers={'content-type':'application/json'}
return Response(json.dumps({"test":"json string"}),headers=headers)
but accessing http://localhost:8089/api/json_get_request in postman gives me Invalid json data then I have check postman console in that response header -> Content-Type: "text/html" is shown.
Not understand after sending data in the json type why json data not recieved.
After accessing http://localhost:8089/api/json_get_request as http request on postman gives me correct json data.
Please give me suggestion
Thanks in advance
First thing method that handle 'json' request should return a dictionary directly:
#http.route('/api/json_get_request', auth='public', type='json', csrf=False)
def printjson(self, **kw):
return {'attribute': 'test'}
this will return a result like this:
{
"jsonrpc": "2.0",
"id": null,
"result": {
"attribute": "test"
}
}
You are having this error because you didn't send any json data with the request.
You could test your code using the requests module:
>>> import requests
>>> import json
>>> url = 'http://localhost:8069/api/json_get_request'
>>> data = {'params': {'test': 100}}
>>> headers={'content-type':'application/json'}
>>> requests.get(url, data=json.dumps(data), headers=headers)
<Response [404]>
It gives me this error because there is no database selected, and I have more than one database.
If you have only one database you see the correct result.
but when you test it from Postman extension I used RESTED
Because I'm all ready logged to Odoo the method was executed successfully , Just look for a better postman extension or search how to send json data with your current postman. You need to send some json data in the request.
I was successful in publishing (POST) a JSON file in Zapier and creating a Storage for it. However, I´d like to access the JSON in Zapier Storage using a Python code run locally. I am able to access the storage with Python3, see that is something written there, but I cannot access the JSON contents.
import urllib
import json
import codecs
reader = codecs.getreader("utf-8")
access_token = "password"
def GetStorage(page_id, access_token):
url = 'https://hooks.zapier.com/url/'
response = urllib.request.urlopen(url)
data = json.load(reader(response))
return data
a=GetStorage(url, access_token)
print(a)
All I get is:
{'attempt': '5a539a49-65eb-44f8-a30e-e171faf7a680',
'id': '1b38d21a-0150-46df-98c1-490a0d04b565',
'request_id': '5a539a49-65eb-44f8-a30e-e171faf7a680',
'status': 'success'}
When in fact I need:
{'Name':'value',
'Address': 'value'
}
Any ideas ?
David here, from the Zapier Platform team.
You're close! hooks.zapier.com is the url we use for incoming webhooks, so we always reply with a 200 and the response body you're seeing.
Instead, use store.zapier.com. You'll also want to make sure to include your secret. A full request URL will look like:
https://store.zapier.com/api/records?secret=test
which will return arbitrary json data:
{
"name": "david",
"job": "programmer"
}
The full docs are in json here: https://store.zapier.com/
I've run into a problem which I can't seem to solve while trying to use an API from within R. So I'm hoping you guys can give me a hint here what's wrong.
What I'm trying to do is make the Botometer API (https://market.mashape.com/OSoMe/botometer) accessible from within R. I know that there's a Python package for it, but since I've just learned R I thought it'd be fun to make it run with R. I've written a function to gather all the relevant information about an user with the rtweet package and parse it into a json file that should have a format accepted by the API. However, when trying to upload the json data, I get the following result from the API:
Response [https://osome-botometer.p.mashape.com/2/check_account]
Date: 2017-09-17 17:19
Status: 400
Content-Type: text/html; charset=UTF-8
Size: 192 B
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this ser...
When using the example payload provided by the API devs, however, I get the proper result:
Response [https://osome-botometer.p.mashape.com/2/check_account]
Date: 2017-09-17 15:59
Status: 200
Content-Type: application/json
Size: 298 B
{
"categories": {
"content": 0.39,
"friend": 0.54,
"network": 0.44,
"sentiment": 0.33,
"temporal": 0.56,
"user": 0.24
},
"scores": {
...
The most likely cause of the error right now seems to be that the Content-Type is not considered JSON for my request and as thus not accepted by the server (correct me if I'm wrong).
Here's the code I'm using to make the JSON file (utilizing jsonlite and rtweet):
Tweetson <- function(x) {
user.json <- lookup_users(x, token = twitter_token, parse = FALSE) %>% toJSON(null = "null", na = "null", pretty = TRUE)
timeline.json <- get_timeline(x, token = twitter_token, parse = FALSE, n = 200) %>% toJSON(null = "null", na = "null", pretty = TRUE)
mentions.json <- search_tweets(paste("#",x), token = twitter_token, parse = FALSE, n = 100) %>% toJSON(null = "null", na = "null", pretty = TRUE)
payload <- paste("{\n\"timeline\":", timeline.json, ",\"mentions\":", mentions.json, ",\"user\":", user.json, "\n}")
write(payload, "testload.json")
}
I then upload the file to the API, running into the aforementioned problem. (This will later be done in one function, since the issue seems to be in the making of the json file, however, I didn't include the rest here).
Checking with external json viewers or the isValidJSON() function it seems that "testload.json" is a valid json file, so how comes the API disagrees and denies my request? Am I oblivious to something obvious in the working of json files?
Oh, and one more thing I can't figure out (which probably has an obvious solution): how do I make R show me the API's whole answer without restricting it as seen above?
I'm fairly new to R and coding as a whole, so any help would be greatly appreciated!
Edit: This is the code I'm using to make the request to the API:
POST(botometerURL,
add_headers(
"X-Mashape-Key"=MashapeKey,
"Content-Type" = "application/json",
"Accept" = "application/json"),
body=upload_file("testload.json"),
encode = "json")
As mentioned, this works fine with the example payload but fails me with my self-generated ones.