I just began exploring APIs. This is my code so far. For locu API this works but for Zomato they use curl header request which I don't know how to use. Could someone guide or show me how?
import json
import urllib2
Key = 'Zomato_key'
url = 'https://developers.zomato.com/api/v2.1/categories'
json_obj = urllib2.urlopen(url)
data = json.load(json_obj)
print data
By looking at the Zomato API docs, it seems that the parameter user-key has to be set in the header.
The following works:
import json
import urllib2
Key = '<YOUR_ZOMATO_API_KEY>'
url = "https://developers.zomato.com/api/v2.1/categories"
request = urllib2.Request(url, headers={"user-key" : Key})
json_obj = urllib2.urlopen(request)
data = json.load(json_obj)
print data
If you want a more elegant way to query APIs, have a look at requests module (you can install using pip install requests).
I suggest you the following:
import json
import requests
Key = <YOUR_ZOMATO_API_KEY>'
url = "https://developers.zomato.com/api/v2.1/categories"
if __name__ == '__main__':
r = requests.get(url, headers={'user-key': Key})
if r.ok:
data = r.json()
print data
NB: I suggest you remove your Key from StackOverflow if you care about keeping it to yourself.
this didn't work for me can u suggest some other method for me.
-->the code when tried to compile is taking long time and returning an traceback error in request method that is in built
but curl command is working
curl -X GET --header "Accept: application/json" --header "user-key: c5062d18e16b9bb9d857391bb32bb52f" "https://developers.zomato.com/api/v2.1/categories"
Related
I can query all occurances of certain base url within a given common crawl index, saving them all to a file and get a specific article (test_article_num) using the code below. However, I have not come across a way to extract the raw html for that article from the specific crawl-data ('filename' in output), even though I know the offset and length of the data I want. I feel like there should be a way to do this in python similar to this, maybe using requests and warcio (perhaps something akin to this), but I'm not sure. Any help is greatly appreicated.
EDIT:
I found exactly what I needed here.
import requests
import pathlib
import json
news_website_base = 'hobbsnews.com'
URL = "https://index.commoncrawl.org/CC-MAIN-2022-05-index?url="+news_website_base+"/*&output=json"
website_output = requests.get(URL)
pathlib.Path('data.json').write_bytes(website_output.content)
news_articles = []
test_article_num=300
for line in open('data.json', 'r'):
news_articles.append(json.loads(line))
print(news_articles[test_article_num])
news_URL=news_articles[test_article_num]['url']
news_warc_file=news_articles[test_article_num]['filename']
news_offset=news_articles[test_article_num]['offset']
news_length=news_articles[test_article_num]['length']
Code output:
{'urlkey': 'com,hobbsnews)/2020/03/22/no-new-positive-covid-19-tests-in-lea-in-last-24-hours/{{%20data.link', 'timestamp': '20220122015439', 'url': 'https://www.hobbsnews.com/2020/03/22/no-new-positive-covid-19-tests-in-lea-in-last-24-hours/%7B%7B%20data.link', 'mime': 'text/html', 'mime-detected': 'text/html', 'status': '404', 'digest': 'GY2UDG4G3V3S5TXDL3H7HE6VCSRBD3XR', 'length': '40062', 'offset': '21016412', 'filename': 'crawl-data/CC-MAIN-2022-05/segments/1642320303729.69/crawldiagnostics/CC-MAIN-20220122012907-20220122042907-00614.warc.gz'}
https://www.hobbsnews.com/2020/03/22/no-new-positive-covid-19-tests-in-lea-in-last-24-hours/%7B%7B%20data.link
crawl-data/CC-MAIN-2022-05/segments/1642320300343.4/crawldiagnostics/CC-MAIN-20220117061125-20220117091125-00631.warc.gz
21016412
40062
With the WARC URL, and WARC record offset and length it's simply:
download the range from offset until offset+length-1
pass the downloaded bytes to a WARC parser
Using curl and warcio CLI:
curl -s -r250975924-$((250975924+6922-1)) \
https://data.commoncrawl.org/crawl-data/CC-MAIN-2021-10/segments/1614178365186.46/warc/CC-MAIN-20210303012222-20210303042222-00595.warc.gz \
>warc_temp.warc.gz
warcio extract --payload warc_temp.warc.gz 0
Or with Python requests and warcio (cf. here):
import io
import requests
import warcio
warc_filename = 'crawl-data/CC-MAIN-2021-10/segments/1614178365186.46/warc/CC-MAIN-20210303012222-20210303042222-00595.warc.gz'
warc_record_offset = 250975924
warc_record_length = 6922
response = requests.get(f'https://data.commoncrawl.org/{warc_filename}',
headers={'Range': f'bytes={warc_record_offset}-{warc_record_offset + warc_record_length - 1}'})
with io.BytesIO(response.content) as stream:
for record in warcio.ArchiveIterator(stream):
html = record.content_stream().read()
i have a server with POST e REST services. Using micropython i need to perform POST request.
GET queries arrive at the REST services and respond correctly using the following format:
s.send(b"GET /sensordata/premises HTTP/1.1\r\nHost:XX.XXX.XXX.XXX\r\n" + "Accept: application/json\r\n\r\n")
but for the POST looks like the request arrived to server, but the body is empty. For some reason the JSON body of the request is not interpreted in a correct way.
from network import WLAN
import socket
import machine
import time
import struct
import json
import pycom
wlan = WLAN(mode=WLAN.STA)
wlan.connect("*****", auth=(WLAN.WPA2, "*****"), timeout=5000)
while not wlan.isconnected():
machine.idle()
print("Connected to WiFi\n")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
url = 'XX.XXX.XXX.XXX'
sockaddr = socket.getaddrinfo(url, 80) [0][-1]
s.connect(sockaddr)
print('socket connected')
httpreq = b'POST /sensordata/insertrecords HTTP/1.1 \r\n Host:XX.XXX.XXX.XXX \r\n' + 'Accept: application/json \r\n' + 'Content-Type: application/json\r\n' + ' {\"node_id\":\"1\",\"value\":[{\"measure_time_stamp\":\"2020-10-06T09:25:43\",\"temp\":\"14\",\"humidity\":\"75\",\"ph1\":11,\"ph2\":12,\"ph3\":13}]}\r\n\r\n'
s.send(httpreq)
time.sleep(1)
rec_bytes = s.recv(4096)
print("RESPONSE = " + str(rec_bytes))
print('end')
s.close()
Are you aware of the urequests library? It's an HTTP client library inspired by the popular Python requests. A POST request is simply:
import urequests
r = urequests.post('https://your/endpoint')
I haven't found the docs for that use case. How can I get the request body, ensure it's a valid JSON (any valid JSON, including numbers, string, booleans, and nulls, not only objects and arrays) and get the actual JSON.
Using Pydantic forces the JSON to have a specific structure.
You can find nearly everything inside the Request object
You are able to get request body with request.json(), which will give you the parsed JSON as dictionary.
from fastapi import Request, FastAPI
#app.post("/dummypath")
async def get_body(request: Request):
return await request.json()
If you want access the body as string, you can use request.body()
The accepted answer is valid as well, but FastAPI provides a built-in way to do that - check the Singular values in body section in docs.
A parameter with the default Body gets all the payload that doesn't match passed Pydantic-typed parameters (the whole payload in our case) and converts it to the dict. In case of invalid JSON, a standard validation error would be produced.
from fastapi import Body, FastAPI
app = FastAPI()
#app.post('/test')
async def update_item(
payload: dict = Body(...)
):
return payload
UPD: Note on ... (Ellipsis) - it allows marking a value as required. Read more in the Required with Ellipsis docs section
If you are confident that the incoming data is "a valid JSON", you can create a simple type annotation structure to receive the arbitrary JSON data.
from fastapi import FastAPI
from typing import Any, Dict, AnyStr, List, Union
app = FastAPI()
JSONObject = Dict[AnyStr, Any]
JSONArray = List[Any]
JSONStructure = Union[JSONArray, JSONObject]
#app.post("/")
async def root(arbitrary_json: JSONStructure = None):
return {"received_data": arbitrary_json}
Examples
1. JSON object
curl -X POST "http://0.0.0.0:6022/" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"test_key\":\"test_val\"}"
Response:
{
"received_data": {
"test_key": "test_val"
}
}
2. JSON array
curl -X POST "http://0.0.0.0:6022/" -H "accept: application/json" -H "Content-Type: application/json" -d "[\"foo\",\"bar\"]"
Response:
{
"received_data": [
"foo",
"bar"
]
}
If you are not sure about the content type of the incoming data, better to parse the request body.
It can be done as,
from fastapi import FastAPI, Request
app = FastAPI()
#app.post("/")
async def root(request: Request):
return {"received_request_body": await request.body()}
The advantage of this method is that the body will contain any kind of data, JSON, form-data, multipart-form-data, etc.
from fastapi import Request
async def synonyms__select(request: Request):
return await request.json()
will return a JSON object.
This is an example to print the content of a Request, it will print the json body (if it is json parsable) otherwise just print the raw bytes of the body.
async def print_request(request):
print(f'request header : {dict(request.headers.items())}' )
print(f'request query params : {dict(request.query_params.items())}')
try :
print(f'request json : {await request.json()}')
except Exception as err:
# could not parse json
print(f'request body : {await request.body()}')
#app.post("/printREQUEST")
async def create_file(request: Request):
try:
await print_request(request)
return {"status": "OK"}
except Exception as err:
logging.error(f'could not print REQUEST: {err}')
return {"status": "ERR"}
FastAPI has a JSON encoder.
There are some cases where you might need to convert a data type (like
a Pydantic model) to something compatible with JSON
from fastapi import FastAPI
from fastapi.encoders import jsonable_encoder
from pydantic import BaseModel
import simplejson as json
class SubmitGeneral(BaseModel):
controllerIPaddress: str
readerIPaddress: str
ntpServer: str
#app.post("/submitGeneral")
async def submitGeneral(data: SubmitGeneral):
data = jsonable_encoder(data)
#data = json.loads(data.json()) # same as above line
print(f"data = {json.dumps(data)}")
# you have to access the properties with brackets, not by dot notation
query = f"update LocalPLC set ControllerIpAddress = '{data['controllerIPaddress']}', ReaderIPAddress = '{data['readerIPaddress']}'"
return {"status": "OK"}
For those of you using BaseModel and want to have a JSON field, you can import Json from pydantic
from fastapi import FastAPI
from pydantic import BaseModel, Json, Field
app = FastAPI()
class MockEndpoint(BaseModel):
endpoint: str = Field(description="API endpoint to mock")
response: Json = Field(description="Example response of the endpoint")
#app.get("/")
async def root():
return {"message": "Hello World"}
#app.post("/mock")
async def mock_request(mock_endpoint: MockEndpoint):
return mock_endpoint
I have the following Python code in AWS Lambda to verify if an event received is indeed from Slack:
import hmac
import json
def verifySignature(header,body):
h = hmac.new(key=os.getenv('sign_secret').encode(), \
msg=f'v0:{header.get("X-Slack-Request-Timestamp")}:{body}'.encode(), \
digestmod="sha256")
result = hmac.compare_digest('v0='+h.hexdigest(),header.get('X-Slack-Signature'))
print('v0='+h.hexdigest(),header.get('X-Slack-Signature'))
return result
def lambda_handler(event, context):
body = json.loads(event.get('body'))
if verifySignature(event.get('headers'),body):
do_something()
Slack's authentication protocol is outlined here. However, I keep getting mismatching signatures (result == False). Does anyone know why?
There is a high chance the issue is coming from the encoding / decoding. There is pip package to verify the slack signature.
But the verification code is simple:
import hashlib
import hmac
def verify_slack_signature(slack_post_request, slack_signing_secret):
slack_signing_secret = bytes(slack_signing_secret, 'utf-8')
slack_signature = slack_post_request['headers']['X-Slack-Signature']
slack_request_timestamp = slack_post_request['headers']['X-Slack-Request-Timestamp']
request_body = slack_post_request["body"]
basestring = f"v0:{slack_request_timestamp}:{request_body}".encode('utf-8')
my_signature = 'v0=' + hmac.new(slack_signing_secret, basestring, hashlib.sha256).hexdigest()
return hmac.compare_digest(my_signature, slack_signature)
I am a newbie to python and am trying to create a script to login to crucible and use the token to pass to other services.
1) I am able to make a xml request and get a response but as soon as I pass the headers to my conn.request it says HTTP Error 415, unsupported Media Type.
I have done quiet a bit of research on this topic and found out that the rest API might not be supporting the json reques, but Crucible says that there API supports json so seems to be some other issue,
2) when trying to pass the args generated using feauth the auth token is not getting used , for now I have appended it to url and it works.
Please help me with the same , below is my script
import httplib
import urllib
import json
from xml.etree.ElementTree import XML
import xml.dom.minidom
conn = httplib.HTTPSConnection("fisheye")
args=urllib.urlencode({'userName':'UNAME', 'password':'PWD'})
headers={'content-type':'application/json', 'accept':'application/json'}
#headers={'Authorization' : 'Basic %s' % base64.b64encode("username:password")}
r1 = conn.request("post", "/rest-service/auth-v1/login", args)
#status = r1[u'headers']['status']
#conn.connect()
r2 = conn.getresponse()
print r1,r2.status,r2.reason,r2
r3=r2.read()
print(r3)
r4=str(r3)
print r4
data = XML(r4).find("token").text
print data
# data1=urllib.quote_plus(data, safe=":")
# print data1
args=urllib.urlencode({'FEAUTH':data}).replace("%3A", ":")
print "args is", args
#args={}
req = conn.request("get","/rest-service/reviews-v1")
r3 = conn.getresponse()
status = r3.status
print "the url is"#, r3.getheader('Location')
url=r3.getheader('location', '')
print url
url1=r3.msg#.dict['location']
print url1
#print req.url
#print req.get_method()
print dir(req) # list lots of other stuff in Request
print "after sending open review request"
print r3
print req,r3.status,r3.reason,r3
r4=r3.read()
print(r4)
r5=str(r4)
print r5
# json_ob=json.loads(r3.read())
# print json_ob
I was able to resolve the issue by
1) removing the Content-Type from the headers and changed the accept to Accept(sentence cased).
2) The login request was a get request and hence it supports data transfer by URL append, it is only for post request that we can pass an argument.
In the header of the request, try to specify the media type:
headers = { 'Content-Type' : 'application/json' }
req = urllib2.Request(url, headers=headers)