I am sending a Post request with a json body to a server but can not extract the json file when it arrives. I have does exhaustive searches but to no avail. I have provided both client and server scripts to illustrate what is happening.
All I need is to extract the json portion at the end of the received string so I can analyze the request and return the appropriate data.
I'm sure it's simple but I can't seem to find the answer. Any direction would be appreciated
***
CLIENT: script to test Server
import json
import requests
def info_send():
url = 'http:1234abcd.ngrok.io'
payload = {
'command': '["command", "status", "off", None]',
'userID': 'userID string',
'status': 'current status',
}
requests.post(url, data=json.dumps(payload))
info_send()
***
SERVER: receives json POST request
HOST, PORT = '', 5000
listen_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listen_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
listen_socket.bind((HOST, PORT))
listen_socket.listen(1)
print('Listening on port %s' % PORT)
while True:
client_connection, client_address = listen_socket.accept()
request = client_connection.recv(1024).decode('utf-8')
print(request)
***
This is what is printed at the server
POST / HTTP/1.1
Host: 1234abcd.ngrok.io
User-Agent: python-requests/2.18.4
Accept-Encoding: gzip, deflate
Accept: /
Content-Length: 112
X-Forwarded-For: 112.162.214.265
{"command": "[\"command\", \"status\", \"off\", None]", "userID": "userID string", "deviceID": "current status"}
Related
I'm trying to get some JSON data from this API - https://ped.uspto.gov/api/queries
This cURL request works fine and returns what is expected:
curl -X POST "https://ped.uspto.gov/api/queries" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"searchText\":\"*:*\", \"fq\":[ \"totalPtoDays:[1 TO 99999]\", \"appFilingDate:[2005-01-01T00:00:00Z TO 2005-12-31T23:59:59Z]\" ], \"fl\":\"*\", \"mm\":\"100%\", \"df\":\"patentTitle\", \"facet\":\"true\", \"sort\":\"applId asc\", \"start\":\"0\"}"
I have this python script to do the same thing:
from requests.structures import CaseInsensitiveDict
import json
url = "https://ped.uspto.gov/api/queries"
headers = CaseInsensitiveDict()
headers["accept"] = "application/json"
headers["Content-Type"] = "application/json"
data = json.dumps({
"searchText":"*:*",
"fq":[
"totalPtoDays:[1 TO 99999]",
"appFilingDate:[2005-01-01T00:00:00Z TO 2005-12-31T23:59:59Z]"
],
"fl":"*",
"mm":"100%",
"df":"patentTitle",
"facet":"true",
"sort":"applId asc",
"start":"0"
})
resp = requests.post(url, headers=headers, data=data)
print(resp.status_code)
but it returns a 403 error code and the following response header:
"Date":"Mon, 24 Oct 2022 16:13:58 GMT",
"Content-Type":"text/html",
"Content-Length":"919",
"Connection":"keep-alive",
"X-Cache":"Error from cloudfront",
"Via":"1.1 d387fec28536c5aa92926c56363afe9a.cloudfront.net (CloudFront)",
"X-Amz-Cf-Pop":"LHR50-P8",
"X-Amz-Cf-Id":"RMd69prehvXNAl97mo0qyFtuBIiY8r9liIxcQEmbdoBV1zwXLhirXA=="
I'm at quite a loss at what to do, because I really don't understand what my Python is missing to replicate the cURL request.
Thanks very much.
I was interested in this. I got an account with uspto.gov and acquired an access key. Their other API's work well. But the PEDS API? I kept getting the Cloudflare Gateway Timeout 503 error. While I was on their website, I looked into the PEDS API, I could not load any link to a https://ped.uspto.gov page.
I called them and they gave me an email address. I got this reply:
The PEDS API was taken down, because repeated data mining was bringing the entire PEDS System down.
The PEDS Team is working on a solution to fix the PEDS API, so that it can be re-enabled.
I tried it using PHP.
Cloudflare has been causing a lot of problems for curl.
I got a timeout.
I may have gotten past the 403 Forbidden, but did not have credentials and so the server dropped the connection.
An HTTP 504 status code (Gateway Timeout) indicates that when
CloudFront forwarded a request to the origin (because the requested
object wasn't in the edge cache), one of the following happened: The
origin returned an HTTP 504 status code to CloudFront. The origin
didn't respond before the request expired.
AWS Cloudflare Curl Issues
bypassing CloudFlare 403
How to Fix Error 403 Forbidden on Cloudflare
403 Forbidden cloudflare
██████████████████████████████████████████████████████████████
This is a conversion from you curl.
The Content-Type:application/data is added by default when you send JSON data.
I do not know about your json_data.dump or you putting the JSON in parentheses.
import requests
headers = {
'accept': 'application/json',
}
json_data = {
'searchText': '*:*',
'fq': [
'totalPtoDays:[1 TO 99999]',
'appFilingDate:[2005-01-01T00:00:00Z TO 2005-12-31T23:59:59Z]',
],
'fl': '*',
'mm': '100%',
'df': 'patentTitle',
'facet': 'true',
'sort': 'applId asc',
'start': '0',
}
response = requests.post('https://ped.uspto.gov/api/queries', headers=headers, json=json_data)
I am getting a bad request response to my request. I have checked with an online JSON validator my dictionary data to be correct, and everything seems fine.
My code is the following:
// Parse datetime to timestamp and include data in a dict
let data_dict = {
"stop_date": Date.parse(sup_limit.value),
"start_date": Date.parse(inf_limit.value)
}
// Send the Ajax request
let request = $.ajax({
url: url,
type: 'POST',
data: data_dict,
contentType: 'application/json;charset=UTF-8',
});
Backend receive endpoint:
#dashboard_bp.route('/download_last_test_influx<mode>', methods=['GET', 'POST'])
#login_required
def download_last_test_influx(mode: str):
# Check if request comes from a custom or test event
if mode == 'custom':
start_date = int(request.json.get('start_date'))
stop_date = int(request.json.get('stop_date'))
# Check if time range is valid, if not return server internal error
if stop_date - start_date <= 0:
return jsonify({'message': 'Time range must be grater than 0'}), 500
# Create response header
response = make_response(send_file(spock_comm_mgr
.test_backup_influx_manager
.get_last_test_influx_record(start_date=start_date, stop_date=stop_date)))
response.headers['Content-Type'] = 'application/gzip'
response.headers['Content-Encoding'] = 'gzip'
return response
Request header:
POST /download_last_test_influxcustom HTTP/1.1
Host: 0.0.0.0:5000
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json;charset=UTF-8
X-Requested-With: XMLHttpRequest
Content-Length: 48
Origin: http://0.0.0.0:5000
Connection: keep-alive
Referer: http://0.0.0.0:5000/influx_management
Cookie: *********************
Request payload:
stop_date=1623758400000&start_date=1623708000000
Response message:
Bad Request
The browser (or proxy) sent a request that this server could not understand.
You are telling your server, you are sending JSON data, but the request body is not a JSON string but a url-encoded string (because that's the default behaviour of $.ajax() when you pass an object as data).
Use JSON.stringify, to pass a correct JSON body
let request = $.ajax({
url: url,
type: 'POST',
data: JSON.stringify(data_dict),
contentType: 'application/json;charset=UTF-8',
});
I'm using a URL query string to debug my viewer-request and viewer-response lambda#edge functions by returning the event as JSON to the frontend (FYI so I can check for the presence/absence of certain things via an external monitoring tool).
This works fine with the viewer-request: if I go to https://example.org/?debug_viewer_request_event I get a JSON of the viewer-request event:
import json
def lambda_handler(event, context):
request = event["Records"][0]["cf"]["request"]
if "debug_viewer_request_event" in request["querystring"]:
response = {
"status": "200",
"statusDescription": "OK",
"headers": {
"cache-control": [
{
"key": "Cache-Control",
"value": "no-cache"
}
],
"content-type": [
{
"key": "Content-Type",
"value": "application/json"
}
]
},
"body": json.dumps(event)
}
return response
# rest of viewer-request logic...
Testing with cURL:
curl -i https://example.org/?debug_viewer_request_event
HTTP/2 200
content-type: application/json
content-length: 854
server: CloudFront
date: Mon, 26 Apr 2021 06:05:28 GMT
cache-control: no-cache
x-cache: LambdaGeneratedResponse from cloudfront
via: 1.1 xxxxxxxxxxx.cloudfront.net (CloudFront)
x-amz-cf-pop: AMS50-C1
x-amz-cf-id: pU0ItvQA1-r5v3yR1Dl6Z3VpPW_EuuUCHhnOD60uLhng...
{"Records": [{"cf": {"config": {"distributionDomainName": "xxxxxxx.cloudfront.net", "distributionId": "xxxxxxx", "eventType": "viewer-request", "requestId": "pU0ItvQA1-r5v3yR1Dl6Z3VpPW_EuuUCHhnOD60uLhng...
However when I do the same with the viewer-response I get a 502 error:
the code is the same except debug_viewer_request_event is debug_viewer_response_event
if I don't include the debug query string, the response is 200 OK so I know overall both lambdas are working properly (with the exception of the debug for the viewer-response)
Here is the cURL output:
curl -i https://example.org/?debug_viewer_response_event
HTTP/2 502
content-type: text/html
content-length: 1013
server: CloudFront
date: Mon, 26 Apr 2021 06:07:39 GMT
x-cache: LambdaValidationError from cloudfront
via: 1.1 xxxxxxxxx.cloudfront.net (CloudFront)
x-amz-cf-pop: AMS50-C1
x-amz-cf-id: NqXQ-FFEsIX-fEt8IvlHFTYoQdrZSGPScq1H-KNwVWR0-xxxxxx
The Lambda function result failed validation: The function tried to add, delete, or change a read-only header
If I look at the docs, the list of "Read-only Headers for CloudFront Viewer Response Events" is:
Content-Encoding
Content-Length
Transfer-Encoding
Warning
Via
As far as I can see I'm not directly changing any of these headers, but I'm guessing because I'm modifying the response, headers such as Content-Length are modified
Q: Is there a way to return the viewer-response event as JSON to the frontend for debugging or is it simply not possible due to not being able to change Content-Length?
As far as I can see I'm not directly changing any of these headers,
but I'm guessing because I'm modifying the response, headers such as
Content-Length are modified
I agree, I think your issue is that you are returning the response instead of calling
callback(null, response);
where callback should be the third argument to your lambda handler func:
def lambda_handler(event, context, callback):
Since content-length is not mutable, we should assume (and I checked this is true in practice at least for viewer request functions), cloudfront will generate it for you when you generate a response in the edge function.
I'm getting 403 watchDenied errors when I try to make File Watch requests in a Batch Request. Sending them normally is fine. I'm presuming that batching of File Watch requests is permitted?
Additional info: If I set an incorrect Address, I get the usually unpermitted webhook address error, and if I set a dummy file ID I get a 404, so it doesn't feel like the request I'm making is incorrect.
Example code (using the C# client library):
var request = new BatchRequest(service);
request.Queue<Channel>(service.Files.Watch(
new Channel
{
Id = [Guid being used],
Type = "web_hook",
Address = [notification endpoint],
Token = [token being used],
Expiration = DateTime.Now.AddHours(1).ToUnixTimeMilliseconds()
}, fileId)
, (content, error, i, message) =>
{
}
);
await request.ExecuteAsync();
API Request
Headers:
POST: https://www.googleapis.com/batch
User-Agent: OverDRIVE google-api-dotnet-client/1.16.0.0 (gzip)
Authorization: Bearer [token]
Content-Type: multipart/mixed; boundary="8d7a0653-4d9b-4c09-b701-9794341f882d"
Host: www.googleapis.com
Content-Length: 988
Accept-Encoding: gzip, deflate
Body:
--8d7a0653-4d9b-4c09-b701-9794341f882d
Content-Type: application/http
POST https://www.googleapis.com/drive/v2/files/[fileId]/watch
Content-Type: application/json; charset=utf-8
Content-Length: 241
{"address":"[notification endpoint]","expiration":1474036096103,"id":"938bd983-b071-4e63-8535-f20d7f39e248","token":"[token being used]","type":"web_hook"}
--8d7a0653-4d9b-4c09-b701-9794341f882d
Content-Type: application/http
POST https://www.googleapis.com/drive/v2/files/[fileId]/watch
Content-Type: application/json; charset=utf-8
Content-Length: 241
{"address":"[notification endpoint]","expiration":1474036096105,"id":"79303874-5f6c-49aa-a601-93950895ac0f","token":"[token being used]","type":"web_hook"}
--8d7a0653-4d9b-4c09-b701-9794341f882d--
Error example
{
"error": {
"errors": [
{
"domain": "push",
"reason": "watchDenied",
"message": "Watch request denied by backend"
}
],
"code": 403,
"message": "Watch request denied by backend"
}
}
Had an update from the developer support team for a ticket I submitted to them.
Unfortunately watch requests are known to largely incompatible with
Drive batch requests. Our Drive team recommended to make watch
requests separate from other requests.
I trying to understand how to deal with HTTP error codes using web.py as a REST framework. I can easily use try/catch blocks to return HTTP 404, 400, 500, etc... but I am having a hard time sending a custom JSON message with it.
import web
import json
urls = (
'/test/(.*)', 'Test'
)
app = web.application(urls, globals())
def notfound():
return web.notfound(json.dumps({'test': 'test'}))
class Test:
def GET(self, id):
web.header('Content-Type', 'application/json')
return self.get_resource(str(id))
def get_resource(self, id):
result = {}
if id == '1':
result = {'1': 'one'}
elif id == '3':
return web.notfound()
return json.dumps(result)
if __name__ == '__main__':
web.config.debug = False
app.notfound = notfound
app.run()
This works fine, but when id == 3, I cannot override the behaviour, and the Content-Type header is duplicated:
# curl -i -H "Accept: application/json" http://localhost:8080/test/3
HTTP/1.1 404 Not Found
Content-Type: application/json
Content-Type: text/html
Transfer-Encoding: chunked
Date: Mon, 09 Sep 2013 23:59:28 GMT
Server: localhost
404 Not Found
How can I return a JSON Content-Type, with a custom message?
HTTP Errors in web.py should be raised as exceptions.
I used this class to decorate json error (it has specific output format in my app, so you may adopt it to your needs):
class NotFoundError(web.HTTPError):
'''`404 Not Found` error.'''
headers = {'Content-Type': 'application/json'}
def __init__(self, note='Not Found', headers=None):
status = '404 Not Found'
message = json.dumps([{'note': note}])
web.HTTPError.__init__(self, status, headers or self.headers,
unicode(message))
I have created a simple json api with web.py you may want to check, believe me it has some interesting ideas.