I trying to understand how to deal with HTTP error codes using web.py as a REST framework. I can easily use try/catch blocks to return HTTP 404, 400, 500, etc... but I am having a hard time sending a custom JSON message with it.
import web
import json
urls = (
'/test/(.*)', 'Test'
)
app = web.application(urls, globals())
def notfound():
return web.notfound(json.dumps({'test': 'test'}))
class Test:
def GET(self, id):
web.header('Content-Type', 'application/json')
return self.get_resource(str(id))
def get_resource(self, id):
result = {}
if id == '1':
result = {'1': 'one'}
elif id == '3':
return web.notfound()
return json.dumps(result)
if __name__ == '__main__':
web.config.debug = False
app.notfound = notfound
app.run()
This works fine, but when id == 3, I cannot override the behaviour, and the Content-Type header is duplicated:
# curl -i -H "Accept: application/json" http://localhost:8080/test/3
HTTP/1.1 404 Not Found
Content-Type: application/json
Content-Type: text/html
Transfer-Encoding: chunked
Date: Mon, 09 Sep 2013 23:59:28 GMT
Server: localhost
404 Not Found
How can I return a JSON Content-Type, with a custom message?
HTTP Errors in web.py should be raised as exceptions.
I used this class to decorate json error (it has specific output format in my app, so you may adopt it to your needs):
class NotFoundError(web.HTTPError):
'''`404 Not Found` error.'''
headers = {'Content-Type': 'application/json'}
def __init__(self, note='Not Found', headers=None):
status = '404 Not Found'
message = json.dumps([{'note': note}])
web.HTTPError.__init__(self, status, headers or self.headers,
unicode(message))
I have created a simple json api with web.py you may want to check, believe me it has some interesting ideas.
Related
Python 3.6 - Scrapy 1.5
I'm scraping the John Deere warranty webpage to watch all new PMP's and its expiration date. Looking inside network communication between browser and webpage I found a REST API that feed data in webpage.
Now, I'm trying to get json data from API rather scraping the javascript page's content. However, I'm getting a Internal Server Error and I don't know why.
I'm using scrapy to log in and catch data.
import scrapy
class PmpSpider(scrapy.Spider):
name = 'pmp'
start_urls = ['https://jdwarrantysystem.deere.com/portal/']
def parse(self, response):
self.log('***Form Request***')
login ={
'USERNAME':*******,
'PASSWORD':*******
}
yield scrapy.FormRequest.from_response(
response,
url = 'https://registration.deere.com/servlet/com.deere.u90950.registrationlogin.view.servlets.SignInServlet',
method = 'POST', formdata = login, callback = self.parse_pmp
)
self.log('***PARSE LOGIN***')
def parse_pmp(self, response):
self.log('***PARSE PMP***')
cookies = response.headers.getlist('Set-Cookie')
for cookie in cookies:
cookie = cookie.decode('utf-8')
self.log(cookie)
cook = cookie.split(';')[0].split('=')[1]
path = cookie.split(';')[1].split('=')[1]
domain = cookie.split(';')[2].split('=')[1]
yield scrapy.Request(
url = 'https://jdwarrantysystem.deere.com/api/pip-products/collection',
method = 'POST',
cookies = {
'SESSION':cook,
'path':path,
'domain':domain
},
headers = {
"Accept":"application/json",
"accounts":["201445","201264","201167","201342","201341","201221"],
"excludedPin":"",
"export":"",
"language":"",
"metric":"Y",
"pipFilter":"OPEN",
"pipType":["MALF","SAFT"]
},
meta = {'dont_redirect': True},
callback = self.parse_pmp_list
)
def parse_pmp_list(self, response):
self.log('***LISTA PMP***')
self.log(response.body)
Why am I getting an error? How to get data from this API?
2018-07-05 17:26:19 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <POST https://jdwarrantysystem.deere.com/api/pip-products/collection> (failed 1 times): 500 Internal Server Error
2018-07-05 17:26:20 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <POST https://jdwarrantysystem.deere.com/api/pip-products/collection> (failed 2 times): 500 Internal Server Error
2018-07-05 17:26:21 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <POST https://jdwarrantysystem.deere.com/api/pip-products/collection> (failed 3 times): 500 Internal Server Error
2018-07-05 17:26:21 [scrapy.core.engine] DEBUG: Crawled (500) <POST https://jdwarrantysystem.deere.com/api/pip-products/collection> (referer: https://jdwarrantysystem.deere.com/portal/)
2018-07-05 17:26:21 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <500 https://jdwarrantysystem.deere.com/api/pip-products/collection>: HTTP status code is not handled or not allowed
I found the problem: This is a POST request that must have a body data in json format, because unlike a GET request, the parameters don't go in the URI. The request header need too a "content-type": "application/json". See: How parameters are sent in POST request and Rest POST in python. So, editing the function parse_pmp:
def parse_pmp(self, response):
self.log('***PARSE PMP***')
cookies = response.headers.getlist('Set-Cookie')
for cookie in cookies:
cookie = cookie.decode('utf-8')
self.log(cookie)
cook = cookie.split(';')[0].split('=')[1]
path = cookie.split(';')[1].split('=')[1]
domain = cookie.split(';')[2].split('=')[1]
data = json.dumps({"accounts":["201445","201264","201167","201342","201341","201221"],"excludedPin":"","export":"","language":"","metric":"Y","pipFilter":"OPEN","pipType":["MALF","SAFT"]}) # <----
yield scrapy.Request(
url = 'https://jdwarrantysystem.deere.com/api/pip-products/collection',
method = 'POST',
cookies = {
'SESSION':cook,
'path':path,
'domain':domain
},
headers = {
"Accept":"application/json",
"content-type": "application/json" # <----
},
body = data, # <----
meta = {'dont_redirect': True},
callback = self.parse_pmp_list
)
Everything works fine!
I am sending a Post request with a json body to a server but can not extract the json file when it arrives. I have does exhaustive searches but to no avail. I have provided both client and server scripts to illustrate what is happening.
All I need is to extract the json portion at the end of the received string so I can analyze the request and return the appropriate data.
I'm sure it's simple but I can't seem to find the answer. Any direction would be appreciated
***
CLIENT: script to test Server
import json
import requests
def info_send():
url = 'http:1234abcd.ngrok.io'
payload = {
'command': '["command", "status", "off", None]',
'userID': 'userID string',
'status': 'current status',
}
requests.post(url, data=json.dumps(payload))
info_send()
***
SERVER: receives json POST request
HOST, PORT = '', 5000
listen_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listen_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
listen_socket.bind((HOST, PORT))
listen_socket.listen(1)
print('Listening on port %s' % PORT)
while True:
client_connection, client_address = listen_socket.accept()
request = client_connection.recv(1024).decode('utf-8')
print(request)
***
This is what is printed at the server
POST / HTTP/1.1
Host: 1234abcd.ngrok.io
User-Agent: python-requests/2.18.4
Accept-Encoding: gzip, deflate
Accept: /
Content-Length: 112
X-Forwarded-For: 112.162.214.265
{"command": "[\"command\", \"status\", \"off\", None]", "userID": "userID string", "deviceID": "current status"}
I am constrained to using AFNetworking, but I am working in Swift. I am trying to retrieve data from a POST API request that is supposed to be returning JSON. However it seems that no matter how I make the request I am not able to see what is being returned and that it does not seem to be in proper JSON format. Here's what I've tried.
(1) The canonical approach:
manager.responseSerializer = AFJSONResponseSerializer.init(readingOptions: .AllowFragments)
manager.responseSerializer.acceptableContentTypes = serialiazationTypes
This produces the error: Invalid value around character 2
Optional(Error Domain=NSCocoaErrorDomain Code=3840 "Invalid value around character 2." UserInfo={NSDebugDescription=Invalid value around character 2., NSUnderlyingError=0x60800004ad70 {Error Domain=com.alamofire.error.serialization.response Code=-1011 "Request failed: bad request (400)" UserInfo={com.alamofire.serialization.response.error.response=<NSHTTPURLResponse: 0x608000a2c140> { URL: https://API_ADDRESS} { status code: 400, headers {
Connection = "keep-alive";
"Content-Length" = 2261;
"Content-Type" = "text/html; charset=utf-8";
Date = "Wed, 07 Dec 2016 15:46:26 GMT";
Server = "nginx/1.10.1";
} },
(2) Using the standard parser. So basically I removed any explicit setting of the response serializer, which produced this error:
Optional(Error Domain=com.alamofire.error.serialization.response Code=-1011 "Request failed: bad request (400)" UserInfo={NSUnderlyingError=0x60000004ace0 {Error Domain=com.alamofire.error.serialization.response Code=-1016 "Request failed: unacceptable content-type: text/html" UserInfo={com.alamofire.serialization.response.error.response=<NSHTTPURLResponse: 0x61800082e6a0> { URL: https://API_ADDRESS } { status code: 400, headers {
Connection = "keep-alive";
"Content-Length" = 2261;
"Content-Type" = "text/html; charset=utf-8";
Date = "Wed, 07 Dec 2016 15:49:12 GMT";
Server = "nginx/1.10.1";
(3) So then I tried to read as text/html, not requiring JSON, so that I could see what the problem is, but I don't seem to be able to retrieve this and just see what the content is either. So with this code:
manager.responseSerializer = AFHTTPResponseSerializer()
I ended up with this response, which does not even indicate why this is a bad request:
"Request failed: bad request (400)"
{ status code: 400, headers {
Connection = "keep-alive";
"Content-Length" = 2261;
"Content-Type" = "text/html; charset=utf-8";
Date = "Wed, 07 Dec 2016 15:51:10 GMT";
Server = "nginx/1.10.1";
} }
Here are my questions
(1) In cases 1 & 2 above are these problems with parsing the response code whereas in case 3 I am just seeing the server tell me that my request is bad?
(2) How can I start figuring out why my request is bad? This is an internal API, but I'm not getting a lot of help with how it works. I have a sample curl command that does work with the API, but I'm not clear on how I am departing from it:
curl -X POST -v -s -H "Session-Token: EYYW$YW$YWYW" -H "Content-type: application/json" -d#sample.json https://API_ADDRESS
What is different between that and what I am doing below:
manager.requestSerializer = AFJSONRequestSerializer()
manager.responseSerializer = AFHTTPResponseSerializer()
manager.requestSerializer.setValue(sessionToken, forHTTPHeaderField: tokenString)
manager.requestSerializer.setValue("application/json", forHTTPHeaderField: "Content-type")
manager.POST(
user_url,
parameters: nil,
success: { (operation: AFHTTPRequestOperation?,
responseObject: AnyObject!) in
print("JSON FROM POST: \(responseObject)")
},
failure: { (operation: AFHTTPRequestOperation?, error: NSError?) in
print("there was an error in POST: \(error)")
}
)
I am trying to POST JSON Parameter to REST HTTPS URL using HTTP Builder in my Jenkins Job using Groovy script.
Below is my script :-
import groovyx.net.http.HTTPBuilder
import static groovyx.net.http.ContentType.URLENC
def http = new HTTPBuilder( 'http://restservice.appshop.com/' )
def postBody = [name: 'bob', title: 'customer'] // will be url-encoded
http.post( path: '/', body: postBody,
requestContentType: URLENC ) { resp ->
println "POST Success: ${resp.statusLine}"
assert resp.statusLine.statusCode == 201
}
I got below error :-
unable to resolve class groovyx.net.http.httpbuilder
I tried using #Grab also but still got the error only.
Kindly help me out to use this HTTP Builder.
I'm using tastypie with django. I have one line of code:
data = self.deserialize(request, request.body, format=request.META.get('CONTENT_TYPE', 'application/json'))
I use this code from the command line to send a post request to my webserver:
curl -X post -d "{ 'username' : 'user', 'password' : 'password' }" http://127.0.0.1:8000/api/employee/login/ --header "Content-Type:application/json"
When I run this, it results in a json response of
{"error": ""}
Looking at my server logs I see:
[15/Feb/2014 20:39:49] "post /api/user/login/ HTTP/1.1" 400 13
A log message logged immediately before the deserialize line will be logged successfully, but a log message logged immediately after the deserialize line will not be logged, so I am pretty sure the deserialize is wrong. Does anyone know what could be wrong or if I should consider something else as the problem?
Your JSON is not valid. Please check it here. The 400 (bad request) status should give you clue about that. It should be: {"username": "user", "password": "password"}. Here you have some solutions how to escape " char in CURL command. Tastypie unfortunately raises exception without message here but we can easily fix that for future to save time for other people which will use your API.
from tastypie.exceptions import BadRequest
from tastypie.serializers import Serializer
class VerboseSerializer(Serializer):
"""
Gives message when loading JSON fails.
"""
# Tastypie>=0.9.6,<=0.11.0
def from_json(self, content):
"""
Override method of `Serializer.from_json`. Adds exception message when loading JSON fails.
"""
try:
return json.loads(content)
except ValueError as e:
raise BadRequest(u"Incorrect JSON format: Reason: \"{}\" (See www.json.org for more info.)".format(e.message))
class MyResource(BaseModelResource):
class Meta:
serializer = VerboseSerializer(formats=['json'])