I am trying to crawl a ajax site using Scrapy, the url is http://www.target.com/p/bounty-select-a-size-white-paper-towels-12-mega-rolls/-/A-14920157#prodSlot=medium_1_2&term=bounty
My goal is to get the store id. I did that by checking all the XHR request in chrome developer tool and find the one with name ("v1?request_type=availability&key=.....") to be the one I want.
My questions are:
In developer tool, there is "Query string parameter" which seems to be the part after ? of the request url. There is also Request Payload section, which is a json.
So which one should I use to send to the server? If i need request payload, how can I send a json file?
when i send the whole url to get json:
https://api.target.com/available_to_promise_aggregator/v1?request_type=availability&key=q0jGNkIyuqUTYIlzZKoCfK6ugaNGSP8h
I get "Request method 'GET' not supported", so should I use POST instead or there is something wrong I did?
You must send the query string as part of the URL after a ?, as you guessed.
To include a JSON payload in a request, and send the request as a POST request, use the method and body parameters of the Request class.
Related
I tried to POST a request to a web API. The website help says if I POST a JSON like this, it will response with a token:
{
"client_id": "0xDMgCIF9bW3bRBKlUicpivvOoLGExCJqbOHqPYI",
"client_secret": "BQKCoBO3iNXo52cMmiKPmuyYW0mE1PpPJOHXxQFgcJuWzg92PC",
"grant_type": "client_credentials"
}
But after I use Postman to POST the request, I put the above string in the body section, it always return 400 BAD REQUEST response. I have never used POSTMAN before. Can someone help to check if the string content is correct or not?
you have to choose body type as json not formdata:
choose body>raw>json and replace the content with your json
I am always getting JSON response for my HTTP request to access one ORDS-Oracle Rest Data Services API:
http://localhost:8080/ords/hr/employees/
However my requirement is to get XML response. I tried to choose XML from drop down list available at POSTMAN, however content is not changing to XML.
I need a way to specify parameter as part of HTTP URL so that response is XML than JSON and I tried to change URL like below:
Way1:
http://localhost:8080/ords/hr/employees?format=xml
Way2:
http://abcdef.us.oracle.com:8001/claims-ws/api/generic/lineofbusinesses/421?Content-Type:application/xml
Way3:
http://abcdfef.us.oracle.com:8001/claims-ws/api/generic/lineofbusinesses/421?Accept:application/xml
Way4:
http://abcdef.us.oracle.com:8001/claims-ws/api/generic/lineofbusinesses/421?Accept=application/xml
However nothing is working and response is always JSON.
Any suggestion?
I have to append parameter in JSON response after request processing is completed and before sending. I can do in after exec filter. Here I defined filter like this,
beego.InsertFilter("*", beego.AfterExec, AddRequestAfterExec, false)
Now in this AddRequestAfterExec method, I am unable to read JSON response. I need to read JSON response and add parameter to it. I searched many things, but did not find any thing useful. So far I have left it empty.
func AddRequestAfterExec(c *context.Context) {}
Can anybody help, how to read Response in this function and modify it?
Also I am sending response from API controller like this
authController.ServeJSON()
Opening https://www.coinome.com/exchange/data.json in browser shows a json,but while trying to fetch response by making a GET request,the returned response is in html format.
How to get JSON response instead of html response?
that's because the first time you perform the request to that url, the server returns an html form to check whether youare human or not.
Then after you compile and submit the form a cookie is returned.
You have to save that cookie and redo the request to the same url while putting the cookie in the request header in order to get the response in json format.
I am trying to use HTTP Post to post the data by passing following Header and Body part,
Body:
One image URL having file path. Type as 'File'
Metadata in JSON format. Type as 'Text'
Headers:
Content-type = multipart/form-data
Authorization = Bearer 5412
Here is the screenshot of complete request,
This is how I have setup the HTTP Post task in Pentaho,
I found related post here using REST Client but this also didn't help: Pentaho HTTP Post using JSON
Really appreciate your help.
I think the problem is that "Post a file" doesn't mean include a file in the request, but rather to get the entire request from that file.
I don't know the specifics of how the data should look in the Post request, but the rough approach should be:
Pass the filename field to a Calculator step with the operation "Load file content to binary" to get a binary type field (ex: myimage)
Base64 (or other) encode the data with a Javascript step like this:
var encString = new Packages.java.lang.String( Packages.org.apache.commons.codec.binary.Base64.encodeBase64( myimage ) );
Wrap it in some content-type string indicating the encoding
Include the field in your request.
There is a patch on https://jira.pentaho.com/browse/PDI-14743 to enable proper binary transfer instead of String when using Rest Client.