Playframework handling post request - json

In my routes:
POST /forms/FormValidator1/validateForm controllers.FormValidator1.validateForm(jsonForm:String)
There is a controller method defined for that route:
def validateForm(jsonForm:String) = Action { ...
Then I try to send POST request by chrome POSTMAN plugin (see pic above).
I use:
url: http://localhost:9000/forms/FormValidator1/validateForm
headers: Content Type: application/json
json data: {name: "me", surname: "my"}
So, sending this POST request I can not reach controller's method by mentioned route / url. Why?
UPDATE:
Interestly enough: after I got it working on my laptop (see my answer below) then push it on gitHub and pull it to another machine it starts working differently. Now it complains than Bad Request is [Invalid XML] nevertheless I use "application/json" header and did not change any line of code after commit. I wonder maybe it is a bug.

It seems I got it.
Here:
https://groups.google.com/forum/#!topic/play-framework/XH3ulCys_co
And here:
https://groups.google.com/forum/#!msg/play-framework/M97vBcvvL58/216pTqm22HcJ
There is wrong and correct way explained:
Doesn't work: curl -d "name=sam" http://localhost:9000/test
Works: curl -d "" http://localhost:9000/test?name=sam
This is the way how POST params are passing..in play. (second link is explanation WHY):
Sometimes you have to make compromises. In Play 1 you could bind your
action parameters from any parameter extracted from the URL path,
query string or even the request body. It was highly productive but
you had no way to control the way the form was uploaded. I mean, if a
user uploads a big file you needed to load the entire request in
memory to be able to handle it.
In Play 2 you can control the request body submission. You can reject
it early if something is wrong with the user, you can process big
files or streams without filling your memory with more than one HTTP
chunk… You gain a high control of what happens and it can help you to
scale you service. But, the other side of the coin is that when a
request is routed, Play 2 only uses the request header to make its
decision: the request body is not available yet, hence the inability
to directly bind an action parameter from a parameter extracted from
the request body.
UPDATE:
Interestly enough: after I got it working on my laptop then push it on gitHub and pull it to another machine it starts working differently. Now it complains than Bad Request is [Invalid XML] nevertheless I use "application/json" header and did not change any line of code after commit.
UPDATE 2
So I fixed it like this:
On angular side (we even can comment dataType and headers):
var data = $scope.fields
$http({
url: '/forms/FormValidator1/validateForm',
method: "POST",
//dataType: "json",
data: data,
//headers: {'Content-Type': 'application/json'}
}).success(function (data, status, headers, config) {
console.log("good")
}).error(function (data, status, headers, config) {
console.log("something wrong")
});
On playFramework side: (use BodyParser)
def validateForm = Action { request =>
val body: AnyContent = request.body
val jsonBody: Option[JsValue] = body.asJson
// Expecting text body
jsonBody.map { jsValue =>
val name = (jsValue \ "name")
val surname = (jsValue \ "surname")
....
}
Routes (don't define parameters at all !):
POST /forms/FormValidator1/validateForm controllers.FormValidator1.validateForm

Related

NodeJS and HTTP: How to send JSON content in a POST/PUT call as a string rather than creating a new key

I built an API service that will process REST requests and use them to perform CRUD operations on a MongoDB instance. This application is standalone (by design) and should be a passthrough for anything that calls it. My other application that I built in Angular is calling this API to interact with my MongoDB instance. I have been trying to construct my JSON payload from a form, which works fine. I get something like:
{ "_id":"111111111", "name":"herp", "address":"derp", "city":"foo", "state":"bar", "zip":"11111", "phone":"111-222-3333"}
I am then trying to take that JSON and send it along to the service, but something is getting lost in translation once the service gets a hold of it and my variable name that contains the JSON object is being turned into an actual key in the request, with the JSON as its value. I am calling the service like this:
const httpOptions = {
headers: new HttpHeaders({ 'Content-Type': 'application/json', 'Cache-Control': 'no-cache' })
};
updateStuff(update){
console.log("Sending: " + JSON.stringify(update) + " for update");
return this.http.put('http://localhost:3000/api/test/_update', {dbName:"testDb",collection:"testing",update}, httpOptions);
}
Which logs:
Sending: {"name":"blah","address":"111 Anystreet","city":"MyCity","state":"NY","zip":"11111","phone":"555-111-2222","_id":"5ba914df13236f7a6ea3e233"} for update
So I know that right before the call is made, the data is fine. However, on the other side, it sees the following when it gets the data:
Received request: {"dbName":"testDb","collection":"testing","update":{"name":"blah","address":"111 Anystreet","city":"MyCity","state":"NY","zip":"11111","phone":"555-111-2222","_id":"5ba914df13236f7a6ea3e233"}}
instead of what I intended, which is below:
{"dbName":"testDb","collection":"testing","name":"blah","address":"111 Anystreet","city":"MyCity","state":"NY","zip":"11111","phone":"555-111-2222","_id":"5ba914df13236f7a6ea3e233"}
How do I tell the HTTP request to send the data itself rather than constructing a new key with the name "update" and sticking the payload in there as its value? I tried JSON.stringify, but that ends up sending the same thing, but with a bunch of backslashes in front of all the parenthesis. It still sends it all in a key with the name "update" as well. Any help would be greatly appreciated.
Your problem is here:
{dbName:"testDb",collection:"testing",update}
The statement above is shorthand for this:
{dbName:"testDb",collection:"testing",update:update}
What you're looking to do is this:
{dbName:"testDb",collection:"testing",...update}
Which is shorthand for this:
const data = {dbName:"testDb",collection:"testing"};
for (let key in update) {
if (update.hasOwnProperty(key)) {
data[key] = update[key];
}
}

Scraping - catch json response with python

I need to scrape a website with a "load more" button.
I need to catch the json response (which is invisible in the html code) and parse it to build URLs
This is the JSON post request response
I'm using Selenium, python.
how do I ?
tHX
You can bypass actually clicking on the "load more" button by reading the API call that the website is sending when you click the button and then sending it via Selenium. If you send it through Selenium, you can capture the response. Here's what I've been using an Angular website. You'll have to modify it to work with the website you're using, but this should get you started.
call = """
$http = angular.element(document.body).injector().get('$http');
var done = arguments[0];
$http({
method: 'POST',
headers: {
"Content-Type": "application/json"
},
data: {
foo: "bar"
},
url: "https://request.url/"
}).then(data => done(data));
"""
json_response = driver.execute_async_script(call)
The execute_async_script method will make the call and wait for a JSON response.
You can also right-click on the xhr in Chrome DevTools and copy the API call, which should make it easier to recreate it with selenium.
Let me know if you have follow-up questions.

NodeJS request doesn't encode the entire form

The task is rather simple, I request the endpoint with POST request (https://banana.com/endpoint/swap.php), give it my form: { banana: ["China's Red", "Sweden's Gray"], apples: [] } and send it.
However, the Request module for NodeJS that I am using does not encode the empty array (in this case "apples") and if the endpoint doesn't receive the "apples" array, it returns an error - "Invalid JSON". I have tried doing this with already encoded strings and it works just fine. I am also unable to stringify my json and then use encodeURI(), as it will then give "bananas" and "apples" quotes around them, which will get encoded - needless to say, the endpoint doesn't like that either.
I'd really appreciate if somebody could at least point me in the right direction. As I am unsure on how to proceed with this, without creating some awful spaghetti code.
data = { banana: ["China's Red", "Sweden's Gray"], apples: [] }
result = JSON.parse(JSON.stringify(data)) .
You wouldn't get double around banana and apple and if you need to access then access it
console.log(result.banana)
console.log(result.apple)
So if you need to feed this result in post request then -
url = 'your url';
const options = {
url: url,
method: 'POST',
headers: {
Accept: 'application/json',
'Accept-Charset': 'utf-8'
},
json: result
};
request.post(options, function (err, response, body) {
// do something with your data
})
Let me know if this works.

Node-red - Posting data to influxdb via http

Im trying to post data to an Influxdb via Node-red.
Via CURL i can post this:
curl -i -XPOST 'http://localhost:8086/write?db=waterlevel' --data-binary 'vattenstand,lake=siljan,region=dalarna value=160.80'
and it puts data to InfluxDb.
When I try to post via Node-red and an HTTP request I get the error:
{"error":"unable to parse '{\"url\":\"http://192.168.1.116:8086/write?db=waterlevel\",\"method\":\"POST\",\"body\":\"vattenstand,lake=siljan,region=dalarna value=160.80\",}': missing tag value"}
I use this code in a function in Node-red and pass it to the HTTP request:
var dataString = 'vattenstand,lake=siljan,region=dalarna value=160.80';
msg.payload = {
'url': 'http://192.168.1.116:8086/write?db=waterlevel',
'method': 'POST',
'body': dataString,
};
msg.headers = {
Accept: "application/json"
};
return msg;
The sidebar help for the node details the msg properties you should be setting to configure the node.
You are passing in URL, method and body as properties of msg.payload. That is not correct.
They should be set as msg.url, msg.method for the first two, and msg.payload should be the body of the request.
In this instance, you have already configured the node with a URL and method directly, so there's no need to pass them in with the message. In fact, as you have configured the URL in the node you will find you cannot override it with msg.url. if you want to set the URL with each message, you must leave the node's URL field blank in the editor.
You may also need to set the content-type header.
Assuming you are happy to leave the URL and method hard coded in the node, you function should be something like:
msg.payload = 'vattenstand,lake=siljan,region=dalarna value=160.80';
msg.headers = {
Accept: "application/json"
};
msg.headers['Content-type'] = 'application/x-www-form-urlencoded';
return msg;
Why don't you use the spezial influxdb node?
https://flows.nodered.org/node/node-red-contrib-influxdb
Advantage: The http header need not be created. You can reuse the defined connection for other data.

NodeJS : Validating request type (checking for JSON or HTML)

I would like to check if the type that is requested by my client is JSON or HTML, as I want my route to satisfy both human and machine needs.
I have read the Express 3 documentation at:
http://expressjs.com/api.html
And there are two methods req.accepts() and req.is(), used like this:
req.accepts('json')
or
req.accepts('html')
Since these are not working as they should, I tried using:
var requestType = req.get('content-type');
or
var requestType = req.get('Content-Type');
requestType is always undefined...
Using the suggestion at this thread:
Express.js checking request type with .is() doesn't work
does not work either. what am I doing wrong?
Edit 1: I have checked for correct client HTML negotiation. Here are my two different request headers (taken from the debugger inspector):
HTML : Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
JSON : Accept: application/json, text/javascript, */*; q=0.01
Solution (thanks to Bret):
Turns out I was specifying the Accept headers wrong, and the */* was the issue. Here is the code that works!
//server (now works)
var acceptsHTML = req.accepts('html');
var acceptsJSON = req.accepts('json');
if(acceptsHTML) //will be null if the client does not accept html
{}
I am using JSTREE, a jQuery plugin that uses jQuery Ajax calls underneath). The parameters passed to the Ajax call are in the "ajax" field, and I have replaced the "accepts" parameter with a complete "headers" object. Now it works, and should solve the problem when you are using plain jQuery if it should ever occur.
//client
.jstree({
// List of active plugins
"plugins" : [
"themes","json_data","ui","crrm","cookies","dnd","search","types","hotkeys","contextmenu"
],
"json_data" : {
"ajax" : {
// the URL to fetch the data
"url" : function(n) {
var url = n.attr ? IDMapper.convertIdToPath(n.attr("id")) : "<%= locals.request.protocol + "://" + locals.request.get('host') + locals.request.url %>";
return url;
},
headers : {
Accept : "application/json; charset=utf-8",
"Content-Type": "application/json; charset=utf-8"
}
}
}
})
var requestType = req.get('Content-Type'); definitely works if a content-type was actually specified in the request (I just re-verified this a minute ago). If no content-type is defined, it will be undefined. Keep in mind that typically only POST and PUT requests will specify a content type. Other requests (like GET) will often specify a list of accepted types, but that's obviously not the same thing.
EDIT:
Okay, I understand your question better now. You're talking about the Accept: header, not content-type.
Here's what's happening: notice the */*;q=0.8 at the end of listed accept types? That essentially says "I'll accept anything." Therefore, req.accepts('json') is always going to return "json" because technically it's an accepted content type.
I think what you want is to see if application/json is explicitly listed, and if so, respond in json, otherwise respond in html. This can be done a couple of ways:
// a normal loop could also be used in place of array.some()
if(req.accepted.some(function(type) {return type.value === 'application/json';}){
//respond json
} else {
//respond in html
}
or using a simple regular expression:
if(/application\/json;/.test(req.get('accept'))) {
//respond json
} else {
//respond in html
}
According to the documentation, this the proper solution
req.accepts(['html', 'json']) === 'json'
Hope it helps!
Please define "not working as they should", because they do for me.
In other words:
// server.js
console.log('Accepts JSON?', req.accepts('json') !== undefined);
// client sends 'Accept: application/json ...', result is:
Accepts JSON? true
// client sends 'Accept: something/else', result is:
Accepts JSON? false
The Content-Type header as sent by a client isn't used for content negotiation, but to declare the content type for any body data (like with a POST request). Which is the reason why req.is() isn't the correct method to call in your case, because that checks Content-Type.
To throw my hat in the ring, I wanted easily readable routes without having .json suffixes everywhere or having to have each route hide these details in the handler.
router.get("/foo", HTML_ACCEPTED, (req, res) => res.send("<html><h1>baz</h1><p>qux</p></html>"))
router.get("/foo", JSON_ACCEPTED, (req, res) => res.json({foo: "bar"}))
Here's how those middlewares work.
function HTML_ACCEPTED (req, res, next) { return req.accepts("html") ? next() : next("route") }
function JSON_ACCEPTED (req, res, next) { return req.accepts("json") ? next() : next("route") }
Personally I think this is quite readable (and therefore maintainable).
$ curl localhost:5000/foo --header "Accept: text/html"
<html><h1>baz</h1><p>qux</p></html>
$ curl localhost:5000/foo --header "Accept: application/json"
{"foo":"bar"}
Notes:
I recommend putting the HTML routes before the JSON routes because some browsers will accept HTML or JSON, so they'll get whichever route is listed first. I'd expect API users to be capable of understanding and setting the Accept header, but I wouldn't expect that of browser users, so browsers get preference.
The last paragraph in ExpressJS Guide talks about next('route'). In short, next() skips to the next middleware in the same route while next('route') bails out of this route and tries the next one.
Here's the reference on req.accepts.