How can I send data back to node, from a process launched via execfile in nodeJS? preferably in a JSON format.
This is how my code looks like right now:
//index.js NodeJS
var execFile = require('child_process').execFile;
var express = require('express');
app.get('/', function(req, res) {
var lchecker = execFile('/usr/local/bin/casperjs', [myprogram, myargs]);
lchecker.stdout.on('data', function(data) {
var dataObject = JSON.parse(data); //This throws an error.
});
});
The casperjs script that I'm calling returns JSON-like strings, like this:
console.log("[{href: targetLink.href, anchor: targetLink.anchor, isLive: 0, isFollowed: null}]");
This is the error that I get
When I'm trying to parse the JSON-like string, I get an error that says:
19 Jun 16:46:43 - [nodemon] starting node index.js
undefined:1
[{href: targetLink.href, anchor: targetLink.anchor, isLive: 1, isFollow: 1}]
^
Unexpected token h
So my JSON is invalid, and sincerely, I'm sure that there's a better way to send data back to node from casperjs, but I don't know how.
I've been thinking about creating a new route in express, and then make casperjs visit that route and pass the information via GET, and then manipulate that information in node. Is this a good way to achieve this?
Even though I received good and viable answers, I ultimately ended up outputting everything to stdout in casperjs, to send it back to PHP through a JSON array.
so in casperjs I wrote something like:
console.log(JSON.stringify(targetLink))
And then in node, I could just access that through JSON.parse and manipulate the data in any way I want.
EDIT:
I've run into this situation more often than not, so alternatively you can make CasperJS POST the information to a web endpoint, it's sometimes cleaner, but it adds overhead if you are worried about security and you need to make sure that only authorized scrapers can post data to your endpoint.
I have used CasperJS on NodeJS by running CasperJs as a service.
Basically NodeJS through http.get() makes a request to CasperJS script which return a JSON object as response.
Here an example and more details about how a CasperJS script can start a web server:
CasperJS passing data back to PHP
You may probably prefer to use something like SpookyJS (https://github.com/WaterfallEngineering/SpookyJS) which offer the ability to use CasperJs inside a Node.js program.
I don't know if you will find the feature you want but it's probably cleaner anyway.
Related
I'm using Superagent in my react app, and i'm making some call's to the IPFS api. Specifically, I am uploading files to my IPFS server. Now, everything works, When I upload one or multiple files the call goes through and the files show up in IPFS no problem.
A problem occurs when I upload multiple files though, the response seems to come back as plain text, instead of JSON, and superagent throws the error
client.js:399 Uncaught (in promise) Error: Parser is unable to parse the response
at Request.<anonymous> (client.js:399)
at Request.Emitter.emit (index.js:133)
at XMLHttpRequest.xhr.onreadystatechange (client.js:708)
So to be clear, when uploading a single file, I get a nice JSON response, but when I upload multiple files, the response is in plain text.
Can I force Superagent to give me the response back and parse it myself? Or can I set something when making the call so that it forces a json parse? Below is my superagent request function
add : acceptedFiles => {
const url = ipfs.getUrl("add")
const req = request.post(url)
acceptedFiles.forEach(file => req.attach(file.name, file))
req.then(res => {
return console.log(res);
})
}
I'm searching for this for a more elegant solution, but before I would have found it , I'd like to provide my own solution.
I think this problem caused by wrong responsive Content-Type set, but I've not confirmed this opinion yet.
However, you can try this:
req.catch(function (err) {
console.log(err.rawResponse)
})
At least, this solves my problem.
According to their docs you can specify custom parser that will take precedence over built-in parser:
You can set a custom parser (that takes precedence over built-in parsers) with the .buffer(true).parse(fn) method. If response buffering is not enabled (.buffer(false)) then the response event will be emitted without waiting for the body parser to finish, so response.body won't be available.
I tried and it worked well for me.
superagent.get('....')
.buffer(true)
.parse(({ text }) => JSON.parse(text))
.then(...)
I am relatively new to nodejs and running into an issue while parsing a Json post request.
Here is the JSON format of the post request:
{"parameters":{"issuerId":[96409],"source":["'XYZ'"]}}
And here is my code to read it.
function getSearchData(req, res, next) {
console.log("req is" + req.body);
try {
JSON.parse(reqJSON);
} catch (e) {
console.log(e);
}
}
This parsing works fine and I am able to parse it and do my further logic. However, if I change my format of post request(same request with additional new lines) it fails to parse as it adds additional commas in place of each new line in the request.
{
"parameters": {
"issuerId": [96409],
"source":["'XYZ'"]
}
}
Here's the output from the code with the second request.
req is{,"parameters":{"id":[96409],,"source":["'XYZ'"]}}
[SyntaxError: Unexpected token ,]
If you notice, an extra comma gets added at each new line, which was never in the request to begin with.
What am I doing wrong here?
You should never have to parse the JSON yourself, unless you're concatenating the request body stream yourself.
Hint 1: Do you use any framework like Express? Do you use body parser?
Hint 2: How do you create the JSON?
Hint 3: Do you use correct content-type?
Hint 4: How do you create req.body from the request stream?
You didn't include the entire code so it's impossible to give you a specific solution.
What am I doing wrong here?
Whatever you're doing wrong here, it's not included in the question.
However, if I change my format of post request(same request with additional new lines)
It would be useful if you included more details of how you do it.
I see two potential sources of that problem:
either the commas are introduced during the on the client side
or they are introduced during the request reading on the server side
You didn't show us any of those two parts - you didn't show the serializing code and the code that sends the data, and you didn't include the code that gets the data, possibly joins it from chunks and parses the JSON. But the problem is likely in one of those parts.
Update
Here is an example on how to do what you need using Express. You didn't answer whether you use any framework like Express or not, but I think that you should if you can't achieve that simple task without it, so here is a working example:
const express = require('express');
const bodyParser = require('body-parser');
const app = express();
function getSearchData(req, res, next) {
console.log('req body is', req.body);
console.log('req body JSON is', JSON.stringify(req.body));
res.json({ ok: true });
}
app.use(bodyParser.json());
app.use(getSearchData);
app.listen(3335, () => console.log('Listening on http://localhost:3335/'));
It shows how to correctly get a parsed JSON request body as an object req.body, how to print the data as a standard console.log representation and serialized again as JSON. See how it works and compare it to your own solution. This is all I can do because not having seen your entire solution I cannot tell you more than the hints that I've already given.
I'm doing a web app based on original MEAN.js framework. When I want to request local json test file using $http.get() method in my AngularJS file, it returned my index html content.Is it a routing problem? I didnot change the original mean.js routing code(https://github.com/meanjs/mean), just added a $http.get() method in home.client.controller.js file. Can anyone help me with this? Thanks!
That is most likely happening, because you didn't define an endpoint for that particular GET request in your app.
Everytime you make a request to your server (for example a GET request to /my-request) nodejs/express are configured in MEAN.js so that your server will try to find the endpoint for that request, if it does not find any, that request will be handled by this particular code block (specified in /modules/core/server/routes/core.server.routes.js):
// Define application route
app.route('/*').get(core.renderIndex);
Which will basically render the index view.
I'm not sure if you're using a custom module or not, eitherway, if you want that request to be handled in a different way in MEAN.js, you can specify your endpoint in your custom module routes file (or in core.server.controller.js) like so:
// Define application route
app.route('/my-request').get(core.sendMyJSON);
Be careful, because this route must be placed before the one I mentioned earlier, otherwise your request will still be handled the same way and the index view will be rendered and served again.
Then you will have to create the controller that should be called to handle that request:
exports.sendMyJSON = function (req, res) {
// logic to serve the JSON file
};
This way you should be able to get it done with a few adjustments.
Side note:
I'm not entirely sure but I think if you place your JSON file in the public directory of your app you should be able to directly access it without the need for the extra logic.
I have a http request I am trying to make on an afterSave method in my Cloud Code. I have been able to create my request, and when I console.log(response) it outputs a block that contains the information that I am after. I am aware that response.text is a string so I am trying to run JSON.parse(response.text) so I can access my API response.
I can print out what appears to be an object after running JSON.parse, but much of the data within my response is stripped out. I know it is not the fault of the API because I have another function that runs on the client with the same query and it works correctly.
What is the correct way to parse the response.text from a Parse.Cloud.httpRequest to maintain my data.
Try var result = JSON.parse(response['text']).
I'm trying to deal with: "Requests made from Javascript running on your web pages must be proxied to avoid same-origin policy conflicts."
I know how to work with the JSON once I've got it. But aside from copy-pasting the JSON results via my browser, I don't know how to localize it for use.
Did you tried
$.getJSON('url', function(data){
//do smth with data
})
?
After the request is complete, data will be an object with all JSON response and you can access it as regular js object: data.something and so on.