not getting full body with npm request package - json

I have the following function which returns a promise to get data. I am using the request npm module.
let getData = function (user) {
return new Promise(function (resolve, reject) {
let url = 'https://someurl.com/' + user;
request(url, function (error, res, body) {
if (error) reject(error);
try{
resolve(cleanData(JSON.parse(body).items))
}catch(e){
console.log(body)
console.log(url)
console.log(e);
}
})
})
}
When I resolve my promise, I sometimes get something like this:
SyntaxError: Unexpected token u in JSON at position 0
That happens when my body is returned as undefined. But other times I also get something like this:
SyntaxError: Unexpected token f in JSON at position 11203
And other times (most of the times), it goes through perfectly.
I've been able to double check this while debugging and it seems that there are times when I get an incomplete body. I know for a fact that the body from the source url is not incomplete. I checked this by going to the url directly with my browser and making sure the json was complete and valid.
What is going on? It is my understanding that the callback on the request function is only called when the response is ready to be consumed.

No - the callback is triggered as soon as there is a response - see
HTTP request API documentation. You will need to implement a response listener as:
res.on('data', (chunk) => {
// do the data accumulation here
});
res.on('end', () => {
// do the resolve here
});

In request module if you expect binary data from response, then set encoding: null in request options.
Excerpt from:
encoding - encoding to be used on setEncoding of response data. If null, the body is returned as a Buffer. Anything else (including the default value of undefined) will be passed as the encoding parameter to toString() (meaning this is effectively utf8 by default). (Note: if you expect binary data, you should set encoding: null.)
https://www.npmjs.com/package/request

Related

Proper way of handling errors and sending JSON back for API

I am new to js/node/express, and I have been working on this application where I have the following code to handle user registration:
const Account = require('../models/Account.js')
module.exports = {
// TODO: Check why Postman hangs on POST request for this
async register (req, res, next) {
if (req.body.email && req.body.password) {
var newAccount = new Account()
newAccount.email = req.body.email
newAccount.password = newAccount.generateHash(req.body.password)
const account = Account.create({email: newAccount.email, password: newAccount.password}, function (err, res) {
if (err) {
console.log('could not insert. Check error.')
// CANT CALL res.status(400).send({ error: 'email already exists'})
res.status(500)
return next(err)
}
res.status(400).send({
error: 'exists'
})
})
console.log(`inserted account ${newAccount.email}`)
res.send(account.toJSON())
}
}
}
I read this post about how to properly send JSON data back in order to build a proper REST API but ran into some issues.
When I do the call to res.status(400) I get an error that res.status is not a function. Is that because res is not available in that if statement? If it isn't how then, do I properly send a 400 (or any error status) in a case like this?
I want to be able to send an error message if the saving into my mongo db fails, or send back the created user if the insertion was successful.
If there is anything out there that I can read Id love to read some of that as well.
When I do the call to res.status(400) I get an error that res.status is not a function.
That's because you are defining res as an argument to the callback in this line:
const account = Account.create({email: newAccount.email, password: newAccount.password},
function (err, res) {
And that res hides the higher scoped res. The solution is to not have a name conflict. Change the name of this res to be accountRes or something like that. You have to be aware of name conflicts in declared argument names when nesting inline functions.
It also looks like:
res.send(account.toJSON())
is in the wrong place. You will send that BEFORE Account.create() finishes its asynchronous work. That probably needs to be inside the callback.
Speaking of proper error handling, if this if (req.body.email && req.body.password) test fails, then you don't send any response at all. You need to always send some sort of response to an http request. I'd suggest adding an else to that if and send an appropriate response.

Sending larger JSON strings via HTTP using Node

I am working on sending JSON data from my HTTP server to my client. I have been successful in sending smaller sized JSON responses to the client, but once I have to wait and collect all the data before sending, my code does not function properly. I am using the .on('data', function (){}) method to collect the data and build it back up and the .on('end', function(){}) to try sending the data, but my code never enters either of these methods.
My code for sending the larger sized JSON data is below:
exports.sendJson = function (req, resp, data) {
resp.writeHead(200, "Valid EndPoints", { "Content-Type": "application/json" });
var payload = '';
resp.on('data', function(data){
payload += data;
})
.on('end', function(){
resp.write(JSON.stringify(payload));
});
resp.end();
};
My code that works fine for smaller sized JSON data is as follows:
exports.sendJson = function (req, resp, data) {
resp.writeHead(200, { "Content-Type": "application/json" });
if(data) {
resp.write(JSON.stringify(data));
}
resp.end();
};
Thanks in advance. I'm actually pretty happy I got this far.
A HTTP server response is a Writable; it does not emit data events.
If the thing you pass in to sendJson's data parameter is an object, you just do exactly what you're doing in the second example. There is no need to buffer anything. (If the thing you pass to sendJson is a stream, just pipe it to the response.)
(When parsing received JSON, you must buffer the data you get from a Readable -- the HTTP response on the client.)

Node Exports Function Returning Undefined

I have a an exports function I'm calling that should return a json array of draft results. In the route below in app.js, when I console.log draft_results, I get undefined
app.get('/draft-results', function(req, res) {
var draft_results = fantasy.getDraftResults(req, res);
console.log(util.inspect(draft_results, false, null));
//looks in views folder by default
res.render('draft-results', {
draft_results: draft_results
});
});
In my other file, this is the function that should be returning the json array. If i console.log draft, the data is there.
exports.getDraftResults = function(req, res, cb) {
oauth.get(
"http://fantasysports.yahooapis.com/fantasy/v2/league/" + conf.LEAGUE_ID + "/draftresults?format=json",
req.user.accessToken,
req.user.tokenSecret,
function(e, data, resp) {
if (e) console.error(e);
data = JSON.parse(data);
var draft = data.fantasy_content.league[1].draft_results;
res.json(draft);
}
);
};
I feel like I am returning the data incorrectly, and I can't seem to find any other good examples out there. Could someone please assist?
getDraftResults() is asynchronous. That means the results it generates occur sometime later. Thus, it cannot return its results directly from the function like you are trying to use.
It is unclear what you want to be doing here. Inside of getDraftResults() you are creating a JSON response back to the web request that started all this. That, in itself would be fine and will work as you have it (except the error handling is missing).
But, in your app.get() handler, you have completely different code that seems to thing that getDraftResults() is going to return a value (it has no return value at all) and then you will later use that return value.
So, if you just want getDraftResults to make a JSON response to the original web request, it's already doing that and you can remove the rest of what you have in the app.get() handler. If that's not really what you want to do and you want to use the response from getDraftResults() inside of the app.get() handler, then you will have to change the design of both functions and likely pass a callback to getDraftResults() so the callback can supply the asynchronous response and you can then continue the rest of the app.get() functionality in that callback.
If you're trying to do the latter, then here's a scaffolding (I don't know exactly what you're trying to accomplish so I can't be too detailed here):
app.get('/draft-results', function(req, res) {
fantasy.getDraftResults(req, function(err, draft_results) {
if (err) {
// send some sort of error response here
console.error(err);
return;
}
console.log(util.inspect(draft_results, false, null));
//looks in views folder by default
res.render('draft-results', {
draft_results: draft_results
});
});
});
exports.getDraftResults = function(req, cb) {
oauth.get(
"http://fantasysports.yahooapis.com/fantasy/v2/league/" + conf.LEAGUE_ID + "/draftresults?format=json",
req.user.accessToken,
req.user.tokenSecret,
function(e, data, resp) {
if (e) {
console.error(e);
cb(e);
return;
}
data = JSON.parse(data);
var draft = data.fantasy_content.league[1].draft_results;
// send results back to caller
cb(null, draft);
}
);
};

Mongoose Populate with express res.json() breaks

So I'm selecting Activities from the mongodb and populating User for each.
var query = Activity.find(query).populate("user");
return query.sort({created:"desc"}).exec(function(err, activities) {
debugger;
if (!err) {
return res.json(activities);
} else {
res.status(400).json(err);
}
});
As you can see I have a debugger; breakpoint is there, When I'm pring activities it prints an array of activities with the user object populated.
Also when I'm calling something like activities[0].toJSON() I get everything good!
But the response comes back with the user property empty !
I looked into the source of express.response.json(OBJ) and saw this line:
var body = JSON.stringify(val, replacer, spaces);
val is my activities
When calling JSON.stringify(activities) it will create a json with an empty user field.. any suggestions ?
Try the lean option. That gives back plain JS objects with no mongoose weirdness. Also, your error handling seems a little awkward, can be simplified.
var query = Activity.find(query).populate("user");
query.sort({created:"desc"}).lean().exec(function(err, activities) {
if (err) return res.status(400).json(err);
res.json(activities);
});
I would go even further, not hard-coding error sending in routes but simply passing along via if (err) return next(err) to error-handling middleware defined elsewhere in your app. You can still set the status, then use detection in your middleware, something like this:
app.use(function(err, req, res, next){
err.status = err.status || 500;
res.status(err.status).json(err);
});

Node Express 4 get header in middleware missing

I have a middleware function using Node's Express4 to log each request & response for debugging. I use the res.json call in the request handler to send back JSON to the client for all but static files. So I do not want to log the response for static files, but only the JSON responses. I have the following code:
function logRequests(req, res, next) {
// do logging (will show user name before authentication)
logger.reqLog('IN '+req.method+' '+req.url, req);
var oldEnd = res.end,
oldWrite = res.write,
chunks = [];
res.write = function(chunk) {
chunks.push(chunk);
oldWrite.apply(res, arguments);
};
res.end = function(chunk, encoding) {
if(chunk) {
chunks.push(chunk);
}
oldEnd.apply(res, arguments);
// the content-type prints "undefined" in some cases
// even though the browser shows it returned as "application/json"
console.log('type='+res.get('content-type'));
if(res.get('content-type') === 'application/json') {
var body = Buffer.concat(chunks).toString('utf8');
logger.info(body, req);
}
logger.reqLog('OUT '+req.method+' '+req.path, req);
};
next(); // make sure we go to the next routes and don't stop here
}
So why do some requests show the correct content type in the middleware meaning they also print the response fine and others do not? All of them look good in the REST client when inspecting the returned headers.
EDIT: Some more info discovered tonight while trying to figure this out - if I append any character as a dummy request parameter, it logs the response type correctly:
http://localhost:8081/node/ionmed/api/logout?0 WORKS
where
http://localhost:8081/node/ionmed/api/logout DOES NOT
Also, I can always get a response type logging in the middleware function if I replace the .json() call with .end() so this:
res.json({ item: 'logout', success: true });
becomes:
res.set('content-type', 'application/json');
res.end({ item: 'logout', success: true });