What's the most performant way to parse JSON in Node.js?
What's the advantage of using process.nextTick and slightly delaying the answer in the following snippet and shouldn't the whole try/catch be encapsulated into process.nextTick?
function parseJSON(json, done) {
try {
var result = JSON.parse(json);
process.nextTick(function() { done(null, result); });
} catch (err) {
process.nextTick(function() { done(err); });
}
}
First, because JSON.parse is synchronous, there's no point in using a callback.
These two blocks of code essentially do the same thing. The bottom one is more efficient:
parseJSON(json, function(err, result) {
if (err) {
//ERROR
}
else {
//PARSED
}
});
vs
try {
var result = JSON.parse(json);
//PARSED
}
catch (e) {
//ERROR
}
Now onto your actual question(s):
What's the advantage of using process.nextTick...
Basically there is no "advantage" except that if you don't use process.nextTick then your function call becomes more synchronous. Take this example for instance:
console.log('1');
parseJSON(json, function(err, result) {
console.log('parsed');
});
console.log('2');
When using process.nextTick you'll get this output:
1
2
parsed
This happens because process.nextTick puts the callback function at the end of the event loop. Meaning it'll let other things happen first then execute the callback.
If you didn't use process.nextTick then you'd get this output:
1
parsed
2
This happens because parseJSON won't exit until the callback is called.
Both versions are valid, but with callbacks you usually to want them to be truly asynchronous.
As for the try-catch, it's fine where it's at if you're trying to catch errors coming from JSON.parse. Since you appear to want to handle the errors (because you're passing the error parameter to the callback if it throws an exception) then I'd say it's correct as is.
First of all
. process.nextick is async... Try catch may not work as you expext.. JSON.parse is preferred way to parse json string
--> What's the most performant way to parse JSON in Node.js?
JSON stream probably, becasue of the low memory footprint.
See the work done by dominictarr or mafintosh
https://github.com/mafintosh/json-format-stream
https://github.com/mafintosh/through-json
https://github.com/dominictarr/JSONStream
Related
I have written an async Flutter/Dart function which behaves unexpectedly in my opinion. Following code structure:
static Future<bool> verifySometing() async {
try {
await getCloudData().then((snapshot) {
if (snapshot.exists && snapshot.hasData) {
bool dataValid = await validateData(snapshot.data);
if (dataValid) {
print('Data is correct');
return true;
}
}
});
} catch (e) {
print('Error $e');
return false;
}
print('Something went wrong');
return false;
}
The expected result would be that the function awaits the cloud data, then awaits validation and returns true if the data is valid. In this case, the console would show the following and the function would return true:
Data is correct
What happens in practice is that the console shows the following output and the function first returns true and then false:
Data is correct
Something went wrong
This goes against anything I thought to know about funtions in Dart because I always assumed that once return is fired, the function is done. Any ideas how this happens?
The issue is with this line.
await getCloudData().then((snapshot) {
Here, instead of just awaiting, you have also attached a then callback. So in actuality, whatever you are returning is return value of the callback function ie., (snapshot) {}.
The callback function that you need to pass into the then takes that return true and gives it to us as the result of await.
So, if you would've put something like
var bool = await getCloudData().then((snapshot) { ..... });
Then this bool, would've been equal to true. That's it. No return from the your main function.
Change it to this,
var snapshot = await getCloudData();
if (snapshot.exists && snapshot.hasData) {
bool dataValid = await validateData(snapshot.data);
if (dataValid) {
print('Data is correct');
return true;
}
}
Hope, I was able to explain clearly.
There are a few of faults in your assumptions.
First, you've attached a then to the future returned by getCloudData(). This means that you aren't awaiting getCloudData(), but instead the additional future returned by getCloudData().then(...), and that future will return when the callback function passed to it completes. (And unless the first future throws an error, the callback will be called.)
Second, the callback function operates on its own scope. So this code is not doing what you think it's doing:
bool dataValid = await validateData(snapshot.data);
if (dataValid) {
print('Data is correct');
return true;
}
This return will affect the callback function, not the verifySomething function.
Given these, the order of operation is as follows:
The validateSomething function awaits getCloudData().then(...).
getCloudData() gets called.
getCloudData() returns, the callback passed to then is called.
(Assuming the snapshot has data) validateData is called.
(Assuming data is successfully validated) "Data is correct" gets printed and the callback function returns true.
validateSomething is notified that the awaited future is complete, so execution resumes.
"Something went wrong" gets printed and the validateSomething function returns false.
Generally speaking, these kinds of errors are common when mixing async/await and then patterns. Unless you know what you're doing, stick with either one or the other, preferably the async/await pattern. For example, a refactor of your code to eliminate the call to then is as follows:
static Future<bool> verifySometing() async {
try {
final snapshot = await getCloudData();
if (snapshot.exists && snapshot.hasData) {
bool dataValid = await validateData(snapshot.data);
if (dataValid) {
print('Data is correct');
return true;
}
}
} catch (e) {
print('Error $e');
return false;
}
print('Something went wrong');
return false;
}
Now that there isn't a pesky closure to deal with, return will return from validateSomething as expected and you don't need to deal with issues like callbacks and scope.
They say the V8 engine is a proper Javascript ES5 engine. Does it support ES5 features like Promise?
No. Promises are not functionally supported. But all promises related syntax is valid and doesn't throw any error. However everything runs synchronously.
async function promise1_() {
Logger.log("Start")
Utilities.sleep(10000);
return "done";
}
function test1(){
promise1_();
Logger.log("End")
}
If promises worked, "End" should be logged before "Start", but that's not the case.
Apps script V8 recognizes the new function definition formats, for example:
let letAsyncFunction = async function() { //Your logic here }
It asynchronously returns the result of the function evaluations, pretty much like a Promise.
In other words, when the function async is called, it returns a promise. Also await is used for calling an async function and wait for it to resolve or reject
References:
async function
V8 runtime Overview
Its a bit old yet still relevant.
My answer is No (Apr. 2022).
Although syntactically and the way the code runs yes, but practically it is synchronous flow.
example:
function main() {
console.log(Date(),"start");
getStatAsync();
console.log(Date(),"end");
}
function getStatAsync(){
let p = new Promise((res,rej)=>{
let url = "some url";
let response = UrlFetchApp.fetch(url);
let ans = parse(response);
if (ans){
res(ans);
}else{
rej("fetch faild");
}
})
//resolved
p.then((res) =>{
console.log("fetch resolved",res);
})
//rejected
p.catch((msg) => {
console.log("fetch failure", msg );
})
function parse(response){
let ans = null;
let code = response.getResponseCode();
if (code == "200" ) {
ans = JSON.parse(response.getContentText());
} else {
console.error(` fetch failed code=${code}`);
}
return ans;
}
}
main()
this is semi code fetching for a URL from within a Promise.
The promise is wrapping UrlFetchApp is within a promise that logs the fetch results either is the then or the catch callbacks.
The output for that would be:
start
end
for successful fetch: fetch resolved
The end message is logged before the promise logging, indicating asynchronous execution.
Without the promise end message become the last log line and the fetch resolved becomes last as in standard synchronous flow
However, assuming the fetch takes 10 sec, both end and fetch resolved are logged after those 10 sec.
The main code is suspended until fetch returned, although the log lines seems are printed like it is asynchronous flow
In my Flutter app I'd like to make multiple network calls simultaneously and then do something when they all have finished. For this I use Future.wait(), which does what I want. However when a call fails it throws an exception, which is somehow not caught in the exception handler (i.e. uncaught exception).
When I do await _fetchSomeData() separately (outside Future.wait()) the exception does get called by the exception handler as expected.
Future<bool> someMethod() async {
try {
var results = await Future.wait([
_fetchSomeData(),
_fetchSomeOtherData()
]);
//do some stuf when both have finished...
return true;
}
on Exception catch(e) {
//does not get triggered somehow...
_handleError(e);
return false;
}
}
What do I need to do to catch the exceptions while using Future.wait()?
Update:
I have narrowed down the issue. Turns out if you use another await statement in the method that is called by the Future.wait() it causes the issue. Here an example:
void _futureWaitTest() async {
try {
//await _someMethod(); //using this does not cause an uncaught exception, but the line below does
await Future.wait([ _someMethod(), ]);
}
on Exception catch(e) {
print(e);
}
}
Future<bool> _someMethod() async {
await Future.delayed(Duration(seconds: 0), () => print('wait')); //removing this prevents the uncaught exception
throw Exception('some exception');
}
So if you either remove the await line from _someMethod() or if you just call _someMethod() outside of Future.wait() will prevent the uncaught exception. This is most unfortunate of course, I need await for an http call... some bug in Dart?
I have the Uncaught Exceptions breakpoints enabled. If I turn this off the issue seems to be gone. Perhaps it's an issue with the debugger. I am using Visual Studio Code and the latest flutter.
What do I need to do to catch the exceptions while using Future.wait()?
What I found out when I used the same code as you the code inside of each procedure which is used in Future.wait() must be wrapped with try/catch and on catch must return Future.error(). Also eagerError must be set to true.
try {
await Future.wait([proc1, ...], eagerError: true);
} on catch(e) {
print('error: $e')
}
/// Proc 1
Future<void> proc1() async {
try {
final result = await func();
} on SomeException catch(e) {
return Future.error('proc 1 error: $');
}
}
I think you are a bit mislead by the Future.wait() naming. Future.wait() returns another future that will have a List of elements returned by each future when it completes with success.
Now since the Future.wait() is still a future. You can handle it in two ways:
Using await with try catch.
Using onError callback.
Tis will be something like
Future.wait([futureOne, futureTwo])
.then((listOfValues) {
print("ALL GOOD")
},
onError: (error) { print("Something is not ok") }
The jsonix library does not follow the first argument must be an error convention so I decided to use bluebird and promisify it like so:
return new Promise(function(resolve, reject) {
try {
unmarshaller.unmarshalString(xmlResponse,
function (unmarshalled) {
...
resolve(unmarshalled);
});
}
catch (error) {
reject(error);
}
});
But this hangs indefinitely! Whereas if I simply save the xmlResponse to a file and then process it with a different method: unmarshalFile ... the promisification seems to work just fine!
return new Promise(function(resolve, reject) {
try {
unmarshaller.unmarshalFile('test1.xml',
function (unmarshalled) {
...
resolve(unmarshalled);
});
}
catch (error) {
reject(error);
}
});
So my question is why would promisification fail for one method but not another?
When I look at the code for jsonix, I don't see any callback function for .unmarshalString() and looking at the implementation, there's nothing async in the implementation and nothing that calls a callback. It just returns the answer directly. So, the function is synchronous, not async and returns its value directly as a return value.
For reference, .unmarshalURL() and .unmarshalFile() do accept the callback and do have an async implementation - .unmarshalString() is just different.
So, you don't need to use promises with unmarshaller.unmarshalString(xmlResponse) at all. You could just return the straight value:
return unmarshaller.unmarshalString(xmlResponse);
If you want to wrap it in a promise for consistency of interface among all three methods, you can do this:
try {
return Promise.resolve(unmarshaller.unmarshalString(xmlResponse));
} catch(err) {
return Promise.reject(err);
}
Or, you can use Bluebird's Promise.method() to wrap it for you:
return Promise.method(unmarshaller.unmarshalString.bind(unmarshaller, xmlResponse));
Disclaimer: I'm the author of Jsonix.
unmarshalURL and unmarshalFile are async (and have to be) but unmarshalString or unmarshalDocument are not async (and don't have to be).
I have a an exports function I'm calling that should return a json array of draft results. In the route below in app.js, when I console.log draft_results, I get undefined
app.get('/draft-results', function(req, res) {
var draft_results = fantasy.getDraftResults(req, res);
console.log(util.inspect(draft_results, false, null));
//looks in views folder by default
res.render('draft-results', {
draft_results: draft_results
});
});
In my other file, this is the function that should be returning the json array. If i console.log draft, the data is there.
exports.getDraftResults = function(req, res, cb) {
oauth.get(
"http://fantasysports.yahooapis.com/fantasy/v2/league/" + conf.LEAGUE_ID + "/draftresults?format=json",
req.user.accessToken,
req.user.tokenSecret,
function(e, data, resp) {
if (e) console.error(e);
data = JSON.parse(data);
var draft = data.fantasy_content.league[1].draft_results;
res.json(draft);
}
);
};
I feel like I am returning the data incorrectly, and I can't seem to find any other good examples out there. Could someone please assist?
getDraftResults() is asynchronous. That means the results it generates occur sometime later. Thus, it cannot return its results directly from the function like you are trying to use.
It is unclear what you want to be doing here. Inside of getDraftResults() you are creating a JSON response back to the web request that started all this. That, in itself would be fine and will work as you have it (except the error handling is missing).
But, in your app.get() handler, you have completely different code that seems to thing that getDraftResults() is going to return a value (it has no return value at all) and then you will later use that return value.
So, if you just want getDraftResults to make a JSON response to the original web request, it's already doing that and you can remove the rest of what you have in the app.get() handler. If that's not really what you want to do and you want to use the response from getDraftResults() inside of the app.get() handler, then you will have to change the design of both functions and likely pass a callback to getDraftResults() so the callback can supply the asynchronous response and you can then continue the rest of the app.get() functionality in that callback.
If you're trying to do the latter, then here's a scaffolding (I don't know exactly what you're trying to accomplish so I can't be too detailed here):
app.get('/draft-results', function(req, res) {
fantasy.getDraftResults(req, function(err, draft_results) {
if (err) {
// send some sort of error response here
console.error(err);
return;
}
console.log(util.inspect(draft_results, false, null));
//looks in views folder by default
res.render('draft-results', {
draft_results: draft_results
});
});
});
exports.getDraftResults = function(req, cb) {
oauth.get(
"http://fantasysports.yahooapis.com/fantasy/v2/league/" + conf.LEAGUE_ID + "/draftresults?format=json",
req.user.accessToken,
req.user.tokenSecret,
function(e, data, resp) {
if (e) {
console.error(e);
cb(e);
return;
}
data = JSON.parse(data);
var draft = data.fantasy_content.league[1].draft_results;
// send results back to caller
cb(null, draft);
}
);
};