I have written an async Flutter/Dart function which behaves unexpectedly in my opinion. Following code structure:
static Future<bool> verifySometing() async {
try {
await getCloudData().then((snapshot) {
if (snapshot.exists && snapshot.hasData) {
bool dataValid = await validateData(snapshot.data);
if (dataValid) {
print('Data is correct');
return true;
}
}
});
} catch (e) {
print('Error $e');
return false;
}
print('Something went wrong');
return false;
}
The expected result would be that the function awaits the cloud data, then awaits validation and returns true if the data is valid. In this case, the console would show the following and the function would return true:
Data is correct
What happens in practice is that the console shows the following output and the function first returns true and then false:
Data is correct
Something went wrong
This goes against anything I thought to know about funtions in Dart because I always assumed that once return is fired, the function is done. Any ideas how this happens?
The issue is with this line.
await getCloudData().then((snapshot) {
Here, instead of just awaiting, you have also attached a then callback. So in actuality, whatever you are returning is return value of the callback function ie., (snapshot) {}.
The callback function that you need to pass into the then takes that return true and gives it to us as the result of await.
So, if you would've put something like
var bool = await getCloudData().then((snapshot) { ..... });
Then this bool, would've been equal to true. That's it. No return from the your main function.
Change it to this,
var snapshot = await getCloudData();
if (snapshot.exists && snapshot.hasData) {
bool dataValid = await validateData(snapshot.data);
if (dataValid) {
print('Data is correct');
return true;
}
}
Hope, I was able to explain clearly.
There are a few of faults in your assumptions.
First, you've attached a then to the future returned by getCloudData(). This means that you aren't awaiting getCloudData(), but instead the additional future returned by getCloudData().then(...), and that future will return when the callback function passed to it completes. (And unless the first future throws an error, the callback will be called.)
Second, the callback function operates on its own scope. So this code is not doing what you think it's doing:
bool dataValid = await validateData(snapshot.data);
if (dataValid) {
print('Data is correct');
return true;
}
This return will affect the callback function, not the verifySomething function.
Given these, the order of operation is as follows:
The validateSomething function awaits getCloudData().then(...).
getCloudData() gets called.
getCloudData() returns, the callback passed to then is called.
(Assuming the snapshot has data) validateData is called.
(Assuming data is successfully validated) "Data is correct" gets printed and the callback function returns true.
validateSomething is notified that the awaited future is complete, so execution resumes.
"Something went wrong" gets printed and the validateSomething function returns false.
Generally speaking, these kinds of errors are common when mixing async/await and then patterns. Unless you know what you're doing, stick with either one or the other, preferably the async/await pattern. For example, a refactor of your code to eliminate the call to then is as follows:
static Future<bool> verifySometing() async {
try {
final snapshot = await getCloudData();
if (snapshot.exists && snapshot.hasData) {
bool dataValid = await validateData(snapshot.data);
if (dataValid) {
print('Data is correct');
return true;
}
}
} catch (e) {
print('Error $e');
return false;
}
print('Something went wrong');
return false;
}
Now that there isn't a pesky closure to deal with, return will return from validateSomething as expected and you don't need to deal with issues like callbacks and scope.
Why does the console.log(jsonData) within the async function return the json object in the format I want, and the console.log(jsondata) outside the async function return a message saying Promise { < pending> }.
How do I save the JSON file that I fetched, so that I may store it and read it outside of the function?
async function returnData() {
const response = await fetch(data_url)
if (!response.ok) {
const message = 'An error has occured!'
throw new Error(message);
}
const jsonData = await response.json();
console.log(jsonData)
return jsonData;
}
const jsondata = returnData();
console.log(jsondata)
returnData is an async function so the return value will be a promise.
You can get the result value by writing:
(async () => {
console.log(await returnData())
})()
Take a look at this answer: How to return values from async functions using async-await from function?
In short
You cannot access promise data outside of the async block without stopping the browser entirely.
Read below for workarounds.
But why?
Asynchronous code, or async for short, gives you the ability to run a function (eg. making a server request) without stopping script execution.
This is crucial when dealing with servers and APIs because server requests take time.
You don't want to stop the browser entirely every time you make a request.
This is why asynchronous code exists, to enable you to do other things while a request is pending.
The reason console.log gave you a pending promise, is because it hasn't finished executing yet.
You cannot access promise data outside of the async block without stopping the browser entirely.
Conclusion / Workarounds
Your best bet is to make a callback function that the async function (eg. fetch) "calls back" to when it's finished executing. An example of that would be:
fetch(url).then(callBackFunc);
function callBackFunc(data) { // It automatically "feeds" the function with data
// Do something with data
}
You can also use an async function with await, as you did.
What this does is it pauses script execution, but only inside the function it is in.
Then you can call a callback function from there:
async function getData() {
var data = await fetch(url);
callBackFunc(data);
}
function callBackFunc(data) {
// Do something with data
}
References
But seriously, give them a look. They will give you a better understanding of how async works, show why it's important and explain why this was all even created.
Asynchronous Programming in JavaScript
MDN Article: Asynchronous JavaScript
Just for reference: Promises Guide: W3
I have an ordinary
var express = require('express')
Node express www page, using session, pug, etc as usual. My db calls
var db = require('./scripts/myHappyMysqlScript')
I'm naturally using mysql, so in the db script
var mysql = require('mysql')
So for example
app.get('/catPhotos', (req, response) => {
response.render('catPhotos.pug');
})
Say a page has a table showing something from the petNames database,
app.get('/pets', function(req, res, next) {
db.allPetNames(function(err, petsList) {
res.render('pets.pug',
{
'petsList': petsList,
'pretty' : true
})
})
all good so far.
But here's a case with three tables on the pug page, and three different database calls:
db.cats(function(err, c) {
db.dogs(function(err, d) {
db.budgies(function(err, b) {
res.render('bigScreen.pug',
{
'cats' : c,
'k9s': d,
'budgies': b,
'pretty' : true
})
})
})
})
I just nest them like that.
This does seem to work perfectly.
It correctly waits sequentially. Errors fall through and are handled properly, and so on.
But is there a better syntax, better way?
What's the Node Way for real™ Node, not-Swift, programmers?!
Perhaps given that I'm using the mysql library, if that's relevant.
Note, one better way overall is to use something like Ajax to just stream in each "part" of the web page. Indeed I do that all the time. What I'm asking here, assuming at res.render I indeed want to return all that info at once, is there something better than nesting like that? Cheers
You can get rid of nested database calls by using promises.
Since you mentioned that you are using mysql library for interacting with the database, unfortunately, this library doesn't provide a promise-based API. So to get rid of nested database calls in your code, you need to create a promise-based wrapper around the callback version of database calls.
For a general overview of what promises are and how they work, see the following links:
MDN - Promise.
MDN - Using Promises
Following is an example of how you can create a promise-based wrapper and then use that wrapper to get rid of nested database calls.
This promise-based wrapper is just a function that returns a promise. It creates a promise instance, wraps the underlying database call, and eventually when the database call returns the data, it notifies your code.
function getCats() {
return new Promise((resolve, reject) => {
// make the database call
db.cats((error, cats) => {
// in case of an error, reject the promise by
// calling "reject" function
// Also pass the "error" object to the "reject" function
// as an argument to get access to the error message
// in the code that calls this "getCats" function
if (error) {
reject(error);
return;
}
// if there was no error, call the "resolve" function
// to resolve the promise. Promise will be resolved
// in case of a successful database call
// Also pass the data to the "resolve" function
// to access this data in the code that calls this
// "getCats" function
resolve(cats);
});
});
}
Now in your route handler function, instead of calling db.cats(...), call this getCats wrapper function.
There are two ways you can call the function that returns a promise:
Promise-chaining (For details, visit the links mentioned above)
async-await syntax (Recommended)
The following code example uses async-await syntax. For this, first, mark the route handler function as async by using the async keyword before the function keyword. By doing this, we can use the await keyword inside this route handler function.
app.get('/pets', async function(req, res, next) {
try {
const cats = await getCats();
// similar wrappers for other database calls
const dogs = await getDogs();
const budgies = await getBudgies();
// render the pub template, passing in the data
// fetched from the database
...
catch (error) {
// catch block will be invoked if the promise returned by
// the promise-based wrapper function is rejected
// handle the error appropriately
}
});
The above code example only shows how to wrap the db.cats(...) database call in a promise-based wrapper and use that wrapper to get the data from the database. Similarly, you can create wrappers for db.dogs(...) and db.budgies(...) calls.
Instead of creating a separate promise-based wrapper for each database call, ideally, you should create a re-usable promise-based wrapper function that takes in a function to call and wraps that function call in a promise just like shown in the above code example, i.e. getCats function.
Parallel Database calls
One important thing to note in the above code is the route handler function
const cats = await getCats();
const dogs = await getDogs();
const budgies = await getBudgies();
is that this will lead to sequential database calls which may or may not be what you want.
If these database calls do not depend on each other, then you can call the promise-based wrappers in parallel using Promise.all() method.
The following code example shows how you can call your promise-based wrapper functions in parallel using Promise.all().
app.get('/pets', async function(req, res, next) {
try {
// "petsData" will be an array that will contain all the data from
// three database calls.
const petsData = await Promise.all([getCats(), getDogs(), getBudgies()]);
// render the pub template, passing in the data
// fetched from the database
...
catch (error) {
...
}
});
I hope this is enough to help you get rid of the nested database calls in your current code and start using promises in your code.
If you're trying to use MySQL with Nodejs, the module you should be looking for is mysql2 rather than mysql.
mysql2 provides a promise based approach and is a much refined version of mysql module for nodejs.
For example, for executing a query,
in mysql
con.query(sql_query, (err, rows, field)=>{ //some code here }
in mysql2, you can use the async approach as well as promise approach. Also, prepared statements in mysql2 are more easier than mysql.
//async approach
class A {
static async fn(sql, params){
const [data] = await con.execute(sql, [params]);
return data;
}
}
//promise approach remains same as **mysql** itself.
Here's the documentation for
mysql2 & more docs
If your database calls returned promises instead of using callbacks, you could:
const cats = await db.cats();
const dogs = await db.dogs();
const budgies = await db.budgies();
res.render('bigScreen.pug', {
cats : cats,
k9s: dogs,
budgies: budgies,
pretty : true
});
// Or request them all in parallel instead of waiting for each to finish
const [
cats,
dogs,
budgies
] = Promise.all([
dg.cats(),
dg.dogs(),
db.budgies()
]);
Simply convert the mysql functions into promises using the nodejs standard lib util.promisify
example:
const { promisify } = require('util');
const catsPromise = promisify(db.cats);
const dogsPromise = promisify(db.dogs);
const budgiesPromise = promisify(db.budgies);
async function routeHandler() {
let err = null;
try {
const cats = await catsPromise();
const dogs = await dogsPromise();
const budgies = await budgiesPromise();
} catch(error) {
err = error;
}
if (err) {
console.log(err);
// you should res.end() or res.render(someErrorPage) here
// failure to do so will leave the request open
} else {
res.render('bigScreen.pug', {
'cats' : cats,
'k9s': dogs,
'budgies': budgies,
'pretty' : true
});
}
}
Promise.all() method seems a more famous and cleaner way to make multiple calls in parallel like your use case.
But there is one more alternate way. : Multiple statement queries
To use this feature you have to enable it for your connection:
var connection = mysql.createConnection({multipleStatements: true});
Once enabled, you can execute multiple statement queries like any other query:
db.query('SELECT cats; SELECT dogs', function (error, results, fields) {
if (error) throw error;
// `results` is an array with one element for every statement in the query:
console.log(results[0]); // [{cat1,cat2}]
console.log(results[1]); // [{dog1,dog2}]
});
It is technically more efficient as requires less back and forth with MySQL connection.
(However, this feature is disabled by default as it allows for SQL injection attacks if values are not properly escaped). To use this feature you have to enable it for your connection.)
What's the most performant way to parse JSON in Node.js?
What's the advantage of using process.nextTick and slightly delaying the answer in the following snippet and shouldn't the whole try/catch be encapsulated into process.nextTick?
function parseJSON(json, done) {
try {
var result = JSON.parse(json);
process.nextTick(function() { done(null, result); });
} catch (err) {
process.nextTick(function() { done(err); });
}
}
First, because JSON.parse is synchronous, there's no point in using a callback.
These two blocks of code essentially do the same thing. The bottom one is more efficient:
parseJSON(json, function(err, result) {
if (err) {
//ERROR
}
else {
//PARSED
}
});
vs
try {
var result = JSON.parse(json);
//PARSED
}
catch (e) {
//ERROR
}
Now onto your actual question(s):
What's the advantage of using process.nextTick...
Basically there is no "advantage" except that if you don't use process.nextTick then your function call becomes more synchronous. Take this example for instance:
console.log('1');
parseJSON(json, function(err, result) {
console.log('parsed');
});
console.log('2');
When using process.nextTick you'll get this output:
1
2
parsed
This happens because process.nextTick puts the callback function at the end of the event loop. Meaning it'll let other things happen first then execute the callback.
If you didn't use process.nextTick then you'd get this output:
1
parsed
2
This happens because parseJSON won't exit until the callback is called.
Both versions are valid, but with callbacks you usually to want them to be truly asynchronous.
As for the try-catch, it's fine where it's at if you're trying to catch errors coming from JSON.parse. Since you appear to want to handle the errors (because you're passing the error parameter to the callback if it throws an exception) then I'd say it's correct as is.
First of all
. process.nextick is async... Try catch may not work as you expext.. JSON.parse is preferred way to parse json string
--> What's the most performant way to parse JSON in Node.js?
JSON stream probably, becasue of the low memory footprint.
See the work done by dominictarr or mafintosh
https://github.com/mafintosh/json-format-stream
https://github.com/mafintosh/through-json
https://github.com/dominictarr/JSONStream
I would like to fetch multiple files at once using the new fetch api (https://fetch.spec.whatwg.org/). Is is possible natively? If so, how should I do it leveraging the promises?
var list = [];
var urls = ['1.html', '2.html', '3.html'];
var results = [];
urls.forEach(function(url, i) { // (1)
list.push( // (2)
fetch(url).then(function(res){
results[i] = res.blob(); // (3)
})
);
});
Promise
.all(list) // (4)
.then(function() {
alert('all requests finished!'); // (5)
});
This is untested code! Additionally, it relies on Array.prototype.forEach and the new Promise object of ES6. The idea works like this:
Loop through all URLs.
For each URL, fetch it with the fetch API, store the returned promise in list.
Additionally, when the request is finished, store the result in results.
Create a new promise, that resolves, when all promises in list are resolved (i.e., all requests finished).
Enjoy the fully populated results!
While implementing Boldewyn's solution in Kotlin, I pared it down to this:
fun fetchAll(vararg resources: String): Promise<Array<out Response>> {
return Promise.all(resources.map { fetch(it) }.toTypedArray())
}
Which roughly translates to this in JavaScript:
function fetchAll(...resources) {
var destination = []
resources.forEach(it => {
destination.push(fetch(it))
})
return Promise.all(destination)
}
Earlier, I tried to use map instead of forEach + pushing to a new array, but for some reason that simply didn't work.