I am new to NodeJS with MySQL Database and I want to execute my code for nested queries.
The scenario is that I have to get the list of incomplete trades and then by using iterative loop to iterate the list which I have received from database.
In the loop there are more queries executing and a 3rd party API is called to fetch the data which returns data in a callback. Now the issue is that callback execute asynchronously and the loop doesnt wait for the callback to return data and it moves on.
Kindly guide me as I am stucked in this situation.
Here is my code
var sql = incompleteTradesQuery.getIncompleteTrades();
sqlConn.query(sql, function (err, data) {
if (err) {
console.log(err);
}
else {
for (var i = 0; i < data.length; i++) {
bittrexExchange.getOrder(data.order_uuid, function(err, order_data) {
if (order_data.result.IsOpen != true) {
var order_sql = tradesQuery.insertTrade(order_data.result.OrderUuid, order_data.result.Exchange, data.customer_id, order_data.result.Quantity, order_data.result.QuantityRemaining, order_data.result.Limit, order_data.result.Reserved, order_data.result.ReserveRemaining, order_data.result.CommissionReserved, order_data.result.CommissionReserveRemaining, order_data.result.CommissionPaid, order_data.result.Price, order_data.result.PricePerUnit, order_data.result.Opened, order_data.result.Closed, order_data.result.IsOpen, null, data.commission_fee, data.total_transfer, new Date());
sqlConn.query(order_sql);
var incomplete_trades_query = incompleteTradesQuery.deleteIncompleteTradesById(data.id);
sqlConn(incomplete_trades_query);
}
});
}
}
});
Since, NodeJS fundamentally works in asynchronous nature, your nested queries will also be asynchronous and it will be a painful task to write chain of callbacks. One simple answer to your question will be use promises.
Additionally, I would recommend you using asynchronous way to handle your multiple queries which will undoubtably, work faster than the synchronous way of handling queries/requests.
NodeJS also provides async.js module that will solve your problem. Q and Step are also good packages to handle your nested callback code.
https://code.tutsplus.com/tutorials/managing-the-asynchronous-nature-of-nodejs--net-36183
Related
I'm pretty new to nodejs which is probably why I'm asking this question. I recently discovered that calls being made with nodejs to any database are async.
As a former C# .Net programmer this is little bit a surprise for me. I'm just used to code synchronous and it's ok to wait a little.
Currently I want to make a database call and with the returned result the code should continue to run. How to do this best? I found something about promises but I can't find the proper solution yet.
What I really want is something like this:
var requestLoop = setInterval(function(){
console.log('Trading bot (re)started..');
var wlist = [];
wlist = db_connection.getWatchList_DB() ==> Database call here
if(wlist.length > 0){
// Perform the rest of the code
}
}, 5000);//300000 five minutes
So, for me it's ok to wait for the database call and continue with the fetched results. Is there any simple solution for this?
You can try this mysql2 module which has inbuilt support for the promises.Code snippet from the official documentation
async function main() {
// get the client
const mysql = require('mysql2');
// create the pool
const pool = mysql.createPool({host:'localhost', user: 'root', database: 'test'});
// now get a Promise wrapped instance of that pool
const promisePool = pool.promise();
// query database using promises
const [rows,fields] = await promisePool.query("SELECT 1");
}
Also if you are very new to Node and async programming I would suggest you to learn about Callbacks,Promises and ofcourse Async-Await
I see that JSON.stringify and JSON.parse are both sycnhronous.
I would like to know if there a simple npm library that does this in an asynchonous way .
Thank you
You can make anything "asynchronous" by using Promises:
function asyncStringify(str) {
return new Promise((resolve, reject) => {
resolve(JSON.stringify(str));
});
}
Then you can use it like any other promise:
asyncStringfy(str).then(ajaxSubmit);
Note that because the code is not asynchronous, the promise will be resolved right away (there's no blocking operation on stringifying a JSON, it doesn't require any system call).
You can also use the async/await API if your platform supports it:
async function asyncStringify(str) {
return JSON.stringify(str);
}
Then you can use it the same way:
asyncStringfy(str).then(ajaxSubmit);
// or use the "await" API
const strJson = await asyncStringify(str);
ajaxSubmit(strJson);
Edited: One way of adding true asynchrnous parsing/stringifying (maybe because we're parsing something too complex) is to pass the job to another process (or service) and wait on the response.
You can do this in many ways (like creating a new service that shares a REST API), I will demonstrate here a way of doing this with message passing between processes:
First create a file that will take care of doing the parsing/stringifying. Call it async-json.js for the sake of the example:
// async-json.js
function stringify(value) {
return JSON.stringify(value);
}
function parse(value) {
return JSON.parse(value);
}
process.on('message', function(message) {
let result;
if (message.method === 'stringify') {
result = stringify(message.value)
} else if (message.method === 'parse') {
result = parse(message.value);
}
process.send({ callerId: message.callerId, returnValue: result });
});
All this process does is wait a message asking to stringify or parse a JSON and then respond with the right value.
Now, on your code, you can fork this script and send messages back and forward. Whenever a request is sent, you create a new promise, whenever a response comes back to that request, you can resolve the promise:
const fork = require('child_process').fork;
const asyncJson = fork(__dirname + '/async-json.js');
const callers = {};
asyncJson.on('message', function(response) {
callers[response.callerId].resolve(response.returnValue);
});
function callAsyncJson(method, value) {
const callerId = parseInt(Math.random() * 1000000);
const callPromise = new Promise((resolve, reject) => {
callers[callerId] = { resolve: resolve, reject: reject };
asyncJson.send({ callerId: callerId, method: method, value: value });
});
return callPromise;
}
function JsonStringify(value) {
return callAsyncJson('stringify', value);
}
function JsonParse(value) {
return callAsyncJson('parse', value);
}
JsonStringify({ a: 1 }).then(console.log.bind(console));
JsonParse('{ "a": "1" }').then(console.log.bind(console));
Note: this is just one example, but knowing this you can figure out other improvements or other ways to do it. Hope this is helpful.
Check this out, another npm package-
async-json is a library that provides an asynchronous version of the standard JSON.stringify.
Install-
npm install async-json
Example-
var asyncJSON = require('async-json');
asyncJSON.stringify({ some: "data" }, function (err, jsonValue) {
if (err) {
throw err;
}
jsonValue === '{"some":"data"}';
});
Note-Didn't test it, you need to manually check it's dependency and
required packages.
By asynchronous I assume you actually mean non-blocking asynchronous - i.e., if you have a large (megabytes large) JSON string, and you stringify, you don't want your web server to hard freeze and block newly incoming web requests for 500+ milliseconds while it processes the object.
Option 1
The generic answer is to iterate through your object piece by piece, and to then call setImmedate whenever a threshold is reached. This then allows other functions in the event queue to run for a bit.
For JSON (de)serialization, the yieldable-json library does this very well. It does however drastically sacrifice JSON processing time (which is somewhat intentional).
Usage example from the yieldable-json readme:
const yj = require('yieldable-json')
yj.stringifyAsync({key:"value"}, (err, data) => {
if (!err)
console.log(data)
})
Option 2
If processing speed is extremely important (such as with real-time data), you may want to consider spawning multiple Node threads instead. I've used used the PM2 Process Manager with great success, although initial setup was quite daunting. Once it works however, the final result is magic, and does not require modifying your source code, just your package.json file. It acts as a proxy, load balancer, and monitoring tool for Node applications. It's somewhat analogous to Docker swarm, but bare metal, and does not require a special client on the server.
Disclaimer: I'm new to NoSQL databases, if something is not clear, appreciate comments and questions to clear things up.
I'm calling some 3rd party web services to get JSON response but need a way to cache the response to avoid repeated calling since each response is constant across remote entities.
Question.
I've selected mongodb, is it the right tool for the job, the response entities have lengthy schemas and mongoose is forcing me to define one, is there a way to avoid having to define schema and just save the response by some id and read it later. if someone can kindly help with condition for checking the A. cache and B. saving.
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/test');
if() { // A. condition to check if cached response is available
} else { // call web service
http.request({}, function (response) {
var buffers = [];
response.on('data', function (chunk) {
buffers.push(chunk)
});
response.on('end', function () {
var data = Buffer.concat(buffers).toString();
// B. how to save data to a mongoose db
});
})
}
If approach is not right, and I should be using something else then please also enlighten.
If you don't have a schema available then don't use mongoose. Just store the json as mongo documents directly.
https://docs.mongodb.com/getting-started/node/client/
https://docs.mongodb.com/getting-started/node/update/
Why is the new Monsters object only being filled while being called inside the mysql function scope?
Monsters = {};
db.query('SELECT * from rpg_monsters ', function(err, results) {
for(i=0; i < results.length; i++){
Monsters[results[i].monster_id] = {
monster_id: results[i].monster_id,
monster_name: results[i].monster_name,
monster_filename: results[i].monster_filename,
monster_hp: results[i].monster_hp,
monster_chp: results[i].monster_hp,
monster_type: results[i].monster_type,
monster_level: results[i].monster_level,
xPosition: results[i].monster_xPosition,
yPosition: results[i].monster_yPosition
}
}
console.log(Monsters); // This Returns True with all the objects properties!
db.end();
});
console.log(Monsters); // This Returns Empty?
Is it possible use the Monsters object (or any) outside of the mysql callback?
Short answer: No.
Long answer: Welcome to asynchronous programming in NodeJS! Your error is not due to a scoping issue, but a time issue! Your asynchronous code does not run in source order. Your code does this:
initialize global variable Monsters reference an empty Javascript object
submit an asynchronous request to db.query, and pass a callback taking params err and results
console.log the contents of Monsters, which still references an empty array
NodeJS reaches the end of it's main loop and ticks. This may happen many times while waiting for db.query to finish its asynchronous IO. At some point in the future we resume:
db.query resolves and runs your callback function.
clone all elements in results into object at global variable Monsters
log value of Monsters
You will need to restructure your code to follow the asynchronous callback structure, or you can invesetigate alternatives like Promises or coroutines.
Please see How to return the response from an asynchronous call?, it has quite the explanation of your issue.
First, I need to tell you that I am very new to the wonders of nodejs, socketstream, angularjs and JavaScript in general. I come from a Java background and this might explain my ignorance of the correct way of doing things async.
To toy around with things I installed the ss-angular-demo from americanyak. My problem is now that the Rpc seems to be a synchronous interface and my call the the mysql database has an asynchronous interface. How can I return the database results upon a call of the Rpc?
Here is what I did so far with socketstream 0.3:
In app.js I successfully tell ss to allow my mysql database connection to be accessed by putting ss.api.add('coolStore',mysqlConn); in there at the right place (as explained in the socketstream docs). I use the mysql npm, so I can call mysql within the Rpc
server/rpc/coolRpc.js
exports.actions = function (req, res, ss) {
// use session middleware
req.use('session');
return {
get: function(threshold){
var sql = "SELECT cool.id, cool.score, cool.data FROM cool WHERE cool.score > " + threshold;
if (!ss.arbStore) {
console.log("connecting to mysql arb data store");
ss.coolStore = ss.coolStore.connect();
}
ss.coolStore.query(sql, function(err, rows, fields) {
if(err) {
console.log("error fetching stuff", err);
} else {
console.log("first row = "+rows[0].id);
}
});
var db_rows = ???
return res(null, db_rows || []);
}
}
The console logs the id of my database entry, as expected. However, I am clueless how I can make the Rpc's return statement return the rows of my query. What is the right way of addressing this sort of problem?
Thanks for your help. Please be friendly with me, because this is also my first question on stackoverflow.
It's not synchronous. When your results are ready, you can send them back:
exports.actions = function (req, res, ss) {
// use session middleware
req.use('session');
return {
get: function(threshold){
...
ss.coolStore.query(sql, function(err, rows, fields) {
res(err, rows || []);
});
}
}
};
You need to make sure that you always call res(...) from an RPC function, even when an error occurs, otherwise you might get dangling requests (where the client code keeps waiting for a response that's never generated). In the code above, the error is forwarded to the client so it can be handled there.