I have a MySQL query that returns ~11,000 results that then need further queries run on each result. While the code is running it doesn't allow other users to log in to the website.
Potential problems I've seen are the use of callback functions or foreach loops but that doesn't seen to help.
code looks like this:
query(sql, params, function(err, result) {
result.foreach(
query(generate_sql(result), params, function(err, result) {
//do stuff
});
)
});
I recommend you using Promises
Then your code would be something like that:
var Promise = require('bluebird');
_query = Promise.promisify(query);
_query(sql, params)
.map(function(singleRow) {
// you get here every single row of your query
})
.then(function() {
// you are finished
});
Your problem is having that much queries. 11,001 query is a very large number, and I don't think that MySQL can handle that number of queries in parallel.
So another way is to use subqueries to handle your problem.
Related
I am trying to implement unions that can dynamically take query strings or builder as parameters. See the code below:
function dynamicUnion(queryString||builder){
baseQuery.union(function () {
//I want to use query string or querybuilder here instead.
this.select('*').from('users').whereNull('first_name');
})
}
In the place of the line: this.select('*').from('users').whereNull('first_name'), I would like to implement something like: this.raw(queryString||builder) or any working alternative though I've not come across .raw() method when working with this keyword in that block.
I am implementing it this way since the select queries that are to be used in the union will will vary and it's efficient if passed dynamically.
Sounds like a bug in knex that this.raw is not working in this case. Here is one way to pass raw query to union:
const Knex = require('knex');
const knex = Knex({
client: 'mysql',
});
knex('foo').union(knex.raw('foo bar')).toSQL();
// outputs "select * from `foo` union foo bar"
https://runkit.com/embed/10boda0lt1it
I've created a simple function that runs a query and fetches a field value from a MySQL database in Node. I'm using the normal 'mysql' library. For some reason though, I can't pass the resulting field value out to the function. What am I doing wrong?
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'localhost',
user : 'root',
password : '',
database : 'mydb'
});
//This is my function to fetch the field
function getinfofromdb(inputID){
connection.query('SELECT * FROM `mytable` WHERE ? ORDER BY `ID` DESC LIMIT 1;', {identifierID: inputID}, function (err, rows, fields) {
if(rows[0]['filename'].length > 0){
console.log(rows[0]['filename']); // This works fine! I just can't pass it as response. :(
response = rows[0]['filename'];
}else{
response = 'none found';
}
});
return response;
}
//However, I always get that 'response' is undefined
console.log(getinfofromdb(1));
Furthermore, returning from the inner function also yields nothing.
There is no problem with the query, as I can console.log it just fine, but it just doesn't return it to the function.
if(rows[0]['filename']){
console.log(rows[0]['filename']); //This prints out just fine
return rows[0]['filename']; //Doesn't return anything
}
Update: Everything I'm trying is not yielding anything. Would someone please show me the proper way of writing a function in nodejs using the mysql library (https://github.com/mysqljs/mysql) that receives an input, runs a simple query using that input, and the value of a field in the response row? I'm going nuts.
I found the solution -- this is impossible to do.
Coming from a PHP background, not everything I'm used to write is asynchronous.
This Node JS MySQL library runs its queries asynchronously, and therefore the callback function cannot return anything. It just prints stuff such as console.log.
I guess I'm gonna revert to PHP.
How to return value from an asynchronous callback function?
I have a function that is supposed to check if a license plate already exists in my MySQL database, but on the then-promise-return the result comes as:
result: [object Object]. How do you actually get the response of this knex query and parse it to check for a license plate existing?
var licensePlateExists = function(licensePlate){
knex.select('plate').from('licensePlates').where({'plate': licensePlate}).limit(1).then(function(result){
console.log("Result: " + result);
if(!result){
return true;
}
return false;
}).catch(function(error) {
console.error(error);
});
}
I think I might have an error related to the query itself, but I tested the raw query string in MySQL CLI and it outputs the row as expected. Maybe ordering matters in the knex query builder?
P.S. This doesn't error, it executes all the way through.
Try with
knex.select('plate').from('licensePlates').where('plate', licensePlate)
Actually using count query would be better
I wish to use connection pooling using NodeJS with MySQL database. According to docs, there are two ways to do that: either I explicitly get connection from the pool, use it and release it:
var pool = require('mysql').createPool(opts);
pool.getConnection(function(err, conn) {
conn.query('select 1+1', function(err, res) {
conn.release();
});
});
Or I can use it like this:
var mysql = require('mysql');
var pool = mysql.createPool({opts});
pool.query('select 1+1', function(err, rows, fields) {
if (err) throw err;
console.log('The solution is: ', rows[0].solution);
});
If I use the second options, does that mean, that connections are automatically pulled from the pool, used and released? And if so, is there reason to use the first approach?
Yes, the second one means that the pool is responsible to get the next free connection do a query on that and then release it again. You use this for "one shot" queries that have no dependencies.
You use the first one if you want to do multiple queries that depend on each other. A connection holds certain states, like locks, transaction, encoding, timezone, variables, ... .
Here an example that changes the used timezone:
pool.getConnection(function(err, conn) {
function setTimezone() {
// set the timezone for the this connection
conn.query("SET time_zone='+02:00'", queryData);
}
function queryData() {
conn.query( /* some query */, queryData);
}
function restoreTimezoneToUTC() {
// restore the timezone to UTC (or what ever you use as default)
// otherwise this one connection would use +02 for future request
// if it is reused in a future `getConnection`
conn.query("SET time_zone='+00:00'", releseQuery);
}
function releaseQuery() {
// return the query back to the pool
conn.release()
}
setTimezone();
});
In case anyone else stumbles upon this:
When you use pool.query you are in fact calling a shortcut which does what the first example does.
From the readme:
This is a shortcut for the pool.getConnection() -> connection.query() -> connection.release() code flow. Using pool.getConnection() is useful to share connection state for subsequent queries. This is because two calls to pool.query() may use two different connections and run in parallel.
So yes, the second one is also calling connection.release() you just don't need to type it.
I'm fairly new to knex and databases in general, so this is a beginner question.
I found no clear mention in the knex docs about this.
Are non-raw knex queries automatically "safe"?
Secondly, for raw queries, I have several raw statements similar to this:
var condition = _.map(ids, function(id) {
return '`id`=' + id;
}).join(' OR ');
knex('categories')
.whereRaw(condition)
.select('*')
.catch(_error.bind(null, cb))
.then(function(res) { ... });
Would escaping the id in the condition with a function described here be sufficient to escape that query?
What else to look out fo in such a scenario?
All knex queries are safe, also the knex.raw() queries if you use parameter binding syntax where ? are replaced with escaped values (http://knexjs.org/#Raw).
Query that you are doing would be better be done without raw as follows
knex('categories').whereIn('id', ids).catch(...).then(...);
If you want to use automatic escaping of column reference a.k.a identifier you may use whereRaw('?? = ?', ['id', value]) which escapes first part as identifier and second part as value.
So with parameter escaping your example would be something like this:
var condition = _.map(ids, function() {
return '?? = ?';
}).join(' OR ');
var conditionParameters = _.flatten(_.map(ids, function(id) {
return ['id', id];
}));
knex('categories')
.whereRaw(condition, conditionParameters)
.select('*')
.catch(_error.bind(null, cb))
.then(function(res) { ... });
However I have to say that there is pretty much always better ways to do the queries in knex than using raw conditions made this way.