How to enforce 2 MySQL queries to run in series - mysql

I need every day to get a list of all Customers past due and insert them into a new table, then i need to move the dates with one month, so they will be processed next month again.
Currently i am running 2 SQL Queries in series
insert into history select * from customers where nextdate<CURDATE()
update customers set nextdate=calculation() where nextdate<CURDATE()
But sometimes customers are updated, but not inserted into history.
In think that the update begin to run before mysql finished the select.
I am using Node js, he current method i am using for serialization, is that i am running the update in the callback from the insert, but probably the insert is called back before the code actually runs.
It doesn't happen every time but i think that making a Stored procedure with both in it can help, anyone had some experience with it?

There are two issues here. The first is that your SELECT and UPDATE don't wait for each because SELECT does not lock the table. You can force locked reading by using:
INSERT INTO history SELECT * FROM customers WHERE nextdate<CURDATE() FOR UPDATE;
See https://dev.mysql.com/doc/refman/5.7/en/innodb-locking-reads.html for more information on Locking Reads.
The second is that you probably also want to learn about transactions, which let you issue a batch of queries such that either all of them get committed, or none of them do.
In the case of your code, you want to add the start and end commands for issueing a transaction:
START TRANSACTION;
INSERT INTO history SELECT * FROM customers WHERE nextdate<CURDATE() FOR UPDATE;
UPDATE customers SET nextdate=calculation() WHERE nextdate<CURDATE();
COMMIT;
See https://dev.mysql.com/doc/refman/5.7/en/commit.html for all the details on this mechanism, which anything from MySQL to Postgres to SQLite supports.

you can use promise provided by nodejs to make them in series or you can use Async series
Async Series
async.series([
function(callback) {
//run your First SQL Query based on response send success or failure callback
callback(null, 'one');
},
function(callback) {
//run your Second SQL Query based on response send success or failure callback
callback(null, 'two');
}
]
// your call call back should close connection if needed.
or you can go with first option
Using Promise
(I will recommend to use this one)
var conn = db.config(mysql);
run_query(conn,[ 'insert into history select * from customers where nextdate<CURDATE()' ]).then(function(result){
console.log(result); // result of 1st query
return run_query(conn,[ 'update customers set nextdate=calculation() where nextdate<CURDATE()']);
}).then(function(result){
console.log(result.something); // result of 2nd query
conn.end();
}).catch(function(err){
// will run if any error occur
console.log('there was an error', err);
});

Related

Node JS Multiple statement queries

I have started a project using Node.js and Express to connect a MySQL DB to my app, but in some cases, I need multiple SQL Statements to fetch data for the dropdown menus as they come from dimensional tables.
The query works and returns all the data I need but one thing in the documentation called my attention and I'm not sure I fully understand what is meant: (it allows for SQL injection attacks if values are not properly escaped)
https://www.npmjs.com/package/mysql#multiple-statement-queries
What do they mean by "properly escaped"? Could someone please give me an example?
The query I'm running is very similar to this:
let SQL = 'SELECT * FROM dim_val1; SELECT * FROM dim_val2; SELECT * FROM dim_val3;'
connection.query(SQL, function (error, results) {
if (error) throw error;
// `results` is an array with one element for every statement in the query:
console.log(results[0]); // [{1: 1}]
console.log(results[1]); // [{2: 2}]
console.log(results[2]); // [{2: 2}]
});
Is it risky? Is there any workaround to fetch data from multiple tables?
Thanks!

MySQL executes sleep command when UPDATE query is used

I have created a discord bot that interacts with a mysql database but when you run a command that uses the UPDATE query it doesnt execute the update query but executes sleep , meaning the data in the DB isnt chnaged.
(from comment)
#client.command()
async def SetJob(ctx, uid: str, rank: str):
disout = exec("UPDATE users SET 'job'='{0}' WHERE identifier='{1}'".format(rank,uid))
if ctx.message.author == client.user:
return
if ctx.message.author.id not in whitelisted:
await ctx.send(embed=discord.Embed(title="You are not authorized to use this bot", description='Please contact Not Soviet Bear to add you to the whitelisted members list', color=discord.Color.red()))
return
else:
await ctx.send(embed=discord.Embed(title="Job Change", description="Job changed to '{0}' for Identifier'{1}'".format(rank,uid), color=discord.Color.blue()))
I assume your "bot" is periodically doing SHOW PROCESSLIST? Well, the UPDATE probably finished so fast that it did not see the query.
The Sleep says that the connection is still sitting there, but doing nothing. (There is no "sleep command"; "Sleep" indicates that no query is running at the instant.)
So, perhaps the question is "why did my update not do anything?". In order to debug that (or get help from us),
Check for errors after running the update. (You should always do this.)
Figure out the exact text of the generated SQL. (Sometimes there is an obvious syntax error or failure to escape, say, quotes.)

How to run 'SELECT FOR UPDATE' in Laravel 3 / MySQL

I am trying to execute SELECT ... FOR UPDATE query using Laravel 3:
SELECT * from projects where id = 1 FOR UPDATE;
UPDATE projects SET money = money + 10 where id = 1;
I have tried several things for several hours now:
DB::connection()->pdo->exec($query);
and
DB::query($query)
I have also tried adding START TRANSACTION; ... COMMIT; to the query
and I tried to separate the SELECT from the UPDATE in two different parts like this:
DB::query($select);
DB::query($update);
Sometimes I get 0 rows affected, sometimes I get an error like this one:
SQLSTATE[HY000]: General error: 2014 Cannot execute queries while other unbuffered queries are active. Consider using PDOStatement::fetchAll(). Alternatively, if your code is only ever going to run against mysql, you may enable query buffering by setting the PDO::MYSQL_ATTR_USE_BUFFERED_QUERY attribute.
SQL: UPDATE `sessions` SET `last_activity` = ?, `data` = ? WHERE `id` = ?
I want to lock the row in order to update sensitive data, using Laravel's database connection.
Thanks.
In case all you need to do is increase money by 10, you don't need to lock the row before update. Simply executing the update query will do the job. The SELECT query will only slow down your script and doesn't help in this case.
UPDATE projects SET money = money + 10 where id = 1;
I would use diferent queries for sure, so you can have control on what you are doing.
I would use a transaction.
If we read this simple explanations, pdo transactions are quite straightforward. They give us this simple but complete example, that ilustrates how everithing is as we should expect (consider $db to be your DB::connection()->pdo).
try {
$db->beginTransaction();
$db->exec("SOME QUERY");
$stmt = $db->prepare("SOME OTHER QUERY?");
$stmt->execute(array($value));
$stmt = $db->prepare("YET ANOTHER QUERY??");
$stmt->execute(array($value2, $value3));
$db->commit();
}
catch(PDOException $ex) {
//Something went wrong rollback!
$db->rollBack();
echo $ex->getMessage();
}
Lets go to your real statements. For the first of them, the SELECT ..., i wouldn't use exec, but query, since as stated here
PDO::exec() does not return results from a SELECT statement. For a
SELECT statement that you only need to issue once during your program,
consider issuing PDO::query(). For a statement that you need to issue
multiple times, prepare a PDOStatement object with PDO::prepare() and
issue the statement with PDOStatement::execute().
And assign its result to some temp variable like
$result= $db->query ($select);
After this execution, i would call $result->fetchAll(), or $result->closeCursor(), since as we can read here
If you do not fetch all of the data in a result set before issuing
your next call to PDO::query(), your call may fail. Call
PDOStatement::closeCursor() to release the database resources
associated with the PDOStatement object before issuing your next call
to PDO::query().
Then you can exec the update
$result= $db->exec($update);
And after all, just in case, i would call again $result->fetchAll(), or $result->closeCursor().
If the aim is
to lock the row in order to update sensitive data, using Laravel's database connection.
Maybe you can use PDO transactions :
DB::connection()->pdo->beginTransaction();
DB::connection()->pdo->commit();
DB::connection()->pdo->rollBack();

MySQL to select or insert based on condition

How to do this in a single MySQL query:
if (select count(*)..)=10
select a record from the same table
else
insert a record into the same table
If you want to use one single SQL command (but I don't know why) you can use a Stored Procedure:
CREATE PROCEDURE `select_or_insert`()
MODIFIES SQL DATA
COMMENT 'blah blah'
BEGIN
IF ((SELECT COUNT(*) FROM `your_table`) = 10) THEN
SELECT ... FROM ... ;
ELSE
INSERT INTO ... ;
END IF;
END;
To invoke the Procedure you will issue the following command:
CALL `select_or_insert`();
If the SELECT is executed, the statement will return a resultset.
As long as only one webserver is involved, I would try APC http://php.net/manual/de/book.apc.php with some nice system of timeout.
Say for example: if an IP has already sent a request in the last 2 seconds, it's current request should be refused and the timeout changes to 4 seconds, after that 10 seconds etc.
if ($timeoutLevel = apc_fetch("locked_" . $ip)){
$timeoutLevel++;
$timeout = getNextTimeout($timeoutLevel);
apc_store("locked_".$ip, $timeoutLevel, $timeout);
show_my_error_page(get_friendly_text("please do not try again for $timeout seconds! You are Blocked!"));
exit();
}
else{
$timeoutLevel = 1;
$timeout = INITIAL_PAGE_TIMEOUT;
apc_store("locked_".$ip, $timeoutLevel, $timeout);
}
That should cost at max around 50 Byte per ip of the last x seconds, so if it is not a DDOS then the webserver should have that RAM.
But be careful: some html-pages contain references to css, javascript, images, sounds, ajax-calls might come later, json-requests etc. pp.
After $timeout seconds APC automatically drops that value, so no need to hire a cleaning women.

Node.js and node-mysql DB Query - need synchronous code help

I am writing some non-web app helper, and came across a need for a synchronous query call.
Basically, within a loop I need to check the database to see if the value exists. If it doesn't then insert the value. Currently, with node-mysql I can only get it to work with a callback. Because of that, node.js treats the call as asynchronous and keeps processing my request before the query is finished. This is a big issue because in the end it could be inserting duplicates because they were in the queue.
Ideal Solution - doesn't work. Results is actually the object of client, and I can't find the actual results within. However this does make it synchronous.
results = client.query('SELECT COUNT(md5) as md5Count FROM table WHERE md5 = "' + md5 + '"')
The following does not work. Node.js treats it as asynchronous, and outerResult is still the object of client.
outerResult = client.query('SELECT COUNT(md5) as md5Count FROM board WHERE md5 = "' + md5 + '"', function selectCb(err, results, fields) {console.log(results);});
Any help is appreciated.
Basically, within a loop I need to check the database to see if the value exists. If it doesn't then insert the value.
This is a problem best served with SQL. You don't solve this problem by talking to the database repeatedly, you solve this problem by having SQL only insert where the index value doesn't already exist.
INSERT INTO mytable ( name, address )
SELECT #name, #address FROM DUAL
WHERE NOT EXISTS (SELECT * FROM mytable WHERE name = #name, address = #address)
This is a super simplified example, and not the most optimized. You can do the same thing here with sets of data, instead of record by record, if you like.
Basically, within a loop I need to
check the database to see if the value
exists. If it doesn't then insert the
value. Currently, with node-mysql I
can only get it to work with a
callback. Because of that, node.js
treats the call as asynchronous and
keeps processing my request before the
query is finished. This is a big issue
because in the end it could be
inserting duplicates because they were
in the queue.
There is an asynchronous solution, there always is.
basically your worried that duplicate entries could be entered.
I presume you have an array of data to loop through. Your problem is solved with _.uniq or some other filter solution.
So you simply call _.uniq(md5s).forEach(function() { })