Hey guys, here is one I am not able to figure out. We have a table in database, where PHP inserts records. I created a trigger to compute a value to be inserted as well. The computed value should be unique. However it happens from time to time that I have exact same number for few rows in the table. The number is combination of year, month and day and a number of the order for that day. I thought that single operation of insert is atomic and table is locked while transaction is in progress. I need the computed value to be unique...The server is version 5.0.88. Server is Linux CentOS 5 with dual core processor.
Here is the trigger:
CREATE TRIGGER bi_order_data BEFORE INSERT ON order_data
FOR EACH ROW BEGIN
SET NEW.auth_code = get_auth_code();
END;
Corresponding routine looks like this:
CREATE FUNCTION `get_auth_code`() RETURNS bigint(20)
BEGIN
DECLARE my_auth_code, acode BIGINT;
SELECT MAX(d.auth_code) INTO my_auth_code
FROM orders_data d
JOIN orders o ON (o.order_id = d.order_id)
WHERE DATE(NOW()) = DATE(o.date);
IF my_auth_code IS NULL THEN
SET acode = ((DATE_FORMAT(NOW(), "%y%m%d")) + 100000) * 10000 + 1;
ELSE
SET acode = my_auth_code + 1;
END IF;
RETURN acode;
END
I thought that single operation of
insert is atomic and table is locked
while transaction is in progress
Either table is locked (MyISAM is used) or records may be locked (InnoDB is used), not both.
Since you mentioned "transaction", I assume that InnoDB is in use.
One of InnoDB advantages is absence of table locks, so nothing will prevent many triggers' bodies to be executed simultaneously and produce the same result.
Related
I am working on the backend of an application that needs to protect an external API from too many requests per user per month. So I need to keep track of number of requests from each user. I have a lot of experience with concurrent programming but almost no experience with db management or MySQL,
So, suppose I want to execute the equivalent of the following pseudocode, where I mix SQL statements with application-level logic, and where lookups is a table:
mutex mtx;
set #userid = 'usrid1';
set #date = CURDATE();
set #month = CONCAT_WS('-', YEAR(#date), MONTH(#date));
mtx.lock()
select counter from lookups where userid=#userid and month=#month;
if returned rows == 0:
insert into lookups set month=#month, userid=#userid, counter=1;
else:
update lookups set counter=counter+1;
mtx.unlock()
Except, of course, I don't have access to that mutex. At first I thought it would be enough to just wrap the whole thing inside a transaction, but upon closer inspection of the MySQL reference it seems that may not be enough to avoid possible race conditions, such as two threads/processes reading the same counter value. Is it good enough then, in mysql with default settings, to do the following:
set #userid = 'usrid1';
set #date = CURDATE();
set #month = CONCAT_WS('-', YEAR(#date), MONTH(#date));
start transaction;
select counter from lookups where userid=#userid and month=#month for update;
if returned rows == 0:
insert into lookups set month=#month, userid=#userid, counter=1;
else:
update lookups set counter=counter+1;
commit;
From what I can glean from the reference, it looks like it should be enough, and it should cause neither race conditions nor deadlocks, but the reference is long winded and complex, so I wanted to ask here to be sure. Performance isn't important. The reference states that MySQL's default isolation level is REPEATABLE READ.
I suggest this solution:
create table lookups (userid varchar(20), yearmonth date, counter int, primary key (userid, yearmonth));
insert into lookups set userid = 'usrid1',
yearmonth = date_format(curdate(), '%Y-%m-01'),
counter = last_insert_id(1)
on duplicate key update
counter = last_insert_id(counter + 1);
select last_insert_id(); -- returns the new value, whether 1 or the updated value.
This means you don't have to check if a row exists, it will either insert it or update it atomically.
The last_insert_id(<expression>) trick is documented at the end of the entry for that function: https://dev.mysql.com/doc/refman/8.0/en/information-functions.html#function_last-insert-id
I am running event to fetch data from main table and inserting it into Summary table
I am running MySQL event every 2 minute and calling stored procedure inside my event for some grouping, parsing and calculations from main table to summary table.
But while selecting 2 minutes records, some time last record got missed while creating summary occasionally
Frequency of missing record is in 1 day 90-100 records (1 day record count in main table is 30K).
MySQL select query inside stored procedure:
SELECT ID, COUNT(*) AS TOT_COUNT
FROM CUSTOMER
WHERE (TIMESTAMP > (DATE_FORMAT((NOW() - INTERVAL 3 MINUTE),"%Y-%m-%d
%H:%i:00"))
AND TIMESTAMP <= (DATE_FORMAT((NOW() - INTERVAL 1 MINUTE),"%Y-%m-%d %H:%i:00")))
GROUP BY ID, NAME;
Note: selecting previous 2 minutes record, skipping current minute records
I have tried updating select statement where condition as below (missing record frequency reduced to 50%)
SELECT ID, NAME, COUNT(*) AS TOT_COUNT
FROM CUSTOMER
WHERE (SUBSTRING_INDEX(TIMESTAMP, ':', 2) != SUBSTRING_INDEX((NOW()), ':', 2)
AND SUBSTRING_INDEX(TIMESTAMP,':',2) >= SUBSTRING_INDEX((NOW() - INTERVAL 2 MINUTE),':',2))
GROUP BY ID,NAME;
Also try to catch exception by using below statement in stored procedure.
DECLARE EXIT HANDLER FOR SQLEXCEPTION
BEGIN
GET DIAGNOSTICS CONDITION 1 #sqlstate = RETURNED_SQLSTATE,
#errno = MYSQL_ERRNO, #text = MESSAGE_TEXT;
SET #full_error = CONCAT("ERROR ", #errno, " (", #sqlstate, "): ", #text);
SELECT #full_error;
INSERT INTO ERRORS_TABLE (Level,Code,Message,INSERT_TMST) VALUES ("ERROR ", #errno, #full_error, now());
END;
No event or stored procedure fail log or MySQL error log found.
also checked for null values.
If I call the stored procedure manually for that missed minute record set manually then it returns the correct count.
Can you please help me how can I debug this issue, any other perfect select query to fetch 2 minutes record or i am missing something ?
Thanks in advance
You simply cannot rely on events running precisely on time. You just can't. You have learned what happens if you do.
You're trying to create a so-called materialized view of your data in your summary table. What can you do about this? A few different things.
You can use an ordinary, non-materialized, VIEW of your data. If you have the correct indexes on your table the VIEW will most likely perform well. And it's a robust solution.
You can rewrite the stored code in your event so it handles everything since the last event ran. To do that you may need a tiny one-row table with the TIMESTAMP used in the previous run.
Switch to a RDBMS that supports materialized views natively. That's probably Oracle, so it will cost a fortune.
I am writing a web application and came across the problem to keep a value in column that is identical in two rows only and both goes in single batch of execution. One way I came up with solution to read the MAX value in the column and increment by 1. Thus, I end-up writing the procedure to lock the table so that other user should not get the dirty read of MAX value.
Create table D(Id int , Name varchar(100))
Begin Tran
DECLARE #i int = (SELECT MAX(ID) FROM D with (tablockx, holdlock))
Print #i ;
Insert into D values ((#i + 1), 'ANAS')
SELECT * FROM D
--COMMIT
Rollback
This code lock the table until query commits or rollback. I have two question from this 1) Is this code guarantee to have exclusive lock on table? 2) In my quick read tablockx can help perform lock for read also whereas holdlock help to prevent the changes in row I am working in locked session, but is this rightly used because I think holdlock may actually not required
Not sure you are asking the right question. You need the current value not to be changed and no insert. With SERIALIZABLE you may not need (updlock).
I am not positive about this answer. You should do your own testing.
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
Begin Tran
DECLARE #i int = (SELECT MAX(ID) FROM D with (updlock))
Print #i ;
Insert into D values ((#i + 1), 'ANAS')
SELECT * FROM D
--COMMIT
Rollback
identity or sequence number are better approaches
I am building a "poor man's queuing system" using MySQL. It's a single table containing jobs that need to be executed (the table name is queue). I have several processes on multiple machines whose job it is to call the fetch_next2 sproc to get an item off of the queue.
The whole point of this procedure is to make sure that we never let 2 clients get the same job. I thought that by using the SELECT .. LIMIT 1 FOR UPDATE would allow me to lock a single row so that I could be sure it was only updated by 1 caller (updated such that it no longer fit the criteria of the SELECT being used to filter jobs that are "READY" to be processed).
Can anyone tell me what I'm doing wrong? I just had some instances where the same job was given to 2 different processes so I know it doesn't work properly. :)
CREATE DEFINER=`masteruser`#`%` PROCEDURE `fetch_next2`()
BEGIN
SET #id = (SELECT q.Id FROM queue q WHERE q.State = 'READY' LIMIT 1 FOR UPDATE);
UPDATE queue
SET State = 'PROCESSING', Attempts = Attempts + 1
WHERE Id = #id;
SELECT Id, Payload
FROM queue
WHERE Id = #id;
END
Code for the answer:
CREATE DEFINER=`masteruser`#`%` PROCEDURE `fetch_next2`()
BEGIN
SET #id := 0;
UPDATE queue SET State='PROCESSING', Id=(SELECT #id := Id) WHERE State='READY' LIMIT 1;
#You can do an if #id!=0 here
SELECT Id, Payload
FROM queue
WHERE Id = #id;
END
The problem with what you are doing is that there is no atomic grouping for the operations. You are using the SELECT ... FOR UPDATE syntax. The Docs say that it blocks "from reading the data in certain transaction isolation levels". But not all levels (I think). Between your first SELECT and UPDATE, another SELECT can occur from another thread. Are you using MyISAM or InnoDB? MyISAM might not support it.
The easiest way to make sure this works properly is to lock the table.
[Edit] The method I describe right here is more time consuming than using the Id=(SELECT #id := Id) method in the above code.
Another method would be to do the following:
Have a column that is normally set to 0.
Do an "UPDATE ... SET ColName=UNIQ_ID WHERE ColName=0 LIMIT 1. That will make sure only 1 process can update that row, and then get it via a SELECT afterwards. (UNIQ_ID is not a MySQL feature, just a variable)
If you need a unique ID, you can use a table with auto_increment just for that.
You can also kind of do this with transactions. If you start a transaction on a table, run UPDATE foobar SET LockVar=19 WHERE LockVar=0 LIMIT 1; from one thread, and do the exact same thing on another thread, the second thread will wait for the first thread to commit before it gets its row. That may end up being a complete table blocking operation though.
MySQL provides an automatic mechanism to increment record IDs. This is OK for many purposes, but I need to be able to use sequences as offered by ORACLE. Obviously, there is no point in creating a table for that purpose.
The solution SHOULD be simple:
1) Create a table to hosts all the needed sequences,
2) Create a function that increases the value of a specific sequence and returns the new value,
3) Create a function that returns the current value of a sequence.
In theory, it looks simple... BUT...
When increasing the value of a sequence (much the same as nextval in Oracle), you need to prevent other sessions to perform this operation (or even fetch the current value) till the updated is completed.
Two theoretical options:
a - Use an UPDATE statement that would return the new value in a single shot, or
b - Lock the table between the UPDATE and SELECT.
Unfortunately, it would appear that MySQL does not allow to lock tables within functions / procedures, and while trying to make the whole thing in a single statement (like UPDATE... RETURNING...) you must use #-type variables which survive the completion of the function/procedure.
Does anyone have an idea/working solution for this?
Thanks.
The following is a simple example with a FOR UPDATE intention lock. A row-level lock with the INNODB engine. The sample shows four rows for next available sequences that will not suffer from the well-known INNODB Gap Anomaly (the case where gaps occur after failed usage of an AUTO_INCREMENT).
Schema:
-- drop table if exists sequences;
create table sequences
( id int auto_increment primary key,
sectionType varchar(200) not null,
nextSequence int not null,
unique key(sectionType)
) ENGINE=InnoDB;
-- truncate table sequences;
insert sequences (sectionType,nextSequence) values
('Chassis',1),('Engine Block',1),('Brakes',1),('Carburetor',1);
Sample code:
START TRANSACTION; -- Line1
SELECT nextSequence into #mine_to_use from sequences where sectionType='Carburetor' FOR UPDATE; -- Line2
select #mine_to_use; -- Line3
UPDATE sequences set nextSequence=nextSequence+1 where sectionType='Carburetor'; -- Line4
COMMIT; -- Line5
Ideally you do not have a Line3 or bloaty code at all which would delay other clients on a Lock Wait. Meaning, get your next sequence to use, perform the update (the incrementing part), and COMMIT, ASAP.
The above in a stored procedure:
DROP PROCEDURE if exists getNextSequence;
DELIMITER $$
CREATE PROCEDURE getNextSequence(p_sectionType varchar(200),OUT p_YoursToUse int)
BEGIN
-- for flexibility, return the sequence number as both an OUT parameter and a single row resultset
START TRANSACTION;
SELECT nextSequence into #mine_to_use from sequences where sectionType=p_sectionType FOR UPDATE;
UPDATE sequences set nextSequence=nextSequence+1 where sectionType=p_sectionType;
COMMIT; -- get it and release INTENTION LOCK ASAP
set p_YoursToUse=#mine_to_use; -- set the OUT parameter
select #mine_to_use as yourSeqNum; -- also return as a 1 column, 1 row resultset
END$$
DELIMITER ;
Test:
set #myNum:= -1;
call getNextSequence('Carburetor',#myNum);
+------------+
| yourSeqNum |
+------------+
| 4 |
+------------+
select #myNum; -- 4
Modify the stored procedure accordingly for you needs, such as having only 1 of the 2 mechanisms for retrieving the sequence number (either the OUT parameter or the result set). In other words, it is easy to ditch the OUT parameter concept.
If you do not adhere to ASAP release of the LOCK (which obviously is not needed after the update), and proceed to perform time consuming code, prior to the release, then the following can occur after a timeout period for other clients awaiting a sequence number:
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting
transaction
Hopefully this is never an issue.
show variables where variable_name='innodb_lock_wait_timeout';
MySQL Manual Page for innodb_lock_wait_timeout.
On my system at the moment it has a value of 50 (seconds). A wait of more than a second or two is probably unbearable in most situations.
Also of interest during TRANSACTIONS is that section of the output from the following command:
SHOW ENGINE INNODB STATUS;