Intermittently getting old data from mysql innodb queries - mysql

We have various tables to represent various types of data. Each table has a corresponding revisions table to track history of this data. Each revision (entry in a revisions table) has a unique number. This number is stored in a change metadata table. Each of these tables references a parent_id. Before we make any changes to the tables we lock the parent row with SELECT … FOR UPDATE.
After making an update/insert we also increment the change number and write that number to the change metadata table. To do so we do a SELECT MAX on the change metadata number and then increment it.
The issue we’re seeing is that somehow a transaction is getting an old change number from the select max statement. To illustrate:
Transaction 1:
START TRANSACTION
lock with FOR UPDATE
do stuff...
Get Latest Change Number (9)
Insert Revision with Number 10
COMMIT
Transaction 2:
START TRANSACTION
lock with FOR UPDATE
do stuff...
Get Latest Change Number (7)
Insert Revision with Number 8
COMMIT
This causes the revision insert for transaction 2 to fail as the change number is a unique key. I’m leaning towards it being an issue of repeatable reads but I’m not sure how the old data can persist across transactions in such a way. For each transactions there's a START TRANSACTION statement and then immediately the parent id is locked with FOR UPDATE. We have a high traffic site with multiple concurrent transactions. It's possible there are many waiting on the lock at any one time. I'd be happy to clarify any point and would appreciate any insight anyone could offer.

SELECT MAX on the change metadata number
That needs FOR UPDATE, too.
Another approach:
Have a "sequence number generator" table.
CREATE TABLE Sequence (
pk TINYINT NOT NULL,
seq INT UNSIGNED NOT NULL AUTO_INCREMENT,
PRIMARY KEY(pk), -- For ON DUPLICATE KEY UPDATE
INDEX(seq) -- Sufficient for AUTO_INCREMENT
);
The only action (once initialized) should be
INSERT INTO Sequence (pk, seq) VALUE (1, 0)
ON DUPLICATE KEY UPDATE seq := LAST_INSERT_ID(seq+1);
That will update the one row atomically. Then (in the same connection), do this to get the new seq:
SELECT LAST_INSERT_ID();
That statement is tied to the connection, so there is no chance of someone else getting your number.

Related

Prevent duplicate increment values during parallel transactions

I use transactions in MySQL to store orders. Each order has OrderID (BIGINT), which looks like this: XXXXXX0001, with last four digits incrementing (1620200001, 1620200002, 1620200003, ...).
The transaction works as follows:
start transaction
get new OrderID (increment by 1)
do some stuff
commit/rollback
Saving the transaction can take up to several seconds and if multiple orders are created in very short timespan, duplicate OrderID's can be inserted into database. Before first order is commited, second is assigned the same OrderID, which, at the moment is next in line.
What is best way to prevent this? Having UNIQUE OrderID does not solve it (there would be rollback in second order). I could get rid of transaction and save OrderID quicker, but this leads to other potential problems and does not entirely solve this (just reduces chances of problem happening).
Any help would be appreciated.
Read about AUTO_INCREMENT. Search for it in the manual on CREATE TABLE. It's a long page, but AUTO_INCREMENT is documented about 1/4 of the way down the page.
Briefly, you just declare the primary key with a column option:
CREATE TABLE mytable (
id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY,
...other columns...
);
The initial value is 1, or you can make it start at a higher value:
ALTER TABLE mytable AUTO_INCREMENT=1620200001;
A table with an auto-increment column ensures that each concurrent transaction gets a unique, increasing value. There is no race condition, because the INSERT acquires a brief table-lock during which it increments the value. Unlike transaction-based locks, the auto-increment table lock is released immediately. So concurrent sessions don't have to wait for your transaction to finish.
Auto-increment is guaranteed to be unique. That is, the same value will not be allocated to multiple sessions. However, it's not guaranteed to allocate consecutive values. Also, it may allocate a value to one session, but that session decides to rollback its transaction. The value it had allocated is NOT returned to any kind of queue of values, because there has probably other sessions that have allocated the next few values in the meantime. So it's possible to "lose" values and then your table has "gaps" or non-consecutive values.
Do not worry about gaps. These could also happen even if the values were consecutive, because you might delete a row later.

MySQL concurrency and auto_incrementing key

I have a MySQL table of Users, and a table of Actions performed by the Users (linked to that User by a the primary key, userid ). The Actions table has an incrementing key indx. Whenever I add a new row to that table, I then update the latest column of the relevant Users row with the indx of the row I just added to the Actions table. So something like:
INSERT INTO actions(indx,actionname,userid) VALUES(default, "myaction", 1);
UPDATE users SET latest=LAST_INSERT_ID() WHERE userid=1;
The idea being that I can check for updates for a User by seeing if the latest is higher then the last time I checked.
My issue is that if more than one connection is opened on the database and they try and add an Action for the same User at the same time, connection2 could conceivably run their INSERT and UPDATE between the INSERT and update of connection1, and the latest entry of the user they're both trying to update will no longer have the indx of the most recent action entry.
I've been reading up on transaction, isolation levels, etc. But haven't really found a way around this (though my understanding of how these work exactly is pretty shaky, so maybe I just misunderstood). I think I need a way to lock the Actions table until the User table is updated. This application only gets used by a few hundred users tops, so I don't think the performance hit due to momentarily locking the table will be too bad.
So is that something that can be done in MySQL? Is there a better solution? I imagine this general pattern must be pretty common: having one table with a bunch of varieties of rows, and a second table with a row that tracks meta data for each variety in table A and needs to be updated atomically each time that first table is changed. So I'm hoping there's a solution that isn't too complex
Use SELECT ... FOR UPDATE to lock the row in order to serialize the access to the table and prevent from race conditions:
START TRANSACTION;
SELECT any_column FROM users WHERE userid=1 FOR UPDATE;
INSERT INTO actions(indx,actionname,userid) VALUES(default, "myaction", 1);
UPDATE users SET latest=LATEST_INSERT_ID() WHERE userid=1;
COMMIT;
However this will slown down your INSERTing rate, because all these transactions from all sessions will be serialized.
The better option is to not store the last ID in users table at all. Just use SELECT max( id ) FROM actions WHERE userid = xxxx in all places where this number is required. With an index on actions( userid ) this query will be very fast (assuming that id column is the primary key in this table), and the inserts will not be slowed down

What is the correct implementation of handling concurrent SQL inserts while ensuring a non-primary key column (i.e. wid) unique and auto-incremental?

What is the correct implementation of handling concurrent SQL inserts while ensuring a non-primary key column (i.e. wid) unique and auto-incremental?
Note: There was a design constraint that led this field (i.e. wid) in not becoming the primary key. A separate auto-increment PK is included in the column called id. Changing the wid to PK is not feasible as the database design is now heavily coupled to multiple software projects. So I need to devise a workaround.
I'm thinking of 3 possible ways:
1.) Reserving the wid by executing a transaction and proceeding with other SQL statements which should have been included in the same transaction. The reservation is executed in a separate transaction to make the locking as short as possible. However, if the subsequent transaction failed (e.g. due to database connection issues), then the invalidated row containing the reserved wid will remain in the table.
START TRANSACTION;
SET #wid_d = 0;
SELECT COALESCE(MAX(`wid`), 0) INTO #wid_d FROM table
WHERE `wid >= 0 AND `wid <= 1000 FOR UPDATE;
INSERT INTO table (`wid`) VALUES (IF (#wid_d = 0, 1, #wid_d+1) )
COMMIT;
2.) Including the reservation of wid in the entire transaction to eliminate the problem of insertion of invalidated row, should the transaction fail but this approach prolongs the locking of tables.
3.) Rely on the failure of transaction calls for re-executing the transaction until it performs successfully; should the transaction fail due to the race condition of inserting the same MAX(wid) + 1. This approach will prevent insertion of invalidated rows but the problem is the number of times the transaction should be re-run. There is also no distinction of knowing the problem that is caused by the conflicting wid or other problems, hence, this is a problematic approach.
Well, depending on how brave you are you have other options, like create separate sequence table that will be used to maintain WID values:
CREATE TABLE wid_seq (wid INT PRIMARY KEY AUTO_INCREMENT);
and in your transaction:
START TRANSACTION;
INSERT INTO wid_seq VALUES (null);
INSERT yourtable (WID) values (LAST_INSERT_ID());
COMMIT;

MySQL Auto Increment Columns on TRANSACTION, COMMIT, and ROLLBACK

When using MySQL START TRANSACTION and the decision is made by MySQL to roll back -
In the case that a table had an AUTO_INCREMENT column - does the column get... decremented during the roll back?
Or should it?
I am having some issues where the transaction data is being properly rolled back - but it looks like the table was auto incremented and not decremented in the rollback.
# BOTH TABLES START OUT EMPTY // TABLE1 ID is **auto_increment**
START TRANSACTION;
INSERT INTO `TABLE1` (`ID` ,`NAME`) VALUES (NULL , 'Ted'); # MySQL TABLE1 **ID** is Auto incremented to 1
INSERT INTO `TABLE2` (`ID` ,`WRONGVALUE`) VALUES (NULL , 'some value'); # error. This TRANSACTION will not succeed
COMMIT; # Because of the error - this TRANSACTION is now rolled back and Ted is NOT added
Because MySQL will auto_increment the ID on the first table - regardless of if the transaction succeeds or fails - is the standard practice for this to decrement the table yourself?
No, auto-increment mechanisms must work outside the scope of transactions, because another user may be inserting to the same table before you finish your transaction. The other user's transaction must be able to allocate the next value, before knowing whether your transaction is using the value you just allocated.
Re your comment: If I may say it more clearly, any change in the scope of a transaction may be rolled back. The auto-increment counter is not rolled back, so it doesn't obey atomicity of transactions. Nor does it obey isolation because another transaction gets the next value even though your transaction hasn't committed yet.
The way auto-increment works means that sometimes, if you insert some rows and then roll back your transaction, the values you allocated with auto-increment are lost forever!
But this is okay. Primary key values must be unique, but they don't need to be consecutive. In other words, they are not row numbers, and you shouldn't use them like that. So you should never need to decrement values created by auto-increment.
Disclosure: I am relatively new to SQL so some of this may be wrong. But this is how I understand it.
Auto increment must function outside of transactions otherwise you can compromise data. Let us say you have 2 users both trying to connect to your dB. Both are creating accounts. When Account 1 is being created it increments up to 1. Then Account 2 is created and it increments to 2. So far it would look like the following:
(1, Account1)
(2, Account2)
Increment: 2.
Now let us say that Account 2 commits, but Account 1 fails. Now your table looks as follows:
(2, Account2)
Increment: 1.
Because you decremented your Auto Increment, you will fail at having a unique value when someone tries to register a new account, because the pair will be:
(2,Account2)
Attempting to Insert (2, NewAccount)
Which will fail because 2, the primary key, will not be unique.
If you need to fix your table (for development purposes) you can always run the following code:
ALTER TABLE `DB_NAME`.`TABLE_NAME`
AUTO_INCREMENT = 1 (or some other number);

MySQL AUTO_INCREMENT does not ROLLBACK

I'm using MySQL's AUTO_INCREMENT field and InnoDB to support transactions. I noticed when I rollback the transaction, the AUTO_INCREMENT field is not rollbacked? I found out that it was designed this way but are there any workarounds to this?
It can't work that way. Consider:
program one, you open a transaction and insert into a table FOO which has an autoinc primary key (arbitrarily, we say it gets 557 for its key value).
Program two starts, it opens a transaction and inserts into table FOO getting 558.
Program two inserts into table BAR which has a column which is a foreign key to FOO. So now the 558 is located in both FOO and BAR.
Program two now commits.
Program three starts and generates a report from table FOO. The 558 record is printed.
After that, program one rolls back.
How does the database reclaim the 557 value? Does it go into FOO and decrement all the other primary keys greater than 557? How does it fix BAR? How does it erase the 558 printed on the report program three output?
Oracle's sequence numbers are also independent of transactions for the same reason.
If you can solve this problem in constant time, I'm sure you can make a lot of money in the database field.
Now, if you have a requirement that your auto increment field never have gaps (for auditing purposes, say). Then you cannot rollback your transactions. Instead you need to have a status flag on your records. On first insert, the record's status is "Incomplete" then you start the transaction, do your work and update the status to "compete" (or whatever you need). Then when you commit, the record is live. If the transaction rollsback, the incomplete record is still there for auditing. This will cause you many other headaches but is one way to deal with audit trails.
Let me point out something very important:
You should never depend on the numeric features of autogenerated keys.
That is, other than comparing them for equality (=) or unequality (<>), you should not do anything else. No relational operators (<, >), no sorting by indexes, etc. If you need to sort by "date added", have a "date added" column.
Treat them as apples and oranges: Does it make sense to ask if an apple is the same as an orange? Yes. Does it make sense to ask if an apple is larger than an orange? No. (Actually, it does, but you get my point.)
If you stick to this rule, gaps in the continuity of autogenerated indexes will not cause problems.
I had a client needed the ID to rollback on a table of invoices, where the order must be consecutive
My solution in MySQL was to remove the AUTO-INCREMENT and pull the latest Id from the table, add one (+1) and then insert it manually.
If the table is named "TableA" and the Auto-increment column is "Id"
INSERT INTO TableA (Id, Col2, Col3, Col4, ...)
VALUES (
(SELECT Id FROM TableA t ORDER BY t.Id DESC LIMIT 1)+1,
Col2_Val, Col3_Val, Col4_Val, ...)
Why do you care if it is rolled back? AUTO_INCREMENT key fields are not supposed to have any meaning so you really shouldn't care what value is used.
If you have information you're trying to preserve, perhaps another non-key column is needed.
I do not know of any way to do that. According to the MySQL Documentation, this is expected behavior and will happen with all innodb_autoinc_lock_mode lock modes. The specific text is:
In all lock modes (0, 1, and 2), if a
transaction that generated
auto-increment values rolls back,
those auto-increment values are
“lost.” Once a value is generated for
an auto-increment column, it cannot be
rolled back, whether or not the
“INSERT-like” statement is completed,
and whether or not the containing
transaction is rolled back. Such lost
values are not reused. Thus, there may
be gaps in the values stored in an
AUTO_INCREMENT column of a table.
If you set auto_increment to 1 after a rollback or deletion, on the next insert, MySQL will see that 1 is already used and will instead get the MAX() value and add 1 to it.
This will ensure that if the row with the last value is deleted (or the insert is rolled back), it will be reused.
To set the auto_increment to 1, do something like this:
ALTER TABLE tbl auto_increment = 1
This is not as efficient as simply continuing on with the next number because MAX() can be expensive, but if you delete/rollback infrequently and are obsessed with reusing the highest value, then this is a realistic approach.
Be aware that this does not prevent gaps from records deleted in the middle or if another insert should occur prior to you setting auto_increment back to 1.
INSERT INTO prueba(id)
VALUES (
(SELECT IFNULL( MAX( id ) , 0 )+1 FROM prueba target))
If the table doesn't contain values or zero rows
add target for error mysql type update FROM on SELECT
If you need to have the ids assigned in numerical order with no gaps, then you can't use an autoincrement column. You'll need to define a standard integer column and use a stored procedure that calculates the next number in the insert sequence and inserts the record within a transaction. If the insert fails, then the next time the procedure is called it will recalculate the next id.
Having said that, it is a bad idea to rely on ids being in some particular order with no gaps. If you need to preserve ordering, you should probably timestamp the row on insert (and potentially on update).
Concrete answer to this specific dilemma (which I also had) is the following:
1) Create a table that holds different counters for different documents (invoices, receipts, RMA's, etc..); Insert a record for each of your documents and add the initial counter to 0.
2) Before creating a new document, do the following (for invoices, for example):
UPDATE document_counters SET counter = LAST_INSERT_ID(counter + 1) where type = 'invoice'
3) Get the last value that you just updated to, like so:
SELECT LAST_INSERT_ID()
or just use your PHP (or whatever) mysql_insert_id() function to get the same thing
4) Insert your new record along with the primary ID that you just got back from the DB. This will override the current auto increment index, and make sure you have no ID gaps between you records.
This whole thing needs to be wrapped inside a transaction, of course. The beauty of this method is that, when you rollback a transaction, your UPDATE statement from Step 2 will be rolled back, and the counter will not change anymore. Other concurrent transactions will block until the first transaction is either committed or rolled back so they will not have access to either the old counter OR a new one, until all other transactions are finished first.
SOLUTION:
Let's use 'tbl_test' as an example table, and suppose the field 'Id' has AUTO_INCREMENT attribute
CREATE TABLE tbl_test (
Id int NOT NULL AUTO_INCREMENT ,
Name varchar(255) NULL ,
PRIMARY KEY (`Id`)
)
;
Let's suppose that table has houndred or thousand rows already inserted and you don't want to use AUTO_INCREMENT anymore; because when you rollback a transaction the field 'Id' is always adding +1 to AUTO_INCREMENT value.
So to avoid that you might make this:
Let's remove AUTO_INCREMENT value from column 'Id' (this won't delete your inserted rows):
ALTER TABLE tbl_test MODIFY COLUMN Id int(11) NOT NULL FIRST;
Finally, we create a BEFORE INSERT Trigger to generate an 'Id' value automatically. But using this way won't affect your Id value even if you rollback any transaction.
CREATE TRIGGER trg_tbl_test_1
BEFORE INSERT ON tbl_test
FOR EACH ROW
BEGIN
SET NEW.Id= COALESCE((SELECT MAX(Id) FROM tbl_test),0) + 1;
END;
That's it! You're done!
You're welcome.
$masterConn = mysql_connect("localhost", "root", '');
mysql_select_db("sample", $masterConn);
for($i=1; $i<=10; $i++) {
mysql_query("START TRANSACTION",$masterConn);
$qry_insert = "INSERT INTO `customer` (id, `a`, `b`) VALUES (NULL, '$i', 'a')";
mysql_query($qry_insert,$masterConn);
if($i%2==1) mysql_query("COMMIT",$masterConn);
else mysql_query("ROLLBACK",$masterConn);
mysql_query("ALTER TABLE customer auto_increment = 1",$masterConn);
}
echo "Done";