We're using MySQL with InnoDB storage engine and transactions a lot, and we've run into a problem: we need a nice way to emulate Oracle's SEQUENCEs in MySQL. The requirements are:
- concurrency support
- transaction safety
- max performance (meaning minimizing locks and deadlocks)
We don't care if some of the values won't be used, i.e. gaps in sequence are ok. There is an easy way to archieve that by creating a separate InnoDB table with a counter, however this means it will take part in transaction and will introduce locks and waiting. I am thinking to try a MyISAM table with manual locks, any other ideas or best practices?
If auto-increment isn't good enough for your needs, you can create a atomic sequence mechanism with n named sequences like this:
Create a table to store your sequences:
CREATE TABLE sequence (
seq_name varchar(20) unique not null,
seq_current int unsigned not null
);
Assuming you have a row for 'foo' in the table you can atomically get the next sequence id like this:
UPDATE sequence SET seq_current = (#next := seq_current + 1) WHERE seq_name = 'foo';
SELECT #next;
No locks required. Both statements need to be executed in the same session, so that the local variable #next is actually defined when the select happens.
The right way to do this is given in the MySQL manual:
UPDATE child_codes SET counter_field = LAST_INSERT_ID(counter_field + 1);
SELECT LAST_INSERT_ID();
We are a high transaction gaming company and need these sort of solutions for our needs. One of the features of Oracle sequences was also the increment value that could also be set.
The solution uses DUPLICATE KEY.
CREATE TABLE sequences (
id BIGINT DEFAULT 1,
name CHAR(20),
increment TINYINT,
UNIQUE KEY(name)
);
To get the next index:
Abstract the following with a stored procedure or a function sp_seq_next_val(VARCHAR):
INSERT INTO sequences (name) VALUES ("user_id") ON DUPLICATE KEY UPDATE id = id + increment;<br/>
SELECT id FROM sequences WHERE name = "user_id";
Won't the MySQL Identity column on the table handle this?
CREATE TABLE table_name
(
id INTEGER AUTO_INCREMENT PRIMARY KEY
)
Or are you looking to use it for something other than just inserting into another table?
If you're writing using a procedural language as well (instead of just SQL) then the other option would be to create a table containing a single integer (or long integer) value and a stored procedure which locked it, selected from it, incremented it and unlocked it before returning the value.
(Note - always increment before you return the value - it maximise the chance of not getting duplicates if there are errors - or wrap the whole thing in a transaction.)
You would then call this independently of your main insert / update (so it doesn't get caught in any transactions automatically created by the calling mechanism) and then pass it as a parameter to wherever you want to use it.
Because it's independent of the rest of the stuff you're doing it should be quick and avoid locking issues. Even if you did see an error caused by locking (unlikely unless you're overloading the database) you could just call it a second / third time.
Related
I have a table that looks something like:
name: posts
columns:
- id
- sequence_id
- text
- like_count
The ID is a standard auto-incremented unique integer index.
The sequence ID should be similar — it is also a unique integer index.
The difference is that I want to increment it to the new maximum value in the table on update or insert not just insert.
Currently I accomplish this with a Redis counter that I increment before inserting into the database.
I’d like to drop the Redis dependency, though, and do this with purely MySQL if possible.
One option I thought of was creating a post_updates table which just has an auto-incrementing ID which I use the same way but this feels worse.
Another option is doing a full column scan to do max(sequence_id) + 1 but that isn’t really scalable and it would have race conditions.
Are there some better options I’m not aware of?
There's a solution in the manual to simulate a sequence object in MySQL:
CREATE TABLE sequence (id INT NOT NULL);
INSERT INTO sequence VALUES (0);
The sequence table doesn't need an auto-increment itself, and it stores only one row.
When you are ready to increment your sequence_id in the table you describe, you first update the sequence value in the following manner:
UPDATE sequence SET id = LAST_INSERT_ID(id+1);
Now you can use that value when inserting/updating your table:
INSERT INTO posts SET sequence_id = LAST_INSERT_ID(), text = '...';
or
UPDATE posts SET sequence_id = LAST_INSERT_ID(), like_count = like_count+1;
Note that this is a little bit more heavy than an auto-increment, because updating the sequence table creates a row lock, not just an auto-increment lock. This could put an upper limit on the rate of traffic, because many concurrent clients trying to access this table would queue up on each other.
If you want a very high-throughput solution, I'd recommend to continue using Redis.
I was developing a database in SQL Server where I was using an identity column as a seed for a primary key field. The intention was to reset the identity to 1 at the beginning of every year. This would allow us to create a PK of the Year - Identity Column.
Create Table Issues (
IssueID AS RIGHT(CONVERT(VARCHAR, Year(getdate()), 4),2) + '-' + RIGHT(REPLICATE('0', 2) +
CONVERT(VARCHAR, RecordID),3) NOT NULL PRIMARY KEY,
RecordID int Identity (1,1),.........)
The result would be
IssueID RecordID
20-001 1
20-002 2
20-003 3
21-001 1
etc....
Now I've been told we are going to use a MySQL database instead.
Can an Auto-Increment field in MySQL contain duplicate values like it can in SQL Server?
If Not, how can I do what I need to do in MySQL?
In MySQL, you can't use the default auto-increment feature for what you describe, a incrementing value that starts over per year.
This was a feature of the MyISAM storage engine years ago. An auto-increment that was the second column of a multi-column primary key would start counting from one for each distinct value in the first column of the PK. See the example under "MyISAM Notes" on this page: https://dev.mysql.com/doc/refman/8.0/en/example-auto-increment.html
But it's considered not a good idea to use MyISAM because it does not support ACID. In general, I would find another way of solving this task. I would not use MyISAM.
In InnoDB, there's no way the table will generate a value that is a duplicate of a value currently in the table, or even a value less than the maximum value previously generated for that table. In other words, there's no way to "fill in the gaps" using auto-increment.
You can use ALTER TABLE mytable AUTO_INCREMENT=1 to reset the counter, but the value you set it will automatically advance to the max value currently in the table + 1.
So you'll have to generate it using either another table, or else something other than the MySQL database. For example, I've seen some people use memcached, which supports an atomic "increment and return counter" operation.
Another thing to consider: If you need a row counter per year, this is actually different from using MySQL's auto-increment feature. It's not easy to use the latter as a row counter. Besides, what happens if you roll back a transaction or delete a row? You'd end up with non-consecutive RecordId values, with unexplained "gaps." It's also a fact about the auto-increment feature that it guarantees that subsequent id's will be greater, but it does not guarantee to generate all values consecutively. So you'll get gaps eventually anyway.
In MySQL a table can have only one auto_increment column and this column must be a part of the primary key. See details here.
Technical workaround for your task would be creating of a table with a single auto_increment column, and you can obtain auto_increment value by inserting a record into this table and immediately calling standard MySQL function last_inser_id(). When time comes you should truncate the table - in this case the auto_increment count will be reset.
I have following table similar to Oracle user_sequences.
I have logic of sequence prefix/suffix something, but for simplicity, I'm skipping as matters less here.
create table my_seq(
min_value integer,
Max_value integer,
last_value integer,
increment_by tinyint,
customer_id integer);
Assume in current table there are two records.
insert into my_seq(min_value,max_value,last_value,increment_by,customer_id)
values(1,99999999,1,1,'foo#',1),(1,999999999,100,1,'foo#',2);
My foo table structure is like,
create table foo(id Auto_increment,foo_number varchar(20),customer_id integer);
Constrained:
I can't use MySQL AUTO_INCREMENT columns as foo contains different customers data, and every customer could opt foo_number auto generation or manual entry and there should be gap if customer opted for auto_generation. So customer=1 has opted for it, foo# should be 1,2,3,4 etc, no gaps are allowed.
So far so good, with auto increment logic that we have implemented if my app runs in single thread. We generate foo_number and populate in foo table, along with other data points.
I simply do a query to get the next auto#.
select last_number from my_seq where customer_id=?;
reads the # and the update the record.
update my_seq set last_number=last_number+increment_by where customer_id=?;
Problem:
When multiple concurrent session tries the run select last_number from my_seq..., it returns same foo_number multiple times. Also, I can't enforce single thread in application because of application side limitation and performance bottleneck, hence need to solve it in database side.
Please suggest, how I could avoid duplicate numbers? Please help, thanks in advance.
I did google, many stackoverflow links suggests get_last_id(), as you could see, I can't use it.
I was able to solve this problem by just combining suggestions of #Akina and #RickJames , thank you both for thier support.
create table my_seq(
min_value integer,
Max_value integer,
last_value integer,
increment_by tinyint,
customer_id integer)ENGINE = InnoDB;
Here ENGINE=InnoDB is very important.
In order to make sure there is table level locking while reading, I have modified my app code to:
Auto-Commit=FALSE
Then,
//very import to begin the transaction
begin;
select last_number from my_seq where customer_id=? FOR UPDATE;
Read the result in App.
update my_seq set last_number=last_number+1 where customer_id=?;
commit;
This was generating the unique sequence number even in case of multiple concurrent sessions.
I have faced another problem, that this solution has slowed down other are where I do generate sequence#. I have solved it enabling a row level lock instead of table level lock by indexing customer_id.
ALTER TABLE TABLE_NAME ADD INDEX (customer_id);
Hope this will be help full to others.
In MySQL, I am using an InnoDB table that contains unique names, and IDs for those names. Clients need to atomically check for an existing name, insert a new one if it does not exist, and get the ID. The ID is an AUTO_INCREMENT value, and it must not increment out-of-control when checking for existing values regardless of the setting of "innodb_autoinc_lock_mode"; this is because very often the same name will be checked (e.g. "Alice"), and every now and then some new name will come along (e.g. "Bob").
The "INSERT...ON DUPLICATE KEY UPDATE" statement causes an AUTO_INCREMENT increase even in the duplicate-key case, depending on "innodb_autoinc_lock_mode", and is thus unacceptable. The ID will be used as the target of a Foreign-Key Constraint (in another table), and thus it is not okay to change existing IDs. Clients must not deadlock when they do this action concurrently, regardless of how the operations might be interleaved.
I would like the processing during the atomic operation (e.g. checking for the existing ID and deciding whether or not to do the insert) to be done on the server-side rather than the client-side, so that the delay for other sessions attempting to do the same thing simultaneously is minimal and does not need to wait for client-side processing.
My test table to demonstrate this is named FirstNames:
CREATE TABLE `FirstNames` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`FirstName` varchar(45) COLLATE utf8mb4_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `FirstName_UNIQUE` (`FirstName`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
The best solution that I have come up with thus far is as follows:
COMMIT;
SET #myName='Alice';
SET #curId=NULL;
SET autocommit=0;
LOCK TABLES FirstNames WRITE;
SELECT Id INTO #curId FROM FirstNames WHERE FirstName = #myName;
INSERT INTO `FirstNames` (`FirstName`) SELECT #myName FROM DUAL WHERE #curId IS NULL;
COMMIT;
UNLOCK TABLES;
SET #curId=IF(#curId IS NULL, LAST_INSERT_ID(), #curId);
SELECT #curId;
This uses "LOCK TABLES...WRITE" following the instructions given in the MySQL "Interaction of Table Locking and Transactions" documentation for the correct way to lock InnoDB tables. This solution requires the user to have the "LOCK TABLES" privilege.
If I run the above query with #myName="Alice", I obtain a new ID and then continue to obtain the same ID no matter how many times I run it. If I then run with #myName="Bob", I get another ID with the next AUTO_INCREMENT value, and so on. Checking for a name that already exists does not increase the table's AUTO_INCREMENT value.
I am wondering if there is a better solution to accomplish this, perhaps one that does not require the "LOCK TABLES"/"UNLOCK TABLES" commands and combines more "rudimentary" commands (e.g. "INSERT" and "SELECT") in a more clever way? Or is this the best methodology that MySQL currently has to offer?
Edit
This is not a duplicate of "How to 'insert if not exists' in MySQL?". That question does not address all of the criteria that I stated. The issue of keeping the AUTO_INCREMENT value stable is not resolved there (it is only mentioned in passing).
Many of the answers do not address getting the ID of the existing/inserted record, some of the answers do not provide an atomic operation, and some of the answers have the logic being done on the client-side rather than the server-side. A number of the answers change an existing record, which is not what I'm looking for. I am asking for either a better method to meet all of the criteria stated, or confirmation that my solution is the optimal one with existing MySQL support.
The question is really about how to normalize data when you expect there to be duplicates. And then avoid "burning" ids.
http://mysql.rjweb.org/doc.php/staging_table#normalization discusses a 2-step process and is aimed at mass updates due to high-speed ingestion of rows. It degenerates to a single row, but still requires the 2 steps.
Step 1 INSERTs any new rows, creating new auto_inc ids.
Step 2 pulls back the ids en masse.
Note that the work is best done with autocommit=ON and outside the main transaction that is loading the data. This avoids an extra cause for burning ids, namely potential rollbacks.
You can use a conditional INSERT in a single statement:
INSERT INTO FirstNames (FirstName)
SELECT i.firstName
FROM (SELECT 'Alice' AS firstName) i
WHERE NOT EXISTS (SELECT * FROM FirstNames t WHERE t.FirstName = i.firstName);
The next AUTO_INCREMENT value stays untouched in case of existance. But I can't tell you that would be the case in any (future) version or for every configuration. However, it is not much different from what you did - Just in a single statement and without locking the table.
At this point you can be sure that the name exists and just select the corresponding Id:
SELECT Id FROM FirstNames WHERE FirstName = 'Alice';
I'm using MySQL's AUTO_INCREMENT field and InnoDB to support transactions. I noticed when I rollback the transaction, the AUTO_INCREMENT field is not rollbacked? I found out that it was designed this way but are there any workarounds to this?
It can't work that way. Consider:
program one, you open a transaction and insert into a table FOO which has an autoinc primary key (arbitrarily, we say it gets 557 for its key value).
Program two starts, it opens a transaction and inserts into table FOO getting 558.
Program two inserts into table BAR which has a column which is a foreign key to FOO. So now the 558 is located in both FOO and BAR.
Program two now commits.
Program three starts and generates a report from table FOO. The 558 record is printed.
After that, program one rolls back.
How does the database reclaim the 557 value? Does it go into FOO and decrement all the other primary keys greater than 557? How does it fix BAR? How does it erase the 558 printed on the report program three output?
Oracle's sequence numbers are also independent of transactions for the same reason.
If you can solve this problem in constant time, I'm sure you can make a lot of money in the database field.
Now, if you have a requirement that your auto increment field never have gaps (for auditing purposes, say). Then you cannot rollback your transactions. Instead you need to have a status flag on your records. On first insert, the record's status is "Incomplete" then you start the transaction, do your work and update the status to "compete" (or whatever you need). Then when you commit, the record is live. If the transaction rollsback, the incomplete record is still there for auditing. This will cause you many other headaches but is one way to deal with audit trails.
Let me point out something very important:
You should never depend on the numeric features of autogenerated keys.
That is, other than comparing them for equality (=) or unequality (<>), you should not do anything else. No relational operators (<, >), no sorting by indexes, etc. If you need to sort by "date added", have a "date added" column.
Treat them as apples and oranges: Does it make sense to ask if an apple is the same as an orange? Yes. Does it make sense to ask if an apple is larger than an orange? No. (Actually, it does, but you get my point.)
If you stick to this rule, gaps in the continuity of autogenerated indexes will not cause problems.
I had a client needed the ID to rollback on a table of invoices, where the order must be consecutive
My solution in MySQL was to remove the AUTO-INCREMENT and pull the latest Id from the table, add one (+1) and then insert it manually.
If the table is named "TableA" and the Auto-increment column is "Id"
INSERT INTO TableA (Id, Col2, Col3, Col4, ...)
VALUES (
(SELECT Id FROM TableA t ORDER BY t.Id DESC LIMIT 1)+1,
Col2_Val, Col3_Val, Col4_Val, ...)
Why do you care if it is rolled back? AUTO_INCREMENT key fields are not supposed to have any meaning so you really shouldn't care what value is used.
If you have information you're trying to preserve, perhaps another non-key column is needed.
I do not know of any way to do that. According to the MySQL Documentation, this is expected behavior and will happen with all innodb_autoinc_lock_mode lock modes. The specific text is:
In all lock modes (0, 1, and 2), if a
transaction that generated
auto-increment values rolls back,
those auto-increment values are
“lost.” Once a value is generated for
an auto-increment column, it cannot be
rolled back, whether or not the
“INSERT-like” statement is completed,
and whether or not the containing
transaction is rolled back. Such lost
values are not reused. Thus, there may
be gaps in the values stored in an
AUTO_INCREMENT column of a table.
If you set auto_increment to 1 after a rollback or deletion, on the next insert, MySQL will see that 1 is already used and will instead get the MAX() value and add 1 to it.
This will ensure that if the row with the last value is deleted (or the insert is rolled back), it will be reused.
To set the auto_increment to 1, do something like this:
ALTER TABLE tbl auto_increment = 1
This is not as efficient as simply continuing on with the next number because MAX() can be expensive, but if you delete/rollback infrequently and are obsessed with reusing the highest value, then this is a realistic approach.
Be aware that this does not prevent gaps from records deleted in the middle or if another insert should occur prior to you setting auto_increment back to 1.
INSERT INTO prueba(id)
VALUES (
(SELECT IFNULL( MAX( id ) , 0 )+1 FROM prueba target))
If the table doesn't contain values or zero rows
add target for error mysql type update FROM on SELECT
If you need to have the ids assigned in numerical order with no gaps, then you can't use an autoincrement column. You'll need to define a standard integer column and use a stored procedure that calculates the next number in the insert sequence and inserts the record within a transaction. If the insert fails, then the next time the procedure is called it will recalculate the next id.
Having said that, it is a bad idea to rely on ids being in some particular order with no gaps. If you need to preserve ordering, you should probably timestamp the row on insert (and potentially on update).
Concrete answer to this specific dilemma (which I also had) is the following:
1) Create a table that holds different counters for different documents (invoices, receipts, RMA's, etc..); Insert a record for each of your documents and add the initial counter to 0.
2) Before creating a new document, do the following (for invoices, for example):
UPDATE document_counters SET counter = LAST_INSERT_ID(counter + 1) where type = 'invoice'
3) Get the last value that you just updated to, like so:
SELECT LAST_INSERT_ID()
or just use your PHP (or whatever) mysql_insert_id() function to get the same thing
4) Insert your new record along with the primary ID that you just got back from the DB. This will override the current auto increment index, and make sure you have no ID gaps between you records.
This whole thing needs to be wrapped inside a transaction, of course. The beauty of this method is that, when you rollback a transaction, your UPDATE statement from Step 2 will be rolled back, and the counter will not change anymore. Other concurrent transactions will block until the first transaction is either committed or rolled back so they will not have access to either the old counter OR a new one, until all other transactions are finished first.
SOLUTION:
Let's use 'tbl_test' as an example table, and suppose the field 'Id' has AUTO_INCREMENT attribute
CREATE TABLE tbl_test (
Id int NOT NULL AUTO_INCREMENT ,
Name varchar(255) NULL ,
PRIMARY KEY (`Id`)
)
;
Let's suppose that table has houndred or thousand rows already inserted and you don't want to use AUTO_INCREMENT anymore; because when you rollback a transaction the field 'Id' is always adding +1 to AUTO_INCREMENT value.
So to avoid that you might make this:
Let's remove AUTO_INCREMENT value from column 'Id' (this won't delete your inserted rows):
ALTER TABLE tbl_test MODIFY COLUMN Id int(11) NOT NULL FIRST;
Finally, we create a BEFORE INSERT Trigger to generate an 'Id' value automatically. But using this way won't affect your Id value even if you rollback any transaction.
CREATE TRIGGER trg_tbl_test_1
BEFORE INSERT ON tbl_test
FOR EACH ROW
BEGIN
SET NEW.Id= COALESCE((SELECT MAX(Id) FROM tbl_test),0) + 1;
END;
That's it! You're done!
You're welcome.
$masterConn = mysql_connect("localhost", "root", '');
mysql_select_db("sample", $masterConn);
for($i=1; $i<=10; $i++) {
mysql_query("START TRANSACTION",$masterConn);
$qry_insert = "INSERT INTO `customer` (id, `a`, `b`) VALUES (NULL, '$i', 'a')";
mysql_query($qry_insert,$masterConn);
if($i%2==1) mysql_query("COMMIT",$masterConn);
else mysql_query("ROLLBACK",$masterConn);
mysql_query("ALTER TABLE customer auto_increment = 1",$masterConn);
}
echo "Done";