I have following table similar to Oracle user_sequences.
I have logic of sequence prefix/suffix something, but for simplicity, I'm skipping as matters less here.
create table my_seq(
min_value integer,
Max_value integer,
last_value integer,
increment_by tinyint,
customer_id integer);
Assume in current table there are two records.
insert into my_seq(min_value,max_value,last_value,increment_by,customer_id)
values(1,99999999,1,1,'foo#',1),(1,999999999,100,1,'foo#',2);
My foo table structure is like,
create table foo(id Auto_increment,foo_number varchar(20),customer_id integer);
Constrained:
I can't use MySQL AUTO_INCREMENT columns as foo contains different customers data, and every customer could opt foo_number auto generation or manual entry and there should be gap if customer opted for auto_generation. So customer=1 has opted for it, foo# should be 1,2,3,4 etc, no gaps are allowed.
So far so good, with auto increment logic that we have implemented if my app runs in single thread. We generate foo_number and populate in foo table, along with other data points.
I simply do a query to get the next auto#.
select last_number from my_seq where customer_id=?;
reads the # and the update the record.
update my_seq set last_number=last_number+increment_by where customer_id=?;
Problem:
When multiple concurrent session tries the run select last_number from my_seq..., it returns same foo_number multiple times. Also, I can't enforce single thread in application because of application side limitation and performance bottleneck, hence need to solve it in database side.
Please suggest, how I could avoid duplicate numbers? Please help, thanks in advance.
I did google, many stackoverflow links suggests get_last_id(), as you could see, I can't use it.
I was able to solve this problem by just combining suggestions of #Akina and #RickJames , thank you both for thier support.
create table my_seq(
min_value integer,
Max_value integer,
last_value integer,
increment_by tinyint,
customer_id integer)ENGINE = InnoDB;
Here ENGINE=InnoDB is very important.
In order to make sure there is table level locking while reading, I have modified my app code to:
Auto-Commit=FALSE
Then,
//very import to begin the transaction
begin;
select last_number from my_seq where customer_id=? FOR UPDATE;
Read the result in App.
update my_seq set last_number=last_number+1 where customer_id=?;
commit;
This was generating the unique sequence number even in case of multiple concurrent sessions.
I have faced another problem, that this solution has slowed down other are where I do generate sequence#. I have solved it enabling a row level lock instead of table level lock by indexing customer_id.
ALTER TABLE TABLE_NAME ADD INDEX (customer_id);
Hope this will be help full to others.
Related
I am trying to build an API and one of the endpoints will return a random row from my database. In the database I have a table in which I want a "views" column to be updated every time I run a SELECT query on a row.
My table looks something like this:
CREATE TABLE `movies` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(256) NOT NULL,
`description` text,
`views` int(11) NOT NULL DEFAULT 0,
PRIMARY KEY (`id`)
);
The row is selected by ordering the table with rand() and then limiting the result by 1, like so:
SELECT * FROM table ORDER BY rand() LIMIT 1;
Is something like this below possible?
SELECT * FROM table ORDER BY rand() LIMIT 1
UPDATE table SET views = +1 WHERE (selected row?);
I'm new to SQL queries, so I don't know if this is the best way or even possible at all. Should I run a new query after this one has completed that updates the value instead?
Usually, every table has a Primary Key, i.e. a unique ID of every single row. Since you have a result of your SELECT query and it's only 1 row, you always can make a consequent update query like UPDATE table SET views = views + 1 WHERE id = <returned_record_id>. Here we assume that the column id is a Primary Key column. This pair of queries need to be issued by the application code. If you want to achieve SELECT + UPDATE functionality as a single SQL statement, consider using stored procedures.
While the aforementioned approach is technically possible, it might have a few performance problems. First of, ORDER BY rand() often has a poor performance. Also, having an update on each select could have bad performance implications.
No what you want is not possible .as, select and update commands can not be used togethor in a single transaction.
You can do it seperately
You need to create a procedure for this in your database like:
CREATE PROCEDURE `procedure_name`()
BEGIN
SELECT * FROM table ORDER BY rand() LIMIT 1 ;
UPDATE table SET views = +1 WHERE (selected row?) ;
END
and then call it
call procedure_name();
You can check only as there are many ways to write a procedure.
Thanks
Unfortunately, what you want to do is not possible, at least not without a lot of work. SQL in general -- and MySQL in particular -- offer a capability called triggers.
Triggers allow you to do take actions when something happens in the database. For instance, if you want to check that values are correct, you can write an insert/update trigger to check the values and reject improper ones. Or, if you want to stash deleted records into an audit table, a trigger is the way to go.
What you are describing could be implemented using a trigger on a "select". Such a beast does not exist.
What are your options? Well, the simplest is to do this in your application. When a movie is selected, then you can update views. Of course, that only increments the views where you have the code.
You can move this code into a stored procedure. This simplifies the application code. It just has to "know" to use the stored procedure. But, there is no enforcement mechanism.
You can make this more enforceable by using permissions. Basically, don't allow access to the underlying table except through the stored procedure. This is closest to what you want.
In MySQL, I am using an InnoDB table that contains unique names, and IDs for those names. Clients need to atomically check for an existing name, insert a new one if it does not exist, and get the ID. The ID is an AUTO_INCREMENT value, and it must not increment out-of-control when checking for existing values regardless of the setting of "innodb_autoinc_lock_mode"; this is because very often the same name will be checked (e.g. "Alice"), and every now and then some new name will come along (e.g. "Bob").
The "INSERT...ON DUPLICATE KEY UPDATE" statement causes an AUTO_INCREMENT increase even in the duplicate-key case, depending on "innodb_autoinc_lock_mode", and is thus unacceptable. The ID will be used as the target of a Foreign-Key Constraint (in another table), and thus it is not okay to change existing IDs. Clients must not deadlock when they do this action concurrently, regardless of how the operations might be interleaved.
I would like the processing during the atomic operation (e.g. checking for the existing ID and deciding whether or not to do the insert) to be done on the server-side rather than the client-side, so that the delay for other sessions attempting to do the same thing simultaneously is minimal and does not need to wait for client-side processing.
My test table to demonstrate this is named FirstNames:
CREATE TABLE `FirstNames` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`FirstName` varchar(45) COLLATE utf8mb4_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `FirstName_UNIQUE` (`FirstName`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
The best solution that I have come up with thus far is as follows:
COMMIT;
SET #myName='Alice';
SET #curId=NULL;
SET autocommit=0;
LOCK TABLES FirstNames WRITE;
SELECT Id INTO #curId FROM FirstNames WHERE FirstName = #myName;
INSERT INTO `FirstNames` (`FirstName`) SELECT #myName FROM DUAL WHERE #curId IS NULL;
COMMIT;
UNLOCK TABLES;
SET #curId=IF(#curId IS NULL, LAST_INSERT_ID(), #curId);
SELECT #curId;
This uses "LOCK TABLES...WRITE" following the instructions given in the MySQL "Interaction of Table Locking and Transactions" documentation for the correct way to lock InnoDB tables. This solution requires the user to have the "LOCK TABLES" privilege.
If I run the above query with #myName="Alice", I obtain a new ID and then continue to obtain the same ID no matter how many times I run it. If I then run with #myName="Bob", I get another ID with the next AUTO_INCREMENT value, and so on. Checking for a name that already exists does not increase the table's AUTO_INCREMENT value.
I am wondering if there is a better solution to accomplish this, perhaps one that does not require the "LOCK TABLES"/"UNLOCK TABLES" commands and combines more "rudimentary" commands (e.g. "INSERT" and "SELECT") in a more clever way? Or is this the best methodology that MySQL currently has to offer?
Edit
This is not a duplicate of "How to 'insert if not exists' in MySQL?". That question does not address all of the criteria that I stated. The issue of keeping the AUTO_INCREMENT value stable is not resolved there (it is only mentioned in passing).
Many of the answers do not address getting the ID of the existing/inserted record, some of the answers do not provide an atomic operation, and some of the answers have the logic being done on the client-side rather than the server-side. A number of the answers change an existing record, which is not what I'm looking for. I am asking for either a better method to meet all of the criteria stated, or confirmation that my solution is the optimal one with existing MySQL support.
The question is really about how to normalize data when you expect there to be duplicates. And then avoid "burning" ids.
http://mysql.rjweb.org/doc.php/staging_table#normalization discusses a 2-step process and is aimed at mass updates due to high-speed ingestion of rows. It degenerates to a single row, but still requires the 2 steps.
Step 1 INSERTs any new rows, creating new auto_inc ids.
Step 2 pulls back the ids en masse.
Note that the work is best done with autocommit=ON and outside the main transaction that is loading the data. This avoids an extra cause for burning ids, namely potential rollbacks.
You can use a conditional INSERT in a single statement:
INSERT INTO FirstNames (FirstName)
SELECT i.firstName
FROM (SELECT 'Alice' AS firstName) i
WHERE NOT EXISTS (SELECT * FROM FirstNames t WHERE t.FirstName = i.firstName);
The next AUTO_INCREMENT value stays untouched in case of existance. But I can't tell you that would be the case in any (future) version or for every configuration. However, it is not much different from what you did - Just in a single statement and without locking the table.
At this point you can be sure that the name exists and just select the corresponding Id:
SELECT Id FROM FirstNames WHERE FirstName = 'Alice';
I've got a bit of a stupid question. The thing is my program has to have the function to delete data from my database. Yay, not really the problem. But how can I delete data without the danger that others can see, that there has been something deleted.
User Table:
U_ID U_NAME
1 Chris
2 Peter
OTHER TABLE
ID TIMESTAMP FK_U_D
1 2012-12-01 1
2 2012-12-02 1
Sooooo the ID's are AUTO_INCREMENT, so if I delete one of them there's a gap. Furthermore, the timestamp is also bigger than the row before, so ascending.
I want to let the data with ID 1 disappear from the user's profile (U_ID 1).
If I delete it, there is a gap. If I just change the FK_U_ID to 2 (Peter) it's obvious, because when I insert data, there are 20 or 30 data rows with the same U_ID...so it's obvious that there has been a modification.
If I set the FK_U_ID NULL --> same sh** like when I change it to another U_ID.
Is there any solution to get this work? I know that if nobody but me has access to the database, it's just no problem. But just in case, if somebody controls my program it should not be obvious that there has been modifications.
So here we go.
For the ID gaps issue you can use GUIDs as #SLaks suggests, but then you can't use the native RDBMS auto_increment which means you have to create the GUID and insert it along with the rest of the record data upon creation. Of course, you don't really need the ID to be globally unique, you could just store a random string of 20 characters or something, but then you have to do a DB read to see if that ID is taken and repeat (recursively) that process until you find an unused ID... could be quite taxing.
It's not at all clear why you would want to "hide" evidence that a delete was performed. That sounds like a really bad idea. I'm not a fan of promulgating misinformation.
Two of the characteristics of an ideal primary key are:
- anonymous (be void of any useful information, doesn't matter what it's set to)
- immutable (once assigned, it will never be changed.)
But, if we set that whole discussion aside...
I can answer a slightly different question (an answer you might find helpful to your particular situation)
The only way to eliminate a "gap" in the values in a column with an AUTO_INCREMENT would be to change the column values from their current values to a contiguous sequence of new values. If there are any foreign keys that reference that column, the values in those columns would need to be updated as well, to preserve the relationship. That will likely leave the current auto_increment value of the table higher than the largest value of the id column, so I'd want to reset that as well, to avoid a "gap" on the next insert.
(I have done re-sequencing of auto_increment values in development and test environments, to "cleanup" lookup tables, and to move the id values of some tables to ranges that are distinct from ranges in other tables... that let's me test SQL to make sure the SQL join predicates aren't inadvertently referencing the wrong table, and returning rows that look correct by accident... those are some reasons I've done reassignment if auto_increment values)
Note that the database can "automagically" update foreign key values (for InnnoDB tables) when you change the primary key value, as long as the foreign key constraint is defined with ON UPDATE CASCADE, and FOREIGN_KEY_CHECKS is not disabled.
If there are no foreign keys to deal with, and assuming that all of the current values of id are positive integers, then I've been able to do something like this: (with appropriate backups in place, so I can recover if things don't work right)
UPDATE mytable t
JOIN (
SELECT s.id AS old_id
, #i := #i + 1 AS new_id
FROM mytable s
CROSS
JOIN (SELECT #i := 0) i
ORDER BY s.id
) c
ON t.id = c.old_id
SET t.id = c.new_id
WHERE t.id <> c.new_id
To reset the table AUTO_INCREMENT back down to the largest id value in the table:
ALTER TABLE mytable AUTO_INCREMENT = 1;
Typically, I will create a table and populate it from that query in the inline view (aliased as c) above. I can then use that table to update both foreign key columns and the primary key column, first disabling the FOREIGN_KEY_CHECKS and then re-enabling it. (In a concurrent environment, where other processes might be inserting/updating/deleting rows from one of the tables, I would of course first obtain an exclusive lock on all of the tables to be updated.)
Taking up again, the discussion I set aside earlier... this type of "administrative" function can be useful in a test environment, when setting up test cases. But it is NOT a function that is ever performed in a production environment, with live data.
I've got a mysql table where each row has its own sequence number in a "sequence" column. However, when a row gets deleted, it leaves a gap. So...
1
2
3
4
...becomes...
1
2
4
Is there a neat way to "reset" the sequencing, so it becomes consecutive again in one SQL query?
Incidentally, I'm sure there is a technical term for this process. Anyone?
UPDATED: The "sequence" column is not a primary key. It is only used for determining the order that records are displayed within the app.
If the field is your primary key...
...then, as stated elsewhere on this question, you shouldn't be changing IDs. The IDs are already unique and you neither need nor want to re-use them.
Now, that said...
Otherwise...
It's quite possible that you have a different field (that is, as well as the PK) for some application-defined ordering. As long as this ordering isn't inherent in some other field (e.g. if it's user-defined), then there is nothing wrong with this.
You could recreate the table using a (temporary) auto_increment field and then remove the auto_increment afterwards.
I'd be tempted to UPDATE in ascending order and apply an incrementing variable.
SET #i = 0;
UPDATE `table`
SET `myOrderCol` = #i:=#i+1
ORDER BY `myOrderCol` ASC;
(Query not tested.)
It does seem quite wasteful to do this every time you delete items, but unfortunately with this manual ordering approach there's not a whole lot you can do about that if you want to maintain the integrity of the column.
You could possibly reduce the load, such that after deleting the entry with myOrderCol equal to, say, 5:
SET #i = 5;
UPDATE `table`
SET `myOrderCol` = #i:=#i+1
WHERE `myOrderCol` > 5
ORDER BY `myOrderCol` ASC;
(Query not tested.)
This will "shuffle" all the following values down by one.
I'd say don't bother. Reassigning sequential values is a relatively expensive operation and if the column value is for ordering purpose only there is no good reason to do that. The only concern you might have is if for example your column is UNSIGNED INT and you suspect that in the lifetime of your application you might have more than 4,294,967,296 rows (including deleted rows) and go out of range, even if that is your concern you can do the reassigning as a one time task 10 years later when that happens.
This is a question that often I read here and in other forums. As already written by zerkms this is a false problem. Moreover if your table is related with other ones you'll lose relations.
Just for learning purpose a simple way is to store your data in a temporary table, truncate the original one (this reset auto_increment) and than repopulate it.
Silly example:
create table seq (
id int not null auto_increment primary key,
col char(1)
) engine = myisam;
insert into seq (col) values ('a'),('b'),('c'),('d');
delete from seq where id = 3;
create temporary table tmp select col from seq order by id;
truncate seq;
insert into seq (col) select * from tmp;
but it's totally useless. ;)
If this is your PK then you shouldn't change it. PKs should be (mostly) unchanging columns. If you were to change them then not only would you need to change it in that table but also in any foreign keys where is exists.
If you do need a sequential sequence then ask yourself why. In a table there is no inherent or guaranteed order (even in the PK, although it may turn out that way because of how most RDBMSs store and retrieve the data). That's why we have the ORDER BY clause in SQL. If you want to be able to generate sequential numbers based on something else (time added into the database, etc.) then consider generating that either in your query or with your front end.
Assuming that this is an ID field, you can do this when you insert:
INSERT INTO yourTable (ID)
SELECT MIN(ID)
FROM yourTable
WHERE ID > 1
As others have mentioned I don't recommend doing this. It will hold a table lock while the next ID is evaluated.
We're using MySQL with InnoDB storage engine and transactions a lot, and we've run into a problem: we need a nice way to emulate Oracle's SEQUENCEs in MySQL. The requirements are:
- concurrency support
- transaction safety
- max performance (meaning minimizing locks and deadlocks)
We don't care if some of the values won't be used, i.e. gaps in sequence are ok. There is an easy way to archieve that by creating a separate InnoDB table with a counter, however this means it will take part in transaction and will introduce locks and waiting. I am thinking to try a MyISAM table with manual locks, any other ideas or best practices?
If auto-increment isn't good enough for your needs, you can create a atomic sequence mechanism with n named sequences like this:
Create a table to store your sequences:
CREATE TABLE sequence (
seq_name varchar(20) unique not null,
seq_current int unsigned not null
);
Assuming you have a row for 'foo' in the table you can atomically get the next sequence id like this:
UPDATE sequence SET seq_current = (#next := seq_current + 1) WHERE seq_name = 'foo';
SELECT #next;
No locks required. Both statements need to be executed in the same session, so that the local variable #next is actually defined when the select happens.
The right way to do this is given in the MySQL manual:
UPDATE child_codes SET counter_field = LAST_INSERT_ID(counter_field + 1);
SELECT LAST_INSERT_ID();
We are a high transaction gaming company and need these sort of solutions for our needs. One of the features of Oracle sequences was also the increment value that could also be set.
The solution uses DUPLICATE KEY.
CREATE TABLE sequences (
id BIGINT DEFAULT 1,
name CHAR(20),
increment TINYINT,
UNIQUE KEY(name)
);
To get the next index:
Abstract the following with a stored procedure or a function sp_seq_next_val(VARCHAR):
INSERT INTO sequences (name) VALUES ("user_id") ON DUPLICATE KEY UPDATE id = id + increment;<br/>
SELECT id FROM sequences WHERE name = "user_id";
Won't the MySQL Identity column on the table handle this?
CREATE TABLE table_name
(
id INTEGER AUTO_INCREMENT PRIMARY KEY
)
Or are you looking to use it for something other than just inserting into another table?
If you're writing using a procedural language as well (instead of just SQL) then the other option would be to create a table containing a single integer (or long integer) value and a stored procedure which locked it, selected from it, incremented it and unlocked it before returning the value.
(Note - always increment before you return the value - it maximise the chance of not getting duplicates if there are errors - or wrap the whole thing in a transaction.)
You would then call this independently of your main insert / update (so it doesn't get caught in any transactions automatically created by the calling mechanism) and then pass it as a parameter to wherever you want to use it.
Because it's independent of the rest of the stuff you're doing it should be quick and avoid locking issues. Even if you did see an error caused by locking (unlikely unless you're overloading the database) you could just call it a second / third time.