Related
I have a table with UID is the primary key. In the old system, it wasn't the primary key. So, people can insert data to that field, but we don't want to do it anymore.
In this table, I have a gap between UID 2000 and 2005 (2003 is taken). How do I get the list integers inside that gap?
UPDATED
I actually don't want the list of consecutive numbers between 2 uids. Assuming that I may have some UIDs existing between 2 numbers but I don't know that. I just want to get the list of available UIDs between 2 UIDs
I want this list to return:
MISSING
2001
2002
2004
See Generating a range of numbers in MySQL for how to create a table that lists all the numbers in a range. Then do:
set #start = 2000;
set #end = 2005;
SELECT n AS missing
FROM number_table AS nt
LEFT JOIN your_table AS t ON nt.n = t.uid
WHERE n BETWEEN #start AND #end
AND t.uid IS NULL
Summary of sections seen below.
Section 1 mimicks a table that has gaps in id like your question
Section 2 shows a fast way to put a 4 million row table together with incrementing pk.
completely not used for this but perhaps useful.
if it seems left like a half-thought it is because it was not useful entirely
Section 3 creates a table inspired by section 2 to leave you with a table of
it is just a worktable where ordering in important, both to insert into it and processing it
The new id to use (the pk)
Your current id (the one that is gap-prone)
and a column that says whether or not has been processed so you can do them in batches
section 3 is where the action is
xxxxxxxxxxxxxxxxxxxxxxxxxx
Section 1:
create table tbl1
( // this mimicks your current table. naturally you have one already
id bigint not null auto_increment primary key,
thing varchar(100) -- whatever columns you have
)engine=MyISAM;
insert tbl1(thing) values('a'),('a'),('b');
show table status from test; -- auto_increment=4, max is 3
alter table tbl1 auto_increment=2000;
insert tbl1(thing) values('a'),('a'),('b');
alter table tbl1 auto_increment=100000; -- 100k
insert tbl1(thing) values('a'),('a'),('b');
alter table tbl1 auto_increment=110000; -- 110k
insert tbl1(thing) values('a'),('a'),('b');
alter table tbl1 auto_increment=2000000; -- 2m
insert tbl1(thing) values('a'),('a'),('b');
show table status from test; -- auto_increment=2000003, max is 2000002
select count(*) from tbl1 -- 15 rows
xxxxxxxxxxxxxxxxxxxxxxxxxx
Section 2:
create table idFix
( colIs bigint auto_increment primary key, -- Is your Key
colShouldBe bigint null -- Should be your new Key
processedYet tinyint null -- 1 if processed
)engine=myisam;
insert into idFix(colIs) values(null); -- prime it with 1 row
-- this is pretty fast, don't laugh
-- run the following line 22 times
insert into idFix(colIs) select null from idFix;
-- you now have 4.2m rows in tbl2 (4,194,304)
select count(*) from idFix
select count(*) from idFix
where colIs not in (select id from tbl1)
-- 4,194,289
xxxxxxxxxxxxxxxxxxxxxxxxxx
Section 3:
Backup data first. Then perform tests in a scratch database of the following
create table idFix2
( yourIdShouldBe bigint auto_increment primary key, -- This becomes what your new key should be
yourIdIs bigint null, -- This is what your gap-prone id is right now
processedYet tinyint null -- 1 if processed, null otherwise
)engine=myisam;
-- the order by is important
insert into idFix2(yourIdIs,processedYet)
select id,null from tbl1 order by id
-- again order by in above stmt is important
Now you have a table that what your key should be, what your key is, and processedYet is null.
Do them in batches in a stored proc or front end code (say java/c#, whatever)
It is important to do them top to bottom. any other way will screw up your data
Did i mention it is important to do it top to bottom?
I will leave my thought out of it about getting everyone out of system and requiring a table lock
only you know your system not us.
select *
from idFix2
where processedYet is null and yourIdShouldBe<>yourIdIs
order by yourIdShouldBe -- order is important
limit 2 -- you might want to choose a bigger number :>
Did i mention it is important to do it top to bottom ??
Here is the flow using result set from above select stmt
(a) get next row in result set
(b) insert new parent record using data from tbl1 data back into tbl1 using
data from row yourIdIs but the insert will be pk=yourIdShouldBe
The insert will guarantee you won't have foreign key constraints in children tweaked below
(c) update children that use the old yourIdIs to hang under the new yourIdShouldBe
in their tables (there can be scads of these tables). the children's foreign key constraints
will be honored because the new parent row is in place already from step(b)
(d) delete the parent row from tbl1 where pk is yourIdIs. fear not that this will cause even more
gaps because those will be filled based on looping thru (a) which will slot fill them
(e) update idFix2 set processedYet=1 for the row your are processing from step (a) result set
(f) GoTo (a)
When you have no more processedYet=null you are almost done
Set new auto_increment value to what it should be (1 more than max(id) in tbl1, let's call that number nnnn)
alter table tbl1 auto_increment=nnnn;
xxxxxxxxxxxxxxxxxxxxxxxxxx
Note the following
show table status from test where name like 'tbl1%'; -- auto_increment=2000003
I have nothing in slot4, 2000 will become slot 4
insert tbl1(id,thing) values(4,'stuff from record2000 you get the drift');
show table status from test where name like 'tbl1%'; -- auto_increment=2000003 is left as is
So you are free to fill the gaps without screwing with auto_increment until the end
There it is and your gaps go away. If it fails, consider taking a vacation day.
Oh, I forgot, you were testing this first in a scratch database anyway.
Good luck!
I am running into a dead lock during moderate-to-high load situations. Here are the details.
MySQL-5.5.21-55
Engine: InnoDB
Table: Order
# Field, Type, Null, Key, Default, Extra
id, bigint(20) unsigned, NO, PRI, , auto_increment
sno, varchar(32), NO, MUL, ,
misc1, int, NO, , 0,
Table: OrderItem
# Field, Type, Null, Key, Default, Extra
id, bigint(20) unsigned, NO, PRI, , auto_increment
order_id, bigint(20), YES, MUL, ,
f1, varchar(50), YES, , ,
f2, varchar(100), YES, , ,
misc2, int, NO, , 0,
Order.sno is unique
OrderItem.order_id is not defined as foreign key but used as foreign key in the application
Order has one-to-many relation with OrderItem
OrderItem.order_id + OrderItem.f1 + OrderItem.f2 is unique
Usecase:
Whenever any record in Order or OrderItem needs to be updated, I want to invalidate the old records (or delete) and insert new ones.
It may so happen that earlier, one record in Order (e.g order1) has 3 records in OrderItem (e.g orderItem1, orderItem2, orderItem3). But now I want to have it as order1->orderItem1, orderItem4, orderItem5 or entirely new set. This is the reason why i want to invalidate the old records all together and insert new ones as finding out what is changed in OrderItem is complicated.
Multiple threads will be doing this operation; but they will work on different record sets. I'm operating on 25 Order s at a time
What I tried:
Insert into Order; on duplicate key update Order and delete all children from OrderItem and insert the OrderItem s.
Have another column called is_active in Order and mark all the records for the same sno as 0 and insert the new records in Order; insert new children into OrderItem.
Delete from Order for the given sno; delete from OrderItem for the same sno.Insert into both the tables freshly.
All the above approaches resulted in deat lock, some or the other time.
No other thread or process is working on these tables.
Observation:
Went through the following links
InnoDB record level locks
InnoDB locks set
and found that updating/deleting multiple records causes MySQL to obtain Next Key locks at REPEATABLE_READ isolation level (which is the default). In my opinion this causes the problem.
Appreciate if you can provide some direction on solving this.
Since you haven't mention it, I'll give it a try, have you tried "FOR UPDATE" ?
set connection auto commit to false.
at the beginning of program, use this to lock all relevent data.
SELECT o.* , oi.*
FROM order o
INNER JOIN orderitem oi ON (o.id=oi.order_id)
WHERE o.id = <order id to update>
FOR UPDATE;
then you can do whatever you want to those entries.
then commit.
EDIT: I think the root cause of the problem is multi threading (obviously), I was thinking if it's better if you remove that element from the equation.
imagine a system where you have multiple receiver that receives request. Those receiver will only do something like:
//select to check if there is a existing record(no need to lock), if no, return fail as response
SELECT o.* , oi.*
FROM order o
INNER JOIN orderitem oi ON (o.id=oi.order_id)
WHERE o.sno = <sno to update>;
insert into request_buffer (request_id, sno, new_order_item,create_date .......)
values
(1, abc , orderitem1......);
//return success after inserting buffer.
where you have a seperate single thread program that pools this table and handle those buffer entries.
In that case it would seperate the multithread element from the DB updateing process. I am not sure about the amount of incoming request but I think if you handle a few more request per cycle/query then the preformance wouldn't be THAT significantly different?
Finally, (perhaps a workaround, in-line with #JackyCheng's suggestion), Im,
Inserting into Order table (multiple records at a time); no update on duplicate key
Inserting into OrderItem table (multiple records at a time);
Select all Order.ids that are to be updated
Mark the identified Order (ids) as inactive one-by-one (this is the key part as it uses index key lock and not the next key lock)
Overall the transaction execution time remains below 30ms for 25 Orders.
I have a table called events where all new information goes. This table works as a reference for all queries for news feed(s) so event items are selected from there and information corresponding to that event is retrieved from the correct tables.
Now, here's my problem. I have E_ID's in the events table which correspond to the ID of an event in a different table, be it T_ID for tracks, S_ID for status and so on... These ID's could be the same so for the time being I just used a different auto_increment value for each table so status started on 500 tracks on 0 etc. Obviously, I don't want to do that as I have no idea yet of which table is going to have the most data in it. I would assume status would quickly exceed tracks.
The information is inserted into the event table with triggers. Here's an example of one;
BEGIN
INSERT INTO events (action, E_ID, ID)
VALUES ('has some news.', NEW.S_ID, NEW.ID);
END
That ones for he status table.
Is there an addition to that trigger I can make to ensure the NEW.S_ID != an E_ID currently in events and if it does change the S_ID accordingly.
Alternatively, is there some kind of key I can use to reference events when auto incrementing the S_ID so that the S_ID is not incremented to a value of E_ID.
Those are my thoughts, I think the latter solution would be better but I doubt it is possible or it is but would require another reference table and would be too complex.
It's really uncommon to require a unique id across tables, but here's a solution that will do it.
/* Create a single table to store unique IDs */
CREATE TABLE object_ids (
id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
object_type ENUM('event', ...) NOT NULL
) ENGINE=InnoDB;
/* Independent object tables do not auto-increment, and have a FK to the object_ids table */
CREATE TABLE events (
id INT UNSIGNED NOT NULL PRIMARY KEY,
...
CONSTRAINT FOREIGN KEY (id) REFERENCES object_ids (id)
) ENGINE=InnoDB;
/* When creating a new record, first insert your object type into the object_ids table */
INSERT INTO object_ids(object_type) VALUES ('event');
/* Then, get the auto-increment id. */
SET #id = LAST_INSERT_ID();
/* And finally, create your object record. */
INSERT INTO events (id, ...) VALUES (#id, ...);
Obviously, you would duplicate the structure of the events table for your other tables.
You could also just use a Universal Unique Identifier (UUID).
A UUID is designed as a number that is globally unique in space and time. Two calls to UUID() are expected to generate two different values, even if these calls are performed on two separate computers that are not connected to each other.
Please read more about it in the manual.
There's also a shorter version.
UUID_SHORT() should do the trick.
It will generate 64-bit unsigned integers for you.
According to the doc the generator logic is:
(server_id & 255) << 56
+ (server_startup_time_in_seconds << 24)
+ incremented_variable++;
The value of UUID_SHORT() is guaranteed to be unique if the following conditions hold:
The server_id value of the current server is between 0 and 255 and is unique among your set of master and slave servers
You do not set back the system time for your server host between mysqld restarts
You invoke UUID_SHORT() on average fewer than 16 million times per second between mysqld restarts
mysql> SELECT UUID_SHORT();
-> 92395783831158784
If you curious what is your server id you can use either of these:
SELECT ##server_id
SHOW VARIABLES LIKE 'server_id';
I want to know if I can repopulate the autoincrement value in mysql.
Because, I have records that look similar:
ID Name
1 POP
3 OLO
12 lku
Basically , I want a way to update the ID to this
ID Name
1 POP
2 OLO
3 lku
Is there any way to do this in mysql?
Thanks.
It's not best practice to fiddle your primary keys - better to let your DB handle it itself. There can be issues if, in between the UPDATE and ALTER, another record is added. Because of this, you must LOCK the table, which might hang other queries and spike load on a busy production server.
LOCK TABLES table WRITE
UPDATE table SET id=3 WHERE id=12;
ALTER TABLE table AUTO_INCREMENT=4;
UNLOCK TABLES
OR - for thousands of rows (with no foriegn key dependencies):
CREATE TEMPORARY TABLE nameTemp( name varchar(128) not null )
INSERT INTO name SELECT name FROM firstTable
TRUNCATE firstTable
INSERT INTO firstTable SELECT name FROM nameTemp
The latter method will only work where you have no foreign keys. If you do, you'll require a lookup table.
CREATE TEMPORARY TABLE lookup( newId INTEGER AUTO_INCREMENT, oldId INTEGER, PRIMARY KEY newId( newId ) );
INSERT INTO lookup (oldId) SELECT id FROM firstTable
[do temp table queries above]
You now have a lookup table with the old to new ids which you can use to update the foreign key dependencies on your other tables (on a test server!)
Changing the primary key is a very bad idea as it endangers your referential integrity (what if another table uses the id without having a foreign key with proper "on change"?).
If you really, really have to do it and don't care about bad side-effects:
Create a second table with identical structure
INSERT INTO new_table (id, [other fields]) SELECT NULL, [other fields] FROM old_table;
DROP old_table;
RENAME new_table old_table;
Warning:
This will damage every other table that has foreign keys on this table (but if you had such then you wouldn't be doing this anyways).
You may want to try something like...
Create Temporary table MyBackup
( ID as your autoincrement,
OldID as Int for backlinking/retention,
RestOfFields as their type )
insert into MyBackup
( OldID
RestOfFields )
select
ID as OldID,
RestOfFields
from
YourOriginalTable
order by
ID (this is your original ID)
Then you'll have a new table with an autoincrement with new IDs assigned, yet have a full copy of their original ID. Then, you can do correlated updates against other tables and set the ID = ID where ID = OldID. By keeping your insert via order by the original ID, it will keep the numbers from replacing out of sequence.. Ex: if your table was orderd as
Old ID = 3, new ID = 1
Old ID = 1, new ID = 3
Old ID = 12, new ID = 2
Your old 3's will become 1's, then the 1's would become 3's, and 12's become 2's
Old ID = 1, new ID = 1
Old ID = 3, new ID = 2
Old ID = 12, new ID = 3
your 3's won't overwrite the higher number, and the 12's won't conflict with the 3's since the threes were already lowered to 2's.
I've read some posts about this but none cover this issue.
I guess its not possible, but I'll ask anyway.
I have a table with more than 50.000 registers. It's an old table where various insert/delete operations have taken place.
That said, there are various 'holes' some of about 300 registers. I.e.: ..., 1340, 1341, 1660, 1661, 1662,...
The question is. Is there a simple/easy way to make new inserts fill these 'holes'?
I agree with #Aaron Digulla and #Shane N. The gaps are meaningless. If they DO mean something, that is a flawed database design. Period.
That being said, if you absolutely NEED to fill these holes, AND you are running at least MySQL 3.23, you can utilize a TEMPORARY TABLE to create a new set of IDs. The idea here being that you are going to select all of your current IDs, in order, into a temporary table as such:
CREATE TEMPORARY TABLE NewIDs
(
NewID INT UNSIGNED AUTO INCREMENT,
OldID INT UNSIGNED
)
INSERT INTO NewIDs (OldId)
SELECT
Id
FROM
OldTable
ORDER BY
Id ASC
This will give you a table mapping your old Id to a brand new Id that is going to be sequential in nature, due to the AUTO INCREMENT property of the NewId column.
Once this is done, you need to update any other reference to the Id in "OldTable" and any foreign key it utilizes. To do this, you will probably need to DROP any foreign key constraints you have, update any reference in tables from the OldId to the NewId, and then re-institute your foreign key constraints.
However, I would argue that you should not do ANY of this, and just understand that your Id field exists for the sole purpose of referencing a record, and should NOT have any specific relevance.
UPDATE: Adding an example of updating the Ids
For example:
Let's say you have the following 2 table schemas:
CREATE TABLE Parent
(
ParentId INT UNSIGNED AUTO INCREMENT,
Value INT UNSIGNED,
PRIMARY KEY (ParentId)
)
CREATE TABLE Child
(
ChildId INT UNSIGNED AUTO INCREMENT,
ParentId INT UNSIGNED,
PRIMARY KEY(ChildId),
FOREIGN KEY(ParentId) REFERENCES Parent(ParentId)
)
Now, the gaps are appearing in your Parent table.
In order to update your values in Parent and Child, you first create a temporary table with the mappings:
CREATE TEMPORARY TABLE NewIDs
(
Id INT UNSIGNED AUTO INCREMENT,
ParentID INT UNSIGNED
)
INSERT INTO NewIDs (ParentId)
SELECT
ParentId
FROM
Parent
ORDER BY
ParentId ASC
Next, we need to tell MySQL to ignore the foreign key constraint so we can correctly UPDATE our values. We will use this syntax:
SET foreign_key_checks = 0;
This causes MySQL to ignore foreign key checks when updating the values, but it will still enforce the correct value type is used (see MySQL reference for details).
Next, we need to update our Parent and Child tables with the new values. We will use the following UPDATE statement for this:
UPDATE
Parent,
Child,
NewIds
SET
Parent.ParentId = NewIds.Id,
Child.ParentId = NewIds.Id
WHERE
Parent.ParentId = NewIds.ParentId AND
Child.ParentId = NewIds.ParentId
We now have updated all of our ParentId values correctly to the new, ordered Ids from our temporary table. Once this is complete, we can re-institute our foreign key checks to maintain referential integrity:
SET foreign_key_checks = 1;
Finally, we will drop our temporary table to clean up resources:
DROP TABLE NewIds
And that is that.
What is the reason you need this functionality? Your db should be fine with the gaps, and if you're approaching the max size of your key, just make it unsigned or change the field type.
You generally don't need to care about gaps. If you're getting to the end of the datatype for the ID it should be relatively easy to ALTER the table to upgrade to the next biggest int type.
If you absolutely must start filling gaps, here's a query to return the lowest available ID (hopefully not too slowly):
SELECT MIN(table0.id)+1 AS newid
FROM table AS table0
LEFT JOIN table AS table1 ON table1.id=table0.id+1
WHERE table1.id IS NULL
(remember to use a transaction and/or catch duplicate key inserts if you need concurrent inserts to work.)
INSERT INTO prueba(id)
VALUES (
(SELECT IFNULL( MAX( id ) , 0 )+1 FROM prueba target))
IFNULL for skip null on zero rows count
add target for skip error mysql "error clause FROM)
There is a simple way but it doesn't perform well: Just try to insert with an id and when that fails, try the next one.
Alternatively, select an ID and when you don't get a result, use it.
If you're looking for a way to tell the DB to automatically fill the gaps, then that's not possible. Moreover, it should never be necessary. If you feel you need it, then you're abusing an internal technical key for something but the single purpose it has: To allow you to join tables.
[EDIT] If this is not a primary key, then you can use this update statement:
update (
select *
from table
order by reg_id -- this makes sure that the order stays the same
)
set reg_id = x.nextval
where x is a new sequence which you must create. This will renumber all existing elements preserving the order. This will fail if you have foreign key constraints. And it will corrupt your database if you reference these IDs anywhere without foreign key constraints.
Note that during the next insert, the database will create a huge gap unless you reset the identity column.
As others have said, it doesn't matter, and if it does then something is wrong in your database design. But personally I just like them to be in order anyway!
Here is some SQL that will recreate your IDs in the same order, but without the gaps.
It is done first in a temp_id field (which you will need to create), so you can see that it is all good before overwriting your old IDs. Replace Tbl and id as appropriate.
SELECT #i:=0;
UPDATE Tbl
JOIN
(
SELECT id
FROM Tbl
ORDER BY id
) t2
ON Tbl.id = t2.id
SET temp_id = #i:=#i+1;
You will now have a temp_id field with all of your shiny new IDs. You can make them live by simply:
UPDATE Tbl SET id = temp_id;
And then dropping your temp_id column.
I must admit I'm not quite sure why it works, since I would have expected the engine to complain about duplicate IDs, but it didn't when I ran it.
You might wanna clean up gaps in a priority column.
The way below will give an auto increment field for the priority.
The extra left join on the same tabel will make sure it is added in the same order as (in this case) the priority
SET #a:=0;
REPLACE INTO footable
(id,priority)
(
SELECT tbl2.id, #a
FROM footable as tbl
LEFT JOIN footable as tbl2 ON tbl2.id = tbl.id
WHERE (select #a:=#a+1)
ORDER BY tbl.priority
)