I have table structure like this
when I insert row to the table I'm using this query:
INSERT INTO table_blah ( material_item, ... hidden ) VALUES ( data, ... data ) ON DUPLICATE KEY UPDATE id = id, material_item = data, ... hidden = data;
when I first insert data without triggering the ON DUPLICATE KEY the id increments fine:
but when the ON DUPLICATE KEY triggers and i INSERT A NEW ROW the id looks odd to me:
How can I keep the auto increment, increment properly even when it triggers ON DUPLICATE KEY?
This behavior is documented (paragraph in parentheses):
If you specify ON DUPLICATE KEY UPDATE, and a row is inserted that
would cause a duplicate value in a UNIQUE index or PRIMARY KEY, MySQL
performs an UPDATE of the old row. For example, if column a is
declared as UNIQUE and contains the value 1, the following two
statements have similar effect:
INSERT INTO table (a,b,c) VALUES (1,2,3) ON DUPLICATE KEY UPDATE c=c+1;
UPDATE table SET c=c+1 WHERE a=1;
(The effects are not identical for
an InnoDB table where a is an auto-increment column. With an
auto-increment column, an INSERT statement increases the
auto-increment value but UPDATE does not.)
Here is a simple explanation. MySQL attempts to do the insert first. This is when the id gets auto incremented. Once incremented, it stays. Then the duplicate is detected and the update happens. But the value gets missed.
You should not depend on auto_increment having no gaps. If that is a requirement, the overhead on the updates and inserts is much larger. Essentially, you need to put a lock on the entire table, and renumber everything that needs to be renumbered, typically using a trigger. A better solution is to calculate incremental values on output.
This question is a fairly old one, but I answer it maybe it helps someone, to solve the auto-incrementing problem use the following code before insert/on duplicate update part and execute them all together:
SET #NEW_AI = (SELECT MAX(`the_id`)+1 FROM `table_blah`);
SET #ALTER_SQL = CONCAT('ALTER TABLE `table_blah` AUTO_INCREMENT =', #NEW_AI);
PREPARE NEWSQL FROM #ALTER_SQL;
EXECUTE NEWSQL;
together and in one statement it should be something like below:
SET #NEW_AI = (SELECT MAX(`the_id`)+1 FROM `table_blah`);
SET #ALTER_SQL = CONCAT('ALTER TABLE `table_blah` AUTO_INCREMENT =', #NEW_AI);
PREPARE NEWSQL FROM #ALTER_SQL;
EXECUTE NEWSQL;
INSERT INTO `table_blah` (`the_col`) VALUES("the_value")
ON DUPLICATE KEY UPDATE `the_col` = "the_value";
I had the same frustration of gaps in the auto increment but I found a way to avoid it.
In terms of previouslly discussed "overheads". When I first wrote my DB query code, it did so many separate queries that it took 5 hours. Once I put on
"ON DUPLICATE KEY UPDATE"
it got it down to about 50 seconds. Amazing! Anyway the way I solved it was by using 2 queries. Which doulbles the time it takes to 2 minutes, which is still fine.
First I did an sql query for writing all the data (updates and inserts), but I included "IGNORE" in the first query, so this just bypasses the updates and only inserts the new stuff. So assuming your auto_increment previously has no gaps then it will still have no gaps because its only new records. I believe it is updates that cause the gaps. So for inserts:
"INSERT IGNORE INTO mytablename(stuff,stuff2) VALUES "
Next I did the "ON DUPLICATE KEY UPDATE" variation of that sql query. It will keep the ID's in tact because all the records being updated have ID's already. The only thing it breaks is the auto_increment value, which gets incremented when a new record is added (or updated). So the solution is to patch this auto_increment value back to what it was before, once you have applied the updates.
To patch the auto increment value use this sql in your php:
"ALTER TABLE mytablename AUTO_INCREMENT = " . ($TableCount + 1);
This works because when you do the updates you are not increasing the amount of records. Therefore we can use the tablecount to know what the next ID should be. You set $TableCount to the table count, then we add 1 and that's the next auto increment number.
This is cheap and dirty but it seems to work. Could be bad using this while something else is writing to the db though.
Change database engine from InnoDB to MyIsam will resolve your issue.
I often deal with this by creating a temporary table, recording in the temporary table whether the record is new or not, doing an UPDATE only on the rows that are not new, and doing an INSERT with the new rows. Here's a complete example:
## THE SETUP
# This is the table we're trying to insert into
DROP TABLE IF EXISTS items;
CREATE TABLE items (
id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(100) UNIQUE,
price INT
);
# Put a few rows into the table
INSERT INTO items (name, price) VALUES
("Bike", 200),
("Basketball", 10),
("Fishing rod", 25)
;
## THE INSERT/UPDATE
# Create a temporary table to help with the update
DROP TEMPORARY TABLE IF EXISTS itemUpdates;
CREATE TEMPORARY TABLE itemUpdates (
name VARCHAR(100) UNIQUE,
price INT,
isNew BOOLEAN DEFAULT(true)
);
# Change the price of the Bike and Basketball and add a new Tent item
INSERT INTO itemUpdates (name, price) VALUES
("Bike", 150),
("Basketball", 8),
("Tent", 100)
;
# For items that already exist, set isNew false
UPDATE itemUpdates
JOIN items
ON items.name = itemUpdates.name
SET isNew = false;
# UPDATE the already-existing items
UPDATE items
JOIN itemUpdates
ON items.name = itemUpdates.name
SET items.price = itemUpdates.price
WHERE itemUpdates.isNew = false;
# INSERT the new items
INSERT INTO items (name, price)
SELECT name, price
FROM itemUpdates
WHERE itemUpdates.isNew = true;
# Check the results
SELECT * FROM items;
# Results:
# ID | Name | Price
# 1 | Bike | 150
# 2 | Basketball | 8
# 3 | Fishing rod | 25
# 4 | Tent | 100
The INSERT IGNORE INTO approach is simpler, but it ignores any error, which isn't what I want. And I agree that this is strange behavior on the part of MySQL but it's what we've got to work with.
I just thought I'd add, as i was trying to find an answer to my problem.
I could not stop the duplicate warning and found it was because I had it set it to TINYINT which only allows 127 entries, changing to SMALL/MED/BIGINT allows for many more
I don't think this is a problem with MySQL 5.6. See this example.
ON DUPLICATE KEY UPDATE id=LAST_INSERT_ID(id)
Adding less of a direct answer and more of a fix to the end results.
If you don't use your autoincrement as an identification field within your application (and you really shouldn't be. A UUID or something of that nature is better practice), and of course, if you don't have multi-billions of lines, you can reset your autoincrement field fairly easily.
SET SQL_SAFE_UPDATES = 0;
SET #num := 0;
UPDATE my_table SET id = #num := (#num+1);
ALTER TABLE my_table AUTO_INCREMENT =1;
I kinda hate that this is a thing when doing an INSERT UPDATE in MySQL.
This is not my code. I got it some somewhere on SO but it was so long ago...
Additional note, this is not really an answer to this issue. Its more to help fix an out-of-control autoincrement field.
INSERT INTO table_blah ( material_item, ... hidden ) VALUES ( data, ... data ) ON DUPLICATE KEY UPDATE material_item = data, ... hidden = data
Yes remove the ID=ID as it will automaticly add where PRIMARY KEY = PRIMARY KEY...
I have a lists table that has an order field.
When I insert a new record, is it possible to find the order of the previous row and increment the new row?
Or should I go about it myself in PHP by doing an OrderBy('order') query and getting the max() value of that?
When you declare a table with MySQL you can use an auto-increment id so you won't have to deal about its incrementation:
CREATE TABLE people (
id MEDIUMINT NOT NULL AUTO_INCREMENT,
name CHAR(30) NOT NULL,
PRIMARY KEY (id)
);
As explained in the documentation,
An integer or floating-point column can have the additional attribute
AUTO_INCREMENT. When you insert a value of NULL (recommended) or 0
into an indexed AUTO_INCREMENT column, the column is set to the next
sequence value. Typically this is value+1, where value is the largest
value for the column currently in the table. AUTO_INCREMENT sequences
begin with 1.
I suggest you to ommit the field completly when inserting new records.
You can then retrieve the last id inserted with LAST_INSERT_ID() SQL function (or the mysqli_insert_id function of PHP languagefor example).
But since it's not what you wanted, probably because of one of the reasons quoted from MarioZ's comment:
If you are already using auto-increment for the ID you can use it for
the order (that can be one reason). For auto-increment the column
must be set as primary and unique, can't be repeated values. The auto-increment is from the number in the record, if you inserted 10
rows and you delete 2, the next insert with auto-increment will be
11(if the last now is 8 you'd want it to be 9). Those are posible
reasons not to use it for what #Notflip wants :P
... You'll have to use PHP, with LOCK TABLE and UNLOCK TABLE SQL instructions before and after the retrieving of the last order then the updating of the new order, to avoid having simultaneous records with the same "order".
I have an existing sql table with 3 columns and 100+ entries/rows. There is an id column with autoincrement.
Now, I want to add 10 new rows at the beginning of the table with id from 1 to 10. But I cannot lose any existing row. So, how do I do it?
One idea that just came to my mind is perhaps I can increase the existing id by adding 10, like 1+10 becomes 11, 25+10 becomes 35, and then I can add rows at the beginning. What will be the script for this IF this is possible?
All you need to do for this is to set the auto_increment for that table to whatever number you need to create space for the new records you want to insert.
For example, if you inserted rows with id's 1-100, you might:
Check the next auto_increment value by running:
select auto_increment as val from information_schema.tables where table_schema='myschema' and table_name='mytable';
Let's assume that value would be 101 (the value that would be used if you inserted a new row). You can "advance" the auto_increment value by running:
alter table myschema.mytable auto_increment = 111;
If you insert a new row like this:
insert into mytable (not_the_id_column) values ('test');
It will get the "next" id of 111. But if you specify id values manually, you are ok in this case as long as you use any value less than 111, so you could insert your desired records like this:
insert into mytable (id, not_the_id_column) values (101, 'test101');
insert into mytable (id, not_the_id_column) values (102, 'test102');
... -- more inserts as needed
Now, you still must take proper precautions when updating PK values, or any value that has dependencies on it (Foreign Key or otherwise), but it is completely legitimate to forcibly advance and/or backfill the id values, as long as the resulting auto_increment value doesn't duplicate one that's already in the table.
I agree with juergen d's comment that you should not do this, but I realize there are situations where this kind of thing must be done.
SELECT MAX(id)-MIN(id)+1 INTO #x FROM theTable;
UPDATE theTable SET id = id + #x;
SELECT MIN(id) INTO #x FROM theTable;
UPDATE theTable SET id = 10 + id - #x;
If the id is the primary key, value collisions within an update can cause MySQL to reject the update. (Hence the pair of updates to avoid such a possibility.)
Edit: Factoring N.B.'s strong objection into this, it would also probably be good to verify the table's next auto-increment value is not going to collide with the updated records after the update is completed. I don't have an appropriate database on hand to verify whether UPDATE statements affect it; and even if they do affect it, you may end up wanting to reduce it so as to not create an unnecessary gap (gaps should ideally not be a problem, but if they are or you are just mildly OCD, it is worth looking into).
I have 2 tables - sensors and readings. There is a one to many relation from sensors to readings.
I need to query for all rows from sensors and then get the newest (i.e MAX timestamp) data from readings for each row. I've tried with:
SELECT sensors.*, readings.value, readings.timestamp
FROM sensors
LEFT JOIN readings ON readings.sensor_id = sensors.id
GROUP BY readings.sensor_id
The problem is, I have 6 million rows of data and the query is taking nearly two minutes to execute. Is there a more effecient way I can get hold of the last reading/value for each sensor?
This is how I'd go about the problem:
it involves a trigger that populates latest_readings table
it involves another table that I named latest_readings.
The table
I made sensor_id unique because I assumed you have one reading per sensor. This can be categorized by types by adding an additional column.
Reason for unique index: we'll be using MySQL's INSERT INTO ... ON DUPLICATE KEY UPDATE to have all the hard work done for us. If there's a reading for a particular sensor, it gets updated - otherwise, it gets inserted (in one query).
You can also make sensor_id a foreign key. I skipped that part.
CREATE TABLE latest_readings (
id int unsigned not null auto_increment,
sensor_id int unsigned not null,
reading_id int unsigned not null,
primary key(id),
unique (sensor_id)
) ENGINE = InnoDB;
The trigger
Trigger type is after insert. I will assume that the table is named readings and that it contains sensor_id column. Adjust accordingly.
DELIMITER $$
CREATE
TRIGGER `readings_after_insert` AFTER INSERT ON `readings`
FOR EACH ROW BEGIN
INSERT INTO readings
(sensor_id, reading_id)
VALUES
(NEW.sensor_id, NEW.id)
ON DUPLICATE KEY UPDATE reading_id = NEW.id
;
END;
$$
DELIMITER ;
How to query for latest sensor reading
Once more, I assumed what column names were, so adjust accordingly.
SELECT
r.reading_value
FROM readings r
INNER JOIN latest_readings latest
ON latest.sensor_id = r.sensor_id
WHERE r.sensor_id = 12345;
Disclaimer: this is just an example and it probably contains bugs, which means it's not a copy paste solution. If something doesn't work, and it's easy to fix - please do it :)
There is a necessity when inserting into a table of values to change the auto-increment field to another that no two of the same id in these tables. It is necessary for the data output from the third table based on the recording and going to not add to the table a column indicating. Here's my trigger, but it does not work
CREATE TRIGGER `update_id` AFTER INSERT ON `table1`
FOR EACH ROW BEGIN
ALTER TABLE `table2` AUTO_INCREMENT = NEW.id;
END;
It's not entirely clear what problem you are trying to solve.
But it sounds as if you have two tables with an id column, and you want to ensure that the same value of id is not used in both tables. That is, if id value 42 exists in table1, you want to ensure that 42 is not used as an id value in table2.
Unforunately, MySQL does not provide any declarative constraint for this.
It sounds as if you want an Oracle-style SEQUENCE object. And unfortunately, MySQL doesn't provide an equivalent.
But what we can do is emulate that. Create an extra "sequence" table that contains an AUTO_INCREMENT column. The purpose of this table is to be used to generate id values, and to keep track of the highest generated id value:
CREATE TABLE mysequence (id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY);
Then, we'd remove the AUTO_INCREMENT attribute from the id columns of the two tables we want to generate distinct id values for.
For those tables, we'd create BEFORE INSERT triggers that will obtain distinct id values and assign it to the id column. To generate a unique value, we can insert a row to the new mysequence table, and then retrieve the auto_increment value using the LAST_INSERT_ID function.
Something like this:
CREATE TRIGGER table1_bi
BEFORE INSERT ON table1
FOR EACH ROW
BEGIN
DECLARE generated_id INT UNSIGNED;
-- do we need to generate a value for id column?
IF NEW.id IS NULL THEN
-- generate unique id value with insert into sequence table
INSERT INTO mysequence (id) VALUES (NULL);
-- retrieve inserted id value
SELECT LAST_INSERT_ID() INTO generated_id;
-- assign the retrieved value to the id columns of the row being inserted
SET NEW.id = generated_id;
END IF
END$$
(That's just a rough outline, likely there's at least one syntax error in there somewhere.)
You'd need to create a BEFORE INSERT trigger for each of the tables.
This is one approach to generating distinct values for the id columns.
Note that it wouldn't be necessary to keep ALL of the rows in the mysequence table, it's only necessary to keep the row with the largest id value.
Also note that this doesn't enforce any constraint on either tables; some session could supply a value for id that is already in the other table. To prevent that, the trigger could raise an error if a non-NULL id value is supplied. It might also be possible to allow non-NULL values, and to perform a query to check if the supplied id value already exists in the other table, and raise an error if it does. But that query would be subject to a race condition... two concurrent sessions doing inserts to the tables, and you'd need to implement some concurrency killing locking mechanisms to prevent concurrent inserts.