Normally I can insert a row into a MySQL table and get the last_insert_id back. Now, though, I want to bulk insert many rows into the table and get back an array of IDs. Does anyone know how I can do this?
There are some similar questions, but they are not exactly the same. I don't want to insert the new ID to any temporary table; I just want to get back the array of IDs.
Can I retrieve the lastInsertId from a bulk insert?
Mysql mulitple row insert-select statement with last_insert_id()
Old thread but just looked into this, so here goes: if you are using InnoDB on a recent version of MySQL, you can get the list of IDs using LAST_INSERT_ID() and ROW_COUNT().
InnoDB guarantees sequential numbers for AUTO INCREMENT when doing bulk inserts, provided innodb_autoinc_lock_mode is set to 0 (traditional) or 1 (consecutive).
Consequently you can get the first ID from LAST_INSERT_ID() and the last by adding ROW_COUNT()-1.
The only way I can think it could be done is if you store a unique identifier for each set of rows inserted (guid)
then select the row ids.
e.g:
INSERT INTO t1
(SELECT col1,col2,col3,'3aee88e2-a981-1027-a396-84f02afe7c70' FROM a_very_large_table);
COMMIT;
SELECT id FROM t1
WHERE guid='3aee88e2-a981-1027-a396-84f02afe7c70';
You could also generate the guid in the database by using uuid()
Lets assume we have a table called temptable with two cols uid, col1 where uid is an auto increment field. Doing something like below will return all the inserted id's in the resultset. You can loop through the resultset and get your id's. I realize that this is an old post and this solution might not work for every case. But for others it might and that's why I'm replying to it.
# lock the table
lock tables temptable write;
#bulk insert the rows;
insert into temptable(col1) values(1),(2),(3),(4);
#get the value of first inserted row. when bulk inserting last_insert_id() #should give the value of first inserted row from bulk op.
set #first_id = last_insert_id();
#now select the auto increment field whose value is greater than equal to #the first row. Remember since you have write lock on that table other #sessions can't write to it. This resultset should have all the inserted #id's
select uid from temptable where uid >=#first_id;
#now that you are done don't forget to unlock the table.
unlock tables;
It's worth noting that #Dag Sondre Hansen's answer can also be implemented in case you have innodb_autoinc_lock_mode set to 2 by simply locking the table before insert.
LOCK TABLE my_table WRITE;
INSERT INTO my_table (col_a, col_b, col_c) VALUES (1,2,3), (4,5,6), (7,8,9);
SET #row_count = ROW_COUNT();
SET #last_insert_id = LAST_INSERT_ID();
UNLOCK TABLES;
SELECT id FROM my_table WHERE id >= #last_insert_id AND id <= #last_insert_id + (#row_count - 1);
Here's a fiddle demonstrating: https://www.db-fiddle.com/f/ahXAhosYkkRmwqR9Y4mAsr/0
I wouldn't be sure that auto increment value will increase item by 1. and there will be huge problems if your DB will have Master // Master replication and to resolve auto_increment duplicate exclusion. AI will be +2 instead of +1, also if there will be one more master it will come to +3. so relay on thing like AUTO_INCREMENT is going up for 1 is killing your project.
I see only some good options to do that.
this SQL snippet will have no problems with multiple masters and give good results until you will need only inserted records. on multiple requests without transactions can catch other inserts records.
START TRANSACTION;
SELECT max(id) into #maxLastId FROM `main_table`;
INSERT INTO `main_table` (`value`) VALUES ('first'), ('second') ON DUPLICATE KEY UPDATE `value` = VALUES(`value`);
SELECT `id` FROM `main_table` WHERE id > #maxLastId OR #maxLastId IS NULL;
COMMIT;
(if you will need also updated records by DUPLICATE KEY UPDATE) you will need to refactor database a bit and SQL will look like next, (safe for transactions and no transactions inside one connection.)
#START TRANSACTION
INSERT INTO bulk_inserts VALUES (null);
SET #blukTransactionId = LAST_INSERT_ID();
SELECT #blukTransactionId, LAST_INSERT_ID();
INSERT INTO `main_table` (`value`, `transaction_id`) VALUES ('first', #blukTransactionId), ('second', #blukTransactionId) ON DUPLICATE KEY UPDATE `value` = VALUES(`value`), `transaction_id` = VALUES(`transaction_id`);
SELECT #blukTransactionId, LAST_INSERT_ID();
SELECT id FROM `main_table` WHERE `transaction_id` = #blukTransactionId;
#COMMIT
both cases are safe to transnational. first will show you only inserted records and second will give you all records even updated.
also those options will work even with INSERT IGNORE ...
This thread is old but all these solutions did not help me so I came up with my own.
First, count how many rows you want to insert
let's say we need to add 5 rows:
LOCK TABLE tbl WRITE;
SELECT `AUTO_INCREMENT` FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = 'my_db' AND TABLE_NAME = 'tbl'
then use the auto_increment just selected to do next query:
ALTER TABLE tbl AUTO_INCREMENT = {AUTO_INCREMENT}+5;
UNLOCK TABLES;
Finally do your inserts
Use the reserved autoincrement range to insert with id.
Warning: this solution requires elevated access level to the tables. But usually bulk inserts are run by crons and importer scripts and what not that may use special access anyway. You would not use this for just a few inserts.
This may leave unused id's if you use ON DUPLICATE KEY UPDATE.
I think you will have to either handle the transaction id in your application, or the item id in your application in order to do this flawlessly.
One way to do this which could work, assuming that all your inserts succeed (!), is the following :
You can then get the inserted id's with a loop for the number of affected rows, starting with lastid (which is the first inserted id of the bulk insert).
And thus, i checked it works perfectly .. just be careful that HeidiSQL for example will not return the correct value for ROW_COUNT(), probably because it's a crappy GUI doing random shit we don't ask it - however it's perfectly correct from either command line or PHP mysqli -
START TRANSACTION;
BEGIN;
INSERT into test (b) VALUES ('1'),('2'),('3');
SELECT LAST_INSERT_ID() AS lastid,ROW_COUNT() AS rowcount;
COMMIT;
In PHP it looks like this (local_sqle is a straight call to mysqli_query, local_sqlec is a call to mysqli_query + convert resultset to PHP array) :
local_sqle("START TRANSACTION;
BEGIN;
INSERT into test (b) VALUES ('1'),('2'),('3');");
$r=local_sqlec("SELECT LAST_INSERT_ID() AS lastid,ROW_COUNT() AS rowcount;");
local_sqle("
COMMIT;");
$i=0;
echo "last id =".($r[0]['lastid'])."<br>";
echo "Row count =".($r[0]['rowcount'])."<br>";
while($i<$r[0]['rowcount']){
echo "inserted id =".($r[0]['lastid']+$i)."<br>";
$i++;
}
The reason the queries are separated is because I wouldn't otherwise get my result using my own functions, if you do this with standard functions, you can put it back in one statement and then retrieve the result you need (it should be result number 2 - assuming you use an extension which handles more than one result set / query).
For anyone using java with JDBC, it is possible. I am getting ids back with batch-insert doing it like this:
PreparedStatement insertBatch = null;
Connection connection = ....;
for (Event event : events) {
if (insertBatch == null){
insertBatch = connection.prepareStatement("insert into `event` (game, `type`, actor, target, arg1, arg2, arg3, created) " +
"values (?, ?, ?, ?, ?, ?, ?, ?)", Statement.RETURN_GENERATED_KEYS);
}
insertBatch.setObject(1, event.game);
insertBatch.setString(2, event.type);
insertBatch.setObject(3, event.actor);
insertBatch.setObject(4, event.target);
insertBatch.setString(5, event.arg1);
insertBatch.setObject(6, event.arg2);
insertBatch.setObject(7, event.arg3);
insertBatch.setTimestamp(8, new Timestamp(event.created.getTime()));
insertBatch.addBatch();
}
}
if (insertBatch != null){
insertBatch.executeBatch();
ResultSet generatedKeys = insertBatch.getGeneratedKeys();
for (Event event : events) {
if ( generatedKeys == null || ! generatedKeys.next()){
logger.warn("Unable to retrieve all generated keys");
}
event.id = generatedKeys.getLong(1);
}
logger.debug("events inserted");
}
Source: "Using MySQL I can do it with JDBC this way:" - Plap - https://groups.google.com/g/jdbi/c/ZDqnfhK758g?pli=1
I have to actually add this to my JDBC url: rewriteBatchedStatements=true. Or else the actual inserts show up in the mysql "general query log" as separate rows. With 7000 rows inserted, I got 2m11s for regular inserts, 46s without rewrite.. on and 1.1s with rewrite.. on. Also, it does not make other people's inserts block (I tested that). When I inserted 200k rows, it grouped them into about 36k per line ie insert into abc(..) values(..),(..),(..)....
I am actually using JDBCTemplate so the way to access the PreparedStatement is:
ArrayList<Long> generatedIds = (ArrayList<Long>) jdbcTemplate.execute(
new PreparedStatementCreator() {
#Override
public PreparedStatement createPreparedStatement(Connection connection) throws SQLException {
return connection.prepareStatement(insertSql, Statement.RETURN_GENERATED_KEYS);
}
},
new PreparedStatementCallback<Object>() {
#Override
public Object doInPreparedStatement(PreparedStatement ps) throws SQLException, DataAccessException {
// see above answer for setting the row data
...
ps.executeBatch();
ResultSet resultSet = ps.getGeneratedKeys();
ArrayList<Long> ids = new ArrayList<>();
while (resultSet.next()) {
ids.add(resultSet.getLong(1));
}
return ids;
}
}
);
$query = "INSERT INTO TABLE (ID,NAME,EMAIL) VALUES (NULL,VALUE1, VALUE2)";
$idArray = array();
foreach($array as $key) {
mysql_query($query);
array_push($idArray, mysql_insert_id());
}
print_r($idArray);
Table foo has id and name.
Transaction A select for update by id 1.
Transaction B select for update by id 1 then wait.
Transaction A select for update by name anything(even though not exists) cause Transaction B deadlock.
Why this happen ?
Below scripts reproduce deadlock.
create table foo (id int primary key, name varchar(100));
insert into foo values (1, 'foo1');
-- transaction A
start transaction;
select * from foo where id=1 for update;
-- transaction B
start transaction;
select * from foo where id=1 for update;
-- now waiting
-- transaction A
select * from foo where name='xxxxx' for update;
-- transaction B dead lock occer
I figured out close answer.
MySQL locks all records when select for update searched by no indexed column.
https://dev.mysql.com/doc/refman/5.7/en/innodb-locks-set.html
If you have no indexes suitable for your statement and MySQL must scan the entire table to process the statement, every row of the table becomes locked, which in turn blocks all inserts by other users to the table.
But, I don't know why this causes deadlock.
By the way, I resolved my problem that every select for update to search by primary key.
I got MySQL update query like this :
"INSERT INTO $tbl_name
(status, name, ...)
VALUES
('$status', '$name'...)
ON duplicate KEY UPDATE
status=values(status), name=values(name)...
";
It have a lot of table columns so to keep it short I`ll get right to the issue:
Instead of updating the current table row it duplicates it with a new ID.
Note : There are unique fields in the row. Any ideas what am I doing wrong ?
Check existence before doing any Insert/Update
IF (SELECT 1 = 1 FROM Table WHERE Key=...) THEN
BEGIN
UPDATE Table Set status=values(status), name=values(name)... WHERE Key=...;
END;
ELSE
BEGIN
INSERT INTO Table (FieldValue) VALUES('');
END;
END IF;
Why not to use
UPDATE table_name
SET column1=value1,column2=value2,...
WHERE some_column=some_value;
?
Assume I've got an users table with 1M users on MySQL/InnoDB:
users
userId (Primary Key, Int)
status (Int)
more data
If I would want to have an exact count of the amount of users with status = 1 (denoting an activate account), what would be the way to go for big tables, I was thinking along the lines of:
usercounts
status
count
And then run an TRIGGER AFTER INSERT on users that updates the appropiate columns in usercounts
Would this be the best way to go?
ps. An extra small question: Since you also need an TRIGGER AFTER UPDATE on users for when status changes, is there a syntax available that:
Covers both the TRIGGER AFTER INSERT and TRIGGER AFTER UPDATE on status?
Increments the count by one if a count already is present, else inserts a new (status, count = 0) pair?
Would this be the best way to go?
Best (opinion-based) or not but it's definitely a possible way to go.
is there a syntax available that: covers both the TRIGGER AFTER INSERT and TRIGGER AFTER UPDATE on status?
No. There isn't a compound trigger syntax in MySQL. You'll have to create separate triggers.
is there a syntax available that: increments the count by one if a count already is present, else inserts a new (status, count = 0) pair?
Yes. You can use ON DUPLICATE KEY clause in INSERT statement. Make sure that status is a PK in usercounts table.
Now if users can be deleted even if only for maintenance purposes you also need to cover it with AFTER DELETE trigger.
That being said your triggers might look something like
CREATE TRIGGER tg_ai_users
AFTER INSERT ON users
FOR EACH ROW
INSERT INTO usercounts (status, cnt)
VALUES (NEW.status, 1)
ON DUPLICATE KEY UPDATE cnt = cnt + 1;
CREATE TRIGGER tg_ad_users
AFTER DELETE ON users
FOR EACH ROW
UPDATE usercounts
SET cnt = cnt - 1
WHERE status = OLD.status;
DELIMITER $$
CREATE TRIGGER tg_au_users
AFTER UPDATE ON users
FOR EACH ROW
BEGIN
IF NOT NEW.status <=> OLD.status THEN -- proceed ONLY if status has been changed
UPDATE usercounts
SET cnt = cnt - 1
WHERE status = OLD.status;
INSERT INTO usercounts (status, cnt) VALUES (NEW.status, 1)
ON DUPLICATE KEY UPDATE cnt = cnt + 1;
END IF;
END$$
DELIMITER ;
To initially populate usercounts table use
INSERT INTO usercounts (status, cnt)
SELECT status, COUNT(*)
FROM users
GROUP BY status
Here is SQLFiddle demo
I think there are simpler options available to you.
Just add an index to the field you'd like to count on.
ALTER TABLE users ADD KEY (status);
Now a select should be very fast.
SELECT COUNT(*) FROM users WHERE status = 1
I have a table with 1000 rows.
I need to delete rows from 5 to 1000, so the first 4 remain intact and eveythin after 4 is deleted.
However if I write something like:
DELETE FROM table_name WHERE id > 4
the numbering of the next record after INSERT is 1001, which I do not want.
How to delete rows, but force the numbering of all next rows to the number of my last id?
P.S. I can not use ALTER TABLE to drop all the table.
Thanks in advance.
Can you do this?
ALTER TABLE theTableInQuestion AUTO_INCREMENT=5
If your id - auto_increment field you need to use ALTER TABLE to change it's value...
But if it's impossible you can try not to delete rows but to set there value to NULL or "" or 0 and then just to update there value
not INSERT but UPDATE ... WHERE id = 5
With MySQL? You don't. AUTO_INCREMENT never... decrements unless you call ALTER TABLE. Have you considered use of a trigger DELETE, however?
(You may need to debug this)
delimiter |
CREATE TRIGGER testref AFTER DELETE ON theTableInQuestion
FOR EACH ROW BEGIN
IF SELECT COUNT(*) FROM test1 < 5 THEN
ALTER TABLE theTableInQuestion AUTO_INCREMENT=5
END IF;
END;
|
DBCC CHECKIDENT ( 'table_name', RESEED, new_reseed_value ) ==> Works with MS-SQL
whereas
ALTER TABLE table_name AUTO_INCREMENT = 5 ==> Works with MySQL
Think the table is in auto_increment . If you cant ALTER TABLE you need to manualy give the id in each query, something like
INSERT INTO table (id,value) VALUES ((SELECT MAX(id)+1 FROM table), value);
(see comment vis-a-vis lock table)