Related
Normally I can insert a row into a MySQL table and get the last_insert_id back. Now, though, I want to bulk insert many rows into the table and get back an array of IDs. Does anyone know how I can do this?
There are some similar questions, but they are not exactly the same. I don't want to insert the new ID to any temporary table; I just want to get back the array of IDs.
Can I retrieve the lastInsertId from a bulk insert?
Mysql mulitple row insert-select statement with last_insert_id()
Old thread but just looked into this, so here goes: if you are using InnoDB on a recent version of MySQL, you can get the list of IDs using LAST_INSERT_ID() and ROW_COUNT().
InnoDB guarantees sequential numbers for AUTO INCREMENT when doing bulk inserts, provided innodb_autoinc_lock_mode is set to 0 (traditional) or 1 (consecutive).
Consequently you can get the first ID from LAST_INSERT_ID() and the last by adding ROW_COUNT()-1.
The only way I can think it could be done is if you store a unique identifier for each set of rows inserted (guid)
then select the row ids.
e.g:
INSERT INTO t1
(SELECT col1,col2,col3,'3aee88e2-a981-1027-a396-84f02afe7c70' FROM a_very_large_table);
COMMIT;
SELECT id FROM t1
WHERE guid='3aee88e2-a981-1027-a396-84f02afe7c70';
You could also generate the guid in the database by using uuid()
Lets assume we have a table called temptable with two cols uid, col1 where uid is an auto increment field. Doing something like below will return all the inserted id's in the resultset. You can loop through the resultset and get your id's. I realize that this is an old post and this solution might not work for every case. But for others it might and that's why I'm replying to it.
# lock the table
lock tables temptable write;
#bulk insert the rows;
insert into temptable(col1) values(1),(2),(3),(4);
#get the value of first inserted row. when bulk inserting last_insert_id() #should give the value of first inserted row from bulk op.
set #first_id = last_insert_id();
#now select the auto increment field whose value is greater than equal to #the first row. Remember since you have write lock on that table other #sessions can't write to it. This resultset should have all the inserted #id's
select uid from temptable where uid >=#first_id;
#now that you are done don't forget to unlock the table.
unlock tables;
It's worth noting that #Dag Sondre Hansen's answer can also be implemented in case you have innodb_autoinc_lock_mode set to 2 by simply locking the table before insert.
LOCK TABLE my_table WRITE;
INSERT INTO my_table (col_a, col_b, col_c) VALUES (1,2,3), (4,5,6), (7,8,9);
SET #row_count = ROW_COUNT();
SET #last_insert_id = LAST_INSERT_ID();
UNLOCK TABLES;
SELECT id FROM my_table WHERE id >= #last_insert_id AND id <= #last_insert_id + (#row_count - 1);
Here's a fiddle demonstrating: https://www.db-fiddle.com/f/ahXAhosYkkRmwqR9Y4mAsr/0
I wouldn't be sure that auto increment value will increase item by 1. and there will be huge problems if your DB will have Master // Master replication and to resolve auto_increment duplicate exclusion. AI will be +2 instead of +1, also if there will be one more master it will come to +3. so relay on thing like AUTO_INCREMENT is going up for 1 is killing your project.
I see only some good options to do that.
this SQL snippet will have no problems with multiple masters and give good results until you will need only inserted records. on multiple requests without transactions can catch other inserts records.
START TRANSACTION;
SELECT max(id) into #maxLastId FROM `main_table`;
INSERT INTO `main_table` (`value`) VALUES ('first'), ('second') ON DUPLICATE KEY UPDATE `value` = VALUES(`value`);
SELECT `id` FROM `main_table` WHERE id > #maxLastId OR #maxLastId IS NULL;
COMMIT;
(if you will need also updated records by DUPLICATE KEY UPDATE) you will need to refactor database a bit and SQL will look like next, (safe for transactions and no transactions inside one connection.)
#START TRANSACTION
INSERT INTO bulk_inserts VALUES (null);
SET #blukTransactionId = LAST_INSERT_ID();
SELECT #blukTransactionId, LAST_INSERT_ID();
INSERT INTO `main_table` (`value`, `transaction_id`) VALUES ('first', #blukTransactionId), ('second', #blukTransactionId) ON DUPLICATE KEY UPDATE `value` = VALUES(`value`), `transaction_id` = VALUES(`transaction_id`);
SELECT #blukTransactionId, LAST_INSERT_ID();
SELECT id FROM `main_table` WHERE `transaction_id` = #blukTransactionId;
#COMMIT
both cases are safe to transnational. first will show you only inserted records and second will give you all records even updated.
also those options will work even with INSERT IGNORE ...
This thread is old but all these solutions did not help me so I came up with my own.
First, count how many rows you want to insert
let's say we need to add 5 rows:
LOCK TABLE tbl WRITE;
SELECT `AUTO_INCREMENT` FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = 'my_db' AND TABLE_NAME = 'tbl'
then use the auto_increment just selected to do next query:
ALTER TABLE tbl AUTO_INCREMENT = {AUTO_INCREMENT}+5;
UNLOCK TABLES;
Finally do your inserts
Use the reserved autoincrement range to insert with id.
Warning: this solution requires elevated access level to the tables. But usually bulk inserts are run by crons and importer scripts and what not that may use special access anyway. You would not use this for just a few inserts.
This may leave unused id's if you use ON DUPLICATE KEY UPDATE.
I think you will have to either handle the transaction id in your application, or the item id in your application in order to do this flawlessly.
One way to do this which could work, assuming that all your inserts succeed (!), is the following :
You can then get the inserted id's with a loop for the number of affected rows, starting with lastid (which is the first inserted id of the bulk insert).
And thus, i checked it works perfectly .. just be careful that HeidiSQL for example will not return the correct value for ROW_COUNT(), probably because it's a crappy GUI doing random shit we don't ask it - however it's perfectly correct from either command line or PHP mysqli -
START TRANSACTION;
BEGIN;
INSERT into test (b) VALUES ('1'),('2'),('3');
SELECT LAST_INSERT_ID() AS lastid,ROW_COUNT() AS rowcount;
COMMIT;
In PHP it looks like this (local_sqle is a straight call to mysqli_query, local_sqlec is a call to mysqli_query + convert resultset to PHP array) :
local_sqle("START TRANSACTION;
BEGIN;
INSERT into test (b) VALUES ('1'),('2'),('3');");
$r=local_sqlec("SELECT LAST_INSERT_ID() AS lastid,ROW_COUNT() AS rowcount;");
local_sqle("
COMMIT;");
$i=0;
echo "last id =".($r[0]['lastid'])."<br>";
echo "Row count =".($r[0]['rowcount'])."<br>";
while($i<$r[0]['rowcount']){
echo "inserted id =".($r[0]['lastid']+$i)."<br>";
$i++;
}
The reason the queries are separated is because I wouldn't otherwise get my result using my own functions, if you do this with standard functions, you can put it back in one statement and then retrieve the result you need (it should be result number 2 - assuming you use an extension which handles more than one result set / query).
For anyone using java with JDBC, it is possible. I am getting ids back with batch-insert doing it like this:
PreparedStatement insertBatch = null;
Connection connection = ....;
for (Event event : events) {
if (insertBatch == null){
insertBatch = connection.prepareStatement("insert into `event` (game, `type`, actor, target, arg1, arg2, arg3, created) " +
"values (?, ?, ?, ?, ?, ?, ?, ?)", Statement.RETURN_GENERATED_KEYS);
}
insertBatch.setObject(1, event.game);
insertBatch.setString(2, event.type);
insertBatch.setObject(3, event.actor);
insertBatch.setObject(4, event.target);
insertBatch.setString(5, event.arg1);
insertBatch.setObject(6, event.arg2);
insertBatch.setObject(7, event.arg3);
insertBatch.setTimestamp(8, new Timestamp(event.created.getTime()));
insertBatch.addBatch();
}
}
if (insertBatch != null){
insertBatch.executeBatch();
ResultSet generatedKeys = insertBatch.getGeneratedKeys();
for (Event event : events) {
if ( generatedKeys == null || ! generatedKeys.next()){
logger.warn("Unable to retrieve all generated keys");
}
event.id = generatedKeys.getLong(1);
}
logger.debug("events inserted");
}
Source: "Using MySQL I can do it with JDBC this way:" - Plap - https://groups.google.com/g/jdbi/c/ZDqnfhK758g?pli=1
I have to actually add this to my JDBC url: rewriteBatchedStatements=true. Or else the actual inserts show up in the mysql "general query log" as separate rows. With 7000 rows inserted, I got 2m11s for regular inserts, 46s without rewrite.. on and 1.1s with rewrite.. on. Also, it does not make other people's inserts block (I tested that). When I inserted 200k rows, it grouped them into about 36k per line ie insert into abc(..) values(..),(..),(..)....
I am actually using JDBCTemplate so the way to access the PreparedStatement is:
ArrayList<Long> generatedIds = (ArrayList<Long>) jdbcTemplate.execute(
new PreparedStatementCreator() {
#Override
public PreparedStatement createPreparedStatement(Connection connection) throws SQLException {
return connection.prepareStatement(insertSql, Statement.RETURN_GENERATED_KEYS);
}
},
new PreparedStatementCallback<Object>() {
#Override
public Object doInPreparedStatement(PreparedStatement ps) throws SQLException, DataAccessException {
// see above answer for setting the row data
...
ps.executeBatch();
ResultSet resultSet = ps.getGeneratedKeys();
ArrayList<Long> ids = new ArrayList<>();
while (resultSet.next()) {
ids.add(resultSet.getLong(1));
}
return ids;
}
}
);
$query = "INSERT INTO TABLE (ID,NAME,EMAIL) VALUES (NULL,VALUE1, VALUE2)";
$idArray = array();
foreach($array as $key) {
mysql_query($query);
array_push($idArray, mysql_insert_id());
}
print_r($idArray);
I want to insert a value into a table. However that value comes from that table too. And I want to check if there is a duplicate key on that table. Since this value come from that table too, the query says that a column name is ambiguous.
$result2 = "INSERT INTO estock_saldo
(items, customer_id, quantity , reference_no, size)
SELECT
items, '".$member_id."', '".$quantity[$i]."', reference_no, size
FROM
estock_saldo
WHERE id in ({$order_id[$i]})
ON DUPLICATE KEY
UPDATE estock_saldo.quantity = estock_saldo.quantity - '".$quantity[$i]."'";
$res2 = $mysqli->query($result2);
if(!$res2){ printf("Errormessage 2: %s\n", $mysqli->error); die(); }
The ambiguous come from estock_saldo.quantity. I have tried to alias the column name. However you can't do that in insert table.
Problem persist in the below shown code snippet. You can't use column alias in INSERT statement.
INSERT INTO estock_saldo
(items, customer_id, quantity AS asquan
<-- HERE
What you are trying will always have a duplicate entry, since you are inserting the same record again. Instead of INSERT statement, you actually meant to do a UPDATE like
UPDATE estock_saldo SET
quantity = quantity - '".$quantity[$i]."'"
WHERE id in ({$order_id[$i]});
Ik execute a query that inserts some values in de table, if a combination of ID, Year, rownum (unique index) exists that i do a regular ON DUPLICATE KEY UPDATE and so the row is updated. The query looks like this:
INSERT INTO data_advertenties_prijzen (`ID`, `year`, `rownum`, `status_prijs`,
`datum_dag`, `timestamp_dag`)
VALUES (100,2014,1,1,'2014-01-01',1388534400),
(100,2014,2,1,'2014-07-16',1405468800),
(100,2014,3,1,'2014-07-26',1406332800)
ON DUPLICATE KEY UPDATE
status_prijs = VALUES(status_prijs),
datum_dag = VALUES(datum_dag),
timestamp_dag = VALUES(timestamp_dag)
Nothing difficults there, but….
I also want to do a ON DUPLICATE IGNORE for 1 value in the same query. I Also want to insert one row for 2015. For example: (100,2015,1,1,'2015-01-01',1405468800)…
If there is already a row with ID=100, Year=2015 And rownum=1 the insert of that row must be ignored.
How to do that?
You could change the values conditionally in the ON DUPLICATE clause.
I did this experiment to make sure it works:
INSERT INTO data_advertenties_prijzen VALUES (100, 2014, 1, 7, now(), 1406332800)
ON DUPLICATE KEY UPDATE
status_prijs = IF((id,year,rownum)=(100,2015,1), status_prijs, VALUES(status_prijs)),
datum_dag = IF((id,year,rownum)=(100,2015,1), datum_dag, VALUES(datum_dag)),
timestamp_dag = IF((id,year,rownum)=(100,2015,1), timestamp_dag, VALUES(timestamp_dag));
So if I try to insert a specific trio of id/year/rownum, it just uses the existing value, else if it's some other id/year/rownum, it uses the VALUES I specify in the INSERT.
Unfortunately, you must repeat the expression for each column you want to update.
Im running a database log and every day I log on a new row. My Mysql query therefore checks if the day (date (the unique key)) already exists, and if so, it tries to increment all the loggable values of the log-row by one. If the date record doesnt eyist yet, it will create a new row.
My SQL query is:
INSERT INTO `log` (`date`,`hits`,`stale`)
VALUES ('2012-03-06',1,1)
ON DUPLICATE KEY
UPDATE `hits`=`hits`+1,`stale`=`stale`+1
WHERE `date`='2012-03-06';"
All columns have 0 as default value, so if this query runs directly after midnight only 'stale' and 'hits' are set to 1. Otherwise 'stale' and 'hits' are both incremented.
I wish! (it doesn't work).
What am I missing? Which separator other then a comma should I use between 'hits' = 'hits' +1 and 'stale'='stale'+1?
Just get rid of the WHERE clause:
INSERT INTO `log` (`date`,`hits`,`stale`)
VALUES ('2012-03-06',1,1)
ON DUPLICATE KEY
UPDATE `hits`=`hits`+1,`stale`=`stale`+1;
Your separator is correct, but the UPDATE has already found the duplicate row to be able to trigger the ON DUPLICATE KEY, so you don't need to try to select it again using WHERE.
INSERT INTO `log` (`date`,`hits`,`stale`)
VALUES ('2012-03-06',1,1)
ON DUPLICATE KEY
UPDATE `hits`=`hits`+1,`stale`=`stale`+1
Demo here.
You shouldn't have the WHERE clause. ON DUPLICATE KEY UPDATE automatically limits the row it affects to the one that has the existing key.
Remove it and your query should work fine.
If you only want to do the update if some specific expression is true, you can do it with two statements:
INSERT IGNORE INTO x VALUES (.....);
UPDATE x SET ..... WHERE .....;
The INSERT will silently fail if there is a duplicate key.
I have a table with columns record_id (auto inc), sender, sent_time and status.
In case there isn't any record of a particular sender, for example "sender1", I have to INSERT a new record otherwise I have to UPDATE the existing record which belongs to "user1".
So if there isn't any record already stored, I would execute
# record_id is AUTO_INCREMENT field
INSERT INTO messages (sender, sent_time, status)
VALUES (#sender, time, #status)
Otherwise I would execute UPDATE statement.
Anyway.. does anyone know how to combine these two statements in order to insert a new record if there isn't any record where the field sender value is "user1" otherwise update the existing record?
MySQL supports the insert-on-duplicate syntax, f.e.:
INSERT INTO table (key,col1) VALUES (1,2)
ON DUPLICATE KEY UPDATE col1 = 2;
If you have solid constraints on the table, then you can also use the REPLACE INTO for that. Here's a cite from MySQL:
REPLACE works exactly like INSERT, except that if an old row in the table has the same value as a new row for a PRIMARY KEY or a UNIQUE index, the old row is deleted before the new row is inserted.
The syntax is basically the same as INSERT INTO, just replace INSERT by REPLACE.
INSERT INTO messages (sender, sent_time, status) VALUES (#sender, time, #status)
would then be
REPLACE INTO messages (sender, sent_time, status) VALUES (#sender, time, #status)
Note that this is a MySQL-specific command which doesn't occur in other DB's, so keep portability in mind.
As others have mentioned, you should use "insert...on duplicate key update", sometimes referred to as an "upsert". However, in your specific case you don't want to use a static value in the update, but rather the values you pass in to the values clause of the insert statement.
Specifically, I think you want to update two columns if the row already exists:
1) sent_time
2) status
In order to do this, you would use an "upsert" statement like this (using your example):
INSERT INTO messages (sender, sent_time, status)
VALUES (#sender, time, #status)
ON DUPLICATE KEY UPDATE
sent_time = values(sent_time),
status = values(status);
Check out "Insert on Duplicate Key Update".
INSERT INTO table (a,b,c) VALUES (1,2,3)
ON DUPLICATE KEY UPDATE c=c+1;
UPDATE table SET c=c+1 WHERE a=1;
One options is using on duplicate update syntax
http://dev.mysql.com/doc/refman/5.1/en/insert-on-duplicate.html
Other options is doing select to figure out if record exists and then doind inser/update accordingly. Mind that if you're withing transaction select will not explicitly terminate the transaction so it's safe using it.
use merge statement :
merge into T1
using T2
on (T1.ID = T2.ID)
when matched
then update set
T1.Name = T2.Name
when not matched
then insert values (T2.ID,T2.Name);