In MySQL, can you insert multiple rows into a table and get a list back of the rows that succeeded or failed?
Or must you insert each row individually in its own statement in order to determine if the insert succeeded?
(Can multiple inserts be submitted in a batch but the success of each individually accounted for in any other DB? E.g. PostgreSQL?)
There is no way to find out which rows failed when doing a bulk insert. Usually a query will fail if you are doing multiple inserts and one of them fails for any reason.
However, there are a few options you can take to ensure your query succeeds.
1) Use REPLACE INTO rather than INSERT INTO. This will cause MySQL to over write the existing row if the primary key already exists.
2) Use INSERT IGNORE INTO to skip over existing rows. The downside is you won't know which rows have been skipped.
3) Use INSERT INTO ... ON DUPLICATE KEY UPDATE .... This will allow you to update one, two, or all fields of the row which already exists. You can also probably use this as a pseudo failure indicator if you have an empty field.
// clear out failures from previous runs
UPDATE table SET failures = 0;
// insert the row if we can, otherwise record the failure and move on without updating the row
INSERT INTO table
(id, name) VALUES
(1, 'John'),
(2, 'Steve'),
...
(500, 'Tim')
ON DUPLICATE KEY UPDATE failures = failures + 1;
// check failures
SELECT id, name, failures FROM table;
Related
What is the best way to combine two statements (INSERT or (BACKUP and UPDATE)) and perform them atomically in golang?
I found this similar question:
https://codereview.stackexchange.com/questions/186909/query-select-and-insert-if-not-exists?newreg=067063956a834327883542c3171a22d4
But the Solution does there does not fulfil 2 of the the following requirements:
perform an backup of the value ON DUPLICATE KEY,
use standard SQL
not use store procedures but
remain atomic.
This is more a SQL question/answer than Go specific so the possible solutions are SQL based.
Possible solutions:
(1) REPLACE INTO
REPLACE INTO books
(id, title, author, year_published)
VALUES
(1, 'Green Eggs and Ham', 'Dr. Seuss', 1960);
This would overwrite an existing record. Works on unique constraint(s). Though when a matching record is found it will be deleted and thus it might a not wanted behavior.
(2) INSERT IGNORE
INSERT IGNORE INTO books
(id, title, author, year_published)
VALUES
(1, 'Green Eggs and Ham', 'Dr. Seuss', 1960);
This would add if the record does not exist. Works on unique constraint(s).
From the handbook:
If you use the IGNORE modifier, errors that occur while executing the
INSERT statement are ignored. For example, without IGNORE, a row that
duplicates an existing UNIQUE index or PRIMARY KEY value in the table
causes a duplicate-key error and the statement is aborted. With
IGNORE, the row is discarded and no error occurs. Ignored errors may
generate warnings instead, although duplicate-key errors do not.
(3) INSERT ... ON DUPLICATE KEY UPDATE
INSERT INTO books
(id, title, author, year_published)
VALUES
(1, 'Green Eggs and Ham', 'Dr. Seuss', 1960);
ON DUPLICATE KEY UPDATE
title = 'No Green Eggs and Ham';
If the insert fails the values from ON DUPLICATE KEY will be used to make an update statement.
To do a backup create a history table ( a table with the same structure but amended with columns to get the change date ) and do a INSERT ... SELECT. To be atomic you would probable need to use transactions with the correct locking strategy - not sure how to get this right for MySQL.
Reference:
https://chartio.com/resources/tutorials/how-to-insert-if-row-does-not-exist-upsert-in-mysql/ for sample
https://dev.mysql.com/doc/refman/5.5/en/insert.html
https://dev.mysql.com/doc/refman/5.5/en/replace.html
https://dev.mysql.com/doc/refman/5.5/en/insert-on-duplicate.html
I have a simpe query like so:
INSERT INTO myTable (col1, col2) VALUES
(1,2),
(1,3),
(2,2)
I need to do a check that no duplicate values have been added BUT the check needs to happen across both column: if a value exists in col1 AND col2 then I don't want to insert. If the value exists only in one of those columns but not both then then insert should go through..
In other words let's say we have the following table:
+-------------------------+
|____col1____|___col2_____|
| 1 | 2 |
| 1 | 3 |
|______2_____|_____2______|
Inserting values like (2,3) and (1,1) would be allowed, but (1,3) would not be allowed.
Is it possible to do a WHERE NOT EXISTS check a single time? I may need to insert 1000 values at one time and I'm not sure whether doing a WHERE check on every single insert row would be efficient.
EDIT:
To add to the question - if there's a duplicate value across both columns, I'd like the query to ignore this specific row and continue onto inserting other values rather than throwing an error.
What you might want to use is either a primary key or a unique index across those columns. Afterwards, you can use either replace into or just insert ignore:
create table myTable
(
a int,
b int,
primary key (a,b)
);
-- Variant 1
replace into myTable(a,b) values (1, 2);
-- Variant 2
insert ignore into myTable(a,b) values (1,2);
See Insert Ignore and Replace Into
Using the latter variant has the advantage that you don't change any record if it already exists (thus no need to rebuild any index) and would best match your needs regarding your question.
If, however, there are other columns that need to be updated when inserting a record violating a unique constraint, you can either use replace into or insert into ... on duplicate key update.
Replace into will perform a real deletion prior to inserting a new record, whereas insert into ... on duplicate key update will perform an update instead. Although one might think that the result will be same, so why is there a statement for both operations, the answer can be found in the side-effects:
Replace into will delete the old record before inserting the new one. This causes the index to be updated twice, delete and insert triggers get executed (if defined) and, most important, if you have a foreign key constraint (with on delete restrict or on delete cascade) defined, your constraint will behave exactly the same way as if you deleted the record manually and inserted the new version later on. This means: Either your operation fails because the restriction is in place or the delete operation gets cascaded to the target table (i.e. deleting related records there, although you just changed some column data).
On the other hand, when using on duplicate key update, update triggers will get fired, the indexes on changed columns will be rewritten once and, if a foreign key is defined on update cascade for one of the columns being changed, this operation is performed as well.
To answer your question in the comments, as stated in the manual:
If you use the IGNORE modifier, errors that occur while executing the INSERT statement are ignored. For example, without IGNORE, a row that duplicates an existing UNIQUE index or PRIMARY KEY value in the table causes a duplicate-key error and the statement is aborted. With IGNORE, the row is discarded and no error occurs. Ignored errors may generate warnings instead, although duplicate-key errors do not.
So, all violations are treated as warnings rather than errors, causing the insert to complete. Otherwise, the insert would be applied partially (except when using transactions). Violations of duplicate key, however, do not even produce such a warning. Nonetheless, all records violating any constraint won't get inserted at all, but ignore will ensure all valid records get inserted (given that there is no system failure or out-of-memory condition).
I ve already seen some questions regarding this like below
MySQL “good” way to insert a row if not found, or update it if it is found
Now i have a summary table which gets updated with the qty every time say a sale occurs. so out of 1000 sales of an item only first time the insert executes and the rest of the times it would be update. My understanding is in Insert on Duplicate Key Update it tries to insert first and if it fails updates. so all 999 times the insert is not successfull
1) Is there a method to check Update first and if not updated then insert in a single statement?
2) which of the below methods would be desirable considering most of the cases update will be successfull
a) using Insert on Duplicate Key Update
b) Call Update; if no rows affected call insert
Right now i am using the second option(b). performance gain is very important here and also i m testing the first option. ill post the results here once done
INSERT ON DUPLICATE KEY UPDATE is the way to go. It does not perform a full insert or how you put it.
For this statement to work there has to be a primary key or unique key on the table and the corresponding columns have to be involved in the statement.
Before it's decided whether an insert or an update statement has to be done, the said keys are checked. This is usually really fast.
With your "update first" approach you gain nothing, no it gets even worse. The primary key lookup has to be done anyway. In the worst case you wasted time by having to look up the primary key two times. First for the update statement (which may not be necessary), then for the insert statement.
I'm trying to make 2 values unique, like if I have the values (5, 10) the same values can't be added again.
I'm currently selecting from the table the values x and y, checking if they both together exists on the table if they don't exists insert them, in other words
"Select * from location where x=? and y=?"
if no result is returned it will continue to insert the values.
This is typically accomplished by creating a unique index on both columns combined (a multi-column index).
Then, MySQL will prevent you from inserting duplicates. You can go ahead and try to insert the record, and if you get a duplicate key error, you know it already exists.
Alternatively, another way to handle it is to use INSERT IGNORE, so that no error occurs if you try to insert a duplicate row. Still, it won't insert, so you simply check the affected ROW_COUNT() to see if the insert was successful.
Using a unique index and catching the failure on the insert is more performant than selecting then trying to insert because in the case you do insert, MySQL only has to perform one search, rather than two.
I have a single procedure that has two insert statements in it for two different tables. I must insert data into table1 before I can insert into table2. I'm using PHP to do the data collection. What I'd like to know is how to insert multiple rows into table2, which can have many rows associated with table1. How would I do this?
I want to only store the person in table1 just one time but table2 requires multiple rows. If these insert statements were in separate procedures, I wouldn't have a problem but I just don't know how I would insert more than one row into table2 without table1 rejecting a second duplicate record.
BEGIN
INSERT INTO user(name, address, city) VALUES(Name, Address, City);
INSERT INTO order(order_id, desc) VALUES(OrderNo, Description);
END
I'd suggest you do it separately, otherwise you'd need a complicated solution which is prone to error if something changes.
The complicated solution is:
join all orderno and descriptions with a separator. (orderno#description)
join all orders with a different separator. (orderno#description/orderno#description/...)
pass it to the procedure
in the procedure, split the string by order separator, then loop through each of them
for each order, split the string by the first separator, then insert into the appropriate columns
As you can see, this is bad.
I am sorry, but what's stopping you from inserting data into these (seemingly unrelated) tables in separate queries? If you don't like the idea of it failing halfway through, you can wrap it into a transaction. I know, mysqli and pdo can do that just fine.
Answering your question directly, insert's ignore mode turns errors during insertion into warnings, so upon attempting to insert a duplicate row the warning is issued and the row is not inserted, but there is no error.
You could use the IGNORE keyword on the first statement.
http://dev.mysql.com/doc/refman/5.1/en/insert.html:
If you use the IGNORE keyword, errors that occur while executing the INSERT statement are treated as warnings instead. For example, without IGNORE, a row that duplicates an existing UNIQUE index or PRIMARY KEY value in the table causes a duplicate-key error and the statement is aborted. With IGNORE, the row still is not inserted, but no error is issued.But somehow this seems rather inefficient to me, a "stabbed from behind through the chest in the eye"-solution.