I am copying records from one table to another and there is a chance that some records may already be in the second table i am copying the records to.
Since there are lots of rows i am copying,i am looking a way to ignore all the record already exists messages from mysql and continue executing the mysql file.
Is there a way i can suppress the error messages?.
As documented under INSERT Syntax:
The INSERT statement supports the following modifiers:
[ deletia ]
If you use the IGNORE keyword, errors that occur while executing the INSERT statement are treated as warnings instead. For example, without IGNORE, a row that duplicates an existing UNIQUE index or PRIMARY KEY value in the table causes a duplicate-key error and the statement is aborted. With IGNORE, the row still is not inserted, but no error is issued.
Try this:
zcat db.sql.gz | sed -e 's/INSERT/INSERT IGNORE/' | mysql -u user -p dbname
Related
Let's say I have to do records update and insertion offline.
How can I import updated records without making duplication and also update existing records?
Will my following command work?
cat newly_updated_data.sql | /usr/bin/mysql -u root --password=xxx DBNAME
Only if the INSERT commands have an "ON DUPLICATE KEY UPDATE" clause and, of course, each record has a unique key.
This will make MySQL create a new record if one with the specified key does not exist. If a record with the given key is found, the "ON DUPLICATE KEY UPDATE" clause will execute and the specified fields will be updated.
Bottom line: in general, you cannot reapply the SQL commands and expect no duplicates.
I try to import to table data from sql files using command line.
This data contents duplicates in filed url.
But field url in table is unique. So when I try to insert data I get error "Dublicate entry"
How to inmport all data skipping this error?
You can to use the --force (-f) flag.
mysql -u userName -p -f -D dbName < script.sql
From man mysql:
ยท --force, -f
Continue even if an SQL error occurs.
Create a staging table with the same structure as your destination
table but without the constraints (unique index included).
Manually check the duplicates and decide on the way you want to
choose between duplicates rows / merge rows.
Write the appropriate query and use "insert into ... select ...".
How to inmport all data skipping this error?
Drop the index for time being -> run your batch insert -> recreate the index back
If you are using insert, the you can ignore errors using ignore error or on duplicate key update (preferable because it only ignores duplicate key errors).
If you are using load data infile, then you can use the ignore key word. As described in the documentation:
If you specify IGNORE, rows that duplicate an existing row on a
unique key value are discarded. For more information, see Comparison
of the IGNORE Keyword and Strict SQL Mode.
Or, do as I would normally do:
Load the data into a staging table.
Validate the staging table and only load the appropriate data into the final table.
I am using table type for bulk insertion or updating, but I am unable to deal with the case a single error in 100 records. In that case for one entry my 99 entries will fail what to do in that case?
You can commit for update commands after each update operation and use insert ignore into instead of insert into command for insertion and commit in the end. For better performance on insert you can use mysql specific syntax :
insert ignore into t1(a,b) values (1,2),(3,4),(5,6);
Original Question
MySQL workbench allows one to define "inserts": rows to be inserted into the database on creation. It does this by adding lines such as
START TRANSACTION;
USE `someDB`;
INSERT INTO `someDB`.`countries` (`name`) VALUES ('South Africa');
COMMIT;
However, if the database, table and entry exists, this throws an error. Creation of tables does not, as workbench uses CREATE IF NOT EXISTS for those. Is there a way to get workbench to insert using INSERT...ON DUPLICATE KEY UPDATE?
Half Solution
Running the script with the force argument:
mysql user=xx password=xx --force < script.sql
Ignores such errors, and is thus a solution in my particular case. However, the actual question of modifying the type of INSERTS still stands (for interest)
See here
I was once told that it is faster to just run an insert and let the insert fail than to check if a database entry exists and then inserting if it is missing.
I was also told that that most databases are heavily optimized for reading reading rather than writing, so wouldn't a quick check be faster than a slow insert?
Is this a question of the expected number of collisions? (IE it's faster to insert only if there is a low chance of the entry already existing.) Does it depend on the database type I am running? And for that matter, is it bad practice to have a method that is going to be constantly adding insert errors to my error log?
Thanks.
If the insert is going to fail because of an index violation, it will be at most marginally slower than a check that the record exists. (Both require checking whether the index contains the value.) If the insert is going to succeed, then issuing two queries is significantly slower than issuing one.
You can use INSERT IGNORE so that if the key already exist, the insert command would just be ignored, else the new row will be inserted. This way you need to issue a single query, which checks the duplicate values as well inserts new values too.
still Be careful with INSERT IGNORE as it turns EVERY error into a warning. Read this post for insert ignore
On duplicate key ignore?
I think INSERT IGNORE INTO .... can be used here, either it will insert or ignore it.
If you use the IGNORE keyword, errors that occur while executing the INSERT statement are treated as warnings instead. For example, without IGNORE, a row that duplicates an existing UNIQUE index or PRIMARY KEY value in the table causes a duplicate-key error and the statement is aborted. With IGNORE, the row still is not inserted, but no error is issued.
If you want to delete the old value and insert a new value you can use REPLACE You can use REPLACE instead of INSERT to overwrite old rows.
REPLACE works exactly like INSERT, except that if an old row in the table has the same value as a new row for a PRIMARY KEY or a UNIQUE index, the old row is deleted before the new row is inserted.
Else use the INSERT IGNORE as it will either inserts or ignores.
a row that duplicates an existing UNIQUE index or PRIMARY KEY value in the table causes a duplicate-key error and the statement is aborted. With IGNORE, the row still is not inserted, but no error is issued.
If your intension is to Insert if its a new record OR Update the record if it already exists then how about doing an UPSERT?
Check out - http://vadivel.blogspot.com/2011/09/upsert-insert-and-update-in-sql-server.html
Instead of checking whether the record exists or not we can try to Update it directly. If there is no matching record then ##RowCount would be 0. Based on that we can Insert it as a new record. [In SQL Server 2008 you can use MERGE concept for this]
EDIT: Please note, I know this works for MS SQL Server and I don't know about MySQL or ORACLE