I am using table type for bulk insertion or updating, but I am unable to deal with the case a single error in 100 records. In that case for one entry my 99 entries will fail what to do in that case?
You can commit for update commands after each update operation and use insert ignore into instead of insert into command for insertion and commit in the end. For better performance on insert you can use mysql specific syntax :
insert ignore into t1(a,b) values (1,2),(3,4),(5,6);
Related
I have a python code that will do a bulk insert into a table. If my table already contains the same data, I will get integrity error. How to check all the rows and whatever the rows are getting I need to update those as a bulk update.
i.e., In MySql we have a query Insert into ... IF exists Update... (Similar query). So I want to use that as a bulk insert in SQLAlchemy. Can anyone help me?
Is there a SQL statement (or atomic sequence of statements) supported by both MySQL and HSQLDB to insert values if they aren't already there?
I'm working on an app that uses MySQL as its production database and HSQLDB for unit tests; I'd like to have a single "initial data import when the tables are empty" script.
MySQL supports INSERT IGNORE, REPLACE INTO and INSERT INTO ... ON DUPLICATE KEY UPDATE ..., but HSQLDB doesn't; conversely, HSQLDB supports MERGE but MySQL doesn't.
HSQLDb from version 2.3.4 adds support for insert ignore.
http://hsqldb.org/
Version 2.3.4 added the UUID type for columns, SYNONYM for tables and
functions, PERIOD predicates, and auto-updated TIMESTAMP columns on
row updates. Other new features included the ability to cancel
long-running statements from JDBC as well as from admin sessions, and
UTF-16 file support for text table sources, in addition to 8-bit text
files. MySQL compatibility for REPLACE, INSERT IGNORE and ON
DUPLICATE KEY UPDATE statements.
And
http://hsqldb.org/doc/guide/guide.pdf (page 260).
HyperSQL supports and translates INSERT IGNORE, REPLACE and ON
DUPLICATE KEY UPDATE variations of INSERT into predictable and
error-free operations. When INSERT IGNORE is used, if any of the
inserted rows would violate a PRIMARY KEY or UNIQUE constraint, that
row is not inserted. With multi-row inserts, the rest of the rows are
then inserted only if there is no other violation such as long strings
or type mismatch, otherwise the appropriate error is returned. When
REPLACE or ON DUPLICATE KEY UPDATE is used, the rows that need
replacing or updating are updated with the given values. This works
exactly like an UPDATE statement for those rows. Referential
constraints and other integrity checks are enforced and update
triggers are activated. The row count returned is simply the total
number of rows inserted and updated.
If someone still has this problem you can enable syntax support for MySQl by adding the following to your script
SET DATABASE SQL SYNTAX MYS TRUE
Original Question
MySQL workbench allows one to define "inserts": rows to be inserted into the database on creation. It does this by adding lines such as
START TRANSACTION;
USE `someDB`;
INSERT INTO `someDB`.`countries` (`name`) VALUES ('South Africa');
COMMIT;
However, if the database, table and entry exists, this throws an error. Creation of tables does not, as workbench uses CREATE IF NOT EXISTS for those. Is there a way to get workbench to insert using INSERT...ON DUPLICATE KEY UPDATE?
Half Solution
Running the script with the force argument:
mysql user=xx password=xx --force < script.sql
Ignores such errors, and is thus a solution in my particular case. However, the actual question of modifying the type of INSERTS still stands (for interest)
See here
I was once told that it is faster to just run an insert and let the insert fail than to check if a database entry exists and then inserting if it is missing.
I was also told that that most databases are heavily optimized for reading reading rather than writing, so wouldn't a quick check be faster than a slow insert?
Is this a question of the expected number of collisions? (IE it's faster to insert only if there is a low chance of the entry already existing.) Does it depend on the database type I am running? And for that matter, is it bad practice to have a method that is going to be constantly adding insert errors to my error log?
Thanks.
If the insert is going to fail because of an index violation, it will be at most marginally slower than a check that the record exists. (Both require checking whether the index contains the value.) If the insert is going to succeed, then issuing two queries is significantly slower than issuing one.
You can use INSERT IGNORE so that if the key already exist, the insert command would just be ignored, else the new row will be inserted. This way you need to issue a single query, which checks the duplicate values as well inserts new values too.
still Be careful with INSERT IGNORE as it turns EVERY error into a warning. Read this post for insert ignore
On duplicate key ignore?
I think INSERT IGNORE INTO .... can be used here, either it will insert or ignore it.
If you use the IGNORE keyword, errors that occur while executing the INSERT statement are treated as warnings instead. For example, without IGNORE, a row that duplicates an existing UNIQUE index or PRIMARY KEY value in the table causes a duplicate-key error and the statement is aborted. With IGNORE, the row still is not inserted, but no error is issued.
If you want to delete the old value and insert a new value you can use REPLACE You can use REPLACE instead of INSERT to overwrite old rows.
REPLACE works exactly like INSERT, except that if an old row in the table has the same value as a new row for a PRIMARY KEY or a UNIQUE index, the old row is deleted before the new row is inserted.
Else use the INSERT IGNORE as it will either inserts or ignores.
a row that duplicates an existing UNIQUE index or PRIMARY KEY value in the table causes a duplicate-key error and the statement is aborted. With IGNORE, the row still is not inserted, but no error is issued.
If your intension is to Insert if its a new record OR Update the record if it already exists then how about doing an UPSERT?
Check out - http://vadivel.blogspot.com/2011/09/upsert-insert-and-update-in-sql-server.html
Instead of checking whether the record exists or not we can try to Update it directly. If there is no matching record then ##RowCount would be 0. Based on that we can Insert it as a new record. [In SQL Server 2008 you can use MERGE concept for this]
EDIT: Please note, I know this works for MS SQL Server and I don't know about MySQL or ORACLE
I'm fetching data from a text file or log periodically and it gets inserted in the database every time fetched. Is there a way in MySQL that the insert is only done when the log files are updated or I have to do it using the programming language ? I mean Is there a type of insert that when It sees a duplicate primary key, It doesn't give an error of "Duplicate Entry" .. It just ignore.
Put the fetch in a logrotate postrotate script, and fetch from the just rotated log.
Ignoring duplicates can be done with either INSERT IGNORE OR INSERT .... ON DUPLICATE KEY UPDATE syntax (which will either ignore the lines causing a duplcate unique key, or give you the possibility to alter some values in the existing row.)