I have a python code that will do a bulk insert into a table. If my table already contains the same data, I will get integrity error. How to check all the rows and whatever the rows are getting I need to update those as a bulk update.
i.e., In MySql we have a query Insert into ... IF exists Update... (Similar query). So I want to use that as a bulk insert in SQLAlchemy. Can anyone help me?
Related
I have a large text file full of INSERT SQL statements that I want to insert into a phpMyAdmin database. The problem I am having is that many of these INSERT statements within this file are identical, resulting in “Duplicate Key” error occuring.
Is there a way to make phpMyAdmin ignore the repeated SQL statements? I have tried running the file through a .vbs script that removes duplicate lines but it failed to deliver.
Logic that I am thinking of so far is the following:
Run the file through a script that removes duplicate lines.
Find a solution in which phpMyAdmin ignores repeated lines.
Has anyone got any other ideas or suggestions on how I could solve this problem?
The easy way is by using INSERT IGNORE statement, but you will not know which record is duplicate.
another way, by create new table like 'table2' with no primary key or unique key, insert all the data into it, then INSERT IGNORE to your main table before, and compare which row are duplicate. Or maybe you can use the COUNT() function to get the duplicate row by.
I am using the node.js mysql package where I can create a connection and after connection.query(..) .., close the connection.
I have a collection of rows that I want to update in mysql database, and I don't know if the rows exist or not. Here I have two solutions but I don't know which one is better:
delete all the rows I want to update in mysql database and insert all rows using one connection.query(..)..
sudo code: 1. delete from table where id = ..[],
2. insert into table set ...
check each row that I want to update/insert, if the id exists in mysql database, then send query to update, otherwise insert.
sudo code: 1. for each row in list
2. check if the row exists in mysql
3. if exists, then send query to update;
if not exists, then send query to insert
for the second solution, I concerned that might be too time-consuming. Any one have some suggestions? What is the best way to do this?
I'm new to these stuff so I don't know if it is better to send multiple queries inside a for loop, especially in node each connection.query is a function with callback. Which one will have better overall performance?, or any other solutions?
Use insert ... on duplicate key update
Example:
insert into `tableName` (col1, col2, col3) values ( 'v1', 'v2', 'v3') on dupicate key update `col3`='v3'
*This wouldn't work if the key in this case is an auto_increment counter.
Source: http://dev.mysql.com/doc/refman/5.7/en/insert-on-duplicate.html
I am using table type for bulk insertion or updating, but I am unable to deal with the case a single error in 100 records. In that case for one entry my 99 entries will fail what to do in that case?
You can commit for update commands after each update operation and use insert ignore into instead of insert into command for insertion and commit in the end. For better performance on insert you can use mysql specific syntax :
insert ignore into t1(a,b) values (1,2),(3,4),(5,6);
I'm developing a php script in which I insert multiple data into a mysql table, but some of this data may already be inserted. I can try to insert each of the individual data and detect the #1062 error (duplicate entry), but it would be very inefficient since it can be more than 100 entries. So, is there any way to do this in one query or must I use a query for each entry to be inserted?
Thanks a lot.
You'll want to use the INSERT ... ON DUPLICATE KEY UPDATE command to accomplish this. In the "ON DUPLICATE KEY UPDATE" section, you can just do some kind of update that changes nothing. For example, "SET fieldName=fieldName", just so nothing is actually changed.
http://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html
i wanna migrate my origin database to new database(mariaDB)
but new database has some information
so i have to check
if some datas are in new database origin should not migrate to new one
i wanna seperate each situation, i wanna avoid
how can i check? just using sql query
take a look at
INSERT ............ ON DUPLICATE KEY UPDATE <---if you set UPDATE clause to do something useless, it'll do what you want.
INSERT IGNORE <---- skip if duplicate.
I personally suggest the on duplicate key approach as insert ignore will also ignore some other errors and doesn't give you error message.