I want to create or update objects from a list of dicts using sqlalchemy.
Doing something like: session.bulk_update_mappings(MyObject, list_of_dicts)
And the problem is that I am getting integrity error, for cases when I there are some from the list_of_dicts are not in the database.
So I am kind of looking for a way to combine bulk_update_mappings and bulk_insert_mappings for one set of objects.
This is DB-specific. Take a look at this answer: SQLite UPSERT - ON DUPLICATE KEY UPDATE
In MySQL we solve this problem by using
INSERT ... ON DUPLICATE KEY UPDATE ...
looks like you can achieve the same result in sqlite by using
INSERT OR IGNORE INTO ... UPDATE ...
make sure you have a proper primary (or unique) key defined so the DB can identify the dups.
Related
I have an app that has to import TONS of data from a remote source. From 500 to 1500 entries per call.
Sometimes some of the data coming in will need to replace data already stored in the dB. If I had to guess, I would say once in 300 or 400 entries would one need to be replaced.
Each incoming entry has a unique ID. So I am trying to figure out if it is more efficient to always issue a delete command based on this ID or to check if there is already an entry THEN delete.
I found this SO post where it talks about the heavy work a dB has to do to delete something. But it is discussing a different issue so I'm not sure if it applies here.
Each incoming entry has a unique ID. So I am trying to figure out if it is more efficient to always issue a delete command based on this ID or to check if there is already an entry THEN delete.
Neither. Use INSERT ... ON DUPLICATE KEY UPDATE ....
Since you are using MySQL and you have a unique key then let MySQL do the work.
You can use
INSERT INTO..... ON DUPLICATE KEY UPDATE......
MySQL will try to insert a new record in the table, is the unique value exists in the table then MySQL will update all the field that you have set after the update
You can read more about the INSERT INTO..... ON DUPLICATE KEY UPDATE...... syntax on
http://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html
i wanna migrate my origin database to new database(mariaDB)
but new database has some information
so i have to check
if some datas are in new database origin should not migrate to new one
i wanna seperate each situation, i wanna avoid
how can i check? just using sql query
take a look at
INSERT ............ ON DUPLICATE KEY UPDATE <---if you set UPDATE clause to do something useless, it'll do what you want.
INSERT IGNORE <---- skip if duplicate.
I personally suggest the on duplicate key approach as insert ignore will also ignore some other errors and doesn't give you error message.
I'm trying to create a code for a single button where it will perform either of two actions where it will add to the database if the user currently don't have the record while it will update the user's record if the user has records already. I've done it like this:
if() {
mysql_query("INSERT INTO table...");
}
else {
mysql_query("UPDATE table SET...");
}
Is it possible?
Yes, what you've written will work. If you have a way to know if there already exists a row or not without making an additional query just for this bit of code, then do exactly as you wrote.
If, however, you planned to first SELECT from the table to see if a row exists, then conditionally INSERT or UPDATE, you will perform more queries than necessary.
It would be better to either:
Have a PRIMARY KEY or other constraint on the table prevent duplicate INSERTs. Then issue an INSERT ... ON DUPLICATE KEY UPDATE query. This will attempt to INSERT the row, and if it is a duplicate, automatically perform the specified UPDATE to that row instead.
Issue the UPDATE query and check mysql_affected_rows to see if it updated an existing row. If not, then issue the INSERT query to create the new row.
Which one is more appropriate depends on your application.
you can use INSERT ... ON DUPLICATE KEY UPDATE Syntax like:
INSERT INTO table (a,b,c) VALUES (1,2,3)
ON DUPLICATE KEY UPDATE c=c+1;
http://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html
If you have properly set unique keys, you should use REPLACE so you could remove the if.
REPLACE INTO table VALUE (...);
Pay attention that this is a MySQL extension, thus not portable to other DBs.
Yes, you could try the insert then if it fails try the update.
But you could use the MYSQL sql "REPLACE" keyword, which will insert a new record if it doesn't exist or delete the existing record and insert your new one if it does.
You could also use the INSERT ... ON DUPLICATE KEY UPDATE syntax
(explained here - Link to MYSQL ref which seems to be the closest fit to your requirement.
yes it is possible
first write a query for check that record is already exist or not.
Yes it is possible , it will work
I have a table that has three fields
field_one
field_two
field_three
I'd like to do an insert/update, but instead of checking if one of the key fields already exists, I need to check if (field_one,field_two) combination is already in the database, if so, then update instead of inserting.
Create unique index your_index_name on yourtable (field_one,field_two) (see docs) and use INSERT...ON DUPLICATE KEY UPDATE.
MySQL will do the rest automagically.
Multiple ways to do this. Easiest is probably something like this:
Obtain the existing fields
Insert all your fields that are not in the existing fields
Update the rest
Sound to me like you can use REPLACE INTO or ON DUPLICATE KEY UPDATE as long as there is a unique constraint on the two fields.
MySql doesn't support the MERGE statement, so need either a unique constraint or some external code.
I have two INSERT commands, that are useless to me like that because the two sets of rows - the ones that are already in the table, and the ones I have as INSERT commands - are not disjunct. Both commands insert lots of rows, and lots of values.
Therefore I get the duplicate entry error if I want to execute those lines.
Is there any easy way to 'convert' those commands into UPDATE?
I know this sounds stupid, because why do I make INSERT commands, if I want to UPDATE. Just to make it a clear scenario: another developer gave me the script:)
Thanks in advance,
Daniel
EDIT - problem solved
First I created a table and filled it up with my INSERT commands, then I used the following REPLACE command:
REPLACE
INTO table_1
SELECT *
FROM table_2;
This can originally be found at: How can I merge two MySQL tables?
MySQL's REPLACE keyword does this. Simply replace the INSERT keyword in your queries with the word REPLACE and it should update the rows instead of inserting new ones. Please note that it will only work if you're inserting a primary key or unique key column.
You would have to rewrite them to updates by hand. If I encouter such a problem, I query for the count of certain primary key first, if none is found I insert a generic dataset and update it afterwards. By this, new data can be added and already existing data will be updated, and you don't have to differentiate between inserting new data and updating data.
For MySQL, you can use either the INSERT IGNORE or the INSERT ... ON DUPLICATE UPDATE syntaxes. See the MySQL reference manual
You can easily modify your queries to update duplicate rows, see INSERT ... ON DUPLICATE KEY syntax in MySQL