I have a members table. Half the data/fields are populated through an online CMS.
But for the member's core contact detail fields, they come from a CSV exported from a desktop database.
I wanted to be able to upload this CSV and use the LOAD DATA command to update the members contact detail fields (matching on id) but without touching/erasing the other fields.
Is there a way to do this or must I instead loop through each row of the CSV and UPDATE... (if that's the case, any tips for the best way to do it?)
The Load Data Infile command supports the REPLACE keyword. This might be what you're looking for. From the manual:
REPLACE works exactly like INSERT,
except that if an old row in the table
has the same value as a new row for a
PRIMARY KEY or a UNIQUE index, the old
row is deleted before the new row is
inserted
The Load Data Infile command also has options where you can specify which columns to update, so perhaps you can upload the data, only specifying the columns which you want to update.
Related
I have a CSV file that I am loading into my database. I want the previous data in the table to be overwritten and not appended every time I load my CSV file. Is it possible to do this within a single query?
Is the only solution to TRUNCATE the table and then utilize the LOAD DATA INFILE queries?
Assuming you have a primary key, you can use REPLACE. As the documentation states:
The REPLACE and IGNORE modifiers control handling of input rows that
duplicate existing rows on unique key values:
If you specify REPLACE, input rows replace existing rows. In other words, rows that have the same value for a primary key or unique index
as an existing row. See Section 13.2.9, “REPLACE Statement”.
However, if you want to replace the existing table, then truncate the table first and then load.
Ok, so I have a database in my testing environment called 'Food'. In this database, there is a table called 'recipe', with a column called 'source'.
This same database exists in my local environment. However, I just received an updated database (in my local environment) where all the column values (for 'source') have changed.
Is there any way I can migrate the 'source' column from my local to my test environment, without changing the values for any other column? There are 1186 rows in the 'Food' database 'recipe' table in my test environment that need to be updated ONLY with the 'source' column.
You need some way to uniquely identify your Recipes. If both tables have a surrogate key that remained constant, use that. Otherwise figure out some way to match up the new data with your test data: you might already have a unique index in mind or you might need to decide on a combination of fields that uniquely identify your Recipes.
On a side note, why can't you just overwrite all the columns? It is just test data, right?
If only a column has changed and you have IDs (or keys) on your rows, you could follow these steps:
create an intermediate table locally
insert keys and new source values there (either those which have changed or all)
use mysqldump to selectively export the table from the local database
copy the dumped table to the remote database server
import it there
join it with the production table in an update statement to replace the values
drop the intermediate table on the server
I get a report in a tab delimited file which stores some SKUs and the current quantities of them.
Which means most of the time the inventory is the same and we just have to update the quantities.
But it can happen, that a new SKU is in the list which we have to insert instead of updating.
We are using an INNODB table for storing those SKUs. At the moment we just cut the file by tabs and line breaks and make an INSERT ... ON DUPLICATE KEY UPDATE query which is quite inefficient, because INSERT is expensive at INNODB, right? Also tricky because when a list with a lot of SKUs coming in > 20k it just take some minutes.
So my resolution for now is to just make a LOAD DATA INFILE into an tmp table and afterwards do the INSERT ... ON DUPLICATE KEY UPDATE, which should be faster i think.
Also is there another solution which does a simple UPDATE in the first place and only if there are some left, it performs and INSERT? This would be perfect, but yet i could not find anything about it. Is there a way to delete rows which returned an update: 1?
Sort the CSV file by the PRIMARY KEY of the table.
LOAD DATA INFILE into a separate table (as you said)
INSERT INTO real_table SELECT * FROM tmp_table ON DUPLICATE KEY UPDATE ... -- Note: This is a single INSERT.
Caveat: This may block the table from other uses during step 3. A solution: Break the CSV into 1000-row chunks. COMMIT after each chunk.
I just want to get suggestions about what is the best way to do the following:
I have a csv file, this file contains users and their information. Before I enter that to the MySQL database, I need to compare the email column of the CSV file with the email column in the database, it if exist, I don't insert it, if it doesn't then I insert it. How would I do this process other than manually? I would highly appreciate ideas.
Just to mention, the way I am doing it is so manually (I know it is stupid but I am not that good with SQL) what I do is, I sign in to my wordpress, go to users, and in the search bar I search for every email to make sure it does not exist.
If there's a unique index on the email column, you can use INSERT IGNORE. This will skip any records that have duplicate keys.
Insert the csv to a tempoary table. then insert into the target table by select all the records that is not exist in the target table from source table by using insert into select.
When using LOAD DATA INFILE, is there a way to get the same functionality provided by ON DUPLICATE KEY UPDATE of regular INSERT statements?
What I want to do is: for each line of my file, if the row doesn't exist, a new row is inserted, otherwise the selected fields are updated.
My table has 5 columns: A, B, C, D and E. A is the primary key. Sometimes, I have to insert new rows with all the values, but sometimes I have to update only B and C, for example. But, the point is that I want to regroup all the INSERT or UPDATE in the same file.
Thanks
If you want to insert/update some of fields, then you should load data into additional table, and then use INSERT, UPDATE or INSERT...SELECT+ON DUPLICATE KEY UPDATE statement to copy/modify data; otherwise other fields will be set to NULL.
The REPLACE option in LOAD DATA INFILE won't help you in this case.
Also, you can use Data Import tool (CSV format) in dbForge Studio for MySQL (free express edition), just choose Append/Update import mode and specify fields mapping in the Data Import wizard.