I have a CSV file that I am loading into my database. I want the previous data in the table to be overwritten and not appended every time I load my CSV file. Is it possible to do this within a single query?
Is the only solution to TRUNCATE the table and then utilize the LOAD DATA INFILE queries?
Assuming you have a primary key, you can use REPLACE. As the documentation states:
The REPLACE and IGNORE modifiers control handling of input rows that
duplicate existing rows on unique key values:
If you specify REPLACE, input rows replace existing rows. In other words, rows that have the same value for a primary key or unique index
as an existing row. See Section 13.2.9, “REPLACE Statement”.
However, if you want to replace the existing table, then truncate the table first and then load.
Suppose I have a MySQL table with three fields: key, value1, value2
I want to load data for two fields (key,value1) from file inserts.txt.
Content of inserts.txt:
1;2
3;4
with:
LOAD DATA LOCAL INFILE
"inserts.txt"
REPLACE
INTO TABLE
`test_insert_timestamp`
FIELDS TERMINATED BY ';'
But in case of REPLACE, I want to leave the value2 unchanged.
How could I achieve this?
The REPLACE statement consists in the following algorithm:
MySQL uses the following algorithm for REPLACE (and LOAD DATA ... REPLACE):
Try to insert the new row into the table
While the insertion fails because a duplicate-key error occurs for a
primary key or unique index:
Delete from the table the conflicting row that has the duplicate key
value
Try again to insert the new row into the table
(https://dev.mysql.com/doc/refman/5.7/en/replace.html)
So you can't keep a value from a line which is going to be deleted.
What you want to do is emulating a "ON DUPLICATE KEY UPDATE" logic.
You can't do that within a single LOAD DATA query. What you have to do is to load your data in a temporary table first, then to make an INSERT from your temporary table to your destination table, where you will be able to use the "ON DUPLICATE KEY UPDATE" feature.
The whole process is fully detailed in the most upvoted answer of this question : MySQL LOAD DATA INFILE with ON DUPLICATE KEY UPDATE
I have a table with 18 columns and I want to update all rows of the column page_count using LOAD DATA INFILE. The file with the data is extremely simple, just \n separated values. I'm not trying to pull off anything impressive here; I just want to update the values in that one single column - about 3000 records. The code I'm using is
LOAD DATA INFILE '/tmp/sbfh_counter_restore' REPLACE INTO TABLE obits FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' (page_count);
And all this does is add one new row with the entire contents of the file dumped into the column page_count. What am I doing wrong? Shoud I use phpmyadmin? I'd be happy to use that as it better fits my skill set ;)
I created the file using SELECT page_count FROM obits INTO outfile '/tmp/sbfh_counter_restore'
based what I can understand from MySQL document, it does not support loading data into ONE COLUMN, but it will require ALL COLUMNS to be present in the file in correct order.
At least, you should use SELECT * FROM obits INTO outfile, then load it back as it will ensure the column order is consistent.
As all your file content was loaded into a new row, I think you should check the primary key (or unique key) of your table. The rows will be matched by the key and update or insert based on the matching result. It is likely that page_count is not your primary or unique key.
Hope that helps.
Best way to illustrate my problem is with a quick example.
Imagine the following file (loaded into MySQL using "Load Data Infile in Table" command)
Color,Shape
red,square
blue,triangle
green,circle
(Note: My primary key = Color. Unique Key = Shape)
No matter how many times I use the command I still (correctly) just have 3 records, as it doesn't allow duplicate records.
However, if I amend record 3 within MySQL and change it from circle to diamond and re-run the Load Data command I end up with 4 records.
Color,Shape
red,square
blue,triangle
green,diamond
green,circle
I now have 2 x green values in my Primary Key field. If I try to edit one of them I get a "Duplicate Entry for Primary Key Field" error.
I would have expected the Load Data Infile command to skip the record as it creates a duplicate value in the Primary Key field. Instead it seems to only ignore it if the entire record is a duplicate. It doesn't seem to validate fields to ensure that the Primary Key field is always unique.
Why is it failing to do this?
I have one table spread across two servers running MySql 4. I need to merge these into one server for our test environment.
These tables literally have millions of records each, and the reason they are on two servers is because of how huge they are. Any altering and paging of the tables will give us too huge of a performance hit.
Because they are on a production environment, it is impossible for me to alter them in any way on their existing servers.
The issue is the primary key is a unique auto incrementing field, so there are intersections.
I've been trying to figure out how to use the mysqldump command to ignore certain fields, but the --disable-keys merely alters the table, instead of getting rid of the keys completely.
At this point it's looking like I'm going to need to modify the database structure to utilize a checksum or hash for the primary key as a combination of the two unique fields that actually should be unique... I really don't want to do this.
Help!
To solve this problem, I looked up this question, found #pumpkinthehead's answer, and realized that all we need to do is find+replace the primary key in each row with the NULL so that mysql will use the default auto_increment value instead.
(your complete mysqldump command) | sed -e "s/([0-9]*,/(NULL,/gi" > my_dump_with_no_primary_keys.sql
Original output:
INSERT INTO `core_config_data` VALUES
(2735,'default',0,'productupdates/configuration/sender_email_identity','general'),
(2736,'default',0,'productupdates/configuration/unsubscribe','1'),
Transformed Output:
INSERT INTO `core_config_data` VALUES
(NULL,'default',0,'productupdates/configuration/sender_email_identity','general'),
(NULL,'default',0,'productupdates/configuration/unsubscribe','1'),
Note: This is still a hack; For example, it will fail if your auto-increment column is not the first column, but solves my problem 99% of the time.
if you don't care what the value of the auto_increment column will be, then just load the first file, rename the table, then recreate the table and load the second file. finally, use
INSERT newly_created_table_name (all, columns, except, the, auto_increment, column)
SELECT all, columns, except, the, auto_increment, column
FROM renamed_table_name
You can create a view of the table without the primary key column, then run mysqldump on that view.
So if your table "users" has the columns: id, name, email
> CREATE VIEW myView AS
SELECT name, email FROM users
Edit: ah I see, I'm not sure if there's any other way then.
Clone Your table
Drop the column in clone table
Dump the clone table without the structure (but with -c option to get complete inserts)
Import where You want
This is a total pain. I get around this issue by running something like
sed -e "s/([0-9]*,/(/gi" export.sql > expor2.sql
on the dump to get rid of the primary keys and then
sed -e "s/VALUES/(col1,col2,...etc.) VALUES/gi" LinxImport2.sql > LinxImport3.sql
for all of the columns except for the primary key. Of course, you'll have to be careful that ([0-9]*, doesn't replace anything that you actually want.
Hope that helps someone.
SELECT null as fake_pk, `col_2`, `col_3`, `col_4` INTO OUTFILE 'your_file'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM your_table;
LOAD DATA INFILE 'your_file' INTO TABLE your_table
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n';
For added fanciness, you can set a before insert trigger on your receiving table that sets the new primary key for reach row before the insertion occurs, thereby using regular dumps and still clearing your pk. Not tested, but feeling pretty confident about it.
Use a dummy temporary primary key:
Use mysqldump normally --opts -c. For example, your primary key is 'id'.
Edit the output files and add a row "dummy_id" to the structure of your table with the same type as 'id' (but not primary key of course). Then modify the INSERT statement and replace 'id' by 'dummy_id'. Once imported, drop the column 'dummy_id'.
jimyi was on the right track.
This is one of the reasons why autoincrement keys are a PITA. One solution is not to delete data but add to it.
CREATE VIEW myView AS
SELECT id*10+$x, name, email FROM users
(where $x is a single digit uniquely identifying the original database) either creating the view on the source database (which you hint may not be possible) or use an extract routine like that described by Autocracy or load the data into staging tables on the test box.
Alternatively, don't create the table on the test system - instead put in separate tables for the src data then create a view which fetches from them both:
CREATE VIEW users AS
(SELECT * FROM users_on_a) UNION (SELECT * FROM users_on_b)
C.
The solution I've been using is to just do a regular SQL export of the data I'm exporting, then removing the primary key from the insert statements using a RegEx find&replace editor. Personally I use Sublime Text, but I'm sure TextMate, Notepad++ etc. can do the same.
Then I just run the query in which ever database the data should be inserted to by copy pasting the query into HeidiSQL's query window or PHPMyAdmin. If there's a LOT of data I save the insert query to an SQL file and use file import instead. Copy & paste with huge amounts of text often makes Chrome freeze.
This might sound like a lot of work, but I rarely use more than a couple of minutes between the export and the import. Probably a lot less than I would use on the accepted solution. I've used this solution method on several hundred thousand rows without issue, but I think it would get problematic when you reach the millions.
I like the temporary table route.
create temporary table my_table_copy
select * from my_table;
alter table my_table_copy drop id;
// Use your favorite dumping method for the temporary table
Like the others, this isn't a one-size-fits-all solution (especially given OP's millions of rows) but even at 10^6 rows it takes several seconds to run but works.