Given a table 'products' with the following fields:
id
name
cost
user_id
I want to dump a CSV file containing 'name' and 'cost', and read it back in.
I'm using SELECT 'name', 'cost' INTO 'data.csv' FROM products;
How can I use LOAD DATA INFILE 'data.csv' INTO TABLE products; to read it back in, since some columns are not defined?
Assuming your id and user_id columns have default values set (or accept NULL), the statement is simple:
LOAD DATA INFILE 'rows.csv' (name,cost);
If those columns need values set, then you can set them per-row at load time:
LOAD DATA INFILE 'rows.csv' (name,cost) SET id=MD5(name),user_id=NULL;
MySQL is quite powerful when it comes to filling in values from a source CSV. Here's a blog article that shows many of the features in the context of a real world example.
Related
Is there a way to ignore certain information when loading a local file in the DB?
Example: I have a file with 2 columns "PHONE" and "VALIDATED" where a column of "VALIDATED" receives the status of 'YES' or 'NO'.
As I don't need negative cases, I wanted to ignore them to save space and improve query times.
Is there any way for me to load only phones where VALIDATED = YES?
DB: MYSQL 6.3
Import: LOAD DATA LOCAL INFILE
Crossposting a translation of my original answer at Stack Overflow in Portuguese:
It's always possible to load all data to a temporary table and later copy only desired rows to the destination table.
Assuming a table called contacts:
CREATE TEMPORARY TABLE contacts_temp LIKE contacts;
LOAD DATA LOCAL INFILE 'my_file' INTO TABLE contacts_temp;
INSERT INTO contacts (phones, validated, dt_imp)
SELECT phone, validated, dt_imp
FROM contacts_temp
WHERE validated = 'YES';
-- drop statement is only useful if you are planning to keep the
-- current session open
DROP TEMPORARY TABLE contacts_temp;
I have a large (1000s) list of IDs in a text file. I want to return the rows from my database that correspond to these IDs. How can I do this?
My current method is to simply paste the whole list into a gigantic SQL query and run that. It works, but I feel like there must be a better way.
As the list of values goes bigger and bigger, a better solution is to load it into a table, that you can then use it your query. In MySQL, the load data statement syntax comes handy for this.
Consider something like:
create temporary table all_ids (id int);
load data infile 'myfile.txt' into table all_ids;
create index idx_all_ids on all_ids(id); -- for performance
select t.*
from mytable t
where exists (select 1 from all_ids a where a.id = t.id)
The load data syntax accepts many options to accommodate the format of the input file - you can read the documentation for more information.
I am trying to load the data from CSV file to MYSql database through bulk insert option. Here are the below create table syntax and CSV file
CREATE TABLE discounts (
id INT NOT NULL ,
title VARCHAR(10) NOT NULL,
expired_date DATE NOT NULL,
amount VARCHAR(255 ) NOT NULL
);
CSV file format:
"475","Back","20140401","FFFF"
"476","bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","20140901","DDD"
SQL Query :
LOAD DATA INFILE 'C:\Users\karthick\Desktop\data.csv'
INTO TABLE discounts
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n';
In above create table syntax i have specified the column "title" data length to "10". But the value in data file for second row exceeds the length 10.
When i executed the SQL query the data are loaded successfully to MySQL database and here are the below output & My values in second row is getting truncated for the field "title". Could you please suggest how to stop loading the row without truncating it. Also it should load the next consecutive row without terminating if the data are appropriate. Please suggest
Database Output :
'475', 'Back', '2014-04-01', 'FFFF'
'476', 'bbbbbbbbbb', '2014-09-01', 'DDD'
Here is trick you may use. Assuming the maximum width title you want to persist is 100 characters, you may create the table as follows:
CREATE TABLE discounts (
id INT NOT NULL,
title VARCHAR(101) NOT NULL, -- slightly larger than desired max width
expired_date DATE NOT NULL,
amount VARCHAR(255 ) NOT NULL
);
Then load your data as you were doing. Records having titles which exceed a width of 100 would in fact have a width of 101 in your database table. Then, you may target such records for deletion:
DELETE
FROM discounts
WHERE LENGTH(title) > 100;
If you want, you can also now resize the title column to a width of exactly 100:
ALTER TABLE discounts MODIFY COLUMN title VARCHAR(100);
There might be a way to do this from LOAD DATA, but in general this tool is fairly simple and designed to just blindly load data into a MySQL table. LOAD DATA does have the ability to transform data as it is read, but I am not sure if it can block it.
As per my understanding, below are few points that you want to achieve:
1) Data should not get truncated if title length is more than specified field length as per table structure.
2) If title length is more, then that record should get skipped while doing an importing of records & rest of the process should continue ahead.
Answer as per mysql database taken into consideration:
You can make use of sql_mode as TRADITIONAL (Make MySQL behave like a “traditional” SQL database system. A simple description of this mode is “give an error instead of a warning” when inserting an incorrect value into a column. Reference: https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html)
Now after setting this mode, while doing records import, error will occur if any incorrect data or value out of range is getting to insert into table.
Next part, for out of range values their is no way to skip the error rows. You can check existing discussion link: Skip error lines while loading data to mysql table from delimited file
For skipping rows which are breaking unique constraints or possibly creating duplicate records, can be skipped using IGNORE keyword along with LOAD DATA INFILE.
Refer: https://dev.mysql.com/doc/refman/5.5/en/load-data.html
How to add a custom value using LOAD DATA LOCAL INFILE?
The column time_added is the 7th column and the file has only 2 values for the first and the second column. For the 7th column, time_added I want to use the unix timestamp when loading from file.
This code isn't working:
$result = mysql_query("LOAD DATA LOCAL INFILE '{$myFile}' INTO TABLE {$table} FIELDS TERMINATED BY ':' LINES TERMINATED BY '\n' SET `time_added`=unix_timestamp()");
Why wouldn't this work?
LOAD DATA INFILE 'file.txt'
INTO TABLE t1
(column1, column2)
SET column7 = unix_timestamp();
The answer given by #iouri indicates the key element to address your question, namely the explicit listing of the columns populated by the .csv file, (column1, column2). This line informs the LOAD function to only consider these columns when loading data from the .csv file and avoids an error similar to Row 1 doesn't contain data for all columns.
You will still need to list all columns, including custom columns, in the table definition. Also, the column names listed in the parentheses should match the names of the columns defined in the table definition. For example, if the table definition specifies two columns named user and id then you would need to have the line (user, id) above the SET column7 = unix_timestamp() line.
You may also want to double check that you want LOAD DATA LOCAL INFILE instead of LOAD DATA INFILE (no LOCAL). As specified in the documentation for load-data, the LOCAL keyword affects the expected location of the file and both the server and client must be configured properly to allow for using the LOCAL option.
I am trying to load a text file into an existing table by issuing the following command
load data infile "test.txt" into table m_c;
The table has 5 columns: id, title, official, genre and platform
where the id is the primary key with auto_increment set.
The file was added to the table, but the content was not. Instead i got NULL as values for all columns.
I really need to know why!
LOAD DATA INFILE 'test.txt'
INTO TABLE m_c
(title, genre, platform, official)
SET gameid = NULL;
Reference
Also, how is your file formatted? Tab delimited? CSV? You may need file or line terminators.
See the manual.
You do not need an ID field in your text file, the system will automatically give you a new id for each row you insert.