I need to load data that is coming from a csv file to a particular table. I am loading 5 fields of the csv file into the table. I need to apply regular expression for a particular field value in the csv file. IF it doesnt match i need to reject that record. Is it possible ?
This is my load data query:
LOAD DATA LOCAL INFILE ''/test.csv''
INTO TABLE TEST_TABLE FIELDS
TERMINATED BY '',''
LINES TERMINATED BY ''\n''
(#FIELD1,#FIELD2,#FIELD3,#FIELD4,#FIELD5)
SET
FIELD1=STR_TO_DATE(#FIELD1,''%d-%m-%Y''), FIELD2=nullif(#FIELD2,''''),
FIELD3=nullif(#FIELD3,''''), FIELD4=nullif(#FIELD4,''''),
FIELD5=nullif(#FIELD5,'''');
If the values that is coming in field4 in csv file is equal to either 200 or 300, i need to consider that record and load other values otherwise i need to reject the record.
Sample file::
1),234232323,STATUS,200,33
2),45454545,STATUS,300,33
3),646546445,STATUS,100,33
here 1st and 2nd record should be considered and 3rd record should be rejected.
LOAD ...;
DELETE TEST_TABLE WHERE field4 NOT IN (200,300);
Related
I have a tab delimited data file with many missing values and I need to import it into a table in mariadb(10.4.5).
I used this command:
load data infile 'c:/path to file/file.txt' into table table_name fields terminated by '\t' lines terminated by '\n' ignore 1 rows;
But I get this error:
SQL Error (1366): Incorrect double value: '' for column db_name.table_name.col_name1 at row 10
When I examine the text data file, col_name1 at row 10 is a missing value - ie. nothing between the two tab delimiters.
I have spent hours trying to solve this issue - I would appreciate any help: Is there any way of the data including importing missing values (empty strings) into the mysql table?
Do I need to pre-process the text file before using LOAD DATA INFILE? And if so, what would be the best way to pre-process?
Do I need to pre-process the text file before using LOAD DATA INFILE? And if so, what would be the best way to pre-process?
You must do it during the importing. Something like:
LOAD DATA INFILE 'c:/path to file/file.txt'
INTO TABLE table_name
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
-- the fields which values are set directly,
-- and intermediate variables for values which must be processed,
-- positioned according to CSV structure
(field1, field2, #variable3, field4, ...)
-- process the values in the variables and set fields values
SET field3 = CASE WHEN #variable3 = '' THEN 0 ELSE #variable3 END;
(field1, field2, #variable3, field4, ...) is the destination of data fields parsed from each line of source CSV file.
I.e. first parsed value from the source line which is currently processed will be assigned directly to the field field1 of destination table. The same with second value and field2.
The third value parsed will be assigned to user-defined local variable #variable3.
The 4th parsed value again will be assigned to the table field. And so on if more data and code is present.
After the whole line parsed due to specification explained above the next processing directive is executed: SET field3 = CASE WHEN #variable3 = '' THEN 0 ELSE #variable3 END.
It is simple. If a value of variable #variable3 was assigned to empty string, then the value 0 is assigned to the field field3 of the record currently parsed, else the value parsed from current line of source file is assigned to this field without modification.
After both lines processed the whole record (all fields which were assigned to some value to) are stored into one new record in destination table by common way (assigning defaults to non-listed fields, checks, triggers...).
After storing the record the next line from CSV is readed, parsed, processed, stored, then the next line ... and so on, until the end of file or some error.
I need to enter a text value that to represent the year (will be the same for every row) in a set of data being imported from a csv file. I am getting a syntax error each time. How do I specify the text value so that it will populate the column properly?
Load data local infile 'C:/Users/Candace.....csv'
into table estimate(State, '2010', Population)
fields terminated by ',';
Not tested, though according to the documentation it should work:
LOAD DATA INFILE 'file.csv'
INTO TABLE estimate
(State, Population)
SET Year = 2010;
Relevant part from the doc:
The SET clause can be used to supply values not derived from the input file.
I have a problem within load a CSV file into MySQL database
the CSV file is like this:
stuID,stuName,degreeProg
6902101,A001,null
6902102,A002,null
6902103,A003,null
6902104,A004,null
6902105,A005,null
I have write a script like this:
LOAD DATA LOCAL INFILE 'demo.csv' INTO TABLE `table`
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(`col1`, `col2`, `col3`)
What troubles me is that:
the third column in file is null but when loading into the table, it becomes 'null' (the string)
at the end of the file, there is a extra empty line, which will be also loaded and assigned with null
How should I write the script to deal with those 2 questions? (It is forbidden to modify the csv file) (and it's better to try to reduce the warning from MySQL when runs this script )
1) one option is to have the LOAD DATA assign the value of the third field (i.e. the string 'null') into a user defined variable, and use the"SET col = expr"form to assign a value to the columncol3`.
As an example:
(`col1`, `col2`, #field3)
SET col3 = IF(#field3='null',NULL,#field3)
2) There's no way to have MySQL LOAD DATA "skip" the last record in the file. To have MySQL ignore the last line, that would be better handled outside MySQL. For example, have MySQL LOAD DATA read from a named pipe, and have a separate concurrent process read the CSV file and write to that named pipe.
If you could modify the CSV file, simply add FIELDS ENCLOSED BY '"' and change null to NULL (upper case) to get them to load as NULL. Alternatively, use \N to load in NULL.
Also, obviously, delete the empty line at the end (which is most likely causing the warnings):
stuID,stuName,degreeProg
6902101,A001,\N
6902102,A002,\N
6902103,A003,\N
6902104,A004,\N
6902105,A005,\N
I have a csv file with a couple thousand game dates in it, but they are all in the MM/DD/YYYY format
2/27/2011,3:05 PM,26,14
(26 and 14 are team id #s), and trying to put them into SQL like that just results in 0000-00-00 being put into the date field of my table. This is the command I tried using:
LOAD DATA LOCAL INFILE 'c:/scheduletest.csv' INTO TABLE game
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(`date`, `time`, `awayteam_id`, `hometeam_id`);
but again, it wouldn't do the dates right. Is there a way I can have it convert the date as it tries to insert it? I found another SO question similar to this, but I couldn't get it to work.
Have you tried the following:
LOAD DATA LOCAL INFILE 'c:/scheduletest.csv' INTO TABLE game
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(#DATE_STR, `time`, `awayteam_id`, `hometeam_id`)
SET `date` = STR_TO_DATE(#DATE_STR, '%c/%e/%Y');
For more information, the documentation has details about the use of user variables with LOAD DATA (about half-way down - search for "User variables in the SET clause" in the page)
You can use variables to load the data from the csv into and run functions on them before inserting, like:
LOAD DATA INFILE 'file.txt'
INTO TABLE t1
(#datevar, #timevar, awayteam_id, hometeam_id)
SET date = STR_TO_DATE(#datevar, '%m/%d/%Y'),
SET time = etc etc etc;
My suggestion would be to insert the file into a temporary holding table where the date column is a character datatype. Then write a query with theSTR_TO_DATE conversion to move the data from the holding table to your final destination.
Convert field that you are using for the date to varchar type so it will play friendly with any format
Import CSV
Convert the dates to a valid mysql date format using something like:
UPDATE table SET field = STR_TO_DATE(field, '%c/%e/%Y %H:%i');
Then revert field type to date
Use a function to convert the format as needed.
I'm not an expert on MySQL, but http://dev.mysql.com/doc/refman/5.0/en/date-and-time-functions.html#function_str-to-date looks promising.
If you can't do that in the load command directly, you may try creating a table that allows you to load all the values as VARCHAR and then to do an insert into your game table with a select statement with the appropriate conversion instead.
If you file is not too big, you can use the Excel function TEXT. If, for example, your date is in cell A2, then the formula in a temporary column next to it would be =TEXT(A2,"yyyy-mm-dd hh:mm:ss"). This will do it and then you can paste the values of the formula's result back into the column and then delete the temporary column.
I have created a database and a table. I have also created all the fields I will be needing. I have created 46 fields including one that is my ID for the row. The CSV doesn't contain the ID field, nor does it contain the headers for the columns. I am new to all of this but have been trying to figure this out. I'm not on here being lazy asking for the answer, but looking for directions.
I'm trying to figure out how to import the CSV but have it start importing data starting at the 2nd field, since I'm hoping the auto_increment will fill in the ID field, which is the first field I created.
I tried these instructions with no luck. Can anyone offer some insight?
The column names of your CSV file must match those of your table
Browse to your required .csv file
Select CSV using LOAD DATA options
Check box 'ON' for Replace table data with file
In Fields terminated by box, type ,
In Fields enclosed by box, "
In Fields escaped by box, \
In Lines terminated by box, auto
In Column names box, type column name separated by , like column1,column2,column3
Check box ON for Use LOCAL keyword.
Edit:
The CSV file is 32.4kb
The first row of my CSV is:
Test Advertiser,23906032166,119938,287898,,585639051,287898 - Engager - 300x250,88793551,Running,295046551,301624551,2/1/2010,8/2/2010,Active,,Guaranteed,Publisher test,Maintainer test,example-site.com,,All,All,,Interest: Dental; custom geo zones: City,300x250,-,CPM,$37.49 ,"4,415","3,246",3,0,$165.52 ,$121.69 ,"2,895",805,0,0,$30.18 ,$37.49 ,0,$0.00 ,IMPRESSIONBASED,NA,USD
You can have MySQL set values for certain columns during import. If your id field is set to auto increment, you can set it to null during import and MySQL will then assign incrementing values to it. Try putting something like this in the SQL tab in phpMyAdmin:
LOAD DATA INFILE 'path/to/file.csv' INTO TABLE your_table FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' SET id=null;
Please look at this page and see if it has what you are looking for. Should be all you need since you are dealing with just one table. MYSQL LOAD DATA INFILE
So for example you might do something like this:
LOAD DATA INFILE 'filepath' INTO TABLE 'tablename' FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (column2, column3, column4);
That should give you an idea. There are of course more options that can be added as seen in the above link.
be sure to use LOAD DATA LOCAL INFILE if the import file is local. :)