In my code I have a query that looks like this:
$load_query = "LOAD DATA LOCAL INFILE '{$file}' INTO TABLE `{$table}`
FIELDS TERMINATED BY ',' ENCLOSED BY '\"';";
Here is an example row included in the file that I am trying to load:
"MC318199","06160","1","P","00750","00000","TN598792","04/16/2009","91X"
You will notice that at the end of the example row there are quite a few spaces. These spaces all occur before the new line character "\n" that terminates the line. The problem is that these spaces get entered into the database.
How can I remove the additional spaces when running the LOAD DATA INFILE command? Is there a way to use RTRIM?
Edit
As ajreal suggested I had to re-prepare the file. After the file was prepared it was inserted into the database correctly. I modified the bash script found at: http://gabeanderson.com/2008/02/01/unixlinux-find-replace-in-multiple-files/ to accomplish this. The code is shown below:
#!/bin/bash
for fl in *.txt; do
mv $fl $fl.old
sed 's/[ \t]*$//' $fl.old > $fl
rm -f $fl.old
done
$load_query = "LOAD DATA LOCAL INFILE '{$file}' INTO TABLE `{$table}`
FIELDS TERMINATED BY ',' ENCLOSED BY '\"' (col1, col2, ... , #col_with_spaces)
SET col_with_spaces = RTRIM(#col_with_spaces);";
That way you won't need to pre-process the file nor create other updates on the table.
dun think too complicate, right after u loaded the data
update $table set $last_column=rtrim($last_column);
or manually remove the space in the $file using vi (or any editor)
or re-preapare the $file
How about simply trimming the imported data in a second step?
$fix_query = "UPDATE `{$table}` SET LastFieldName = RTRIM(LastFieldName);";
use LINES TERMINATED BY '\n'
$load_query = "LOAD DATA LOCAL INFILE '{$file}' INTO TABLE `{$table}`
FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n' " ;
Related
I have two records for example like this
11,avec myName à EX,Ex,0,2021-06-25
22,"andone \"ttt\"",Ex,0,2021-06-25
I am trying to load this into a table with MySQL with load data, and every time I try it, the quotes get cut off, or the back spaces don't show up.
I need to know if this is even possible. Can those two records go into a table and look exactly like they are in the CSV file?
I am using MySQL and trying
LOAD DATA LOCAL INFILE 'example.csv' INTO TABLE example;
Use the ENCLOSED BY and ESCAPED BY options.
LOAD DATA LOCAL INFILE 'example.csv'
INTO TABLE example
FIELDS OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '\\';
LOAD DATA LOCAL INFILE 'example.csv' INTO TABLE example
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '\\'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(col1, col2, col3.col4,col5);
In mysql 8 this will get you an error please read
https://dev.mysql.com/doc/refman/8.0/en/load-data.html#load-data-file-location
Windws uses LINES TERMINATED BY '\r\n'
if the file comes from a linux system use `LINES TERMINATED BY '\n'
it os hard to tell without seeing the file what parameters would help exactly so you should try some variants out`.
also a editor with hexfile capability helps in such cases to analyse the struicture
I've got the CSV file which I need to load to mysql database. Problem is that every row of data is followed by empty row:
open_test_uuid|open_uuid|asn|bytes_download
O0037c645-0c7b-4bd0-a6dc-1983e6d0f814|Pf13e1f22-92f6-4a49-9bd3-2882373d0266|25255|11704458
O0037c645-0c7b-4bd0-a6dc-1983e6d0f814|Pf13e1f22-92f6-4a49-9bd3-2882373d0266|25255|11704458
I tried differend combinations of LOAD command but nothing works.
LOAD DATA LOCAL INFILE '/Users/BauchMAC/Desktop/details_201301.csv'
INTO TABLE netztest
FIELDS TERMINATED BY '|'
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n\r' <--HERE should be some magic line
IGNORE 1 ROWS;
I have no idea how to solve that...
Thank you for every idea
You could try
LINES TERMINATED BY '\n\n'
or
LINES TERMINATED BY '\r\n\r\n'
and if that doesn't work you could pre-process the file before feed it to MySQL.
If you're on linux you could run this:
cat file.csv | grep -v '^\s*$' > fixed.csv
UPDATE
Perhaps a little Perl will do the trick:
perl -e '$_=`cat file.csv`; s/[\r\n]+/\n/g; print'
I'm working on a component where I am loading data from a CSV file to the mySQL database. I'm using LOAD DATA LOCAL INFILE. It works great except for one thing. In Joomla the query replaces all of the data in the table. (It truncates the table and then add the new data). I just want it to insert the new rows at the bottom of the table.
Here's the code that I am using:
$db = JFactory::getDbo();
$loaddata_query = "LOAD DATA LOCAL INFILE 'c:/data/mydata.csv'"
."\n INTO TABLE myDataTable"
."\n FIELDS TERMINATED BY ','"
."\n OPTIONALLY ENCLOSED BY '" . "\"" . "'"
."\n LINES TERMINATED BY '\\n'"
."\n IGNORE 1 LINES"
."\n (column1,column2,column3);
$db->setQuery($loaddata_query);
$result = $db->query();
Like I say, this works great for importing the data, but replaces all of the records. However if I echo $loaddata_query and I copy and paste it into phpMyAdmin, it works correctly. Here's the echoed query:
LOAD DATA LOCAL INFILE 'c:/data/mydata.csv'
TO TABLE myDataTable
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(column1,column2,column3)
I can't seem to figure out the difference. Does anyone have a suggestion?
Thanks in advance for your help!
Normally this shouldn't be happening as phpmyadmin is php, so maybe joomla has a TRUNCATE before the LOAD but in all cases try using IGNORE keyword
Like this:
LOAD DATA LOCAL INFILE IGNORE 'c:/data/mydata.csv'
TO TABLE myDataTable
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(column1,column2,column3)
More info here: http://dev.mysql.com/doc/refman/5.0/en/load-data.html
Here is my MySQL table structure
id | tracking_number | order_id
Here is the structure of the CSV file:
(Sometimes the order_id is missing, and this seems to be causing issues)
"1R2689Y603406","33097"
"1R2689Y603404","33096"
"1R2689Y603414",
"1R2689Y603429","33093"
"1R2689Y603452",
Here is my current SQL Query which isn't working:
(The file is being uploaded, and is being read correctly, it's the query itself which is causing issues)
$sql = 'LOAD DATA LOCAL INFILE "'.$_FILES['my_file']['tmp_name'].'"
INTO TABLE table_tracking
FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY "\""
LINES TERMINATED BY "\n"
(tracking_number,order_id)';
mysql_query($sql) or die(myqsl_error());
What is wrong with my query? Thanks for any help!
Edit: Changed CSV structure to represent missing data that sometimes occurs.
Changed query to match the one I am now using
Just trying here (not 100% confident) but did you try adding the destination columns like this:
mysql_query("LOAD DATA LOCAL INFILE '".$_FILES['my_file']['tmp_name']."'
INTO TABLE table_tracking FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
(tracking_number,order_id)");
You have three columns in your table and are only providing two data.
This worked for me:
$sql = 'load data local infile "c:/users/ramon/desktop/1.csv"
into table test fields terminated by ","
optionally enclosed by "\""
lines terminated by "\n"';
mysql_query($sql) or die(myqsl_error());
You also probably need to make sure that your third column has a default value.
Instead of $_FILES['my_file']['tmp_name'], try
dirname(__FILE__)."/".$_FILES['my_file']['tmp_name']
dirname(__FILE__) will give the full directory path like 'C:/path/to/directory'.
What are the potential choices for LINES TERMINATED BY in MySQL?
I have a CSV file that I am trying to upload and neither \r\n, \n, or \r work.
Try inspecting the file using a binary-displaying utility like od (http://swoolley.org/man.cgi/1/od). That will show you imediately how your lines are terminated.
I agree with brian.
Query which i used to work are as follows.
load data local infile 'test.csv'
into table test_field fields
terminated by ','
lines terminated by '\n'