I'm using the following to load a csv file to a mysql database:
LOAD DATA LOCAL INFILE 'file_location.csv'
INTO TABLE social_spend
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 ROWS
I have a text field that is written as: 'D,i,Y', which, after being uploaded shows in my database as the same:'D,i,Y'
I have a numeric field that is written as: '1,234' that is automatically trunacted on upload to: '1'
Why are text field commas preserved, but numeric fields are not?
When MySQL encounters a text string in a numeric context, it coerces it to a number. Sloppily.
So if you use the string 00134abc in a numeric context, you get the number 134. And, in your case it sees 1,234 and coerces it to 1.
Yeah, we all know about this being a pain in the xxx neck.
You may be able to do your load with something like this (NOT debugged!)
LOAD DATA LOCAL INFILE 'file_location.csv'
INTO TABLE social_spend
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 ROWS
(col1, col2, #yourNumber, col4)
SET numberColumn = CAST(REPLACE(#yourNumber, ',', '') AS SIGNED INTEGER);
Between the parentheses you can list columns of your table to correspond to columns in your CSV. If you put #something you can use it in an expression in a SET clause. The expression I showed strips out the commas from your numbers.
Related
There is an example of my csv file:
direction,latitude,longitude,metrictimestamp,odometer,routecode,speed,device_deviceid,vehicle_vehicleid
180,-3.724404,-38.557694,20180201025934,161245809,0,0,148469,33089
0,-3.878595,-38.533493,20180201025955,19030291,0,0,8155064,34489
135,-3.744851,-38.545571,20180201025959,55697826,0,3,134680,32040
And there is the execution of the import query:
Any ideas of why this lines are being skipped??
You can use SHOW WARNINGS after LOAD DATA to show what went wrong. Please note that by default it doesn't show all 35 million warnings but just a first few.
That said, your data doesn't include anything for 'id' so that's probably at least part of the issue. Another is that you don't enclose the fields in double quotes but do have ENCLOSED BY '"'
See LOAD DATA documentation to see how you can choose which fields to populate:
LOAD DATA LOCAL INFILE 'C:/Users/Public/Downloads/Dados.csv'
INTO TABLE data
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(direction, latitude, longnitude, metrictimestamp,
odometer, routecode, speed, deviceid, vehicleid)
As id is AUTO INCREMENT, it will be populated automatically.
I'm trying to load csv files into mysql table.
Delimiter : ,(comma)
As part of the source data few of the field values are enclosed in double quotes and inside the double quotes we have ,
There are few records for which / is part of the field data and we need to escape it.
By default / is getting escaped and when I specified the " as escape character " is getting escaped. As we have multiple special characters inside the same file, we need to escape multiple special characters.
Any suggestion
Eg:
id name location
1 A "Location , name here"
2 B "Different Location"
3 C Another Location
4 D Location / with escape character
LOAD DATA LOCAL INFILE 'data.csv' INTO TABLE table_name FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES;
I think it's not possible. Referring to LOAD DATA reference
Any of the field- or line-handling options can specify an empty string (''). If not empty, the FIELDS [OPTIONALLY] ENCLOSED BY and FIELDS ESCAPED BY values must be a single character.
Only a single char is supported for ESCAPED BY field.
My proposal is to use any programming language (e.g. PHP, C# etc.) for opening and processing file line-by-line using regexp
I'm exporting a database report with a shell file. If I run the query in PHPMyAdmin the file comes out fine, new lines at the end of each row in the database only.
However when I run the query in my shell script using outfile to generate the file I get /n, /r and /r/n in some of the columns content. I can't work out what causes this or how to avoid it.
The issue only seems to be caused in the colour column which is the third in the example export.
Query:
mysql $MYSQLOPTS << EOFMYSQL
SELECT Product_Name, Item_Size, Item_Colour, Item_Price, Current_Stock, Item_Price * Current_Stock AS Stock_Value
FROM Items
ORDER BY Product_Name
INTO OUTFILE '$FILE'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
EOFMYSQL
Example result:
"Scarf_in_Peach","ONE SIZE","12/04-B2B2 ",10.00,3,30.00
"Scarf_in_Pink","ONE SIZE ","11/06-odds-C1C12100",10.00,0,0.00
"Scarf_in_Red","ONE SIZE ","11/06-B7B2-C1C12100",10.00,0,0.00
"Scarf_in_Sand_","ONE SIZE","11/06-B1I3-C1C12100
",10.00,0,0.00
"Scarf_in_Sand_/_Blue_Flowers","ONE SIZE","12/04-B2E2-C1C12100 ",10.00,4,40.00
"Scarf_in_Teal","ONE SIZE","11/06-B5G1-C1C12100
",10.00,0,0.00
"Scarf_in_Teal_/_Red_Flowers","ONE SIZE","12/04 - B2B2 ",10.00,1,10.00
"Sunrise_Skinnies","16","ODD-R1S009-1-BLUE",20.00,0,0.00
"Sunrise_Skinnies","8","ODD-R1S009-1
BLUE",20.00,0,0.00
You have 2 options:
Replace carriage return and line feed characters with empty string within your query. Pro: it is completely up to you what characters you filter out and from which fields. Con: you have to create expression for each affected field manually.
Use FIELDS ESCAPED BY character option of the SELECT ... INTO OUTFILE ... command:
FIELDS ESCAPED BY controls how to write special characters. If the
FIELDS ESCAPED BY character is not empty, it is used when necessary to
avoid ambiguity as a prefix that precedes following characters on
output:
The FIELDS ESCAPED BY character
The FIELDS [OPTIONALLY] ENCLOSED BY character
The first character of the FIELDS TERMINATED BY and LINES TERMINATED BY values
ASCII NUL (the zero-valued byte; what is actually written following the escape character is ASCII “0”, not a zero-valued byte)
The FIELDS TERMINATED BY, ENCLOSED BY, ESCAPED BY, or LINES TERMINATED
BY characters must be escaped so that you can read the file back in
reliably. ASCII NUL is escaped to make it easier to view with some
pagers.
Pro: this is a fast and standard approach, that you can easily apply to all export functionality using this approach. Con: less flexible. For example, if the lines terminated by option is set to \n, then \r is not going to be escaped, which can still cause some issues on some systems.
I have imported a CSV file where a specific column has a decimal number.
In the original excel file (before saving it to a CSV), the first number of the column shows up as 218,790. When I choose the cell, the number shows up as 218790.243077911.
In the CSV file the number shows up as 218790 and when I choose the cell it is 218,790.
When I import the file on mySQL and show the table I created, the number shows up as 218.000000000.
Here is the code I used:
create table Apolo_Test(
Leads decimal (15,9)
);
LOAD DATA LOCAL INFILE 'C:/Users/SCRIPTS/file.csv'
INTO TABLE Apolo_Test
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 7 ROWS
;
I tried updating the format with this :
update Apolo_Test set Leads = format(Leads, 10, 'de_DE');
but it did not work. I have never had a case where files had a comma before. I guess it is the UK version of numerical fields.
How is it possible to make it work on mySQL without using any MACROS in excel?
UPD:
It works but I get some warnings although I double checked the csv file and the fields :
create table Apolo_Test(
Ad_Group varchar(50),
Impacts int,
Leads decimal (10,3)
);
LOAD DATA LOCAL INFILE 'C:/Users/me/Desktop/SCRIPTS/11/Adalyser.csv'
INTO TABLE Apolo_Test
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 7 ROWS
(Ad_Group, Impacts, #Leads)
SET Leads = replace(#Leads, ',', '');
;
alter table Apolo_Test ADD IPL decimal (10,6) after Leads;
update Apolo_Test set IPL=Impacts/Leads;
select * from Apolo_Test;
You have to use this syntax:
LOAD DATA LOCAL INFILE 'C:/path/to/mytable.txt' IGNORE
INTO TABLE mytable
FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\r\n'
(int_col, #float_col)
SET float_col = replace(#float_col, ',', '.');
For more information read here
The thousands-separator should not matter when moving data around -- Excel internal values and CSV files and MySQL internal values do not include it. Only "formatted" output includes it. And you should not use formatted output for moving numbers around.
Be careful with locale, such as de_DE.
The German "218.790" is the same as English "218,790".
"218790.243077911" is likely to be what Excel had internally for the number.
"218,790" is likely to be the English representation on the screen; note the English thousands separator.
In the CSV file the number shows up as 218790 and when I choose the cell it is 218,790.
What do you mean? Perhaps that there no comma or dot in the file, itself? But what you mean by "choose the cell"?
I can't see how to get "218.000000000" without truncation going on somewhere.
I've a csv file containing data in this format:
SATURN,6459,"50,486",27184
I'm using this command to import this file into the table:
LOAD DATA LOCAL INFILE 'D:\\temp.csv' INTO TABLE `test`.`tmp` FIELDS ESCAPED BY '\\'
TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n'
(`description`,`file_no`,`net`,`run_no`);
all the fields are being imported correctly but the net column always having the data like
50.00 // it should be 50,486
Data type of this column is Decimal(10,2). I've also tried Numeric(10,2) but no luck. Any help is highly appreciated. Thanks a lot in advance.
Please have a try with this one:
LOAD DATA LOCAL INFILE 'D:\\temp.csv' INTO TABLE `test`.`tmp` FIELDS ESCAPED BY '\\'
TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n'
(`description`, `file_no`, #a_variable, `run_no`)
SET `net` = REPLACE(#a_variable, ',', '');
MySQL cannot deal with commas as 1000 separators when inserting numbers. You need to parse out these commas before loading.
If you need the numbers formatted this way when querying, use FORMAT(). For examples see Apply comma to number field in MySQL