Different number of rows exporting a table to a csv file in MySQL - mysql

I have loaded a table in mysql (xampp) with around 40,000,000 rows, with it I created another table with around 6,000,000 rows and I exported it to a csv file using:
(SELECT ...)
UNION
(SELECT ...
FROM ctr_train0
INTO OUTFILE 'C:/.../file.csv'
FIELDS ENCLOSED BY '"' TERMINATED BY ',' ESCAPED BY '"'
LINES TERMINATED BY '\n');
no errors, but this command creates a csv file with around 200,000 rows less than the original table, What happens? How can I export all the 6,000,000 rows?. Thanks in advance.

My best guess -- given the limited information -- is the use of union. This removes duplicates from the output, and the duplicates are both between tables and within tables. So, if your data has duplicates, this removes them.
Try running the query with union all instead:
(SELECT ...)
UNION ALL
(SELECT ...
FROM ctr_train0
INTO OUTFILE 'C:/.../file.csv'
FIELDS ENCLOSED BY '"' TERMINATED BY ',' ESCAPED BY '"'
LINES TERMINATED BY '\n');

Related

MySQL Query CSV to output not a file

I want to make a select to mySQL and return the resulting recordset as CSV but not saving it into a file.
I have this:
SELECT order_id,product_name,qty
FROM orders
WHERE foo = 'bar'
INTO OUTFILE '/var/lib/mysql-files/orders.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n';
I have tried to change it into:
SELECT order_id,product_name,qty
FROM orders
WHERE foo = 'bar'
INTO #out
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n';
But it doesn't work.
The use case is to give users the option to download CSV files from data driven web site. To do so I need to output the CSV file/data to the browser.
How to do?
You can not directly and you need to import data first in a table.Mysql command for this https://dev.mysql.com/doc/refman/8.0/en/load-data.html

MySQL Load Data Query: Issue with Backslash ( \ )

I have I have a huge csv file with 149 column and 25K+ rows to upload this file in MySQL table I am using MySQL LOAD DATA Query
MY Query is:
LOAD DATA local INFILE '/Dir/file.csv' INTO TABLE my_table FIELDS TERMINATED BY ',' ENCLOSED BY '"' ESCAPED BY '\\' LINES TERMINATED BY '\n' IGNORE 1 LINES
My query is working fine, the problem comes when my file has any Backslash (\) characters, the column value get disturb and file cell value not inserting in correct columns. is there any fix for this issue.
Thanks
LOAD DATA local INFILE '/Dir/file.csv' INTO TABLE my_table FIELDS TERMINATED BY ',' ENCLOSED BY '"' ESCAPED BY '\b' LINES TERMINATED BY '\n' IGNORE 1 LINES
Use '\b' as escape character instead of '\' in your query.

ignore first two characters on a column while importing csv to mysql

I am trying to import a csv file to mysql table, But I need to remove First two characters on particular column before importing to mysql.
This is my statment :
string strLoadData = "LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets FIELDS terminated by ',' ENCLOSED BY '\"' lines terminated by '\n' IGNORE 1 LINES (SiteId,DateTime,Serial,DeviceId,AgentAID,VehicleRegistration,CarPark,SpaceNumber,GpsAddress,VehicleType,VehicleMake,VehicleModel,VehicleColour,IssueReasonCode,IssueReason,NoticeLocation,Points,Notes)";
Column IssueReasoncode' has data like 'LU12' , But i need to remove the first 2 characters it should have only integers on it and not alpha numeric .
I need to remove 'LU' from that column.
Is it possible to write like this on left(IssueReasonCode +' '2). This column is varchar(45) and cant be changed now because of large data on it.
Thanks
LOAD DATA INFILE has the ability to perform a function on the data for each column as you read it in (q.v. here). In your case, if you wanted to remove the first two characters from the IssueReasonCode column, you could use:
RIGHT(IssueReasonCode, CHAR_LENGTH(IssueReasonCode) - 2)
to remove the first two characters. You specify such column mappings at the end of the LOAD DATA statement using SET. Your statement should look something like the following:
LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets
FIELDS terminated by ','
ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(SiteId, DateTime, Serial, DeviceId, AgentAID, VehicleRegistration, CarPark, SpaceNumber,
GpsAddress, VehicleType, VehicleMake, VehicleModel, VehicleColour, IssueReasonCode,
IssueReason, NoticeLocation, Points, Notes)
SET IssueReasonCode = RIGHT(IssueReasonCode, CHAR_LENGTH(IssueReasonCode) - 2)
Referencing this and quoting this example , you can try the below to see if it works
User variables in the SET clause can be used in several ways. The
following example uses the first input column directly for the value
of t1.column1, and assigns the second input column to a user variable
that is subjected to a division operation before being used for the
value of t1.column2:
LOAD DATA INFILE 'file.txt' INTO TABLE t1 (column1, #var1) SET
column2 = #var1/100;
string strLoadData = "LOAD DATA LOCAL INFILE 'E:/park/Export.csv' INTO TABLE tickets FIELDS terminated by ',' ENCLOSED BY '\"' lines terminated by '\n' IGNORE 1 LINES (SiteId,DateTime,Serial,DeviceId,AgentAID,VehicleRegistration,CarPark,SpaceNumber,GpsAddress,VehicleType,VehicleMake,VehicleModel,VehicleColour,#IRC,IssueReason,NoticeLocation,Points,Notes) SET IssueReasonCode = substr(#IRC,2) ;";

Is it possible to perform functions during LOAD DATA INFILE without creating a stored procedure?

I have two columns, q1 and q2, that I'd like to sum together and put in the destination column, q.
The way I do it now, I put the data in an intermediate table, then sum during loading, but I'm wondering if it's possible to do it during extraction instead?
Here's my script:
LOAD DATA INFILE 'C:\temp\foo.csv'
INTO TABLE new_foo
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(q1,q1)
INSERT INTO foo (q) SELECT q1+q2 AS q
FROM foo_temp;
Try:
LOAD DATA INFILE 'C:\temp\foo.csv'
INTO TABLE `foo`
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(#`q1`, #`q2`)
SET `q` = #`q1` + #`q2`;

How to output to csv file using mysql?

This is a sample table of my original table USER_DETAILS, i want to export the user_id and user_phone fields to a csv file for further implementation in my project.
user_id user_phone
1 9977660050
2 9977660051
3 9977660042
4 9977660080
P.S.: Please answer me the query for this, instead of giving abrupt answers and suggestions. i guess the table is quite clear to understand.
If you don't really need this to be in PHP, just run:
SELECT user_id, user_phone INTO OUTFILE '/your/filepath'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM user_details;
Obviously, replace '/your/filepath' with the path to the file you want to save to.
Not sure if you want to do this in PHP or just output it to a CSV file. Here's how to do the latter:
SELECT user_id, user_phone
FROM user_details
INTO OUTFILE '/tmp/user_details.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
SELECT 'User','User Phone'
UNION
SELECT user_id,user_phone INTO OUTFILE 'users.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
FROM USER_DETAILS;