I have a MySQL table that needs to be exported into several separate files. The table should be grouped by a particular column and files should have names of the corresponding values of this column. Format is not relevant I just need a suitable technique, program, whatever.
Any help would be much appreciated!
If it's less than 10 files or so, it's easy to manually craft a script like:
SELECT *
FROM YourTable
WHERE col1 = 'alfa'
INTO OUTFILE 'c:\result-alfa.txt'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
If typing it out is too tedious, consider a query like this to generate the script:
SELECT concat('SELECT * FROM YourTable WHERE col1 = ''',
col1, ''' INTO OUTFILE '''c:\result-', col1, '.txt'' ',
'FIELDS TERMINATED BY '','' OPTIONALLY ENCLOSED BY ''"''',
'LINES TERMINATED BY ''\n'';')
FROM YourTable
GROUP BY col1
Related
1st question: I am trying to load test1.csv, test2.csv and test3.csv to table1, table2 and table3 respectively using SQLLDR. Please bear with my lack if knowledge in this area, I couldn't get it quite right while defining this in .ctl file, only I can think of is the below code but this is not correct. so my question is how can I make this right or is this possible?
OPTIONS (SKIP=1)
LOAD DATA
INFILE 'test1.csv'
INFILE 'test2.csv'
INFILE 'test2.csv'
TRUNCATE
INTO TABLE table1
fields terminated by ',' optionally enclosed by '"'
TRAILING NULLCOLS
(
Col1 "TRIM(:Col1)",
Col2 "TRIM(:Col2)"
)
INTO TABLE table2
fields terminated by ',' optionally enclosed by '"'
TRAILING NULLCOLS
(
Colx "TRIM(:Colx)",
Coly "TRIM(:Coly)"
)
INTO TABLE table3
fields terminated by ',' optionally enclosed by '"'
TRAILING NULLCOLS
(
Colp "TRIM(:Colp)",
Colq "TRIM(:Colq)"
)
2nd question: This is an alternative to this first question. Since I couldn't figure it out the first one, what I have done is splitting the loads for each table into multiple .ctl files and calling those all three in a .bat file. This works at least but my question is there a way to process all these 3 .ctl files in a session without mentioning user/password 3 times as below?
sqlldr userid=user/pass#server control=test1.ctl
sqlldr userid=user/pass#server control=test2.ctl
sqlldr userid=user/pass#server control=test3.ctl
If there is a field that can be used which can indicate which table that file's data is intended for, you can do something like this using multiple INFILE satements. Let's say the first field is that indicator and it will not be loaded (define it as a FILLER so sqlldr will ignore it):
...
INTO TABLE table1
WHEN (01) = 'TBL1'
fields terminated by ',' optionally enclosed by '"'
TRAILING NULLCOLS
(
rec_skip filler POSITION(1),
Col1 "TRIM(:Col1)",
Col2 "TRIM(:Col2)"
)
INTO TABLE table2
WHEN (01) = 'TBL2'
fields terminated by ',' optionally enclosed by '"'
TRAILING NULLCOLS
(
rec_skip filler POSITION(1),
Colx "TRIM(:Colx)",
Coly "TRIM(:Coly)"
)
INTO TABLE table3
WHEN (01) = 'TBL3'
fields terminated by ',' optionally enclosed by '"'
TRAILING NULLCOLS
(
rec_skip filler POSITION(1),
Colp "TRIM(:Colp)",
Colq "TRIM(:Colq)"
)
So logically, each INTO table WHEN section handles each file. Not that flexible though and arguably harder to maintain. For ease of maintenance you may just want to have a control file for each file? If all files and tables are the same layout you could also load them all into the same staging table (with the indicator) for ease of loading, then split them out programatically into the separate tables afterwards. Pros of that method are faster and easier loading and more control over the splitting into the separate table part of the process. Just some other ideas. I have done each method, depends on the requirements and what you are able to change.
Is there a way to explicitly set formatting for DATE/DATETIME/TIMESTAMP type columns in MySQL INTO OUTFILE command?
select * from orders LIMIT 100
INTO OUTFILE 'c:/tmp/orders.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '
';
Maybe in shell or command itself?
The key thing here is that the first part of the query, the select is pretty much a full featured standard select. So instead of select * youo could have
SELECT col1, col2, date_format(date_col, 'someformat') ...
INTO OUTFILE 'c:/tmp/orders.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '
';
And produce output in the format of your choice. DATE_FORMAT reference here:
https://dev.mysql.com/doc/refman/5.7/en/date-and-time-functions.html#function_date-format
I have loaded a table in mysql (xampp) with around 40,000,000 rows, with it I created another table with around 6,000,000 rows and I exported it to a csv file using:
(SELECT ...)
UNION
(SELECT ...
FROM ctr_train0
INTO OUTFILE 'C:/.../file.csv'
FIELDS ENCLOSED BY '"' TERMINATED BY ',' ESCAPED BY '"'
LINES TERMINATED BY '\n');
no errors, but this command creates a csv file with around 200,000 rows less than the original table, What happens? How can I export all the 6,000,000 rows?. Thanks in advance.
My best guess -- given the limited information -- is the use of union. This removes duplicates from the output, and the duplicates are both between tables and within tables. So, if your data has duplicates, this removes them.
Try running the query with union all instead:
(SELECT ...)
UNION ALL
(SELECT ...
FROM ctr_train0
INTO OUTFILE 'C:/.../file.csv'
FIELDS ENCLOSED BY '"' TERMINATED BY ',' ESCAPED BY '"'
LINES TERMINATED BY '\n');
This is a sample table of my original table USER_DETAILS, i want to export the user_id and user_phone fields to a csv file for further implementation in my project.
user_id user_phone
1 9977660050
2 9977660051
3 9977660042
4 9977660080
P.S.: Please answer me the query for this, instead of giving abrupt answers and suggestions. i guess the table is quite clear to understand.
If you don't really need this to be in PHP, just run:
SELECT user_id, user_phone INTO OUTFILE '/your/filepath'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM user_details;
Obviously, replace '/your/filepath' with the path to the file you want to save to.
Not sure if you want to do this in PHP or just output it to a CSV file. Here's how to do the latter:
SELECT user_id, user_phone
FROM user_details
INTO OUTFILE '/tmp/user_details.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
SELECT 'User','User Phone'
UNION
SELECT user_id,user_phone INTO OUTFILE 'users.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
FROM USER_DETAILS;
I want to modify the following code:
SELECT * INTO OUTFILE 'C:\\my_excel_table.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
FROM mytable;
so that it dumps the result into longtext/variable rather than to a output file.
Not sure if you can, but there is the INSERT ... SELECT FROM syntax, which lets you insert the results of a select into another table. Problem is that SELECT is going to return rows, which would be inserted one-by-one into the other table, instead of going in as a single long chunk of text.
use insert select
INSERT INTO mytable (my_field) select my_other_field from my_other_table