Is there anyway to export data in MySQL table with column headers?
I find a way to do this by hard-coding the headers into query but if there are about 60 or even 100 columns in a table then it is impossible.
I tried the query below but I can't get the result as the first part of the query return a concatenated string of all headers in the table. It doesn't give me a desired result:
(select concat(group_concat(COLUMN_NAME separator ','), "\n")
from information_schema.COLUMNS
where table_name = '<table name>'
and table_schema = '<DB name>'
order by ORDINAL_POSITION)
union all
(select * from <table name> into outfile "E:\\test.csv" fields terminated by "," lines terminated by "\n");
There are few ways to fetch the field names of table in mysql, but below method works the best for outfile.
The filename of the csv file is dynamically created based on the datestamp, and for this prepared statement is used.
-- ensure mysql user has write permission on below location
SET #effectiveFileName = CONCAT('/home/myhome/temp-dev/', 'mytable','_', DATE_FORMAT(NOW(), '%Y-%m-%d'), '.csv');
-- group concat default is 1024, to avoid field names getting truncated we increase this value
SET SESSION group_concat_max_len = 10000;
SET #queryStr = (
SELECT
CONCAT('SELECT * INTO OUTFILE \'',
#effectiveFileName,
'\' FIELDS TERMINATED BY \',\' OPTIONALLY ENCLOSED BY \'"\' LINES TERMINATED BY \'\n\' FROM (SELECT ',
GROUP_CONCAT(CONCAT('\'', COLUMN_NAME, '\'')),
'UNION ALL SELECT * FROM myschema.mytable WHERE myschema.mytable.myfield <=\'',
CURDATE(),
'\') as tmp')
FROM
INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_NAME = 'mytable' AND
TABLE_SCHEMA = 'myschema'
ORDER BY ORDINAL_POSITION
);
PREPARE stmt FROM #queryStr;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
Below is simple version of above query with static file name.
-- group concat default is 1024, to avoid field names getting truncated we increase this value
SET SESSION group_concat_max_len = 10000;
SELECT
CONCAT('SELECT * INTO OUTFILE \'/home/myuser/myfile.csv\' FIELDS TERMINATED BY \',\' OPTIONALLY ENCLOSED BY \'"\' ESCAPED BY \'\' LINES TERMINATED BY \'\n\' FROM (SELECT ',
GROUP_CONCAT(CONCAT('\'', COLUMN_NAME, '\'')),
' UNION select * from YOUR_TABLE) as tmp')
FROM
INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_NAME = 'YOUR_TABLE'
AND TABLE_SCHEMA = 'YOUR_SCHEMA'
ORDER BY ORDINAL_POSITION;
(SELECT 'Order Number','Order Date','Status')
UNION
(SELECT orderNumber,orderDate, status
FROM orders
INTO OUTFILE 'C:/tmp/orders.csv'
FIELDS ENCLOSED BY '"' TERMINATED BY ';' ESCAPED BY '"'
LINES TERMINATED BY '\r\n');
Related
Working in the query-box of phpMyAdmin I want to write an outfile 'protokoll' for each table in a MySQL 5.5 database. As I have many databases that contain the same tables and are different by name only, I want the filename of the outfile to look like
/tmp/dbname_protokoll_tablename_.csv
This works:
SELECT DATABASE() into #client;
SET #dir = '/tmp/';
SET #table = 'Adressen';
SET #stmt = CONCAT( 'SELECT * from ', #table, ' WHERE MarkDel=1 into outfile ''', #dir, #client , '_Protokoll_', #table , '.csv'' CHARACTER SET utf8 FIELDS TERMINATED BY '','' OPTIONALLY ENCLOSED BY ''"'' LINES TERMINATED BY ''\r\n'' ');
PREPARE DoExport from #stmt;
EXECUTE DoExport;
DEALLOCATE PREPARE DoExport;
From what I read about prepare, I should be able to use a '?' inside the statement, like
SET #stmt = CONCAT( 'SELECT * from ? WHERE MarkDel=1 into outfile ...
and then execute this with a list of arguments like
EXECUTE DoExport USING 'Adressen', 'Familien', 'Kinder';
but I can't get this to work, all I receive is an unspecific syntax error. How do I have to rewrite this?
I have some queries like this
(SELECT 'title', 'created_at', 'modified_at')
UNION
(select title, created_at, modified_at from table1 order by created_at
INTO OUTFILE 'result.csv'
CHARACTER SET utf8
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' LINES TERMINATED BY '\r\n');
and the the result is more than 10000 rows.
Is there a way to split this into multiple files?
I would like to limit the number of rows.
I know that I can do something like this:
select * from table1 limit 0, 1000 into outfile result1.csv
select * from table1 limit 1000, 1000 into outfile result2.csv
select * from table1 limit 2000, 1000 into outfile result3.csv
...
since there's more than 10000 rows, I am wondering if I can use some kind of loop.
Any advice and help will be greatly appreciated!
::EDIT::
I wrote repeat procedure but it gives me a syntax error.
DELIMITER ;;
CREATE PROCEDURE `reapeatloop`()
BEGIN
declare x int;
declare filename varchar(255);
declare lowlimit int;
SET x = 0;
REPEAT
set lowlimit = x*1000;
(select * from table1 limit lowlimit, 1000
INTO OUTFILE concat('post',x,'.csv')
CHARACTER SET utf8
FIELDS TERMINATED BY ',' ENCLOSED BY '"' ESCAPED BY '"' LINES
TERMINATED BY '\r\n');
SET x = x + 1;
UNTIL x > 19 END REPEAT;
END;;
DELIMITER ;
Inside this query, I used local variable to define csv and limit value and this gives syntax error. can anyone see what's wrong with this query?
as soon as I change this variable to some number, or string, it creates procedure successfully.
You have some mistake in your procedure. Specially check the SELECT INTO OUTFILE.
Be careful about the position of FROM.
In your case try this:
DROP PROCEDURE reapeatloop;
DELIMITER $$
CREATE PROCEDURE `reapeatloop`()
BEGIN
declare x int;
declare filename varchar(255);
declare lowlimit int;
SET x = 0;
REPEAT
set lowlimit = x*1000;
SET #fullOutputPath := CONCAT('"D:/post',x,'.csv"');
set #q1 := concat("select * INTO OUTFILE ", #fullOutputPath,
" CHARACTER SET utf8 FIELDS TERMINATED BY ',' ENCLOSED BY '\"' ESCAPED BY '\"' LINES TERMINATED BY '\r\n'
from table1 limit ", lowlimit, ",1000");
select #q1;
prepare s1 from #q1;
execute s1;deallocate prepare s1;
SET x = x + 1;
UNTIL x > 19 END REPEAT;
END$$
DELIMITER ;
I created a Stored procedure with a cursor in order to get the info from a few tables into a temp table and then export the result into multiple files, one for each Client ID, the cursor and the loop get all the info right but I'm having syntax issues when creating each file
BEGIN
DECLARE id_cli INT;
DECLARE cur_id_cli CURSOR FOR SELECT id FROM cliente ORDER BY id;
OPEN cur_id_cli;
read_loop: LOOP
FETCH cur_id_cli INTO id_cli;
DROP TEMPORARY TABLE IF EXISTS tmp_reporte_enviadas_sucursal_mensual;
CREATE TEMPORARY TABLE tmp_reporte_enviadas_sucursal_mensual
SELECT * FROM
((SELECT 'Promo_id', 'Mensaje', 'Sucursal_id', 'Direccion') UNION ALL (SELECT
p.id,
p.mensaje,
s.id,
s.direccion
FROM
Usuario_Promo AS up, Promo_Sucursal AS ps, Cliente_Sucursal AS cs, Cliente AS c, Promo AS p, Sucursal AS s
WHERE
p.id = ps.Promo_id
AND up.promo_id = ps.id
AND up.recibido = 1
AND ps.Cliente_Sucursal_id = cs.id
AND cs.cliente_id = id_cli
AND DATE(up.fecha_recibido) BETWEEN (NOW() - INTERVAL 30 DAY) AND NOW()
AND s.id = cs.sucursal_id
ORDER BY p.id)) AS tmp_reporte_enviadas_sucursal_mensual;
SET #idCli = CAST(id_cli AS CHAR);
SET #fullOutputPath = CONCAT('D:/Octagon/Apps/Flyermob/Clientes/Reportes_Sql/reporte_enviadas_sucursal_mensual_',#idCli,'.csv');
SELECT * FROM
INTO OUTFILE #fullOutputPath
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
FROM tmp_reporte_enviadas_sucursal_mensual;
SELECT * FROM tmp_reporte_enviadas_sucursal_mensual;
END LOOP;
CLOSE cur_id_cli;
END
here's where I'm having the issue:
SELECT * FROM
INTO OUTFILE #fullOutputPath
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
FROM tmp_reporte_enviadas_sucursal_mensual;
Whenever I try to save, it says that there's a syntax problem near #fullOutputPath if I remove this section, the procedure run perfectly and I get all the results for each Client Id.
Just to get it out of the way id_cli is a primary key, so there's no chance of repeating and I have full permissions on #fullOutputPath.
Try this:
SELECT * FROM tmp_reporte_enviadas_sucursal_mensual
INTO OUTFILE #fullOutputPath
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"';
I managed to make it work with
SET #idCli = CAST(id_cli AS CHAR);
SET #fullOutputPath = CONCAT('flyermob/cliente/reporte_enviadas_sucursal_mensual_cliente_',#idCli,'.csv');
set #q1 := concat("SELECT * INTO OUTFILE '",#fullOutputPath,
"' FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
FROM tmp_reporte_enviadas_sucursal_mensual");
prepare s1 from #q1;
execute s1;
deallocate prepare s1;
It creates the file in the MySql default folder
I am trying to write SP query results to file nad get a consistent error code: 1086 File already exists. This is despite the fact that the file name has to be unique because it's containing a random generate number.
Here is part of my code:
SET fullOutputPath = CONCAT(user,'_',FLOOR(1000+RAND()*9999),'.txt');
SELECT fullOutputPath;
-- write the resultset to the file
SELECT node_concat
INTO OUTFILE ",fullOutputPath,"
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY ''
LINES TERMINATED BY '\n'
FROM sn_neighbour_tmp;
Any ideas ?
You need to use 13.5 SQL Syntax for Prepared Statements.
Example:
...
SET `fullOutputPath` := CONCAT(USER, '_', FLOOR(1000 + RAND() * 9999), '.txt');
SET #`qry` := CONCAT('SELECT `node_concat`
INTO OUTFILE ', `fullOutputPath`, '
FIELDS TERMINATED BY \',\'
OPTIONALLY ENCLOSED BY \'\'
LINES TERMINATED BY \'\n\'
FROM `sn_neighbour_tmp`');
PREPARE `stmt` FROM #`qry`;
SET #`qry` := NULL;
EXECUTE `stmt`;
DEALLOCATE PREPARE `stmt`;
...
I have this event
CREATE DEFINER=`frontend`#`%` EVENT `general_log_event`
ON SCHEDULE EVERY 5 MINUTE STARTS '2013-03-05 16:08:54' DO BEGIN
SET #query = CONCAT("
SELECT * INTO OUTFILE '/Users/Admin/logs/log_",
DATE_FORMAT(NOW(),"%M-%d-%y_%h-%m-%s"), ".csv'",
"
FIELDS TERMINATED BY ','
ENCLOSED BY '\"\'",
"LINES TERMINATED BY '\\n\'",
"FROM mysql.general_log;");
PREPARE statement FROM #query;
EXECUTE statement;
END
but every time it runs it gives me this error
[ERROR] Event Scheduler: [root#%][ge.general_log] File '/Users/Admin/logs/log_03-05-13_04-03-54.csv' already exists
its the same exact error every time. it shouldnt have the same file it should be changing every time.
used FLOOR(1052 + RAND() * 3564) to create a random value and it seems to be working. Here is the final piece.
SET #query = CONCAT("
SELECT SQL_NO_CACHE * INTO OUTFILE '/Users/Admin/logs/General_Log/log_",
DATE_FORMAT(NOW(),"%M-%d-%y_%h-%m-%s_"), FLOOR(1052 + RAND() * 3564), ".csv'",
"
FIELDS TERMINATED BY ','
ENCLOSED BY '\"\'",
"LINES TERMINATED BY '\\n\'",
"FROM mysql.general_log;");
PREPARE statement FROM #query;
EXECUTE statement;
SELECT #query;