I am trying to write SP query results to file nad get a consistent error code: 1086 File already exists. This is despite the fact that the file name has to be unique because it's containing a random generate number.
Here is part of my code:
SET fullOutputPath = CONCAT(user,'_',FLOOR(1000+RAND()*9999),'.txt');
SELECT fullOutputPath;
-- write the resultset to the file
SELECT node_concat
INTO OUTFILE ",fullOutputPath,"
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY ''
LINES TERMINATED BY '\n'
FROM sn_neighbour_tmp;
Any ideas ?
You need to use 13.5 SQL Syntax for Prepared Statements.
Example:
...
SET `fullOutputPath` := CONCAT(USER, '_', FLOOR(1000 + RAND() * 9999), '.txt');
SET #`qry` := CONCAT('SELECT `node_concat`
INTO OUTFILE ', `fullOutputPath`, '
FIELDS TERMINATED BY \',\'
OPTIONALLY ENCLOSED BY \'\'
LINES TERMINATED BY \'\n\'
FROM `sn_neighbour_tmp`');
PREPARE `stmt` FROM #`qry`;
SET #`qry` := NULL;
EXECUTE `stmt`;
DEALLOCATE PREPARE `stmt`;
...
Related
Working in the query-box of phpMyAdmin I want to write an outfile 'protokoll' for each table in a MySQL 5.5 database. As I have many databases that contain the same tables and are different by name only, I want the filename of the outfile to look like
/tmp/dbname_protokoll_tablename_.csv
This works:
SELECT DATABASE() into #client;
SET #dir = '/tmp/';
SET #table = 'Adressen';
SET #stmt = CONCAT( 'SELECT * from ', #table, ' WHERE MarkDel=1 into outfile ''', #dir, #client , '_Protokoll_', #table , '.csv'' CHARACTER SET utf8 FIELDS TERMINATED BY '','' OPTIONALLY ENCLOSED BY ''"'' LINES TERMINATED BY ''\r\n'' ');
PREPARE DoExport from #stmt;
EXECUTE DoExport;
DEALLOCATE PREPARE DoExport;
From what I read about prepare, I should be able to use a '?' inside the statement, like
SET #stmt = CONCAT( 'SELECT * from ? WHERE MarkDel=1 into outfile ...
and then execute this with a list of arguments like
EXECUTE DoExport USING 'Adressen', 'Familien', 'Kinder';
but I can't get this to work, all I receive is an unspecific syntax error. How do I have to rewrite this?
I am looking for a way to add the current time to my filename after every new export.
Current code:
SELECT * INTO OUTFILE 'C:/Output/data_.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\r\n' FROM eloge_collector;
What i want to achieve:
SELECT * INTO OUTFILE 'C:/Output/data_2018-05-10-15-14.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\r\n' FROM collector;
or something like this.
you can use prepared statements to create your query string then executing it
https://dev.mysql.com/doc/refman/8.0/en/sql-syntax-prepared-statements.html
this is my solution
first create create variable to hold the query string.
use concat function to inject current date time to the query string.
use DATE_FORMAT and now functions to get current date time
set #sql = concat("SELECT * INTO OUTFILE 'C:/filePrefix_",DATE_FORMAT(NOW(), '%Y%m%d%H%i%s'),".fileExtension' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\r\n' FROM `table`");
prepare s1 from #sql; --create statment from variable
execute s1; -- execute prepared statements
Is there anyway to export data in MySQL table with column headers?
I find a way to do this by hard-coding the headers into query but if there are about 60 or even 100 columns in a table then it is impossible.
I tried the query below but I can't get the result as the first part of the query return a concatenated string of all headers in the table. It doesn't give me a desired result:
(select concat(group_concat(COLUMN_NAME separator ','), "\n")
from information_schema.COLUMNS
where table_name = '<table name>'
and table_schema = '<DB name>'
order by ORDINAL_POSITION)
union all
(select * from <table name> into outfile "E:\\test.csv" fields terminated by "," lines terminated by "\n");
There are few ways to fetch the field names of table in mysql, but below method works the best for outfile.
The filename of the csv file is dynamically created based on the datestamp, and for this prepared statement is used.
-- ensure mysql user has write permission on below location
SET #effectiveFileName = CONCAT('/home/myhome/temp-dev/', 'mytable','_', DATE_FORMAT(NOW(), '%Y-%m-%d'), '.csv');
-- group concat default is 1024, to avoid field names getting truncated we increase this value
SET SESSION group_concat_max_len = 10000;
SET #queryStr = (
SELECT
CONCAT('SELECT * INTO OUTFILE \'',
#effectiveFileName,
'\' FIELDS TERMINATED BY \',\' OPTIONALLY ENCLOSED BY \'"\' LINES TERMINATED BY \'\n\' FROM (SELECT ',
GROUP_CONCAT(CONCAT('\'', COLUMN_NAME, '\'')),
'UNION ALL SELECT * FROM myschema.mytable WHERE myschema.mytable.myfield <=\'',
CURDATE(),
'\') as tmp')
FROM
INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_NAME = 'mytable' AND
TABLE_SCHEMA = 'myschema'
ORDER BY ORDINAL_POSITION
);
PREPARE stmt FROM #queryStr;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
Below is simple version of above query with static file name.
-- group concat default is 1024, to avoid field names getting truncated we increase this value
SET SESSION group_concat_max_len = 10000;
SELECT
CONCAT('SELECT * INTO OUTFILE \'/home/myuser/myfile.csv\' FIELDS TERMINATED BY \',\' OPTIONALLY ENCLOSED BY \'"\' ESCAPED BY \'\' LINES TERMINATED BY \'\n\' FROM (SELECT ',
GROUP_CONCAT(CONCAT('\'', COLUMN_NAME, '\'')),
' UNION select * from YOUR_TABLE) as tmp')
FROM
INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_NAME = 'YOUR_TABLE'
AND TABLE_SCHEMA = 'YOUR_SCHEMA'
ORDER BY ORDINAL_POSITION;
(SELECT 'Order Number','Order Date','Status')
UNION
(SELECT orderNumber,orderDate, status
FROM orders
INTO OUTFILE 'C:/tmp/orders.csv'
FIELDS ENCLOSED BY '"' TERMINATED BY ';' ESCAPED BY '"'
LINES TERMINATED BY '\r\n');
I have this event
CREATE DEFINER=`frontend`#`%` EVENT `general_log_event`
ON SCHEDULE EVERY 5 MINUTE STARTS '2013-03-05 16:08:54' DO BEGIN
SET #query = CONCAT("
SELECT * INTO OUTFILE '/Users/Admin/logs/log_",
DATE_FORMAT(NOW(),"%M-%d-%y_%h-%m-%s"), ".csv'",
"
FIELDS TERMINATED BY ','
ENCLOSED BY '\"\'",
"LINES TERMINATED BY '\\n\'",
"FROM mysql.general_log;");
PREPARE statement FROM #query;
EXECUTE statement;
END
but every time it runs it gives me this error
[ERROR] Event Scheduler: [root#%][ge.general_log] File '/Users/Admin/logs/log_03-05-13_04-03-54.csv' already exists
its the same exact error every time. it shouldnt have the same file it should be changing every time.
used FLOOR(1052 + RAND() * 3564) to create a random value and it seems to be working. Here is the final piece.
SET #query = CONCAT("
SELECT SQL_NO_CACHE * INTO OUTFILE '/Users/Admin/logs/General_Log/log_",
DATE_FORMAT(NOW(),"%M-%d-%y_%h-%m-%s_"), FLOOR(1052 + RAND() * 3564), ".csv'",
"
FIELDS TERMINATED BY ','
ENCLOSED BY '\"\'",
"LINES TERMINATED BY '\\n\'",
"FROM mysql.general_log;");
PREPARE statement FROM #query;
EXECUTE statement;
SELECT #query;
Is there a way to do something like the following ? which doesn't work but shows what I want to do
SET #OutputPath = '/Users/jo/Documents'
SET #fullOutputPath = CONCAT(#OutputPath,'/','filename.csv')
SET #fullOutputPath2 = CONCAT(#OutputPath,'/','filename2.csv')
SELECT * INTO OUTFILE #fullOutputPath
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
FROM database.tableName;
SELECT * INTO OUTFILE #fullOutputPath2
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
FROM database.tableName2;
Edit: Saving data(e.g. a table) into file without using variable (only constant values)
-- folder_path could could be like => c:/users/sami
-- choose the directory/folder already available in system
-- and make sure you have access to write the file there
SELECT * INTO OUTFILE 'folder_path/filename.csv'
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
FROM database.tableName;
Now using variable
Whenever you have to use a variable name in sql, you need dynamic sql (which is applicable in stored procedures only, neither in simple sql query nor in triggers or functions)
SET #OutputPath := 'Users/jo/Documents'; //or any folder_path
SET #fullOutputPath := CONCAT(#OutputPath,'/','filename.csv');
SET #fullOutputPath2 := CONCAT(#OutputPath,'/','filename2.csv');
set #q1 := concat("SELECT * INTO OUTFILE ",#fullOutputPath,
" FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
FROM database.tableName");
set #q2 := concat("SELECT * INTO OUTFILE ",#fullOutputPath2,
" FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
FROM database.tableName2");
prepare s1 from #q1;
execute s1;deallocate prepare s1;
prepare s1 from #q2;
execute s1;deallocate prepare s1;
As you had both ' and " in your query already, so I concatenated your query using " and used \ to escape your original " to ensure its use as a literal character and not used for concatenation
I just told the use of variable in sql. First You should make sure if your query works like example at the top (without using variable)
Conclusion: If your above query works fine then my told dynamic sql will work as well given that you are using it in some stored procedure.
I have a low carma so I'm posting an answer that should go as a comment to Sami's post - you need to enclose the file name by quotes (note added ' before and after #fullOutputPath):
set #q1 := concat("SELECT * INTO OUTFILE '",#fullOutputPath,
"' FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
FROM database.tableName");
If you want to do this from bash, i.e. export some data from mysql in csv to a file with dynamic name, it maybe easier and more readable like the following.
The SQL with embedded bash variables:
where (e.timestamp >= ${begin_ts} and e.timestamp < ${end_ts}) order by ed.timestamp ASC ) a
INTO OUTFILE '${export_path}' FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n';
And the bash script that runs the sql file. Notice the envsubst command that evaluates the sql script and substitutes the variables.
#!/bin/bash
mysql_db="dbname"
mysql_user="mysqlpass"
mysql_pass="password"
export_path="./data.csv"
begin_ts="1478278490"
current_ts=$(date +%s -u)
sql=`export_path=${export_path} begin_ts=${last_ts} end_ts=${current_ts} envsubst < export.sql`
mysql $mysql_db -u $mysql_user -p$mysql_pass -e"${sql}"
You cannot do it in mysql CLI but in this way it works
mysql -e "SELECT * FROM database.tableName;" -u user -p database > filename.csv