can I use a variable to specify OUTFILE in mysql - mysql

Is there a way to do something like the following ? which doesn't work but shows what I want to do
SET #OutputPath = '/Users/jo/Documents'
SET #fullOutputPath = CONCAT(#OutputPath,'/','filename.csv')
SET #fullOutputPath2 = CONCAT(#OutputPath,'/','filename2.csv')
SELECT * INTO OUTFILE #fullOutputPath
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
FROM database.tableName;
SELECT * INTO OUTFILE #fullOutputPath2
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
FROM database.tableName2;

Edit: Saving data(e.g. a table) into file without using variable (only constant values)
-- folder_path could could be like => c:/users/sami
-- choose the directory/folder already available in system
-- and make sure you have access to write the file there
SELECT * INTO OUTFILE 'folder_path/filename.csv'
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
FROM database.tableName;
Now using variable
Whenever you have to use a variable name in sql, you need dynamic sql (which is applicable in stored procedures only, neither in simple sql query nor in triggers or functions)
SET #OutputPath := 'Users/jo/Documents'; //or any folder_path
SET #fullOutputPath := CONCAT(#OutputPath,'/','filename.csv');
SET #fullOutputPath2 := CONCAT(#OutputPath,'/','filename2.csv');
set #q1 := concat("SELECT * INTO OUTFILE ",#fullOutputPath,
" FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
FROM database.tableName");
set #q2 := concat("SELECT * INTO OUTFILE ",#fullOutputPath2,
" FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
FROM database.tableName2");
prepare s1 from #q1;
execute s1;deallocate prepare s1;
prepare s1 from #q2;
execute s1;deallocate prepare s1;
As you had both ' and " in your query already, so I concatenated your query using " and used \ to escape your original " to ensure its use as a literal character and not used for concatenation
I just told the use of variable in sql. First You should make sure if your query works like example at the top (without using variable)
Conclusion: If your above query works fine then my told dynamic sql will work as well given that you are using it in some stored procedure.

I have a low carma so I'm posting an answer that should go as a comment to Sami's post - you need to enclose the file name by quotes (note added ' before and after #fullOutputPath):
set #q1 := concat("SELECT * INTO OUTFILE '",#fullOutputPath,
"' FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
FROM database.tableName");

If you want to do this from bash, i.e. export some data from mysql in csv to a file with dynamic name, it maybe easier and more readable like the following.
The SQL with embedded bash variables:
where (e.timestamp >= ${begin_ts} and e.timestamp < ${end_ts}) order by ed.timestamp ASC ) a
INTO OUTFILE '${export_path}' FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n';
And the bash script that runs the sql file. Notice the envsubst command that evaluates the sql script and substitutes the variables.
#!/bin/bash
mysql_db="dbname"
mysql_user="mysqlpass"
mysql_pass="password"
export_path="./data.csv"
begin_ts="1478278490"
current_ts=$(date +%s -u)
sql=`export_path=${export_path} begin_ts=${last_ts} end_ts=${current_ts} envsubst < export.sql`
mysql $mysql_db -u $mysql_user -p$mysql_pass -e"${sql}"

You cannot do it in mysql CLI but in this way it works
mysql -e "SELECT * FROM database.tableName;" -u user -p database > filename.csv

Related

Adding current time to INTO OUTFILE 'C:/Output/data_.csv'

I am looking for a way to add the current time to my filename after every new export.
Current code:
SELECT * INTO OUTFILE 'C:/Output/data_.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\r\n' FROM eloge_collector;
What i want to achieve:
SELECT * INTO OUTFILE 'C:/Output/data_2018-05-10-15-14.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\r\n' FROM collector;
or something like this.
you can use prepared statements to create your query string then executing it
https://dev.mysql.com/doc/refman/8.0/en/sql-syntax-prepared-statements.html
this is my solution
first create create variable to hold the query string.
use concat function to inject current date time to the query string.
use DATE_FORMAT and now functions to get current date time
set #sql = concat("SELECT * INTO OUTFILE 'C:/filePrefix_",DATE_FORMAT(NOW(), '%Y%m%d%H%i%s'),".fileExtension' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\r\n' FROM `table`");
prepare s1 from #sql; --create statment from variable
execute s1; -- execute prepared statements

mysql stored procedure outfile

I am trying to write SP query results to file nad get a consistent error code: 1086 File already exists. This is despite the fact that the file name has to be unique because it's containing a random generate number.
Here is part of my code:
SET fullOutputPath = CONCAT(user,'_',FLOOR(1000+RAND()*9999),'.txt');
SELECT fullOutputPath;
-- write the resultset to the file
SELECT node_concat
INTO OUTFILE ",fullOutputPath,"
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY ''
LINES TERMINATED BY '\n'
FROM sn_neighbour_tmp;
Any ideas ?
You need to use 13.5 SQL Syntax for Prepared Statements.
Example:
...
SET `fullOutputPath` := CONCAT(USER, '_', FLOOR(1000 + RAND() * 9999), '.txt');
SET #`qry` := CONCAT('SELECT `node_concat`
INTO OUTFILE ', `fullOutputPath`, '
FIELDS TERMINATED BY \',\'
OPTIONALLY ENCLOSED BY \'\'
LINES TERMINATED BY \'\n\'
FROM `sn_neighbour_tmp`');
PREPARE `stmt` FROM #`qry`;
SET #`qry` := NULL;
EXECUTE `stmt`;
DEALLOCATE PREPARE `stmt`;
...

unable to add outfile prepared statement mysql

This is my query:
SET #query2 = CONCAT('
SELECT * FROM table_name
INTO OUTFILE "',arg_file_path,'/',#var_table_name,'_CURRENT_TIMESTAMP.csv" fields terminated by "," optionally enclosed by ''"'' lines terminated by "\n" ');
But it produces following output:
SELECT * FROM lcs_tbl_test
INTO OUTFILE "/data/test_outfile/lcs_tbl_test_CURRENT_TIMESTAMP.csv" fields terminated by "," optionally enclosed by '"' lines terminated by "
"
Basically I want "\n" to print as it is in my prepared statement. It is executing \n as line separator in current code.
You need to use back slash as double means for '\' you need to use '\'. Please use below statement.
SET #query2 = CONCAT('
SELECT * FROM table_name
INTO OUTFILE "',arg_file_path,'/',#var_table_name,'_CURRENT_TIMESTAMP.csv" fields terminated by "," optionally enclosed by ''"'' lines terminated by "\\n" ');
Use \ as extra escape character to not interpret \n as new line character.
And, you may also require to change current_timestamp string from literal to dynamic value. Change your concat parameters as below:
SET #query2 = CONCAT(
'SELECT * FROM table_name INTO OUTFILE "',
arg_file_path, '/', #var_table_name, '_', ( CURRENT_TIMESTAMP + 0 ), '.csv"
fields terminated by ","
optionally enclosed by ''"''
lines terminated by "\\n" ');
It produces output as:
SELECT * FROM lcs_tbl_test
INTO OUTFILE "/data/test_outfile/lcs_tbl_test_20140410132736.csv"
fields terminated by ","
optionally enclosed by '"'
lines terminated by "\n"

CONCAT string and INT in MYSQL and export big mysql table to small csv files

I am new in MYSQL programming. I want to select into outfile a very big table into several csv files using MYSQL loop script. My script is as following:
BEGIN
SET #t_lines=0;
SET #t_count=0;
SET #t_filepath='/home/ab/path/table_name_02212013_';
WHILE t_lines<=5000000
SET #t_filename=CONCAT(#t_filepath,CAST(#t_count as CHAR));
select * into outfile #t_filename fields terminated by ',' optionally enclosed by '"' lines terminated by '\n' from table_name limit #t_lines,#t_lines+300000;
SET #t_lines= #t_lines+300000;
SET #t_count= #t_count+1;
END WHILE;
COMMIT;
END
I got a syntax error in line:
SET #t_filename=CONCAT(#t_filepath,CAST(#t_count as CHAR));
The syntax error caused by WHILE ... DO. Thanks for quick reply.
Here I still have syntax error at "#t_lines,#t_lines+300000" after "limit".
I figure it out, seems "limit" not allowed "#t_lines+300000" to give the line range. It can be fixed by giving a new variable:
BEGIN
DECLARE t_lines INT DEFAULT 0;
DECLARE t_count INT DEFAULT 0;
DECLARE t_endlines INT DEFAULT 300000;
DECLARE t_linerange INT DEFAULT 300000;
SET #t_filepath='/home/ab/path/table_name_02212013_';
WHILE t_lines<=5000000 DO
SET #t_filename=CONCAT(#t_filepath,CAST(#t_count as CHAR));
select * into outfile '#t_filename' fields terminated by ',' optionally enclosed by '"' lines terminated by '\n' from table_name limit t_lines,t_endlines;
SET t_lines= t_lines+t_linerange;
SET t_endlines= t_endlines+t_linerange
SET t_count= t_count+1;
END WHILE;
COMMIT;
END
Thanks very much
The syntax for a WHILE loop in MySQL requires a DO;
WHILE t_lines<=5000000 DO
...
END WHILE;
The filename argument for INTO OUTFILE must be a literal string. You can't use variables or expressions as the filename.
The arguments to LIMIT must be integer constants. You can use parameter placeholders (?), or stored procedure parameters or local procedure variables (the ones you DECLARE). But you can't use session variables (the ones with # prefix), nor can you use expressions.
You're going to have to create the query as an SQL string, interpolating the values into the string.
Then use that SQL string with PREPARE and EXECUTE.
Try :
convert(columnName, char)
-->
BEGIN
SET #t_lines=0;
SET #t_count=0;
SET #t_filepath='/home/ab/path/table_name_02212013_';
WHILE t_lines<=5000000
SET #t_filename=CONCAT(#t_filepath,convert(#t_count, CHAR));
select * into outfile #t_filename fields terminated by ',' optionally enclosed by '"' lines terminated by '\n' from table_name limit #t_lines,#t_lines+300000;
SET #t_lines= #t_lines+300000;
SET #t_count= #t_count+1;
END WHILE;
COMMIT;
END

load data infile with unknown system variable error

I´ve been trying to use load data infile procedure to load data from a csv file to mysql, althougt I'ven been unable to due it because of a unknown variable error in the set statement.
LOAD DATA LOCAL INFILE 'D:\\Bets\\BD\\tables\\match.csv'
INTO TABLE `bets`.`match`
FIELDS TERMINATED BY ';' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n'
(`idMatch`, `idChampionship`, #date, `homeTeam`, `awayTeam`, `homeTeamGoals`, `awayTeamGoals`, `matchType` );
SET matchDate = STR_TO_DATE(#date, 'YYYY-MM-DD')
The date field to import is in format 2011-08-07.
when running the query the error given is: SQL Error (1193): Unknown system variable 'matchDate'.
any help would be pappreciated... thanks!
When SET matchDate it should be like this:
SET matchDate = STR_TO_DATE(#date, '%Y-%m-%d');
instead of SET matchDate = STR_TO_DATE(#date, 'YYYY-mm-dd');
Example of MySQL STR_TO_DATE function you can read here.
I've stumbled upon the same issue. The SET line must be part of the LOAD DATA INFILE ... statement. In your code, the column expression is terminated by a semicolon, which executes the SET command after the LOAD DATA INFILE finishes. The user variables created in the statement are not existing any more after the statement completes. Correct statement would be as follows,
LOAD DATA LOCAL INFILE 'D:\\Bets\\BD\\tables\\match.csv'
INTO TABLE `bets`.`match`
FIELDS TERMINATED BY ';' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'
LINES TERMINATED BY '\r\n'
(`idMatch`, `idChampionship`, #date, `homeTeam`,
`awayTeam`,`homeTeamGoals`, `awayTeamGoals`, `matchType`)
SET matchDate = STR_TO_DATE(#date, 'YYYY-MM-DD');