I have a MySQL query that is exported as a CSV file. The person who receives the CSV wants me to include the column headings. Obviously I can hard code them in the query but wondered if there was a way to return a row that has the names of all of the columns that have been queried?
My current query is below - I wish to programatically include the column headings too if I can just in case I forget to update it when changes are made.
Thanks
SET #sql_text =
CONCAT (
"SELECT
cf_id as FoundUsFrom,
firstname as CustomerName1,
lastname as CustomerName2,
address1 as CAddress1,
address2 as CAddress2,
'' as CAddress3,
towncity as CCity,
postcode as CPostCode,
email as 'email',
telephone as HomePhone,
mobile as MobilePhone,
'' as WorkPhone,
chidname as PName,
'' as ToBeAged,
partyaddress1 as PAddress1,
partyaddress2 as PAddress2,
'' as PAddress3,
partytowncity as PCity,
partypostcode as PPostCode,
/*NEED VALIDATION*/
celebrating as PType,
partydate as PDate,
partyboffinstart as PTime,
specialinstructions as SpecialInst,
numberguests as Guests,
IF(airexp = '','N','Y') as Air,
IF(chemcocklngexp = '','N','Y') as CocktailLong,
IF(chemcockshtexp = '','N','Y') as CocktailShort,
IF(chemreact = '','N','Y') as Reactions,
IF(dryicebolton = '','N','Y') as DryIce,
IF(electricitybolton = '','N','Y') as Electricity,
IF(flightbolton = '','N','Y') as Flight,
IF(fourelementsbolton = '','N','Y') as Elements,
IF(magicbolton = '','N','Y') as Magic,
IF(slimebolton = '','N','Y') as Slime,
IF(sweetbolton = '','N','Y') as Sweets,
IF(miniimpsbolton = '','N','Y') as MiniImps,
IF(extravaganzabolton = '','N','Y') as Extravaganza,
IF(dryicecreambolton = '','N','Y') as AIceCream,
IF(rocketbolton = '','N','Y') as ARocket,
IF(foodbreakbolton = '','N','Y') as AFood,
partybags as PBags,
deluxepartybags as PDBags
INTO OUTFILE 'PATHTO/data/bookings",
DATE_FORMAT( NOW(), '%Y%m%d%s'),
".csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
FROM sb_chronoforms_data_bookingformcombined"
);
PREPARE s1 FROM #sql_text;
EXECUTE s1;
DROP PREPARE s1;
loging this separately since it's another way to solve the issue:
select 'col_1', 'col_2','col_3','col_4' from dual UNION all select
col_1,col_2,col_3,col_4 from table_1;
that's a strange question :) try
SELECT
*
FROM
information_schema.COLUMNS
where
table_chema = 'your_schema'
and table_name = 'your_table';
than use ordinal_position to display the relevant column name
Related
So, I have more or less this structure of columns in my table:
Name Age Operator
---- --- --------
Jhon 35 >
Michael 30 =
Jess 27 <
Based on that I want to make a query like this
SELECT * FROM mytable WHERE Name = 'John' AND Age > 40
obviosly this will return no results, and thats fine, but my problem is that I want to use Jhon's "Operator" value (> in this case) to make that condition.
Is it possible?
Thank you!
You can simply do it like this:
SELECT
*
FROM Table1
WHERE Name = 'Jhon'AND CASE
WHEN Operator = '>' THEN Age > 10
WHEN Operator = '<' THEN Age < 10
WHEN Operator = '=' THEN Age = 10
END
see it working live in an sqlfiddle
You also could use MySQL's PREPARE and EXECUTE statements to make dynamic SQL.
SET #name = 'Jhon';
SET #operator = NULL;
SET #age = 10;
SELECT
Operator
INTO
#operator
FROM
Table1
WHERE
Name = #name;
SET #SQL = CONCAT(
"SELECT"
, " * "
, " FROM "
, " Table1 "
, " WHERE "
, " name = '", #name, "' AND age ", #operator, ' ' , #age
);
SELECT #SQL; # not needed but you can see the generated SQL code which will be executed
PREPARE s FROM #SQL;
EXECUTE s;
see demo https://www.db-fiddle.com/f/3Z59Lxaoy1ZXC4kdNCtpsr/1
Is it possible to include the headers somehow when using the MySQL INTO OUTFILE?
You'd have to hard code those headers yourself. Something like:
SELECT 'ColName1', 'ColName2', 'ColName3'
UNION ALL
SELECT ColName1, ColName2, ColName3
FROM YourTable
INTO OUTFILE '/path/outfile'
The solution provided by Joe Steanelli works, but making a list of columns is inconvenient when dozens or hundreds of columns are involved. Here's how to get column list of table my_table in my_schema.
-- override GROUP_CONCAT limit of 1024 characters to avoid a truncated result
set session group_concat_max_len = 1000000;
select GROUP_CONCAT(CONCAT("'",COLUMN_NAME,"'"))
from INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'my_table'
AND TABLE_SCHEMA = 'my_schema'
order BY ORDINAL_POSITION
Now you can copy & paste the resulting row as first statement in Joe's method.
For complex select with ORDER BY I use the following:
SELECT * FROM (
SELECT 'Column name #1', 'Column name #2', 'Column name ##'
UNION ALL
(
// complex SELECT statement with WHERE, ORDER BY, GROUP BY etc.
)
) resulting_set
INTO OUTFILE '/path/to/file';
This will alow you to have ordered columns and/or a limit
SELECT 'ColName1', 'ColName2', 'ColName3'
UNION ALL
SELECT * from (SELECT ColName1, ColName2, ColName3
FROM YourTable order by ColName1 limit 3) a
INTO OUTFILE '/path/outfile';
You can use prepared statement with lucek's answer and export dynamically table with columns name in CSV :
--If your table has too many columns
SET GLOBAL group_concat_max_len = 100000000;
--Prepared statement
SET #SQL = ( select CONCAT('SELECT * INTO OUTFILE \'YOUR_PATH\' FIELDS TERMINATED BY \',\' OPTIONALLY ENCLOSED BY \'"\' ESCAPED BY \'\' LINES TERMINATED BY \'\\n\' FROM (SELECT ', GROUP_CONCAT(CONCAT("'",COLUMN_NAME,"'")),' UNION select * from YOUR_TABLE) as tmp') from INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 'YOUR_TABLE' AND TABLE_SCHEMA = 'YOUR_SCHEMA' order BY ORDINAL_POSITION );
--Execute it
PREPARE stmt FROM #SQL;
EXECUTE stmt;
Thank lucek.
I simply make 2 queries, first to get query output (limit 1) with column names (no hardcode, no problems with Joins, Order by, custom column names, etc), and second to make query itself, and combine files into one CSV file:
CSVHEAD=`/usr/bin/mysql $CONNECTION_STRING -e "$QUERY limit 1;"|head -n1|xargs|sed -e "s/ /'\;'/g"`
echo "\'$CSVHEAD\'" > $TMP/head.txt
/usr/bin/mysql $CONNECTION_STRING -e "$QUERY into outfile '${TMP}/data.txt' fields terminated by ';' optionally enclosed by '\"' escaped by '' lines terminated by '\r\n';"
cat $TMP/head.txt $TMP/data.txt > $TMP/data.csv
This is an alternative cheat if you are familiar with Python or R, and your table can fit into memory.
Import the SQL table into Python or R and then export from there as a CSV and you'll get the column names as well as the data.
Here's how I do it using R, requires the RMySQL library:
db <- dbConnect(MySQL(), user='user', password='password', dbname='myschema', host='localhost')
query <- dbSendQuery(db, "select * from mytable")
dataset <- fetch(query, n=-1)
write.csv(dataset, 'mytable_backup.csv')
It's a bit of a cheat but I found this was a quick workaround when my number of columns was too long to use the concat method above. Note: R will add a 'row.names' column at the start of the CSV so you'll want to drop that if you do need to rely on the CSV to recreate the table.
I faced similar problem while executing mysql query on large tables in NodeJS. The approach which I followed to include headers in my CSV file is as follows
Use OUTFILE query to prepare file without headers
SELECT * INTO OUTFILE [FILE_NAME] FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED
BY '\"' LINES TERMINATED BY '\n' FROM [TABLE_NAME]
Fetch column headers for the table used in point 1
select GROUP_CONCAT(CONCAT(\"\",COLUMN_NAME,\"\")) as col_names from
INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = [TABLE_NAME] AND TABLE_SCHEMA
= [DATABASE_NAME] ORDER BY ORDINAL_POSITION
Append the column headers to the file created in step 1 using prepend-file npm package
Execution of each step was controlled using promises in NodeJS.
I think if you use a UNION it will work:
select 'header 1', 'header 2', ...
union
select col1, col2, ... from ...
I don't know of a way to specify the headers with the INTO OUTFILE syntax directly.
Since the 'include-headers' functionality doesn't seem to be build-in yet, and most "solutions" here need to type the columns names manually, and/or don't even take joins into account, I'd recommand to get around the problem.
The best alternative I found so far is using a decent tool (I use HeidiSQL).
Put your request, select the grid, just right click and export to a file. It got all necessary options for a clean export, ans should handle most needs.
In the same idea, user3037511's approach works fine, and can be automated easily.
Just launch your request with some command line to get your headers. You may get the data with a SELECT INTO OUTFILE... or by running your query without the limit, yours to choose.
Note that output redirect to a file works like a charm on both Linux AND Windows.
This makes me want to highlight that 80% of the time, when I want to use SELECT FROM INFILE or SELECT INTO OUTFILE, I end-up using something else due to some limitations (here, the absence of a 'headers options', on an AWS-RDS, the missing rights, and so on.)
Hence, I don't exactly answer to the op's question... but it should answer his needs :)
EDIT : and to actually answer his question : no
As of 2017-09-07, you just can't include headers if you stick with the SELECT INTO OUTFILE command :|
The easiest way is to hard code the columns yourself to better control the output file:
SELECT 'ColName1', 'ColName2', 'ColName3'
UNION ALL
SELECT ColName1, ColName2, ColName3
FROM YourTable
INTO OUTFILE '/path/outfile'
Actually you can make it work even with an ORDER BY.
Just needs some trickery in the order by statement - we use a case statement and replace the header value with some other value that is guaranteed to sort first in the list (obviously this is dependant on the type of field and whether you are sorting ASC or DESC)
Let's say you have three fields, name (varchar), is_active (bool), date_something_happens (date), and you want to sort the second two descending:
select
'name'
, 'is_active' as is_active
, date_something_happens as 'date_something_happens'
union all
select name, is_active, date_something_happens
from
my_table
order by
(case is_active when 'is_active' then 0 else is_active end) desc
, (case date when 'date' then '9999-12-30' else date end) desc
So, if all the columns in my_table are a character data type, we can combine the top answers (by Joe, matt and evilguc) together, to get the header added automatically in one 'simple' SQL query, e.g.
select * from (
(select column_name
from information_schema.columns
where table_name = 'my_table'
and table_schema = 'my_schema'
order by ordinal_position)
union all
(select * // potentially complex SELECT statement with WHERE, ORDER BY, GROUP BY etc.
from my_table)) as tbl
into outfile '/path/outfile'
fields terminated by ',' optionally enclosed by '"' escaped by '\\'
lines terminated by '\n';
where the last couple of lines make the output csv.
Note that this may be slow if my_table is very large.
an example from my database
table name sensor with colums (id,time,unit)
select ('id') as id, ('time') as time, ('unit') as unit
UNION ALL
SELECT * INTO OUTFILE 'C:/Users/User/Downloads/data.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM sensor
If you are using MySQL Workbench:
Select all the columns from the SCHEMAS tab -> Right Click -> Copy to
Clipboard -> Name
Paste it in any text editor and, Replace " ` " with " ' "
Copy it back and use it in your UNION query (as mentioned in the accepted
answer):
SELECT [Paste your text here]
UNION ALL
SELECT *
FROM table_name
INTO OUTFILE 'file_path'
I was writing my code in PHP, and I had a bit of trouble using concat and union functions, and also did not use SQL variables, any ways I got it to work, here is my code:
//first I connected to the information_scheme DB
$headercon=mysqli_connect("localhost", "USERNAME", "PASSWORD", "information_schema");
//took the healders out in a string (I could not get the concat function to work, so I wrote a loop for it)
$headers = '';
$sql = "SELECT column_name AS columns FROM `COLUMNS` WHERE table_schema = 'YOUR_DB_NAME' AND table_name = 'YOUR_TABLE_NAME'";
$result = $headercon->query($sql);
while($row = $result->fetch_row())
{
$headers = $headers . "'" . $row[0] . "', ";
}
$headers = substr("$headers", 0, -2);
// connect to the DB of interest
$con=mysqli_connect("localhost", "USERNAME", "PASSWORD", "YOUR_DB_NAME");
// export the results to csv
$sql4 = "SELECT $headers UNION SELECT * FROM YOUR_TABLE_NAME WHERE ... INTO OUTFILE '/output.csv' FIELDS TERMINATED BY ','";
$result4 = $con->query($sql4);
Here is a way to get the header titles from the column names dynamically.
/* Change table_name and database_name */
SET #table_name = 'table_name';
SET #table_schema = 'database_name';
SET #default_group_concat_max_len = (SELECT ##group_concat_max_len);
/* Sets Group Concat Max Limit larger for tables with a lot of columns */
SET SESSION group_concat_max_len = 1000000;
SET #col_names = (
SELECT GROUP_CONCAT(QUOTE(`column_name`)) AS columns
FROM information_schema.columns
WHERE table_schema = #table_schema
AND table_name = #table_name);
SET #cols = CONCAT('(SELECT ', #col_names, ')');
SET #query = CONCAT('(SELECT * FROM ', #table_schema, '.', #table_name,
' INTO OUTFILE \'/tmp/your_csv_file.csv\'
FIELDS ENCLOSED BY \'\\\'\' TERMINATED BY \'\t\' ESCAPED BY \'\'
LINES TERMINATED BY \'\n\')');
/* Concatenates column names to query */
SET #sql = CONCAT(#cols, ' UNION ALL ', #query);
/* Resets Group Contact Max Limit back to original value */
SET SESSION group_concat_max_len = #default_group_concat_max_len;
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
I would like to add to the answer provided by Sangam Belose. Here's his code:
select ('id') as id, ('time') as time, ('unit') as unit
UNION ALL
SELECT * INTO OUTFILE 'C:/Users/User/Downloads/data.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM sensor
However, if you have not set up your "secure_file_priv" within the variables, it may not work. For that, check the folder set on that variable by:
SHOW VARIABLES LIKE "secure_file_priv"
The output should look like this:
mysql> show variables like "%secure_file_priv%";
+------------------+------------------------------------------------+
| Variable_name | Value |
+------------------+------------------------------------------------+
| secure_file_priv | C:\ProgramData\MySQL\MySQL Server 8.0\Uploads\ |
+------------------+------------------------------------------------+
1 row in set, 1 warning (0.00 sec)
You can either change this variable or change the query to output the file to the default path showing.
MySQL alone isn't enough to do this simply. Below is a PHP script that will output columns and data to CSV.
Enter your database name and tables near the top.
<?php
set_time_limit( 24192000 );
ini_set( 'memory_limit', '-1' );
setlocale( LC_CTYPE, 'en_US.UTF-8' );
mb_regex_encoding( 'UTF-8' );
$dbn = 'DB_NAME';
$tbls = array(
'TABLE1',
'TABLE2',
'TABLE3'
);
$db = new PDO( 'mysql:host=localhost;dbname=' . $dbn . ';charset=UTF8', 'root', 'pass' );
foreach( $tbls as $tbl )
{
echo $tbl . "\n";
$path = '/var/lib/mysql/' . $tbl . '.csv';
$colStr = '';
$cols = $db->query( 'SELECT COLUMN_NAME AS `column` FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = "' . $tbl . '" AND TABLE_SCHEMA = "' . $dbn . '"' )->fetchAll( PDO::FETCH_COLUMN );
foreach( $cols as $col )
{
if( $colStr ) $colStr .= ', ';
$colStr .= '"' . $col . '"';
}
$db->query(
'SELECT *
FROM
(
SELECT ' . $colStr . '
UNION ALL
SELECT * FROM ' . $tbl . '
) AS sub
INTO OUTFILE "' . $path . '"
FIELDS TERMINATED BY ","
ENCLOSED BY "\""
LINES TERMINATED BY "\n"'
);
exec( 'gzip ' . $path );
print_r( $db->errorInfo() );
}
?>
You'll need this to be the directory you'd like to output to. MySQL needs to have the ability to write to the directory.
$path = '/var/lib/mysql/' . $tbl . '.csv';
You can edit the CSV export options in the query:
INTO OUTFILE "' . $path . '"
FIELDS TERMINATED BY ","
ENCLOSED BY "\""
LINES TERMINATED BY "\n"'
At the end there is an exec call to GZip the CSV.
I had no luck with any of these, so after finding a solution, I wanted to add it to the prior answers. Python==3.8.6 MySQL==8.0.19
(Forgive my lack of SO formatting foo. Somebody please clean up.)
Note a couple of things:
First, the query to return column names is unforgiving of punctuation. Using ` backticks or leaving out ' quote around the 'schema_name' and 'table_name' will throw an "unknown column" error.
WHERE TABLE_SCHEMA = 'schema' AND TABLE_NAME = 'table'
Second, the column header names return as a single-entity tuple with all the column names concatenated in one quoted string. Convert to quoted list was easy, but not intuitive (for me at least).
headers_list = headers_result[0].split(",")
Third, cursor must be buffered or the "lazy" thing will not fetch your results as you need them. For very large tables, memory could be an issue. Perhaps chunking would solve that problem.
cur = db.cursor(buffered=True)
Last, all types of UNION attempts yielded errors for me. By zipping the whole mess into a list of dicts, it became trivial to write to a csv, using csv.DictWriter.
headers_sql = """
SELECT
GROUP_CONCAT(CONCAT(COLUMN_NAME) ORDER BY ORDINAL_POSITION)
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = 'schema' AND TABLE_NAME = 'table';
""""
cur = db.cursor(buffered=True)
cur.execute(header_sql)
headers_result = cur.fetchone()
headers_list = headers_result[0].split(",")
rows_sql = """ SELECT * FROM schema.table; """"
data = cur.execute(rows_sql)
data_rows = cur.fetchall()
data_as_list_of_dicts = [dict(zip(headers_list, row)) for row in data_rows]
with open(csv_destination_file, 'w', encoding='utf-8') as destination_file_opened:
dict_writer = csv.DictWriter(destination_file_opened, fieldnames=headers_list)
dict_writer.writeheader() for dict in dict_list:
dict_writer.writerow(dict)
Solution using python but no need to install a python package to read sql files if you already use another tool.
If you are not familiar with python you can run the python codes in a colab notebook, all the required packages are already installed. It automates Matt and Joe's solutions.
Firstly execute this SQL script to get a csv with all table names :
SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_SCHEMA='your_schema'
INTO OUTFILE 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/tables.csv';
Then move tables.csv to a suitable directory and execute this python code after having replaced 'path_to_tables' and 'your_schema'. It will generate a sql script to export all tables headers:
import pandas as pd
import os
tables = pd.read_csv('tables.csv',header = None)[0]
text_file = open("export_headers.sql", "w")
schema = 'your_schema'
sql_output_path = 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/'
for table in tables :
path = os.path.join(sql_output_path,'{}_header.csv'.format(table))
string = "(select GROUP_CONCAT(COLUMN_NAME)\nfrom INFORMATION_SCHEMA.COLUMNS\nWHERE TABLE_NAME = '{}'\nAND TABLE_SCHEMA = '{}'\norder BY ORDINAL_POSITION)\nINTO OUTFILE '{}';".format(table,schema,path)
n = text_file.write(string)
n = text_file.write('\n\n')
text_file.close()
Then execute this python code which will generate a sql script to export the values of all tables:
text_file = open("export_values.sql", "w")
for table in tables :
path = os.path.join(sql_output_path,'{}.csv'.format(table))
string = "SELECT * FROM {}.{}\nINTO OUTFILE '{}';".format(schema,table,path)
n = text_file.write(string)
n = text_file.write('\n\n')
text_file.close()
Execute the two generated sql scripts and move the header csvs and values csvs in directories of your choice.
Then execute this last python code :
#Respectively the path to the headers csvs, the values csv and the path where you want to put the csvs with headers and values combined
headers_path, values_path, tables_path = '', '', ''
for table in tables :
header = pd.read_csv(os.path.join(headers_path,'{}_header.csv'.format(table)))
df = pd.read_csv(os.path.join(values_path,'{}.csv'.format(table)),names = header.columns,sep = '\t')
df.to_csv(os.path.join(tables_path,'{}.csv'.format(table)),index = False)
Then you got all your table exported in csv with the headers without having to write or copy paste all the tables and columns names.
Inspired by pivot table example from Rick James.
SET #CSVTABLE = 'myTableName',
#CSVBASE = 'databaseName',
#CSVFILE = '/tmp/filename.csv';
SET #sql = (SELECT CONCAT("SELECT ", GROUP_CONCAT(CONCAT('"', COLUMN_NAME, '"')), " UNION SELECT * FROM ", #CSVBASE, ".", #CSVTABLE) FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME=#CSVTABLE AND TABLE_SCHEMA=#CSVBASE);
prepare stmt from CONCAT(#sql, " INTO OUTFILE '", #CSVFILE, "' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\\n';");
execute stmt;
It gets list of columns from INFORMATION_SCHEMA.COLUMNS table, and uses GROUP_CONCAT to prepare SELECT statement with list of strings with column names.
Next UNION is added with SELECT * FROM specified database.table - this creates query text that will output both column names and column values in result.
Now the statement is prepared using previously created query (stored in #sql variable), CSV output specific "things" are appended to query and finally statement is executed with execute stmt
SELECT 'ColName1', 'ColName2', 'ColName3'
UNION ALL
SELECT ColName1, ColName2, ColName3
FROM YourTable
INTO OUTFILE 'c:\\datasheet.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n'
I am trying to export all the user information / permissions to a file for documentation. I trying to find a script out this task, is there anyway to pull all the permission from all the SQL databases from a server at one time? Working with SQL 2008 and 2012.
#Damaged I used powershell to export "Database-Level Object Permissions".. you can try below soluition.
[string]$SrvIns = 'YourServerName'
[string]$db = 'YourDatabaseName'
$sql = " SELECT
usr.name AS 'User',
CASE WHEN perm.state <> 'W' THEN perm.state_desc ELSE 'GRANT' END AS PermType,
perm.permission_name,
USER_NAME(obj.schema_id) AS SchemaName,
obj.name AS ObjectName,
CASE obj.Type
WHEN 'U' THEN 'Table'
WHEN 'V' THEN 'View'
WHEN 'P' THEN 'Stored Proc'
WHEN 'FN' THEN 'Function'
ELSE obj.Type END AS ObjectType,
CASE WHEN cl.column_id IS NULL THEN '--' ELSE cl.name END AS ColumnName,
CASE WHEN perm.state = 'W' THEN 'X' ELSE '--' END AS IsGrantOption
FROM
sys.database_permissions AS perm
INNER JOIN sys.objects AS obj
ON perm.major_id = obj.[object_id]
INNER JOIN sys.database_principals AS usr
ON perm.grantee_principal_id = usr.principal_id
LEFT JOIN sys.columns AS cl
ON cl.column_id = perm.minor_id AND cl.[object_id] = perm.major_id
WHERE
obj.Type <> 'S'
ORDER BY
usr.name, perm.state_desc ASC, perm.permission_name ASC"
Invoke-Sqlcmd -ServerInstance $SrvIns -Database $db -Query $sql | Export-Csv C:\permission.csv
Good Luck**
Here is a script that someone found for me it works perfect. I'm posting so other people can use.
set nocount on
declare #permission table (
Database_Name sysname,
User_Role_Name sysname,
Account_Type nvarchar(60),
Action_Type nvarchar(128),
Permission nvarchar(60),
ObjectName sysname null,
Object_Type nvarchar(60)
)
declare #dbs table (dbname sysname)
declare #Next sysname
insert into #dbs
select name from sys.databases order by name
select top 1 #Next = dbname from #dbs
while (##rowcount<>0)
begin
insert into #permission
exec('use [' + #Next + ']
declare #objects table (obj_id int, obj_type char(2))
insert into #objects
select id, xtype from master.sys.sysobjects
insert into #objects
select object_id, type from sys.objects
SELECT ''' + #Next + ''', a.name as ''User or Role Name'', a.type_desc as ''Account Type'',
d.permission_name as ''Type of Permission'', d.state_desc as ''State of Permission'',
OBJECT_SCHEMA_NAME(d.major_id) + ''.'' + object_name(d.major_id) as ''Object Name'',
case e.obj_type
when ''AF'' then ''Aggregate function (CLR)''
when ''C'' then ''CHECK constraint''
when ''D'' then ''DEFAULT (constraint or stand-alone)''
when ''F'' then ''FOREIGN KEY constraint''
when ''PK'' then ''PRIMARY KEY constraint''
when ''P'' then ''SQL stored procedure''
when ''PC'' then ''Assembly (CLR) stored procedure''
when ''FN'' then ''SQL scalar function''
when ''FS'' then ''Assembly (CLR) scalar function''
when ''FT'' then ''Assembly (CLR) table-valued function''
when ''R'' then ''Rule (old-style, stand-alone)''
when ''RF'' then ''Replication-filter-procedure''
when ''S'' then ''System base table''
when ''SN'' then ''Synonym''
when ''SQ'' then ''Service queue''
when ''TA'' then ''Assembly (CLR) DML trigger''
when ''TR'' then ''SQL DML trigger''
when ''IF'' then ''SQL inline table-valued function''
when ''TF'' then ''SQL table-valued-function''
when ''U'' then ''Table (user-defined)''
when ''UQ'' then ''UNIQUE constraint''
when ''V'' then ''View''
when ''X'' then ''Extended stored procedure''
when ''IT'' then ''Internal table''
end as ''Object Type''
FROM [' + #Next + '].sys.database_principals a
left join [' + #Next + '].sys.database_permissions d on a.principal_id = d.grantee_principal_id
left join #objects e on d.major_id = e.obj_id
order by a.name, d.class_desc')
delete #dbs where dbname = #Next
select top 1 #Next = dbname from #dbs
end
set nocount off
select ##SERVERNAME as Server_name,* from #permission
In particular table there exists a field of SET type with specific legal values:
personType SET('CUSTOMER','SUPPLIER','EMPLOYEE', 'CONTRACTOR') NOT NULL
Is there any way to query MySQL to get a list of the valid values? In the MySQL interpreter I would just run DESCRIBE someTable; however if there is a more direct method that one could use programmatically without lots of parsing it would be nice.
Thanks.
Now, this simply freaks out, but it is MySQL-only and it works!
SELECT TRIM("'" FROM SUBSTRING_INDEX(SUBSTRING_INDEX(
(SELECT TRIM(')' FROM SUBSTR(column_type, 5)) FROM information_schema.columns
WHERE table_name = 'some_table' AND column_name = 'some_column'),
',', #r:=#r+1), ',', -1)) AS item
FROM (SELECT #r:=0) deriv1,
(SELECT ID FROM information_schema.COLLATIONS) deriv2
HAVING #r <=
(SELECT LENGTH(column_type) - LENGTH(REPLACE(column_type, ',', ''))
FROM information_schema.columns
WHERE table_name = 'some_table' AND column_name = 'some_column');
Just replace "some_table" and "some_column" for your specific table/column, and see the magic!
You will see a weird usage of "information_schema.COLLATIONS" - this is because we need a table there - any table - containing at least N rows, where N is the number of elements in your set.
SELECT
column_type
FROM
information_schema.columns
WHERE
table_name = 'some_table'
AND
column_name = 'some_column';
Returns:
column_type
------------------
set('this','that')
The function below returns an array containing all available options for SET with some parsing but not "lots of parsing"... :)
function get_set_values($table_name, $field_name)
{
$sql = 'DESCRIBE ' . $table_name . ' ' . $field_name;
$result = mysql_query($sql);
$row = mysql_fetch_array($result);
return str_getcsv( trim( substr( $row['Type'], 3 ), '()' ), ',', "'" );
}
Remember that in a set column you may have a combination of values or even an empty value (these are also valid).
Is it possible to include the headers somehow when using the MySQL INTO OUTFILE?
You'd have to hard code those headers yourself. Something like:
SELECT 'ColName1', 'ColName2', 'ColName3'
UNION ALL
SELECT ColName1, ColName2, ColName3
FROM YourTable
INTO OUTFILE '/path/outfile'
The solution provided by Joe Steanelli works, but making a list of columns is inconvenient when dozens or hundreds of columns are involved. Here's how to get column list of table my_table in my_schema.
-- override GROUP_CONCAT limit of 1024 characters to avoid a truncated result
set session group_concat_max_len = 1000000;
select GROUP_CONCAT(CONCAT("'",COLUMN_NAME,"'"))
from INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'my_table'
AND TABLE_SCHEMA = 'my_schema'
order BY ORDINAL_POSITION
Now you can copy & paste the resulting row as first statement in Joe's method.
For complex select with ORDER BY I use the following:
SELECT * FROM (
SELECT 'Column name #1', 'Column name #2', 'Column name ##'
UNION ALL
(
// complex SELECT statement with WHERE, ORDER BY, GROUP BY etc.
)
) resulting_set
INTO OUTFILE '/path/to/file';
This will alow you to have ordered columns and/or a limit
SELECT 'ColName1', 'ColName2', 'ColName3'
UNION ALL
SELECT * from (SELECT ColName1, ColName2, ColName3
FROM YourTable order by ColName1 limit 3) a
INTO OUTFILE '/path/outfile';
You can use prepared statement with lucek's answer and export dynamically table with columns name in CSV :
--If your table has too many columns
SET GLOBAL group_concat_max_len = 100000000;
--Prepared statement
SET #SQL = ( select CONCAT('SELECT * INTO OUTFILE \'YOUR_PATH\' FIELDS TERMINATED BY \',\' OPTIONALLY ENCLOSED BY \'"\' ESCAPED BY \'\' LINES TERMINATED BY \'\\n\' FROM (SELECT ', GROUP_CONCAT(CONCAT("'",COLUMN_NAME,"'")),' UNION select * from YOUR_TABLE) as tmp') from INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 'YOUR_TABLE' AND TABLE_SCHEMA = 'YOUR_SCHEMA' order BY ORDINAL_POSITION );
--Execute it
PREPARE stmt FROM #SQL;
EXECUTE stmt;
Thank lucek.
I simply make 2 queries, first to get query output (limit 1) with column names (no hardcode, no problems with Joins, Order by, custom column names, etc), and second to make query itself, and combine files into one CSV file:
CSVHEAD=`/usr/bin/mysql $CONNECTION_STRING -e "$QUERY limit 1;"|head -n1|xargs|sed -e "s/ /'\;'/g"`
echo "\'$CSVHEAD\'" > $TMP/head.txt
/usr/bin/mysql $CONNECTION_STRING -e "$QUERY into outfile '${TMP}/data.txt' fields terminated by ';' optionally enclosed by '\"' escaped by '' lines terminated by '\r\n';"
cat $TMP/head.txt $TMP/data.txt > $TMP/data.csv
This is an alternative cheat if you are familiar with Python or R, and your table can fit into memory.
Import the SQL table into Python or R and then export from there as a CSV and you'll get the column names as well as the data.
Here's how I do it using R, requires the RMySQL library:
db <- dbConnect(MySQL(), user='user', password='password', dbname='myschema', host='localhost')
query <- dbSendQuery(db, "select * from mytable")
dataset <- fetch(query, n=-1)
write.csv(dataset, 'mytable_backup.csv')
It's a bit of a cheat but I found this was a quick workaround when my number of columns was too long to use the concat method above. Note: R will add a 'row.names' column at the start of the CSV so you'll want to drop that if you do need to rely on the CSV to recreate the table.
I faced similar problem while executing mysql query on large tables in NodeJS. The approach which I followed to include headers in my CSV file is as follows
Use OUTFILE query to prepare file without headers
SELECT * INTO OUTFILE [FILE_NAME] FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED
BY '\"' LINES TERMINATED BY '\n' FROM [TABLE_NAME]
Fetch column headers for the table used in point 1
select GROUP_CONCAT(CONCAT(\"\",COLUMN_NAME,\"\")) as col_names from
INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = [TABLE_NAME] AND TABLE_SCHEMA
= [DATABASE_NAME] ORDER BY ORDINAL_POSITION
Append the column headers to the file created in step 1 using prepend-file npm package
Execution of each step was controlled using promises in NodeJS.
I think if you use a UNION it will work:
select 'header 1', 'header 2', ...
union
select col1, col2, ... from ...
I don't know of a way to specify the headers with the INTO OUTFILE syntax directly.
Since the 'include-headers' functionality doesn't seem to be build-in yet, and most "solutions" here need to type the columns names manually, and/or don't even take joins into account, I'd recommand to get around the problem.
The best alternative I found so far is using a decent tool (I use HeidiSQL).
Put your request, select the grid, just right click and export to a file. It got all necessary options for a clean export, ans should handle most needs.
In the same idea, user3037511's approach works fine, and can be automated easily.
Just launch your request with some command line to get your headers. You may get the data with a SELECT INTO OUTFILE... or by running your query without the limit, yours to choose.
Note that output redirect to a file works like a charm on both Linux AND Windows.
This makes me want to highlight that 80% of the time, when I want to use SELECT FROM INFILE or SELECT INTO OUTFILE, I end-up using something else due to some limitations (here, the absence of a 'headers options', on an AWS-RDS, the missing rights, and so on.)
Hence, I don't exactly answer to the op's question... but it should answer his needs :)
EDIT : and to actually answer his question : no
As of 2017-09-07, you just can't include headers if you stick with the SELECT INTO OUTFILE command :|
The easiest way is to hard code the columns yourself to better control the output file:
SELECT 'ColName1', 'ColName2', 'ColName3'
UNION ALL
SELECT ColName1, ColName2, ColName3
FROM YourTable
INTO OUTFILE '/path/outfile'
Actually you can make it work even with an ORDER BY.
Just needs some trickery in the order by statement - we use a case statement and replace the header value with some other value that is guaranteed to sort first in the list (obviously this is dependant on the type of field and whether you are sorting ASC or DESC)
Let's say you have three fields, name (varchar), is_active (bool), date_something_happens (date), and you want to sort the second two descending:
select
'name'
, 'is_active' as is_active
, date_something_happens as 'date_something_happens'
union all
select name, is_active, date_something_happens
from
my_table
order by
(case is_active when 'is_active' then 0 else is_active end) desc
, (case date when 'date' then '9999-12-30' else date end) desc
So, if all the columns in my_table are a character data type, we can combine the top answers (by Joe, matt and evilguc) together, to get the header added automatically in one 'simple' SQL query, e.g.
select * from (
(select column_name
from information_schema.columns
where table_name = 'my_table'
and table_schema = 'my_schema'
order by ordinal_position)
union all
(select * // potentially complex SELECT statement with WHERE, ORDER BY, GROUP BY etc.
from my_table)) as tbl
into outfile '/path/outfile'
fields terminated by ',' optionally enclosed by '"' escaped by '\\'
lines terminated by '\n';
where the last couple of lines make the output csv.
Note that this may be slow if my_table is very large.
an example from my database
table name sensor with colums (id,time,unit)
select ('id') as id, ('time') as time, ('unit') as unit
UNION ALL
SELECT * INTO OUTFILE 'C:/Users/User/Downloads/data.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM sensor
If you are using MySQL Workbench:
Select all the columns from the SCHEMAS tab -> Right Click -> Copy to
Clipboard -> Name
Paste it in any text editor and, Replace " ` " with " ' "
Copy it back and use it in your UNION query (as mentioned in the accepted
answer):
SELECT [Paste your text here]
UNION ALL
SELECT *
FROM table_name
INTO OUTFILE 'file_path'
I was writing my code in PHP, and I had a bit of trouble using concat and union functions, and also did not use SQL variables, any ways I got it to work, here is my code:
//first I connected to the information_scheme DB
$headercon=mysqli_connect("localhost", "USERNAME", "PASSWORD", "information_schema");
//took the healders out in a string (I could not get the concat function to work, so I wrote a loop for it)
$headers = '';
$sql = "SELECT column_name AS columns FROM `COLUMNS` WHERE table_schema = 'YOUR_DB_NAME' AND table_name = 'YOUR_TABLE_NAME'";
$result = $headercon->query($sql);
while($row = $result->fetch_row())
{
$headers = $headers . "'" . $row[0] . "', ";
}
$headers = substr("$headers", 0, -2);
// connect to the DB of interest
$con=mysqli_connect("localhost", "USERNAME", "PASSWORD", "YOUR_DB_NAME");
// export the results to csv
$sql4 = "SELECT $headers UNION SELECT * FROM YOUR_TABLE_NAME WHERE ... INTO OUTFILE '/output.csv' FIELDS TERMINATED BY ','";
$result4 = $con->query($sql4);
Here is a way to get the header titles from the column names dynamically.
/* Change table_name and database_name */
SET #table_name = 'table_name';
SET #table_schema = 'database_name';
SET #default_group_concat_max_len = (SELECT ##group_concat_max_len);
/* Sets Group Concat Max Limit larger for tables with a lot of columns */
SET SESSION group_concat_max_len = 1000000;
SET #col_names = (
SELECT GROUP_CONCAT(QUOTE(`column_name`)) AS columns
FROM information_schema.columns
WHERE table_schema = #table_schema
AND table_name = #table_name);
SET #cols = CONCAT('(SELECT ', #col_names, ')');
SET #query = CONCAT('(SELECT * FROM ', #table_schema, '.', #table_name,
' INTO OUTFILE \'/tmp/your_csv_file.csv\'
FIELDS ENCLOSED BY \'\\\'\' TERMINATED BY \'\t\' ESCAPED BY \'\'
LINES TERMINATED BY \'\n\')');
/* Concatenates column names to query */
SET #sql = CONCAT(#cols, ' UNION ALL ', #query);
/* Resets Group Contact Max Limit back to original value */
SET SESSION group_concat_max_len = #default_group_concat_max_len;
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
I would like to add to the answer provided by Sangam Belose. Here's his code:
select ('id') as id, ('time') as time, ('unit') as unit
UNION ALL
SELECT * INTO OUTFILE 'C:/Users/User/Downloads/data.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM sensor
However, if you have not set up your "secure_file_priv" within the variables, it may not work. For that, check the folder set on that variable by:
SHOW VARIABLES LIKE "secure_file_priv"
The output should look like this:
mysql> show variables like "%secure_file_priv%";
+------------------+------------------------------------------------+
| Variable_name | Value |
+------------------+------------------------------------------------+
| secure_file_priv | C:\ProgramData\MySQL\MySQL Server 8.0\Uploads\ |
+------------------+------------------------------------------------+
1 row in set, 1 warning (0.00 sec)
You can either change this variable or change the query to output the file to the default path showing.
MySQL alone isn't enough to do this simply. Below is a PHP script that will output columns and data to CSV.
Enter your database name and tables near the top.
<?php
set_time_limit( 24192000 );
ini_set( 'memory_limit', '-1' );
setlocale( LC_CTYPE, 'en_US.UTF-8' );
mb_regex_encoding( 'UTF-8' );
$dbn = 'DB_NAME';
$tbls = array(
'TABLE1',
'TABLE2',
'TABLE3'
);
$db = new PDO( 'mysql:host=localhost;dbname=' . $dbn . ';charset=UTF8', 'root', 'pass' );
foreach( $tbls as $tbl )
{
echo $tbl . "\n";
$path = '/var/lib/mysql/' . $tbl . '.csv';
$colStr = '';
$cols = $db->query( 'SELECT COLUMN_NAME AS `column` FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = "' . $tbl . '" AND TABLE_SCHEMA = "' . $dbn . '"' )->fetchAll( PDO::FETCH_COLUMN );
foreach( $cols as $col )
{
if( $colStr ) $colStr .= ', ';
$colStr .= '"' . $col . '"';
}
$db->query(
'SELECT *
FROM
(
SELECT ' . $colStr . '
UNION ALL
SELECT * FROM ' . $tbl . '
) AS sub
INTO OUTFILE "' . $path . '"
FIELDS TERMINATED BY ","
ENCLOSED BY "\""
LINES TERMINATED BY "\n"'
);
exec( 'gzip ' . $path );
print_r( $db->errorInfo() );
}
?>
You'll need this to be the directory you'd like to output to. MySQL needs to have the ability to write to the directory.
$path = '/var/lib/mysql/' . $tbl . '.csv';
You can edit the CSV export options in the query:
INTO OUTFILE "' . $path . '"
FIELDS TERMINATED BY ","
ENCLOSED BY "\""
LINES TERMINATED BY "\n"'
At the end there is an exec call to GZip the CSV.
I had no luck with any of these, so after finding a solution, I wanted to add it to the prior answers. Python==3.8.6 MySQL==8.0.19
(Forgive my lack of SO formatting foo. Somebody please clean up.)
Note a couple of things:
First, the query to return column names is unforgiving of punctuation. Using ` backticks or leaving out ' quote around the 'schema_name' and 'table_name' will throw an "unknown column" error.
WHERE TABLE_SCHEMA = 'schema' AND TABLE_NAME = 'table'
Second, the column header names return as a single-entity tuple with all the column names concatenated in one quoted string. Convert to quoted list was easy, but not intuitive (for me at least).
headers_list = headers_result[0].split(",")
Third, cursor must be buffered or the "lazy" thing will not fetch your results as you need them. For very large tables, memory could be an issue. Perhaps chunking would solve that problem.
cur = db.cursor(buffered=True)
Last, all types of UNION attempts yielded errors for me. By zipping the whole mess into a list of dicts, it became trivial to write to a csv, using csv.DictWriter.
headers_sql = """
SELECT
GROUP_CONCAT(CONCAT(COLUMN_NAME) ORDER BY ORDINAL_POSITION)
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = 'schema' AND TABLE_NAME = 'table';
""""
cur = db.cursor(buffered=True)
cur.execute(header_sql)
headers_result = cur.fetchone()
headers_list = headers_result[0].split(",")
rows_sql = """ SELECT * FROM schema.table; """"
data = cur.execute(rows_sql)
data_rows = cur.fetchall()
data_as_list_of_dicts = [dict(zip(headers_list, row)) for row in data_rows]
with open(csv_destination_file, 'w', encoding='utf-8') as destination_file_opened:
dict_writer = csv.DictWriter(destination_file_opened, fieldnames=headers_list)
dict_writer.writeheader() for dict in dict_list:
dict_writer.writerow(dict)
Solution using python but no need to install a python package to read sql files if you already use another tool.
If you are not familiar with python you can run the python codes in a colab notebook, all the required packages are already installed. It automates Matt and Joe's solutions.
Firstly execute this SQL script to get a csv with all table names :
SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_SCHEMA='your_schema'
INTO OUTFILE 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/tables.csv';
Then move tables.csv to a suitable directory and execute this python code after having replaced 'path_to_tables' and 'your_schema'. It will generate a sql script to export all tables headers:
import pandas as pd
import os
tables = pd.read_csv('tables.csv',header = None)[0]
text_file = open("export_headers.sql", "w")
schema = 'your_schema'
sql_output_path = 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/'
for table in tables :
path = os.path.join(sql_output_path,'{}_header.csv'.format(table))
string = "(select GROUP_CONCAT(COLUMN_NAME)\nfrom INFORMATION_SCHEMA.COLUMNS\nWHERE TABLE_NAME = '{}'\nAND TABLE_SCHEMA = '{}'\norder BY ORDINAL_POSITION)\nINTO OUTFILE '{}';".format(table,schema,path)
n = text_file.write(string)
n = text_file.write('\n\n')
text_file.close()
Then execute this python code which will generate a sql script to export the values of all tables:
text_file = open("export_values.sql", "w")
for table in tables :
path = os.path.join(sql_output_path,'{}.csv'.format(table))
string = "SELECT * FROM {}.{}\nINTO OUTFILE '{}';".format(schema,table,path)
n = text_file.write(string)
n = text_file.write('\n\n')
text_file.close()
Execute the two generated sql scripts and move the header csvs and values csvs in directories of your choice.
Then execute this last python code :
#Respectively the path to the headers csvs, the values csv and the path where you want to put the csvs with headers and values combined
headers_path, values_path, tables_path = '', '', ''
for table in tables :
header = pd.read_csv(os.path.join(headers_path,'{}_header.csv'.format(table)))
df = pd.read_csv(os.path.join(values_path,'{}.csv'.format(table)),names = header.columns,sep = '\t')
df.to_csv(os.path.join(tables_path,'{}.csv'.format(table)),index = False)
Then you got all your table exported in csv with the headers without having to write or copy paste all the tables and columns names.
Inspired by pivot table example from Rick James.
SET #CSVTABLE = 'myTableName',
#CSVBASE = 'databaseName',
#CSVFILE = '/tmp/filename.csv';
SET #sql = (SELECT CONCAT("SELECT ", GROUP_CONCAT(CONCAT('"', COLUMN_NAME, '"')), " UNION SELECT * FROM ", #CSVBASE, ".", #CSVTABLE) FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME=#CSVTABLE AND TABLE_SCHEMA=#CSVBASE);
prepare stmt from CONCAT(#sql, " INTO OUTFILE '", #CSVFILE, "' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\\n';");
execute stmt;
It gets list of columns from INFORMATION_SCHEMA.COLUMNS table, and uses GROUP_CONCAT to prepare SELECT statement with list of strings with column names.
Next UNION is added with SELECT * FROM specified database.table - this creates query text that will output both column names and column values in result.
Now the statement is prepared using previously created query (stored in #sql variable), CSV output specific "things" are appended to query and finally statement is executed with execute stmt
SELECT 'ColName1', 'ColName2', 'ColName3'
UNION ALL
SELECT ColName1, ColName2, ColName3
FROM YourTable
INTO OUTFILE 'c:\\datasheet.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n'