Related
I have a mysqldump backup of my mysql database consisting of all of our tables which is about 440 megs. I want to restore the contents of just one of the tables from the mysqldump. Is this possible? Theoretically, I could just cut out the section that rebuilds the table I want but I don't even know how to effectively edit a text document that size.
You can try to use sed in order to extract only the table you want.
Let say the name of your table is mytable and the file mysql.dump is the file containing your huge dump:
$ sed -n -e '/CREATE TABLE.*`mytable`/,/Table structure for table/p' mysql.dump > mytable.dump
This will copy in the file mytable.dump what is located between CREATE TABLE mytable and the next CREATE TABLE corresponding to the next table.
You can then adjust the file mytable.dump which contains the structure of the table mytable, and the data (a list of INSERT).
I used a modified version of uloBasEI's sed command. It includes the preceding DROP command, and reads until mysql is done dumping data to your table (UNLOCK). Worked for me (re)importing wp_users to a bunch of Wordpress sites.
sed -n -e '/DROP TABLE.*`mytable`/,/UNLOCK TABLES/p' mydump.sql > tabledump.sql
This can be done more easily? This is how I did it:
Create a temporary database (e.g. restore):
mysqladmin -u root -p create restore
Restore the full dump in the temp database:
mysql -u root -p --one-database restore < fulldump.sql
Dump the table you want to recover:
mysqldump restore mytable > mytable.sql
Import the table in another database:
mysql -u root -p database < mytable.sql
A simple solution would be to simply create a dump of just the table you wish to restore separately. You can use the mysqldump command to do so with the following syntax:
mysqldump -u [user] -p[password] [database] [table] > [output_file_name].sql
Then import it as normal, and it will only import the dumped table.
One way or another, any process doing that will have to go through the entire text of the dump and parse it in some way. I'd just grep for
INSERT INTO `the_table_i_want`
and pipe the output into mysql. Take a look at the first table in the dump before, to make sure you're getting the INSERT's the right way.
Edit: OK, got the formatting right this time.
Backup
$ mysqldump -A | gzip > mysqldump-A.gz
Restore single table
$ mysql -e "truncate TABLE_NAME" DB_NAME
$ zgrep ^"INSERT INTO \`TABLE_NAME" mysqldump-A.gz | mysql DB_NAME
You should try #bryn command but with the ` delimiter otherwise you will also extract the tables having a prefix or a suffix, this is what I usually do:
sed -n -e '/DROP TABLE.*`mytable`/,/UNLOCK TABLES/p' dump.sql > mytable.sql
Also for testing purpose, you may want to change the table name before importing:
sed -n -e 's/`mytable`/`mytable_restored`/g' mytable.sql > mytable_restored.sql
To import you can then use the mysql command:
mysql -u root -p'password' mydatabase < mytable_restore.sql
One possible way to deal with this is to restore to a temporary database, and dump just that table from the temporary database. Then use the new script.
sed -n -e '/-- Table structure for table `my_table_name`/,/UNLOCK TABLES/p' database_file.sql > table_file.sql
This is a better solution than some of the others above because not all SQL dumps contain a DROP TABLE statement. This one will work will all kinds of dumps.
This tool may be is what you want: tbdba-restore-mysqldump.pl
https://github.com/orczhou/dba-tool/blob/master/tbdba-restore-mysqldump.pl
e.g. Restore a table from database dump file:
tbdba-restore-mysqldump.pl -t yourtable -s yourdb -f backup.sql
Table should present with same structure in both dump and database.
`zgrep -a ^"INSERT INTO \`table_name" DbDump-backup.sql.tar.gz | mysql -u<user> -p<password> database_name`
or
`zgrep -a ^"INSERT INTO \`table_name" DbDump-backup.sql | mysql -u<user> -p<password> database_name`
This may help too.
# mysqldump -u root -p database0 > /tmp/database0.sql
# mysql -u root -p -e 'create database database0_bkp'
# mysql -u root -p database0_bkp < /tmp/database0.sql
# mysql -u root -p database0 -e 'insert into database0.table_you_want select * from database0_bkp.table_you_want'
Most modern text editors should be able to handle a text file that size, if your system is up to it.
Anyway, I had to do that once very quickly and i didnt have time to find any tools. I set up a new MySQL instance, imported the whole backup and then spit out just the table I wanted.
Then I imported that table into the main database.
It was tedious but rather easy. Good luck.
You can use vi editor. Type:
vi -o mysql.dump mytable.dump
to open both whole dump mysql.dump and a new file mytable.dump.
Find the appropriate insert into line by pressing / and then type a phrase, for example: "insert into `mytable`", then copy that line using yy. Switch to next file by ctrl+w then down arrow key, paste the copied line with pp. Finally save the new file by typing :wq and quite vi editor by :q.
Note that if you have dumped the data using multiple inserts you can copy (yank) all of them at once using Nyy in which N is the number of lines to be copied.
I have done it with a file of 920 MB size.
I tried a few options, which were incredibly slow. This split a 360GB dump into its tables in a few minutes:
How do I split the output from mysqldump into smaller files?
The 'sed' solutions mentioned earlier are nice but as mentioned not 100% secure
You may have INSERT commands with data containing:
... CREATE TABLE...(whatever)...mytable...
or even the exact string "CREATE TABLE `mytable`;"
if you are storing DML commands for instance!
(and if the table is huge you don't want to check that manually)
I would verify the exact syntax of the dump version used, and have a more restrictive pattern search:
Avoid ".*" and use "^" to ensure we start at the begining of the line.
And I'd prefer to grab the initial 'DROP'
All in all, this works better for me:
sed -n -e '/^DROP TABLE IF EXISTS \`mytable\`;/,/^UNLOCK TABLES;/p' mysql.dump > mytable.dump
Get a decent text editor like Notepad++ or Vim (if you're already proficient with it). Search for the table name and you should be able to highlight just the CREATE, ALTER, and INSERT commands for that table. It may be easier to navigate with your keyboard rather than a mouse. And I would make sure you're on a machine with plenty or RAM so that it will not have a problem loading the entire file at once. Once you've highlighted and copied the rows you need, it would be a good idea to back up just the copied part into it's own backup file and then import it into MySQL.
The chunks of SQL are blocked off with "Table structure for table my_table" and "Dumping data for table my_table."
You can use a Windows command line as follows to get the line numbers for the various sections. Adjust the searched string as needed.
find /n "for table `" sql.txt
The following will be returned:
---------- SQL.TXT
[4384]-- Table structure for table my_table
[4500]-- Dumping data for table my_table
[4514]-- Table structure for table some_other_table
... etc.
That gets you the line numbers you need... now, if I only knew how to use them... investigating.
You can import single table using terminal line as given below.
Here import single user table into specific database.
mysql -u root -p -D my_database_name < var/www/html/myproject/tbl_user.sql
I admire some of the ingenuity here, but there is literally no reason to use sed at all to address the OP's question.
The comment "use --one-database" is the correct answer, built into MySQL/MariaDB. No need for third-party hacks.
mysql -u root -p databasename --one-database < localhost.sql will just import the desired database.
I also found in some cases, when using this to import a series of databases, it would create the next database in the list for me (but not put anything in it). Not sure why it did that, but it made the restore easier.
With this command, enter the password interactively and it will import the requested database.
I have a Magento database in which I want to search for a particular string/pattern.
But the database's size is too large so I cannot export the database to .sql file and then search into that file(editor even Geany crashes opening such large files).
So how can I do a search the database for a perfect match of [string/pattern] and display fulltext information as result, through only using command-line and MySQL Database credentials ?
I tried below command, but it requires username to be given as -u[USERNAME], also it doesn't display full query or result in terminal window.
mysqldump -p[PASSWORD] [DATABASE] --extended=FALSE | grep [pattern] | less -S
Anyone have any solutions for this ?
You can first log into MySQL CLI as especified in http://dev.mysql.com/doc/refman/5.7/en/connecting.html
mysql --host=localhost --user=myname --password=mypass mydb
So, you can use a query command to find your pattern. If you know the table you want to search such as the column it make the thinks easy. The SELECT statement is like this:
SELECT column FROM table WHERE column LIKE '%pattern%';
http://dev.mysql.com/doc/en/select.html
If you don't know the table's name, you can list all and try to find by the meaning.
SHOW TABLES;
Edited with better code
You didn't say if this was a one off or not but this will check all tables in a schema for a value.
First in your home directory set up a file named .my.cnf with the following contents and change its permissions to 700 (Replace [USERNAME] and [PASSWORD] with your username and password.
[client]
user=[USERNAME]
password="[PASSWORD]"
Then execute the following (Replacing [DATABASE] and [CHECKSTRING] with your database and the check string)
mysql [DATABASE] --silent -N -e "show tables;"|while read table; do mysql [DATABASE] --silent -N -e "select * from ${table};"|while read line;do if [[ "${line}" == *"[CHECKSTRING]"* ]]; then echo "${table}***${line}";fi;done;done
If checking for 51584 the result would be something like
test_table***551584,'column 2 value','column 3 value'
test_table5***'column 1 value',251584,'column 3 value'
If you want to know which column had the value then select from INFORMATION_SCHEMA.COLUMNS and add another nest.
mysql [DATABASE] --silent -N -e "show tables;"|while read table; do mysql [DATABASE] --silent -N -e "select column_name from information_schema.columns where table_schema='[DATABASE]' and table_name = '${table}';"|while read column; do mysql [DATABASE] --silent -N -e "select ${column} from ${table};"|while read line;do if [[ "${line}" == *"[CHECKSTRING]"* ]]; then echo "${table}***${column}***${line}";fi;done;done;done
If checking for 51584 the result would be something like
test_table***column1***551584
test_table5***column2***251584
First of all you need to login into database with correct username and password by below command.
sudo mysql -u root -p
then check the database in which you want to operate operation.
eg.
SHOW DATABASES;
USE Test;
now your database is ready for operation through terminal. Here I assume my database name is "Test".
Now for String/pattern matching use command as below or follow the link http://www.mysqltutorial.org/mysql-regular-expression-regexp.aspx.
SELECT
column_list
FROM
table_name
WHERE
string_column REGEXP pattern;
I have a large SQL file with one database and about 150 tables. I would like to use mysqlimport to import that database, however, I would like the import process to ignore or skip over a couple of tables. What is the proper syntax to import all tables, but ignore some of them? Thank you.
The accepted answer by RandomSeed could take a long time! Importing the table (just to drop it later) could be very wasteful depending on size.
For a file created using
mysqldump -u user -ppasswd --opt --routines DBname > DBdump.sql
I currently get a file about 7GB, 6GB of which is data for a log table that I don't 'need' to be there; reloading this file takes a couple of hours. If I need to reload (for development purposes, or if ever required for a live recovery) I skim the file thus:
sed '/INSERT INTO `TABLE_TO_SKIP`/d' DBdump.sql > reduced.sql
And reload with:
mysql -u user -ppasswd DBname < reduced.sql
This gives me a complete database, with the "unwanted" table created but empty. If you really don't want the tables at all, simply drop the empty tables after the load finishes.
For multiple tables you could do something like this:
sed '/INSERT INTO `TABLE1_TO_SKIP`/d' DBdump.sql | \
sed '/INSERT INTO `TABLE2_TO_SKIP`/d' | \
sed '/INSERT INTO `TABLE3_TO_SKIP`/d' > reduced.sql
There IS a 'gotcha' - watch out for procedures in your dump that might contain "INSERT INTO TABLE_TO_SKIP".
mysqlimport is not the right tool for importing SQL statements. This tool is meant to import formatted text files such as CSV. What you want to do is feed your sql dump directly to the mysql client with a command like this one:
bash > mysql -D your_database < your_sql_dump.sql
Neither mysql nor mysqlimport provide the feature you need. Your best chance would be importing the whole dump, then dropping the tables you do not want.
If you have access to the server where the dump comes from, then you could create a new dump with mysqldump --ignore-table=database.table_you_dont_want1 --ignore-table=database.table_you_dont_want2 ....
Check out this answer for a workaround to skip importing some table
For anyone working with .sql.gz files; I found the following solution to be very useful. Our database was 25GB+ and I had to remove the log tables.
gzip -cd "./mydb.sql.gz" | sed -r '/INSERT INTO `(log_table_1|log_table_2|log_table_3|log_table_4)`/d' | gzip > "./mydb2.sql.gz"
Thanks to the answer of Don and comment of Xosofox and this related post:
Use zcat and sed or awk to edit compressed .gz text file
Little old, but figure it might still come in handy...
I liked #Don's answer (https://stackoverflow.com/a/26379517/1446005) but found it very annoying that you'd have to write to another file first...
In my particular case this would take too much time and disc space
So I wrote a little bash script:
#!/bin/bash
tables=(table1_to_skip table2_to_skip ... tableN_to_skip)
tableString=$(printf "|%s" "${tables[#]}")
trimmed=${tableString:1}
grepExp="INSERT INTO \`($trimmed)\`"
zcat $1 | grep -vE "$grepExp" | mysql -uroot -p
this does not generate a new sql script but pipes it directly to the database
also, it does create the tables, just doesn't import the data (which was the problem I had with huge log tables)
Unless you have ignored the tables during the dump with mysqldump --ignore-table=database.unwanted_table, you have to use some script or tool to filter out the data you don't want to import from the dump file before passing it to mysql client.
Here is a bash/sh function that would exclude the unwanted tables from a SQL dump on the fly (through pipe):
# Accepts one argument, the list of tables to exclude (case-insensitive).
# Eg. filt_exclude '%session% action_log %_cache'
filt_exclude() {
local excl_tns;
if [ -n "$1" ]; then
# trim & replace /[,;\s]+/ with '|' & replace '%' with '[^`]*'
excl_tns=$(echo "$1" | sed -r 's/^[[:space:]]*//g; s/[[:space:]]*$//g; s/[[:space:]]+/|/g; s/[,;]+/|/g; s/%/[^\`]\*/g');
grep -viE "(^INSERT INTO \`($excl_tns)\`)|(^DROP TABLE (IF EXISTS )?\`($excl_tns)\`)|^LOCK TABLES \`($excl_tns)\` WRITE" | \
sed 's/^CREATE TABLE `/CREATE TABLE IF NOT EXISTS `/g'
else
cat
fi
}
Suppose you have a dump created like so:
MYSQL_PWD="my-pass" mysqldump -u user --hex-blob db_name | \
pigz -9 > dump.sql.gz
And want to exclude some unwanted tables before importing:
pigz -dckq dump.sql.gz | \
filt_exclude '%session% action_log %_cache' | \
MYSQL_PWD="my-pass" mysql -u user db_name
Or you could pipe into a file or any other tool before importing to DB.
If desired, you can do this one table at a time:
mysqldump -p sourceDatabase tableName > tableName.sql
mysql -p -D targetDatabase < tableName.sql
Here is my script to exclude some tables from mysql dump
I use it to restore DB when need to keep orders and payments data
exclude_tables_from_dump.sh
#!/bin/bash
if [ ! -f "$1" ];
then
echo "Usage: $0 mysql_dump.sql"
exit
fi
declare -a TABLES=(
user
order
order_product
order_status
payments
)
CMD="cat $1"
for TBL in "${TABLES[#]}";do
CMD+="|sed 's/DROP TABLE IF EXISTS \`${TBL}\`/# DROP TABLE IF EXIST \`${TBL}\`/g'"
CMD+="|sed 's/CREATE TABLE \`${TBL}\`/CREATE TABLE IF NOT EXISTS \`${TBL}\`/g'"
CMD+="|sed -r '/INSERT INTO \`${TBL}\`/d'"
CMD+="|sed '/DELIMITER\ \;\;/,/DELIMITER\ \;/d'"
done
eval $CMD
It avoid DROP and reCREATE of tables and inserting data to this tables.
Also it strip all FUNCTIONS and PROCEDURES that stored between DELIMITER ;; and DELIMITER ;
I would not use it on production but if I would have to import some backup quickly that contains many smaller table and one big monster table that might take hours to import I would most probably "grep -v unwanted_table_name original.sql > reduced.sql
and then mysql -f < reduced.sql
I have a mysqldump backup of my mysql database consisting of all of our tables which is about 440 megs. I want to restore the contents of just one of the tables from the mysqldump. Is this possible? Theoretically, I could just cut out the section that rebuilds the table I want but I don't even know how to effectively edit a text document that size.
You can try to use sed in order to extract only the table you want.
Let say the name of your table is mytable and the file mysql.dump is the file containing your huge dump:
$ sed -n -e '/CREATE TABLE.*`mytable`/,/Table structure for table/p' mysql.dump > mytable.dump
This will copy in the file mytable.dump what is located between CREATE TABLE mytable and the next CREATE TABLE corresponding to the next table.
You can then adjust the file mytable.dump which contains the structure of the table mytable, and the data (a list of INSERT).
I used a modified version of uloBasEI's sed command. It includes the preceding DROP command, and reads until mysql is done dumping data to your table (UNLOCK). Worked for me (re)importing wp_users to a bunch of Wordpress sites.
sed -n -e '/DROP TABLE.*`mytable`/,/UNLOCK TABLES/p' mydump.sql > tabledump.sql
This can be done more easily? This is how I did it:
Create a temporary database (e.g. restore):
mysqladmin -u root -p create restore
Restore the full dump in the temp database:
mysql -u root -p --one-database restore < fulldump.sql
Dump the table you want to recover:
mysqldump restore mytable > mytable.sql
Import the table in another database:
mysql -u root -p database < mytable.sql
A simple solution would be to simply create a dump of just the table you wish to restore separately. You can use the mysqldump command to do so with the following syntax:
mysqldump -u [user] -p[password] [database] [table] > [output_file_name].sql
Then import it as normal, and it will only import the dumped table.
One way or another, any process doing that will have to go through the entire text of the dump and parse it in some way. I'd just grep for
INSERT INTO `the_table_i_want`
and pipe the output into mysql. Take a look at the first table in the dump before, to make sure you're getting the INSERT's the right way.
Edit: OK, got the formatting right this time.
Backup
$ mysqldump -A | gzip > mysqldump-A.gz
Restore single table
$ mysql -e "truncate TABLE_NAME" DB_NAME
$ zgrep ^"INSERT INTO \`TABLE_NAME" mysqldump-A.gz | mysql DB_NAME
You should try #bryn command but with the ` delimiter otherwise you will also extract the tables having a prefix or a suffix, this is what I usually do:
sed -n -e '/DROP TABLE.*`mytable`/,/UNLOCK TABLES/p' dump.sql > mytable.sql
Also for testing purpose, you may want to change the table name before importing:
sed -n -e 's/`mytable`/`mytable_restored`/g' mytable.sql > mytable_restored.sql
To import you can then use the mysql command:
mysql -u root -p'password' mydatabase < mytable_restore.sql
One possible way to deal with this is to restore to a temporary database, and dump just that table from the temporary database. Then use the new script.
sed -n -e '/-- Table structure for table `my_table_name`/,/UNLOCK TABLES/p' database_file.sql > table_file.sql
This is a better solution than some of the others above because not all SQL dumps contain a DROP TABLE statement. This one will work will all kinds of dumps.
This tool may be is what you want: tbdba-restore-mysqldump.pl
https://github.com/orczhou/dba-tool/blob/master/tbdba-restore-mysqldump.pl
e.g. Restore a table from database dump file:
tbdba-restore-mysqldump.pl -t yourtable -s yourdb -f backup.sql
Table should present with same structure in both dump and database.
`zgrep -a ^"INSERT INTO \`table_name" DbDump-backup.sql.tar.gz | mysql -u<user> -p<password> database_name`
or
`zgrep -a ^"INSERT INTO \`table_name" DbDump-backup.sql | mysql -u<user> -p<password> database_name`
This may help too.
# mysqldump -u root -p database0 > /tmp/database0.sql
# mysql -u root -p -e 'create database database0_bkp'
# mysql -u root -p database0_bkp < /tmp/database0.sql
# mysql -u root -p database0 -e 'insert into database0.table_you_want select * from database0_bkp.table_you_want'
Most modern text editors should be able to handle a text file that size, if your system is up to it.
Anyway, I had to do that once very quickly and i didnt have time to find any tools. I set up a new MySQL instance, imported the whole backup and then spit out just the table I wanted.
Then I imported that table into the main database.
It was tedious but rather easy. Good luck.
You can use vi editor. Type:
vi -o mysql.dump mytable.dump
to open both whole dump mysql.dump and a new file mytable.dump.
Find the appropriate insert into line by pressing / and then type a phrase, for example: "insert into `mytable`", then copy that line using yy. Switch to next file by ctrl+w then down arrow key, paste the copied line with pp. Finally save the new file by typing :wq and quite vi editor by :q.
Note that if you have dumped the data using multiple inserts you can copy (yank) all of them at once using Nyy in which N is the number of lines to be copied.
I have done it with a file of 920 MB size.
I tried a few options, which were incredibly slow. This split a 360GB dump into its tables in a few minutes:
How do I split the output from mysqldump into smaller files?
The 'sed' solutions mentioned earlier are nice but as mentioned not 100% secure
You may have INSERT commands with data containing:
... CREATE TABLE...(whatever)...mytable...
or even the exact string "CREATE TABLE `mytable`;"
if you are storing DML commands for instance!
(and if the table is huge you don't want to check that manually)
I would verify the exact syntax of the dump version used, and have a more restrictive pattern search:
Avoid ".*" and use "^" to ensure we start at the begining of the line.
And I'd prefer to grab the initial 'DROP'
All in all, this works better for me:
sed -n -e '/^DROP TABLE IF EXISTS \`mytable\`;/,/^UNLOCK TABLES;/p' mysql.dump > mytable.dump
Get a decent text editor like Notepad++ or Vim (if you're already proficient with it). Search for the table name and you should be able to highlight just the CREATE, ALTER, and INSERT commands for that table. It may be easier to navigate with your keyboard rather than a mouse. And I would make sure you're on a machine with plenty or RAM so that it will not have a problem loading the entire file at once. Once you've highlighted and copied the rows you need, it would be a good idea to back up just the copied part into it's own backup file and then import it into MySQL.
The chunks of SQL are blocked off with "Table structure for table my_table" and "Dumping data for table my_table."
You can use a Windows command line as follows to get the line numbers for the various sections. Adjust the searched string as needed.
find /n "for table `" sql.txt
The following will be returned:
---------- SQL.TXT
[4384]-- Table structure for table my_table
[4500]-- Dumping data for table my_table
[4514]-- Table structure for table some_other_table
... etc.
That gets you the line numbers you need... now, if I only knew how to use them... investigating.
You can import single table using terminal line as given below.
Here import single user table into specific database.
mysql -u root -p -D my_database_name < var/www/html/myproject/tbl_user.sql
I admire some of the ingenuity here, but there is literally no reason to use sed at all to address the OP's question.
The comment "use --one-database" is the correct answer, built into MySQL/MariaDB. No need for third-party hacks.
mysql -u root -p databasename --one-database < localhost.sql will just import the desired database.
I also found in some cases, when using this to import a series of databases, it would create the next database in the list for me (but not put anything in it). Not sure why it did that, but it made the restore easier.
With this command, enter the password interactively and it will import the requested database.
I'd like to avoid mysqldump since that outputs in a form that is only convenient for mysql to read. CSV seems more universal (one file per table is fine). But if there are advantages to mysqldump, I'm all ears. Also, I'd like something I can run from the command line (linux). If that's a mysql script, pointers to how to make such a thing would be helpful.
If you can cope with table-at-a-time, and your data is not binary, use the -B option to the mysql command. With this option it'll generate TSV (tab separated) files which can import into Excel, etc, quite easily:
% echo 'SELECT * FROM table' | mysql -B -uxxx -pyyy database
Alternatively, if you've got direct access to the server's file system, use SELECT INTO OUTFILE which can generate real CSV files:
SELECT * INTO OUTFILE 'table.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM table
In MySQL itself, you can specify CSV output like:
SELECT order_id,product_name,qty
FROM orders
INTO OUTFILE '/tmp/orders.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
From http://www.tech-recipes.com/rx/1475/save-mysql-query-results-into-a-text-or-csv-file/
You can dump a whole database in one go with mysqldump's --tab option. You supply a directory path and it creates one .sql file with the CREATE TABLE DROP IF EXISTS syntax and a .txt file with the contents, tab separated. To create comma separated files you could use the following:
mysqldump --password --fields-optionally-enclosed-by='"' --fields-terminated-by=',' --tab /tmp/path_to_dump/ database_name
That path needs to be writable by both the mysql user and the user running the command, so for simplicity I recommend chmod 777 /tmp/path_to_dump/ first.
The select into outfile option wouldn't work for me but the below roundabout way of piping tab-delimited file through SED did:
mysql -uusername -ppassword -e "SELECT * from tablename" dbname | sed 's/\t/","/g;s/^/"/;s/$/"/' > /path/to/file/filename.csv
Here is the simplest command for it
mysql -h<hostname> -u<username> -p<password> -e 'select * from databaseName.tableNaame' | sed 's/\t/,/g' > output.csv
If there is a comma in the column value then we can generate .tsv instead of .csv with the following command
mysql -h<hostname> -u<username> -p<password> -e 'select * from databaseName.tableNaame' > output.csv
If you really need a "Backup" then you also need database schema, like table definitions, view definitions, store procedures and so on. A backup of a database isn't just the data.
The value of the mysqldump format for backup is specifically that it is very EASY to use it to restore mysql databases. A backup that isn't easily restored is far less useful. If you are looking for a method to reliably backup mysql data to so you can restore to a mysql server then I think you should stick with the mysqldump tool.
Mysql is free and runs on many different platforms. Setting up a new mysql server that I can restore to is simple. I am not at all worried about not being able to setup mysql so I can do a restore.
I would be far more worried about a custom backup/restore based on a fragile format like csv/tsv failing. Are you sure that all your quotes, commas, or tabs that are in your data would get escaped correctly and then parsed correctly by your restore tool?
If you are looking for a method to extract the data then see several in the other answers.
You can use below script to get the output to csv files. One file per table with headers.
for tn in `mysql --batch --skip-page --skip-column-name --raw -uuser -ppassword -e"show tables from mydb"`
do
mysql -uuser -ppassword mydb -B -e "select * from \`$tn\`;" | sed 's/\t/","/g;s/^/"/;s/$/"/;s/\n//g' > $tn.csv
done
user is your user name, password is the password if you don't want to keep typing the password for each table and mydb is the database name.
Explanation of the script: The first expression in sed, will replace the tabs with "," so you have fields enclosed in double quotes and separated by commas. The second one insert double quote in the beginning and the third one insert double quote at the end. And the final one takes care of the \n.
If you want to dump the entire db as csv
#!/bin/bash
host=hostname
uname=username
pass=password
port=portnr
db=db_name
s3_url=s3://buckera/db_dump/
DATE=`date +%Y%m%d`
rm -rf $DATE
echo 'show tables' | mysql -B -h${host} -u${uname} -p${pass} -P${port} ${db} > tables.txt
awk 'NR>1' tables.txt > tables_new.txt
while IFS= read -r line
do
mkdir -p $DATE/$line
echo "select * from $line" | mysql -B -h"${host}" -u"${uname}" -p"${pass}" -P"${port}" "${db}" > $DATE/$line/dump.tsv
done < tables_new.txt
touch $DATE/$DATE.fin
rm -rf tables_new.txt tables.txt
Check out mk-parallel-dump which is part of the ever-useful maatkit suite of tools. This can dump comma-separated files with the --csv option.
This can do your whole db without specifying individual tables, and you can specify groups of tables in a backupset table.
Note that it also dumps table definitions, views and triggers into separate files. In addition providing a complete backup in a more universally accessible form, it also immediately restorable with mk-parallel-restore
Two line PowerShell answer:
# Store in variable
$Global:csv = (mysql -uroot -p -hlocalhost -Ddatabase_name -B -e "SELECT * FROM some_table") `
| ConvertFrom-Csv -Delimiter "`t"
# Out to csv
$Global:csv | Export-Csv "C:\temp\file.csv" -NoTypeInformation
Boom-bata-boom
-D = the name of your database
-e = query
-B = tab-delimited
There's a slightly simpler way to get all the tables into tab delimited fast:
#!/bin/bash
tablenames=$(mysql your_database -e "show tables;" -B |sed "1d")
IFS=$'\n'
tables=($tablenames)
for table in ${tables[#]}; do
mysql your_database -e "select * from ${table}" -B > "${table}.tsv"
done
Here's a basic python script that does the work! You can also choose to export only the headers (column names) or headers & data both.
Just change the database credentials and run the script. It will output all the data to the output folder.
To run the script -
Run: pip install mysql-connector-python
Change database credentials in the "INPUT" section
Run: python filename.py
import mysql.connector
from pathlib import Path
import csv
#========INPUT===========
databaseHost=""
databaseUsername=""
databasePassword=""
databaseName=""
outputDirectory="./WITH-DATA/"
exportTableData=True #MAKING THIS FIELD FALSE WILL STORE ONLY THE TABLE HEADERS (COLUMN NAMES) IN THE CSV FILE
#========INPUT END===========
Path(outputDirectory).mkdir(parents=True, exist_ok=True)
mydb = mysql.connector.connect(
host=databaseHost,
user=databaseUsername,
password=databasePassword
)
mycursor = mydb.cursor()
mycursor.execute("USE "+databaseName)
mycursor.execute("SHOW TABLES")
tables = mycursor.fetchall()
tableNames=[table[0] for table in tables]
print("================================")
print("Total number of tables: "+ str(len(tableNames)))
print(tableNames)
print("================================")
for tableName in tableNames:
print("================================")
print("Processing: "+ str(tableName))
mydb = mysql.connector.connect(
host=databaseHost,
user=databaseUsername,
password=databasePassword
)
mycursor = mydb.cursor()
mycursor.execute("USE "+databaseName)
if exportTableData:
mycursor.execute("SELECT * FROM "+tableName)
else:
mycursor.execute("SELECT * FROM "+tableName+" LIMIT 1")
print(mycursor.column_names)
with open(outputDirectory+tableName+".csv", 'w', newline='') as csvfile:
csvwriter = csv.writer(csvfile)
csvwriter.writerow(mycursor.column_names)
if exportTableData:
myresult = mycursor.fetchall()
csvwriter.writerows(myresult)