I want to export a list of tables starting with a certain prefix using a wild card.
Ideally I would do:
mysqldump -uroot -p mydb table_prefix_* > backup.sql
Obviously that doesn’t work. But what is the correct way to do this?
If it has a prefix, make a user with select and lock permissions to the table with GRANT like this
GRANT SELECT, LOCK TABLES ON `table\_prefix\_%` . * TO 'backup-user'#'localhost';
then instead of running mysql dump as root, run it as backup-user with the option --all-databases
as backup user only has select and lock permission on these tables, They are the only ones which will be there.
its also safer using a user like this rather than the root account for everything
Ok. Here is a solution using a Bash script (I've tested it in Linux).
Create a text file named "dumpTablesWithPrefix.script" and put this in it:
tableList=$(mysql -h $1 -u $2 -p$3 $4 -e"show tables" | grep "^$5.*" | sed ':a;N;$!ba;s/\n/ /g')
mysqldump -h $1 -u $2 -p$3 $4 $tableList
Save it, and make it executable:
$ chmod +x dumpTablesWithPrefix.script
Now you can run it:
$ ./dumpTablesWithPrefix.script host user pwd database tblPrefix > output.sql
Now let me explain each piece:
The command-line arguments
Bash scripts store command-line arguments in the variables $1, $2, $3 and so on. So the first thing I need to tell you is the order of the arguments you need:
The host (if the mysql server is in your machine, write localhost)
Your user
Your password
The name of your database
The prefix of the tables you want to export
The first line of the script
I'll split this line in three pieces:
mysql -h $1 -u $2 -p$3 $4 -e"show tables"
This piece retrieves the full table list of your database, one table per line.
| grep "^$5.*"
This piece filters the table list, using the prefix you specified.
| sed ':a;N;$!ba;s/\n/ /g'
This final piece (a gem I found here: How can I replace a newline (\n) using sed? ) replaces all the "new line" characters with spaces.
This three pieces, strung together, throw the filtered table list and store it in the tableList variable.
The second line of the script
mysqldump -h $1 -u $2 -p$3 $4 $tableList
This line is simply the MySQL Dump command that does what you need.
Hope this helps
Related
Currently, I have a usrpwd.txt file with users and passwords separated by 'space'. I wish to add these users to my mysql server and change their privileges. Since there are 2000 users, it becomes unpractical to add them manually. I tried to use awk in a bash script:
awk '{mysql -u root -proot -e "CREATE USER '\''"$1"'\''#'\''%'\'' IDENTIFIED BY '\''"$2"'\'';"}' ./conf/usrpwd.txt ;
Unfortunately, the users are not being added. No error is reported.
Regards, Andre
My first instinct wouldn't be to solve this in awk, but anyway, here it is:
$ cat tst.awk
{
cmd = sprintf("mysql -u root -p root -h localhost -e \"CREATE USER '%s'#'%' IDENTIFIED BY '%s'\"", $1, $2)
if ((cmd | getline line) > 0)
print line
close(cmd)
}
You can call this like:
$ awk -f tst.awk usrpwd.txt
or you could put this in a bash script, if you wanted to.
I have a script to dump my databases, like this :
#!/usr/bin/env sh
PATH=/usr/bin:/bin
LOG="mybackup/log/backup_$(date +%Y-%m-%d_%H:%M:%S).log"
# some other command
# backup mysql database
echo "Backing up database" > ~/$LOG
mysqldump -u myusername -pmypassword --ignore-table={db1.table1,db1.table2,db1.table3,db1.table4} db1 -r ~/mybackup/db/db1_$(date +%Y-%m-%d_%H:%M:%S).sql >> ~/$LOG 2>&1
# some other command
When I run the command in terminal, it successfully dump my database without ignored tables. But when I run the command through the script, it dump all tables in the database.
I have tried to escape the curly braces :
mysqldump -u myusername -pmypassword --ignore-table=\{db1.table1,db1.table2,db1.table3,db1.table4\} db1 -r ~/mybackup/db/db1_$(date +%Y-%m-%d_%H:%M:%S).sql
but it still dump all table.
My question is how to escape braces/brackets properly in shell script?
I can't answer why it DOES work on command line, but the documentation says to use the option multiple times:
mysqldump --ignore-table documentation
man mylsqdump
--ignore-table=db_name.tbl_name
Do not dump the given table, which must be specified using both the
database and table names. To ignore multiple tables, use this option
multiple times. This option also can be used to ignore views.
I have a large SQL file with one database and about 150 tables. I would like to use mysqlimport to import that database, however, I would like the import process to ignore or skip over a couple of tables. What is the proper syntax to import all tables, but ignore some of them? Thank you.
The accepted answer by RandomSeed could take a long time! Importing the table (just to drop it later) could be very wasteful depending on size.
For a file created using
mysqldump -u user -ppasswd --opt --routines DBname > DBdump.sql
I currently get a file about 7GB, 6GB of which is data for a log table that I don't 'need' to be there; reloading this file takes a couple of hours. If I need to reload (for development purposes, or if ever required for a live recovery) I skim the file thus:
sed '/INSERT INTO `TABLE_TO_SKIP`/d' DBdump.sql > reduced.sql
And reload with:
mysql -u user -ppasswd DBname < reduced.sql
This gives me a complete database, with the "unwanted" table created but empty. If you really don't want the tables at all, simply drop the empty tables after the load finishes.
For multiple tables you could do something like this:
sed '/INSERT INTO `TABLE1_TO_SKIP`/d' DBdump.sql | \
sed '/INSERT INTO `TABLE2_TO_SKIP`/d' | \
sed '/INSERT INTO `TABLE3_TO_SKIP`/d' > reduced.sql
There IS a 'gotcha' - watch out for procedures in your dump that might contain "INSERT INTO TABLE_TO_SKIP".
mysqlimport is not the right tool for importing SQL statements. This tool is meant to import formatted text files such as CSV. What you want to do is feed your sql dump directly to the mysql client with a command like this one:
bash > mysql -D your_database < your_sql_dump.sql
Neither mysql nor mysqlimport provide the feature you need. Your best chance would be importing the whole dump, then dropping the tables you do not want.
If you have access to the server where the dump comes from, then you could create a new dump with mysqldump --ignore-table=database.table_you_dont_want1 --ignore-table=database.table_you_dont_want2 ....
Check out this answer for a workaround to skip importing some table
For anyone working with .sql.gz files; I found the following solution to be very useful. Our database was 25GB+ and I had to remove the log tables.
gzip -cd "./mydb.sql.gz" | sed -r '/INSERT INTO `(log_table_1|log_table_2|log_table_3|log_table_4)`/d' | gzip > "./mydb2.sql.gz"
Thanks to the answer of Don and comment of Xosofox and this related post:
Use zcat and sed or awk to edit compressed .gz text file
Little old, but figure it might still come in handy...
I liked #Don's answer (https://stackoverflow.com/a/26379517/1446005) but found it very annoying that you'd have to write to another file first...
In my particular case this would take too much time and disc space
So I wrote a little bash script:
#!/bin/bash
tables=(table1_to_skip table2_to_skip ... tableN_to_skip)
tableString=$(printf "|%s" "${tables[#]}")
trimmed=${tableString:1}
grepExp="INSERT INTO \`($trimmed)\`"
zcat $1 | grep -vE "$grepExp" | mysql -uroot -p
this does not generate a new sql script but pipes it directly to the database
also, it does create the tables, just doesn't import the data (which was the problem I had with huge log tables)
Unless you have ignored the tables during the dump with mysqldump --ignore-table=database.unwanted_table, you have to use some script or tool to filter out the data you don't want to import from the dump file before passing it to mysql client.
Here is a bash/sh function that would exclude the unwanted tables from a SQL dump on the fly (through pipe):
# Accepts one argument, the list of tables to exclude (case-insensitive).
# Eg. filt_exclude '%session% action_log %_cache'
filt_exclude() {
local excl_tns;
if [ -n "$1" ]; then
# trim & replace /[,;\s]+/ with '|' & replace '%' with '[^`]*'
excl_tns=$(echo "$1" | sed -r 's/^[[:space:]]*//g; s/[[:space:]]*$//g; s/[[:space:]]+/|/g; s/[,;]+/|/g; s/%/[^\`]\*/g');
grep -viE "(^INSERT INTO \`($excl_tns)\`)|(^DROP TABLE (IF EXISTS )?\`($excl_tns)\`)|^LOCK TABLES \`($excl_tns)\` WRITE" | \
sed 's/^CREATE TABLE `/CREATE TABLE IF NOT EXISTS `/g'
else
cat
fi
}
Suppose you have a dump created like so:
MYSQL_PWD="my-pass" mysqldump -u user --hex-blob db_name | \
pigz -9 > dump.sql.gz
And want to exclude some unwanted tables before importing:
pigz -dckq dump.sql.gz | \
filt_exclude '%session% action_log %_cache' | \
MYSQL_PWD="my-pass" mysql -u user db_name
Or you could pipe into a file or any other tool before importing to DB.
If desired, you can do this one table at a time:
mysqldump -p sourceDatabase tableName > tableName.sql
mysql -p -D targetDatabase < tableName.sql
Here is my script to exclude some tables from mysql dump
I use it to restore DB when need to keep orders and payments data
exclude_tables_from_dump.sh
#!/bin/bash
if [ ! -f "$1" ];
then
echo "Usage: $0 mysql_dump.sql"
exit
fi
declare -a TABLES=(
user
order
order_product
order_status
payments
)
CMD="cat $1"
for TBL in "${TABLES[#]}";do
CMD+="|sed 's/DROP TABLE IF EXISTS \`${TBL}\`/# DROP TABLE IF EXIST \`${TBL}\`/g'"
CMD+="|sed 's/CREATE TABLE \`${TBL}\`/CREATE TABLE IF NOT EXISTS \`${TBL}\`/g'"
CMD+="|sed -r '/INSERT INTO \`${TBL}\`/d'"
CMD+="|sed '/DELIMITER\ \;\;/,/DELIMITER\ \;/d'"
done
eval $CMD
It avoid DROP and reCREATE of tables and inserting data to this tables.
Also it strip all FUNCTIONS and PROCEDURES that stored between DELIMITER ;; and DELIMITER ;
I would not use it on production but if I would have to import some backup quickly that contains many smaller table and one big monster table that might take hours to import I would most probably "grep -v unwanted_table_name original.sql > reduced.sql
and then mysql -f < reduced.sql
I have a big mySQL database dump named forum.sql. I want to restore only one table, but when I restore the full database, it takes a long time to import the "post" table.
Is there any option to restore this database skipping the "post" table?
If you are restoring from a dump file, you can easily build a new dumpfile without this table, just by writting down the line numbers.
Initial line
> grep dumpfile.sql -ne "Dumping data for table \`avoid_tablename\`" -m 1
43:-- Dumping data for table `avoid_tablename`
Total lines
> wc -l dumpfile.sql
63 dumpfile.sql
Make a new file
> head -n 43 dumpfile.sql > dumpfile-lite.sql
> tail -n 20 dumpfile.sql >> dumpfile-lile.sql
20 comes from substracting 63 - 43
not clean, but usefull
Alternatively, extract the table(s) that need to be restored from fulldump.sql using sed:
sed -n -e '/CREATE TABLE.*tableName1/,/CREATE TABLE/p' fulldump.sql > temp.sql
sed -n -e '/CREATE TABLE.*tableName2/,/CREATE TABLE/p' fulldump.sql >> temp.sql
...etc
Now restore from temp.sql.
Restore a single table
First, do the restore with zero inserts:
cat dump.sql | grep -v '^INSERT INTO' | mysql -u <user> -p<pw> <dbname>
Using grep -v here will exclude any statements matching the pattern. The pattern in this case uses ^ to match at the beginning of a line. The result should be a restored schema with zero data.
Next, restore only your desired INSERT statements:
cat dump.sql | grep '^INSERT INTO \\\`<table>\\\`' | mysql -u <user> -p<pw> <dbname>
That will restore data only for the table named <table>. Note the triple backslashes. You need a backslash to escape a backtick and then you need to escape the backslash with 2 more backslashes.
Restore everything except one table
Another technique I use all the time when I want to restore an entire database but exclude the data from a table or two is this... You can filter out any unwanted INSERT statements by passing your dump through a filter before pushing into the db. Here's an example using grep as the filter:
nohup sh -c "cat dump.sql | grep -v 'INSERT INTO \\\`<table>\\\`' | mysql -u <user> -p<pw> <dbname>" &
The nohup command will keep the sh command running even if you log out of your shell. That can be pretty handy if you have a large dump file that will take quite some time to restore.
The -v flag for grep will exclude anything matching the pattern.
The & at the end will send the command to the background.
As far as I know, no.
You would have to manually edit the CREATE and INSERT statements of the undesired table out of the dump file.
I don't think you can do it. But you can dump tables separately when necessary, using --tables myqsldump option. So, you can generate a dump for post table and another dump for the remaining tables.
Example:
mysqldump -u USERNAME -pPASSWORD --tables TABLE_NAME database_name > TABLE_NAME.sql
You could alter your dump statement so that it uses ignore table? http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html#option_mysqldump_ignore-table
I have a mysqldump backup of my mysql database consisting of all of our tables which is about 440 megs. I want to restore the contents of just one of the tables from the mysqldump. Is this possible? Theoretically, I could just cut out the section that rebuilds the table I want but I don't even know how to effectively edit a text document that size.
You can try to use sed in order to extract only the table you want.
Let say the name of your table is mytable and the file mysql.dump is the file containing your huge dump:
$ sed -n -e '/CREATE TABLE.*`mytable`/,/Table structure for table/p' mysql.dump > mytable.dump
This will copy in the file mytable.dump what is located between CREATE TABLE mytable and the next CREATE TABLE corresponding to the next table.
You can then adjust the file mytable.dump which contains the structure of the table mytable, and the data (a list of INSERT).
I used a modified version of uloBasEI's sed command. It includes the preceding DROP command, and reads until mysql is done dumping data to your table (UNLOCK). Worked for me (re)importing wp_users to a bunch of Wordpress sites.
sed -n -e '/DROP TABLE.*`mytable`/,/UNLOCK TABLES/p' mydump.sql > tabledump.sql
This can be done more easily? This is how I did it:
Create a temporary database (e.g. restore):
mysqladmin -u root -p create restore
Restore the full dump in the temp database:
mysql -u root -p --one-database restore < fulldump.sql
Dump the table you want to recover:
mysqldump restore mytable > mytable.sql
Import the table in another database:
mysql -u root -p database < mytable.sql
A simple solution would be to simply create a dump of just the table you wish to restore separately. You can use the mysqldump command to do so with the following syntax:
mysqldump -u [user] -p[password] [database] [table] > [output_file_name].sql
Then import it as normal, and it will only import the dumped table.
One way or another, any process doing that will have to go through the entire text of the dump and parse it in some way. I'd just grep for
INSERT INTO `the_table_i_want`
and pipe the output into mysql. Take a look at the first table in the dump before, to make sure you're getting the INSERT's the right way.
Edit: OK, got the formatting right this time.
Backup
$ mysqldump -A | gzip > mysqldump-A.gz
Restore single table
$ mysql -e "truncate TABLE_NAME" DB_NAME
$ zgrep ^"INSERT INTO \`TABLE_NAME" mysqldump-A.gz | mysql DB_NAME
You should try #bryn command but with the ` delimiter otherwise you will also extract the tables having a prefix or a suffix, this is what I usually do:
sed -n -e '/DROP TABLE.*`mytable`/,/UNLOCK TABLES/p' dump.sql > mytable.sql
Also for testing purpose, you may want to change the table name before importing:
sed -n -e 's/`mytable`/`mytable_restored`/g' mytable.sql > mytable_restored.sql
To import you can then use the mysql command:
mysql -u root -p'password' mydatabase < mytable_restore.sql
One possible way to deal with this is to restore to a temporary database, and dump just that table from the temporary database. Then use the new script.
sed -n -e '/-- Table structure for table `my_table_name`/,/UNLOCK TABLES/p' database_file.sql > table_file.sql
This is a better solution than some of the others above because not all SQL dumps contain a DROP TABLE statement. This one will work will all kinds of dumps.
This tool may be is what you want: tbdba-restore-mysqldump.pl
https://github.com/orczhou/dba-tool/blob/master/tbdba-restore-mysqldump.pl
e.g. Restore a table from database dump file:
tbdba-restore-mysqldump.pl -t yourtable -s yourdb -f backup.sql
Table should present with same structure in both dump and database.
`zgrep -a ^"INSERT INTO \`table_name" DbDump-backup.sql.tar.gz | mysql -u<user> -p<password> database_name`
or
`zgrep -a ^"INSERT INTO \`table_name" DbDump-backup.sql | mysql -u<user> -p<password> database_name`
This may help too.
# mysqldump -u root -p database0 > /tmp/database0.sql
# mysql -u root -p -e 'create database database0_bkp'
# mysql -u root -p database0_bkp < /tmp/database0.sql
# mysql -u root -p database0 -e 'insert into database0.table_you_want select * from database0_bkp.table_you_want'
Most modern text editors should be able to handle a text file that size, if your system is up to it.
Anyway, I had to do that once very quickly and i didnt have time to find any tools. I set up a new MySQL instance, imported the whole backup and then spit out just the table I wanted.
Then I imported that table into the main database.
It was tedious but rather easy. Good luck.
You can use vi editor. Type:
vi -o mysql.dump mytable.dump
to open both whole dump mysql.dump and a new file mytable.dump.
Find the appropriate insert into line by pressing / and then type a phrase, for example: "insert into `mytable`", then copy that line using yy. Switch to next file by ctrl+w then down arrow key, paste the copied line with pp. Finally save the new file by typing :wq and quite vi editor by :q.
Note that if you have dumped the data using multiple inserts you can copy (yank) all of them at once using Nyy in which N is the number of lines to be copied.
I have done it with a file of 920 MB size.
I tried a few options, which were incredibly slow. This split a 360GB dump into its tables in a few minutes:
How do I split the output from mysqldump into smaller files?
The 'sed' solutions mentioned earlier are nice but as mentioned not 100% secure
You may have INSERT commands with data containing:
... CREATE TABLE...(whatever)...mytable...
or even the exact string "CREATE TABLE `mytable`;"
if you are storing DML commands for instance!
(and if the table is huge you don't want to check that manually)
I would verify the exact syntax of the dump version used, and have a more restrictive pattern search:
Avoid ".*" and use "^" to ensure we start at the begining of the line.
And I'd prefer to grab the initial 'DROP'
All in all, this works better for me:
sed -n -e '/^DROP TABLE IF EXISTS \`mytable\`;/,/^UNLOCK TABLES;/p' mysql.dump > mytable.dump
Get a decent text editor like Notepad++ or Vim (if you're already proficient with it). Search for the table name and you should be able to highlight just the CREATE, ALTER, and INSERT commands for that table. It may be easier to navigate with your keyboard rather than a mouse. And I would make sure you're on a machine with plenty or RAM so that it will not have a problem loading the entire file at once. Once you've highlighted and copied the rows you need, it would be a good idea to back up just the copied part into it's own backup file and then import it into MySQL.
The chunks of SQL are blocked off with "Table structure for table my_table" and "Dumping data for table my_table."
You can use a Windows command line as follows to get the line numbers for the various sections. Adjust the searched string as needed.
find /n "for table `" sql.txt
The following will be returned:
---------- SQL.TXT
[4384]-- Table structure for table my_table
[4500]-- Dumping data for table my_table
[4514]-- Table structure for table some_other_table
... etc.
That gets you the line numbers you need... now, if I only knew how to use them... investigating.
You can import single table using terminal line as given below.
Here import single user table into specific database.
mysql -u root -p -D my_database_name < var/www/html/myproject/tbl_user.sql
I admire some of the ingenuity here, but there is literally no reason to use sed at all to address the OP's question.
The comment "use --one-database" is the correct answer, built into MySQL/MariaDB. No need for third-party hacks.
mysql -u root -p databasename --one-database < localhost.sql will just import the desired database.
I also found in some cases, when using this to import a series of databases, it would create the next database in the list for me (but not put anything in it). Not sure why it did that, but it made the restore easier.
With this command, enter the password interactively and it will import the requested database.