Script for MySQL backup to multiple files - mysql

I am creating a script to back up the MySQL running on a Windows 2012 server, using PowerShell. Unlike other tips found here, I want to generate a .sql file for each of the databases.
This post shows how to create multiple files. I adapted it to PowerShell:
Get-ChildItem -Path "$($MYSQL_HOME)\data" | cmd /C "$($MYSQL_HOME)\bin\ mysql -u -s -r $($dbuser) -p$($dbpass) -e 'databases show' '| while read dbname; cmd /C "$($MYSQL_HOME)\bin\mysqldump.exe --user = $($dbuser) --password = $($dbpass) --databases $dbname> $($BKP_FOLDER)\$dbname($BACKUPDATE).sql "
but it returns error in while.
What should I change so that you can generate multiple .sql, one for each database?

Your entire commandline is b0rken. Throw it away and start over. Try something like this:
Set-Location "$MYSQL_HOME\bin"
& .\mysql.exe -N -s -r -u $dbuser -p$dbpass -e 'show databases' | % {
& .\mysqldump.exe -u $dbuser -p$dbpass --single-transaction $_ |
Out-File "$BKP_FOLDER\${_}$BACKUPDATE.sql" -Encoding Ascii
}

Example using an input from file result of mysqldump.
#Variables#####
# Debe especificar la ruta completa, por ejemplo C:\Temp\archivo.txt
$file = $args[0]
$c = ""
$i = 0
$startSafe = 0;
###############
New-Item -ItemType Directory -Force -Path .\data
$x = 0;
foreach ($f in [System.IO.File]::ReadLines($file)) {
if ($f -like '-- Dumping data for table *') {
$startSafe = 1;
$x = $f.split('`')[1];
write $x;
}
if ($startSafe) {
if($f -eq "UNLOCK TABLES;"){$i += 1; $f >> .\data\$x.out.sql; $startSafe = 0; $x = $i}
if($f -ne "UNLOCK TABLES;"){ $f >> .\data\$x.out.sql;}
}
}

Related

Redirecting mysql output to prompt using shell

I'm writing a shell code to automaticaly run multiple sql queries in multiple databases. My code is working well, but beside "select" queries, all other queries aren't displaying anything in the prompt while executing. How could I force query outputs to redirect to prompt?
That's some of my code:
for x in "${db[#]}"
do
found=0
for enreg in `cat /home/dbfile.csv`
do
#extracting database data from csv
DBNAME=`echo $enreg | awk -F";" '{ print $4 }'`
if [ $x = $DBNAME ]
then
PASS=`echo $enreg | awk -F";" '{ print $2 }'`
HOST=`echo $enreg | awk -F";" '{ print $3 }'`
USERNAME=`echo $enreg | awk -F";" '{ print $1 }'`
# Running queries in database $DBNAME
for y in "${req[#]}"
do
echo "
"
mysql -u $USERNAME -p$PASS -h $HOST $DBNAME -e "$y"
echo "
"
done
found=1
break
fi
done
if [ $found -eq 0 ]
then
echo "Database $x doesn't exist"
fi
done

get multiple variables from a line using shell script

i am trying to monitor my hosts through ping. the hosts information are in a mysql table. i m using fping command the code is as followings
#/bin/sh
id=$(mysql -B --column-names=0 -uroot -pPassword -D monitor -e "SELECT ipv4 FROM nics WHERE icmp=1");
result=$(fping -c 10 $id |grep 'xmt/rcv/%loss');
#echo $result;
for line in $result;
do
echo $line
done
the output is
111.125.140.6 : xmt/rcv/%loss = 10/10/0%, min/avg/max = 234/234/235
123.135.140.7 : xmt/rcv/%loss = 10/0/100%
111.125.140.1 : xmt/rcv/%loss = 10/10/0%, min/avg/max = 230/231/231
111.125.130.2 : xmt/rcv/%loss = 10/10/0%, min/avg/max = 234/234/234
now i want to get IP, loss and average record from each line and input the data in a new table
thanks in advance
Using a while loop (instead of a for loop) with awk:
while read -r line
do declare $(awk '{split($0,a,"[%/ ,]"); print \
"ip="a[1],"send="a[8],"rec="a[9],"loss="a[10],"min="a[17],"avg="a[18],"max="a[19]}' <<< $line);
## variables created: [ip, send, rec, loss, min, avg, max]
## test with: echo $ip, $send, $rec, $loss, $min, $avg, $max
## now you can insert the variables into your new SQL table:
echo "INSERT INTO newtable (IP,SEND,REC,LOSS,MIN,AVG,MAX) \
VALUES ('$ip','$send','$rec','$loss','$min','$avg','$max');"
done< <(printf '%s\n' "$result") | mysql -uroot -pPassword foobar
This will to declare individual variables from your $line strings which you can insert into SQL.

Create mysql database and user in bash script

This seems like it should be simple and I swear this code has worked for months but it's not working now. I'm sure I'm just overly tired but I would appreciate a knowing nudge.
# create random password
PASSWDDB="$(openssl rand -base64 12)"
# replace "-" with "_" for database username
MAINDB=${USER_NAME//[^a-zA-Z0-9]/_}
# create database and user
mysql -e "CREATE DATABASE $MAINDB"
mysql -e "GRANT ALL PRIVILEGES ON $MAINDB.* TO $MAINDB#localhost IDENTIFIED BY '$PASSWDDB!'"
MAINDB is supplied previously in the script. What I end up with is a database, a user, and proper permissions and the user has a password - it's just not the password defined in $PASSWORDDB.
MariaDB 10.0.22
Ubuntu 14.04
This is what I use: https://raw.githubusercontent.com/saadismail/useful-bash-scripts/master/db.sh
In your case you can use this:
# create random password
PASSWDDB="$(openssl rand -base64 12)"
# replace "-" with "_" for database username
MAINDB=${USER_NAME//[^a-zA-Z0-9]/_}
# If /root/.my.cnf exists then it won't ask for root password
if [ -f /root/.my.cnf ]; then
mysql -e "CREATE DATABASE ${MAINDB} /*\!40100 DEFAULT CHARACTER SET utf8 */;"
mysql -e "CREATE USER ${MAINDB}#localhost IDENTIFIED BY '${PASSWDDB}';"
mysql -e "GRANT ALL PRIVILEGES ON ${MAINDB}.* TO '${MAINDB}'#'localhost';"
mysql -e "FLUSH PRIVILEGES;"
# If /root/.my.cnf doesn't exist then it'll ask for root password
else
echo "Please enter root user MySQL password!"
echo "Note: password will be hidden when typing"
read -sp rootpasswd
mysql -uroot -p${rootpasswd} -e "CREATE DATABASE ${MAINDB} /*\!40100 DEFAULT CHARACTER SET utf8 */;"
mysql -uroot -p${rootpasswd} -e "CREATE USER ${MAINDB}#localhost IDENTIFIED BY '${PASSWDDB}';"
mysql -uroot -p${rootpasswd} -e "GRANT ALL PRIVILEGES ON ${MAINDB}.* TO '${MAINDB}'#'localhost';"
mysql -uroot -p${rootpasswd} -e "FLUSH PRIVILEGES;"
fi
I have created a one which you may find useful:
#!/bin/bash
#
# Script to create MySQL db + user
#
# #author Raj KB <magepsycho#gmail.com>
# #website http://www.magepsycho.com
# #version 0.1.0
################################################################################
# CORE FUNCTIONS - Do not edit
################################################################################
#
# VARIABLES
#
_bold=$(tput bold)
_underline=$(tput sgr 0 1)
_reset=$(tput sgr0)
_purple=$(tput setaf 171)
_red=$(tput setaf 1)
_green=$(tput setaf 76)
_tan=$(tput setaf 3)
_blue=$(tput setaf 38)
#
# HEADERS & LOGGING
#
function _debug()
{
[ "$DEBUG" -eq 1 ] && $#
}
function _header()
{
printf "\n${_bold}${_purple}========== %s ==========${_reset}\n" "$#"
}
function _arrow()
{
printf "➜ $#\n"
}
function _success()
{
printf "${_green}✔ %s${_reset}\n" "$#"
}
function _error() {
printf "${_red}✖ %s${_reset}\n" "$#"
}
function _warning()
{
printf "${_tan}➜ %s${_reset}\n" "$#"
}
function _underline()
{
printf "${_underline}${_bold}%s${_reset}\n" "$#"
}
function _bold()
{
printf "${_bold}%s${_reset}\n" "$#"
}
function _note()
{
printf "${_underline}${_bold}${_blue}Note:${_reset} ${_blue}%s${_reset}\n" "$#"
}
function _die()
{
_error "$#"
exit 1
}
function _safeExit()
{
exit 0
}
#
# UTILITY HELPER
#
function _seekConfirmation()
{
printf "\n${_bold}$#${_reset}"
read -p " (y/n) " -n 1
printf "\n"
}
# Test whether the result of an 'ask' is a confirmation
function _isConfirmed()
{
if [[ "$REPLY" =~ ^[Yy]$ ]]; then
return 0
fi
return 1
}
function _typeExists()
{
if [ $(type -P $1) ]; then
return 0
fi
return 1
}
function _isOs()
{
if [[ "${OSTYPE}" == $1* ]]; then
return 0
fi
return 1
}
function _checkRootUser()
{
#if [ "$(id -u)" != "0" ]; then
if [ "$(whoami)" != 'root' ]; then
echo "You have no permission to run $0 as non-root user. Use sudo"
exit 1;
fi
}
function _printPoweredBy()
{
cat <<"EOF"
Powered By:
__ ___ ___ __
/ |/ /__ ____ ____ / _ \___ __ ______/ / ___
/ /|_/ / _ `/ _ `/ -_) ___(_-</ // / __/ _ \/ _ \
/_/ /_/\_,_/\_, /\__/_/ /___/\_, /\__/_//_/\___/
/___/ /___/
>> Store: http://www.magepsycho.com
>> Blog: http://www.blog.magepsycho.com
################################################################
EOF
}
################################################################################
# SCRIPT FUNCTIONS
################################################################################
function generatePassword()
{
echo "$(openssl rand -base64 12)"
}
function _printUsage()
{
echo -n "$(basename $0) [OPTION]...
Create MySQL db & user.
Version $VERSION
Options:
-h, --host MySQL Host
-d, --database MySQL Database
-u, --user MySQL User
-p, --pass MySQL Password (If empty, auto-generated)
-h, --help Display this help and exit
-v, --version Output version information and exit
Examples:
$(basename $0) --help
"
_printPoweredBy
exit 1
}
function processArgs()
{
# Parse Arguments
for arg in "$#"
do
case $arg in
-h=*|--host=*)
DB_HOST="${arg#*=}"
;;
-d=*|--database=*)
DB_NAME="${arg#*=}"
;;
-u=*|--user=*)
DB_USER="${arg#*=}"
;;
-p=*|--pass=*)
DB_PASS="${arg#*=}"
;;
--debug)
DEBUG=1
;;
-h|--help)
_printUsage
;;
*)
_printUsage
;;
esac
done
[[ -z $DB_NAME ]] && _error "Database name cannot be empty." && exit 1
[[ $DB_USER ]] || DB_USER=$DB_NAME
}
function createMysqlDbUser()
{
SQL1="CREATE DATABASE IF NOT EXISTS ${DB_NAME};"
SQL2="CREATE USER '${DB_USER}'#'%' IDENTIFIED BY '${DB_PASS}';"
SQL3="GRANT ALL PRIVILEGES ON ${DB_NAME}.* TO '${DB_USER}'#'%';"
SQL4="FLUSH PRIVILEGES;"
if [ -f /root/.my.cnf ]; then
$BIN_MYSQL -e "${SQL1}${SQL2}${SQL3}${SQL4}"
else
# If /root/.my.cnf doesn't exist then it'll ask for root password
_arrow "Please enter root user MySQL password!"
read rootPassword
$BIN_MYSQL -h $DB_HOST -u root -p${rootPassword} -e "${SQL1}${SQL2}${SQL3}${SQL4}"
fi
}
function printSuccessMessage()
{
_success "MySQL DB / User creation completed!"
echo "################################################################"
echo ""
echo " >> Host : ${DB_HOST}"
echo " >> Database : ${DB_NAME}"
echo " >> User : ${DB_USER}"
echo " >> Pass : ${DB_PASS}"
echo ""
echo "################################################################"
_printPoweredBy
}
################################################################################
# Main
################################################################################
export LC_CTYPE=C
export LANG=C
DEBUG=0 # 1|0
_debug set -x
VERSION="0.1.0"
BIN_MYSQL=$(which mysql)
DB_HOST='localhost'
DB_NAME=
DB_USER=
DB_PASS=$(generatePassword)
function main()
{
[[ $# -lt 1 ]] && _printUsage
_success "Processing arguments..."
processArgs "$#"
_success "Done!"
_success "Creating MySQL db and user..."
createMysqlDbUser
_success "Done!"
printSuccessMessage
exit 0
}
main "$#"
_debug set +x
Usage:
./mysql-create-db-user.sh --host=localhost --database=test-db --user=test-user
For more details:
http://www.blog.magepsycho.com/bash-script-create-mysql-database-user-optional-password/
mysql -uroot -p$*PASSWORD_OF_MYSQL_ROOT_USER* -e "CREATE DATABASE $MAINDB"
mysql -uroot -p$*PASSWORD_OF_MYSQL_ROOT_USER* -e "GRANT ALL PRIVILEGES ON $MAINDB.* TO $MAINDB#localhost IDENTIFIED BY '$PASSWDDB!'"
This will work for you.
You could use heredoc to improve the lisibility of your SQL:
# create random password
PASSWDDB="$(openssl rand -base64 12)"
# replace "-" with "_" for database username
MAINDB=${USER_NAME//[^a-zA-Z0-9]/_}
mysql -u root -p<<MYSQL_SCRIPT
CREATE DATABASE $MAINDB;
CREATE DATABASE ${MAINDB} /*\!40100 DEFAULT CHARACTER SET utf8 */;
CREATE USER ${MAINDB}#localhost IDENTIFIED BY '${PASSWDDB}';
GRANT ALL PRIVILEGES ON ${MAINDB}.* TO '${MAINDB}'#'localhost';
FLUSH PRIVILEGES;
MYSQL_SCRIPT

Why powershell doesn't update database?

I have the following Powershell script:
$a = Get-Content C:\users\diana\desktop\names.txt
snmpwalk -v 2c -c root $a .1.3.6.1.2.1.25.3.3.1.2 > c:\users\diana\desktop\cpu.txt
snmpwalk -v 2c -c root $a .1.3.6.1.2.1.25.5.1.1.2 > c:\users\diana\desktop\ramvid.txt
snmpwalk -v 2c -c root $a .1.3.6.1.2.1.25.2.2 > c:\users\diana\desktop\ram.txt
get-content C:\users\diana\desktop\ramvid.txt | %{ [int]$used+=$_.split(' ')[3]; } ; echo $used > C:\users\diana\desktop\naujas.txt
get-content C:\users\diana\desktop\ram.txt | %{ [int]$total=$_.split(' ')[3]; } ; echo $total > C:\users\diana\desktop\ramfiltruotas.txt
[decimal]$b=($used*100)/$total
[math]::floor($b) > C:\users\diana\desktop\naujas2.txt
get-content C:\users\diana\desktop\cpu.txt | %{ [int]$array=$_.split(' ')[3]; }
$c=($array | Measure-Object -Average).average
echo $c > C:\users\diana\desktop\naujas3.txt
[void][system.reflection.Assembly]::LoadWithPartialName(“MySQL.Data”)
$myconnection = New-Object MySql.Data.MySqlClient.MySqlConnection
$myconnection.ConnectionString = "database=db;server=localhost;Persist Security Info=false;user id=root;pwd= "
$myconnection.Open()
$command = $myconnection.CreateCommand()
$command.CommandText = "UPDATE db.server SET (cpu='$c',ram='$b') WHERE server_name like '192.168.95.139'";
$myconnection.Close()
The upper part of the code works great, but when it comes to MySQL nothing happens. Not a single error, nothing, the table doesn't update.
Can someone point me where is the problem here?
Looks like you're not executing the command, perhaps:
$command.ExecuteNonQuery()
I think you should also dispose:
$command.Dispose()

dump all mysql tables into separate files automatically?

I'd like to get dumps of each mysql table into separate files. The manual indicates that the syntax for this is
mysqldump [options] db_name [tbl_name ...]
Which indicates that you know the table names before hand. I could set up the script that knows each table name now, but say I add a new table down the road and forget to update the dump script. Then I'm missing dumps for one or more table.
Is there a way to automagically dump each existing table into a separate file? Or am I going to have to do some script-fu; query the database, get all the table names, and dump them by name.
If I go the script-fu route, what scripting langauges can access a mysql database?
Here's a script that dumps table data as SQL commands into separate, compressed files. It does not require being on the MySQL server host, doesn't hard-code the password in the script, and is just for a specific db, not all db's on the server:
#!/bin/bash
# dump-tables-mysql.sh
# Descr: Dump MySQL table data into separate SQL files for a specified database.
# Usage: Run without args for usage info.
# Author: #Trutane
# Ref: http://stackoverflow.com/q/3669121/138325
# Notes:
# * Script will prompt for password for db access.
# * Output files are compressed and saved in the current working dir, unless DIR is
# specified on command-line.
[ $# -lt 3 ] && echo "Usage: $(basename $0) <DB_HOST> <DB_USER> <DB_NAME> [<DIR>]" && exit 1
DB_host=$1
DB_user=$2
DB=$3
DIR=$4
[ -n "$DIR" ] || DIR=.
test -d $DIR || mkdir -p $DIR
echo -n "DB password: "
read -s DB_pass
echo
echo "Dumping tables into separate SQL command files for database '$DB' into dir=$DIR"
tbl_count=0
for t in $(mysql -NBA -h $DB_host -u $DB_user -p$DB_pass -D $DB -e 'show tables')
do
echo "DUMPING TABLE: $DB.$t"
mysqldump -h $DB_host -u $DB_user -p$DB_pass $DB $t | gzip > $DIR/$DB.$t.sql.gz
tbl_count=$(( tbl_count + 1 ))
done
echo "$tbl_count tables dumped from database '$DB' into dir=$DIR"
The mysqldump command line program does this for you - although the docs are very unclear about this.
One thing to note is that ~/output/dir has to be writable by the user that owns mysqld. On Mac OS X:
sudo chown -R _mysqld:_mysqld ~/output/dir
mysqldump --user=dbuser --password --tab=~/output/dir dbname
After running the above, you will have one tablename.sql file containing each table's schema (create table statement) and tablename.txt file containing the data.
If you want a dump with schema only, add the --no-data flag:
mysqldump --user=dbuser --password --no-data --tab=~/output/dir dbname
You can accomplish this by:
Get the list of databases in mysql
dump each database with mysqldump
# Optional variables for a backup script
MYSQL_USER="root"
MYSQL_PASS="something"
BACKUP_DIR=/srv/backup/$(date +%Y-%m-%dT%H_%M_%S);
test -d "$BACKUP_DIR" || mkdir -p "$BACKUP_DIR"
# Get the database list, exclude information_schema
for db in $(mysql -B -s -u $MYSQL_USER --password=$MYSQL_PASS -e 'show databases' | grep -v information_schema)
do
# dump each database in a separate file
mysqldump -u $MYSQL_USER --password=$MYSQL_PASS "$db" | gzip > "$BACKUP_DIR/$db.sql.gz"
done
Here is the corresponding import.
#!/bin/bash
# import-files-mysql.sh
# Descr: Import separate SQL files for a specified database.
# Usage: Run without args for usage info.
# Author: Will Rubel
# Notes:
# * Script will prompt for password for db access.
[ $# -lt 3 ] && echo "Usage: $(basename $0) <DB_HOST> <DB_USER> <DB_NAME> [<DIR>]" && exit 1
DB_host=$1
DB_user=$2
DB=$3
DIR=$4
DIR=$DIR/*
echo -n "DB password: "
read -s DB_pass
echo
echo "Importing separate SQL command files for database '$DB' into '$DB'"
file_count=0
for f in $DIR
do
echo "IMPORTING FILE: $f"
gunzip -c $f | mysql -h $DB_host -u $DB_user -p$DB_pass $DB
(( file_count++ ))
done
echo "$file_count files importing to database '$DB'"
#!/bin/bash
for i in $(mysql -uUser -pPASSWORD DATABASE -e "show tables;"|grep -v Tables_in_);do mysqldump -uUSER -pPASSWORD DATABASE $i > /backup/dir/$i".sql";done
tar -cjf "backup_mysql_"$(date +'%Y%m%d')".tar.bz2" /backup/dir/*.sql
I have had recently the need to backup a big database (more than 250GB uncompressed dump file) and I found the answers to this question really helpful.
I started using #Trutane approach and it worked like a charm. But I was concerned about dumping tables in different mysql sessions because that could, in some moment, drive to a non-consistent backup.
After some research and testing, I have developed a different solution based on gawk. The basic idea is creating a dump of the whole database using mysqldump with --single-transaction=true and then process the output with gawk to produce a different file for every table.
So I can call:
mysqldump --single-transaction=true -u DBUSERNAME -p DBNAME | \
gawk -v 'database=DBNAME' -f 'backup.awk' -
And it produces, in current folder, a bunch of $database.$table.sql files with the schema of every table and $database.$table.sql.gz files with the content of every table. Thanks to the param --single-transaction=true, all the dump happens in a single transaction and data consistency is ensured.
The content of backup.awk is:
# Split mysqldump output in different files, two per table:
# * First file is named $database.$table.sql and it contains the table schema
# * Second file is named $database.$table.sql.gz and it contains the table data
# The 'database' variable is expected to be provided in command-line
BEGIN {
insert=0
filename=sprintf("%s.header.sql", database);
}
# A line starting with "INSERT INTO" activates inserting mode
/INSERT INTO/ { insert=1 }
# A line containing "-- Table structure for table `name-of-table`" finishes inserting mode
# It is also used to detect table name and change file names accordingly
match($0, /-- Table structure for table `(.*)`/, m) {
insert=0;
table=m[1];
filename=sprintf("%s.%s.sql", database, table);
print sprintf("Dumping table %s\n", table);
}
# If in inserting mode, line is piped to a gzipped file,
# if it is not, it is redirected to an uncompressed schema file
{
if (insert == 1) {
output = sprintf("gzip > %s.gz", filename);
print | output
} else {
print > filename;
}
}
It looks everybody here forgot of autocommit=0;SET unique_checks=0;SET foreign_key_checks=0; that is suppose to speed up the import process ...
#!/bin/bash
MYSQL_USER="USER"
MYSQL_PASS="PASS"
if [ -z "$1" ]
then
echo "Dumping all DB ... in separate files"
for I in $(mysql -u $MYSQL_USER --password=$MYSQL_PASS -e 'show databases' -s --skip-column-names);
do
echo "SET autocommit=0;SET unique_checks=0;SET foreign_key_checks=0;" > "$I.sql"
mysqldump -u $MYSQL_USER --password=$MYSQL_PASS $I >> "$I.sql";
echo "SET autocommit=1;SET unique_checks=1;SET foreign_key_checks=1;commit;" >> "$I.sql"
gzip "$I.sql"
done
echo "END."
else
echo "Dumping $1 ..."
echo "SET autocommit=0;SET unique_checks=0;SET foreign_key_checks=0;" > "$1.sql"
mysqldump -u $MYSQL_USER --password=$MYSQL_PASS $1 >> "$1.sql";
echo "SET autocommit=1;SET unique_checks=1;SET foreign_key_checks=1;commit;" >> "$1.sql"
gzip "$1.sql"
fi
If You want to dump all tables from all databases just combine Elias Torres Arroyo's and Trutane's answer:
And if You don't want to give Your password on terminal, just store Your password in an extra config file (chmod 0600)- see Mysqldump launched by cron and password security
#!/bin/bash
# this file
# a) gets all databases from mysql
# b) gets all tables from all databases in a)
# c) creates subfolders for every database in a)
# d) dumps every table from b) in a single file
# this is a mixture of scripts from Trutane (http://stackoverflow.com/q/3669121/138325)
# and Elias Torres Arroyo (https://stackoverflow.com/a/14711298/8398149)
# usage:
# sk-db.bash parameters
# where pararmeters are:
# d "dbs to leave"
# t " tables to leave"
# u "user who connects to database"
# h "db host"
# f "/backup/folder"
user='root'
host='localhost'
backup_folder=''
leave_dbs=(information_schema mysql)
leave_tables=()
while getopts ":d:t:u:h:f:" opt; do
case $opt in
d) leave_dbs=( $OPTARG )
;;
t) leave_tables=( $OPTARG )
;;
u) user=$OPTARG
;;
h) host=$OPTARG
;;
f) backup_folder=$OPTARG
;;
\?) echo "Invalid option -$OPTARG" >&2
;;
esac
done
echo '****************************************'
echo "Database Backup with these options"
echo "Host $host"
echo "User $user"
echo "Backup in $backup_folder"
echo '----------------------------------------'
echo "Databases to emit:"
printf "%s\n" "${leave_dbs[#]}"
echo '----------------------------------------'
echo "Tables to emit:"
printf "%s\n" "${leave_tables[#]}"
echo '----------------------------------------'
BACKUP_DIR=$backup_folder/$(date +%Y-%m-%dT%H_%M_%S);
CONFIG_FILE=/root/db-config.cnf
function contains() {
local n=$#
local value=${!n}
for ((i=1;i < $#;i++)) {
if [ "${!i}" == "${value}" ]; then
echo "y"
return 0
fi
}
echo "n"
return 1
}
test -d "$BACKUP_DIR" || mkdir -p "$BACKUP_DIR"
# Get the database list, exclude information_schema
database_count=0
tbl_count=0
for db in $(mysql --defaults-extra-file=$CONFIG_FILE -B -s -u $user -e 'show databases' )
do
if [ $(contains "${leave_dbs[#]}" "$db") == "y" ]; then
echo "leave database $db as requested"
else
# dump each database in a separate file
(( database_count++ ))
DIR=$BACKUP_DIR/$db
[ -n "$DIR" ] || DIR=.
test -d $DIR || mkdir -p $DIR
echo
echo "Dumping tables into separate SQL command files for database '$db' into dir=$DIR"
for t in $(mysql --defaults-extra-file=$CONFIG_FILE -NBA -h $host -u $user -D $db -e 'show tables')
do
if [ $(contains "${leave_tables[#]}" "$db.$t") == "y" ]; then
echo "leave table $db.$t as requested"
else
echo "DUMPING TABLE: $db.$t"
# mysqldump --defaults-extra-file=$CONFIG_FILE -h $host -u $user $db $t > $DIR/$db.$t.sql
tbl_count=$(( tbl_count + 1 ))
fi
done
echo "Database $db is finished"
echo '----------------------------------------'
fi
done
echo '----------------------------------------'
echo "Backup completed"
echo '**********************************************'
And also, this helped:
Check if bash array contains value
arrays in bash
named arguments in script
I'm not bash master, but I'd just do it with a bash script. Without hitting MySQL, with knowledge of the data directory and database name, you could just scan for all .frm files (one for every table in that db/directory) for a list of tables.
I'm sure there are ways to make it slicker and accept arguments or whatnot, but this worked well for me.
tables_in_a_db_to_sql.sh
#!/bin/bash
database="this_is_my_database"
datadir="/var/lib/mysql/"
datadir_escaped="\/var\/lib\/mysql\/"
all_tables=($(ls $datadir$database/*.frm | sed s/"$datadir_escaped$database\/"/""/g | sed s/.frm//g))
for t in "${all_tables[#]}"; do
outfile=$database.$t.sql
echo "-- backing up $t to $outfile"
echo "mysqldump [options] $database $t > $outfile"
# mysqldump [options] $database $t > $outfile
done
Fill in the [options] and desired outfile convention as you need, and uncomment the last mysqldump line.
For Windows Servers, you can use a batch file like so:
set year=%DATE:~10,4%
set day=%DATE:~7,2%
set mnt=%DATE:~4,2%
set hr=%TIME:~0,2%
set min=%TIME:~3,2%
IF %day% LSS 10 SET day=0%day:~1,1%
IF %mnt% LSS 10 SET mnt=0%mnt:~1,1%
IF %hr% LSS 10 SET hr=0%hr:~1,1%
IF %min% LSS 10 SET min=0%min:~1,1%
set backuptime=%year%-%mnt%-%day%-%hr%-%min%
set backupfldr=C:\inetpub\wwwroot\backupfiles\
set datafldr="C:\Program Files\MySQL\MySQL Server 5.5\data"
set zipper="C:\inetpub\wwwroot\backupfiles\zip\7za.exe"
set retaindays=21
:: Switch to the data directory to enumerate the folders
pushd %datafldr%
:: Get all table names and save them in a temp file
mysql --skip-column-names --user=root --password=mypassword mydatabasename -e "show tables" > tables.txt
:: Loop through all tables in temp file so that we can save one backup file per table
for /f "skip=3 delims=|" %%i in (tables.txt) do (
set tablename = %%i
mysqldump --user=root --password=mypassword mydatabasename %%i > "%backupfldr%mydatabasename.%backuptime%.%%i.sql"
)
del tables.txt
:: Zip all files ending in .sql in the folder
%zipper% a -tzip "%backupfldr%backup.mydatabasename.%backuptime%.zip" "%backupfldr%*.sql"
echo "Deleting all the files ending in .sql only"
del "%backupfldr%*.sql"
echo "Deleting zip files older than 21 days now"
Forfiles /p %backupfldr% /m *.zip /d -%retaindays% /c "cmd /c del /q #path"
Then schedule it using Windows Task Scheduler.
Also, if you want to exclude certain tables in your backup, note that you can use a where clause on the "show tables" statement, but the column name depends on your database name.
So for example, if your database name is "blah" then your column name in the "show tables" result set will be "tables_in_blah". Which means you could add a where clause something similar to this:
show tables where tables_in_blah <> 'badtable'
or
show tables where tables_in_blah like '%goodtable%'
They complete the route where they would store the backups. East
case we are creating one per day of the week, so we have 7 days of backup and they are recycled.
check how many databases it has and then how many tables each database has. and create a file named by db.tablename.sql
which can then be restored.
regards
#!/bin/bash
USER="root"
MYSQL_PASSWORD="password"
RUTA=/hdd/backup/mysql
diasemana=$(date +\%w)
mkdir -m 7777 $RUTA
mkdir -m 7777 $RUTA/infodb
mkdir -m 7777 $RUTA/$diasemana
mysql -u$USER -p$MYSQL_PASSWORD -e "SHOW DATABASES where \`Database\` <> 'information_schema' and \`Database\` <> 'mysql' and \`Database\` <> 'sys' and \`Database\` <> 'performance_schema';" -N > $RUTA/infodb/db.txt;
for i in $(cat $RUTA/infodb/db.txt);
do
mysql -u$USER -p$MYSQL_PASSWORD -e "USE $i;show tables;" -N >$RUTA/infodb/$i.txt;
for j in $(cat $RUTA/infodb/$i.txt);
do
mysqldump -u$USER -p$MYSQL_PASSWORD $i $j > $RUTA/$diasemana/$i"_"$j".sql";
echo $RUTA/$diasemana/$i"_"$j".sql"
done
done
See the following article by Pauli Marcus:
Howto split a SQL database dump into table-wise files
Splitting a sql file containing a whole database into per-table files
is quite easy: Grep the .sql for any occurence of DROP TABLE. Generate
the file name from the table name that is included in the DROP TABLE
statement. Echo the output to a file. Here is a little script that
expects a .sql file as input:
#!/bin/bash
file=$1 # the input file
directory="$file-splitted" # the output directory
output="$directory/header" # the first file containing the header
GREP="DROP TABLE" # what we are looking for
mkdir $directory # create the output directory
while read line
do
# if the current line contains the wanted statement
if [ $(echo "$line" | grep -c "$GREP") == "1" ]
then
# extract the file name
myfile=$(echo $line | awk '{print $5}' | sed -e 's/`//g' -e 's/;//g')
# set the new file name
output="$directory/$myfile"
fi
echo "$line" >> $output # write to file
done < $file