I am using MySQL on centos7. I have 50 databases Like database1, database2...., database50.
How can I set a cronjob for take a dump every day of all database same as database name [ Like database1.sql, database2.sql .... database50.sql ] using single command or script.
Please provide some adequate solution that will be appreciated.
Thanks.
Convert the current date to an integer number of days since some starting date.
Take that modulo 50. This gives you 0 .. 49.
Add 1 and concatenate. Now you have database1 .. database50. Put that in the shell variable db
mysqldump ... $db >$db.sql
I am using this script
#! /bin/bash
# MySQL database backup (databases in separate files) with daily, weekly and monthly rotation
# Sebastian Flippence (http://seb.flippence.net) originally based on code from: Ameir Abdeldayem (http://www.ameir.net)
# You are free to modify and distribute this code,
# so long as you keep the authors name and URL in it.
# Modified by IVO GELOV
# How many backups do you want to keep?
MAX_DAYS=5
# Date format that is appended to filename
DATE=`date +'%Y-%m-%d'`
DATSTR=`date '+%Y%m%d' -d "-$MAX_DAYS days"`
# MySQL server's name
SERVER=""
# Directory to backup to
BACKDIR="/var/db_arhiv/mysql"
#----------------------MySQL Settings--------------------#
# MySQL server's hostname or IP address
HOST="localhost"
# MySQL username
USER="user"
# MySQL password
PASS="password"
# List all of the MySQL databases that you want to backup,
# each separated by a space. Or set the option below to backup all database
DBS="db1 db2"
# Set to 'y' if you want to backup all your databases. This will override
# the database selection above.
DUMPALL="y"
# Custom path to system commands (enable these if you want use a different
# location for PHP and MySQL or if you are having problems running this script)
MYSQL="/usr/local/mysql/bin/mysql"
MYSQLDUMP="/usr/local/mysql/bin/mysqldump"
function checkMysqlUp() {
$MYSQL -N -h $HOST --user=$USER --password=$PASS -e status > /dev/null
}
trap checkMysqlUp 0
function error() {
local PARENT_LINENO="$1"
local MESSAGE="$2"
local CODE="${3:-1}"
if [[ -n "$MESSAGE" ]] ; then
echo "Error on or near line ${PARENT_LINENO}: ${MESSAGE}; exiting with status ${CODE}"
else
echo "Error on or near line ${PARENT_LINENO}; exiting with status ${CODE}"
fi
exit "${CODE}"
}
trap 'error ${LINENO}' ERR
# Check backup directory exists
# if not, create it
if [ ! -e "$BACKDIR/$DATE" ]; then
mkdir -p "$BACKDIR/$DATE"
echo "Created backup directory (${BACKDIR}/${DATE})"
fi
if [ $DUMPALL = "y" ]; then
echo "Creating list of databases on: ${HOST}..."
$MYSQL -N -h $HOST --user=$USER --password=$PASS -e "show databases;" > ${BACKDIR}/dbs_on_${SERVER}.txt
# redefine list of databases to be backed up
DBS=`sed -e ':a;N;$!ba;s/\n/ /g' -e 's/Database //g' ${BACKDIR}/dbs_on_${SERVER}.txt`
fi
echo "Backing up MySQL databases..."
#cd ${LATEST}
for database in $DBS; do
if [ ${database} = "information_schema" ] || [ ${database} = "performance_schema" ] || [ ${database} = "pinba" ]
then
continue
fi
echo "${database}..."
$MYSQLDUMP --host=$HOST --user=$USER --password=$PASS --default-character-set=utf8 --routines --triggers --lock-tables --disable-keys --force --single-transaction --allow-keywords --dump-date $database > ${BACKDIR}/${DATE}/${SERVER}$database.sql
done
if [ $DUMPALL = "y" ]; then
rm -f ${BACKDIR}/dbs_on_${SERVER}.txt
fi
# dump privileges
$MYSQL -N -h $HOST --user=$USER --password=$PASS --skip-column-names -A -e "SELECT CONCAT('SHOW GRANTS FOR ''',user,'''#''',host,''';') FROM mysql.user" | $MYSQL -N -h $HOST --user=$USER --password=$PASS --skip-column-names -A > ${BACKDIR}/${DATE}/${SERVER}_grants.sql
# delete older files
for x in `find ${BACKDIR}/20* -type d`
do
xd=`basename "${x//-/}"`
if [[ $xd < $DATSTR ]]
then
rm -rf "$x"
fi
done
echo "MySQL backup is complete"
Related
I'm running an MySQL server docker container using a docker-compose YAML file.
Here is how the file looks like:
version: '3.1'
services:
db:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
volumes:
- ./mysql-dump/samples:/docker-entrypoint-initdb.d
environment:
MYSQL_ROOT_PASSWORD: example
MYSQL_DATABASE: db_example
adminer:
image: adminer
restart: always
ports:
- 8080:8080
In the db service, the volumne is set to ./mysql-dump/samples:/docker-entrypoint-initdb.d this takes .sql files from ./mysql-dump/sample to inject them to the database.
In my case I have two files file2.sql for the sql schema of the database, and file1.sql for the data.
Since the file appear to be injected in order, I get a NO SUCH TABLE ERROR, surely because the schema is injected last (because it's name is file2.sql)
Is there a way to reverse the order of the injection beside changing the names of the files?
If you go through the documentation of mysql Dockerhub it clearly mentioned that it will dump file in alphabetical order.
When a container is started for the first time, a new database with
the specified name will be created and initialized with the provided
configuration variables. Furthermore, it will execute files with
extensions .sh, .sql and .sql.gz that are found in
/docker-entrypoint-initdb.d. Files will be executed in alphabetical
order. You can easily populate your mysql services by mounting a SQL
dump into that directory and provide custom images with contributed
data. SQL files will be imported by default to the database specified
by the MYSQL_DATABASE variable.
You need to replace file name, suppose db.sql and table.sql so it will first dump db.sql then table.sql
Updated:
To reverse the order of MySQL dump, you have to modify the docker file and entry point.
FROM mysql:8
#From mysql
COPY docker-entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/docker-entrypoint.sh
ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
EXPOSE 3306 33060
CMD ["mysqld"]
ENTRYPOINT:
#!/bin/bash
set -x
set -eo pipefail
shopt -s nullglob
# if command starts with an option, prepend mysqld
if [ "${1:0:1}" = '-' ]; then
set -- mysqld "$#"
fi
# skip setup if they want an option that stops mysqld
wantHelp=
for arg; do
case "$arg" in
-'?'|--help|--print-defaults|-V|--version)
wantHelp=1
break
;;
esac
done
# usage: file_env VAR [DEFAULT]
# ie: file_env 'XYZ_DB_PASSWORD' 'example'
# (will allow for "$XYZ_DB_PASSWORD_FILE" to fill in the value of
# "$XYZ_DB_PASSWORD" from a file, especially for Docker's secrets feature)
file_env() {
local var="$1"
local fileVar="${var}_FILE"
local def="${2:-}"
if [ "${!var:-}" ] && [ "${!fileVar:-}" ]; then
echo >&2 "error: both $var and $fileVar are set (but are exclusive)"
exit 1
fi
local val="$def"
if [ "${!var:-}" ]; then
val="${!var}"
elif [ "${!fileVar:-}" ]; then
val="$(< "${!fileVar}")"
fi
export "$var"="$val"
unset "$fileVar"
}
# usage: process_init_file FILENAME MYSQLCOMMAND...
# ie: process_init_file foo.sh mysql -uroot
# (process a single initializer file, based on its extension. we define this
# function here, so that initializer scripts (*.sh) can use the same logic,
ls -r
process_init_file() {
local f="$1"; shift
local mysql=( "$#" )
case "$f" in
*.sh) echo "$0: running $f"; . "$f" ;;
*.sql) echo "$0: running $f"; "${mysql[#]}" < "$f"; echo ;;
*.sql.gz) echo "$0: running $f"; gunzip -c "$f" | "${mysql[#]}"; echo ;;
*) echo "$0: ignoring $f" ;;
esac
echo
}
_check_config() {
toRun=( "$#" --verbose --help )
if ! errors="$("${toRun[#]}" 2>&1 >/dev/null)"; then
cat >&2 <<-EOM
ERROR: mysqld failed while attempting to check config
command was: "${toRun[*]}"
$errors
EOM
exit 1
fi
}
# Fetch value from server config
# We use mysqld --verbose --help instead of my_print_defaults because the
# latter only show values present in config files, and not server defaults
_get_config() {
local conf="$1"; shift
"$#" --verbose --help --log-bin-index="$(mktemp -u)" 2>/dev/null \
| awk '$1 == "'"$conf"'" && /^[^ \t]/ { sub(/^[^ \t]+[ \t]+/, ""); print; exit }'
# match "datadir /some/path with/spaces in/it here" but not "--xyz=abc\n datadir (xyz)"
}
# allow the container to be started with `--user`
if [ "$1" = 'mysqld' -a -z "$wantHelp" -a "$(id -u)" = '0' ]; then
_check_config "$#"
DATADIR="$(_get_config 'datadir' "$#")"
mkdir -p "$DATADIR"
chown -R mysql:mysql "$DATADIR"
exec gosu mysql "$BASH_SOURCE" "$#"
fi
if [ "$1" = 'mysqld' -a -z "$wantHelp" ]; then
# still need to check config, container may have started with --user
_check_config "$#"
# Get config
DATADIR="$(_get_config 'datadir' "$#")"
if [ ! -d "$DATADIR/mysql" ]; then
file_env 'MYSQL_ROOT_PASSWORD'
if [ -z "$MYSQL_ROOT_PASSWORD" -a -z "$MYSQL_ALLOW_EMPTY_PASSWORD" -a -z "$MYSQL_RANDOM_ROOT_PASSWORD" ]; then
echo >&2 'error: database is uninitialized and password option is not specified '
echo >&2 ' You need to specify one of MYSQL_ROOT_PASSWORD, MYSQL_ALLOW_EMPTY_PASSWORD and MYSQL_RANDOM_ROOT_PASSWORD'
exit 1
fi
mkdir -p "$DATADIR"
echo 'Initializing database'
"$#" --initialize-insecure
echo 'Database initialized'
if command -v mysql_ssl_rsa_setup > /dev/null && [ ! -e "$DATADIR/server-key.pem" ]; then
# https://github.com/mysql/mysql-server/blob/23032807537d8dd8ee4ec1c4d40f0633cd4e12f9/packaging/deb-in/extra/mysql-systemd-start#L81-L84
echo 'Initializing certificates'
mysql_ssl_rsa_setup --datadir="$DATADIR"
echo 'Certificates initialized'
fi
SOCKET="$(_get_config 'socket' "$#")"
"$#" --skip-networking --socket="${SOCKET}" &
pid="$!"
mysql=( mysql --protocol=socket -uroot -hlocalhost --socket="${SOCKET}" )
for i in {30..0}; do
if echo 'SELECT 1' | "${mysql[#]}" &> /dev/null; then
break
fi
echo 'MySQL init process in progress...'
sleep 1
done
if [ "$i" = 0 ]; then
echo >&2 'MySQL init process failed.'
exit 1
fi
if [ -z "$MYSQL_INITDB_SKIP_TZINFO" ]; then
# sed is for https://bugs.mysql.com/bug.php?id=20545
mysql_tzinfo_to_sql /usr/share/zoneinfo | sed 's/Local time zone must be set--see zic manual page/FCTY/' | "${mysql[#]}" mysql
fi
if [ ! -z "$MYSQL_RANDOM_ROOT_PASSWORD" ]; then
export MYSQL_ROOT_PASSWORD="$(pwgen -1 32)"
echo "GENERATED ROOT PASSWORD: $MYSQL_ROOT_PASSWORD"
fi
rootCreate=
# default root to listen for connections from anywhere
file_env 'MYSQL_ROOT_HOST' '%'
if [ ! -z "$MYSQL_ROOT_HOST" -a "$MYSQL_ROOT_HOST" != 'localhost' ]; then
# no, we don't care if read finds a terminating character in this heredoc
# https://unix.stackexchange.com/questions/265149/why-is-set-o-errexit-breaking-this-read-heredoc-expression/265151#265151
read -r -d '' rootCreate <<-EOSQL || true
CREATE USER 'root'#'${MYSQL_ROOT_HOST}' IDENTIFIED BY '${MYSQL_ROOT_PASSWORD}' ;
GRANT ALL ON *.* TO 'root'#'${MYSQL_ROOT_HOST}' WITH GRANT OPTION ;
EOSQL
fi
"${mysql[#]}" <<-EOSQL
-- What's done in this file shouldn't be replicated
-- or products like mysql-fabric won't work
SET ##SESSION.SQL_LOG_BIN=0;
ALTER USER 'root'#'localhost' IDENTIFIED BY '${MYSQL_ROOT_PASSWORD}' ;
GRANT ALL ON *.* TO 'root'#'localhost' WITH GRANT OPTION ;
${rootCreate}
DROP DATABASE IF EXISTS test ;
FLUSH PRIVILEGES ;
EOSQL
if [ ! -z "$MYSQL_ROOT_PASSWORD" ]; then
mysql+=( -p"${MYSQL_ROOT_PASSWORD}" )
fi
file_env 'MYSQL_DATABASE'
if [ "$MYSQL_DATABASE" ]; then
echo "CREATE DATABASE IF NOT EXISTS \`$MYSQL_DATABASE\` ;" | "${mysql[#]}"
mysql+=( "$MYSQL_DATABASE" )
fi
file_env 'MYSQL_USER'
file_env 'MYSQL_PASSWORD'
if [ "$MYSQL_USER" -a "$MYSQL_PASSWORD" ]; then
echo "CREATE USER '$MYSQL_USER'#'%' IDENTIFIED BY '$MYSQL_PASSWORD' ;" | "${mysql[#]}"
if [ "$MYSQL_DATABASE" ]; then
echo "GRANT ALL ON \`$MYSQL_DATABASE\`.* TO '$MYSQL_USER'#'%' ;" | "${mysql[#]}"
fi
echo 'FLUSH PRIVILEGES ;' | "${mysql[#]}"
fi
echo
ls -r /docker-entrypoint-initdb.d/ > /dev/null
for f in $(ls -r /docker-entrypoint-initdb.d/*); do
process_init_file "$f" "${mysql[#]}"
done
if [ ! -z "$MYSQL_ONETIME_PASSWORD" ]; then
"${mysql[#]}" <<-EOSQL
ALTER USER 'root'#'%' PASSWORD EXPIRE;
EOSQL
fi
if ! kill -s TERM "$pid" || ! wait "$pid"; then
echo >&2 'MySQL init process failed.'
exit 1
fi
echo
echo 'MySQL init process done. Ready for start up.'
echo
fi
fi
exec "$#"
If you run the container, You will see the file is in processing reverse order
I can backup all my db in separate files using the following script:
#!/bin/bash
MYSQL_USER="USER"
MYSQL_PASS="PASSWORD"
if [ -z "$1" ]
then
echo "Dumping all DB ..."
for I in $(mysql -u $MYSQL_USER --password=$MYSQL_PASS -e 'show databases' -s --skip-column-names);
do
echo "SET autocommit=0;SET unique_checks=0;SET foreign_key_checks=0;" > "$I.sql"
mysqldump -u $MYSQL_USER --password=$MYSQL_PASS $I >> "$I.sql";
echo "SET autocommit=1;SET unique_checks=1;SET foreign_key_checks=1;commit;" >> "$I.sql"
gzip "$I.sql"
done
echo "END."
else
echo "Dumping $1 ..."
echo "SET autocommit=0;SET unique_checks=0;SET foreign_key_checks=0;" > "$1.sql"
mysqldump -u $MYSQL_USER --password=$MYSQL_PASS $1 >> "$1.sql";
echo "SET autocommit=1;SET unique_checks=1;SET foreign_key_checks=1;commit;" >> "$1.sql"
gzip "$1.sql"
fi
I'm looking for the reverse command:
create a DB with the same file name
uncompress/import the sql.gz backup
You can use this script:
for f in *.sql.gz; do
db="${f%%.*}"
echo "creating database $db"
mysql -h localhost -u root -p mysql --password=passwd <<< "create database $db"
echo "restoring database $db"
gunzip "$f"
mysql -h localhost -u root -ppasswd "$db" < "$db.sql"
done
My bash script won't crate the Database. What am I doing wrong here?
Please have a look:
#!/bin/bash -x
set -x
function deebee() {
EXPECTED_ARGS=2
E_BADARGS=65
MYSQL=`which mysql`
Q1="CREATE DATABASE IF NOT EXISTS $1;"
Q2="GRANT USAGE ON *.* TO $2#localhost IDENTIFIED BY '$3';"
Q3="GRANT ALL PRIVILEGES ON $1.* TO $2#localhost;"
Q4="FLUSH PRIVILEGES;"
SQL="${Q1}${Q2}${Q3}${Q4}"
if [ $# -ne $EXPECTED_ARGS ]
then
echo "Usage: $0 dbname dbuser dbpass"
exit $E_BADARGS
fi
$MYSQL -uroot -p -e "$SQL"
}
deebee $1 $2 $3
I'm calling the script as I've put it in a function, but it just spits out the expected arguments telling me the syntax, i.e that I should type in the bashscrip name, dbname, dbuser dbpass, but obviously there' something wrong with the script of my login permissions or user so that I can't automate this... What's going on, I'd love to know!
Thanks!
What I would do :
creating a mysql config file with credentials (easier to automate now) :
cat ~/.my.cnf
[client]
host = localhost
user = root
password = xxx
And the script :
#!/bin/bash
set -x
deebee() {
EXPECTED_ARGS=3
E_BADARGS=65
MYSQL=$(type -p mysql)
if (($# != $EXPECTED_ARGS))
then
echo "Usage: $0 dbname dbuser dbpass"
exit $E_BADARGS
fi
$MYSQL <<EOF
CREATE DATABASE IF NOT EXISTS '$1';
GRANT USAGE ON *.* TO '$2'#'localhost' IDENTIFIED BY '$3';
GRANT ALL PRIVILEGES ON '$1'.* TO '$2'#'localhost';
FLUSH PRIVILEGES;
EOF
}
# TODO tests on input args
deebee "$1" "$2" "$3"
And like Denis said in comments, take care of what users can put as arguments to prevent sql injection.
You should add some tests on input args.
My database had lots of views and it was impossible to drop them one by one.
I would like to just drop them all because the database doesn't refresh structure changes of the tables in the view that select from them.
If you want to do this in the MySQL client, you can dynamically generate the DDL statements using information_schema, dump them to a SQL script, and then execute that script.
Example:
select concat('drop view ',table_schema,'.',table_name,';') as ddl
into outfile '/tmp/drop_all_views.sql'
from information_schema.views
where table_schema = 'your_schema';
\. /tmp/drop_all_views.sql
After searching on the web, I found a shell script to drop all tables here: http://www.cyberciti.biz/faq/how-do-i-empty-mysql-database/
Then, I changed that script to drop all views of the database.
This is the final result:
#!/bin/bash
PREFIX=""
SUFFIX=""
HOST="localhost"
PORT="3306"
while getopts p:s:h:P: OPCAO; do
case "${OPCAO}" in
p) PREFIX="${OPTARG}" ;;
s) SUFFIX="${OPTARG}" ;;
h) HOST="${OPTARG}" ;;
P) PORT="${OPTARG}" ;;
esac
done
shift $((OPTIND-1))
MUSER="$1"
MPASS="$2"
MDB="$3"
# Detect paths
MYSQL=$(which mysql)
AWK=$(which awk)
GREP=$(which grep)
if [ $# -eq 0 ]
then
echo "Usage: $0 [-h Host] [-P Port] [-p Prefix-View-Name] [-s Suffix-View-Name] {MySQL-User-Name} {MySQL-User-Password} {MySQL-Database-Name}"
echo "Drops all views from a MySQL"
exit 1
fi
TABLES=$($MYSQL -h $HOST -P $PORT -u $MUSER -p$MPASS $MDB -e "SELECT table_name FROM information_schema.views WHERE table_schema = '$MDB' AND table_name LIKE '$PREFIX%$SUFFIX';" | $AWK '{ print $1}')
for t in $TABLES
do
echo "Deleting $t view from $MDB database..."
$MYSQL -h $HOST -P $PORT -u $MUSER -p$MPASS $MDB -e "drop view $t"
done
This solution will work just for who uses Unix-like systems.
To call the script, I used:
./script.sh [-h host] [-P port] [-p prefixViewName] [-s suffixViewName] username password databaseName
EDIT: I improved the script to accept option parameters, like the host, port and also a prefix and a suffix to filter what views will be dropped.
I am using rsnapshot to backup our remote servers and I need to add MySQL backups.
rsnapshot calls a remote script that dumps the mysql DB's on the server. The problem is, the script does not exclude the information_schema database and therefore it dies )or appears to die and I have a second problem.)
I'm not sure how I would exclude the information_schema DB from this script:
### SETUP MYSQL LOGIN ###
MUSER='USER'
MPASS='PASSWORD'
MHOST="127.0.0.1"
### Set to 1 if you need to see progress while dumping dbs ###
VERBOSE=0
### Set bins path ###
GZIP=/bin/gzip
MYSQL=/usr/bin/mysql
MYSQLDUMP=/usr/bin/mysqldump
RM=/bin/rm
MKDIR=/bin/mkdir
MYSQLADMIN=/usr/bin/mysqladmin
GREP=/bin/grep
### Setup dump directory ###
BAKRSNROOT=/tmp/rsnapshot/mysql
#####################################
### ----[ No Editing below ]------###
#####################################
### Default time format ###
TIME_FORMAT='%H_%M_%S%P'
### Make a backup ###
backup_mysql_rsnapshot(){
local DBS="$($MYSQL -u $MUSER -h $MHOST -p$MPASS -Bse 'show databases')"
local db="";
[ ! -d $BAKRSNROOT ] && ${MKDIR} -p $BAKRSNROOT
${RM} -f $BAKRSNROOT/* >/dev/null 2>&1
[ $VERBOSE -eq 1 ] && echo "*** Dumping MySQL Database ***"
[ $VERBOSE -eq 1 ] && echo -n "Database> "
for db in $DBS
do
local tTime=$(date +"${TIME_FORMAT}")
local FILE="${BAKRSNROOT}/${db}.${tTime}.gz"
[ $VERBOSE -eq 1 ] && echo -n "$db.."
${MYSQLDUMP} -u ${MUSER} -h ${MHOST} -p${MPASS} $db | ${GZIP} -9 > $FILE
done
[ $VERBOSE -eq 1 ] && echo ""
[ $VERBOSE -eq 1 ] && echo "*** Backup done [ files wrote to $BAKRSNROOT] ***"
}
### Die on demand with message ###
die(){
echo "$#"
exit 999
}
### Make sure bins exists.. else die
verify_bins(){
[ ! -x $GZIP ] && die "File $GZIP does not exists. Make sure correct path is set in $0."
[ ! -x $MYSQL ] && die "File $MYSQL does not exists. Make sure correct path is set in $0."
[ ! -x $MYSQLDUMP ] && die "File $MYSQLDUMP does not exists. Make sure correct path is set in $0."
[ ! -x $RM ] && die "File $RM does not exists. Make sure correct path is set in $0."
[ ! -x $MKDIR ] && die "File $MKDIR does not exists. Make sure correct path is set in $0."
[ ! -x $MYSQLADMIN ] && die "File $MYSQLADMIN does not exists. Make sure correct path is set in $0."
[ ! -x $GREP ] && die "File $GREP does not exists. Make sure correct path is set in $0."
}
### Make sure we can connect to server ... else die
verify_mysql_connection(){
$MYSQLADMIN -u $MUSER -h $MHOST -p$MPASS ping | $GREP 'alive'>/dev/null
[ $? -eq 0 ] || die "Error: Cannot connect to MySQL Server. Make sure username and password are set correctly in $0"
}
### main ####
verify_bins
verify_mysql_connection
backup_mysql_rsnapshot
Look into mysqldump and its --all-databases parameter. Don't reinvent the wheel!
mysqldump does not dump theINFORMATION_SCHEMA database. If you name that database explicitly on the command line,mysqldump silently ignores it.
If you have a version of mysqldump that does really not exclude INFORMATION_SCHEMA and you need all DBs in seperate files, just exclude it from your loop:
for db in $DBS
do
if [ $db -ne "INFORMATION_SCHEMA" ] ; do
YOUR_STUFF()
done
dome
Replace the following line:
local DBS="$($MYSQL -u $MUSER -h $MHOST -p$MPASS -Bse 'show databases')"
with
local DBS="$($MYSQL -u $MUSER -h $MHOST -p$MPASS -Bse 'show databases' -s --skip-column-names|grep -vi information_schema)"