I am currently running a cronjob executing a bash-script for a mysqldump.
#!/bin/bash
#
# define variables
TIMESTAMP=$(date +%Y-%m-%d_%H:%M)
USER=mysqluser
PASSWORD='myPassword'
BACKUP_DIR=/backup
# create directory with timestamp
mkdir -p "$BACKUP_DIR/$TIMESTAMP"
# dump database
# compress file with GZip
mysqldump --opt --user=$USER --password=$PASSWORD --host='host3.mydbserver.com' db1 | gzip -9 > ${BACKUP_DIR}/$TIMESTAMP/backup-db1-$TIMESTAMP.sql.gz
mysqldump --opt --user=$USER --password=$PASSWORD --host='host7.mydbserver.com' db2 | gzip -9 > ${BACKUP_DIR}/$TIMESTAMP/backup-db2-$TIMESTAMP.sql.gz
# find directories older than 120 minutes and delete recursively
find $BACKUP_DIR/* -type d -mmin +120 -exec rm -r {} \;
The script seems to be working fine dumping the data into sql files in a timestamped directory. After executing the dump the script is to delete all folders older than 120 minutes, but I am getting an error code 1 from the cron service saying the directory of the script is nonexistent. The output for the find command is
find: '/backup/2020-05-22_13:30': No such file or directory
Any idea how I can avoid this error?
EDIT 1: Once every couple of times I'm not getting the error code, without any changes in the code (it is currently running every 5 minutes for test purposes). I'm getting a feeling maybe this is a timing thing?
EDIT 2: I'm a noob regarding all things Linux, scripting, bash etc. and I spent days getting as far as I did. So any downvoters please leave a comment to let me know what obvious thing I overlooked.
Try to add -depth to the find options:
find $BACKUP_DIR -depth -type d -mmin +120 -exec rm -r {} \;
Related
I have a bash script called copydata which does the following to do a MySQL dump of specific tables from our production MySQL server to a local file, and then push it into my local MySQL database.
#!/bin/sh
#set up tunnel
ssh -f -i ~/.ssh/ec2-eu-keypair.pem -o CompressionLevel=9 -o ExitOnForwardFailure=yes -L 3307:elr2.our-id.eu-west-1.rds.amazonaws.com:3306 username#example.com
echo "Dumping tables \"$#\" to /tmp/data.sql"
#dump tables to local file
mysqldump -u root -h 127.0.0.1 -pmypass -P 3307 live_db_name --extended-insert --single-transaction --default-character-set=utf8 --skip-set-charset $# > /tmp/data.sql
pv /tmp/data.sql | mysql -u root local_db_name --default-character-set=utf8 --binary-mode --force
So, it is called like copydata table1 table2
It works, but the mysqldump part can take a very long time, and it would be nice to have some visual feedback on progress. One thing which occurred to me is that I could show the size of /tmp/data.sql while the dump is in progress - if I just keep doing the following, in a seperate tab, for example, I can see it going up at the rate of approx 2mb per second:
ls -lh /tmp/data.sql
Can I add the above command, or something similar, to the above script so that I can see the file size updating while i'm waiting for the mysqldump line to complete?
Thanks to #YuriLachin in the comments, I did the following:
added a & to the mysqldump line, so it becomes asynchronous, ie the script carries on to the next line while the mysqldump continues in the background
added this line, to repeatedly call ls -lh on the local file:
pid=$!; while [ -d "/proc/$pid" ] ; do echo -n "$(ls -lh /tmp/data.sql)"\\r; sleep 1; done
Lets break that down, to aid my own learning as much as anything else:
#get the process id of that last backgrounded task (the mysqldump) so we
#can tell when it's finished running
pid=$!
#while it *is* still running
while [ -d "/proc/$pid" ] ; do
#get the size of the file, with ls, but do it inside an echo command.
#Wrapping it like this allows us to use the `-n` option which means "omit newline",
#or don't go onto the next line. Then, at the end, do \\r which is a carriage return,
#meaning 'go back to the start of the current line', so the next line will
#overwrite the first one.
#Now it updates in place rather than spewing out loads of lines.
echo -n "$(ls -lh /tmp/data.sql)"\\r
#then do nothing for 1 second, to avoid wasting cpu time.
sleep 1
done
I just found Vigrant, and I am trying to make a script that set up fully my development environment. I am using Ubuntu 14.04.01 like a server. With scrip now I install LAMP, set up a password for MySQL, and change the LAMP server public directory. Now I have a directory with many .sql dump file. I want to find the newest one and import it to my database. I have a problem with importing my last created .sql dump of database. I found a command that find last created file in directory.
find /vagrant/VagrantFiles/DB/ -type f -exec stat -c "%n" {} + | sort -r | head -n1
But when add mysql command to import it mysql --user=root --password=pass < {} and execute line like this:
find /vagrant/VagrantFiles/DB/ -type f -exec stat -c "%n" {} + | sort -r | head -n1 | mysql --user=root --password=pass < {}
I get the error in terminal -bash: {}: No such file or directory
How I can make this work?
Hi after two days of searching the solution I found it.
I just separate this command in two part. Firs find newest file and remember it to variable, then execute mysql command with this variable. My inspiration for this solution is this post. And here it is the solution:
last_dump=$(find /vagrant/VagrantFiles/DB/ -type f -exec stat -c "%n" {} + | sort -r | head -n1)
mysql --user=root --password=pass < $last_dump
After months of trying to get this to happen I found a shell script that will get the job done.
Heres the code I'm working with
#!/bin/bash
### MySQL Server Login Info ###
MUSER="root"
MPASS="MYSQL-ROOT-PASSWORD"
MHOST="localhost"
MYSQL="$(which mysql)"
MYSQLDUMP="$(which mysqldump)"
BAK="/backup/mysql"
GZIP="$(which gzip)"
### FTP SERVER Login info ###
FTPU="FTP-SERVER-USER-NAME"
FTPP="FTP-SERVER-PASSWORD"
FTPS="FTP-SERVER-IP-ADDRESS"
NOW=$(date +"%d-%m-%Y")
### See comments below ###
### [ ! -d $BAK ] && mkdir -p $BAK || /bin/rm -f $BAK/* ###
[ ! -d "$BAK" ] && mkdir -p "$BAK"
DBS="$($MYSQL -u $MUSER -h $MHOST -p$MPASS -Bse 'show databases')"
for db in $DBS
do
FILE=$BAK/$db.$NOW-$(date +"%T").gz
$MYSQLDUMP -u $MUSER -h $MHOST -p$MPASS $db | $GZIP -9 > $FILE
done
lftp -u $FTPU,$FTPP -e "mkdir /mysql/$NOW;cd /mysql/$NOW; mput /backup/mysql/*; quit" $FTPS
Everything is running great, however there are a few things I'd like to fix but am clueless when it comes to shell scripts. I'm not asking anyone to write it. Just some pointers. First of all the /backup/mysql directory on my server stacks the files everytime it backs up. Not to big of a deal. But after a year of nightly backups it might get a little full. So id like it to clear that directory after uploading. Also I don't want to overload my hosting service with files so id like it to clear the remote servers dir before uploading. Lastly I would like it to upload to a subdirectory on the remote server such as /mysql
Why reinvent the wheel? You can just use Debian's automysqlbackup package (should be available on Ubuntu as well).
As for cleaning old files the following command might be of help:
find /mysql -type f -mtime +16 -delete
Uploading to remote server can be done using scp(1) command;
To avoid password prompt read about SSH public key authentication
Take a look at Backup, it allows you to model your backup jobs using a Ruby DSL, very powerful.
Support multiple DBs and most popular online storages, and lots of cool features.
I have written a shell script which starts MySQL when its killed/terminated. I am running this shell script using a crontab.
My cron looks for the script file named mysql.sh under /root/mysql.sh
sh /root/mysql.sh
mysql.sh:
cd /root/validate-mysql-status
sh /root/validate-mysql-status/validate-mysql-status.sh
validate-mysql-status.sh:
# mysql root/admin username
MUSER="xxxx"
# mysql admin/root password
MPASS="xxxxxx"
# mysql server hostname
MHOST="localhost"
MSTART="/etc/init.d/mysql start"
# path mysqladmin
MADMIN="$(which mysqladmin)"
# see if MySQL server is alive or not
# 2&1 could be better but i would like to keep it simple
$MADMIN -h $MHOST -u $MUSER -p${MPASS} ping 2>/dev/null 1>/dev/null
if [ $? -ne 0 ]; then
# MySQL's status log file
MYSQL_STATUS_LOG=/root/validate-mysql-status/mysql-status.log
# If log file not exist, create a new file
if [ ! -f $MYSQL_STATUS_LOG ]; then
cat "Creating MySQL status log file.." > $MYSQL_STATUS_LOG
now="$(date)"
echo [$now] error : MySQL not running >> $MYSQL_STATUS_LOG
else
now="$(date)"
echo [$now] error : MySQL not running >> $MYSQL_STATUS_LOG
fi
# Restarting MySQL
/etc/init.d/mysql start
now1="$(date)"
echo [$now1] info : MySQL started >> $MYSQL_STATUS_LOG
cat $MYSQL_STATUS_LOG
fi
When I run the above mysql shell script manually using webmin's crontab, MySQL started successfully (when its killed).
However, when I schedule it using a cron job, MySQL doesn't starts. The logs are printed properly (it means my cron runs the scheduled script successfully, however MySQL is not restarting).
crontab -l displays:
* * * * * sh /root/mysql.sh
I found from URL's that we should give absolute path to restart MySQL through schedulers like cron. However, it haven't worked for me.
Can anyone please help me!
Thank You.
First, crontab normaly looks like this:
* * * * * /root/mysql.sh
So remove the surplus sh and put it at the beginning of the script - #!/bin/bash I suppose (why are you referring to sh instead of bash?) and don't forget to have an execute permission on the file (chmod +x /root/mysql.sh)
Second, running scripts within crontab is tricky, because the environment is different! You have to set it manually. We start with PATH: go to console and do echo $PATH, and then copy-paste the result into export PATH=<your path> to your cron script:
mysql.sh:
#!/bin/bash
export PATH=.:/bin:/usr/local/bin:/usr/bin:/opt/bin:/usr/games:./:/sbin:/usr/sbin:/usr/local/sbin
{
cd /root/validate-mysql-status
/root/validate-mysql-status/validate-mysql-status.sh
} >> OUT 2>> ERR
Note that I also redirected all the output to files so that you don't receive emails from cron.
Problem is how to know which other variables (besides PATH) matter. Try to go through set | less and try to figure out which variables might be important to set in the cron script too. If there are any MYSQL related variables, you must set them! You may also examine the cron script environment by putting set > cron.env to the cron script and then diff-ing it against console environment to look for significant differences.
I need a script that automatically makes a backup of a MySql Database. I know there are a lot of posts and scripts out there on this topic already but here is where mine differs.
The script needs to run on the machine hosting the MySql database (It is a linux machine).
The backups must be saved onto the same server that the database is on.
A backup needs to be made every 30 minutes.
When a backup is older than a week it is deleted unless it is the very first backup created that week. i.e out of these backups backup_1_12_2010_0-00_Mon.db, backup_1_12_2010_0-30_Mon.db, backup_1_12_2010_1-00_Mon.db ... backup_7_12_2010_23-30_Sun.db etc only backup_1_12_2010_0-00_Mon.db is kept.
Anyone have anything similar or any ideas where to start?
Answer: A cron
Description:
Try creating a file something.sh with this:
#!/bin/sh
mysqldump -u root -p pwd --opt db1.sql > /respaldosql/db1.sql
mysqldump -u root -p pwd --opt db2.sql > /respaldosql/db2.sql
cd /home/youuser/backupsql/
tar -zcvf backupsql_$(date +%d%m%y).tgz *.sql
find -name '*.tgz' -type f -mtime +2 -exec rm -f {} \;
Give the adequate permission to the file
chmod 700 mysqlrespaldo.sh
or
sudo chmod 700 something.sh
and then create a cron with
crontab -e
setting it like
**0 1 * * *** /home/youruser/coolscripts/something.sh
Remember that the numbers or '*' characters have this structure:
Minutes (range 0-59)
Hours (0-23)
Day of month (1-31)
Month (1-12)
Day of the week (0-6 being 0=Domingo)
Absolute path to script or program to run
You can also use the helper folder available in newer versions of linux distros, where you find /etc/cron.daily, /etc/cron.hourly, /etc/cron.weekly, etc. In this case, you can create a symlink to your script into the chosen folder and OS will take charge of running it with the promised recurrence (from a powerful comment by #Nick).
Create a shell script like the one below:
#!/bin/bash
mysqldump -u username -p'password' dbname > /my_dir/db_$(date+%m-%d-%Y_%H-%M-%S).sql
find /mydir -mtime +10 -type f -delete
Replace username, password and your backup directory(my_dir). Save it in a directory(shell_dir) as filename.sh
Schedule it to run everyday using crontab -e like:
30 8 * * * /shell_dir/filename.sh
This will run everyday at 8:30 AM and backup the database. It also deletes the backup which is older than 10 days. If you don't wanna do that just delete the last line from the script.
Doing pretty much the same like many people.
The script needs to run on the machine hosting the MySql database (It is a linux machine).
=> Create a local bash or perl script (or whatever) "myscript" on this machine "A"
The backups must be saved onto the same server that the database is on.
=> in the script "myscript", you can just use mysqldump. From the local backup, you may create a tarball that you send via scp to your remote machine. Finally you can put your backup script into the crontab (crontab -e).
Some hints and functions to get you started as I won't post my entire script, it does not fully do the trick but not far away :
#!/bin/sh
...
MYSQLDUMP="$(which mysqldump)"
FILE="$LOCAL_TARBALLS/$TARBALL/mysqldump_$db-$SNAPSHOT_DATE.sql"
$MYSQLDUMP -u $MUSER -h $MHOST -p$MPASS $db > $FILE && $GZIP $GZ_COMPRESSION_LEVEL $FILE
function create_tarball()
{
local tarball_dir=$1
tar -zpcvf $tarball_dir"_"$SNAPSHOT_DATE".tar.gz" $tarball_dir >/dev/null
return $?
}
function send_tarball()
{
local PROTOCOLE_="2"
local IPV_="4"
local PRESERVE_="p"
local COMPRESSED_="C"
local PORT="-P $DESTINATION_PORT"
local EXECMODE="B"
local SRC=$1
local DESTINATION_DIR=$2
local DESTINATION_HOST=$DESTINATION_USER"#"$DESTINATION_MACHINE":"$DESTINATION_DIR
local COMMAND="scp -$PROTOCOLE_$IPV_$PRESERVE_$COMPRESSED_$EXECMODE $PORT $SRC $DESTINATION_HOST &"
echo "remote copy command: "$COMMAND
[[ $REMOTE_COPY_ACTIVATED = "Yes" ]] && eval $COMMAND
}
Then to delete files older than "date", you can look at man find and focus on the mtime and newer options.
Edit: as said earlier, there is no particular interest in doing a local backup except a temproray file to be able send a tarball easily and delete it when sent.
You can do most of this with a one-line cronjob set to run every 30 minutes:
mysqldump -u<user> -p<pass> <database> > /path/to/dumps/db.$(date +%a.%H:%M).dump
This will create a database dump every 30 minutes, and every week it'll start overwriting the previous week's dumps.
Then have another cronjob that runs once a week that copies the most recent dump to a separate location where you're keeping snapshots.
After a brief reading the question and the good answers i would add few more points. Some of them are mentioned already.
The backup process can involve next steps:
Create a backup
Compress the backup file
Encrypt the compressed backup
Send the backup to a cloud (DropBox, OneDrive, GoogleDrive, AmazonS3,...)
Get a notification about results
Setup a schedule to run the backup process periodically
Delete the old backup files
To compound a script to cover all the backup steps you need an effort and knowledge.
I would like to share a link to an article (i'm one of the writers) which describes the most used ways to backup MySQL databases with some details:
Bash script
# Backup storage directory
backup_folder=/var/backups
# Notification email address
recipient_email=<username#mail.com>
# MySQL user
user=<user_name>
# MySQL password
password=<password>
# Number of days to store the backup
keep_day=30
sqlfile=$backup_folder/all-database-$(date +%d-%m-%Y_%H-%M-%S).sql
zipfile=$backup_folder/all-database-$(date +%d-%m-%Y_%H-%M-%S).zip
# Create a backup
sudo mysqldump -u $user -p$password --all-databases > $sqlfile
if [ $? == 0 ]; then
echo 'Sql dump created'
else
echo 'mysqldump return non-zero code' | mailx -s 'No backup was created!' $recipient_email
exit
fi
# Compress backup
zip $zipfile $sqlfile
if [ $? == 0 ]; then
echo 'The backup was successfully compressed'
else
echo 'Error compressing backup' | mailx -s 'Backup was not created!' $recipient_email
exit
fi
rm $sqlfile
echo $zipfile | mailx -s 'Backup was successfully created' $recipient_email
# Delete old backups
find $backupfolder -mtime +$keep_day -delete
Automysqlbackup
sudo apt-get install automysqlbackup
wget https://github.com/sixhop/AutoMySQLBackup/archive/master.zip
mkdir /opt/automysqlbackup
mv AutoMySQLBackup-master.zip
cd /opt/automysqlbackup
tar -zxvf AutoMySQLBackup-master.zip
./install.sh
sudo nano /etc/automysqlbackup/automysqlbackup.conf
CONFIG_configfile="/etc/automysqlbackup/automysqlbackup.conf"
CONFIG_backup_dir='/var/backup/db'
CONFIG_mysql_dump_username='root'
CONFIG_mysql_dump_password='my_password'
CONFIG_mysql_dump_host='localhost'
CONFIG_db_names=('my_db')
CONFIG_db_exclude=('information_schema')
CONFIG_mail_address='mail#google.com'
CONFIG_rotation_daily=6
CONFIG_rotation_weekly=35
CONFIG_rotation_monthly=150
automysqlbackup /etc/automysqlbackup/automysqlbackup.conf
Third party tools
Hope it would be helpful!
My preference is for AutoMySQLBackup which comes with Debian. It's really easy and creates daily backups, which can be configured. As well, it stores on weekly and then one monthly backup as well.
I have had this running for a while and it's super easy to configure and use!
You might consider this Open Source tool, matiri, https://github.com/AAFC-MBB/matiri which is a concurrent mysql backup script with metadata in Sqlite3. Features (more than you were asking for...):
Multi-Server: Multiple MySQL servers are supported whether they are co-located on the same or separate physical servers.
Parallel: Each database on the server to be backed up is done separately, in parallel (concurrency settable: default: 3)
Compressed: Each database backup compressed
Checksummed: SHA256 of each compressed backup file stored and the archive of all files
Archived: All database backups tar'ed together into single file
Recorded: Backup information stored in Sqlite3 database
Full disclosure: original matiri author.