Backup MySQL db with Coldfusion - mysql

What I want to do is run a backup task in Coldfusion (probably in a scheduled task) which will back up the structure and data in a MySql database.
The hosting server I use always blocks the use of cfexecute for security purposes so I can't use mysqldump.
e.g.
<cfexecute name="c:\program files\mysql\mysql server 4.1\bin\mysqldump"
arguments="--user=xxx --password=yyy dharma"
outputfile="#expandPath("./ao.sql")#" timeout="30"/>
(From Raymond Camden)
Are there any other options available to me ?

Backing up database files is a good idea, but if you back them up to the same drive, and the drive fails, you are screwed. I backup my databases daily to my local system. Here is the script I use in a .bat file
#ECHO OFF
#REM Set dir variables. Use ~1 format in win2k
SET basedir={directory where zip files will be put}
SET workdir={Working directory}
SET mysqldir=c:\PROGRA~1\mysql\mysqls~1.5\bin
SET gzipdir=c:\PROGRA~2\GnuWin32\bin
SET mysqlpassword={db password}
SET mysqluser={db user}
SET host={host IP or domain name}
for /f "tokens=1-4 delims=/ " %%a in ('date/t') do (
set mm=%%a
set dd=%%b
set yy=%%c
)
ECHO Check connection
PING -n 1 %host%|find "Reply from " >NUL
IF NOT ERRORLEVEL 1 goto :SUCCESS
IF ERRORLEVEL 1 goto :END
:SUCCESS
ECHO Connection found, run backup
#REM Change to mysqldir
CD %mysqldir%
#REM dump database. This is all one line
mysqldump -h %host% -u %mysqluser% -p%mysqlpassword% --databases {space delimited list of databases to backup >%workdir%\backup.sql
#REM Change to workdir
CD %workdir%
#REM Zip up database
%gzipdir%\gzip.exe backup.sql
#REM Move to random file name
MOVE backup.sql.gz %basedir%\%yy%_%mm%_%dd%_backup.gz
#REM Change back to base dir
CD %basedir%
:END
ECHO No connection, do not run
I use Windows task scheduler to run this every night. You could probably update it to remove older backups.
You will need to make sure you have gzip installed.
This will put copies of the DB on your local system - I then use a backup service to back up the backups to another offsite system.

Related

Not able to monitor file changes via batch script for MySQL Logs

I am not able to monitor file changes for a MySQL slow query log on Windows. The batch script works perfectly fine to monitor other files, but not for this one
#echo off
:loop
for %%i in (C:\ProgramData\MySQL\MySQL Server 8.0\Data\slowlog.txt) do echo %%~ai|find "a">nul || goto :loop
echo file was changed
attrib -a C:\ProgramData\MySQL\MySQL Server 8.0\Data\slowlog.txt
goto :loop
This is the batch file to see if the "slowlog.txt" has been changed.
The slowlog is present at
C:\ProgramData\MySQL\MySQL Server 8.0\Data\slowlog.txt

Iterating through multiple subdirectories to load large batch data

newbie to coding and am making a legislative database for use in academic work. I have downloaded the California legislative information into a directory on a partitioned portion of my HD. Loaded the schema to the MySQL DB with no issues, downloaded the data and am having problems getting it uploaded. Lets call my workspace home directory home, within that directory are my modules (I have node in there but I would love to avoid using it until I make an app), my json package and settings files and a subdirectory called pubinfo. This is all set up.
Within the pubinfo directory are my sql table files, and shell commands for loading the data into mysql where I have a DB with tables ready for data insertion, as well as subdirectories for legislative sessions labeled from 2001-2019 by sessions (2001, 2003, and so on by 2 years). The loadData.sh file is below, and the instructions from the California data website said to download these files, unzip them, then to run them on my pubinfo directory...
if [ $# -gt 0 ]; then
echo Usage: .loadData.sh
exit 1
fi
if [ -z "$MYSQL_PWD" ]; then
read -p "Please enter root password:" MYSQL_PWD
export MYSQL_PWD=${MYSQL_PWD}
fi
do
if [ -e ${lcTable}.dat ]; then
echo Processing table: ${lcTable}
if [ -z "$MYSQL_PWD" ]; then
mysql -uroot -p -Dcapublic -v -v -f < ${lcTable}.sql 2>&1 > ${lcTable}.log
else
mysql -uroot -Dcapublic -v -v -f < ${lcTable}.sql 2>&1 > ${lcTable}.log
fi
fi
done < "tables_lc.lst"
When ran, the out put on my zsh terminal is '/usr/local/bin/loadData.sh: line 29: location_code_tbl.sql: No such file or directory', I also have to add that I symlinked the shell file into my path so that the variable could be called in a global setting. I plan to eliminate it once this is all uploaded. I suppose I could symlink all the sql tables as well, but I know there has to be an easier way to iterate through subdirectories while using the sql tables and files in my main directory. I just am not familiar with zsh or bash, I had to take an Udemy course just to set up the MySQL DB. Anyways, I was hoping someone would be able to help, if you have any questions that I did not address here I can answer. Oh and if there is any question on my machine, it is a newer Mac book pro, running the most current mysql version and my editor is visual studio code in addition to the good old terminal.
Thanks!

How to create a capistrano task to download latest database_backup.tgz and import locally?

I'm trying to do something with a Capistrano task that is similar to the heroku db:pull functionality if you are familiar with it.
I have a remote server. On that server I have a bunch of backups in the /path/db_backups/ folder. And in that folder there's a backup of the database everyday.
All I want to do is
Download the latest backup on the client machine.
Untar it.
Import it into local mysql db.
Anyone know of a good way to handle this? Is there a gem I am unaware of? Is there a script you have handy?
I'm not sure if there is a gem for that. I usually copy/pastle this task on capistrano (config/deploy.rb) to pull a compressed database from the server and store it on my development environment
namespace :utils do
desc 'pull the DB from the server'
task :pull_db, :roles => :db, :only => { :primary => true } do
website = "http://www.my_website.com"
filename = "#{application}.dump.#{Time.now.to_f}.sql"
filename_bz2 = "#{filename}.bz2"
remote_file = "#{current_path}/public/#{filename_bz2}"
text = capture "cat #{deploy_to}/current/config/database.yml"
yaml = YAML::load(text)
on_rollback { run "rm #{remote_file}" }
run "mysqldump -h#{yaml[rails_env]['host']} -u #{yaml[rails_env]['username']} -p #{yaml[rails_env]['database']} | bzip2 -c > #{remote_file}" do |ch, stream, out|
ch.send_data "#{yaml[rails_env]['password']}\n" if out =~ /^Enter password:/
end
local_text = run_locally("cat config/database.yml")
local_yaml = YAML::load(local_text)
run_locally("wget #{website}/#{filename_bz2}")
run_locally("bzip2 -d #{filename_bz2}")
run_locally("bundle exec rake db:drop")
run_locally("bundle exec rake db:create")
if local_yaml['development']['password'] && !local_yaml['development']['password'].blank?
run_locally("mysql -h#{local_yaml['development']['host']} -u#{local_yaml['development']['username']} -p#{local_yaml['development']['password']} #{local_yaml['development']['database']} < #{filename}")
else
run_locally("mysql -h#{local_yaml['development']['host']} -u#{local_yaml['development']['username']} #{local_yaml['development']['database']} < #{filename}")
end
run_locally("rm #{filename}")
run "rm #{remote_file}"
end
end
The following script should achieve that:
# Find out which file to copy and save its name in a local text file:
# ssh allows you to specify a command that should be executed on the remote
# machine instead of opening a terminal session on it. I use this to get
# a sorted (ls -t sorts by modification date) list of all backups. I then
# truncate this list to one entry using head -1 and save the file name in a
# local file (filename.txt).
# (12.34.56.78 is a placeholder for the ip/hostname of your server)
ssh 12.34.56.78 ls -t /path/to/backups/ | head -1 > filename.txt
# Copy the backup specified in filename.txt to the tmp dir on your local machine.
scp 12.34.56.78:/path/to/backups/`cat filename.txt` /tmp/db_backup.sql.tar
# Untar the backup archive.
cd /tmp && tar -xf db_backup.sql.tar
# Import into database of choice.
mysql -u your_username -p database_to_import_to < /tmp/db_backup.sql
(This assumes that you are on a UNIX system and have scp and tar installed...)

Automatically Backup MySQL database on linux server

I need a script that automatically makes a backup of a MySql Database. I know there are a lot of posts and scripts out there on this topic already but here is where mine differs.
The script needs to run on the machine hosting the MySql database (It is a linux machine).
The backups must be saved onto the same server that the database is on.
A backup needs to be made every 30 minutes.
When a backup is older than a week it is deleted unless it is the very first backup created that week. i.e out of these backups backup_1_12_2010_0-00_Mon.db, backup_1_12_2010_0-30_Mon.db, backup_1_12_2010_1-00_Mon.db ... backup_7_12_2010_23-30_Sun.db etc only backup_1_12_2010_0-00_Mon.db is kept.
Anyone have anything similar or any ideas where to start?
Answer: A cron
Description:
Try creating a file something.sh with this:
#!/bin/sh
mysqldump -u root -p pwd --opt db1.sql > /respaldosql/db1.sql
mysqldump -u root -p pwd --opt db2.sql > /respaldosql/db2.sql
cd /home/youuser/backupsql/
tar -zcvf backupsql_$(date +%d%m%y).tgz *.sql
find -name '*.tgz' -type f -mtime +2 -exec rm -f {} \;
Give the adequate permission to the file
chmod 700 mysqlrespaldo.sh
or
sudo chmod 700 something.sh
and then create a cron with
crontab -e
setting it like
**0 1 * * *** /home/youruser/coolscripts/something.sh
Remember that the numbers or '*' characters have this structure:
Minutes (range 0-59)
Hours (0-23)
Day of month (1-31)
Month (1-12)
Day of the week (0-6 being 0=Domingo)
Absolute path to script or program to run
You can also use the helper folder available in newer versions of linux distros, where you find /etc/cron.daily, /etc/cron.hourly, /etc/cron.weekly, etc. In this case, you can create a symlink to your script into the chosen folder and OS will take charge of running it with the promised recurrence (from a powerful comment by #Nick).
Create a shell script like the one below:
#!/bin/bash
mysqldump -u username -p'password' dbname > /my_dir/db_$(date+%m-%d-%Y_%H-%M-%S).sql
find /mydir -mtime +10 -type f -delete
Replace username, password and your backup directory(my_dir). Save it in a directory(shell_dir) as filename.sh
Schedule it to run everyday using crontab -e like:
30 8 * * * /shell_dir/filename.sh
This will run everyday at 8:30 AM and backup the database. It also deletes the backup which is older than 10 days. If you don't wanna do that just delete the last line from the script.
Doing pretty much the same like many people.
The script needs to run on the machine hosting the MySql database (It is a linux machine).
=> Create a local bash or perl script (or whatever) "myscript" on this machine "A"
The backups must be saved onto the same server that the database is on.
=> in the script "myscript", you can just use mysqldump. From the local backup, you may create a tarball that you send via scp to your remote machine. Finally you can put your backup script into the crontab (crontab -e).
Some hints and functions to get you started as I won't post my entire script, it does not fully do the trick but not far away :
#!/bin/sh
...
MYSQLDUMP="$(which mysqldump)"
FILE="$LOCAL_TARBALLS/$TARBALL/mysqldump_$db-$SNAPSHOT_DATE.sql"
$MYSQLDUMP -u $MUSER -h $MHOST -p$MPASS $db > $FILE && $GZIP $GZ_COMPRESSION_LEVEL $FILE
function create_tarball()
{
local tarball_dir=$1
tar -zpcvf $tarball_dir"_"$SNAPSHOT_DATE".tar.gz" $tarball_dir >/dev/null
return $?
}
function send_tarball()
{
local PROTOCOLE_="2"
local IPV_="4"
local PRESERVE_="p"
local COMPRESSED_="C"
local PORT="-P $DESTINATION_PORT"
local EXECMODE="B"
local SRC=$1
local DESTINATION_DIR=$2
local DESTINATION_HOST=$DESTINATION_USER"#"$DESTINATION_MACHINE":"$DESTINATION_DIR
local COMMAND="scp -$PROTOCOLE_$IPV_$PRESERVE_$COMPRESSED_$EXECMODE $PORT $SRC $DESTINATION_HOST &"
echo "remote copy command: "$COMMAND
[[ $REMOTE_COPY_ACTIVATED = "Yes" ]] && eval $COMMAND
}
Then to delete files older than "date", you can look at man find and focus on the mtime and newer options.
Edit: as said earlier, there is no particular interest in doing a local backup except a temproray file to be able send a tarball easily and delete it when sent.
You can do most of this with a one-line cronjob set to run every 30 minutes:
mysqldump -u<user> -p<pass> <database> > /path/to/dumps/db.$(date +%a.%H:%M).dump
This will create a database dump every 30 minutes, and every week it'll start overwriting the previous week's dumps.
Then have another cronjob that runs once a week that copies the most recent dump to a separate location where you're keeping snapshots.
After a brief reading the question and the good answers i would add few more points. Some of them are mentioned already.
The backup process can involve next steps:
Create a backup
Compress the backup file
Encrypt the compressed backup
Send the backup to a cloud (DropBox, OneDrive, GoogleDrive, AmazonS3,...)
Get a notification about results
Setup a schedule to run the backup process periodically
Delete the old backup files
To compound a script to cover all the backup steps you need an effort and knowledge.
I would like to share a link to an article (i'm one of the writers) which describes the most used ways to backup MySQL databases with some details:
Bash script
# Backup storage directory
backup_folder=/var/backups
# Notification email address
recipient_email=<username#mail.com>
# MySQL user
user=<user_name>
# MySQL password
password=<password>
# Number of days to store the backup
keep_day=30
sqlfile=$backup_folder/all-database-$(date +%d-%m-%Y_%H-%M-%S).sql
zipfile=$backup_folder/all-database-$(date +%d-%m-%Y_%H-%M-%S).zip
# Create a backup
sudo mysqldump -u $user -p$password --all-databases > $sqlfile
if [ $? == 0 ]; then
echo 'Sql dump created'
else
echo 'mysqldump return non-zero code' | mailx -s 'No backup was created!' $recipient_email
exit
fi
# Compress backup
zip $zipfile $sqlfile
if [ $? == 0 ]; then
echo 'The backup was successfully compressed'
else
echo 'Error compressing backup' | mailx -s 'Backup was not created!' $recipient_email
exit
fi
rm $sqlfile
echo $zipfile | mailx -s 'Backup was successfully created' $recipient_email
# Delete old backups
find $backupfolder -mtime +$keep_day -delete
Automysqlbackup
sudo apt-get install automysqlbackup
wget https://github.com/sixhop/AutoMySQLBackup/archive/master.zip
mkdir /opt/automysqlbackup
mv AutoMySQLBackup-master.zip
cd /opt/automysqlbackup
tar -zxvf AutoMySQLBackup-master.zip
./install.sh
sudo nano /etc/automysqlbackup/automysqlbackup.conf
CONFIG_configfile="/etc/automysqlbackup/automysqlbackup.conf"
CONFIG_backup_dir='/var/backup/db'
CONFIG_mysql_dump_username='root'
CONFIG_mysql_dump_password='my_password'
CONFIG_mysql_dump_host='localhost'
CONFIG_db_names=('my_db')
CONFIG_db_exclude=('information_schema')
CONFIG_mail_address='mail#google.com'
CONFIG_rotation_daily=6
CONFIG_rotation_weekly=35
CONFIG_rotation_monthly=150
automysqlbackup /etc/automysqlbackup/automysqlbackup.conf
Third party tools
Hope it would be helpful!
My preference is for AutoMySQLBackup which comes with Debian. It's really easy and creates daily backups, which can be configured. As well, it stores on weekly and then one monthly backup as well.
I have had this running for a while and it's super easy to configure and use!
You might consider this Open Source tool, matiri, https://github.com/AAFC-MBB/matiri which is a concurrent mysql backup script with metadata in Sqlite3. Features (more than you were asking for...):
Multi-Server: Multiple MySQL servers are supported whether they are co-located on the same or separate physical servers.
Parallel: Each database on the server to be backed up is done separately, in parallel (concurrency settable: default: 3)
Compressed: Each database backup compressed
Checksummed: SHA256 of each compressed backup file stored and the archive of all files
Archived: All database backups tar'ed together into single file
Recorded: Backup information stored in Sqlite3 database
Full disclosure: original matiri author.

Rails Database Back-up Script

I currently use the script below to back-up a website but it could be improved dramatically! Please could you suggest any improvements, or perhaps alternative solutions?
Currently, I only delete items after a massive amount has been reached - and this is not good. Does anyone know how I can delete items that are a month old, or start deleting when there are fifty backups and start deleting the oldest items first?
require 'find'
require 'ftools'
namespace :db do desc "Backup the database to a file. Options: DIR=base_dir
RAILS_ENV=development MAX=20"
task :backup => [:environment] do
datestamp = Time.now.strftime("%d-%m-%Y_%H-%M-%S")
base_path = ENV["DIR"] || "db"
backup_base = File.join(base_path, 'backup')
backup_folder = File.join(backup_base, datestamp)
backup_file = File.join(backup_folder, "#{RAILS_ENV}_dump.sql.gz")
File.makedirs(backup_folder)
db_config = ActiveRecord::Base.configurations[RAILS_ENV]
sh "mysqldump -u #{db_config['username'].to_s} #{'-p' if db_config[
'password']}#{db_config['password'].to_s} --opt #{db_config['database']} |
gzip -c > #{backup_file}"
dir = Dir.new(backup_base)
all_backups = (dir.entries - ['.', '..']).sort.reverse
puts "Created backup: #{backup_file}"
max_backups = ENV["MAX"] || 10000000
unwanted_backups = all_backups[max_backups.to_i..-1] || []
for unwanted_backup in unwanted_backups
FileUtils.rm_rf(File.join(backup_base, unwanted_backup))
puts "deleted #{unwanted_backup}"
end
puts "Deleted #{unwanted_backups.length} backups, #{all_backups.length -
unwanted_backups.length} backups available"
end
end
We use this script, which isn't quite as complex as yours but does more or less the same thing:
#!/usr/bin/env ruby
require "date"
DBS = %w( list the databases to back up )
USER = "" # Username with rights to all those databases, might be root
PW = "" # Password for that username
today_s = Date.today().to_s
yesterday_s = (Date.today()-(2)).to_s
DBS.each do |db|
system "/usr/bin/mysqldump --user=#{USER} --password=#{PW} --add-drop-database --opt -icC #{db} > ~/dbs/#{today_s}-#{db}.sql"
if File.exist?("/path/to/backups/dbs/#{yesterday_s}-#{db}.sql")
File.unlink("/path/to/backups/dbs/#{yesterday_s}-#{db}.sql")
end
end
We then run that with cron on a regular basis (4x/day, but obviously we only keep the most-recent one from each day, because later ones for each day will overwrite earlier ones). It keeps two days worth of backups; we have a remote server which uses scp to copy the entire /path/to/backups/dbs/ directory twice daily, and that one keeps backups until we have time to burn them to DVD-ROM.
Notice that if it misses a deletion the file will hang around for quite a while--the script only deletes "yesterday's" file, not "all files older than X," which your script does. But you can probably take some ideas from this and incorporate them in your script.
why dont use git with cron job ?
git setup:
cd /PATH/TO/EXPORTFILE/
git init .
git add .
git commit -am "init commit"
cron job:
mysqldump -uUSER -pPASSWORD --skip-extended-insert DBNAME > /PATH/TO/EXPORTFILE/FILENAME.SQL && \
cd /PATH/TO/EXPORTFILE/ && \
git add . && \
git commit -am "MYSQL BACKUP" | mail -s "MYSQL BACKUP CRON JOB" your#emailaddress.com
no deleting file, history for ALL mysqls dumps depending on cron job execution times...
Since you already put timestamp in your back up folder name, why don't you parse folder name and delete whatever has timestamp that's older than 30 days?