There're 4 shell commands that im using for mysql import and export
cmd1 > exports table structure
cmd2 > exports table data
cmd4 > imports table strcuture
cmd5 > imports table data
below 4 commands works well if i directly run on commandline. each command should execute one by one that is based on previous cmd output. cmd2 and cmd4 may take time since it is export/import table data. cmd2 not executing properly. i only seeing empty outputfile. but same command working fine if i run directly on shell.
I have tried suggestions as given here python subprocess and mysqldump. but getting same result as before.
import subprocess
print("subprocess started....")
cmd1 = "mysqldump -h hostnamegoeshere -u FPS_DEV_User -pPASSWORD --single-transaction --no-data --triggers --routines novatime --ignore-table=novatime.novatime_detail_paycom --ignore-table=novatime.novatime_schedules --ignore-table=novatime.novatime_schedules_new > /home/ec2/dba/subprocess/mysql-migration/novatime_mysqldump_so.sql 2> /home/ec2/dba/subprocess/mysql-migration/novatime_mysqldump_so_err.sql"
cmd2 = "mysqldump --host hostnamegoeshere -u FPS_DEV_User -pPASSWORD --compress --single-transaction --no-create-info --add-locks --lock-tables --quick --disable-keys novatime > /home/ec2/dba/subprocess/mysql-migration/novatime_mysqldump_do.sql 2> /home/ec2/dba/subprocess/mysql-migration/novatime_mysqldump_do_err.sql"
cmd4 = "mysql --host hostnamegoeshere -u admin -PASSWORD -f novatime < /home/ec2/dba/subprocess/mysql-migration/novatime_mysqldump_so.sql 2> /home/ec2/dba/subprocess/mysql-migration/novatime_mysqldump_so_err.sql"
cmd5 = "mysql --host hostnamegoeshere -u admin -PASSWORD -f novatime < /home/ec2/dba/subprocess/mysql-migration/novatime_mysqldump_do.sql 2> /home/ec2/dba/subprocess/mysql-migration/novatime_mysqldump_do.sql"
try:
print('process1')
process1 = subprocess.Popen(cmd1,stdout=subprocess.PIPE, shell=True)
process1.wait()
print('process2')
process2 = subprocess.Popen(cmd2,stdout=subprocess.PIPE, shell=True)
process2.wait()
print('process4')
process4 = subprocess.Popen(cmd4,stdout=subprocess.PIPE, shell=True)
process4.wait()
print('process5')
process5 = subprocess.Popen(cmd5,stdout=subprocess.PIPE, shell=True)
process5.wait()
except:
print('error')
You should close the previous processes' stdout before the wait.
print('process1')
process1 = subprocess.Popen(cmd1,stdout=subprocess.PIPE, shell=True)
process1.stdout.close()
process1.wait()
print('process2')
process2 = subprocess.Popen(cmd2,stdout=subprocess.PIPE, shell=True)
The last line of the script was not executed.
I tried to execute the code manually on the instance created and it was successful.
#!/bin/bash
#install tools
apt-get update -y
apt-get install mysql-client -y
#Create MySQL config file
echo "[mysql]" >> ~/.my.cnf
echo "user = poc5admin" >> ~/.my.cnf
echo "password = poc5password" >> ~/.my.cnf
#test
echo "endpoint = ${rds_endpoint}" >> ~/variables
hostip=$(hostname -I)
endpoint=${rds_endpoint}
echo "$hostip" >> ~/variables
#I have created a table here but I will remove the code since it is unnecessary...
#Create User
echo "CREATE USER 'poc5user'#'%' IDENTIFIED BY 'poc5pass';" >> ~/mysqlscript.sql
echo "GRANT EVENT ON * . * TO 'poc5user'#'%';" >> ~/mysqlscript.sql
cp mysqlscript.sql /home/ubuntu/mysqlscript.sql
mysql -h $endpoint -u poc5admin < ~/mysqlscript.sql
Expected result: There should be a Database, Table and User created on the RDS instance.
You can insert or create Database like this from the bash script but it is not recommended an approach to work with RDS. better to place your data over s3 and import from the s3.
Here is the example, that will create DB
resource "aws_db_instance" "db" {
allocated_storage = 20
storage_type = "gp2"
engine = "mysql"
engine_version = "5.7"
instance_class = "db.t2.micro"
name = "mydb"
username = "foo"
password = "foobarbaz"
parameter_group_name = "default.mysql5.7"
s3_import {
source_engine = "mysql"
source_engine_version = "5.6"
bucket_name = "mybucket"
bucket_prefix = "backups"
ingestion_role = "arn:aws:iam::1234567890:role/role-xtrabackup-rds-restore"
}
}
~/.my.cnf why you need this? better to place these script in the s3 file.
second thing, If you still interesting to run from your local environment then you can insert from local-exec
resource "null_resource" "main_db_update_table" {
provisioner "local-exec" {
on_failure = "fail"
interpreter = ["/bin/bash", "-c"]
command = <<EOT
mysql -h ${aws_rds_cluster.db.endpoint} -u your_username -pyour_password your_db < mysql_script.sql
EOT
}
}
But better to with s3.
If you want to import from remote, you can explore remote-exec.
With user-data, you can do this but it seems your MySQL script not generating properly. better to cp script to remote and then run with local exec in remote.
There is no such thing as terraform "user_data". User data is a bootstrap script for the EC2 instances which you can use to install software/binaries or to execute your script at the boot time.
The script will be executed by the cloud-init, not by the terraform itself. The responsibility of the terraform is to set user-data for the ec2 instances.
You may check the cloud-init output logs which should have the result of your user-data script also.
From your code, I am not able to understand which step you have copied the below file.
cp mysqlscript.sql /home/ubuntu/mysqlscript.sql
mysql -h $endpoint -u poc5admin < ~/mysqlscript.sql
I am assuming that you are creating a new server and it does not have any file.
Thank you for your inputs. I have found an answer by moving the config file to /etc/mysql/my.cnf and then executing
mysql -h $endpoint -u poc5admin < ~/mysqlscript.sql
I am using MySQL on centos7. I have 50 databases Like database1, database2...., database50.
How can I set a cronjob for take a dump every day of all database same as database name [ Like database1.sql, database2.sql .... database50.sql ] using single command or script.
Please provide some adequate solution that will be appreciated.
Thanks.
Convert the current date to an integer number of days since some starting date.
Take that modulo 50. This gives you 0 .. 49.
Add 1 and concatenate. Now you have database1 .. database50. Put that in the shell variable db
mysqldump ... $db >$db.sql
I am using this script
#! /bin/bash
# MySQL database backup (databases in separate files) with daily, weekly and monthly rotation
# Sebastian Flippence (http://seb.flippence.net) originally based on code from: Ameir Abdeldayem (http://www.ameir.net)
# You are free to modify and distribute this code,
# so long as you keep the authors name and URL in it.
# Modified by IVO GELOV
# How many backups do you want to keep?
MAX_DAYS=5
# Date format that is appended to filename
DATE=`date +'%Y-%m-%d'`
DATSTR=`date '+%Y%m%d' -d "-$MAX_DAYS days"`
# MySQL server's name
SERVER=""
# Directory to backup to
BACKDIR="/var/db_arhiv/mysql"
#----------------------MySQL Settings--------------------#
# MySQL server's hostname or IP address
HOST="localhost"
# MySQL username
USER="user"
# MySQL password
PASS="password"
# List all of the MySQL databases that you want to backup,
# each separated by a space. Or set the option below to backup all database
DBS="db1 db2"
# Set to 'y' if you want to backup all your databases. This will override
# the database selection above.
DUMPALL="y"
# Custom path to system commands (enable these if you want use a different
# location for PHP and MySQL or if you are having problems running this script)
MYSQL="/usr/local/mysql/bin/mysql"
MYSQLDUMP="/usr/local/mysql/bin/mysqldump"
function checkMysqlUp() {
$MYSQL -N -h $HOST --user=$USER --password=$PASS -e status > /dev/null
}
trap checkMysqlUp 0
function error() {
local PARENT_LINENO="$1"
local MESSAGE="$2"
local CODE="${3:-1}"
if [[ -n "$MESSAGE" ]] ; then
echo "Error on or near line ${PARENT_LINENO}: ${MESSAGE}; exiting with status ${CODE}"
else
echo "Error on or near line ${PARENT_LINENO}; exiting with status ${CODE}"
fi
exit "${CODE}"
}
trap 'error ${LINENO}' ERR
# Check backup directory exists
# if not, create it
if [ ! -e "$BACKDIR/$DATE" ]; then
mkdir -p "$BACKDIR/$DATE"
echo "Created backup directory (${BACKDIR}/${DATE})"
fi
if [ $DUMPALL = "y" ]; then
echo "Creating list of databases on: ${HOST}..."
$MYSQL -N -h $HOST --user=$USER --password=$PASS -e "show databases;" > ${BACKDIR}/dbs_on_${SERVER}.txt
# redefine list of databases to be backed up
DBS=`sed -e ':a;N;$!ba;s/\n/ /g' -e 's/Database //g' ${BACKDIR}/dbs_on_${SERVER}.txt`
fi
echo "Backing up MySQL databases..."
#cd ${LATEST}
for database in $DBS; do
if [ ${database} = "information_schema" ] || [ ${database} = "performance_schema" ] || [ ${database} = "pinba" ]
then
continue
fi
echo "${database}..."
$MYSQLDUMP --host=$HOST --user=$USER --password=$PASS --default-character-set=utf8 --routines --triggers --lock-tables --disable-keys --force --single-transaction --allow-keywords --dump-date $database > ${BACKDIR}/${DATE}/${SERVER}$database.sql
done
if [ $DUMPALL = "y" ]; then
rm -f ${BACKDIR}/dbs_on_${SERVER}.txt
fi
# dump privileges
$MYSQL -N -h $HOST --user=$USER --password=$PASS --skip-column-names -A -e "SELECT CONCAT('SHOW GRANTS FOR ''',user,'''#''',host,''';') FROM mysql.user" | $MYSQL -N -h $HOST --user=$USER --password=$PASS --skip-column-names -A > ${BACKDIR}/${DATE}/${SERVER}_grants.sql
# delete older files
for x in `find ${BACKDIR}/20* -type d`
do
xd=`basename "${x//-/}"`
if [[ $xd < $DATSTR ]]
then
rm -rf "$x"
fi
done
echo "MySQL backup is complete"
I have an existing MySQL instance (test), containing 2 databases and a few users each having different access privileges to each database.
I now need to duplicate one of the databases (into production) and the users associated with it.
Duplicating the database was easy:
Export:
mysqldump --no-data --tables -u root -p secondb >> secondb_schema.sql
Import:
mysql -u root -p -h localhost secondb < secondb_schema.sql
I didn't find, however, a straightforward way to export and import users, from the command line (either inside or outside mysql).
How do I export and import a user, from the command line?
Update: So far, I have found manual (and thus error prone) steps for accomplishing this:
-- lists all users
select user,host from mysql.user;
Then find its grants:
-- find privilege granted to a particular user
show grants for 'root'#'localhost';
Then manually create user with the grants listed in the result of the 'show grants' command above.
I prefer a safer, more automated way. Is there one?
One of the easiest ways I've found to export users is using Percona's tool pt-show-grants. The Percona tool kit is free, easy to install, and easy to use, with lots of documentation.
It's an easy way to show all users, or specific users. It lists all of their grants and outputs in SQL format. I'll give an example of how I would show all grants for test_user:
shell> pt-show-grants --only test_user
Example output of that command:
GRANT USAGE ON *.* TO 'test_user'#'%' IDENTIFIED BY PASSWORD '*06406C868B12689643D7E55E8EB2FE82B4A6F5F4';
GRANT ALTER, INSERT, LOCK TABLES, SELECT, UPDATE ON `test`.* TO 'test_user'#'%';
I usually rederict the output into a file so I can edit what I need, or load it into mysql.
Alternatively, if you don't want to use the Percona tool and want to do a dump of all users, you could use mysqldump in this fashion:
shell> mysqldump mysql --tables user db > users.sql
Note: --flush-privileges won't work with this, as the entire db isn't being dumped. this means you need to run it manually.
shell> mysql -e "FLUSH PRIVILEGES"
mysql -u<user> -p<password> -h<host> -e"select concat('show grants for ','\'',user,'\'#\'',host,'\'') from mysql.user" > user_list_with_header.txt
sed '1d' user_list_with_header.txt > ./user.txt
while read user; do mysql -u<user> -p<password> -h<host> -e"$user" > user_grant.txt; sed '1d' user_grant.txt >> user_privileges.txt; echo "flush privileges" >> user_privileges.txt; done < user.txt
awk '{print $0";"}' user_privileges.txt >user_privileges_final.sql
rm user.txt user_list_with_header.txt user_grant.txt user_privileges.txt
Above script will run in linux environment and output will be user_privileges_final.sql that you can import in new mysql server where you want to copy user privileges.
UPDATE: There was a missing - for the user of the 2nd mysql statement.
In mysql 5.7 and later you can use this.
mysqlpump -uroot -p${yourpasswd} --exclude-databases=% --users
This will generate a sql format output that you can redirect to mysql_users.sql.
Note that it is mysqlpump not mysqldump.
Yet another bash one-liner for linux that you can use instead of the Percona tool:
mysql -u<user> -p<password> -h<host> -N mysql -e "select concat(\"'\", user, \"'#'\", host, \"'\"), coalesce(password, authentication_string) from user where not user like 'mysql.%'" | while read usr pw ; do echo "GRANT USAGE ON *.* TO $usr IDENTIFIED BY PASSWORD '$pw';" ; mysql -u<user> -p<password> -h<host> -N -e "SHOW GRANTS FOR $usr" | grep -v 'GRANT USAGE' | sed 's/\(\S\)$/\1;/' ; done
In complement of #Sergey-Podushkin 's answer, this shell script code is workin for me:
mysql -u<user> -p<password> -N mysql -e "select concat(\"'\", user, \"'#'\", host, \"'\"), authentication_string from user where not user like 'root'" | while read usr pw ; do mysql -u<user> -p<password> -N -e "SHOW GRANTS FOR $usr" | sed 's/\(\S\)$/\1;/'; done
PhpMyAdminYou can use phpMyAdmin.
Login and Go to your database or a table where the user has access.
Select privileges
All users with access are there.
Select Export. And a little window with all the GRANTS are there ready to copy and paste.
I tackled this with a small C# program. There is code here to generate a script or apply the grants directly from source to destination. If porting from a Windows -> *nix environment you may have to consider case sensitivity issues.
using System;
using MySql.Data.MySqlClient;
using System.Configuration;
using System.IO;
using System.Collections.Generic;
namespace GenerateUsersScript
{
class Program
{
static void Main(string[] args)
{
List<string> grantsQueries = new List<string>();
// Get A Show Grants query for each user
using (MySqlConnection sourceConn = OpenConnection("sourceDatabase"))
{
using (MySqlDataReader usersReader = GetUsersReader(sourceConn))
{
while (usersReader.Read())
{
grantsQueries.Add(String.Format("SHOW GRANTS FOR '{0}'#'{1}'", usersReader[0], usersReader[1]));
}
}
Console.WriteLine("Exporting Grants For {0} Users", grantsQueries.Count);
using (StreamWriter writer = File.CreateText(#".\UserPermissions.Sql"))
{
// Then Execute each in turn
foreach (string grantsSql in grantsQueries)
{
WritePermissionsScript(sourceConn, grantsSql, writer);
}
//using (MySqlConnection destConn = OpenConnection("targetDatabase"))
//{
// MySqlCommand command = destConn.CreateCommand();
// foreach (string grantsSql in grantsQueries)
// {
// WritePermissionsDirect(sourceConn, grantsSql, command);
// }
//}
}
}
Console.WriteLine("Done - Press A Key to Continue");
Console.ReadKey();
}
private static void WritePermissionsDirect(MySqlConnection sourceConn, string grantsSql, MySqlCommand writeCommand)
{
MySqlCommand cmd = new MySqlCommand(grantsSql, sourceConn);
using (MySqlDataReader grantsReader = cmd.ExecuteReader())
{
while (grantsReader.Read())
{
try
{
writeCommand.CommandText = grantsReader[0].ToString();
writeCommand.ExecuteNonQuery();
}
catch (Exception ex)
{
Console.WriteLine(grantsReader[0].ToString());
Console.WriteLine(ex.Message);
}
}
}
}
private static void WritePermissionsScript(MySqlConnection conn, string grantsSql, StreamWriter writer)
{
MySqlCommand command = new MySqlCommand(grantsSql, conn);
using (MySqlDataReader grantsReader = command.ExecuteReader())
{
while (grantsReader.Read())
{
writer.WriteLine(grantsReader[0] + ";");
}
}
writer.WriteLine();
}
private static MySqlDataReader GetUsersReader(MySqlConnection conn)
{
string queryString = String.Format("SELECT User, Host FROM USER");
MySqlCommand command = new MySqlCommand(queryString, conn);
MySqlDataReader reader = command.ExecuteReader();
return reader;
}
private static MySqlConnection OpenConnection(string connName)
{
string connectionString = ConfigurationManager.ConnectionStrings[connName].ConnectionString;
MySqlConnection connection = new MySqlConnection(connectionString);
connection.Open();
return connection;
}
}
}
with an app.config containing ...
<connectionStrings>
<add name="sourceDatabase" connectionString="server=localhost;user id=hugh;password=xxxxxxxx;persistsecurityinfo=True;database=MySql" />
<add name="targetDatabase" connectionString="server=queeg;user id=hugh;password=xxxxxxxx;persistsecurityinfo=True;database=MySql" />
</connectionStrings>
Here's what I'm using these days as part of my daily backup scripts (requires root shell and MySQL access, linux shell, and uses the mysql built-in schema:
First, I create a file /var/backup/mysqlroot.cnf containing the root password so I can automate my scripts and not hardcode any passwords in them:
[client]
password=(put your password here)
Then I create an export script which dumps create user commands and grants like this:
touch /var/backup/backup_sql.sh
chmod 700 /var/backup/backup_sql.sh
vi /var/backup/backup_sql.sh
And then write the following contents:
#!/bin/bash
mysql --defaults-extra-file=/var/backup/mysqlroot.cnf -sNe " \
SELECT \
CONCAT( 'CREATE USER \'', User, '\'#\'', Host, '\' IDENTIFIED BY \'', authentication_string, '\'\;' ) AS User \
FROM mysql.user \
WHERE \
User NOT LIKE 'mysql.%' AND CONCAT( User, Host ) <> 'rootlocalhost' AND User <> 'debian-sys-maint' \
"
mysql --defaults-extra-file=/var/backup/mysqlroot.cnf -sNe " \
SELECT \
CONCAT( '\'', User, '\'#\'', Host, '\'' ) as User FROM mysql.user \
WHERE \
User NOT LIKE 'mysql.%' \
AND CONCAT( User, Host ) <> 'rootlocalhost' \
AND User <> 'debian-sys-maint' \
" | sort | while read u ;
do echo "-- $u"; mysql --defaults-extra-file=/var/backup/mysqlroot.cnf -sNe "show grants for $u" | sed 's/$/;/'
done
Then I just have to run it like this:
/var/backup/backup_sql.sh > /tmp/exportusers.sql
A PHP script to loop over your users to get the grant commands would be as such:
// Set up database root credentials
$host = 'localhost';
$user = 'root';
$pass = 'YOUR PASSWORD';
// ---- Do not edit below this ----
// Misc settings
header('Content-type: text/plain; Charset=UTF-8');
// Final import queries goes here
$export = array();
// Connect to database
try {
$link = new PDO("mysql:host=$host;dbname=mysql", $user, $pass);
} catch (PDOException $e) {
printf('Connect failed: %s', $e->getMessage());
die();
}
// Get users from database
$statement = $link->prepare("select `user`, `host`, `password` FROM `user`");
$statement->execute();
while ($row = $statement->fetch())
{
$user = $row[0];
$host = $row[1];
$pass = $row[2];
$export[] = "CREATE USER '{$user}'#'{$host}' IDENTIFIED BY '{$pass}'";
// Fetch any permissions found in database
$statement2 = $link->prepare("SHOW GRANTS FOR '{$user}'#'{$host}'");
$statement2->execute();
while ($row2 = $statement2->fetch())
{
$export[] = $row2[0];
}
}
$link = null;
echo implode(";\n", $export);
Gist: https://gist.github.com/zaiddabaeen/e88a2d10528e31cd6692
pass=your_password_here; \
MYSQL_PWD=$pass mysql -B -N -uroot -e "SELECT CONCAT('\'', user,'\' ','\'', host,'\' ','\'', authentication_string,'\' ','\'', plugin,'\'') FROM mysql.user WHERE user != 'debian-sys-maint' AND user != 'root' AND user != 'mysql.sys' AND user != 'mysql.session' AND user != ''" > mysql_all_users.txt; \
while read line; do linearray=(${line}); \
MYSQL_PWD=$pass mysql -B -N -uroot -e "SELECT CONCAT('CREATE USER \'',${linearray[0]},'\'#\'',${linearray[1]},'\' IDENTIFIED WITH \'',${linearray[3]},'\' AS \'',${linearray[2]},'\'')"; \
done < mysql_all_users.txt > mysql_all_users_sql.sql; \
while read line; do linearray=(${line}); \
MYSQL_PWD=$pass mysql -B -N -uroot -e "SHOW GRANTS FOR ${linearray[0]}#${linearray[1]}"; \
done < mysql_all_users.txt >> mysql_all_users_sql.sql; \
sed -e 's/$/;/' -i mysql_all_users_sql.sql; \
echo 'FLUSH PRIVILEGES;' >> mysql_all_users_sql.sql; \
unset pass
First mysql command : export all users to file and exclude some.
Second mysql command : loop users from file to write a sql command 'create user' to an exported file (with authentication credentials).
Third mysql command : loop users from file to append their privileges to the exported file.
sed command to append a ";" to end of lines and flush privileges to finish.
To import : MYSQL_PWD=$pass mysql -u root < mysql_all_users_sql.sql
SELECT CONCAT('\create user ', user,'\'#\'', host, '\' identified by ', "'", authentication_string, "'"'\;') FROM user WHERE user != 'mysql.session' AND user !='mysql.sys' AND user != 'root' AND user != '';
I had the same problem. The solution is that after the import of the backup you need to do a "flush privileges;". Then the privileges of the users will be active as in the original database.
So execute:
mysql -u root -p -h localhost secondb < secondb_schema.sql
mysql -u root
then in mysql:
flush privileges;
My database had lots of views and it was impossible to drop them one by one.
I would like to just drop them all because the database doesn't refresh structure changes of the tables in the view that select from them.
If you want to do this in the MySQL client, you can dynamically generate the DDL statements using information_schema, dump them to a SQL script, and then execute that script.
Example:
select concat('drop view ',table_schema,'.',table_name,';') as ddl
into outfile '/tmp/drop_all_views.sql'
from information_schema.views
where table_schema = 'your_schema';
\. /tmp/drop_all_views.sql
After searching on the web, I found a shell script to drop all tables here: http://www.cyberciti.biz/faq/how-do-i-empty-mysql-database/
Then, I changed that script to drop all views of the database.
This is the final result:
#!/bin/bash
PREFIX=""
SUFFIX=""
HOST="localhost"
PORT="3306"
while getopts p:s:h:P: OPCAO; do
case "${OPCAO}" in
p) PREFIX="${OPTARG}" ;;
s) SUFFIX="${OPTARG}" ;;
h) HOST="${OPTARG}" ;;
P) PORT="${OPTARG}" ;;
esac
done
shift $((OPTIND-1))
MUSER="$1"
MPASS="$2"
MDB="$3"
# Detect paths
MYSQL=$(which mysql)
AWK=$(which awk)
GREP=$(which grep)
if [ $# -eq 0 ]
then
echo "Usage: $0 [-h Host] [-P Port] [-p Prefix-View-Name] [-s Suffix-View-Name] {MySQL-User-Name} {MySQL-User-Password} {MySQL-Database-Name}"
echo "Drops all views from a MySQL"
exit 1
fi
TABLES=$($MYSQL -h $HOST -P $PORT -u $MUSER -p$MPASS $MDB -e "SELECT table_name FROM information_schema.views WHERE table_schema = '$MDB' AND table_name LIKE '$PREFIX%$SUFFIX';" | $AWK '{ print $1}')
for t in $TABLES
do
echo "Deleting $t view from $MDB database..."
$MYSQL -h $HOST -P $PORT -u $MUSER -p$MPASS $MDB -e "drop view $t"
done
This solution will work just for who uses Unix-like systems.
To call the script, I used:
./script.sh [-h host] [-P port] [-p prefixViewName] [-s suffixViewName] username password databaseName
EDIT: I improved the script to accept option parameters, like the host, port and also a prefix and a suffix to filter what views will be dropped.