It is a super weird issue. I have a simple playbook to execute the mysqldump command to backup MySQL:
tasks:
- name: Run mysqldump SQL server
ansible.builtin.shell:
cmd: mysqldump --max_allowed_packet=512M --set-gtid-purged=OFF -u root -p{{ mysql_root_password }} myDB > /tmp/myDB-{{ ansible_date_time.date|replace('-','') }}.sql
If I manually run this playbook, everything is good. The size of my backup file is 800MB. But If I run this playbook by cronjob. I lost some data and tables in the DB. The size of the backup file only has 450MB
* 11 * * * ansible-playbook -i ~/my_hosts.ini --vault-password-file ~/ansible_vault_password ~/mysql_backup.yml
Does anyone have the same issue?
I am very new to Ansible, any help is appreciated!
Related
I have an mysql instance running in a docker container. I mount a volume in /var/lib/mysql to preserve the data after shutting down the container. I think i have two options to backup my database to my host system:
Backup the complete volume:
docker run --rm --volumes-from db -v {BACKUP_PATH_ON_HOST_SYSTEM}:/backup ubuntu tar cvf /backup/backup.tar /var/lib/mysql
Only backup a mysqldump
Basically run above command but instead of backing up the volume i create a mysqldump which i would copy to /backup.
Which option is better?
I have a similar requirement. In my case, I'm using an old mysql Docker image on purpose like so:
db:
image: mysql:5.6
container_name: ${COMPOSE_SITE_NAME}_mysql
volumes:
- db_files:/var/lib/mysql
# Load the initial SQL dump into the DB when it is created.
# This only runs once if the DB is empty.
- ${SQL_DUMP_FILE}:/docker-entrypoint-initdb.d/dump.sql
environment:
MYSQL_ROOT_PASSWORD: ${WORDPRESS_DB_PASS}
...
volumes:
db_files:
name: ${COMPOSE_SITE_NAME}_db_files
If the volume is lost, then it can be recreated with a dump file. In my case, I prefer to make a dump file instead of preserving the cacophony of SQL files in that /var/lib/mysql folder.
docker-compose exec db sh -c '\
mysqldump -uroot -p$MYSQL_ROOT_PASSWORD --all-databases --routines --triggers \
' | gzip -c > /path/outside/docker/backup-`date '+%Y-%m-%d'`.sql.gz
This will create a compressed dump file on your host outside Docker due to the stdout redirect (>). I use the sh -c '' so I can reuse the MYSQL_ROOT_PASSWORD env var in the container. Feel free to adjust this to suit your MySQL requirements, like specifying a limited user.
With the default flags, the dump file will have DROP TABLE IF EXISTS statements so you can replace an existing DB without deleting the volume (docker-compose down then docker volume rm ...).
I'm building a helm chart with a MySQL dependency. It's being set up okay when empty, but I want to run an sh file on the pod that would copy some data from somewhere else. For this I want to use the initializationFiles constructions from the documentation https://github.com/helm/charts/tree/master/stable/mysql
My values.yaml part related to this dependendency looks like this:
mysql:
mysqlRootPassword: somePass
mysqlPassword : somePass
mysqlUser: user
appPassword: somePass
initializationFiles:
db.sh: |-
#!/bin/sh
touch dump.sql
Looks like the code under initializationFiles stanza is tried to be executed and causes the pod to fail. Since this stanza is executed only once by design, the second attempt succeeds, and when the pod is running I don't see any new file when I do kubectl exec -it pod_name -- bash -c ls
I have tried this:
...
initializationFiles:
- db.sh
and put the db.sh file in the same folder as values.yaml, but this still didn't work.
What is the correct way to execute an sh file when the dependency MySQL pod is being set up?
Kubernetes version 1.17, helm 3.5.0
I see that the file is actually copied to the docker-entrypoint-initdb.d, but it's not executed, no new file created.
root#mypod:/docker-entrypoint-initdb.d# ls -la
lrwxrwxrwx 1 root root 12 Mar 6 00:14 db.sh -> ..data/db.sh
root#mypod:/docker-entrypoint-initdb.d# cat db.sh
#!/bin/sh
touch dump.sql
I've tried to run the file manually, and got permission denied:
root#mypod:/# ./docker-entrypoint-initdb.d/db.sh
bash: ./docker-entrypoint-initdb.d/db.sh: Permission denied
If I change my command to echo test then the sh file is executed, I can see this in the logs of the pod. Looks like changing the filesystem is prohibited, but doing touch /dump.sql or touch /home/dump.sql doesn't work either.
MUAHAHA I did it.
The scripts inside ./docker-entrypoint-initdb.d/ are executed by the mysql user.
After poking around inside the pod I've found a directory for which the mysql user has write permissions. So I just write the file there and delete afterwards to initialize the dbs.
Now the whole stanza looks like this:
mysql:
mysqlRootPassword: myPass
mysqlPassword : myPass
mysqlUser: user
appPassword: myPass
initializationFiles:
db.sh: |-
#!/bin/sh
mysqldump -h remoteDbUrl -u remoteUser -pRemotePass --databases db1 db2 > /var/lib/mysql/dump.sql
mysql -uroot -pmyPass < /var/lib/mysql/dump.sql
rm /var/lib/mysql/dump.sql
echo "GRANT ALL PRIVILEGES ON *.* TO 'user'#'%' IDENTIFIED BY 'myPass';" | mysql -uroot -pmyPass
I know how to restore a dump file from mysqldump. Now, I am attempting to do that using kubernetes and a docker container. The database files are in persistent (nfs) mount. The docker cannot be accessed outside of the cluster as there is no need for anything external to touch it.
I tried:
kubectl run -i -t dbtest --image=mariadb --restart=Never --rm=true --command -- mysql -uroot -ps3kr37 < dump.sql
and
kubectl exec mariadb-deployment-3614069618-mn524 -i -t -- mysql -u root -p=s3kr37 < dump.sql
But neither commands worked -- errors about TTY, sockets, and other things hinting that I am missing something vital here.
What am I not understanding here?
I could just stop the deployment, scp the database files, and restart the container and hope for the best. However, what can go right?
The question Install an sql dump file to a docker container with mariaDB sure looks like a duplicate but is not: first, I am on Linux not Windows and more importantly the answers all are about initialising with a dump. I want to be able to trash the data and revert to the dump data. This is a test system that will eventually be the "live" so I need to restore from many potential dumps.
As described in here you can use the following command to restore a DB on kubernetes pod from a dump in your machine
$ kubectl exec -it {{podName}} -n {{namespace}} -- mysql -u {{dbUser}} -p{{password}} {{DatabaseName}} < <scriptName>.sql
Example :
$ kubectl exec -it mysql-58 -n sql -- mysql -u root -proot USERS < dump_all.sql
What I did was this:
Create an NFS mount with two sub0drectories: mysql and initd.
In initd, I added several ,sql files, including the dump.
Mount initd as /docker-entrypoint-initdb.d in the deployment.This causes all the files to be read at initialisation time provided that it is the first time we run.
The mysql directory is mounted as /var/lib/mysql and contains all the mariaDB files.
If I need to revert, I trash all the contents of the mysql directory and re-create the deployment.
This should work:
kubectl --kubeconfig=k8s-XXXXXXX-kubeconfig.yaml exec -i ddevdb-XXXXX -- mysql -u root -h mysqlservice -proot drupal < you-dump.sql
kubeconfig is optional, digitalocean for examples provides that so you can run your commands from your local.
To see if everything looks good:
kubectl --kubeconfig=k8s-XXXXXXX-kubeconfig.yaml run -it --rm --image=mariadb:10.4 --restart=Never mysql -- mysql -h mysqlservice -proot
After which you'll have a terminal inside mysql.
I need to take a backup of all existing MySQL databases on my server with Ansible.
I'm aware of mysql_db module. It takes the names of the databases I'd like to manipulate on one by one, so I must get the list of existing databases before using that module.
Is there any way to backup all MySQL databases at once or to get a list of existing databases with Ansible?
A patch to adds name=all that allows a user to dump or import all data was merged into devel recently, it's not available yet in 1.9.1, but it's already shown in this part of the documentation.
# Dumps all databases to hostname.sql
- mysql_db: state=dump name=all target=/tmp/{{ inventory_hostname }}.sql
Hopefully this will soon be available in a stable release.
(Run sudo pip install ansible --upgrade to upgrade.)
The mysql_db module uses the mysqldump executable under the hood, which in turn provides an --all-databases switch, it's just that the Ansible module does not provide an option to use it via the module.
I would suggest using mysqldump executable via command module for now and in the meantime file a feature request on Ansible's GitHub to add support for it.
Something like this should get you going for now:
- name: Dump all MySQL databases to a single file
command: mysqldump --opt -uroot --all-databases --result-file=/tmp/all-dbs.sql
Adjust the options to mysqldump as desired: http://dev.mysql.com/doc/refman/5.5/en/mysqldump.html
Update Nov 26, 2016:
A patch adding name=all was added to the mysql_db module on May 12, 2015, so the recommended way to dump all databases is:
# Dumps all databases to hostname.sql
- mysql_db: state=dump name=all target=/tmp/{{ inventory_hostname }}.sql
Alternative way, each database in separate file.
---
# This playbook backups all mysql databases into separate files.
- name: backup mysql
vars:
- exclude_db:
- "Database"
- "information_schema"
- "performance_schema"
- "mysql"
tasks:
- name: get db names
shell: 'mysql -u root -p{{ vault_root_passwd }} -e "show databases;" '
register: dblist
- name: backup databases
mysql_db:
state: dump
name: "{{ item }}"
target: "/tmp/{{ item }}.sql"
login_user: root
login_password: "{{ vault_root_passwd }}"
with_items: "{{ dblist.stdout_lines | difference(exclude_db) }}"
I'm trying to setup a Dockerfile for my LAMP project, but i'm having a few problems when starting MySQL. I have the folowing lines on my Dockerfile:
VOLUME ["/etc/mysql", "/var/lib/mysql"]
ADD dump.sql /tmp/dump.sql
RUN /usr/bin/mysqld_safe & sleep 5s
RUN mysql -u root -e "CREATE DATABASE mydb"
RUN mysql -u root mydb < /tmp/dump.sql
But I keep getting this error:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
Any ideas on how to setup database creation and dump import during a Dockerfile build?
The latest version of the official mysql docker image allows you to import data on startup. Here is my docker-compose.yml
data:
build: docker/data/.
mysql:
image: mysql
ports:
- "3307:3306"
environment:
MYSQL_ROOT_PASSWORD: 1234
volumes:
- ./docker/data:/docker-entrypoint-initdb.d
volumes_from:
- data
Here, I have my data-dump.sql under docker/data which is relative to the folder the docker-compose is running from. I am mounting that sql file into this directory /docker-entrypoint-initdb.d on the container.
If you are interested to see how this works, have a look at their docker-entrypoint.sh in GitHub. They have added this block to allow importing data
echo
for f in /docker-entrypoint-initdb.d/*; do
case "$f" in
*.sh) echo "$0: running $f"; . "$f" ;;
*.sql) echo "$0: running $f"; "${mysql[#]}" < "$f" && echo ;;
*) echo "$0: ignoring $f" ;;
esac
echo
done
An additional note, if you want the data to be persisted even after the mysql container is stopped and removed, you need to have a separate data container as you see in the docker-compose.yml. The contents of the data container Dockerfile are very simple.
FROM n3ziniuka5/ubuntu-oracle-jdk:14.04-JDK8
VOLUME /var/lib/mysql
CMD ["true"]
The data container doesn't even have to be in start state for persistence.
Each RUN instruction in a Dockerfile is executed in a different layer (as explained in the documentation of RUN).
In your Dockerfile, you have three RUN instructions. The problem is that MySQL server is only started in the first. In the others, no MySQL are running, that is why you get your connection error with mysql client.
To solve this problem you have 2 solutions.
Solution 1: use a one-line RUN
RUN /bin/bash -c "/usr/bin/mysqld_safe --skip-grant-tables &" && \
sleep 5 && \
mysql -u root -e "CREATE DATABASE mydb" && \
mysql -u root mydb < /tmp/dump.sql
Solution 2: use a script
Create an executable script init_db.sh:
#!/bin/bash
/usr/bin/mysqld_safe --skip-grant-tables &
sleep 5
mysql -u root -e "CREATE DATABASE mydb"
mysql -u root mydb < /tmp/dump.sql
Add these lines to your Dockerfile:
ADD init_db.sh /tmp/init_db.sh
RUN /tmp/init_db.sh
What I did was download my sql dump in a "db-dump" folder, and mounted it:
mysql:
image: mysql:5.6
environment:
MYSQL_ROOT_PASSWORD: pass
ports:
- 3306:3306
volumes:
- ./db-dump:/docker-entrypoint-initdb.d
When I run docker-compose up for the first time, the dump is restored in the db.
Here is a working version using v3 of docker-compose.yml. The key is the volumes directive:
mysql:
image: mysql:5.6
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_USER: theusername
MYSQL_PASSWORD: thepw
MYSQL_DATABASE: mydb
volumes:
- ./data:/docker-entrypoint-initdb.d
In the directory that I have my docker-compose.yml I have a data dir that contains .sql dump files. This is nice because you can have a .sql dump file per table.
I simply run docker-compose up and I'm good to go. Data automatically persists between stops. If you want remove the data and "suck in" new .sql files run docker-compose down then docker-compose up.
If anyone knows how to get the mysql docker to re-process files in /docker-entrypoint-initdb.d without removing the volume, please leave a comment and I will update this answer.
I used docker-entrypoint-initdb.d approach (Thanks to #Kuhess)
But in my case I want to create my DB based on some parameters I defined in .env file so I did these
1) First I define .env file something like this in my docker root project directory
MYSQL_DATABASE=my_db_name
MYSQL_USER=user_test
MYSQL_PASSWORD=test
MYSQL_ROOT_PASSWORD=test
MYSQL_PORT=3306
2) Then I define my docker-compose.yml file. So I used the args directive to define my environment variables and I set them from .env file
version: '2'
services:
### MySQL Container
mysql:
build:
context: ./mysql
args:
- MYSQL_DATABASE=${MYSQL_DATABASE}
- MYSQL_USER=${MYSQL_USER}
- MYSQL_PASSWORD=${MYSQL_PASSWORD}
- MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
ports:
- "${MYSQL_PORT}:3306"
3) Then I define a mysql folder that includes a Dockerfile. So the Dockerfile is this
FROM mysql:5.7
RUN chown -R mysql:root /var/lib/mysql/
ARG MYSQL_DATABASE
ARG MYSQL_USER
ARG MYSQL_PASSWORD
ARG MYSQL_ROOT_PASSWORD
ENV MYSQL_DATABASE=$MYSQL_DATABASE
ENV MYSQL_USER=$MYSQL_USER
ENV MYSQL_PASSWORD=$MYSQL_PASSWORD
ENV MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD
ADD data.sql /etc/mysql/data.sql
RUN sed -i 's/MYSQL_DATABASE/'$MYSQL_DATABASE'/g' /etc/mysql/data.sql
RUN cp /etc/mysql/data.sql /docker-entrypoint-initdb.d
EXPOSE 3306
4) Now I use mysqldump to dump my db and put the data.sql inside mysql folder
mysqldump -h <server name> -u<user> -p <db name> > data.sql
The file is just a normal sql dump file but I add 2 lines at the beginning so the file would look like this
--
-- Create a database using `MYSQL_DATABASE` placeholder
--
CREATE DATABASE IF NOT EXISTS `MYSQL_DATABASE`;
USE `MYSQL_DATABASE`;
-- Rest of queries
DROP TABLE IF EXISTS `x`;
CREATE TABLE `x` (..)
LOCK TABLES `x` WRITE;
INSERT INTO `x` VALUES ...;
...
...
...
So what happening is that I used "RUN sed -i 's/MYSQL_DATABASE/'$MYSQL_DATABASE'/g' /etc/mysql/data.sql" command to replace the MYSQL_DATABASE placeholder with the name of my DB that I have set it in .env file.
|- docker-compose.yml
|- .env
|- mysql
|- Dockerfile
|- data.sql
Now you are ready to build and run your container
edit: I had misunderstand the question here. My following answer explains how to run sql commands at container creation time, but not at image creation time as desired by OP.
I'm not quite fond of Kuhess's accepted answer as the sleep 5 seems a bit hackish to me as it assumes that the mysql db daemon has correctly loaded within this time frame. That's an assumption, no guarantee. Also if you use a provided mysql docker image, the image itself already takes care about starting up the server; I would not interfer with this with a custom /usr/bin/mysqld_safe.
I followed the other answers around here and copied bash and sql scripts into the folder /docker-entrypoint-initdb.d/ within the docker container as this is clearly the intended way by the mysql image provider. Everything in this folder is executed once the db daemon is ready, hence you should be able rely on it.
As an addition to the others - since no other answer explicitely mentions this: besides sql scripts you can also copy bash scripts into that folder which might give you more control.
This is what I had needed for example as I also needed to import a dump, but the dump alone was not sufficient as it did not provide which database it should import into. So in my case I have a script named db_custom_init.sh with this content:
mysql -u root -p$MYSQL_ROOT_PASSWORD -e 'create database my_database_to_import_into'
mysql -u root -p$MYSQL_ROOT_PASSWORD my_database_to_import_into < /home/db_dump.sql
and this Dockerfile copying that script:
FROM mysql/mysql-server:5.5.62
ENV MYSQL_ROOT_PASSWORD=XXXXX
COPY ./db_dump.sql /home/db_dump.sql
COPY ./db_custom_init.sh /docker-entrypoint-initdb.d/
Based on Kuhess response, but without hard sleep:
RUN /bin/bash -c "/usr/bin/mysqld_safe --skip-grant-tables &" && \
while ! mysqladmin ping --silent; do sleep 1; echo "wait 1 second"; done && \
mysql -u root -e "CREATE DATABASE mydb" && \
mysql -u root mydb < /tmp/dump.sql
any file or script added to /docker-entrypoint-initdb.d will executed
at the starting of the container
make sure that you do not add or run any sql or sh file that can use
mysql servies from the Dockerfile .they will fail and stop the image
build becuase mysql servies did not start yet when this files or
scripts called .the best way to add .sh file is to ADD them on
/docker-entrypoint-initdb.d directory from your Dockerfile
working exmple
FROM mysql
ADD mysqlcode.sh /docker-entrypoint-initdb.d/mysqlcode.sh
ADD db.sql /home/db.sql
RUN chmod -R 775 /docker-entrypoint-initdb.d
ENV MYSQL_ROOT_PASSWORD mypassword
and the mysqlcode.sh will do some command when mysql service is active
mysqlcode.sh
#!/bin/bash
mysql -u root -pmypassword --execute "CREATE DATABASE IF NOT EXISTS mydatabase;"
mysql -u root -pmypassword mydatabase < /home/db.sql
I have experienced the same problem, but managed to get it working by separating the MySQL start-up commands:
sudo docker build -t MyDB_img -f Dockerfile.dev
sudo docker run --name SomeDB -e MYSQL_ROOT_PASSWORD="WhatEver" -p 3306:3306 -v $(pwd):/app -d MyDB_img
Then sleep for 20 seconds before running the MySQL scripts, it works.
sudo docker exec -it SomeDB sh -c yourscript.sh
I can only presume that the MySQL server takes a few seconds to start up before it can accept incoming connections and scripts.