What is the optimal way to load in a sql dump when using docker-compose + docker automated builds?
Have been ignoring docker-compose for a moment and trying to understand docker and it's automated builds at first but have come to realize that i will probably need docker-compose if i want to accomplish my project goal that is to use one 1 command and from that have a fully working 3 site Docker cluster
1xHAProxy
3xUbuntu/wp
3xMysqld
In my Dockerfile i can just include the db.sql from my Github repo like
ADD db.sql /tmp/db.sql
Failing to find a best practise how i should load my DB without writing any commands outside of build.
Want to know your solution to this using Dockerfile or Compose
By just executing one of the commands below a mysql FROM mysql with ADD db.sql db.sql should be build / run while loading db.sql in to mysql db wp
Dockerfile
$docker run -d user/repo:tag
docker-compose.yml
$docker-compose up
If am totally on the wrong path here please give me some references. Could also mention that am planning to use CoreOS once i feel OK with Docker. So if best practices on a CoreOS > Docker setup is something else, let me know!
There are two options for initializing a SQL file during build or run time:
The first would be to just base your MySQL image on the official image and place your SQL file in /docker-entrypoint-initdb.d (using something like ADD my.sql /docker-entrypoint-initdb.d/ in the Dockerfile). The official image has a fairly complex entrypoint script (https://github.com/docker-library/mysql/blob/master/5.7/docker-entrypoint.sh) which starts MySQL, initializes a username and password, and scripts from the /docker-entrypoint-initdb.d folder.
The other option would be to do something like the answer at https://stackoverflow.com/a/25920875/684908 and just add a command such as:
COPY dump.sql /tmp/
RUN /bin/bash -c "/usr/bin/mysqld_safe &" && \
sleep 5 && \
mysql -u root -e "CREATE DATABASE mydb" && \
mysql -u root mydb < /tmp/dump.sql
Related
I am trying to get a bash script to run when my MySQL container starts. Not the initial time when there are no databases to create, but subsequent times (so placing the files in docker-entrypoint-initdb.d will not work).
My objective is to re-build my container with some database upgrade scripts (schema changes, etc). The thought being I deploy the container with the initial scripts and deploy subsequent updates with my database upgrades as the application ages. It seems like this would be an easy task, but I am having trouble.
Most of the things I have tried came from suggestions I found googling. Here are things I have tried with no success:
Modify the entrypoint.sh (and /usr/local/bin/docker-entrypoint.sh) in the Dockerfile build to add in a call to my script.
This does not even seem to be called, which I suspect is a sign, but my database starts (also note it creates my schema fine the first time)
I do this with a RUN sed in my Dockerfile and have confirmed my changes exist after the container starts
Tried running my script on startup by:
adding a script to /etc/r.d/rc.local
adding a restart cron job (well, I tried, but the Oracle Linux distro doesn’t have it)
— Modifying the /etc/bashrc
— Adding a script to /etc/profile.d/
— Appending to /etc/profie.d/sh.local
Tried adding a command to my docker-compose.yml, but it said that wasn’t found.
My actual database upgrade script works great when I log in to the container manually and execute it. All of my experiments above have been just touching a file or echoing to a file as a proof of concept. Once I get that working, I'll add in the logic to wait for MySQL to start and then run my actual script.
Dockerfile:
FROM mysql:8.0.32
VOLUME /var/lib/mysql
## these are my experiments
RUN sed -i '/main "$#"/a echo "run the script here" > /usr/tmp/XXX' /entrypoint.sh
RUN sed -i '/main "$#"/a echo "run the script here" > /usr/tmp/XXX' /usr/local/bin/docker-entrypoint.sh
RUN echo "touch /usr/tmp/XXX" >> /etc/profile.d/sh.local
RUN sed -i '/doublesourcing/a echo “run the script here > /usr/tmp/XXX' etc/bashrc
I build and run it using:
docker build -t mysql-database -f Dockerfile .
docker run -it --rm -d -p 3306:3306 --name database -v ~/Docker-Volume-Share/database:/var/lib/mysql mysql-database
Some other information that may be useful
I am using a volume on the host. I’ve run my experiments with an existing schema as well as by deleting this directory so it starts fresh
I am using mysql:8.0.32 as the image (Oracle Linux Server release 8.7)
Docker version 20.10.22, build 3a2c30b
Host OS is macOS 13.2.1
Thanks in advance for any tips and guidance!
It sounds like you are trying to run a script after the MySQL container has started and the initial setup has been completed. Here are a few suggestions:
1-Use a custom entrypoint script
You can create a custom entrypoint script that runs after the default entrypoint script included in the MySQL container image. In your Dockerfile, copy your custom entrypoint script into the container and set it as the entrypoint. Here's an example:
FROM mysql:8.0.32
COPY custom-entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/custom-entrypoint.sh
ENTRYPOINT ["custom-entrypoint.sh"]
In your custom entrypoint script, you can check if the database already exists and run your upgrade script if it does. Here's an example:
#!/bin/bash
set -e
# Run the default entrypoint script
/docker-entrypoint.sh "$#"
# Check if the database already exists
if mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "use my_database"; then
# Run your upgrade script
/path/to/upgrade-script.sh
fi
2-Use a Docker Compose file
If you're using Docker Compose, you can specify a command to run after the container has started. Here's an example:
version: '3'
services:
database:
image: mysql:8.0.32
volumes:
- ~/Docker-Volume-Share/database:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: mypassword
command: >
bash -c "
/docker-entrypoint.sh mysqld &
while ! mysqladmin ping -hlocalhost --silent; do sleep 1; done;
/path/to/upgrade-script.sh
"
This command runs the default entrypoint script in the background, waits for MySQL to start, and then runs your upgrade script.
I hope these suggestions help you achieve your goal!
I'm using the mysql/mysql-server image to create a mysql server in docker. Since I want to setup my database (add users, create tables) automatically, I've created a SQL file that does that for me. In order to automatically run that script, I extended the image with this dockerfile:
FROM mysql/mysql-server:latest
RUN mkdir /scripts
WORKDIR /scripts
COPY ./db_setup.sql .
RUN mysql -u root -p password < cat db_setup.sql
but for some reason, this happens:
/bin/sh: cat: No such file or directory
ERROR: Service 'db' failed to build : The command '/bin/sh -c mysql -u root -p password < cat db_setup.sql' returned a non-zero code: 1
How do I fix this?
You can just remove the cat command from your RUN command:
RUN mysql -u root -p password < db_setup.sql
No such file or directory is returned since cat cannot be found in the current directory set by WORKDIR. You can just redirect the stdin of mysql to be from the db_setup.sql file. Edited to clarify < sh redirection is expecting the file name to use for input.
EDIT 2: Keep in mind your example is a RUN command that is attempting to run mysql and creating a layer at docker image build time. You may want to have this run during the mysql entrypoint script at runtime instead (e.g. scripts are run from thedocker-entrypoint-initdb.d/ directory by the docker-entrypoint.sh script of the official mysql image) or using other features that are documented for the official image.
RUN is a build time command. MySQL isn't running at this point.
If you where/are using a standard image there is a location for database initialization:
FROM mysql:8.0
COPY db_setup.sql /docker-entrypoint-initdb.d
Command cat is not present in mysql/mysql-server:latest image.
Moreover, you would only need to provide filename afetr redirection.
RUN mysql -u root -p password < db_setup.sql
I'm dumping a database into a sql dump:
docker exec mysql sh -c 'exec mysqldump --all-databases -uroot -ppassword' > all-databases.sql
Then I'm using a Dockerfile to build a mysql image and run as a container:
FROM mysql:5.6.41
# needed for intialization
ENV MYSQL_ROOT_PASSWORD=whateverPassword
ADD all-databases.sql /docker-entrypoint-initdb.d/
EXPOSE 3306
When I run the container if I exec into the container, can I access the all-databases.sql file and see the contents of my database in plaintext in the docker image?
Currently if I look into /docker-entrypoint-initdb.d/ it says all-databases.sql but I don't know where that file is stored/if it's encrypted.
If you docker exec into the container, the file will be unencrypted. (It's just a text file and you can look at it with more on most image bases.)
However, if you can run any Docker command at all, then generally it's trivial to get unrestricted root access on the system. (Consider using docker run -v /etc:/host-etc to add yourself to /etc/sudoers or to allow root logins with no password.)
Also remember that anyone who has the image can docker run it and see the file there, if that matters to your security concerns. If you're looking for a single file with root access on the system anyways, you can find it without too much effort in /var/lib/docker. They can also easily run docker history to see the database root password you've set.
I know how to restore a dump file from mysqldump. Now, I am attempting to do that using kubernetes and a docker container. The database files are in persistent (nfs) mount. The docker cannot be accessed outside of the cluster as there is no need for anything external to touch it.
I tried:
kubectl run -i -t dbtest --image=mariadb --restart=Never --rm=true --command -- mysql -uroot -ps3kr37 < dump.sql
and
kubectl exec mariadb-deployment-3614069618-mn524 -i -t -- mysql -u root -p=s3kr37 < dump.sql
But neither commands worked -- errors about TTY, sockets, and other things hinting that I am missing something vital here.
What am I not understanding here?
I could just stop the deployment, scp the database files, and restart the container and hope for the best. However, what can go right?
The question Install an sql dump file to a docker container with mariaDB sure looks like a duplicate but is not: first, I am on Linux not Windows and more importantly the answers all are about initialising with a dump. I want to be able to trash the data and revert to the dump data. This is a test system that will eventually be the "live" so I need to restore from many potential dumps.
As described in here you can use the following command to restore a DB on kubernetes pod from a dump in your machine
$ kubectl exec -it {{podName}} -n {{namespace}} -- mysql -u {{dbUser}} -p{{password}} {{DatabaseName}} < <scriptName>.sql
Example :
$ kubectl exec -it mysql-58 -n sql -- mysql -u root -proot USERS < dump_all.sql
What I did was this:
Create an NFS mount with two sub0drectories: mysql and initd.
In initd, I added several ,sql files, including the dump.
Mount initd as /docker-entrypoint-initdb.d in the deployment.This causes all the files to be read at initialisation time provided that it is the first time we run.
The mysql directory is mounted as /var/lib/mysql and contains all the mariaDB files.
If I need to revert, I trash all the contents of the mysql directory and re-create the deployment.
This should work:
kubectl --kubeconfig=k8s-XXXXXXX-kubeconfig.yaml exec -i ddevdb-XXXXX -- mysql -u root -h mysqlservice -proot drupal < you-dump.sql
kubeconfig is optional, digitalocean for examples provides that so you can run your commands from your local.
To see if everything looks good:
kubectl --kubeconfig=k8s-XXXXXXX-kubeconfig.yaml run -it --rm --image=mariadb:10.4 --restart=Never mysql -- mysql -h mysqlservice -proot
After which you'll have a terminal inside mysql.
I am trying to automate the installation and running of set of linked docker containers using fig. The configuration is composed of a container running RStudio linked to a container running MySQL, such that I can query the MySQL database from RStudio.
On first run, I would like to create the MySQL container from the base MySQL image, and populate it with a user and database. From the command line, something like this:
#Get the latest database file
wget -P /tmp http://ergast.com/downloads/f1db.sql.gz && gunzip -f /tmp/f1db.sql.gz
#Create the database container with user, password and database
docker run --name ergastdb -e MYSQL_USER=ergast -e MYSQL_ROOT_PASSWORD=mrd -e MYSQL_DATABASE=f1db -d mysql
#Populate the database
docker run -it --link=ergastdb:mysql -v /tmp:/tmp/import --rm mysql sh -c 'exec mysql -h$MYSQL_PORT_3306_TCP_ADDR -P$MYSQL_PORT_3306_TCP_PORT -uergast -pmrd f1db < /tmp/import/f1db.sql'
#Fire up RStudio and link to the MySQL db
docker run --name f1djd -p 8788:8787 --link ergastdb:db -d rocker/hadleyverse
If I could get hold of a database image with the data preloaded, I guess that something like the following fig.yml script could link the elements?
gdrive:
command: echo created
image: busybox
volumes:
- "~/Google Drive/shareddata:/gdrive"
dbdata:
image: mysql_preloaded
environment:
MYSQL_USER=ergast
MYSQL_ROOT_PASSWORD=mrd
MYSQL_DATABASE=f1db
rstudio:
image: rocker/hadleyverse
links:
- dbdata:db
ports:
- "8788:8787"
volumes_from:
- gdrive
My question is, can I use a one-shot fig step to create the dbdata container, then perhaps mount a persistent volume, link to it and initialise the database, presumably as part of an initial fig up. If I then start and stop containers, I don't want to run the db initialisation step again, just link to the data volume container that contains the data I previously installed.
I also notice that the MySQL docker image looks like it will support arbitrary datadir definitions (Update entrypoints to read DATADIR from the MySQL configuration directly instead of assuming /var/lib/docker). As I understand it, the current definition of the MySQL image prevents mounting (and hence persisting) the database contents within the database container. I guess this might make it possible to create a mysql_preloaded image, but I don't think the latest version of the MySQL docker script has been pushed to dockerhub just yet and I can't quite think my way to how fig might then be able to make use of this alternative pathway?
Some options:
Edit the fig.yml to run a custom command that is different than the default image command/entrypoint.
From http://www.fig.sh/yml.html (example):
command: bundle exec thin -p 3000
Start the container locally, modify it and then commit it as a new image.
Modify the MySQL image docker-entrypoint.sh file to do your custom initialization.
https://github.com/docker-library/mysql/blob/567028d4e177238c58760bcd69a8766a8f026e2a/5.7/docker-entrypoint.sh
Couldn't you just roll your own version of the MySQL docker image? The official one from MySQL "upstream" is available at https://github.com/mysql/mysql-docker/blob/mysql-server/5.7/Dockerfile
What if you simply make your own copy of that, remove the VOLUME line (line 11) and then you can
docker build -t my_mysql .
docker run -d --name=empty_db my_mysql ...
# add data to the database running in the container
docker commit empty_db primed_db
docker rm -v empty_db
docker run -d --name=instance1 primed_db
docker run -d --name=instance2 primed_db
which should leave you with two running "identical" but fully isolated instances.