I want to make a mysql docker image that imports some initial data in the build process.
Afterwards, when used in a container, the container stays stateless, meaning the data added while the container is running does not survive destroying/starting the container again but the inital data is still there.
Is this possible? How would I a setup such an image and container?
I suggest creating the MySQL tables as needed in a SQL script, or directly in a local MySQL instance and exporting them to a file.
With this file in hand, create a Dockerfile which builds on the MySQL container. Add to this another entrypoint script which injects the SQL script into the database.
You don't write anything about mounting volumes. You may want a data volume for the database or configure MySQL for keeping everything in memory.
For added "statelessness" you may want to DROP all tables in your SQL script too.
I think what you need is a multi-stage build:
FROM mysql:5.7 as builder
# needed for intialization
ENV MYSQL_ROOT_PASSWORD=somepassword
ADD initialize.aql /docker-entrypoint-initdb.d/
# That file does the DB initialization but also runs mysql daemon, by removing the last line it will only init
RUN ["sed", "-i", "s/exec \"$#\"/echo \"not running $#\"/", "/usr/local/bin/docker-entrypoint.sh"]
RUN ["/usr/local/bin/docker-entrypoint.sh", "mysqld", "--datadir", "/initialized-db"]
FROM mysql:5.7
COPY --from=builder /initialized-db /var/lib/mysql
You can put your initialization scripts in initialize.sql (or choose a different way to initialize your database).
The resulting image is a database that is already initialised. You can use it and throw it away as you like.
You can also use this process to create different images (tag them differently) for different use cases.
Hope this answers your question.
Related
Can I run a docker container with mysql, and save my database (data), outside the container?
Yes, you can. You can use bind mounts when creating the docker container to mount a path on the host to some path inside the container:
https://docs.docker.com/storage/bind-mounts/
You could, for example, mount the host OS' /home//mysqldata as /var/lib/mysql inside the container. When a process inside the docker container tries to read/write files in /var/lib/mysql inside the container, that will actually be reading/writing data in the host OS' /home//mysqldata directory/folder. For example:
docker run -it --mount type=bind,source=/home/bob/mysqldata,target=/var/lib/mysql <some_image_name>
Do note that docker volumes can also be used for this although those work differently than bind mounts, so make sure you're using a bind mount (type=bind).
Also, I've seen at least one scenario where using a bind mount won't work for MySQL data. In my case it was using a bind mount for a docker container that was running inside a Vagrant box using a directory that was a VirtualBox shared folder. In that case I was getting some kernel/block level errors that prevented MySQL from setting certain file modes or making low-level calls to some of the files in the data dir which ultimately prevented MySQL from starting. I forget now exactly what error it was throwing (I can go back and check) but I had to switch to a volume instead of a bind mount. That was fine for my use case but just be aware if you use a bind mount and MySQL fails to start due to some lower-level disk call.
I should also add that it's not clear from your question /why/ you want to do this so I can't advocate that doing this will be good/do what you want. Only one MySQL process should be writing to the MySQL data directory at a time and the files are binary files so trying to read them with something other than MySQL seems odd. But, if you have a use case where you want something outside of Docker to read the MySQL data files, the bind mount might do what you want.
I am developing an application and try to implement the microservice architecture. For information about locations (cities, zip codes, etc.) I downloaded a database dump for mysql from opengeodb.org.
Now I want to provide the database as a docker container.
I set up a mysql image with following Dockerfile as mentioned in the docs for the mysql image:
FROM mysql
ENV MYSQL_ROOT_PASSWORD=mypassword
ENV MYSQL_DATABASE geodb
WORKDIR /docker-entrypoint-initdb.d
ADD ${PWD}/sql .
EXPOSE 3306
The "sql"-folder contains sql scripts with the raw data as insert statements, so it creates the whole database.The problem is, that the database is really huge and it takes really long to set it up.
So I thought, maybe there is a possibility to save the created database inside an image, because it is an static database for read-only operations only.
I am fairly new to docker and not quite sure how to achieve this.
I'm using docker on a Windows 10 machine.
EDIT:
I achieved my goal by doing the following:
I added the sql dump file as described above.
I ran the container and built the whole database with a local directory (the 'data' folder) mounted to /var/lib/mysql.
Then stopped the container and edited the Dockerfile:
FROM mysql
ENV MYSQL_ROOT_PASSWORD=mypassword
ENV MYSQL_DATABASE geodb
WORKDIR /var/lib/mysql
COPY ${PWD}\data .
EXPOSE 3306
So the generated Database is now beeing copied from local system into the container.
You could create a volume with your container to persist the database on your local machine. When you first create the container, the SQL in /docker-entrypoint-initdb.d will be executed, and the changes will be stored to the volume. Next time you start the container, MySQL will see that the schema already exists and it won't run the scripts again.
https://docs.docker.com/storage/volumes/
In principle you could achieve it like this:
start the container
load the database
perform a docker commit to build an image of the current state of the container.
The other option would be to load in the database during the image build time, but for this you would have to start mysql similarly to how it's done in the entrypoint script.
start mysql in background
wait for it to initialize
load in the data using mysql < sql file
I have a smallish webapp running in a Docker container. It uses a mariadb database running in another container on the same box, based on the official "mariadb" image.
When I first set up these containers, I started the mariadb container using an "internal" database. I gave the "/var/lib/mysql" a volume name, but I didn't map it to a directory on the host ("-v vol-name:/var/lib/mysql"). Actually, I'm not even sure why I gave it a volume name. I set this up several months ago, and I'm not sure why I would have done that specifically.
In any case, I've concluded that having a database internal to the container wasn't a good idea. I've decided I really need to have the actual database stored on the host and use a volume mapping to refer to it. I know how to do this if I was setting this up from scratch, but now that the app is running, I need to move the database to the host and restart the container to point to that. I'm not certain of all the proper steps to make this happen.
In addition, I'm also going to need to set up a second instance of this application, using containers based on the same images. The second database will also be stored on the host, in a directory next to the other one. I can initialize the second db with the backup file from the first one, but I'll likely manually empty most of the tables in the second instance.
I did use mysqldump inside the container to dump the database, then I copied that backup file to the host.
I know how to set a volume mapping in "docker run" to map /var/lib/mysql in the container to a location on the host.
At this point, I'm not certain exactly what to do with this backup file so I can restart the container with the modified volume mapping. I know I can run "mysql dbname < backup.sql", but I'm not sure of the consequences of that.
While the container is running, run docker cp-a CONTAINER:/var/lib/mysql /local/path/to/folder to copy the MariaDB databases from the container to your local machine. Replace "CONTAINER" with the name or ID of your MariaDB container.
Once you've done that, you can stop the container and restart it binding /local/path/to/folder to the container's /var/lib/mysql path.
If you're using an older version of docker that does not support the -a or --archive flag, you can copy the files without that flag but you'll need to make sure that the folder on the host machine has the proper ownership: the UID and GID of the folder must match the UID and GID of the folder in the Docker container.
Note: if you're using SELinux, you might need to set the proper permissions as well, as the documentation for the MariaDB image states:
Note that users on host systems with SELinux enabled may see issues with this. The current workaround is to assign the relevant SELinux policy type to the new data directory so that the container will be allowed to access it:
$ chcon -Rt svirt_sandbox_file_t /my/own/datadir
our IT broke the mysql container and now it can not be started.
I understand that I can commit a new version and run it without entrypoint, so I can "exec -it" to enter and check what's wrong.
but how can I recover my data? inspect the old container and copy all files from mounted volume? (it seems a overkill for this problem, can I 'start' my container without entrypoint?)
what's the best practice for this problem?
If you have a mounted volume, your data is in a volume directory in your host, and there will be unless you delete it. So, fix your MySQL image and then create another MySQL container.
You should be able to fix your container by using docker attach or docker exec. You can even change container entrypoint using something like this: How to start a stopped Docker container with a different command?
But that's not a good approach. As stated in Best practices for writing Dockerfiles, Docker containers should be ephemeral, meaning this that they can be replaced easily for new ones. So, best option is destroy your container and create a new one.
I think as #kstromeiraos says you should first fix your Dockerfile if at all it's broken and again build and run the container using:
docker build
docker run -v xxx
Since you have used volumes from your MySQL data seems to be backed off properly, so the new container which comes up should have the backed up data.
You can do:
docker exec -it bash
and get into the container and check the logs and data.
I want to create containers w/ a MySQL db and a dump loaded for integration tests. Each test should connect to a fresh container, with the DB in the same state. It should be able to read and write, but all changes should be lost when the test ends and the container is destroyed. I'm using the "mysql" image from the official docker repo.
1) The image's docs suggests taking advantage of the "entrypoint" script that will import any .sql files you provide on a specific folder. As I understand, this will import the dump again every time a new container is created, so not a good option. Is that correct?
2) This SO answer suggests extending that image with a RUN statement to start the mysql service and import all dumps. This seems to be the way to go, but I keep getting
mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
followed by
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
when I run build, even though I can connect to mysql fine on containers of the original image. I tried sleep 5 to wait for the mysqld service to startup, and adding -h with 'localhost' or the docker-machine ip.
How can I fix "2)"? Or, is there a better approach?
If re-seeding the data is an expensive operation another option would be starting / stopping a Docker container (previously build with the DB and seed data). I blogged about this a few months ago Integration Testing using Spring Boot, Postgres and Docker and although the blog focuses on Postgres, the idea is the same and could be translated to MySQL.
The standard MySQL image is pretty slow to start up so might be useful to use something that has been prepared more for this situation like this:
https://github.com/awin/docker-mysql
You can include data or use with a Flyway situation too, but it should speed things up a bit.
How I've solved this before is using a Database Migration tool, specifically flyway: http://flywaydb.org/documentation/database/mysql.html
Flyway is more for migrating the database schema opposed to putting data into it, but you could use it either way. Whenever you start your container just run the migrations against it and your database will be setup however you want. It's easy to use and you can also just use the default MySQL docker container without messing around with any settings. Flyway is also nice for many other reasons, like having a way to have version control for a database schema, and the ability to perform migrations on production databases easily.
To run integration tests with a clean DB I would just have an initial dataset that you insert before the test, then afterwards just truncate all the tables. I'm not sure how large your dataset is, but I think this is generally faster than restarting a mysql container every time,.
Yes, the data will be imported every time you start a container. This could take a long time.
You can view an example image that I created
https://github.com/kliewkliew/mysql-adventureworks
https://hub.docker.com/r/kliew/mysql-adventureworks/
My Dockerfile builds an image by installing MySQL, imports a sample database (from a .sql file), and sets the entrypoint to auto-start MySQL server. When you start a container from this image, it will have the data pre-loaded in the database.