For internal dev productivity usecase, we are building docker images for every build by installing our application which includes glassfish application server and mysql database and keeping the application server and database in stopped state before saving the docker image. On container startup, database and application server are started in that order.
In order to reduce the docker image size, i am planning to compress the mysql data folder and keep the .tar.gz file only in the docker image. Container startup will uncompress the data folder before starting the database. Are there any issues with this approach in case anyone has gone down this path?
Yeah that is a fairly good approach to go with.
You can compress the data into tar either by going inside the container and running tar command. Then you can just edit rc.local where you can put a restart script which will run when the container boots up and untars the tar file in the appropriate directory and then Later you can make an image out of this.
OR
Create image of your container. Then use Dockerfile to do what I explained above. put this image over DockerHub and pull it on a different machine, run it and it should run what you want. You can also make use of CMD to start database.
OR
Simply just tar the data, ship the Docker Image. Pull it on which ever server you want to, keep and untar command handy. Run the container, get into it, untar the file and then start your database server.
Related
I've been plugging around with Docker for the last few days and am hoping to move my Python-MySQL webapp over to Docker here soon.
The corollary is that I need to use Docker volumes and have been stumped lately. I can create a volume directly by
$ docker volume create my-vol
Or indirectly by referencing a nonexistent volume in a docker run call, but I cannot figure out how to populate these volumes with my .sql database file, without copying the file over via a COPY call in the Dockerfile.
I've tried directly creating the volume within the directory containing the .sql file (first method mentioned above) and mounting the directory containing the .sql file in my 'docker run' call, which does move the .sql file to the container (I've seen it by navigaating the bash shell inside the container) but when running a mariadb container connecting to the database-containing mariadb container (as suggested in the mariadb docker readme file), it only has the standard databases (information_schema, mysql, performance_schema)
How can I create a volume containing my pre-existing .sql database?
When working with mariadb in a docker container, the image supports running .sql files as a part of the first startup of the container. This allows you to push data into the database before it is made accessible.
From the mariadb documentation:
Initializing a fresh instance
When a container is started for thefirst time, a new database with the specified name will be created and
initialized with the provided configuration variables. Furthermore, it
will execute files with extensions .sh, .sql and .sql.gz that are
found in /docker-entrypoint-initdb.d. Files will be executed in
alphabetical order. You can easily populate your mariadb services by
mounting a SQL dump into that directory and provide custom images with
contributed data. SQL files will be imported by default to the
database specified by the MYSQL_DATABASE variable.
This means that if you want to inject data into the container, when it starts up for the first time. In your Dockerfile, COPY the .sql file into the container at the path /docker-entrypoint-initdb.d/myscript.sql - and it will be invoked on the database that you specified in the environment variable MYSQL_DATABASE.
Like this:
FROM mariadb
COPY ./myscript.sql /docker-entrypoint-initdb.d/myscript.sql
Then:
docker run -e MYSQL_DATABASE=mydb mariadb
There is then the question of how you want to manage the database storage. You basically have two options here:
Create a volume binding to the host, where mariadb stores the database. This will enable you to access the database storage files easily from the host machine.
An example with docker run:
docker run -v /my/own/datadir:/var/lib/mysql mariadb
Create a docker volume and bind it to the storage location in the container. This will be a volume that is managed by docker. This volume will persist the data between restarts of the container.
docker volume create my_mariadb_volume
docker run -v my_mariadb_volume:/var/lib/mysql mariadb
The is also covered in the docs for the mariadb docker image. I can recommend reading it from top to bottom if you are going to use this image.
Before I keep going down this path (explained below) can someone verify I am on the right track? Or How do you work with test data for local development and refresh it regularly.
For our local development we run a standalone mysql and use an import script to load sanitized test data. The import script takes over 2 hours. I am at the point where I have mysql running in a container and I can load the test data inside. The container fully loaded is 50GB. I am having trouble saving it with docker commit and docker export/docker import. "Error processing tar file(exit status 1): unexpected EOF" From researching this error I need to be on the latest version of Docker 18.09.3. Right now I am using an AWS ami that only goes to version 18.06.1-ce. So currently I am spinning up another EC2 server with a Centos ami to load the newest version on docker.
I have been working on this project for 2 weeks and would appreciate any advice.
If container size if 50GB, I assume you are loading full mysql data in the docker image or load data in container and now want to same container state.
Best way would be to mount host directory to container to hold mysql data. In this way, your mysql image is immutable and data is mounted on the image.
This data can be copied on any other server and run the new mysql container.
I am developing an application and try to implement the microservice architecture. For information about locations (cities, zip codes, etc.) I downloaded a database dump for mysql from opengeodb.org.
Now I want to provide the database as a docker container.
I set up a mysql image with following Dockerfile as mentioned in the docs for the mysql image:
FROM mysql
ENV MYSQL_ROOT_PASSWORD=mypassword
ENV MYSQL_DATABASE geodb
WORKDIR /docker-entrypoint-initdb.d
ADD ${PWD}/sql .
EXPOSE 3306
The "sql"-folder contains sql scripts with the raw data as insert statements, so it creates the whole database.The problem is, that the database is really huge and it takes really long to set it up.
So I thought, maybe there is a possibility to save the created database inside an image, because it is an static database for read-only operations only.
I am fairly new to docker and not quite sure how to achieve this.
I'm using docker on a Windows 10 machine.
EDIT:
I achieved my goal by doing the following:
I added the sql dump file as described above.
I ran the container and built the whole database with a local directory (the 'data' folder) mounted to /var/lib/mysql.
Then stopped the container and edited the Dockerfile:
FROM mysql
ENV MYSQL_ROOT_PASSWORD=mypassword
ENV MYSQL_DATABASE geodb
WORKDIR /var/lib/mysql
COPY ${PWD}\data .
EXPOSE 3306
So the generated Database is now beeing copied from local system into the container.
You could create a volume with your container to persist the database on your local machine. When you first create the container, the SQL in /docker-entrypoint-initdb.d will be executed, and the changes will be stored to the volume. Next time you start the container, MySQL will see that the schema already exists and it won't run the scripts again.
https://docs.docker.com/storage/volumes/
In principle you could achieve it like this:
start the container
load the database
perform a docker commit to build an image of the current state of the container.
The other option would be to load in the database during the image build time, but for this you would have to start mysql similarly to how it's done in the entrypoint script.
start mysql in background
wait for it to initialize
load in the data using mysql < sql file
Trying to set up a docker mysql server with phpmyadmin and an existing company_dev.sql file to import, in an effort to dockerize my dev environment.
My first question is how do I go about setting this up? Do I need to specify an OS, i.e. Ubuntu in my Dockerfile, then add sudo apt-get install mysql-server and install phpmyadmin? Or am I better off running an existing docker image from the docker repo and building on top of that?
Upon making CRUD operations to this database, I would like to save its state for later use. Would using docker commit be appropriate for this use case? I know using dockerfile is best practice.
I appreciate any advice.
First of all, with docker you should have a single service/Daemon per container. In your case, mysql and phpmyadmin should go in different containers. This is not mandatory (there are workarounds) but makes things a lot easier.
In order to avoid reinventing the wheel, you should IMHO always use existing images for the wanted service, expecially if they're official ones. But again, you can choose for any reason to start from scratch (a base image such as "Ubuntu" or "Debian" just to name two) and install the needed stuff.
About the storage question: docker containers should always be immutable. If a container needs to save it's state, it should use volumes. Volumes are a way to share a folder between the container and the host. For instance, the official mysql image uses a volume to store the database files.
So, to summarize, you should use ready images when possible and no, using git commit to store mysql data is not a good practice.
Previously I have used this Dockerfile in order to restore MySQL data.
GitHub - stormcat24/docker-mysql-remote
My first question is how do I go about setting this up?
This dockerfile is using mysqldump to load from real env/save to docker env. You can also do that. Actually, it will load/save whole tables in your specified database.
Do I need to specify an OS, i.e. Ubuntu in my Dockerfile, then add sudo apt-get install mysql-server and install phpmyadmin?
You can see this docker image is created from DockerHub - library/mysql , we don't need to prepare basic middle-wares except phpmyadmin.
Or am I better off running an existing docker image from the docker repo and building on top of that?
It's better to use already existing Docker repository !
Upon making CRUD operations to this database, I would like to save its state for later use. Would using docker commit be appropriate for this use case? I know using dockerfile is best practice.
I have tried this also. After some testing, I successfully saved a docker image contains MySQL DB. To do that, we just need to exec docker commit xxx after finished your build. However be careful, don't push your image file to DockerHub.
our IT broke the mysql container and now it can not be started.
I understand that I can commit a new version and run it without entrypoint, so I can "exec -it" to enter and check what's wrong.
but how can I recover my data? inspect the old container and copy all files from mounted volume? (it seems a overkill for this problem, can I 'start' my container without entrypoint?)
what's the best practice for this problem?
If you have a mounted volume, your data is in a volume directory in your host, and there will be unless you delete it. So, fix your MySQL image and then create another MySQL container.
You should be able to fix your container by using docker attach or docker exec. You can even change container entrypoint using something like this: How to start a stopped Docker container with a different command?
But that's not a good approach. As stated in Best practices for writing Dockerfiles, Docker containers should be ephemeral, meaning this that they can be replaced easily for new ones. So, best option is destroy your container and create a new one.
I think as #kstromeiraos says you should first fix your Dockerfile if at all it's broken and again build and run the container using:
docker build
docker run -v xxx
Since you have used volumes from your MySQL data seems to be backed off properly, so the new container which comes up should have the backed up data.
You can do:
docker exec -it bash
and get into the container and check the logs and data.