Mysql database gone on each docker restart - mysql

Background: I was updating my dockerfile and to make sure it works I rebuild my docker from scratch after purging everything using system prune -a. After it was built successfully I tried making a database called "database_A" and using "show databases" it does show that "database_A" was created. However whenever I restart the container it will be gone and checking using the query "show databases" the database "database_A" does not exist.
This is how I run the mysql container:
echo "runing mysql container..."
docker run --rm -it \
--network host \
-e MYSQL_ROOT_PASSWORD=secret \
--volume $(pwd)/mysql_data:/var/lib/mysql \
--volume $(pwd)/mysql.conf:/etc/mysql/mysql.conf.d \
--name mysql \
-p 3307:3307 \
-d mysql:5.6
in mysql.conf.d/mysqld.cnf it was specified
datadir = /var/lib/mysql
Using the bash terminal on the mysql container I checked the /var/lib/mysql folder and the ibdata1 file wasn't update at all. I tried creating a folder in the /var/lib/mysql and it does shows up in /mysql_data in host. So it's using the right volume but saving in different location? How can I verify it?

Try to use permanent folder for storing mysql db data files. Using $(pwd)/mysql_data seems temporary and dependent of where you run the container.
Below flow shows that created database is retained after container removal and starting new one with the same bind mount:
$ docker run --name some-mysql -v ~/private/mysql-data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=secret -d mysql:5.6
$docker exec -it some-mysql bash
root#0873b9898e23:/# mysql -uroot -psecret
Welcome to the MySQL monitor...
mysql> crete database test;
Query OK, 1 row affected (0.01 sec)
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| test |
+--------------------+
4 rows in set (0.01 sec)
mysql> ^DBye
root#0873b9898e23:/# exit
# let's remove the container and create new one
$ docker rm -f some-mysql
some-mysql
$docker run --name some-mysql -v ~/private/mysql-data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=secret -d mysql:5.6
docker exec -it some-mysql bash
root#6f77e071321e:/# mysql -uroot -psecret
Welcome to the MySQL monitor....
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| test |
+--------------------+
4 rows in set (0.01 sec)
Also, note mysql docker docs.

Related

Cannot find mysql docker data files on my volume local host

I created a simple mysql docker instance as
docker run -d
--name mysql3
-p 3306:3306
-v db:/var/lib/mysql
-e MYSQL_ROOT_PASSWORD=rsecret
-e MYSQL_DATABASE=mkt_data_db
mysql/mysql-server:8.0
After that I logged in and created tables
$docker exec -it mysql3 bash
#mysql -u root -p
use mkt_data_db;
create table price (ticker char(30), eod_date date, close float);
insert into price(ticker, eod_date, close) values ('xyz', curdate(), 100.1);
When I logout and check db directory, its empty
$ ls -al db/
total 0
drwxr-xr-x 2 accnt staff 64 21 Jun 23:38 .
drwxr-xr-x 6 accnt staff 192 21 Jun 23:38 ..
when I stop+rm+run docker again, I do see the the 'price' table and the data within it
Where is the database stored on my local host (mac)?
I would ideally like to move these database files (data) to another computer, provide new DB volume on docker run there and see the data on that machine.
This is an interesting problem, and kudos to you for posting the answer!
I'd be curious to know what the result of docker inspect mysql3 would show (look for the "Mounts" section in the large amount of output) when NOT specifying the full path (like in your original question).
An afterthought: I like to use pwd when creating a shared volume (assuming you are running the command in the desired directory):
docker run -d
--name mysql3
-p 3306:3306
-v $(pwd):/var/lib/mysql
-e MYSQL_ROOT_PASSWORD=rsecret
-e MYSQL_DATABASE=mkt_data_db
-e MYSQL_USER=mkt_data_usr
-e MYSQL_PASSWORD=usecret
mysql/mysql-server:8.0
The above problem was solved by providing absolute path with -v flag
i.e.
docker run -d
--name mysql3
-p 3306:3306
-v /Users/accnt/db:/var/lib/mysql
-e MYSQL_ROOT_PASSWORD=rsecret
-e MYSQL_DATABASE=mkt_data_db
-e MYSQL_USER=mkt_data_usr
-e MYSQL_PASSWORD=usecret
mysql/mysql-server:8.0
i.e. change -v db:/var/lib/mysql to -v /Users/accnt/db:/var/lib/mysql
After that $ls -l /Users/accnt/db showed my entire db contents (including mkt_data_db and price table inside

CircleCI job creates docker MySQL 8 but nothing can connect

(See UPDATE at end of post for potentially helpful debug info.)
I have a CircleCI job that deploys MySQL 8 via - setup_remote_docker+docker-compose and then attempts to start a Java app to communicate with MySQL 8. Unfortunately, even though docker ps shows the container is up and running, any attempt to communicate with MySQL--either through the Java app or docker exec--fails, saying the container is not running (and Java throws a "Communications Link Failure" exception). It's a bit confusing because the container appears to be up, and the exact same commands work on my local machine.
Here's my CircleCI config.yml:
Build and Test:
<<: *configure_machine
steps:
- *load_repo
- ... other unrelated stuff ...
- *load_gradle_wrapper
- run:
name: Install Docker Compose
environment:
COMPOSE_VERSION: '1.29.2'
command: |
curl -L "https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-$(uname -s)-$(uname -m)" -o ~/docker-compose
chmod +x ~/docker-compose
sudo mv ~/docker-compose /usr/local/bin/docker-compose
- setup_remote_docker
- run:
name: Start MySQL docker
command: docker-compose up -d
- run:
name: Check Docker MySQL
command: docker ps
- run:
name: Query MySQL #test that fails
command: docker exec -it mysql8_test_mysql mysql mysql -h 127.0.0.1 --port 3306 -u root -prootpass -e "show databases;"
And here's my docker-compose.yml that is run in one of the steps:
version: "3.1"
services:
# MySQL Dev Image
mysql-migrate:
container_name: mysql8_test_mysql
image: mysql:8.0
command:
mysqld --default-authentication-plugin=mysql_native_password
--character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
--log-bin-trust-function-creators=true
environment:
MYSQL_DATABASE: test_db
MYSQL_ROOT_PASSWORD: rootpass
ports:
- "3306:3306"
volumes:
- "./docker/mysql/data:/var/lib/mysql"
- "./docker/mysql/my.cnf:/etc/mysql/conf.d/my.cnf"
- "./mysql_schema_v1.sql:/docker-entrypoint-initdb.d/mysql_schema_v1.sql"
It's a fairly simple setup and the output from CircleCI is positive until it reaches the docker exec, which I added to test the connection. Here is what the output from CircleCI says per step:
Start MySQL Docker:
#!/bin/bash -eo pipefail
docker-compose up -d
Creating network "project_default" with the default driver
Pulling mysql-migrate (mysql:8.0)...
8.0: Pulling from library/mysql
5158dd02: Pulling fs layer
f6778b18: Pulling fs layer
a6c74a04: Pulling fs layer
4028a805: Pulling fs layer
7163f0f6: Pulling fs layer
cb7f57e0: Pulling fs layer
7a431703: Pulling fs layer
5fe86aaf: Pulling fs layer
add93486: Pulling fs layer
960383f3: Pulling fs layer
80965951: Pulling fs layer
Digest: sha256:b17a66b49277a68066559416cf44a185cfee538d0e16b5624781019bc716c122 121B/121BkBBB
Status: Downloaded newer image for mysql:8.0
Creating mysql8_******_mysql ...
Creating mysql8_******_mysql ... done
So we know MySQL 8 was pulled fine (and therefore the previous step worked). Next step is to ask Docker what's running.
Check Docker MySQL:
#!/bin/bash -eo pipefail
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cb6b7941ad65 mysql:8.0 "docker-entrypoint.s…" 1 second ago Up Less than a second 0.0.0.0:3306->3306/tcp, 33060/tcp mysql8_test_mysql
CircleCI received exit code 0
Looks good so far. But now let's actually try to run a command against it via docker exec.
Query MySQL:
#!/bin/bash -eo pipefail
docker exec -it mysql8_test_mysql mysql mysql -h 127.0.0.1 --port 3306 -u root -prootpass -e "show databases;"
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:3306' (111)
Exited with code exit status 1
CircleCI received exit code 1
So now we can't connect to MySQL even though docker ps showed it up and running. I even tried adding an absurd step to wait in case MySQL needed more time:
- run:
name: Start MySQL docker
command: docker-compose up -d
- run:
name: Check Docker MySQL
command: docker ps
- run:
name: Wait Until Ready
command: sleep 120
- run:
name: Query MySQL
command: docker exec -it mysql8_test_mysql mysql mysql -h 127.0.0.1 --port 3306 -u root -prootpass -e "show databases;"
Of course adding a 2 minute wait for MySQL to spin up didn't help. Any ideas as to why this is so difficult in CircleCI?
Thanks in advance.
UPDATE 1: I can successfully start MySQL if I SSH into the job's server and run the same command myself:
docker-compose up
Then in another terminal run this:
docker exec -it mysql8_test_mysql mysql mysql -h localhost --port 3306 -u root -prootpass -e "show databases;"
+--------------------+
| Database |
+--------------------+
| information_schema |
| test_db |
| mysql |
| performance_schema |
| sys |
+--------------------+
So it is possible to start MySQL. It's just not working right when through job steps.
UPDATE 2: I moved the two minute wait between docker-compose up -d and docker ps and now it shows nothing is running. So the container must be starting then crashing and that's the reason for why it's not available moments later.
The cause of the problem was the volumes entry in my docker-compose.yml with this line:
- "./mysql_schema_v1.sql:/docker-entrypoint-initdb.d/mysql_schema_v1.sql"
The container appeared to be up when I checked immediately after docker-compose up -d but in actuality it would crash seconds later because CircleCI appears to have an issue with Docker volume, potentially related to this: https://discuss.circleci.com/t/docker-compose-doesnt-mount-volumes-with-host-files-with-circle-ci/19099.
To make it work I removed that volume entry and added run commands to copy and import the schema like so:
- run:
name: Start MySQL docker
command: docker-compose up -d
# Manually copy schema file instead of using docker-compose volumes (has issues with CircleCI)
- run:
name: Copy Schema
command: docker cp mysql_schema_v1.sql mysql8_mobile_mysql:docker-entrypoint-initdb.d/mysql_schema_v1.sql
- run:
name: Import Schema
command: docker exec mysql8_mobile_mysql /bin/sh -c 'mysql -u root -prootpass < docker-entrypoint-initdb.d/mysql_schema_v1.sql'
With this new setup I've been able to create the tables and connect to MySQL. However, there appears to be an issue running tests against MySQL causing hangups but that might be unrelated. I will follow up with more information, but at least I hope this can help someone else.

Docker command through SSH from a remote machine

I'm trying to run mysql command into a docker container from a remote machine, through SSH. In particular the command is this one:
ssh user#servername /usr/bin/sudo /usr/bin/docker exec CONTAINER '/usr/bin/mysql --user=USER --password=PASSWORD -e "show databases;"'
I receive this response: ERROR 1049 (42000): Unknown database 'databases;"'
The same command (/usr/bin/sudo /usr/bin/docker exec CONTAINER /usr/bin/mysql --user=USER --password=PASSWORD -e "show databases;") launched on the machine where the container is running, works!
Someone could help me? thank you
use this command
ssh user#servername 'sudo docker exec CONTAINER mysql --user=USER --password=PASSWORD -e "show databases;"'
We have to pass the complete command (to be executed on remote shell) in quotes('').
I resolved using /usr/bin/mysqlshow instad of /usr/bin/mysql. Now it returns the database list correctly. I post the entire command i use to save the mysql docker container database list remotely:
LIST_DB=($(ssh -q USER#SERVERNAME /usr/bin/sudo /usr/bin/docker exec CONTAINER "/usr/bin/mysqlshow -u USER -pPASSWORD" 2>&1 | sed 's/|//g' | sed 's/\-//g' | sed 's/\+//g' |sed -e 's/^[[:blank:]]*//g' | grep -v "Using a password" | grep -Ev "_schema|tmp|innodb|sys|Databases"))

mysqldump in Jenkinsfile can't connect to server

This is the stage in Jenkinsfile where the problem comes from :
stage ('Build & Run container') {
imageMysql = docker.build('backend-server-mysql-dev', '--no-cache -f build/docker/mysql/Dockerfile .')
containerMysql = imageMysql.run("--name backend-server-mysql-dev -p 3306:3306 -e MYSQL_ROOT_PASSWORD=root -e MYSQL_ROOT_USER=root -e MYSQL_PASSWORD=mahmoud -e MYSQL_DATABASE=soextremedb")
sh 'docker ps | docker exec -it backend-server-mysql-dev /bin/bash | ls -l | mysqldump -u root -proot soextremedb < soextremedb.sql'
}
This is the error message:
Shell Script -- docker ps | docker exec -it backend-server-mysql-dev /bin/bash | ls -l | mysqldump -u root -proot soextremedb < soextremedb.sql -- (self time 566ms)
[soextremeBackEnd_Dev-MBC6SQWYSNVE6ADN2QOAOGZ4YYVT5E6K7Y2FUP6ROOROWRMCPFOA] Running shell script
+ docker ps
+ docker exec -it backend-server-mysql-dev /bin/bash
+ ls -l
+ mysqldump -u root -proot soextremedb
**mysqldump: Got error: 2002: "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")" when trying to connect the input device is not a TTY**
I think there are a couple of issues with the sh command.
First, | is used to send the output of one command on to the next command, but it looks like you're just trying to execute a sequence of commands. For that, you can use ; or &&. You might take a look at this answer for a great summary of shell operators.
Then, for your docker exec command, I think you actually want to call a series of commands non-interactively: leave off off the -it and use /bin/bash -c to pass a command string to the shell.
This will give you something like:
sh 'docker ps ; docker exec backend-server-mysql-dev /bin/bash -c "ls -l ; mysqldump -u root -proot soextremedb < soextremedb.sql"'

Accessing docker container mysql databases

I am trying to access mysql databases from my docker host to the container.
It's my own dockerfile which install a database expose on port 3306.
I launch my docker with docker-compose, and my compose file is mapping 3308 host port on 3306 container port.
I can access to mysql from the host like this :
mysql -h localhost -P 3308 -u root -pMyPassword
It's working well, but what I can't figure out, is why I can't see any datas from my container?
From inside the container, I have a test databases which I can connect to without any problem. But when I connect from the host to the container mysql process, It seems to show me the mysql datas from the host machine, not from the container one.
Any ideas?
Thanks :)
EDIT 1 :
So here is the first way I can connect to mysql into the container :
docker exec -it MyContainer mysql -uroot -pMyPassword
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| test_db |
+--------------------+
It show me my db : test_db
But If i access from :
mysql -h localhost -P 3308 -u root -pMyPassword
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
+--------------------+
My test_db isn't here.
And the result of docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a0de6a691d72 MyContainer "docker-entrypoint.sh" 3 hours ago Up 3 hours 9000/tcp, 0.0.0.0:8085->80/tcp, 0.0.0.0:3308->3306/tcp, 0.0.0.0:8084->8000/tcp, 0.0.0.0:8086->8080/tcp MyContainer
EDIT 2 :
I am developing a standard docker container for web hosting production environnement. Each host is controlled by ajenti. The host work with an nginx reverse proxy which redistribute websites on correct container. Every thing is wokring well. So here is my Dockerfile :
FROM php:5.6-fpm
RUN apt-get update && apt-get install -y --no-install-recommends \
git \
libxml2-dev \
python \
build-essential \
make \
gcc \
python-dev \
locales \
python-pip
RUN dpkg-reconfigure locales && \
locale-gen C.UTF-8 && \
/usr/sbin/update-locale LANG=C.UTF-8
ENV LC_ALL C.UTF-8
ARG MYSQL_ROOT_PASSWORD
RUN export DEBIAN_FRONTEND=noninteractive; \
echo mysql-server mysql-server/root_password password $MYSQL_ROOT_PASSWORD | debconf-set-selections; \
echo mysql-server mysql-server/root_password_again password $MYSQL_ROOT_PASSWORD | debconf-set-selections;
RUN apt-get update && apt-get install -y -q mysql-server php5-mysql
RUN rm /etc/apt/apt.conf.d/docker-gzip-indexes
RUN apt-get update && apt-get install -y wget
RUN wget http://repo.ajenti.org/debian/key -O- | apt-key add -
RUN echo "deb http://repo.ajenti.org/debian main main debian" > /etc/apt/sources.list.d/ajenti.list
RUN apt-get update && apt-get install -y ajenti cron unzip ajenti-v ajenti-v-php-fpm ajenti-v-mysql ajenti-v-nginx
RUN apt-get install -y python-setuptools python-dev \
&& easy_install -U gevent==1.1b3 \
&& sed -i -e s/ssl_version=PROTOCOL_SSLv3/ssl_version=PROTOCOL_SSLv23/ /usr/local/lib/python2.7/dist-packages/gevent-1.1b3-py2.7-linux-x86_64.egg/gevent/ssl.py
EXPOSE 80 8000 8080 3306
RUN mkdir /tmp/tempfiles \
&& mv /srv /tmp/tempfiles \
&& mv /var/lib/mysql /tmp/tempfiles \
&& mv /var/log /tmp/tempfiles \
&& mv /etc/ajenti /tmp/tempfiles
COPY docker-entrypoint.sh /usr/local/bin/
RUN ln -s /usr/local/bin/docker-entrypoint.sh /entrypoint.sh # backwards compat
ENTRYPOINT ["docker-entrypoint.sh"]
As I said, I wanted to be able do deploy a new container easily. So I created a docker-entrypoint.sh which copy wanted files to my volume when I start the container :
#!/bin/bash
DIR="/var/lib/mysql"
# look for empty dir
if [ ! "$(ls -A $DIR)" ]; then
cp -avr /tmp/tempfiles/mysql /var/lib/
fi
# rest of the logic
DIR="/srv"
# look for empty dir
if [ ! "$(ls -A $DIR)" ]; then
cp -avr /tmp/tempfiles/srv /
fi
# rest of the logic
DIR="/var/log"
# look for empty dir
if [ ! "$(ls -A $DIR)" ]; then
cp -avr /tmp/tempfiles/log /var/
fi
# rest of the logic
DIR="/etc/ajenti"
# look for empty dir
if [ ! "$(ls -A $DIR)" ]; then
cp -avr /tmp/tempfiles/ajenti /etc/
fi
# rest of the logic
Finally, my docker-compose.yml to launch everything and map ports :
version: '2'
services:
ajenti:
build:
context: ./dockerfiles/
args:
MYSQL_ROOT_PASSWORD: MyPassword
volumes:
- ./logs:/var/log
- ./html:/srv
- ./ajenti:/etc/ajenti
- ./mysql-data:/var/lib/mysql
- ./apache2:/etc/apache2
ports:
- "8084:8000"
#NGINX
- "8085:80"
#APACHE
- "8086:8080"
- "3308:3306"
Hope this will help to find a solution !
I finally found a solution and it was pretty simple...
First of all, I need to let mysql bind external address, so I changed the line bind-address to '0.0.0.0' inside the container.
Next I just changed the command line with mysql -h 127.0.0.1 -P 3308 -u root -pMyPassword
Now it's fine, I can access container mysql data from the host.
Thanks all for your help :)
In my case I was confused because docker used a different host and port. So you need to find them then do this:
mysql -P <portnumber> -h <host IP> -u db_name -p
Most people would put the docker DB related variables into the environment of the docker container so do this:
sudo docker exec -it container_name env
See if there's a variable called DB_HOST or DB_PORT or something like that. If not then look thru the source code. If it's a PHP project then find a config directory and look in main.php and see
if you execute MySQL operation as entrypoint in the dockerfile file, you will only see that operation when you connect to the container. try changing the entrypoint.
https://docs.docker.com/engine/reference/builder/#entrypoint