I am importing a large application to Docker. Part of the application is a database. I have dumped the database into .sql file and now I am trying to import it to the docker container running official mysql image by mounting a directory from host machine and issuing command
mysql -u myUsername -p myDB < /mountdir/databasedump.sql
The database dump is very large, more than 10GB. Everything goes well for an hour, but then it issues error
loop: Write error at byte offset
I have a feeling that the size of the container runs out.
Is there a smarter way in accomplishing the dockerization of the database? In case not, how can I import the enormous database to the container?
Don't use a devicemapper loop file for anything serious! Docker has warnings about this.
For a database that size you may want try mounting a host directory as a volume or creating a local volume to avoid docker's file system overhead. The two are essentially the same thing underneath.
If you are on RHEL, you can use an adequately sized LVM thin pool directly. There's a process in this answer for changing storage drivers.
The problem was that any container couldn't store the whole imported database, which was 26Gb and the error was due to the container ran out of disk space.
I solved the problem by mounting a directory from host as a external volume using the -v switch and editing the MySQL config to store its databases there.
This solution of course 'un-virtualizes' the database and it might be a security risk. Database server runs well virtualized though. In my situation the slightly weakened security wasn't an issue.
Related
Use Case: A MySQL instance will be running on production with required databases. I need to configure this running container as a master and spin up one more MySQL instance as a slave.
Challenges: Facing issues in configuring running MySQL instance as a master. The issue is not able to create replication user and not able to append the master/slave configuration to my.cnf file. The reason is,
To create replication user or to execute any custom SQL commands in container, we have to place initdb.sql with required SQL commands inside docker-entrypoint-initdb.d. So when container starting it execute the file present in docker-entrypoint-initdb.d and executes it, if the database had not created, if the database had created already it skips executing this .sql file residing in docker-entrypoint-initdb.d. This is the root for failing to configure master because MySQL is running with databases in production. So I cannot take this solution to configure as MySQL.
After facing this issue we planned to put the configuration SQL commands in .sh and keep in docker-entrypoint-initdb.d and execute them by patching the deployment. In this scenario we are facing some permission issues when executing the .sh files.
I need to configure replication(master-slave) for MySQL instance(s) in kubernetes world. I gone through the lot of posts to understand how to implement this. Nothing worked out as I am expecting and as I explained above. Along with this I found a custom image(bitnami/mysql) which supports setting up the replication, which I don't want use because finally I would not be able to implement this in production env.
So it will be very grateful if anyone helps me by suggesting any approach to solve this problem.
Thank you very much in advance.!!!
I want to create containers w/ a MySQL db and a dump loaded for integration tests. Each test should connect to a fresh container, with the DB in the same state. It should be able to read and write, but all changes should be lost when the test ends and the container is destroyed. I'm using the "mysql" image from the official docker repo.
1) The image's docs suggests taking advantage of the "entrypoint" script that will import any .sql files you provide on a specific folder. As I understand, this will import the dump again every time a new container is created, so not a good option. Is that correct?
2) This SO answer suggests extending that image with a RUN statement to start the mysql service and import all dumps. This seems to be the way to go, but I keep getting
mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
followed by
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
when I run build, even though I can connect to mysql fine on containers of the original image. I tried sleep 5 to wait for the mysqld service to startup, and adding -h with 'localhost' or the docker-machine ip.
How can I fix "2)"? Or, is there a better approach?
If re-seeding the data is an expensive operation another option would be starting / stopping a Docker container (previously build with the DB and seed data). I blogged about this a few months ago Integration Testing using Spring Boot, Postgres and Docker and although the blog focuses on Postgres, the idea is the same and could be translated to MySQL.
The standard MySQL image is pretty slow to start up so might be useful to use something that has been prepared more for this situation like this:
https://github.com/awin/docker-mysql
You can include data or use with a Flyway situation too, but it should speed things up a bit.
How I've solved this before is using a Database Migration tool, specifically flyway: http://flywaydb.org/documentation/database/mysql.html
Flyway is more for migrating the database schema opposed to putting data into it, but you could use it either way. Whenever you start your container just run the migrations against it and your database will be setup however you want. It's easy to use and you can also just use the default MySQL docker container without messing around with any settings. Flyway is also nice for many other reasons, like having a way to have version control for a database schema, and the ability to perform migrations on production databases easily.
To run integration tests with a clean DB I would just have an initial dataset that you insert before the test, then afterwards just truncate all the tables. I'm not sure how large your dataset is, but I think this is generally faster than restarting a mysql container every time,.
Yes, the data will be imported every time you start a container. This could take a long time.
You can view an example image that I created
https://github.com/kliewkliew/mysql-adventureworks
https://hub.docker.com/r/kliew/mysql-adventureworks/
My Dockerfile builds an image by installing MySQL, imports a sample database (from a .sql file), and sets the entrypoint to auto-start MySQL server. When you start a container from this image, it will have the data pre-loaded in the database.
For some technical reasons, I have to migrate our actual server, running on CentOS, to a new server, but running on Ubuntu.
I MUST keep the SQL DB as it is now, same version of MySQL, etc.
I have tried several dump scripts and methods, but impossible to import the dumps to a newer version of MySQL.
So I'm thinking of copying the /var/lib/MySQL folder and other MySQL folders to the new server.
What's your opinion about this ? Will it work ?
I was also thinking of going to the recovery console of the old server and doing an rsync to the new server, this way I copy the entire system to the new server.
But that's a bit heavy in my opinion. and I don't know Unix enough to remount discs and perform all tasks that would make the copy work.
There are actually several ways to do this. The first and by far the best is what the commenters have said. Use mysqldump to create a sql file then import it onto the other machine. If space is an issue on one of the machines you can import more directly like so: mysqldump {YOUR OPTIONS HERE} | mysql -h {NEWSERVER} -u root -p
You should do the above method if at all possible. It will work better 100% of the time when doing cross version imports. Just fix the errors you see. You may need to rewrite triggers and stored procedures if the syntax differs across versions.
If you know the danger and are ok with losing your data, you can try the following xtrabackup method:
Install xtrabackup on the older server (http://www.percona.com/software/percona-xtrabackup)
Get a full and consistent backup from xtrabackup (read the docs)
Shutdown mysql on the new server
Copy the backup from xtrabackup to that new server
Start up mysql again, it may take a while as it upgrades the tables
I have done this before across mysql server versions, BUT this is not reliable, and you quite possibly could end up with unreliable or corrupted data. Do it at your own risk.
During development, how our local WAMP servers get up-to-date data from the test server is that a dump of the database is made and we upload that dump using the source command to load the .sql file.
Recently, at the very end of the import we have been getting errors about the #old variables which stored the original settings like foreign key constraints before they’re changed (so turning off foreign key constraints so that the import doesn’t throw errors when it recreates tables and attempts to create foreign keys when one of the tables has yet to be created). I have worked out that the cause is that the product table is getting more and more data and at a point the session has timed out during the import.
I’m wondering what setting can I set (either as part of the SQL query on in the my.ini file) that will stop all timeouts, in effect making a session last forever while we are signed in.
Strategies for importing large MySQL databases
PHPMyAdmin Import
Chances are if you’re reading this, PHPMyAdmin was not an option for your large MySQL database import. Nonetheless it is always worth a try, right? The most common cause of failure for PHPMyAdmin imports is exceeding the import limit. If you’re working locally or have your own server, you can try changing the MySQL ini settings usually found in the my.ini file located in the MySQL install folder. If you’re working with WAMP on Windows, you can access that file using the WAMP control panel under MySQL > my.ini. Remember to restart WAMP so your new settings will be used. Settings you may want to increase here include:
max_allowed_packet
read_buffer_size
Even with enhanced MySQL import settings you may still find that imports time out due to PHP settings. If you have access to PHP.ini, you can make edits to the maximum execution time and related settings. In WAMP, access the PHP.ini file under the WAMP control panel at PHP > php.ini. Consider raising the limits on the following settings while trying large MySQL imports:
max_execution_time
max_input_time
memory_limit
Using Big Dump staggered MySQL dump importer
If basic PHPMyAdmin importing does not work, you may want to try the Big Dump script from Ozerov.de for staggered MySQL imports. What this useful script does is run your import in smaller blocks, which is exactly what is often needed to successfully import a large MySQL dump. It is a free download available at http://www.ozerov.de/bigdump/.
The process of using Big Dump is fairly simple: you basically position your SQL import file and the Big Dump script together on the server, set a few configs in the Big Dump script and then run the script. Big Dump handles the rest!
One key point about this otherwise great option, is that it will not work at all on MySQL exports that contain extended inserts. So if you have the option to prevent extended inserts, try it. Otherwise you will have to use another method for importing your large MySQL file.
Go command line with MySQL console
If you’re running WAMP (and even if you’re not) there is always the option to cut to the chase and import your large MySQL database using the MySQL console. I’m importing a 4GB database this way as I write this post. Which is actually why I have some time to spend writing, because even this method takes time when you have a 4GB SQL file to import!
Some developers (usually me) are intimidated by opening up a black screen and typing cryptic commands into it. But it can be liberating, and when it comes to MySQL databases it often the best route to take. In WAMP we access the MySQL console from the WAMP control panel at MySQL > MySQL Console. Now let’s learn the 2 simple MySQL Console commands you need to import a MySQL database, command-line style:
use `db_name`
Command use followed by the database name will tell the MySQL console which database you want to use. If you have already set up the database to which you are importing, then you start by issuing the use command. Suppose your database is named my_great_database. In this case, issue the following command in the MySQL Console. Note that commands must end with a semi-colon.
mysql-> use my_great_database;
mysql-> source sql_import_file.sql
Command source followed by the location of a SQL file will import the SQL file to the database you previously specified with the use command. You must provide the path, so if you’re using WAMP on your local server, start by putting the SQL file somewhere easy to get at such as C:\sql\my_import.sql. The full command with this example path would be:
mysql-> source C:\sql\my_import.sql;
After you run that command, the SQL file should begin to be imported. Let the queries run and allow the import to complete before closing the MySQL console.
Further documentation for MySQL command line can be found here: http://dev.mysql.com/doc/refman/5.5/en/mysql.html.
Another solution is to use MySQL Workbench.
This solution worked for me:
max_allowed_packet <-- --> upped size to 8M
read_buffer_size <-- --> upped from 256 to 512
Using Xampp control panel on localhost. After making the changes to the my.ini file in MySQL config, don’t forget to quit Xampp (or Wamp) and restart it for changes to take effect.
(Four days of head-banging and I finally got it fixed!)
Symptoms were on Import: #2006 MySql server went away. However, only 10 table rows were being imported out of 87 table rows.
Consider using MySQL Workbench, it's free and handles very large script very well (from the menu choose: File -> Open SQL Script - if it's large, it will ask you if you'd like run it). Has served me well over the years when working with large SQL dumps.
I have a test database on a separate remote server than my production DB. Every once in awhile, I want to try and test things by uploading a copy of my production DB to my testing DB. Unfortunately, the backup file is now half a gig and I'm having trouble transferring it via FTP or SSH. Is there an easy way that I can use the mysql restore command between servers? Also, is there another way to move over large files that I'm not considering? Half a gig doesn't seem that big, I would imagine that people run into this issue frequently.
Thanks!
Are the servers accessible to each other?
If so, you can just pipe the data from one db to another without using a file.
ex: mysqldump [options] | mysql -h test -u username -ppasswd
0.Please consider whether you really need production data (especially if it contains some sensitive information)
1.The simplest solution is to compress the backup on the source server (usually gzip), transfer it across the wire, then decompress on the target server.
http://www.techiecorner.com/44/how-to-backup-mysql-database-in-command-line-with-compression/
2.If you don't need the exact replica of production data (e.g. you don't need some application logs, errors, some other technical stuff) you can consider creating a backup and restore on a source server to a different DB name, delete all unnecessary data and THEN take a backup that you will use.
3.Restore full backup once on your reference server in your Dev environment and then copy transaction logs only (to replay them on the reference server). Depending on the usage pattern transaction logs may take a lot less space as the whole database.
Mysql allows you to connect to a remote database server to run sql commands. Using this feature, we can pipe the output from mysqldump and ask mysql to connect to the remote database server to populate the new database.
mysqldump -u root -p rootpass SalesDb | mysql --host=185.32.31.96 -C SalesDb
Use an efficient transfer method, rather than ftp.
If you have a dump file created by mysqldump, on the test db server, and you update it every so often. I think you could save time (if not disk space) by using rsync to transfer it. Rsync will use ssh and compress data for the transfer, but I think both the local and remote files should/could be uncompressed.
Rsync will only transfer the changed portion of a file.
It may take some time to decide what, precisely, has changed in a dump file, but the transfer should be quick.
I must admit though, I've never done it with a half-gigabyte dump file.