Use Case: A MySQL instance will be running on production with required databases. I need to configure this running container as a master and spin up one more MySQL instance as a slave.
Challenges: Facing issues in configuring running MySQL instance as a master. The issue is not able to create replication user and not able to append the master/slave configuration to my.cnf file. The reason is,
To create replication user or to execute any custom SQL commands in container, we have to place initdb.sql with required SQL commands inside docker-entrypoint-initdb.d. So when container starting it execute the file present in docker-entrypoint-initdb.d and executes it, if the database had not created, if the database had created already it skips executing this .sql file residing in docker-entrypoint-initdb.d. This is the root for failing to configure master because MySQL is running with databases in production. So I cannot take this solution to configure as MySQL.
After facing this issue we planned to put the configuration SQL commands in .sh and keep in docker-entrypoint-initdb.d and execute them by patching the deployment. In this scenario we are facing some permission issues when executing the .sh files.
I need to configure replication(master-slave) for MySQL instance(s) in kubernetes world. I gone through the lot of posts to understand how to implement this. Nothing worked out as I am expecting and as I explained above. Along with this I found a custom image(bitnami/mysql) which supports setting up the replication, which I don't want use because finally I would not be able to implement this in production env.
So it will be very grateful if anyone helps me by suggesting any approach to solve this problem.
Thank you very much in advance.!!!
Related
I can't seem to get Flask to migrate my models. I'm following along with the Flask Mega-Tutorial series. At the database setup part of the tutorial, I'm just trying to substitute MySQL for SQLite that is used in the tutorial.
I followed SQLAlchemy's instructions for connecting to a MySQL database, and I've included mysqlclient in my pipfile.
But when I run:
flask db init
flask db migrate
I get the following:
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
ERROR [root] Error: Can't locate revision identified by '7bb962b87f19'
I've tried deleting the migrations folder, and I've deleted the environment and recreated to see if that would fix anything. But it keeps saying it can't locate revision. I thought maybe it did something with the database, I've read other solutions saying you have to flush the database. But, there aren't any tables created or any schemas in the database.
Other info:
DATABASE_URL="mysql://root:mypassword#localhost/flask_tutorial"
SQLite database runs fine and makes the migrations. So it must be something I've done with the mysql setup.
Any suggestions or ideas that may lead me in the right direction would be greatly appreciated.
The session was storing incorrect environmental variables exported. Had to completely restart my machine for the settings to reset. I even tried unset for the variables, which didn't remove the variables. Once Flask was able to pick up the correct variable settings for my configuration everything worked as planned.
Please make sure you have used this "SQLALCHEMY_TRACK_MODIFICATIONS= True"
Did you install MySqldb lib. to connect with your mysql db ?
I had the same issue. I had deleted the migrations folder but hadn't done the same for the app.db file. So, every time I tried to create any db migrations, it failed because it was using the old (and still existing) app.db file.
This is how my database was configured:
class Config(object):
# Database configuration
SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL') or \
'sqlite:///' + os.path.join(basedir, 'app.db')
SQLALCHEMY_TRACK_MODIFICATIONS = False
Deleting the app.db file solved the error. Subsequent db migrations and upgrades worked well.
It started with installation of wampserver for php, before that I had already running MySQL server 5.7 with all databases properly showing in the side schema panel. When I launched MySQL server after installation of wampserver 5.7.14, the password for root was overridden and no database except sys is showing. Also I found another user (mysql.sys) has been created (found in Users and Privileges). Then I looked for databases in path "ProgramData\MySQL\MySQL Server 5.7\Data", all databases are present in the folder (thank god they are not deleted). Problem is now for my website I get this error:
"Unable to update database to match the current model because there are pending changes and automatic migration is disabled. Either write the pending model changes to a code-based migration or enable automatic migration. Set DbMigrationsConfiguration.AutomaticMigrationsEnabled to true to enable automatic migration"
Exception Details: System.Data.Entity.Migrations.Infrastructure.AutomaticMigrationsDisabledException: Unable to update database to match the current model because there are pending changes and automatic migration is disabled. Either write the pending model changes to a code-based migration or enable automatic migration. Set DbMigrationsConfiguration.AutomaticMigrationsEnabled to true to enable automatic migration.
I know this is not related to migrations because my database already exists but I don't know how to solve the problem that MySQL can see the databases in its "ProgramData\MySQL\MySQL Server 5.7\Data" again.
When you installed WAMPServer on a system that already had a MYSQL Server instance installed on it, you created a second MySQL Server instance. See your service you will have one called wampmysqld or wampmysqld64 and another called MYSQL.
Now, whichever service you start first will be the MySQL Server that Workbench, or anything else is talking to. And you will only see the databases that this MySQL Server instance knows about.
Correct Solution:
Stop WAMPServer
Start the MYSQL installed with Workbench
Backup all YOUR databases, NOT the ones created by MySQL like mysql, sys, information_schema, performance_schema
Stop this MYSQL Server.
Uninstall this MySQL Server
Start WAMPServer
Restore your databases to this MySQL Server instance
All should start working as expected now.
Connect your Workbench to WAMPServers MySQL Server
Update
Did a robust workaround, though not very suggestible but database is now accessible in MySQL workbench. Though I'm having the same error when opening the website on local web server. One thing that I did not mention before that environment is Windows based. what I did is copied database from the ProgramData MySQL path to "..\wamp64\bin\mysql\mysql5.7.14\data" path and refreshed schema in workbench. Any advice why I'm still having problem in web server.
I'm working on changing my environment from vagrant to docker and I came across one hitch. With vagrant I have a bash file that will pull data from ftp and restore it in my local so that I can work with the most up to date data.
This is my code
php artisan db:restore --database=mysql --source=ftp --sourcePath=$(date +'%Y')"/"$(date +'%m')"/"$(date +%m-%d-%Y -d "yesterday")".gz" --compression=gzip
php artisan migrate
Inside of my work container I run this and it wont find the mysql command because mysql is in a different container. What can I do to fix my work flow?
This answer is about rails migrations, but you could take a similar approach to laravel. Create a container that has the required laravel tools and required MySQL tools (likely this) to connect to the database.
You could then combine it with a suggestion in this answer and create a migration script that you can add to your image. As larsks points out in the comments, it would allow you to encapsulate the logic for where to get the backup data and how to restore it in one place. Assuming you named this script restore, you could run your restore command like this
docker run myimage restore
I am importing a large application to Docker. Part of the application is a database. I have dumped the database into .sql file and now I am trying to import it to the docker container running official mysql image by mounting a directory from host machine and issuing command
mysql -u myUsername -p myDB < /mountdir/databasedump.sql
The database dump is very large, more than 10GB. Everything goes well for an hour, but then it issues error
loop: Write error at byte offset
I have a feeling that the size of the container runs out.
Is there a smarter way in accomplishing the dockerization of the database? In case not, how can I import the enormous database to the container?
Don't use a devicemapper loop file for anything serious! Docker has warnings about this.
For a database that size you may want try mounting a host directory as a volume or creating a local volume to avoid docker's file system overhead. The two are essentially the same thing underneath.
If you are on RHEL, you can use an adequately sized LVM thin pool directly. There's a process in this answer for changing storage drivers.
The problem was that any container couldn't store the whole imported database, which was 26Gb and the error was due to the container ran out of disk space.
I solved the problem by mounting a directory from host as a external volume using the -v switch and editing the MySQL config to store its databases there.
This solution of course 'un-virtualizes' the database and it might be a security risk. Database server runs well virtualized though. In my situation the slightly weakened security wasn't an issue.
I want to create containers w/ a MySQL db and a dump loaded for integration tests. Each test should connect to a fresh container, with the DB in the same state. It should be able to read and write, but all changes should be lost when the test ends and the container is destroyed. I'm using the "mysql" image from the official docker repo.
1) The image's docs suggests taking advantage of the "entrypoint" script that will import any .sql files you provide on a specific folder. As I understand, this will import the dump again every time a new container is created, so not a good option. Is that correct?
2) This SO answer suggests extending that image with a RUN statement to start the mysql service and import all dumps. This seems to be the way to go, but I keep getting
mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
followed by
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
when I run build, even though I can connect to mysql fine on containers of the original image. I tried sleep 5 to wait for the mysqld service to startup, and adding -h with 'localhost' or the docker-machine ip.
How can I fix "2)"? Or, is there a better approach?
If re-seeding the data is an expensive operation another option would be starting / stopping a Docker container (previously build with the DB and seed data). I blogged about this a few months ago Integration Testing using Spring Boot, Postgres and Docker and although the blog focuses on Postgres, the idea is the same and could be translated to MySQL.
The standard MySQL image is pretty slow to start up so might be useful to use something that has been prepared more for this situation like this:
https://github.com/awin/docker-mysql
You can include data or use with a Flyway situation too, but it should speed things up a bit.
How I've solved this before is using a Database Migration tool, specifically flyway: http://flywaydb.org/documentation/database/mysql.html
Flyway is more for migrating the database schema opposed to putting data into it, but you could use it either way. Whenever you start your container just run the migrations against it and your database will be setup however you want. It's easy to use and you can also just use the default MySQL docker container without messing around with any settings. Flyway is also nice for many other reasons, like having a way to have version control for a database schema, and the ability to perform migrations on production databases easily.
To run integration tests with a clean DB I would just have an initial dataset that you insert before the test, then afterwards just truncate all the tables. I'm not sure how large your dataset is, but I think this is generally faster than restarting a mysql container every time,.
Yes, the data will be imported every time you start a container. This could take a long time.
You can view an example image that I created
https://github.com/kliewkliew/mysql-adventureworks
https://hub.docker.com/r/kliew/mysql-adventureworks/
My Dockerfile builds an image by installing MySQL, imports a sample database (from a .sql file), and sets the entrypoint to auto-start MySQL server. When you start a container from this image, it will have the data pre-loaded in the database.