My company uses a product that uses MySQL 5.5 for its backend database. The product automatically installs and configures MySQL during it's installation process. The product can be configured to run in a Hot Standby Redundant configuration. In these cases, the same installation process is performed on 2 separate servers and then during the products initial configuration redundant mode is selected. The product internally handles all the processes of duplicating the database data and keeping the 2 databases in sync. MySQL has know knowledge of the redundant setup. The MySQL installation on both server are identical, same location and same structure. The product does not have a very elegant/efficient way to sync a large, say 300G is size with 3K tables, database from the Primary server to the Backup server in cases where this is required, such as when creating a redundant system from a Single/Primary server config that has already been running for a while. My question is as follows.
Is there a safe/supported way to just manually copy the database/files from the Primary server to the Backup server considering that the MySQL installation on both servers are identical? BTW, this is on Production Windows Servers. I know I can do a full Export of the database from the Primary and then Import it on the BU server, but this can take hours. I am hoping there is a faster supported way to just copy the files from one server to the other, but in researching this I see conflicting info.
System Info
Windows
MySQL 5.5
Identical installation on both servers
"C:\ProgramData\MySQL\MySQL Server 5.5\data"
Innodb
File per table = true
Thanks in advance for any advice.
I once tried to just copy the Database Folder that contains all the innodb table files, "C:\ProgramData\MySQL\MySQL Server 5.5\data\Mydbase", from one server to another but mysql would not start up and had errors.
Yes: shut down the MySQL Server service on both computers. Then you can move the files in the datadir in any way you want. But this incurs some downtime while you do the file transfer.
If you must have no downtime, it's also possible, but requires more steps.
What I do is use Percona XtraBackup to make a physical backup of the source instance, but this won't work as easily for you because you are on Windows. XtraBackup doesn't work on Windows. Some people use tricks to run XtraBackup in a Docker container on Windows.
Then restore the XtraBackup to your new computer in the normal way, and configure it as a replica of the source instance. See https://docs.percona.com/percona-xtrabackup/8.0/howtos/setting_up_replication.html
By making the new instance a replica, you can let it get updated with the most recent changes that have occurred on the source instance while you were setting up the replica.
Then at some point you decide to switch to the new instance. Then you set the source instance to read-only mode, to prevent client applications from making any new changes. Let the replica catch up with the last final changes (this should only take a second if the replica was keeping up with changes already). Now you can change your client applications to use the replica instead of the former source. Then un-configure replication on the new instance with RESET SLAVE because the last thing you want is for any more changes to occur on the former source and replicate to the new instance.
If you try this procedure, I suggest you test it on a test instance — NOT your production instance — until you are comfortable with the tools.
P.S.: In addition to not supporting Windows, I have no idea if the current version of XtraBackup works with MySQL 5.5. That version was released in 2010, and reached its end of life in 2018. So I think you will need to research which version of XtraBackup still can read a MySQL 5.5 instance. You might have to use an old version of XtraBackup.
Related
In my setup there are two debian servers. The first one is the old production server and the second is the new one. On the first (old) one runs a mysql v5.5 db-server and an old application which lags support. It cannot be ported easily to the new server. The new server runs mariadb v10.1 and all the other applications were ported from the old server to this new one. These applications have to work also with the data of the application that cannot be ported.
The ported application can only access local databases. So there is no easy way of changing the connection for these apps to the old db server.
My idea:
I want to replicate (master->slave) the data of the one database (used by the old application that is not portable) of the mysql v5.5 db server to the maraidb v10.1 db-server.
No problem so far.
But the applications on the new server not only read the data of the old application, they can also modify them. And they also have there own databases that only exists on the new server. This is a problem as far as I know and can lead to the break of the replication in some situations if the applications would try to write at the replicated database on the slave.
My next thought to solve this was that I can make use of a sql dispatcher proxy and found some interesting ones (mariadb maxscale, haproxy, proxySQL) but as far as I understood they can split read and write operations but I couldn't find a way to route write operations for different databases to different servers.
Can Anybody give me a hint to solve this problem?
Setting:
Server 1 - Mysql v5.5 - database_1
Server 2 - Mariadb v10.1 - database_1, database_2, database_3
An application on server 1 is writing and reading data from database_1 on server 1.
Other applications on server 2 are reading and writing data to database_1 on server 2.
So the data of database_1 have to be replicated from server 1 to server 2 and could be changed there.
A master-master replication instead of master-slave could work, but in reason of auto_increment fields that could break the replication and in reason of the fact that the changed data from server 2 doesn't have to exist on server 1, I think this is not the way to go. (I'm aware that I could set the auto_increment interval to two to avoid this problem, but it's an already running production system, so changes like this are not so easy).
At the moment we're doing backups by hand and copy them over but that's way to slow and I'm sure there is a better way ;)
You can use write to a replication slave (server 2) for databases like database_2 and database_3 that will never appear in the replication scream.
If you started updating database_1 you probably would end up in trouble.
You are replicating between two database server of over a major version difference so there is the possibility that a deprecated SQL statement gets replicated to a server that has it removed and the replication will stop. Keep an eye out for this in the weeks after deployment. binlog_format=ROW may mitigate some of the SQL that could got incorrectly.
I have a MySQL DB running in Ubuntu Server (a live server), and it goes well. But I copy the DB folder for development and will run in Windows. When I just copied that DBfolder into my windows-based XAMPP, it doesn't go well. Some table error and "in use" error info on the collation column. The error table engine is InnoDB, and the rest is MyISAM. I'm wondering why it happens.
You shouldn't be coping any folder there is a export and import utility for migrating databases from one system database to another system.
For InnoDB, there are three choices:
Copy the entire tree, not just one table or database. And do it with MySQL shutdown. This does not allow for any kind of mix and match of tables.
Replication -- One server is the Master, the Other is the Slave. But thin, all writes must go to the Master. And it provides only for maintaining consistency, not for initially providing it.
"Transportable tablespaces". This is a way to disconnect a single table (or partition) so that you can copy a file to another server. Then you perform some other magic to connect the table on the other server.
https://dev.mysql.com/doc/refman/5.6/en/tablespace-copying.html
https://dev.mysql.com/doc/refman/5.7/en/innodb-transportable-tablespace-examples.html
I have been doing some research on best backup procedures for largish (27.678gb) MYSQL database tables.
Currently we are using a program called Rapidsync (which is a offsite backup tool) but it is slow and it locks the tables it's currently backing up therefore causing downtime/slowness of sql.
Our current server is running Windows 2008 r2 with SQL server 2008 (on the same box) also.
Hardware specs for the dedicated server are:
16gb Ram
CPU intel xeon E3-1230 V2 # 3.30GHz
1 TB hard drive
In terms of databases we have 58 in total varying in size in which some need to backed up weekly ideally or even daily.
Through a program we use called Navicat you can tunnel to a database using SSH and copy databases manually, is this a reliable and feasible option if we were to install it on our local machine and copy them across? Or would it be more secure/efficient to use SQL Dump maybe?
I hope I have given all of the necessary info but please do ask if you need to know more.
P.S Only free options at this moment as we are on a tight budget! :)
Thanks in advance
You mentioned SSH, so I suppose the backups can be done also on other server than the database server itself. For Unix, you can use great tool Percona xtrabackup, which is free and supports online (without locking) backup of InnoDB database files and also incremental backups. Maybe it is possible to compile it also on Windows (part of it is written in C, part in Perl).
So you can setup weekly full backup and daily incremental backups. The tool will keep track of which pages of the InnoDB datafile has been changed and will copy only those.
I launched an Amazon EC2 with Amazon Linux and Amazon-EBS as root volume. I also started tomcat7 and mysql 5.5 on this EBS volume.
Later I decided to change from Amazon Linux to Ubuntu. To do that I need to launch another Amazon EC2 instance with a new EBS root volume. Now I want to copy tomcat7 and mysql from older EBS volume to new one. I have tables and data in mysql which I don't want to loose and an application running on tomcat. How to go about it?
A couple of thoughts and suggestions.
First, if you are going to be having any kind of significant load on your database, running it on EBS-backed volume is probably not a great idea as EBS-backed storage is incredibly slow relative to the machine's local/ephemeral storage (/mnt). Now obviously you don't want DB data on ephemeral storage, so there is really nothing you can do about it if you want to run MySQL on EC2. So my suggestion would be to utilize an RDS instance for your DB if your infrastructure requirements allow for it.
Second, if this is a production application, you are undoubtedly going to have some down time as you make this transition. The question is whether you need to absolutely minimize the amount of downtime. If so, then you need to have an idea as to the size of your database. Is it going to take a long time to dump/load? If not, you could probably just get your new instance up and running, and tested on an older copy of your database and then just dump and load the current database at the time of cutover.
If it is a large database then perhaps you can turn on MySQL binary logging. Then make a dump of the database at a known binary log position. Then install this dump on your new instance. Then when ready to cutover, you can replay the binary logs on the new instance to bring it current. Similarly, you could just set up the DB on the new instance as a replica until the cutover, at which point you make it the master.
You may even consider just using rsync to sync the physical database files if you don't want to mess with binary logging, though this can be a problematic approach if you are not that familiar with dealing with the actual physical database files.
As far as your application goes, that should be much simpler to migrate assuming it is just a collection of files. I would not copy the Tomcat7 installation itself, but rather just install Tomcat on Ubuntu and then adjust the configuration to match current.
As far as the cutover itself goes, this should be pretty straightforward and would vary in approach depending on whether you are using an elastic IP for your server or whether it is behind a load balancer,
I have a MySQL install on a shared server and have access through phpMyAdmin. I want to make a continuous, real time clone of that database to a cloud mySQL database (we have created an Nginx-ready MySQL server specially for this database) I want to create a real time clone of the old one, then update code to point to the new database...
I think you will have difficulty doing real-time replication of a MySQL in a shared server environment. Since you appear to be moving db servers, I would be inclined to do a hot copy of your data, and install that on the new db server. At the same time as taking that copy, you should switch on query logging on your application.
Your switch over would then consist of running logged queries against the new database (faster than they were logged!) and finally, at a point that all logged queries have been run, switching the configuration of the app so that the new db is used.
Edit: the problem with a hot copy is that data is being written to the db at the same time as it is being copied. That means that the 'last updated' time will be different for each table. On that basis, is it possible in your application to set up a 'last_updated' column for each row? If so you will be able to tell for each table which logged queries still need to be copied.
What you're looking for is replication. It has far to many options to cover here in a single post.
http://dev.mysql.com/doc/refman/5.5/en/replication.html
If your going to do replication over the internet you'll want to secure it.Your host might allow a virtual local area network So this doesn't use up your bandwidth resources.
A great set of tools from percona you should look at are maatkit
https://launchpad.net/percona-toolkit
Documentation and usage examples
http://www.maatkit.org/doc/
It's good for other tasks but it also allows you to replicate a live database quickly.
When your working with live databases make sure your backups are upto date.