I'm working on a project that has a MySQL transactional database backing up a web application. The company uses SQL Server for back office and reporting applications. What is the best way to update SQL Server with the data from MySQL? Right now, we are performing a dump of the MySQL data and doing a full restore. This may not be feasible much longer due to the increasing size of the database.
I would prefer a solution that copies only newly inserted and updated rows. I also need the SQL Server database to be static after the updates are applied. Basically, it should change once a day. I can update SQL Server from a local copy of MySQL (i.e. not production) Is there a way to apply MySQL replication to a slave server at specified intervals? A perfect solution is to run a once daily update on MySQL that syncs the database as of a point in time.
Can you find a way to snapshot the mySQL DB and then do the copy? It would make an instant logical copy of the database which would be frozen in time.
http://aspiringsysadmin.com/blog/2007/08/13/consistent-mysql-backups-using-zfs-snapshots/
ZFS filesystem can do this - but you haven't mentioned your hardware/OS.
Also, perhaps you could restrict the data you are pulling - whatever is time sensitive so that your pull will only get data that is older than 1 hour if your pull takes 45 minutes. Or to make things a little safer - how about just pulling the day before?
I believe SSIS 2008 has a new module called 'maintain' table that does the common task of getting updated/inserted records and optionally deletes.
Look into DTS, Microsoft's ETL tool. It's rather nice. Do the mapping, schedule it as a cron job, and Bob's your uncle.
Regardless of how you do the import to SqlServer from the MySQL clone, I don't think you need to worry about restricting MySQL replication to specific times.
MySQL replication only requires one thread in the master server and basically just transfers the transaction log to the slave. If you can, put the master and slave MySQL servers on a private LAN segment so that replication traffic does not impact the web traffic.
if you have SQL Server Standard or higher, SQL Server will take care of all of your needs.
use ssis to grab the data
use agent to schedule your timed tasks
btw - I'm doing the exact same thing that you are doing. SQL Server is awesome - it was easy to setup (I'm a noob to SSIS) and it worked on the first shot.
It sounds like what you need to do is to set up a script to start and stop replication on a slave database. If you can do that via a script, then you can establish a workflow in SSIS such as follows:
Stop Replication to Slave MySQL Database
If Replication has Stopped, then Take Snapshot of Slave MySQL Database
If Snapshot has been Taken, then
a= Start Replication to Slave MySQL Database
b= Import Slave MySQL Database Replica into SQL Server
NB: 3a and 3b can run in parallel.
I think your best bet in such a scenario would be to use SSIS to enable and disable MySQL database replication to the slave as well as to take a snapshot of the slave database. Then you can drive the whole thing from the SQL Server Agent mechanism.
Hope this helps
Related
In my setup there are two debian servers. The first one is the old production server and the second is the new one. On the first (old) one runs a mysql v5.5 db-server and an old application which lags support. It cannot be ported easily to the new server. The new server runs mariadb v10.1 and all the other applications were ported from the old server to this new one. These applications have to work also with the data of the application that cannot be ported.
The ported application can only access local databases. So there is no easy way of changing the connection for these apps to the old db server.
My idea:
I want to replicate (master->slave) the data of the one database (used by the old application that is not portable) of the mysql v5.5 db server to the maraidb v10.1 db-server.
No problem so far.
But the applications on the new server not only read the data of the old application, they can also modify them. And they also have there own databases that only exists on the new server. This is a problem as far as I know and can lead to the break of the replication in some situations if the applications would try to write at the replicated database on the slave.
My next thought to solve this was that I can make use of a sql dispatcher proxy and found some interesting ones (mariadb maxscale, haproxy, proxySQL) but as far as I understood they can split read and write operations but I couldn't find a way to route write operations for different databases to different servers.
Can Anybody give me a hint to solve this problem?
Setting:
Server 1 - Mysql v5.5 - database_1
Server 2 - Mariadb v10.1 - database_1, database_2, database_3
An application on server 1 is writing and reading data from database_1 on server 1.
Other applications on server 2 are reading and writing data to database_1 on server 2.
So the data of database_1 have to be replicated from server 1 to server 2 and could be changed there.
A master-master replication instead of master-slave could work, but in reason of auto_increment fields that could break the replication and in reason of the fact that the changed data from server 2 doesn't have to exist on server 1, I think this is not the way to go. (I'm aware that I could set the auto_increment interval to two to avoid this problem, but it's an already running production system, so changes like this are not so easy).
At the moment we're doing backups by hand and copy them over but that's way to slow and I'm sure there is a better way ;)
You can use write to a replication slave (server 2) for databases like database_2 and database_3 that will never appear in the replication scream.
If you started updating database_1 you probably would end up in trouble.
You are replicating between two database server of over a major version difference so there is the possibility that a deprecated SQL statement gets replicated to a server that has it removed and the replication will stop. Keep an eye out for this in the weeks after deployment. binlog_format=ROW may mitigate some of the SQL that could got incorrectly.
I am trying to configure MySQL databases using the Master-Slave replication. Before I realized that I had to set up my environment using this replication, I already have 2 separate servers running their own MySQL DB. Each of these servers are configured the exact same. The MySQL DB are configured with hundreds of tables.
Is there a way that i can set up (Master-Slave) Replication using the configured DB's? Or will i have to start from scratch and configure the replication first and then load in all the DB tables?
You can delete all data from one of the servers. Remaining one with the data will be your Master. Then use mysqldump to backup all the data and insert it to the slave.
Take a look for the detailed instructions on the page below:
https://livecaller.io/blog/how-to-set-up-mysql-master-slave-replication/
If the data is exactly same in both the MySQL database then you can start master slave replication, but you need to be sure that the data is same. MySQL will not check that, and if there is some discrepancy in the primary key then it will throw error immediately after next DML statement.
To be on a safer side, drop the database from one server, and restore it using the MySQL dump of another server. This will give the surety that database is same on both the server.
Take the reference from the below link to establish replication between two MySQL servers.
https://www.digitalocean.com/community/tutorials/how-to-set-up-master-slave-replication-in-mysql
I maintain a server that runs daily cron jobs to aggregate data sources and generate reports, accessible by a private Ruby on Rails application.
One of our data sources is a partial dump of one of our partner's databases. The partner runs an active application and the MySQL DB has hundreds of tables. They have given us read-only access to a relatively underpowered readonly slave of their application DB.
Because of latency issues and performance bottlenecking on their slave DB, we have been maintaining a limited local copy of their DB. We only need about 20 tables for our reports, so I only dump those tables. We also only need the data to a daily granularity, so realtime sync is not a requirement.
For a few months, I had implemented a nightly cron which streamed the dump of the necessary tables into a local production_tmp database. Then, when all tables were imported, I dropped production and renamed production_tmp to production. This was working until the DB grew to over 25GB, and we started running into disk space limitations.
For now, I have removed the redundancy step and am just streaming the dump straight into production on our local server. This feels a bit flimsy to me, and I would like to implement a safer approach. Also, currently doing the full dump/load takes our server over 2 hours, and I'd like to implement an approach that doesn't take as long. The database will only keep growing, so I'd like to implement something future proof.
Any suggestions would be appreciated!
I take it you have never heard of, or considered MySQL Replication?
The idea is that you do your backup & restore once, and then configure the replica to "subscribe" to a continuous stream of changes as they are made on the primary MySQL instance. Any change applied to the primary is applied automatically to the replica within seconds. You don't have to do the backup & restore procedure again, unless the replica gets damaged.
It takes some care to set up and keep working, but it's a much more efficient method of keeping two instances in sync.
#SusannahPotts mentions hot backup and/or incremental backup. You can get both of these features for free, without paying for MySQL Enterprise using Percona XtraBackup.
You can also consider using MySQL Transportable Tablespaces.
You'll need filesystem access to run either Percona XtraBackup or MySQL Enterprise Backup. It's not possible to use these physical backup tools for Amazon RDS, for example.
One alternative is to create a replication slave in the same network as the live system, and run Percona XtraBackup on that slave, where you do have filesystem access.
Another option is to stream the binary logs to another host (see https://dev.mysql.com/doc/refman/5.6/en/mysqlbinlog-backup.html) and then transfer them periodically to your local instance and replay them.
Each of these solutions has pros and cons. It's hard to recommend which solution is best for you, because you aren't sharing full details about your requirements.
This was working until the DB grew to over 25GB, and we started running into disk space limitations.
Some question marks "here":
Why don't you just increase the available Diskspace for your database? 25 GB seems nothing when it comes down to disk-space?
Why don't you modify your script to: download table1, import table1_tmp, drop table1_prod, rename table1_tmp to table1_prod; rinse and repeat.
Other than that:
Why don't you ask your partner for a system with enough performance to run your reports on? I'm quite sure, he would prefer this rather than having YOU download sensitive data every day to your "local site"?
Last thought (requires MySQL Enterprise Backup https://www.mysql.de/products/enterprise/backup.html):
Rather than dumping, downloading and importing 25 GB every day:
Create a full backup
Download and import
Use Differential or incremental backups from now.
The next day you download (and import) only the data-delta: https://dev.mysql.com/doc/mysql-enterprise-backup/4.0/en/mysqlbackup.incremental.html
Is there a way to add a new database to the replication without having to restart the master server? I can't seem to find a way and I would like to minimize the downtime to a minimum.
BTW, I tried using this to speed up the restart. Bringing down the value close to 0 on my master takes forever, plus it adds a lot of load on the machine.
I am using MySQL 5.5.20.
You shouldn't need to do anything to enable a new database for replication. All DDL and DML statements should be recorded in the binary log, and be read by the replica and executed by the replica's SQL replication thread.
It's another story if you use replication filtering on the master (--binlog-do-db) or filtering on the replica (--replicate-do-db).
You can use wildcards in any of these replication filtering options, so that you include (or ignore) new databases automatically if the database name matches some naming conventions you design.
I need to set up 4 MySQL servers. Each of them needs to support both reads and writes, so a master server (that accepts only writes) is out of the question. I need the data between these 4 servers to be synchronized. It does not matter to me if they have a constant connection open between themselves or if they each connect periodically. I looked at the MySQL replication page but did not find it useful for what I need. What is the best way to do this?
It's called multi master replication, there are tools like mmm to help accomplish this.
Though be warn, it can be very lengthy manual process to recover from replication failures, as you'll end up with situation where you're unsure which copy of the data is more up to date.