I am moving to a new server and thinking about how to keep my 2 MySQL server data consistent is causing me to lose both sleep and hair.
I was thinking about using a Master-Master setup to ensure that I lose nothing in the process. How viable is that. Any potential gotchas?
Why does the old server ever need to be aware of data written to the new server? For this reason, make it a master-slave setup.
You do have to deal with the same type of configuration, for instance.. make sure the old server only uses odd id's, and the new server only uses even id's.
As soon as you shut down the old server (master), make sure nobody can write there anymore.
I'm assuming your entire website uses 1 server for both the DB and the webhosting. If this is the case, I want to add the following:
Don't rely on DNS to migrate your site, as this can take a very long time for certain users.
Consider the following:
old.example.org is the site on the old machine
new.example.org is the site on the new machine.
www.example.org is a CNAME to old.example.org.
When you do the cutover, you will perform the following steps:
The old DB server is shut down, or set to read-only.
www.example.org becomes a CNAME to new.example.org
old.example.org should now host a website that automatically redirects people to new.example.org.
This means that your users might for a while browse the url new.example.org directly. When the DNS is fully propagated your users will no longer be redirected, and automatically hit the new server when using www.example.org.
If you have a low-traffic site.. this can be much easier.. Simply point your old application to use the new MySQL database. Sure, it might seem a bit crazy to connect to a mysql server over the net; but if you're not dealing with too much data this is so much easier than any other solution..
Related
I have several databases hosted on a shared server, and a local testing server which I use for development.
I would like to keep both set of databases somewhat synchronized (more or less daily).
So far, my ideas to solve the problem seem very clumsy. Anyway, for reference, here is what I have considered so far:
Make a database dump from online databases, trash local databases, and recreate the databases from the dump. It's a lot of work and requires a lot of download time (which guarantees I won't do it as much as I would like it to be done)
Write a small web service to access the new data, and write a small application locally to communicate with said web service, download the newest data, and update the local databases.
Both solutions sound like a lot of work for a problem that is probably already solved a zillion times over. Or maybe it's even an existing feature which I completely overlooked.
Is there an easy way to keep databases more or less in synch? Ideally something that I can set up once, schedule and forget about.
I am using MySQL 5 (MyISAM) databases on both servers.
=============
Edit: I had a look at replication, but it seems that I can't go that route because the shared hosting does not give me enough control on the server itself (I got most permissions on my databases, but not on the MySQL server itself)
I only need to keep the data synchronized, nothing else. Is there any other solution that doesn't require full control on the server?
Edit 2:
Sorry, I forgot to mention I am running on a LAMP stack on the shared server, so Windows-only solutions won't work.
I am surprised to see that there is no obvious off-the-shelves solution for this problem.
Have you considered replication? It's not to be trifled with but may be what you want. See here for more details... http://dev.mysql.com/doc/refman/5.0/en/replication-configuration.html
Take a look at Microsoft Sync Framework - you will need to code in .net, but it can resolve your issues.
http://msdn.microsoft.com/en-in/sync/default(en-us).aspx
Here is a sample for SQL server, but it can be adapted to mysql as well using ado.net provider for Mysql.
http://code.msdn.microsoft.com/sync/Release/ProjectReleases.aspx?ReleaseId=4835
You will need the additional tables for change tracking and anchors (keeping track of last synchronization) for this to work, in your mysql database, but you wont need full control as long as you can access the db.
Replication would have simpler :), but this might just work in your case.
I have a large production database at the moment (7GB+), much of the data I require to test my development branches needs to be up to date.
I would like to create a local DB so I can implement a CI server and have the development DB separate to the production DB, however with a DB of this size how do I ensure it's always in sync?
I have thought about replication, but what if the connection is down to my local server or I have changed the DB structure how will this effect the replication?
Thanks,
Gavin
how do you plan to keep a development database in sync with production, considering that you may make table changes in development long before they get into production?
yeah that's part of the problem, I
don't know if its even possible to do
that given the schema could change
quite a bit..
It will be very difficult to have any automatic synchronization that will not wipe out your or cause problems with your development changes.
So, I recommend that you restore a production backup manually and only when you need to. Set up a job to do the work, and just run it as necessary (don't put it on a schedule). Develop as necessary using the current data as long as you can. When you need a refresh, make sure you have implemented/saved to files all of your development changes, and then restore a backup to development. If you have make "releases" to production, then restore after that. You'll need to determine what works best. However, just make sure you don't lose any development changes when restoring!
If you need to have a local version that is up to date for support and debugging production problems, set up a "Support" database and use replication. Don't do develop work there, have a dedicated development database.
Consider dumping the whole database to text files in form of sql statements (schema and actual data) and then putting them under some version control system that will track changes.
This is a methodology question from a freelancer, with a corollary on MySQL.. Is there a way to migrate from an old dedicated server to a new one without losing any data in-between - and with no downtime? In the past, I've had to lose MySQL data between the time when the new server goes up (i.e., all files transferred, system up and ready), and when I take the old server down (data still transferred to old until new one takes over). There is also a short period where both are down for DNS, etc., to refresh.
Is there a way for MySQL/root to easily transfer all data that was updated/inserted between a certain time frame?
I'd make a sorry page, put it up on the old server, transfer all data to the new one and then switch DNS. Though there will be a downtime.
What I like to do is close the site and starting to move DB to other server using these commands: 2, then move all files (php ..etc) to the other server (if you have some store data or change files every hour, like image upload). and point the old server to the new DB server while the DNS is changing to all to the new server.
The longest downtime is from DNS switch - can take several hours and even days till all clients caches are expired.
To avoid it:
set up application on new server to access DB on old one, or just proxy http requests with nginx to the old one, depending on what is more acceptable.
then goes DNS switch, some clients go to ld server, some to new, here you can wait for 24+ hours to make sure all requests go to new server
While DNS switches - rehearse mysql transition.
make a 'sorry/maintanance page', there're plenty of guides how to do that usung rewrites. You'll need it anyway
measure how fast you can dump-transfer-restore db, if time is acceptable - this is the simplest, but remember to give some margin
if previous is too slow - you can try binlog method suggested in previos answer
minimal downtime can be achieved by making new server a mysql slave to the old one, under the hood it is just downloads binlog from master on-the-fly and you will save time on transferring the whole log, most probably during minimal load slave will be just several seconds behind master and catch up very quickly once app is taken down, see how to force slave to catch up.
Write a script, that does all transition for you - enables maintenance mode, locks master db, waits till slave catches up, makes slave a new master, replaces app config with new db, disables maintenance, switches app etc. This way you save time on typing commands youself, test on staging environment to avoid possible errors (also remember to set larger mysql timeout, just in case slave is a lot behind)
here goes the transition itself, by running script from previous step
Also if you use file uploads to a local filesystem - these need to be synced too and on lots of files this is more pain than with db, because even rsync scan for changes can take a lot of time.
Check out the MySQL binary log.
Sure. Enable bin logging on the source server. After that is started, make a DB dump and transfer it to the new server and import it. Then, when you're ready to make the switch, change DNS (let the change propagate while you're working), then take the site down on both servers. Copy the binlogs to the new server and run them again starting at the date/time of the dump.
I have a website using cPanel on a dedicated account, I would like to be able to automatically sync the website to a second hosting company or perhaps to a local (in house ) server.
Basically this is a type of replication. The website is database driven (MySQL), so Ideally it would sync everything (content, database, emails etc.) , but most importantly is syncing the website files and its database.
I'm not so much looking for a fail-over solution as an automatic replication solution, so if the primary site (Server) goes off-line, I can manually bring up the replicated site quickly.
I'm familiar with tools like unison and rsync, but most of these only sync file(s) and do not do too well with open database connections.
Don't use one tool when two is better; Use rsync for files, but use replication for MySQL.
If, for some reason, you don't want to use replication, you might want to consider using DRBD. This is of course only applicable if you're running Linux. DRBD is now part of the main kernel (since version 2.6.33).
And yes - I am aware of at least one large enterprise deployment of DRBD which is used, among other things, store MySQL database files. In fact, MySQL website even has relevant page on this topic.
You might also want to Google for articles against DRBD/MySQL combination; I remember reading few posts of that.
I'm working on a SaaS project and mysql is our main database. Our applications is written on c# .net and runs under an windows 2003 server.
Considering maintainance, cost, options and performance, which server plattaform can I decide for MySQL hosting, windows or Unix/Linux/Ubuntu/Debian?
The scenario is as following:
The server I run today has a modarate transaction volume. Databases increase 5MB daily and we expect to increase 50MB in couple of months and it is mission critical.
I don't know how big the database is going to be. We rent a VPS to host application and database server.
Most of our queries are simple but our ORM Tool makes constantly use of subqueries. Also we run reports simple and heavy ones. Some them runs after user click, but most runs in order to the queue.
Buy an extra co-lo space will be nice as we got more clients. That's SaaS project after all.
When developing, you can use your Windows box to also run a MySQL server. If and when you
want to have your DBMS in a separate server it can be in either a Windows or Linux server.
MySql and supporting tools for backup etc probably have more choices in Linux.
There are also 3rd party suppliers who will host your MySQL database on their servers. The benefit is they will handle backups, maintenance etc.
Also: look into phpMyAdmin for use as a great admin tool.
Larry
I think you need more information to make an informed decision. It's hard to just pull out a "best" answer based on no specific information.
What is your expected transaction volume?
How big will the database get?
How complex are your queries, ie are they long running or relatively quick?
Are you hosting the application on your own server at your own location? If you have to buy extra co-lo space maybe an extra server isn't the best option.
How "mission critical" is this database? Ie maybe you need replicated servers to ensure stability.
There is a server sizing tool online at http://www.sizinglounge.com/, so you should check that out. It sounds like your server could be smaller than their smallest tier, but it should be a good place to start.
If this is a mission critical application you need to do some kind of replication to an extra server in case the primary one fails, so you are definitely looking at two systems. This has to be in addition to a good backup plan.
Given that you are uncertain about how big it could get you might just continue renting a server. For your backup one idea would be to look at running MySQL on an Amazon EC2 instance. BTW it is important to have a remote replicated server. If you have two systems next to each other and an environmental problem comes up, they could both be out of commission at the same time. But with a remote copy your options are open to potentially working around it.
If you run a lot of read-only queries locally and have your site hosted somewhere, it might make sense to set up a local replicated database copy to query against. That could potentially improve both your website and local performance quite a bit. Plus it would give you some good piece of mind having a local copy under your control.
HTH,
Brandon