I have several slave dbs replicated from the same master db, however, for one of the slaves, i would like to keep it as a backup db, which will never have rows updated or deleted.
basically the purpose is to have a backup db with all rows stored by using the replication(mysqldump is waaay slow to do the backup), no update/delete query get replicated, insert query only. i know there will be some conflicts going on no doubt, but still wonder if any filtering options on statement/query on the slave end or any other solutions.
You should never run a production database without a working backup scheme in place - at least as long as you value your data. If you fear that a wrong sql instruction can ruin your database, then you may try point in time recovery.
If you already use replication your master server will log all write/update operations to its binlog - which it will send to the slave servers for replication. You can do for example nightly backups of you complete database. If you destroy your database in the morning, you can import the backup from the night and reapply the instructions from the binlog from after the backup till before the instruction that killed your database.
You could then skip this instruction and apply the instructions that came afterwards. This can also cause consistency issues, as the instruction after the skipped instruction see different data in the database as they did when they were originally executed.
I have similar problem. I know it's old thread but it can help others:
link: mysql replication works only if I choose database by USE database
Related
I've set up a slave replication of MySQL database. And for development requirements, I want to write sth into the slave database, but it would cause the broken of replication.
Since the database is huge, I don't want to restore the slave database from MySQL dump file every time after I finished some development work.
My requirement:
All the changes in the slave database can be reverted by a simple command.
The replication keeps working.
One method is to use LVM filesystem snapshots. Before you begin testing:
Stop replication.
Take an LVM snapshot.
Do your tests. Replication is still off, but data is up to date
After you finish testing:
Stop mysqld.
Restore the snapshot. This reverts all files to the state they were at the moment you created the LVM snapshot above.
Start mysqld and start replication. It will need to catch up and apply all changes since you stopped replication before your testing. This will take a little while, depending on how many changes happened on your master database.
See https://www.tecmint.com/take-snapshot-of-logical-volume-and-restore-in-lvm/ for a nice tutorial on using LVM snapshots.
This method only works if your development database instance is on Linux.
Insert the new records using a primary key that is not expected to be used by the master database (e.g. add a sufficiently large offset like 2^10 or negative numbers if allowed...).
In this way, the insertions coming from the master won't clash.
I have an AWS RDS MySQL 5.7 database with MyISAM tables that I would like to migrate to another RDS in a custom VPC, and once migrated, convert those MyISAM tables to InnoDB.
If I undertood correctly, the only way to create a correct automatic backup is using the following procedure explained here: "Automated Backups with Unsupported MySQL Storage Engines"
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html#Overview.BackupDeviceRestrictions
Stop all activity to your MyISAM tables (that is, close all sessions).
You can close all sessions by calling the mysql.rds_kill command for each process that is returned from the SHOW FULL PROCESSLIST command.
Lock and flush each of your MyISAM tables
Create a snapshot of your DB instance. When the snapshot has completed, release the locks and resume activity on the MyISAM tables
Has someone done this procedure before?
How is that the snapshots are being created successfully every night from the current RDS DBInstance, even though it contains MyISAM tables?
Thanks!
The problem isn't with snapshot creation. It's what can go wrong when you actually try to use one of the snapshots.
RDS snapshots work by capturing a snapshot if your RDS instance's underlying EBS volume (you can't see this volume, but it's there -- RDS runs on EC2, with "hidden" instances and volumes).
EBS snapshots capture the entire contents of the hard drive exactly as they happened to exist at the moment in time when the snapshot process starts.
What ends up on the snapshot is essentially the same thing that you would have on a MySQL Server if you executed sudo killall -9 mysqld -- it is as if the server had halted everything, immediately, without doing any of the things it normally does to clean up for a graceful shutdown. With RDS, things are not quite that dramatic, because RDS does take some precautions, but fundamentally, this is the nature of what is happening.
When you create an RDS instance from a snapshot, the first thing that happens when the instance starts up is the same thing your hypothetical server would do when you restarted the killed MySQL Server daemon: InnoDB Crash Recovery.
InnoDB Crash Recovery
To recover from a MySQL server crash, the only requirement is to restart the MySQL server. InnoDB automatically checks the logs and performs a roll-forward of the database to the present. InnoDB automatically rolls back uncommitted transactions that were present at the time of the crash.
https://dev.mysql.com/doc/refman/5.7/en/innodb-recovery.html#innodb-crash-recovery
Crash recovery is InnoDB's mechanism for bringing everything back into harmony in it internal data structures and ensure that all data is intact, exactly as your application left it. It's possible because InnoDB is a transactional storage engine. That means a lot of different things, but what it specifically means in this case is that InnoDB doesn't just change table data when you change a table. It goes through a process that can be simplified something like this:
store the proposed changes to disk¹
actually make the changes
mark the changes as complete
What this means is that until the changes are finalized, InnoDB can be interrupted and will subsequently be able to pick up where it left off, without corrupting or losing data.
MyISAM has no such mechanisms. It just writes to the data files, directly. Even if a MyISAM table isn't actively being used, it may still need to be repaired when the server comes up, to clean up its structures. In some circumstances, repairing the table can be impossible, and all or part of the data in the table will be lost.
If your MyISAM tables are flushed and locked when the snapshot occurs, they are in a quiescent state on the disk, as though the server had actually been gracefully shut down before the snapshot had occurred, so they will be stable on the snapshot.
But the snapshot process will always appear to succeed, because the snapshot is just duplicating whatever is on the disk, as it appears at the moment in time when the snapshot gets underway.
The problem is that what the snapshot captured may not be usable, and you have no way of knowing whether the snapshot will be fully viable.
¹ Note that the first step, "store the proposed changes to disk" is related to the system variable innodb_flush_log_at_trx_commit which makes the system slower if set to 1 but also is the safest setting, because your query doesn't actually succeed until that first step is done. A setting of 2 is still reasonably safe, because it still writes the changes but continues without requiring that the operating system confirm that they have actually been written to the hard drive before your query returns success... but in a crash, a transaction your application thinks was committed may or may not have survived.
We've got a database (mysql) driven application which contains business critical information, were looking at building a system that will allow us to backup the db frequently (say every 15 mins) essentially so that we minigate the danger of any data loss. Where torn between two setups :
Adding a backup jobs too a queue every 15 mins on a cron and storing these backups on another server. (To save space we would then delete most of these backups after 3 days, but keep the 06:00, 12:00, 18;00 hour versions.)
or
Is there a RAID like setup were all our data will be automatically copied to another hard drive or in this case server, in which case what would happen if we lost data, would the loss be carried to the other server (we would also run standard daily backups for our archives in edition to this) ?
or
Is there another established method for creating frequent backups ?
In my opinion, the optimal backup scheme would be following.
Delayed slave. It allows you quickly restore your database in case of master failure. It may help in case of DROP DATABASE or other wrong SQL. So, you need something additionally.
Incremental backups every day with Xtrabackup from the delayed slave. Optionally you could also check TwinDB for incremental backups.
As long as you need 15 minutes granularity you may pull binary logs from the master with mysqlbinlog from MySQL 5.6 (even if the master is 5.5 or 5.1). So, mysqlbinlog runs on a remote host and pulls logs from the master.
If you need to restore the database you have two ways.
If you can restore from the delayed slave you use that slave as a new master.
If on some reason you can't use the delayed slave (you missed the DROP command) then you restore last night copy from the incremental backup and apply binary logs since the last backup up to the moment of accident (again, if the accident is wrong DROP table you replay logs up to the last event before the DROP).
This schema will be optimal from performance standpoint (no impact on an application) and allows no data loss at all.
If you're doing backups more often than one hour, what you need is replication. Setting up a secondary database server that can serve as a hot-standby is a lot better than abusing your database with repeated reads.
If you're backing up your database frequently, look at innobackupex to snapshot your tables, or possibly LVM snapshots.
I have a ~80GB MySql DB.
I want to create a replication on that DB while having the current DB as master and setting up a slave for it.
My main question is how can i move the data (all 80GB) of it from the master to the new slave with as minimal downtime as possible, preferably none.
my initial thought was to stop the DB (after taking the log position), and then copy the files from the mysqldata lib, and then re start the server but just copying the files would take ~2 hours.
any thoughts?
On July 8, 2011 I addressed a similar question. I wrote scripts that would zap binary logs and starting performing an rsync.
On June 16, 2011, I wrote a post contrasting doing an rsync versus using XtraBackup.
On May 23, 2011, I discussed what considerations to make when doing this kind of backup.
Rather than reinvent the wheel and rewrite in the information I already wrote in those posts, I simply provided the links to my own posts that address this question.
Please read them carefully.
Give it a Try !!!
CAVEAT
The only downtime in my rsync algorithm is when after you have performed multiple rsyncs as specified, you shutdown mysql, perform one more rsync, and then start up mysql.
I would like to clarify the reason for the shutdown:
When you shutdown mysql:
All open MyISAM tables are closed, There is a header that marks how many file handles are open to the MyISAM table. That must be at zero(0) for the table to be OK. Otherwise, a closed MyISAM tables with a nonzero value in this header field marks the table as crashed and in need of a table repair. Shutting down mysql cleans all of that up.
All InnoDB tables that have either data pages or index pages in the Buffer Pool that are marked dirty needs to be flushed to disk. Performing a shutdown triggers a full flush of the Buffer Pool. Naturally, the bigger the pool and the higher the number of dirty pages, the longer the Buffer Pool flush time will be. To shorten this phase of the mysqld's shutdown, run SET GLOBAL innodb_max_dirty_pages_pct = 0; before performing any of the rsyncs. All transactions are completed (either commited or rolled back).
I think you have some misunderstanding.
before it start, you must enable binary log on the master
restart mysql on master
login to master
lock ALL tables from write
record the master binary position
copy binary data from master (DIRECTLY copy *.MYI, *.MYD...etc, you can copy to another location in master database)
after copy is completed, remove write lock
scp data to slave (depends on the network distance)
setup relevant master information into slave (binary log position, and remember to disable binary log)
start slave
After that, it should have huge delay on slave,
and slave will try to catch up with master automatically,
once it catch up, your slave is ready!
So, the down-time is only when you locking table and copy the binary data into another location in your master database.
docs:- http://dev.mysql.com/doc/refman/5.1/en/replication-howto.html
I've found the following tool to be of GREAT help and efficiency. The author currently works for facebook and used to work for dema in japan.
It's quite easy to set-up and you will reach 4 9's HA. ;-)
MHA tool for MySQL replication high availability
I have to say though that MySQL cluster is better, lol ;-)
I have a scenario where there are multiple mysql 4.x servers. These databases were supposed to be replicating to another server. After checking things out on a slave it appears that this slave has not replicated any databases in some time.
Some of these databases are > 4G in size and one is 43G(which resides on another server). Has anyone out there replicated databases without creating a snapshot to copy over to a slave? I cannot shutdown the master server because of the downtime. It will probably take over an hour and 40 minutes to create a snapshot. So this is out of the question.
I was going to perform a load data from master on the slave to pull everything from scratch. Any idea how long this will take on databases ranging from 1-4G and the 43G database will be for another day. All of the tables on the master are myIsam so I don't think I will have a problem with the load from master method.
What are the best methods on the slave to clean things up or reset things so I can just start from a clean slate?
Any suggestions?
Thanks in advance
You need a snapshot to start replication. Snapshots require either the database to be locked (at least) read-only. So you can have a consistent place to start from.
Downtime is a necessary thing, customers usually understand it as long as it doesn't happen too often.