I've got a huge InnoDB table(>500 millions rows) which I'd like to partition by hash in order to decrease the index size. I'd like to achieve this with a minimal downtime(e.g 10 minutes is acceptable), what are the possible options?
I was thinking about something as follows:
create the new partitioned table
insert into this new table all the data from the old one using "insert ... select ..."
make server unavailable for clients
somehow sync changes which happened to the old table during step 2 with the new table
replace the old table with the new one
make server available for clients
The main question is what tool can be used in the step 4. The problem is that during step 2 there can be lots of changes to the original table: new inserts, updates, deletes - the sync tool should take all of this into account...
Another possible way, I believe, is:
setup a replicating slave server
sync this slave server with master
switch master/slave roles and re-configure all clients to connect to the new master
alter table on the previous master
wait for master/slave synchronization
switch master/slave roles again, re-configure all clients
Which one would you recommend?
I would go with master/slave replication. If the master and slave can be on the same subnet, I would also add a new IP to the master, change the clients to point to the new IP. Then when you are about to switch to the slave, just:
stop mysql on the master
ifconfig down the extra IP on the master
ifconfig up the extra IP on the new master
Clients will just connect to the new master without any client reconfig. Then you do the same thing when you switch back to the original master (if you switch back).
I recommend that a slave always be equivalent hardware to its master so that when it takes over for the master you don't find out it's so much slower that it can't keep up and your whole system fails. If you do that, then you need only switch once (from the current master to the new master).
Related
I've set up a slave replication of MySQL database. And for development requirements, I want to write sth into the slave database, but it would cause the broken of replication.
Since the database is huge, I don't want to restore the slave database from MySQL dump file every time after I finished some development work.
My requirement:
All the changes in the slave database can be reverted by a simple command.
The replication keeps working.
One method is to use LVM filesystem snapshots. Before you begin testing:
Stop replication.
Take an LVM snapshot.
Do your tests. Replication is still off, but data is up to date
After you finish testing:
Stop mysqld.
Restore the snapshot. This reverts all files to the state they were at the moment you created the LVM snapshot above.
Start mysqld and start replication. It will need to catch up and apply all changes since you stopped replication before your testing. This will take a little while, depending on how many changes happened on your master database.
See https://www.tecmint.com/take-snapshot-of-logical-volume-and-restore-in-lvm/ for a nice tutorial on using LVM snapshots.
This method only works if your development database instance is on Linux.
Insert the new records using a primary key that is not expected to be used by the master database (e.g. add a sufficiently large offset like 2^10 or negative numbers if allowed...).
In this way, the insertions coming from the master won't clash.
I'm running MySQL as the database on Ubuntu instances. I'm using MySQL Master-Slave replication where master's changes will be written to slave and slave's changes will not be reflected on the master. That's fine. I'm using a HAProxy load balancer to front the MySQL instances where all the requests will be sent to master MySQL instance. If the master MySQL instance is down slave MySQL instance will act as master and HAProxy will send all the requests to salve. Active-Passive scenario.
HAProxy - 192.168.A.ABC
MySQL Master - 192.168.A.ABD
MySQL Slave - 192.168.A.ABE
Let's assume that the MySQL master(192.168.A.ABD) is down. Now all the requests will be sent to MySQL slave(192.168.A.ABE) by HAProxy where now he acts as the master MySQL server for the time being.
My problems are
What happens when original master MySQL instance(192.168.A.ABD) is up?
Will changes written to new MySQL master (192.168.A.ABE) be replicated to original master(192.168.A.ABD) again?
How should I address this scenario?
First of all I should say that I have never used HA Proxy so con't comment on that directly.
However, in your current setup the Master (ABD) will be out of sync, and wont catch up. You will have to rebuild that using mysqlDump or similar tool.
What you would need is a Master < > Master setup (as opposed to Master > Slave), which enables you to write to either database and have it reflected in the other. This isn't quite as straight forward as it sounds though.
Assuming you already have your master > slave setup, and they are in sync
On the Master (ABD) you want to add:
auto_increment_increment=2
auto_increment_offset=1
log-slave-updates
On the Slave (ABE) add:
auto_increment_increment=2
auto_increment_offset=2
log-slave-updates
to your my.cnf files. Restart the Database. This will help to prevent Duplicate Key Errors. (n.b. that log-slave-updates isn't strictly required but makes it easier to add another slave in future)
Next you want to tell the Master (ABD) to replicate from the Slave (ABE).
Depending on what version of MySQL and if you are using GTID etc. the exact process differs slightly. But basically you are going to issue a CHANGE MASTER statement on the Master so it replicates from the slave.
And away you go. You probably want to avoid writing to both at the same time as that opens up a whole other kettle of fish. But if the Master goes down, you can switch your writes to the slave, and when the master comes back up, it will simply start replicating the missing data.
I am considering you scenario
Master - 192.168.A.ABD
Slave - 192.168.A.ABE
You cannot directly add the master in system. To Add master in system you need to perform below steps:
1) When master is up you can add this as a slave. So now this happens
Master - 192.168.A.ABE
Slave - 192.168.A.ABD
2) Then Now U can put master Down. Means You can put 192.168.A.ABD Down
3) Then Again Add this as slave. So After this You will get below scenarion
Master - 192.168.A.ABD
Slave - 192.168.A.ABE
You can refer this link
https://dev.mysql.com/doc/refman/5.5/en/replication-solutions-switch.html
Is there a way to add a new database to the replication without having to restart the master server? I can't seem to find a way and I would like to minimize the downtime to a minimum.
BTW, I tried using this to speed up the restart. Bringing down the value close to 0 on my master takes forever, plus it adds a lot of load on the machine.
I am using MySQL 5.5.20.
You shouldn't need to do anything to enable a new database for replication. All DDL and DML statements should be recorded in the binary log, and be read by the replica and executed by the replica's SQL replication thread.
It's another story if you use replication filtering on the master (--binlog-do-db) or filtering on the replica (--replicate-do-db).
You can use wildcards in any of these replication filtering options, so that you include (or ignore) new databases automatically if the database name matches some naming conventions you design.
I have mysql replication set up with one master and one slave. Due to a bug in the code, somewhere in the middle the entries started to get written on slave server and it was detected a few days later on.
Now I am thinking of how to switch it correctly without any hassle or minimal down time, what would be the best way to do this? Lets consider only one table...
Solution 1
Simply start writing to master from now on after setting auto_increment to slave's last id. Wondering if it will be troublesome to keep master and slave out of sync.
Solution 2
Clear all the data from master, stop the app from making any more entries refill the data using mysqldump and then switching the app back on with correct config.
stop slave
// load the dump
start slave
Will this stop master from re-attempting to write to slave the same data?
Any help appreciated. Any other solutions also welcomed.
Thanks
Sushil
I think you are on the correct track with solution 2. Simply stopping the slave will not prevent the master from writing to it's binary log. So when you start the slave again it will just replicate all the SQL statements from the master.
However, you can use this to your advantage if you have included 'DROP TABLE' before each table creation. This will mean that you have the following:
1) Stop the app from making any more entries in the master table(s)
2) Dump data from slave (ensure that mysqldump includes 'DROP TABLE' before each table import - it should do as it is a default option of mysqldump)
3) Run dump against master
4) Check slave status using SHOW SLAVE STATUS\G. Once Seconds_Behind_Master reaches 0 then you are good to switch on the app again (make sure it is writing to the master!!)
Step 3 will drop and recreate the tables on the master using the data from the slave. This drop and recreate will be replicated on to the slave so you should end up with the two in sync and a correct master slave set up.
Good luck!
I think your best option is to reset the slave/master completely. If the data on the slave is correct reload the data from it and then export export a new dump from the master and import it to the slave, then execute a new "CHANGE MASTER TO..." command
I would recommend setting the "read_only" global variable on the slave.
http://dev.mysql.com/doc/refman/5.1/en/replication-options-slave.html#option_mysqld_read-only
I'm looking for a way to have 2 databases running; 1 at the office and 1 at a data center. When at the office, employees would connect locally but when outside the office they would connect remotely to the data center. Both databases would be fully synchronized. That means that an employee could log in and update a record on the data center and that change would replicate instantly to the office server (or vice-versa). So, either user could edit the same record.
So, the typical scenario of auto_increment_offset etc. won't work because, in this case, each server has to be able to update the same record.
Am I missing something obvious? I can't think of a viable way to handle this. 2 users on 1 db can modify the same records so there HAS to be a way to do the same in this type of setup. I just can't think of one.
You can not do this reliably with asynchronous replication. Consider this simple example:
site A: update foo set bar = 1
site B: update foo set bar = 2
Some time later (milliseconds or hours depending) the transactions are replicated to each site:
site A: update foo set bar = 2
site B: update foo set bar = 1
foo.bar will now be 2 on site A and 1 on site B. There is nothing in MySQL that will detect this or prevent it from happening.
Either partition the problem so the masters "own" different parts of the data or elect a primary master for all updates. Or implement some global locking mechanism.
With one database things are completely different since there is synchronous locking going on internally.
Here's how we did it:
In master-master configuration there are 2 masters and 2 slaves. Both are masters and slaves at the same time:
master1 => slave1[master2]
[slave1]master1 <= slave2
enable binlogging on master server - my.cnf file under [mysqld] section:
log-bin=mysql-bin.log
relay_log=relay-bin.log
server-id = 1
log-slave-updates
auto_increment_increment = 10
auto_increment_offset = 1
You may need to add:
skip-slave-start option too, to prevent from start slave at database startup.
Note:
Each server should use different server-id
auto_increment_offset should differs
Auto increment options:
auto_increment_increment controls the increment between successive AUTO_INCREMENT values.
auto_increment_offset determines the starting point for AUTO_INCREMENT column values.
With auto_increment_increment = 10 and auto_increment_offset=5, auto_inc values would be 5, 15, 25..
Dedicated user:
GRANT REPLICATION SLAVE ON . TO 'replication'#'%' IDENTIFIED BY 'some_password';
Create consistent, binary backup of MySQL database (needed to setup a replication) using LVM but the same steps will apply to ZFS/UFS snapshot and other popular techniques:
connect to MySQL database on master (server1) and run "FLUSH TABLES WITH READ LOCK". It might take few second to complete flush operation and it will lock all tables! It is imporant to keep it in mind, because such operation may impact in production load in some cases. It's better to schedule replication setup to peak-off hours.
Create LVM snapshot (server1): lvcreate -L10G -s -n mysqlbackup /dev/vg/lvm_mysql_partition
Get information about master position. Using previously created connection to database run : SHOW MASTER STATUS; to get all information about current binlog, position and so on.
Copy-paste output somewhere - for future usage.
Uunlock all tables: UNLOCK TABLES;
Mount previously created filesystem: mount /dev/vg/mysqlbackup (this will allow you to access shapshot of data created).
Now you can just copy this data to the second server directly to data dir. Before you start up your database add previously mentioned parameters to my.cnf file, changing value of server-id.
Now, with master information copied somewhere and database files propagated to slave you can remove snapshot: lvremove -f /dev/vg/mysqlbackup on master (server1).
Now, you can log into server2 to check for permissions (depends on which user was used to copy files you'll need to correct it) and after that, startup MySQL instance. Database should start, perform InnoDB recovery (if you're signed it) and after a bit, you will be able to log into using command line client. Because of skip-slave-start your slave will not start by default.
Now, on both servers you need to configure replication thread by setting up master hostname, master port, password, user and info about position:
CHANGE MASTER TO master_host='IP_addr_of_server1', master_port=3306, master_user='replication', master_password='some_password', master_log_file='info_from_MASTER_STATUS', master_log_pos='info_from_MASTER_STATUS';
Now you can start replication: START SLAVE;
Check to make sure: SHOW SLAVE STATUS\G (on new machine)
Seconds_Behind_Master might not be equal to 0 because of changes done to database after you release all tables but with replication working slave should catch master pretty fast.
After your slave server catches up with master, start master-master (server1-server2) setup. Lock all tables on server2 ( FLUSH TABLES WITH READ LOCK;) and run SHOW MASTER STATUS;. On server1 set-up replication by CHANGE MASTER TO..
of course you need to change master_host to valid IP address, master_log_file and master_log_pos values. After that, UNLOCK TABLES on server2 and START SLAVE on server1.
Now you should have master-master-slave configuration with master-master replication between server1 and server2.