i tried master-master replication; as a part of it first I tried master- slave replication. The replication connection is successful but when I made changes on master, the slave is throwing a lot of duplicate errors.
Last_SQL_Error: Error 'Duplicate entry '1576' for key 'PRIMARY'' on query. Default database: 'nagiosxi'. Query: 'INSERT INTO xi_auditlog (log_time,source,user,type,message,ip_address,details) VALUES ('2018-05-15 16:34:19','Nagios XI','NULL',32,'cmdsubsys: User [nagiosadmin] applied a new configuration to Nagios Core','localhost','')' Replicate_Ignore_Server_Ids:
As per my understandings the master cannot overwrite the values in the slave.
the databases which am replicating are application databases, both master and slave databases throwing a lot of errors, as the values cannot be overridden.
Can someone please help me in master master replication.
Do I need to script for this purpose?
To make master-master or Galera safe to use AUTO_INCREMENT one should use the system variables auto_increment_increment and auto_increment_offset to generate unique values for each server.
https://mariadb.com/kb/en/library/auto_increment/
Set auto_increment_increment to 2 on both masters. Set auto_increment_offset to 2 on one server and to 1 (default) on the other.
This causes one server to only create even-numbered values, and the other to create only odd-numbered values. They will still tend to be sequential, because if one server creates 1, 3, 5, 7, and 9, and the next insert occurs on the other server after all of these have replicated, the next id will be 10. The gaps are inevitable, but as any experienced DBA will tell you, do not obsess about the gaps.
You should set your binlog_format to ROW or MIXED -- not STATEMENT -- on both servers. If you are backing up your binlogs as you should be, then log_slave_updates should also be enabled. If you have subtended replicas, the master to which they are connected must have this option enabled.
Also, you're going to need to destroy and rebuild one of the servers at this point, because their data will never be consistent.
The recommended practice for master/master is to only write to one server at a time. Your application can switch at will, but it is most stable if only one is treated as writable.
Related
I'm following this article to create a master-to-master replication on Amazon RDS.
The master and slave MySQL servers are both created using Amazon RDS.
One thing I'm concerned about is whether changing auto_increment_increment and auto_increment_offset of the production environment database will cause any error to the production database or not.
The official document says that they are used to control the operation of AUTO_INCREMENT columns. So I'm worried that this will affect the existing production database.
Has anyone ever changed these values and generated any error?
According to the documentation of Replication and Variables these variables are automatically replicated to slaves.
So if you set them on the master, the slaves should pick up the changes, and they'll assign auto increment IDs the same as the master.
Note that this is only an issue when using statement-based replication. If you use log-based replication, the IDs that the master assigns will automatically be replicated.
I'm mysql newbie.
when it comes to fail-over, which slave should be promoted to the new master?
For example, A is master, B and C are slaves, and A does async replication to B and C.
At some point of time, B receives more data from A than C, A crashes.
If we promote C to new master, and changes B's master to C, then what happens to B? It truncates its data to match C?
Obviously, B is the best new master candidate, but my question is, how to determine this fact?
From the MySQL documentation, there two ways to set up a master-slave architecture. The traditional way, using the log files to replicate transactions and the new version (5.6+) using GTIDs (global transaction identifiers).
If you choose to use GTIDs to make the failover handling you will use the mysqlfailover utility. The utility handles fails of master in one of three ways defined by the database administrator:
auto (default): A search is made in a list of prefered slaves to become master, if none are available another slave is chosen. The chosen slave first becomes the slave to all others slaves and has all the changes from the other slaves copied to it, this way the new master will be the most up to date version possible.
elect: the same as above, except that if no slaves from the list are available it returns an error and finishes (no failover)
fail: No failover happens mysqlfailover will just monitor the database and return an error if a fail happens.
The traditional way requires that you implement your own script to database management and is better explained here.
The Relay_Master_Log_File and Exec_Master_Log_Pos in SHOW SLAVE STATUS is used to determine the best slave as new master: the bigger values win.
Without GTID, I think we must first sync other slaves with the best slave we chose. The obvious sync source is the relay logs. On each slave, determine the differences of relay log from the best slave, download those files and replay the SQL statements. Once all slaves catch up, the slaves could CHANGE MASTER TO the best slave. The MASTER_LOG_FILE and MASTER_LOG_POS would be the tail of the last binlog on the best slave.
With GTID, it's very simple: just CHANGE MASTER TO with MASTER_AUTO_POSITION=1.
We're running a standard Mysql Master/Slave replication system which has been working nicely for a couple of years. So nicely in fact that we were able to recover from a server outage (which rendered the Master offline) by using the Slave DB (after turning it into a Master).
Anyway, the issue that is causing our problem occurred after we recovered the server and returned the original Master to being a Master and the original slave back to being a Slave.
It was caused by me being an idiot - not by anything else!
Basically a write occurred on the Slave - meaning that a write on the same table on the Master cannot be replicated across due to a Duplicated Primary Key error.
As I said, my fault.
I can delete the record from the Slave that was erroneously written there - thus removing the conflicting ID from the Slave but I want the record from the Master DB.
My question is this: I know that I can 'fix' the replication by skipping over the relevant query from the Master (by setting the SQL_SLAVE_SKIP_COUNTER to 1) - but that will mean I lose the record from the Master.
So, is there a way of 'replaying' the errored replication write? Basically pointing the replication to be one query back?
Or do I have to go through the whole rigmarole of dropping my Slave, repopulating it from my last successful backup and set it to be a Slave again?
Many thanks in advance.
If it's a simple modification/update/... why don't you simply play it on the master (manually or taking it from the slave binary log if you have one), with slave's slave process off, then set the new replication pointer (file & position) on the slave (show master status; on master), and restart the slave?
The answer was actually staring me in the face (isn't it always!).
The duplicate Primary key error was caused by me doing an insert on the Slave and an insert to the same table on the master - meaning both new records had the same ID.
Luckily for me, I did not need the insert on the Slave, so I simply deleted it - meaning that there was only one record with the primary key value.
I just needed a way to get the replication to attempt to replicate the record across - and this was easy.
Just restart the Slave!
So:
mysql root:(none)>START SLAVE;
Query OK, 0 rows affected (0.03 sec)
mysql root:(none)> _
I am new to MySQL and after a long search I am able to configure master-slave ROW based replication. I thought it would be safe and I would not have to recheck it again and again.
But today when I did SHOW SLAVE STATUS; on slave then I found following
could not execute Write_rows event on
table mydatabasename.atable; Duplicate
entry '174465' for key 'PRIMARY',
Error_code: 1062; handler error
HA_ERR_FOUND_DUPP_KEY; the event's
master log mysql-bin.000004,
end_log_pos 60121977
Can someone tell me how this can even come when master has no such error and schema on both server is the same then how could this happen. And how to fix it to make this work again and how to prevent such thing in future.
Please also let me know what else unexpected I should expect other than this.
It would never happen on master, why?
The series of SQL are replicated from master,
if the record already exist in master, mysql reject on master
but on slave, if fails and the replication position does not advanced to next SQL (it just halted)
Reason?
The insert query of that record is write directly into slave without using replication from the master
How to fix?
Skip the error on slave, like
SET GLOBAL sql_slave_skip_counter = N;
details - http://dev.mysql.com/doc/refman/5.0/en/set-global-sql-slave-skip-counter.html
Or delete the duplicate record on slave, resume the slave again (let the replication do the insertion)
The worse scenario, required you to re-do the setup again to ensure data integrity on slave.
How to prevent?
Check application level, make sure no write directly into slave
This including how you connect to mysql in command prompt
Split mysql user that can do write and read,
So, your application should use read user (master and slave) when does not require write.
Use write user (master only) for action require write to database.
skip counter is not a viable solution always, you are skipping the records but it might affect the further records.
Here is the complete details on why sql slave skip counter is bad.
http://www.mysqlperformanceblog.com/2013/07/23/another-reason-why-sql_slave_skip_counter-is-bad-in-mysql/
You can delete bigger than duplicate rows in slave db;
DELETE FROM mydatabasename.atable WHERE ID>=174465;
then
START SLAVE;
I'm looking for a way to have 2 databases running; 1 at the office and 1 at a data center. When at the office, employees would connect locally but when outside the office they would connect remotely to the data center. Both databases would be fully synchronized. That means that an employee could log in and update a record on the data center and that change would replicate instantly to the office server (or vice-versa). So, either user could edit the same record.
So, the typical scenario of auto_increment_offset etc. won't work because, in this case, each server has to be able to update the same record.
Am I missing something obvious? I can't think of a viable way to handle this. 2 users on 1 db can modify the same records so there HAS to be a way to do the same in this type of setup. I just can't think of one.
You can not do this reliably with asynchronous replication. Consider this simple example:
site A: update foo set bar = 1
site B: update foo set bar = 2
Some time later (milliseconds or hours depending) the transactions are replicated to each site:
site A: update foo set bar = 2
site B: update foo set bar = 1
foo.bar will now be 2 on site A and 1 on site B. There is nothing in MySQL that will detect this or prevent it from happening.
Either partition the problem so the masters "own" different parts of the data or elect a primary master for all updates. Or implement some global locking mechanism.
With one database things are completely different since there is synchronous locking going on internally.
Here's how we did it:
In master-master configuration there are 2 masters and 2 slaves. Both are masters and slaves at the same time:
master1 => slave1[master2]
[slave1]master1 <= slave2
enable binlogging on master server - my.cnf file under [mysqld] section:
log-bin=mysql-bin.log
relay_log=relay-bin.log
server-id = 1
log-slave-updates
auto_increment_increment = 10
auto_increment_offset = 1
You may need to add:
skip-slave-start option too, to prevent from start slave at database startup.
Note:
Each server should use different server-id
auto_increment_offset should differs
Auto increment options:
auto_increment_increment controls the increment between successive AUTO_INCREMENT values.
auto_increment_offset determines the starting point for AUTO_INCREMENT column values.
With auto_increment_increment = 10 and auto_increment_offset=5, auto_inc values would be 5, 15, 25..
Dedicated user:
GRANT REPLICATION SLAVE ON . TO 'replication'#'%' IDENTIFIED BY 'some_password';
Create consistent, binary backup of MySQL database (needed to setup a replication) using LVM but the same steps will apply to ZFS/UFS snapshot and other popular techniques:
connect to MySQL database on master (server1) and run "FLUSH TABLES WITH READ LOCK". It might take few second to complete flush operation and it will lock all tables! It is imporant to keep it in mind, because such operation may impact in production load in some cases. It's better to schedule replication setup to peak-off hours.
Create LVM snapshot (server1): lvcreate -L10G -s -n mysqlbackup /dev/vg/lvm_mysql_partition
Get information about master position. Using previously created connection to database run : SHOW MASTER STATUS; to get all information about current binlog, position and so on.
Copy-paste output somewhere - for future usage.
Uunlock all tables: UNLOCK TABLES;
Mount previously created filesystem: mount /dev/vg/mysqlbackup (this will allow you to access shapshot of data created).
Now you can just copy this data to the second server directly to data dir. Before you start up your database add previously mentioned parameters to my.cnf file, changing value of server-id.
Now, with master information copied somewhere and database files propagated to slave you can remove snapshot: lvremove -f /dev/vg/mysqlbackup on master (server1).
Now, you can log into server2 to check for permissions (depends on which user was used to copy files you'll need to correct it) and after that, startup MySQL instance. Database should start, perform InnoDB recovery (if you're signed it) and after a bit, you will be able to log into using command line client. Because of skip-slave-start your slave will not start by default.
Now, on both servers you need to configure replication thread by setting up master hostname, master port, password, user and info about position:
CHANGE MASTER TO master_host='IP_addr_of_server1', master_port=3306, master_user='replication', master_password='some_password', master_log_file='info_from_MASTER_STATUS', master_log_pos='info_from_MASTER_STATUS';
Now you can start replication: START SLAVE;
Check to make sure: SHOW SLAVE STATUS\G (on new machine)
Seconds_Behind_Master might not be equal to 0 because of changes done to database after you release all tables but with replication working slave should catch master pretty fast.
After your slave server catches up with master, start master-master (server1-server2) setup. Lock all tables on server2 ( FLUSH TABLES WITH READ LOCK;) and run SHOW MASTER STATUS;. On server1 set-up replication by CHANGE MASTER TO..
of course you need to change master_host to valid IP address, master_log_file and master_log_pos values. After that, UNLOCK TABLES on server2 and START SLAVE on server1.
Now you should have master-master-slave configuration with master-master replication between server1 and server2.