I'm using MySQL 5.7 with GTID master-master replication and I'm experiencing a strange error.
Randomly one of my masters will stop replicating with: "Cannot replicate anonymous transaction when ##GLOBAL.GTID_MODE = ON"
When I check there is indeed an anonymous entry in the binlog, but what isn't clear is how it got there since the other master also has GTID_MODE=ON and that should not allow any Anonymous transactions to execute or make their way in to the binlog.
enforce_gtid_consistency is also ON so queries that would result in an Anonymous transaction should be failing.
It's also only ever a single query/transaction. The previous and next queries in the binlog always have GTID's.
I ran into this same issue. The first time I saw it was when I upgraded to mysql 5.7.12.
I don't have a workaround/fix yet either. What version of mysql are you on?
Setting GTID_MODE to ON_PERMISSIVE helped me either on mysql 8.0.12 with the same error.
I left it at ON_PERMISSIVE because I don't see any disadvantage of it.
SET ##GLOBAL.GTID_MODE = ON_PERMISSIVE;
Related
I have a problem with MySQL replication - there is one table on master server which doesn't appear on slave server. Both master and slave has the same master_log_file and master_log_position, both slave_io and slave_sql threads are running, I even tried to add an empty table to the master database but it does appear on the slave database. It's not the first time I got such error but before that my symptopms were new data did not appear in slave database. Are there any other solutions for this problem than stopping replication on the slave, dropping the database, dumping it on master server, rsyncing to slave server and restarting replication from new file/position?
I noticed using
SHOW SLAVE STATUS;
that Relay_Log_Pos is smaller than Read_Master_Log_Pos and Relay_Log_File differ from Master_Log_File but Slave_SQL_Running_State says
Slave has read all relay log; waiting for more updates
Seconds_Behind_Master says 0.
MySQL officially only supports replication to the next higher version (although it will work for 5.7.13+), see Replication Compatibility Between MySQL Versions:
MySQL supports replication from one release series to the next higher release series. For example, you can replicate from a master running MySQL 5.5 to a slave running MySQL 5.6, from a master running MySQL 5.6 to a slave running MySQL 5.7, and so on.
However, you may encounter difficulties when replicating from an older master to a newer slave if the master uses statements or relies on behavior no longer supported in the version of MySQL used on the slave.
By default, replication will stop if an error occurs, and you have to restart it (after fixing the error). If you use the --slave-skip-errors=all-option however, it will skip these errors:
Normally, replication stops when an error occurs on the slave, which gives you the opportunity to resolve the inconsistency in the data manually. This option causes the slave SQL thread to continue replication when a statement returns any of the errors listed in the option value.
Do not use this option unless you fully understand why you are getting errors. If there are no bugs in your replication setup and client programs, and no bugs in MySQL itself, an error that stops replication should never occur. Indiscriminate use of this option results in slaves becoming hopelessly out of synchrony with the master, with you having no idea why this has occurred.
MySQL 5.5. and 5.7. will actually behave differently for a lot of statements, so enabling this option in this scenario will require even more care.
Without seeing your actual table create-statement, it is unclear what exactly caused that problem and how to fix it (or if it is possible), but you should especially check your configuration settings. MySQL 5.7. enables strict mode by default, so a usual suspect for incompatibilites is e.g. a zero default value for date/timestamp-columns like default '0000-00-00' (either explicit or implicit), which is not allowed anymore, see no_zero_date.
Even if you seem to not be too keen about 100% replication (which can snowball very fast, but that is up to you to evaluate for your scenario), resetting your slave (after fixing e.g. the configuration settings) at least once is probably the easiest solution, as there might have been other things you may have missed, and, if executed without errors, will also doublecheck if your tables and data up to that point are compatible with your 5.7-slave now.
I have been trying to start replication in a mysql database. I followed all the steps from the mysql manual to setup and configure the replication.
http://dev.mysql.com/doc/refman/5.1/en/replication-howto.html
I could start the replication without any trouble. Newly inserted data was replicated properly.
But after a while I observed that though INSERT are working, the UPDATEs are not replicating. So, my replicated database contains data just as it was inserted first time.
My master database is on an UNIX server with MYSQL-5.1.56. The slave is used in Windows using WAMP package with MYSQL 5.5.8. I have also tried a slave with lower version mysql with WAMP5, with same effect.
Please share your ideas and experience on this. Thanks.
I have similar problem. I know it's old thread but it can help others:
link: mysql replication works only if I choose database by USE database
I sorted it out a few months ago. The problem was, I tried to replicate only one single database, and set binlog_do_db and binlog-ignore-db options in master to select only that database. But that left out any statement which was executed without selecting database by USE database, as kayn said. Using replicate-ignore-db and replicate-do-db options in slave would do the same. So I finally fixed it by using the replicate-wild-do-table option in the slave, referenced here. Added the following line in the configuration of the slave server.
replicate-wild-do-table=mydb.%
I have a master/slave replication on my MySql DB.
my slave DB was down for a few hours and is back up again (master was up all the time), when issuing show slave status I can see that the slave is X seconds behind the master.
the problem is that the slave dont seem to catch up with the master, the X seconds behind master dont seem to drop...
any ideas on how I can help the slave catch up?
Here is an idea
In order for you to know that MySQL is fully processing the SQL from the relay logs. Try the following:
STOP SLAVE IO_THREAD;
This will stop replication from downloading new entries from the master into its relay logs.
The other thread, known as the SQL thread, will continue processing the SQL statements it downloaded from the master.
When you run SHOW SLAVE STATUS\G, keep your eye on Exec_Master_Log_Pos. Run SHOW SLAVE STATUS\G again. If Exec_Master_Log_Pos does not move after a minute, you can go ahead run START SLAVE IO_THREAD;. This may reduce the number of Seconds_Behind_Master.
Other than that, there is really nothing you can do except to:
Trust Replication
Monitor Seconds_Behind_Master
Monitor Exec_Master_Log_Pos
Run SHOW PROCESSLIST;, take note of the SQL thread to see if it is processing long running queries.
BTW Keep in mind that when you run SHOW PROCESSLIST; with replication running, there should be two DB Connections whose user name is system user. One of those DB Connections will have the current SQL statement being processed by replication. As long as a different SQL statement is visible each time you run SHOW PROCESSLIST;, you can trust mysql is still replicating properly.
What binary log format are you using ? Are you using ROW or STATEMENT ?
SHOW GLOBAL VARIABLES LIKE 'binlog_format';
If you are using ROW as a binlog format make sure that all your tables has Primary or Unique Key:
SELECT t.table_schema,t.table_name,engine
FROM information_schema.tables t
INNER JOIN information_schema .columns c
on t.table_schema=c.table_schema
and t.table_name=c.table_name
and t.table_schema not in ('performance_schema','information_schema','mysql')
GROUP BY t.table_schema,t.table_name
HAVING sum(if(column_key in ('PRI','UNI'), 1,0)) =0;
If you execute e.g. one delete statement on the master to delete 1 million records on a table without a PK or unique key then only one full table scan will take place on the master's side, which is not the case on the slave.
When ROW binlog_format is being used, MySQL writes the rows changes to the binary logs (not as a statement like STATEMENT binlog_format) and that change will be applied on the slave's side row by row, which means a 1 million full table scan will take place on the slave's to reflect only one delete statement on the master and that is causing slave lagging problem.
"seconds behind" isn't a very good tool to find out how much behind the master you really is. What it says is "the query I just executed was executed X seconds ago on the master". That doesn't mean that you will catch up and be right behind the master the next second.
If your slave is normally not lagging behind and the work load on the master is roughly constant you will catch up, but it might take some time, it might even take "forever" if the slave is normally just barely keeping up with the master. Slaves operate on one single thread so it is by design much slower than the master, also if there are some queries that take a while on the master they will block replication while running on the slave.
Just check if you have same time and timezones on both the servers, i.e., Master as well as Slave.
If you are using INNODB tables, check that you have innodb_flush_log_at_trx_commit to a value different that 0 at SLAVE.
http://dev.mysql.com/doc/refman/4.1/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit
We had exactly the same issue after setting up our slave from a recent backup.
We had changed the configuration of our slave to be more crash-safe:
sync_binlog = 1
sync_master_info = 1
relay_log_info_repository = TABLE
relay_log_recovery = 1
I think that especially the sync_binlog = 1 causes the problem, as the specs of this slave is not so fast as in the master. This config option forces the slave to store every transaction in the binary lo before they are executed (instead of the default every 10k transactions).
After disabling these config options again to their default values I see that the slave is catching up again.
Just to add the findings in my similar case.
There were few bulk temporary table insert/update/delete were happening in master which occupied most of the space from relay log in slave. And in Mysql 5.5, since being single threaded, CPU was always in 100% and took lot of time to process these records.
All I did was to add these line in mysql cnf file
replicate-ignore-table=<dbname>.<temptablename1>
replicate-ignore-table=<dbname>.<temptablename2>
and everything became smooth again.
Inorder to figure out which tables are taking more space in relay log, try the following command and then open in a text editor. You may get some hints
cd /var/lib/mysql
mysqlbinlog relay-bin.000010 > /root/RelayQueries.txt
less /root/RelayQueries.txt
If u have multiple schema's consider using multi threaded slave replication.This is relatively new feature.
This can be done dynamically without stopping server.Just stop the slave sql thread.
STOP SLAVE SQL_THREAD;
SET GLOBAL slave_parallel_threads = 4;
START SLAVE SQL_THREAD;
I have an issue similar to this. and both of my MySQL server hosted on AWS EC2 (master and replication). by increasing EBS disk size (which automatically increased IOPS) for MySQL slave server, its turned out the solution for me. R/W Throughput and bandwidth is increased R/W latency were decreased.
now my MySQL database replication is catching up to the master. and Seconds_Behind_Master was decreased (it was got increased from day to day).
so if you have MySQL hosted on EC2. I suggest you tried to increase EBS disk size or its IOPS on the slave.
I know it's been a while since OP asked but it would have helped me to read the following answer.
In /etc/mysql/mysql.cnf :
[mysql]
disable_log_bin
innodb_flush_log_at_trx_commit=2
innodb_doublewrite = 0
sync_binlog=0
disable_log_bin REALLY carried the trick for me.
Is there any way to get a fault tolerant MySQL replication? I am in an environment that has many networking issues. It appears that replication gets an error and just stops. I need it to continue to work and recover from these faults. There is some wrapper software that checks the state of replication and restarts it in the case of losing its log position. Is there an alternative?
Note:
Replication is done from an embedded computer with MySQL 4.1 to a external computer that has MySQL 5.0.45
What error are you getting? You also haven't described what replication scheme or Mysql version you're using. The errors you're getting are also important.
Replication usually stops when there's a primary/unique key conflict in a Master-Master replication. Other than that on a typical Master-Slave replication setup, networking issues shouldn't cause problems.
Try using Mysql 5.1 or newer, since replication in 5.0 is statement-based and causes problems in Master-Master setups, or when you're using stored-procedures.
(Also, stay away from Mysql Cluster ... noticed the advice on another comment).
Replication errors only happen if the databases get out of sync somehow, having the server simply continue would mean incoherent databases, I really doubt you'd want that.
In my experience, the only time you end up with such errors is if one of the master servers did not complete a query and the slave noticed.
In any case, if you really want to have the slave continue via some sort of chron job, you could always have a query run every few minuts asking the slave "SHOW SLAVE STATUS" then checking the error column, if it's present, send a "STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER; START SLAVE;" command. But it would probably be much more apt to send an email to an admin when mysql encounters an error instead, so he/she can investigate the source of the problem and make sure the databases are actually in sync, otherwise you're likely to see more errors in the near future as the databases become more and more out of sync.
Consider MySQL Cluster using the NDB storage engine, it's meant to be shared-nothing and fault tolerant
MySQL replication will normally detect problems and reconnect anyway, continuing from where it left off.
If you're getting replication errors, it's likely that the source is something else. MySQL replication effectively does a "tail -f" on the query log and replays it on the slave (it's slightly smarter than that, but not much).
If the databases become out of sync, MySQL replication will neither detect nor repair this, but it may eventually cause it to break as a subsequent update cannot proceed due to conflicting data on the slave.
The default timeouts on the replication slave are much too long - it waits hours (or something) - you'll want to reduce this.
Data becoming out of sync is difficult to avoid, mitigation steps are:
Monitor replication using something like mk-table-checksum from Maatkit
Audit all your code for replication-unsafe queries
If using 5.1, switch to row-based replication, which is less likely to suffer from this problem
I'm running a master-slave MySQL binary log replication system (phew!) that, for some data, is not in sync (meaning, the master holds more data than the slave). But the slave stops very frequently on the slightest MySQL error, can this be disabled? (perhaps a my.cnf setting for the replicating slave ignore-replicating-errors or some of the sort ;) )
This is what happens, every now and then, when the slave tries to replicate an item that does not exist, the slave just dies. a quick check at SHOW SLAVE STATUS \G; gives
Slave-IO-Running: Yes
Slave-SQL-Running: No
Replicate-Do-DB:
Last-Errno: 1062
Last-Error: Error 'Duplicate entry '15218' for key 1' on query. Default database: 'db'. Query: 'INSERT INTO db.table ( FIELDS ) VALUES ( VALUES )'
which I promptly fix (once I realize that the slave has been stopped) by doing the following:
STOP SLAVE;
RESET SLAVE;
START SLAVE;
... lately this has been getting kind of tiresome, and before I spit out some sort of PHP which does this for me, i was wondering if there's some my.cnf entry which will not kill the slave on the first error.
Cheers,
/mp
stop slave; set global sql_slave_skip_counter=1; start slave;
You can ignore only the current error and continue the replication process.
Yes, with --slave-skip-errors=xxx in my.cnf, where xxx is 'all' or a comma sep list of error codes.
First, do you really want to ignore errors? If you get an error, it is likely that the data is not in sync any more. Perhaps what you want is to drop the slave database and restart the sync process when you get an error.
Second, I think the error you are getting is not when you replicate an item that does not exist (what would that mean anyway?) - it looks like you are replicating an item that already exists in the slave database.
I suspect the problem mainly arises from not starting at a clean data copy. It seems that the master has been copied to the slave; then replication has been turned off (or failed); and then it has started up again, but without giving the slave the chance to catch up with what it missed.
If you ever have a time when the master can be closed for write access long enough to clone the database and import it into the slave, this might get the problems to go away.
Modern mysqldump commands have a couple options to help with setting up consistent replication. Check out --master-data which will put the binary log file and position in the dump and automatically set when loaded into slave. Also --single-transaction will do the dump inside a transaction so that no write lock is needed to do a consistent dump.
If the slave isn't used for any writes other than the replication, the authors of High Performance MySQL recommend adding read_only on the slave server to prevent users from mistakenly changing data on the slave as this is will also create the same errors you experienced.
i think you are doing replication with out sync the database first sync the database and try for replication and servers are generating same unique ids and try to set auto incerment offset