I understand that mysql replication uses two different threads on slave -
Slave I/O thread to read binlog from master and write it to local relay log
Slave SQL thread to execute the sql statements from the local relay log. This thread is for executing update, delete and create sqls.
What about select queries on slave? Can SELECT queries interfere with the replication process? Or is there a different thread that execute SELECT queries?
I mean, can slow select queries on slave make the replication to lag behind master ?
In short queries can interfere with the replication it is not the threading that matters here but the locking being applied (ACID vs. threading). A update query from the master that is being replicated to the slave can be blocked by a select query on the slave. However the replication sub-system will deal with these query locking issues most of the time. If you don’t mind dirty reads you can set the transaction serialization isolation level on the slave to something less restrictive to mitigate the risk. However make sure dirty reads is acceptable see this link for more information: http://dev.mysql.com/doc/refman/5.0/en/set-transaction.html
You are concerned about lag and that is not something you can eliminate in any replication schema there will be lag. There is almost always a network in between the master and slave. This will introduce lag right from the start. For example large replicated statements could saturate the network bandwidth and this is probably going to happen more often than a query blocking the replication. Replication never was and never will be instantaneous. So your point about lag can be answered like this lag is something you HAVE to deal with not something you can completely eliminate.
Dont get me wrong replication can be fast but it is NEVER instantaneous.
Another thing to keep in mind is that replication can fail and you need to plan for that as well. It will happen at some point and being prepared for it is essential. So basically you will have lag no matter how you do replication and you need to be able to deal with it. Also be prepared for replication failing at some point and how you will recover from it.
While replication can be useful in many places you need to make sure you are preparing for it on many levels such as adequate network infrastructure, how to deal with it during disaster recovery(failover), monitoring it during production and how to get it back online when it breaks.
Related
I am using Mysql/MariaDB with Innodb storage engine version 10.x.
I want to setup a cluster with master-slave configuration. There is an option to read data from slave using --innodb-read-only or --read-only.
However in addition to the above, client needs to read the data from slave if and only if max slave lag is less than x seconds.
Slaves can lag behind the primary due to network congestion, low disk throughput, long-running operations, etc. The read preference with max allowed staleness option should let application specify a maximum replication lag, or “staleness”, for reads from slaves. When a secondary’s estimated staleness exceeds, the client stops using it for read operations from slaves and start reading from master.
I would like to know if there is an option in MySql/InnoDB?
There's no automatic option for switching the query to the master. This is handled by application logic.
You can run a query SHOW SLAVE STATUS and one of the fields returned is Seconds_Behind_Master. You would have to write application code to check this, and if the lag is greater than your threshold, query the master instead.
You might find some type of proxy that can do this logic for you. See https://mydbops.wordpress.com/2018/02/19/proxysql-series-mysql-replication-read-write-split-up/
It's not always the best option to treat a replica with X seconds of lag as unusable. Some queries are perfectly okay regardless of the lag. I wrote a presentation about this some years ago, and it includes some example queries. Read / Write Splitting with MySQL and PHP (Percona webinar 2013)
There are many Proxy products that may have code for such.
If you automatically switch to the Master, then it may get overwhelmed, thereby leading to worse system problem.
If you try to switch to another Slave, it is too easy to get into a flapping situation.
Galera has a way to deal with "critical read", if you wanted to go to a Cluster setup instead of Master + Slaves.
If part of the problem is the distance between Master and Slave, and if you switch to the Master, where is the Client? If it is near the Slave, won't the added time to reach the master cancel out some of the benefit?
Avoid long-running queries, beef up the Slave to avoid slow disks, speed up queries that are hitting the disk a lot, look into network improvements.
In summary, I don't like the idea of attempt to move a query to the Master; I would work on dealing with the underlying problem.
MariaDB MaxScale has multiple ways of dealing with replication lag.
The simplest method is to limit the maximum allowed replication lag with the max_slave_replication_lag parameter. This works exactly the way you described: if a slave is too many seconds behind the master, other slaves and, as a last resort, the master is used. This is the most common method of dealing with replication lag in MaxScale.
Another option is to use the causal_reads feature which leverages the MASTER_GTID_WAIT and other features found in MariaDB 10.2 and newer versions. This allows read consistency without adding additional load on the master. This does come at the cost of latency: if the server is lagging several seconds behind the read could take longer. This option is used when data consistency is critical but the request latency is not as important.
The third option is to use the CCRFilter to force reads to the master after a write happens. This is a simpler approach compared to causal_reads but it provides data consistency at the cost of increased load on the master.
In MySQL 5.7, there is a new feature called Group replication used for fault tolerant system.
Group Replication Doc
I implemented the group replication with 1 master and 3 slaves and it working fine. But i feel it little slow. Slow means query taking time in execution on master. It may be because of master waits for acknowledgement from slave or time in certification acceptance etc.
I know it may be broad question but if you guide some parameters related to group replication to improve the performance then it will be helpful.
Thanks
As you say, the Master does have to communicate with its Slave(s). So, yes, that does slow down each transaction.
Each COMMIT is delayed by the coordination among the servers. So, two things might be possible in your app:
Fewer COMMITs; that is more statements in each transaction. (But not too many.)
More parallelism at the client, hence more transactions running in parallel.
We have to show a difference to show the advantages of using replication. We have two computers, linked by teamviewer so we can show our class what we are doing exactly.
Is it possible to show a difference in performance? (How long it takes to execute certain queries?)
What sort queries should we test? (in other words, where is the difference between using/not using replication the biggest)
How should we fill our database? How much data should be there?
Thanks a lot!
I guess the answer to the above questions depends on factors such as which storage engine you are using, size of the database, as well as your chosen replication architecture.
I don't think replication will have much of an impact on query execution for simple master->slave architecture. If however, you have an architecture where there are two masters: one handling writes, replicating to another master which exclusively handles reads, and then replication to a slave which handles backups, then you are far more likely to be able to present some of the more positive scenarios. Have a read up on locks and storage engines, as this might influence your choices.
One simple way to show how Replication can be positive is to demonstrate a simple backup strategy. E.g. Taking hourly backups on a master server itself can bring the underlying application to a complete halt for the duration of the backup (Taking backups using mysqldump locks the tables so that no read/write operations can occur). Whereas replicating to a slave, then taking backups from there negates this affect.
If you want to show detailed statistics, it's probably better to look into some benchmarking/profiling tools (sysbench,mysqlslap,sql-bench to name a few). This can become quite complex though.
Also might be worth looking at the Percona Toolkit and the Percona monitoring plugins here: http://www.percona.com/software/
Replication has several advantages:
Robustness is increased with a master/slave setup. In the event of problems with the master, you can switch to the slave as a backup
Better response time for clients can be achieved by splitting the load for processing client queries between the master and slave servers
Another benefit of using replication is that you can perform database backups using a slave server without disturbing the master.
Using replication always a safe thing to do you should be replicating your Production server always incase of failure it will be helpful.
You can show seconds_behind_master value while showing replication performance, this shows indication of how “late” the slave is this value should not be more than 600-800 seconds but network latency does matter here.
Make sure that Master and Slave servers are configured correctly now
You can stop slave server and let Master server has some updates/inserts (bulk inserts) happening and now start slave server you will see larger seconds_behind_master value it should be keep on decreasing till reaches 0 value.
There is a tool called MONyog - MySQL Monitor and Advisor which shows Replication status in real-time.
Also what kind of replication to use whether statement based or row based has been explained here
http://dev.mysql.com/doc/refman/5.1/en/replication-sbr-rbr.html
I've honestly tried this left and right and still find that my mirror server, set up as a replication slave still lags behind. My app's user base keeps growing and now Ive reached the point where I can't keep "shutting down" to "resync" databases (not even on weekends).
Anyways, my question: Are there any plausible, affordable, alternatives to binlog replication? I have two servers so wouldn't consider buying a third for load balancing just yet, unless its the only option.
Cheers,
/mp
Your master executes in parallel and your slave executes in serial. If your master can process 1.5 hours of inserts/updates/executes in 1 real hour, your slave will fall behind.
If you can't find ways to improve the write performance on your slave (more memory, faster disks, remove unnecessary indexes), you've hit a limitation in your applications architecture. Eventually you will hit a point that you can't execute the changes in real time as fast as your master can execute them in parallel.
A lot of big sites shard their databases: consider splitting your master+slave into multiple master+slave clusters. Then split your customer base across these clusters. When a slave starts falling behind, it's time to add another cluster.
It's not cheap, but unless you can find a way to make binlog replication execute statements in parallel you probably won't find a better way of doing it.
Update (2017): MySQL now support parallel slave worker threads. There are still many variables that will cause a slave to fall behind, but slaves no longer need to write in serial order. Choosing to preserve the commit order of parallel slave threads is an important option to look at if the exact state of the slave at any point in time is critical.
Have you tried :
1) SET innodb_flush_log_at_trx_commit=0
2) SET sync_binlog=0
Both will help to speed up your Slave with a small level of added risk if you have a server failure.
Adding memory to the slave would probably help. We went from 32 to 128 megs and the lagging more or less went away. But its neither cheap nor will it be enough in all situations.
Buying a third server will probably not help that much though, you will most likely just get another lagging slave.
I am currently working for a company that has a website running mysql/php (all tables are also using the MYISAM table type).
We would like to implement replication, but I have read in the mysql docs and elsewhere on the internet that this will lock the tables when doing the writes to the binary log (which the slave dbs will eventually read from).
Will these locks cause a problem on a live site that is fairly write-heavy? Also, is there a way to enable replication without having to lock the tables?
If you change your table types to innodb, row level locking is used. Also, your replication will be more stable, as updates will be transactional. MyISAM replication is a long-term pain.
Be sure that your servers are version-matched, and ALWAYS be sure to shut down the master before shutting down the slaves. You can bring the master up again immediately after shutting down the slaves, but you do have to take it down.
Also, make sure you use appropriate autoextend options for InnoDB. And, while you're at it, you'll probably want to migrate away from float and double to 'decimal' (which means mysql 5.1.) That will save you some replication headaches.
That's probably a bit more than you asked for. Enjoy.
P.s., yes the myisam locks can cause problems. Also, innodb is slower than myisam, unless myisam is blocking for a huge select.
In my experience DBAing a write-heavy site, writing a binary log adds no perceivable problems with locking or performance on the master. If you want to benchmark it, simply turn binary logging on. I really don't think tables are locked to write queries to the binary log.
Table locking on the slave is quite another thing, however. Replication is serial: each query runs to completion before the slave runs the next one. So long updates will cause replication to fall behind temporarily. If your application is intending to use replication for scale-out, it needs to know how to accomodate this.
The solution with the myisam table type is not 'better'. However, you can get by with it.
The best you can do, is make sure your slave and master run on the same hardware (FPU differences can create replication errors), as well as making sure you are running the same version numbers on your MySQL servers.
The following link answers your questions. Specifically, locks in MyISAM tables have less of a chance of blocking writes if there are no deletes going on. So a table that doesn't have delete holes in it will perform faster in a replicated setup.
http://dev.mysql.com/doc/refman/5.1/en/internal-locking.html
You can mitigate the effect of 'holes' by have a DBA export/import periodically during scheduled downtimes (especially after mass deletes.) Also, make sure your slave databases don't go down with the master still running. That will save you many, many issues.