I have a primary mysql server and 2 slaves as backups.
One of the slaves has been outfitted with solid state storage and due to this is used heavily for reporting.
Some of the data generated takes some time (around half an hour to an hour in some cases) and uses and generates allot of data (on the order of a couple gigs, which makes me hesitant to use transactions). The reporting tables are but a small subset of the total database so fully shutting down replication is somewhat out of the question.
The issue at hand is reports generated while the data is being generated are obviously incomplete and wrong.
What would be the best way to lock the tables on both the master and the reporting server?
Would the "LOCK TABLES" statement be replicated to the slaves or would my best course of action be to generate the data in temporary tables and then copy them to the final table in one INSERT ... SELECT statement.
Try the following
You could try the following:
Step 01) On the Master, run this
FLUSH TABLES WITH READ LOCK; SELECT CONNECTION_ID(); SELECT SLEEP(300);
Step 02) SHOW SLAVE STATUS\G on both Slaves (or just the Reporting Slave)
Step 03) Repeat Step 02 Until
Relay_Log_Space stops changing
Relay_Log_Pos stops changing
Seconds_Behind_Master is 0
At this point, since both Slaves have not received any new SQL to process, you have effectively frozen MySQL on the Slaves at the Same Point-In-Time as the Master
Step 04) On the Slaves (or just the Reporting Slave), run STOP SLAVE;
Step 05) On the Master, (if the CONNECTION_ID() return 789) run mysql> KILL 789; in another mysql session.
Step 06) Run your reports
Step 07) Run START SLAVE; on the Slaves (or just the Reporting Slave)
UPDATE 2012-06-05 15:15 EDT
Since this seems a little heavy handed for the sake of a few tables in one particular schema, the simplest thing would just be to run STOP SLAVE; on the Slave you do the Reporting from.
Related
Running a regular OPTIMIZE TABLE query on a MYISAM table that usually takes a few minutes has now taken about 3 days. This also happened a week or so ago, but it did complete successfully and I wrongly assumed that the table had been repaired okay, and this time it wouldn't present a problem. Not so...
Last time, the process did not respond to KILL PROCESS and I ended up having to restart mysqld. The error log reported warnings such as:
[Warning] /usr/sbin/mysqld: Forcing close of thread 24974085 user: 'XXXX'
And when mysqld restarted, a number of other tables had been left corrupted and had to be rebuilt. Some of these tables are large, and I want to avoid this happening again. I understood that restarting mysqld should safely close tables, but it didn't seem to.
The web services that use the mySQL server will be put into maintenance mode to prevent new queries hitting the DB. But how can I safely close all open tables given that FLUSH TABLES is being blocked by the OPTIMIZE query?
SHOW OPEN TABLES;
[etc...]
114 rows in set (0.01 sec)
I really don't want 114 tables to have to be repaired. Any advice?
NB: I'm quite aware that the table being optimized will be corrupted and can live with that. I just want to minimize problems with the other tables.
Thanks.
UPDATE:
So FLUSH TABLES accepts a comma-separated list of tables... Maybe it could be as simple as flushing all open tables, but I'm still open to reassurance about this, or any other ideas. Flushing a specific table does remove it from the open tables list so this is looking promising.
http://dev.mysql.com/doc/refman/5.1/en/flush.html
FWIW, this all went very smoothly. Steps followed were something like this. I'm not sure if flushing and closing each table separately was overkill, but no tables crashed and total downtime was about 45 mins, so it was worth it:
Make a full slave backup
Redirect FQDN for the master to a dead IP so all services are down and master receives no new queries
When master is receiving no new queries, flush and close all tables individually
Stop mysql on master
Ensure OPTIMIZE TABLE does not cause problems for slave if it is replicated out
Stop mysql on slave
Copy all blog tables from slave to master
Restart master
Restart slave
Check for any corrupt tables and allow them to be repaired
Map FQDN back to the master internal IP
Sanity check of posts on main blog
Restart replication on slave and masters
Make sure replication progresses correctly (multi-master)
Make a full master backup
We're running a standard Mysql Master/Slave replication system which has been working nicely for a couple of years. So nicely in fact that we were able to recover from a server outage (which rendered the Master offline) by using the Slave DB (after turning it into a Master).
Anyway, the issue that is causing our problem occurred after we recovered the server and returned the original Master to being a Master and the original slave back to being a Slave.
It was caused by me being an idiot - not by anything else!
Basically a write occurred on the Slave - meaning that a write on the same table on the Master cannot be replicated across due to a Duplicated Primary Key error.
As I said, my fault.
I can delete the record from the Slave that was erroneously written there - thus removing the conflicting ID from the Slave but I want the record from the Master DB.
My question is this: I know that I can 'fix' the replication by skipping over the relevant query from the Master (by setting the SQL_SLAVE_SKIP_COUNTER to 1) - but that will mean I lose the record from the Master.
So, is there a way of 'replaying' the errored replication write? Basically pointing the replication to be one query back?
Or do I have to go through the whole rigmarole of dropping my Slave, repopulating it from my last successful backup and set it to be a Slave again?
Many thanks in advance.
If it's a simple modification/update/... why don't you simply play it on the master (manually or taking it from the slave binary log if you have one), with slave's slave process off, then set the new replication pointer (file & position) on the slave (show master status; on master), and restart the slave?
The answer was actually staring me in the face (isn't it always!).
The duplicate Primary key error was caused by me doing an insert on the Slave and an insert to the same table on the master - meaning both new records had the same ID.
Luckily for me, I did not need the insert on the Slave, so I simply deleted it - meaning that there was only one record with the primary key value.
I just needed a way to get the replication to attempt to replicate the record across - and this was easy.
Just restart the Slave!
So:
mysql root:(none)>START SLAVE;
Query OK, 0 rows affected (0.03 sec)
mysql root:(none)> _
I have several slave dbs replicated from the same master db, however, for one of the slaves, i would like to keep it as a backup db, which will never have rows updated or deleted.
basically the purpose is to have a backup db with all rows stored by using the replication(mysqldump is waaay slow to do the backup), no update/delete query get replicated, insert query only. i know there will be some conflicts going on no doubt, but still wonder if any filtering options on statement/query on the slave end or any other solutions.
You should never run a production database without a working backup scheme in place - at least as long as you value your data. If you fear that a wrong sql instruction can ruin your database, then you may try point in time recovery.
If you already use replication your master server will log all write/update operations to its binlog - which it will send to the slave servers for replication. You can do for example nightly backups of you complete database. If you destroy your database in the morning, you can import the backup from the night and reapply the instructions from the binlog from after the backup till before the instruction that killed your database.
You could then skip this instruction and apply the instructions that came afterwards. This can also cause consistency issues, as the instruction after the skipped instruction see different data in the database as they did when they were originally executed.
I have similar problem. I know it's old thread but it can help others:
link: mysql replication works only if I choose database by USE database
I have a master/slave replication on my MySql DB.
my slave DB was down for a few hours and is back up again (master was up all the time), when issuing show slave status I can see that the slave is X seconds behind the master.
the problem is that the slave dont seem to catch up with the master, the X seconds behind master dont seem to drop...
any ideas on how I can help the slave catch up?
Here is an idea
In order for you to know that MySQL is fully processing the SQL from the relay logs. Try the following:
STOP SLAVE IO_THREAD;
This will stop replication from downloading new entries from the master into its relay logs.
The other thread, known as the SQL thread, will continue processing the SQL statements it downloaded from the master.
When you run SHOW SLAVE STATUS\G, keep your eye on Exec_Master_Log_Pos. Run SHOW SLAVE STATUS\G again. If Exec_Master_Log_Pos does not move after a minute, you can go ahead run START SLAVE IO_THREAD;. This may reduce the number of Seconds_Behind_Master.
Other than that, there is really nothing you can do except to:
Trust Replication
Monitor Seconds_Behind_Master
Monitor Exec_Master_Log_Pos
Run SHOW PROCESSLIST;, take note of the SQL thread to see if it is processing long running queries.
BTW Keep in mind that when you run SHOW PROCESSLIST; with replication running, there should be two DB Connections whose user name is system user. One of those DB Connections will have the current SQL statement being processed by replication. As long as a different SQL statement is visible each time you run SHOW PROCESSLIST;, you can trust mysql is still replicating properly.
What binary log format are you using ? Are you using ROW or STATEMENT ?
SHOW GLOBAL VARIABLES LIKE 'binlog_format';
If you are using ROW as a binlog format make sure that all your tables has Primary or Unique Key:
SELECT t.table_schema,t.table_name,engine
FROM information_schema.tables t
INNER JOIN information_schema .columns c
on t.table_schema=c.table_schema
and t.table_name=c.table_name
and t.table_schema not in ('performance_schema','information_schema','mysql')
GROUP BY t.table_schema,t.table_name
HAVING sum(if(column_key in ('PRI','UNI'), 1,0)) =0;
If you execute e.g. one delete statement on the master to delete 1 million records on a table without a PK or unique key then only one full table scan will take place on the master's side, which is not the case on the slave.
When ROW binlog_format is being used, MySQL writes the rows changes to the binary logs (not as a statement like STATEMENT binlog_format) and that change will be applied on the slave's side row by row, which means a 1 million full table scan will take place on the slave's to reflect only one delete statement on the master and that is causing slave lagging problem.
"seconds behind" isn't a very good tool to find out how much behind the master you really is. What it says is "the query I just executed was executed X seconds ago on the master". That doesn't mean that you will catch up and be right behind the master the next second.
If your slave is normally not lagging behind and the work load on the master is roughly constant you will catch up, but it might take some time, it might even take "forever" if the slave is normally just barely keeping up with the master. Slaves operate on one single thread so it is by design much slower than the master, also if there are some queries that take a while on the master they will block replication while running on the slave.
Just check if you have same time and timezones on both the servers, i.e., Master as well as Slave.
If you are using INNODB tables, check that you have innodb_flush_log_at_trx_commit to a value different that 0 at SLAVE.
http://dev.mysql.com/doc/refman/4.1/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit
We had exactly the same issue after setting up our slave from a recent backup.
We had changed the configuration of our slave to be more crash-safe:
sync_binlog = 1
sync_master_info = 1
relay_log_info_repository = TABLE
relay_log_recovery = 1
I think that especially the sync_binlog = 1 causes the problem, as the specs of this slave is not so fast as in the master. This config option forces the slave to store every transaction in the binary lo before they are executed (instead of the default every 10k transactions).
After disabling these config options again to their default values I see that the slave is catching up again.
Just to add the findings in my similar case.
There were few bulk temporary table insert/update/delete were happening in master which occupied most of the space from relay log in slave. And in Mysql 5.5, since being single threaded, CPU was always in 100% and took lot of time to process these records.
All I did was to add these line in mysql cnf file
replicate-ignore-table=<dbname>.<temptablename1>
replicate-ignore-table=<dbname>.<temptablename2>
and everything became smooth again.
Inorder to figure out which tables are taking more space in relay log, try the following command and then open in a text editor. You may get some hints
cd /var/lib/mysql
mysqlbinlog relay-bin.000010 > /root/RelayQueries.txt
less /root/RelayQueries.txt
If u have multiple schema's consider using multi threaded slave replication.This is relatively new feature.
This can be done dynamically without stopping server.Just stop the slave sql thread.
STOP SLAVE SQL_THREAD;
SET GLOBAL slave_parallel_threads = 4;
START SLAVE SQL_THREAD;
I have an issue similar to this. and both of my MySQL server hosted on AWS EC2 (master and replication). by increasing EBS disk size (which automatically increased IOPS) for MySQL slave server, its turned out the solution for me. R/W Throughput and bandwidth is increased R/W latency were decreased.
now my MySQL database replication is catching up to the master. and Seconds_Behind_Master was decreased (it was got increased from day to day).
so if you have MySQL hosted on EC2. I suggest you tried to increase EBS disk size or its IOPS on the slave.
I know it's been a while since OP asked but it would have helped me to read the following answer.
In /etc/mysql/mysql.cnf :
[mysql]
disable_log_bin
innodb_flush_log_at_trx_commit=2
innodb_doublewrite = 0
sync_binlog=0
disable_log_bin REALLY carried the trick for me.
I'm looking for a way to have 2 databases running; 1 at the office and 1 at a data center. When at the office, employees would connect locally but when outside the office they would connect remotely to the data center. Both databases would be fully synchronized. That means that an employee could log in and update a record on the data center and that change would replicate instantly to the office server (or vice-versa). So, either user could edit the same record.
So, the typical scenario of auto_increment_offset etc. won't work because, in this case, each server has to be able to update the same record.
Am I missing something obvious? I can't think of a viable way to handle this. 2 users on 1 db can modify the same records so there HAS to be a way to do the same in this type of setup. I just can't think of one.
You can not do this reliably with asynchronous replication. Consider this simple example:
site A: update foo set bar = 1
site B: update foo set bar = 2
Some time later (milliseconds or hours depending) the transactions are replicated to each site:
site A: update foo set bar = 2
site B: update foo set bar = 1
foo.bar will now be 2 on site A and 1 on site B. There is nothing in MySQL that will detect this or prevent it from happening.
Either partition the problem so the masters "own" different parts of the data or elect a primary master for all updates. Or implement some global locking mechanism.
With one database things are completely different since there is synchronous locking going on internally.
Here's how we did it:
In master-master configuration there are 2 masters and 2 slaves. Both are masters and slaves at the same time:
master1 => slave1[master2]
[slave1]master1 <= slave2
enable binlogging on master server - my.cnf file under [mysqld] section:
log-bin=mysql-bin.log
relay_log=relay-bin.log
server-id = 1
log-slave-updates
auto_increment_increment = 10
auto_increment_offset = 1
You may need to add:
skip-slave-start option too, to prevent from start slave at database startup.
Note:
Each server should use different server-id
auto_increment_offset should differs
Auto increment options:
auto_increment_increment controls the increment between successive AUTO_INCREMENT values.
auto_increment_offset determines the starting point for AUTO_INCREMENT column values.
With auto_increment_increment = 10 and auto_increment_offset=5, auto_inc values would be 5, 15, 25..
Dedicated user:
GRANT REPLICATION SLAVE ON . TO 'replication'#'%' IDENTIFIED BY 'some_password';
Create consistent, binary backup of MySQL database (needed to setup a replication) using LVM but the same steps will apply to ZFS/UFS snapshot and other popular techniques:
connect to MySQL database on master (server1) and run "FLUSH TABLES WITH READ LOCK". It might take few second to complete flush operation and it will lock all tables! It is imporant to keep it in mind, because such operation may impact in production load in some cases. It's better to schedule replication setup to peak-off hours.
Create LVM snapshot (server1): lvcreate -L10G -s -n mysqlbackup /dev/vg/lvm_mysql_partition
Get information about master position. Using previously created connection to database run : SHOW MASTER STATUS; to get all information about current binlog, position and so on.
Copy-paste output somewhere - for future usage.
Uunlock all tables: UNLOCK TABLES;
Mount previously created filesystem: mount /dev/vg/mysqlbackup (this will allow you to access shapshot of data created).
Now you can just copy this data to the second server directly to data dir. Before you start up your database add previously mentioned parameters to my.cnf file, changing value of server-id.
Now, with master information copied somewhere and database files propagated to slave you can remove snapshot: lvremove -f /dev/vg/mysqlbackup on master (server1).
Now, you can log into server2 to check for permissions (depends on which user was used to copy files you'll need to correct it) and after that, startup MySQL instance. Database should start, perform InnoDB recovery (if you're signed it) and after a bit, you will be able to log into using command line client. Because of skip-slave-start your slave will not start by default.
Now, on both servers you need to configure replication thread by setting up master hostname, master port, password, user and info about position:
CHANGE MASTER TO master_host='IP_addr_of_server1', master_port=3306, master_user='replication', master_password='some_password', master_log_file='info_from_MASTER_STATUS', master_log_pos='info_from_MASTER_STATUS';
Now you can start replication: START SLAVE;
Check to make sure: SHOW SLAVE STATUS\G (on new machine)
Seconds_Behind_Master might not be equal to 0 because of changes done to database after you release all tables but with replication working slave should catch master pretty fast.
After your slave server catches up with master, start master-master (server1-server2) setup. Lock all tables on server2 ( FLUSH TABLES WITH READ LOCK;) and run SHOW MASTER STATUS;. On server1 set-up replication by CHANGE MASTER TO..
of course you need to change master_host to valid IP address, master_log_file and master_log_pos values. After that, UNLOCK TABLES on server2 and START SLAVE on server1.
Now you should have master-master-slave configuration with master-master replication between server1 and server2.