I have a master and slave database running on different nodes. The master DB is subjected to huge no. of inserts/updates. The master DB size is close to 6 GB, while the log files are now occupying a space of more than 120 GB. I am running out of disk space and need to get rid of the log files.
Will deleting the log files in anyway affect the slave DB ? Presently, the slave is just a couple of seconds behind the master.
Is there someplace where I can see what steps I need to follow to delete those files
eg.
1)Shut down the slave
2)Shut down the master
3)Delete the log files
4)Start the Master
5)Start the Slave
Do I need to inform the slave that the log files have been deleted ?? If yes, what is the way to do it ?
Any help would be appreciated.
Thanks
Yes, you can delete the OLD bin_log files. Make sure they're super old.
Also, I would do mysql flush_logs
You should also set your config file to expire your log files after X days.
This MySql documentation has all the information beautifully laid out
http://dev.mysql.com/doc/refman/4.1/en/purge-binary-logs.html
Related
I need to make working mysql replication from master to slave. (tried it once already)
The database is quite large (over 100GB) and it will take some hours to make it ready for new slave.
The database has MyIsam and innoDB engine and both are being written
I think my only choice is to copy the data files from master to a new slave? (or make a database dump which im referring later in the topic of ROUND 2)
Before that I have to run down all the services which uses the database and
make writelock for tables or should i shut down the whole database?
After data directory sync to the new replication server I started it up and the database with the tables was there. First error that I got rid off by changing bin.log to 007324 and position to 0.
Error 1:
140213 4:52:07 [ERROR] Got fatal error 1236: 'Could not find first log file name in binary log index file' from master when reading data from binary log
140213 4:52:07 [Note] Slave I/O thread exiting, read up to log 'bin-log.007323', position 46774422
After that I got new problems from database and this error came out from every table.
Error 2:
Error 'Incorrect information in file: './database/table.frm'' on query. Default database: 'database'.
Seems that something went wrong.
ROUND 2!
After this scene I started to think that can this be done without long service break.
Master database has been already configured and it works ok to another slave.
So i did some googling and this is what i came up with.
Making read lock to tables:
FLUSH TABLES WITH READ LOCK;
Taking dump:
mysqldump --skip-lock-tables --single-transaction --flush-logs --master-data=2 -A > dbdump.sql
Packaging and moving:
gzip (pigz) the the dbdump and moving it to slave server after that finding the MASTER_LOG_FILE and MASTER_LOG_POS from the dump.
After that i don't think that i want to import the dbdump.sql because its over 100GB and
will take time. So i think SOURCE would be ok option for it.
On SLAVE server:
CREATE DATABASE dbdump;
USE dbdump;
SOURCE dbdump.db;
CHANGE MASTER TO MASTER_HOST='x.x.x.x',MASTER_USER='replication',MASTER_PASSWORD='slavepass',
MASTER_LOG_FILE='mysql-bin.000001',MASTER_LOG_POS=X;
start slave;
SHOW SLAVE STATUS \G
I haven't tested this yet, am I on to something?
--bp
Realize that issuing a SOURCE command is the same as running an import of the dumped SQL from shell. Either way, it is going to take a long time. Outside of that, you have the steps correct - flush table with read lock on master, make a database dump of master, make sure you note master binlog coordinates, import dump on slave, set binlog coordinates, start replication. Do not work with the raw binaries unless you REALLY know what you are doing (especially for INNODB tables).
If you have a number of large tables (i.e. not just one big one), you could consider parallelizing your dumps/imports by table (or groups of tables) to speed things along. There are actually tools out there to help you do this.
You CAN work with the raw binaries, but it is not for the faint of heart. In the past, I have used rsync to differentially update the raw binaries between master and slave (you still must use flush table with read lock and gather master binlog coordinates before doing this). For MyISAM tables this works pretty well actually. For InnoDB, it can be more tricky. I prefer to use the option to set InnoDB to write index and data files per table. You would need to rsync the ibdata* files. You would delete ib_logfile* files from slave.
This whole thing is a bit of a high wire act, so I would not resort to doing this unless you have no other viable options. Absolutely take a traditional SQL dump before even thinking about attempting a binary file sync, and each time until you are VERY comfortable that you actually know what you are doing.
I have a ~80GB MySql DB.
I want to create a replication on that DB while having the current DB as master and setting up a slave for it.
My main question is how can i move the data (all 80GB) of it from the master to the new slave with as minimal downtime as possible, preferably none.
my initial thought was to stop the DB (after taking the log position), and then copy the files from the mysqldata lib, and then re start the server but just copying the files would take ~2 hours.
any thoughts?
On July 8, 2011 I addressed a similar question. I wrote scripts that would zap binary logs and starting performing an rsync.
On June 16, 2011, I wrote a post contrasting doing an rsync versus using XtraBackup.
On May 23, 2011, I discussed what considerations to make when doing this kind of backup.
Rather than reinvent the wheel and rewrite in the information I already wrote in those posts, I simply provided the links to my own posts that address this question.
Please read them carefully.
Give it a Try !!!
CAVEAT
The only downtime in my rsync algorithm is when after you have performed multiple rsyncs as specified, you shutdown mysql, perform one more rsync, and then start up mysql.
I would like to clarify the reason for the shutdown:
When you shutdown mysql:
All open MyISAM tables are closed, There is a header that marks how many file handles are open to the MyISAM table. That must be at zero(0) for the table to be OK. Otherwise, a closed MyISAM tables with a nonzero value in this header field marks the table as crashed and in need of a table repair. Shutting down mysql cleans all of that up.
All InnoDB tables that have either data pages or index pages in the Buffer Pool that are marked dirty needs to be flushed to disk. Performing a shutdown triggers a full flush of the Buffer Pool. Naturally, the bigger the pool and the higher the number of dirty pages, the longer the Buffer Pool flush time will be. To shorten this phase of the mysqld's shutdown, run SET GLOBAL innodb_max_dirty_pages_pct = 0; before performing any of the rsyncs. All transactions are completed (either commited or rolled back).
I think you have some misunderstanding.
before it start, you must enable binary log on the master
restart mysql on master
login to master
lock ALL tables from write
record the master binary position
copy binary data from master (DIRECTLY copy *.MYI, *.MYD...etc, you can copy to another location in master database)
after copy is completed, remove write lock
scp data to slave (depends on the network distance)
setup relevant master information into slave (binary log position, and remember to disable binary log)
start slave
After that, it should have huge delay on slave,
and slave will try to catch up with master automatically,
once it catch up, your slave is ready!
So, the down-time is only when you locking table and copy the binary data into another location in your master database.
docs:- http://dev.mysql.com/doc/refman/5.1/en/replication-howto.html
I've found the following tool to be of GREAT help and efficiency. The author currently works for facebook and used to work for dema in japan.
It's quite easy to set-up and you will reach 4 9's HA. ;-)
MHA tool for MySQL replication high availability
I have to say though that MySQL cluster is better, lol ;-)
I've got a small database, around 50mb. It's a master replicating to a remote server. I noticed that the relay-bin files total to over 5GB. Is it safe to delete them?
I think a better answer is relay logs can be 'deleted' but mysql should manage it automatically.
One way to do this is to check for the value of relay_log_purge.
It should be set to 1 if you want mysql to manage them:
set global relay_log_purge=1;
You would probably need to flush the logs:
flush logs;
This does not affect the binary logs.
Maybe try to resync your master and slave.
If possible, cleanup the slave by running a
reset slave
it will purge all relay binary logs.
Then set the replication again with change master to ...
You may have too much lag between your master and slave.
From the MySQL manual:
The SQL thread automatically deletes each relay log file after it has
executed all events in the file and no longer needs it. There is no
explicit mechanism for deleting relay logs because the SQL thread
takes care of doing so. However, FLUSH LOGS rotates relay logs, which
influences when the SQL thread deletes them.
No, do not delete the relay-bin files manually. What you can do is to purge the binary files using MySQL commands. See the MySQL 5.0 Manual for more on the PURGE BINARY LOGS command.
We have recently deployed a Drupal 6 install and the directory storing the mysqld-bin.XXX files /var/run/mysqld has caused the mysql server to crash. The size of these files (log) are over 1g in size each and there are tons of them. I would like to know a way we can either enable log rotation or something similar to insure this does not happen again.
Any insight is greatly appreciated.
Thanks in advance,
JN
Short answer, yes you can:
http://dev.mysql.com/doc/refman/5.0/en/binary-log.html
http://dev.mysql.com/doc/refman/5.0/en/replication-options-binary-log.html
But you need to make sure you have a good backup system in place so you can take snapshots of your DB, then recover with your binlog.
Specifically, check the
max_binlog_size
option
Set this to something like 128 or 256Mb
then you can then just add on a cron job to copy old binlogs off with each database dump, then delete old binlogs.
I am new in Mysql Replication,
My problem is : When i Give SHOW SLAVE STATUS Command on Slave, Master-bin-log files are not updated, i reset it and update the Master-bin-log file with the command CHANGE MASTER TO MASTER , still master-bin-log files are not updated when i check with the SHOW SLAVE STATUS command.
Addition to that, at the appropriate path Master-bin-log files are generated but not updated.
Please help me get out of this problem...
Thanks in advance!!!!
Riddhi
From what you are saying, I assume that you want binary logging on the slave.
I think the problem stems from something that has bitten me many times.
Please add this to /etc/my.cnf on the slave:
[mysqld]
log-slave-updates
Without this, the data is updated from the relay logs but the transaction is not recorded in the slave's binary logs.
Adding this is crucial, especially if you plan to use the slave as a master someday.