I have got master - to master replication installed.
It all works fine, until you break the link between two servers.
Once it's broken the data won't go from one to another if it exceeds some kind of limit.
Max_packet size is set to 500mb.
However, if you restart the server (mysqld) where the data is clogged (on secondary) it will push it throught.
Nothing in the log file related to this.. Any ideas?
Configuration for primary:
[mysqld]
innodb_file_per_table=1
server-id=6935500
log_bin=mysql-bin
binlog_format=ROW
log_error=/var/log/mysql-bin.err
#binlog_do_db=devjump_dev
binlog_do_db=crm_live
max_allowed_packet=500M
Configuration for secondary
[mysqld]
bind-address=0.0.0.0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
server-id=574271
log_bin=mysql-bin.log
binlog_format=ROW
binlog_do_db=crm_live
max_allowed_packet=500M
log_error=/var/log/mysql-bin.err
relay-log=mysqld-relay-bin
So apparently, after we break a link between these two some big packets are not being pushed into the binlog... From Primary to Secondary.
Thanks Guys,
Roman
After spending lots of hours reading and searching I have discovered that there was an issue with slave-net-timeout
It was defaulted to 3600 so it took 1 hour for the Secondary to find out there was a lag in the connection.
Resetting this to 60 seconds helped big time.
Hopefully someone might find this post useful and it will save their time.
Related
The Environment
We have a staging/test WordPress site on a CentOS server running MariaDB (version 10.3.8). We've been experimenting with a plugin called GeoDirectory (https://wpgeodirectory.com/) The plugin creates a variety of tables in the database. The database also is using the innodb_file_per_table set to ON, so the database has generated IBD files for each tablespace in the database.
The Issue
The staging server had 80 GB of storage. After making a particular change to the settings of the plugin (updating the Place Settings), we noticed that about ten minutes later the staging site was timing out when we tried to access it via the browser.
Logging in via SSH, we noticed that the machine was completely out of space. Looking for the largest files in /var, I noticed that one file was now taking up 70 GB of space (wp_g1a4rar7xx_geodir_gd_place_detail.ibd). This is the IBD file that corresponded to the table where the settings were updated.
Logging into SQL and running a count(*) on the number of records in that table, there were only 6,000 records.
Because the table had blown up to take up the entirety of the disk, even trying OPTIMIZE TABLE wouldn't work because of the lack of space.
The Question
What could have happened to make that file balloon so large so quickly? How can we avoid this happening in the future?
Reckless Speculation
Based on searches, we believe it might have something to do with the rollback/logging that InnoDB performs, and that we might be able to avoid this by making some changes in my.cnf to limiting logging. Again, it could be something completely different.
Here's the current my.cnf from /etc/my.cnf:
[mysqld]
bind-address = 0.0.0.0
# bind-address = ::ffff:127.0.0.1
local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security
risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
Thanks in advance for your help!
Based on the name of the plugin, I am guessing that the issue could be related to updates of indexed geometry columns. MariaDB Server 10.2 imported the InnoDB support for SPATIAL INDEX from MySQL 5.7, and it looks like the purge of version history is sometimes being skipped, depending on concurrent DML activity.
I have mentioned this peculiar design choice in MDEV-15284, which reports that SELECT on a SPATIAL INDEX can return inconsistent results if a concurrent ROLLBACK is being executed.
I see that mysql creates a lot of files that are named like mysql-bin.000001, mysql-bin.000002, mysql-bin.000003 etc.
I've found here that my log_bin setting is "ON".
I'm wondering how I can set this to off and if that's a smart thing to do.
I've tried this: set log_bin = "OFF"; but it resulted to this: ERROR 1238 (HY000): Variable 'log_bin' is a read only variable
On multiple websites I've found this solution and changed it in "/etc/my.cnf":
Disable MySQL binlogging
If you are not replicating, you can disable binlogging by changing
your my.ini or my.cnf file. Open your my.ini or /etc/my.cnf
(/etc/mysql/my.cnf), enter:
# vi /etc/my.cnf
Find a line that reads "log_bin" and remove or comment it as follows:
# log_bin = /var/log/mysql/mysql-bin.log
You also need to remove or comment following lines:
# expire_logs_days = 10
# max_binlog_size = 100M
Close and save the file.
Finally, restart mysql server:
# service mysql restart
But this doesn't seem to have any affect at all.
As far as I could find, these files mainly exist for replication, but I'm not replicating anything.
If you want to disable the binary logging (necessary for replication, useful for recovery), do it by commenting out the log_bin = ... line in your my.cnf, and restarting the server. mysql will run fine without it.
You can manually view the binary logs within mysql by executing show binary logs, and you can get rid of them with the purge binary logs command, if you want to leave it active and deal with them from time to time.
Turned out that there was a setting server-id=1 that I had to remove. Problem is solved now.
What I was trying to do is to tune my MySql server. Because, I need to read and load data from huge (more than 1GB) csv file. I change some lines in my.ini file. Firstly it works fine. But after I shutdown the machine next time mysql server is not opening and giving an error,
i.e. Error:1067 the process terminated unexpectedly in your machine..
I changed some lines like-
innodb_file_per_table
innodb_flush_method=O_DIRECT
innodb_buffer_pool_size=2G
innodb_additional_mem_pool_size=20M
innodb_log_file_size=256M
innodb_log_buffer_size=8M
innodb_lock_wait_timeout=120
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
innodb_purge_threads=1
innodb_flush_log_at_trx_commit=2
Please help me....
This error message appears if the Windows service launcher has quit immediately after being started.
This problem usually happens because the license key has not been correctly deployed(license.txt file in the license folder). More details about this can found here.
I have successfully synced my primary and secondary databases using Mysql Master - Master Replication( guide: https://www.digitalocean.com/community/tutorials/how-to-set-up-mysql-master-master-replication ). Well I am stuck with a weird issue. Every time the mysql service is restarted the value of mysql-bin.xxxxxx file gets incremented(eg: mysql-bin.000005 to mysql-bin.000006) and it stops the master - master replication process. How can I prevent the mysql-bin file value rotation or is there any way to deal with this increment issue so that even if the rotation happens the replication wont get affected?
Thanks :)
Update
As usual I had to find the answer for my own question, here is the solution for those who are facing the same situation. Just added the line log_slave_updates=1 to my.cnf file in both the servers.
Master - Master Repication:
Primary my.cnf:
#bind-address=127.0.0.1
server-id=1
log_bin=/var/log/mysql/mysql-bin.log
binlog_do_db=db_name_1
binlog_do_db=db_name_2
binlog_do_db=db_name_3
max_binlog_size=100M
log_slave_updates=1
auto_increment_increment=2
auto_increment_offset=1
Secondary my.cnf:
#bind-address=127.0.0.1
server-id=2
log_bin=/var/log/mysql/mysql-bin.log
binlog_do_db=db_name_1
binlog_do_db=db_name_2
binlog_do_db=db_name_3
max_binlog_size=100M
log_slave_updates=1
auto-increment-increment=2
auto-increment-offset=2
I have a MySQL 5.1.61 database running behind two load balanced Apache webservers hosting a fairly busy (100K uniques per day) Wordpress sites. I'm caching with Cloudflare, W3TC, and Varnish. Most of the time, the database server handles traffic very well. "show full processlist" shows 20-40 queries at any given time, with most being in the sleep state.
Periodically, though (particularly when traffic spikes or when a large number of comments are cleared), MySQL stops responding. I'll find 1000-1500 queries running, many "sending data", etc. No particular query seems to be straining the database (they're all standard Wordpress queries), but it just seems like the simultaneous volume of requests causes all queries to hang up. I'm (usually) still able to log in, to run "show full processlist", or other queries, but the 1000+ queries already in there just sit. The only solution seems to be to restart mysql (sometimes violently via kill -9 if I can't connect).
All tables are innodb, server has 8 cores, 24GB RAM, plenty of disk space, and the following is my my.cnf:
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
port=3306
skip-external-locking
skip-name-resolve
user=mysql
query_cache_type=1
query_cache_limit=16M
wait_timeout = 300
query_cache_size=128M
key_buffer_size=400M
thread_cache_size=50
table_cache=8192
skip-name-resolve
max_heap_table_size = 256M
tmp_table_size = 256M
innodb_file_per_table
innodb_buffer_pool_size = 5G
innodb_log_file_size=1G
#innodb_commit_concurrency = 32
#innodb_thread_concurrency = 32
innodb_flush_log_at_trx_commit = 0
thread_concurrency = 8
join_buffer_size = 256k
innodb_log_file_size = 256M
#innodb_concurrency_tickets = 220
thread_stack = 256K
max_allowed_packet=512M
max_connections=2500
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
#2012-11-03
#attempting a ram disk for tmp tables
tmpdir = /db/tmpfs01
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
Any suggestions how I can potentially improve MySQL config, or other steps to maintain database stability under heavy load?
Like has been said, think outside the box and do sone rooting around why these queries are slow or somehow hung. An oldie but a good source of problems even for (supposedly;) intelligent system engineers is load balancing causing issues across webserver or database sessions. With all that caching and load balancing going on, are you sure everything is always connecting end-to-end as intended?
I agree with alditis & Bjoern
I'm pretty noobish with mysql but running mysqltuner can reveal some config optimisations based on recent queries of the DB https://github.com/rackerhacker/MySQLTuner-perl
And if possible store the DB files on a physically separate partition from the OS, the OS can consume IO which slows the DB. Like with Bjoern's logrotate issue.
First have a look at basic system behavior at the moment of problems. Use both vmstat and iostat if you can find any issues. See if the system starts swapping (pi,po columns in vmstat) and if lots of IO is happening. This is the first step in debugging your problem.
Another source of useful information is SHOW INNODB STATUS. See for http://www.mysqlperformanceblog.com/2006/07/17/show-innodb-status-walk-through/ on how to interpret the output.
It might be that at a certain point in time your writes are killing read performance because they flush the query cache.