(2006) MySQL server has gone away - mysql

I've read so many threads as well as the MySQL documentation about this issue and nothing suggested seems to work.
Here's my.cnf
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
default-storage-engine=INNODB
character-set-server=utf8
collation-server=utf8_bin
interactive_timeout = 2880000
wait_timeout = 2880000
net_write_timeout = 6000
net_read_timeout = 6000
delayed_insert_timeout = 6000
key_buffer = 256M
key-buffer-size = 32M
max_allowed_packet = 600M
thread_stack = 256K
thread_cache_size = 8
max-connections = 500
thread-cache-size = 50
open-files-limit = 65535
table-definition-cache = 4096
table-open-cache = 10240
query-cache-type = 0
query_cache_limit = 2M
query_cache_size = 32M
myisam-recover = BACKUP
innodb_buffer_pool_size = 384M
innodb_additional_mem_pool_size = 20M
innodb_log_file_size = 10M
innodb_log_buffer_size = 64M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 180
log_error = /var/log/mysql/error.log
expire_logs_days = 10
max_binlog_size = 100M
[mysqldump]
quick
quote-names
max_allowed_packet = 64M
[isamchk]
key_buffer = 32M
In addition I ran queries in the MySQL CLI to make sure my settings were sticking, and they appear to be:
mysql> select ##global.wait_timeout, ##session.wait_timeout;
+-----------------------+------------------------+
| ##global.wait_timeout | ##session.wait_timeout |
+-----------------------+------------------------+
| 2880000 | 2880000 |
+-----------------------+------------------------+
mysql> select ##global.max_allowed_packet, ##session.max_allowed_packet;
+-----------------------------+------------------------------+
| ##global.max_allowed_packet | ##session.max_allowed_packet |
+-----------------------------+------------------------------+
| 629145600 | 629145600 |
+-----------------------------+------------------------------+
Server environment: Ubuntu Server 14.04LTS
MySQL version: 5.6
This is a dedicated MySQL server, it has no other apps on it.
I am not running out of memory:
MemTotal: 32948824 kB
MemFree: 31494136 kB
Cached: 281624 kB
SwapCached: 0 kB
SwapTotal: 33550332 kB
SwapFree: 33550332 kB

I was finally able to fix this issue but bypassing the MySQL Workbench Migration Tool and using mysqldup to generate the database .sql file used to restore to the server. Here's what I did:
Deleted all databases from the MySQL server that was timing out
Exported all the data from our backup database via mysqldump
Restored that dump file to the new MySQL server (the one that was
timing out)
Ran mysql_upgrade, all tables OK Executed the query: everything now
works
Here's the strange thing: mysqlcheck and mysql_upgrade, when run on the database that was somehow timing out, were returning status OK and not finding any errors. I do not know why, but I see this is a pretty strange and annoying problem, considering that's exactly what mysqlcheck is for.
Anyway if you're having this problem try to restore the database from an older backup (if you have one) and see if that works.

Related

mariadb high Disk IO and IO Wait

We have two mariadb servers Master and Slave each on PR and DR, We are observing high disk io above 90% and iowait some time above 20, when mariadb backup start at night iowait goes above 30 and system become unresponsive causing watch dog process timeout and restart mariadb. when running dd command
sudo dd if=/dev/zero of=/data/test2.img bs=512 count=1000 oflag=dsync
it write data between 300 to 500 Kb/s, we have checked storage level as well but no issue found at storage level as same storage has been in used for other DB's (Oracle) and no such IO issues there. i need help to identify this issue.
following is server.cnf of mariadb
[mysqld]
symbolic_links = 0
local_infile = 0
basedir = /usr
datadir = /data/mdb_data
pid_file = /var/lib/mysql/mysqld.pid
log_error = /var/log/mariadb/mysqld.log
bind-address = ::
port = 3306
userstat = 1
plugin-load-add = server_audit=server_audit.so
server_audit = FORCE_PLUS_PERMANENT
server_audit_logging = on
server_audit_events = CONNECT,QUERY_DCL,QUERY_DDL
server_audit_output_type = syslog
cracklib_password_check = off
log_bin_trust_function_creators = 1
lower_case_table_names = 1
character-set-server = utf8
init_connect = SET NAMES utf8
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 1
skip-name-resolve
innodb_autoinc_lock_mode = 2
max_connections = 3000
innodb_buffer_pool_size = 58G
innodb_buffer_pool_instances = 8
innodb_log_file_size = 1024m
key_buffer_size = 16M
log-error = /var/log/mariadb/mysqld.log
skip-external-locking
slow_query_log = on
long_query_time = 1
slow_query_log_file = /var/log/mariadb/slowSQLs.log
performance_schema = on
innodb_flush_method = O_DIRECT
sync_binlog = 1
max_allowed_packet = 512m
slave_max_allowed_packet = 200m
innodb_read_io_threads = 32
innodb_write_io_threads = 32
innodb_io_capacity = 600
slave_parallel_threads = 50
slave_parallel_max_queued = 2097152
slave_parallel_mode = optimistic
rpl_semi_sync_master_enabled = 0
rpl_semi_sync_slave_enabled = 1
rpl_semi_sync_master_wait_point = AFTER_SYNC
binlog_commit_wait_usec = 10000
binlog_commit_wait_count = 1
server_audit_excl_users = repl#b.c,maxscale
slave-skip-errors = 1062,1032
[mysqld_safe]
syslog
[mariadb]
server_id = 228575792
log_slave_updates = on
log-bin
log-basename = db2
report_host = db2
from top
KiB Mem : 74046272 total, 32528492 free, 26723664 used, 14794120 buff/cache
KiB Swap: 33554428 total, 31482364 free, 2072064 used. 33507036 avail Mem

Why does Master think it's a Slave on Reboot?

In a simple MySQL replication Master-Slave configuration I have a problem where Master tries to connect to itself as a slave on reboot.
So when I restart MySQL on Master, I see errors related to the same server trying to replicate to itself and I have to manually run mysql -e "STOP SLAVE;" every time I restart MySQL.
How can I disable slave on master for good?
Here's the relevant portion of my.cnf:
## Logging
binlog_format = mixed
log_bin = /var/log/mysql/mysql-bin.log
sync_binlog = 1
pid_file = /var/run/mysqld/mysqld.pid
log_error = /var/log/mysql/error.log
#general_log = 0
#general_log_file = /var/log/mysql/general.log
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 3
expire_logs_days = 14
sql_mode = STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
# sql_mode = ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
## Replication
server_id = 200
## Master Configuration
binlog-do-db = my_db_1
binlog-do-db = my_db_2
binlog-do-db = my_db_3
binlog-do-db = my_db_4
binlog-do-db = my_db_5
binlog-do-db = my_db_6
Also, when I run SELECT * FROM mysql.user; I don't see the repl user that's allegedly a "slave" on Master.
BUT, I do see that localhost has replication grants:
mysql> select Host, User, grant_priv, Repl_slave_priv, Repl_client_priv from mysql.user;
+-----------------+---------------+------------+-----------------+------------------+
| Host | User | grant_priv | Repl_slave_priv | Repl_client_priv |
+-----------------+---------------+------------+-----------------+------------------+
| localhost | root | Y | Y | Y |
| localhost | mysql.sys | N | N | N |
Here's an example of the errors I see on Reboot (before I run STOP SLAVE; on Master):
2016-09-01T15:22:23.845505Z 384 [Note] Access denied for user 'repl'#'192.168.100.200' (using password: YES)
2016-09-01T15:22:23.845761Z 1 [ERROR] Slave I/O for channel '': error connecting to master 'repl#192.168.100.200:3306' - retry-time: 30 retries: 8, Error_code: 1045
2016-09-01T15:22:50.191636Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 6843ms. The settings might not be optimal. (flushed=15210 and evicted=0, during the time.)
Apart from this, replication is running fine. Writes to Master show up flawlessly on the real, read-only, Slave.
Full my.cnf:
[mysql]
default_character_set = utf8
[mysqld]
datadir = /var/lib/mysql
socket = /var/lib/mysql/mysql.sock
symbolic-links = 0
## Custom Configuration
skip_external_locking = 1
skip_name_resolve
open_files_limit = 20000
## Cache
thread_cache_size = 16
query_cache_type = 1
query_cache_size = 256M
query_cache_limit = 4M
## Per-thread Buffers
sort_buffer_size = 32M
read_buffer_size = 4M
read_rnd_buffer_size = 8M
join_buffer_size = 2M
## Temp Tables
tmp_table_size = 1024M
max_heap_table_size = 1024M
## Networking
back_log = 250
max_connections = 512
max_connect_errors = 100000
max_allowed_packet = 128M
interactive_timeout = 1800
wait_timeout = 1800
character_set_client_handshake = FALSE
character_set_server = utf8mb4
collation_server = utf8mb4_unicode_ci
### Storage Engines
default_storage_engine = InnoDB
innodb = FORCE
## MyISAM
key_buffer_size = 128M
myisam_sort_buffer_size = 16M
## InnoDB
innodb_buffer_pool_size = 46G
innodb_buffer_pool_instances = 64
innodb_log_files_in_group = 2
innodb_log_buffer_size = 32M
innodb_log_file_size = 64M
innodb_file_per_table = 1
innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 1
## Logging
binlog_format = mixed
log_bin = /var/log/mysql/mysql-bin.log
sync_binlog = 1
pid_file = /var/run/mysqld/mysqld.pid
log_error = /var/log/mysql/error.log
#general_log = 0
#general_log_file = /var/log/mysql/general.log
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 3
expire_logs_days = 14
sql_mode = STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
# sql_mode = ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
## Replication
# Master Server ID:
server_id = 200
# Slave Server ID:
# server_id = 300
## Master Configuration
# Comment out on Slave
binlog-do-db = db_1
binlog-do-db = db_2
binlog-do-db = db_3
binlog-do-db = db_4
binlog-do-db = db_5
binlog-do-db = db_6
## Slave Configuration
# Uncomment the following on Slave
# relay-log = /var/log/mysql/mysql-relay-bin.log
# binlog-do-db = db_1
# binlog-do-db = db_2
# binlog-do-db = db_3
# binlog-do-db = db_4
# binlog-do-db = db_5
# binlog-do-db = db_6
# log_slave_updates = 1
# read_only = 1
# slave_skip_errors = 1062
[mysqld_safe]
datadir = /var/lib/mysql
socket = /var/lib/mysql/mysql.sock
symbolic-links = 0
pid_file = /var/run/mysqld/mysqld.pid
log_error = /var/log/mysql/error.log
Also:
mysql> SHOW GLOBAL VARIABLES LIKE '%master_info_repository%';
+------------------------+-------+
| Variable_name | Value |
+------------------------+-------+
| master_info_repository | FILE |
+------------------------+-------+
For managing this kind of setups I recommend to use MHA manager. For this specific situation you may want to clean up the master_info_repository (located by default in master.info). Also, you can use --skip-slave-start on the master host to avoid this situations after failover.
I think that you must have set the master information on the master server (maybe this was a slave at some point or refreshed from one). Run
SHOW SLAVE STATUS
on the master. If the entries are not all empty then this is the cause and on reboot (without skip-slave-start being set) MySQL will try to start the slave.
To fix this, on the master, stop the slave if you did not already and run
RESET SLAVE ALL
to clear the master settings - assuming that you are using 5.5.16 or higher otherwise leave off the ALL.
This can be confirmed with another SHOW SLAVE STATUS which should show all the entries as empty.
When you reboot now the slave will not try to start.
If you prefer for some reason to leave the master settings in place, add skip-slave-start to your my.cnf under [mysqld] and the settings will then be ignored on start-up.

mysql - usage of memory still grow up

I have problem with my production mysql server. Usage of memory still grow up and I don`t know why. Trouble started when we changed the server.
My mysql version: 5.5.44-0+deb8u1-log - (Debian).
My my.cnf file:
[mysqld_safe]
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
skip-external-locking
key_buffer_size = 5M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
tmp_table_size = 384M
max_heap_table_size = 384M
table_open_cache = 7000
open_files_limit = 14000
interactive_timeout=3600
wait_timeout=3600
myisam-recover_options = BACKUP
max_connections = 150
query_cache_limit = 8M
query_cache_size = 127M
slow_query_log_file = /var/log/mysql/mysql-slow.log
slow_query_log = 1
long_query_time = 2
expire_logs_days = 10
max_binlog_size = 100M
innodb_file_per_table
innodb_buffer_pool_instances = 9
innodb_buffer_pool_size = 10000M
innodb_log_file_size = 2000M
It is something wrong in my configuration?
EDIT
For production we have dedicated server:
4 x Intel(R) Xeon(R) CPU E5-2680 v2 # 2.80GHz
20 GB RAM memory
40 GB na DHH - data base have now 15 GB
in the data base is on this moment 650 tables on the innoDB engine
Screenshot of htop:
htop of proces on server

How can wait_timeout/interactive_timeout be ignored?

My mySQL server is not respecting wait_timeout or interactive_timeout of 15 seconds. Queries just keep going and going past the respective 15 seconds. Below is the my.cnf -
[mysqld]
# Settings user and group are ignored when systemd is used (fedora >= 15).
# If you need to run mysqld under different user or group,
# customize your systemd unit file for mysqld according to the
# instructions in http://fedoraproject.org/wiki/Systemd
user=mysql
#skip-innodb
#ignore-builtin-innodb
#default-storage-engine = myisam
#log-queries-not-using-indexes
#key_buffer = 6M
key_buffer_size = 1024M
max_allowed_packet = 64M
thread_stack = 256K
thread_cache_size = 200
max_connections = 200
table_cache = 128K
tmp_table_size = 24M
max_heap_table_size = 24M
join_buffer_size = 1M
query_cache_limit = 32M
query_cache_size = 8M
read_buffer_size = 1M
# concurrent_insert = ALWAYS
general_log = 0
general_log_file = /var/log/mysql/general.log
low_priority_updates=1
log_warnings=2
#log_error=/var/log/mysql/mysql_error.log
slow-query-log = 1
slow-query-log-file = /var/log/mysql/mysql-slow.log
long_query_time=1
wait_timeout=15
interactive_timeout=15
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# InnoDB Settings
innodb_buffer_pool_size = 768M
innodb_log_file_size = 100M
innodb-file-per-table = 1
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Semisynchronous Replication
# http://dev.mysql.com/doc/refman/5.5/en/replication-semisync.html
# uncomment next line on MASTER
#;plugin-load=rpl_semi_sync_master=semisync_master.so
# uncomment next line on SLAVE
#;plugin-load=rpl_semi_sync_slave=semisync_slave.so
# Others options for Semisynchronous Replication
#;rpl_semi_sync_master_enabled=1
#;rpl_semi_sync_master_timeout=10
#;rpl_semi_sync_slave_enabled=1
# http://dev.mysql.com/doc/refman/5.5/en/performance-schema.html
#;performance_schema
#log-queries-not-using-indexes
default-storage-engine=MyISAM
#log-queries-not-using-indexes
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
I ran the command on the server, but when I do mytop - there are queries that are more then 15 seconds.
mysql> select ##global.wait_timeout, ##session.wait_timeout;
+-----------------------+------------------------+
| ##global.wait_timeout | ##session.wait_timeout |
+-----------------------+------------------------+
| 15 | 15 |
+-----------------------+------------------------+
1 row in set (0.00 sec)
Any ideas what I am doing wrong?
wait_timeout is documented as follows:
The number of seconds the server waits for activity on a
noninteractive connection before closing it. This timeout applies only
to TCP/IP and Unix socket file connections, not to connections made
using named pipes, or shared memory.
From your question it sounds like you're trying to use it and interactive_timeout abruptly to abandon long-running queries. But that's not what it's for.

mysql my.cnf for a 50gb of RAM server

I am having load problems on my database server (dual quad core 50 gb ram)
load average: 4.26, 6.78, 11.25
here is the current my.cnf, can i optimize it?
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/english
skip-external-locking
key_buffer = 150M
key_buffer_size = 512M
max_allowed_packet = 16M
thread_stack = 128K
thread_cache_size = 128
open-files = 8192
myisam-recover = BACKUP
max_connections = 1000
table_cache = 8096
join_buffer_size = 2M
tmp_table_size = 10G
max_heap_table_size = 10G
innodb_buffer_pool_size = 10G
innodb_additional_mem_pool_size = 128M
innodb_log_file_size = 100M
log-slow-queries = /var/log/mysql/slow.log
sort_buffer_size = 5M
net_buffer_length = 5M
read_buffer_size = 2M
read_rnd_buffer_size = 12M
thread_concurrency = 16
ft_max_word_len = 30
ft_min_word_len = 2
#thread_concurrency = 10
query_cache_limit = 4M
query_cache_size = 512M
The website has about 2000 users online.
I always start out with a watch "mysqladmin pr -vv" to see which statements are the worst... then you should get an idea what variable to adjust ... most of the time it's a missing index.
You use mytop and mysqltuner to analyze using mysql resources (RAM and CPU).
Too enable the option to log slow queries:
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 3
And check out this post about ntpd service:
MySQL high CPU usage
I have a dedicated server for a high rate of transactions:
max_allowed_packet=16M
key_buffer_size=8M
innodb_additional_mem_pool_size=10M
innodb_buffer_pool_size=512M
join_buffer_size=40M
table_open_cache=1024
query_cache_size=40M
table_definition_cache=256
innodb_additional_mem_pool_size=10M
key_buffer_size=16M
max_allowed_packet=32M
max_connections = 300
query_cache_limit = 10M
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 3
Regards.