Ubuntu mysql stops working - mysql

I am have vps setup with ubuntu 14 server. I have lamp stack installed. Everything works perfect but mysql server stops frequently and i have to restart it manually. Here is my my.cnf mysql config file.
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
And following is my mysqltuner output.
-------- General Statistics ------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.49-0ubuntu0.14.04.1
[OK] Operating on 64-bit architecture
-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 38K (Tables: 34)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[--] Data in InnoDB tables: 11M (Tables: 101)
[!!] Total fragmented tables: 101
-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned
-------- Performance Metrics -------------------------------------------------
[--] Up for: 1h 13m 27s (2K q [0.464 qps], 130 conn, TX: 5M, RX: 332K)
[--] Reads / Writes: 81% / 19%
[--] Total buffers: 192.0M global + 2.7M per thread (151 max threads)
[!!] Maximum possible memory usage: 597.8M (121% of installed RAM)
[OK] Slow queries: 0% (0/2K)
[OK] Highest usage of available connections: 7% (12/151)
[OK] Key buffer size / total MyISAM indexes: 16.0M/196.0K
[OK] Query cache efficiency: 33.9% (493 cached / 1K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 77 sorts)
[!!] Temporary tables created on disk: 33% (166 on disk / 498 total)
[OK] Thread cache hit rate: 90% (12 created / 130 connections)
[OK] Table cache hit rate: 94% (178 open / 189 opened)
[OK] Open file limit used: 11% (116/1K)
[OK] Table locks acquired immediately: 100% (1K immediate / 1K locks)
[OK] InnoDB data size / buffer pool: 11.9M/128.0M
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours - recommendations may be inaccurate
Reduce your overall MySQL memory footprint for system stability
Enable the slow query log to troubleshoot bad queries
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Variables to adjust:
*** MySQL's maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
tmp_table_size (> 16M)
max_heap_table_size (> 16M)
And here is my memory uses
Every 5.0s: free -m Sun Jul 3 03:12:12 2016
total used free shared buffers cached
Mem: 490 476 13 64 10 183
-/+ buffers/cache: 281 208
Swap: 0 0 0
What can be wrong here. Is there some misconfig on my my.cnf or something else is killing mysql. Please help me to find the issue.

Its possible that mysql runs out of memory. It happens when your system lacks swapping ( i.e. When memory gets exhausted, programs can't be transferred from main memory to a "swap" file in order to prevent system from memory failures ).
Swap explanation from Linux.com:
Linux divides its physical RAM (random access memory) into chucks of
memory called pages. Swapping is the process whereby a page of memory
is copied to the preconfigured space on the hard disk, called swap
space, to free up that page of memory. The combined sizes of the
physical memory and the swap space is the amount of virtual memory
available.
You can tweak "innodb_buffer_pool_size" to reduce indexing and caching footprints in the memory.
Ideally, you set the size of the buffer pool to as large a value as
practical, leaving enough memory for other processes on the server to
run without excessive paging. The larger the buffer pool, the more
InnoDB acts like an in-memory database, reading data from disk once
and then accessing the data from memory during subsequent reads.
Buffer pool size is configured using the innodb_buffer_pool_size
configuration option.
According to MySql documentation:
Solution:
Add this under [mysqld] : innodb_buffer_pool_size=64M
then make a swap file:
dd if=/dev/zero of=/swapfile bs=100M count=40
mkswap /swapfile
swapon /swapfile
and add this to /etc/fstab:
/swapfile none swap sw 0 0

Related

Mariadb server constanlty crashing [Error 2013]

I know this problem was asked several times, but no solution worked for me so far. I am struggling with this for over 2 weeks and I am out of options.
System:Ubuntu 18.04 LTS
MySQL version:mysql Ver 15.1 Distrib 10.4.11-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
I suspect it may be a problem with one large database (over 4GB) I had to recover from frm idb. I know the method of recovery was successful after recovery of single table I checked and the data was there, not sure if with the other tables something is not corrupted BUT:
I tried recovery method described here and problem is even when I do mysqlcheck --all-databases I get error: 2013: Lost connection to MySQL server during query when executing 'CHECK TABLE ... ' so unable to check.
From what I see in syslog server also crashed when I log into mysql and do use epffilm;
Using other advice from SO i created /etc/my.cnf :
[mysql]
connect_timeout = 43200
max_allowed_packet = 2048M
net_buffer_length = 512M
debug-info = TRUE
also content of my /etc/mysql/my.cnf:
client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
#socket = /run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
max_connections = 100
connect_timeout = 1000000
wait_timeout = 600
max_allowed_packet = 1073741824
thread_cache_size = 128
sort_buffer_size = 4M
bulk_insert_buffer_size = 16M
tmp_table_size = 32M
max_heap_table_size = 32M
net_read_timeout = 31536000
net_write_timeout = 31536000
#
# * MyISAM
#
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched. On error, make copy and try a repair.
myisam_recover_options = BACKUP
key_buffer_size = 128M
#open-files-limit = 2000
table_open_cache = 400
myisam_sort_buffer_size = 512M
concurrent_insert = 2
read_buffer_size = 2M
read_rnd_buffer_size = 1M
#
# * Query Cache Configuration
#
# Cache only tiny result sets, so we can fit more in the query cache.
query_cache_limit = 128K
query_cache_size = 64M
# for more write intensive setups, set to DEMAND or OFF
#query_cache_type = DEMAND
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
general_log_file = /var/log/mysql/mysql.log
general_log = 1
#
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# we do want to know about network errors and such
log_warnings = 2
#
# Enable the slow query log to see queries with especially long duration
#slow_query_log[={0|1}]
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 10
log_slow_rate_limit = 1000
log_slow_verbosity = query_plan
#log-queries-not-using-indexes
#log_slow_admin_statements
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#report_host = master1
#auto_increment_increment = 2
#auto_increment_offset = 1
log_bin = /var/log/mysql/mariadb-bin
log_bin_index = /var/log/mysql/mariadb-bin.index
# not fab for performance, but safer
#sync_binlog = 1
expire_logs_days = 7
max_binlog_size = 100M
# If applications support it, this stricter sql_mode prevents some
# mistakes like inserting invalid dates etc.
#sql_mode = NO_ENGINE_SUBSTITUTION,TRADITIONAL
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
default_storage_engine = InnoDB
innodb_buffer_pool_size = 6G
innodb_log_buffer_size = 512M
innodb_file_per_table = 1
innodb_open_files = 400
innodb_io_capacity = 400
innodb_flush_method = O_DIRECT
#
#
# Helpful with recovery
#
innodb_force_recovery=3
innodb_purge_threads=0
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
#
# Allow server to accept connections on all interfaces.
#
#bind-address=0.0.0.0
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0
[mysqldump]
quick
quote-names
max_allowed_packet = 1024M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completion
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!include /etc/mysql/mariadb.cnf
!includedir /etc/mysql/conf.d/
Any insight is appreciated.
Update:
I put a crash log on pastenbin
Hello this has already occurred to me ...
In my case my server was low on memory ...
And my database was full of posts.
That is ... he had a lot of information on two types of decodes. (MyISAM & Innodb) in the same bank ...
The solution I found was to export this database ...
This is a backup ...
Create a database from scratch by deciding on single encoding (Innodb) only.
And doing the import little by little checking all the tables and possible errors.
Another option I made was to remove what was old and put it in another database. To be part of another site.
Data integration would be through links between the two sites ...
I hope it helped you!
Strong hug. Good luck!
Send news!
Where did you get these?? (I will comment on it.)
max_allowed_packet = 2048M
net_buffer_length = 512M
They are much too large. These could explain running out of memory and crashing.
Remove those from my.cnf. If some large values are needed for a single load, then set them only within that connection. And shrink innodb_buffer_pool_size by 2.5G to allow room for such a one-time task.
How much RAM do you have?

Mysql in Docker container never releases memory and regularly crashes cause out of memory

Host with 32GB RAM. Mysql 5.7.21 works in Docker container, runs with the following command:
docker run --restart=always --name mydb5721 -v ... -e MYSQL_ROOT_PASSWORD=... -e MYSQL_USER=... -e MYSQL_PASSWORD=... -p ...:3306 --memory 14G --memory-swap 14G --health-interval=10s --health-timeout=10s --health-retries=3 --health-cmd='/bin/bash /var/lib/mysql/healthcheck.sh' mysql:5.7.21 --verbose &
so without swap and with 14GB RAM.
Memory usage almost always grows and is never cleared, even after my large requests are completed. And it crashes when exceeds limit 14GB. "Flush query cache" never changes this graph, "flush tables" usually not greatly affected and then grows fast again. When the db-requests are large and heavy, the limit is reached quickly, otherwise - slowly.
/etc/mysql/mysql.conf.d:
[mysqld]
innodb_force_recovery = 6
user=root
max_allowed_packet=1500M
bind-address=0.0.0.0
key_buffer_size=1024M
max_connections=200
table_open_cache=64
query_cache_limit=4M
query_cache_size=512M
innodb_buffer_pool_size=7G
server-id=2
log-slave-updates=1
innodb_log_buffer_size = 16M
innodb_log_file_size = 250M
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = /var/lib/mysql
log-error = /var/log/mysql/error.log
symbolic-links=0
skip-host-cache
skip-name-resolve
sql-mode="NO_AUTO_VALUE_ON_ZERO"
innodb_file_per_table = 1
table_open_cache = 2048
innodb_open_files = 2048
sort_buffer_size = 128M
read_buffer_size = 128M
read_rnd_buffer_size = 1M
thread_stack = 128M
query_cache_type = 0
thread_cache_size = 32
max_heap_table_size = 256M
tmp_table_size = 1G
innodb_buffer_pool_instances = 4
innodb_read_io_threads = 8
innodb_write_io_threads = 8
performance_schema = 0
innodb_flush_log_at_trx_commit = 2
slow_query_log=1
long_query_time=40
slow-query-log-file=/var/log/mysql/slow_queries.log
general_log=0
innodb_flush_method=O_DIRECT
innodb_tmpdir=/tmp
secure-file-priv = ""
My main questions:
- is the Docker guilty or Mysql (cause I never have this problem with mysql without docker)?
- why memory never releases even after my queries are completed?
- will the solution migration to MariaDB?
Your ulimit -a report indicates Open files limit of 1024. In LX, ulimit -n 24000 will enable more file handles for MySQL to use.
For the new limit to be persistent over LX shutdown, restart review this URL
https://glassonionblog.wordpress.com/2013/01/27/increase-ulimit-and-file-descriptors-limit/
Your details may be slightly different.
Review my recent COMMENT about DOCKER missing CLOSE() to release resources.
Rate Per Second=RPS Suggestions to consider for your my.cnf [mysqld] section
# 20180924 1442 mysqlservertuning.com
read_rnd_buffer_size=256K # from 1M to reduce handler_read_rnd_next RPS
read_buffer_size=256K # from 128M will increase handler_read_next RPS
innodb_io_capacity_max=20000 # from 2000 to take advantage of SSD IOPS capacity
innodb_io_capacity=10000 # from 200 to take advantage of SSD IOPS capacity
tmp_table_size-256M # from 1G to be matched to max_heap_table_size
key_buffer_size=64M # from 1G to conserve RAM, key_blocks_used is less than 1%
innodb_lru_scan_depth=100 # from 1024 to conserve CPU cycles every SECOND
innodb_thread_concurrency=24 # from 0 (unlimited) for your 16 cpu's reported by iostat
table_open_cache=4096 # from 2048 to reduce opened_tables count
table_definition_cache=2048 # from 1424 to reduce opened_table_definitions count
query_cache_size=0 # from ~512M because query_cache_type=0 (off) to conserve RAM
query_cache_limit=0 # from ~4M because query_cache_type=0 (off) to conserve RAM
Considering your situation, backup your current my.cnf, copy the whole block, starting
with # date time, my web address to END of [mysqld] section, lead SAME NAMED variable with # AND spacebar to disable the line prior to new BLOCK of variables, stop service / start service.
Until the docker world gets the CLOSE() in place, you will still have crashes, but a little bit later in the day. Many of your PER CONNECTION variables were OVERPROVISIONED, some may still be, but likely survivable.
For additional suggestions please view my profile, Network profile for contact information including my Skype ID. Look forward to hearing from you.
I tend to think that it was mysql v5 bug cause when I upgrade mysql to v8.0.12 the situation became good - memory regularly releases as expected even during the heavy request.

Finding right configuration for a dedicated mariadb server - for huge innodb tables

I have been working on a separate server to configure the mariadb but still queries are taking sooo long time to return and mostly not returning.
Mariadb version: 10.1.31-1
Server has 50GB Ram and 2200 Ghz * 10 core
Centos7 as OP
The target database currently has very huge innodb tables and not indexed well. there is no any other table type just innodb.
Currently i am making some nested select queries and saving results to another empty clone database with insert into or replace into.
as an example:
REPLACE INTO db1.table1
SELECT * FROM db0.table1 WHERE ART_ID IN (
SELECT ART_ID FROM db0.table2 WHERE BRA_ID IN (
SELECT BRA_ID FROM db0.table3 WHERE BRAND IN (
SELECT BRAND FROM db0.table4 WHERE ID IN (... bunch of comma seperated ids)
)
)
);
The only connection is from local and/or just one remote connection.
The only task of the server is to filter the huge data to another db, no web server etc.
I put the /etc/my.cnf file down below. I made lots of tests on this file so you may see some useless additions.
The question is: Is there any miss configurations in the cnf file; and is there any other configuration tips etc to run those bunch of queries in reasonable time.
It was working better but after i change the configurations or / and update the server everything got much slower. (yes i update the tables as well after updating db version)
# The MySQL server
[mysqld]
performance_schema=0
bind-address = 0.0.0.0
max_connections = 32
socket = /var/lib/mysql/mysql.sock
skip-external-locking
max_allowed_packet = 1024M
connect_timeout=300
wait_timeout=1200
interactive_timeout=300
key_buffer_size = 2G
bulk_insert_buffer_size=256M
sort_buffer_size = 32M
#read_buffer_size = 8M
#read_rnd_buffer_size = 32M
myisam_sort_buffer_size = 32M
join_buffer_size=32M
#max_heap_table_size= 8G
tmp_table_size=8G
concurrent_insert=2
#external-locking=FALSE
#open_files_limit=50000
default-storage-engine=innodb
innodb_file_per_table=1
#lower_case_table_names=1
# Try number of CPU's*2 for thread_concurrency
innodb_write_io_threads=10
innodb_read_io_threads=20
#innodb_buffer_pool_instances=8
#thread_stack = 1M
#thread_cache_size = -1
innodb_buffer_pool_size = 16G
#query_cache_limit = 1073741824
#query_cache_size = 1073741824
#query_cache_type = 1
query_cache_size=0
query_cache_type=0
#innodb_additional_mem_pool_size = 20M
# Set .._log_file_size to 25 % of buffer pool size
innodb_fast_shutdown=0
#innodb_log_file_size = 1G
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 0
innodb_lock_wait_timeout = 1073741823
innodb_thread_concurrency=0
innodb_commit_concurrency=0
innodb_flush_method=O_DSYNC
innodb_log_compressed_pages = 0
#innodb_flush_neighbors = 0
#innodb_adaptive_hash_index_partitions = 8
log-bin=mysql-bin
server-id = 1
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[isamchk]
key_buffer = 384M
sort_buffer = 384M
read_buffer = 256M
write_buffer = 256M
[myisamchk]
key_buffer = 384M
sort_buffer = 384M
read_buffer = 256M
write_buffer = 256M
[mysqlhotcopy]
interactive-timeout
I used percona wizard and merge the advices.
the current my.cnf is below.
I run the php application yesterday night (2018-03-05 22:41:30 UTC), but still mysql could not return any result.
the application is only adding limit 10000 and offset (i did not want to start from beginning, i am currently tracking the last offset and starting from that point.)
currently i am planning to delete and reinstall mariadb :(
(it will be emotional reaction)
# Generated by Percona Configuration Wizard (http://tools.percona.com/) version REL5-20120208
[mysql]
# CLIENT #
port = 9999
socket = /var/lib/mysql/mysql.sock
[mysqld]
# GENERAL #
port = 9999
user = mysql
default-storage-engine = InnoDB
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/mysql.pid
# MyISAM # - there is no myisam table
key-buffer-size = 32M
myisam-recover-options = FORCE,BACKUP
# SAFETY #
max-allowed-packet = 16M
max-connect-errors = 1000000
skip-name-resolve
# DATA STORAGE #
datadir = /var/lib/mysql/
# BINARY LOGGING #
log-bin = /var/lib/mysql/mysql-bin
expire-logs-days = 14
sync-binlog = 1
# CACHES AND LIMITS #
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 30
#thread-cache-size = 16
open-files-limit = 65535
table-definition-cache = 1024
table-open-cache = 2048
# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 5G
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 42G
innodb-buffer-pool-instances = 42
thread_cache_size=100
innodb_lru_scan_depth=100
innodb_purge_threads=4
innodb_read_io_threads=64
innodb_write_io_threads=64
innodb_thread_concurrency=0
max_seeks_for_key=32
max_write_lock_count=16
thread_concurrency=35
innodb_fast_shutdown=0
innodb_file_per_table=1
default-storage-engine=innodb
concurrent_insert=2
join_buffer_size=32M
tmp_table_size=8G
read_buffer_size = 8M
# LOGGING #
log-error = /var/lib/mysql/mysql-error.log
log-queries-not-using-indexes = 0
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log
The my.cnf data in the Question does not match SHOW GLOBAL VARIABLES; values.
Please use this link
https://mariadb.com/kb/en/library/configuring-mariadb-with-mycnf/
to help get accurate matched values for assistance.
Then, Suggestions to consider for your my.cnf [mysqld] section include:
innodb_buffer_pool_size=8G # from > 40G (95% of RAM will not work well)
innodb_lru_scan_depth=100 # from 1024 see refman every second digging too deep
log_warnings=2 # from 1 to include connection errors more detail
max_connect_errors=10 # from 1,000,000 no need to waste a million cycles
thread_cache_size=100 # from 16 CAP per V8 MySQ to be prepared for volume
have_symlink=NO # to protect you server, unless you NEED it
innodb_purge_threads=4 # from 1 to accommodate when needed
innodb_read_io_threads=64 # from 4 see dba.stackexhange.com Q 5666 9/12/11 Rolando
innodb_write_io_threads=64 # from 4 per Rolando to use multi-core
innodb_thrad_concurrency=0 # another part of multi-core enabling
max_seeks_for_key=32 # from huge number, not found in 32, will not be found
max_write_lock_count=16 # give RD opportunity after nn locks
thread_concurrency=35 # to limit concurrent processing load
#Erce, To be applied in the correct active my.cnf, please. Withdrawn my offer to redo your mysqld section.
I used percona wizard and merge the advices. the current my.cnf is below.
I run the php application yesterday night (2018-03-05 22:41:30 UTC), but still mysql could not return any result.
the application is only adding limit 10000 and offset (i did not want to start from beginning, i am currently tracking the last offset and starting from that point.)
[mysql]
# CLIENT #
port = 9999
socket = /var/lib/mysql/mysql.sock
[mysqld]
# GENERAL #
port = 9999
user = mysql
default-storage-engine = InnoDB
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/mysql.pid
# MyISAM # - there is no myisam table
key-buffer-size = 32M
myisam-recover-options = FORCE,BACKUP
# SAFETY #
max-allowed-packet = 16M
max-connect-errors = 1000000
skip-name-resolve
# DATA STORAGE #
datadir = /var/lib/mysql/
# BINARY LOGGING #
log-bin = /var/lib/mysql/mysql-bin
expire-logs-days = 14
sync-binlog = 1
# CACHES AND LIMITS #
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 30
#thread-cache-size = 16
open-files-limit = 65535
table-definition-cache = 1024
table-open-cache = 2048
# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 5G
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 42G
innodb-buffer-pool-instances = 42
thread_cache_size=100
innodb_lru_scan_depth=100
innodb_purge_threads=4
innodb_read_io_threads=64
innodb_write_io_threads=64
innodb_thread_concurrency=0
max_seeks_for_key=32
max_write_lock_count=16
thread_concurrency=35
innodb_fast_shutdown=0
innodb_file_per_table=1
default-storage-engine=innodb
concurrent_insert=2
join_buffer_size=32M
tmp_table_size=8G
read_buffer_size = 8M
# LOGGING #
log-error = /var/lib/mysql/mysql-error.log
log-queries-not-using-indexes = 0
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log
#erce
These THREE lines NEED TO BE REMOVED (just before LOGGING area)
join_buffer_size=32M
tmp_table_size=8G
read_buffer_size = 8M
LOGGING
tmp_table_size was accurately matched earlier with max_heap_table_size.
This line later takes charge and get you unbalanced which is BAD.
The join_buffer_size and read_buffer_size are killing your RAM footprint.
Use mysqlcalculator.com to understand why, please.
The performance problem is primarily in the formulation of the query, not necessarily in the tuning or indexes.
Query Formulation
In general, turn IN ( SELECT ... ) into JOIN.
Tackling the last step:
SELECT bra_id
FROM t3
JOIN t4 ON t3.brand = t4.brand
WHERE t4.id IN (...commalist...)
with indexes:
t4: INDEX(id, brand)
t3: INDEX(brand, bra_id)
This starts with t4, looking for the various ids, then mapping them to brands. Then it reaches into t3 by brand to get bra_id.
I'll let you practice by doing the next two steps to get through t2 to t1.
Tuning advice
Don't blindly increase values in my.cnf, it can get you into trouble.
innodb_buffer_pool_size = 35G -- the most important
42G is probably OK -- if you don't have too many other apps running on the same server. 42 instances may work. Some say 16 instances should be the max.
The rest of the tunables can probably be left alone.
max_heap_table_size= 8G
tmp_table_size=8G
Those are dangerously high. With them, 8GB could be allocated once (or more than once) per connection. But you have only 50GB of RAM. It would be better to set them to no more than 1% of RAM (500M or less; 32M should be just fine).
#lower_case_table_names=1
Leave commented out. Any attempt to override what is needed for the OS is likely to get you into deep weeds.
port = 9999
What was wrong with the default 3306?
Other
"application grabbed 10000 items in 6 hours. for the next block of items 8 hour passed" -- Please provide SHOW CREATE TABLE for the 4 tables being read, plus the table being written to. And tell us how big the tables are. I suspect you are thrashing over index updates in the target table. There is a simple solution.
Check to see if mysqld is swapping.
I guess i need to change select query style, i will try the offered query alternatives because it did not work, queries are still to slow (i assume)
But these are current server stats (49GB ram, 10 core) and mysqls virtual memory is about 47~GB which is fine, no any other app is running. I think its alright ?
tuning-primer.sh
https://pastebin.com/bcjS1PWf
mysqltuner.pl
https://pastebin.com/mXUTLPQ6
my.cnf
https://pastebin.com/9jMLzhZS
Suggestions for your my.cnf-ini [mysqld] section using data posted 3/15/18
thread_cahce_size=40 # from 20, 8 required for overhead, room to wiggle
innodb_io_capacity=400 # from 200 new iostat -x will confirm use
# read_buffer_size=8388608 # lead with # for default of 128K - extreme at 8M
read_rnd_buffer_size=128K # from 256K will affect handler_read_rnd_next
sync_binlog=32 # from 1 sec frequency, unless you fear recovery required
# join_buffer_size=4M # lead with # for default of 128K
key_buffer_size=1M # from 32M less than 1M used - primarily Innodb tables
innodb_adapative_max_sleep_delay=15 # from 150000 (15 secs - n0 other users)
innodb_buffer_pool_instances=8 # from 42 to minimize mgmt. overhead
innodb_buffer_pool_size=40G # from 44G allow OS a little breathing room
innodb_change_buffer_max_size=3 # from 25% - no need to set aside 10G when not changing anything significant
innodb_doublewrite=OFF # from ON you are not a PROD effort - crash recovery NA for run of this instance
innodb_print_all_deadlocks=ON # from OFF you always need in error log
innodb_read_ahead_threshold=8 # from 56, why wait to read NXT extent?
innodb_stats_sample_pages=32 # from 8 for more accurate index cardinality
log_warnings=2 # from 1 to record addl connection error details
max_join_size=1000000000 # from more huge number to 1B rows max for join
sql_select_limit=1000000000 # from more huge number to 1B rows selected
query_alloc_block_size=32K # from 16K - minimize RAM alloc AvgQrySz 20K
query_cache_type=0 # for OFF, not used
query_cache_size=0 # from 8M not needed for one time selections
query_cache_limit=1K # from 256K, just to keep the framework, in case QC needed
query_cache_min_res_unit=512 # from 2K for optimal QC storage when QC used
query_prealloc_size=32K # from 24K to minimize RAM alloc for parsing
transaction_alloc_block_size=32K # from 8K to minimize RAM alloc
transaction_prealloc_size=32K # from 4K to minimize RAM alloc
updatable_views_with_limit=NO # from YES to reduce handler_external_lock count
this one could have to go back to YES - it is DYNAMIC variable
Please find your ONE com_stmt_prepare CODE and ensure it has CLOSE before leaving the session to release resources per refman PREPARE or mysql_stmt_prepare for API.
Once you are running, before running your troubled query, consider using
ALTER TABLE tbl_name ENGINE=INNODB
for each table to take advantage of additional pages for better cardinality,
then try your long running query.

MySQL CPU Maxing

Currently have a MySQL database and experiencing an issue with MySQL running at 600% of the CPU usage.
Specs:
2.3 GHz Intel Xeon® E5-2686 v4 (Broadwell) processors or 2.4 GHz Intel Xeon® E5-2676 v3 (Haswell) processors
8 vCPU's
32GB of RAM
100GB Hard Drive.
Instance is currently hosted with AWS, running Ubuntu 14.04.4 LTS and MySQL version 5.6.33-0ubuntu0.14.04.1-log.
Please see below my.cnf configuration:
# The MySQL database server configuration file.
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
log_error=/var/log/mysql/mysql_error.log
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
innodb_io_capacity = 2000
innodb_read_io_threads = 64
innodb_thread_concurrency = 0
innodb_write_io_threads = 64
innodb_buffer_pool_size = 2G
innodb_log_file_size = 1G
# Skip reverse DNS lookup of clients
skip-name-resolve
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 11G
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 16M
query_cache_type = 0
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
general_log_file = /var/log/mysql/mysql.log
general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/mysql_error.log
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 512M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
First, start by disabling the general_log, no need to have it enabled all the time. The general log is very useful for troubleshooting and/or testing purposes, but it is a performance query killer on production database.
Also, if your are using mixed engines (MyISAM and InnoDB), you can give the innodb_buffer_pool_size more memory. But if you are only using Innodb, you can set this variable to 24G for example, and the key_buffer_size a few hundred of megabytes.
If you are using MyIsam only, please deactivate Innodb engine completely (skip-innodb) and increase Myisam memroy related parameters like key_buffer_size.
Hope it helps.

Mysql terrible performance

I recently got new hardware but I'm having problem with mysql. It's slower than it was on the old server. When I ran sysbench tool on new server I got this:
sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password= --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 8
Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 7 times)
Done.
OLTP test statistics:
queries performed:
read: 365694
write: 0
other: 52242
total: 417936
transactions: 26121 (435.14 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 365694 (6091.93 per sec.)
other operations: 52242 (870.28 per sec.)
Test execution summary:
total time: 60.0293s
total number of events: 26121
total time taken by event execution: 479.9747
per-request statistics:
min: 0.52ms
avg: 18.38ms
max: 117.91ms
approx. 95 percentile: 53.21ms
Threads fairness:
events (avg/stddev): 3265.1250/19.37
execution time (avg/stddev): 59.9968/0.01
Old server produced this:
sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password= --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 8
Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 7 times)
Done.
OLTP test statistics:
queries performed:
read: 952294
write: 0
other: 136042
total: 1088336
transactions: 68021 (1133.58 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 952294 (15870.18 per sec.)
other operations: 136042 (2267.17 per sec.)
Test execution summary:
total time: 60.0052s
total number of events: 68021
total time taken by event execution: 479.5141
per-request statistics:
min: 3.27ms
avg: 7.05ms
max: 31.63ms
approx. 95 percentile: 8.60ms
Threads fairness:
events (avg/stddev): 8502.6250/11.10
execution time (avg/stddev): 59.9393/0.00
Old server has E3-1230 cpu and 16GB ram. New server has E5-1660 v3 and 32GB ram. OS is debian wheezy on both, and mysql version is 5.5.43. My.cnf below
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 256M
key_buffer_size = 256M
sort_buffer_size = 2M
read_rnd_buffer_size = 4M
innodb_buffer_pool_size = 1G
max_allowed_packet = 32M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
max_connections = 300
table_cache = 256M
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 200M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#slow_query_log_file = /var/log/mysql/mysql-slow.log
#slow_query_log = 1
#long_query_time = 2
#log_queries_not_using_indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
Any ideas?
query_cache_size = 200M
Turn off the Query cache. Even if you choose to keep it on, don't set the size bigger than 50M because of the cost of pruning.
table_cache = 256M
OUCH! You do not need a quarter billion tables!! Change to a few hundred (no suffix).
Do this on both machines, then diff the outputs:
SHOW VARIABLES LIKE 'innodb%';
Some defaults have changed over time; this should point them out.
Agree with Rick that your config is probably not optimal, but if you really are running the same mysql config on both machines, then that does not explain the discrepancy. All the potential reasons for this would take far too long to list. Assuming that you gathered the stats in the same way on both systems (cold caches) then the most likely cause is IO related, you could verify this using a disk benchmarking tool - I would recommend afio if you don't already have something you know well which is appropriate for the task. Fixing the problem is an even more complex question.