We have a dedicated database server (MariaDB 5.5.31) hosting our database used by our java application (accessing the db via hibernate from another server). While our database itself is using about 12gb of RAM, the operating system (CentOS 6.4) is caching around 25gb additionally.
Can you give me any advice, how we can influence this behaviour?
While most of our 100 tables have less than 1000 entries, we have some tables with above 10 million. These tables experience lots of reads and writes.
These are our db settings:
[server]
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
character-set-server = utf8
collation-server=utf8_general_ci
skip-external-locking
back_log = 50
max_connections = 100
max_connect_errors = 10
table_open_cache = 2048
max_allowed_packet = 16M
max_heap_table_size = 64M
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 16M
thread_cache_size = 8
thread_concurrency = 8
query_cache_size = 128M
query_cache_limit = 2M
thread_stack = 240K
transaction_isolation = REPEATABLE-READ
tmp_table_size = 64M
binlog_cache_size = 1M
log-bin=mysql-bin
binlog_format=mixed
expire_logs_days = 2
#*** MyISAM Specific options
key_buffer_size = 384M
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
myisam_recover
# *** INNODB Specific options ***
innodb_additional_mem_pool_size = 16M
innodb_buffer_pool_size = 10G
innodb_data_home_dir = /var/lib/mysql
innodb_write_io_threads = 8
innodb_read_io_threads = 8
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 8M
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 120
[embedded]
[mysqld-5.5]
[mariadb]
[mariadb-5.5]
If the tables with millions of records are transactional tables then following are the foremost things to be done:
List the child tables for those tables.
Discuss with client and have a backup plan, i.e., only 1/1.5/2 years of data to be kept in the table.
If there is lots of child table's data that can be backed up along with then have 2 options:
a) Have one denormalized table for the backup data
b) Have one table for each can be named original_table_name_bk
There should be a backup process planned periodically say every 3 months, when preferrably during lean period the process will run and move data from transactional table to backup table.
Now you have both historical and current data in your database without impacting performance of transactional operations.
Related
This question was migrated from Stack Overflow because it can be answered on Database Administrators Stack Exchange.
Migrated 18 days ago.
I use a VPS with 56 CPU cores and 64 GB of memory, a with MySQL database, but the CPU usage is always high for my apps - user 10.000.
This is in my mysqld.cnf. What's wrong with my settings?
innodb_buffer_pool_size=50G
innodb_change_buffering=all
innodb_log_file_size = 3125M
innodb_log_buffer_size = 3125M
innodb_file_per_table = ON
innodb_log_files_in_group =4
innodb_flush_method = O_DSYNC
innodb_lock_wait_timeout = 50
innodb_buffer_pool_instances = 50
innodb_flush_log_at_trx_commit = 0
innodb_thread_concurrency=112
innodb_stats_on_metadata = OFF
innodb_thread_sleep_delay=1000
innodb_purge_threads=8
innodb_read_io_threads = 32
innodb_write_io_threads = 32
innodb_io_capacity = 5000
innodb_io_capacity_max=15000
key_buffer_size = 4G
max_allowed_packet = 1G
thread_stack = 5M
sort_buffer_size = 50M
read_buffer_size = 50M
read_rnd_buffer_size = 20M
myisam_sort_buffer_size = 20M
join_buffer_size = 1G
myisam-recover-options = BACKUP
max_connections = 1000
max_user_connections = 500
thread_cache_size = 1000
query_cache_limit = 0
query_cache_size = 0
long_query_time = 10
expire_logs_days = 5
max_binlog_size = 200M
innodb_log_buffer_size - I would keep that under 1% of RAM. (innodb_log_file_size looks OK.)
innodb_log_files_in_group there is some evidence that more that "2" degrades performance.
innodb_flush_method -- this depends on MySQL version and disk filesystem type. (Your choice is rarely picked by other DBAs.)
Did you also set innodb_adaptive_max_sleep_delay? I see that MariaDB abandoned innodb_thread_sleep_delay in 10.5 -- either it was obviated by some improvement, or it was deemed not useful. I don't know MySQL's stand on the two.
innodb_io_capacity_max=15000 -- Do you have a super-duper SSD?
key_buffer_size = 4G: Unless you are using MyISAM (you should not be), set this to only 50M.
thread_stack -- Leave at the default value
long_query_time = 1 to make better use of the slowlog. After a day, analyze it with pt-query-digest. SlowLog
But the real way to deal with high CPU (or Load Average) is to find the 'worst' queries (according to the slowlog) and work on improving them. (Random tuning of settings can only get you into trouble.)
For further analysis of the settings, please provide SHOW GLOBAL STATUS; and SHOW VARIABLES; after running at least a day. ( http://mysql.rjweb.org/doc.php/mysql_analysis#tuning )
I'm using MySQL 5.7 almost always and I never had performances issues.
But the project I'm working on lately is realy making me doubt about MySQL.
This project in the last year generated a pretty huge DB with tables sometime with 15mln records.
I'm thinking about partitioning but I need to get rid of foreign keys so this is all another problem that I'll probably discuss here in the next future with you guys.
So techinically my staff and I are trying to optmize all the tables, indexes and queries in order to perform.
What I've been taught since I was a student is "let do the database what the database is capable of doing" and that's always been my creed.
First of I noticed that MySQL does not perform with datetime columns in the right way. The first optimization was to convert all the datetime columns into integer values (converting datetime into int timestamp values) and with this little modification and by indexing those columns I decreased significantly the amount of time to get the same result.
Moreover I noticed that with some queries using the "GROUP BY" inside a query or doing a group using the programming language in charge to elaborate results the database is significantly slower.
So first datetime columns (maybe treated like strings don't allows MySQL core to perform) and in second place the "GROUP BY" which I think is one of the most used command.
I'm starting thinking that probably what's wrong here is MySQL tuning so here I share my configuration file (my.cnf):
symbolic-links=0
query_cache_size = 32M
thread_cache_size = 8
myisam_sort_buffer_size = 64M
read_rnd_buffer_size = 8M
read_buffer_size = 2M
sort_buffer_size = 2M
table_open_cache = 512
max_allowed_packet = 1M
key_buffer_size = 384M
sql-mode = "STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
max_allowed_packet = 32M
max_connections = 2000
open_files_limit = 10000
tmp_table_size = 64M
max_heap_table_size = 64M
tmpdir = /home/mysql/tmp
default_storage_engine = InnoDB
skip_name_resolve
query_cache_type=0
query_cache_size=0
log_bin
server_id = 1
max_binlog_size = 100M
expire_logs_days = 7
sync_binlog = 0
binlog_format = MIXED
innodb_buffer_pool_size = 26G
innodb_log_file_size = 3GB
innodb_log_buffer_size = 16M
innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
innodb_buffer_pool_instances = 8
innodb_thread_concurrency = 8
innodb_stats_on_metadata = 0
#innodb_buffer_pool_dump_at_shutdown = 1
#innodb_buffer_pool_load_at_startup = 1
innodb_buffer_pool_dump_pct = 75
innodb_adaptive_hash_index = ON
innodb_adaptive_hash_index_parts = 16
innodb_read_io_threads = 16
innodb_write_io_threads = 16
innodb_file_per_table
innodb_flush_neighbors = 0
innodb_page_cleaners = 8
long_query_time = 4
slow_query_log = 1
So as you can see I talked about simple query operations such "GROUP BY" and DATETIME columns, that's why I probabily think the problem is related to MySQL tuning.
I'm also thinking about migrating to MySQL 8.0 but I need to try this database on a test machine before going on a production environment.
I don't know where I can hit my head to get a solution.
Intro:
I have a Debian 8 VPS with SSD and 512MB RAM (1024MB Burst) and use it only with MySQL. I have turned off all unnecessary services and gave every resources on db and system. I have 5 workstations only. The ping and network are stable.
The main problem is that if I do 10 consecutive queries like this:
(SELECT IFNULL(SUM(Qtty), 0) FROM operations
WHERE blablabla
FROM (goods LEFT JOIN store ON goods.ID = store.GdID)
WHERE Deleted <> -1
GROUP BY goods.ID;
Randomly 5-6 of them are executed for 2s, but the rest for 8s or more! I see no reason for this behavior. When i activate slow_query_log, it have just the same query, nothing else.
My InnoDB data is 98MB is there any reason to give more RAM to innodb_buffer_pool_size or 128 is enough?
innodb_log_file_size must be 25% - 50%, which size would be better for me?
tmp dir is tmpfs, so if tmp_table_size is placed there, is there any reason to give it more RAM or is enough to use 1M for example?
table_open_cache - I have about 200 tables, is there any reason to make it 20000 for example, as the mysqltuner advises me?
Can i tuning something else?
my.cnf
skip-external-locking
skip-name-resolve
performance_schema = OFF
thread_stack = 192K
thread_cache_size = 10 #this is managed globally and is equal to max_connections
max_connections = 10
max_connect_errors = 10
connect_timeout = 20
wait_timeout = 20
interactive_timeout = 20
sql_mode = TRADITIONAL
default_storage_engine = InnoDB
innodb_buffer_pool_instances = 1
innodb_buffer_pool_size = 250M
innodb_log_file_size = 100M
innodb_strict_mode = ON
innodb_flush_method = O_DIRECT
innodb_flush_neighbors = 0
innodb_read_io_threads = 4
innodb_write_io_threads = 4
innodb_io_capacity = 2000
innodb_io_capacity_max = 3000
key_buffer = 128K
delay_key_write = ON
tmp_table_size = 10M # 32M = 53% temp
max_heap_table_size = 10M
query_cache_type = 0
query_cache_size = 0
table_open_cache = 5000
table_definition_cache = 3000
Suggestions to consider for your my.cnf [mysqld] section
max_connections=20 # from 10 to affect thread_cache_size
. because 8 threads used by MySQL + 5 concurrent = 13 and you will have a few extra, please
tmp_table_size=16M # from 10M to reduce 41% temp tables created on disk
max_heap_table_size=16M # from 10M should always be = tmp_table_size
innodb_read_io_threads=64 # from 4 to let the ponies run
innodb_write_io_threads=64 # from 4 to push on through
for additional assistance, pls see my Profile, clk on Network Profile for contact info.
Again thanks for the detailed answer and sorry for the delay. Unfortunately, I travel often without internet and contact by Skype is almost impossible.
I noticed something that can help: often, when running even very small queries, the server responds with "ERROR 2006 (HY000): MySQL server has gone away". If i repeat the operation - everything is fine. The problem is on the server, not in db?
The db is about 100MB, so i increase and max_allowed_packet to 512MB, innodb io_threads to 64. I tried with increased net_read_timeout, net_write_timeout, wait_timeout and interactive_timeout to 28800 and there is an improvement.
This is my timeouts now, does it make sense to change something?
connect_timeout 120
delayed_insert_timeout 300
innodb_flush_log_at_timeout 1
innodb_lock_wait_timeout 50
innodb_rollback_on_timeout OFF
interactive_timeout 28800
lock_wait_timeout 31536000
net_read_timeout 30
net_write_timeout 60
slave_net_timeout 3600
thread_pool_idle_timeout 60
wait_timeout 28800
In the last few days we are experiencing some strange performance issues with MySQL (old version 5.1.69).
The slow log shows stuff like:
# User#Host: jboss[jboss] # localhost [127.0.0.1]
# Query_time: 46.595796 Lock_time: 0.000022 Rows_sent: 0 Rows_examined: 0
SET timestamp=1496127084;
insert into ACTIVE_RENT (LINE_ID, CHARGED_UP_TO, EVENT_ID, NUMBER, RENT_START, TYPE) values (149914, '2017-05-02 00:00:00', 625751, 'ABCD1234567', '2013-08-02 00:00:00', 'DETENTION');
# User#Host: jboss[jboss] # localhost [127.0.0.1]
# Query_time: 19.401896 Lock_time: 0.000041 Rows_sent: 0 Rows_examined: 0
SET timestamp=1496127084;
delete from ACTIVE_RENT where ACTIVE_RENT_ID=463965;
There is no cascade specified on delete or update and it only has one foreign key to another table and there are no triggers for this table. The table really is quite basic. And there are only about 14k rows.
Normally these insert or deletes are very quick but the last few days they can be very very slow in peak times.
We have increased the innodb_buffer_size to 20G but that didn't change much (with respect to this problem, other stuff is faster). The DB size is about 40GB at the moment.
Below is the my.conf. Any idea where the bottle neck could come from and what to do about it? We are planning on upgrading MySQL to 5.5 but that is not going to happen for a few weeks.
[client]
default-character-set=utf8
socket=/home/mysql-data/mysql/mysql.sock
[mysql]
default-character-set=utf8
max_allowed_packet=32M
[mysqld]
# these are the install settings
local-infile=0
open_files_limit=8192
query_cache_size = 256M
query_cache_type = 1
query_cache_limit = 6M
query_prealloc_size = 16384
query_alloc_block_size = 2048
tmp_table_size = 128M
max_heap_table_size = 256M
table_cache = 1024
thread_cache_size = 512
key_buffer = 256M
join_buffer_size = 8M
read_buffer_size = 8M
sort_buffer_size = 8M
innodb_lock_wait_timeout = 120
binlog_format=row
max_allowed_packet=32M
log-bin=mysql-bin
server-id=1
#Allow group_concat to return longer data types
group_concat_max_len=16384
default-character-set = utf8
skip-character-set-client-handshake
collation-server = utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server = utf8
innodb_buffer_pool_size = 20GB
query_cache_size = 256M -- too big. Whenever a write occurs, all entries in the QC for that table must be purged. The time for such is roughly proportional to the size. Lower to 50M.
Are your tables InnoDB? If the server has 64GB and is used mostly for MySQL, then set the buffer_pool to 70% of that.
Is ACTIVE_RENT_ID the PRIMARY KEY?
A one-row INSERT into InnoDB will not take 46s unless a long-running ALTER is going on. Look around; something else is involved. Or maybe a long-running transaction that eventually timed out at 50 seconds? Are you using autocommit=0? (I hope not.) Are you using BEGIN...COMMIT?
How big is the table ACTIVE_RENT?
This is really maddening. I have followed every instruction for settings that I have found on the interwebs, and I can't get past this.
Basically, I have a table with about 8 million rows. I need to create a backup of this table like so:
create table mytable_backup like mytable
And that takes several hours on my production server, which is an Amazon EC2 instance running through EngineYard. It takes only minutes on my MacBook Pro. This is another one of those annoying things that MySQL does in the background, and you can't guess how it is making the decision to do something so stupidly slow.
BTW, there is over 330G available in the tmp directory, so that is not the issue.
But here is what "free -m" yields:
deploy#domU-12-31-39-02-35-31 ~ $ free -m
total used free shared buffers cached
Mem: 1740 1728 11 0 14 1354
-/+ buffers/cache: 359 1380
Swap: 895 2 893
I don't know how to read that, but the "11" under the free column doesn't look very good.
I am running:
Server version: 5.0.51-log Gentoo Linux mysql-community-5.0.51
Here is my configuration file:
# /etc/mysql/my.cnf: The global mysql configuration file.
# $Header: /var/cvsroot/gentoo-x86/dev-db/mysql/files/my.cnf-4.1,v 1.3 2006/05/05 19:51:40 chtekk Exp $
# The following options will be passed to all MySQL clients
[client]
port = 3306
[mysql]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqladmin]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqlcheck]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqldump]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqlimport]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[mysqlshow]
character-sets-dir=/usr/share/mysql/charsets
default-character-set=utf8
[myisamchk]
character-sets-dir=/usr/share/mysql/charsets
[myisampack]
character-sets-dir=/usr/share/mysql/charsets
[mysqld_safe]
err-log = /db/mysql/log/mysql.err
# To allow table cache to be raised
open-file-limit = 4096
[mysqld]
max_connections = 300
innodb_file_per_table = 1
log-slow-queries = /db/mysql/log/slow_query.log
long_query_time = 2000000
ft_min_word_len = 3
max_heap_table_size = 64M
tmp_table_size = 64M
server-id = 1
log-bin = /db/mysql/master-bin
log-bin-index = /db/mysql/master-bin.index
# END master/slave configuration
character-set-server = utf8
default-character-set = utf8
user = mysql
port = 3306
socket = /var/run/mysqld/mysqld.sock
pid-file = /var/run/mysqld/mysqld.pid
log-error = /db/mysql/log/mysqld.err
basedir = /usr
datadir = /db/mysql
key_buffer = 32M
max_allowed_packet = 32M
table_cache = 1024
thread_cache = 512
sort_buffer_size = 100M
net_buffer_length = 64K
read_buffer_size = 1M
read_rnd_buffer_size = 1M
myisam_sort_buffer_size = 100M
myisam_max_sort_file_size = 2G
myisam_repair_threads = 1
language = /usr/share/mysql/english
# security:
# using "localhost" in connects uses sockets by default
# skip-networking
# bind-address = 127.0.0.1
# point the following paths to different dedicated disks
tmpdir = /mnt/mysql/tmp
# log-update = /path-to-dedicated-directory/hostname
# you need the debug USE flag enabled to use the following directives,
# if needed, uncomment them, start the server and issue
# #tail -f /tmp/mysqld.sql /tmp/mysqld.trace
# this will show you *exactly* what's happening in your server ;)
#log = /tmp/mysqld.sql
#gdb
#debug = d:t:i:o,/tmp/mysqld.trace
#one-thread
# the rest of the innodb config follows:
# don't eat too much memory, we're trying to be safe on 64Mb boxes
# you might want to bump this up a bit on boxes with more RAM
innodb_buffer_pool_size = 1275M
# this is the default, increase it if you have lots of tables
innodb_additional_mem_pool_size = 16M
#
# i'd like to use /var/lib/mysql/innodb, but that is seen as a database :-(
# and upstream wants things to be under /var/lib/mysql/, so that's the route
# we have to take for the moment
#innodb_data_home_dir = /var/lib/mysql/
#innodb_log_arch_dir = /var/lib/mysql/
#innodb_log_group_home_dir = /var/lib/mysql/
# you may wish to change this size to be more suitable for your system
# the max is there to avoid run-away growth on your machine
innodb_data_file_path = ibdata1:20M:autoextend
# we keep this at around 25% of of innodb_buffer_pool_size
# sensible values range from 1MB to (1/innodb_log_files_in_group*innodb_buffer_pool_size)
innodb_log_file_size = 96M
# this is the default, increase it if you have very large transactions going on
innodb_log_buffer_size = 8M
# this is the default and won't hurt you
# you shouldn't need to tweak it
innodb_log_files_in_group = 2
# see the innodb config docs, the other options are not always safe
# This is not good for performance when used with bin_sync. Disabling.
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_lock_wait_timeout = 50
query_cache_size = 16M
query_cache_type = 1
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
# uncomment the next directive if you are not familiar with SQL
#safe-updates
[isamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[myisamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
ft_min_word_len = 3
[mysqlhotcopy]
interactive-timeout
For what it's worth, 11 megs free is perfectly fine. That's 11 megs of memory not being used for anything, and "wasted" as far as the hardware is concerned. The real number is the "1380" used in caches, PLUS the 11 megs unused. Caches can be blown away as necessary.
Your system has nearly 1400 MB of RAM available.
You could try
create table backup_table as (select * from production table) engine=myisam
This should create the table with only the data and none of the keys. You can then add the keys on by doing
alter table backup_table add index(column_name)
I've done this successfully several times, and it is usually a factor of 2 times faster than inserting with the keys in place.
You have to look your setting for myisam_max_sort_file_size and myisam_sort_buffer_size
If the sum of all the keys is less that myisam_max_sort_file_size, a sort will, in the worst case, land in a MyISAM table, which is a good thing.
Otherwise, it will revert to the keycache. That means loading the necessary .MYI index pages into the keycache and traversing those index pages in memory. Nobody wants that !!!!
Your current setting for this variable says 2G.
Look at the keys being built. Add them up. If the sum of all the key sizes exceed 2G, keycache all the way !!! You will have to up this value. You could up this value for the session to 4G with
SET myisam_max_sort_file_size = 1024 * 1024 * 1024 * 4;
SET myisam_sort_buffer_size = 1024 * 1024 * 1024 * 4;
or you could plant the number directly like this:
SET myisam_max_sort_file_size = 4294967296;
SET myisam_sort_buffer_size = 4294967296;
before doing the ENABLE KEYS;
If you are just interested in backing up the data, why index it to begin with ??? Try using the ARCHIVE storage engine. It has no indexing whatsoever. Do the following:
CREATE TABLE mytable_backup LIKE mytable;
ALTER TABLE mytable_backup ENGINE=ARCHIVE;
INSERT INTO mytable_backup SELECT * FROM mytable;
I also noticed you are using Amazon EC2. I have never been in EC2 before. Run this command:
SHOW ENGINES;
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
| Engine | Support | Comment | Transactions | XA | Savepoints |
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
| InnoDB | DEFAULT | Supports transactions, row-level locking, and foreign keys | YES | YES | YES |
| MRG_MYISAM | YES | Collection of identical MyISAM tables | NO | NO | NO |
| BLACKHOLE | YES | /dev/null storage engine (anything you write to it disappears) | NO | NO | NO |
| CSV | YES | CSV storage engine | NO | NO | NO |
| MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO |
| FEDERATED | YES | Federated MySQL storage engine | NO | NO | NO |
| ARCHIVE | YES | Archive storage engine | NO | NO | NO |
| MyISAM | YES | Default engine as of MySQL 3.23 with great performance | NO | NO | NO |
+------------+---------+----------------------------------------------------------------+--------------+------+------------+
If the ARCHIVE storage engine appears in the list and Support is Yes, you have the option to backup to an ARCHIVE table. If not, you must get the myisam_max_sort_file_size and myisam_sort_buffer_size adjusted.