my.cnf optimization according to the configuration - mysql

I am trying many problems with MySQL, with high memory usage and especially with high CPU usage.
I have a dedicated server with the following configuration:
8 CPU Intel(R) Xeon(R) CPU E3-1240 v6 # 3.70GHz
16GB DDR3
SO Linux with cPanel/WHM + MySQL
Following is my.cnf file:
[mysqld]
max_connections=1000
wait_timeout=1000
interactive_timeout=1000
long_query_time=50
slow_query_log = 0
#slow_query_log_file = /var/log/mysql/slow-query.log
default-storage-engine=MyISAM
log-error=/var/lib/mysql/dc.leilaoweb.com.err
max_allowed_packet=268435456
local-infile=0
event_scheduler = on
tmp_table_size=300M
max_heap_table_size=128M
open_files_limit=65000
performance-schema=1
innodb_file_per_table=1
innodb_log_file_size=512M
innodb_buffer_pool_size=8G
key_buffer_size=512M
innodb_buffer_pool_instances=8
# config cache
query_cache_limit=8M
query_cache_size=256M
query_cache_type=1
table_open_cache=6000
table_cache=5000
thread_cache_size=96
#bind-address=127.0.0.1
#skip-networking
#performance_schema=ON
skip-name-resolve
How could I improve this setting to make queries faster and not raise server load?

It's a funny question that asks for help with query optimization, in which no specific query is mentioned.
Here are some tips on configuration options:
default-storage-engine=MyISAM
Change the default storage engine to InnoDB, and make sure all your existing tables are InnoDB. Don't use MyISAM.
query_cache_size=256M
query_cache_type=1
Set the query cache size and type to 0. The query cache is useful in such rare conditions that it has become deprecated, and removed in MySQL 8.0. It's better to cache query results in your application code, on a case-by-case basis.
innodb_buffer_pool_size=8G
If you have a lot more data than 8G, consider increasing the size of the buffer pool. The more of your data and indexes that resides in RAM, the better it will be for performance. But there's no further benefit to adding RAM once your data and indexes are 100% cached in the buffer pool.
And of course do not overallocate the buffer pool such that it causes the server to start swapping. That will kill performance (or else Linux's OOM killer will terminate mysqld if you have no swap).
key_buffer_size=512M
No need for extra memory allocated to the key buffer if you don't use MyISAM.
There may be other tuning parameters that can give benefit, but since you have said nothing about your queries or server activity, there's no way to guess what those would be.
You're better off focusing on indexes and query design.
In general, optimization naturally improves some queries at the expense of other queries. So you can make an optimization strategy only after you know which queries you need to optimize for.

Related

Want to set innodb configuration variables setting properly ? want to know what variable size will best for mysql 5.5

I used Mysql 5.5.57 and used innodb
Currency I face performance related issues due to which our system gets crashed.
and got many errors like lockwait time out and deadlock problems on many tables.
I trying to solve that.
but i think if i change my innodb configuration default setting then also system performance gets increased but i dont know what type of variable needs to change so please help me
below is our default innodb configuration setting:
The below is the settings in my.cnf:
innodb_file_format=barracuda
innodb_file_format_max=barracuda
innodb_file_per_table=1
query-cache-size = 64M
thread_cache_size = 8
default-time-zone = '+05:30'
query_cache_limit = 10M
character_set_server=utf8mb4
collation_server=utf8mb4_general_ci
innodb_thread_concurrency=8
key_buffer_size=183500800
group_concat_max_len=50000
innodb_log_file_size=178257920
#innodb_lock_wait_timeout=150
innodb_buffer_pool_size=134 MB
innodb_thread_concurrency=8
innodb_thread_sleep_delay=10000
innodb_concurrency_tickets=500
innodb lock_wait_time_out=150
We have 132GB Ram on the server with 2 processors and each has 6 cores.
16GB should be more than enough for MySql.
please help to set proper variables size for each parameter i provided here
I suggest you read this atrticle https://www.percona.com/blog/2013/09/20/innodb-performance-optimization-basics-updated/
You will find example for each variable there, i think this is what you need.
I've got a significant performance boost by setting
innodb_flush_log_at_trx_commit=0
innodb_buffer_pool_size=2G
innodb_log_file_size=256M
innodb_log_buffer_size=8M
innodb_buffer_pool_instances=1
Specifically, innodb_flush_log_at_trx_commit=0 speeds up loading a dump file from minutes to seconds! This setting is not recommended for "mission-critical" environments but is pretty fine for a development machine.

MySQL: Exclusive lock

I have a mysql error and I didn't find the answer till now on the internet: "Wait on a lock was aborted due to a pending exclusive lock"
I have a lot of transaction in the same time. These based on an innodb table, that has a few triggers. These triggers read and write other innodb and myisam tables.
I know what is the "wait lock", but I don't have enough information about "exclusive lock". What should I do to terminate this error, because these transactions are not finishing and not updateing the rows in the this table.
I have a Windows Server 2012 R2 Standard and a MySQL 5.6.15 on it. The database is on an SSD drive and the logs are on a simple HDD.
This is my ini file:
[mysqld]
port=3306
basedir="C:/MySQL/mysql-5.6.15-winx64/"
datadir="C:/DB/myCustomer/data"
innodb_data_home_dir="C:/DB/myCustomer/innodb/"
log-bin = "E:/Log/mysqldblog/myCustomer/mysql-bin/log"
log-error = "E:/Log/mysqldblog/myCustomer/errorlog/error"
tmpdir ="E:/MySQLTEMP/"
innodb_buffer_pool_size=1024M
join_buffer_size = 4M
sort_buffer_size = 2M
read_rnd_buffer_size = 2M
innodb_log_buffer_size = 4M
innodb_flush_log_at_trx_commit=1
max_allowed_packet = 100M
tmp_table_size=64M
max_heap_table_size=64M
innodb_log_file_size = 256M
innodb_thread_concurrency = 16
federated=1
default-storage-engine=INNODB
event-scheduler = ON
max_sp_recursion_depth = 50
character-set-server=utf8
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
binlog_format=mixed
transaction-isolation = READ-COMMITTED
log-warnings = 0
I would apprisiate if somebody could help me to solve this problem.
Thanks in advence.
kind regards,
András
Exclusive locks are described in MySQL glossary:
A kind of lock that prevents any other transaction from locking the same row. Depending on the transaction isolation level, this kind of lock might block other transactions from writing to the same row, or might also block other transactions from reading the same row. The default InnoDB isolation level, REPEATABLE READ, enables higher concurrency by allowing transactions to read rows that have exclusive locks, a technique known as consistent read.
We cannot tell you how to avoid specifically this instance of the error, we can only provide generic guidance on how to debug it and what MySQL settings may help you to reduce the risk of receiving such errors. The specific solution may require restructuring your data, or changing the application logic, or even upscaling your hw - you have to make the call.
To debug the error, use SHOW ENGINE INNODB STATUS statement (from MySQL v5.7.4 use innodb information_schema tables instead, but the question is applicable to v5.6).
Under the LATEST DETECTED DEADLOCK section you will find information on what transactions and statements caused the deadlock.
Out of the dozens of MySQL settings, innodb_lock_wait_timeout has the most direct impact on these kind of errors, since you can give more time to MySQL to wait until the situation is resolved. However, you need to be careful because increasing this timeout setting may lead to slowing down of your application and may simply mask the issues your application is facing.
Sok szerencsét!

Mysql "memory usage" increasing and increasing

I have a really big website built old fashioned with PHP & MYSQL.
I have more than 1,000 different queries in my website, on different PHP pages, and it's really hard to update all of them to MYSQLI.
I bought VPS server with 4GB RAM and in the past months I experience really slow page loads.
When I restart my server, everything runs smoothly, but after couple of hours/days the website is getting muchu slower with loading time of 3+ seconds for a page load. I notice that the mysqld service is increasing and increasing in memory usage, from 80MB on server restart it reached about 400MB and more of usage.
I put in the end of my index.php mysql_close() but it seem like the connection number still increasing.
Questions
What can cause unlimited increment in mysql memory usage?
Updating all my queries to MYSQLI may improve the performance?
Some information:
innodb_version
5.5.31
protocol_version
10
slave_type_conversions
version
5.5.31-log
version_comment
MySQL Community Server (GPL)
version_compile_machine
x86_64
version_compile_os
Linux
storage engine: Mixed (Somes tables are INNODB,some tables are MyISAM.
my.cnf:
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
max-connections=100000
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
thread_cache_size=5
table_open_cache=99390
sort_buffer_size=512M
read_rnd_buffer_size=512M
query_cache_size=512M
query_cache_limit = 16M
query_cache_type = 1
slow_query_log=1
slow_query_log_file=slow_query_log.log #
long_query_time=5
log-queries-not-using-indexes=1
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
I have about ~6-7 queries running when I use show processlist
max-connections=100000 -- Yikes! Drop to 1000
table_open_cache=99390 -- drop to, say, 2000
sort_buffer_size=512M -- drop to 1% of RAM, say, 40M
read_rnd_buffer_size=512M -- ditto
query_cache_size=512M -- too big; slows things down; drop to 40M
long_query_time=5 -- not low enough to catch much; drop to 2
log-queries-not-using-indexes=1 -- clutters the slowlog without providing much info; change to 0
You did not say which Engine you are using. Read this for advice on MyISAM and InnoDB.
1000 pages -- that's not too many.
Which web server? If Apache, don't set MaxClients to more than 20.
2022 postscript: The query_cache_size and query_cache_type variables have been removed from mySQL 8.0.3+.

Periodic MySQL lockup when Wordpress is under heavy load

I have a MySQL 5.1.61 database running behind two load balanced Apache webservers hosting a fairly busy (100K uniques per day) Wordpress sites. I'm caching with Cloudflare, W3TC, and Varnish. Most of the time, the database server handles traffic very well. "show full processlist" shows 20-40 queries at any given time, with most being in the sleep state.
Periodically, though (particularly when traffic spikes or when a large number of comments are cleared), MySQL stops responding. I'll find 1000-1500 queries running, many "sending data", etc. No particular query seems to be straining the database (they're all standard Wordpress queries), but it just seems like the simultaneous volume of requests causes all queries to hang up. I'm (usually) still able to log in, to run "show full processlist", or other queries, but the 1000+ queries already in there just sit. The only solution seems to be to restart mysql (sometimes violently via kill -9 if I can't connect).
All tables are innodb, server has 8 cores, 24GB RAM, plenty of disk space, and the following is my my.cnf:
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
port=3306
skip-external-locking
skip-name-resolve
user=mysql
query_cache_type=1
query_cache_limit=16M
wait_timeout = 300
query_cache_size=128M
key_buffer_size=400M
thread_cache_size=50
table_cache=8192
skip-name-resolve
max_heap_table_size = 256M
tmp_table_size = 256M
innodb_file_per_table
innodb_buffer_pool_size = 5G
innodb_log_file_size=1G
#innodb_commit_concurrency = 32
#innodb_thread_concurrency = 32
innodb_flush_log_at_trx_commit = 0
thread_concurrency = 8
join_buffer_size = 256k
innodb_log_file_size = 256M
#innodb_concurrency_tickets = 220
thread_stack = 256K
max_allowed_packet=512M
max_connections=2500
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
#2012-11-03
#attempting a ram disk for tmp tables
tmpdir = /db/tmpfs01
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
Any suggestions how I can potentially improve MySQL config, or other steps to maintain database stability under heavy load?
Like has been said, think outside the box and do sone rooting around why these queries are slow or somehow hung. An oldie but a good source of problems even for (supposedly;) intelligent system engineers is load balancing causing issues across webserver or database sessions. With all that caching and load balancing going on, are you sure everything is always connecting end-to-end as intended?
I agree with alditis & Bjoern
I'm pretty noobish with mysql but running mysqltuner can reveal some config optimisations based on recent queries of the DB https://github.com/rackerhacker/MySQLTuner-perl
And if possible store the DB files on a physically separate partition from the OS, the OS can consume IO which slows the DB. Like with Bjoern's logrotate issue.
First have a look at basic system behavior at the moment of problems. Use both vmstat and iostat if you can find any issues. See if the system starts swapping (pi,po columns in vmstat) and if lots of IO is happening. This is the first step in debugging your problem.
Another source of useful information is SHOW INNODB STATUS. See for http://www.mysqlperformanceblog.com/2006/07/17/show-innodb-status-walk-through/ on how to interpret the output.
It might be that at a certain point in time your writes are killing read performance because they flush the query cache.

MySQL restore performance

I have what seems to be a slowing MySQL restore, and am looking for some tuning advice (I am a PostgreSQL and SQL Server guy).
The dev server has 48GB of RAM, 8 cores, running Centos 6.2 64-bit and MySQL 5.1.61 (same as production MySQL), and 4 x 7200 RPM SAS drives in software managed RAID-10 / XFS. The only MySQL client process is the restore. The dump was taken with a plain mysqldump of all databases on the production server.
I have applied some of the options from http://derwiki.tumblr.com/post/24490758395/loading-half-a-billion-rows-into-mysql, including setting FOREIGN_KEY_CHECKS and UNIQUE_CHECKS to zero. I have included my.cnf below.
Monitoring the restore with mytop and pv (pv backup.sql | mysql -u root -p), it appears that the INSERT INTO statements begin to progressively get slower. qps shown by mytop starts at 3, and drops to 0 at 60% through the dump file. Not sure how accurate mytop is in this case, as 3 inserts (with values) still seems slow. htop shows < 10% CPU utilization on the CPU used by MySQL, and less than 8GB of the 48GB of RAM is being utilized.
Different databases, but similar restore techniques, run about 5-10x faster on the same server using PostgreSQL.
Ideas?
[mysqld]
# my.cnf
socket=/var/lib/mysql/mysql.sock
user=mysql
symbolic-links=0
slow-query-log
long_query_time = 60
log-slow-admin-statements
slow_query_log_file = /var/log/mysql_slow.log
innodb_buffer_pool_size = 2G
max_allowed_packet = 1G
key_buffer_size = 1G
concurrent_insert = 1
innodb_flush_log_at_trx_commit = 2
bulk_insert_buffer_size = 1G
innodb_flush_method = O_DIRECT
Sounds like your innodb indexes are slowing you down. If can change the way you dump the database you can remove all non-primary key indexes load the data then re-add them. Better still order the data to be loaded by the primary key. This is probably too much to ask.
Sounds like you are already aware of these tips: http://dev.mysql.com/doc/refman/5.5/en/optimizing-innodb-bulk-data-loading.html
The flush to disk operation (innodb_flush_log_at_trx_commit = 2) may be happening many times a second. Check your innodb_log_file_size * innodb_log_files_in_group is sufficient to avoid writing to disk too often.
(I assumed you are using Innodb from your settings)