MySQL Query cache efficiency - mysql

I'm using MySQLTuner.pl to optimize my site.... though I'm not entirely sure how to resolve some of these issues and am wondering if someone can help me out.
I have about 4GB ram free if needed for MySQL with the following MySQL settings:
key_buffer = 100M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 800
myisam-recover = BACKUP
max_connections = 750
table_cache = 125000
thread_concurrency = 500
query_cache_type=1
query_cache_limit = 128M
query_cache_size = 128M
tmp_table_size = 300M
max_heap_table_size = 300M
innodb_buffer_pool_size = 2G
innodb_file_per_table = 0
innodb_flush_log_at_trx_commit = 2
Here's the output of my tuner
-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.24-0ubuntu0.12.04.1
[OK] Operating on 64-bit architecture
-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 7K (Tables: 10)
[--] Data in InnoDB tables: 27M (Tables: 5)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[!!] Total fragmented tables: 5
-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned
-------- Performance Metrics -------------------------------------------------
[--] Up for: 15d 5h 4m 19s (1B q [1K qps], 208M conn, TX: 172B, RX: 98B)
[--] Reads / Writes: 71% / 29%
[--] Total buffers: 2.5G global + 2.7M per thread (750 max threads)
[OK] Maximum possible memory usage: 4.5G (57% of installed RAM)
[OK] Slow queries: 0% (0/1B)
[OK] Highest usage of available connections: 15% (118/750)
[OK] Key buffer size / total MyISAM indexes: 100.0M/119.0K
[OK] Key buffer hit rate: 100.0% (22M cached / 0 reads)
[!!] Query cache efficiency: 0.1% (703K cached / 519M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (13 temp sorts / 13M sorts)
[OK] Temporary tables created on disk: 25% (4M on disk / 18M total)
[OK] Thread cache hit rate: 99% (752 created / 208M connections)
[OK] Table cache hit rate: 74% (992 open / 1K opened)
[OK] Open file limit used: 0% (68/250K)
[OK] Table locks acquired immediately: 100% (216M immediate / 216M locks)
[OK] InnoDB data size / buffer pool: 27.8M/2.0G
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
Enable the slow query log to troubleshoot bad queries
Variables to adjust:
query_cache_limit (> 128M, or use smaller result sets)
The output of SHOW STATUS LIKE '%cache%' is
Binlog_cache_disk_use 0
Binlog_cache_use 0
Binlog_stmt_cache_disk_use 0
Binlog_stmt_cache_use 0
Com_assign_to_keycache 0
Qcache_free_blocks 19
Qcache_free_memory 134026728
Qcache_hits 704192
Qcache_inserts 143569852
Qcache_lowmem_prunes 0
Qcache_not_cached 11043040
Qcache_queries_in_cache 94
Qcache_total_blocks 217
Ssl_callback_cache_hits 0
Ssl_session_cache_hits 0
Ssl_session_cache_misses 0
Ssl_session_cache_mode NONE
Ssl_session_cache_overflows 0
Ssl_session_cache_size 0
Ssl_session_cache_timeouts 0
Ssl_used_session_cache_entries 0
Threads_cached 748
Is there anything I also can improve for better performance?
Thank you

"Query cache efficiency: 0.1% (703K cached / 519M selects)" means that you have no repeated queries within the cache lifetime cycle.
This may be due to three factors: the cache lifetime is too small for repeats to show, or your queries are too large or different to be cached.
For example, some frameworks will run a large SELECT and then "filter" it server side (maybe even cache it server side). What MySQL sees is a large query that might be deemed best not cached, and MySQL efficiency goes down while overall efficiency stays the same.
Or you might have lots of different clients and every one of them runs some query with custom data (e.g., their message table contents). If you have one thousand customers, each occupying one megabyte, and every one of them checks for message every minute, if you have one gigabyte to dedicate to this category of queries you will see one thousand misses and fifty-nine thousand hits per hour. But if you have only 999 megs, the last client will flush the first client out from the cache, then the first client comes again and gets a miss AND flushes the second client out, and so on; and in that same one hour you see sixty thousand misses, a drop in efficiency from 98% to 0%.
So first of all you need to get a hard look at what queries you're running. Maybe some of them can be optimized, cache-wise: one big query run twice and two refinements instead of two big queries, and the first one gets cached. But keep in mind that you may be doing this at the expense of efficiency down the line.
From your other results, this seems to be the case. You have lots of inserts, lots of free query cache memory, and yet very few hits. So apparently all your queries are different one from the other: in this situation you want to decrease query cache memory to free up memory for any other purpose. Even if the hits decrease further, Amdahl's Law will ensure that query-related performances will not be so relevant (you lose 50% of 0.01%, you're really losing 0.005% overall; do not let that "fifty percent loss!" scare you).
Qcache_free_memory 134,026,728
Qcache_hits 704,192
Qcache_inserts 143,569,852
What surely can't hurt is to check what queries will probably not be worthwhile, and those can be upgraded with a SQL_NO_CACHE. They will still become cache misses, but won't degrade performance of other queries, which may find more memory and improve their performance. This is at zero cost down the line anyway, so can always be done (for queries big enough that they're worth the trouble, of course!).
After you've done this, you can experiment with the query_cache_size parameter. Check how much memory the other components are using, and the performances of the file system cache. You don't want to speed up a query and see the page coming out in twice the time because the view and controller components retrieval has got it in the neck.
Another thing you may want to look at are the indexes. You can lower the slow query limit and check out whether there are queries running significantly slower than others, and check why by looking through their query plan.

Related

MySQL High CPU Consumption (600%)

We have a high traffic server running mysql. This database has run fine for years with no significant spikes in traffic or changes to the database but recently Mysql CPU consumption has spiked to a consistent 600%.
We don't see any slow queries being logged but we get the following recommendations.
We're not very knowledgeable with MySQL and everything we've tried to fix the CPU consumption has not worked.
If anyone can provide some recommended settings that would be appreciated.
top - 08:08:59 up 313 days, 12:47, 2 users, load average: 7.21, 6.48, 6.08
Tasks: 294 total, 1 running, 293 sleeping, 0 stopped, 0 zombie
Cpu(s): 29.9%us, 10.6%sy, 3.9%ni, 47.1%id, 3.1%wa, 0.0%hi, 5.5%si, 0.0%st
Mem: 8018016k total, 7869300k used, 148716k free, 587632k buffers
Swap: 0k total, 0k used, 0k free, 5894136k cached
[mysqld]
# skip-networking
#tmpdir=/dev/shm
general_log = 1
slow_query_log = 1
log-slow-queries = /var/log/mysql-slow.log
slow-query_log_file = /var/log/mysql-slow.log
long_query_time = 2
sort_buffer_size = 4M
max_connections = 151
max_allowed_packet = 100M
query_cache_size = 128M
query_cache_limit = 5M
join_buffer_size = 10M
tmp_table_size = 64M
max_heap_table_size = 64M
thread_cache_size = 4
table_cache = 20k
innodb_buffer_pool_size = 512M
key_buffer_size = 128M
local-infile=0
open_files_limit=2058
wait_timeout = 500
connect_timeout = 500
interactive_timeout = 500
performance_schema = on
>> MySQLTuner 1.4.0 - Major Hayden <major#mhtx.net>
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with '--help' for additional options and output filtering
[OK] Logged in using credentials passed on the command line
[OK] Currently running supported MySQL version 5.5.50-log
[OK] Operating on 64-bit architecture
-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MRG_MYISAM
[--] Data in MyISAM tables: 453M (Tables: 169)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[!!] InnoDB is enabled but isn't being used
[!!] Total fragmented tables: 5
-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned
-------- Performance Metrics -------------------------------------------------
[--] Up for: 22m 39s (763K q [561.834 qps], 22K conn, TX: 506M, RX: 95M)
[--] Reads / Writes: 71% / 29%
[--] Total buffers: 848.0M global + 14.6M per thread (151 max threads)
[OK] Maximum possible memory usage: 3.0G (39% of installed RAM)
[OK] Slow queries: 0% (1/763K)
[OK] Highest usage of available connections: 52% (80/151)
[OK] Key buffer size / total MyISAM indexes: 128.0M/273.3M
[OK] Key buffer hit rate: 100.0% (72M cached / 29K reads)
[OK] Query cache efficiency: 79.1% (472K cached / 596K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (8 temp sorts / 43K sorts)
[OK] Temporary tables created on disk: 0% (37 on disk / 10K total)
[!!] Thread cache hit rate: 47% (12K created / 22K connections)
[OK] Table cache hit rate: 97% (249 open / 256 opened)
[OK] Open file limit used: 0% (277/41K)
[!!] Table locks acquired immediately: 53%
-------- Recommendations -----------------------------------------------------
General recommendations:
Add skip-innodb to MySQL configuration to disable InnoDB
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours - recommendations may be inaccurate
Optimize queries and/or use InnoDB to reduce lock wait
Variables to adjust:
thread_cache_size (> 4)
Suboptimal caching method.
You are using the MySQL Query cache with a fairly high traffic database. It might be worth considering to use memcached instead of the MySQL Query cache, especially if you have multiple slaves.
Less than 80% of the query cache is being utilized.
This might be caused by query_cache_limit being too low. Flushing the query cache might help as well.
The query cache is considerably fragmented.
Severe fragmentation is likely to (further) increase Qcache_lowmem_prunes. This might be caused by many Query cache low memory prunes due to query_cache_size being too small. For a immediate but short lived fix you can flush the query cache (might lock the query cache for a long time). Carefully adjusting query_cache_min_res_unit to a lower value might help too, e.g. you can set it to the average size of your queries in the cache using this formula: (query_cache_size - qcache_free_memory) / qcache_queries_in_cache
There are lots of rows being sorted.
While there is nothing wrong with a high amount of row sorting, you might want to make sure that the queries which require a lot of sorting use indexed columns in the ORDER BY clause, as this will result in much faster sorting
There are too many joins without indexes.
This means that joins are doing full table scans. Adding indexes for the columns being used in the join conditions will greatly speed up table joins
The rate of reading the first index entry is high.
This usually indicates frequent full index scans. Full index scans are faster than table scans but require lots of CPU cycles in big tables, if those tables that have or had high volumes of UPDATEs and DELETEs, running 'OPTIMIZE TABLE' might reduce the amount of and/or speed up full index scans. Other than that full index scans can only be reduced by rewriting queries.
The rate of reading data from a fixed position is high.
This indicates that many queries need to sort results and/or do a full table scan, including join queries that do not use indexes. Add indexes where applicable.
The rate of reading the next table row is high.
This indicates that many queries are doing full table scans. Add indexes where applicable.
Many temporary tables are being written to disk instead of being kept in memory.
Increasing max_heap_table_size and tmp_table_size might help. However some temporary tables are always being written to disk, independent of the value of these variables. To eliminate these you will have to rewrite your queries to avoid those conditions (Within a temporary table: Presence of a BLOB or TEXT column or presence of a column bigger than 512 bytes) as mentioned in the MySQL Documentation
MyISAM key buffer (index cache) % used is low.
You may need to decrease the size of key_buffer_size, re-examine your tables to see if indexes have been removed, or examine queries and expectations about what indexes are being used.
The rate of opening tables is high.
Opening tables requires disk I/O which is costly. Increasing table_open_cache might avoid this.
The rate of opening files is high.
Consider increasing open_files_limit, and check the error log when restarting after changing open_files_limit.
Too many table locks were not granted immediately.
Optimize queries and/or use InnoDB to reduce lock wait.
Too many table locks were not granted immediately.
Optimize queries and/or use InnoDB to reduce lock wait.
Too many connections are aborted.
Connections are usually aborted when they cannot be authorized. This article might help you track down the source.
The InnoDB log file size is not an appropriate size, in relation to the InnoDB buffer pool.
Especially on a system with a lot of writes to InnoDB tables you should set innodb_log_file_size to 25% of innodb_buffer_pool_size. However the bigger this value, the longer the recovery time will be when database crashes, so this value should not be set much higher than 256 MiB. Please note however that you cannot simply change the value of this variable. You need to shutdown the server, remove the InnoDB log files, set the new value in my.cnf, start the server, then check the error logs if everything went fine. See also this blog entry
EDIT
-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MRG_MYISAM
[--] Data in MyISAM tables: 461M (Tables: 168)
[--] Data in InnoDB tables: 16K (Tables: 1)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[!!] Total fragmented tables: 8
-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned
-------- Performance Metrics -------------------------------------------------
[--] Up for: 25d 22h 33m 27s (1B q [533.080 qps], 35M conn, TX: 801B, RX: 148B)
[--] Reads / Writes: 71% / 29%
[--] Total buffers: 1020.0M global + 14.6M per thread (151 max threads)
[OK] Maximum possible memory usage: 3.2G (41% of installed RAM)
[OK] Slow queries: 0% (7/1B)
[OK] Highest usage of available connections: 55% (84/151)
[OK] Key buffer size / total MyISAM indexes: 300.0M/317.3M
[OK] Key buffer hit rate: 100.0% (35B cached / 18K reads)
[OK] Query cache efficiency: 79.6% (743M cached / 933M selects)
[!!] Query cache prunes per day: 34376
[OK] Sorts requiring temporary tables: 0% (48K temp sorts / 68M sorts)
[OK] Temporary tables created on disk: 0% (1K on disk / 17M total)
[OK] Thread cache hit rate: 76% (8M created / 35M connections)
[OK] Table cache hit rate: 98% (504 open / 511 opened)
[OK] Open file limit used: 1% (678/41K)
[!!] Table locks acquired immediately: 74%
[OK] InnoDB buffer pool / data size: 512.0M/16.0K
[OK] InnoDB log waits: 0
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
Optimize queries and/or use InnoDB to reduce lock wait
Variables to adjust:
query_cache_size (> 128M)
Upgrade -- Support of 5.5 ended about 7 years ago.
Tip-top recommendation: Switch your tables from MyISAM to InnoDB. And be sure to adjust key_buffer_size down to 20M and innodb_buffer_pool_size higher, but not so high that it causes swapping.
Top recommendation: Use the SlowLog that you have been capturing. Digest it and let's discuss the 'worst' couple of queries.
Well, maybe there is nothing much in the slowlog ("Slow queries: 0% (1/763K)"). So lower long_query_time to 0.4
The query cache may be hurting more than it is helping. It is hard to say.
Not much of the 8MB is taken up by MySQL; do you have other apps running on the same machine? Changing memory size will not relieve CPU issues. Using a little more memory to raise table_open_cache, open_files_limit, etc, may help with the CPU.
If you are not using MyISAM, lower key_buffer_size to 20M.
Ignore the note about "fragmented tables".
Turn off the general_log; it rapidly fills the disk. (Not much CPU impact.)
"Up for: 22m 39s" -- The analysis is mostly focused on "startup" activity; hence does not adequately address your question. Wait 24 hours.
Deeper analysis: http://mysql.rjweb.org/doc.php/mysql_analysis#tuning
Before I get started, I'll mention that everything in the recommended list is worth looking at and I agree strongly with the other answer that 5.5 is crazy-old — to the point of not even getting security patches. Upgrading is a very good idea.
That out of the way, I want to focus on this:
This database has run fine for years with no significant spikes in traffic or changes to the database
With that in mind, I know two common reasons you can see a sudden drop in performance like this:
The disks are on the verge of failing, such that some disk I/O operations are slow or repeating. If this is your issue, you'll likely find other evidence of it in the core server (not MySql) logs... but not always!
As you see data growth — not just over time but in terms of business growth (hopefully the business has more customers than it did when the server was installed) — it takes more memory to keep active records buffered in RAM instead of frequently reading from the (much slower) disks than it used to. You can hit a tipping point, where suddenly many of your queries that used to finish entirely in RAM now need to spill out to disk.
Yes, yes, I know both of these are disk issues, while the problem presents as CPU use. However, disk I/O problems often show up as CPU load issues, with processes waiting on disk to complete.
The solution to both of these usually involves hardware upgrades... but you can delay this if it's the second reason by working the list of recommendations in your question.
Master00, since you need and want to stay with MyISAM, consider the following suggestions to IMPROVE your response time with MyISAM.
Corrected variable name to key_cache_age_threshold - my humble apologies. Wilson
key_cache_block_size=16384 # from 1024 to reduce overhead managing key_buffer data
key_cache_division_limit=50 # from 100 percent to enable Hot / Warm cache separation
key_cache_age_threshold=7200 # from 300 seconds before AGE_OUT causes another READ
and the these three changes will reduce your key_reads Rate Per Second significantly.
For additional useful tips, view my profile for contact info and get in touch.

mysqltuner output, recommendations for settings?

I do not have 250 GB of RAM (way our of budget), I currently have 70GB and it fills to 57GB after an hour of usage (mysql buffers).
I am mostly unsure about:
temporary table buffers
query cache prunes
join_buffer and query_cache size
The innodb buffer is probably not too small, a large part of those 240GB are rarely accessed. The mostly access stuff is all buffered I think.
Some of those values are already at the edge where I read that it's not beneficial (like the 128MB tmp_table or 1MB join_buffer)
[--] Up for: 9h 51m 30s (78M q [2K qps], 921K conn, TX: 110B, RX: 26B)
[--] Reads / Writes: 49% / 51%
[--] Total buffers: 46.6G global + 1.9M per thread (1500 max threads)
[OK] Maximum possible memory usage: 49.3G (82% of installed RAM)
[OK] Slow queries: 0% (1K/78M)
[OK] Highest usage of available connections: 28% (428/1500)
[OK] Key buffer size / total MyISAM indexes: 256.0M/1.6G
[OK] Key buffer hit rate: 99.9% (105M cached / 55K reads)
[OK] Query cache efficiency: 26.9% (11M cached / 44M selects)
[!!] Query cache prunes per day: 4482061
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 10M sorts)
[!!] Joins performed without indexes: 603
[!!] Temporary tables created on disk: 49% (1M on disk / 2M total)
[OK] Thread cache hit rate: 99% (6K created / 921K connections)
[OK] Table cache hit rate: 83% (1K open / 2K opened)
[OK] Open file limit used: 12% (633/5K)
[OK] Table locks acquired immediately: 99% (9M immediate / 9M locks)
[!!] InnoDB buffer pool / data size: 46.0G/240.6G
[OK] InnoDB log waits: 0
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours - recommendations may be inaccurate
Increasing the query_cache size over 128M may reduce performance
Adjust your join queries to always utilize indexes
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Variables to adjust:
query_cache_size (> 180M) [see warning above]
join_buffer_size (> 1.0M, or always use indexes with joins)
tmp_table_size (> 128M)
max_heap_table_size (> 128M)
innodb_buffer_pool_size (>= 240G)
config:
max_allowed_packet = 64M
thread_stack = 256K
thread_cache_size = 16
max_connections = 1500
max_user_connections = 850
query_cache_limit = 3M
query_cache_size = 180M
query_cache_type = 1
table_open_cache = 2500
key_buffer_size = 256M # index in memory for myisam
innodb_buffer_pool_size = 46G
innodb_log_file_size = 256M
tmp_table_size = 128M
max_heap_table_size = 128M
join_buffer_size = 1M
Let's try to work within the 70GB of RAM. "InnoDB buffer pool / data size: 46.0G/240.6G" is fine if you are not thrashing too badly. Usually improving indexes and/or queries can help.
What I see of the Query cache specs is not too bad. But how big is it? What is the value of query_cache_size? If it is more than 50M, it is too big because the prunes are costly.
Don't change the other settings that mysqltuner recommends.
You have about 50 query_cache_prunes per second, which is too high.
So, although the hit rate is high, I suspect the QC is not well tuned. Recommendations:
Discover which queries are likely to be repeated exactly, and which are not. Change the frequently used ones to SELECT SQL_CACHE ... and change the infrequent ones ones to SELECT SQL_NO_CACHE .... And change to query_cache_type = DEMAND.
To dig deeper, and to fix some slow queries, please follow the suggestions here.
In your configuration file things to do:
thread_cache_size=100 # from 16 for V8 suggested CAP on busy system
innodb_buffer_pool_instances=8 # from default 1 to reduce mutex contention
query_cache_min_res_unit=512 # from 4K to increase capacity of QC
query_cache_limit=1M # from 3M - 180M QC could only hold 60 of them
query_prealloc_size=32768 # from def 8192 to reduce malloc load
max_write_lock_count=2 # from a huge number to give RD opportunity every 2
max_connections=500 # from 1500 will leave 20% growth room
thread_concurrency=30 # to prevent saturation
key_age_threshold=64800 # why discard a key at 5 minutes? keep keep 18 hrs
key_cache_division_limit=50 # from 100% for warm cache to manage
key_cache_block_size=32768 # from 1024 to reduce CPU overhead

Optimizing mysql my.cnf - memory usage is dangerously high

I have tunet up mysql by following some tips on the internet.
but the websites are loading slow and time time-to-first-byte is very high. therefore i started investigation and as far as i see, it is casued by mysql and high memory usage
every time i make changes to the my.cnf according to the suggestions i get from
./mysqltuner.pl AND ./tuning-primer.sh
it gives another suggestions. Actually some values should have balance to each other. I hope someone have an idea how to get high performance usage of mysql and in the same time, take care of the server health
MariaDB server version: 10.0.20-MariaDB-log MariaDB Server
Server information:
Intel Core i7-3770
2x HDD 3,0 TB SATA
4x RAM 8192 MB DDR3
Cloudlinux + Cpanel installed
Apache/2.4.16 + eAccelerator + mod_pagespeed
SLOW QUERIES
Error: slow_query_log_file=/var/log/mysql/log-slow-queries.log
Current long_query_time = 10.000000 sec.
You have 1944 out of 25401054 that take longer than 10.000000 sec. to complete
**Your long_query_time seems to be fine**
BINARY UPDATE LOG
The binary update log is NOT enabled.
**You will not be able to do point in time recovery**
See http://dev.mysql.com/doc/refman/10.0/en/point-in-time-recovery.html
WORKER THREADS
Current thread_cache_size = 8
Current threads_cached = 6
Current threads_per_sec = 0
Historic threads_per_sec = 0
**Your thread_cache_size is fine**
MAX CONNECTIONS
Current max_connections = 151
Current threads_connected = 6
Historic max_used_connections = 43
The number of used connections is 28% of the configured maximum.
**Your max_connections variable seems to be fine.**
No InnoDB Support Enabled!
MEMORY USAGE
Max Memory Ever Allocated : 20.96 G
Configured Max Per-thread Buffers : 24.81 G
Configured Max Global Buffers : 13.89 G
Configured Max Memory Limit : 38.70 G
Physical Memory : 31.12 G
**Max memory limit exceeds 90% of physical memory**
KEY BUFFER
Current MyISAM index space = 29 M
Current key_buffer_size = 384 M
Key cache miss rate is 1 : 870
Key buffer free ratio = 78 %
**Your key_buffer_size seems to be fine**
QUERY CACHE
Query cache is enabled
Current query_cache_size = 512 M
Current query_cache_used = 222 M
Current query_cache_limit = 1.00 G
Current Query cache Memory fill ratio = 43.41 %
Current query_cache_min_res_unit = 4 K
**MySQL won't cache query results that are larger than query_cache_limit in size**
SORT OPERATIONS
Current sort_buffer_size = 16 M
Current read_rnd_buffer_size = 8 M
**Sort buffer seems to be fine**
JOINS
./tuning-primer.sh: line 402: export: `2097152': not a valid identifier
Current join_buffer_size = 128.00 M
You have had 10199 queries where a join could not use an index properly
join_buffer_size >= 4 M
**This is not advised
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.**
OPEN FILES LIMIT
Current open_files_limit = 16162 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
**Your open_files_limit value seems to be fine**
TABLE CACHE
Current table_open_cache = 8000 tables
Current table_definition_cache = 8000 tables
You have a total of 7347 tables
You have 8000 open tables.
**Current table_cache hit rate is 21%
, while 100% of your table cache is in use
You should probably increase your table_cache**
TEMP TABLES
Current max_heap_table_size = 16 M
Current tmp_table_size = 16 M
Of 592173 temp tables, 36% were created on disk
**Perhaps you should increase your tmp_table_size and/or max_heap_table_size
to reduce the number of disk-based temporary tables
Note! BLOB and TEXT columns are not allow in memory tables.
If you are using these columns raising these values might not impact your
ratio of on disk temp tables.**
TABLE SCANS
Current read_buffer_size = 16 M
Current table scan ratio = 74 : 1
**read_buffer_size is over 8 MB there is probably no need for such a large read_buffer**
TABLE LOCKING
Current Lock Wait ratio = 1 : 4366
**You may benefit from selective use of InnoDB.
If you have long running SELECT's against MyISAM tables and perform
frequent updates consider setting 'low_priority_updates=1'
If you have a high concurrency of inserts on Dynamic row-length tables
consider setting 'concurrent_insert=ALWAYS'.**
root#my [~]# ./mysqltuner.pl
>> MySQLTuner 1.4.0 - Major Hayden <major#mhtx.net>
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with '--help' for additional options and output filtering
[!!] Currently running unsupported MySQL version 10.0.20-MariaDB-log
[OK] Operating on 64-bit architecture
-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +Aria +BLACKHOLE +CSV +FEDERATED +InnoDB +MRG_MyISAM
[--] Data in MyISAM tables: 82M (Tables: 925)
[--] Data in InnoDB tables: 7G (Tables: 6334)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 52)
[--] Data in MEMORY tables: 0B (Tables: 2)
[!!] Total fragmented tables: 159
-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned
-------- Performance Metrics -------------------------------------------------
[--] Up for: 14h 42m 18s (25M q [480.374 qps], 81K conn, TX: 71B, RX: 6B)
[--] Reads / Writes: 98% / 2%
[--] Total buffers: 13.9G global + 168.3M per thread (151 max threads)
**[!!] Maximum possible memory usage: 38.7G (124% of installed RAM)**
[OK] Slow queries: 0% (1K/25M)
[OK] Highest usage of available connections: 28% (43/151)
[OK] Key buffer size / total MyISAM indexes: 384.0M/29.8M
[OK] Key buffer hit rate: 99.9% (12M cached / 14K reads)
[OK] Query cache efficiency: 44.9% (20M cached / 44M selects)
**[!!] Query cache prunes per day: 2013573**
[OK] Sorts requiring temporary tables: 0% (1K temp sorts / 1M sorts)
**[!!] Joins performed without indexes: 10207
[!!] Temporary tables created on disk: 58% (345K on disk / 592K total)**
[OK] Thread cache hit rate: 98% (1K created / 81K connections)
[OK] Table cache hit rate: 21% (8K open / 38K opened)
[OK] Open file limit used: 12% (1K/16K)
[OK] Table locks acquired immediately: 99% (10M immediate / 10M locks)
[OK] InnoDB buffer pool / data size: 10.0G/7.7G
[OK] InnoDB log waits: 0
-------- Recommendations -----------------------------------------------------
General recommendations:
**Run OPTIMIZE TABLE to defragment tables for better performance
Reduce your overall MySQL memory footprint for system stability
Increasing the query_cache size over 128M may reduce performance
Adjust your join queries to always utilize indexes
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses**
Variables to adjust:
*** MySQL's maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
query_cache_size (> 512M) [see warning above]
join_buffer_size (> 128.0M, or always use indexes with joins)
tmp_table_size (> 16M)
max_heap_table_size (> 16M)
And here is the my.cnf settings which i have set
[mysqld]
#http://blog.secaserver.com/2011/08/mysql-recommended-my-cnf-settings-innodb-engine/
# GENERAL #
default-storage-engine=InnoDB
tmpdir=/tmp_mysql
group_concat_max_len=10000000
local-infile=1
# LOGGING #
slow_query_log = 1
slow_query_log_file=/var/log/mysql/log-slow-queries.log
long_query_time = 10
log-error = /var/log/error.log
log-queries-not-using-indexes
# CACHES AND LIMITS AND SAFETY #
max_allowed_packet = 512M #16
query_cache_size = 512M
query_cache_limit = 1024M
thread_cache_size = 8
table_definition_cache = 8000
table_open_cache = 8000
sort_buffer_size = 16M
read_buffer_size = 16M #2
read_rnd_buffer_size = 8M
join_buffer_size = 128M
thread_concurrency = 0 # Try number of CPU's*2 for thread_concurrency
key_buffer = 256M
# INNODB #
innodb_file_per_table=1
innodb_file_format = Barracuda
innodb_sort_buffer_size = 128M
innodb_data_home_dir = /var/lib/mysql
innodb_log_group_home_dir = /var/lib/mysql
innodb_thread_concurrency=0
innodb_flush_method=O_DIRECT
innodb_lock_wait_timeout = 120
innodb_buffer_pool_size=10G
innodb_log_file_size = 1536M # Set .. innodb_log_file_size to 25 % of innodb_buffer_pool_size -You can set .. innodb_buffer_pool_size up to 50 - 80 %
innodb_log_buffer_size = 3072M
innodb_additional_mem_pool_size = 20M
#innodb_read_io_threads=16
#innodb_write_io_threads=16
#innodb_io_capacity=500
#innodb_flush_log_at_trx_commit=1
#sync_binlog=1
#innodb_data_file_path = ibdata1:2000M;ibdata2:10M:autoextend
# MyISAM #
key_buffer_size = 384M
myisam_sort_buffer_size = 64M
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
Summary
Don't worry about the memory "problem"
Look at the slowlog - you have some serious optimizations to deal with
Why thousands of tables? This is probably a design flaw.
Ignore the comments about fragmentation.
Convert the rest of your tables to InnoDB.
Details
Your long_query_time seems to be fine
No, crank long_query_time down to 2 and keep an eye on the slow queries.
No InnoDB Support Enabled!
[--] Data in InnoDB tables: 7G (Tables: 6334)
Those statments contradict each other!
InnoDB is the preferred engine. The other outputs seem to say that InnoDB is in use. (Bug in the script?)
Max memory limit exceeds 90% of physical memory
[!!] Maximum possible memory usage: 38.7G (124% of installed RAM)
There is no forumla without flaws. You are probably OK.
Your key_buffer_size seems to be fine
The key_buffer should be the largest memory usage in a MyISAM_only server. But you have a tiny database, so no problem.
The innodb_buffer_pool_size should be the largest memory user when using InnoDB.
Your current values are good:
innodb_buffer_pool_size = 10G
key_buffer_size = 384M
Current query_cache_size = 512 M
[OK] Query cache efficiency: 44.9% (20M cached / 44M selects)
[!!] Query cache prunes per day: 2013573
The 512M hurts performance; limit it to 50M.
The other two lines say that the QC is not that useful, and has a lot pruning overhead.
Conclusion: Cautiously try
query_cache_size = 0
query_cache_type = OFF
./tuning-primer.sh: line 402: export: `2097152': not a valid identifier
Bug in the script?
You should enable "log-queries-not-using-indexes"
No, it just clutters the slowlog.
A one-line table with no index, or even a 100-row table with no index is likely to be "fast enough".
Instead, look at the existing slow queries to decide what should be worked on.
You have a total of 7347 tables
(Tables: 6334)
That's a design flaw in your database.
Of 592173 temp tables, 36% were created on disk
Again, the slowlog can help identify the worst queries. Then we can work on fixing them, either by adding an index or by reformulating the query.
[!!] Total fragmented tables: 159
Run OPTIMIZE TABLE to defragment tables
Ignore -- virtually all tables are virtually always fragmented. No action item here.
[480.374 qps]
A good reason to go to InnoDB.
[!!] Joins performed without indexes: 10207
[!!] Temporary tables created on disk: 58% (345K on disk / 592K total)**
Gotta see the slowlog entries. Let's start with the 10 second ones you have already caught.
#innodb_flush_log_at_trx_commit=1
Suggest setting to 2.
(This is the first time I have seen tuning-primer.sh; I am not impressed.)
These two blogs are likely to be useful:
MyISAM to InnoDB
Cookbook for creating INDEXes
It seems more likely to be an incomplete indexing issue.
verify all fields in WHERE clause of slow queries to be indexed properly
if you have Nfields-in-one indexes try spreading them separately (if they make subject of separate condition) as only the first one will be really indexed or all N together one a WHERE clause
avoid redundant indexing (enum like cases or advanced bit)
db structure might need a little touch too - but this really depends on your db profile and of course the way data gets updated and viewed (you can improve using mixed type of engines per case - but you gotta know how to tweak your my.cnf properly)

What does "MySQL's maximum memory usage is dangerously high" mean by mysqltuner?

I am trying to optimize my mysql runs on a 2GB mem VPS, I used mysqltuner, and I dont quite understand how to deal with the following recommendations, especially the one says:
MySQL's maximum memory usage is dangerously high, deal with this one? Can someone help explain? Thanks.
-------- Performance Metrics -------------------------------------------------
[--] Up for: 3h 17m 7s (49K q [4.190 qps], 1K conn, TX: 70M, RX: 7M)
[--] Reads / Writes: 60% / 40%
[--] Total buffers: 314.0M global + 6.4M per thread (300 max threads)
[!!] Maximum possible memory usage: 2.2G (119% of installed RAM)
[OK] Slow queries: 1% (785/49K)
[OK] Highest usage of available connections: 85% (256/300)
[!!] Cannot calculate MyISAM index size - re-run script as root user
[OK] Query cache efficiency: 92.4% (38K cached / 41K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 633 sorts)
[!!] Temporary tables created on disk: 45% (315 on disk / 699 total)
[OK] Thread cache hit rate: 74% (359 created / 1K connections)
[OK] Table cache hit rate: 95% (141 open / 148 opened)
[OK] Open file limit used: 12% (189/1K)
[OK] Table locks acquired immediately: 99% (6K immediate / 6K locks)
-------- Recommendations -----------------------------------------------------
General recommendations:
Add skip-innodb to MySQL configuration to disable InnoDB
MySQL started within last 24 hours - recommendations may be inaccurate
Reduce your overall MySQL memory footprint for system stability
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Variables to adjust:
*** MySQL's maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
tmp_table_size (> 32M)
max_heap_table_size (> 32M)
"[!!] Maximum possible memory usage: 2.2G (119% of installed RAM)"
This means you essentially lied to MySQL, telling it you have more memory available than you really have, 2.2G > 2G. This might work for weeks or months, but it's a bad idea. If MySQL doesn't have the memory you told it to use, MySQL will crash randomly at the worst possible time.
If you add "skip-innodb" to your /etc/my.cnf file, that might save you some memory. I assume you're not using InnoDB. This is a tangent but I strongly advise you convert your data from MyISAM to InnoDB. MyISAM is old technology. InnoDB is the more modern engine.
Look for anything in your my.cnf that you can lower to save memory. The first thing I typically look at, unused connections. 15% of your connections aren't being used, but listen to the "started within 24 hours" warning. Typically lowering (unused) connections in my.cnf will save lots of memory. I don't know what your application does, but 256 connections sounds high to me. So I'd make sure your application really needs that many connections. Maybe you have 256 PHP children on your server and that could be cut way down 12 children. More children != faster response. If you have 12 PHP children, maybe you only need 13 database connections.
119% is obviously too high but I think 96% is too high also. (Which is why I'm here looking for the best % to use.) Obviously the operating system needs some memory too. How much memory should you leave unused for your operating system, I would like to know! I would ask this as a separate question here, if it hasn't been asked already. (Please post the link here if you do this.) Or you can just listen to mysqltuner's recommendation.
Just testing here:
"[!!] Maximum possible memory usage: 3.4G (88% of installed RAM)"
Lower my.cnf settings again.
"[!!] Maximum possible memory usage: 3.3G (86% of installed RAM)"
Still too high?
"[OK] Maximum possible memory usage: 3.2G (83% of installed RAM)"
Take the advice from mysqltuner with a grain of salt. It's making incorrect estimates about the maximum possible memory usage. And it can't make a correct estimate.
See http://www.percona.com/blog/2009/02/12/how-much-memory-can-mysql-use-in-the-worst-case/ for an explanation.
It's true that there each connection uses some memory, but how much varies. You won't always have 300 connections in use, and even when they are, they won't always be running queries simultaneously, and even if they are, the queries won't always be using all possible buffers to the maximum size.
Mysqltuner is warning about a theoretical maximum memory usage that will never happen.
Another way of looking at it: I've analyzed hundreds of MySQL configurations, and every one of them could in theory allocate more memory than the physical RAM on the server.

MySQL Query Cache: Yes or No for large website building platform

Wading through so much information and conflicting advice on MySQL's query cache can be really overwhelming and hard to tell how it pertains to my use case.
We have a large custom platform running on a LAMP stack that hosts many websites. Each website has its content stored in a table row. For the most part they're all in the same table.
I've read that all cached queries are invalidated whenever a table is updated, but I've also read conflicting things about it.
Say somebody visits Website A and its content is loaded from the database and cached in the process. Another person visits right after and the site loads faster for them because the data were cached. Now Website B's content is changed, which is a row in the same table as Website A. Is all the cached data from Website A now invalidated? And if so, would we actually see a performance increase by turning off the query cache entirely?
I've been reading up on tuning the query cache and again, very overwhelming. I tried a few things but it's hard to tell how much of an effect they had. Here is a current paste from the MySQLTunerscript:
-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.1.62-cll
[OK] Operating on 64-bit architecture
-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 4G (Tables: 1977)
[--] Data in InnoDB tables: 384K (Tables: 16)
[!!] Total fragmented tables: 33
-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned
-------- Performance Metrics -------------------------------------------------
[--] Up for: 5d 1h 30m 33s (53M q [122.486 qps], 1M conn, TX: 125B, RX: 13B)
[--] Reads / Writes: 83% / 17%
[--] Total buffers: 8.1G global + 5.5M per thread (500 max threads)
[OK] Maximum possible memory usage: 10.8G (46% of installed RAM)
[OK] Slow queries: 0% (2K/53M)
[OK] Highest usage of available connections: 5% (28/500)
[OK] Key buffer size / total MyISAM indexes: 8.0G/1.2G
[OK] Key buffer hit rate: 99.7% (1B cached / 3M reads)
[!!] Query cache efficiency: 16.2% (6M cached / 41M selects)
[!!] Query cache prunes per day: 2869188
[OK] Sorts requiring temporary tables: 0% (11 temp sorts / 609K sorts)
[OK] Temporary tables created on disk: 0% (11K on disk / 2M total)
[OK] Thread cache hit rate: 99% (28 created / 1M connections)
[!!] Table cache hit rate: 1% (1K open / 88K opened)
[OK] Open file limit used: 3% (2K/65K)
[OK] Table locks acquired immediately: 99% (41M immediate / 41M locks)
[OK] InnoDB data size / buffer pool: 384.0K/8.0M
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
Enable the slow query log to troubleshoot bad queries
Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
query_cache_limit (> 6M, or use smaller result sets)
query_cache_size (> 96M)
table_cache (> 1024)
Thanks in advance for any advice.
The answer would be very yes.
It is invalidated, and it's completely transparent unless you disable query_cache_wlock_invalidate which you may want to check out because it could still be worth disabling if you use MyISAM tables.
Yet it's saving 16% of your read queries and solving them immediately without bothering the storage engine, which in the case of MyISAM most often means without bothering the I/O system too. This is great enough! As a matter of fact the query cache is about the best thing to do with your RAM for caching, so set the query cache to something like 128 MB, and after some normal/heavy operation, SHOW GLOBAL STATUS LIKE '%qcache%' shows a low or high Qcache_free_memory: if it's low, increase your query cache, even at the expense of other database caches such as the InnoDB buffer pool; if it's high, reduce your query cache by almost that much because unfortunately you won't get to use more of it in your work set.
Last time I was looking at query cache it was a definite no-no to me. While adding ability to cache results it also slows MySQL down a bit as it takes time to invalidate cache entries. I wouldn't recommend turning it on explicitly unless you test your workload against it (as manual suggests: http://dev.mysql.com/doc/refman/5.1/en/query-cache.html).
Good way for testing it would be snapshotting database and capturing the following 1 hour of requests. This way you can setup another server and run all test there without heavily loading your production machine.