I'm trying to tune my Magento DB for optimal performance.
I'm running nginx, php-fpm and mysql on a 4GB RAM, 8CPU core virtual machine with 4GB of RAM.
I've ran the Mysql Tuning Primer and everything looks good apart from my Table Cache:
TABLE CACHE
Current table_open_cache = 1000 tables
Current table_definition_cache = 400 tables
You have a total of 2510 tables
You have 1000 open tables.
Current table_cache hit rate is 3%
, while 100% of your table cache is in use
You should probably increase your table_cache
You should probably increase your table_definition_cache value.
and from mysqltuner
[!!] Table cache hit rate: 9% (1K open / 10K opened)
[!!] Query cache efficiency: 0.0% (0 cached / 209 selects)
The relevant settings from the my.cnf file:
table_cache = 1000
query_cache_limit = 1M
query_cache_size = 64M
The thing is, no matter what I increase my table_cache to - it seems to be consumed almost immediately. Is this normal for Magento? It seems abnormally high?
Does anyone have any tips about what I can do to improve this?
Thanks,
Ed
Check your MySQL config's query cache type setting:
http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html#sysvar_query_cache_type
If you set it to 0 or 2 then it will either not cache any queries or only cache the ones that you have specifically asked to cache. That means Magento would have to explicitly ask for cached query results (I'm not sure it does that). If you set it to 1 then it will cache all queries except those that explicitly ask for no query cache.
Table cache refers to potential open file pointers. It could be consumed rather quickly, and will just roll off unused entries as needed. From MySQL's documentation:
The table_cache and max_connections system variables affect the
maximum number of files the server keeps open. If you increase one or
both of these values, you may run up against a limit imposed by your
operating system on the per-process number of open file descriptors.
Many operating systems permit you to increase the open-files limit,
although the method varies widely from system to system. Consult your
operating system documentation to determine whether it is possible to
increase the limit and how to do so.
table_cache is related to max_connections. For example, for 200
concurrent running connections, you should have a table cache size of
at least 200 * N, where N is the maximum number of tables per join in
any of the queries which you execute. You must also reserve some extra
file descriptors for temporary tables and files.
Make sure that your operating system can handle the number of open
file descriptors implied by the table_cache setting. If table_cache is
set too high, MySQL may run out of file descriptors and refuse
connections, fail to perform queries, and be very unreliable. You also
have to take into account that the MyISAM storage engine needs two
file descriptors for each unique open table. You can increase the
number of file descriptors available to MySQL using the
--open-files-limit startup option to mysqld. See Section C.5.2.18, “'File' Not Found and Similar Errors”.
The cache of open tables is kept at a level of table_cache entries.
The default value is 64; this can be changed with the --table_cache
option to mysqld. Note that MySQL may temporarily open more tables
than this to execute queries.
MySQL closes an unused table and removes it from the table cache under
the following circumstances:
When the cache is full and a thread tries to open a table that is not
in the cache.
When the cache contains more than table_cache entries and a table in
the cache is no longer being used by any threads.
When a table flushing operation occurs. This happens when someone
issues a FLUSH TABLES statement or executes a mysqladmin flush-tables
or mysqladmin refresh command.
When the table cache fills up, the server uses the following procedure
to locate a cache entry to use:
Tables that are not currently in use are released, beginning with the
table least recently used.
If a new table needs to be opened, but the cache is full and no tables
can be released, the cache is temporarily extended as necessary. When
the cache is in a temporarily extended state and a table goes from a
used to unused state, the table is closed and released from the cache.
Related
in mysql there is feature to cache large chunk of database in Memory. Using the mysql configuration file. (my.ini)
Database in RAM InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and row data. The bigger you set this the less disk I/O is needed to access data in tables. On a dedicated database server you may set this parameter up to 80% of the machine physical memory size. Do not set it too large, though, because competition of the physical memory may cause paging in the operating system. Note that on 32bit systems you might be limited to 2-3.5G of user level memory per process, so do not set it too high.
innodb_buffer_pool_size=6000M
Query caching Size in my.ini
Query cache is used to cache SELECT results and later return them without actual executing the same query once again. Having the query cache enabled may result in significant speed improvements, if your have a lot of identical queries and rarely changing tables. See the "Qcache_lowmem_prunes" status variable to check if the current value is high enough for your load.
query_cache_type = 1
query_cache_size = 80M
They drastically boost db performance for a medium scale database.
Do we have similar features in SQL Server?
I have a 16GB dedicated Mysql server database.My innodb_buffer_pool_size is set to around 11GB ,i am implementing query cache in my system ,which has a size of 80mb. From where should i make this space ,innodb_buffer_pool_size or RAM ?
Back in Jun 2014 I answered https://dba.stackexchange.com/questions/66774/why-query-cache-type-is-disabled-by-default-start-from-mysql-5-6/66796#66796
In that post, I discussed how InnoDB micromanages changes between the InnoDB Buffer Pool and the Query Cache.
NOT USING THE QUERY CACHE
The simplest answer would be to just disable the query cache, but there is a catch: You must set both query_cache_size and query_cache_type to zero(0).
If you don't set query_cache_type to zero(0), the micromanagement of changes still occurs. This is verified by what the Paragraph 8 of the MySQL Documentation "Query Cache Configuration" says
If query_cache_size is 0, you should also set query_cache_type variable to 0. In this case, the server does not acquire the query cache mutex at all, which means that the query cache cannot be enabled at runtime and there is reduced overhead in query execution.
USING THE QUERY CACHE
If you really wish to use the Query Cache, then you need to study your data. Basically, you need to know the size of your results. You would then set query_cache_limit and query_cache_min_res_unit to accommodate the average size result set as well as the maximum size result set.
As an alternative, you can enable the Query Cache globally but disable its use from your DB Session as stated in Paragraph 9:
Setting the GLOBAL query_cache_type value determines query cache behavior for all clients that connect after the change is made. Individual clients can control cache behavior for their own connection by setting the SESSION query_cache_type value. For example, a client can disable use of the query cache for its own queries like this:
mysql> SET SESSION query_cache_type = OFF;
YOUR ACTUAL QUESTION
The Query Cache, InnoDB Buffer Pool, Per Connection Threads (See my post https://dba.stackexchange.com/questions/16969/how-costly-is-opening-and-closing-of-a-db-connection/16973#16973), and OS all compete for RAM.
When looking at just the Query Cache and the InnoDB Buffer Pool, you need to see how much space is free in the InnoDB Buffer Pool after MySQL has been running for days or weeks.
You can run this the query
SELECT variable_value / 64 free_mb
FROM information_schema.global_status
WHERE variable_name='Innodb_buffer_pool_pages_free';
This will tell you how much you can lower the InnoDB Buffer Pool. You could then increase the Query Cache by that size. If this query returns less that 1024, it is probably not worth the effort. You will either have get the additional space from RAM or just go with disabling the Query Cache.
NO. The Query cache runs out of steam after perhaps 80MB -- it spends more an more time pruning entries.
I would suggest that over 90% of production systems do (or should) turn off the QC completely.
Keep in mind that every write to a table purges all entries in the QC for that table.
11G on a 16GB dedicated server is a good number for innodb_buffer_pool_size.
I have a 16GB dedicated Mysql server database.My innodb_buffer_pool_size is set to around 11GB ,i am implementing query cache in my system ,which has a size of 80mb. From where should i make this space ,innodb_buffer_pool_size or RAM ?
Back in Jun 2014 I answered https://dba.stackexchange.com/questions/66774/why-query-cache-type-is-disabled-by-default-start-from-mysql-5-6/66796#66796
In that post, I discussed how InnoDB micromanages changes between the InnoDB Buffer Pool and the Query Cache.
NOT USING THE QUERY CACHE
The simplest answer would be to just disable the query cache, but there is a catch: You must set both query_cache_size and query_cache_type to zero(0).
If you don't set query_cache_type to zero(0), the micromanagement of changes still occurs. This is verified by what the Paragraph 8 of the MySQL Documentation "Query Cache Configuration" says
If query_cache_size is 0, you should also set query_cache_type variable to 0. In this case, the server does not acquire the query cache mutex at all, which means that the query cache cannot be enabled at runtime and there is reduced overhead in query execution.
USING THE QUERY CACHE
If you really wish to use the Query Cache, then you need to study your data. Basically, you need to know the size of your results. You would then set query_cache_limit and query_cache_min_res_unit to accommodate the average size result set as well as the maximum size result set.
As an alternative, you can enable the Query Cache globally but disable its use from your DB Session as stated in Paragraph 9:
Setting the GLOBAL query_cache_type value determines query cache behavior for all clients that connect after the change is made. Individual clients can control cache behavior for their own connection by setting the SESSION query_cache_type value. For example, a client can disable use of the query cache for its own queries like this:
mysql> SET SESSION query_cache_type = OFF;
YOUR ACTUAL QUESTION
The Query Cache, InnoDB Buffer Pool, Per Connection Threads (See my post https://dba.stackexchange.com/questions/16969/how-costly-is-opening-and-closing-of-a-db-connection/16973#16973), and OS all compete for RAM.
When looking at just the Query Cache and the InnoDB Buffer Pool, you need to see how much space is free in the InnoDB Buffer Pool after MySQL has been running for days or weeks.
You can run this the query
SELECT variable_value / 64 free_mb
FROM information_schema.global_status
WHERE variable_name='Innodb_buffer_pool_pages_free';
This will tell you how much you can lower the InnoDB Buffer Pool. You could then increase the Query Cache by that size. If this query returns less that 1024, it is probably not worth the effort. You will either have get the additional space from RAM or just go with disabling the Query Cache.
NO. The Query cache runs out of steam after perhaps 80MB -- it spends more an more time pruning entries.
I would suggest that over 90% of production systems do (or should) turn off the QC completely.
Keep in mind that every write to a table purges all entries in the QC for that table.
11G on a 16GB dedicated server is a good number for innodb_buffer_pool_size.
In the below status i have opened files count to be '95349'.
this value is increasing rapidly.
mysql> show global status like 'open_%';
Open_files = 721
Open_streams = 0
Open_table_definitions = 706
Open_tables = 741
Opened_files = 95349
Opened_table_definitions = 701
Opened_tables = 2851
also see this.
mysql>show variables like '%open%';
have_openssl = DISABLED
innodb_open_files = 300
open_files_limit = 8502
table_open_cache = 4096
and
max_connection = 300
is there any relation to open files and opened files. will there be any performance issues because of increasing opened_files value. This is a server of 8 GD RAM and 500 GB hardisk with processor: Intel(R) Xeon(R) CPU E3-1220 V2 # 3.10GHz. It is a dedicated mysql server.
here for the command
ulimit -n;
1024 was the count
the server is hanging often. using some online tools i have optimised some parameters already. need to know what else should be optimized ? in what case the opened files count will reduce? is it necessary that opened files count should be with in some limit. if so how to find the appropriate limit for my server. if am not clear some where please help me by asking more questions.
Opened_files is a counter of how many times you have opened a table since the last time you restarted mysqld (see status variable Uptime for the number of seconds since last restart).
Open_files is not a counter; it's the current number of open files.
If your Opened_files counter is increasing rapidly, you may be able to gain improvement to performance by increasing the size of the table_open_cache.
For some tips on the performance implications of this variable (and some cautions about setting it too high), see:
http://www.mysqlperformanceblog.com/2009/11/16/table_cache-negative-scalability/ (the problem described there seems to be solved finally in MySQL 5.6)
Re your comments:
You misunderstand the purpose of the counter. It always increases. It counts the number of times a particular operation has occurred since the last restart of mysqld. In this case, opening a file for a table.
Having a high value in a counter isn't necessarily a problem. It could mean simply that your mysqld has been running for many days or weeks without a restart. So you have to look at that number compared to your Uptime (that is, MySQL status variable Uptime, not Linux uptime).
What is more meaningful is the rate of increase of a counter, that is how fast does it grow in a given interval of time. That could indicate that you are re-opening tables rapidly.
Normally, MySQL shouldn't have to re-open tables, because it retains an open table handle for each table. But it can only have a finite number of those. That's what table_open_cache is for. In your case, your MySQL instance can "remember" that it has already opened up to 4096 tables at a time. If you need another table opened, it closes one of the file descriptors and opens the table you requested.
So if you have many thousands of tables (or partitions of tables) and you access a wide variety of them rapidly, you could see a lot of turnover in that table open cache. That would be indicated by the counter Opened_tables increasing rapidly.
Therefore sizing the table_open_cache higher means that MySQL can retain more open table handles, and possibly decrease the rate of turnover.
SO the solution is either to increase my hardware (especially RAM) so that i will be able to increase the table_open_cache beyond 4096 or to optimize the query.
I was wondering if there's a way to decrease the opened files in mysql.
Details :
mysql 5.0.92
engine used : MyISAM
SHOW GLOBAL STATUS LIKE 'Opened_tables' : 150K
SHOW VARIABLES LIKE '%open%' :
open_files_limit 200000
table_open_cache 40000
Solutions tried :
restart server : it works the opened tables counter is 0 but this isn't a good solution from my pov since you will need a restart every week because the counter will increase fast
FLUSH TABLES : like the mysql doc said it should force all tables in use to close but this doesn't happen
So any thoughts on this matter?
Generally, many open tables are nothing to worry about. If you come close to OS limits, you can increase this limits in the kernel settings:
How do I change the number of open files limit in Linux?
MySQL opens tables for each session independently to have better concurrency.
The table_open_cache and max_connections system variables affect the maximum number of files the server keeps open. If you increase one or both of these values, you may run up against a limit imposed by your operating system on the per-process number of open file descriptors. Many operating systems permit you to increase the open-files limit, although the method varies widely from system to system.
In detail, this is explained here
http://dev.mysql.com/doc/refman/5.5/en/table-cache.html
EDIT
To verify your assumption you could decrease max_connections and table_open_cache temporarily by SET GLOBAL table_open_cache := newValue.
The value can be adjusted dynamically without a server restart.
Prior MySQL 5.1 this variable is called table_cache
What I was trying to tell, is, that decreasing this value will probably even have a negative impact on performance in terms of less possible concurrent reads (queue get's longer), instead you should try to increase the OS limit and increase max_open_files, but maybe I just don't see the point here