I have 81 tables in an innodb database (MySQL).
The data in them amounts to 2GB on disk.
My queries rarely join more than 3 tables together at once. My innodb_buffer_pool size is about 2.1 GB.
Running mysqltuner.pl I get the following !!
[!!] Table cache hit rate: 7% (274 open / 3K opened)
From mysqlreport I see that I indeed have 274 open, have had 3K opened and that my ceiling for open is 400.
However, doing this
show status like '%open%'
gets this result
...
Open_table_definitions 161
Open_tables 274
Opened_files 150232
Opened_table_definitions 0
Opened_tables 0
Two questions:
1) Shouldn't the "opened tables" say 3K and not zero in the above result from show status like '%open%'?
2) Any advice on what I need to do to remedy this !! i.e. the low table cache hit rate?
Thanks
PS. If it helps, the second !! I have in mysqltuner.pl is this:
[!!] Temporary tables created on disk: 29% (35K on disk / 119K total)
show status like '%open%' shows the status for the current session rather then for the whole MySQL DB(SHOW STATUS Syntax). For getting global status, use show global status like '%open%' instead.
One problem I encountered for mysqltuner.pl is whenever it is run, it opens all the tables in the database, thus increasing the opened_tables statistics. If it is not the case, MySQL manual suggested set table_open_cache to * .
1) Shouldn't the "opened tables" say 3K and not zero in the above
result from show status like '%open%'?
YES, result came mostly from SHOW STATUS and SHOW VARIABLES and some basic calculus operations.
2) Any advice on what I need to do to remedy this !! i.e. the low
table cache hit rate?
Table cache hit is due to:
1. too few table opened
2. Total number of table in all databases <<< open table cache size.
Related
We are facing a problem, our DB instance MySQL 8.0 (Production environment) is continuously showing an alert that number of open tables is equal to table_open_cache value. The number of open tables is increased more than 43,200 in 24 hour observation period which makes total count of open tables equals to 2845063.
Please help me how to reduce this, If I go for Flush tables command with read only or with read lock will it cause any data loss or performance issues. I have to implement this to my production Database, Is it a good practice to use Flush tables manually once a day.
Posted a question regarding MySQL DB instance open tables, need to know how to reduce the same by any method. Is it a good practice to use Flush tables manually once a day.
I am attaching an image for reference :-
image1
Misses/Hits is about 2% -- reasonable.
Apparently that screenshot should be talking about "opened" tables, not "open" tables. Only 4K are currently "open", limited by table_open_cache.
The image shows 43.2K vs 2.8M -- it is unclear what each means. 43.2K/24h is exactly 1 per 2 seconds. This is suspect.
2.8M openings of tables in 24 hours is high, but not necessarily "bad. (It's about the 95th percentile.)
Suggest increasing table_open_cache to 8000. What activity is going on? Perhaps you are opening a connection, performing a single operation (which involves opening one or more tables), then disconnecting? Can you cut back on the rapidity of creating connections?
Please provide SHOW GLOBAL STATUS LIKE 'Connection'; 50 per second is "high".
I await seeing Opened_tables and Uptime fetched at the 'same' time.
No, I don't think FLUSH is the answer.
UPDATE 11.15.2022
Per the request of Wilson Hauck,I have added text files showing the results of the following queries:
SELECT COUNT(*), sum(data_length), sum(index_length), sum(data_free)
FROM information_schema.tables; (https://pastebin.com/uHG0MynW)
SHOW STATUS; (https://pastebin.com/kJd3Pq6J)
SHOW GLOBAL VARIABLES; (https://pastebin.com/51M84TBR)
SHOW FULL PROCESSLIST; (https://pastebin.com/p7YsSDA2)
SHOW ENGINE INNODB STATUS;
CREATE for table person_history_work (https://pastebin.com/HQSrBcXH)
SHOW TABLE STATUS WHERE name LIKE "person_history_work"; (https://pastebin.com/nyA5uQUw)
NOTE SHOW ENGINE INNODB STATUS is blank. Fields are empty.
This is on a Win 10 x64 system M.2 SSD with 500GB free and 5.3GHz CPU (32GB RAM).
I was going to add a column to an existing table (200MB in size with 300,000 rows) and the MySQL IDE hung. SHOW PROCESSLIST shows it stuck with the state of "copy to tmp table" on:
ALTER TABLE tablename ADD COLUMN columnname TEXT DEFAULT NULL
Looking in the MySQL data directory, these 4 files are constantly shuffling and being updated (as date modified) but never change in size:
I've restarted MySQL, and even updated to 8.0.31. This is an INNODB table I've never had an issue with before.
I did change my ##tmp_table_size to 700MB (it was 70MB). No effect.
I also changed ##buffer_pool_size to 1GB. No effect.
Here's the benchmarks of my SSD:
So right now I'm stuck, unable to ALTER the table. I can query, select, and look at the data no problem.
Only way to stop this is to KILL the process or restart the MySQL service. I've waited more than 20 minutes to no avail. Given the fact the file sizes don't change at all, it makes me think MySQL is stuck/hung. Since it's InnoDB I can't really repair - STATUS shows the table as fine.
Any ideas?
Observations may help you avoid stall/hanging,
Your SHOW CREATE TABLE person_history_work includes a line
AVG_ROW_LENGTH = 81,
consider removing this line because SHOW TABLE STATUS indicates ARL is 727.
In your my.ini consider
max_heap_table_size=734003200 # from 16M to match tmp_table_size - best practice
innodb_io_capacity=500 # from 200 to use more of SSD IOPS capacity
innodb_flush_neighbors=2 # to clear EXTENT activity reducing innodb_buffer_pool_pages_dirty ASAP
innodb_max_dirty_pages_pct_lwm=0.00001 # from 10% to reduce IBP-pages-dirty of 17,348
innodb_max_dirty_pages_pct=0.00001 # from 90% to reduce dirty-pages overhead faster.
max_allowed_packet=256M # from 4M to reduce # packets handled
Please view profile for contact info. Additional global variables need to be adjusted.
In the below status i have opened files count to be '95349'.
this value is increasing rapidly.
mysql> show global status like 'open_%';
Open_files = 721
Open_streams = 0
Open_table_definitions = 706
Open_tables = 741
Opened_files = 95349
Opened_table_definitions = 701
Opened_tables = 2851
also see this.
mysql>show variables like '%open%';
have_openssl = DISABLED
innodb_open_files = 300
open_files_limit = 8502
table_open_cache = 4096
and
max_connection = 300
is there any relation to open files and opened files. will there be any performance issues because of increasing opened_files value. This is a server of 8 GD RAM and 500 GB hardisk with processor: Intel(R) Xeon(R) CPU E3-1220 V2 # 3.10GHz. It is a dedicated mysql server.
here for the command
ulimit -n;
1024 was the count
the server is hanging often. using some online tools i have optimised some parameters already. need to know what else should be optimized ? in what case the opened files count will reduce? is it necessary that opened files count should be with in some limit. if so how to find the appropriate limit for my server. if am not clear some where please help me by asking more questions.
Opened_files is a counter of how many times you have opened a table since the last time you restarted mysqld (see status variable Uptime for the number of seconds since last restart).
Open_files is not a counter; it's the current number of open files.
If your Opened_files counter is increasing rapidly, you may be able to gain improvement to performance by increasing the size of the table_open_cache.
For some tips on the performance implications of this variable (and some cautions about setting it too high), see:
http://www.mysqlperformanceblog.com/2009/11/16/table_cache-negative-scalability/ (the problem described there seems to be solved finally in MySQL 5.6)
Re your comments:
You misunderstand the purpose of the counter. It always increases. It counts the number of times a particular operation has occurred since the last restart of mysqld. In this case, opening a file for a table.
Having a high value in a counter isn't necessarily a problem. It could mean simply that your mysqld has been running for many days or weeks without a restart. So you have to look at that number compared to your Uptime (that is, MySQL status variable Uptime, not Linux uptime).
What is more meaningful is the rate of increase of a counter, that is how fast does it grow in a given interval of time. That could indicate that you are re-opening tables rapidly.
Normally, MySQL shouldn't have to re-open tables, because it retains an open table handle for each table. But it can only have a finite number of those. That's what table_open_cache is for. In your case, your MySQL instance can "remember" that it has already opened up to 4096 tables at a time. If you need another table opened, it closes one of the file descriptors and opens the table you requested.
So if you have many thousands of tables (or partitions of tables) and you access a wide variety of them rapidly, you could see a lot of turnover in that table open cache. That would be indicated by the counter Opened_tables increasing rapidly.
Therefore sizing the table_open_cache higher means that MySQL can retain more open table handles, and possibly decrease the rate of turnover.
SO the solution is either to increase my hardware (especially RAM) so that i will be able to increase the table_open_cache beyond 4096 or to optimize the query.
I'm trying to tune my Magento DB for optimal performance.
I'm running nginx, php-fpm and mysql on a 4GB RAM, 8CPU core virtual machine with 4GB of RAM.
I've ran the Mysql Tuning Primer and everything looks good apart from my Table Cache:
TABLE CACHE
Current table_open_cache = 1000 tables
Current table_definition_cache = 400 tables
You have a total of 2510 tables
You have 1000 open tables.
Current table_cache hit rate is 3%
, while 100% of your table cache is in use
You should probably increase your table_cache
You should probably increase your table_definition_cache value.
and from mysqltuner
[!!] Table cache hit rate: 9% (1K open / 10K opened)
[!!] Query cache efficiency: 0.0% (0 cached / 209 selects)
The relevant settings from the my.cnf file:
table_cache = 1000
query_cache_limit = 1M
query_cache_size = 64M
The thing is, no matter what I increase my table_cache to - it seems to be consumed almost immediately. Is this normal for Magento? It seems abnormally high?
Does anyone have any tips about what I can do to improve this?
Thanks,
Ed
Check your MySQL config's query cache type setting:
http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html#sysvar_query_cache_type
If you set it to 0 or 2 then it will either not cache any queries or only cache the ones that you have specifically asked to cache. That means Magento would have to explicitly ask for cached query results (I'm not sure it does that). If you set it to 1 then it will cache all queries except those that explicitly ask for no query cache.
Table cache refers to potential open file pointers. It could be consumed rather quickly, and will just roll off unused entries as needed. From MySQL's documentation:
The table_cache and max_connections system variables affect the
maximum number of files the server keeps open. If you increase one or
both of these values, you may run up against a limit imposed by your
operating system on the per-process number of open file descriptors.
Many operating systems permit you to increase the open-files limit,
although the method varies widely from system to system. Consult your
operating system documentation to determine whether it is possible to
increase the limit and how to do so.
table_cache is related to max_connections. For example, for 200
concurrent running connections, you should have a table cache size of
at least 200 * N, where N is the maximum number of tables per join in
any of the queries which you execute. You must also reserve some extra
file descriptors for temporary tables and files.
Make sure that your operating system can handle the number of open
file descriptors implied by the table_cache setting. If table_cache is
set too high, MySQL may run out of file descriptors and refuse
connections, fail to perform queries, and be very unreliable. You also
have to take into account that the MyISAM storage engine needs two
file descriptors for each unique open table. You can increase the
number of file descriptors available to MySQL using the
--open-files-limit startup option to mysqld. See Section C.5.2.18, “'File' Not Found and Similar Errors”.
The cache of open tables is kept at a level of table_cache entries.
The default value is 64; this can be changed with the --table_cache
option to mysqld. Note that MySQL may temporarily open more tables
than this to execute queries.
MySQL closes an unused table and removes it from the table cache under
the following circumstances:
When the cache is full and a thread tries to open a table that is not
in the cache.
When the cache contains more than table_cache entries and a table in
the cache is no longer being used by any threads.
When a table flushing operation occurs. This happens when someone
issues a FLUSH TABLES statement or executes a mysqladmin flush-tables
or mysqladmin refresh command.
When the table cache fills up, the server uses the following procedure
to locate a cache entry to use:
Tables that are not currently in use are released, beginning with the
table least recently used.
If a new table needs to be opened, but the cache is full and no tables
can be released, the cache is temporarily extended as necessary. When
the cache is in a temporarily extended state and a table goes from a
used to unused state, the table is closed and released from the cache.
Could you, please, explain what is the meaning of the following MySQL metric:
table cache hit rate = open_tables / opened_tables.
As I understand open_tables is the current value of opened tables and opened_tables is a counter and there is no any correlation between these two status variables.
open_tables is the number of tables you have open right now; opened_tables is the total number of table-opening operations since the server started.
For example, if you have performed 100 table opening operations and have 25 tables open now, your table cache hit rate is 25/100 = 1/4.
The rationale is that you are trying to measure whether your table cache is big enough or not, but the ratio of open to opened tables doesn't give you the whole picture. Read "How MySQL Opens and Closes Pages" (http://dev.mysql.com/doc/refman/5.0/en/table-cache.html) to understand this better.
What you want to do is look at the value of opened tables over time - if it is growing rapidly while your system is busy, you might want to increase your table cache size. But be careful about making the table cache too large - it takes time for MySQL to check a large number of cached table descriptors to figure out which one to close next.