hi I want to change my /etc/my.cnf file (mysql's config file).
What should the below values for better performance on my queries.
query_cache_type = 1
query_cache_limit = 1M
query_cache_size = 16M
Is there an optimal ratio for cache_size / RAM ? I have 8GB of ram on my ubuntu machine.
If there were a well-defined optimum there would be no need for a configuration option. MySQL would use that optimum by default. The query cache is also only useful for very specific circumstances (you read a lot more from the table than you write to it) because the cache is emptied on a per-table basis every time you write anything to the table. It also only works if you state the exact same queries, with the same parameters, over and over.
The optimal value for you needs to be measured out and depends a lot on your use case. If you have a lot of InnoDB tables you will get much more use out of the InnoDB buffer pool: innodb_buffer_pool_size. Set this variable as high as possible (and on a MySQL-only, InnoDB-only machine this might mean as much as 80% of your available RAM).
We host hundreds of small websites on our 8GB RAM server, which runs both database and web server on the same machien, with a mixture of MyISAM and InnoDB tables. Here is our configuration for comparison:
innodb_file_per_table=1
open_files_limit=50000
max_allowed_packet=268435456
innodb_buffer_pool_size=1G
innodb_log_file_size=256M
innodb_flush_method=O_DIRECT
innodb_io_capacity=1000
innodb_old_blocks_time=1000
innodb_open_files=5000
key_buffer_size=16M
read_buffer_size=256K
read_rnd_buffer_size=256K
query_cache_size=256M
query_cache_limit=5M
join_buffer_size=4M
sort_buffer_size=4M
max_heap_table_size=64M
tmp_table_size=64M
table_open_cache=4500
table_definition_cache=4000
thread_cache_size=50
If your machine has a lot of writes, turn the Query cache completely off (type=0, size=0). That is because every write to a table causes all entries in the QC for that table to be removed.
As a corollary to that, having too big a QC can be "slow". I recommend no more than 50M for query_cache_size.
I hope that explains why I did not address your title question about percent of RAM.
This depends on the size of the query results. If you have a query cache limit of 5M and query_cache_size of 256M a worst case scenario will let you end up with 55 query results of 5M in your cache.
Depending on the type of queries you run most you are better of setting a smaller query_cache_limit (64k) giving you a total of 4096 smaller query results in the cache. On top of this the results in the cache are smaller and will not lock the query cache longer then needed.
The query cache of MySQL uses a single thread that locks the cache on every request. If to many requests connect to the query cache the overall performance will drop.
Related
we are trying to improve the efficiency of our database server.
One of the recommendations of MySQLTunner is rising the innodb_log_file_size to 12 GB. As we see this change could improve significantly the speed and the performance of our queries. The problem came when we increase this parameter more than 1 GB, the service wont start, we delete the logs, stop it cleanly, and still cant start with this parameter above 1 GB.
Someinfo:
mysql Ver 14.14 Distrib 5.5.62, for debian-linux-gnu
innodb_buffer_pool_size = 100G
innodb_file_per_table = ON
innodb_buffer_pool_instances = 64
innodb_stats_on_metadata = OFF
innodb_log_file_size = 1G
innodb_log_buffer_size = 8M
Theres algo enough space in partitions for keeping this log size
Thanks!
In MySQL 5.5, you can't increase the innodb log file size over 4GB total. The innodb_log_file_size can only be 4GB / innodb_log_files_in_group (which is 2 by default, and there's no benefit to change that). So you can set the log file to a max of 2GB.
See https://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_log_file_size
The max combined size of the log files increased to 512GB in 5.6.3. Again, innodb_log_file_size should be the size of one log file, so if you use multiple log files, the total cannot exceed 512GB.
I agree with Rick James' answer that increasing the log file size is not a magical solution to make queries run faster. It won't do that.
It's sometimes useful to increase the innodb log file size, if the bottleneck is that you run out of log space faster than dirty pages can be flushed to the tablespace, because you have very high write traffic. That's affected by the rate of writes, not the speed of individual writes.
For most apps, two 2GB log files is more than enough. If it isn't, it's probably time to run multiple MySQL instances, and distribute your write traffic over them as evenly as you can.
It's tricky to change the log_file_size in that old version. For the steps, see https://dba.stackexchange.com/questions/1261/how-to-safely-change-mysql-innodb-variable-innodb-log-file-size/4103#4103
However, I predict that changing the log_file_size will not help much. "Performance" usually implies a few slow queries. Do you have the slowlog turned on? with a low value of long_query_time? Find the few worst queries; we can probably improve performance by tackling them. Steps on that: http://mysql.rjweb.org/doc.php/mysql_analysis
If you change the innodb_log_file_size parameter, you need to remove the old log files. Otherwise, Innodb won't start successfully if the existing files do not match the specified size in the config file.
On the other hand, The innodb_buffer_pool_size should be set to about 70% of available RAM if you are running InnoDB only.
I have a 16GB dedicated Mysql server database.My innodb_buffer_pool_size is set to around 11GB ,i am implementing query cache in my system ,which has a size of 80mb. From where should i make this space ,innodb_buffer_pool_size or RAM ?
Back in Jun 2014 I answered https://dba.stackexchange.com/questions/66774/why-query-cache-type-is-disabled-by-default-start-from-mysql-5-6/66796#66796
In that post, I discussed how InnoDB micromanages changes between the InnoDB Buffer Pool and the Query Cache.
NOT USING THE QUERY CACHE
The simplest answer would be to just disable the query cache, but there is a catch: You must set both query_cache_size and query_cache_type to zero(0).
If you don't set query_cache_type to zero(0), the micromanagement of changes still occurs. This is verified by what the Paragraph 8 of the MySQL Documentation "Query Cache Configuration" says
If query_cache_size is 0, you should also set query_cache_type variable to 0. In this case, the server does not acquire the query cache mutex at all, which means that the query cache cannot be enabled at runtime and there is reduced overhead in query execution.
USING THE QUERY CACHE
If you really wish to use the Query Cache, then you need to study your data. Basically, you need to know the size of your results. You would then set query_cache_limit and query_cache_min_res_unit to accommodate the average size result set as well as the maximum size result set.
As an alternative, you can enable the Query Cache globally but disable its use from your DB Session as stated in Paragraph 9:
Setting the GLOBAL query_cache_type value determines query cache behavior for all clients that connect after the change is made. Individual clients can control cache behavior for their own connection by setting the SESSION query_cache_type value. For example, a client can disable use of the query cache for its own queries like this:
mysql> SET SESSION query_cache_type = OFF;
YOUR ACTUAL QUESTION
The Query Cache, InnoDB Buffer Pool, Per Connection Threads (See my post https://dba.stackexchange.com/questions/16969/how-costly-is-opening-and-closing-of-a-db-connection/16973#16973), and OS all compete for RAM.
When looking at just the Query Cache and the InnoDB Buffer Pool, you need to see how much space is free in the InnoDB Buffer Pool after MySQL has been running for days or weeks.
You can run this the query
SELECT variable_value / 64 free_mb
FROM information_schema.global_status
WHERE variable_name='Innodb_buffer_pool_pages_free';
This will tell you how much you can lower the InnoDB Buffer Pool. You could then increase the Query Cache by that size. If this query returns less that 1024, it is probably not worth the effort. You will either have get the additional space from RAM or just go with disabling the Query Cache.
NO. The Query cache runs out of steam after perhaps 80MB -- it spends more an more time pruning entries.
I would suggest that over 90% of production systems do (or should) turn off the QC completely.
Keep in mind that every write to a table purges all entries in the QC for that table.
11G on a 16GB dedicated server is a good number for innodb_buffer_pool_size.
Summary:
I haven't yet been able to get MySQL to use more than 1 core for a select statement and it doesn't get above 10 or 15 GB of RAM.
The machine:
I have a dedicated Database server running MariaDB using MySQL 5.6. The machine is strong with 48 cores and 192GB of RAM.
The data:
I have about 250 million rows in one large table (also several other tables ranging from 5-100 million rows). I have been doing a lot of reading from the tables, sometimes inserting into a new table to denormalize the data a bit. I am not setting this system up as a transactional system, rather, it will be used more similarly to a data warehouse with few connections.
The problem:
When I look at my server's stats, it looks like CPU is at around 70% for one core with a select query running, and memory is at about 5-8%. There is no IO waiting, so I am convinced that I have a problem with MySQL memory allocation. After searching on how to increase the usage of memory in MySQL I have noticed that the config file may be the way to increase memory usage.
The solution I have tried based on my online searching:
I have changed the tables to MyISAM engine and added many indexes. This has helped performance, but querying these tables is still incredibly slow. The write speed using load data infile is very fast, however, running a mildly complex select query takes hours or even days.
I have also tried adjusting the following configurations:
key-buffer-size = 64G
read_buffer_size = 1M
join_buffer_size = 4294967295
read_rnd_buffer_size = 2M
key_cache_age_threshold = 400
key_cache_block_size = 800
myisam_data_pointer_size = 7
preload_buffer_size = 2M
sort_buffer_size = 2M
myisam_sort_buffer_size = 10G
bulk_insert_buffer_size = 2M
myisam_repair_threads = 8
myisam_max_sort_file_size = 30G
max-allowed-packet = 256M
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 500
thread-cache-size = 150
open-files-limit = 65535
table-definition-cache = 1024
table-open-cache = 2048
These config changes have slightly improved the amount of memory being used, but I would like to be able to use 80% of memory or so... or as much as possible to get maximum performance. Any ideas on how to increase the memory allocation to MySQL?
As you have already no IO waiting you are using a good amount of memory. Your buffers also seem quite big. So I would doubt that you can have significant CPU savings with using additional memory. You are limited by the CPU power of a single core.
Two strategies could help:
Use EXPLAIN or query analyzers to find out if you can optimize your queries to save CPU time. Adding missing indexes could help a lot. Sometimes you also might need combined indexes.
Evaluate an alternative storage engine (or even database) that is better suited for analytical queries and can use all of your cores. MariaDB supports InfiniDB but there are also other storage engines and databases available like Infobright, MonetDB.
Use show global variables like "%thread%" and you may get some clues on enabling thread concurrency options.
read_rnd_buffer_size at 2M tested at 16384 with your data may produce significant reduction in time required to complete your query.
I have been trying to create an index on a varchar(20) column with 100K rows, and it's been running for 30 minutes so far. On an 8 core i7 processor with 16GB of memory and an SSD drive, I just don't understand what's taking it so long.
Any ideas? I'm a bit new to MySQL, but this is just a basic vanilla index on a relatively small table. The one other index on the same table took only a few seconds to generate.
How does one debug this sort of thing in MySQL?
What's the total size in memory of the table? If it's big enough that you're getting a lot of hard drive calls, it could still take a while. Also, is your site live while you're doing this?
As far as debugging goes, you could check your SQL process on the system to see how many resources it's using.
Finally, have you looked at creating a multi-column index rather than two single-column indexes?
It turns out that the default Ubuntu server LAMP install of MySQL has incredibly low memory allocated, requiring an enormous amount of disk swapping, even on machines that have obvious excess memory.
Please note that I did not experiment to see which setting(s) solved the issue, but the commands I was running on 100K rows, which previously ran for hours, now only take seconds.
[mysqld]
key_buffer_size = 256M
max_allowed_packet = 16M
# Added
innodb_buffer_pool_size = 2G
innodb_log_buffer_size = 64M
innodb_log_file_size = 64M
skip_name_resolve
query_cache_limit = 16M
query_cache_size = 64M
I have a pretty vanilla mysql 5.1 setup, and I am trying to tune it. I found this handy script
It made the following suggestions:
query_cache_limit (> 1M, or use smaller result sets)
query_cache_size (> 16M)
join_buffer_size (> 128.0K, or always use indexes with joins)
table_cache (> 64)
innodb_buffer_pool_size (>= 14G)
In reading up on what these mean and what they are currently set to, I found that I can run "mysqladmin variables"
My current values are:
query_cache_limit | 1048576
query_cache_size | 16777216
join_buffer_size | 131072
innodb_buffer_pool_size | 8388608
How do I read these, are they Kbytes? so is that 1M, 16M, 13M and 8M?
My box is only 4G of Ram and on a normal day only had a few hundred megs free of memory. Should I follow these suggestions and do:
#innodb_buffer_pool_size = 15G
#table_cache = 128
#join_buffer_size = 32M
#query_cache_size = 64M
#query_cache_limit = 2M
Im confused by the 15G, is this a disk space thing, not a memory thing? If so then the recommendations are not very good right?
Should I get more memory for my box?
More Info:
- My db size is 34Gigs, I use all innodb, I have 71 tables, 4 of them are huge, the rest are small. Ive been thinking of moving the big ones to SOLR and doing all queries from there, but wanted to see what I can do with basic tuning.
thanks
Joel
You should not set your innodb buffer pool higher than your available memory. The script probably recommended that based on the number of records in your table and their physical size. Innodb performance is very much memory based, if it can fit the indexes in memory, performance is going to drop quickly and noticeably. So setting innodb_buffer_pool_size high is almost always good advice.
Innodb is not the best table type for everything when it comes to mysql. Very large tables that generally have a lot of inserts, but few reads and updates (i.e. logging) are better off as MyISAM tables. Your very active tables (inserts, updates, deletes, selects) are better off Innodb. There may be a flame war on this advice, and it is generic advice.
But that said, no script is going to being to tell you what your settings should be. It can only make a best guest. The best settings are based on your data access patterns. You really have to read up on what all the variables are. mysqlperformanceblog.com is an excellent place for learning about mysql, in addition to the manual.
When in mysql, use "show variables" and "show status" to see what's going on. You can also run "show innodb status", but you may not understand that output if you don't know what the variables are.