I'm trying to optimise my MySQL database.
I've got around 90 tables most of which are hardly ever used.
Only 10 or so do the vast bulk of the work running my website.
MySQL status statistics show approx 2M queries over 2.5 days and reports "Opened_tables" of 1.7k (with Open_tables 256). I have the table_cache set at 256, increased from 32.
I presume most of the opened tables are either multiple instances of the same tables from different connections or some temporary tables.
In the same period it reports "Created_tmp_tables" of 19.1 k and more annoyingly Created_tmp_disk_tables of 5.7k. I have max_heap_table_size and tmp_table_size both set at 128M.
I've tried to optimise my indexes & joins as best i can, and i've tried to avoid BLOB and TEXT fields in the tables to avoid disk usage.
Is there anything you can suggest to improve things?
First of all, don't conclude your MySQL database is performing poorly based on these internal statistics. There's nothing wrong with tmp tables. In fact, queries involving ordering or summaries require their creation.
It's like trying to repair your vehicle after analyzing the amount of time it spent in second gear. Substantially less than 1% of your queries are generating tmp tables. That is good. That number is low enough that these queries might be for backups or some kind of maintenance operation, rather than production.
If you are having performance problems, you will know that because certain queries are working too slowly, and certain pages on your web app are slow. Can you figure out which queries have problems? There's a slow query log that might help you.
http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html
You might try increasing tmp_table_size if you have plenty of RAM. Why not take it up to a couple of megabytes and see if things get better? But, they probably won't change noticeably.
Related
We are having troubles tuning MySQL 5.6 on AWS, and are looking for pointers to solve some performance issues.
We used to have dedicated servers and we could configure them how we like.
Our application uses a lot of temporary tables and we had no performance issue on that regard.
Until we switched to AWS RDS instance.
Now, many slow queries show up in the logs and it slows down the whole application.
Previously we worked with MySQL 5.4 and now it's 5.6.
Looking through the docs, we discovered some changes regarding the temporary tables default format.
It is InnoDB by default, and we set it back to MYISAM like we are used to and constated improvements on that regard.
first aspect:
Also, our DB is quite large and we have hundreds of simultaneous access to our application, and some tables require real-time computation. Join and Unions are used in that cases.
When developing the application (with MySQL 5.4), we found that splitting the larger queries into 2 or more steps and using intermediate tables, the over-whole performance improved.
Explain showed file sort and temporary file so we could get rid of those with temporary tables.
Is splitting queries into temporary tables really a good idea?
Edit We know conditions about the implicit conversion of MEMORY temporary tables to disk.
Another thing we are not sure is what makes query using temporary files?
We know using correct indexes is one way to go (correct order usage in where clause, correct usage of order by etc…), but is there anything else we could do?
second aspect:
Regarding some settings we are using, we have some hundred Mb for max_heap_table_size and tmp_table_size so we hoped that our temporary tables could hold in memory.
We also found articles describing to look at ReadIOPS and WriteIOPS.
The reads are stable and low, but the writes are showing unstable and high numbers.
Here is a graph:
The values on the vertical axis are operations/sec.
How can we interpret those numbers?
One thing to know about our application, is that every user action is logged into one big logs table. But it should be once per page load.
third aspect:
How far can we go with those settings so temporary tables can be used in Memory?
For instance, we read some articles explaining they set few Gb of max_heap_table_size on a dedicated MySQL server with about 12Gb of ram. (sounded like 80% or so)
Is that really something we can try? (Are same settings applicable on RDS?)
Also, can we set innodb_buffer_pool_size at the same value as well?
Note I can't find there article where I found that info, but I might have confused some parameter names. I'll edit the question if I find the source again.
The server settings are very different from what we used to have (the new servers on AWS are not set by us) and many settings values have been increased, and some decreased.
We fear it's not a good thing…
Here are the some noticable changes:
innodb_buffer_pool_size (increased *6)
innodb_log_buffer_size (decreased /4)
innodb_additional_mem_pool_size (decreased /4)
sort_buffer_size (decreased /2)
myisam_sort_buffer_size (increased *1279) (we have only innoDB tables, do we need to touch that?)
read_buffer_size (increased *2)
join_buffer_size (decreased /4)
read_rnd_buffer_size (increased *4)
tmp_table_size + max_heap_table_size (increased *8)
Some changes look weird to us (like myisam_sort_buffer_size), we think that using the same settings would have been better in the first place.
Can we have some pointers on those variables? (Sorry, we can't provide the exact numbers)
Also, is there a good article we could read that sums up a good balance between all those parameters?
Because we are concerned about the temporary tables not fitting in memory, I made that query to see what percentage of queries are actually written to disk:
select
tmp_tables
,tmp_tables_disk
,round( (tmp_tables_disk / tmp_tables) * 100, 2 ) as to_disk_percent
from
(select variable_value as tmp_tables from information_schema.GLOBAL_STATUS where variable_name = 'Created_tmp_tables') as s1
,(select variable_value as tmp_tables_disk from information_schema.GLOBAL_STATUS where variable_name =
'Created_tmp_disk_tables') as s2
The result is 10~12% (depending on the instance). Is that high?
TL;DR
I tried to add as many details to the question as possible about the current situation, I hope it's not more confusing that anything...
The issues are about writes.
Is there a way to diagnose what causes so many writes? (maybe linked to temporary tables?)
MySQL 5.4 never existed. Perhaps MariaDB 5.4?
Temp tables, even if seemingly not hurting performance, are a clue that things could be improved.
Do you have TEXT columns that you don't need to fetch. (This can force an in-memory temp table to turn into on-disk.)
Do you have TEXT columns that could be a smaller VARCHAR? (Same reason.)
Do you understand that INDEX(a,b) may be better than INDEX(a), INDEX(b)? ("composite" index)
The buffer_pool is very important for performance. Did the value of innodb_buffer_pool_size change? How much data do/did you have? How much RAM do/did you have.
In general, "you cannot tune your way out of a performance problem".
Is this a Data Warehouse application? Building and maintaining Summary tables is often a huge performance boost for DW "reports".
MyISAM only has table locking. This leads to blocking, which leads to all sorts of performance problems, especially slowing down queries. I'm surprised that InnoDB was slower for you. MyISAM is a dead end. By the time you get to 8.0, there will be even stronger reasons to switch.
Let's see one of the slow queries that benefited from splitting up. There are several techniques for making complex queries run faster (at least in InnoDB).
The order of ANDed things in WHERE does not matter. It does matter in composite indexes.
Changing tmp_table_size rarely helps or hurts. It should, however, be kept under 1% of RAM -- this is to avoid running out of RAM. Swapping is the worst thing to happen.
Graphs - what are the Y-axis units? Where did the graphs come from? They look useless.
"Gb of max_heap_table_size on a dedicated MySQL server with about 12Gb of ram. (sounded like 80% or so)" -- Check that article again. I recommend 120M for that setting on that machine.
When exclusively using MyISAM on a 12GB machine, key_buffer_size should be about 2400M and innodb_buffer_pool_size should be 0.
When exclusively using InnoDB on a 12GB machine, key_buffer_size should be about 20M and innodb_buffer_pool_size should be 8G. Did you change those when you tested InnoDB?
innodb_additional_mem_pool_size -- Unused, and eventually removed.
myisam_sort_buffer_size -- 250M on a 12G machine. If using only InnoDB, the setting can be ignored since that buffer won't be used.
As for the *6 and /4 -- I need to see the actual sizes to judge anything.
"sums up a good balance between all those parameters" -- Two things:
http://mysql.rjweb.org/doc.php/memory
Leave the rest at their defaults.
For more tuning advice, plus Slowlog advice, see http://mysql.rjweb.org/doc.php/mysql_analysis
I have a single processor dedicated server with 4GB RAM and a 400MB Mysql database (Myisam) who has big performance problems.
The database is used by an ecommerce.
I already tryied to tune it using the mysqltuner script, but without good results.
Because the variable settings have been modified several times, I would like to have a basic configuration to start from, thereafter try to tune it.
Try this tool, it always show good results for performance tuning.
https://tools.percona.com/wizard
For ecommerce, you need InnoDB. If you don't change, you will be burned badly when a crash occurs at just the wrong instant in a monetary transaction.
Make that change, then
key_buffer_size = 20M
innodb_buffer_pool_size = 1000M
read my blog on moving from MyISAM to InnoDB.
When you then find that things are not working fast enough, do
long_query_time = 1
turn on the slowlog
wait a day
run pt-query-digest to find the worst couple of queries
present them to us for critique. The solution could be as simple as adding an composite index. Or maybe reformulating a SELECT.
I have redirected you toward slow queries because you cannot "tune" your way out of bad schema, bad queries, etc.
After running mysqltuner on my Magento db it says me
[!!] Total fragmented tables: 1203
In suggestion it says
Run OPTIMIZE TABLE to defragment tables for better performance
I just made a db backup and I was wondering which is the best way to defragment Magneto db.
I saw on some forums that i can't use OPTIMIZE option because is an InnoDB.
My question is:
1) Which is the best command to optimize Magento InnoDB ?
I can't do it one by one if they are 1203 tables.
2)Can I do optimization on all database?
Bogus! That tool always complains about fragmented tables. Virtually all tables in the universe are "fragmented" to some level. Virtually no MySQL system suffers from fragmentation.
I have studied literally thousands of slow queries. Only two could be traced to fragmentation. The rest were solved by indexes, reformulation of queries, redesign of the schema, etc. Not defragmentation.
Most of the other output from that tool is reasonably good. Did anything else warrant a [!!]?
Is the system running slowly?
The best tool for diagnosing slow MySQL is the slowlog. Turn it on, set long_query_time to 2 (or less), wait a day, run pt-query-digest, show us the top 1 or 2 slow queries. With luck, we can give you a fix that will significantly speed up the system.
And if you want a more thorough analysis of the tunables, provide me with SHOW GLOBAL STATUS (after being up at least a day), SHOW VARIABLES and how much RAM you have.
All of what I find on the web for advice on tuning MySQL for performance deals with production databases that have a high number of connections and many repeated queries. This is not my workload, instead, I'm doing data investigation with MySQL where I am the only user, the data doesn't change very often (bulk imports only), and the number of connections I might have at any given time is < 20. The data I have is largish (several hundred gigs, tables with 50M rows with a bunch of strings in them), but the queries I write are rarely run more than a few times each.
I have the O'Reilly Schwartz et al. book on MySQL and it has been a godsend for understanding how to make some things (like indices) work to my advantage. Yet I feel much less comfortable with the server parameters for this kind of workload, as I can find few examples on the web. Here are the non-stock (MySQL 5.5, Ubuntu) parameters I am running with:
max_heap_table_size=32G
tmp_table_size=32G
join_buffer_size=6G
innodb_buffer_pool_size=10G
innodb_buffer_pool_instances=2
sort_buffer_size=100M
My server is a multi-core (quad, seems wasted on MySQL but sometimes I'll hit up a couple of queries at once) 32GB of RAM machine. Right now it looks like MySQL is limiting itself to 12GB of ram, likely because of the innodb_buffer_pool size. I set tmp_table_size and heap size to be just fantastical because I had been doing some queries where I stored a lot in memory.
Are there any good resources to tune MySQL to this kind of workload? Are there suggestions on what parameters I should set for innodb?
I don't think you have to tune your InnoDB engine performance any more. The real performance gain will be in the way you structure tables, and the queries you write. Be sure that the columns you select on are indexed, sensible primary keys are chosen, etc. Tables with 50M rows shouldn't be a problem as long as you have a good primary key.
If you haven't run into any performance bottlenecks yet, then I think there is no reason to worry.
I've a database with 3 of the tables having rows in excess of 20 million each. I've used GUIDs as primary keys (unfortunately). Now our database is about 20GB and growing 5GB per month.
It takes about 2 hrs to take full backup of the database, and 30hrs to restore on a box with 4GB RAM.
We once have all the tables from database disappeared. other mysql databases in same server were alright except one - for which only data was disappeared leaving empty tables.
A select query (among many slow queries) - which get max of a date column in one of 20m table takes about 5 mins to return result. This query used pretty frequently.
What I'm looking answers for
recommended db design changes
ways to improved select query performance - max date column on 20m records
other queries' performance
how to go about handling future db growth
Thanks all for your attention.
I've seen setups of larger size (with InnoDB as storage engine and a GUID as a primary key), and there were no such problems.
We once have all the tables from database disappeared. other mysql databases in same server were alright except one - for which only data was disappeared leaving empty tables.
The tables may seem empty if the system LSN has gone below the each page's LSN. This may happen if the InnoDB logfiles are corrupt. InnoDB, however, will issue a warning in this case.
A select query (among many slow queries) - which get max of a date column in one of 20m table takes about 5 mins to return result. This query used pretty frequently.
Create an index on this column, the query will be instant.
Please post the exact query and I'll tell you how to create the best index.
I see no problem in the DB design as such, most probably it's something with your server.
Is it possible to reproduce this behavior on another server with a clean vanilla MySQL installation?
You may also want to try to split data between the tables. Set innodb_file_per_table and restore from the backup.
A free alternaive to innodb hot backup is Percona XtraBackup Tool.
For backup, you could use the innodb hot backup tool. This not only lets you do consistent backups while your database is up, but the restore is much faster than the one you're doing (I'm assuming mysqldump?). It does cost money.
You might also try Mydumper: http://www.mydumper.org/
It is a great tool and is free and open source