I want to cache data on MySQL
SET GLOBAL query_cache_size = SOME_SIZE;
Is it all the thing required for caching data [efficiently] in MySQL ?
Do I need to add something extra to use the cache efficiently ?
I don't have good knowledge on data caching but still need to use for performance issue, so if I've missed to give some vital info, answer this question assuming the system is in default state.
I don't usually recommend using the MySQL query cache. It sounds great in theory, but unfortunately isn't a great win for caching efficiently, because access to it from queries is governed by a mutex. That means many concurrent queries queue up to get access to the query cache, and this harms more than it helps if you have a lot of concurrent clients.
It even harms INSERT/UPDATE/DELETE, even though these queries don't have result sets, because they purge query results from the query cache if they update the same table(s). And this purging is subject to the same queueing on the mutex.
A better strategy is to use memcached for scalable caching of specific query results, but this requires you to think about what you want to cache and to write application code to access memcached and fail back to MySQL if the data isn't present in the cache. That's more work, but if you do it right it gives better results.
See TANSTAAFL.
There are quite a few settings used for caching different things within MySQL. This is a good guide to optimizing MySQL:
http://www.fromdual.com/mysql-performance-tuning-key
Be careful, the query cache is very specific in what it does:
The query cache stores the text of a
SELECT statement together with the
corresponding result that was sent to
the client. If an identical statement
is received later, the server
retrieves the results from the query
cache rather than parsing and
executing the statement again.
http://dev.mysql.com/doc/refman/5.6/en/query-cache.html
Therefore, if anything in the related tables change, or the query is even reworded, the cache isn't used. So select * from T where id in (1,2) and select * from T where id in (2,1) are different.
SHOW VARIABLES LIKE '%query_cache%';
Will show you the current settings for the cache. But its not as simple as just turning it on, the queries you run need to have result sets that are cacheable and it would take more than this comments box to explain that.
If you have a particular query that you think should be cached then post it and we may be able to determine if it is cacheable.
Related
I have one MySQL server running on Linux.
It has a few critical apps, and we have set the querycache as high as we could afford.
I also have a few non critical databases, (wordpress / etc).
Questions:
Is it possible to fine tune query cache on a per database level?
Is it possible to fine tune query cache on a per table basis?
Is it even worth doing? Having a fine tuned query cache on critical tables, will the db still stutter when accessing non important data?
Thanks in advance
Unfortunately, there are very few options that lets you manipulate the MySQL query cache.
The query_cache_limit option instructs MySQL to not cache query results larger than a set limit. Reasons why you would want to lower this value:
some relatively rare queries return large result sets
most slower queries typically return small result sets
The SQL_NO_CACHE keyword, immediately placed after the SELECT statement, instructs MySQL to not cache this result.
Conversely, you could set the query_cache_type server option to 2 so that only queries using the SQL_CACHE keyword are cached.
I would also advise to make sure your query cache is actually fully used. SHOW STATUS LIKE 'Qcache_free_memory'; gives you this information. If a high proportion of your query cache is free, then it probably means that your data changes too frequently for the results in cache to be reused.
It could also mean that most of your queries return results sets larger than query_cache_limit (which in turn probably suggests badly designed queries).
There are a few other tips here.
However, you are correctly wondering whether this is worth the hassle., In my opinion the query cache is, at best, a secondary factor for a fast database. Appropriate indexing is the first factor. Moreover, in many cases, your memory would be better used for caching indexes (the most important parameters being the innodb_buffer_pool_size for InnoDB tables, or the key_buffer_size for MyISAM tables)
Does mysql server keeps records of queries executed on it, if it does so , is it possible to retrieve those queries.
Thanks.
You can use the General Query Log, which logs all queries in plain text.
We have this enabled on our development environment only. It shouldn't be a problem to turn this on temporarily on a production server during off-peak times.
If you're looking for slow queries, you may as well use the Slow Query Log.
If you want to keep record of all queries that are executed, you can enable either the General Query Log or the Slow Query Log and set the threshold to 0 seconds (any query that takes more than 0 seconds will be logged).
Another option is the Binary log. However, binary log does not keep record of queries like SELECT or SHOW that do not modify data.
Note that these logs get pretty big pretty fast in a server with traffic. And that both might contain passwords so they have to be protected from unauthorized eyes.
You can use MySQL Proxy which stands between client app and RDBMS itself.
http://forge.mysql.com/wiki/MySQL_Proxy
You can see online queries and also it allows you to rewrite queries based on rules.
There's another option - use a profiler.
For instance: http://www.jetprofiler.com/
I have mysql Proxy running which takes a query, performs an md5 on it, and caches the result into a memcached DB. the problem occurs when an update happens in the rails app that would invalidate that cache. Any ideas on how to invalidate all of the proper keys in the cache at that time?
The core of the problem, is you don't know what the key is since it is md5 generated.
However, you can mitigate the problem by not storing data for that query.
You query may look like this "SELECT my_data.* FROM my_data WHERE conditions"
However, you can reduce the redudeancy of data by use this query instead
SELECT my_data.id FROM my_data WHERE conditions
Which is then followed up by
Memcache.mget( ids )
This won't prohibit the return on data that no longer matches the conditions, but may mitigate returning stale data.
--
Another option is to look into using namespaces: See here:
http://code.google.com/p/memcached/wiki/NewProgrammingTricks#Namespacing
You can namespace all of your major queries. You won't be able to delete the keys, but you can change the key version id, which will in effect expire your data.
Logistically messy, but you could use it on a few bad queries.
--
lastly, you could store those queries in a different memcache server and flush on a more frequent basis.
If the same sql run many times from different sessions, will mysql parse the same sql many times? In oracle/sql server, the plan for a sql is cached and can be reused. Since it is told that parse and creating sql plan is costly, if mysql doesn't cache them, will it be a problem to parse it many time which could potentially cost a lot?
For execution plan caching: I don't believe MySQL currently offers this feature.
MySQL does have a query cache: http://dev.mysql.com/doc/refman/5.1/en/query-cache.html
The query cache stores the text of a SELECT statement together with the corresponding result that was sent to the client. If an identical statement is received later, the server retrieves the results from the query cache rather than parsing and executing the statement again. The query cache is shared among sessions, so a result set generated by one client can be sent in response to the same query issued by another client.
I'm not sure how up to date this article is (2006), but it talks about these issues in detail:
http://www.mysqlperformanceblog.com/2006/07/27/mysql-query-cache/
To the best of my knowledge, not much has changed since then in this regard.
This is an existing MySQL Feature Request.
However, the last comments (in 2009) where along the lines that it's not clear it would offer any significant performance improvements and that it could lead to deadlock conditions.
If you are concerned about this, you might want to look into using prepared statements.
I'm running a mySQL query that joins various tables of 500,000+ rows. Sometimes it takes a second, other times around 15 seconds! This is on my local machine. I have experienced similarly varied times before on other intensive queries, does anyone know why this is?
Thanks
Thanks for the replies - I am using appropriate indexes, inner and left joins and have a WHERE clause range of one week out of possible 2 year period of invoices. If I keep varying it (so presumably query results are not cached) and re-running, time varies a lot, even if no. of rows retrieved is similar. The server is not busy. A few scheduled queries every minute but not intensive, take around 200ms.
The explain plan shows that a table of around 2000 rows is always fully scanned. So maybe these rows are sometimes cached, or maybe indexes are cached - didnt know indexes could be cached. I will try again with caching turned off.
Editing again - query cache is in fact off, I'm using InnoDB so looks like increasing innodb_buffer_pool_size is way to go
Same query each time?
It's hard to tell, based on what you've posted. If we assume that the schema and data aren't changing, I'd guess that there's something else running on your machine when the queries are long that would explain the difference. It could be that the state of memory is different, so paging is going on; an anti-virus program is running; some other service has started. It's impossible to answer.
Try to do an
Optimize Table
That should help to refresh some data useful for the query planner.
You have not give us much information, if you're using MyISAM tables, it may be a matter of locks.
Are you using ANSI INNER JOINs? Little basic, but don't use "cross joins". Those are the joins with the comma, like
SELECT * FROM t1, t2 WHERE t1.id_t1=t2.id_t1
Last things you may want to try. Increase your buffers (innodb), your key_buffers (myisam), and some query cache buffers.
Here's some common reasons(bar your server simply being too busy)
The slow query is hitting the harddrive. In the fast case the indexes and data are already cached in MySQL or the OS file cache.
Retrieving the data gets locked by updates/inserts, for MyISAM tables the whole table gets locked whenever someone inserts/updates data in it in some cases.
Table statistics are out of date and/or the wrong index gets selected. running analyze oroptimize on the table can help.
You have the query cache enabled, fetching the result of a cached query is fast, fetching it if it's not in the cache might be slow. Try turning off the query cache to check if the query is always slow if its not fetched from the cache.
In any case, you should show the output of EXPLAIN on your queries to verify indexes are getting used properly - even if they're not, queries can be fast if everything is in ram but grinding to a halt if it needs to hit the hardddrive.