I would like to run the same query multiple times to see how much time it takes without the aid of cache memory.
Running the command RESET QUERY CACHE seems not to work because the query takes a really short time on the second run even after the cache reset.
What am i missing?
Your OS is caching a whole lot of data. Then the DBMS itself caches a whole of data. The query cache only holds the output of a previous query. The data the query output is created from may all be in RAM. If you want to run the query without any caching then switch off the host between executions or run a query which will read enough data from disk to overwrite all the cache. But it won't be a realistic measure of ho.w your query will perform in the wild.
Two "caches" affects the query. Query Cache and Buffer Cache(buffer_pool with InnoDB and key_buffer with MyISAM). Query cache affects much more than Buffer Cache because it caches the result of your query, so the same query will not execute again.
To avoid Query Cache, uses SQL_NO_CACHE is a good idea, and you can disable Query Cache in my.cnf and restart mysql.
Buffer cache is managed by mysql to cache data in memory so that your query will not read data from disk(SSD or HDD). If you want to clear it, try to set the 'Buffer cache' small enough and fill it with other data(use SELECT).
You can use the SQL_NO_CACHE in the SELECT query.
https://dev.mysql.com/doc/refman/5.5/en/query-cache-in-select.html
Keep it in mind that Query cache has been removed in later versions of MySQL.
Depending on the storage engine, (e.g innodb), it loads table data into memory as part the buffer pool. This part you can't really control on what tables to load and which tables not to load (or at least not easily).
On an ancient version of MySQL (v.5.1), I had a problem with a view being cached, which would never refresh. I tried RESET QUERY CACHE , FLUSH TABLES, SELECT SQL_NO_CACHE..., etc. And nothing worked. Then, I changed the storage engine for the underlying (single) table I was querying in this view, from InnoDB to MyISAM, and it worked as desired implicitly! There was no need to jump through any hoops to clear or prevent caching!
I'm not sure if this was simply a bug with that old version / storage engine? Please leave comments if you have any knowledge to share on the matter.
Related
Do we need to drop OS cache?
I want to read data from disk not from cache I have disabled
1. query_cache_type=OFF
2. query_cache_size=0
even then when i perform select operation for Id =2 , innodb_buffer_pool_reads changes . If i select Id=3 no change for innodb_buffer_pool_reads.
How do I read next value from disk? Is there any other way to verify whether data is being read from the disk?
[Edit] Thank you all for your response.
I m trying to perform reverse engineering , want to test the execution speed of a select query without cache . So want to disable all cache and read data from disk?
Yes, to completely turn off the Query cache, make both of those settings.
To disable the Query cache for a single SELECT, do SELECT SQL_NO_CACHE ....
But... The QC is not the only caching mechanism. For InnoDB, the buffer_pool caches data and indexes. (MyISAM uses its key_cache and the OS's cache)
Typically, the first time you perform a query (after restarting the server), the disk will need to be hit. Typically, that query (or similar queries) performed after that will not need to hit the disk. Because of "caching", MySQL will hit the disk as little as necessary.
If some other connection in modifying the data you are about to SELECT, do not worry. MySQL will make sure to change the cached copy and/or the disk copy. You will always get the correct value.
InnoDB does things in "blocks" (16KB, typically about 100 rows). That is the unit of disk I/O. Ids 1,2,3 are probably in the same block. Again, MySQL takes care of fetches and changes. It will probably read the block once, cache it for a long time, and eventually write it once, even if there are a lot of changes to the rows in the block.
So how does "Durability" happen? Magic. It involves the InnoDB log file and some extra writes that are done to it. That is another topic; it would take much too long to explain it all.
I know that SQL query will use query cache to receive data instead of reprocess all of the data. Here the question I would like to ask,
I working with a server of database and I'm one of the developer that working on it and I need to do performance testing on queries that i handling
If I clear the query cache
example using FLUSH QUERY CACHE; or RESET QUERY CACHE;,
will it affect others developer or it only clears away my local query cache?
If it will affect others, is there any way to clear locally or allow my query won't use the query cache for testing
Two clarifications to begin with:
MySQL query cache is a server-side feature, there's no such thing as "local cache". You're probably confused by the LOCAL keyword in FLUSH command. As docs explain it's just an alias for NO_WRITE_TO_BINLOG (thus it's related to replication and "local" means "this server").
MySQL will only return cached data if you've enabled the feature and either made it default or opted-in with the SQL_CACHE hint. In my experience, most servers do not have it by default.
Let's now answer your question. At The MySQL Query Cache we can read:
The query cache is shared among sessions, so a result set generated by
one client can be sent in response to the same query issued by another
client.
Which makes sense: a cache that cannot reuse stored data is not as useful.
I don't know what you want to test exactly. Your data should always be fresh:
The query cache does not return stale data. When tables are modified,
any relevant entries in the query cache are flushed.
However you might want to get an idea of how long the query takes to run. You can always opt out with the SQL_NO_CACHE keyword:
The server does not use the query cache. It neither checks the query
cache to see whether the result is already cached, nor does it cache
the query result.
Just take into account that a query that runs for the second time might run faster even without cache because part of the data segments might be already loaded into RAM.
Try using the SQL_NO_CACHE option in your query.This will stop MySQL caching the results
SELECT SQL_NO_CACHE * FROM TABLE
With SQL Server for cached data, you can use DBCC DROPCLEANBUFFERS and force a manul CHECKPOINT.
However it works at the Server (instance) level:
Use DBCC DROPCLEANBUFFERS to test queries with a cold buffer cache without shutting down and restarting the server.
To drop clean buffers from the buffer pool, first use CHECKPOINT to produce a cold buffer cache. This forces all dirty pages for the current database to be written to disk and cleans the buffers. After you do this, you can issue DBCC DROPCLEANBUFFERS command to remove all buffers from the buffer pool.
edited*
SQL Query buffer cache is global but not local. If the buffer or query cache is drop, it drops it globally, and will affect all user using the database server.
I have one MySQL server running on Linux.
It has a few critical apps, and we have set the querycache as high as we could afford.
I also have a few non critical databases, (wordpress / etc).
Questions:
Is it possible to fine tune query cache on a per database level?
Is it possible to fine tune query cache on a per table basis?
Is it even worth doing? Having a fine tuned query cache on critical tables, will the db still stutter when accessing non important data?
Thanks in advance
Unfortunately, there are very few options that lets you manipulate the MySQL query cache.
The query_cache_limit option instructs MySQL to not cache query results larger than a set limit. Reasons why you would want to lower this value:
some relatively rare queries return large result sets
most slower queries typically return small result sets
The SQL_NO_CACHE keyword, immediately placed after the SELECT statement, instructs MySQL to not cache this result.
Conversely, you could set the query_cache_type server option to 2 so that only queries using the SQL_CACHE keyword are cached.
I would also advise to make sure your query cache is actually fully used. SHOW STATUS LIKE 'Qcache_free_memory'; gives you this information. If a high proportion of your query cache is free, then it probably means that your data changes too frequently for the results in cache to be reused.
It could also mean that most of your queries return results sets larger than query_cache_limit (which in turn probably suggests badly designed queries).
There are a few other tips here.
However, you are correctly wondering whether this is worth the hassle., In my opinion the query cache is, at best, a secondary factor for a fast database. Appropriate indexing is the first factor. Moreover, in many cases, your memory would be better used for caching indexes (the most important parameters being the innodb_buffer_pool_size for InnoDB tables, or the key_buffer_size for MyISAM tables)
I noticed that my database server supports the Memory database engine. I want to make a database I have already made running InnoDB run completely in memory for performance.
How do I do that? I explored PHPMyAdmin, and I can't find a "change engine" functionality.
Assuming you understand the consequences of using the MEMORY engine as mentioned in comments, and here, as well as some others you'll find by searching about (no transaction safety, locking issues, etc) - you can proceed as follows:
MEMORY tables are stored differently than InnoDB, so you'll need to use an export/import strategy. First dump each table separately to a file using SELECT * FROM tablename INTO OUTFILE 'table_filename'. Create the MEMORY database and recreate the tables you'll be using with this syntax: CREATE TABLE tablename (...) ENGINE = MEMORY;. You can then import your data using LOAD DATA INFILE 'table_filename' INTO TABLE tablename for each table.
It is also possible to place the MySQL data directory in a tmpfs in thus speeding up the database write and read calls. It might not be the most efficient way to do this but sometimes you can't just change the storage engine.
Here is my fstab entry for my MySQL data directory
none /opt/mysql/server-5.6/data tmpfs defaults,size=1000M,uid=999,gid=1000,mode=0700 0 0
You may also want to take a look at the innodb_flush_log_at_trx_commit=2 setting. Maybe this will speedup your MySQL sufficently.
innodb_flush_log_at_trx_commit changes the mysql disk flush behaviour. When set to 2 it will only flush the buffer every second. By default each insert will cause a flush and thus cause more IO load.
Memory Engine is not the solution you're looking for. You lose everything that you went to a database for in the first place (i.e. ACID).
Here are some better alternatives:
Don't use joins - very few large apps do this (i.e Google, Flickr, NetFlix), because it sucks for large sets of joins.
A LEFT [OUTER] JOIN can be faster than an equivalent subquery because
the server might be able to optimize it better—a fact that is not
specific to MySQL Server alone.
-The MySQL Manual
Make sure the columns you're querying against have indexes. Use EXPLAIN to confirm they are being used.
Use and increase your Query_Cache and memory space for your indexes to get them in memory and store frequent lookups.
Denormalize your schema, especially for simple joins (i.e. get fooId from barMap).
The last point is key. I used to love joins, but then had to run joins on a few tables with 100M+ rows. No good. Better off insert the data you're joining against into that target table (if it's not too much) and query against indexed columns and you'll get your query in a few ms.
I hope those help.
If your database is small enough (or if you add enough memory) your database will effectively run in memory since it your data will be cached after the first request.
Changing the database table definitions to use the memory engine is probably more complicated than you need.
If you have enough memory to load the tables into memory with the MEMORY engine, you have enough to tune the innodb settings to cache everything anyway.
"How do I do that? I explored PHPMyAdmin, and I can't find a "change engine" functionality."
In direct response to this part of your question, you can issue an ALTER TABLE tbl engine=InnoDB; and it'll recreate the table in the proper engine.
In place of the Memory storage engine, one can consider MySQL Cluster. It is said to give similar performance but to support disk-backed operation for durability. I've not tried it, but it looks promising (and been in development for a number of years).
You can find the official MySQL Cluster documentation here.
Additional thoughts :
Ramdisk - setting the temp drive MySQL uses as a RAM disk, very easy to set up.
memcache - memcache server is easy to set up, use it to store the results of your queries for X amount of time.
I have a 5GB database, all tables are MyISAM. It runs into heavy load time from 01:30AM to 8:30AM (100+ selects, 150+ updates, 200+ cache hits per second) to do data analysis, during other time, load is moderate (10 selects, 5 inserts per second).
Problem is after a few days, data analysis during heavy load time appears to be slow down maybe due to query cache prunes (iowait increases). Current query cache is set to 1.5G while total RAM is 4G. It runs fast again after manually restart mysql server.
Is there a way to do regular optimization or cleaning up on mysql server to keep it running in a efficiently without a restart
It sounds to me like your application is busy updating the tables and you might have table contention. Do you have mytop running, or does SHOW PROCESSLIST give you any insight as to what part of your application is doing the most work? Have you enabled --slow-query-log setting?
Also, your database table engine might be an issue. Are you using MyISAM or InnoDB? You want to look out for table locking during updates, and how much of a backup that can create.
If you are issuing FLUSH QUERY CACHE, that can lead to badness, many versions of MySQL exhibit near-lockup when running that command.
Also, running top and checking /var/log/cron for cronjobs that might be affecting system load could help. If you are running updatedb or logrotate on your server, that could affect iowait.
It seems like your query cache size is far too large. While the query cache is usually a good thing, if it is too large it can hurt more then it helps.
This behavior is discussed in this article:
The issue here was that the customer had a moderate level of write traffic, and the current query cache implementation invalidates all result sets for a given table whenever that table is updated. As the query cache grows in size, the number of entries that must be invalidated for a given table may grow as well. In addition, the coarse locking on the cache can lead to lock contention that can kill performance, particularly on multi-core hardware.
I would recommend lowering the size of your query cache to somewhere between 16-128MB and see how that effects performance.
Another possibility is that the queries are generating really small result sets which is causing memory fragmentation. More information on this is available here, look for the "query_cache_min_res_unit" setting.