When i disable query cache in mysql, queries still cached. As I understand it is because of OS filesystem cache. How can i prevent filesystem on cache this data. I working on WIndows 7 but it might be the Linux.
There is no query filesystem cache in MySQL.
When i disable query cache in mysql, queries still cached
How do you disable it and how do you know queries are still cached? Why you don't want them to be cached?
SET SESSION query_cache_type = OFF;
Well now i can answer my question by myself. To prevent caching second and next queries need to set innodb_buffer_pool_size=0 config option. This buffer used by mysql to swapping data into memory and all next queries operates with memory instead HD.
You need buffer pool a bit (say 10%) larger than your data (total size of Innodb TableSpaces) because it does not only contain data pages – it also contain adaptive hash indexes, insert buffer, locks which also take some time. Though it is not as critical – for most workloads if you will have your Innodb Buffer Pool 10% less than your database size you would not loose much anyway
Related
After a few weeks running, mysqld (mariadb) is reserving more and more memory as you can see on the screenshot below
htop screenshot
Some answers says that disabling query caching will help, so i did add to /etc/mysql/my.cnf
[mysqld]
query_cache_size = 0
query_cache_type = 0
and tried to restart mysqld but the reserved memory is still the same after the restart.
I didn't touch anything else on the my.conf and using both myisam and innodb tables
Also RESET QUERY CACHE on selected db does nothing at all
This is normal, and 209MB of RAM is very small for a typical MySQL deployment. Did you change any settings away from defaults? That memory consumption looks about right if you haven't. How big is your data?
Some things that are allocated in RAM grow up to some limit, then stick with that setting. The buffer_pool (cf innodb_buffer_pool_size) is the main memory allocation. When you start mysqld, only a little space is allocated for it. As you process data, tables are opened, etc, and the buffer_pool grows. This is probably what you are seeing.
The Query cache is generally useless and should be left off. (There are rare exceptions.) Even if you use it, it would probably not explain what you are seeing. It quickly fills up, fragments and remains (I think) stuck at query_cache_size.
If you need the memory for other apps, then do decrease innodb_buffer_pool_size. With only 170MB of data (including indexes?), a setting of 250M is probably more than adequate. Possibly 100M would work OK, but I would not go that low without adequate need for the RAM elsewhere.
The buffer_pool is used for a variety of things, but it is a "cache". That is, to handle all your data, it needs to be more then 170M, but being a cache, it is find to set it less than that. (But understand that may lead to more I/O.)
The Htop shows that not all of your 8GB of RAM is in use.
Also, lower max_connections; this will decrease the per-connection memory allocations. For most users 20 is sufficient. (Then watch for error messages something like 'cannot get a connection'.)
For my development machine I need no data consistency in case of a crash. Is there a config for a Debian-like system, that optimizes MySQL for speed (even if it sacrifices reliability)?
So something like: Cache the last 1 GB in RAM. Don't touch the disk with data until the 1 GB is used.
What kind of queries are going on? One of my mantras: "You cannot configure your way out of a performance problem."
Here's one thing that speeds up InnoDB, wrt transactions:
innodb_flush_log_at_trx_commit = 2
There is a simple way to speed up single-row inserts by a factor of 10.
Some 'composite' indexes can speed up a SELECT by a factor of 100.
Reformulating a WHERE can sometimes speed up a query by a factor of 100.
You can disable many of the InnoDB configurations for durability, at the risk of increased risk of losing data. But sometimes you want to operate the database in Running with scissors mode because the original data is safely stored somewhere else, and the copy in your test database is easily recreated.
This blog describes Reducing MySQL durability for testing. You aren't going to see any official MySQL recommendation to do this for any purpose other than testing!
Here's a summary of changes you can make in your /etc/my.cnf:
[mysqld]
# log_bin (comment this out to disable the binary log)
# sync_binlog=0 (irrelevant if you don't use the binary log)
sync_frm=0
innodb_flush_log_at_trx_commit=0
innodb_doublewrite=0
innodb_checksums=0
innodb_support_xa=0
innodb_log_file_size=2048M # or more
He also recommends to increase innodb_buffer_pool_size, but the size depends on your available RAM.
For what it's worth, I recently tried to set innodb_flush_log_at_trx_commit=0 in the configuration in the default Vagrant box I built for developers on my team, but I had to back out that change because it was causing too much lost time for developers who were getting corrupted databases. Just food for thought. Sometimes it's not a good tradeoff.
This doesn't do exactly what you asked (keep the last 1GB of data in RAM), as it still operates InnoDB with transaction logging and the log flushes to disk once per second. There's no way to turn that off in MySQL.
You could try using MyISAM, which uses buffered writes for data and index, and relies on the filesystem buffer. Therefore it could cache some of your data (in practice I have found that the buffer flushes to disk pretty promptly, so you're unlikely to have a full 1GB in RAM at any time). MyISAM has other problems, like lack of support for transactions. Developing with MyISAM and then using InnoDB in production can set you up for some awkward surprises.
Here's a couple of other changes you could make in your MySQL sessions for the sake of performance, but I don't recommend these even for development, because it can change your application behavior.
set session unique_checks=0;
set session foreign_key_checks=0;
Some people recommend using the MEMORY storage engine. That has its own problems, like size limits, table-locking, and lack of support for transactions.
I've also experimented with trying to put tables or tmpdir onto a tmpfs, but I found that didn't give nearly the performance boost you might expect. There's overhead in an RDBMS that is not directly related to disk I/O.
You might also like to experiment with MyRocks, a version of MySQL including the RocksDB storage engine for MySQL. Facebook developed it and released it as open-source. See Facebook rocks an open source storage engine for MySQL (InfoWorld). They promise it reduces I/O, it compresses data, and does other neat things.
But again, it's a good rule of thumb to make your development environment as close as possible to your production environment. Using a different storage engine creates a risk of not discovering some bugs until your code reaches production.
Bottom line: Tuning MySQL isn't a magic bullet. Maybe you should consider designing your application to make more use of microservices, caches, and message queues, and less reliance on direct SQL queries.
Also, I'd recommend to always supply your developers the fastest SSD-based workstation you can afford. Go for the top of the line on CPU and RAM and disk speed.
#Bill Karwin's answer has useful mysql settings to improve performance. I have used them all and was able to achieve a roughly 2x performance improvement.
However, what gave me the biggest performance boost (nearly 15x faster) for my use case -- which was reloading a mysql dump -- was to mount the underlying filesystem (ext4) using the nobarriers option.
mount -o remount,nobarrier /
More info here
You should only consider this if you have a separate partition (or logical volume) mounted at /var/lib/mysql, so that you can make this tradeoff only for MySQL, not your entire system.
Although this answer may not hit exactly the questions you ask, consider creating your tables with MEMORY engine as documented here: http://dev.mysql.com/doc/refman/5.7/en/memory-storage-engine.html
A typical use case for the MEMORY engine involves these
characteristics:
Operations involving transient, non-critical data such as session
management or caching. When the MySQL server halts or restarts, the
data in MEMORY tables is lost.
In-memory storage for fast access and low latency. Data volume can fit
entirely in memory without causing the operating system to swap out
virtual memory pages.
A read-only or read-mostly data access pattern (limited updates).
Give that a shot.
My recommendation, even for a development machine, would be to use the default InnoDB. If you choose to do transactions, InnoDB will be helpful.
This blog can help you run MySQL off of tmpfs: http://jotschi.de/2014/02/03/high-performance-mysql-testdatabase/. User Jotschi also speaks about that in a SO answer #10692398
Aurora has two query-cache related metrics :
Buffer cache hit ratio : The percentage of requests that are served by the Buffer cache.
Resultset cache hit ratio : The percentage of requests that are served by the Resultset cache.
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Monitoring.html
But I can't find the documentation that explains the difference between "Buffer cache" and "Resultset cache".
What are they?
"Resultset Cache Hit Ratio" is related to the query cache, which is a feature that enables caching the read queries' results (that's why called result set cache hit). So,if the engine started to execute a new read query, it will check the cached results before executing the query itself and if it found that this same query has been executed before and that its result wasn't invalidated yet, then it will serve the result of the new query from the cache. This is generally useful & shows up high in number when the workload contains a lot of similar select queries that has the similar values and conditions.
On the other hand, "Buffer Cache Hit Ratio" is more related to the innodb page caching hit ratio (& not the query result cache), and this should increase with increasing all types of read queries, as this process is called by bufferpool warm up which will cause the engine to load all the needed pages from the storage to the memory for faster access to the data. However, with increased amount of writes to the writer, this will make the readers to invalidate there in memory pages then load these pages again from the storage when needed. The "ratio" here depends on the percentage of hitting the in memory pages which should be very high ex: more than 99%.
Query cache is generally considered with low connections, similar type of queries over & over again (based on few observations on mysql/aurora, query cache might be actually bad for performance if you have high no. of connections & lots of adhoc style, changing queries).
There's not a ton of info from Amazon other than what I found here: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Monitoring.html
Buffer cache hit ratio: The percentage of requests that are served by the Buffer cache.
Resultset cache hit ratio: The percentage of requests that are served by the Resultset cache.
I have some reporting queries that are rarely run, which I need to be performant without relying on them being cached anywhere in the system. In testing various schema and sproc changes I'll typically see the first run be very slow and subsequent runs fast, so I know there's some caching going on that's making it cumbersome to test changes. Restarting mysqld or running several other large queries are the only reliable ways to reproduce it. I'm wondering if there's a better way.
The MySQL Query Cache is turned OFF.
Monitoring the disk, I don't see any reads happening except on the first run. I'm not that familiar with disk cache but I would expect if that's where the caching is happening I'd still see disk reads, they'd just be very fast.
MONyog gives me what I think is the definitive proof, which is the InnoDB cache hit ratio. Monitoring it I see that when the query's fast it's hitting the InnoDB buffer, when it's slow it's hitting disk.
On a live system I'll gladly let InnoDB do this, but for development and test purposes I'm interested in worst case scenarios.
I'm using MySQL 5.5 on Windows Server 2008R2
I found a post on the Percona blog that says:
For MySQL Caches you can restart MySQL and this is the only way to clean all of the caches. You can do FLUSH TABLES to clean MySQL table cache (but not Innodb table meta data) or you can do “set global key_buffer_size=0; set global key_buffer_size=DEFAULT” to zero out key buffer but there is no way to clean Innodb Buffer Pool without restart.
In the comments he goes on to say:
Practically everything has caches. To do real profiling you need to profile real query mix which will have each query having appropriate cache/hit ratio not running one query in the loop and assuming results will be fine.
I guess that sums it up. It does make it hard to test individual queries. My case is that I want to try forcing different indices to make sure the query planner is picking the right one, and apparently I'll have to restart MySQL between tests to take the cache out of the equation!
I want to load a MYSQL-database into the RAM of my computer, is there a way to do this? I am running this database under Linux. Also, if this is possible, is there a good way to make backups, because if the computer is unexpectedly shut down, I would lose all my data.
If you buffer pool is big enough, you data is -- effectively -- an in-memory database with a disk backup copy. Don't fool around with RAM databases, simply make the buffer pool size as large as you can make it.
Read this:
http://dev.mysql.com/doc/refman/5.1/en/innodb-parameters.html#sysvar_innodb_buffer_pool_size
Yes, you can use the MEMORY engine. As far as backups, it's your call. You never said, e.g. how often you want to store to disk. But you can use traditional MySQL replication or your own solution.
absolutely, for example under linux you can mount your database in a tmpfs
If you're using innodb tables then I recommend adjusting the buffer pool size like S.Lott suggested above. Make it 110% or so of your database size if you have the ram.
If your database is > 50mb you'll also want to look at increasing the innodb_log_file_size. See http://dev.mysql.com/doc/refman/5.1/en/innodb-parameters.html#sysvar_innodb_log_file_size Perhaps to around 25 - 50% of your buffer pool size, but 1gb max.
The innodb_log_file_size is a bit tricky to adjust. You need to shut the db down, move the current log files into a backup location, and let mysql recreate them when it's restarted (i.e. after you've changed the values in my.cnf). Google it and you'll find some answers.