Imagine we have a MYSQL DB that's data size is 500 MB.
If I will set the innodb_buffer_pool_size at 500MB (or more), is it correct to think that all the data will be cached in RAM, and my queries won't touch disk?
Is effective_cache_size in POSTGRESS is the same as MYSQL's buffer_pool and it also can help avoid reading from disc?
I believe you are on the right track in regards to MySQL innoDB tables. But you must remember that when measuring the size of a database, there are two components: data length and index length.
MySQL database size.
You also have no control over which databases are loaded into memory. If you want to guarantee a particular DB is loaded, then you must make sure the buffer pool is large enough to hold all of them, with some room to spare just in case.
MySQL status variables can then be used to see how the buffer pool is functioning.
I also highly recommend you use the buffer pool load/save variables so that the buffer pool is saved on shutdown and reloaded on startup of the MySQL server. Those variables are available from version 5.6 and up, I believe.
Also, check this out in regards to sizing your buffer pool.
Is "effective_cache_size", a parameter to indicate the planner as to what OS is actually doing ?
http://www.cybertec.at/2013/11/effective_cache_size-better-set-it-right/
and for caching the tables, do we not need to configure "shared_buffers" ?
And with regards to MySQL, yes the "innodb_buffer_pool" size will cache the data for Innodb tables and preventing disc reads. Make sure its configured adequate to hold all the data in memory.
Related
İ am trying to understand the mysql architecture and I came acrosa two notions.
The first one is query cache, which I understood that it stores the queries that were run at least once, and if the query processor sees the query cached there, it no longer goes to the parser and takes the results directly to the cache.
But then, there is also the buffer pool, part of the Storage Engine buffer manager, which kinda does the same thing from my understanding.
So my question would be, if there is a cache in the logical layer, why do we need one in the physical layer also? İ am thinking that if a query is found in the query cache it will never be searched in the buffer pool, and if the query is not found in cache, then it will never be also retreived from the buffer pool. Am I missing something?
For query cache, you got it spot on. Its based on the raw text of the query mapping to the exact query results. It has major scaling problems which is why MySQL-8.0 removed it.
innodb buffer pool, is a storage of the low level data and index pages of the database. It ensures that all the recently used data is off disk and able to be queried without resorting to the much slower (by comparison to ram) storage.
So buffer pools serve all queries on the same data, while query caches only serve a particular query (at a large scaleability cost).
Adding some context to #danblack's answer, query cache stores the query and actual data associated with the query. But in buffer pool which we call as innodb_buffer_pool stores the physical (01,10) or low-level data or say pages. Whenever query executes it checks in the buffer pool and if required data is not present then it will proceed towards the disk (i.e. your secondary storage) and puts data in the buffer pool.
With query cache, there is a disadvantage of invalidating query cache if query cache size being set quite high without analyzing the situations. By "invalidating query cache" I mean marking the data or entry in query cache as invalid because the underlying table has been changed by DML statements. I have personally experienced many times for example under "show processlist" when replication is stuck for long at this particular state i.e. invalidation query cache and once it invalidates all the entries, things start catching up.
"Why do we need one in the physical layer?"
It is because having data in query cache can seriously impact the performance IF underlying table changes quite often which can affect the overall database performance. So if your table is not changing frequently query cache is useful. But now the concept of query cache has been removed in MySQL 8 (which is not a part of the discussion).
Bufferpool is only used to store pages coming from the secondary store.
CPU can not fetch data from secondary storage so the Database management system makes a pool in RAM and then CPU keeps access data from this buffer pool from RAM.
and DBMS uses a replacement algorithm to replace pages from this buffer pool.
Cache of data is something else.
There are other data structs and techniques for data cache.
I've two different MySQL servers with the same database (a copy), both with Ubuntu x64, 4Gb RAM. Both are virtual machines hosted in the same VMWare server.
The first is our old server with MySQL 5.6.33-0ubuntu0.14.04.1-log, and the new one have the version 5.7.17-0ubuntu0.16.04.1 installed.
I'm comparing the performance of some SQL scripts and I noticed that the new server have bigger fetch times with the exact same SQL. Can you help to determinate possible causes?
Maybe the 5.7 engine analyses the SQL in a different and less efficient way?
Maybe some MySQL configuration need to be tuned differently? I only changed innodb_buffer_pool_size = 2G and innodb_buffer_pool_instances = 2 (same as the old server)
Ideas?
Thx
I suspect your problem is that your buffer pool is allocated, but not yet full of data. As you run queries, it has to fetch data from disk, which is much slower than RAM. As you run those queries again and again, the data required will already be in the buffer pool, and MySQL will take advantage of that. Data that is already in the buffer pool can be read without touching the disk.
You can check how much is in your buffer pool. Here's an example from my test instance (I put "..." because the output is long, and I'm showing an excerpt).
mysql> SHOW ENGINE INNODB STATUS\G
...
----------------------
BUFFER POOL AND MEMORY
----------------------
...
Buffer pool size 65528
Free buffers 64173
Database pages 1339
...
These numbers are in "pages" of 16KB each. You can see I have 64*1024 pages = 1GB allocated, but nearly all of it is free, i.e. unoccupied by data. Only 2% of my buffer pool pages have data in them. It's likely that if I run queries now, it will have to read from the disk to load data. Unless perhaps I have very little data in my database on disk too, and it only fills 2% of my buffer pool even when it's fully loaded.
Anyway, assuming you have more data than the size of your buffer pool, it will gradually fill the buffer pool as you run queries. Then you'll see the ratio of "Database pages" to "Free buffers" change over time (I don't know why they say both pages and buffers, since they refer to the same thing). Subsequent queries should run faster.
Recently I updated my VPS from 1GB to 4GB memory. I'd hoped that the queries (MYSQL/InnoDB) were running faster with more memory, but unfortunately that's not the case. Does mysql automatically takes more memory when a server has more memory or do I have to change some settings in my.cnf? And if so, what changes should I make?
MySQL will not automatically take the benefit of more memory installed.
In your case (given that you are using InnoDB) you can do at least these to improve the performance of mysql:
increase innodb_buffer_pool_size (default value for this option is 128MB). This defines how much memory is dedicated to mysql innodb to cache its data tables and idexes. Which means if you can allocate more memory mysql will cache more of its data resulting in faster queries (because mysql will look in memory instead of doing I/O operations for data lookup).
Of course you should allocate reasonable amount of memory (not the whole 4G :)) may be not more than 2G. You should try and test it on the server for more accurate result. (read this for more info, before you change this option https://dev.mysql.com/doc/refman/5.7/en/innodb-buffer-pool-resize.html)
increase innodb_buffer_pool_instances. For you case may be 1 or 2 instances are more than enough. (you can read more here: https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_buffer_pool_instances)
But before starting with editing of my.ini do your calculations for your case. Consider your mysql server load, slow queries etc. for more accurate setup of the options in my.ini
I'm working on a system that incluces exporting large amounts of data into csv files. We are using InnoDB for the our tables. InnoDB buffers previous queries/results in some manor.
Now on a production enviroment that is a really good thing but while testing the performance of an export in my dev enviroment it is not.
The buffer pool size seems to be Around 128MB.
I couldn't find much about this on google except that you can change some MySQL settings when the server boots up.
Anyone knows a workaround of maybe there is a sql statement that prevents it from being put into the buffer?
It's a non-problem (since 5.1.41)
It is impossible to prevent any InnoDB activity from going through the buffer_pool. It is too deeply engrained in the design.
The buffer_pool caches data and index blocks, not queries/results. The Query cache plays with queries/results. But the QC should normally be disabled for production systems.
innodb_old_blocks_pct (default = 37, meaning % of buffer_pool) prevents wiping out the buffer pool from certain operations such as the reads needed for your 'export'.
See http://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_old_blocks_pct
and the links in that section.
and what about set the buffer pool to a very small value (ex: 1MB)
I am running a mysql server. I would like to somehow make sure that the whole DB is loaded into the ram as I heard it would be alot faster.
Is this true? and how do I vertify it?
If you are using InnoDB, then you can set innodb_buffer_pool_size to be at least as large as your data and indexes. Then as you access data, it will be cached into memory and all subsequent accesses will occur via memory.
If you wanted to pre-cache the data, you could do some of your common queries, or full table/index scans to force it to load all of the data into memory.
There is an init-file option for mysqld which can have the server automatically execute some commands on startup, where you could include the pre-caching queries to execute.
Keep in mind you don't want to run out of memory, so be careful with setting innodb_buffer_pool_size too large.
Well, one option is to use a ramdrive.. though it's not permanent.
http://www.linuxforums.org/forum/misc/670-how-create-ram-drive-linux.html
(Just for completeness,) You could use HEAP as table engine. But keep in mind that whatever data you put in a HEAP table is gone when you reboot the server.
Tables in RAM are very fast.
You can change your existing tables by altering their storage engine. But make a backup if you want to use the data after a server restart.
ALTER TABLE test ENGINE = MEMORY;
Keep in mind, that MEMORY tables cannot contain BLOB or TEXT columns and data and index size is limited to available memory.