I have a multi-threaded (OpenMP) application using an embedded MariaDB database. Sometimes I use Aria and other times I use InnoDB. When I set the page cache buffer size for Aria (aria_pagecache_buffer_size) or the InnoDB buffer pool size (innodb-buffer-pool-size), will this memory be shared across all of the OpenMP threads for the embedded database? Or are they private for each connection?
InnoDB:
The buffer_pool is shared across all connections. (From MariaDB's point of view, "connections" are significant; it does not care about what is going on in the client.)
How much RAM? Keep in mind that Aria and InnoDB don't share much. So you must split most of the memory between the buffer_pool and the pagecache. A lot of the smaller things are 'common'.
Related
The general rule of thumb I've observed is configuring this property to use 70% of available RAM for dedicated SQL servers with over 4GB RAM. However I'm working on what basically amounts to a shared hosting environment experiencing a ton of traffic lately and I want to optimize this - these are dedicated mySQL servers, but have databases for 200-1000 different sites. Should I still configure using this rule?
You may have many tables and schemas in your MySQL instance, but a single buffer pool is used for all of them. It makes no difference what they're used for — one website or many websites or some database that is not for a website at all. Basically everything that is stored in a page of an InnoDB tablespace on that MySQL instance must be loaded into the buffer pool before it can be read or updated.
The recommendation of 70% of available RAM is not a magic number.
For example, it assumes you have a lot more data on storage than can fit in RAM. If you had 100GB of RAM and 2GB of data on storage, it would be unnecessary overkill to make a 70GB buffer pool. The pages from storage will only be copied into the buffer pool once, therefore for 2GB of data, your 70GB buffer pool would be mostly empty.
It also assumes that the remaining 30% of RAM is enough to support your operating system and other processes besides MySQL.
70% is just a starting suggestion. You need to understand your memory needs to size it properly.
MySQL's default storage engine, InnoDB, maintains an internal buffer pool of database pages. In newer versions of MySQL (e.g. 5.7+) the space and page IDs of the pages in the buffer pool are persisted to disk in the "ib_buffer_pool" file.
I'm curious about how this file is constructed, and in particular if the relative young-ness/old-ness of the pages in the buffer pool persists across restarts. In other words, if some page in the pool is younger than some other page, will that relationship hold after the file is written to, and then read from, the disk?
A broader question is the following: how much of the state of the InnoDB buffer pool persists across restarts?
Most of what you ask does not matter.
That file contains pointers, not data blocks. Each "pointer" probably contain the tablespace id (ibdata1 versus individual .ibd files) and block number. It would be handy, but not absolutely necessary to include the LRU info.
The goal is to quickly refill the RAM-based "buffer pool" after a restart. The buffer pool is a cache; in the past is was simply not reloaded. During normal activity, the blocks in the buffer pool are organized based (somewhat) on "least recently used". This helps prevent bumping out a block "too soon".
If all the block pointers are stored in that file before shutting down, then the buffer pool can be restored to essentially where it was. At restart, this takes some disk activity, but after that, each query should be as fast as if the restart had not occurred..
If, because of whatever, some block is inappropriately reloaded, it will be a minor performance hit, but nothing will be "wrong". That block will soon be bumped out of the buffer pool.
How much state persists across a restart? Well, the absolute requirement is that the integrity of the data in the database be maintained -- even across a power failure. Anything beyond that is just performance optimizations. So, to fully answer the question, one needs to understand iblog* (needed after a crash; not needed after clean shutdown), the new tmp table file (not needed), the "double buffer" (used to recover from 'torn page' after abrupt crash), etc.
I created Clustered Column Store index on a table. I want to improve memory performance using buffer pool. Is it possible to set buffer pool on sql server, a table having clustered column Store index in SQL Server 2014? What will be the performance gain after setting buffer on it.
This is one of those "it depends" questions. The answer depends on a lot of things, such as how much RAM you already have in your system and your workload.
Real RAM is always going to give you better performance than the buffer pool extension, if you're running standard edition you can now use 128Gb with SQL server and filling your server with more RAM will give you much better performance improvements than using the buffer pool extensions.
You may also find that for your workload, using the SSDs for your data files rather than the buffer pool gives better results (especially for read heavy workloads with low cache hit ratios so it has to read a lot from disk)
Here is a link to a test done on buffer pool extensions by the Brent Ozar team:
SQL Server 2014 Buffer Pool Extensions
You may get different results with your workload and hardware, so you really need to test it yourself to get an accurate idea of how much it will help.
It is pretty much a given though that it won't help as much as adding more RAM.
I've tried reading some of the material out there but they are a bit over my head. However, from what I understand, if you have a lot of memory allocated to the buffer pool, then the writes to memory are happening faster than the disk can keep up with, and therefore, there are "pages to be flushed" still? Additionally, if I restart the mySQL server, will that cause any issues?
InnoDB performs certain tasks in the background, including flushing of dirty pages (those pages that have been changed but are not yet written to the database files) from the buffer pool, a task performed by the master thread.
For more information you can refer:
http://dev.mysql.com/doc/refman/5.6/en/innodb-performance.html#innodb-performance-adaptive_flushing
Having dirty pages is something normal. When you update a row, MySQL updates it in the buffer pool, marking the page as dirty. The change is written in the binary log as well, so in case of crash, MySQL will replay the log and data won't be lost. Writting to the binary log is a append-only operation, while the actual update involve random writes, and random write is slower. MySQL flushes dirty pages to disk when it needs to load new data in the buffer pool. So, having dirty pages in InnoDB is something normal - it's how it works and it's done to improve the overall performance.But if you really want to get rid of them, set innodb_max_dirty_pages_pct value to 0
If you are using MySQL v5.6 then you can enable this variable innodb_buffer_pool_dump_at_shutdown which
Specifies whether to record the pages cached in the InnoDB buffer pool when the MySQL server is shut down, to shorten the warmup process at the next restart. you must use this variable in conjunction with innodb_buffer_pool_load_at_startup.
What's the difference between memcache and buffer pool in MySQL?
It seems to me that both use memory as buffer/cache so as to reduce disk I/O.
Thanks!
Memcache
memcached is a simple, highly scalable key-based cache that stores data and objects wherever dedicated or spare RAM is available for quick access by applications, without going through layers of parsing or disk I/O. To use, you run the memcached command on one or more hosts and then use the shared cache to store objects.
InnoDB buffer pool
The memory area that holds cached InnoDB data for both tables and indexes. For efficiency of high-volume read operations, the buffer pool is divided into pages that can potentially hold multiple rows. For efficiency of cache management, the buffer pool is implemented as a linked list of pages; data that is rarely used is aged out of the cache, using a variation of the LRU algorithm. On systems with large memory, you can improve concurrency by dividing the buffer pool into multiple buffer pool instances.
Several InnoDB status variables, information_schema tables, and performance_schema tables help to monitor the internal workings of the buffer pool. Starting in MySQL 5.6, you can also dump and restore the contents of the buffer pool, either automatically during shutdown and restart, or manually at any time, through a set of InnoDB configuration variables such as innodb_buffer_pool_dump_at_shutdown and innodb_buffer_pool_load_at_startup.
Memcache integration with InnoDB
MySQL 5.6 includes a NoSQL interface, using an integrated memcached daemon that can automatically store data and retrieve it from InnoDB tables, turning the MySQL server into a fast “key-value store” for single-row insert, update, or delete operations. You can still also access the same tables through SQL for convenience, complex queries, bulk operations, application compatibility, and other strengths of traditional database software.
http://dev.mysql.com/doc/refman/5.6/en/innodb-memcached-benefits.html
http://dev.mysql.com/doc/refman/5.6/en/innodb-memcached.html
For more information on memcache refer:
http://dev.mysql.com/doc/refman/5.0/en/ha-memcached-using.html