I am attempting to run the same query multiple times on the same mysql 8 database and table.
I need to carry out experiments to determine if tweaking the query and or table itself improves performance. However after the first attempt the response time is much faster, i assume becuase the data is cached.
mysql 8 innodb
What options do i have to clear the cache so the data is fetched from scratch.
It appears the answers that have been proposed before are all related to mysql 5 and not mysql 8. Most of the commands seem to now be deprecated.
Clear MySQL query cache without restarting server
The question you link to is about the query cache, which is removed in MySQL 8.0 so there's no need to clear it anymore.
Your wording suggests you are asking about the buffer pool, which is not the same as the query cache. The buffer pool caches data and index pages, whereas the query cache (when it existed) cached results of queries.
There is no command to clear the buffer pool without restarting the MySQL Server. Pages remain cached in the buffer pool until they are evicted by other pages.
The buffer pool is in RAM so its contents are cleared if you restart the MySQL Server process. So if you want to start from scratch, you would need to restart that process (you don't need to reboot the whole OS, just restart the MySQL service).
The caveat is that in MySQL 8.0, the contents of the buffer pool are not entirely cleared when you restart. A percentage of the content of the buffer pool is saved during shutdown, and reloaded automatically on startup. This feature is enabled by default, but you can optionally disable it.
Read more about this:
https://dev.mysql.com/doc/refman/8.0/en/innodb-preload-buffer-pool.html
https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_buffer_pool_dump_at_shutdown
Related
MariaDB 10.3.22 innodb_status_output keeps turning on automatically
Per MySQL docs, https://dev.mysql.com/doc/refman/8.0/en/innodb-troubleshooting.html,
"InnoDB temporarily enables standard InnoDB Monitor output under the following conditions:
1.A long semaphore wait
2.InnoDB cannot find free blocks in the buffer pool
3.Over 67% of the buffer pool is occupied by lock heaps or the adaptive hash index "
MariaDB docs don't mention "InnoDB temporarily enables standard InnoDB Monitor", https://mariadb.com/kb/en/xtradb-innodb-monitors/
Running the commands below does turn off the monitors, but they come back on, probably due to the conditions mentioned above:
SET GLOBAL innodb_status_output=OFF;
SET GLOBAL innodb_status_output_locks=OFF;
I'd like to prevent MariaDB from temporarily turning on InnoDB Monitor. I understand we could fix our db to prevent the conditions above, but we'd like to not have InnoDB Monitor turned on automatically. -Thanks for the help.
I didn't find an exact solution. I had to turn off all logging to stop the output of "standard InnoDB Monitor". I could not find a way to output error logging without MariaDB automatically turning on "standard InnoDB Monitor output". Note that "standard InnoDB Monitor" outputs millions of lines per day for a medium active db. I'm still interested if anyone should find a solution, thanks -K
What are the recommended configurations for allocating RAM to MySQL?
My environment: Ubuntu 14 machine with 96G RAM, MySQL Ver 14.14 Distrib 5.7.
And how to set those configurations?
Thank you in advance!
Assuming you are are using InnoDB as engine than please see here https://www.percona.com/blog/2013/09/20/innodb-performance-optimization-basics-updated/
MySQL Innodb Settings
From 5.5 InnoDB is the default engine, so these parameters are even more important for performance than before. The most important ones are:
innodb_buffer_pool_size: InnoDB relies heavily on the buffer pool and should be set correctly, so be sure to allocate enough memory to it. Typically a good value is 70%-80% of available memory. More precisely, if you have RAM bigger than your dataset setting it bit larger should be appropriate with that keep in account of your database growth and re-adjust innodb buffer pool size accordingly. Further, there is improvement in code for InnoDB buffer scalability if you are using Percona Server 5.1 or Percona Server 5.5 You can read more about it here.
innodb_buffer_pool_instances: Multiple innodb buffer pools introduced in InnoDB 1.1 and MySQL 5.5. In MySQL 5.5 the default value for it was 1 which is changed to 8 as new default value in MySQL 5.6. Minimum innodb_buffer_pool_instances should be lie between 1 (minimum) & 64 (maximum). Enabling innodb_buffer_pool_instances is useful in highly concurrent workload as it may reduce contention of the global mutexes.
Dump/Restore Buffer Pool: This feature speed up restarts by saving and restoring the contents of the buffer pool. This feature is first introduced in Percona Server 5.5 you can read about it here. Also Vadim benchmark this feature You can read more about it in this post. Oracle MySQL also introduced it in version 5.6, To automatically dump the database at startup and shutdown set innodb_buffer_pool_dump_at_shutdown & innodb_buffer_pool_load_at_startup parameters to ON.
innodb_log_file_size: Large enough InnoDB transaction logs are crucial for good, stable write performance. But also larger log files means that recovery process will slower in case of crash. However this is not such big issue since great improvements in 5.5. Default value has been changed in MySQL 5.6 to 50 MB from 5 MB (old default), but it’s still too small size for many workloads. Also, in MySQL 5.6, if innodb_log_file_size is changed between restarts then MySQL will automatically resize the logs to match the new desired size during the startup process. Combined log file size is increased to almost 512 GB in MySQL 5.6 from 4 GB. To get the optimal logfile size please check this blog post.
innodb_log_buffer_size: Innodb writes changed data record into lt’s log buffer, which kept in memory and it saves disk I/O for large transactions as it not need to write the log of changes to disk before transaction commit. 4 MB – 8 MB is good start unless you write a lot of huge blobs.
innodb_flush_log_at_trx_commit: When innodb_flush_log_at_trx_commit is set to 1 the log buffer is flushed on every transaction commit to the log file on disk and provides maximum data integrity but it also has performance impact. Setting it to 2 means log buffer is flushed to OS file cache on every transaction commit. The implication of 2 is optimal and improve performance if you are not concerning ACID and can lose transactions for last second or two in case of OS crashes.
innodb_thread_concurrency: With improvements to the InnoDB engine, it is recommended to allow the engine to control the concurrency by keeping it to default value (which is zero). If you see concurrency issues, you can tune this variable. A recommended value is 2 times the number of CPUs plus the number of disks. It’s dynamic variable means it can set without restarting MySQL server.
innodb_flush_method: DIRECT_IO relieves I/O pressure. Direct I/O is not cached, If it set to O_DIRECT avoids double buffering with buffer pool and filesystem cache. Given that you have hardware RAID controller and battery-backed write cache.
innodb_file_per_table: innodb_file_per_table is ON by default from MySQL 5.6. This is usually recommended as it avoids having a huge shared tablespace and as it allows you to reclaim space when you drop or truncate a table. Separate tablespace also benefits for Xtrabackup partial backup scheme.
I have a droplet on DigitalOcean created using Laravel Forge and since a few days ago the MySQL server just crashes and the only way to make it work again is by rebooting the server (MySQL makes the server unresponsive).
When I type htop to see the list of processes is showing a few of /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysql.pid (currently is showing 33 of them).
The error log is bigger than 1GB (yes, I know!) and shows this message hundreds of times:
[Warning] InnoDB: Difficult to find free blocks in the buffer pool (21
search iterations)! 21 failed attempts to flush a page! Consider
increasing the buffer pool size. It is also possible that in your Unix
version fsync is very slow, or completely frozen inside the OS kernel.
Then upgrading to a newer version of your operating system may help.
Look at the number of fsyncs in diagnostic info below. Pending flushes
(fsync) log: 0; buffer pool: 0. 167678974 OS file reads, 2271392 OS
file writes, 758043 OS fsyncs. Starting InnoDB Monitor to print
further diagnostics to the standard output.
This droplet has been running during 6 months but this problem only started last week. The only thing that changed recently is now we send weekly notifications to customers (only the ones that subscribed to it) to let them know about certain events happening in the current week. This is kind of a intensive process, because we have a few thousands of customers, but we take advantage of Laravel Queues in order to process everything.
Is this a MySQL-settings related issue?
Try increasing innodb_buffer_pool_size in my.cnf
The recommendation for a dedicated DB server is 80% - if you're already at that level then you should consider moving to a bigger instance type.
in the my.cnf set this value:
innodb_buffer_pool_size = 12G
innodb_buffer_pool_instances = 12
innodb_page_cleaners = 12
I am currently using MySQL. I have noticed several times my front application runs slow after some usage. when i checked server status in MySQL workbench. I have noticed that innodb buffer usage was going to 100% . so I increased parameter innodb_buffer_pool_size to 1G in my.ini file of xampp. but innodb is not flushing the buffer and application runs slow after some time. is there any other parameters to change as-well?
consider using a size for innodb_buffer_pool_size of 70%-80% of available ram. Depending on how big your dataset is, you should increase the size.
I'm trying to add a Fulltext index to a table. When I run the query,
ALTER TABLE thisTable ADD FULLTEXT(thisText);
I get the message
SQL Error (2013): Lost connection to MySQL server during query
and the mysql service will indeed be stopped. If I restart the service and try to add the index again I get another error.
SQL Error (1813): Tablespace for table 'thisTable/#sql-ib21134' exists. Please DISCARD the tablespace before IMPORT.
The engine is InnoDb and I run MySQL 5.6.12, so Fulltext index should be supported. The column is a TEXT column.
I'd be very grateful if someone could point me in the right direction where the error is coming from.
The problem is related to sort buffer size. It is known bug of mysql/mariadb/percona.
Even after many months I have reported this bug it was not fixed (we are using latest Mariadb)
The second error happens because the table (or the fulltext index table) was partially modified (or created) when the server crashed. Drop and re-create your table from scratch.
Now, why did the server crash? Hard to tell for sure, but it is likely that some buffer reached capacity. The usual suspect is innodb_buffer_pool_size. Try to increase it progressively.
On Ubuntu server 20.04,
despite reasonable logic to increase innodb_buffer_pool_size I have idenified that OOM Killer killed mysql when that buffer is huge.
OOM Killer is a function of the Linux kernel that is intended to kill rogue processes that are requesting more memory that the OS can allocate, so that the system can survive.
Excerpt from syslog:
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mysql.service,task=mysqld,pid=98524,uid=114
Jan 10 06:55:40 vps-5520a8d5 kernel: [66118.230690] Out of memory:
Killed process 98524 (mysqld) total-vm:9221052kB, anon-rss:5461436kB,
file-rss:0kB, shmem-rss:0kB, UID:114 pgtables:11232kB oom_score_adj:0
So this means, due tu huge innodb_buffer_pool_size setting, it is trying to allocate too big chunks of memory, too much agressiveley.
So using that fact, logic dictates to reduce size or set default values for innodb_buffer_pool_size, I have completley removed it from /etc/mysql/mysql.conf.d/mysqld.conf then restarted server with
systemctl restart mysql
and now adding fulltext on my 61M table is not crashing.
Bad luck pal...
InnoDB tables do not support FULLTEXT indexes.
Source - http://dev.mysql.com/doc/refman/5.5/en/innodb-restrictions.html