I found a very slow MySQL query in my web app. The weird thing is that the query is only slow the first time it's executed, despite the fact that the query_cache is set to its default (query_cache_size 0) like so:
mysql> show variables like 'query%';
+------------------------------+---------+
| Variable_name | Value |
+------------------------------+---------+
| query_alloc_block_size | 8192 |
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 0 |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
| query_prealloc_size | 8192 |
+------------------------------+---------+
The even weirder thing is that this speedup persists even after the MySQL server has been stopped and restarted (I'm using OSX, and perform this restart using the system preferences pane.) The only way I can re-create the poor performance of the initial query is by rebooting the system.
So my question is: how is this happening? Obviously some sort of caching at work, but where? And how does it persist across database restarts? This query is mediated through our web app, which comes via PHP/Apache, but there are no extra bells and whistles, and the curious caching also persists across Apache restarts.
Help?
wild guess is operating system, or hard disk, caching the file
Related
When I Query ProxySQL Client Connections :
select * from stats.stats_mysql_global where variable_name like 'Client_connection%';
+-------------------------------------+----------------+
| Variable_Name | Variable_Value |
+-------------------------------------+----------------+
| Client_Connections_aborted | 0 |
| Client_Connections_connected | 495 |
| Client_Connections_created | 43785 |
| Client_Connections_non_idle | 495 |
| Client_Connections_hostgroup_locked | 0 |
+-------------------------------------+----------------+
Connections non idle is always same with connections non idle;
But when I query on Process List :
show processlist;
in the command fields, all status is Sleep.
The Client_Connections_non_idle must be zero, but not in my case.
What is wrong with my thought ?
Thanks for any explanation of my problems.
I am using ProxySql v2.4.2
According to the ProxySQL documentation
Client_Connections_non_idle : number of client connections that are currently handled by the main worker threads. If ProxySQL isn’t running with “–idle-threads”, Client_Connections_non_idle is always equal to “Client_Connections_connected”
So unless you run ProxySQL with --idle-threads this is expected behaviour and nothing to worry about. You can read more about idle threads in the documentation
I already find the solution of my problem. Before i set wait_timeout too big, after I set wait_timeout into 30000 (30 second) it is normal.
Many of us who are working on their home or pet projects and who use databases for storing structured data may encounter performance issues when trying to dump/restore data. It can annoying just to sit and wait for another dump restore operation for dozens of minutes or even for hours.
I have quite typical machine specs - 4 core i5 7300, 8 Gb RAM, quite fast M2 drive and Windows 10/MySQL 5.7.
The problem was that trying to restore ~4.5Gb file it took more than 4 hours. That was ridiculous and I wondered if mysqld process isn't using even a half of system resources - CPU/Memory/Disk I/O
Generally speaking, this post relates to some kind of summary of related issues including credits to many other posts which I put below
I performed a number of experiments with MySQL parameters for better dump restore operations
+--------------------------------+---------+---------+-----------------------+---------------------+
| Parameter | Default | Changed | Performance (minutes) | Perfomance gain (%) |
+--------------------------------+---------+---------+-----------------------+---------------------+
| All default | - | - | 259 min | - |
| innodb_buffer_pool_size | 8M | 4G | 32 min | +709% |
| innodb_buffer_pool_size | 4G | 6G | 32 min | ~0% |
| innodb_log_file_size | 48M | 1G | 11 min | +190% |
| innodb_log_file_size | 1G | 2G | 10 min | +10% |
| max_allowed_packet | 4M | 128M | 10 min | ~0% |
| innodb_flush_log_at_trx_commit | 1 | 0 | 9 min 25 sec | +5% |
| innodb_thread_concurrency | 9 | 0 | 9 min 27 sec | ~0% |
| innodb_double_write | - | off | 8 min 5 sec | +18% |
+--------------------------------+---------+---------+-----------------------+---------------------+
Summary (for best dump restore performance):
Set innodb_buffer_pool_size to half of RAM
Set innodb_log_file_size to 1G
Set innodb_flush_log_at_trx_commit to 0
Disabling innodb_double_write recommended only for fastest performance, it should be enabled on production. I also found, that changing another related parameter innodb_flush_method didn't change performance. But this can be an issue of Windows platform.
If you have complex structure with a lot of foreign keys for example, you can try Bulk Data Loading for InnoDB Tables tricks, link is listed at bottom of page
As you can see, I tried to increase CPU utilization by setting innodb_thread_concurrency to 0 (and also setting innodb_read_io_threads to maximum of 64) but results didn't change - it seems that mysqld process is already quite efficient for multi-core environment.
Restoring only data (without table structure) also didn't affect performance
I also changed a number of other parameters, but those above are most relevant ones for dump restore operation so far.
It may seem obvious, but novice question can be - where I can find and set those settings?
In Windows, my.ini file is located at ProgramData/MySQL/MySQL Server <version>/my.ini. You won't find some settings there (like innodb_double_write) - it's ok, just add to the end of the file.
The best way to change settings is to use MySQL Workbench (Server > Options file > InnoDB).
I pay my credits to following posts (and a lot of similar ones), which I found very useful:
https://www.percona.com/blog/2018/02/22/restore-mysql-logical-backup-maximum-speed/
https://www.percona.com/blog/2014/01/28/10-mysql-performance-tuning-settings-after-installation/
https://dev.mysql.com/doc/refman/5.5/en/optimizing-innodb-bulk-data-loading.html
https://dba.stackexchange.com/questions/86636/when-is-it-safe-to-disable-innodb-doublewrite-buffering
https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html
My mariadb is increase disk usage every day until full the disk up. Actually data inserting to the other disk.
Only reclaim the used disk space when stop mariadb daemon.(or restart)
Mariadb is work fine until the disk full up.
Mariadb Env.
Version : 10.3.x
OS : Ubuntu 18.04 LTS
Related DB Configurations.
| Variable_name | Value |
+------------------------+--------------------------------------+
| basedir | /usr |
| innodb_temp_data_file_path | ibtmp1:12M:autoextend |
| datadir | /mysql_data |
| innodb_tmpdir | |
| max_tmp_tables | 32 |
| slave_load_tmpdir | /mysql_data/mysql_tmp |
| tmp_disk_table_size | 4294967295 |
| tmp_memory_table_size | 33554432 |
| tmp_table_size | 33554432 |
| tmpdir | /mysql_data/mysql_tmp |
+----------------------------+----------------------------------+
'/mysql_data' is physically separated disk with '/'.
I understand that increased '/mysql_data' disk by inserted data but don't under stand why increased every day about 2G size usage to '/' disk.
When stop mariadb daemon then reclaim the space clearly at the disk '/'.
Everyday full backup the database by mysqldump command to the '/mysql_data' disk.
Searching mariadb log but there's noting error or noticeable messages.
I think mariadb daemon is grabbed system space('/'), but don't know why.
I need your help. How can i solve this?
I have a MySQL 5.6.27-0ubuntu0.14.04.1 that run on a Google Compute instance with 4 CPU.
I noticed that if I have a connection that Sleep for a long time, then the CPU of the server will increase in a linear way. I don't understand why? If I kill the Sleep connection then CPU just restore to a correct usage.
So to summary I have the following:
I notice the CPU of my instance is increasing:
Then I check the processlist on my server
mysql> show processlist
-> ;
+-------+--------+-------------------+----------------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+--------+-------------------+----------------+---------+------+-------+------------------+
| 85949 | nafora | paper-eee-2:58461 | state_recorder | Sleep | 1300 | | NULL |
| 85956 | nafora | paper-eee-2:58568 | state_recorder | Sleep | 64 | | NULL |
| 85959 | root | localhost | NULL | Query | 0 | init | show processlist |
+-------+--------+-------------------+----------------+---------+------+-------+------------------+
You can see I have just 2 connection that Sleep and one is here from 1300 seconds (because I have a process that is stuck with the connection open)
So I kill the connection 85949, and the CPU just fall down.
Can someone explain me why a single connection that is sleeping can impact my database like this.
Thanks.
Some non-closed connections or long running slow queries might cause this behavior. You could limit the non-closed connections by configuring the global variable wait_timeout as reasonable value and also set another related variable interactive_timeout as high as per best practice.
Stateful applications that use a connection pool (Java, .NET, etc.) will need to adjust wait_timeout to match their connection pool settings. The default 8 hours (wait_timeout = 28800) works well with properly configured connection pools.
Configure the wait_timeout to be slightly longer than the application connection pool’s expected connection lifetime. This is a good safety check. Also profile the queries accordingly to observe the performance of MySQL instance can help you to avoid I/O bottlenecks.
I was working with netbean's database explorer (services tab) and accidentally deleted one of the databases.
I was trying to connect to a database and there were 2 options in the context menu for connecting.
1)Connect
2)Delete
I accidentally hit delete. It did not even ask me for confirmation. Is there anyway to recover the database?
I don't have access to mysql server. I can just connect to the database using mysql workbench.
Thanks a ton!
If you have the binary logging enabled, then you might have a chance to use it to restore your database.
Else use an older backup and restore it !
To find binary logs status :
mysql> show variables like '%bin%';
+---------------------------------+----------------------+
| Variable_name | Value |
+---------------------------------+----------------------+
| binlog_cache_size | 32768 |
| innodb_locks_unsafe_for_binlog | OFF |
| log_bin | OFF |
| log_bin_trust_function_creators | OFF |
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_size | 1073741824 |
| sync_binlog | 0 |
+---------------------------------+----------------------+
7 rows in set (0.00 sec)