Weird behavior optimizing query indices (MariaDB + InnoDB) - mysql

Question

I'm currently trying to optimize the indices for a quite large table of a project and experiencing a very counter intuitive behavior between the explain result and the actual query runtime.
The server is running MariaDB version 10.1.26-MariaDB-0+deb9u1 with the following configuration options:
key_buffer_size = 5G
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
myisam_sort_buffer_size = 512M
read_buffer_size = 2M
read_rnd_buffer_size = 1M
query_cache_type = 0
query_cache_limit = 256K
query_cache_min_res_unit = 2k
query_cache_size = 0M
join_buffer_size = 8M
sort_buffer_size = 8M
tmp_table_size = 64M
max_heap_table_size = 64M
table_open_cache = 4K
performance_schema = ON
innodb_buffer_pool_size = 30G
innodb_log_buffer_size = 4MB
innodb_log_file_size = 1G
innodb_buffer_pool_instances = 10
The table looks contains about 6.8 million rows summing up to 12.1GB and looks like this:
CREATE TABLE `ad_master_test` (
`ID_AD_MASTER` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
/* Some more attribute fields (mainly integers) ... */
`FK_KAT` BIGINT(20) UNSIGNED NOT NULL,
/* Some more content fields (mainly varchars/integers) ... */
`STAMP_START` DATETIME NULL DEFAULT NULL,
`STAMP_END` DATETIME NULL DEFAULT NULL,
PRIMARY KEY (`ID_AD_MASTER`),
INDEX `TEST1` (`STAMP_START`, `FK_KAT`),
INDEX `TEST2` (`FK_KAT`, `STAMP_START`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
ROW_FORMAT=DYNAMIC
AUTO_INCREMENT=14149037;
I already simplyfied the query as far as possible to better illustrate the Problem. I'm using FORCE INDEX to illustrate my issue here.
This first index is optimized using the explain statement and looks pretty promising (regarding the explain output):
SELECT *
FROM `ad_master_test`
FORCE INDEX (TEST1)
WHERE FK_KAT IN
(94169,94163,94164,94165,94166,94167,94168,94170,94171,94172,
94173,94174,94175,94176,94177,94162,99606,94179,94180,94181,
94182,94183,94184,94185,94186,94187,94188,94189,94190,94191,
94192,94193,94194,94195,94196,94197,94198,94199,94200,94201,
94202,94203,94204,94205,94206,94207,94208,94209,94210,94211,
94212,94213,94214,94215,94216,94217,94218,94219,94220,94221,
94222,94223,94224,94225,94226,94227,94228,94229,94230,94231,
94232,94233,94234,94235,94236,94237,94238,94239,94240,94241,
94178,94161)
ORDER BY STAMP_START DESC
LIMIT 24
Results in this explain:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE ad_master_test index (NULL) TEST1 14 (NULL) 24 Using where
And this profile:
Status Duration
starting 0.000180
checking permissions 0.000015
Opening tables 0.000041
After opening tables 0.000013
System lock 0.000011
Table lock 0.000013
init 0.000115
optimizing 0.000044
statistics 0.000050
preparing 0.000039
executing 0.000009
Sorting result 0.000016
Sending data 4.827512
end 0.000023
query end 0.000008
closing tables 0.000004
Unlocking tables 0.000014
freeing items 0.000011
updating status 0.000132
cleaning up 0.000021
The second index is just the fields reversed (the way I understood it here: https://dev.mysql.com/doc/refman/8.0/en/order-by-optimization.html ) which looks pretty horrible (regarding the explain output):
SELECT *
FROM `ad_master_test`
FORCE INDEX (TEST2)
WHERE FK_KAT IN (94169,94163,94164,94165,94166,94167,94168,94170,94171,94172,94173,94174,94175,94176,94177,94162,99606,94179,94180,94181,94182,94183,94184,94185,94186,94187,94188,94189,94190,94191,94192,94193,94194,94195,94196,94197,94198,94199,94200,94201,94202,94203,94204,94205,94206,94207,94208,94209,94210,94211,94212,94213,94214,94215,94216,94217,94218,94219,94220,94221,94222,94223,94224,94225,94226,94227,94228,94229,94230,94231,94232,94233,94234,94235,94236,94237,94238,94239,94240,94241,94178,94161)
ORDER BY STAMP_START DESC
LIMIT 24
Results in this explain:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE ad_master_test range TEST2 TEST2 8 (NULL) 497.766 Using index condition; Using filesort
And this profile:
Status Duration
starting 0.000087
checking permissions 0.000007
Opening tables 0.000021
After opening tables 0.000007
System lock 0.000006
Table lock 0.000005
init 0.000058
optimizing 0.000023
statistics 0.000654
preparing 0.000480
executing 0.000008
Sorting result 0.433607
Sending data 0.001681
end 0.000010
query end 0.000007
closing tables 0.000003
Unlocking tables 0.000011
freeing items 0.000010
updating status 0.000158
cleaning up 0.000021
Edit: When not using force index the explain changes as following:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE ad_master_test index TEST2 TEST1 14 (NULL) 345 Using where
The profile and runtime stays (as expected) the same it was when using FORCE INDEX on the TEST1 index.
/Edit
I honestly can't wrap my head around this. Why does the explain and the actual query performance differ that extremely. What does the server do while the 5 seconds "Sending data"?

Answer 1

It looks like there are some TEXT or BLOB or even large VARCHAR columns?? 12.1GB/6.8M = 1.8KB. If you don't need them, don't fetch them; this may speed up any such query. How much RAM do you have?
The two indexes do seem to take a different time (4.8s vs 0.4s).
(STAMP_START, FK_KAT)
This avoids the "filesort" by scanning the index BTree in the desired order. It has to check every entry for a matching fk_kat. I think it will stop after 24 (see LIMIT) matching rows, but that could be the first 24 (fast), the last 24 (very slow), or something in between.
(FK_KAT, STAMP_START)
This show go directly to all 82 ids, scan each (assuming not unique), collecting perhaps hundreds of rows. Then do a "filesort". (Note: this will be a disk sort if any TEXT columns are being fetched.) Then deliver the first 24. (Oops; I don't think MariaDB 10.1 has that feature.)
Even though this takes more steps, by avoiding the full index scan it turns out to be faster.
Other Notes
key_buffer_size = 20G - Don't use MyISAM. But if you do, change this to 10% of RAM. If you don't, change it to 30M and give 70% of RAM to innodb_buffer_pool_size.
If you want to discuss further, please provide EXPLAIN FORMAT=JSON SELECT ... for each query. This will have the "cost" analysis, which should explain why it picked the worse index.
Another experiment
Instead of SELECT *, run the timings and EXPLAINs with just SELECT ID_AD_MASTER. If that proves to be "fast", then reformulate the query thus:
SELECT b.* -- (or selected columns from `b`)
FROM ( SELECT ID_AD_MASTER FROM ... ) AS a
JOIN ad_master_test AS b USING(ad_master_test)
ORDER BY STAMP_START DESC ; -- (yes, repeat the ORDER BY)

Answer 2

Suggestions to consider for your my.cnf [mysqld] section
(RPS is RatePerSecond)
thread_handling=pool-of-threads # from one-thread-per-connection see refman
max_connections=100 # from 151 because max_used_connections < 60
read_rnd_buffer_size=256K # from 1M to reduce RAM used, < handler_read_rnd_next RPS
aria_pagecache_division_limit=50 # from 100 for WARM cache for < aria_pagecache_reads RPS
key_cache_division_limit=50 # from 100 for WARM cache for < key_reads
key_buffer_size=2G # from 5G Mysqltuner reports 1G used (this could be WRONG-test it)
innodb_io_capacity=30000 # from 200 since you have SSD
innodb_buffer_pool_instances=8 # from 16 for your volume of data
innodb_lru_scan_depth=128 # from 1024 to conserve CPU every SECOND see refman
innodb_buffer_pool_size=36G # from 30G for effective size of 32G when
innodb_change_buffer_pool_size=10 # from 25% set aside for Del,Ins,Upd activities
for additional suggestions, view profile, Network profile, for contact info including my Skype ID. There are additional opportunities available to improve your configuration.
Remember the advice ONLY one change per day, monitor, if positive results proceed to next suggestion. Otherwise let me know any seriously adverse result and which change seemed to cause the problem, please.

Answer 3

Analysis of VARIABLES and GLOBAL STATUS:
Observations:
Version: 10.1.26-MariaDB-0+deb9u1
64 GB of RAM
Uptime = 7d 22:50:19
You are not running on Windows.
Running 64-bit version
You appear to be running entirely (or mostly) InnoDB.
The More Important Issues:
A "Load Average" of 1 (or more) usually indicates an inefficient query. This is further confirmed by the large value for Created_tmp_disk_tables and Handler_read_rnd_next for a "mere" 91 queries per second. Let's see the slowest query. See Recommendations for further investiataion.
thread_cache_size = 20
Having gotten rid of MyISAM, there is no need to such a large key_buffer_size; decrease from 5G to 50M.
I'm not a fan of ROW_FORMAT=COMPRESSED; this has two relevant impacts for your Question: Increased CPU for compress/uncompress, and need for extra buffer_pool space. On the other hand, the GLOBAL STATUS does not indicate that 30GB is "too small". Is there a need for shrinking the disk space usage?
You have turned off some optimizations? Was this in response to some other problem?
Details and other observations:
( (key_buffer_size - 1.2 * Key_blocks_used * 1024) / _ram ) = (5120M - 1.2 * 25 * 1024) / 65536M = 7.8% -- Percent of RAM wasted in key_buffer.
-- Decrease key_buffer_size.
( Key_blocks_used * 1024 / key_buffer_size ) = 25 * 1024 / 5120M = 0.00% -- Percent of key_buffer used. High-water-mark.
-- Lower key_buffer_size to avoid unnecessary memory usage.
( innodb_buffer_pool_size / _ram ) = 30720M / 65536M = 46.9% -- % of RAM used for InnoDB buffer_pool
( table_open_cache ) = 4,096 -- Number of table descriptors to cache
-- Several hundred is usually good.
( Innodb_os_log_written / (Uptime / 3600) / innodb_log_files_in_group / innodb_log_file_size ) = 6,714,002,432 / (687019 / 3600) / 2 / 1024M = 0.0164 -- Ratio
-- (see minutes)
( Uptime / 60 * innodb_log_file_size / Innodb_os_log_written ) = 687,019 / 60 * 1024M / 6714002432 = 1,831 -- Minutes between InnoDB log rotations Beginning with 5.6.8, this can be changed dynamically; be sure to also change my.cnf.
-- (The recommendation of 60 minutes between rotations is somewhat arbitrary.) Adjust innodb_log_file_size. (Cannot change in AWS.)
( default_tmp_storage_engine ) = default_tmp_storage_engine =
( Innodb_rows_deleted / Innodb_rows_inserted ) = 1,319,619 / 2015717 = 0.655 -- Churn
-- "Don't queue it, just do it." (If MySQL is being used as a queue.)
( innodb_thread_concurrency ) = 0 -- 0 = Let InnoDB decide the best for concurrency_tickets.
-- Set to 0 or 64. This may cut back on CPU.
( innodb_print_all_deadlocks ) = innodb_print_all_deadlocks = OFF -- Whether to log all Deadlocks.
-- If you are plagued with Deadlocks, turn this on. Caution: If you have lots of deadlocks, this may write a lot to disk.
( innodb_buffer_pool_populate ) = OFF = 0 -- NUMA control
( query_prealloc_size / _ram ) = 24,576 / 65536M = 0.00% -- For parsing. Pct of RAM
( query_alloc_block_size / _ram ) = 16,384 / 65536M = 0.00% -- For parsing. Pct of RAM
( net_buffer_length / max_allowed_packet ) = 16,384 / 16M = 0.10%
( bulk_insert_buffer_size / _ram ) = 8M / 65536M = 0.01% -- Buffer for multi-row INSERTs and LOAD DATA
-- Too big could threaten RAM size. Too small could hinder such operations.
( Created_tmp_tables ) = 19,436,364 / 687019 = 28 /sec -- Frequency of creating "temp" tables as part of complex SELECTs.
( Created_tmp_disk_tables ) = 17,887,832 / 687019 = 26 /sec -- Frequency of creating disk "temp" tables as part of complex SELECTs
-- increase tmp_table_size and max_heap_table_size.
Check the rules for temp tables on when MEMORY is used instead of MyISAM. Perhaps minor schema or query changes can avoid MyISAM.
Better indexes and reformulation of queries are more likely to help.
( Created_tmp_disk_tables / Questions ) = 17,887,832 / 62591791 = 28.6% -- Pct of queries that needed on-disk tmp table.
-- Better indexes / No blobs / etc.
( Created_tmp_disk_tables / Created_tmp_tables ) = 17,887,832 / 19436364 = 92.0% -- Percent of temp tables that spilled to disk
-- Maybe increase tmp_table_size and max_heap_table_size; improve indexes; avoid blobs, etc.
( tmp_table_size ) = 64M -- Limit on size of MEMORY temp tables used to support a SELECT
-- Decrease tmp_table_size to avoid running out of RAM. Perhaps no more than 64M.
( Handler_read_rnd_next ) = 703,386,895,308 / 687019 = 1023824 /sec -- High if lots of table scans
-- possibly inadequate keys
( Handler_read_rnd_next / Com_select ) = 703,386,895,308 / 58493862 = 12,024 -- Avg rows scanned per SELECT. (approx)
-- Consider raising read_buffer_size
( Select_full_join ) = 15,981,913 / 687019 = 23 /sec -- joins without index
-- Add suitable index(es) to tables used in JOINs.
( Select_full_join / Com_select ) = 15,981,913 / 58493862 = 27.3% -- % of selects that are indexless join
-- Add suitable index(es) to tables used in JOINs.
( Select_scan ) = 1,510,902 / 687019 = 2.2 /sec -- full table scans
-- Add indexes / optimize queries (unless they are tiny tables)
( sort_buffer_size ) = 8M -- One per thread, malloced at full size until 5.6.4, so keep low; after that bigger is ok.
-- This may be eating into available RAM; recommend no more than 2M.
( binlog_format ) = binlog_format = STATEMENT -- STATEMENT/ROW/MIXED. ROW is preferred; it may become the default.
( slow_query_log ) = slow_query_log = OFF -- Whether to log slow queries. (5.1.12)
( long_query_time ) = 10 -- Cutoff (Seconds) for defining a "slow" query.
-- Suggest 2
( Threads_created / Connections ) = 3,081 / 303642 = 1.0% -- Rapidity of process creation
-- Increase thread_cache_size (non-Windows)
Abnormally large:
Connection_errors_peer_address = 2
Handler_icp_attempts = 71206 /sec
Handler_icp_match = 71206 /sec
Handler_read_next / Handler_read_key = 283
Handler_read_prev = 12522 /sec
Handler_read_rnd_deleted = 16 /sec
Innodb_rows_read = 1255832 /sec
Key_blocks_unused = 4.24e+6
Performance_schema_table_instances_lost = 32
Select_range / Com_select = 33.1%
Sort_scan = 27 /sec
Tc_log_page_size = 4,096
innodb_lru_scan_depth / innodb_io_capacity = 5.12
innodb_max_dirty_pages_pct_lwm = 0.10%
max_relay_log_size = 100MB
myisam_sort_buffer_size = 512MB
Abnormal strings:
Compression = ON
innodb_cleaner_lsn_age_factor = HIGH_CHECKPOINT
innodb_empty_free_list_algorithm = BACKOFF
innodb_fast_shutdown = 1
innodb_foreground_preflush = EXPONENTIAL_BACKOFF
innodb_log_checksum_algorithm = INNODB
myisam_stats_method = NULLS_UNEQUAL
opt_s__engine_condition_pushdown = off
opt_s__mrr = off
opt_s__mrr_cost_based = off