Query doesn't end if limit is not added - AWS RDS MariaDB - mysql

Question

I am currently migrating some databases from a physical server running MySQl over to AWS RDS. Current server uses MySQL 5.5 and the new RDS one uses MariaDB 10.1. Everything has been going smoothly until I tried to run an application against the new server. The following query will not complete even given 10 minutes, unless I add a large limit to it.
SELECT al.*
FROM US.Products p
JOIN US.Products_Contributors pc ON p.Product_id = pc.Product_id
JOIN US.Contributors c ON pc.Contributor_Id = c.Contributor_Id
JOIN US.Products_Category pca ON pca.Product_id = p.Product_id
JOIN US.Categories ca ON ca.Category_Id = pca.Category_Id
JOIN US.Asset_Links al ON (al.Asset_link_Id = p.Product_id) OR (al.Asset_link_Id = c.Contributor_ID)
WHERE p.Product_ISBN13 is not null
AND (
ca.Category_Code_3 in ("JNF","JUV")
)
AND al.Asset_Link_Table in ("Contributors","Products")
AND al.Asset_id != 0
GROUP BY al.Asset_Links_Id;
The query on the old server completes in around 11 seconds. If I add 'LIMIT 900000' to the query on the new server it finishes in around 7 seconds. There are ~800,000 rows returned before the GROUP by and ~150,000 rows returned after the group by. If I put the LIMIT to anything over 900000 then the query will not complete.
Things I have tried:
Been through DB parameters and increased buffer sizes on new server to be the same as the old server
Profiling queries: All of the time is spent in 'Copying to tmp table', which is consistently the same amount of time when a LIMIT clause is added as long as it is below 900000.
EXPLAIN outputs the same on both servers
Repairing and optimising tables on new server
Increasing RDS instance size - its now db.m4.xlarge
Setting RDS disk to Provisioned IOPS at 1000 IOPS
Running query on a different MySQL 5.5 server and it runs the same as it does on my original source server (not RDS).
Installing MySQL 5.7 on my EC2 and it runs the query very slowly unless the LIMIT is added.
So it seems to me that this is an issue with the MySQL version being used? But why would adding the LIMIT resolve the issue? And why does it only work up to 900000.
Any help would be greatly appreciated!
Thanks
UPDATE: 16:19 17/12/2017
EXPLAIN on original server MySQL 5.5:
EXPLAIN on RDS MariaDB 10.1 (No LIMIT):
{
"query_block": {
"select_id": 1,
"filesort": {
"temporary_table": {
"function": "buffer",
"table": {
"table_name": "p",
"access_type": "ALL",
"possible_keys": ["PRIMARY", "ISBN", "Product_Id"],
"rows": 120108,
"filtered": 99.999,
"attached_condition": "(p.Product_ISBN13 is not null)"
},
"table": {
"table_name": "pc",
"access_type": "ref",
"possible_keys": ["Products_contributor", "Contributor_Id"],
"key": "Products_contributor",
"key_length": "4",
"used_key_parts": ["Product_Id"],
"ref": ["US.p.Product_Id"],
"rows": 1,
"filtered": 100
},
"table": {
"table_name": "c",
"access_type": "eq_ref",
"possible_keys": ["PRIMARY"],
"key": "PRIMARY",
"key_length": "4",
"used_key_parts": ["Contributor_Id"],
"ref": ["US.pc.Contributor_Id"],
"rows": 1,
"filtered": 100,
"using_index": true
},
"table": {
"table_name": "pca",
"access_type": "ref",
"possible_keys": ["Products_Category", "Category_Id"],
"key": "Products_Category",
"key_length": "4",
"used_key_parts": ["Product_Id"],
"ref": ["US.p.Product_Id"],
"rows": 2,
"filtered": 100
},
"table": {
"table_name": "ca",
"access_type": "eq_ref",
"possible_keys": ["PRIMARY"],
"key": "PRIMARY",
"key_length": "4",
"used_key_parts": ["Category_Id"],
"ref": ["US.pca.Category_Id"],
"rows": 1,
"filtered": 100,
"index_condition": "(ca.Category_Id = pca.Category_Id)",
"attached_condition": "(ca.Category_Code_3 in ('JNF','JUV'))"
},
"block-nl-join": {
"table": {
"table_name": "al",
"access_type": "ALL",
"possible_keys": ["Asset_Link_Id", "Asset_Id"],
"rows": 908975,
"filtered": 95.517,
"attached_condition": "((al.Asset_Link_Table in ('Contributors','Products')) and (al.Asset_Id <> 0))"
},
"buffer_type": "flat",
"buffer_size": "1024Kb",
"join_type": "BNL",
"attached_condition": "((al.Asset_Link_Id = p.Product_Id) or (al.Asset_Link_Id = pc.Contributor_Id))"
}
}
}
}
}
EXPLAIN RDS MariaDB 10.1 (LIMIT 908974):
{
"query_block": {
"select_id": 1,
"filesort": {
"temporary_table": {
"function": "buffer",
"table": {
"table_name": "p",
"access_type": "ALL",
"possible_keys": ["PRIMARY", "ISBN", "Product_Id"],
"rows": 120108,
"filtered": 99.999,
"attached_condition": "(p.Product_ISBN13 is not null)"
},
"table": {
"table_name": "pc",
"access_type": "ref",
"possible_keys": ["Products_contributor", "Contributor_Id"],
"key": "Products_contributor",
"key_length": "4",
"used_key_parts": ["Product_Id"],
"ref": ["US.p.Product_Id"],
"rows": 1,
"filtered": 100
},
"table": {
"table_name": "c",
"access_type": "eq_ref",
"possible_keys": ["PRIMARY"],
"key": "PRIMARY",
"key_length": "4",
"used_key_parts": ["Contributor_Id"],
"ref": ["US.pc.Contributor_Id"],
"rows": 1,
"filtered": 100,
"using_index": true
},
"table": {
"table_name": "pca",
"access_type": "ref",
"possible_keys": ["Products_Category", "Category_Id"],
"key": "Products_Category",
"key_length": "4",
"used_key_parts": ["Product_Id"],
"ref": ["US.p.Product_Id"],
"rows": 2,
"filtered": 100
},
"table": {
"table_name": "ca",
"access_type": "eq_ref",
"possible_keys": ["PRIMARY"],
"key": "PRIMARY",
"key_length": "4",
"used_key_parts": ["Category_Id"],
"ref": ["US.pca.Category_Id"],
"rows": 1,
"filtered": 100,
"index_condition": "(ca.Category_Id = pca.Category_Id)",
"attached_condition": "(ca.Category_Code_3 in ('JNF','JUV'))"
},
"range-checked-for-each-record": {
"keys": ["Asset_Link_Id", "Asset_Id"],
"table": {
"table_name": "al",
"access_type": "ALL",
"possible_keys": ["Asset_Link_Id", "Asset_Id"],
"key": "Asset_Id",
"key_length": "4",
"used_key_parts": ["Asset_Id"],
"rows": 908975,
"filtered": 95.517
}
}
}
}
}
}
What I did notice is that setting the limit to any number which is 1 less than the reported number of joined ros then the query uses "Range checked for each record (index map: 0x6)", where as on MySQL 5.5 it uses that if there is a limit or not. I have found that if I add force index(Asset_Link_Id) to the last join it will always use the "Range checked..." and then the query will complete.
Whilte modifying and optimising all the queries is the ideal solution it is not the best in this case. The reason I don't really want to modify the query as the server I am migrating has hundreds of dfferent scripts / applications on it and if I have to modify a lot of queries in lots of different applications then this is going to take me a very long time and I won't be able to meet the migration deadline. So at this point, if this behaviour cannot be controlled by a setting, then I will probably resort to using MySQL 5.5 on the new server instead of MariaDB 10.1.
Can it be explained why the query optimiser chooses a different route with a large / undefined limit in 5.7 where it didn't in 5.5? Also after reading about dynamic range and the join_buffer why it is slower using the buffer over the range? From what I have read I would have thought this was more performant?

Answer 1

(This Answer does not directly address the "why did this slow down" question, but, as a consolation prize, addresses other performance issues.)
I see what looks like two many-to-many mapping tables. The typical implementation of such is less efficient than it could be.
Please follow the tips in https://mariadb.com/kb/en/library/building-the-best-index-for-a-given-select/#many-to-many-mapping-table -- then see if performance improves with or without the LIMIT.
The EXPLAIN is likely to change; let's see it.
Profiling -- Yeah, that is usually useless; it has a couple of uninformative messages where it spends 99% of its time.
Do not increase buffer sizes to the point of causing swapping; that will hurt a lot.
On the versions that allow for such, please provide EXPLAIN FORMAT=JSON SELECT ...
OR is often a performance killer; turn it into UNION:
( SELECT ...
JOIN US.Asset_Links al ON al.Asset_link_Id = p.Product_id
...
) UNION DISTINCT
( SELECT ...
JOIN US.Asset_Links al al.Asset_link_Id = c.Contributor_ID
...
)
(Because of GROUP BY, I may not have mapped the OR into UNION correctly.)