Sorting left join results on large open schema tables - mysql

Question

I am designing an open schema database with the following table definitions
mysql> desc orders;
+-------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+---------+------+-----+---------+----------------+
| ID | int(11) | NO | PRI | NULL | auto_increment |
| json | text | NO | | NULL | |
+-------+---------+------+-----+---------+----------------+
mysql> desc ordersnames;
+-------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+----------------+
| ID | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(330) | NO | UNI | NULL | |
+-------+--------------+------+-----+---------+----------------+
with an index on name
mysql> desc orderskeys;
+-----------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+----------------+
| ID | int(11) | NO | PRI | NULL | auto_increment |
| reference | int(11) | NO | MUL | NULL | |
| nameref | int(11) | NO | MUL | NULL | |
| value | varchar(330) | NO | | NULL | |
+-----------+--------------+------+-----+---------+----------------+
with indices on:
reference,nameref,value
nameref,value
reference
All json fields (1 dimension only) have entry in the orderskeys table per existing field, whereby nameref is a reference to the field name as defined in ordersname.
I would typically query like this:
SELECT
orderskeysdeliveryPostcode.value deliveryPostcode,
orders.ID,
orderskeysCN.value CN
FROM
orders
JOIN ordersnames as ordersnamesCN
on ordersnamesCN.name = 'CN'
JOIN orderskeys as orderskeysCN
on orderskeysCN.nameref = ordersnamesCN.ID
and orderskeysCN.reference = orders.ID
and orderskeysCN.value = '10094'
JOIN ordersnames as ordersnamesdeliveryPostcode
on ordersnamesdeliveryPostcode.name = 'deliveryPostcode'
JOIN orderskeys as orderskeysdeliveryPostcode
on orderskeysdeliveryPostcode.nameref = ordersnamesdeliveryPostcode.ID
and orderskeysdeliveryPostcode.reference = orders.ID
order by deliveryPostcode
limit 0,1000
yielding a result set like this
+------------------+--------+-------+
| deliveryPostcode | ID | CN |
+------------------+--------+-------+
| NULL | 251018 | 10094 |
| NULL | 157153 | 10094 |
| NULL | 95419 | 10094 |
| B-5030 | 172944 | 10094 |
+------------------+--------+-------+
-> lightning fast even with 400k + orders records
However, not all record do contain all fields, so the above query will not yield the records that do not have a 'deliveryPostcode field', so I have to query like this
SELECT
orderskeysdeliveryPostcode.value deliveryPostcode,
orders.ID,
orderskeysCN.value CN
FROM
orders
JOIN ordersnames as ordersnamesCN
on ordersnamesCN.name = 'CN'
JOIN orderskeys as orderskeysCN
on orderskeysCN.nameref = ordersnamesCN.ID
and orderskeysCN.reference = orders.ID
and orderskeysCN.value = '10094'
JOIN ordersnames as ordersnamesdeliveryPostcode
on ordersnamesdeliveryPostcode.name = 'deliveryPostcode'
LEFT JOIN orderskeys as orderskeysdeliveryPostcode
on orderskeysdeliveryPostcode.nameref = ordersnamesdeliveryPostcode.ID
and orderskeysdeliveryPostcode.reference = orders.ID
limit 0,1000
-> equally fast, but as soon as I add an ORDER BY clause on the key value from a left joined table, mysql wants to do the sorting externally (temporary, filesort) instead of using an existing index.
SELECT
orderskeysdeliveryPostcode.value deliveryPostcode,
orders.ID,
orderskeysCN.value CN
FROM
orders
JOIN ordersnames as ordersnamesCN
on ordersnamesCN.name = 'CN'
JOIN orderskeys as orderskeysCN
on orderskeysCN.nameref = ordersnamesCN.ID
and orderskeysCN.reference = orders.ID
and orderskeysCN.value = '10094'
JOIN ordersnames as ordersnamesdeliveryPostcode
on ordersnamesdeliveryPostcode.name = 'deliveryPostcode'
LEFT JOIN orderskeys as orderskeysdeliveryPostcode
on orderskeysdeliveryPostcode.nameref = ordersnamesdeliveryPostcode.ID
and orderskeysdeliveryPostcode.reference = orders.ID
ORDER BY deliveryPostCode
limit 0,1000
-> very slow ...
In fact the sorting operation itself is not much different , as all NULL values for column deliveryPostcode would be at the beginning (ASC) or the end (DESC) while the rest of the dataset would have the same order as with JOIN instead of LEFT JOIN.
How can I query (and order) such tables efficiently? Do I need different relations or indices ?
Much obliged ...

Answer 1

With INNER JOINs, to reduce the number of lookups, MySQL is going to start with the table with the fewest rows (see the EXPLAIN result to see which table MySQL starts with).
If you order by anything other than a column in that first table, or there is no index to satisfy the ORDER BY clause on that first table, MySQL is going to have to do a filesort.
The use of a temporary table is much more likely when text columns are involved, and not just an in-memory temporary table, but a dreadful on-disk temporary table.
Use STRAIGHT_JOIN to force the order that MySQL performs inner joins.

Answer 2

I am not sure what logic do you have in some parts of your query.
I think it still can be optimized.
But just to resolve the issue you have, try just switch it to RIGHT JOIN for now:
SELECT
orderskeysdeliveryPostcode.value deliveryPostcode,
o.id,
o.CN
FROM orderskeys as orderskeysdeliveryPostcode
INNER JOIN ordersnames as ord_n
on ord_n.id = orderskeysdeliveryPostcode.nameref
AND ord_n.name = 'deliveryPostcode'
RIGHT JOIN (
SELECT
orders.ID,
orderskeysCN.CN
FROM
orders
LEFT JOIN
(SELECT
orderskeys.value as CN,
orderskeys.reference
FROM
orderskeys
INNER JOIN ordersnames as ordersnamesCN
ON ordersnamesCN.id = orderskeys.nameref
AND ordersnamesCN.name = 'CN'
WHERE orderskeys.value = '12209'
) as orderskeysCN
ON
orderskeysCN.reference = orders.ID
limit 0,1000
) as o
on
orderskeysdeliveryPostcode.reference = o.ID
ORDER BY deliveryPostCode;
and here is sqlfiddle we can play with. Just need you to add data inserts there.