Efficient multiple subqueries from multiple tables with OR and ORDER BY/LIMIT - mysql

Question

Question covers doubts on efficient SQL query for multiple subqueries:
I have 3 tables. I want to get details from table 1, based on filtering done from table 2 and table 3. Currently I am using IN clause on table 2 and table 3 but it takes around 6 seconds for 2M users. I tried join also but it was slower than subquery.
Table1:
mysql> describe users;
Field | Type | Null | Key | Default
| uuid | varchar(36) | NO | PRI | NULL
| firstname | varchar(512) | YES | | NULL
| status | varchar(512) | YES | | NULL
| createdAt | timestamp | YES | | CURRENT_TIMESTAMP
Table 2:
describe homes;
| Field | Type | Null | Key | Default | Extra
| uuid | varchar(50) | NO | PRI | NULL
| phoneNumberHash | varchar(512) | YES | MUL | NULL
| secondaryPhoneNumberHash | varchar(512) | YES | MUL | NULL
Table 3:
describe utility_tags:
| Field | Type | Null | Key | Default |
| tag_name | varchar(50) | NO | MUL | NULL |
| tag_value | varchar(50) | NO | MUL | NULL |
| user_id | varchar(50) | NO | MUL | NULL |
I have index on all the required fields ie.
User Table : Index on uuid
Home Table : Separate Index on phoneNumberHash and secondaryPhoneNumberHash
Utility_Tags: Separate Index on tag_name and tag_value
Query I am running:
SELECT uuid, firstname
FROM users
WHERE ( uuid in (
SELECT `uuid`
FROM `homes`
WHERE ( ( `phoneNumberHash` = '02c' OR `secondaryPhoneNumberHash` = '02c' ))
)
OR uuid in (
SELECT `user_id`
FROM `utility_tags`
WHERE ( `tag_name` = 'ACCOUNT_NUMBER' AND `tag_value`= '13' )
))
AND `status` != 'DELETED'
ORDER BY `createdAt` DESC LIMIT 10 OFFSET 0;
The query is slow and takes around 6 sec when there are 2M rows in user and homes table.
I tried join query:
SELECT users.uuid, firstname
FROM users inner join homes on homes.uuid=users.uuid
inner join utility_tags on utility_tags.user_id=users.uuid
WHERE ( phoneNumberHash = '02c' OR secondaryPhoneNumberHash = '02cd0' )
OR ( tag_name = 'ACCOUNT_NUMBER' AND tag_value= '1311851988' )
AND `status` != 'DELETED'
ORDER BY `createdAt` DESC
LIMIT 10 OFFSET 0;
This takes around 30 seconds.
Any help is highly appreciated.

Answer 1

You are selecting certain rows from your users table based on matches in your other tables. You're using a complex IN( ... ) clause for that.
Let's look at the contents of that clause for optimization possibilities. Here's one way you generate a set of uuid values.
SELECT uuid
FROM homes
WHERE phoneNumberHash = '02c'
OR secondaryPhoneNumberHash = '02c'
Here's the other
SELECT user_id
FROM utility_tags
WHERE tag_name = 'ACCOUNT_NUMBER'
AND tag_value= '13'
Let's recast all this as a UNION of several sets of uuid values, like this.
SELECT uuid FROM homes WHERE phoneNumberHash = '02c'
UNION
SELECT uuid FROM homes WHERE secondaryPhoneNumberHash = '02c'
UNION
SELECT user_id AS uuid
FROM utility_tags
WHERE tag_name = 'ACCOUNT_NUMBER'
AND tag_value= '13'
That union of three queries does the same thing as all your OR clauses. The first two of those queries should (if you're using InnoDB) be optimized by the indexes on phoneNumberHash and secondaryPhoneNumberHash respectively. The third query in that union needs a compound index on (tag_name, tag_value, user_id) to perform efficiently.
The cool thing about UNION is it does the same sort of set creation as OR, but lets you write queries within the UNION that are more likely to use indexes. I suggest you experiment with this UNION query and appropriate indexes until you're happy with its performance. Then you can use it in your outer query.
(It's possible that the query planner has become smart enough to handle phoneNumberHash = '02c' OR secondaryPhoneNumberHash = '02c' as a UNION all by itself, exploiting your two indexes one after the other. Recent MySQL versions have made great progress in query planning.)
So that leaves us with the outer query:
SELECT uuid, firstname
FROM users
WHERE matching uuids
AND status != 'DELETED'
ORDER BY createdAt DESC
LIMIT 10 OFFSET 0
This is hard to make sargable. The query planner doesn't like != operators. It likes = best because index equality scans are cheap. It likes <, <=, >=, and > OK because range scans are almost as cheap. But you're stuck with !=.
Also, the query planner hates ORDER BY ... LIMIT because it has to sort a whole mess of rows just to discard all except a tiny number.
The following compound covering index MAY optimize this query: (createdAt, status, uuid, firstname). The query planner may be able to dodge the separate ORDER BY if it has an index that provides both the match criteria and the needed results. It's also possible that this index will be better. (createdAt, status, uuid, status, firstname) You'll need to try them both. Don't keep them both, only the one that helps best.
Putting it all together:
SELECT u.uuid, u.firstname
FROM users u
JOIN (
SELECT uuid FROM homes WHERE phoneNumberHash = '02c'
UNION
SELECT uuid FROM homes WHERE secondaryPhoneNumberHash = '02c'
UNION
SELECT user_id AS uuid
FROM utility_tags
WHERE tag_name = 'ACCOUNT_NUMBER'
AND tag_value= '13'
) s ON s.uuid = u.uuid
WHERE status != 'DELETED'
ORDER BY createdAt DESC
LIMIT 10 OFFSET 0
Things get interesting on megarow tables when you want subsecond query response. http://use-the-index-luke.com/ is a fine reference for this stuff.

Answer 2

Your main problem is you're selecting from users first - move it to last so its index can be used (subqueries can't be indexed).
Also, SQL OR is notorious, mainly because (almost always) at most 1 index can be used.
Select from the subquery first, so the index into users can be used
Ensure there are indexes on all looked-up columns, ie (uuid), (phoneNumberHash), (secondaryPhoneNumberHash) and (tag_name, tag_value)
Break up your query to eradicate OR
Try this:
SELECT uuid, firstname
FROM (
SELECT uuid
FROM homes
WHERE phoneNumberHash = '02c'
UNION
SELECT uuid
FROM homes
WHERE secondaryPhoneNumberHash = '02c'
SELECT user_id
FROM utility_tags
WHERE tag_name = 'ACCOUNT_NUMBER'
AND tag_value = 13
) x
JOIN users ON users.uuid = x.uuid
AND status != 'DELETED'
ORDER BY createdAt DESC
LIMIT 10 OFFSET 0
Notice also that the test for status != 'DELETED' is in the join condition (not the WHERE clause), so it's executed at join time, not post-join, which will boost performance especially if there are a lot of deleted users.