Optimize Join, Sum, Subqueries - mysql

Question

I'm building a Tinder clone for a study project and I'm trying to do something very simple conceptually but it appears that my request is really too heavy.
Data Structure
I've created this simple fiddle to visualize the database structure.
I've tried to put indexes on user.id user.gender * user.orientation match.user1 match.user2 match.createdAt with no luck.
Expected result
I want to find the people who have the less number of matches depending on gender, orientation, lastLogin and calendar date.
Users musn't be part of more than 4 matches during 24h so I look for users with <= 3 matches during the last 24h.
Values in the following are hard coded for easy editing of the request and because I didn't took time to do this part for now.
A match is composed of 2 users (user1 and user2).
The limit of 4 matches on the same day is a sum of when they appear as user1 and user2.
SELECT total_sum, userId
FROM (
SELECT u.id as userId, u.orientation as userOrientation, u.gender as userGender, m1.sum1, m2.sum2, (m1.sum1 + m2.sum2) AS total_sum
FROM user u
INNER JOIN (
SELECT user1, COUNT(user1) as sum1
FROM `match`
WHERE createdAt > DATE('2017-12-11 00:00:00')
GROUP BY user1
) m1
ON m1.user1 = u.id
INNER JOIN (
SELECT user2, COUNT(user1) as sum2
FROM `match`
WHERE createdAt > DATE('2017-12-11 00:00:00')
GROUP BY user2
) m2
ON m2.user2 = u.id
WHERE u.gender IN ('female')
AND u.orientation IN ('hetero', 'bi')
AND u.lastLogin > 1512873464582
) as total
WHERE total_sum < 4
ORDER BY total_sum ASC
LIMIT 8
The issue
With tiny tables, request takes few ms but with medium tables (50k users, 200k matches), request takes ages (170s).
Optimizing
According to #Thorsten Kettner response, this is the explain plan of his request when I run it into my test db after setting the indexes he advised:
Solution
I've ended up doing something easier.
First I flatened my match table by removing user2 column. It double the size because now 1 match become 2 rows but allow me to do something very simpler and very efficient with proper indexes.
The first query is to manage users with no matches and the second one to handle user with matches. I don't have anymore the matchesLimit into the query as it add extra work for mysql and I just need to check the first result to see if matchNumber is <= 3.
(SELECT u.id, mc.id as nb_match, u.gender, u.orientation
FROM user u
LEFT JOIN match_composition mc
ON (mc.matchedUser = u.id AND mc.createdAt > DATE('2017-12-11 00:00:00'))
WHERE u.lastLogin > 1512931740721
AND u.orientation IN ('bi', 'hetero')
AND u.gender IN ('female')
AND mc.id IS NULL
ORDER BY u.lastLogin DESC)
UNION ALL
(SELECT u.id, count(mc.id) as nb_match, u.gender, u.orientation
FROM match_composition mc
JOIN user u
ON u.id = matchedUser
WHERE mc.createdAt > DATE('2017-12-11 00:00:00')
AND u.lastLogin > 1512931740721
AND u.orientation IN ('bi', 'hetero')
AND u.gender IN ('female')
GROUP BY matchedUser
ORDER BY nb_match ASC
LIMIT 8)
thanks for your help

Answer 1

A user can be matched as user1 or user2. We can use UNION ALL to get one record per user:
select user1 as userid from match union all select user2 as userid from match;
The complete query:
select
u.id as userid,
coalesce(um.total, 0) as total
from user u
left join
(
select userid, count(*) as total
from
(
select user1 as userid from match where createdat > date '2017-12-11'
union all
select user2 as userid from match where createdat > date '2017-12-11'
) m
group by userid
) um on um.userid = u.id
where u.gender IN ('female')
and u.orientation in ('hetero', 'bi')
and u.lastlogin > 1512873464582
and coalesce(um.total, 0) < 4
order by coalesce(um.total, 0);
You would have the following indexes for this:
create index idx_m1 on match (createdat, user1);
create index idx_m2 on match (createdat, user2);
create index idx_u on user (lastlogin, gender, orientation, id);

Answer 2

I guess you were right about your SQL skills. This is what I came up with:
SELECT u.id as userId,
u.orientation as userOrientation,
u.gender as userGender,
count(m.user1) total_sum
FROM user u
LEFT JOIN `match` m on (u.id in (m.user1, m.user2)
and m.createdAt > DATE('2017-12-11 00:00:00'))
WHERE u.gender IN ('female')
AND u.orientation IN ('hetero', 'bi')
AND u.lastLogin > 1512873464582
having count(m.user1) <=4
ORDER BY total_sum ASC
LIMIT 8;
Edit: Covered also cases with no matches
Try to play around with indexing match table columns user1, user1 and also with User table columns(or column combinations) you use in filters (gender for example), see what brings better performance.

Answer 3

From what you provide, I would create indexes on:
- match.user1
- match.user2
- match.createdAt
- user.id (unique, and probably a PK)
- user.lastLogin
I would also try to replace COUNT(user1) by COUNT(*), but it won't probably have a big impact.
Indexes on user.gender and user.orientation are probably useless: the efficiency of an index is somehow proportional to the variance of its underlying values. Therefore an index on a field with 2-3 distinct values is more costly than useful.
As for the DLL, try the following. I tried to force the filtering on user to be done BEFORE the joins with match, in case the query optimizer does not work properly (I have little experience with non MS databases)
SELECT total_sum, userId
FROM (SELECT u.id as userId, u.orientation as userOrientation, u.gender as userGender, m1.sum1, m2.sum2, (m1.sum1 + m2.sum2) AS total_sum
FROM (SELECT * FROM user
WHERE gender = 'female'
AND orientation IN ('hetero', 'bi')
AND lastLogin > 1512873464582
) u
INNER JOIN (SELECT user1, COUNT(*) as sum1
FROM `match`
WHERE createdAt > DATE('2017-12-11 00:00:00')
GROUP BY user1
) m1 ON m1.user1 = u.id
INNER JOIN (SELECT user2, COUNT(*) as sum2
FROM `match`
WHERE createdAt > DATE('2017-12-11 00:00:00')
GROUP BY user2
) m2 ON m2.user2 = u.id
) as total
WHERE total_sum < 4
ORDER BY total_sum ASC
LIMIT 8