Optimizing rand query with join - mysql

Question

I have a rand query which runs very slow like almost every rand query. I researched all stackoverflow but cannot find any good solution for my query
SELECT u.id
, u.is_instagram_connected
, u.tokens
, u.username
, u.name
, u.photo
, u.bio
, u.voice
, u.mobile_update
, 1584450999 - l.time idleTime
FROM mobile_login_list l
JOIN users u
ON l.username = u.username
JOIN mobile_token_list t
ON t.username = l.username
WHERE l.time > 1584393399
AND l.username NOT IN ('enesdoo')
AND u.username NOT IN (
SELECT blocked_username
FROM hided_mobile_users_from_shuffle
WHERE username = 'enesdoo'
)
AND u.ban_status = 0
AND u.perma_ban = 0
AND u.mobile_online_status = 1
AND u.lock_status = 0
GROUP
BY l.username
ORDER
BY RAND( )
LIMIT 27
If i remove the order by rand line, this runs very very quick like 100 times faster.
How can i speed up this query?
mobile_login_list has > 50k rows
users has > 1m rows
Edit:
Explain:
My table:
CREATE TABLE IF NOT EXISTS `mobile_login_list` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`username` varchar(30) COLLATE utf8_bin NOT NULL,
`key` varchar(32) COLLATE utf8_bin NOT NULL,
`time` int(11) NOT NULL,
`ip` int(11) NOT NULL,
`version` smallint(4) NOT NULL,
`messaged` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
KEY `kontrol` (`username`,`key`),
KEY `username` (`username`),
KEY `time` (`time`),
KEY `username_2` (`username`,`time`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=3351637 ;

Answer 1

In the lingo of random retrieval, this is called a deal operation (deal 27 different cards from a shuffled deck of 4k or so. The other random operation is called roll: it allows duplicates.)
You're using SELECT mess-of-columns FROM mess-of-joins WHERE mess-of-criteria ORDER BY RAND() LIMIT small-number to do shuffle and deal operation. That is a notorious performance antipattern. It causes some extra work for the server because it must order a fairly large result set then discard almost all of it (with the LIMIT).
A way to save some of the trouble is to defer the joins to the details. Shuffle only the ids. Then take the small number of results and fetch the details you need. Something like this.
SELECT u.id /* just the id values */
FROM mobile_login_list l
JOIN users u
ON l.username = u.username
JOIN mobile_token_list t
ON t.username = l.username
WHERE l.time > 1584393399
AND l.username NOT IN ('enesdoo')
AND u.username NOT IN (
SELECT blocked_username
FROM hided_mobile_users_from_shuffle
WHERE username = 'enesdoo'
)
AND u.ban_status = 0
AND u.perma_ban = 0
AND u.mobile_online_status = 1
AND u.lock_status = 0
ORDER
BY RAND( )
LIMIT 27
You can debug, run EXPLAIN and optimize this subquery by changing indexes and maybe tightening up your selection criteria. It's the one doing all the hard work of shuffling and dealing.
Then join that resultset to your detail tables to choose the data you need. This outer query only needs to process your 27 rows. Be sure to shuffle again.
SELECT u.id
, u.is_instagram_connected
, u.tokens
, u.username
, u.name
, u.photo
, u.bio
, u.voice
, u.mobile_update
, 1584450999 - l.time idleTime
FROM mobile_login_list l
JOIN users u
ON l.username = u.username
JOIN (
/* the subquery from above */
) selected ON u.id = selected.id
ORDER BY RAND()
Putting it all together, you get this big repetitive mess of a query. But it should be a little faster.
SELECT u.id
, u.is_instagram_connected
, u.tokens
, u.username
, u.name
, u.photo
, u.bio
, u.voice
, u.mobile_update
, 1584450999 - l.time idleTime
FROM mobile_login_list l
JOIN users u
ON l.username = u.username
JOIN (
SELECT u.id
FROM mobile_login_list l
JOIN users u
ON l.username = u.username
JOIN mobile_token_list t
ON t.username = l.username
WHERE l.time > 1584393399
AND l.username NOT IN ('enesdoo')
AND u.username NOT IN (
SELECT blocked_username
FROM hided_mobile_users_from_shuffle
WHERE username = 'enesdoo'
)
AND u.ban_status = 0
AND u.perma_ban = 0
AND u.mobile_online_status = 1
AND u.lock_status = 0
ORDER
BY RAND( )
LIMIT 27
) selected ON u.id = selected.id
ORDER BY RAND()
A more performant way to deal records is this, if you do the dealing a lot.
Add a FLOAT column to the table you're dealing from, let's call it deal. Put an index on it.
Every few hours, or maybe overnight or even once a week, shuffle the table by running this query UPDATE users SET deal = RAND(); It will take a while; it needs to change the deal value in every row.
When you need to deal, do ...WHERE deal >= RAND() * 0.9 ... ORDER BY deal LIMIT n. The multiplication by 0.9 helps ensure you don't hit the end of the table by choosing a random number too close to 1.
This is equivalent, in cardshark terms, to shuffling the deck every few hours and then just cutting it for every deal. It's the way Wikipedia implements their "show a random article" feature.

Answer 2

Can we see the EXPLAIN for this instead...?
SELECT DISTINCT u.id
, u.is_instagram_connected
, u.tokens
, u.username
, u.name
, u.photo
, u.bio
, u.voice
, u.mobile_update
, 1584450999 - l.time idleTime
FROM mobile_login_list l
JOIN users u
ON l.username = u.username
JOIN mobile_token_list t
ON t.username = l.username
LEFT
JOIN hided_mobile_users_from_shuffle x
ON x.blocked_username = u.username
AND x.username = 'enesdoo'
WHERE l.time > 1584393399
AND l.username NOT IN ('enesdoo')
AND x.blocked_username IS NULL
AND u.ban_status = 0
AND u.perma_ban = 0
AND u.mobile_online_status = 1
AND u.lock_status = 0
ORDER
BY RAND( )
LIMIT 27
Given my limited knowledge of query optimisation, I would simply define the table as follows, but maybe someone else can suggest further improvements:
CREATE TABLE IF NOT EXISTS mobile_login_list
(id SERIAL PRIMARY KEY
,username varchar(30) COLLATE utf8_bin NOT NULL
,`key` varchar(32) COLLATE utf8_bin NOT NULL
,time int NOT NULL
,ip int NOT NULL
,version smallint NOT NULL
,messaged int NOT NULL DEFAULT 0
,KEY username_2 (username,time) -- or (time,username)
);
Note that key is a reserved word (and time is a 'keyword') rendering it a poor choice for a table/column identifier