Leaderboard position SQL optimization - mysql

I'm offering an experience leaderboard for a Discord bot I actively develop with stuff like profile cards showing one's rank. The SQL query I'm currently using works flawlessly, however I notice that this query takes a rather long processing time.
SELECT id,
discord_id,
discord_tag,
xp,
level
FROM (SELECT #rank := #rank + 1 AS id,
discord_id,
discord_tag,
xp,
level
FROM profile_xp,
(SELECT #rank := 0) r
ORDER BY xp DESC) t
WHERE discord_id = '12345678901';
The table isn't too big (roughly 20k unique records), but this query is taking anywhere between 300-450ms on average, which piles up relatively fast with a lot of concurrent requests.
I was wondering if this query can be optimized to increase performance. I've isolated this to this query, the rest of the MySQL server is responsive and swift.
I'd be happy about any hint and thanks in advance! :)

You're scanning 20,000 rows to assign "row numbers" then selecting exactly one row from it. You can use aggregation instead:
SELECT *, (
SELECT COUNT(*)
FROM profile_xp AS x
WHERE xp > profile_xp.xp
) + 1 AS rnk
FROM profile_xp
WHERE discord_id = '12345678901'
This will give you rank of the player. For dense rank use COUNT(DISTINCT xp). Create an index on xp column if necessary.

Not an answer; too long for a comment:
I usually write this kind of thing exactly the same way that you have done, because it's quick and easy, but actually there's a technical flaw with this method - although it only becomes apparent in certain situations.
By way of illustration, consider the following:
DROP TABLE IF EXISTS ints;
CREATE TABLE ints (i INT NOT NULL PRIMARY KEY);
INSERT INTO ints VALUES
(0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
Your query:
SELECT a.*
, #i:=#i+1 rank
FROM ints a
JOIN (SELECT #i:=0) vars
ORDER
BY RAND() DESC;
+---+------+
| i | rank |
+---+------+
| 3 | 4 |
| 2 | 3 |
| 5 | 6 |
| 1 | 2 |
| 7 | 8 |
| 9 | 10 |
| 4 | 5 |
| 6 | 7 |
| 8 | 9 |
| 0 | 1 |
+---+------+
Look, the result set isn't 'random' at all. rank always corresponds to i
Now compare that with the following:
SELECT a.*
, #i:=#i+1 rank
FROM
( SELECT * FROM ints ORDER by RAND() DESC) a
JOIN (SELECT #i:=0) vars;
+---+------+
| i | rank |
+---+------+
| 5 | 1 |
| 2 | 2 |
| 8 | 3 |
| 7 | 4 |
| 4 | 5 |
| 6 | 6 |
| 0 | 7 |
| 1 | 8 |
| 3 | 9 |
| 9 | 10 |
+---+------+

Assuming discord_id is the primary key for the table, and you're just trying to get one entry's "rank", you should be able to take a different approach.
SELECT px.discord_id, px.discord_tag, px.xp, px.level
, 1 + COUNT(leaders.xp) AS rank
, 1 + COUNT(DISTINCT leaders.xp) AS altRank
FROM profile_xp AS px
LEFT JOIN profile_xp AS leaders ON px.xp < leaders.xp
WHERE px.discord_id = '12345678901'
GROUP BY px.discord_id, px.discord_tag, px.xp, px.level
;
Note I have "rank" and "altRank". rank should give you a similar position to what you were originally looking for; your results could have fluctuated for "ties", this rank will always put tied players at their highest "tie". If 3 records tie for 2nd place, those (queried separately with this) will show 2nd place, the next xp down would should 5th place (assuming 1 in 1st, 2,3,4 in 2nd, 5 in 5th). The altRank would "close the gaps" putting 5 in the 3rd place "group".
I would also recommend an index on xp to speed this up further.

Related

Alternative to "ntile" for MySQL version lower than 8?

I am trying the below code, which analyses and scores customers based on recency, frequency and monetary value of transactions.
select customer_id, rfm_recency, rfm_frequency, rfm_monetary
from
(
select customer_id,
ntile(4) over (order by last_order_date) as rfm_recency,
ntile(4) over (order by count_order) as rfm_frequency,
ntile(4) over (order by sum_amount) as rfm_monetary
from
(
select customer_id,
max(local_date) as last_order_date,
count(*) as count_order,
sum(amount) as sum_amount
from transaction
group by customer_id) as T
) as P
However ntile is not available in my MySQL version (v5) as apparently it's a "window function" which works on v8+ only.
I can't find a working alternative to this function. I am very new to SQL so I'm having a hard time figuring it out myself.
Is there an ntile alternative that I can use? The code works fine if i remove the ntile segment.
You should really upgrade to MySQL 8.0 if you need features in MySQL 8.0. They are bound to be easier and more optimized.
I found a way to simulate the ntile query shown in the documentation:
SELECT
val,
ROW_NUMBER() OVER w AS 'row_number',
NTILE(2) OVER w AS 'ntile2',
NTILE(4) OVER w AS 'ntile4'
FROM numbers
WINDOW w AS (ORDER BY val);
Here's a solution:
SELECT val, #r:=#r+1 AS rownum,
FLOOR((#r-1)*2/9)+1 AS ntile2,
FLOOR((#r-1)*4/9)+1 AS ntile4
FROM (SELECT #r:=0,#n:=0) AS _init, numbers
The 2 and 4 factors are for the ntile(2) and ntile(4) respectively. The 9 value is because there are 9 rows in this example table. You must know the count of the table before you can run this query. The solution also requires user defined variables, which are always kind of tricky.
Result:
+------+--------+--------+--------+
| val | rownum | ntile2 | ntile4 |
+------+--------+--------+--------+
| 1 | 1 | 1 | 1 |
| 1 | 2 | 1 | 1 |
| 2 | 3 | 1 | 1 |
| 3 | 4 | 1 | 2 |
| 3 | 5 | 1 | 2 |
| 3 | 6 | 2 | 3 |
| 4 | 7 | 2 | 3 |
| 4 | 8 | 2 | 4 |
| 5 | 9 | 2 | 4 |
+------+--------+--------+--------+
I'll leave it as an exercise for you to adapt this technique to your query and your table, or to decide that it's time to upgrade to MySQL 8.0.
You can enumerate rows and use arithmetic. Unfortunately, you'll need to do this three times:
select floor(seqnum * 4 / #rn) as ntile_recency, t.*
from (select (#rn := #rn + 1) as seqnum, t.*
from (select customer_id, max(local_date) as last_order_date, count(*) as count_order,
sum(amount) as sum_amount
from transaction
group by customer_id
order by last_order_date
) t cross join
(select #rn := 0) params
) t;

combine 3 queries to one, SELECT / COUNT / INSERT

I need help to optimize my 3 queries into one.
I have 2 tables, the first has a list of image processing servers I use, so different servers can handle different simultaneous job loads at a time, so I have a field called quota as seen below.
First table name, "img_processing_servers"
| id | server_url | server_key | server_quota |
| 1 | examp.uu.co | X0X1X2XX3X | 5 |
| 2 | examp2.uu.co| X0X1X2YX3X | 3 |
The second table registers if there is a job being performed at this moment on the server
Second table, "img_servers_lock"
| id | lock_server | timestamp |
| 1 | 1 | 2020-04-30 12:08:09 |
| 2 | 1 | 2020-04-30 12:08:09 |
| 3 | 1 | 2020-04-30 12:08:09 |
| 4 | 2 | 2020-04-30 12:08:09 |
| 5 | 2 | 2020-04-30 12:08:09 |
| 6 | 2 | 2020-04-30 12:08:09 |
Basically what I want to achieve is that my image servers don't go past the max quota and crash, so the 3 queries I would like to combine are:
Select at least one server available that hasn't reached it's quota and then insert a lock record for it.
SELECT * FROM `img_processing_servers` WHERE
SELECT COUNT(timestamp) FROM `img_servers_lock` WHERE `lock_server` = id
! if the count is < than quota, go ahead and register use
INSERT INTO `img_servers_lock`(`lock_server`, `timestamp`) VALUES (id_of_available_server, now())
How would I go about creating this single query?
My goal is to keep my image servers safe from overload.
Join the two tables and put that into an INSERT query.
INSERT INTO img_servers_lock(lock_server, timestamp)
SELECT s.id, NOW()
FROM img_processing_servers s
LEFT JOIN img_servers_lock l ON l.lock_server = s.id
GROUP BY s.id
HAVING IFNULL(COUNT(l.id), 0) < s.server_quota
ORDER BY s.server_quota - IFNULL(COUNT(l.id), 0) DESC
LIMIT 1
The ORDER BY clause makes it select the server with the most available quota.
OK, so I encountered just a small addition that was giving me a bug and it was that the s.server_quota had to be added to GROUP BY for it to work in the HAVING
INSERT INTO img_servers_lock(lock_server, timestamp)
SELECT s.id, NOW()
FROM alpr_servers s
LEFT JOIN img_servers_lock l ON l.lock_server = s.id
GROUP BY s.id, s.server_quota
HAVING IFNULL(COUNT(l.id), 0) < s.server_quota
ORDER BY s.server_quota - IFNULL(COUNT(l.id), 0) DESC
LIMIT 1
Thanks again Barmar!

Retrieve distinct values without reducing number of results

I'm writing a MySQL request for retrieving data from a list of questions.
The table looks like this :
-----------------------------------------------------
| id | answer_name | rating | question_id | answers |
-----------------------------------------------------
Where several rows can have the same answer_name value, since several questions can be asked about the same answer.
Now, for retrieving the data I use a LIMIT clause which is calculated from ratings and the total number of rows.
For example, if I wanna get the data between 80% and 100% of rating, and there are 100 rows, I would use ORDER BY rating LIMIT 80, 20.
My problem is the following : I need to retrieve data with distinct values for answer_name column, but using a GROUP BY clause makes the number of result (e.g. of rows in the table) reduce cause of aggregation, causing the top percentages of rows to return nothing cause of searching rows at a limit that doesn't exist.
Does anyone know if there is a way to keep the number of results the same and still to retrieve distinct results for the answer_name column ?
EDIT :
Here are some sample rows and expected output :
game_data table :
-----------------------------------------------------
| id | answer_name | rating | question_id | answers |
|----|-------------|--------|-------------|---------|
| 1 | A. Merkel | 40 | 1 | [1,2,3] |
| 2 | A. Merkel | 45 | 2 | [2,3,4] |
| 3 | B. Clinton | 55 | 1 | [2,5,8] |
| 4 | B. Clinton | 50 | 2 | [3,5,8] |
| 5 | L. Messi | 17 | 4 | [7,8,9] |
| 6 | L. Messi | 18 | 5 | [7,8,9] |
| 7 | L. Messi | 25 | 6 | [7,8,9] |
| 8 | D. Beckham | 21 | 4 | [6,7,8] |
| 9 | D. Beckham | 52 | 5 | [6,7,8] |
| 10 | D. Beckham | 41 | 6 | [6,7,8] |
-----------------------------------------------------
Where answers is an array of ids referring to another table.
Let's say I wanna retrieve the 50% to 80% of the table, ordered by rating.
SELECT id FROM game_data GROUP BY answer_name ORDER BY rating LIMIT 5, 3
Here the problem is the GROUP BY answer_name is gonna reduce the number of rows of the table, and therefore instead of returning 3 results, will return an empty set.
Also, I want the selected value in the GROUP BY close to be randomly chosen.
Using group by like this goes against pretty much every instinct, but you said you want random values, so it's good enough.
select * from (
select q.*, #rank := #rank + 1 as rank
from (
select * from game_data
group by answer_name
order by rating desc
) q, (select #rank := 0) qq
) qqq
where rank between (#rank * .5) and (#rank * .8)
demo here
How does it work? First (in the innermost query) we group by your answer_name, to get your distinct results, and we order it by the rating as required.
Then in the query wrapping around that one, we give those results a ranking from 1 to however many rows are in the result. Once this level of the query completes, we know our best answer is answer 1, and our 'worst' answer is the last value of our #rank variable.
Then we get to the outermost query. We can use that #rank variable to determine our percentages, which we use to filter the where clause.
In all likelihood this will give you the same results each time you run the same query, but the values chosen are indeterminate - so it could change. If you want truly random (ie changes with each execution) that's a different kettle of fish altogether.
(note, this bit: , (select #rank := 0) qq is purely to initialise the variable)
Simple is That.
Use Group By 'id' not 'answer_name' b/c Group By not get duplicate values
SELECT * FROM game_data GROUP BY id ORDER BY rating

What is SQL to select a property and the max number of occurrences of a related property?

I have a table like this:
Table: p
+----------------+
| id | w_id |
+---------+------+
| 5 | 8 |
| 5 | 10 |
| 5 | 8 |
| 5 | 10 |
| 5 | 8 |
| 6 | 5 |
| 6 | 8 |
| 6 | 10 |
| 6 | 10 |
| 7 | 8 |
| 7 | 10 |
+----------------+
What is the best SQL to get the following result? :
+-----------------------------+
| id | most_used_w_id |
+---------+-------------------+
| 5 | 8 |
| 6 | 10 |
| 7 | 8 |
+-----------------------------+
In other words, to get, per id, the most frequent related w_id.
Note that on the example above, id 7 is related to 8 once and to 10 once.
So, either (7, 8) or (7, 10) will do as result. If it is not possible to
pick up one, then both (7, 8) and (7, 10) on result set will be ok.
I have come up with something like:
select counters2.p_id as id, counters2.w_id as most_used_w_id
from (
select p.id as p_id,
w_id,
count(w_id) as count_of_w_ids
from p
group by id, w_id
) as counters2
join (
select p_id, max(count_of_w_ids) as max_counter_for_w_ids
from (
select p.id as p_id,
w_id,
count(w_id) as count_of_w_ids
from p
group by id, w_id
) as counters
group by p_id
) as p_max
on p_max.p_id = counters2.p_id
and p_max.max_counter_for_w_ids = counters2.count_of_w_ids
;
but I am not sure at all whether this is the best way to do it. And I had to repeat the same sub-query two times.
Any better solution?
Try to use User defined variables
select id,w_id
FROM
( select T.*,
if(#id<>id,1,0) as row,
#id:=id FROM
(
select id,W_id, Count(*) as cnt FROM p Group by ID,W_id
) as T,(SELECT #id:=0) as T1
ORDER BY id,cnt DESC
) as T2
WHERE Row=1
SQLFiddle demo
Formal SQL
In fact - your solution is correct in terms of normal SQL. Why? Because you have to stick with joining values from original data to grouped data. Thus, your query can not be simplified. MySQL allows to mix non-group columns and group function, but that's totally unreliable, so I will not recommend you to rely on that effect.
MySQL
Since you're using MySQL, you can use variables. I'm not a big fan of them, but for your case they may be used to simplify things:
SELECT
c.*,
IF(#id!=id, #i:=1, #i:=#i+1) AS num,
#id:=id AS gid
FROM
(SELECT id, w_id, COUNT(w_id) AS w_count
FROM t
GROUP BY id, w_id
ORDER BY id DESC, w_count DESC) AS c
CROSS JOIN (SELECT #i:=-1, #id:=-1) AS init
HAVING
num=1;
So for your data result will look like:
+------+------+---------+------+------+
| id | w_id | w_count | num | gid |
+------+------+---------+------+------+
| 7 | 8 | 1 | 1 | 7 |
| 6 | 10 | 2 | 1 | 6 |
| 5 | 8 | 3 | 1 | 5 |
+------+------+---------+------+------+
Thus, you've found your id and corresponding w_id. The idea is - to count rows and enumerate them, paying attention to the fact, that we're ordering them in subquery. So we need only first row (because it will represent data with highest count).
This may be replaced with single GROUP BY id - but, again, server is free to choose any row in that case (it will work because it will take first row, but documentation says nothing about that for common case).
One little nice thing about this is - you can select, for example, 2-nd by frequency or 3-rd, it's very flexible.
Performance
To increase performance, you can create index on (id, w_id) - obviously, it will be used for ordering and grouping records. But variables and HAVING, however, will produce line-by-line scan for set, derived by internal GROUP BY. It isn't such bad as it was with full scan of original data, but still it isn't good thing about doing this with variables. On the other hand, doing that with JOIN & subquery like in your query won't be much different, because of creating temporery table for subquery result set too.
But to be certain, you'll have to test. And keep in mind - you already have valid solution, which, by the way, isn't bound to DBMS-specific stuff and is good in terms of common SQL.
Try this query
select p_id, ccc , w_id from
(
select p.id as p_id,
w_id, count(w_id) ccc
from p
group by id,w_id order by id,ccc desc) xxx
group by p_id having max(ccc)
here is the sqlfidddle link
You can also use this code if you do not want to rely on the first record of non-grouping columns
select p_id, ccc , w_id from
(
select p.id as p_id,
w_id, count(w_id) ccc
from p
group by id,w_id order by id,ccc desc) xxx
group by p_id having ccc=max(ccc);

Select a Sum the last 5 rows

I'm building a little quiz game in PHP/MySQL. After asking questions I want a screen to display how many of the last round were answered correctly.
Im storing whether they were answered correctly or not in a table that looks like this:
rowID | questionid | playerid | answercorrect |
1 | 1 | 1 | 1 |
2 | 2 | 1 | 1 |
3 | 3 | 1 | 1 |
4 | 4 | 1 | 1 |
5 | 5 | 1 | 0 |
6 | 6 | 1 | 1 |
7 | 7 | 1 | 1 |
I want to see how many of the last x (usually 5) questions were answered correctly.
I thought this would be simple. I'm trying this:
SELECT sum( answercorrect ) FROM `answersgiven` ORDER BY id DESC LIMIT 5
I thought this would sum the answercorrect column for the last 5 rows, giving me an answer of 4, but it's giving me 7, which is the result for ALL of the rows.
I feel like I'm missing something obvious. It seems like a simple thing to want to do.
Any ideas?
Try this:
SELECT sum(answercorrect)
FROM (SELECT answercorrect FROM `answersgiven` ORDER BY rowID DESC LIMIT 5) t1
Example Fiddle
In your query, the LIMIT clause affects the overall result: So first all are summed up (which results in one row - the sum over all rows) and then the first 5, if available, are taken (which again is just the one row).
The easiest way of achieving your target is to first select just the first 5 rows (in the subselect) and then sum up afterwards.
try this query
SELECT sum(col)
FROM (SELECT col FROM `table` ORDER BY id DESC LIMIT 5) t1
You're almost there. Just sum up those top 5 answers now:
SELECT SUM('top5')
FROM
(SELECT answercorrect AS 'top5'
FROM `answersgiven`
ORDER BY id DESC LIMIT 5) 'x'