Use my sql to randomly select exclusive records - mysql

Question

I have a Table A as below
id (integer)
follow_up (integer, days under observation)
matched_id (integer)
id ; follow_up ; matched_id
1 ; 10 ; 19
1 ; 10 ; 20
1 ; 10 ; 21
2 ; 5 ; 22
2 ; 5 ; 23
2 ; 5 ; 24
2 ; 5 ; 19
2 ; 5 ; 20
3 ; 6 ; 25
3 ; 6 ; 26
3 ; 6 ; 27
4 ; 7 ; 19
4 ; 7 ; 28
4 ; 7 ; 29
I would like to limit to 2 records per id, and the records should be randomly picked up and be exclusive for each id. For, example
matched_id: "19" and "20" were given to id:1, then "19" and "20" should not be given to id:2
matched_id: "19" was given to id:1, then "19" should not be given to id:4
and so on for the rest of the table.
require output
id ; follow_up ; matched_id
1 ; 10 ; 19
1 ; 10 ; 20
2 ; 5 ; 22
2 ; 5 ; 23
3 ; 6 ; 25
3 ; 6 ; 26
4 ; 7 ; 28
4 ; 7 ; 29
Please help me. Thank you so much!

Answer 1

This is a very good and very challenging SQL question.
You have a very challenging set of requirements:
1. No matched_id should appear more than once in the result set
2. No ID be given more than two matches
3. The matching be random
We will stick to a pure SQL solution, assuming that you can't return, say, a larger result set and do some filtering using business logic in your implementation language.
First, let's tackle random assignment. Randomly ordering items inside of groups is a fun question. I decided to tackle it by ordering on a SHA1 hash of the data in the row (id, follow_up, matched_id), which will give a repeatable result with a feeling of randomness. (This would be best if there were a column that contained the date/time created or modified.)
SELECT * FROM
(
SELECT
a.id,
a.follow_up,
a.matched_id,
a.rank_hash,
count(*) rank
FROM
(SELECT *, SHA1(CONCAT(id, follow_up, matched_id)) rank_hash FROM TableA) a
JOIN
(SELECT *, SHA1(CONCAT(id, follow_up, matched_id)) rank_hash FROM TableA) b
ON a.rank_hash >= b.rank_hash
AND a.id = b.id
GROUP BY a.id, a.matched_id
ORDER BY a.id, rank
) groups
WHERE rank <= 2
GROUP BY matched_id
This might suffice for your use case if there are sufficient matched_id values for each id. But what if there is a hidden fourth requirement:
4. If possible, an ID should receive a match.
In other words, what if, as a result of random shuffling, a matched_id was assigned to an id that had several other matches, but further down the result set it was the only match for an id? An optimal solution in which every ID were matched with a matched_id was possible, but it never happened because all the matched_ids were used up earlier in the process?
For example:
CREATE TABLE TableA
(`id` int, `follow_up` int, `matched_id` varchar(1))
;
INSERT INTO TableA
(`id`, `follow_up`, `matched_id`)
VALUES
(1, 10, 'A'),
(1, 10, 'B'),
(1, 10, 'C'),
(2, 5, 'D'),
(2, 5, 'E'),
(2, 5, 'F'),
(3, 5, 'C')
;
In the above set, if IDs and their matches are assigned randomly, if ID 1 gets assigned matched_id C, then ID 3 will not get a matched_id at all.
What if we first find out how many matches an ID received, and order by that first?
SELECT
a.*,
frequency
FROM TableA a
JOIN
( SELECT
matched_id,
count(*) frequency
FROM
TableA
GROUP BY matched_id
) b
ON a.matched_id = b.matched_id
GROUP BY a.matched_id
ORDER BY b.frequency
This is where a middleman programming language might come in handy to help limit the result set.
But note that we also lost our requirement of randomness! As you can see, a pure SQL solution might get pretty ugly. It is indeed possible combining the techniques outlined above.
Hopefully this will get your imagination firing.

Answer 2

Along with RAND() and MySQL user defined variables you can achieve this:
SELECT
t.id,
t.follow_up,
t.matched_id
FROM
(
SELECT
randomTable.*,
IF(#sameID = id, #rn := #rn + 1,
IF(#sameID := id, #rn := 1, #rn := 1)
) AS rowNumber
FROM
(
SELECT
*
FROM tableA
ORDER BY id, RAND()
) AS randomTable
CROSS JOIN (SELECT #sameID := 0, #rn := 0) var
) AS t
WHERE t.rowNumber <= 2
ORDER BY t.id
See Demo

Answer 3

Here's a solution for the specific problem given. It does not scale!
SELECT *
FROM
( SELECT a.matched_id m1
, b.matched_id m2
, c.matched_id m3
, d.matched_id m4
FROM my_table a
JOIN my_table b
ON b.matched_id NOT IN(a.matched_id)
JOIN my_table c
ON c.matched_id NOT IN(a.matched_id,b.matched_id)
JOIN my_table d
ON d.matched_id NOT IN(a.matched_id,b.matched_id,c.matched_id)
WHERE a.id = 1
AND b.id = 2
AND c.id = 3
AND d.id = 4
) x
JOIN
( SELECT a.matched_id n1
, b.matched_id n2
, c.matched_id n3
, d.matched_id n4
FROM my_table a
JOIN my_table b
ON b.matched_id NOT IN(a.matched_id)
JOIN my_table c
ON c.matched_id NOT IN(a.matched_id,b.matched_id)
JOIN my_table d
ON d.matched_id NOT IN(a.matched_id,b.matched_id,c.matched_id)
WHERE a.id = 1
AND b.id = 2
AND c.id = 3
AND d.id = 4
) y
ON y.n1 NOT IN(x.m1,x.m2,x.m3,x.m4)
AND y.n2 NOT IN(x.m1,x.m2,x.m3,x.m4)
AND y.n3 NOT IN(x.m1,x.m2,x.m3,x.m4)
AND y.n4 NOT IN(x.m1,x.m2,x.m3,x.m4)
ORDER
BY RAND() LIMIT 1;
+----+----+----+----+----+----+----+----+
| m1 | m2 | m3 | m4 | n1 | n2 | n3 | n4 |
+----+----+----+----+----+----+----+----+
| 20 | 24 | 27 | 29 | 21 | 23 | 26 | 28 |
+----+----+----+----+----+----+----+----+
So, in this example, the pairs are:
id1: 20,21
id2: 24,23
id3: 27,26
id4: 29,28