Randomly select rows from a table based on weight and probability - mysql

Question

I'm using MySQL. I have a table which looks like that:
id: primary key
name: varchar
weight: int (this can be either 1,2 or 3)
What I want to do is randomly select one row until I get a list of 400 selected rows from a table similar to that below that has 500 rows, but taking into account the weight.
For example, if I have 3 rows:
id, name, weight
1, "some content", 2
2, "other content", 1
3, "something", 3
When creating the list, rows that have a weight of 2 appear 30% of times in the list, rows that have a weight of 1 appear 20% of times in the list and rows with weight of 3 appear 50% of times in the list.
Duplicates are permitted but not back to back.
Is there a way to do that?
If you don't understand something please feel free to ask.
Thanks in advance.

Answer 1

I still havent solve the repetition part. But this will give you a start
SQL Fiddle Demo
most inner select assign a random number
middle select use variables to assign a row_number to each row partition by Weight
last select filter to match the ratio. In this case generate a list of size 50.
the original data has an evenly distribution of ~30 for each category. So size 60 will be the limit to achive 50% Weight = 3
.
SELECT `ID`,`Name`,`Weight`, RowNumber
FROM (
SELECT *,
#row_num := IF(#prev_value = `Weight`,
#row_num + 1,
IF(#prev_value:=`Weight`,
1,
1)
) AS RowNumber
FROM (
SELECT `ID`,`Name`,`Weight`, rand() as rng
FROM `myTable`
ORDER BY `Weight`, rng
) X
CROSS JOIN (SELECT #row_num := 1, #prev_value := 0) y
) T
WHERE ( Weight = 3 and RowNumber <= 50 * 0.5 )
OR ( Weight = 2 and RowNumber <= 50 * 0.3 )
OR ( Weight = 1 and RowNumber <= 50 * 0.2 )
ORDER BY Weight, RowNumber

Answer 2

I suggest you make a temporary table where all records with 1's are repeated 2 times, alle records with 2 are repeated 3 times, and all records with 3's are repeated 5 times. Then make random selections in the temporary table among all the records. This should statistically end up with a distribution very near your target, if the total is large enough (e.g. 400).

Answer 3

In my other answer I solve how assign an ID to each weight. Here I will show you how create a list to handle the duplicates.
I use tables to show the whole process, also you can do select on the demo to validate each result. But with some work can be combine in a single query but wont be easy to read.
SQL FIDDLE DEMO
First we need to create a table to store which row will participate in your list
CREATE TABLE `incr` (
`weight` mediumint,
`row` mediumint
);
Using store procedure we fill the table.
CREATE PROCEDURE dowhile(IN Size INT)
BEGIN
DECLARE v1 INT DEFAULT Size * 0.5;
WHILE v1 >= 0 DO
IF v1 <= (Size - 1) * 0.5 THEN
INSERT incr VALUES (3, v1);
END IF;
IF v1 <= (Size - 1) * 0.3 THEN
INSERT incr VALUES (2, v1);
END IF;
IF v1 <= (Size - 1) * 0.2 THEN
INSERT incr VALUES (1, v1);
END IF;
SET v1 = v1 - 1;
END WHILE;
END//
CALL dowhile(300); -- Indicate List Size
Now create a new table to know the size of each weight category in our sample.
CREATE TABLE maxWeight
SELECT `Weight`, COUNT(*) as mw
FROM `myTable`
GROUP BY `Weight`;
Using % operator we can repeat the rows to fill the required size
CREATE TABLE rowList
SELECT i.weight,
CASE WHEN i.row >= w.mw then i.row % w.mw
ELSE i.row
END newrow
FROM incr i
JOIN maxWeight w
ON i.weight = w.weight;
As you can see here even when my list is only 100 the final result is 300
SELECT weight, count(*)
FROM rowList
GROUP BY weight;
| weight | count(*) |
|--------|----------|
| 1 | 60 |
| 2 | 90 |
| 3 | 150 |
Now join both tables together
CREATE TABLE finalResult
SELECT `ID`,`Name`, T.`Weight`, RowNumber
FROM (
SELECT *,
#row_num := IF(#prev_value = `Weight`,
#row_num + 1,
IF(#prev_value:=`Weight`,
0,
0)
) AS RowNumber
FROM (
SELECT `ID`,`Name`,`Weight`, rand() as rng
FROM `myTable`
ORDER BY `Weight`, rng
) X
CROSS JOIN (SELECT #row_num := 0, #prev_value := 0) y
) T
JOIN rowList
ON T.`RowNumber` = rowList.`newrow`
AND T.`Weight` = rowList.`weight`;
The final result has the desire ratio using repeat the names
SELECT `Weight`, COUNT(*) total, COUNT(DISTINCT `Name`) d_name
FROM finalResult
GROUP BY `Weight`;
| Weight | total | d_name |
|--------|-------|--------|
| 1 | 60 | 36 |
| 2 | 90 | 32 |
| 3 | 150 | 30 |
Even when original table has 37 weight = 1, the tool I use to generate random values duplicate one Name, so d_name = 36