Outliers of data by groups - mysql

Question

I want to analyse outliers a of grouped data. Lets say I have data:
+--------+---------+-------+
| fruit | country | price |
+--------+---------+-------+
| apple | UK | 1 |
| apple | USA | 3 |
| apple | LT | 2 |
| apple | LV | 5 |
| apple | EE | 4 |
| pear | SW | 6 |
| pear | NO | 2 |
| pear | FI | 3 |
| pear | PL | 7 |
+--------+---------+-------+
Lets take pears. If my method of finding outliers would be to take 25% highest prices of pears and lowest 25%, outliers of pears would be
+--------+---------+-------+
| pear | NO | 2 |
| pear | PL | 7 |
+--------+---------+-------+
As for apples:
+--------+---------+-------+
| apple | UK | 1 |
| apple | LV | 5 |
+--------+---------+-------+
That I want is to create a view, which would show table of all fruits outliers union. If I had this view, I could analyse only tails, also intersect view with main table to get table without outliers - that's my goal. Solution to this would be:
(SELECT * FROM fruits f WHERE f.fruit = 'pear' ORDER BY f.price ASC
LIMIT (SELECT ROUND(COUNT(*) * 0.25,0)
FROM fruits f2
WHERE f2.fruit = 'pear')
)
union all
(SELECT * FROM fruits f WHERE f.fruit = 'pear' ORDER BY f.price DESC
LIMIT (SELECT ROUND(COUNT(*) * 0.25,0)
FROM fruits f2
WHERE f2.fruit = 'pear')
)
union all
(SELECT * FROM fruits f WHERE f.fruit = 'apple' ORDER BY f.price ASC
LIMIT (SELECT ROUND(COUNT(*) * 0.25,0)
FROM fruits f2
WHERE f2.fruit = 'apple')
)
union all
(SELECT * FROM fruits f WHERE f.fruit = 'apple' ORDER BY f.price DESC
LIMIT (SELECT ROUND(COUNT(*) * 0.25,0)
FROM fruits f2
WHERE f2.fruit = 'apple')
)
This would give me a table I want, however code after LIMIT doesn't seem to be correct... Another problem is number of groups. In this example there are only two groups(pears,apples), but in my actual data there are around 100 groups. So 'union all' should somehow automatically go thru all unique fruits without writing code for each unique fruit, find number of outliers of each unique fruit, take only that numbe of rows and show it all in another table(view).

Answer 1

You can't supply LIMIT with a value from a subquery, in any RDBMS I'm aware of. Some dbs don't even allow host variables/parameters in their versions of the clause (I'm thinking of iSeries DB2).
This is essentially a greatest-n-per-group problem. Similar queries in most other RDBMSs are solved with what are called Windowing functions - essentially, you're looking at a movable selection of data.
MySQL doesn't have this functionality, so we have to counterfeit it. The actual mechanics of the query will depend on the actual data you need, so I can only speak to what you're attempting here. The techniques should be generally adaptable, but may require rather more creativity than otherwise.
To start with you want a function that will return a number indicating it's position - I'm assuming duplicate prices should be given the same rank (ties), and that doing so won't create a gap in the number. This is essentially the DENSE_RANK() windowing function. We can get these results by doing the following:
SELECT fruit, country, price,
#Rnk := IF(#last_fruit <> fruit, 1,
IF(#last_price = price, #Rnk, #Rnk + 1)) AS Rnk,
#last_fruit := fruit,
#last_price := price
FROM Fruits
JOIN (SELECT #Rnk := 0) n
ORDER BY fruit, price
Example Fiddle
... Which generates the following for the 'apple' group:
fruit country price rank
=============================
apple UK 1 1
apple LT 2 2
apple USA 3 3
apple EE 4 4
apple LV 5 5
Now, you're trying to get the top/bottom 25% of rows. In this case, you need a count of distinct prices:
SELECT fruit, COUNT(DISTINCT price)
FROM Fruits
GROUP BY fruit
... And now we just need to join this to the previous statement to limit the top/bottom:
SELECT RankedFruit.fruit, RankedFruit.country, RankedFruit.price
FROM (SELECT fruit, COUNT(DISTINCT price) AS priceCount
FROM Fruits
GROUP BY fruit) CountedFruit
JOIN (SELECT fruit, country, price,
#Rnk := IF(#last_fruit <> fruit, 1,
IF(#last_price = price, #Rnk, #Rnk + 1)) AS rnk,
#last_fruit := fruit,
#last_price := price
FROM Fruits
JOIN (SELECT #Rnk := 0) n
ORDER BY fruit, price) RankedFruit
ON RankedFruit.fruit = CountedFruit.fruit
AND (RankedFruit.rnk > ROUND(CountedFruit.priceCount * .75)
OR RankedFruit.rnk <= ROUND(CountedFruit.priceCount * .25))
SQL Fiddle Example
...which yields the following:
fruit country price
=======================
apple UK 1
apple LV 5
pear NN 2
pear NO 2
pear PL 7
(I duplicated a pear row to show "tied" prices.)

Answer 2

Does round not need 2 / 3 arguments? I.e. do you not need to put in, to what decimal place you wish to round?
so
...
LIMIT (SELECT ROUND(COUNT(*) * 0.25)
FROM #fruits f2
WHERE f2.fruit = 'apple')
becomes
...
LIMIT (SELECT ROUND(COUNT(*) * 0.25,2)
FROM #fruits f2
WHERE f2.fruit = 'apple')
also, just having a quick look at lunch, but it looks like you're just expecting the min / max values. Could you not just use those functions instead?