SQL- Selecting the most similar product - mysql

Question

Alright, I have a relation which stores two keys, a product Id and an attribute Id. I want to figure out which product is most similar to a given product. (Attributes are actually numbers but it makes the example more confusing so they have been changed to letters to simplify the visual representation.)
Prod_att
Product | Attributes
1 | A
1 | B
1 | C
2 | A
2 | B
2 | D
3 | A
3 | E
4 | A
Initially this seems fairly simple, just select the attributes that a product has and then count the number of attributes per product that are shared. The result of this is then compared to the number of attributes a product has and I can see how similar two products are. This works for products with a large number of attributes relative to their compared products, but issues arise when products have very few attributes. For example product 3 will have a tie for almost every other product (as A is very common).
SELECT Product, count(Attributes)
FROM Prod_att
WHERE Attributes IN
(SELECT Attributes
FROM prod_att
WHERE Product = 1)
GROUP BY Product
;
Any suggestions on how to fix this or improvements to my current query?
Thanks!
*edit: Product 4 will return count() =1 for all Products. I would like to show Product 3 is more similar as it has fewer differing attributes.

Answer 1

Try this
SELECT
a_product_id,
COALESCE( b_product_id, 'no_matchs_found' ) AS closest_product_match
FROM (
SELECT
*,
#row_num := IF(#prev_value=A_product_id,#row_num+1,1) AS row_num,
#prev_value := a_product_id
FROM
(SELECT #prev_value := 0) r
JOIN (
SELECT
a.product_id as a_product_id,
b.product_id as b_product_id,
count( distinct b.Attributes ),
count( distinct b2.Attributes ) as total_products
FROM
products a
LEFT JOIN products b ON ( a.Attributes = b.Attributes AND a.product_id <> b.product_id )
LEFT JOIN products b2 ON ( b2.product_id = b.product_id )
/*WHERE */
/* a.product_id = 3 */
GROUP BY
a.product_id,
b.product_id
ORDER BY
1, 3 desc, 4
) t
) t2
WHERE
row_num = 1
The above query gets the closest matches for all the products, you can include the product_id in the innermost query, to get the results for a particular product_id, I have used LEFT JOIN so that even if a product has no matches, its displayed
SQLFIDDLE
Hope this helps

Answer 2

Try the "Lower bound of Wilson score confidence interval for a Bernoulli parameter". This explicitly deals with the problem of statistical confidence when you have small n. It looks like a lot of math, but actually this is about the minimum amount of math you need to do this sort of thing right. And the website explains it pretty well.
This assumes it is possible to make the step from positive / negative scoring to your problem of matching / not matching attributes.
Here's an example for positive and negative scoring and 95% CL:
SELECT widget_id, ((positive + 1.9208) / (positive + negative) -
1.96 * SQRT((positive * negative) / (positive + negative) + 0.9604) /
(positive + negative)) / (1 + 3.8416 / (positive + negative))
AS ci_lower_bound FROM widgets WHERE positive + negative > 0
ORDER BY ci_lower_bound DESC;

Answer 3

You could write a little view that will give you the total shared attributes between two products.
create view vw_shared_attributes as
select a.product,
b.product 'product_match',
count(*) 'shared_attributes'
from your_table a
inner join test b on b.attribute = a.attribute and b.product <> a.product
group by a.product, b.product
and then use that view to select the top match.
select product,
(select top 1 s.product_match from vw_shared_attributes s where t.product = s.product order by s.shared_attributes desc)
from your_table t
group by product
See http://www.sqlfiddle.com/#!6/53039/1 for an example