MySQL, WHERE NOT IN very slow query? - mysql

Question

SELECT product_id
FROM ebay_shop
LEFT JOIN amazon_shop ON ebay_shop.product_id = amazon_shop.product_id
WHERE ebay_shop.validated = 2007
AND amazon_shop.validated = 2007
Result(= 150 products): products are validated in ebay_shop && amazon_shop in year 2007
SELECT ebay_shop.product_id
FROM ebay_shop
WHERE ebay_shop.validated = 2010
Result(= 4000 products): products are validated in ebay_shop in year 2010
Question: i want to find products which aren't validated to 2010 of 2007 list, products have in 2007 but not in 2010.
My query:
SELECT product_id
FROM ebay_shop
LEFT JOIN amazon_shop ON ebay_shop.product_id = amazon_shop.product_id
WHERE ebay_shop.validated = 2007
AND amazon_shop.validated = 2007
AND NOT IN (SELECT ebay_shop.product_id
FROM ebay_shop
WHERE ebay_shop.validated = 2010)
My query is very slow(~60s) 150 find in 4000 records. I need help to improve this query.

Answer 1

NOT IN is terrible on performance. Probably the best way is to do a LEFT JOIN for your 2010 and then qualify it is NULL... Any record that IS found, throw it out, just keep the ones that return NULL via the join qualifier.
In addition, ensure you have an index on your ebay shop and amazon shops based on YEAR AND PRODUCT via index ( validated, product_id ). Reason to have the year first, to keep all the common year as small set, then cycle all products within that. Then, the index would also be optimized for the respective join too.
SELECT STRAIGHT_JOIN
es.product_id
FROM
ebay_shop eshop
LEFT JOIN amazon_shop ashop
ON eshop.validated = ashop.validated
AND eshop.product_id = ashop.product_id
LEFT JOIN ebay_shop ebay2
on ebay2.validated = 2010
and ebay2.product_id = eshop.product_id
WHERE
eshop.validated = 2007
and ebay2.validated is null

Answer 2

I realize this isn't really an answer but more of a different study case!
I also use "NOT IN" but in a different context. I get the result I what but the query takes more than 10 seconds for a table with about 300 records.
Here is the query:
(
SELECT (
'inscrit'
) AS
TYPE , membre_mandataire.id AS id, membres.prenom AS prenom, membres.nom_de_famille AS nom_de_famille, membre_mandataire.photo AS photo, membre_mandataire.niveau_qualification AS niveau_qualification, membre_mandataire.numero_mandataire AS numero_mandataire
FROM membres, membre_mandataire
WHERE membre_mandataire.id_membre = membres.id
AND membre_mandataire.confirme = 'Y'
AND membre_mandataire.visible_dans_liste = 'Y'
)
UNION (
SELECT (
'actif'
) AS
TYPE , mandataires_actifs.id AS id, mandataires_actifs.prenom AS prenom, mandataires_actifs.nom_de_famille AS nom_de_famille, mandataires_actifs.photo AS photo, mandataires_actifs.niveau_qualification AS niveau_qualification, mandataires_actifs.numero_mandataire AS numero_mandataire
FROM mandataires_actifs, membre_mandataire
WHERE (
mandataires_actifs.numero_mandataire
) NOT
IN (
SELECT membre_mandataire.numero_mandataire
FROM membre_mandataire
WHERE membre_mandataire.confirme = 'Y'
AND membre_mandataire.visible_dans_liste = 'Y'
)
)
ORDER BY nom_de_famille ASC
The field "numero_mandataire" is the constant in both tables. In the result, i Want all records from the table membre_mandataire and UNION or JOIN all records where "numero_mandataire" cannot be found in the table mandataires_actifs.
I guess the solution for me would be to go with "LEFT JOIN" instead of "UNION" but I am still stuck with "NOT IN". I've been trying for a while and I can't make it work.
I am learning MySQL but looks like i'm not quite there yet!
Thanks for any help!
Blaise

Answer 3

#DRapp wrote a fine workaround and I want to add an other idea...
SELECT group_concat(product_id)
FROM ebay_shop
LEFT JOIN amazon_shop ON ebay_shop.product_id = amazon_shop.product_id
WHERE ebay_shop.validated = 2007
AND amazon_shop.validated = 2007
and then:
SELECT product_id
FROM ebay_shop
LEFT JOIN amazon_shop ON ebay_shop.product_id = amazon_shop.product_id
WHERE ebay_shop.validated = 2007
AND amazon_shop.validated = 2007
AND NOT IN (1,2,3,4) /* results from first query */