I'm trying to improve a live product search for the admin side of our online store.
We're currently using the following query:
SELECT p.product_id, p.full_title, p.descript, p.cost, p.no_vat, p.high_pic, p.prod_type, count(p.product_id) AS occurence, t.descript AS type_desc, p.available
FROM gdd_product as p, gdd_prodtype as t, gdd_info as i
LEFT JOIN gdd_keyword as k ON i.product_id = k.product
WHERE p.prod_type = t.prod_type
AND i.product_id = p.product_id
AND replace(concat_ws(p.descript, i.info_search1, i.info_search2, i.info_search3, k.keyword),' ','')
LIKE '%tool%'
GROUP BY p.product_id
ORDER BY occurence DESC, cost ASC LIMIT 30
This works, but omits any results which don't have an entry in the info_search columns.
So I tried changing it so that gdd_info is LEFT JOINed, with this code:
SELECT p.product_id, p.full_title, p.descript, p.cost, p.no_vat, p.high_pic, p.prod_type,
count(p.product_id) AS occurence, t.descript AS type_desc, p.available
FROM gdd_product as p, gdd_prodtype AS t
LEFT JOIN gdd_info as i ON p.product_id = i.product_id
LEFT JOIN gdd_keyword as k ON p.product_id = k.product
WHERE p.prod_type = t.prod_type
AND replace(CONCAT_WS(p.descript, i.info_search1, i.info_search2, i.info_search3, k.keyword),' ','')
LIKE '%tool%'
GROUP BY p.product_id
ORDER BY occurence DESC, cost ASC
LIMIT 30
...but that throws an error:
SQL Error (1054): Unknown column 'p.product_id' in 'on clause'
What am I doing wrong?
1st. Don't mix standards the ANSI standards (ANSI-92 vs ANSI-89). Either use INNER/CROSS/LEFT join or the , notation but not both. It's bad form and may eventually break; and maybe what's causing your p.product error now.
SELECT p.product_id, p.full_title, p.descript, p.cost
, p.no_vat, p.high_pic, p.prod_type
, count(p.product_id) AS occurence, t.descript AS type_desc
, p.available
FROM gdd_product as p
INNER JOIN gdd_prodtype AS t
on p.prod_type = t.prod_type
LEFT JOIN gdd_info as i
ON p.product_id = i.product_id
LEFT JOIN gdd_keyword as k
ON p.product_id = k.product
WHERE replace(CONCAT_WS(p.descript, i.info_search1, i.info_search2, i.info_search3, k.keyword),' ','')
LIKE '%tool%'
GROUP BY p.product_id
ORDER BY occurence DESC, cost ASC
LIMIT 30
2nd. the reason why it's omitting records is likely because NULL concatenated with a string is NULL. Then you search for a string against a null which will never return a result. You need to coalesce the i.info_search1... with a empty set '' coalesce(i.info_earch1,'') and so on... so it takes the 1st non-null value in a series and then a string is compared against a string
replace(CONCAT_WS(
coalesce(p.descript,'')
, coalesce(i.info_search1,'')
, coalesce(i.info_search2,'')
, coalesce(i.info_search3,'')
, coalesce(k.keyword,'')),' ','')
Giving us...
SELECT p.product_id, p.full_title, p.descript
, p.cost, p.no_vat, p.high_pic, p.prod_type
, count(p.product_id) AS occurence, t.descript AS type_desc
, p.available
FROM gdd_product as p
INNER JOIN gdd_prodtype AS t
on p.prod_type = t.prod_type
LEFT JOIN gdd_info as i
ON p.product_id = i.product_id
LEFT JOIN gdd_keyword as k
ON p.product_id = k.product
WHERE replace(CONCAT_WS(
coalesce(p.descript,'')
, coalesce(i.info_search1,'')
, coalesce(i.info_search2,'')
, coalesce(i.info_search3,'')
, coalesce(k.keyword,'')),' ','')
LIKE '%tool%'
GROUP BY p.product_id
ORDER BY occurence DESC, cost ASC
LIMIT 30
Think of it this way...
Say p.descript exists but info doesn't on your left join... so you concat p.descript with null getting null. Nothing will be like null so you get no records as you can't execute an equality check (like) on a null value and expect to get a result.
Now say p.descript doesn't exist and is null, concat it with anything that is null is yet again null so you have the same result.
Since any value could be null in your concat_WS string we need to coalesce all values just in case.
and now we have a valid string compared against your like and thus when your string matches your like, you'll now get results instead of when a column value in your ws_concat being null wiping your record out.
Related
We are using the OpenCart ecommerce platform running on PHP 7.2 with MySQL 5.7.27, with about 5000 products.
We use an extension to search through products in the admin panel and it takes about 70-80 seconds on average to execute the search query.
Raw query:
SELECT
SQL_CALC_FOUND_ROWS pd.*,
p.*,
(
SELECT
price
FROM
product_special
WHERE
product_id = p.product_id
AND
(
date_start = '0000-00-00'
OR date_start < NOW()
AND
(
date_end = '0000-00-00'
OR date_end > NOW()
)
)
ORDER BY
priority,
price LIMIT 1
)
AS special_price,
IF(p.image IS NOT NULL
AND p.image <> ''
AND p.image <> 'no_image.png', 'Igen', 'Nem') AS image_text,
IF(p.status, 'Engedélyezett', 'Letiltott') AS status_text,
GROUP_CONCAT(DISTINCT CONCAT_WS(' > ', fgd.name, fd.name)
ORDER BY
CONCAT_WS(' > ', fgd.name, fd.name) ASC SEPARATOR '
') AS filter_text, GROUP_CONCAT(DISTINCT fd.filter_id ORDER BY CONCAT_WS(' > ', fgd.name, fd.name) ASC SEPARATOR '_') AS filter, GROUP_CONCAT(DISTINCT cat.name ORDER BY cat.name ASC SEPARATOR ' ') AS category_text, GROUP_CONCAT(DISTINCT cat.category_id ORDER BY cat.name ASC SEPARATOR '_') AS category, GROUP_CONCAT(DISTINCT IF(p2s.store_id = 0, 'ButopĂȘa HU', s.name) SEPARATOR ' ') AS store_text, GROUP_CONCAT(DISTINCT p2s.store_id SEPARATOR '_') AS store FROM product p LEFT JOIN product_description pd ON (p.product_id = pd.product_id AND pd.language_id = '2') LEFT JOIN product_to_category p2c ON (p.product_id = p2c.product_id) LEFT JOIN (SELECT cp.category_id AS category_id, GROUP_CONCAT(cd1.name ORDER BY cp.level SEPARATOR ' > ') AS name FROM category_path cp LEFT JOIN category c ON (cp.path_id = c.category_id) LEFT JOIN category_description cd1 ON (c.category_id = cd1.category_id) LEFT JOIN category_description cd2 ON (cp.category_id = cd2.category_id) WHERE cd1.language_id = '2' AND cd2.language_id = '2' GROUP BY cp.category_id ORDER BY name) AS cat ON (p2c.category_id = cat.category_id) LEFT JOIN product_to_category p2c2 ON (p.product_id = p2c2.product_id) LEFT JOIN product_filter p2f ON (p.product_id = p2f.product_id) LEFT JOIN filter f ON (f.filter_id = p2f.filter_id) LEFT JOIN filter_description fd ON (fd.filter_id = p2f.filter_id AND fd.language_id = '2') LEFT JOIN filter_group_description fgd ON (f.filter_group_id = fgd.filter_group_id AND fgd.language_id = '2')
LEFT JOIN
product_filter p2f2
ON (p.product_id = p2f2.product_id)
LEFT JOIN
product_to_store p2s
ON (p.product_id = p2s.product_id)
LEFT JOIN
store s
ON (s.store_id = p2s.store_id)
LEFT JOIN
product_to_store p2s2
ON (p.product_id = p2s2.product_id)
GROUP BY
p.product_id
ORDER BY
pd.name ASC LIMIT 0,
190
I tried using MySQL's EXPLAIN functionality to see what's going on, but nothing catches my attention right away:
My test environment is running on Intel NVME, 2666 MHz DDR4 RAM, and i7 8th gen. CPU, and yet it's still very slow.
I appreciate any hints as to what is slowing this query down.
It looks like some many-to-many mappings being used in that SELECT. WooCommerce (and the underlying Wordpress) have an inefficient way of implementing such.
Here is my discussion of how to change the schema to improve performance of wp_postmeta and similar tables (product_to_category and product_to_store): http://mysql.rjweb.org/doc.php/index_cookbook_mysql#speeding_up_wp_postmeta
This may be a bug:
( date_start = '0000-00-00'
OR date_start < NOW()
AND ( date_end = '0000-00-00'
OR date_end > NOW() )
)
You probably wanted extra parentheses:
( ( date_start = '0000-00-00'
OR date_start < NOW() )
AND ( date_end = '0000-00-00'
OR date_end > NOW() )
)
Also,
( date_start = '0000-00-00' OR date_start < NOW() )
can be simplified to just
( date_start < NOW() )
And, the query seems to have the explode-implode syndrome where the JOINs expand to generate a large temp table, only to have the GROUP BY collapse down to the original size. The workaround is to turn
GROUP_CONCAT(... foo.x ...) AS blah.
...
LEFT JOIN foo ... ON ...
into
( SELECT GROUP_CONCAT(... foo.x ...) FROM foo WHERE ... ) AS blah,
If that eliminates all the LEFT JOINs, then the GROUP BY p.product_id can also be eliminated.
Do not say LEFT JOIN when the 'right' table is not optional. (Instead, say JOIN.)
LEFT JOIN category_description cd1 ON (cp.category_id = cd1.category_id)
WHERE cd1.language_id = '2' -- this invalidates the `LEFT`
cd2 seems not to be used except for checking that cd2.language_id = '2'. Consider removing references to it.
This requires two temp tables since they are different:
GROUP BY p.product_id
ORDER BY pd.name ASC
Am I correct in saying that pd.name is simply the name for p.product_id in 'language' 2? If so, this may be semantically the same, but faster because of eliminating a temp table and sort):
GROUP BY pd.name
ORDER BY pd.name
Once that is done, it may be better to have INDEX(language_id, name) on pd.
The speedup from LIMIT 0, 190 is mostly eliminated by SQL_CALC_FOUND_ROWS.
Over-normalization led to having the two components of this in separate tables?
CONCAT_WS(' > ', fgd.name, fd.name)
Mysql LEFT JOIN query is returning strange result. The query returns a record when the right ON condition "PARTIALLY" matches the left.
SELECT s.id sid
, s.points sCredits
, s.ghs
, s.usd
, s.africa
, o.id oid
, o.user_id
, o.item_id
FROM pi21o_zoo_item s
LEFT
JOIN pi21o_logos_orders o
ON s.id = o.item_id
AND o.user_id = '268'
WHERE s.id = '268'
The problem is the query returns a record when s.id = '268' and o.item_id = '268-AGW'
Your problem is that when you compare an integer (s.id) with a string (o.item_id) MySQL automatically converts the string to an integer (see the manual). Since '268-AGW' starts with an integer, it is successfully converted to 268 which then matches the s.id value. To work around this, cast the s.id value to a string i.e. write
(CAST(`s`.`id` AS CHAR) = `o`.`item_id`) AND (`o`.`user_id` = '268')
I have a query which gives result as below, how to replace duplicate values with NULL
Query:
SELECT
word.lemma,
synset.definition,
synset.pos,
sampletable.sample
FROM
word
LEFT JOIN
sense ON word.wordid = sense.wordid
LEFT JOIN
synset ON sense.synsetid = synset.synsetid
LEFT JOIN
sampletable ON synset.synsetid = sampletable.synsetid
WHERE
word.lemma = 'good'
Result:
Required Result: all the greyed out results as NULL
First, this is the type of transformation that is generally better done at the application level. The reason is that it presupposes that the result set is in a particular order -- and you seem to be assuming this even with no order by clause.
Second, it is often simpler in the application.
However, in MySQL 8+, it is not that hard. You can do:
SELECT w.lemma,
(CASE WHEN ROW_NUMBER() OVER (PARTITION BY w.lemma, ss.definition ORDER BY st.sample) = 1
THEN ss.definition
END) as definition,
ss.pos,
st.sample
FROM word w LEFT JOIN
sense s
ON w.wordid = s.wordid LEFT JOIN
synset ss
ON s.synsetid = ss.synsetid LEFT JOIN
sampletable st
ON ss.synsetid = st.synsetid
WHERE w.lemma = 'good'
ORDER BY w.lemma, ss.definition, st.sample;
For this to work reliably, the outer ORDER BY clause needs to be compatible with the ORDER BY for the window function.
If you are using Mysql 8 try with Rank().. As I didn't have your table or data couldn't test this query.
SELECT
word.lemma
,case when r = 1 synset.definition else null end as definition
,synset.pos
,sampletable.sample
FROM
(
SELECT
word.lemma
,synset.definition
,synset.pos
,sampletable.sample
,RANK() OVER (PARTITION BY synset.definition ORDER BY synset.definition) r
FROM
(
SELECT
word.lemma,
synset.definition,
synset.pos,
sampletable.sample
FROM
word
LEFT JOIN
sense ON word.wordid = sense.wordid
LEFT JOIN
synset ON sense.synsetid = synset.synsetid
LEFT JOIN
sampletable ON synset.synsetid = sampletable.synsetid
WHERE
word.lemma = 'good'
) t
)t1;
i have a query and i'm having trouble to change the name of the last row of columb name to 'TOTAL'. The result gives me the same name of the row above the last row.
Here's my query:
SELECT COALESCE(ticket_types.name,'TOTAL') AS name,
COUNT(1) AS quantity
FROM tr_logs
LEFT JOIN tickets ON tr_logs.value = tickets.id
LEFT JOIN ticket_types ON tickets.ticket_type_id = ticket_types.id
LEFT JOIN transactions ON tr_logs.transaction_id = transactions.id
LEFT JOIN tr_fields_data AS tfd_shipping ON tfd_shipping.transaction_id = transactions.id
WHERE type = 'ADDITEM'
AND transactions.event_id = '46'
AND DATE(tr_logs.created_date)
BETWEEN '2017-03-26' AND '2017-05-24'
AND tfd_shipping.data IN ('0','570','571','771')
AND name IS NOT NULL
GROUP BY ticket_types.id WITH ROLLUP
The result looks like this:
name quantity
premium 56
outlaw 6
outlaw 62
Last row name from rollup is not null.... I need it to be TOTAL and not outlaw
Thanks
You haven't changed the name to TOTAL at all: you've changed the name of the column to name, and you've told it to replace any null values with TOTAL.
If you want to change the name of ticket_types.name to total, you just want
SELECT ticket_types.name AS total ...
(But it would be weird to rename something called name to total, so perhaps you need to clarify your requirements a little.)
This may or not be related to your observed problem, but the WHERE and GROUP BY clauses turn all the outer joins into inner joins. You should simplify the query to:
SELECT COALESCE(tt.name, 'TOTAL') AS name, COUNT(1) AS quantity
FROM tr_logs l JOIN
tickets
ON l.value = t.id JOIN
ticket_types tt
ON t.ticket_type_id = tt.id JOIN
transactions tr
ON l.transaction_id = tr.id JOIN
tr_fields_data fd
ON fd.transaction_id = tr.id
WHERE type = 'ADDITEM' AND
tr.event_id = '46' AND
DATE(l.created_date) BETWEEN '2017-03-26' AND '2017-05-24' AND
fd.data IN ('0', '570', '571', '771') AND
tt.name IS NOT NULL
GROUP BY tt.id WITH ROLLUP
Thanks to Gordon Linoff I have figure out my problem.
The name of the last row was never null beacause i GROUP BY with a different attribute.
Here's the solution.
SELECT COALESCE(tckn,'TOTAL') AS name, quantity FROM
(SELECT tt.name AS tckn, COUNT(1) AS quantity
FROM tr_logs AS l
LEFT JOIN tickets AS t ON l.value = t.id
LEFT JOIN ticket_types AS tt ON t.ticket_type_id = tt.id
LEFT JOIN transactions AS tr ON l.transaction_id = tr.id
LEFT JOIN tr_fields_data AS tfd ON tfd.transaction_id = tr.id
WHERE type = 'ADDITEM'
AND tr.event_id = '46'
AND DATE(l.created_date)
BETWEEN '2017-03-26' AND '2017-05-24'
AND tfd.data IN ('0','570','571','771')
GROUP BY tckn WITH ROLLUP) as sum;
I have the following query:
return $this->createQueryBuilder('s')
->select('DISTINCT s.username')
->addSelect('COUNT(p.id) as HIDDEN c_id')
->leftJoin('s.owner', 'o')
->leftJoin('s.userPictures', 'p')
->leftJoin('o.transactions', 't')
->leftJoin('t.packType', 'pt')
->where('pt.id =:packId')
->setParameter('packId', $packId)
->andWhere('s.expirydate >=:expiryDate')
->setParameter('expiryDate', new \DateTime('now'))
->andWhere('COUNT(p.id) <:numberOfPictures')
->setParameter('numberOfPictures', $numberOfPictures)
->groupBy('p.shop')
->orderBy("c_id", 'ASC')
->getQuery()
->getResult()
;
however when trying to run this I always get invalid use of group function. Why is this?
Here's the real SQL query when translated:
SELECT DISTINCT s.username, COUNT(p.id) as HIDDEN c_id
FROM App\MainBundle\Entity\InstagramShop s
LEFT JOIN s.owner o
LEFT JOIN s.userPictures p
LEFT JOIN o.transactions t
LEFT JOIN t.packType pt
WHERE pt.id =:packId AND s.expirydate >=:expiryDate AND COUNT(p.id) <:numberOfPictures
GROUP BY p.shop
ORDER BY c_id ASC