I have this query i am trying to optimize, I want to replace LIKE that is causing full table scan by a MATCH AGAIN command.
This is the original query, it is working great but too slow
SELECT DISTINCT a.*, (CASE WHEN a.title LIKE :keywords THEN 300 ELSE 0 END) +
(CASE WHEN a.title LIKE :word_0 THEN 10 ELSE 0 END) +
(CASE WHEN a.description LIKE :word_0 THEN 10 ELSE 0 END) +
(CASE WHEN cl.name LIKE :word_0 THEN 5 ELSE 0 END) +
(CASE WHEN cpl.name LIKE :word_0 THEN 2 ELSE 0 END) +
(CASE WHEN cl.description LIKE :word_0 THEN 1 ELSE 0 END) +
(CASE WHEN cpl.description LIKE :word_0 THEN 1 ELSE 0 END) as relevance,
FROM lara237posts as a
INNER JOIN lara237categories as c ON c.id=a.category_id AND c.active=1
LEFT JOIN lara237categories as cp ON cp.id=c.parent_id AND cp.active=1
LEFT JOIN (SELECT MAX(id) max_id, post_id FROM lara237payments WHERE active=1 GROUP BY post_id) mpy ON mpy.post_id = a.id AND a.featured=1
LEFT JOIN lara237payments as py ON py.id=mpy.max_id
LEFT JOIN lara237packages as p ON p.id=py.package_id
LEFT JOIN lara237categories as cl ON cl.translation_of=c.id AND cl.translation_lang = :translationLang
LEFT JOIN lara237categories as cpl ON cpl.translation_of=cp.id AND cpl.translation_lang = :translationLang
WHERE a.country_code = :countryCode AND (a.verified_email = 1 AND a.verified_phone = 1) AND a.archived != 1 AND a.deleted_at IS NULL AND a.reviewed = 1
GROUP BY a.id, relevance
HAVING relevance >= :average
ORDER BY p.lft DESC, relevance DESC, a.created_at DESC
LIMIT 0, 16
I have create the query below to replace Like by MATCH AGAIN, but the second query is giving me no result. I am wondering where the issue is ?
Below is the second query i have created
SELECT DISTINCT a.*, (CASE WHEN MATCH(a.title, a.description) AGAINST(:keywords IN NATURAL LANGUAGE MODE) THEN 300 ELSE 0 END) +
(CASE WHEN MATCH(cl.name) AGAINST(:word_0 IN NATURAL LANGUAGE MODE) THEN 5 ELSE 0 END) +
(CASE WHEN MATCH(cpl.name) AGAINST(:word_0 IN NATURAL LANGUAGE MODE) THEN 2 ELSE 0 END) +
(CASE WHEN MATCH(cl.description) AGAINST(:word_0 IN NATURAL LANGUAGE MODE) THEN 1 ELSE 0 END) +
(CASE WHEN MATCH(cpl.description) AGAINST(:word_0 IN NATURAL LANGUAGE MODE) THEN 1 ELSE 0 END) as relevance, py.package_id as py_package_id
FROM lara237posts as a
INNER JOIN lara237categories as c ON c.id=a.category_id AND c.active=1
LEFT JOIN lara237categories as cp ON cp.id=c.parent_id AND cp.active=1
LEFT JOIN (SELECT MAX(id) max_id, post_id FROM lara237payments WHERE active=1 GROUP BY post_id) mpy ON mpy.post_id = a.id AND a.featured=1
LEFT JOIN lara237payments as py ON py.id=mpy.max_id
LEFT JOIN lara237packages as p ON p.id=py.package_id
LEFT JOIN lara237categories as cl ON cl.translation_of=c.id AND cl.translation_lang = :translationLang
LEFT JOIN lara237categories as cpl ON cpl.translation_of=cp.id AND cpl.translation_lang = :translationLang
WHERE a.country_code = :countryCode AND (a.verified_email = 1 AND a.verified_phone = 1) AND a.archived != 1 AND a.deleted_at IS NULL AND a.reviewed = 1
GROUP BY a.id, relevance
HAVING relevance >= :average
ORDER BY p.lft DESC, relevance DESC, a.created_at DESC
LIMIT 0, 16
Any help or tips will be greatly appreciated.
Thank you very much in advance
Related
I'm writing code for the production report.
I had written this query
SELECT
P.*,
(
SELECT
COUNT(id) AS cnt
FROM
bales
WHERE
create_date < '2019-11-01' AND product_id = P.id AND(TYPE = 'bale' OR TYPE = 'bag')
) AS before_prod,
(
SELECT
COUNT(id) AS cnt
FROM
bales
WHERE
(
dispatched = '0' OR disp_bunch = '0'
) AND dispatch_date < '2019-11-01' AND product_id = P.id AND(TYPE = 'bale' OR TYPE = 'bag')
) AS before_dispatched,
(
SELECT
COUNT(id) AS cnt
FROM
bales
WHERE
create_date BETWEEN '2019-11-01' AND '2019-11-06' AND product_id = P.id AND(TYPE = 'bale' OR TYPE = 'bag')
) AS production,
(
SELECT
COUNT(id) AS cnt
FROM
bales
WHERE
(
dispatched = '0' OR disp_bunch = '0'
) AND dispatch_date BETWEEN '2019-11-01' AND '2019-11-06' AND product_id = P.id AND(TYPE = 'bale' OR TYPE = 'bag')
) AS production_dispatched,
C.name AS category_name
FROM
products P
INNER JOIN category C ON
C.id = P.category
This query is working but as I have too many records in all tables it takes too much time.
also, I need only records where before_prod, before_dispatched, production, production_dispatched all these subquery results should be greater than 0.
I tried to use having clause but it also takes too much time.
I have also tried php for loop, * LOGIC: first all products than in for loop its production. but it was much slower.*
How can I optimize my query?
You can use join instead and select case to sum your data that matches your conditions.
select p.*, t.*
from products p
inner join (
select t2.id, sum(case when create_date < '2019-11-01' then 1 else 0 end) as before_prod
, sum(case when (dispatched = '0' or disp_bunch = '0') and create_date < '2019-11-01' then 1 else 0 end) as before_dispatched
, sum(case when create_date between '2019-11-01' and '2019-11-06' then 1 else 0 end) as production
, sum(case when (dispatched = '0' or disp_bunch = '0') and create_date between '2019-11-01' and '2019-11-06' then 1 else 0 end) as production_dispatched
from bales t1
inner join product t2 on t2.id= t1.product_id
inner join category t3 on t3.id = t2.category
where t1.TYPE in ('bale', 'bag')
group by t2.id) t
on t.id = p.id
Trying to add another column with a new table, would appreciate any help/suggestions!
Current query:
SELECT parent.name AS parentname, a.name AS accname,
parent.code AS parentcode, a.code AS acccode,
parent.guid AS parentguid, a.guid AS accguid,
a.account_type AS accttype,
sum(case when date_format(post_date, '%Y-%m-%d') <= '2017-12-31' then (s.value_num/s.value_denom) else '0' end) AS 'value2017-12-31',
sum(case when date_format(post_date, '%Y-%m-%d') <= '2018-01-25' then (s.value_num/s.value_denom) else '0' end) AS 'value2018-01-25'
FROM transactions AS t INNER JOIN splits AS s ON s.tx_guid = t.guid
INNER JOIN accounts AS a ON a.guid = s.account_guid
INNER JOIN accounts AS parent ON parent.guid = a.parent_guid
WHERE a.hidden = 0 AND a.account_type NOT IN ('INCOME', 'EXPENSE')
AND parent.name <>''
AND a.guid = '3f3fc442a98225f481bb72e0fd526cbb'
GROUP by accname, parentname ORDER by acccode
And that works fine, gives 1 row of results. I'd now like to add another column to the results, I believe a left outer join from the prices table of the record that is the closest date before 2017-12-31. My attempt:
SELECT parent.name AS parentname, a.name AS accname,
parent.code AS parentcode, a.code AS acccode,
parent.guid AS parentguid, a.guid AS accguid,
a.account_type AS accttype,
sum(case when date_format(post_date, '%Y-%m-%d') <= '2017-12-31' then (s.value_num/s.value_denom) else '0' end) AS 'value2017-12-31',
sum(case when date_format(post_date, '%Y-%m-%d') <= '2018-01-25' then (s.value_num/s.value_denom) else '0' end) AS 'value2018-01-25',
MAX(case when date_format(p.date, '%Y-%m-%d') <= '2017-12-31' then (p.value_num/p.value_denom) else '0' end) AS 'price2017-12-31'
FROM transactions AS t INNER JOIN splits AS s ON s.tx_guid = t.guid
INNER JOIN accounts AS a ON a.guid = s.account_guid
INNER JOIN accounts AS parent ON parent.guid = a.parent_guid
LEFT OUTER JOIN prices as p ON p.commodity_guid = a.commodity_guid
WHERE a.hidden = 0 AND a.account_type NOT IN ('INCOME', 'EXPENSE')
AND parent.name <>''
AND a.guid = '3f3fc442a98225f481bb72e0fd526cbb'
GROUP by accname, parentname ORDER by acccode
It's obviously incorrect as it's now changing the values of value2017-12-31 and value2018-01-25. I get the price correctly using a separate query:
SELECT p.value_num/p.value_denom as calcprice
FROM `prices` as p
WHERE commodity_guid = '77be249d12d9889e90f08dde7c671eb0'
AND date_format(p.date, '%Y-%m-%d') <= '2017-12-31'
order
by date DESC
limit 1
Is there any way to combine them rather than using temp tables?
query taking 1 minute to fetch results
SELECT
`jp`.`id`,
`jp`.`title` AS game_title,
`jp`.`game_type`,
`jp`.`state_abb` AS game_state,
`jp`.`location` AS game_city,
`jp`.`zipcode` AS game_zipcode,
`jp`.`modified_on`,
`jp`.`posted_on`,
`jp`.`game_referal_amount`,
`jp`.`games_referal_amount_type`,
`jp`.`status`,
`jp`.`is_flaged`,
`u`.`id` AS employer_id,
`u`.`email` AS employer_email,
`u`.`name` AS employer_name,
`jf`.`name` AS game_function,
`jp`.`game_freeze_status`,
`jp`.`game_statistics`,
`jp`.`ats_value`,
`jp`.`integration_id`,
`u`.`account_manager_id`,
`jp`.`model_game`,
`jp`.`group_id`,
(CASE
WHEN jp.group_id != '0' THEN gm.group_name
ELSE 'NA'
END) AS group_name,
`jp`.`priority_game`,
(CASE
WHEN jp.country != 'US' THEN jp.country_name
ELSE ''
END) AS game_country,
IFNULL((CASE
WHEN
`jp`.`account_manager_id` IS NULL
OR `jp`.`account_manager_id` = 0
THEN
(SELECT
(CASE
WHEN
account_manager_id IS NULL
OR account_manager_id = 0
THEN
`u`.`account_manager_id`
ELSE account_manager_id
END) AS account_manager_id
FROM
user_user
WHERE
id = (SELECT
user_id
FROM
game_user_assigned
WHERE
game_id = `jp`.`id`
LIMIT 1))
ELSE `jp`.`account_manager_id`
END),
`u`.`account_manager_id`) AS acc,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '1'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS somewhatgame,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '2'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS verygamecommitted,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '3'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS notgame,
(SELECT
COUNT(joa.id) AS applicationcount
FROM
game_refer_to_member jrmm
INNER JOIN
game_refer jrr ON jrr.id = jrmm.rid
INNER JOIN
game_applied joa ON jrmm.id = joa.referred_by
WHERE
jrmm.STATUS = '1'
AND jrr.referby_user_id IN (SELECT
ab_testing_user_id
FROM
ab_testing)
AND joa.game_post_id = rpr.recach_limit_game_id
AND (rpr.recach_limit = 1
OR rpr.recach_limit = 2)) AS gamecount
FROM
(`game_post` AS jp)
JOIN
`user_info` AS u ON `jp`.`user_user_id` = `u`.`id`
JOIN
`game_functional` jf ON `jp`.`game_functional_id` = `jf`.`id`
LEFT JOIN
`group_musesm` gm ON `gm`.`group_id` = `jp`.`group_id`
LEFT JOIN
`recach_limit` rpr ON `jp`.`id` = `rpr`.`recach_limit_game_id`
WHERE
`jp`.`status` != '3'
GROUP BY `jp`.`id`
ORDER BY `posted_on` DESC
LIMIT 10
I would first suggest not nesting select statements because this will cause an n^x performance hit on every xth level and I see at least 3 levels of selects inside this query.
Add index
INDEX(status, posted_on)
Move LIMIT inside
Then, instead of saying
FROM (`game_post` AS jp)
say
FROM ( SELECT id FROM game_post
WHERE status != 3
ORDER BY posted_on DESC
LIMIT 10 ) AS ids
JOIN game_post AS jp USING(id)
(I am assuming that the PK of jp is (id)?)
That should efficiently use the new index to get the 10 ids needed. Then it will reach back into game_post to get the other columns.
LEFT
Also, don't say LEFT unless you need it. It costs something to generate NULLs that you may not be needing.
Is GROUP BY necessary?
If you remove the GROUP BY, does it show dup ids? The above changes may have eliminated the need.
IN(SELECT) may optimize poorly
Change
AND jrr.referby_user_id IN ( SELECT ab_testing_user_id
FROM ab_testing )
to
AND EXISTS ( SELECT * FROM ab_testing
WHERE ab_testing_user_id = jrr.referby_user_id )
(This change may or may not help, depending on the version you are running.)
More
Please provide EXPLAIN SELECT if you need further assistance.
This is my query:
SELECT usr.id,
count(DISTINCT sol.id) as 'Asked',
count(DISTINCT ans.id) as 'Answered',
sum(DISTINCT CASE ans.accepted WHEN 1 THEN 1 ELSE 0 end) as 'Accepted'
FROM tbl_users usr
LEFT JOIN tbl_solutions sol on sol.authorID = usr.id
LEFT JOIN tbl_solution_answers ans on ans.authorID = usr.id
group by usr.id, sol.authorID
My above query with the sum(DISTINCT CASE ans.accepted WHEN 1 THEN 1 ELSE 0 end) only ever returns 1 though I know that's not the case. I've tried adding a group clause on the ans.authorID but it has no effect.
How can I get the sum of all rows from the tbl_solution_answers ans table where the authorID is that of tbl_users.id and Accepted is 1.
SELECT usr.id,
count(DISTINCT sol.id) as 'Asked',
count(DISTINCT ans.id) as 'Answered',
count(DISTINCT case ans.accepted when 1 then ans.id end) as 'Accepted'
FROM tbl_users usr
LEFT JOIN tbl_solutions sol on sol.authorID = usr.id
LEFT JOIN tbl_solution_answers ans on ans.authorID = usr.id
group by usr.id, sol.authorID, ans.authorID
After so many permutations count(DISTINCT case ans.accepted when 1 then ans.id end) as 'Accepted' seems to work. Now if an authorID in tbl_solution_answers has 8 rows they'll all be returned as Answered and if say 3 of them are Accepted then 3 is returned as Accepted.
I have
users
------------------------
id | name | other_stuff.....
.
engagement
------------------------
user_id | type_code |
type_code is a varchar, but either A, B, C or NULL
[ EDIT for clarity: Users can have many engagements of each type code. SO I want to count how many they have of each. ]
I want to return ALL user rows, but with a count of A, B and C type engagements. E.g.
users_result
------------------------
user_id | user_name | other_stuff..... | count_A | count_B | count_C |
I've done quite a bit of searching, but found the following issues with other solutions:
The "other_stuff..." is actually grouped / concatenated results from a dozen other joins, so it's a bit of a monster already. So I need to be able to just add the additional fields to the pre-existing "SELECT ...... FROM users..." query.
The three additional required bits of data all come from the same engagement table, each with their own condition. I havent found anything to allow me to use the three conditions on the same related table.
Thanks
[edit]
I tried to simplify the question so people didn't have to look through loads of unnecessary stuff, but seems I might not have given enough info. Here is 'most' of the original query. I've taken out a lot of the selected fields as there are loads, but I've left most of the joins in so you can see basically what is actually going on.
SELECT
user.id,
user.first_name,
user.second_name,
GROUP_CONCAT(DISTINCT illness.id ORDER BY illness.id SEPARATOR ',' ) AS reason_for_treatment,
IF(ww_id=1000003, 1,'') as user_refused_program,
Group_CONCAT(DISTINCT physical_activity.name SEPARATOR ', ') AS programme_options,
COUNT(CASE WHEN engagement_item.type_code LIKE 'wm6%' THEN 1 ELSE NULL END) as count_A,
COUNT(CASE WHEN engagement_item.type_code LIKE 'wm12%' THEN 1 ELSE NULL END) as count_B,
COUNT(CASE WHEN engagement_item.type_code LIKE 'wm6%' THEN 1 ELSE NULL END) as count_C
FROM `user`
LEFT JOIN session AS session_induction ON (user.id = session_induction.user_id AND session_induction.session_type_id = 3)
LEFT JOIN stats AS stats_induction ON session_induction.id = stats_induction.session_id
LEFT JOIN session AS session_interim ON (user.id = session_interim.user_id AND session_interim.session_type_id = 4)
LEFT JOIN stats AS stats_interim ON session_interim.id = stats_interim.session_id
LEFT JOIN session AS session_final ON (user.id = session_final.user_id AND session_final.session_type_id = 5)
LEFT JOIN stats AS stats_final ON session_final.id = stats_final.session_id
LEFT JOIN user_has_illness ON user.ID = user_has_illness.user_id
LEFT JOIN illness ON user_has_illness.illness_id = illness.id
LEFT JOIN user_has_physical_activity ON user.ID = user_has_physical_activity.user_id
LEFT JOIN physical_activity ON user_has_physical_activity.physical_activity_id = physical_activity.id
LEFT JOIN engagement_item ON user.ID = engagement_item.user_ID
WHERE (user.INDUCTION_DATE>='2010-06-09' AND user.INDUCTION_DATE<='2011-06-09' AND user.archive!='1' )
GROUP BY user.id, engagement_item.user_id
It's worth mentioning that it works fine - returns all users with all details required. Except for the count_A B and C cols.
[edit added slightly more simplified query below]
Stripped out the unrelated joins and selects.
SELECT
user.id,
user.first_name,
COUNT(CASE WHEN engagement_item.type_code LIKE 'wm6%' THEN 1 ELSE NULL END) as count_A,
COUNT(CASE WHEN engagement_item.type_code LIKE 'wm12%' THEN 1 ELSE NULL END) as count_B,
COUNT(CASE WHEN engagement_item.type_code LIKE 'wm6%' THEN 1 ELSE NULL END) as count_C
FROM `user`
LEFT JOIN engagement_item ON user.ID = engagement_item.user_ID
GROUP BY user.id, engagement_item.user_id
SELECT e.user_id, u.name,
COUNT(CASE type_code WHEN 'A' THEN 1 ELSE NULL END) as count_A,
COUNT(CASE type_code WHEN 'B' THEN 1 ELSE NULL END) as count_B,
COUNT(CASE type_code WHEN 'C' THEN 1 ELSE NULL END) as count_C
FROM engagement e join users u on (e.user_id = u.id)
GROUP BY e.user_id, u.name
I would use COUNT instead of SUM just because that is what it is made for, counting things when not NULL.
SELECT
user.id,
user.first_name,
user.second_name,
GROUP_CONCAT(DISTINCT illness.id ORDER BY illness.id SEPARATOR ',' ) AS reason_for_treatment,
IF(ww_id=1000003, 1,'') as user_refused_program,
Group_CONCAT(DISTINCT physical_activity.name SEPARATOR ', ') AS programme_options,
ei.count_A, ei.count_B, ei.count_C
FROM `user`
LEFT JOIN ( SELECT user_id
, COUNT(CASE WHEN engagement_item.type_code LIKE 'wm6%' THEN 1 ELSE NULL END) as count_A
, COUNT(CASE WHEN engagement_item.type_code LIKE 'wm12%' THEN 1 ELSE NULL END) as count_B
, COUNT(CASE WHEN engagement_item.type_code LIKE 'wm6%' THEN 1 ELSE NULL END) as count_C
FROM engagement_item
GROUP BY userid ) ei
LEFT JOIN session AS session_induction ON (user.id = session_induction.user_id AND session_induction.session_type_id = 3)
LEFT JOIN stats AS stats_induction ON session_induction.id = stats_induction.session_id
LEFT JOIN session AS session_interim ON (user.id = session_interim.user_id AND session_interim.session_type_id = 4)
LEFT JOIN stats AS stats_interim ON session_interim.id = stats_interim.session_id
LEFT JOIN session AS session_final ON (user.id = session_final.user_id AND session_final.session_type_id = 5)
LEFT JOIN stats AS stats_final ON session_final.id = stats_final.session_id
LEFT JOIN user_has_illness ON user.ID = user_has_illness.user_id
LEFT JOIN illness ON user_has_illness.illness_id = illness.id
LEFT JOIN user_has_physical_activity ON user.ID = user_has_physical_activity.user_id
LEFT JOIN physical_activity ON user_has_physical_activity.physical_activity_id = physical_activity.id
LEFT JOIN engagement_item ON user.ID = engagement_item.user_ID
WHERE (user.INDUCTION_DATE>='2010-06-09' AND user.INDUCTION_DATE<='2011-06-09' AND user.archive!='1' )
GROUP BY user.id, engagement_item.user_id, ei.count_A, ei.count_B, ei.count_C
Something like this perhaps?
select e.user_id, u.name,
sum(case e.type_code when 'A' then 1 else 0 end) as count_A,
sum(case e.type_code when 'B' then 1 else 0 end) as count_B,
sum(case e.type_code when 'C' then 1 else 0 end) as count_C
from engagement e join users u on (e.user_id = u.id)
group by e.user_id, u.name
The interesting part is the use of CASE inside the SUM to split the counting into three chunks.