Query 10 times slower using LEFT JOIN CASE in order by - mysql

I have a relatively long query (posted below for reference).
I have tried to debug why the query is so slow (2 seconds), and I have finally found the cause.
In the end of the query I do:
ORDER BY
-- order by date
DATE(p.date) DESC,
-- order by followed people
CASE WHEN n.id IS NULL THEN '0' ELSE '1' END DESC -- this case takes the query from 20ms to 2 seconds
If I remove the CASE order by, it'll execute in around 20 ms.
Why is this?
When I run the query with EXPLAIN, I notice the CASE will add "Using temporary" into the Extra field.
See EXPLAIN queries below:
Explain query with CASE WHEN n.id IS NULL THEN '0' ELSE '1' END DESC in order by
Explain query without CASE WHEN n.id IS NULL THEN '0' ELSE '1' END DESC in order by
The full query (if it may help):
SELECT
-- feed type
'1' AS feed_type,
-- fetch post data
p.id,
p.receiver,
p.date,
p.message,
p.system_msg,
p.type AS post_type,
-- fetch author data
u.user_id,
u.firstname,
u.lastname,
u.type,
u.permalink,
av.file AS avatar_file,
-- fetch receiever data
u2.user_id AS receiver_user_id,
u2.firstname AS receiver_firstname,
u2.lastname AS receiver_lastname,
u2.permalink AS receiver_permalink,
u2.type AS receiver_type,
-- fetch post comment count
(
SELECT
COUNT(*)
FROM
edu_posts pc
WHERE
pc.comment = p.id
AND pc.deleted IS NULL
) as commentCount,
-- fetch post like count
(
SELECT
COUNT(*)
FROM
edu_likes l
WHERE
l.like_entity = p.id
) as likeCount,
-- user follow state
CASE WHEN n.id IS NOT NULL THEN '1' ELSE '0' END as is_following,
-- check if user likes post
CASE WHEN l.like_id IS NOT NULL THEN '1' ELSE '0' END as user_likes
FROM
edu_posts p
INNER JOIN -- author information
edu_users u ON u.user_id = p.author
LEFT JOIN -- author avatar
edu_avatars av ON av.fk = p.author
AND av.temp = 0
AND av.fk_type = 1
LEFT JOIN -- receiver information (if any)
edu_users u2 ON u2.user_id = p.receiver
LEFT JOIN -- check if author/receiver is followed by current user
edu_notification_list n ON n.user = 1
AND n.following = 1
AND (
n.fk = p.author
OR n.fk = p.receiver
)
AND (
(
n.type = 5
AND p.type = 3
)
OR (
n.type = 2
AND p.type = 1
)
)
LEFT JOIN -- check if user likes the post
edu_likes l ON l.like_entity = p.id
AND l.like_author = 1
WHERE
p.deleted IS NULL
AND p.comment IS NULL
AND (
p.id = p.comment
OR 1 = 1
)
AND (
n.id IS NOT NULL
OR p.system_msg = 0
)
ORDER BY
-- order by date
DATE(p.date) DESC,
-- order by followed people
CASE WHEN n.id IS NULL THEN '0' ELSE '1' END DESC
LIMIT
20 OFFSET 0
NOTE: Please let me know, if you would like to see any of the other tables.

You perform an operation that can't make use of indexes. Instead try
ORDER BY DATE(p.date) DESC,
n.id IS NULL ASC

Related

MySQL query taking too much time

query taking 1 minute to fetch results
SELECT
`jp`.`id`,
`jp`.`title` AS game_title,
`jp`.`game_type`,
`jp`.`state_abb` AS game_state,
`jp`.`location` AS game_city,
`jp`.`zipcode` AS game_zipcode,
`jp`.`modified_on`,
`jp`.`posted_on`,
`jp`.`game_referal_amount`,
`jp`.`games_referal_amount_type`,
`jp`.`status`,
`jp`.`is_flaged`,
`u`.`id` AS employer_id,
`u`.`email` AS employer_email,
`u`.`name` AS employer_name,
`jf`.`name` AS game_function,
`jp`.`game_freeze_status`,
`jp`.`game_statistics`,
`jp`.`ats_value`,
`jp`.`integration_id`,
`u`.`account_manager_id`,
`jp`.`model_game`,
`jp`.`group_id`,
(CASE
WHEN jp.group_id != '0' THEN gm.group_name
ELSE 'NA'
END) AS group_name,
`jp`.`priority_game`,
(CASE
WHEN jp.country != 'US' THEN jp.country_name
ELSE ''
END) AS game_country,
IFNULL((CASE
WHEN
`jp`.`account_manager_id` IS NULL
OR `jp`.`account_manager_id` = 0
THEN
(SELECT
(CASE
WHEN
account_manager_id IS NULL
OR account_manager_id = 0
THEN
`u`.`account_manager_id`
ELSE account_manager_id
END) AS account_manager_id
FROM
user_user
WHERE
id = (SELECT
user_id
FROM
game_user_assigned
WHERE
game_id = `jp`.`id`
LIMIT 1))
ELSE `jp`.`account_manager_id`
END),
`u`.`account_manager_id`) AS acc,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '1'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS somewhatgame,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '2'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS verygamecommitted,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '3'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS notgame,
(SELECT
COUNT(joa.id) AS applicationcount
FROM
game_refer_to_member jrmm
INNER JOIN
game_refer jrr ON jrr.id = jrmm.rid
INNER JOIN
game_applied joa ON jrmm.id = joa.referred_by
WHERE
jrmm.STATUS = '1'
AND jrr.referby_user_id IN (SELECT
ab_testing_user_id
FROM
ab_testing)
AND joa.game_post_id = rpr.recach_limit_game_id
AND (rpr.recach_limit = 1
OR rpr.recach_limit = 2)) AS gamecount
FROM
(`game_post` AS jp)
JOIN
`user_info` AS u ON `jp`.`user_user_id` = `u`.`id`
JOIN
`game_functional` jf ON `jp`.`game_functional_id` = `jf`.`id`
LEFT JOIN
`group_musesm` gm ON `gm`.`group_id` = `jp`.`group_id`
LEFT JOIN
`recach_limit` rpr ON `jp`.`id` = `rpr`.`recach_limit_game_id`
WHERE
`jp`.`status` != '3'
GROUP BY `jp`.`id`
ORDER BY `posted_on` DESC
LIMIT 10
I would first suggest not nesting select statements because this will cause an n^x performance hit on every xth level and I see at least 3 levels of selects inside this query.
Add index
INDEX(status, posted_on)
Move LIMIT inside
Then, instead of saying
FROM (`game_post` AS jp)
say
FROM ( SELECT id FROM game_post
WHERE status != 3
ORDER BY posted_on DESC
LIMIT 10 ) AS ids
JOIN game_post AS jp USING(id)
(I am assuming that the PK of jp is (id)?)
That should efficiently use the new index to get the 10 ids needed. Then it will reach back into game_post to get the other columns.
LEFT
Also, don't say LEFT unless you need it. It costs something to generate NULLs that you may not be needing.
Is GROUP BY necessary?
If you remove the GROUP BY, does it show dup ids? The above changes may have eliminated the need.
IN(SELECT) may optimize poorly
Change
AND jrr.referby_user_id IN ( SELECT ab_testing_user_id
FROM ab_testing )
to
AND EXISTS ( SELECT * FROM ab_testing
WHERE ab_testing_user_id = jrr.referby_user_id )
(This change may or may not help, depending on the version you are running.)
More
Please provide EXPLAIN SELECT if you need further assistance.

SUM with CASE Statement in mysql

I have following Mysql query
SELECT c.`id`
,c.`category_name`
,c.`category_type`
,c.bookmark_count
,f.category_id cat_id
,f.unfollow_at
,(
CASE WHEN c.id = f.follower_category_id
THEN (
SELECT count(`user_bookmarks`.`id`)
FROM `user_bookmarks`
WHERE (`user_bookmarks`.`category_id` = cat_id)
AND ((`f`.`unfollow_at` > `user_bookmarks`.`created_at`) || (`f`.`unfollow_at` = '0000-00-00 00:00:00'))
)
ELSE 0 END
) counter
,c.id
,f.follower_category_id follow_id
,c.user_id
FROM categories c
LEFT JOIN following_follower_categories f ON f.follower_category_id = c.id
WHERE c.user_id = 26
ORDER BY `category_name` ASC
and here is output what i am getting after execuation
now i just want to count . here i have field id having value 172 against it i have counter 30,3, 2 and Bookmark_count is 4( i need to include only once)
and i am accepting output for id 172 is 30+3+2+4(bookmark_count only once).
I am not sure how to do this.
Can anybody help me out
Thanks a lot
The following may be the most inefficient query for that purpose, but I added a cover to your query in order to hint at grouping the results.
(I removed the second c.id, and my example may have errors since I couldn't try it.)
SELECT `id`,
`category_name`,
`category_type`,
max(`bookmark_count`),
`cat_id`,
`unfollow_at`,
sum(`counter`)+max(`bookmark_count`) counter,
follow_id`, `user_id`
FROM
(SELECT c.`id`
,c.`category_name`
,c.`category_type`
,c.bookmark_count
,f.category_id cat_id
,f.unfollow_at
,(
CASE WHEN c.id = f.follower_category_id
THEN (
SELECT count(`user_bookmarks`.`id`)
FROM `user_bookmarks`
WHERE (`user_bookmarks`.`category_id` = cat_id)
AND ((`f`.`unfollow_at` > `user_bookmarks`.`created_at`) || (`f`.`unfollow_at` = '0000-00-00 00:00:00'))
)
ELSE 0 END
) counter
,f.follower_category_id follow_id
,c.user_id
FROM categories c
LEFT JOIN following_follower_categories f ON f.follower_category_id = c.id
WHERE c.user_id = 26)
GROUP BY `id`, `category_name`, `category_type`, `cat_id`, `unfollow_at`, `follow_id`, `user_id`
ORDER BY `category_name` ASC

Sorting by virtual key takes very long time in mysql

SELECT p . * , (
SELECT (
SELECT COUNT( * )
FROM sales s
WHERE s.affiliate != ''
AND s.pid = p.pid
AND s.saletype = 'sale' )
) AS popular
FROM products p
INNER JOIN members m ON m.uname = p.vendor
WHERE (m.mpid = p.pid OR p.marketavail = 'yes')
AND p.showinmarket = 'yes'
AND p.pname != ''
AND p.pdesc != ''
AND p.active = 'yes'
ORDER BY popular DESC
Here, If i use ORDER BY popular , it takes 17 seconds to load. without this ordering , query is executed in 4 seconds.
Please tell me why it is taking too much time while ordering by virtual columns?
All tables has index on required columns, so indexing is not the issue i guess. And if i run select count(*) for single product, it is executing in milliseconds.
And one more error i saw, If i remove SELECT word (second select word in my sql), it takes 105 sec to execute.
Please tell me if i need to give any more information.
Due to such delay in sorting, i am using php instead of mysql for sorting. Please help me to make it better.
Thank you in advance.
please try this query
SELECT p.column1,
p.column2,
p.column3,
COUNT(s.pid) as popular
FROM products p
INNER JOIN members m ON m.uname = p.vendor
LEFT JOIN sales s ON s.pid = p.pid AND s.affiliate != '' AND s.saletype = 'sale'
WHERE (m.mpid = p.pid OR p.marketavail = 'yes')
AND p.showinmarket = 'yes'
AND p.pname != ''
AND p.pdesc != ''
AND p.active = 'yes'
GROUP BY p.column1,p.column2,p.column3
ORDER BY popular DESC
column1,column2,column3 are just examples of columns you want, because you're select * I don't know what column names are from product. so change them to your actual column names.
edit: try this query see if it's any faster
SELECT p.pname, p.vendor, p.pid,
COUNT( s.pid ) AS popular
FROM products p INNER JOIN members m ON m.uname = p.vendor
LEFT JOIN
(SELECT pid FROM sales
WHERE affiliate != ''
AND saletype = 'sale'
)s
ON (s.pid = p.pid)
WHERE ( m.mpid = p.pid OR p.marketavail = 'yes' )
AND p.showinmarket = 'yes' AND p.pname != ''
AND p.pdesc != ''
AND p.active = 'yes'
GROUP BY p.pid, p.pname
ORDER BY popular DESC
if it runs faster you can pre-filter products too like this query and see if it runs even faster
SELECT p.pname, p.vendor, p.pid,
COUNT( s.pid ) AS popular
FROM (SELECT pname,vendor,pid,marketavail
FROM products
WHERE showinmarket = 'yes'
AND pname != ''
AND pdesc != ''
AND active = "yes"
)p
INNER JOIN members m ON m.uname = p.vendor
LEFT JOIN
(SELECT pid FROM sales
WHERE affiliate != ''
AND saletype = 'sale'
)s
ON (s.pid = p.pid)
WHERE ( m.mpid = p.pid OR p.marketavail = 'yes' )
GROUP BY p.pid, p.pname
ORDER BY popular DESC

LEFT JOIN isn't working as expected while retrieving data from 2 tables

I have 2 tables, named subscriptions and tags. Once in a month I will have to create tags for each order.
I have to list subscriptions without a tag for the given month (2011-10-01 or 2011-09-01 etc). This query returns 0 records:
SELECT s.id, s.active, s.status
FROM asw_subscriptions as s
LEFT JOIN asw_tags as t ON s.id = t.subscription_id
WHERE t.subscription_id IS NULL
AND t.deliveryDate = '2011-10-01'
AND s.status = '2'
AND s.active = '1'
ORDER BY s.id DESC
LIMIT 0, 25
Table Structure
subscriptions = id (int / auto), active (enum : 0,1), status (enum : 0,1)
tags = id (int / auto), deliveryDate (date), tagNumber
The problem is in clausule
t.deliveryDate = '2011-10-01' AND
You have not record on the left because condition 'IS NULL' eliminates LEFT JOIN'ed records.
So, above condition will eliminate all join products, because will never be true (there will be always null in t.deliveryDate.
Try something like this:
SELECT s.id, s.active, s.status
FROM asw_subscriptions as s
WHERE s.status = '2'
AND s.active = '1'
AND NOT EXISTS (
SELECT 1
FROM asw_tags as t
WHERE s.id = t.subscription_id
AND t.deliveryDate = '2011-10-01'
)
ORDER BY s.id DESC
LIMIT 0, 25
The problem is that you are checking the tag date on the same query that checks the existence of a tag.
Try this:
SELECT s.id, s.active, s.status
FROM
asw_subscriptions as s
LEFT JOIN (
SELECT subscription_id
FROM asw_tags
WHERE deliveryDate = '2011-10-01'
) as t ON s.id = t.subscription_id
WHERE
t.subscription_id IS NULL
AND s.status = '2'
AND s.active = '1'
ORDER BY s.id DESC
LIMIT 0, 25
Just realize that specifying condition under WHERE - you implement INNER JOIN logic, It is not what you have expected. So place entire WHERE section under ON of OUTER JOIN:
... ON s.id = t.subscription_id
AND t.subscription_id IS NULL AND
t.deliveryDate = '2011-10-01' AND
s.status = '2' AND
s.active = '1'

MySQL Join Optimization

Can you please help me optimize this query. I use this query to get a list of friends along with their details and their status.
It takes about 0.08 secs to process this on a Athlon X2 6000
I cant use materizlized view as well because this is frequently changing.
SELECT p.userid, p.firstname, p.lastname, p.gender, p.dob, x.relationship,
IF(p.picture !=1,
IF(p.gender != 'm','/sc/f-t.jpg','/sc/m-t.jpg'),
concat('/sc/pthumb/', p.userid, '.jpg' )) AS picture
FROM `social` AS p
LEFT JOIN `friendlist` AS f1 ON (f1.`userid` = p.`userid` AND f1.`friendid` = 1 AND `f1`.`status` = 1)
LEFT JOIN `friendlist` AS f2 ON (f2.`friendid` = p.`userid` AND f2.`userid` = 1 AND `f2`.`status` = 1)
LEFT JOIN `x_relationship` AS x ON (x.`id` = p.`relationship`)
LEFT JOIN `auth` as a ON (a.`userid` = p.`userid`)
WHERE 1 AND (f1.`userid` IS NOT NULL OR f2.`userid` IS NOT NULL AND ((a.`banned` != 1 AND a.`deleted` != 1)))
ORDER BY RAND() LIMIT 0,10
This is your original query, formatted. Below it are a few thoughts.
SELECT
p.userid,
p.firstname,
p.lastname,
p.gender,
p.dob,
x.relationship,
IF(p.picture !=1, IF(p.gender != 'm', '/sc/f-t.jpg', '/sc/m-t.jpg'), concat('/sc/pthumb/', p.userid, '.jpg' )) AS picture
FROM
`social` AS p
LEFT JOIN `friendlist` AS f1 ON (f1.`friendid` = 1 AND f1.`userid` = p.`userid` AND `f1`.`status` = 1)
LEFT JOIN `friendlist` AS f2 ON (f2.`friendid` = p.`userid` AND f2.`userid` = 1 AND `f2`.`status` = 1)
LEFT JOIN `x_relationship` AS x ON (x.`id` = p.`relationship`)
LEFT JOIN `auth` AS a ON (a.`userid` = p.`userid`)
WHERE
1
AND (
f1.`userid` IS NOT NULL
OR f2.`userid` IS NOT NULL
AND (a.`banned` != 1 AND a.`deleted` != 1)
)
ORDER BY
RAND()
LIMIT
0,10
Think if some of your left joins could be inner joins. If so, change them.
Your WHERE clause is messed up. You are mixing AND and OR without properly prioritizing with parentheses. Try:
WHERE
a.`banned` != 1
AND a.`deleted` != 1
AND (
f1.`userid` IS NOT NULL
OR f2.`userid` IS NOT NULL
)
You could also try:
INNER JOIN `auth` AS a ON (
a.`userid` = p.`userid`
AND a.`banned` != 1
AND a.`deleted` != 1
)
WHERE
f1.`userid` IS NOT NULL
OR f2.`userid` IS NOT NULL
You seem to be joining against friendlist solely to check for record existence. You might want to give this a try as well:
FROM
`social` AS p
INNER JOIN `auth` AS a ON (
a.`userid` = p.`userid`
AND a.`banned` != 1
AND a.`deleted` != 1
)
LEFT JOIN `x_relationship` AS x ON (
x.`id` = p.`relationship`
)
WHERE
EXISTS (
SELECT 1 FROM `friendlist` WHERE `friendid` = 1 AND `userid` = p.`userid` AND `status` = 1
)
OR EXISTS (
SELECT 1 FROM `friendlist` WHERE `friendid` = p.`userid` AND `userid` = 1 AND `status` = 1
)
ORDER BY RAND() might not be the fastest thing on earth, but if this is what you need... Try ordering by a indexed column to see how much impact ORDER BY RAND() has.
Indexing:
You should create a composite index on friendlist(friendid, userid, status).
Make sure there is an index on relationship(id)
Make sure there is an index on auth(userid)