MySQL query taking too much time - mysql

query taking 1 minute to fetch results
SELECT
`jp`.`id`,
`jp`.`title` AS game_title,
`jp`.`game_type`,
`jp`.`state_abb` AS game_state,
`jp`.`location` AS game_city,
`jp`.`zipcode` AS game_zipcode,
`jp`.`modified_on`,
`jp`.`posted_on`,
`jp`.`game_referal_amount`,
`jp`.`games_referal_amount_type`,
`jp`.`status`,
`jp`.`is_flaged`,
`u`.`id` AS employer_id,
`u`.`email` AS employer_email,
`u`.`name` AS employer_name,
`jf`.`name` AS game_function,
`jp`.`game_freeze_status`,
`jp`.`game_statistics`,
`jp`.`ats_value`,
`jp`.`integration_id`,
`u`.`account_manager_id`,
`jp`.`model_game`,
`jp`.`group_id`,
(CASE
WHEN jp.group_id != '0' THEN gm.group_name
ELSE 'NA'
END) AS group_name,
`jp`.`priority_game`,
(CASE
WHEN jp.country != 'US' THEN jp.country_name
ELSE ''
END) AS game_country,
IFNULL((CASE
WHEN
`jp`.`account_manager_id` IS NULL
OR `jp`.`account_manager_id` = 0
THEN
(SELECT
(CASE
WHEN
account_manager_id IS NULL
OR account_manager_id = 0
THEN
`u`.`account_manager_id`
ELSE account_manager_id
END) AS account_manager_id
FROM
user_user
WHERE
id = (SELECT
user_id
FROM
game_user_assigned
WHERE
game_id = `jp`.`id`
LIMIT 1))
ELSE `jp`.`account_manager_id`
END),
`u`.`account_manager_id`) AS acc,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '1'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS somewhatgame,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '2'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS verygamecommitted,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '3'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS notgame,
(SELECT
COUNT(joa.id) AS applicationcount
FROM
game_refer_to_member jrmm
INNER JOIN
game_refer jrr ON jrr.id = jrmm.rid
INNER JOIN
game_applied joa ON jrmm.id = joa.referred_by
WHERE
jrmm.STATUS = '1'
AND jrr.referby_user_id IN (SELECT
ab_testing_user_id
FROM
ab_testing)
AND joa.game_post_id = rpr.recach_limit_game_id
AND (rpr.recach_limit = 1
OR rpr.recach_limit = 2)) AS gamecount
FROM
(`game_post` AS jp)
JOIN
`user_info` AS u ON `jp`.`user_user_id` = `u`.`id`
JOIN
`game_functional` jf ON `jp`.`game_functional_id` = `jf`.`id`
LEFT JOIN
`group_musesm` gm ON `gm`.`group_id` = `jp`.`group_id`
LEFT JOIN
`recach_limit` rpr ON `jp`.`id` = `rpr`.`recach_limit_game_id`
WHERE
`jp`.`status` != '3'
GROUP BY `jp`.`id`
ORDER BY `posted_on` DESC
LIMIT 10

I would first suggest not nesting select statements because this will cause an n^x performance hit on every xth level and I see at least 3 levels of selects inside this query.

Add index
INDEX(status, posted_on)
Move LIMIT inside
Then, instead of saying
FROM (`game_post` AS jp)
say
FROM ( SELECT id FROM game_post
WHERE status != 3
ORDER BY posted_on DESC
LIMIT 10 ) AS ids
JOIN game_post AS jp USING(id)
(I am assuming that the PK of jp is (id)?)
That should efficiently use the new index to get the 10 ids needed. Then it will reach back into game_post to get the other columns.
LEFT
Also, don't say LEFT unless you need it. It costs something to generate NULLs that you may not be needing.
Is GROUP BY necessary?
If you remove the GROUP BY, does it show dup ids? The above changes may have eliminated the need.
IN(SELECT) may optimize poorly
Change
AND jrr.referby_user_id IN ( SELECT ab_testing_user_id
FROM ab_testing )
to
AND EXISTS ( SELECT * FROM ab_testing
WHERE ab_testing_user_id = jrr.referby_user_id )
(This change may or may not help, depending on the version you are running.)
More
Please provide EXPLAIN SELECT if you need further assistance.

Related

Order by clause not behaving correctly after several joins

Here is my query:
SELECT DISTINCT `post_data`. * , pv.`seller_id` , pv.`islimited` , pv.`isquantity` , pv.`isslider`, `price`.`original_price` , `price`.`discount_percentage` , `timelimit`.`start_date` , `timelimit`.`expire_date` , `quantity`.`in_stock`, `currency`.`currency_symbol`, `seller`.`directory`, `post_to_cat`.`cat_id`, count(`sales`.`sales_id`) as sale FROM `post_view` AS pv
INNER JOIN `post_data` ON pv.`post_id` = `post_data`.`post_id` AND pv.`status` = 1
INNER JOIN `price` ON pv.`post_id` = `price`.`post_id`
INNER JOIN `currency` ON `price`.`currency_id` = `currency`.`currency_id`
INNER JOIN `seller` ON pv.`seller_id` = `seller`.`seller_id`
INNER JOIN `post_to_cat` ON `post_to_cat`.`cat_id` = 1 AND `post_to_cat`.`post_id` = `post_data`.`post_id`
LEFT JOIN `timelimit` ON ( CASE WHEN pv.`islimited` = 1 THEN `timelimit`.`post_id` ELSE -1 END ) = pv.`post_id`
LEFT JOIN `quantity` ON ( CASE WHEN pv.`isquantity` = 1 THEN `quantity`.`post_id` ELSE -1 END ) = pv.`post_id`
LEFT JOIN `sales` ON `sales`.`post_id` = pv.`post_id` AND `sales`.`status` = 1
WHERE pv.`status` = 1
ORDER BY pv.`post_id` DESC LIMIT 1
The ORDER BY DESC is not working, it just returns the first row from the table, but I want to get the highest post_id value row. What is the mistake I am making?
AS #Alex said in the comments you've got a LIMIT 1 at the end, you should probably bracket the last LEFT JOIN also for readability.
As #McAdam331 said we need data sample and sql fiddle to investigate what is wrong with you query. But at the moment I have some suggestions how to improve and debug your query.
First off all, what do I see the main and very left table in your query is post_view so all other tables should be LEFT JOIN if you want to get the max id. You should use INNER JOIN only if you think that other table could filter your main table somehow and order or result could be other table dependend. But in your case I see no reason to use INNER JOIN.
Second point is your very weird ON conditions:
LEFT JOIN `timelimit` ON ( CASE WHEN pv.`islimited` = 1 THEN `timelimit`.`post_id` ELSE -1 END ) = pv.`post_id`
LEFT JOIN `quantity` ON ( CASE WHEN pv.`isquantity` = 1 THEN `quantity`.`post_id` ELSE -1 END ) = pv.`post_id`
I have converted them into another one
CASE WHEN pv.`islimited`=1 THEN `timelimit`.`start_date` ELSE NULL END as start_date ,
CASE WHEN pv.`islimited`=1 THEN `timelimit`.`expire_date` ELSE NULL END as expire_date,
CASE WHEN pv.`isquantity`=1 THEN `quantity`.`in_stock` ELSE NULL END as in_stock,
But I still don't like it. It seems very useless to me. And has no sense when I read CASE WHEN pv.islimited=1 THEN timelimit.start_date ELSE NULL END as start_date so if flag pv.islimited=0 you don't need start_date? Are you sure?
And the last thing I can suggest: try to use my or even your query. But add every table by step while debugging. So First query just:
SELECT
pv.`post_id`, pv.`seller_id` , pv.`islimited` , pv.`isquantity` ,
pv.`isslider`
FROM `post_view` AS pv
WHERE pv.`status` = 1
ORDER BY pv.`post_id` DESC
LIMIT 1
If it returns correct post_id add next table:
SELECT
pv.`post_id`, pv.`seller_id` , pv.`islimited` , pv.`isquantity` ,
pv.`isslider`,
`post_data`. *
FROM `post_view` AS pv
LEFT JOIN `post_data`
ON pv.`post_id` = `post_data`.`post_id`
WHERE pv.`status` = 1
AND `post_data`.`slug` = 'abc'
ORDER BY pv.`post_id` DESC
LIMIT 1
Check the result. And continue step by step.
Yes it takes time. But that is debugging process. It could be the fastest way to get that query done. :-)
SELECT `post_data`. * ,
pv.`post_id`, pv.`seller_id` , pv.`islimited` , pv.`isquantity` ,
pv.`isslider`, `price`.`original_price` , `price`.`discount_percentage` ,
CASE WHEN pv.`islimited`=1 THEN `timelimit`.`start_date` ELSE NULL END as start_date ,
CASE WHEN pv.`islimited`=1 THEN `timelimit`.`expire_date` ELSE NULL END as expire_date,
CASE WHEN pv.`isquantity`=1 THEN `quantity`.`in_stock` ELSE NULL END as in_stock,
`currency`.`currency_symbol`, `seller`.`directory`, `post_to_cat`.`cat_id`, count(`sales`.`sales_id`) as sale
FROM `post_view` AS pv
LEFT JOIN `post_data`
ON pv.`post_id` = `post_data`.`post_id`
LEFT JOIN `price`
ON pv.`post_id` = `price`.`post_id`
LEFT JOIN `currency`
ON `price`.`currency_id` = `currency`.`currency_id`
LEFT JOIN `seller`
ON pv.`seller_id` = `seller`.`seller_id`
LEFT JOIN `post_to_cat`
ON `post_to_cat`.`cat_id` = 1
AND `post_to_cat`.`post_id` = pv.`post_id`
LEFT JOIN `timelimit`
ON `timelimit`.`post_id` = pv.`post_id`
LEFT JOIN `quantity`
ON quantity`.`post_id` = pv.`post_id`
LEFT JOIN `sales`
ON `sales`.`post_id` = pv.`post_id`
AND `sales`.`status` = 1
WHERE pv.`status` = 1
AND `post_data`.`slug` = 'abc'
GROUP BY pv.`post_id`
ORDER BY pv.`post_id` DESC
LIMIT 1
EDIT 1 - last GROUP BY pv.post_id was added as per #McAdam331 notice about count() function without GROUP BY
I believe the issue here is mostly as a result of preforming aggregation (using the COUNT()) function, without any group by. Although, it seems like you don't necessarily need it because you want that count only for the post in question.
If you're trying to gather all of that information for a single post, I would adjust your WHERE clause to have a condition to only gather that information for the post with the largest ID.
Instead of ordering by ID and limiting by 1, use a subquery to get the largest id, like this:
...
WHERE pv.status = 1 AND post_data.slug = 'abc' AND pv.post_id = (SELECT MAX(post_id) FROM post_view);

Query 10 times slower using LEFT JOIN CASE in order by

I have a relatively long query (posted below for reference).
I have tried to debug why the query is so slow (2 seconds), and I have finally found the cause.
In the end of the query I do:
ORDER BY
-- order by date
DATE(p.date) DESC,
-- order by followed people
CASE WHEN n.id IS NULL THEN '0' ELSE '1' END DESC -- this case takes the query from 20ms to 2 seconds
If I remove the CASE order by, it'll execute in around 20 ms.
Why is this?
When I run the query with EXPLAIN, I notice the CASE will add "Using temporary" into the Extra field.
See EXPLAIN queries below:
Explain query with CASE WHEN n.id IS NULL THEN '0' ELSE '1' END DESC in order by
Explain query without CASE WHEN n.id IS NULL THEN '0' ELSE '1' END DESC in order by
The full query (if it may help):
SELECT
-- feed type
'1' AS feed_type,
-- fetch post data
p.id,
p.receiver,
p.date,
p.message,
p.system_msg,
p.type AS post_type,
-- fetch author data
u.user_id,
u.firstname,
u.lastname,
u.type,
u.permalink,
av.file AS avatar_file,
-- fetch receiever data
u2.user_id AS receiver_user_id,
u2.firstname AS receiver_firstname,
u2.lastname AS receiver_lastname,
u2.permalink AS receiver_permalink,
u2.type AS receiver_type,
-- fetch post comment count
(
SELECT
COUNT(*)
FROM
edu_posts pc
WHERE
pc.comment = p.id
AND pc.deleted IS NULL
) as commentCount,
-- fetch post like count
(
SELECT
COUNT(*)
FROM
edu_likes l
WHERE
l.like_entity = p.id
) as likeCount,
-- user follow state
CASE WHEN n.id IS NOT NULL THEN '1' ELSE '0' END as is_following,
-- check if user likes post
CASE WHEN l.like_id IS NOT NULL THEN '1' ELSE '0' END as user_likes
FROM
edu_posts p
INNER JOIN -- author information
edu_users u ON u.user_id = p.author
LEFT JOIN -- author avatar
edu_avatars av ON av.fk = p.author
AND av.temp = 0
AND av.fk_type = 1
LEFT JOIN -- receiver information (if any)
edu_users u2 ON u2.user_id = p.receiver
LEFT JOIN -- check if author/receiver is followed by current user
edu_notification_list n ON n.user = 1
AND n.following = 1
AND (
n.fk = p.author
OR n.fk = p.receiver
)
AND (
(
n.type = 5
AND p.type = 3
)
OR (
n.type = 2
AND p.type = 1
)
)
LEFT JOIN -- check if user likes the post
edu_likes l ON l.like_entity = p.id
AND l.like_author = 1
WHERE
p.deleted IS NULL
AND p.comment IS NULL
AND (
p.id = p.comment
OR 1 = 1
)
AND (
n.id IS NOT NULL
OR p.system_msg = 0
)
ORDER BY
-- order by date
DATE(p.date) DESC,
-- order by followed people
CASE WHEN n.id IS NULL THEN '0' ELSE '1' END DESC
LIMIT
20 OFFSET 0
NOTE: Please let me know, if you would like to see any of the other tables.
You perform an operation that can't make use of indexes. Instead try
ORDER BY DATE(p.date) DESC,
n.id IS NULL ASC

Sorting by virtual key takes very long time in mysql

SELECT p . * , (
SELECT (
SELECT COUNT( * )
FROM sales s
WHERE s.affiliate != ''
AND s.pid = p.pid
AND s.saletype = 'sale' )
) AS popular
FROM products p
INNER JOIN members m ON m.uname = p.vendor
WHERE (m.mpid = p.pid OR p.marketavail = 'yes')
AND p.showinmarket = 'yes'
AND p.pname != ''
AND p.pdesc != ''
AND p.active = 'yes'
ORDER BY popular DESC
Here, If i use ORDER BY popular , it takes 17 seconds to load. without this ordering , query is executed in 4 seconds.
Please tell me why it is taking too much time while ordering by virtual columns?
All tables has index on required columns, so indexing is not the issue i guess. And if i run select count(*) for single product, it is executing in milliseconds.
And one more error i saw, If i remove SELECT word (second select word in my sql), it takes 105 sec to execute.
Please tell me if i need to give any more information.
Due to such delay in sorting, i am using php instead of mysql for sorting. Please help me to make it better.
Thank you in advance.
please try this query
SELECT p.column1,
p.column2,
p.column3,
COUNT(s.pid) as popular
FROM products p
INNER JOIN members m ON m.uname = p.vendor
LEFT JOIN sales s ON s.pid = p.pid AND s.affiliate != '' AND s.saletype = 'sale'
WHERE (m.mpid = p.pid OR p.marketavail = 'yes')
AND p.showinmarket = 'yes'
AND p.pname != ''
AND p.pdesc != ''
AND p.active = 'yes'
GROUP BY p.column1,p.column2,p.column3
ORDER BY popular DESC
column1,column2,column3 are just examples of columns you want, because you're select * I don't know what column names are from product. so change them to your actual column names.
edit: try this query see if it's any faster
SELECT p.pname, p.vendor, p.pid,
COUNT( s.pid ) AS popular
FROM products p INNER JOIN members m ON m.uname = p.vendor
LEFT JOIN
(SELECT pid FROM sales
WHERE affiliate != ''
AND saletype = 'sale'
)s
ON (s.pid = p.pid)
WHERE ( m.mpid = p.pid OR p.marketavail = 'yes' )
AND p.showinmarket = 'yes' AND p.pname != ''
AND p.pdesc != ''
AND p.active = 'yes'
GROUP BY p.pid, p.pname
ORDER BY popular DESC
if it runs faster you can pre-filter products too like this query and see if it runs even faster
SELECT p.pname, p.vendor, p.pid,
COUNT( s.pid ) AS popular
FROM (SELECT pname,vendor,pid,marketavail
FROM products
WHERE showinmarket = 'yes'
AND pname != ''
AND pdesc != ''
AND active = "yes"
)p
INNER JOIN members m ON m.uname = p.vendor
LEFT JOIN
(SELECT pid FROM sales
WHERE affiliate != ''
AND saletype = 'sale'
)s
ON (s.pid = p.pid)
WHERE ( m.mpid = p.pid OR p.marketavail = 'yes' )
GROUP BY p.pid, p.pname
ORDER BY popular DESC

MySQL: How to optimize this query?

I almost spent a day to optimize this query:
SELECT
prod. *,
cat.slug category_slug,
sup.bname bname,
sup.slug bname_slug
FROM bb_admin.bb_directory_products AS prod
LEFT JOIN bb_admin.bb_categories_products AS cat
ON prod.primary_category_id = cat.category_id
LEFT JOIN bb_admin.bb_directory_suppliers AS sup
ON prod.supplier_id = sup.supplier_id
LEFT JOIN bb_admin.bb_directory_suppliers AS credit_sup
ON prod.credit_supplier_id = credit_sup.supplier_id
LEFT JOIN bb_admin.bb_directory_suppliers AS photo_sup
ON prod.photo_supplier_id = photo_sup.supplier_id
WHERE (
prod.status = '1'
OR prod.status = '3'
OR prod.status = '5'
)
AND (
sup.active_package_id != '1'
OR sup.active_package_id != '5'
OR sup.active_package_id != '6'
OR prod.supplier_id = '0'
)
AND (
sup.supplier_id = '1989'
OR credit_sup.supplier_id = '1989'
OR photo_sup.supplier_id = '1989'
)
GROUP BY prod.product_id
ORDER BY prod.priority_index ASC
Can you help me to optimized this query?
Update your column data types to be INT or one of its variants, since the ones you are checking against are all integer IDs (assumption).
Create indexes on following columns(if possible in all tables):
prod.status
supplier_id
active_package_id
Use IN clause instead of concatenating OR segments.
I'll be putting the updated WHERE clause here:
WHERE prod.status IN(1, 3, 5)
AND ( sup.active_package_id NOT IN(1, 5, 6)
OR prod.supplier_id = 0
)
AND 1989 IN (prod.supplier_id, prod.credit_supplier_id, prod.photo_supplier_id)

MySQL 500 million rows table in select query with join

I'm concerned about the performance of the query below once the tables are fully populated. So far it's under development and performs well with dummy data.
The table "adress_zoo" will contain about 500 million records once fully populated. "adress_zoo" table looks like this:
CREATE TABLE `adress_zoo`
( `adress_id` int(11) NOT NULL, `zoo_id` int(11) NOT NULL,
UNIQUE KEY `pk` (`adress_id`,`zoo_id`),
KEY `adress_id` (`adress_id`) )
ENGINE=InnoDB DEFAULT CHARSET=latin1;
The other tables will contain maximum 500 records each.
The full query looks like this:
SELECT a.* FROM jos_zoo_item AS a
JOIN jos_zoo_search_index AS zsi2 ON zsi2.item_id = a.id
WHERE a.id IN (
SELECT r.id FROM (
SELECT zi.id AS id, Max(zi.priority) as prio
FROM jos_zoo_item AS zi
JOIN jos_zoo_search_index AS zsi ON zsi.item_id = zi.id
LEFT JOIN jos_zoo_tag AS zt ON zt.item_id = zi.id
JOIN jos_zoo_category_item AS zci ON zci.item_id = zi.id
**JOIN adress_zoo AS az ON az.zoo_id = zi.id**
WHERE 1=1
AND ( (zci.category_id != 0 AND ( zt.name != 'prolong' OR zt.name is NULL))
OR (zci.category_id = 0 AND zt.name = 'prolong') )
AND zi.type = 'telefoni'
AND zsi.element_id = '44d3b1fd-40f6-4fd7-9444-7e11643e2cef'
AND zsi.value = 'Small'
AND zci.category_id > 15
**AND az.adress_id = 5**
GROUP BY zci.category_id ) AS r
)
AND a.application_id = 6
AND a.access IN (1,1)
AND a.state = 1
AND (a.publish_up = '0000-00-00 00:00:00' OR a.publish_up <= '2012-06-07 07:51:26')
AND (a.publish_down = '0000-00-00 00:00:00' OR a.publish_down >= '2012-06-07 07:51:26')
AND zsi2.element_id = '1c3cd26e-666d-4f8f-a465-b74fffb4cb14'
GROUP BY a.id
ORDER BY zsi2.value ASC
The query will usually return about 25 records.
Based on your experience, will this query perform acceptable (respond within say 3 seconds)?
What can I do to optimise this?
As adviced by #Jack I ran the query with EXPLAIN and got this:
This part is an important limiter:
az.adress_id = 5
MySQL will limit the table to only those records where adress_id matches before joining it with the rest of the statement, so it will depend on how big you think that result set might be.
Btw, you have a UNIQUE(adress_id, zoo_id) and a separate INDEX. Is there a particular reason? Because the first part of a spanning key can be used by MySQL to select with as well.
What's also important is to use EXPLAIN to understand how MySQL will "attack" your query and return the results. See also: http://dev.mysql.com/doc/refman/5.5/en/execution-plan-information.html
To avoid subquery you can try to rewrite your query as:
SELECT a.* FROM jos_zoo_item AS a
JOIN jos_zoo_search_index AS zsi2 ON zsi2.item_id = a.id
INNER JOIN
(
SELECT ** distinct ** r.id FROM (
SELECT zi.id AS id, Max(zi.priority) as prio
FROM jos_zoo_item AS zi
JOIN jos_zoo_search_index AS zsi ON zsi.item_id = zi.id
LEFT JOIN jos_zoo_tag AS zt ON zt.item_id = zi.id
JOIN jos_zoo_category_item AS zci ON zci.item_id = zi.id
**JOIN adress_zoo AS az ON az.zoo_id = zi.id**
WHERE 1=1
AND ( (zci.category_id != 0 AND ( zt.name != 'prolong' OR zt.name is NULL))
OR (zci.category_id = 0 AND zt.name = 'prolong') )
AND zi.type = 'telefoni'
AND zsi.element_id = '44d3b1fd-40f6-4fd7-9444-7e11643e2cef'
AND zsi.value = 'Small'
AND zci.category_id > 15
**AND az.adress_id = 5**
GROUP BY zci.category_id ) AS r
) T
on a.id = T.id
where
AND a.application_id = 6
AND a.access IN (1,1)
AND a.state = 1
AND (a.publish_up = '0000-00-00 00:00:00' OR a.publish_up <= '2012-06-07 07:51:26')
AND (a.publish_down = '0000-00-00 00:00:00' OR a.publish_down >= '2012-06-07 07:51:26')
AND zsi2.element_id = '1c3cd26e-666d-4f8f-a465-b74fffb4cb14'
GROUP BY a.id
ORDER BY zsi2.value ASC
This approach don't perform subquery for each candidate row. Performance may be increased only if T is calculated in few milliseconds.