I have a sql query that is running really slow and I am perplexed as to why. The query is:
SELECT DISTINCT(c.ID),c.* FROM `content` c
LEFT JOIN `content_meta` cm1 ON c.id = cm1.content_id
WHERE 1=1
AND c.site_id IN (14)
AND c.type IN ('a','t')
AND c.status = 'visible'
AND (c.lock = 0 OR c.site_id = 14)
AND c.level = 0
OR
(
( c.site_id = 14
AND cm1.meta_key = 'g_id'
AND cm1.meta_value IN ('12','13','7')
)
OR
( c.status = 'visible'
AND (
(c.type = 'topic' AND c.parent_id IN (628,633,624))
)
)
)
ORDER BY c.date_updated DESC LIMIT 20
The content table has about 1250 rows and the content meta table has about 3000 rows. This isn't a lot of data and I'm not quite sure what may be causing it to run so slow. Any thoughts/opinions would be greatly appreciated.
Thanks!
Is you where clause correct? You are making a series of AND statements and later you are executing a OR.
Wouldn't the correct be something like:
AND (c.lock = 0 OR c.site_id = 14)
AND (
( ... )
OR
( ... )
)
If it is indeed correct, you could think on changing the structure or treating the result in an script or procedure.
It might have to do with your "OR" clause at the end... your up-front are being utilized by the indexes where possible, but then you throw this huge OR condition at the end that can be either one or the other. Not knowing more of the underlying content, I would adjust to have a UNION on the inside so each entity can utilize its own indexes, get you qualified CIDs, THEN join to final results.
select
c2.*
from
( select distinct
c.ID
from
`content` c
where
c.site_id in (14)
and c.type in ('a', 't' )
and c.status = 'visible'
and c.lock in ( 0, 14 )
and c.level = 0
UNION
select
c.ID
from
`content` c
where
c.status = 'visible'
and c.type = 'topic'
and c.parent_id in ( 628, 633, 624 )
UNION
select
c.ID
from
`content` c
join `content_meta` cm1
on c.id = cm1.content_id
AND cm1.meta_key = 'g_id'
AND cm1.meta_value in ( '12', '13', '7' )
where
c.site_id = 14 ) PreQuery
JOIN `content` c2
on PreQuery.cID = c2.cID
order by
c2.date_updated desc
limit
20
I would ensure content table has an index on ( site_id, type, status ) another on (parent_id, type, status)
and the meta table, an index on ( content_id, meta_key, meta_value )
Related
I am encountering a strange issue where this particular MySQL query that we have would run almost 50 times slower after we upgraded our database from MySQL 5.1.73 to 5.6.23.
This is the SQL query:
SELECT `companies`.*
FROM `companies`
LEFT OUTER JOIN `company_texts`
ON `company_texts`.`company_id` = `companies`.`id`
AND `company_texts`.`language` = 'en'
AND `company_texts`.`region` = 'US'
INNER JOIN show_texts
ON show_texts.company_id = companies.id
AND `show_texts`.`is_deleted` = 0
AND `show_texts`.`language` = 'en'
AND `show_texts`.`region` = 'US'
INNER JOIN show_region_counts
ON show_region_counts.show_id = show_texts.show_id
AND show_region_counts.region = 'US'
WHERE ( ( `companies`.`id` NOT IN ( '77', '26' ) )
AND ( `company_texts`.is_deleted = 0 )
AND `companies`.id IN (
SELECT DISTINCT show_texts.company_id AS
id
FROM shows
INNER JOIN `show_rollups`
ON
`show_rollups`.`show_id` = `shows`.`id`
AND ( `show_rollups`.`device_id` = 3 )
AND ( `show_rollups`.`package_group_id` = 2 )
AND ( `show_rollups`.`videos_count` > 0 )
LEFT OUTER JOIN `show_texts` ON
`show_texts`.`show_id` = `shows`.`id`
AND
`show_texts`.`is_deleted` = 0
AND
`show_texts`.`language` = 'en'
AND
`show_texts`.`region` = 'US'
AND
shows.is_browseable = 1
AND
show_texts.show_id IS NOT NULL
AND (
`show_rollups`.`episodes_count` > 0
OR `show_rollups`.`clips_count` > 0
OR `show_rollups`.`games_count` > 0
)
) )
GROUP BY companies.id
ORDER BY Sum(show_region_counts.view_count) DESC
LIMIT 30 offset 30;
Now the problem is when I run this query in MySQL 5.1.73 before the upgrade, the query would only take around 1.5 seconds, but after the upgrade to 5.6.23, it now can take upward to 1 minute.
So I did an EXPLAIN of this query in 5.1.73, and I saw this:
Enlarged version : http://i.stack.imgur.com/c4ko0.jpg
And when I did EXPLAIN in 5.6.23 , I saw this :
Enlarged version : http://i.stack.imgur.com/CgBtA.jpg
I can see that in both cases, there is a full scan (type ALL) of the shows table, but is there something else I am not seeing that is causing the massive slowdown in 5.6?
Thanks
IS
Please provide SHOW CREATE TABLE. Without that, I will guess that you are missing this desirable index:
show_rollups: INDEX(device_id, package_group_id, videos_count)
You have LEFT OUTER JOIN show_texts ... ON ... show_texts.show_id IS NOT NULL. This is probably wrong for two reasons: (a) Don't use LEFT if you are not looking for NULL, and (b) the NULL test should be in the missing WHERE clause, not in the ON clause.
Getting rid of the IN ( SELECT ... ) may help it on both machines:
SELECT c.*
FROM
( SELECT DISTINCT st.company_id AS id
FROM shows
INNER JOIN `show_rollups` AS sr ON sr.`show_id` = `shows`.`id`
AND ( sr.`device_id` = 3 )
AND ( sr.`package_group_id` = 2 )
AND ( sr.`videos_count` > 0 )
LEFT OUTER JOIN `show_texts` AS st ON st.`show_id` = `shows`.`id`
AND st.`is_deleted` = 0
AND st.`language` = 'en'
AND st.`region` = 'US'
AND shows.is_browseable = 1
AND st.show_id IS NOT NULL
AND ( sr.`episodes_count` > 0
OR sr.`clips_count` > 0
OR sr.`games_count` > 0 )
) AS x
JOIN `companies` AS c ON x.id = c.id
LEFT OUTER JOIN `company_texts` AS ct ON ct.`company_id` = c.`id`
AND ct.`language` = 'en'
AND ct.`region` = 'US'
INNER JOIN show_texts AS st ON st.company_id = c.id
AND st.`is_deleted` = 0
AND st.`language` = 'en'
AND st.`region` = 'US'
INNER JOIN show_region_counts AS src ON src.show_id = st.show_id
AND src.region = 'US'
WHERE ( c.`id` NOT IN ( '77', '26' ) )
AND ( ct.is_deleted = 0 )
GROUP BY c.id
ORDER BY Sum(src.view_count) DESC
LIMIT 30 offset 30;
There is some chance that the JOINs will mess with the computation in Sum(src.view_count).
I'm sorry, but I can't make out the EXPLAIN screen-grabs on my display.
Without that, I'd wonder if the schemas for the tables changed at all, or of the database engines changed. In particular, companies.id seems to be used as a primary key.
I have following Mysql query
SELECT c.`id`
,c.`category_name`
,c.`category_type`
,c.bookmark_count
,f.category_id cat_id
,f.unfollow_at
,(
CASE WHEN c.id = f.follower_category_id
THEN (
SELECT count(`user_bookmarks`.`id`)
FROM `user_bookmarks`
WHERE (`user_bookmarks`.`category_id` = cat_id)
AND ((`f`.`unfollow_at` > `user_bookmarks`.`created_at`) || (`f`.`unfollow_at` = '0000-00-00 00:00:00'))
)
ELSE 0 END
) counter
,c.id
,f.follower_category_id follow_id
,c.user_id
FROM categories c
LEFT JOIN following_follower_categories f ON f.follower_category_id = c.id
WHERE c.user_id = 26
ORDER BY `category_name` ASC
and here is output what i am getting after execuation
now i just want to count . here i have field id having value 172 against it i have counter 30,3, 2 and Bookmark_count is 4( i need to include only once)
and i am accepting output for id 172 is 30+3+2+4(bookmark_count only once).
I am not sure how to do this.
Can anybody help me out
Thanks a lot
The following may be the most inefficient query for that purpose, but I added a cover to your query in order to hint at grouping the results.
(I removed the second c.id, and my example may have errors since I couldn't try it.)
SELECT `id`,
`category_name`,
`category_type`,
max(`bookmark_count`),
`cat_id`,
`unfollow_at`,
sum(`counter`)+max(`bookmark_count`) counter,
follow_id`, `user_id`
FROM
(SELECT c.`id`
,c.`category_name`
,c.`category_type`
,c.bookmark_count
,f.category_id cat_id
,f.unfollow_at
,(
CASE WHEN c.id = f.follower_category_id
THEN (
SELECT count(`user_bookmarks`.`id`)
FROM `user_bookmarks`
WHERE (`user_bookmarks`.`category_id` = cat_id)
AND ((`f`.`unfollow_at` > `user_bookmarks`.`created_at`) || (`f`.`unfollow_at` = '0000-00-00 00:00:00'))
)
ELSE 0 END
) counter
,f.follower_category_id follow_id
,c.user_id
FROM categories c
LEFT JOIN following_follower_categories f ON f.follower_category_id = c.id
WHERE c.user_id = 26)
GROUP BY `id`, `category_name`, `category_type`, `cat_id`, `unfollow_at`, `follow_id`, `user_id`
ORDER BY `category_name` ASC
I almost spent a day to optimize this query:
SELECT
prod. *,
cat.slug category_slug,
sup.bname bname,
sup.slug bname_slug
FROM bb_admin.bb_directory_products AS prod
LEFT JOIN bb_admin.bb_categories_products AS cat
ON prod.primary_category_id = cat.category_id
LEFT JOIN bb_admin.bb_directory_suppliers AS sup
ON prod.supplier_id = sup.supplier_id
LEFT JOIN bb_admin.bb_directory_suppliers AS credit_sup
ON prod.credit_supplier_id = credit_sup.supplier_id
LEFT JOIN bb_admin.bb_directory_suppliers AS photo_sup
ON prod.photo_supplier_id = photo_sup.supplier_id
WHERE (
prod.status = '1'
OR prod.status = '3'
OR prod.status = '5'
)
AND (
sup.active_package_id != '1'
OR sup.active_package_id != '5'
OR sup.active_package_id != '6'
OR prod.supplier_id = '0'
)
AND (
sup.supplier_id = '1989'
OR credit_sup.supplier_id = '1989'
OR photo_sup.supplier_id = '1989'
)
GROUP BY prod.product_id
ORDER BY prod.priority_index ASC
Can you help me to optimized this query?
Update your column data types to be INT or one of its variants, since the ones you are checking against are all integer IDs (assumption).
Create indexes on following columns(if possible in all tables):
prod.status
supplier_id
active_package_id
Use IN clause instead of concatenating OR segments.
I'll be putting the updated WHERE clause here:
WHERE prod.status IN(1, 3, 5)
AND ( sup.active_package_id NOT IN(1, 5, 6)
OR prod.supplier_id = 0
)
AND 1989 IN (prod.supplier_id, prod.credit_supplier_id, prod.photo_supplier_id)
I'm concerned about the performance of the query below once the tables are fully populated. So far it's under development and performs well with dummy data.
The table "adress_zoo" will contain about 500 million records once fully populated. "adress_zoo" table looks like this:
CREATE TABLE `adress_zoo`
( `adress_id` int(11) NOT NULL, `zoo_id` int(11) NOT NULL,
UNIQUE KEY `pk` (`adress_id`,`zoo_id`),
KEY `adress_id` (`adress_id`) )
ENGINE=InnoDB DEFAULT CHARSET=latin1;
The other tables will contain maximum 500 records each.
The full query looks like this:
SELECT a.* FROM jos_zoo_item AS a
JOIN jos_zoo_search_index AS zsi2 ON zsi2.item_id = a.id
WHERE a.id IN (
SELECT r.id FROM (
SELECT zi.id AS id, Max(zi.priority) as prio
FROM jos_zoo_item AS zi
JOIN jos_zoo_search_index AS zsi ON zsi.item_id = zi.id
LEFT JOIN jos_zoo_tag AS zt ON zt.item_id = zi.id
JOIN jos_zoo_category_item AS zci ON zci.item_id = zi.id
**JOIN adress_zoo AS az ON az.zoo_id = zi.id**
WHERE 1=1
AND ( (zci.category_id != 0 AND ( zt.name != 'prolong' OR zt.name is NULL))
OR (zci.category_id = 0 AND zt.name = 'prolong') )
AND zi.type = 'telefoni'
AND zsi.element_id = '44d3b1fd-40f6-4fd7-9444-7e11643e2cef'
AND zsi.value = 'Small'
AND zci.category_id > 15
**AND az.adress_id = 5**
GROUP BY zci.category_id ) AS r
)
AND a.application_id = 6
AND a.access IN (1,1)
AND a.state = 1
AND (a.publish_up = '0000-00-00 00:00:00' OR a.publish_up <= '2012-06-07 07:51:26')
AND (a.publish_down = '0000-00-00 00:00:00' OR a.publish_down >= '2012-06-07 07:51:26')
AND zsi2.element_id = '1c3cd26e-666d-4f8f-a465-b74fffb4cb14'
GROUP BY a.id
ORDER BY zsi2.value ASC
The query will usually return about 25 records.
Based on your experience, will this query perform acceptable (respond within say 3 seconds)?
What can I do to optimise this?
As adviced by #Jack I ran the query with EXPLAIN and got this:
This part is an important limiter:
az.adress_id = 5
MySQL will limit the table to only those records where adress_id matches before joining it with the rest of the statement, so it will depend on how big you think that result set might be.
Btw, you have a UNIQUE(adress_id, zoo_id) and a separate INDEX. Is there a particular reason? Because the first part of a spanning key can be used by MySQL to select with as well.
What's also important is to use EXPLAIN to understand how MySQL will "attack" your query and return the results. See also: http://dev.mysql.com/doc/refman/5.5/en/execution-plan-information.html
To avoid subquery you can try to rewrite your query as:
SELECT a.* FROM jos_zoo_item AS a
JOIN jos_zoo_search_index AS zsi2 ON zsi2.item_id = a.id
INNER JOIN
(
SELECT ** distinct ** r.id FROM (
SELECT zi.id AS id, Max(zi.priority) as prio
FROM jos_zoo_item AS zi
JOIN jos_zoo_search_index AS zsi ON zsi.item_id = zi.id
LEFT JOIN jos_zoo_tag AS zt ON zt.item_id = zi.id
JOIN jos_zoo_category_item AS zci ON zci.item_id = zi.id
**JOIN adress_zoo AS az ON az.zoo_id = zi.id**
WHERE 1=1
AND ( (zci.category_id != 0 AND ( zt.name != 'prolong' OR zt.name is NULL))
OR (zci.category_id = 0 AND zt.name = 'prolong') )
AND zi.type = 'telefoni'
AND zsi.element_id = '44d3b1fd-40f6-4fd7-9444-7e11643e2cef'
AND zsi.value = 'Small'
AND zci.category_id > 15
**AND az.adress_id = 5**
GROUP BY zci.category_id ) AS r
) T
on a.id = T.id
where
AND a.application_id = 6
AND a.access IN (1,1)
AND a.state = 1
AND (a.publish_up = '0000-00-00 00:00:00' OR a.publish_up <= '2012-06-07 07:51:26')
AND (a.publish_down = '0000-00-00 00:00:00' OR a.publish_down >= '2012-06-07 07:51:26')
AND zsi2.element_id = '1c3cd26e-666d-4f8f-a465-b74fffb4cb14'
GROUP BY a.id
ORDER BY zsi2.value ASC
This approach don't perform subquery for each candidate row. Performance may be increased only if T is calculated in few milliseconds.
Can you please help me optimize this query. I use this query to get a list of friends along with their details and their status.
It takes about 0.08 secs to process this on a Athlon X2 6000
I cant use materizlized view as well because this is frequently changing.
SELECT p.userid, p.firstname, p.lastname, p.gender, p.dob, x.relationship,
IF(p.picture !=1,
IF(p.gender != 'm','/sc/f-t.jpg','/sc/m-t.jpg'),
concat('/sc/pthumb/', p.userid, '.jpg' )) AS picture
FROM `social` AS p
LEFT JOIN `friendlist` AS f1 ON (f1.`userid` = p.`userid` AND f1.`friendid` = 1 AND `f1`.`status` = 1)
LEFT JOIN `friendlist` AS f2 ON (f2.`friendid` = p.`userid` AND f2.`userid` = 1 AND `f2`.`status` = 1)
LEFT JOIN `x_relationship` AS x ON (x.`id` = p.`relationship`)
LEFT JOIN `auth` as a ON (a.`userid` = p.`userid`)
WHERE 1 AND (f1.`userid` IS NOT NULL OR f2.`userid` IS NOT NULL AND ((a.`banned` != 1 AND a.`deleted` != 1)))
ORDER BY RAND() LIMIT 0,10
This is your original query, formatted. Below it are a few thoughts.
SELECT
p.userid,
p.firstname,
p.lastname,
p.gender,
p.dob,
x.relationship,
IF(p.picture !=1, IF(p.gender != 'm', '/sc/f-t.jpg', '/sc/m-t.jpg'), concat('/sc/pthumb/', p.userid, '.jpg' )) AS picture
FROM
`social` AS p
LEFT JOIN `friendlist` AS f1 ON (f1.`friendid` = 1 AND f1.`userid` = p.`userid` AND `f1`.`status` = 1)
LEFT JOIN `friendlist` AS f2 ON (f2.`friendid` = p.`userid` AND f2.`userid` = 1 AND `f2`.`status` = 1)
LEFT JOIN `x_relationship` AS x ON (x.`id` = p.`relationship`)
LEFT JOIN `auth` AS a ON (a.`userid` = p.`userid`)
WHERE
1
AND (
f1.`userid` IS NOT NULL
OR f2.`userid` IS NOT NULL
AND (a.`banned` != 1 AND a.`deleted` != 1)
)
ORDER BY
RAND()
LIMIT
0,10
Think if some of your left joins could be inner joins. If so, change them.
Your WHERE clause is messed up. You are mixing AND and OR without properly prioritizing with parentheses. Try:
WHERE
a.`banned` != 1
AND a.`deleted` != 1
AND (
f1.`userid` IS NOT NULL
OR f2.`userid` IS NOT NULL
)
You could also try:
INNER JOIN `auth` AS a ON (
a.`userid` = p.`userid`
AND a.`banned` != 1
AND a.`deleted` != 1
)
WHERE
f1.`userid` IS NOT NULL
OR f2.`userid` IS NOT NULL
You seem to be joining against friendlist solely to check for record existence. You might want to give this a try as well:
FROM
`social` AS p
INNER JOIN `auth` AS a ON (
a.`userid` = p.`userid`
AND a.`banned` != 1
AND a.`deleted` != 1
)
LEFT JOIN `x_relationship` AS x ON (
x.`id` = p.`relationship`
)
WHERE
EXISTS (
SELECT 1 FROM `friendlist` WHERE `friendid` = 1 AND `userid` = p.`userid` AND `status` = 1
)
OR EXISTS (
SELECT 1 FROM `friendlist` WHERE `friendid` = p.`userid` AND `userid` = 1 AND `status` = 1
)
ORDER BY RAND() might not be the fastest thing on earth, but if this is what you need... Try ordering by a indexed column to see how much impact ORDER BY RAND() has.
Indexing:
You should create a composite index on friendlist(friendid, userid, status).
Make sure there is an index on relationship(id)
Make sure there is an index on auth(userid)