Optimize mysql query with subqueries - mysql

Below is MySQl query with which I am able to get disired result
But is there any way i could optimize the query
SELECT users.*,
(SELECT country_name FROM country WHERE country_code = users.country_code)
AS country_name,
(SELECT zone_name FROM timezone WHERE timezone_id = users.timezone_id)
AS zone_name,
(SELECT GROUP_CONCAT(list_name)
FROM list LEFT JOIN user_list ON user_list.list_id = list.list_id
WHERE user_list.user_id = users.user_id AND user_list.status = "active")
AS groups,
(SELECT GROUP_CONCAT(promotion_name)
FROM promotion LEFT JOIN promotion_user ON promotion_user.promotion_id = promotion.promotion_id
WHERE promotion_user.user_id = users.user_id AND promotion_user.status = "active")
AS promotions,
(SELECT GROUP_CONCAT(full_name)
FROM users u LEFT JOIN promotion_user ON promotion_user.promotor_id = u.user_id
WHERE promotion_user.user_id = users.user_id AND promotion_user.status = "active")
AS promotors
FROM users WHERE client_id = '2' AND status != 'deleted'
ORDER BY user_id desc
LIMIT 50 OFFSET 0
The Explain Output is
possible key
id select_type table type _keys key _len ref rows Extra
1 PRIMARY users index NULL PRIMARY 4 NULL 1045612 Using where
6 DEPENDENT SUBQUERY promotion_user ALL NULL NULL NULL NULL 16159 Using where
6 DEPENDENT SUBQUERY u eq_ref PRIMARY PRIMARY 4 [1] 1 NULL
5 DEPENDENT SUBQUERY promotion_user ALL NULL NULL NULL NULL 16895 Using where
5 DEPENDENT SUBQUERY promotion ALL PRIMARY NULL NULL NULL 4 Using where; Using join buffer (Block Nested Loop)
4 DEPENDENT SUBQUERY list ALL PRIMARY NULL NULL NULL 1592 NULL
4 DEPENDENT SUBQUERY user_list ALL NULL NULL NULL NULL 159852 Using where; Using join buffer (Block Nested Loop)
3 DEPENDENT SUBQUERY timezone eq_ref PRIMARY PRIMARY 4 [2] 1 NULL
2 DEPENDENT SUBQUERY country ALL NULL NULL NULL NULL 239 Using where
[1] test.promotion_user.promoter_id
[2] test.promotion_user.promoter_id

I would try using non correlated sub queries. However as you are only bringing back the details for a single user (hence a single row probably) this might not help. Beyond probably eliminating one sub query.
Something like this (untested as no data definitions or data examples)
SELECT `users`.*,
country.country_name,
timezone.zone_name,
sub_groups.groups,
sub_promotors.promotions,
sub_promotors.promotors
FROM `users`
INNER JOIN country
ON country.country_code = users.country_code
INNER JOIN timezone
ON timezone.timezone_id = users.timezone_id
INNER JOIN
(
SELECT promotion_user.user_id, GROUP_CONCAT(full_name) AS promotors, GROUP_CONCAT(promotion_name) AS promotions
FROM users u
LEFT JOIN promotion_user ON promotion_user.promotor_id = u.user_id
WHERE promotion_user.status = "active"
GROUP BY promotion_user.user_id
) AS sub_promotors
ON sub_promotors.user_id = users.user_id
INNER JOIN
(
SELECT user_list.user_id, GROUP_CONCAT(list_name) AS groups
FROM list
LEFT JOIN user_list ON user_list.list_id = list.list_id
WHERE user_list.status = "active"
GROUP BY user_list.user_id
) AS sub_groups
ON sub_groups.user_id = users.user_id
WHERE users.client_id = '2'
AND users.status != 'deleted'
ORDER BY users.user_id
DESC LIMIT 50 OFFSET 0
Correlated sub queries effectively forces MySQL to perform themselves once for each returned row. Changing these to non correlated sub queries which are joined means they can be performed once for all returned rows. Down side is that joining onto a sub query is poorly optimised as far as indexes in MySQL.
You might be able to remove the sub queries if the promotor full name, etc are unique.

Related

How the position of LEFT JOIN influence the query and the optimizer?

i see a strange behavior of Mysql when i launch two different query with one small difference, the position of one left join.
The slow query:
SELECT i.id_affiliate, i.id_franchising, i.Codice
FROM network_configuration_affiliate AS c
INNER JOIN franchising AS fr ON fr.id = c.id_franchising
INNER JOIN network_selected_car AS i ON c.id_affiliate = i.id_affiliate
INNER JOIN (
select T1.id_car, T1.id_network, T1.id_franchising, T1.id_agencie
from network_car_destinations as T1
where T1.id_network='12' and ( T1.id_franchising = 968 or T1.id_franchising = 974 )
) AS n ON n.id_franchising=i.id_franchising AND n.id_car=i.id_car AND c.id_network=n.id_network
INNER JOIN affiliate_tipologies AS t ON t.id_tipology_ag=i.idCategory AND t.id_franchising=i.id_franchising
INNER JOIN network_assoc_tipologies AS p ON p.id_network=c.id_network AND p.id_default=t.id_tipology_net
LEFT JOIN network_conf_users as ce on ce.id_affiliate = i.id_affiliate and ce.id_user = i.id_user
WHERE c.id_network='12' and c.code_affiliate='69842' AND c.configured = 'yes' AND i.Code_car not like '' and p.code != '0'
GROUP BY i.Code_car
The quick:
select T2.* from (
SELECT i.`id_affiliate`, i.id_franchising, i.Code_car, i.id_user
FROM network_configuration_affiliate AS c
INNER JOIN franchising AS fr ON fr.id = c.id_franchising
INNER JOIN network_selected_car AS i ON c.`id_affiliate` = i.`id_affiliate`
INNER JOIN (
select T1.id_car, T1.id_network, T1.id_franchising, T1.id_agencie
from network_car_destinations as T1
where T1.id_network='12' and ( T1.id_franchising = 968 or T1.id_franchising = 974 )
) AS n ON n.id_franchising=i.id_franchising AND n.id_car=i.id_car AND c.id_network=n.id_network
INNER JOIN affiliate_tipologies AS t ON t.id_tipology_ag=i.idCategory AND t.id_franchising=i.id_franchising
INNER JOIN network_assoc_tipologies AS p ON p.id_network=c.id_network AND p.id_default=t.id_tipology_net
WHERE c.id_network='12' and c.code_affiliate='69842' AND c.configured = 'yes' AND i.Code_car not like ''
GROUP BY i.Code_car
) as T2
LEFT JOIN network_conf_users as ce on ce.`id_affiliate` = T2.`id_affiliate` and ce.id_user = T2.id_user
WHERE c.id_network='12' and c.code_affiliate='69842' AND c.configured = 'yes' AND i.Code_car not like '' and p.code != '0'
GROUP BY i.Code_car
The EXPLAIN are almost the same, the result is the same for both query, but the first query is taking 20 secs to end, the second 0.02, how the position of the left join can influence so much the execution of the query?
EXPLAIN - Slow query:
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY c index_merge id_affiliate,id_network,configured,code_affiliate code_affiliate,id_network 203,5 NULL 1 100.00 Using intersect(code_affiliate,id_network); Using where; Using temporary; Using filesort
1 PRIMARY fr eq_ref PRIMARY PRIMARY 4 c.id_franchising 1 100.00 Using index
1 PRIMARY <derived2> ref <auto_key1> <auto_key1> 5 const 10 100.00 Using where
1 PRIMARY t ref id_franchising,id_tipology_ag,id_tipology_net id_franchising 5 n.id_franchising 13 100.00 Using where
1 PRIMARY p ref id_network,id_default id_default 5 t.id_tipology 26 100.00 Using where
1 PRIMARY i ref id_car,id_affiliate,id_franchising,idCategory id_car 5 n.id_car 3 100.00 Using where
1 PRIMARY ce ALL id_affiliate,id_user NULL NULL NULL 4 75.00 Using where; Using join buffer (Block Nested Loop)
2 DERIVED T1 ref id_franchising,id_network id_network 5 const 136952 100.00 Using where
SHOW WARNINGS:
Level Code Message
Note 1003 /* select#1 */
select `i`.`id_affiliate` AS `id_affiliate`,
`i`.`id_franchising` AS `id_franchising`,
`i`.`Code_car` AS `Code_car`
from `network_configuration_affiliate` `c`
join `franchising` `fr`
join `network_selected_car` `i`
join (/* select#2 */
select `T1`.`id_car` AS `id_car`,
`T1`.`id_network` AS `id_network`,
`T1`.`id_franchising` AS `id_franchising`,
`T1`.`id_agencie` AS `id_agencie`
from `network_car_destinations` `T1`
where ((`T1`.`id_network` = '12')
and ((`T1`.`id_franchising` = 968)
or (`T1`.`id_franchising` = 974)))
) `n`
join `affiliate_tipologies` `t`
join `network_assoc_tipologies` `p`
left join `network_conf_users` `ce` on(((`ce`.`id_user` = `i`.`id_user`)
and (`ce`.`id_affiliate` = `c`.`id_affiliate`))
)
where ((`fr`.`id` = `c`.`id_franchising`)
and (`i`.`id_affiliate` = `c`.`id_affiliate`)
and (`i`.`id_car` = `n`.`id_car`)
and (`t`.`id_franchising` = `n`.`id_franchising`)
and (`i`.`id_franchising` = `n`.`id_franchising`)
and (`i`.`idCategory` = `t`.`id_tipology_ag`)
and (`p`.`id_default` = `t`.`id_tipology_net`)
and (`n`.`id_network` = `c`.`id_network`)
and (`p`.`id_network` = `c`.`id_network`)
and (`c`.`configured` = 'yes')
and (`c`.`code_affiliate` = '69842')
and (`c`.`id_network` = '12')
and (not((`i`.`Code` like '')))
and (`p`.`code` <> '0')
)
group by `i`.`Code_car`
Fast query:
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 7280 100.00 NULL
1 PRIMARY ce ALL id_affiliate,id_user NULL NULL NULL 4 75.00 Using where; Using join buffer (Block Nested Loop)
2 DERIVED c index_merge id_affiliate,id_network,configured,code_affiliate code_affiliate,id_network 203,5 NULL 1 100.00 Using intersect(codice_affiliato,id_network); Using where; Using temporary; Using filesort
2 DERIVED fr eq_ref PRIMARY PRIMARY 4 c.id_franchising 1 100.00 Using index
2 DERIVED <derived3> ref <auto_key1> <auto_key1> 5 const 10 100.00 Using where
2 DERIVED i ref id_car,id_affiliate,id_franchising,idCategory id_car 5 n.id_car 2 100.00 Using where
2 DERIVED t ref id_franchising,id_tipology_ag,id_tipology_net id_franchising 5 n.id_franchising 14 100.00 Using where
2 DERIVED p ref id_network,id_default id_default 5 t.id_tipology_net 26 100.00 Using where
3 DERIVED T1 ref id_franchising,id_network id_network 5 const 133324 100.00 Using where
SHOW WARNINGS:
Level Code Message
Note 1003 /* select#1 */
select `T2`.id_affiliate AS id_affiliate,
`T2`.`id_franchising` AS `id_franchising`,
`T2`.Code_car AS Code_car,`T2`.id_user AS id_user
from (/* select#2 */
select `i`.id_affiliate AS id_affiliate,
`i`.`id_franchising` AS `id_franchising`,
`i`.Code_car AS Code_car,`i`.id_user AS id_user
from network_configuration_affiliate `c`
join `franchising` `fr`
join network_selected_car `i`
join (/* select#3 */
select `T1`.id_car AS id_car,
`T1`.`id_network` AS `id_network`,
`T1`.`id_franchising` AS `id_franchising`,
`T1`.`id_agencie` AS `id_agencie`
from network_car_destinations `T1`
where ((`T1`.`id_network` = '12')
and ((`T1`.`id_franchising` = 968)
or (`T1`.`id_franchising` = 974)))
) `n`
join affiliate_tipologies `t`
join network_assoc_tipologies `p`
where ((`fr`.`id` = `c`.`id_franchising`)
and (`i`.id_affiliate = `c`.id_affiliate)
and (`i`.id_car = `n`.id_car)
and (`i`.`id_franchising` = `n`.`id_franchising`)
and (`t`.`id_franchising` = `n`.`id_franchising`)
and (`t`.`id_tipology_ag` = `i`.`idCategory`)
and (`p`.`id_default` = `t`.`id_tipology_net`)
and (`n`.`id_network` = `c`.`id_network`)
and (`p`.`id_network` = `c`.`id_network`)
and (`c`.`configured` = 'yes')
and (`c`.`code_affiliate` = '69842')
and (`c`.`id_network` = '12')
and (not((`i`.Code_car like '')))
)
group by `i`.Code_car
) `T2`
left join network_conf_users `ce` on(((`ce`.id_affiliate = `T2`.id_affiliate)
and (`ce`.id_user = `T2`.id_user))
)
where 1
The left join finds those fields to the left and retrieves those from the right, while inner joining will lookup all fields. Keeping it simple by not needing to organize the extra columns the query completes much quicker.
Please show us the EXPLAINs.
The Optimizer will turn LEFT JOIN into JOIN if it decides that it makes no difference. Please do EXPLAIN EXTENDED SELECT ... followed immediately by SHOW WARNINGS; so we can see whether this happened.
The Optimizer will try a variety of orders for JOINing (in the absence of LEFT or RIGHT). So, if the LEFT was really redundant, I would expect the EXPLAINs to have the tables in the same order.
Normally, the Optimizer will start with a "derived" table, n in your examples. But it may be hidden behind the LEFT.
Do you have any "composite" indexes? These are likely to be beneficial:
T1: INDEX(id_network, id_franchising) -- in this order
c: INDEX(id_network, code_affiliate, configured)
Revise
With the recent changes to the query, I see this pattern:
Slow:
SELECT ...
FROM ...
JOIN ...
GROUP BY ...
Quick:
SELECT ...
FROM ( SELECT ...
FROM ...
GROUP BY ... )
JOIN ...
(LEFT is only partially important to the question; any JOIN can exhibit the problem.)
I call the slow one "inflate-deflate". By that I mean that first it did join(s), thereby inflating the number of rows. Then it did a GROUP BY, which deflated the results.
The quick one deflated before doing the final join, there by leading to less effort overall.
index merge intersect can almost always be improved by using a 'composite' index. In this case, the one I suggest for c. (This would speed up both versions of the query.)

Nested SELECT is hanging my SQL statement

I'm running a statement that's selecting stock market data from three tables. The last part of the statement is running a SELECT max(date) on a table that contains rows of stock data that is dated. I need the last date for a chosen stock from this table (tbl_asxd_extended.date). The problem is the statement just hangs and I can't work out why.
If I separate the statements, up to the final SELECT and run them independently they run fine! They just don't play well together when combined.
I'm not sure how to troubleshoot this one.
SELECT tbl_asxd_extended.close, tbl_asxd_extended.mcapintra, tbl_asxco.industry, tbl_asxco.company, tbl_watchlist.*
FROM tbl_watchlist
INNER JOIN tbl_asxco ON tbl_asxco.symbol = tbl_watchlist.symbol
INNER JOIN tbl_asxd_extended ON tbl_asxd_extended.symbol = tbl_watchlist.symbol
WHERE user_email='testuser#test.com'
AND tbl_asxd_extended.date =
(SELECT max(tbl_asxd_extended.date) FROM tbl_asxd_extended
WHERE tbl_watchlist.symbol = tbl_asxd_extended.symbol)
Here is an 'EXPLAIN' of the statement
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY tbl_watchlist ALL NULL NULL NULL NULL 9 Using where
1 PRIMARY tbl_asxco eq_ref symbol_2,symbol symbol_2 32 func 1 Using where
1 PRIMARY tbl_asxd_extended ALL NULL NULL NULL NULL 2195 Using where; Using join buffer
2 DEPENDENT SUBQUERY tbl_asxd_extended ALL NULL NULL NULL NULL 2195 Using where
SELECT tbl_asxd_extended.close, tbl_asxd_extended.mcapintra, tbl_asxco.industry, tbl_asxco.company, tbl_watchlist.*
FROM tbl_watchlist
INNER JOIN tbl_asxco ON tbl_asxco.symbol = tbl_watchlist.symbol
INNER JOIN tbl_asxd_extended ON tbl_asxd_extended.symbol = tbl_watchlist.symbol
WHERE user_email='testuser#test.com'
AND tbl_asxd_extended.date =
(SELECT MAX(tbl_asxd_extended.date) FROM tbl_asxd_extended
WHERE tbl_watchlist.symbol = tbl_asxd_extended.symbol)
this will solve your problem as you're using the max() instead of MAX();
And if possible execute last select query first and store it in any variable say result and just assign result variable to the AND part condition matching
An uncorrelated subquery usually outperforms a correlated one:
SELECT e.close
, e.mcapintra
, s.industry
, s.company
, w.*
FROM tbl_watchlist w
JOIN tbl_asxco a
ON a.symbol = w.symbol
JOIN tbl_asxd_extended e
ON e.symbol = w.symbol
JOIN
( SELECT symbol
, MAX(date) date
FROM tbl_asxd_extended
GROUP
BY symbol
) x
ON x.symbol = e.symbol
AND x.date = e.date
WHERE user_email = 'testuser#test.com'
Further performance improvements may be gained by providing the EXPLAIN for the above together with CREATE TABLE statements for ALL relevant tables.

MySQL is not using prmary index

I have this query:
SELECT SQL_NO_CACHE
COUNT(*) AS `numrows`
FROM
(`citations`)
LEFT JOIN
`projects` ON `projects`.`project_id` = `citations`.`project_id`
LEFT JOIN
`users` ON `users`.`user_id` = `projects`.`user_id`
WHERE
`users`.`role` = '0'
AND `citations`.`created` BETWEEN 1360213200 AND 1360299599
AND `citations`.`in_card` = '0'
AND `citations`.`citation_id` NOT IN (SELECT
user_stats_citations.citation_id
FROM
user_stats_citations,
user_stats FORCE INDEX (user_stats_type_index)
WHERE
user_stats_citations.user_stat_id = user_stats.id
AND user_stats.type IN (69 , 70, 71, 75, 76));
I have those indexes on user table:
users 0 PRIMARY 1 user_id A 42836 (NULL) (NULL) BTREE
users 1 users_industry_id_index 1 industry_id A 118 (NULL) (NULL) YES BTREE
users 1 users_sponsor_index 1 sponsor A 12 (NULL) (NULL) YES BTREE
This is the output of EXPLAIN EXTENDED
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY users ALL PRIMARY \N \N \N 42836 100.00 Using where
1 PRIMARY projects ref PRIMARY\,projects_user_id_index projects_user_id_index 4 citelighter.users.user_id 1 100.00 Using where; Using index
1 PRIMARY citations ref citations_project_id_index citations_project_id_index 4 citelighter.projects.project_id 4 100.00 Using index condition; Using where
2 SUBQUERY user_stats range user_stats_type_index user_stats_type_index 2 \N 410768 100.00 Using where; Using index
2 SUBQUERY user_stats_citations ref user_stats_citations_index_user_stat_id\,user_stats_citations_index_citation_id user_stats_citations_index_user_stat_id 8 citelighter.user_stats.id 1 100.00 \N
I tried to add FORCE INDEX on users LEFT JOIN but the index is not used. Can you help me to solve this, because this query is taking like 10 seconds on my local and 1 second on production environment.
The first thing I notice is that this predicate in the where clause: WHERE users.role = '0' turns your LEFT JOINs to INNER JOINs, so you may as well just make them inner joins.
Secondly, MySQL has problems optimising correlated subqueries, and also can perform poorly with derived tables. e.g. In this simple query:
SELECT *
FROM (SELECT * FROM T) T
JOIN (SELECT * FROM T) T2 ON T.ID = T2.ID;
Even though ID is the primary key on T, the primary key is not used for the join as it can't be cascaded out of the derived table. Similarly sometimes when you write:
SELECT *
FROM T
WHERE Afield NOT IN (SELECT Afield FROM T WHERE AnotherField = 1);
MySQL does not necessarily materialise the subquery and use this, it will often rewrite the query as:
SELECT *
FROM T
WHERE NOT EXISTS (SELECT 1
FROM T T2
WHERE T.Afield = T2.Afield
AND T2.AnotherField = 1);
And the subquery is executed for each row in the outer query, so if you have a large number of rows in the outer query executing the subquery for every row becomes very costly. The solution is to avoid subqueries as far as possible. In your case you can rewrite your query as:
SELECT SQL_NO_CACHE
COUNT(*) AS `numrows`
FROM `citations`
INNER JOIN `projects`
ON `projects`.`project_id` = `citations`.`project_id`
INNER JOIN `users`
ON `users`.`user_id` = `projects`.`user_id`
LEFT JOIN (user_stats_citations
INNER JOIN user_stats
ON user_stats_citations.user_stat_id = user_stats.id
AND user_stats.type IN (69 , 70, 71, 75, 76))
ON user_stats_citations.citation_id = `citations`.`citation_id`
WHERE `users`.`role` = '0'
AND `citations`.`created` BETWEEN 1360213200 AND 1360299599
AND `citations`.`in_card` = '0'
AND user_stats_citations.citation_id IS NULL;
With no subqueries there is no derived tables, or row by row execution of subqueries. This should improve execution time.
What does this give you?
SELECT COUNT(*) numrows
FROM citations c
JOIN projects p
ON p.project_id = c.project_id
JOIN users u
ON u.user_id = p.user_id
LEFT
JOIN
( SELECT uc.citation_id
FROM user_stats_citations uc
JOIN user_stats us
ON uc.user_stat_id = us.id
AND us.type IN (69,70,71,75,76)
) x
ON x.citation_id = c.citation_id
WHERE u.role = 0
AND c.created BETWEEN 1360213200 AND 1360299599
AND c.in_card = 0
AND x.citation_id IS NULL

Mapping table MySQL / Access

I have a short access/mySQL question. I have a mapping table on the format below.
ID Category_A Category_B Category_C Team
1 a b T1
2 a d T2
I have a second table which also includes Category_A, Category_B, and Category_C. I would like to join the Team value to the my second table based on the mappingtable. My problem is that when there is a blank (e.g. ID=2, Category_B) the mapping should assign the T2 to any row that contains Category_A=a and Category_C=d regardless of the value in Category_B.
Can this type of mapping be done?
Grateful for your help!
In MS Access, I think you would need something on the lines of:
SELECT t.ID, m.Team
FROM Team t
INNER JOIN Mapping m
ON (m.Category_C = t.Category_C)
AND (m.Category_B = t.Category_B)
AND (m.Category_A = t.Category_A)
WHERE m.Category_C Is Not Null
AND m.Category_B Is Not Null
AND m.Category_A Is Not Null
UNION ALL
SELECT t.ID, m.Team
FROM Team t
INNER JOIN Mapping m
ON (m.Category_B = t.Category_B)
AND (m.Category_A = t.Category_A)
WHERE m.Category_C Is Null
AND m.Category_B Is Not Null
AND m.Category_A Is Not Null
UNION ALL
SELECT t.ID, m.Team
FROM Team t
INNER JOIN Mapping m
ON (m.Category_C = t.Category_C)
AND (m.Category_A = t.Category_A)
WHERE m.Category_C Is Not Null
AND m.Category_B Is Null
AND m.Category_A Is Not Null
UNION ALL
SELECT t.ID, m.Team
FROM Team t
INNER JOIN Mapping m
ON (m.Category_C = t.Category_C)
AND (m.Category_B = t.Category_B)
WHERE m.Category_C Is Not Null
AND m.Category_B Is Not Null
AND m.Category_A Is Null

mysql query running very slow. Please help me with indexes

I'm trying to run a query which is taking 5 seconds to execute with 100000 rows. The query is given below. I've tried all possible indexes i could. Please suggest me what am i missing.
select distinct db_books.bookid as id
, request_type.name as book_type
, request_type.id as book_type_id
, db_books.subject as subject
, sender_user.uid as sender_user_id
, sender_user.username as sender_user
, sender_company.companyid as sender_company_id
, sender_company.companyname as sender_company
, sender_team_id.teamid as sender_team_id
, sender_team_id.name as sender_team
, GROUP_CONCAT(distinct receiver_user_details.uid separator '|') as receiver_user_id
, GROUP_CONCAT(distinct receiver_user_details.username separator '|') as receiver_user
, GROUP_CONCAT(distinct receiver_company.companyid separator '|') as receiver_company_id
, GROUP_CONCAT(distinct receiver_company.companyname separator '|') as receiver_company
, GROUP_CONCAT(distinct receiver_team_details.teamid separator '|') as receiver_team_id
, GROUP_CONCAT(distinct receiver_team_details.name separator '|') as receiver_team
, status.id as statusid
, status.name as status
, db_books.modifydate as modified_date
, db_books.createddate as creation_date
, state.id as stateid
, state.name as state
, assignee.uid as assignee_user_id
, assignee.username as assignee_user
, purpose.name as purpose
, purpose.id as purposeid
, g.name as entityname
, g.entityid as entityid
from db_books db_books
inner join db_users sender_user on (sender_user.deleted=0 and sender_user.uid=db_books.sndrUserid)
inner join db_companies sender_company on (sender_company.deleted=0 and sender_company.companyid=db_books.sndrCompanyid)
inner join db_companies receiver_company on (receiver_company.deleted=0 and receiver_company.companyid=db_books.target_company_id)
inner join db_request_types request_type on (request_type.id=db_books.book_type_id)
left outer join db_teams sender_team_id on (sender_team_id.deleted=0 and sender_team_id.teamid=db_books.sender_team_id)
left outer join db_books_to_users receiver_user on (receiver_user.bookid=db_books.bookid)
left outer join db_users receiver_user_details on (receiver_user_details.uid=receiver_user.userid)
left outer join db_books_to_teams receiver_teams on (receiver_teams.bookid=db_books.bookid)
left outer join db_teams receiver_team_details on (receiver_team_details.teamid=receiver_teams.teamid)
left outer join db_request_status status on (status.id=db_books.statusid)
left outer join db_request_state_types state on (state.id=db_books.request_state_id)
left outer join db_request_purpose purpose on (purpose.id=db_books.request_purpose_id)
left outer join db_users assignee on (assignee.uid=db_books.assignee)
left outer join db_books_details mdtl on (mdtl.deleted=0 and mdtl.bookid=db_books.bookid)
left outer join db_entities g on (g.deleted=0 and g.entityid=mdtl.entityid)
where 1=1
and
(db_books.sndrUserid=25000000003265
or db_books.sender_team_id in (
select a.teamid from db_team_users a
inner join db_teams b on (b.teamid=a.teamid and b.deleted=0)
where a.userid=25000000003265
)
or db_books.bookid in (
select distinct bookid from db_books_to_users where userid=25000000003265
union
select distinct bookid from db_books_to_teams where teamid in
(
select a.teamid from db_team_users a
inner join db_teams b on (b.teamid=a.teamid and b.deleted=0)
where a.deleted=0 AND a.userid=25000000003265
)
)
)
group by db_books.bookid
limit 20
The explain plan is as given below.
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY sender_user ALL PRIMARY,u2 14573 Using where; Using temporary; Using filesort
1 PRIMARY db_books ref i_db_books_target_company_id,i_db_books_sndrUserid,i_db_books_sndrCompanyid,i_sndrUserid_sender_team_idbookid i_db_books_sndrUserid 7 mde_staging.sender_user.uid 41 Using where
1 PRIMARY sender_company eq_ref PRIMARY,db_companies_icd PRIMARY 7 mde_staging.db_books.sndrCompanyid 1 Using where
1 PRIMARY receiver_company eq_ref PRIMARY,db_companies_icd PRIMARY 7 mde_staging.db_books.target_company_id 1 Using where
1 PRIMARY sender_team_id eq_ref PRIMARY,db_teams_i PRIMARY 7 mde_staging.db_books.sender_team_id 1
1 PRIMARY receiver_user ref i_db_books_to_users_bookid i_db_books_to_users_bookid 7 mde_staging.db_books.bookid 1
1 PRIMARY receiver_user_details eq_ref PRIMARY,u2 PRIMARY 7 mde_staging.receiver_user.userid 1
1 PRIMARY receiver_teams ref i_db_books_to_teams_bookid i_db_books_to_teams_bookid 7 mde_staging.db_books.bookid 1
1 PRIMARY receiver_team_details eq_ref PRIMARY,db_teams_i PRIMARY 7 mde_staging.receiver_teams.teamid 1
1 PRIMARY status eq_ref PRIMARY PRIMARY 4 mde_staging.db_books.statusid 1
1 PRIMARY state eq_ref PRIMARY PRIMARY 4 mde_staging.db_books.request_state_id 1
1 PRIMARY purpose eq_ref PRIMARY PRIMARY 4 mde_staging.db_books.request_purpose_id 1
1 PRIMARY assignee eq_ref PRIMARY,u2 PRIMARY 7 mde_staging.db_books.assignee 1
1 PRIMARY mdtl ref db_books_details_bookid db_books_details_bookid 7 mde_staging.db_books.bookid 1
1 PRIMARY request_type ALL PRIMARY 4 Using where; Using join buffer
1 PRIMARY g eq_ref PRIMARY,db_entities7 PRIMARY 7 mde_staging.mdtl.entityid 1
3 DEPENDENT SUBQUERY db_books_to_users ref i_db_books_to_users_bookid i_db_books_to_users_bookid 7 func 1 Using where; Using temporary
4 DEPENDENT UNION db_books_to_teams ref i_db_books_to_teams_bookid i_db_books_to_teams_bookid 7 func 1 Using where; Using temporary
5 DEPENDENT SUBQUERY b eq_ref PRIMARY,db_teams_i PRIMARY 7 func 1 Using where
5 DEPENDENT SUBQUERY a ref db_team_users_i db_team_users_i 11 func,const 1 Using where
UNION RESULT <union3,4> ALL
2 DEPENDENT SUBQUERY b eq_ref PRIMARY,db_teams_i PRIMARY 7 func 1 Using where
2 DEPENDENT SUBQUERY a ref db_team_users_i db_team_users_i 7 func 1 Using where
If you see the first row of the explain plan, it is not using the possible index and then using file sort etc. Not sure if that is the problem. Please suggest me how to fix this or me what indexes to use??
The biggest problem I see is the subquery qualifiers. Those hit per every row tested. I would then change the WHERE clause portion to just a prequery as the first table and get those resulting books and join to books, then the rest should be fine. In addition, the clause "STRAIGHT_JOIN" tells the engine to do the query in the order you've said. Sometimes, it gets to ahead of you and tries to optimize based on one of the "lookup" reference tables and back-fill find the rest. All that said,
CHANGE the SELECT at the top to
select STRAIGHT_JOIN distinct
and then your from clause from
from
db_books db_books
to
from
( SELECT distinct db.bookid
from
db_books db
left join db_team_users TeamA
ON db.sndrUserID = TeamA.userID
AND db.Sender_Team_ID = TeamA.TeamID
LEFT JOIN db_teams TeamB
ON TeamA.TeamID = TeamB.TeamID
AND TeamB.Deleted = 0
left join db_books_to_users ToUser
ON db.BookID = ToUser.BookID
AND db.sndrUserID = ToUser.userID
left join db_books_to_teams ToTeamA
ON db.TeamID = ToTeamA.TeamID
AND db.sndrUserID = ToTeamA.UserID
AND a.Deleted = 0
left join db_teams ToTeamsB
ON ToTeamA.TeamID = ToTeamB.TeamID
AND b.Deleted = 0
where
db.sndrUserID = 25000000003265
OR NOT TeamB.TeamID IS NULL
OR NOT ToUser.BookID IS NULL
OR NOT ToTeamB.TeamID IS NULL
limit
20 ) PreQualBooks
JOIN db_books
ON PreQualBooks.BookID = db_Books.BookID
And you can remove the Final WHERE clause as this prequery will be done ONCE up front to pre-qualify every POSSIBLE book ID based on user or team relationship with JOINs. By allowing LEFT JOIN, the books table goes through ONCE, with all the respective relationships to team / user status and will only return those records based on the send user OR the lowest level of the respective LEFT JOINs (TeamB, ToUser and ToTeamB). This prequery also applies the limit to 20 books, so the LIMIT clause at the end of your query is not needed either as only 20 books will ever be POSSIBLE.
Leave your Outer GROUP BY due to your group_concat.