My question is about MySQL optimisation and good practice.
I have to get translated text list with order preference : if I can't find text in english, I want to use french text and if it does not exists, I want to use spanish.
In other words, I have :
id lang text
1 fr text_FR
1 en text_EN
1 es text_ES
2 fr text_FR2
3 fr text_FR3
3 es text_ES3
And I want:
id lang text
1 en text_EN
2 fr text_FR2
3 fr text_FR3
So I read some topics like this, this, that, or that. Ok.
I tried this :
SELECT text FROM (
SELECT id, text
FROM translation
ORDER BY FIELD(lang, 'en', 'fr', 'es')
) tt GROUP BY tt.id;
But, when I EXPLAIN this query, I can read :
id select_type table partitions type possible_keys key key_len ref rows Extra
1 PRIMARY <derived2> NULL ALL NULL NULL NULL NULL 4 Using temporary; Using filesort
2 DERIVED translation NULL ALL PRIMARY PRIMARY 52 1641 Using filesort
So it's not optimized, because of the subquery. Can I avoid this subquery ?
I would suggest JOINing on a table by id for each language. You can use a LEFT JOIN to allow for null results. The query will still use temporary/filesort because it has to scan the entire table (you can use a WHERE to limit it more), but only for the first table. Lookups on the joined tables should be quick.
SELECT
COALESCE(en.text, fr.text, es.text, any.text)
FROM
f any
LEFT JOIN f en ON (any.id = en.id AND en.lang = 'en')
LEFT JOIN f fr ON (any.id = fr.id AND fr.lang = 'fr')
LEFT JOIN f es ON (any.id = es.id AND es.lang = 'es')
GROUP BY any.id;
http://sqlfiddle.com/#!2/baafc/1
Related
i see a strange behavior of Mysql when i launch two different query with one small difference, the position of one left join.
The slow query:
SELECT i.id_affiliate, i.id_franchising, i.Codice
FROM network_configuration_affiliate AS c
INNER JOIN franchising AS fr ON fr.id = c.id_franchising
INNER JOIN network_selected_car AS i ON c.id_affiliate = i.id_affiliate
INNER JOIN (
select T1.id_car, T1.id_network, T1.id_franchising, T1.id_agencie
from network_car_destinations as T1
where T1.id_network='12' and ( T1.id_franchising = 968 or T1.id_franchising = 974 )
) AS n ON n.id_franchising=i.id_franchising AND n.id_car=i.id_car AND c.id_network=n.id_network
INNER JOIN affiliate_tipologies AS t ON t.id_tipology_ag=i.idCategory AND t.id_franchising=i.id_franchising
INNER JOIN network_assoc_tipologies AS p ON p.id_network=c.id_network AND p.id_default=t.id_tipology_net
LEFT JOIN network_conf_users as ce on ce.id_affiliate = i.id_affiliate and ce.id_user = i.id_user
WHERE c.id_network='12' and c.code_affiliate='69842' AND c.configured = 'yes' AND i.Code_car not like '' and p.code != '0'
GROUP BY i.Code_car
The quick:
select T2.* from (
SELECT i.`id_affiliate`, i.id_franchising, i.Code_car, i.id_user
FROM network_configuration_affiliate AS c
INNER JOIN franchising AS fr ON fr.id = c.id_franchising
INNER JOIN network_selected_car AS i ON c.`id_affiliate` = i.`id_affiliate`
INNER JOIN (
select T1.id_car, T1.id_network, T1.id_franchising, T1.id_agencie
from network_car_destinations as T1
where T1.id_network='12' and ( T1.id_franchising = 968 or T1.id_franchising = 974 )
) AS n ON n.id_franchising=i.id_franchising AND n.id_car=i.id_car AND c.id_network=n.id_network
INNER JOIN affiliate_tipologies AS t ON t.id_tipology_ag=i.idCategory AND t.id_franchising=i.id_franchising
INNER JOIN network_assoc_tipologies AS p ON p.id_network=c.id_network AND p.id_default=t.id_tipology_net
WHERE c.id_network='12' and c.code_affiliate='69842' AND c.configured = 'yes' AND i.Code_car not like ''
GROUP BY i.Code_car
) as T2
LEFT JOIN network_conf_users as ce on ce.`id_affiliate` = T2.`id_affiliate` and ce.id_user = T2.id_user
WHERE c.id_network='12' and c.code_affiliate='69842' AND c.configured = 'yes' AND i.Code_car not like '' and p.code != '0'
GROUP BY i.Code_car
The EXPLAIN are almost the same, the result is the same for both query, but the first query is taking 20 secs to end, the second 0.02, how the position of the left join can influence so much the execution of the query?
EXPLAIN - Slow query:
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY c index_merge id_affiliate,id_network,configured,code_affiliate code_affiliate,id_network 203,5 NULL 1 100.00 Using intersect(code_affiliate,id_network); Using where; Using temporary; Using filesort
1 PRIMARY fr eq_ref PRIMARY PRIMARY 4 c.id_franchising 1 100.00 Using index
1 PRIMARY <derived2> ref <auto_key1> <auto_key1> 5 const 10 100.00 Using where
1 PRIMARY t ref id_franchising,id_tipology_ag,id_tipology_net id_franchising 5 n.id_franchising 13 100.00 Using where
1 PRIMARY p ref id_network,id_default id_default 5 t.id_tipology 26 100.00 Using where
1 PRIMARY i ref id_car,id_affiliate,id_franchising,idCategory id_car 5 n.id_car 3 100.00 Using where
1 PRIMARY ce ALL id_affiliate,id_user NULL NULL NULL 4 75.00 Using where; Using join buffer (Block Nested Loop)
2 DERIVED T1 ref id_franchising,id_network id_network 5 const 136952 100.00 Using where
SHOW WARNINGS:
Level Code Message
Note 1003 /* select#1 */
select `i`.`id_affiliate` AS `id_affiliate`,
`i`.`id_franchising` AS `id_franchising`,
`i`.`Code_car` AS `Code_car`
from `network_configuration_affiliate` `c`
join `franchising` `fr`
join `network_selected_car` `i`
join (/* select#2 */
select `T1`.`id_car` AS `id_car`,
`T1`.`id_network` AS `id_network`,
`T1`.`id_franchising` AS `id_franchising`,
`T1`.`id_agencie` AS `id_agencie`
from `network_car_destinations` `T1`
where ((`T1`.`id_network` = '12')
and ((`T1`.`id_franchising` = 968)
or (`T1`.`id_franchising` = 974)))
) `n`
join `affiliate_tipologies` `t`
join `network_assoc_tipologies` `p`
left join `network_conf_users` `ce` on(((`ce`.`id_user` = `i`.`id_user`)
and (`ce`.`id_affiliate` = `c`.`id_affiliate`))
)
where ((`fr`.`id` = `c`.`id_franchising`)
and (`i`.`id_affiliate` = `c`.`id_affiliate`)
and (`i`.`id_car` = `n`.`id_car`)
and (`t`.`id_franchising` = `n`.`id_franchising`)
and (`i`.`id_franchising` = `n`.`id_franchising`)
and (`i`.`idCategory` = `t`.`id_tipology_ag`)
and (`p`.`id_default` = `t`.`id_tipology_net`)
and (`n`.`id_network` = `c`.`id_network`)
and (`p`.`id_network` = `c`.`id_network`)
and (`c`.`configured` = 'yes')
and (`c`.`code_affiliate` = '69842')
and (`c`.`id_network` = '12')
and (not((`i`.`Code` like '')))
and (`p`.`code` <> '0')
)
group by `i`.`Code_car`
Fast query:
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 7280 100.00 NULL
1 PRIMARY ce ALL id_affiliate,id_user NULL NULL NULL 4 75.00 Using where; Using join buffer (Block Nested Loop)
2 DERIVED c index_merge id_affiliate,id_network,configured,code_affiliate code_affiliate,id_network 203,5 NULL 1 100.00 Using intersect(codice_affiliato,id_network); Using where; Using temporary; Using filesort
2 DERIVED fr eq_ref PRIMARY PRIMARY 4 c.id_franchising 1 100.00 Using index
2 DERIVED <derived3> ref <auto_key1> <auto_key1> 5 const 10 100.00 Using where
2 DERIVED i ref id_car,id_affiliate,id_franchising,idCategory id_car 5 n.id_car 2 100.00 Using where
2 DERIVED t ref id_franchising,id_tipology_ag,id_tipology_net id_franchising 5 n.id_franchising 14 100.00 Using where
2 DERIVED p ref id_network,id_default id_default 5 t.id_tipology_net 26 100.00 Using where
3 DERIVED T1 ref id_franchising,id_network id_network 5 const 133324 100.00 Using where
SHOW WARNINGS:
Level Code Message
Note 1003 /* select#1 */
select `T2`.id_affiliate AS id_affiliate,
`T2`.`id_franchising` AS `id_franchising`,
`T2`.Code_car AS Code_car,`T2`.id_user AS id_user
from (/* select#2 */
select `i`.id_affiliate AS id_affiliate,
`i`.`id_franchising` AS `id_franchising`,
`i`.Code_car AS Code_car,`i`.id_user AS id_user
from network_configuration_affiliate `c`
join `franchising` `fr`
join network_selected_car `i`
join (/* select#3 */
select `T1`.id_car AS id_car,
`T1`.`id_network` AS `id_network`,
`T1`.`id_franchising` AS `id_franchising`,
`T1`.`id_agencie` AS `id_agencie`
from network_car_destinations `T1`
where ((`T1`.`id_network` = '12')
and ((`T1`.`id_franchising` = 968)
or (`T1`.`id_franchising` = 974)))
) `n`
join affiliate_tipologies `t`
join network_assoc_tipologies `p`
where ((`fr`.`id` = `c`.`id_franchising`)
and (`i`.id_affiliate = `c`.id_affiliate)
and (`i`.id_car = `n`.id_car)
and (`i`.`id_franchising` = `n`.`id_franchising`)
and (`t`.`id_franchising` = `n`.`id_franchising`)
and (`t`.`id_tipology_ag` = `i`.`idCategory`)
and (`p`.`id_default` = `t`.`id_tipology_net`)
and (`n`.`id_network` = `c`.`id_network`)
and (`p`.`id_network` = `c`.`id_network`)
and (`c`.`configured` = 'yes')
and (`c`.`code_affiliate` = '69842')
and (`c`.`id_network` = '12')
and (not((`i`.Code_car like '')))
)
group by `i`.Code_car
) `T2`
left join network_conf_users `ce` on(((`ce`.id_affiliate = `T2`.id_affiliate)
and (`ce`.id_user = `T2`.id_user))
)
where 1
The left join finds those fields to the left and retrieves those from the right, while inner joining will lookup all fields. Keeping it simple by not needing to organize the extra columns the query completes much quicker.
Please show us the EXPLAINs.
The Optimizer will turn LEFT JOIN into JOIN if it decides that it makes no difference. Please do EXPLAIN EXTENDED SELECT ... followed immediately by SHOW WARNINGS; so we can see whether this happened.
The Optimizer will try a variety of orders for JOINing (in the absence of LEFT or RIGHT). So, if the LEFT was really redundant, I would expect the EXPLAINs to have the tables in the same order.
Normally, the Optimizer will start with a "derived" table, n in your examples. But it may be hidden behind the LEFT.
Do you have any "composite" indexes? These are likely to be beneficial:
T1: INDEX(id_network, id_franchising) -- in this order
c: INDEX(id_network, code_affiliate, configured)
Revise
With the recent changes to the query, I see this pattern:
Slow:
SELECT ...
FROM ...
JOIN ...
GROUP BY ...
Quick:
SELECT ...
FROM ( SELECT ...
FROM ...
GROUP BY ... )
JOIN ...
(LEFT is only partially important to the question; any JOIN can exhibit the problem.)
I call the slow one "inflate-deflate". By that I mean that first it did join(s), thereby inflating the number of rows. Then it did a GROUP BY, which deflated the results.
The quick one deflated before doing the final join, there by leading to less effort overall.
index merge intersect can almost always be improved by using a 'composite' index. In this case, the one I suggest for c. (This would speed up both versions of the query.)
The query is as below:
EXPLAIN EXTENDED
SELECT u0_.value AS value0, u1_.property_uri AS property_uri1, u2_.service_id AS sclr2, count(u0_.id) AS sclr3
FROM usc_account_triple u0_
INNER JOIN usc_account_connection u3_ ON ((u0_.service_subscriber_id = u3_.account_2_id))
INNER JOIN usc_service_subscriber u2_ ON ((u3_.account_1_id = u2_.id))
INNER JOIN usc_property u1_ ON u0_.property_id = u1_.id AND (u1_.status = 1)
WHERE (u1_.create_analytics = '2') AND (u0_.status = 1) AND (u3_.status = 1)
AND (u2_.status = 1) GROUP BY u0_.property_id, u0_.value, u2_.service_id;
Here, along with other table's index, I am trying to create index on table 'usc_service_subscriber' table as like below:
CREATE INDEX `temp` ON usc_service_subscriber(id, status, service_id);
But, unfortunately the index isn't getting selected by the optimizer. Here is what explain command return:
| 1 | SIMPLE | u2_ | eq_ref | PRIMARY,temp | PRIMARY | 8 | bootsat.u3_.account_1_id | 1 | 100.00 | Using where
Any Idea why? Is there anything wrong with the combination or ordering of the index?
You can force the use of index using the syntax FORCE INDEX (index_name) but to my knowledge, in your case the index is not used because of too few rows to be selected.
Per your posted explain result, it looks like only 1 row to be fetched/selected and probably that's the very same reason index is not used.
See Here for more information on MySQL Index Hints.
Can someone please help me optimize following query? Its to search for a search term and result data which will be used in an auto complete form. Data will be sent as JSON (probably) or HTML (). But for now, my concern is how I can optimize this query. There will be around 20000 users per day on the site (maybe all at same time) and I am hoping to optimize this as much as I can.
As some might have guessed, these are all Drupal tables and I am generating a custom query.
EXPLAIN SELECT n.nid AS nid, fcs.field_call_sign_value AS field_call_sign_value, old.field_r_13_n_old_value AS field_r_13_n_old_value,
new.field_r_13_n_new_value AS field_r_13_n_new_value,fn.field_name_value AS field_name_value
FROM node n
INNER JOIN field_data_field_call_sign fcs ON n.nid = fcs.entity_id and n.vid=fcs.revision_id
INNER JOIN field_data_field_name fn ON n.nid = fn.entity_id and n.vid=fn.revision_id
INNER JOIN field_data_field_r_13_n_old old ON n.nid = old.entity_id and n.vid=old.revision_id
INNER JOIN field_data_field_r_13_n_new new ON n.nid = new.entity_id and n.vid=new.revision_id
WHERE (n.title LIKE '%APTNHD%' ESCAPE '\\')
ORDER BY n.created DESC
LIMIT 5 OFFSET 0
Explain Result:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE fcs ALL entity_id,revision_id NULL NULL NULL 11 Using temporary; Using filesort
1 SIMPLE old ALL entity_id,revision_id NULL NULL NULL 11 Using where; Using join buffer
1 SIMPLE new ALL entity_id,revision_id NULL NULL NULL 11 Using where; Using join buffer
1 SIMPLE fn ALL entity_id,revision_id NULL NULL NULL 11 Using where; Using join buffer
1 SIMPLE n eq_ref PRIMARY,vid PRIMARY 4 DB.new.entity_id 1 Using where
Please let me know if you need any more information
EDIT: New EXPLAIN result after "Stephan"'s answer
Original Query took "0.0010 sec" (and same on multiple executes)
New Query took "0.0012 sec" (and "0.0007 sec" after running it again)
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE n ALL NULL NULL NULL NULL 44 Using where; Using filesort
1 SIMPLE fcs ref entity_id,revision_id entity_id 4 DB.n.nid 1
1 SIMPLE fn ref entity_id,revision_id entity_id 4 DB.n.nid 1
1 SIMPLE old ref entity_id,revision_id entity_id 4 DB.n.nid 1
1 SIMPLE new ref entity_id,revision_id entity_id 4 DB.n.nid 1
The mysql query optimizer has decided to switch the order the tables are joined in order to have less scanned rows but not in all cases this is good.
I see that for the joined table you have index on entity_id which is good , so you need to maintain the join table order using STRAIGHT_JOIN and LEFT JOIN:
EXPLAIN SELECT STRAIGHT_JOIN n.nid AS nid, fcs.field_call_sign_value AS field_call_sign_value, old.field_r_13_n_old_value AS field_r_13_n_old_value,
new.field_r_13_n_new_value AS field_r_13_n_new_value,fn.field_name_value AS field_name_value
FROM node n
LEFT JOIN field_data_field_call_sign fcs ON n.nid = fcs.entity_id and n.vid=fcs.revision_id
LEFT JOIN field_data_field_name fn ON n.nid = fn.entity_id and n.vid=fn.revision_id
LEFT JOIN field_data_field_r_13_n_old old ON n.nid = old.entity_id and n.vid=old.revision_id
LEFT JOIN field_data_field_r_13_n_new new ON n.nid = new.entity_id and n.vid=new.revision_id
WHERE (n.title LIKE '%APTNHD%' ESCAPE '\\')
ORDER BY n.created DESC
LIMIT 5 OFFSET 0
Furthermore you can add a FULLTEXT INDEX on n.title column because then you can use fulltext-searchs which are faster then (n.title LIKE '%APTNHD%' ESCAPE '\\')
I have a mySQL table (myISAM) containing approximately two million rows - name, address, company data. The first name and surname are held in separate columns, so I also have a second table (linked by the primary key of the first) which holds a single full name column.
The first name, surname, and company name (among others) in the first table are indexed, as is the full name column in the secondary table.
Taking this query as a starting point:
SELECT * FROM table_a INNER JOIN table_b ON table_a.ID = table_b.ID WHERE....
searching exact match or even after-like on the name columns works in milliseconds:
....table_a.first_name = 'Fred'
....table_a.surname = 'Bloggs'
....table_b.fullname = 'Fred Bloggs'
....table_a.first_name LIKE 'Mike%'
just a few examples.
Throw the COMPANY NAME in there as well..... the query suddenly takes 15 to 20 seconds:
....table_a.first_name = 'Fred' OR table_a.company_name = 'Widgets Inc'
for example
Both fields are indexed, it's an exact match.... why would the addition of a second indexed search column slow things down so much? Have I missed something about my table design?
Examples follow - there are a few other tables joined but I'm not sure these are affecting performance:
Example of name-only query which returns in 0.0123 seconds:
SELECT SQL_CALC_FOUND_ROWS
webmaster.dupe_master_id AS webmaster_id,
webmaster.first_name,
webmaster.family_name,
webmaster.job_title,
webmaster.company_name,
webmaster.address_1,
webmaster.address_2,
webmaster.town_city,
webmaster.state_county,
webmaster.post_code,
webmaster.email,
webmaster.ignored,
countries.country_name,
GROUP_CONCAT(DISTINCT titles.code ORDER BY code ASC) AS sub_string,
'' AS expo_string
FROM
(`webmaster`)
LEFT JOIN `countries` ON `countries`.`country_id` = `webmaster`.`country_id`
LEFT JOIN `red_subscriptions` ON `red_subscriptions`.`webmaster_id` = `webmaster`.`webmaster_id` AND red_subscriptions.subscription_status_id = 2
LEFT JOIN `titles` ON `titles`.`title_id` = `red_subscriptions`.`title_id`
LEFT JOIN `webmaster_tags` ON `webmaster_tags`.`webmaster_id` = `webmaster`.`webmaster_id`
LEFT JOIN `tags` ON `tags`.`tag_id` = `webmaster_tags`.`tag_id`
INNER JOIN `webmaster_search_data` ON `webmaster`.`webmaster_id` = `webmaster_search_data`.`webmaster_id`
WHERE
(full_name = '<name>')
GROUP BY
`webmaster`.`dupe_master_id`
LIMIT 50
Add in company_name (also indexed) and the query time goes through the roof:
SELECT SQL_CALC_FOUND_ROWS
webmaster.dupe_master_id AS webmaster_id,
webmaster.first_name,
webmaster.family_name,
webmaster.job_title,
webmaster.company_name,
webmaster.address_1,
webmaster.address_2,
webmaster.town_city,
webmaster.state_county,
webmaster.post_code,
webmaster.email,
webmaster.ignored,
countries.country_name,
GROUP_CONCAT(DISTINCT titles.code ORDER BY code ASC) AS sub_string,
'' AS expo_string
FROM
(`webmaster`)
LEFT JOIN `countries` ON `countries`.`country_id` = `webmaster`.`country_id`
LEFT JOIN `red_subscriptions` ON `red_subscriptions`.`webmaster_id` = `webmaster`.`webmaster_id` AND red_subscriptions.subscription_status_id = 2
LEFT JOIN `titles` ON `titles`.`title_id` = `red_subscriptions`.`title_id`
LEFT JOIN `webmaster_tags` ON `webmaster_tags`.`webmaster_id` = `webmaster`.`webmaster_id`
LEFT JOIN `tags` ON `tags`.`tag_id` = `webmaster_tags`.`tag_id`
INNER JOIN `webmaster_search_data` ON `webmaster`.`webmaster_id` = `webmaster_search_data`.`webmaster_id`
WHERE
(full_name = '<name>' OR company_name '<name>')
GROUP BY
`webmaster`.`dupe_master_id`
LIMIT 50
EXPLAIN on full_name only:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE webmaster_search_data ref webmaster_id,full_name full_name 302 const 94 Using where; Using temporary; Using filesort
1 SIMPLE webmaster eq_ref PRIMARY PRIMARY 4 webmaster_search_data.webmaster_id 1
1 SIMPLE countries eq_ref PRIMARY PRIMARY 2 webmaster.country_id 1
1 SIMPLE red_subscriptions ref webmaster_id,subscription_status_id webmaster_id 4 webmaster_search_data.webmaster_id 1
1 SIMPLE titles eq_ref PRIMARY PRIMARY 2 red_subscriptions.title_id 1
1 SIMPLE webmaster_tags ref webmaster_id webmaster_id 4 webmaster_search_data.webmaster_id 5
1 SIMPLE tags eq_ref PRIMARY PRIMARY 2 webmaster_tags.tag_id 1 Using index
Explain when company_name is added:
1 SIMPLE webmaster index PRIMARY,company_name dupe_master_id 4 NULL 2072015 Using filesort
1 SIMPLE countries eq_ref PRIMARY PRIMARY 2 webmaster.country_id 1
1 SIMPLE red_subscriptions ref webmaster_id,subscription_status_id webmaster_id 4 webmaster.webmaster_id 1
1 SIMPLE titles eq_ref PRIMARY PRIMARY 2 red_subscriptions.title_id 1
1 SIMPLE webmaster_tags ref webmaster_id webmaster_id 4 webmaster.webmaster_id 5
1 SIMPLE tags eq_ref PRIMARY PRIMARY 2 webmaster_tags.tag_id 1 Using index
1 SIMPLE webmaster_search_data eq_ref webmaster_id,full_name webmaster_id 4 webmaster.webmaster_id 1 Using where
MySQL cannot use two indexes at once. When you throw in the company name, MySQL cannot use the index on Firstname, Lastname anymore because now there are more columns it has to check to get an exact result.
It is probably doing a full table scan.
You could split your queries up by doing a Union, that way you can use both columns with the index.
SELECT * FROM
( SELECT * FROM table_a
INNER JOIN table_b ON table_a.ID = table_b.ID
WHERE table_a.first_name = 'Fred'
UNION
SELECT * FROM table_a
INNER JOIN table_b ON table_a.ID = table_b.ID
WHERE table_a.company_name = 'Widgets Inc'
) sub;
Each query should be evaluated separately and use the adequate index. THe UNION will take care of doubles, so you will in the end have the same result.
I have this complex query which produces 3744 rows in about 50ms.
SELECT
srl.event_id as eid
, srl.race_num as rnum
, bts.boat_id as bid_id
, srl.series_year as yr
, srl.id as id
, IFNULL(rfi.fleet,fleet_def) as flt_old,flt_match,s.series_id as sid
, s.series_year as syr
,IFNULL(ovr_pts,POINTS('4',IFNULL(ovr_place,place),num_start)) as points
FROM
(SELECT en1.boat_id,en1.boat_name,MAX(fleet) as fleet_def FROM entries en1
JOIN series_race_list srl1 ON srl1.event_id=en1.event_id
AND srl1.series_year=en1.race_year
LEFT JOIN entries_race er1 ON en1.boat_id= er1.boat_id
AND srl1.event_id=en1.event_id
AND srl1.series_year =en1.race_year
WHERE srl1.series_id ='3' AND srl1.series_year ='2012'
AND en1.entry_deleted='N'
GROUP BY boat_id) bts
JOIN series_race_list srl LEFT JOIN series as s ON s.series_id=srl.series_id
AND s.series_year =srl.series_year
LEFT JOIN entries as en ON srl.event_id=en.event_id
AND srl.series_year =en.race_year AND bts.boat_id =en.boat_id
LEFT JOIN entries_race er ON er.race_id= srl.event_id AND er.race_num=srl.race_num
AND er.yr = srl.series_year AND bts.boat_id =er.boat_id
LEFT JOIN event_race_info as eri ON eri.race_id= srl.event_id
AND eri.race_num=srl.race_num AND eri.yr = srl.series_year
ANd er.line=eri.line AND status REGEXP 'prelim|final'
LEFT JOIN race_results as rr ON srl.event_id=rr.race_id
AND srl.race_num= rr.race_num AND srl.series_year =rr.yr
AND bts.boat_id= rr.boat_id AND checked_in='Y'
LEFT JOIN race_fleet_info as rfi ON rfi.race_id= srl.event_id
AND rfi.yr=srl.series_year AND srl.race_num= rfi.race_num
AND rfi.fleet=rr.flt AND complete='Y'
LEFT JOIN series_pts_override as spo ON srl.id =spo.id AND en.boat_id =spo.bid
WHERE s.series_id ='3' AND s.series_year ='2012' AND approved ='Y'
Sorry for the length. As I said this query executes in around 50ms. Now I want to use this data and perform queries on this 3744 row result. As soon I as wrap this with a query like
SELECT eid FROM(
......previous query here.....
) data
The execution time goes from 50 ms to 2.5 sec Ouch!
I tried creating a temporary table, That was the same. (Actually this is my preferred approach since I will need to do a few different queries on this results set.
Reading on this site I don't think this is a correlated sub query but is seems to be acting like one.
Seems like the act of creating a alias table is my issues, since the sub query has a derived table alias and the temp table is obviously a table.
How can I get access to these 3744 rows of data with out this time penalty?
If it would help I can figure out how to post the Explains.
Explain for the longer query:
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 3744
2 DERIVED s const PRIMARY PRIMARY 5 1
2 DERIVED srl ref series_id,series_id_2 series_id 5 16 Using where
2 DERIVED <derived3> ALL NULL NULL NULL NULL 208 Using join buffer
2 DERIVED en eq_ref PRIMARY,event_id,event_id_2 PRIMARY 9 race_reg_test.srl.event_id,bts.boat_id 1 Using index
2 DERIVED er ref PRIMARY,boat_id,boat_id_2 boat_id_2 5 bts.boat_id 5
2 DERIVED eri eq_ref PRIMARY PRIMARY 13 race_reg_test.srl.event_id,race_reg_test.srl.race_... 1
2 DERIVED rr ref PRIMARY,boat_id boat_id 4 bts.boat_id 9
2 DERIVED rfi eq_ref PRIMARY PRIMARY 31 race_reg_test.srl.event_id,race_reg_test.srl.race_... 1
2 DERIVED spo ref PRIMARY PRIMARY 8 race_reg_test.srl.id,race_reg_test.en.boat_id 1
3 DERIVED srl1 ref series_id,series_id_2 series_id 5 16 Using index; Using temporary; Using filesort
3 DERIVED en1 ref PRIMARY,event_id,event_id_2 PRIMARY 5 race_reg_test.srl1.event_id 11 Using where
3 DERIVED er1 ref boat_id,boat_id_2 boat_id 4 race_reg_test.en1.boat_id 9 Using index
You said you tried creating a temporary table, I am not sure if by that you mean a View or not.
I would create a View with that query and then perform any queries necessary on the View.
CREATE VIEW massive_query_view AS
SELECT
srl.event_id as eid
, srl.race_num as rnum
, bts.boat_id as bid_id
, srl.series_year as yr
, srl.id as id
, IFNULL(rfi.fleet,fleet_def) as flt_old,flt_match,s.series_id as sid
, s.series_year as syr
,IFNULL(ovr_pts,POINTS('4',IFNULL(ovr_place,place),num_start)) as points
FROM
(SELECT en1.boat_id,en1.boat_name,MAX(fleet) as fleet_def FROM entries en1
JOIN series_race_list srl1 ON srl1.event_id=en1.event_id
AND srl1.series_year=en1.race_year
LEFT JOIN entries_race er1 ON en1.boat_id= er1.boat_id
AND srl1.event_id=en1.event_id
AND srl1.series_year =en1.race_year
WHERE srl1.series_id ='3' AND srl1.series_year ='2012'
AND en1.entry_deleted='N'
GROUP BY boat_id) bts
JOIN series_race_list srl LEFT JOIN series as s ON s.series_id=srl.series_id
AND s.series_year =srl.series_year
LEFT JOIN entries as en ON srl.event_id=en.event_id
AND srl.series_year =en.race_year AND bts.boat_id =en.boat_id
LEFT JOIN entries_race er ON er.race_id= srl.event_id AND er.race_num=srl.race_num
AND er.yr = srl.series_year AND bts.boat_id =er.boat_id
LEFT JOIN event_race_info as eri ON eri.race_id= srl.event_id
AND eri.race_num=srl.race_num AND eri.yr = srl.series_year
ANd er.line=eri.line AND status REGEXP 'prelim|final'
LEFT JOIN race_results as rr ON srl.event_id=rr.race_id
AND srl.race_num= rr.race_num AND srl.series_year =rr.yr
AND bts.boat_id= rr.boat_id AND checked_in='Y'
LEFT JOIN race_fleet_info as rfi ON rfi.race_id= srl.event_id
AND rfi.yr=srl.series_year AND srl.race_num= rfi.race_num
AND rfi.fleet=rr.flt AND complete='Y'
LEFT JOIN series_pts_override as spo ON srl.id =spo.id AND en.boat_id =spo.bid
WHERE s.series_id ='3' AND s.series_year ='2012' AND approved ='Y'
Then, you can perform queries on the View.
SELECT * FROM massive_query_view;
Hope that speeds things up. Another thing you can do is check your indexes. Indexes make where clauses faster but inserts slower. For more information, view the MySQL documentation on how MySQL uses indexes: http://dev.mysql.com/doc/refman/5.0/en/mysql-indexes.html.
A few things, but the biggest one I see is in your original query... at the point of
boat_id) bts
JOIN series_race_list srl
LEFT JOIN series as s
You have no "ON" condition between bts and srl which will result in a Cartesian result and probably a big killer to you. For every record in bts, its creating an entry in srl, then from that product joining to series. From srl to series is ok as it is joined on apparent valid criteria / keys.
Next, you have a few fields that are not alias.field, such as max(fleet) in inner-most query that aliases out to "bts". In addition, why the MAX(fleet) if its grouped by the boat ID which I would interpret as a primary key and would be unique... would a boat ever change it's fleet? If so, is this accurate? If you have a table of fleets (also having its own auto-sequence ID), and a boat changes ownership/sponsor ship (whatever) to a pre-existing fleet from say... fleet 93 to a new who already had an ID on file of 47 where even though 47 was the newest relationship, but and older pre-existing ID... is that what you really want? MAX()?
Additional fields for no alias.field: ovr_pts and ovr_place, place in the field list (and what is the POINTS() function... status at the regular expression, checked_in at race results, and complete at race fleet info, and finally approved in the final where clause. Minor, but could be helpful for index optimizing.
Lastly, your query has the WHERE clause on specific "s.series_id... and s.series_year..." yet you have a LEFT-JOIN earlier in the query. This basically cancels out the left-join component of it and turns it into an implied INNER JOIN since you are not allowing for NULL as a valid option of inclusion.
After some clarification, I might even suggest altering the query around some, but the biggest thing I see was from the start... no "ON" condition joining bts and the series_rate_list table.