Can someone please help me optimize following query? Its to search for a search term and result data which will be used in an auto complete form. Data will be sent as JSON (probably) or HTML (). But for now, my concern is how I can optimize this query. There will be around 20000 users per day on the site (maybe all at same time) and I am hoping to optimize this as much as I can.
As some might have guessed, these are all Drupal tables and I am generating a custom query.
EXPLAIN SELECT n.nid AS nid, fcs.field_call_sign_value AS field_call_sign_value, old.field_r_13_n_old_value AS field_r_13_n_old_value,
new.field_r_13_n_new_value AS field_r_13_n_new_value,fn.field_name_value AS field_name_value
FROM node n
INNER JOIN field_data_field_call_sign fcs ON n.nid = fcs.entity_id and n.vid=fcs.revision_id
INNER JOIN field_data_field_name fn ON n.nid = fn.entity_id and n.vid=fn.revision_id
INNER JOIN field_data_field_r_13_n_old old ON n.nid = old.entity_id and n.vid=old.revision_id
INNER JOIN field_data_field_r_13_n_new new ON n.nid = new.entity_id and n.vid=new.revision_id
WHERE (n.title LIKE '%APTNHD%' ESCAPE '\\')
ORDER BY n.created DESC
LIMIT 5 OFFSET 0
Explain Result:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE fcs ALL entity_id,revision_id NULL NULL NULL 11 Using temporary; Using filesort
1 SIMPLE old ALL entity_id,revision_id NULL NULL NULL 11 Using where; Using join buffer
1 SIMPLE new ALL entity_id,revision_id NULL NULL NULL 11 Using where; Using join buffer
1 SIMPLE fn ALL entity_id,revision_id NULL NULL NULL 11 Using where; Using join buffer
1 SIMPLE n eq_ref PRIMARY,vid PRIMARY 4 DB.new.entity_id 1 Using where
Please let me know if you need any more information
EDIT: New EXPLAIN result after "Stephan"'s answer
Original Query took "0.0010 sec" (and same on multiple executes)
New Query took "0.0012 sec" (and "0.0007 sec" after running it again)
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE n ALL NULL NULL NULL NULL 44 Using where; Using filesort
1 SIMPLE fcs ref entity_id,revision_id entity_id 4 DB.n.nid 1
1 SIMPLE fn ref entity_id,revision_id entity_id 4 DB.n.nid 1
1 SIMPLE old ref entity_id,revision_id entity_id 4 DB.n.nid 1
1 SIMPLE new ref entity_id,revision_id entity_id 4 DB.n.nid 1
The mysql query optimizer has decided to switch the order the tables are joined in order to have less scanned rows but not in all cases this is good.
I see that for the joined table you have index on entity_id which is good , so you need to maintain the join table order using STRAIGHT_JOIN and LEFT JOIN:
EXPLAIN SELECT STRAIGHT_JOIN n.nid AS nid, fcs.field_call_sign_value AS field_call_sign_value, old.field_r_13_n_old_value AS field_r_13_n_old_value,
new.field_r_13_n_new_value AS field_r_13_n_new_value,fn.field_name_value AS field_name_value
FROM node n
LEFT JOIN field_data_field_call_sign fcs ON n.nid = fcs.entity_id and n.vid=fcs.revision_id
LEFT JOIN field_data_field_name fn ON n.nid = fn.entity_id and n.vid=fn.revision_id
LEFT JOIN field_data_field_r_13_n_old old ON n.nid = old.entity_id and n.vid=old.revision_id
LEFT JOIN field_data_field_r_13_n_new new ON n.nid = new.entity_id and n.vid=new.revision_id
WHERE (n.title LIKE '%APTNHD%' ESCAPE '\\')
ORDER BY n.created DESC
LIMIT 5 OFFSET 0
Furthermore you can add a FULLTEXT INDEX on n.title column because then you can use fulltext-searchs which are faster then (n.title LIKE '%APTNHD%' ESCAPE '\\')
Related
i see a strange behavior of Mysql when i launch two different query with one small difference, the position of one left join.
The slow query:
SELECT i.id_affiliate, i.id_franchising, i.Codice
FROM network_configuration_affiliate AS c
INNER JOIN franchising AS fr ON fr.id = c.id_franchising
INNER JOIN network_selected_car AS i ON c.id_affiliate = i.id_affiliate
INNER JOIN (
select T1.id_car, T1.id_network, T1.id_franchising, T1.id_agencie
from network_car_destinations as T1
where T1.id_network='12' and ( T1.id_franchising = 968 or T1.id_franchising = 974 )
) AS n ON n.id_franchising=i.id_franchising AND n.id_car=i.id_car AND c.id_network=n.id_network
INNER JOIN affiliate_tipologies AS t ON t.id_tipology_ag=i.idCategory AND t.id_franchising=i.id_franchising
INNER JOIN network_assoc_tipologies AS p ON p.id_network=c.id_network AND p.id_default=t.id_tipology_net
LEFT JOIN network_conf_users as ce on ce.id_affiliate = i.id_affiliate and ce.id_user = i.id_user
WHERE c.id_network='12' and c.code_affiliate='69842' AND c.configured = 'yes' AND i.Code_car not like '' and p.code != '0'
GROUP BY i.Code_car
The quick:
select T2.* from (
SELECT i.`id_affiliate`, i.id_franchising, i.Code_car, i.id_user
FROM network_configuration_affiliate AS c
INNER JOIN franchising AS fr ON fr.id = c.id_franchising
INNER JOIN network_selected_car AS i ON c.`id_affiliate` = i.`id_affiliate`
INNER JOIN (
select T1.id_car, T1.id_network, T1.id_franchising, T1.id_agencie
from network_car_destinations as T1
where T1.id_network='12' and ( T1.id_franchising = 968 or T1.id_franchising = 974 )
) AS n ON n.id_franchising=i.id_franchising AND n.id_car=i.id_car AND c.id_network=n.id_network
INNER JOIN affiliate_tipologies AS t ON t.id_tipology_ag=i.idCategory AND t.id_franchising=i.id_franchising
INNER JOIN network_assoc_tipologies AS p ON p.id_network=c.id_network AND p.id_default=t.id_tipology_net
WHERE c.id_network='12' and c.code_affiliate='69842' AND c.configured = 'yes' AND i.Code_car not like ''
GROUP BY i.Code_car
) as T2
LEFT JOIN network_conf_users as ce on ce.`id_affiliate` = T2.`id_affiliate` and ce.id_user = T2.id_user
WHERE c.id_network='12' and c.code_affiliate='69842' AND c.configured = 'yes' AND i.Code_car not like '' and p.code != '0'
GROUP BY i.Code_car
The EXPLAIN are almost the same, the result is the same for both query, but the first query is taking 20 secs to end, the second 0.02, how the position of the left join can influence so much the execution of the query?
EXPLAIN - Slow query:
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY c index_merge id_affiliate,id_network,configured,code_affiliate code_affiliate,id_network 203,5 NULL 1 100.00 Using intersect(code_affiliate,id_network); Using where; Using temporary; Using filesort
1 PRIMARY fr eq_ref PRIMARY PRIMARY 4 c.id_franchising 1 100.00 Using index
1 PRIMARY <derived2> ref <auto_key1> <auto_key1> 5 const 10 100.00 Using where
1 PRIMARY t ref id_franchising,id_tipology_ag,id_tipology_net id_franchising 5 n.id_franchising 13 100.00 Using where
1 PRIMARY p ref id_network,id_default id_default 5 t.id_tipology 26 100.00 Using where
1 PRIMARY i ref id_car,id_affiliate,id_franchising,idCategory id_car 5 n.id_car 3 100.00 Using where
1 PRIMARY ce ALL id_affiliate,id_user NULL NULL NULL 4 75.00 Using where; Using join buffer (Block Nested Loop)
2 DERIVED T1 ref id_franchising,id_network id_network 5 const 136952 100.00 Using where
SHOW WARNINGS:
Level Code Message
Note 1003 /* select#1 */
select `i`.`id_affiliate` AS `id_affiliate`,
`i`.`id_franchising` AS `id_franchising`,
`i`.`Code_car` AS `Code_car`
from `network_configuration_affiliate` `c`
join `franchising` `fr`
join `network_selected_car` `i`
join (/* select#2 */
select `T1`.`id_car` AS `id_car`,
`T1`.`id_network` AS `id_network`,
`T1`.`id_franchising` AS `id_franchising`,
`T1`.`id_agencie` AS `id_agencie`
from `network_car_destinations` `T1`
where ((`T1`.`id_network` = '12')
and ((`T1`.`id_franchising` = 968)
or (`T1`.`id_franchising` = 974)))
) `n`
join `affiliate_tipologies` `t`
join `network_assoc_tipologies` `p`
left join `network_conf_users` `ce` on(((`ce`.`id_user` = `i`.`id_user`)
and (`ce`.`id_affiliate` = `c`.`id_affiliate`))
)
where ((`fr`.`id` = `c`.`id_franchising`)
and (`i`.`id_affiliate` = `c`.`id_affiliate`)
and (`i`.`id_car` = `n`.`id_car`)
and (`t`.`id_franchising` = `n`.`id_franchising`)
and (`i`.`id_franchising` = `n`.`id_franchising`)
and (`i`.`idCategory` = `t`.`id_tipology_ag`)
and (`p`.`id_default` = `t`.`id_tipology_net`)
and (`n`.`id_network` = `c`.`id_network`)
and (`p`.`id_network` = `c`.`id_network`)
and (`c`.`configured` = 'yes')
and (`c`.`code_affiliate` = '69842')
and (`c`.`id_network` = '12')
and (not((`i`.`Code` like '')))
and (`p`.`code` <> '0')
)
group by `i`.`Code_car`
Fast query:
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 7280 100.00 NULL
1 PRIMARY ce ALL id_affiliate,id_user NULL NULL NULL 4 75.00 Using where; Using join buffer (Block Nested Loop)
2 DERIVED c index_merge id_affiliate,id_network,configured,code_affiliate code_affiliate,id_network 203,5 NULL 1 100.00 Using intersect(codice_affiliato,id_network); Using where; Using temporary; Using filesort
2 DERIVED fr eq_ref PRIMARY PRIMARY 4 c.id_franchising 1 100.00 Using index
2 DERIVED <derived3> ref <auto_key1> <auto_key1> 5 const 10 100.00 Using where
2 DERIVED i ref id_car,id_affiliate,id_franchising,idCategory id_car 5 n.id_car 2 100.00 Using where
2 DERIVED t ref id_franchising,id_tipology_ag,id_tipology_net id_franchising 5 n.id_franchising 14 100.00 Using where
2 DERIVED p ref id_network,id_default id_default 5 t.id_tipology_net 26 100.00 Using where
3 DERIVED T1 ref id_franchising,id_network id_network 5 const 133324 100.00 Using where
SHOW WARNINGS:
Level Code Message
Note 1003 /* select#1 */
select `T2`.id_affiliate AS id_affiliate,
`T2`.`id_franchising` AS `id_franchising`,
`T2`.Code_car AS Code_car,`T2`.id_user AS id_user
from (/* select#2 */
select `i`.id_affiliate AS id_affiliate,
`i`.`id_franchising` AS `id_franchising`,
`i`.Code_car AS Code_car,`i`.id_user AS id_user
from network_configuration_affiliate `c`
join `franchising` `fr`
join network_selected_car `i`
join (/* select#3 */
select `T1`.id_car AS id_car,
`T1`.`id_network` AS `id_network`,
`T1`.`id_franchising` AS `id_franchising`,
`T1`.`id_agencie` AS `id_agencie`
from network_car_destinations `T1`
where ((`T1`.`id_network` = '12')
and ((`T1`.`id_franchising` = 968)
or (`T1`.`id_franchising` = 974)))
) `n`
join affiliate_tipologies `t`
join network_assoc_tipologies `p`
where ((`fr`.`id` = `c`.`id_franchising`)
and (`i`.id_affiliate = `c`.id_affiliate)
and (`i`.id_car = `n`.id_car)
and (`i`.`id_franchising` = `n`.`id_franchising`)
and (`t`.`id_franchising` = `n`.`id_franchising`)
and (`t`.`id_tipology_ag` = `i`.`idCategory`)
and (`p`.`id_default` = `t`.`id_tipology_net`)
and (`n`.`id_network` = `c`.`id_network`)
and (`p`.`id_network` = `c`.`id_network`)
and (`c`.`configured` = 'yes')
and (`c`.`code_affiliate` = '69842')
and (`c`.`id_network` = '12')
and (not((`i`.Code_car like '')))
)
group by `i`.Code_car
) `T2`
left join network_conf_users `ce` on(((`ce`.id_affiliate = `T2`.id_affiliate)
and (`ce`.id_user = `T2`.id_user))
)
where 1
The left join finds those fields to the left and retrieves those from the right, while inner joining will lookup all fields. Keeping it simple by not needing to organize the extra columns the query completes much quicker.
Please show us the EXPLAINs.
The Optimizer will turn LEFT JOIN into JOIN if it decides that it makes no difference. Please do EXPLAIN EXTENDED SELECT ... followed immediately by SHOW WARNINGS; so we can see whether this happened.
The Optimizer will try a variety of orders for JOINing (in the absence of LEFT or RIGHT). So, if the LEFT was really redundant, I would expect the EXPLAINs to have the tables in the same order.
Normally, the Optimizer will start with a "derived" table, n in your examples. But it may be hidden behind the LEFT.
Do you have any "composite" indexes? These are likely to be beneficial:
T1: INDEX(id_network, id_franchising) -- in this order
c: INDEX(id_network, code_affiliate, configured)
Revise
With the recent changes to the query, I see this pattern:
Slow:
SELECT ...
FROM ...
JOIN ...
GROUP BY ...
Quick:
SELECT ...
FROM ( SELECT ...
FROM ...
GROUP BY ... )
JOIN ...
(LEFT is only partially important to the question; any JOIN can exhibit the problem.)
I call the slow one "inflate-deflate". By that I mean that first it did join(s), thereby inflating the number of rows. Then it did a GROUP BY, which deflated the results.
The quick one deflated before doing the final join, there by leading to less effort overall.
index merge intersect can almost always be improved by using a 'composite' index. In this case, the one I suggest for c. (This would speed up both versions of the query.)
So I am dealing with what a query with a decent amount of joins and a lot of many to main relationships.
The only tables with a one to many would be invoice, so, and xc_orders.
Each of these tables also have hundreds of thousands of rows -
invoice has 822,967 rows
invc_fee has 208,021 rows
invc_tender has 821,799 rows
customer has 377,515 rows
cust_address has 665,633
invc_item has 1,975,436 rows
invn_sbs has 122,669 rows
so has 195,169 rows
xc_orders has 267,165 rows
If I split up the query below into two separate queries based on the WHERE conditions it changes the length of time to run the queries from 56.8 seconds to 5.36 seconds for the first query and 5.32 seconds for the second query. I take it this is due to the OR clause? Was just running the queries on their own and looking at the time to run these without caching the results the most obvious way to determine if it's alright to combine the WHERE conditions? Was there something I was missing that would allow for me to speed up the results and still keep the OR conditional statements in there?Thanks for the help.
For what it's worth this was being run a MySQL 5.5 database.
SELECT SQL_NO_CACHE i.invc_no, DATE_FORMAT(i.created_date, '%Y-%m-%d') AS invcdate, IF(i.so_no LIKE '%WEB%', substring(i.so_no,5,10),i.so_no) AS so, format(SUM(it.amt),2) AS invc_amt, i.invc_type, format(ii.qty,0) as qty, isb.description1, format(ii.price,2) As price, replace(isb.dcs_code,' ','') AS dcs, isb.siz, isb.attr, trim(i.note) AS invc_note, trim(so.note) AS so_note, trim(xo.notes) AS xcart_notes, trim(xo.customer_notes) AS xcart_cust_notes
FROM rp.invoice AS i
LEFT JOIN rp.invc_fee AS ife ON i.invc_sid = ife.invc_sid
LEFT JOIN rp.invc_tender AS it ON it.invc_sid = i.invc_sid
LEFT JOIN rp.customer AS c ON i.cust_sid = c.cust_sid
LEFT JOIN rp.cust_address AS ca ON c.cust_sid = ca.cust_sid /* NEW */
LEFT JOIN rp.invc_item AS ii ON ii.invc_sid = i.invc_sid
LEFT JOIN rp.invn_sbs AS isb ON isb.item_sid = ii.item_sid
LEFT JOIN rp.so AS so ON so.so_sid = i.so_sid
LEFT JOIN dev.xc_orders AS xo ON xo.orderid = REPLACE(so.so_no,'WEB0','')
WHERE i.invc_no != '0' AND (c.email_addr = 'email#gmail.com' OR (c.first_name = 'Eric' AND c.last_name = 'MXXXX' AND ca.address1 LIKE '1234%' AND ca.zip = '12345')) AND IFNULL(ife.fee_type, 0) >= 0
GROUP BY i.invc_no, i.created_date, i.so_no, i.invc_type, ii.qty, isb.description1, ii.price, isb.dcs_code, isb.siz, isb.attr, i.note, so.note, xo.notes, xo.customer_notes, ii.item_pos, ii.item_sid
ORDER BY i.created_date desc, i.invc_no, i.invc_type
Here is the explain results
id select table type possible_keys key_len ref rows filtered Extra
1 SIMPLE i ALL INVC_NO 822967 91.92 Using where; Using temporary; Using filesort
1 SIMPLE ife ref PRIMARY 8 rp.i.INVC_SID 2080 100.00 Using where; Using index
1 SIMPLE it ref PRIMARY 8 rp.i.INVC_SID 8217 100.00
1 SIMPLE c eq_ref PRIMARY 8 rp.i.CUST_SID 1 100.00 Using where
1 SIMPLE ca ref PRIMARY 8 rp.c.CUST_SID 6656 100.00 Using where
1 SIMPLE ii ref PRIMARY 8 rp.i.INVC_SID 19754 100.00
1 SIMPLE isb ref PRIMARY 8 rp.ii.ITEM_SID 1226 100.00 Using where
1 SIMPLE so eq_ref PRIMARY 8 rp.i.SO_SID 1 100.00 Using where
1 SIMPLE xo eq_ref PRIMARY 4 func 1 100.00 Using where
To improve performance, I would suggest replacing
LEFT JOIN rp.customer AS c ON i.cust_sid = c.cust_sid
LEFT JOIN rp.cust_address AS ca ON c.cust_sid = ca.cust_sid /* NEW */
....
WHERE i.invc_no != '0' AND (c.email_addr = 'email#gmail.com' OR (c.first_name = 'Eric' AND c.last_name = 'MXXXX' AND ca.address1 LIKE '1234%' AND ca.zip = '12345')) AND IFNULL(ife.fee_type, 0) >= 0
With
LEFT JOIN
( SELECT * FROM rp.customer WHERE c.email_addr = 'email#gmail.com' OR (c.first_name = 'Eric' AND c.last_name = 'MXXXX' ) AS c ON i.cust_sid = c.cust_sid
LEFT JOIN (SELECT * FROM rp.cust_addr WHERE ca.address1 LIKE '1234%' AND ca.zip = '12345') AS ca ON c.cust_sid = ca.cust_sid /* NEW */
....
WHERE i.invc_no != '0' AND IFNULL(ife.fee_type, 0) >= 0
I have this query:
SELECT SQL_NO_CACHE
COUNT(*) AS `numrows`
FROM
(`citations`)
LEFT JOIN
`projects` ON `projects`.`project_id` = `citations`.`project_id`
LEFT JOIN
`users` ON `users`.`user_id` = `projects`.`user_id`
WHERE
`users`.`role` = '0'
AND `citations`.`created` BETWEEN 1360213200 AND 1360299599
AND `citations`.`in_card` = '0'
AND `citations`.`citation_id` NOT IN (SELECT
user_stats_citations.citation_id
FROM
user_stats_citations,
user_stats FORCE INDEX (user_stats_type_index)
WHERE
user_stats_citations.user_stat_id = user_stats.id
AND user_stats.type IN (69 , 70, 71, 75, 76));
I have those indexes on user table:
users 0 PRIMARY 1 user_id A 42836 (NULL) (NULL) BTREE
users 1 users_industry_id_index 1 industry_id A 118 (NULL) (NULL) YES BTREE
users 1 users_sponsor_index 1 sponsor A 12 (NULL) (NULL) YES BTREE
This is the output of EXPLAIN EXTENDED
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY users ALL PRIMARY \N \N \N 42836 100.00 Using where
1 PRIMARY projects ref PRIMARY\,projects_user_id_index projects_user_id_index 4 citelighter.users.user_id 1 100.00 Using where; Using index
1 PRIMARY citations ref citations_project_id_index citations_project_id_index 4 citelighter.projects.project_id 4 100.00 Using index condition; Using where
2 SUBQUERY user_stats range user_stats_type_index user_stats_type_index 2 \N 410768 100.00 Using where; Using index
2 SUBQUERY user_stats_citations ref user_stats_citations_index_user_stat_id\,user_stats_citations_index_citation_id user_stats_citations_index_user_stat_id 8 citelighter.user_stats.id 1 100.00 \N
I tried to add FORCE INDEX on users LEFT JOIN but the index is not used. Can you help me to solve this, because this query is taking like 10 seconds on my local and 1 second on production environment.
The first thing I notice is that this predicate in the where clause: WHERE users.role = '0' turns your LEFT JOINs to INNER JOINs, so you may as well just make them inner joins.
Secondly, MySQL has problems optimising correlated subqueries, and also can perform poorly with derived tables. e.g. In this simple query:
SELECT *
FROM (SELECT * FROM T) T
JOIN (SELECT * FROM T) T2 ON T.ID = T2.ID;
Even though ID is the primary key on T, the primary key is not used for the join as it can't be cascaded out of the derived table. Similarly sometimes when you write:
SELECT *
FROM T
WHERE Afield NOT IN (SELECT Afield FROM T WHERE AnotherField = 1);
MySQL does not necessarily materialise the subquery and use this, it will often rewrite the query as:
SELECT *
FROM T
WHERE NOT EXISTS (SELECT 1
FROM T T2
WHERE T.Afield = T2.Afield
AND T2.AnotherField = 1);
And the subquery is executed for each row in the outer query, so if you have a large number of rows in the outer query executing the subquery for every row becomes very costly. The solution is to avoid subqueries as far as possible. In your case you can rewrite your query as:
SELECT SQL_NO_CACHE
COUNT(*) AS `numrows`
FROM `citations`
INNER JOIN `projects`
ON `projects`.`project_id` = `citations`.`project_id`
INNER JOIN `users`
ON `users`.`user_id` = `projects`.`user_id`
LEFT JOIN (user_stats_citations
INNER JOIN user_stats
ON user_stats_citations.user_stat_id = user_stats.id
AND user_stats.type IN (69 , 70, 71, 75, 76))
ON user_stats_citations.citation_id = `citations`.`citation_id`
WHERE `users`.`role` = '0'
AND `citations`.`created` BETWEEN 1360213200 AND 1360299599
AND `citations`.`in_card` = '0'
AND user_stats_citations.citation_id IS NULL;
With no subqueries there is no derived tables, or row by row execution of subqueries. This should improve execution time.
What does this give you?
SELECT COUNT(*) numrows
FROM citations c
JOIN projects p
ON p.project_id = c.project_id
JOIN users u
ON u.user_id = p.user_id
LEFT
JOIN
( SELECT uc.citation_id
FROM user_stats_citations uc
JOIN user_stats us
ON uc.user_stat_id = us.id
AND us.type IN (69,70,71,75,76)
) x
ON x.citation_id = c.citation_id
WHERE u.role = 0
AND c.created BETWEEN 1360213200 AND 1360299599
AND c.in_card = 0
AND x.citation_id IS NULL
My question is about MySQL optimisation and good practice.
I have to get translated text list with order preference : if I can't find text in english, I want to use french text and if it does not exists, I want to use spanish.
In other words, I have :
id lang text
1 fr text_FR
1 en text_EN
1 es text_ES
2 fr text_FR2
3 fr text_FR3
3 es text_ES3
And I want:
id lang text
1 en text_EN
2 fr text_FR2
3 fr text_FR3
So I read some topics like this, this, that, or that. Ok.
I tried this :
SELECT text FROM (
SELECT id, text
FROM translation
ORDER BY FIELD(lang, 'en', 'fr', 'es')
) tt GROUP BY tt.id;
But, when I EXPLAIN this query, I can read :
id select_type table partitions type possible_keys key key_len ref rows Extra
1 PRIMARY <derived2> NULL ALL NULL NULL NULL NULL 4 Using temporary; Using filesort
2 DERIVED translation NULL ALL PRIMARY PRIMARY 52 1641 Using filesort
So it's not optimized, because of the subquery. Can I avoid this subquery ?
I would suggest JOINing on a table by id for each language. You can use a LEFT JOIN to allow for null results. The query will still use temporary/filesort because it has to scan the entire table (you can use a WHERE to limit it more), but only for the first table. Lookups on the joined tables should be quick.
SELECT
COALESCE(en.text, fr.text, es.text, any.text)
FROM
f any
LEFT JOIN f en ON (any.id = en.id AND en.lang = 'en')
LEFT JOIN f fr ON (any.id = fr.id AND fr.lang = 'fr')
LEFT JOIN f es ON (any.id = es.id AND es.lang = 'es')
GROUP BY any.id;
http://sqlfiddle.com/#!2/baafc/1
I have a mySQL table (myISAM) containing approximately two million rows - name, address, company data. The first name and surname are held in separate columns, so I also have a second table (linked by the primary key of the first) which holds a single full name column.
The first name, surname, and company name (among others) in the first table are indexed, as is the full name column in the secondary table.
Taking this query as a starting point:
SELECT * FROM table_a INNER JOIN table_b ON table_a.ID = table_b.ID WHERE....
searching exact match or even after-like on the name columns works in milliseconds:
....table_a.first_name = 'Fred'
....table_a.surname = 'Bloggs'
....table_b.fullname = 'Fred Bloggs'
....table_a.first_name LIKE 'Mike%'
just a few examples.
Throw the COMPANY NAME in there as well..... the query suddenly takes 15 to 20 seconds:
....table_a.first_name = 'Fred' OR table_a.company_name = 'Widgets Inc'
for example
Both fields are indexed, it's an exact match.... why would the addition of a second indexed search column slow things down so much? Have I missed something about my table design?
Examples follow - there are a few other tables joined but I'm not sure these are affecting performance:
Example of name-only query which returns in 0.0123 seconds:
SELECT SQL_CALC_FOUND_ROWS
webmaster.dupe_master_id AS webmaster_id,
webmaster.first_name,
webmaster.family_name,
webmaster.job_title,
webmaster.company_name,
webmaster.address_1,
webmaster.address_2,
webmaster.town_city,
webmaster.state_county,
webmaster.post_code,
webmaster.email,
webmaster.ignored,
countries.country_name,
GROUP_CONCAT(DISTINCT titles.code ORDER BY code ASC) AS sub_string,
'' AS expo_string
FROM
(`webmaster`)
LEFT JOIN `countries` ON `countries`.`country_id` = `webmaster`.`country_id`
LEFT JOIN `red_subscriptions` ON `red_subscriptions`.`webmaster_id` = `webmaster`.`webmaster_id` AND red_subscriptions.subscription_status_id = 2
LEFT JOIN `titles` ON `titles`.`title_id` = `red_subscriptions`.`title_id`
LEFT JOIN `webmaster_tags` ON `webmaster_tags`.`webmaster_id` = `webmaster`.`webmaster_id`
LEFT JOIN `tags` ON `tags`.`tag_id` = `webmaster_tags`.`tag_id`
INNER JOIN `webmaster_search_data` ON `webmaster`.`webmaster_id` = `webmaster_search_data`.`webmaster_id`
WHERE
(full_name = '<name>')
GROUP BY
`webmaster`.`dupe_master_id`
LIMIT 50
Add in company_name (also indexed) and the query time goes through the roof:
SELECT SQL_CALC_FOUND_ROWS
webmaster.dupe_master_id AS webmaster_id,
webmaster.first_name,
webmaster.family_name,
webmaster.job_title,
webmaster.company_name,
webmaster.address_1,
webmaster.address_2,
webmaster.town_city,
webmaster.state_county,
webmaster.post_code,
webmaster.email,
webmaster.ignored,
countries.country_name,
GROUP_CONCAT(DISTINCT titles.code ORDER BY code ASC) AS sub_string,
'' AS expo_string
FROM
(`webmaster`)
LEFT JOIN `countries` ON `countries`.`country_id` = `webmaster`.`country_id`
LEFT JOIN `red_subscriptions` ON `red_subscriptions`.`webmaster_id` = `webmaster`.`webmaster_id` AND red_subscriptions.subscription_status_id = 2
LEFT JOIN `titles` ON `titles`.`title_id` = `red_subscriptions`.`title_id`
LEFT JOIN `webmaster_tags` ON `webmaster_tags`.`webmaster_id` = `webmaster`.`webmaster_id`
LEFT JOIN `tags` ON `tags`.`tag_id` = `webmaster_tags`.`tag_id`
INNER JOIN `webmaster_search_data` ON `webmaster`.`webmaster_id` = `webmaster_search_data`.`webmaster_id`
WHERE
(full_name = '<name>' OR company_name '<name>')
GROUP BY
`webmaster`.`dupe_master_id`
LIMIT 50
EXPLAIN on full_name only:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE webmaster_search_data ref webmaster_id,full_name full_name 302 const 94 Using where; Using temporary; Using filesort
1 SIMPLE webmaster eq_ref PRIMARY PRIMARY 4 webmaster_search_data.webmaster_id 1
1 SIMPLE countries eq_ref PRIMARY PRIMARY 2 webmaster.country_id 1
1 SIMPLE red_subscriptions ref webmaster_id,subscription_status_id webmaster_id 4 webmaster_search_data.webmaster_id 1
1 SIMPLE titles eq_ref PRIMARY PRIMARY 2 red_subscriptions.title_id 1
1 SIMPLE webmaster_tags ref webmaster_id webmaster_id 4 webmaster_search_data.webmaster_id 5
1 SIMPLE tags eq_ref PRIMARY PRIMARY 2 webmaster_tags.tag_id 1 Using index
Explain when company_name is added:
1 SIMPLE webmaster index PRIMARY,company_name dupe_master_id 4 NULL 2072015 Using filesort
1 SIMPLE countries eq_ref PRIMARY PRIMARY 2 webmaster.country_id 1
1 SIMPLE red_subscriptions ref webmaster_id,subscription_status_id webmaster_id 4 webmaster.webmaster_id 1
1 SIMPLE titles eq_ref PRIMARY PRIMARY 2 red_subscriptions.title_id 1
1 SIMPLE webmaster_tags ref webmaster_id webmaster_id 4 webmaster.webmaster_id 5
1 SIMPLE tags eq_ref PRIMARY PRIMARY 2 webmaster_tags.tag_id 1 Using index
1 SIMPLE webmaster_search_data eq_ref webmaster_id,full_name webmaster_id 4 webmaster.webmaster_id 1 Using where
MySQL cannot use two indexes at once. When you throw in the company name, MySQL cannot use the index on Firstname, Lastname anymore because now there are more columns it has to check to get an exact result.
It is probably doing a full table scan.
You could split your queries up by doing a Union, that way you can use both columns with the index.
SELECT * FROM
( SELECT * FROM table_a
INNER JOIN table_b ON table_a.ID = table_b.ID
WHERE table_a.first_name = 'Fred'
UNION
SELECT * FROM table_a
INNER JOIN table_b ON table_a.ID = table_b.ID
WHERE table_a.company_name = 'Widgets Inc'
) sub;
Each query should be evaluated separately and use the adequate index. THe UNION will take care of doubles, so you will in the end have the same result.