I have the below query, which I appreciate probably isn't well written, but on my local PC with Xampp and MariaDB it executes in 0.1719 seconds, which is about the speed I would hope for.
However, on my development server with Plesk and MariaDB the same query with the same data takes over 12 seconds. Obviously would be no use.
Probably the query could be modified to make it better, but can somebody explain why the performance difference? The server is a VPS, it has no shortage of resources - it isn't live so usage is almost none at all, yet still 12+ seconds for this query.
The query:
SELECT m.id AS match_id, e.event AS event1
FROM matches m
JOIN competitions co ON co.id = m.competition
JOIN clubs h ON h.id = m.hometeam
JOIN clubs a ON a.id = m.awayteam
LEFT JOIN match_events e ON e.match = m.id
AND e.player = '7138'
WHERE (m.hometeam = '1'
OR m.awayteam = '1'
)
AND m.season = '121'
Are you sure you need AND e.player = '7138' in the ON clause of a LEFT JOIN and not in the WHERE clause?
Better indexing
Recommend these composite, covering, indexes:
m: (season, awayteam, hometeam, competition, id)
e: (player, match, event)
Avoiding OR
OR optimizes poorly. A common trick is to turn it into UNION. Such may work for your query:
SELECT ...
FROM matches JOIN ...
WHERE m.season = 121
AND m.hometeam = 1
UNION ALL
SELECT ...
FROM matches JOIN ...
WHERE m.season = 121
AND m.awayteam = 1
And have these two indexes:
INDEX(season, hometeam) -- will be used by one part of the UNION
INDEX(season, awayteam) -- will be used by the other
I chose UNION ALL because it is faster than UNION DISTINCT. But if you get unwanted dups, change it.
Related
I have a problem when IN clause contains too many values. Consider this query
EXPLAIN
SELECT DISTINCT t.entry_id , t.sticky , wd.field_id_104 , t.title
FROM exp_channel_titles AS t
LEFT JOIN exp_channels ON t.channel_id = exp_channels.channel_id
LEFT JOIN exp_channel_data AS wd ON t.entry_id = wd.entry_id
LEFT JOIN exp_members AS m ON m.member_id = t.author_id
INNER JOIN exp_category_posts ON t.entry_id = exp_category_posts.entry_id
INNER JOIN exp_categories ON exp_category_posts.cat_id = exp_categories.cat_id
WHERE t.entry_id !=''
AND t.site_id IN ('1')
AND t.entry_date < 1610109517
AND (t.expiration_date = 0 OR t.expiration_date > 1610109517)
AND t.entry_id IN ('0','649','650','651','652','653','654','655')
;
if there are few values output is following, which is ok
but if IN ('0','649','650','651','652','653','654','655', thousand values)
query run about 1 minute and explain change to this
how to fix that?
UPDATE: range_optimizer_max_mem_size had already set to 0 and isn't issue
We have had similar problems at my company when someone runs a query with a very long list of values in an IN (...) predicate.
We found that MySQL enforces a limit on memory available to the range optimizer. If the list of values is too long, it exceeds the memory limit, and the optimizer cannot finish its analysis to see if it should use the index. So it gives up and says, "forget it! it's a table-scan for you."
We fix it by setting the MySQL Server configuration value range_optimizer_max_mem_size=0 which means there is no limit to the memory that the range optimizer can use.
This creates a risk that if someone were to run a query with a million values in the IN (...) list, it could use a lot of memory, maybe enough to kill the MySQL Server. But so far the tradeoff is preferable, to allow the optimizer to choose the index.
See documentation:
https://dev.mysql.com/doc/refman/5.7/en/range-optimization.html
https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_range_optimizer_max_mem_size
Re your comment:
Another common reason for the optimizer to choose to do a table-scan is that it calculates that your conditions match a large enough portion of the table that it's more expensive to use the index than to simply run a table-scan and examine every row.
The threshold for this isn't documented, and it depends on the implementation of the cost-based optimizer, so it might change from version to version. But my observation is that usually if your conditions match more than 20% of the table, the optimizer chooses the table-scan.
You could use an index hint to tell the optimizer to treat a table-scan as infinitely expensive, so the index is preferred to a table-scan.
Explode-implode. This is a classic problem of an inefficient way to write a query.
JOIN several tables
Filter
Collapse the results -- usually by GROUP BY or LIMIT, but DISTINCT has the same effect.
So... Turn the query inside out.
Find the ids of the desired rows in t
JOIN that to the rest of the tables.
Presumably the DISTINCT will not be needed at all.
SELECT t2.entry_id, t2.sticky, wd.field_id_104, t2.title
FROM ( SELECT id
FROM exp_channel_titles
WHERE entry_id !=''
AND site_id IN ('1')
AND entry_date < 1610109517
AND (expiration_date = 0 OR expiration_date > 1610109517)
AND entry_id IN ('0','649','650','651','652','653','654','655')
) AS t
JOIN exp_channel_titles AS t2 USING(id)
LEFT JOIN exp_channels ON t2.channel_id = exp_channels.channel_id
LEFT JOIN exp_channel_data AS wd ON t2.entry_id = wd.entry_id
;
Another reformulation
Since there is only one use for md, this might be better:
SELECT entry_id,
sticky,
( SELECT wd.field_id_104
FROM exp_channels ON t2.channel_id = exp_channels.channel_id
LEFT JOIN exp_channel_data AS wd ON t.entry_id = wd.entry_id
) AS field_id_104,
title
FROM exp_channel_titles
WHERE entry_id !=''
AND site_id IN ('1')
AND entry_date < 1610109517
AND (expiration_date = 0 OR expiration_date > 1610109517)
AND entry_id IN ('0','649','650','651','652','653','654','655')
;
and have a 5-column index starting with site_id, entry_date
Other...
AND (t.expiration_date = 0 OR t.expiration_date > 1610109517)
OR is not sargeable. Can you redesign the table to avoid this OR?
Without the above reformulation, this may help:
INDEX(site_id, entry_date)
Also, get rid of these, since they seem to be totally useless:
LEFT JOIN exp_channels ON t.channel_id = exp_channels.channel_id
LEFT JOIN exp_members AS m ON m.member_id = t.author_id
And these may be useless:
INNER JOIN exp_category_posts ON t.entry_id = exp_category_posts.entry_id
INNER JOIN exp_categories ON exp_category_posts.cat_id = exp_categories.cat_id
I have one query that is loading db so much and my hosting provider complains
SELECT count(a.id),
a.*,
CASE
WHEN CHAR_LENGTH(a.alias) THEN CONCAT_WS(":", a.id, a.alias)
ELSE a.id
END AS slug,
CASE
WHEN CHAR_LENGTH(cc.alias) THEN CONCAT_WS(":", cc.id, cc.alias)
ELSE cc.id
END AS catslug
FROM jos_chrono_comments AS com,
jos_content AS a
LEFT JOIN jos_content_frontpage AS f ON f.content_id = a.id
INNER JOIN jos_content AS c ON f.content_id = c.id
INNER JOIN jos_categories AS cc ON cc.id = a.catid
INNER JOIN jos_sections AS s ON s.id = a.sectionid
WHERE (a.state = 1
AND s.id > 0)
AND s.published = 1
AND cc.published = 1
AND a.id = com.pageid
AND DATE_SUB(CURDATE(),INTERVAL 30 DAY) <= c.publish_up
GROUP BY (com.pageid)
ORDER BY 1 DESC LIMIT 0,
10
it's Joomla 1.5 related and Chronocomments module for most commented in 30 days
I have some hints here https://goo.gl/0wF2ex but I am not so good to rewerite that in better way without using temp table
Looking for help to make that query not so heavy for mysql server, maybe eliminating group by or any hint will be usefull
Thanks,
K#m0
There are multiple possible answers, this is focusing on the database with no change to the PHP code.
Explain query
With the MySQL EXPLAIN command, that you can use e.g. from phpMyAdmin, you can have optimization tips from MySQL itself, mostly related to adding indexes if required.
Possible indexes
As a wild guess, I would make sure that all the following fields are indexed:
jos_content.catid
jos_content.sectionid
jos_content.state
jos_content_frontpage.content_id (if you have many items in frontpage)
jos_categories.published (if you have many categories)
jos_sections.published (if you have many sections)
jos_chrono_comments.pageid
Final notes
The ORDER BY 1 DESC part seems useless to me.
The increase of performance is really dependant from the size of your tables, so don't expect miracles. But it's definitely worth a try.
The below query is very slow (takes around 1 second), but is only searching approx 2500 records (+ inner joined tables).
if i remove the ORDER BY, the query runs in much less time (0.05 or less)
OR if i remove the part nested select below "# used to select where no ProfilePhoto specified" it also runs fast, but i need both of these included.
I have indexes (or primary key) on :tPhoto_PhotoID, PhotoID, p.Enabled, CustomerID, tCustomer_CustomerID, ProfilePhoto (bool), u.UserName, e.PrivateEmail, m.tUser_UserID, Enabled, Active, m.tMemberStatuses_MemberStatusID, e.tCustomerMembership_MembershipID, e.DateCreated
(do i have too many indexes? my understanding is add them anywhere i use WHERE or ON)
The Query :
SELECT e.CustomerID,
e.CustomerName,
e.Location,
SUBSTRING_INDEX(e.CustomerProfile,' ', 25) AS Description,
IFNULL(p.PhotoURL, PhotoTable.PhotoURL) AS PhotoURL
FROM tCustomer e
LEFT JOIN (tCustomerPhoto ep INNER JOIN tPhoto p ON (ep.tPhoto_PhotoID = p.PhotoID AND p.Enabled=1))
ON e.CustomerID = ep.tCustomer_CustomerID AND ep.ProfilePhoto = 1
# used to select where no ProfilePhoto specified
LEFT JOIN ((SELECT pp.PhotoURL, epp.tCustomer_CustomerID
FROM tPhoto pp
LEFT JOIN tCustomerPhoto epp ON epp.tPhoto_PhotoID = pp.PhotoID
GROUP BY epp.tCustomer_CustomerID) AS PhotoTable) ON e.CustomerID = PhotoTable.tCustomer_CustomerID
INNER JOIN tUser u ON u.UserName = e.PrivateEmail
INNER JOIN tmembers m ON m.tUser_UserID = u.UserID
WHERE e.Enabled=1
AND e.Active=1
AND m.tMemberStatuses_MemberStatusID = 2
AND e.tCustomerMembership_MembershipID != 6
ORDER BY e.DateCreated DESC
LIMIT 12
i have similar queries that but they run much faster.
any opinions would be grateful:
Until we get more clarity on your question between working in other query etc..Try EXPLAIN {YourSelectQuery} in MySQL client and see the suggestions to improve the performance.
I am currently running into an issue where, when I use a "LIKE" in my query I get the result in 2 seconds. But when I use the '=' instead, it takes around 1 minute for the result to show up.
The following is my query:
QUERY1
The following query takes 2 seconds:
`select distinct p.Name from Timeset s
join table1 f on (f.id = s.id)
join table2 p on (p.source=f.table_name)
join table3 d on (d.Name = p.Name) WHERE
s.Active = 'Y' AND **p.sourcefrom like '%sometable%'`
QUERY2
The same query replacing the 'like' by '=' takes 1 minute:
select distinct p.Name from Timeset s
join table1 f on (f.id = s.id)
join table2 p on (p.source=f.table_name)
join table3 d on (d.Name = p.Name) WHERE
s.Active = 'Y' AND **p.sourcefrom = 'sometable'
I am really puzzled because I know that 'LIKE' is usually slower (than '=') since mysql need to look for different possibilities. But I am sure why in my case, "=" is slower with such a substantial difference.
thank you kindly for the help in advance,
regards,
When you use = MySQL is probably using a different index compared to when you use LIKE. Check the output from the two execution plans and see what the differnce is. Then you can FORCE the use of the better performing index. Might be worth running ANALYZE TABLE for each of the tables involved.
I've found info on how to optimize MySQL queries, but most of the tips seem to suggest avoiding things MySQL isn't built for (e.g., calculations, validation, etc.) My query on the other hand is very straight forward but joins a lot of tables together.
Is there an approach to speeding up simple queries with many INNER JOINS? How would I fix my query below?
SELECT t_one.id FROM table_one t_one
INNER JOIN entr_to_state st
INNER JOIN entr_to_country ct
INNER JOIN entr_to_domain dm
INNER JOIN entr_timing t
INNER JOIN entr_to_weather a2w
INNER JOIN entr_to_imp_num a2i
INNER JOIN entr_collection c
WHERE t_one.type='normal'
AND t_one.campaign_id = c.id
AND t_one.status='running'
AND c.status='running'
AND (c.opt_schedule = 'continuous' OR (c.opt_schedule = 'schedulebydate'
AND (c.start_date <= '2011-03-06 14:25:52' AND c.end_date >= '2011-03-06 14:25:52')))
AND t.entr_id = t_one.id AND ct.entr_id = t_one.id
AND st.entr_id = t_one.id AND a2w.entr_id = t_one.id
AND (t_one.targeted_gender = 'male' OR t_one.targeted_gender = 'both')
AND t_one.targeted_min_age <= 23.1 AND t_one.targeted_max_age > 23.1
AND (ct.abbreviation = 'US' OR ct.abbreviation = 'any')
AND (st.abbreviation = 'CO' OR st.abbreviation = 'any')
AND t.sun = 1 AND t.hour_14 = 1
AND (a2w.weather_category_id = 1 OR a2w.weather_category_id = 0)
AND t_one.targeted_min_temp <= 46
AND t_one.targeted_max_temp > 46 GROUP BY t_one.id
Index all relevant fields, of course, which I'm sure you have
Then find which joins are the most costly ones by running EXPLAIN SELECT...
Consider splitting them off into a seperate query i.e. narrow down the record(s) you're looking for, then perform the joins on those records rather than all the records
i.e.
SELECT c.*, ....
FROM (SELECT x, y, z .... ) AS c
You would need to EXPLAIN SELECT the query, check which parts of the query are not using in indices, and then attempt to index those. If possible, break the query down into smaller parts as well.
If you really cannot in any way optimize the underlying DB or your query, you could resort to a flat table that has the data you need for fast access. Then just hook up the main query to update the flat table to run as often as needed.