I have the following in .my.cnf
# LOGGING #
slow_query_log = ON
slow_query_log_file = /var/log/mariadb/slow_query.log
log-queries-not-using-indexes = 1
When I run tuning-primer.sh I get this:
SLOW QUERIES
The slow query log is NOT enabled.
Current long_query_time = 10.000000 sec.
You have 0 out of 36 that take longer than 10.000000 sec. to complete
Your long_query_time seems to be fine
Can someone explain how is this possible?
Apparently the slowlog is working now. Do you know what fixed that issue?
Meanwhile, this has morphed into query tuning...
What is #1 for? Why is it run so often? It returns an average of 156K rows examined (the whole table?), but only 665 rows returned. 665 is a lot of rows; do you really need them all? Could more filtering be done in the SQL?
It sounds like there is no INDEX(autoload) -- add it; it should speed up the query considerably.
#1
SELECT option_name, option_value
FROM wp_options
WHERE autoload = 'S'
What are you doing with the thousands of rows from the following? And you are preforming them thousands of times?
#2
SELECT st.value AS tra, s.value AS org, s.domain_name_context_md5 AS ctx
FROM wp_icl_strings s
LEFT JOIN wp_icl_string_translations st ON s.id=st.string_id
AND st.status=N
AND st.language='S'
AND s.language!='S'
#3
SELECT slug, taxonomy
FROM wp_posts
INNER JOIN wp_term_relationships ON (wp_posts.ID = wp_term_relationships.object_id)
INNER JOIN wp_term_taxonomy ON (wp_term_relationships.term_taxonomy_id =
wp_term_taxonomy.term_taxonomy_id )
INNER JOIN wp_terms ON (wp_term_taxonomy.term_id = wp_terms.term_id )
WHERE wp_posts.ID IN ("S","S","S","S","S","S","S","S","S",...)
ORDER BY wp_terms.name ASC
#4
SELECT t.element_id, tax.term_id, tax.taxonomy
FROM wp_icl_translations t
JOIN wp_term_taxonomy tax ON t.element_id = tax.term_taxonomy_id
AND t.element_type = CONCAT('S', tax.taxonomy)
JOIN wp_terms terms ON terms.term_id = tax.term_id
WHERE tax.term_id != tax.term_taxonomy_id
Why the LEFT in #2? That probably prevents starting with st, which might be more selective with INDEX(language, status).
In #3: wp_terms might benefit from INDEX(name).
In #4: The schema design led to the clumsy CONCAT('S', tax.taxonomy); can that be rectified? That is, can t.element_type and tax.taxonomy look the same -- either both with the prefix or both without? Or maybe the prefix is a separate column?
If you would like to discuss any of these further, please provide SHOW CREATE TABLE and EXPLAIN SELECT ....
Related
I have a problem when IN clause contains too many values. Consider this query
EXPLAIN
SELECT DISTINCT t.entry_id , t.sticky , wd.field_id_104 , t.title
FROM exp_channel_titles AS t
LEFT JOIN exp_channels ON t.channel_id = exp_channels.channel_id
LEFT JOIN exp_channel_data AS wd ON t.entry_id = wd.entry_id
LEFT JOIN exp_members AS m ON m.member_id = t.author_id
INNER JOIN exp_category_posts ON t.entry_id = exp_category_posts.entry_id
INNER JOIN exp_categories ON exp_category_posts.cat_id = exp_categories.cat_id
WHERE t.entry_id !=''
AND t.site_id IN ('1')
AND t.entry_date < 1610109517
AND (t.expiration_date = 0 OR t.expiration_date > 1610109517)
AND t.entry_id IN ('0','649','650','651','652','653','654','655')
;
if there are few values output is following, which is ok
but if IN ('0','649','650','651','652','653','654','655', thousand values)
query run about 1 minute and explain change to this
how to fix that?
UPDATE: range_optimizer_max_mem_size had already set to 0 and isn't issue
We have had similar problems at my company when someone runs a query with a very long list of values in an IN (...) predicate.
We found that MySQL enforces a limit on memory available to the range optimizer. If the list of values is too long, it exceeds the memory limit, and the optimizer cannot finish its analysis to see if it should use the index. So it gives up and says, "forget it! it's a table-scan for you."
We fix it by setting the MySQL Server configuration value range_optimizer_max_mem_size=0 which means there is no limit to the memory that the range optimizer can use.
This creates a risk that if someone were to run a query with a million values in the IN (...) list, it could use a lot of memory, maybe enough to kill the MySQL Server. But so far the tradeoff is preferable, to allow the optimizer to choose the index.
See documentation:
https://dev.mysql.com/doc/refman/5.7/en/range-optimization.html
https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_range_optimizer_max_mem_size
Re your comment:
Another common reason for the optimizer to choose to do a table-scan is that it calculates that your conditions match a large enough portion of the table that it's more expensive to use the index than to simply run a table-scan and examine every row.
The threshold for this isn't documented, and it depends on the implementation of the cost-based optimizer, so it might change from version to version. But my observation is that usually if your conditions match more than 20% of the table, the optimizer chooses the table-scan.
You could use an index hint to tell the optimizer to treat a table-scan as infinitely expensive, so the index is preferred to a table-scan.
Explode-implode. This is a classic problem of an inefficient way to write a query.
JOIN several tables
Filter
Collapse the results -- usually by GROUP BY or LIMIT, but DISTINCT has the same effect.
So... Turn the query inside out.
Find the ids of the desired rows in t
JOIN that to the rest of the tables.
Presumably the DISTINCT will not be needed at all.
SELECT t2.entry_id, t2.sticky, wd.field_id_104, t2.title
FROM ( SELECT id
FROM exp_channel_titles
WHERE entry_id !=''
AND site_id IN ('1')
AND entry_date < 1610109517
AND (expiration_date = 0 OR expiration_date > 1610109517)
AND entry_id IN ('0','649','650','651','652','653','654','655')
) AS t
JOIN exp_channel_titles AS t2 USING(id)
LEFT JOIN exp_channels ON t2.channel_id = exp_channels.channel_id
LEFT JOIN exp_channel_data AS wd ON t2.entry_id = wd.entry_id
;
Another reformulation
Since there is only one use for md, this might be better:
SELECT entry_id,
sticky,
( SELECT wd.field_id_104
FROM exp_channels ON t2.channel_id = exp_channels.channel_id
LEFT JOIN exp_channel_data AS wd ON t.entry_id = wd.entry_id
) AS field_id_104,
title
FROM exp_channel_titles
WHERE entry_id !=''
AND site_id IN ('1')
AND entry_date < 1610109517
AND (expiration_date = 0 OR expiration_date > 1610109517)
AND entry_id IN ('0','649','650','651','652','653','654','655')
;
and have a 5-column index starting with site_id, entry_date
Other...
AND (t.expiration_date = 0 OR t.expiration_date > 1610109517)
OR is not sargeable. Can you redesign the table to avoid this OR?
Without the above reformulation, this may help:
INDEX(site_id, entry_date)
Also, get rid of these, since they seem to be totally useless:
LEFT JOIN exp_channels ON t.channel_id = exp_channels.channel_id
LEFT JOIN exp_members AS m ON m.member_id = t.author_id
And these may be useless:
INNER JOIN exp_category_posts ON t.entry_id = exp_category_posts.entry_id
INNER JOIN exp_categories ON exp_category_posts.cat_id = exp_categories.cat_id
I have read through many tutorials online and here on stackoverflow but I still can't figure out how to solve the problem I'm facing right now.
I would like to tell you guys that I'm a mysql newbie so please forgive my noobness.
Alright, the query is this and it grabs the information that I need from wordpress database
SELECT
product.ID productId,
product.guid productLink,
product.post_title productTitle,
post.ID postId,
post.post_title postTitle,
post.post_content postContent,
post.post_date postDate,
tm.slug typeSlug, tm.name typeName,
tm2.slug langSlug, tm2.name langName,
tm3.slug pubSlug, tm3.name pubName,
IFNULL(wl.id,0) wishlist
FROM wp_posts product
JOIN wp_postmeta meta ON meta.meta_key = 'p2m' AND meta.meta_value=product.ID
JOIN wp_posts post ON post.ID = meta.post_id
JOIN wp_term_relationships tr ON tr.object_id = product.ID
JOIN wp_term_taxonomy tt ON tt.term_taxonomy_id = tr.term_taxonomy_id AND tt.taxonomy = 'mtype'
JOIN wp_terms tm ON tm.term_id = tt.term_id
JOIN wp_term_relationships tr2 ON tr2.object_id = product.ID
JOIN wp_term_taxonomy tt2 ON tt2.term_taxonomy_id = tr2.term_taxonomy_id AND tt2.taxonomy = 'language'
JOIN wp_terms tm2 ON tm2.term_id = tt2.term_id
JOIN wp_term_relationships tr3 ON tr3.object_id = product.ID
JOIN wp_term_taxonomy tt3 ON tt3.term_taxonomy_id = tr3.term_taxonomy_id AND tt3.taxonomy = 'publisher'
JOIN wp_terms tm3 ON tm3.term_id = tt3.term_id
LEFT JOIN wp_yith_wcwl wl ON wl.user_id = 1 AND wl.prod_id = product.ID AND wl.post_id = post.ID
WHERE product.post_type = 'product'
ORDER BY post.post_date DESC LIMIT 0,35
When I remove "ORDER BY post.post_date DESC" the speed of the query gets down to .03 seconds which is freaking amazing.. But with the addition of the "ORDER BY post.post_date DESC" the speed of the query goes to amazing 10+ seconds which is way too long..
I've used EXPLAIN and it seems that there is usage of filesort when the ORDER BY by date gets into the query.
I need to have my query reply back the results according to the post_date so I can't figure out what I could do at this point...
Additionally, I would like to point it out that in Database Description of wordpress there is an INDEX referred as "type_status_date" which could be used in my case. However, I'm totally clueless where to use it and how to do it. If there is anyone who can point out the flaw in the logic of my query or help me out with the optimization of the query (or index) please do so. Thanks for you kind attention!
P.S: I don't know how to create an index too :)
Initial Result of EXPLAIN with ORDER BY
JOIN wp_postmeta meta
ON meta.meta_key = 'p2m' -- filters
AND meta.meta_value=product.ID -- shows relation
is confusing. JOIN...ON is used to say how two tables are related. Filters belong in WHERE:
WHERE ...
AND meta.meta_key = 'p2m'
...
wp_postmeta is not well indexed. More discussion here .
Adding INDEX(post_date) may or may not help performance -- It depends on how quickly 35 good rows are found.
From the EXPLAIN, we see that the worst part is getting into meta -- something like 30K rows to look through. This _estimates that there are 30 rows with meta_key = 'p2m'. How many rows are there?
Unfortunately wp_postmeta is not designed to efficiently start with the meta_key+meta_value. This is a general problem with key-value stores (such as Posts in WP), especially when the 'value' is LONGTEXT.
The index on wp_post for type_status_date, has the date field as the third field of the index,
type_status_date INDEX
post_type
post_status
post_date
ID
So you have a predicate for post type, but post status is not included in your query in any predicate, at best it can therefore do a partial index scan of the index (sometimes called a skip scan or range scan depending on the db, my sql is not going to play ball easily on that) but it will be slower.
That is a best case scenario though, with all those joins and additional fields not covered by the index the cost of the index scan and row lookups could be far too high to consider even touching the index vs a straight scan.
It would help if you post the explain plan, it would help confirm what the optimizer was doing. The above is a more generic DB engine commentary.
type_status_date is a combined index so it's used only if you order by all it's components. It cannot be used by MySQL to order by only post_date. So the best solution is to add an index by post_date.
I have an quite large db (800+Mb dump file) that I imported to my local server. Its a Wordpress db from witch i need to extract certain posts. There is around 160000 posts inside.
Currently im testing a bit with MySql Workbench running simple queries with JOIN and it requires a lot of time, so long actually that Workbench stops processing.
Here`s an example:
SELECT
COUNT(*)
FROM wp_posts
LEFT JOIN wp_term_relationships
ON wp_posts.ID = wp_term_relationships.object_id
LEFT JOIN wp_term_taxonomy
ON wp_term_relationships.term_taxonomy_id = wp_term_taxonomy.term_taxonomy_id
LEFT JOIN wp_terms
ON wp_term_taxonomy.term_id = wp_terms.term_id
WHERE wp_terms.term_id = 195;
Running over 600 seconds.
Here is an wordpress db schema:
Of course chance is I'm just bad at SQL, not really my field...
If you have all indexes on columns you use for joining (wp_posts.ID, wp_term_relationships.object_id etc.), this should not be a problem, and query should be executed in less than 1 sec (1 sec is a lot, too).
Also, there might be waiting on other queries in queue (locks), so you should add this to those queries:
SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED ;
SELECT
COUNT(*)
FROM
wp_posts
LEFT JOIN
wp_term_relationships ON wp_posts.ID = wp_term_relationships.object_id
LEFT JOIN
wp_term_taxonomy ON wp_term_relationships.term_taxonomy_id = wp_term_taxonomy.term_taxonomy_id
LEFT JOIN
wp_terms ON wp_term_taxonomy.term_id = wp_terms.term_id
WHERE
wp_terms.term_id = 195;
COMMIT ;
Create indexes in workbench. I can see here that you are joining on IDs on every table, so because those are ID columns, they should be PRIMARY KEY, and create UNIQUE Clustered index on them.
If you are really just running a count you can get your result with:
SELECT COUNT(*)
FROM wp_term_relationships wtr
JOIN wp_term_taxonomy wtt
ON wtt.term_taxonomy_id = wtr.term_taxonomy_id
WHERE wtt.term_id = 195;
You don't need the information from the obligatory wp_terms or wp_posts records that are attached to these records.
You also don't need a LEFT JOIN as you are discarding NULL tuples with your WHERE condition.
As for the speed, I would suggest running the query prepended with EXPLAIN and checking that everything is indexed correctly and these indices are being used.
The below query is very slow (takes around 1 second), but is only searching approx 2500 records (+ inner joined tables).
if i remove the ORDER BY, the query runs in much less time (0.05 or less)
OR if i remove the part nested select below "# used to select where no ProfilePhoto specified" it also runs fast, but i need both of these included.
I have indexes (or primary key) on :tPhoto_PhotoID, PhotoID, p.Enabled, CustomerID, tCustomer_CustomerID, ProfilePhoto (bool), u.UserName, e.PrivateEmail, m.tUser_UserID, Enabled, Active, m.tMemberStatuses_MemberStatusID, e.tCustomerMembership_MembershipID, e.DateCreated
(do i have too many indexes? my understanding is add them anywhere i use WHERE or ON)
The Query :
SELECT e.CustomerID,
e.CustomerName,
e.Location,
SUBSTRING_INDEX(e.CustomerProfile,' ', 25) AS Description,
IFNULL(p.PhotoURL, PhotoTable.PhotoURL) AS PhotoURL
FROM tCustomer e
LEFT JOIN (tCustomerPhoto ep INNER JOIN tPhoto p ON (ep.tPhoto_PhotoID = p.PhotoID AND p.Enabled=1))
ON e.CustomerID = ep.tCustomer_CustomerID AND ep.ProfilePhoto = 1
# used to select where no ProfilePhoto specified
LEFT JOIN ((SELECT pp.PhotoURL, epp.tCustomer_CustomerID
FROM tPhoto pp
LEFT JOIN tCustomerPhoto epp ON epp.tPhoto_PhotoID = pp.PhotoID
GROUP BY epp.tCustomer_CustomerID) AS PhotoTable) ON e.CustomerID = PhotoTable.tCustomer_CustomerID
INNER JOIN tUser u ON u.UserName = e.PrivateEmail
INNER JOIN tmembers m ON m.tUser_UserID = u.UserID
WHERE e.Enabled=1
AND e.Active=1
AND m.tMemberStatuses_MemberStatusID = 2
AND e.tCustomerMembership_MembershipID != 6
ORDER BY e.DateCreated DESC
LIMIT 12
i have similar queries that but they run much faster.
any opinions would be grateful:
Until we get more clarity on your question between working in other query etc..Try EXPLAIN {YourSelectQuery} in MySQL client and see the suggestions to improve the performance.
I am not an expert in sql.
My wordpress started to return timeouts and respond really slow.
when I started digging, I noticed that the slow_query log has a lot to tell me.
unfortunately I have a lot of slow queries.
for example:
# Time: 140425 17:03:29
# User#Host: geektime[geektime] # localhost []
# Query_time: 7.024031 Lock_time: 0.000432 Rows_sent: 0 Rows_examined: 0
SET timestamp=1398434609;
SELECT wp_posts.*
FROM wp_posts
INNER JOIN wp_postmeta ON (wp_posts.ID = wp_postmeta.post_id)
INNER JOIN wp_postmeta AS mt1 ON (wp_posts.ID = mt1.post_id)
LEFT JOIN wp_postmeta AS order1 ON order1.post_id = wp_posts.ID
AND order1.meta_key = '_event_start_date'
LEFT JOIN wp_postmeta AS order2 ON order2.post_id = wp_posts.ID
AND order2.meta_key = '_event_start_time'
WHERE 1=1
AND wp_posts.post_type = 'event'
AND (wp_posts.post_status = 'publish'
OR wp_posts.post_status = 'future'
OR wp_posts.post_status = 'draft'
OR wp_posts.post_status = 'pending')
AND ((wp_postmeta.meta_key = '_event_start_date'
AND CAST(wp_postmeta.meta_value AS CHAR) BETWEEN '2014-04-11' AND '2014-04-17')
OR (mt1.meta_key = '_event_end_date'
AND CAST(mt1.meta_value AS CHAR) BETWEEN '2014-04-11' AND '2014-04-17'))
GROUP BY wp_posts.ID
ORDER BY order1.meta_value,
order2.meta_value ASC;
The columns post_id, meta_id and meta_key are indexed in wp_postmeta table.
The columns ID, post_name, post_type, post_status, post_date,post_parent, post_author and guid are indexed in wp_posts table.
however, the columns ID and GUID are indexed twice, is it bad?
and there are 4 indexs with the same key_name: type_status_date, is it bad?
How could it be that I have 60K rows in wp_posts and 3M rows in wp_postmeta?
I know its a lot to ask but I really tried to understand from researching online.
thanks in advance.
however, the columns ID and GUID are indexed twice, is it bad?
There are two different columns, so no, unless you're meaning that both have two indexes on them — in which case yes, it's bad and likely a bug in one of your theme or plugins (or a prior bug in WP itself).
and there are 4 indexs with the same key_name: type_status_date, is it bad?
Same as above: if you mean four identical indexes, it's either a theme or plugin or WP bug and you can safely drop the duplications.
How could it be that I have 60K rows in wp_posts and 3M rows in wp_postmeta?
Because the WP meta API sucks and enforces a database anti-pattern called the Entity Attribute Value (also known as EAV):
http://en.wikipedia.org/wiki/Entity-attribute-value_model
Cursory googling SO will yield plenty of threads that explain why it is a bad idea to store data in an EAV or equivalent (json, hstore, xml, whatever) if the stuff ever needs to appear in e.g. a where, join or order by clause.
You can see the inefficiencies first-hand in form of the slow query you highlighted. The query is joining the meta table four times, does so twice with a cast operator to boot — and it casts the value to char instead of date at that. Adding insult to injury, it then proceeds to order rows using values stored within it. It is a recipe for poor performance.
There is, sadly, little means of escaping the repulsive stench of this sewage, short of writing your own plugins that create proper tables to store, index and query the data you need in lieu of using the WP meta API, its wretched quoting madness, and the putrid SQL that results from using it.
One thing that you can do as temporary duct tape and WD-40 measure while you rewrite the plugins you're using from the ground up, is to toss callbacks on one or more of the filters you'll find in the giant mess of a class method that is WP_Query#get_posts(). For instance the posts_request filter, which holds the full and final SQL query, allows you to rewrite anything to your liking using regex-foo. It's no magic bullet: doing so will allow you to fix bugs such as integer values getting sorted lexicographically and such, as well as toss in very occasional query optimizations; little more.
Edit: Upon re-reading your query, methinks you're mostly in luck with respect to that last point. Your particular query features the following abomination:
INNER JOIN wp_postmeta ON (wp_posts.ID = wp_postmeta.post_id)
INNER JOIN wp_postmeta AS mt1 ON (wp_posts.ID = mt1.post_id)
LEFT JOIN wp_postmeta AS order1 ON order1.post_id = wp_posts.ID
AND order1.meta_key = '_event_start_date'
LEFT JOIN wp_postmeta AS order2 ON order2.post_id = wp_posts.ID
AND order2.meta_key = '_event_start_time'
Two of those have _event_start_date in common, so you can factor it out:
SELECT wp_posts.*
FROM wp_posts
INNER JOIN wp_postmeta ON (wp_posts.ID = wp_postmeta.post_id)
AND wp_postmeta.meta_key = '_event_start_date'
INNER JOIN wp_postmeta AS mt1 ON (wp_posts.ID = mt1.post_id)
AND mt1.meta_key = '_event_end_date'
INNER JOIN wp_postmeta AS order2 ON order2.post_id = wp_posts.ID
AND order2.meta_key = '_event_start_time'
WHERE 1=1
AND wp_posts.post_type = 'event'
AND (wp_posts.post_status = 'publish'
OR wp_posts.post_status = 'future'
OR wp_posts.post_status = 'draft'
OR wp_posts.post_status = 'pending')
AND (CAST(wp_postmeta.meta_value AS CHAR) BETWEEN '2014-04-11' AND '2014-04-17'
OR CAST(mt1.meta_value AS CHAR) BETWEEN '2014-04-11' AND '2014-04-17')
GROUP BY wp_posts.ID
ORDER BY wp_postmeta.meta_value,
order2.meta_value ASC;
Among other things, slow performance is caused by the use of functions like this:
AND CAST(wp_postmeta.meta_value AS CHAR) BETWEEN '2014-04-11' AND '2014-04-17')
Assuming that field is a date field, you will get better performance with something like this:
and wp_postmeta.meta_value >= AStartDateVariable
and wp_postmeta.meta_value < TheDayAfterAnEndDateVariable
That will be even more true if meta_value is indexed. I assume you will be sending these variables as query parmameters.
Holy cow! 3 megarows in postmeta? 60k posts? Something is seriously wrong with your installation.
Is it possible that your events table is open to spammers entering rubbish?
Do you have tons of old expired events that could somehow be purged from your system?
You may be able to get your system back on the air by increasing your timeout value. If you know how to handle php.ini, go find the timeout value and increase it, or ask your hosting company for help.
Are you on one of those $5 per month hosting companies? With sixty thousand events to handle, you may need to upgrade.
The proximate cause of the timeout is obvious. This sequence of code is full-scanning that monster post_meta table TWICE!
Why? It has an OR in it. And it is applying functions to the value of a column.
AND ((wp_postmeta.meta_key = '_event_start_date'
AND CAST(wp_postmeta.meta_value AS CHAR) BETWEEN '2014-04-11' AND '2014-04-17')
OR (mt1.meta_key = '_event_end_date'
AND CAST(mt1.meta_value AS CHAR) BETWEEN '2014-04-11' AND '2014-04-17'))
One of the disadvantages of the WordPress schema when you scale up a site is the generic nature of the postmeta table. This query does date range searches, but it's hard to index a key-value repository like postmeta to optimize those.
Do you know your way around the code of the Events Manager plugin you're using? If so, you may want to investigate optimizing this yourself.
If not, seek support from the Events Manager plugin developer.