SQL - Slow SQL Query - mysql

I have a sql query (see below) for wordpress which is taking around 4-5secs to get results. It gives all order ids which have a product/variation id in it.
I want to make it more fast, any help?
SELECT p.ID order_id
FROM wp_posts p
INNER JOIN wp_woocommerce_order_items i ON p.ID=i.order_id
INNER JOIN wp_woocommerce_order_itemmeta im ON i.order_item_id=im.order_item_id
WHERE im.meta_key IN ('_product_id','_variation_id')
AND im.meta_value IN ('703899','981273','981274','981275')
AND p.post_status IN ('wc-completed')
GROUP BY p.ID HAVING COUNT(p.ID)>1
ORDER BY p.post_date desc
LIMIT 0, 20
Above query EXPLAIN:

Why do you join when you only want to select IDs from wp_posts anyway?
SELECT p.ID order_id
FROM wp_posts p
WHERE p.post_status = 'wc-completed'
AND p.ID IN
(
SELECT i.order_id
FROM wp_woocommerce_order_items i
JOIN wp_woocommerce_order_itemmeta im ON im.order_item_id = i.order_item_id
WHERE im.meta_key IN ('_product_id','_variation_id')
AND im.meta_value IN ('703899','981273','981274','981275')
GROUP BY i.order_id
HAVING COUNT(*) > 1
)
ORDER BY p.post_date DESC
LIMIT 0, 20;
Now let's think about how the DBMS can address this. It can look for posts with status 'wc-completed', if there are only few such rows and then check whether they represent an order with more than one of the desired items. This would ask for these indexes:
create index idx1 on wp_posts(post_status, id, post_date);
create index idx2 on wp_woocommerce_order_items(order_id, order_item_id);
create index idx3 on wp_woocommerce_order_itemmeta(order_item_id, meta_key, meta_value);
Or it could look for the desired products, see whether an order contains more than one of them and then check whther this relates to a post with status = 'wc-completed'. That would ask for these indexes:
create index idx4 on wp_woocommerce_order_itemmeta(meta_key, meta_value, order_item_id);
create index idx5 on wp_woocommerce_order_items(order_item_id, order_id);
create index idx6 on wp_posts(id, post_status, post_date);
We don't know which way the DBMS will prefer, so we create all six indexes. Then we look at the explain plan to see which are being used and remove the others. Maybe the DBMS even sees no advantage in using indexes here at all, but I find this unlikely.

The first thing you can try doing is trimming what data you fetch.
That means:
Not fetching fields that you don't need/check
Implementing our constrains before joining
SELECT
p.ID order_id
FROM
(SELECT id, post_status, post_date FROM wp_posts WHERE post_status = 'wc-completed') p,
(SELECT order_id, order_item_id FROM wp_woocommerce_order_items) i,
(
SELECT
order_item_id,
meta_key,
meta_value
FROM
wp_woocommerce_order_itemmeta
WHERE
meta_key IN ('_product_id','_variation_id')
AND meta_value IN ('703899','981273','981274','981275')
) im
WHERE
p.ID = i.order_id
AND i.order_item_id = im.order_item_id
GROUP BY
p.ID
HAVING
COUNT(p.ID)>1
ORDER BY
p.post_date desc
LIMIT
0, 20
Edit:
If Inner joins are necessary, you can try:
SELECT
p.ID order_id
FROM
(SELECT id, post_status, post_date FROM wp_posts WHERE post_status = 'wc-completed') p
INNER JOIN
(SELECT order_id, order_item_id FROM wp_woocommerce_order_items) i
ON
p.ID = i.order_id
INNER JOIN
(
SELECT
order_item_id,
meta_key,
meta_value
FROM
wp_woocommerce_order_itemmeta
WHERE
meta_key IN ('_product_id','_variation_id')
AND meta_value IN ('703899','981273','981274','981275')
) im
ON
i.order_item_id = im.order_item_id
GROUP BY
p.ID
HAVING
COUNT(p.ID)>1
ORDER BY
p.post_date desc
LIMIT
0, 20
ps* I hope my syntax is correct ˙ my SQL is quite rusty

Related

Query Optimization with Indexes

I am needing some assistance on optimizing this WordPress/WooCommerce query:
SELECT
p.ID AS order_id
,DATE(p.post_date) AS order_date
,SUBSTR(comment_content,17) AS csr
,SUBSTR(p.post_status,4) AS order_status
,UCASE(CONCAT((SELECT wp_postmeta.meta_value FROM wp_postmeta WHERE meta_key = '_billing_first_name' and wp_postmeta.post_id = p.ID),' ',(SELECT wp_postmeta.meta_value FROM wp_postmeta WHERE meta_key = '_billing_last_name' and wp_postmeta.post_id = p.ID))) AS customer
,(SELECT GROUP_CONCAT(DISTINCT order_item_name ORDER BY order_item_name ASC SEPARATOR ', ') FROM wp_woocommerce_order_items WHERE order_id = p.ID AND order_item_type = 'line_item' GROUP BY order_id) AS products
,(SELECT GROUP_CONCAT(CONCAT(serial_number,'',serial_feature_code)) FROM wp_custom_serial WHERE wp_custom_serial.order_id = p.ID GROUP BY wp_custom_serial.order_id) AS serials
FROM
wp_posts AS p
INNER JOIN wp_comments AS c ON p.ID = c.comment_post_ID
INNER JOIN wp_postmeta AS pm ON p.ID = pm.post_id
WHERE
p.post_type = 'shop_order'
AND comment_content LIKE 'Order placed by%'
GROUP BY p.ID
ORDER BY SUBSTR(comment_content,17) ASC, p.post_date DESC;
I do not understand what EXPLAIN is telling me and need some guidance on how to speed it up. Can someone describe what, in the EXPLAIN response, indicates where my issue is and where to look for answers?
id
select_type
table
partitions
type
possible_keys
key
key_len
ref
rows
filtered
Extra
1
PRIMARY
c
NULL
ALL
comment_post_ID
NULL
NULL
NULL
20452
11.11
Using where; Using temporary; Using filesort
1
PRIMARY
p
NULL
eq_ref
PRIMARY,post_name,type_status_date,post_parent,post_author
PRIMARY
8
db.c.comment_post_ID
1
50.00
Using where
1
PRIMARY
pm
NULL
ref
post_id
post_id
8
db.c.comment_post_ID
33
100.00
Using index
2
DEPENDENT SUBQUERY
wp_postmeta
NULL
ref
post_id,meta_key
post_id
8
func
33
2.26
Using where
3
DEPENDENT SUBQUERY
wp_postmeta
NULL
ref
post_id,meta_key
post_id
8
func
33
2.30
Using where
4
DEPENDENT SUBQUERY
wp_woocommerce_order_items
NULL
ref
order_id
order_id
8
func
2
10.00
Using where
5
DEPENDENT SUBQUERY
wp_custom_serial
NULL
ALL
NULL
NULL
NULL
NULL
5160
10.00
Using where; Using filesort
Queries are processed in distinct stages. The first clauses processed are the FROM, then WHERE, and then the SELECT clause. Those dependent subqueries mean that for each row that you "have" after processing the FROM and WHERE clauses you are running separate, new subqueries for each row of those results. In your case you are doing that times four.
You can usually rework this to move these queries out of the SELECT clause and into the FROM clause.
Taking one column you have, the serials column, I think you would want to move that into the FROM clause in a way like this
SELECT p.ID AS order_id
, DATE(p.post_date) AS order_date
, SUBSTR(comment_content, 17) AS csr
, SUBSTR(p.post_status, 4) AS order_status
, UCASE(CONCAT((SELECT wp_postmeta.meta_value
FROM wp_postmeta
WHERE meta_key = '_billing_first_name' and wp_postmeta.post_id = p.ID), ' ',
(SELECT wp_postmeta.meta_value
FROM wp_postmeta
WHERE meta_key = '_billing_last_name' and wp_postmeta.post_id = p.ID))) AS customer
, (SELECT GROUP_CONCAT(DISTINCT order_item_name ORDER BY order_item_name ASC SEPARATOR ', ')
FROM wp_woocommerce_order_items
WHERE order_id = p.ID
AND order_item_type = 'line_item'
GROUP BY order_id) AS products
, serials_sub.serials
FROM wp_posts AS p
INNER JOIN wp_comments AS c ON p.ID = c.comment_post_ID
INNER JOIN wp_postmeta AS pm ON p.ID = pm.post_id
LEFT JOIN (
SELECT p.ID as post_id, GROUP_CONCAT(CONCAT(cs.serial_number, '', cs.serial_feature_code)) AS serials
FROM wp_custom_serial cs
JOIN wp_posts AS p ON cs.order_id = p.ID
WHERE p.post_type = 'shop_order'
AND comment_content LIKE 'Order placed by%'
GROUP BY cs.order_id
) as serials_sub ON serials_sub.post_id = p.ID
WHERE p.post_type = 'shop_order'
AND comment_content LIKE 'Order placed by%'
GROUP BY p.ID
ORDER BY SUBSTR(comment_content, 17) ASC, p.post_date DESC;
The difference here is that instead of separate queries being performed for each row, a single subquery is used in the initial FROM clause. So while perhaps looking more unwieldy, in fact this will give you much better performance.
Following this pattern for the other subqueries I think will resolve your issues.
If interested here is the documentation on the EXPLAIN.
https://dev.mysql.com/doc/refman/8.0/en/execution-plan-information.html
And I recommend the book High Performance MySQL.
The outer wp_postmeta does not seem to be used. Remove the JOIN if possible.
Hiding information in the middle of strings leads to SUBSTR() usage, which is inefficient.
The GROUP BY p.ID seems to be unnecessary.
The plugin WP Index Improvements would help with some parts.

Why I keep getting posts.ID unknown column while the column exists?

This is my WooCommerce SQL query below. I keep getting the error,
Unknown column 'posts.ID' is 'on clause'
but my posts.ID column does exist.
SELECT DISTINCT posts.ID as product_id, posts.post_parent as parent_id FROM wp_posts posts,
wp_terms terms, wp_term_relationships term_relationships LEFT JOIN wp_wc_product_meta_lookup
wc_product_meta_lookup ON posts.ID=wc_product_meta_lookup.product_id WHERE
posts.ID=term_relationships.object_id AND term_relationships.term_taxonomy_id=terms.term_id
AND terms.name!='exclude-from-catalog' AND posts.post_type IN ('product')
AND ( ( ( posts.post_title LIKE '%bruno%') OR ( posts.post_excerpt LIKE '%bruno%')
OR ( posts.post_content LIKE '%bruno%' ) OR ( wc_product_meta_lookup.sku LIKE '%bruno%' ) ))
AND posts.post_status IN ('publish') ORDER BY posts.post_parent ASC, posts.post_title ASC;
Never use commas in the FROM clause. Always use proper, explicit, standard JOIN syntax.
So, write the query correctly as:
SELECT DISTINCT p.ID as product_id, p.post_parent as parent_id
FROM wp_posts p JOIN
wp_term_relationships
ON p.id = tr.object_id JOIN
wp_terms t
ON tr.term_taxonomy_id = t.term_id LEFT JOIN
wp_wc_product_meta_lookup pml
ON p.ID = pml.product_id
WHERE t.name <> 'exclude-from-catalog' AND
p.post_type IN ('product') AND
(p.post_title LIKE '%bruno%' OR
p.post_excerpt LIKE '%bruno%' OR
p.post_content LIKE '%bruno%' OR
pml.sku LIKE '%bruno%'
) AND
p.post_status IN ('publish')
ORDER BY p.post_parent ASC, p.post_title ASC;
Notes:
The specific problem is that , affects the scoping of identifiers. You cannot reference a table alias from before the comma in an ON clause after the comma.
I see no reason to duplicate table names.
Using table aliases is good, but the query is simpler if the aliases are shorter.

MySQL query stalling - is there a more efficient solution for this MySQL Query?

SELECT post_title,
count(*) AS c
FROM wp_posts
WHERE post_type = "product"
GROUP BY post_title
HAVING c > 1
ORDER BY c DESC
runs no problem, returns result in < 1 sec. Yet
select * from wp_posts where post_title in (
select post_title from wp_posts WHERE post_type = "product"
group by post_title having count(*) > 1
)
hangs up.
Yet they are fundamentally the same query except for the fact that in the second query I'm trying to pull out the entire record rather than just the post_title.
Have I erred? Is there a more efficient way to achieve the equivalent?
Edit: EXPLAIN query and SHOW CREATE TABLE wp_posts has been appended for your information.
you could avoid the IN clause on the subquery and use an inner join
select a.*
from wp_posts a
INNER JOIN (
select post_title
from wp_posts
WHERE post_type = "product"
group by post_title
having count(*) > 1
) t ON t.post_title = a.post_title
this should be more performant
The most efficient way to write this query is probably using exists . . . assuming wp_posts has a primary key:
select p.*
from wp_posts p
where p.post_type = 'product' and
exists (select 1
from wp_posts p2
where p2.post_title = p.post_title and
p2.post_type = p.post_type and
p2.id <> p.id
);
For performance, yu want an index on wp_posts(post_type, title, id).

SQL - ordering results from joined table

Note: This question is related to a WordPress specific question but I wanted to have an "outside look" at this from a pure SQL point of view: https://wordpress.stackexchange.com/questions/55263/order-posts-by-custom-field-and-if-custom-field-is-empty-return-remaining-posts
Let's say we have to tables with the following strucure:
Tabe posts: ID (key), Title
Table post_metadata: post_ID(FKEY), meta_key, meta_value
And I want to retrieve ID and Title of posts that have:
an entry in post_metadata with key = 'meta_1' and meta_value = 'value_1'
AND an entry in post_metadata with key = 'meta_2' and meta_value = 'value_2'
I want to order the results by the value of a third metadata with meta_key = "meta_3".
Now here is the tricky part:
Not all posts have an entry in post_metadata table with 'meta_3' as meta_key. Since Im not filtering posts by meta_3, only ordering, I wanted to keep these posts in my results, as if they had an empty value for this meta.
How can we achieve that?
Thanks
Edit:
There is SQL fiddle now: https://www.db-fiddle.com/f/kBNaaRFB5xfna5MniuTpaG/1
Perhaps:
Use an Left join once to get meta 3 value if it exists ensuring you keep all posts that have meta1 and meta2 with desired values.
and then use an exist and having to ensure you only get records having both meta1 and 2 with desired values.
UNTESTED...
SELECT P.ID, P.Title, PM.Meta_value
FROM Posts P
LEFT JOIN Post_MetaData PM
on P.ID = PM.Post_ID
and PM.key = 'meta_3'
WHERE exists (SELECT 1
FROM post_meta
WHERE ((Key 'meta_1' and meta_value = 'Value_1') OR
(Key 'meta_2' and meta_value = 'Value_2'))
and P.ID = Post_ID --Either here or an AND in the HAVING clause...need to test to know
GROUP BY Post_ID
HAVING count(*) = 2 )
ORDER BY -PM.meta_value desc, P.ID
This does assume that post_metaData has a unique constraint on the key per Post_ID. otherwise we could get meta_1 with value a and meta1 with value a and the count(*) would be 2; and incorrectly return it in the results.
To ensure nulls are last follow this approach;
MySQL Orderby a number, Nulls last
Doing this as an IN.... but would be slower I would think.
SELECT P.ID, P.Title, PM.Meta_value
FROM Posts P
LEFT JOIN Post_MetaData PM
on P.ID = PM.Post_ID
and PM.key = 'meta_3'
WHERE P.ID in (SELECT Post_meta.Post_ID
FROM post_meta
WHERE ((Key 'meta_1' and meta_value = 'Value_1') OR
(Key 'meta_2' and meta_value = 'Value_2'))
GROUP BY Post_ID
HAVING count(*) = 2 )
ORDER BY -PM.meta_value desc, P.ID

how to get better performance, with not in query

here is my custom sql query ;
SELECT id,
post_title,
post_content,
comment_count,
post_date,
post_status,
post_name
FROM wp_posts
WHERE post_type = 'post'
AND post_status = 'publish'
AND id NOT IN (SELECT DISTINCT post_id
FROM wp_postmeta
WHERE meta_key = 'visible_headlane'
AND meta_value = 'On')
AND id NOT IN (SELECT DISTINCT post_id
FROM wp_postmeta
WHERE meta_key = 'visible_homepage'
AND meta_value = 'On')
ORDER BY post_date DESC
LIMIT 11, 32
i can not use, LIMIT in NOT IN Query so, there is any way to use, or limit SELECT DISTINCT sql query, or any idea ?
Note : I need to optimise this query because, it takes so long , like a slow query.
Thanks.
Switch to NOT EXISTS( SELECT 1 FROM ... ) -- this variation does not need to compute the entire list of things; it only needs to check for the absence. LEFT JOIN .. ON .. IS NULL is also more efficient.
Furthermore, the standard schema for wp_postmeta is quite inefficient; see my suggestions.
Meanwhile, keep it as two tests, unlike the suggestion by 'holder'.
The EXISTS would look something like
AND NOT EXISTS ( SELECT 1 FROM wp_postmeta
WHERE post_id = wp_posts.id
AND meta_key = 'visible_headlane'
AND meta_value = 'On' )
which would make good use of the PRIMARY KEY(post_id, meta_key) that I recommend.
(Is it really 'headlane', not 'headline'?)
Maybe something like this as jarlh suggested
SELECT
ID,post_title,post_content,comment_count,post_date,post_status,post_name
FROM wp_posts t1
left join (
SELECT DISTINCT post_id from wp_postmeta
WHERE meta_key IN ('visible_headlane','visible_homepage') AND meta_value = 'On' ) t2 on t1.ID = t2.post_id
WHERE post_type = 'post'
AND post_status = 'publish'
AND t2.post_id is null
ORDER BY post_date DESC
LIMIT 11, 32