My hoster claims the following SQL query is overloading the server and causing it to perform slowly and threatens to disable my account. I can't seem to find a bottleneck - can anyone help me detect where it all comes from?
Executed 1 min ago for 8 sec on Database --> mydb_name.
SELECT * FROM tsk_posts
LEFT JOIN tsk_term_relationships ON
(tsk_posts.ID = tsk_term_relationships.object_id)
LEFT JOIN tsk_term_taxonomy ON
(tsk_term_relationships.term_taxonomy_id = tsk_term_taxonomy.term_taxonomy_id)
WHERE
tsk_posts.post_status = 'publish'
AND
tsk_posts.post_type = 'post'
AND
tsk_term_taxonomy.taxonomy = 'category'
AND
( tsk_term_taxonomy.term_id = 4 OR tsk_term_taxonomy.term_id = 103 OR
tsk_term_taxonomy.term_id = 19 OR tsk_term_taxonomy.term_id = 20 OR
tsk_term_taxonomy.term_id = 5 OR tsk_term_taxonomy.term_id = 2 OR
tsk_term_taxonomy.term_id = 14 )
ORDER BY RAND()
LIMIT 1
Thank you very much.
"order by rand" can be quite an expensive query. A random number must be generated and then be used to access into the result set.
How many rows are in each of the tables?
Are there any indexes?
Run an explain plan to get a better idea as to where the cost is.
Related
I running into some time issues using my simple select:
SELECT *
FROM ltowert
WHERE bat = 3
AND id >= (SELECT id
FROM ltowert
WHERE bat = 3 AND ident = 'v0'
ORDER BY id DESC
LIMIT 1)
ORDER BY ident;
It takes nearly 12 seconds (depending on the index, etc..)
If I run the subquery (0.00075 sec) and put the result in the statement:
SELECT *
FROM ltowert
WHERE bat = 3 AND id >= 20979399
ORDER BY ident;
it runs in just 0.00095 sec, in addition: 0.0017 sec
So it seems, using the subquery avoid the Optimizer to use the index ?
How can I fix it and get quicker results ?
Thanks for any answers.
JR
I test your proposal, but I run into a problem of understanding. Maybe, it's my issue:
Your idea works, but the result is wrong: Result is only 1 row, but I expect 11 rows. I expect all row from l1, where id >= l2.id.
So I changed to:
SELECT *
FROM ltowert l1
INNER JOIN (
SELECT id
FROM ltowert
WHERE bat = 3 AND ident = 'v0'
order by id desc limit 1
) as l2 on l1.id>=l2.id
order by l1.ident;
This return much rows. l1.bat = 0 which is not the request.
Change to:
A. ") as l2 on l1.id>=l2.id AND l1.bat = 3"
or
B. WHERE l1.bat = 3
runs into a timeout Error 2013.
What is my mistake ?
Additional, due to my 2 step tests, I do a test using a variable:
SELECT #ref_id:=id FROM ltowert WHERE bat = 3 AND ident = 'v0' order by id desc limit 1;
SELECT * FROM ltowert l1 where bat = 3 AND l1.id >= #ref_id order by l1.ident;
It is fast, and give the right results.
Is there a disadvantage to this usage? (I know: 2 separate statements were not maintenance friendly)
You need both of these:
INDEX(bat, ident, id)
INDEX(bat, id)
If they don't suffice, please provide
SHOW CREATE TABLE ltowert;
EXPLAIN SELECT ...;
This query takes 18 seconds
SELECT `wd`.`week` AS `start_week`, `wd`.`hold_code`, COUNT(wd.hold_code) AS hold_code_count
FROM `weekly_data` AS `wd`
JOIN aol_reporting_hold_codes hc ON hc.hold_code = wd.hold_code AND chart = 'GR'
WHERE `wd`.`days` <= 6
AND `wd`.`hold_code` IS NOT NULL
AND NOT `wd`.`hold_code` = ''
AND `wd`.`week` >= '201717'
AND `wd`.`itemgroup` IN ('BOTDTO', 'BOTDWG', 'C&FORG', 'C&FOTO', 'MF-SUB', 'MI-SUB', 'PROPRI', 'PROPTO', 'STRSTO', 'STRSUB')
AND `production_type` = 2
AND `contract` = "1234"
AND `project` = 8
GROUP BY `start_week`, `wd`.`hold_code`
This query takes 4 seconds
SELECT `wd`.`week` AS `start_week`, `wd`.`hold_code`, COUNT(wd.hold_code) AS hold_code_count
FROM `weekly_data` AS `wd`
JOIN aol_reporting_hold_codes hc ON hc.hold_code = wd.hold_code AND chart = 'GR'
WHERE `wd`.`days` <= 6
AND `wd`.`hold_code` IS NOT NULL
AND NOT `wd`.`hold_code` = ''
AND `wd`.`week` >= '201717'
AND `wd`.`itemgroup` IN ('BOTDWG', 'C&FORG', 'C&FOTO', 'MF-SUB', 'MI-SUB', 'PROPRI', 'PROPTO', 'STRSTO', 'STRSUB')
AND `production_type` = 2
AND `contract` = "1234"
AND `project` = 8
GROUP BY `start_week`, `wd`.`hold_code`
All I have done is removed one item from the IN clause. I can remove any one of the items. It runs in 4 seconds as long as there are 9 items or less. It takes 18 seconds to run as soon as I increase to 10 items.
I thought MySQL limited length of command by size i.e. 1MB
More than just the EXPLAIN, use EXPLAIN FORMAT=JSON and get the "Optimizer trace" for the query. I suspect the length of the IN leads to picking a different query plan.
There is virtually no limit to the number of items in IN. I have seen as many as 70K.
That aside, you may be able to speed up even the 4-sec version...
I suggest having this index. Grrr... I can't tell which columns are in which tables. So, if these are all in one table, then make such an index:
INDEX(production_type, contract, project) -- in any order
If those are all in wd, then tack on a 4th column - any of week, itemgroup, days.
Be cautious about COUNT(wd.hold_code).
COUNT(x) checks x for being non-NULL; is that what you want? If not, then simply say COUNT(*).
When JOINing, then GROUP BY, you get an "explode-implode". The number of intermediate rows is big; that is when the COUNT is performed.
It seems wrong to both COUNT(hold_code) and GROUP BY hold_code. What are you trying to do?
For further discussion, please provide SHOW CREATE TABLE and EXPLAIN.
Please note MySql IN clause limit is established with max_allowed_packet value. You may check with NOT IN if results are faster. Also I suggest put values to be checked with IN clause under a buffer string instead of comma separated values and then give a try.
Okay I have look through several posts about SQL running slow and I didn't see anything similar to this, so my apologies if I missed one. I was asked about a month ago to create a view that would allow the user to report what hasn't been billed yet, and the SQL is joining 4 tables together. One is 1.2 million records ish. the rest are between 80K - 250K. (Please note that this should only return around 100 records after the where statements).
SELECT C.Cltnum + '.' + C.CltEng AS [ClientNum],
C.CPPLname,
w.WSCDesc,
MIN(w.Wdate) AS [FirstTDate],
w.WCodeCat,
w.WCodeSub,
SUM(w.Wbilled) AS [Billed],
SUM(w.Whours * w.Wrate) AS [Billable Wip],
sum(ar.[ARProgress]) AS [Progress],
w.Winvnum,
-- dbo.GetInvoiceAmount(w.Winvnum) AS [InvoiceAmount],
SUM(cb.cinvar) AS [AR Balance]
FROM dbo.WIP AS w
--Never join on a select statement
--join BillingAuditCatagoriesT bac on w.WCodeCat = bac.Catagory and w.WCodeSub = bac.Subcatagory
INNER JOIN dbo.Clients AS C ON w.WCltID = C.ID
JOIN dbo.ClientBuckets AS cb on c.cltnum = cb.cltnum
JOIN dbo.AcctsRec AS ar on ar.arapplyto = w.[Winvnum]
-- WHERE w.wcodecat = '1AUDT'
GROUP BY C.Cltnum, C.CltEng, C.CPPLname, w.WCodeCat, w.Wdate, w.Winvnum, w.WCodeSub, w.WSCDesc
so, where I think there may be a problem is that Category is a varchar it is xat, ACT, BID and there are about 15 different Category. this is the same as SubCat. you will notice that there are 3 functions on this and they are GetJamesProgress Which is = (SELECT sum(Amount) From Progress Where inv = w.invnum) and the same with GetInvoiceAmount and GetJamesARBalance. I know that this is bad to do but when I join by invoice number it takes even longer than with out them.
Please help thanks so much!
I got a quite complex query (at least for me).
I want to create a list of users that are ready to be paid. There are 2 conditions that need to be met: order status should be 3 and the total should be more then 50. Currently I got this query (generated with Codeingiter active record):
SELECT `services_payments`.`consultant_id`
, `consultant_userdata`.`iban`
, `consultant_userdata`.`kvk`, `consultant_userdata`.`bic`
, `consultant_userdata`.`bankname`
, SUM(`services_payments`.`amount`) AS amount
FROM (`services_payments`)
JOIN `consultant_userdata`
ON `consultant_userdata`.`user_id` = `services_payments`.`consultant_id`
JOIN `services`
ON `services`.`id` = `services_payments`.`service_id`
WHERE `services`.`status` = 3
AND `services_payments`.`paid` = 0
HAVING `amount` > 50
The services_payments table contains the commissions, consultant_userdata contains the userdata and services keeps the order data. The current query only gives me 1 result while I'm expecting 4 results.
Could someone please tell me what I'm doing wrong and what would be the solution?
For ActiveRecord, rsanchez' answer would be more of
$this->db->group_by('services_payments.consultant_id, consultant_userdata.iban, consultant_userdata.kvk, consultant_userdata.bic, consultant_userdata.bankname');
I'm working on the datafeed script for my Affiliate program and really thought I was "doin' it" but when I run the script it creates the CSV file with only 7 of 28 products. Prior to testing I made all the products meet the conditions of the query so they all should be included.I even removed the stock level and price conditions. Still only 7 of 28 gets added to the csv file 5 from one category and 2 from the other. There are only 5 products in the first category and all of those are returned. But only 2 of the 23 in the second category are returned.
The Goal is to get the specific data for all products from the categories in the query and dump them into a csv file.
This is my first time trying to write a more complex sql query (for a beginner anyway).
Using print_r I found the problem is my query. I thought I had this working before. The difference between when it was working and now is that I included grabbing the categoryid and catparentid. But even if I remove these additions, I still get the same results. Now I'm thinking that I might have thought it was working before on my larger database of products but perhaps it actually wasn't returning all of the products but because there were so many I thought they were.
The full script is pretty long but I created a new script below just to see if the query is even grabbing all of the records... it's not. :(
If I run a query to check for unassociated products (i.e. ones that have no category associations), all products are associated with a category.
All of the products have image & brand associations so I don't think that's the issue either.
Even if I remove the Where clause altogether, only the 7 products are grabbed.
I also ran the query directly within the database and got the same thing:
Showing rows 0 - 6 ( 7 total, Query took 0.0029 sec)
SELECT isc_products.productid, isc_products.prodcode, isc_products.prodname, isc_products.produrl, isc_products.prodcalculatedprice, isc_products.proddesc, isc_products.prodcurrentinv, isc_products.upc, isc_categories.categoryid, isc_categories.catparentid, isc_categories.catname, isc_product_images.imagefilestd, isc_product_images.imagefilethumb, isc_brands.brandname
FROM isc_products
INNER JOIN isc_categories ON isc_products.prodcatids = isc_categories.categoryid
INNER JOIN isc_product_images ON isc_products.productid = isc_product_images.imageprodid
INNER JOIN isc_brands ON isc_products.prodbrandid = isc_brands.brandid
WHERE isc_categories.categoryid =10
OR isc_categories.categoryid =12
Could there be any special characters in the data that would cause this issue?
Or am I using the inner joins incorrectly?
<pre>
<?php
// Make a MySQL Connection
mysql_connect("localhost", "myusername", "mypassword") or die(mysql_error());
mysql_select_db("mydatabase") or die(mysql_error());
$result = mysql_query("SELECT isc_products.productid,
isc_products.prodcode,
isc_products.prodname,
isc_products.produrl,
isc_products.prodcalculatedprice,
isc_products.proddesc,
isc_products.prodcurrentinv,
isc_products.upc,
isc_categories.categoryid,
isc_categories.catparentid,
isc_categories.catname,
isc_product_images.imagefilestd,
isc_product_images.imagefilethumb,
isc_brands.brandname
FROM isc_products
INNER JOIN isc_categories
ON isc_products.prodcatids = isc_categories.categoryid
INNER JOIN isc_product_images
ON isc_products.productid = isc_product_images.imageprodid
INNER JOIN isc_brands
ON isc_products.prodbrandid = isc_brands.brandid
WHERE isc_categories.categoryid = 17
OR isc_categories.categoryid = 15
OR isc_categories.categoryid = 16
OR isc_categories.categoryid = 12
OR isc_categories.categoryid = 13
OR isc_categories.categoryid = 14
OR isc_categories.categoryid = 11
OR isc_categories.categoryid = 10
AND isc_products.prodcurrentinv > 1
AND isc_products.prodcalculatedprice > 1.00
ORDER BY isc_categories.catname
")
or die(mysql_error());
while($row = mysql_fetch_array( $result)){
print_r ($row);
}
// end of file
echo '<br/>****** Script successfully run! ******';
?>
</pre>
Any thoughts on this would be appreciated. Dislcaimer if your first thought is "Why didn't you..." It's because I didn't or don't know about it. :) One day, my friend... one day I will. hahahaha
Thank you!
Your problem is a classic SQL coding problem: OR has precedence over AND, which bacially means you have to put brackets around your ORs, otherwise it's parsed as A OR (B OR (C AND D)) not (A OR B OR C) AND D as you'd hope.
Try this:
WHERE ( -- note opening bracket
isc_categories.categoryid = 17
OR isc_categories.categoryid = 15
OR isc_categories.categoryid = 16
OR isc_categories.categoryid = 12
OR isc_categories.categoryid = 13
OR isc_categories.categoryid = 14
OR isc_categories.categoryid = 11
OR isc_categories.categoryid = 10
) -- note closing bracket
AND isc_products.prodcurrentinv > 1
AND isc_products.prodcalculatedprice > 1.00
Also, you could avoid this problem entirely, and make your SQL easier to read, by simplifying your where clause to:
WHERE isc_categories.categoryid in (10,11,12,13,14,15,16,17)
AND isc_products.prodcurrentinv > 1
AND isc_products.prodcalculatedprice > 1.00
or, because your values are contiguous, even more simply:
WHERE isc_categories.categoryid between 10 and 17
AND isc_products.prodcurrentinv > 1
AND isc_products.prodcalculatedprice > 1.00
Try replacing the inner join with LEFT JOIN, then look at the results. Do you see the 28 you expect? If so (or you see more), the join is incorrect in some way.
Most likely it's the INNER JOIN causing this. INNER joins will return rows where there's NON-NULL fields on both sides of the join. If you want to get categories that have no products attached, you'll want a LEFT or RIGHT join (depending on how you structure the query). Those will return records where there's data on the left or right side of the join, but nulls on the opposite side.