I have a query that takes 15 seconds to get 350 results in a MySQL 5.6 Server and I am unable to diagnose why, I am still very new to database optimizing. U
The EXPLAIN visual does show some non-unique key lookups but each only says one 1 row look up.
The tabular EXPLAIN which I am not able to interpret and I am hoping someone else can here looks like .
I have tried switching the ending LIMIT = 350 to 100, 10, and the query takes exactly the same amount of time to run, about 15 seconds.
I have tried nixing the views but besides making it hard to recreate this query it did not improve performance.
Perhaps related, in other EXPLAIN statements in our MySQL DB, I've seen a view referenced with Materialized next to it, but that does not appear near next to any of the three views used in this query, in fact I don't even see the views referenced at all instead only the tables they reference. Is that a factor?
My last attempt was replacing the final selected column which is a listlineitems.* with the specific columns, since I've read that can improve speed and is just better practice, but I get the sense that is not going to dramatically improve this situation.
Here's the query -
SELECT
0 AS 'Check',
DATE_FORMAT(`listlineitems`.`dateEntered`,
'%Y-%m-%d') AS 'Date Entered',
`listlineitems`.`itemId` AS 'parentTableIdx',
`listlineitems`.`parentProjectId` AS 'parentProjectIdx',
`listlineitems`.`idx` AS 'ID',
IF(`listlineitems`.`active` = 1,
'Active',
'Inactive') AS 'Active/Inactive',
CONCAT(`listUsers`.`FirstName`,
' ',
`listUsers`.`LastName`) AS 'Employee',
CASE `listlineitems`.`type`
WHEN 1 THEN 'Time Entry'
WHEN 2 THEN 'Expense Entry'
END AS 'Type',
`listcustomers`.`name` AS 'Customer',
`listlocations`.`name` AS 'Location',
`listareas`.`name` AS 'Area',
`listassets`.`name` AS 'Asset',
`listprojects`.`name` AS 'Project',
`listprojects`.`number` 'Project #',
`listprojects`.`autoassign` 'autoassign',
`listactivities`.`name` AS 'Activity',
(CASE `listlineitems`.`type`
WHEN 1 THEN `listlineitems`.`qty`
WHEN 2 THEN `listlineitems`.`qty`
END) AS 'Quantity',
`listlineitems`.`taxable` AS 'Taxable',
`listlineitems`.`totalAmount` - `listlineitems`.`taxAmount` AS 'Pre-Tax Amount',
`listlineitems`.`taxAmount` AS 'Tax Amount',
`listlineitems`.`totalAmount` AS 'Total Amount',
`listCustomers`.`idx` AS 'parentCustomerIdx',
`listLocations`.`idx` AS 'parentLocationIdx',
`listAreas`.`idx` AS 'parentAreaIdx',
`listAssets`.`idx` AS 'parentAssetIdx',
CONCAT(`listcustomers`.`name`,
'/',
`listlocations`.`name`,
'/',
`listareas`.`name`,
'/',
`listassets`.`name`,
'/',
`listprojects`.`name`) AS 'Path',
IF(`listlineitems`.`customerViewable` = 1,
'Yes',
'No') AS 'Cust. Viewable',
(CASE
WHEN `listlineitems`.`type` = 2 THEN `listexpenseentry`.`TotalCostToPSI` - `listexpenseentry`.`TaxCostToPSI`
ELSE `listlineitems`.`totalAmount` - `listlineitems`.`taxAmount`
END) AS 'preTaxCostPSI',
(CASE
WHEN `listlineitems`.`type` = 2 THEN `listexpenseentry`.`TaxCostToPSI`
ELSE `listlineitems`.`taxAmount`
END) AS 'taxCostPSI',
(CASE
WHEN `listlineitems`.`type` = 2 THEN `listexpenseentry`.`TotalCostToPSI`
ELSE `listlineitems`.`totalAmount`
END) AS 'totalCostPSI',
view_solinx2.lastAltered AS 'lastalteredSO',
view_polinx2.lastAlteredPO AS 'lastalteredPO',
view_invlinx2.lastAlteredInv AS 'lastalteredInv',
view_solinx2.lastAlteredAfterConfirmation AS 'lastAlteredAfterConfirmation',
view_solinx2.roleIdSO AS 'roleIdSO',
view_polinx2.roleIdPO AS 'roleIdPO',
view_polinx2.userIdPO AS 'userIdPO',
view_polinx2.lastAlteredafterConfirmation AS 'lastAlteredAfterConfirmationPO',
view_invlinx2.roleIdInv AS 'roleIdInv',
view_invlinx2.userIdInv AS 'userIdInv',
view_invlinx2.lastAlteredafterConfirmation AS 'lastAlteredAfterConfirmationInv',
view_solinx2.roleId AS 'roleId',
view_solinx2.userId AS 'userId',
view_solinx2.soId AS 'SOId',
view_solinx2.autoassignSO AS 'autoassignSO',
IF(view_solinx2.notNeeded = 1,
'Not Needed',
view_solinx2.number) AS 'SOname',
view_solinx2.dateEntered AS 'SoDate',
view_solinx2.totalSOAmount AS 'SoTotal',
view_invlinx2.invId AS 'InvId',
IF(view_solinx2.notNeeded = 1,
'------',
view_invlinx2.`number`) AS 'InvName',
view_invlinx2.dateEntered AS 'InvDate',
view_invlinx2.amount AS 'InvTotal',
view_polinx2.poId AS 'POId',
IF(view_solinx2.notNeeded = 1,
'------',
view_polinx2.`number`) AS 'POName',
view_polinx2.dateEntered AS 'PODate',
view_polinx2.amount AS 'POTotal',
(SELECT
listsalesorders.number
FROM
listsalesorders
WHERE
listsalesorders.idx = autoassign) AS 'test',
`listlineitems`.*
FROM
`listlineitems`
LEFT JOIN
`listUsers` ON `listlineitems`.`individualId` = `listUsers`.`idx`
LEFT JOIN
`listprojects` ON `listlineitems`.`parentProjectId` = `listprojects`.`idx`
LEFT JOIN
`listassets` ON `listlineitems`.`parentAssetId` = `listassets`.`idx`
LEFT JOIN
`listareas` ON `listlineitems`.`parentAreaId` = `listareas`.`idx`
LEFT JOIN
`listlocations` ON `listlineitems`.`parentLocationId` = `listlocations`.`idx`
LEFT JOIN
`listcustomers` ON `listlineitems`.`parentCustomerId` = `listcustomers`.`idx`
LEFT JOIN
`listactivities` ON `listactivities`.`idx` = `listlineitems`.`activityCode`
LEFT JOIN
`listexpenseentry` ON (`listexpenseentry`.`idx` = `listlineitems`.`itemId`
AND `listlineitems`.`type` = 2)
LEFT JOIN
view_solinx2 ON view_solinx2.idx = listlineitems.idx
LEFT JOIN
view_polinx2 ON view_polinx2.idx = listlineitems.idx
LEFT JOIN
view_invlinx2 ON view_invlinx2.idx = listlineitems.idx
GROUP BY `listlineitems`.`idx`
ORDER BY `listlineitems`.`dateEntered` DESC
LIMIT 10;
I am at a loss as to what else I can do to improve this and any suggestions are very much appreciated.
You are selecting everything from listlineitems table (100+ K records), joining many tables, then grouping by idx and then throwing out most results.
You can:
Try to add unique index (dateEntered, idx) to listlineitems
Try limit listlineitems by dateEntered if acceptable (WHERE dateEntered > DATE_SUB(NOW(), INTERVAL 30 DAYS)). dateEntered must be indexed
Try to put select from listlineitems + grouping + limit into subquery so MySQL will do joins to only these 10 rows returned by subquery.
Convert dependent subquery (listsalesorders) to left join
Related
We have a scenario where users answer some questions related to a parent entity that we'll call a widget. Each question has both a numeric and word answer. Multiple users answer each question for a given widget.
We then display a row for each widget with the average numeric answer for each question. We do that using a MySQL pseudo-pivot with dynamic columns as detailed here So we end up with something like:
SELECT widget_id, ...
ROUND(IFNULL(AVG(CASE
WHEN LOWER(REPLACE(RQ.question, ' ', '_')) = 'overall_size' THEN
if(RA.num = '', 0, RA.num) END),0) + .0001, 2) AS `raw_avg_overall_size`,
...
... where overall_size would be one of the question types related to the widget and might have "answers" from 5 users like 1,2,2,3,1 to that question for a given widget_id based on the answer options below:
Answers
answer_id
answer_type
num
word
111
overall_size
1
x-large
112
overall_size
2
large
113
overall_size
3
medium
114
overall_size
4
small
115
overall_size
5
x-small
So we would end up with a row that had something like this:
widget_id
average_overall_size
115
1.80
What we can't figure out is then given if we round 1.80 to zero precision we get 2 in this example which is the word value 'large' from our data above. We like to include that in the query output too so that end up with:
widget_id
raw_average_overall_size
average_overall_size
115
1.80
large
The issue is that we do not know the average for the row until the query runs. So how can we then reference the word value for that average answer in the same row when executing the query?
As mentioned we are pivoting into a variable and then run another query for the full execution. So if we join in the pivot section, that subquery looks something like this:
SET #phase_id = 1;
SET SESSION group_concat_max_len = 100000;
SET #SQL = NULL;
SET #NSQL = NULL;
SELECT GROUP_CONCAT(DISTINCT
CONCAT(
'ROUND(IFNULL(AVG(CASE
WHEN LOWER(REPLACE(RQ.short_question, '' '', ''_'')) = ''',
nsq,
''' THEN
if(RA.answer = '''', 0, RA.answer) END),0) + .0001, 2) AS `',
CONCAT('avg_raw_',nsq), '`,
REF.value, -- <- ******* THIS FAILS **** --
ROUND(IFNULL(STDDEV(CASE
WHEN LOWER(REPLACE(RQ.short_question, '' '', ''_'')) = ''',
nsq,
''' THEN RA.answer END), 0) + .0001, 3) AS `',
CONCAT('std_dev_', nsq), '`
'
)
ORDER BY display_order
) INTO #NSQL
FROM (
SELECT FD.ref_value, FD.element_name, RQ.display_order, LOWER(REPLACE(RQ.short_question, ' ', '_')) as nsq
FROM review_questions RQ
LEFT JOIN form_data FD ON FD.id = RQ.form_data_id
LEFT JOIN ref_values RV on FD.ref_value = RV.type
WHERE RQ.phase_id = #phase_id
AND FD.element_type = 'select'
AND RQ.is_active > 0
GROUP BY FD.element_name
HAVING MAX(RV.key_name) REGEXP '^[0-9]+$'
) nq
/****** suggested in 1st answer ******/
LEFT JOIN ref_values REF ON REF.`type` = nq.ref_value
AND REF.key_name = ROUND(CONCAT('avg_raw_',nsq), 0);
So we need the word answer (from the REF join's REF.value field in the above code) in the pivot output, but it fails with 'Unknown column REF.value. If we put REF.value in it's parent query field list, that also fails with the same error.
You'll need to join the table/view/query again to get the 'large' value.
For example:
select a.*, b.word
from (
-- your query here
) a
join my_table b on b.answer_id = a.answer_id
and b.num = round(a.num);
An index on my_table (answer_id, num) will speed up the extra search.
This fails, leading to the default of "2":
LOWER(REPLACE(RQ.question, ' ', '_')) = 'overall_size'
That is because the question seems to be "average_overall_size", not "overall_size".
String parsing and manipulation is the pits in SQL; suggest using the application to handle such.
Also, be aware that you may need a separate subquery to compute aggregate (eg AVG()), else it might not be computed over the set of values you think.
Query into temp table, then join
First query should produce table as follows:
CREATE temp table, temp_average_size
widget_id
average_overall_size
rounded_average_size
115
1.80
2
LEFT JOIN
select s.*, a.word
from temp_average_size s LEFT JOIN answers a
ON (s.rounded_average_size = a.num AND a.answer_type = 'overall_size)
Working with Amazon Aurora MySQL 5.7.12. And got strange issue.
Have next query
SELECT d.BucketDate,
SUM(d.Value),
d.Key,
d.Code,
d.PlC
FROM MonthlyTable d
JOIN (SELECT #getting list of all available Key with last available BucketDate
d1.Key, MAX(d1.BucketDate) AS BucketDT
FROM MonthlyTable d1
JOIN PCTemp AS pc ON d1.SnapshotDate = 'SOME SNAPSHOT DATE' and pc.PC = d1.PC
JOIN PlCTemp AS p ON d1.SnapshotDate = 'SOME SNAPSHOT DATE' and p.PlC = d1.PlC
WHERE d1.SnapshotDate = 'SOME SNAPSHOT DATE'
AND d1.param1 = 1
AND d1.param2 IN (1 , 3)
GROUP BY d1.Key) dd ON d.SnapshotDate = 'SOME SNAPSHOT DATE' and d.Key = dd.Key
WHERE d.SnapshotDate = 'SOME SNAPSHOT DATE'
AND d.param1 = 1
AND d.param2 IN (3 , 1)
AND (d.BucketDate = dd.BucketDT OR d.BucketDate BETWEEN 'START DATE' AND 'END DATE')
GROUP BY d.Key , d.BucketDate;
The table is getting 6-8 millions of new records each day by SnapshotDate and partitioned by key on the SnapshotDate column.
Yesterday that query failed to return a result.
But:
1 - it started returning the correct result once I've changed d.param2 IN (3 , 1) to d.param2 IN (1 , 3)
2 - it started returning the correct result once I've forced the optimal index without query rewrite.
3 - it started returning the correct result after OPTIMIZE TABLE MonthlyTable
Any ideas on what was that?
How to prevent that happening again?
*** SOLUTION IN BOTTOM****
I am trying to add some comment at last column of my query , basically if technical_document.disc_id contains "KN" then i want to add a comment to the last column of query 'OK'
but i cant get it working
SELECT
technical_document.project_no,
technical_document.doc_no,
technical_document.doc_class,
technical_document.disc_id,
tag_document.project_id AS "COMMENT"
CASE WHEN technical_document.disc_id = 'KN' THEN tag_document.project_id = "YES"
FROM technical_document left join tag_document on technical_document.doc_no = tag_document.proj_doc_doc_no
WHERE tag_document.proj_doc_doc_no IS NULL
AND technical_document.project_no like '%%'
AND technical_document.doc_class = ANY ('A','S','SE')
AND technical_document.Lci_Code = ANY ('A','A1','B')
Thanks for your help, here is how it was solved, sorry for confusing/misleading explanation on my part:
SELECT
technical_document.project_no,
technical_document.doc_no,
technical_document.doc_class,
technical_document.disc_id,
CASE
WHEN technical_document.disc_id LIKE 'KN' THEN 'OK'
WHEN technical_document.disc_id LIKE 'K6' THEN 'OK'
ELSE null
END
FROM technical_document left join tag_document on technical_document.doc_no = tag_document.proj_doc_doc_no
WHERE tag_document.proj_doc_doc_no IS NULL
AND technical_document.project_no like '%%'
AND technical_document.doc_class = ANY ('A','S','SE')
AND technical_document.Lci_Code = ANY ('A','A1','B')
I would write the query as:
SELECT d.project_no, d.doc_no, d.doc_class, d.disc_id,
d.project_id AS COMMENT,
(CASE WHEN td.disc_id = 'KN' THEN td.project_id = 'YES' END)
FROM technical_document d LEFT JOIN
tag_document td
ON d.doc_no = td.proj_doc_doc_no
WHERE d.proj_doc_doc_no IS NULL AND
d.project_no like '%%' AND -- not needed
d.doc_class IN ('A', 'S', 'SE') AND
d.Lci_Code IN ('A', 'A1', 'B');
First, you have a WHERE clause that says that there is no match. Hence, including columns in the SELECT from tag_document doesn't make sense. I suspect you intend:
SELECT d.project_no, d.doc_no, d.doc_class, d.disc_id,
d.project_id as COMMENT,
(CASE WHEN td.disc_id = 'KN' THEN td.project_id = 'YES' END)
FROM technical_document d LEFT JOIN
tag_document td
ON d.doc_no = td.proj_doc_doc_no AND td.disc_id = 'KN'
WHERE d.project_no like '%%' AND -- not needed
d.doc_class IN ('A', 'S', 'SE') AND
d.Lci_Code IN ('A', 'A1', 'B');
Note the condition on td in the WHERE clause has been removed.
In addition:
Table aliases make the query easier to write and to read.
= ANY is allowed, but the colloquial operator is IN.
Various small syntax errors are fixed -- missing commas, missing END.
Previously, the following query was limited based on work_category_id. Turns out we need information from as subclass of another work_category_id, so now I'm limiting on job_code_id instead. Job_code_id 29 is work_category_id 88, all other job_code_ids shown are work_category_id 36.
I need hours and performance for each of these job_code_ids, but specifically for the 'cases_per_hr' calculation, I only want to divide cases by all hours except those from job_code_id 29. I tried a nested case when, but that didn't seem to make much sense. Please help!
SELECT
d.user_id as 'employee_ID',
round((sum(d.goal_hours)/sum(d.worked_hours)),2)*100 as 'performance',
round(sum(d.goal_hours),2) as 'goal_hrs',
round(sum(d.worked_hours),2) as 'hrs_worked',
sum(d.cases) as 'total_cases_slctd',
round(sum(d.cases)/sum(d.worked_hours),0) as 'cases_per_hr',
d.metric_dt
FROM
roster r,
prod_detail d
WHERE
d.process_level = r.process_level
and d.accounting_unit = r.accounting_unit
and d.job_code_id in ('29','322','304','303','302','305','181')
-- and d.work_category_id in('36')
If you only want to exclude job_code_id = '29' from your cases_per_hr calculation, you could do the following. It does use CASE WHEN.
SELECT d.user_id as 'employee_ID',
round((sum(d.goal_hours)/sum(d.worked_hours)),2)*100 as 'performance',
round(sum(d.goal_hours),2) as 'goal_hrs',
round(sum(d.worked_hours),2) as 'hrs_worked',
sum(d.cases) as 'total_cases_slctd',
round(
sum(CASE WHEN d.job_code_id <> '29' THEN d.cases ELSE 0 END)
/
sum(CASE WHEN d.job_code_id <> '29' THEN d.worked_hours ELSE 0 END)
,0) as 'cases_per_hr',
d.metric_dt
FROM roster r,
prod_detail d
WHERE
d.process_level = r.process_level
and d.accounting_unit = r.accounting_unit
and d.job_code_id in ('29','322','304','303','302','305','181')
Instead of running hundreds of SELECT queries for one operation, I want to just run one big query, which I'm hoping will ease the load on my server.
SELECT (
(SELECT link_type_id FROM connections WHERE (node_to_id = '0' AND node_from_id = '1') OR (node_from_id = '1' AND node_to_id = '0')),
(SELECT link_type_id FROM connections WHERE (node_to_id = '0' AND node_from_id = '2') OR (node_from_id = '2' AND node_to_id = '0'))
)
There will be many more SELECTS in this query, but even the two aren't working. When I run this code I get the error:
Operand should contain 1 column(s).
Any suggestions? Thanks!
You can try below may be but you may need UNION
SELECT link_type_id FROM connections
WHERE (node_to_id = '0' AND node_from_id = '1')
OR (node_from_id = '1' AND node_to_id = '0')
UNION
SELECT link_type_id FROM connections
WHERE (node_to_id = '0' AND node_from_id = '2')
OR (node_from_id = '2' AND node_to_id = '0')
This is caused by the parentheses outside of the columns. Even something as simple as:
SELECT ((SELECT 1), (SELECT 1))
Will yield this error -- the problem is that MySQL can only display one column per ... well ... column, and the entire () wraps a single column. If you remove the outer parentheses it will display each SELECT in a separate column.