I want to use the default search and pagination in yii2. But the query is complex and I don't know how can I add it to the search model! This is the query:
SELECT p.*,po_sum,rpo_sum,so_sum
FROMproduct p
LEFT JOIN (
SELECT id,product_id , IF(sum(quantity) IS NULL, 0, sum(quantity)) AS po_sum
FROM purchase_order_products inner join purchase_order on purchase_order.id = purchase_order_products.purchase_order_id
Where purchase_order.status = 'Approved'
GROUP BY product_id )
subcount ON p.id = subcount.product_id
LEFT JOIN (
SELECT id,product_id , sum(quantity) AS rpo_sum
FROM return_purchase_order_products inner join return_purchase_order on return_purchase_order.id = return_purchase_order_products.purchase_order_id Where return_purchase_order.status = 'Approved'
GROUP BY product_id )
subcount2 ON p.id = subcount2.product_id
LEFT JOIN (
SELECT product_id , sum(quantity_ordered) AS so_sum
FROM sales_order_item inner join sales_order on sales_order.id = sales_order_item.sales_order_id Where sales_order.order_status = 'complete'
GROUP BY product_id )
subcount3 ON p.id = subcount3.product_id
order by po_sum DESC,rpo_sum DESC
Any help?
If you use MySql >= 5.7.7 the easiest way is to create a view with that query and use it in the tableName method.
You need that version of MySql because you cant use subquery in from clause during view creation in previous versions.
I have a problem with my SQL join query. I have looked up other suggested answers and tried to apply it to my query, but it doesn't seem to be working.
I have this query:
SELECT SUM(p.quantity)
FROM stocktake_scans p
LEFT JOIN (
SELECT stocktake_area_id
FROM stocktake_areas
WHERE stocktake_id =8592 AND area_checked = 1
)d ON d.stocktake_area_id = p.stocktake_area_id
LEFT JOIN (
SELECT user_id
FROM stocktake_scan_edit
WHERE user_id =46521
)e ON e.user_id = p.stocktake_staff_id
WHERE p.stocktake_staff_id = 46521
And it gives me a result of 42, while I should get only 6. What is missing from the query?
I think you may have extra records with the same ID in your joined table that is where you are getting multiple rows returned from which is then calculating wrong in your sum, please try the below.
SELECT SUM(p.quantity) FROM stocktake_scans p LEFT JOIN ( SELECT distinct stocktake_area_id FROM stocktake_areas WHERE stocktake_id =8592 AND area_checked = 1 )d ON d.stocktake_area_id = p.stocktake_area_id LEFT JOIN ( SELECT distinct user_id FROM stocktake_scan_edit WHERE user_id =46521 )e ON e.user_id = p.stocktake_staff_id WHERE p.stocktake_staff_id = 46521
I am attempting to get the sum of 12 columns (in same table) in a subquery in an inner join.
Here is a link to my schema :
SqlFiddle
The query I am attempting to use is this:
SELECT
`inventory`.`part_number`,
`inventory`.`qty`,
`inventory`.`description`,
`reorder`.`reorder_point` * '1' `point`,
`inventory`.`cost`,
`vendor`.`name` AS `vendor_name`, SELECT (SUM(`saleshistory`.`Sales_1_Month_Prior`)+SUM(`saleshistory`.`Sales_2_Month_Prior`)+SUM(`saleshistory`.`Sales_3_Month_Prior`)+SUM(`saleshistory`.`Sales_4_Month_Prior`)+SUM(`saleshistory`.`Sales_5_Month_Prior`)+SUM(`saleshistory`.`Sales_6_Month_Prior`)+SUM(`saleshistory`.`Sales_7_Month_Prior`)+SUM(`saleshistory`.`Sales_8_Month_Prior`)+SUM(`saleshistory`.`Sales_9_Month_Prior`)+SUM(`saleshistory`.`Sales_10_Month_Prior`)+SUM(`saleshistory`.`Sales_11_Month_Prior`)+SUM(`saleshistory`.`Sales_12_Month_Prior`) AS TTL
FROM `inventory`
LEFT JOIN `reorder` ON `inventory`.`part_number` = `reorder`.`part_number`
LEFT JOIN `vendor` ON `inventory`.`vendor` = `vendor`.`vendor_id`
INNER JOIN `saleshistory` ON `saleshistory`.`location` = `inventory`.`location` AND `saleshistory`.`part_number` = `inventory`.`part_number`
WHERE `inventory`.`qty` <= `reorder`.`reorder_point`
AND `inventory`.`location` = '99'
AND `reorder`.`reorder_point` != '0'
GROUP BY `inventory`.`part_number`
ORDER BY `vendor`.`name` ASC
When using this query, it returns all the values for all the records not just the rows.
Something rare to happen is one to one where the second table can have millions of results for the first one. For example, I have a 'radcliente' table that has millions of 'radacct', but need to filter only with the last acct. The following are examples for better explanation:
This is criteria:
$criteria = new CDbCriteria();
$criteria->with = [
'acct', // slow because it will take millions of lines to have only the last
];
$criteria->together = true;
$clientes = Cliente::model()->findAll($criteria);
This is generated query by Yii (very slow, more then 40 seconds, it return millions of rows to use only one in AR):
SELECT
`t`.`id` AS `t0_c0`,
-- ...
`t`.`spc_serasa` AS `t0_c56`,
`acct`.`radacctid` AS `t1_c0`,
-- ...
`acct`.`cliente_id` AS `t1_c27`
FROM
`radcliente` `t`
LEFT OUTER JOIN `radacct` `acct` ON (`acct`.`cliente_id`=`t`.`id`)
ORDER BY
radacctid DESC
After apply my solution limit join to one row (is this fast! 200ms-):
SELECT
`t`.`id` AS `t0_c0`,
..
`t`.`spc_serasa` AS `t0_c56`,
`acct`.`radacctid` AS `t1_c0`,
-- ...
`acct`.`cliente_id` AS `t1_c27`
FROM
`radcliente` `t`
LEFT OUTER JOIN `radacct` `acct` ON (
acct.radacctid = (
SELECT radacctid
FROM `radacct` `acct`
WHERE (acct.cliente_id = t.id)
ORDER BY radacctid DESC
LIMIT 1
)
)
This is the generated query by CActiveDataProvider to total item count with my solution of limit join to one (slow, 10 seconds to count):
SELECT
COUNT(*)
FROM (
SELECT
`t`.`id` AS `t0_c0`,
-- ...
`t`.`spc_serasa` AS `t0_c56`,
`endereco_instalacao`.`id` AS `t1_c0`,
`telefones`.`id` AS `t2_c0`,
`telefones`.`telefone` AS `t2_c3`,
`emails`.`id` AS `t3_c0`,
`emails`.`email` AS `t3_c3`,
`metodo_cobranca`.`id` AS `t4_c0`,
`acct`.`radacctid` AS `t5_c0`,
`acct`.`framedipaddress` AS `t5_c22`
FROM
`radcliente` `t`
LEFT OUTER JOIN `radcliente_endereco_instalacao` `endereco_instalacao` ON (
endereco_instalacao.id = (
SELECT id
FROM `radcliente_endereco_instalacao` `endereco_instalacao`
WHERE (
endereco_instalacao.cliente_id = t.id
)
LIMIT 1
)
)
LEFT OUTER JOIN `radcliente_telefone` `telefones` ON (`telefones`.`cliente_id`=`t`.`id`)
LEFT OUTER JOIN `radcliente_email` `emails` ON (`emails`.`cliente_id`=`t`.`id`)
LEFT OUTER JOIN `radmetodo_cobranca` `metodo_cobranca` ON (
metodo_cobranca.id = (
SELECT id
FROM `radmetodo_cobranca` `metodo_cobranca`
WHERE (metodo_cobranca.cliente_id = t.id)
AND (metodo_cobranca.arquivo = 'nao')
ORDER BY metodo_cobranca.id DESC
LIMIT 1
)
)
LEFT OUTER JOIN `radacct` `acct` ON (
acct.radacctid = (
SELECT radacctid
FROM `radacct` `acct`
WHERE (acct.cliente_id = t.id)
ORDER BY radacctid DESC
LIMIT 1
)
)
GROUP BY t.id
) sq
But the problem is in the count generated by CActiveDataProvider (about 10 seconds to return the result) would have a way to optimize without having to lose the relationship (because I need to filter by a relationship in the future)?
UPDATE
Thank you for your response. I've been doing some tests and noticed that is slow in all cases, the table 'radacct' exacerbates the problem by its size, which should not therefore limit the 1 in the subquery. Follow the models and the link to access the system, if you need to authenticate is:
To access:
http://177.86.111.30/dev2/teste
username: help
password: 1
To download models and schema of radcliente and radacct: http://177.86.111.30/files.zip
Instead of ON id = ( SELECT ... LIMIT 1 ) try adding another JOIN (not LEFT JOIN):
JOIN ( SELECT ... LIMIT 1 ) x ON ...
The fear I have with your code is that it will be evaluating that subquery repeatedly, whenever it needs to check the ON clause. My rewrite will cause the subquery to happen only once.
Your query looks like a "correlated" subquery, so you would need to rephrase it to be non-correlated, if possible.
I have the following query which is actually within a stored procedure, but I removed it as there is too much going on inside the stored procedure. Basically this is the end result which takes ages (more than a minute) to run and I know the reason why - as you will also see from looking at the result of the explain - but I just cannot get it sorted.
Just to quickly explain what this query is doing. It is fetching all products from companies that are "connected" to the company where li.nToObjectID = 37. The result also returns some other information about the other companies like its name, company id, etc.
SELECT DISTINCT
SQL_CALC_FOUND_ROWS
p.id,
p.sTitle,
p.sTeaser,
p.TimeStamp,
p.ExpiryDate,
p.InStoreDate,
p.sCreator,
p.sProductCode,
p.nRetailPrice,
p.nCostPrice,
p.bPublic,
c.id as nCompanyID,
c.sName as sCompany,
m.id as nMID,
m.sFileName as sHighResFileName,
m.nSize,
(
Select sName
FROM tblBrand
WHERE id = p.nBrandID
) as sBrand,
(
Select t.sFileName
FROM tblThumbnail t
where t.nMediaID = m.id AND
t.sType = "thumbnail"
) as sFileName,
(
Select t.nWidth
FROM tblThumbnail t
where t.nMediaID = m.id AND
t.sType = "thumbnail"
) as nWidth,
(
Select t.nHeight
FROM tblThumbnail t
where t.nMediaID = m.id AND
t.sType = "thumbnail"
) as nHeight,
IF (
(
SELECT COUNT(id) FROM tblLink
WHERE
sType = "company"
AND sStatus = "active"
AND nToObjectID = 37
AND nFromObjectID = u.nCompanyID
),
1,
0
) AS bLinked
FROM tblProduct p
INNER JOIN tblMedia m
ON (
m.nTypeID = p.id AND
m.sType = "product"
)
INNER JOIN tblUser u
ON u.id = p.nUserID
INNER JOIN tblCompany c
ON u.nCompanyID = c.id
LEFT JOIN tblLink li
ON (
li.sType = "company"
AND li.sStatus = "active"
AND li.nToObjectID = 37
AND li.nFromObjectID = u.nCompanyID
)
WHERE c.bActive = 1
AND p.bArchive = 0
AND p.bActive = 1
AND NOW() <= p.ExpiryDate
AND (
li.id IS NOT NULL
OR (
li.id IS NULL
AND p.bPublic = 1
)
)
ORDER BY p.TimeStamp DESC
LIMIT 0, 52
Click here to see the output for EXPLAIN. Sorry, just couldn't get the formatting correct.
http://i60.tinypic.com/2hdqjgj.png
And lastly the number of rows for all the tables in this query:
tblProducts
Count: 5392
tblBrand
Count: 194
tblCompany
Count: 368
tblUser
Count: 416
tblMedia
Count: 5724
tblLink
Count: 24800
tblThumbnail
Count: 22207
So I have 2 questions:
1. Is there another way of writing this query which might potentially speed it up?
2. What index combination do I need for tblProducts so that not all the rows are searched through?
UPDATE 1
This is the new query after removing the subqueries and making use of left joins instead:
SELECT DISTINCT DISTINCT
SQL_CALC_FOUND_ROWS
p.id,
p.sTitle,
p.sTeaser,
p.TimeStamp,
p.ExpiryDate,
p.InStoreDate,
p.sCreator,
p.sProductCode,
p.nRetailPrice,
p.nCostPrice,
p.bPublic,
c.id as nCompanyID,
c.sName as sCompany,
m.id as nMID,
m.sFileName as sHighResFileName,
m.nSize,
brand.sName as sBrand,
thumb.sFilename,
thumb.nWidth,
thumb.nHeight,
IF (
(
SELECT COUNT(id) FROM tblLink
WHERE
sType = "company"
AND sStatus = "active"
AND nToObjectID = 37
AND nFromObjectID = u.nCompanyID
),
1,
0
) AS bLinked
FROM tblProduct p
INNER JOIN tblMedia m
ON (
m.nTypeID = p.id AND
m.sType = "product"
)
INNER JOIN tblUser u
ON u.id = p.nUserID
INNER JOIN tblCompany c
ON u.nCompanyID = c.id
LEFT JOIN tblLink li
ON (
li.sType = "company"
AND li.sStatus = "active"
AND li.nToObjectID = 37
AND li.nFromObjectID = u.nCompanyID
)
LEFT JOIN tblBrand AS brand
ON brand.id = p.nBrandID
LEFT JOIN tblThumbnail AS thumb
ON (
thumb.nMediaID = m.id
AND thumb.sType = 'thumbnail'
)
WHERE c.bActive = 1
AND p.bArchive = 0
AND p.bActive = 1
AND NOW() <= p.ExpiryDate
AND (
li.id IS NOT NULL
OR (
li.id IS NULL
AND p.bPublic = 1
)
)
ORDER BY p.TimeStamp DESC
LIMIT 0, 52;
UPDATE 2
ALTER TABLE tblThumbnail ADD INDEX (nMediaID,sType) USING BTREE;
ALTER TABLE tblMedia ADD INDEX (nTypeID,sType) USING BTREE;
ALTER TABLE tblProduct ADD INDEX (bArchive,bActive,ExpiryDate,bPublic,TimeStamp) USING BTREE;
After doing the above changes the explain showed that it is now only searching through 1464 rows on tblProduct instead of 5392.
That's a big query with a lot going on. It's going to take a few steps of work to optimize it. I will take the liberty of just presenting a couple of steps.
First step. Can you get rid of SQL_CALC_FOUND_ROWS and still have your program work correctly? If so, do that. When you specify SQL_CALC_FOUND_ROWS it sometimes means the server has to delay sending you the first row of your resultset until the last row is available.
Second step. Refactor the dependent subqueries to be JOINs instead.
Here's how you might approach that. Part of your query looks like this...
SELECT DISTINCT SQL_CALC_FOUND_ROWS
p.id,
...
c.id as nCompanyID,
...
m.id as nMID,
...
( /* dependent subquery to be removed */
Select sName
FROM tblBrand
WHERE id = p.nBrandID
) as sBrand,
( /* dependent subquery to be removed */
Select t.sFileName
FROM tblThumbnail t
where t.nMediaID = m.id AND
t.sType = "thumbnail"
) as sFileName,
( /* dependent subquery to be removed */
Select t.nWidth
FROM tblThumbnail t
where t.nMediaID = m.id AND
t.sType = "thumbnail"
) as nWidth,
( /* dependent subquery to be removed */
Select t.nHeight
FROM tblThumbnail t
where t.nMediaID = m.id AND
t.sType = "thumbnail"
) as nHeight,
...
Try this instead. Notice how the brand and thumbnail dependent subqueries disappear. You had three dependent subqueries for the thumbnail; they can disappear into a single JOIN.
SELECT DISTINCT SQL_CALC_FOUND_ROWS
p.id,
...
brand.sName,
thumb.sFilename,
thumb.nWidth,
thumb.nHeight,
...
FROM tblProduct p
INNER JOIN tblMedia AS m ON (m.nTypeID = p.id AND m.sType = 'product')
... (other table joins) ...
LEFT JOIN tblBrand AS brand ON p.id = p.nBrandID
LEFT JOIN tblMedia AS thumb ON (t.nMediaID = m.id AND thumb.sType = 'thumbnail')
I used LEFT JOIN rather than INNER JOIN so MySQL will present NULL values if the joined rows are missing.
Edit
You're using a join pattern that looks like this:
JOIN sometable AS s ON (s.someID = m.id AND s.sType = 'string')
You seem to do this for a few tables. You probably can speed up the JOIN operations by creating compound indexes in those tables. For example, try adding the following index to tblThumbnail: (sType, nMediaID). You can do that with this DDL statement.
ALTER TABLE tblThumbnail ADD INDEX (sType, nMediaID) USING BTREE
You can do similar things to other tables with the same join pattern.