I have 2 queries that I'm union like this. But the problem is that ID from table1 and table2 can be different. how can I join them and make the same?
qry1 = (
select(
func.max(table1.id).label("id"),
table1.warehouse_id,
table1.product_id,
)
.group_by(
table1.warehouse_id,
table1.product_id,
)
.order_by(func.max(table1.id))
qry2 = (
select(func.max(table2.id).label("id"),
table2.warehouse_id,
table2.product_id,
.join(
table3,
and_(
table2.code == table3.code,
),
)
.group_by(
table2.warehouse_id,
table2.product_id,
)
.order_by(KcLpnSerialNumberDao.id)
)
stmt = qry1.union(qry2)
result = session.execute(stmt)
Related
I can't find solution to correct this big query, I always receive an error from database.
I have tre tables and I need to make changes on some attribute on some condition:
UPDATE o36t_orders as temp1,
mytable as temp2
SET
temp1.bonifico = 1,
temp2.ultimo = 1
WHERE
temp1.id_order IN (
SELECT
id_order
FROM o36t_orders
LEFT JOIN o36t_address ON (o36t_address.id_address = o36t_orders.id_address_delivery)
LEFT JOIN mytable ON (
mytable.Causale = CONCAT(
o36t_address.lastname,
' ',
o36t_address.firstname
)
)
WHERE
o36t_orders.bonifico <> 1
)
AND temp2.id IN (
SELECT
id
FROM o36t_orders
LEFT JOIN o36t_address ON (o36t_address.id_address = o36t_orders.id_address_delivery)
LEFT JOIN mytable ON (
mytable.Causale = CONCAT(
o36t_address.lastname,
' ',
o36t_address.firstname
)
)
WHERE
o36t_orders.bonifico <> 1
)
Since the subqueries of the 2 IN clauses are identical (except the retuned column), I think that you can do what you want by a straight inner join of the 2 tables and that subquery (returning both columns):
UPDATE o36t_orders temp1
INNER JOIN (
SELECT
id, id_order
FROM o36t_orders
LEFT JOIN o36t_address ON (o36t_address.id_address = o36t_orders.id_address_delivery)
LEFT JOIN mytable ON (
mytable.Causale = CONCAT(
o36t_address.lastname,
' ',
o36t_address.firstname
)
)
WHERE
o36t_orders.bonifico <> 1
) t ON t.id_order = temp1.id_order
INNER JOIN mytable temp2 ON temp2.id = t.id
SET
temp1.bonifico = 1,
temp2.ultimo = 1
When I use left join to perform a grouped query of two tables, the generated count statement is always count(*), I want to generate the count of the specified column. For example: count(InstrumentCode).
Framework: EF Core 3.0-preview9
var resultQuery = from t1 in query
join t2 in orgCrmQuery on t1.CUSTOMER equals t2._ID
join t3 in orgQuery on t2.NAME equals t3.ORGANIZATION_NAME into t4
from t5 in t4.DefaultIfEmpty()
join t6 in instrumentQuery on t5.ORGANIZATION_ID equals t6.ORGANIZATION_ID into t7
from t8 in t7.DefaultIfEmpty()
where t2.RECORD_TYPE == dto.RecordType && t1.MEETING_NO == dto.MeetingNo
group t8.INSTRUMENT_CODE by new {
t1.PERSON_NAME,
t1.SEX,
t1.POSTION,
t1.PHONE,
t1.IS_CIZHU_KEHU,
t2.NAME,
t2.ORG_LEVEL,
} into g
orderby g.Key.NAME
select new {
g.Key.PERSON_NAME,
g.Key.SEX,
g.Key.POSTION,
g.Key.PHONE,
g.Key.IS_CIZHU_KEHU,
g.Key.NAME,
g.Key.ORG_LEVEL,
INSTRUMENT_COUNT = g.Count()
};
SQL statement generated：
SELECT
[t].[PERSON_NAME],
[t].[SEX],
[t].[POSTION],
[t].[PHONE],
[t].[IS_CIZHU_KEHU],
[t0].[NAME],
[t0].[ORG_LEVEL],
COUNT ( * ) AS [INSTRUMENT_COUNT]
FROM
[dbo].[TB_MEETING_PERSONAL] AS [t] WITH ( NOLOCK )
INNER JOIN [dbo].[TB_ORG_CRM] AS [t0] WITH ( NOLOCK ) ON [t].[CUSTOMER] = [t0].[_ID]
LEFT JOIN [dbo].[MS_ORGANIZATION] AS [m] WITH ( NOLOCK ) ON [t0].[NAME] = [m].[ORGANIZATION_NAME]
LEFT JOIN [dbo].[MS_INSTRUMENT] AS [m0] WITH ( NOLOCK ) ON [m].[ORGANIZATION_ID] = [m0].[ORGANIZATION_ID]
WHERE
(
( ( [t0].[RECORD_TYPE] = #__dto_RecordType_0 ) AND ( [t0].[RECORD_TYPE] IS NOT NULL AND #__dto_RecordType_0 IS NOT NULL ) )
OR ( [t0].[RECORD_TYPE] IS NULL AND #__dto_RecordType_0 IS NULL )
)
AND (
( ( [t].[MEETING_NO] = #__dto_MeetingNo_1 ) AND ( [t].[MEETING_NO] IS NOT NULL AND #__dto_MeetingNo_1 IS NOT NULL ) )
OR ( [t].[MEETING_NO] IS NULL AND #__dto_MeetingNo_1 IS NULL )
)
GROUP BY
[t].[PERSON_NAME],
[t].[SEX],
[t].[POSTION],
[t].[PHONE],
[t].[IS_CIZHU_KEHU],
[t0].[NAME],
[t0].[ORG_LEVEL]
ORDER BY
[t0].[NAME] OFFSET 0 ROWS FETCH NEXT 100 ROWS ONLY
I checked some information, but I didn't find a solution.
Based on your linq, same result for .Count(INSTRUMENT_COUNT) and .Count(*) or .Count(1) because this will return the # of rows regardless of the value of your INSTRUMENT_COUNT.
But if you want to get count specifically for INSTRUMENT_COUNT you can try this.
var resultQuery = from t1 in query
join t2 in orgCrmQuery on t1.CUSTOMER equals t2._ID
join t3 in orgQuery on t2.NAME equals t3.ORGANIZATION_NAME into t4
from t5 in t4.DefaultIfEmpty()
join t6 in instrumentQuery on t5.ORGANIZATION_ID equals t6.ORGANIZATION_ID into t7
from t8 in t7.DefaultIfEmpty()
where t2.RECORD_TYPE == dto.RecordType && t1.MEETING_NO == dto.MeetingNo
group t8.INSTRUMENT_CODE by new {
t1.PERSON_NAME,
t1.SEX,
t1.POSTION,
t1.PHONE,
t1.IS_CIZHU_KEHU,
t2.NAME,
t2.ORG_LEVEL,
t8.INSTRUMENT_CODE,
} into g
orderby g.Key.NAME
select new {
g.Key.PERSON_NAME,
g.Key.SEX,
g.Key.POSTION,
g.Key.PHONE,
g.Key.IS_CIZHU_KEHU,
g.Key.NAME,
g.Key.ORG_LEVEL,
INSTRUMENT_COUNT = g.Select(x => x.INSTRUMENT_CODE).Distinct().Count()
};
I have two tables T1 1 000 records and T2 with 500 000 records. I have a query where I run a join between them and fetch data by performing some aggregations. My page seems to be loading slow. Are there any approaches to make this query faster?
I have created indexes on columns for which aggregations are being performed. I think it is a generic statement.
$query = Mymodel::selectRaw("supplier_data.name as distributor,supplier_data.name as name, supplier_data.group_id as group_id, supplier_data.pay,supplier_data.group_id as submitted_group_plan,supplier_data.group_id as group_id_string,
(SELECT sum(t.net_claim) AS trans_number
FROM transactions_data_new as t
JOIN `supplier_data` AS d ON `t`.`member_id` = `d`.`group_id`
WHERE
(
(
t.`submit_date`>= '$date_from' and t.`submit_date`<= '$date_to'
AND t.`member_id` = supplier_data.group_id
)
OR
(
(t.claim_status IS NULL)
AND
(t.submit_date is NULL)
)
)
AND d.id = supplier_data.id
) as trans_number,
(SELECT sum(t.claim) AS trans_number
FROM transactions_data_new as t
JOIN `supplier_data` AS d ON `t`.`member_id` = `d`.`group_id`
WHERE
(
(
t.`submit_date`>= '$date_from' and t.`submit_date`<= '$date_to'
AND t.`member_id` = supplier_data.group_id
)
OR
(
(t.claim_status IS NULL)
AND
(t.submit_date is NULL)
)
)
AND d.id = supplier_data.id
) as claim,
(SELECT sum(t.reversed) AS trans_number
FROM transactions_data_new as t
JOIN `supplier_data` AS d ON `t`.`member_id` = `d`.`group_id`
WHERE
(
(
t.`submit_date`>= '$date_from' and t.`submit_date`<= '$date_to'
AND t.`member_id` = supplier_data.group_id
)
OR
(
(t.claim_status IS NULL)
AND
(t.submit_date is NULL)
)
)
AND d.id = supplier_data.id
) as reversed,
(SELECT sum(t.reversal) AS trans_number
FROM transactions_data_new as t
JOIN `supplier_data` AS d ON `t`.`member_id` = `d`.`group_id`
WHERE
(
(
t.`submit_date`>= '$date_from' and t.`submit_date`<= '$date_to'
AND t.`member_id` = supplier_data.group_id
)
OR
(
(t.claim_status IS NULL)
AND
(t.submit_date is NULL)
)
)
AND d.id = supplier_data.id
) as reversal
");
I don't see the need of this too complex/repeated with same clauses and multiple sub selects for same table which can done using a single left join
SELECT
s.name AS distributor,
s.name AS name,
s.group_id AS group_id,
s.pay,
s.group_id AS submitted_group_plan,
s.group_id AS group_id_string,
SUM(t.net_claim) AS trans_number,
SUM(t.claim) AS claim,
SUM(t.reversed) reversed,
SUM(t.reversal) reversal
FROM
supplier_data s
LEFT JOIN transactions_data_new t
ON `t`.`member_id` = s.`group_id`
AND (
(
t.`submit_date` >= '$date_from'
AND t.`submit_date` <= '$date_to'
)
OR (
t.claim_status IS NULL
AND t.submit_date IS NULL
)
)
GROUP BY s.name,
s.group_id,
s.pay
As I understand it the chunk() method is for use when you need to work with a large dataset and take an action on that data chunk by chunk.
From your question, it sounds like you're performing a query then returning the data as JSON so to me, it doesn't sound like you're taking an action on your dataset that requires chunking.
If you want to break up the returned JSON data you should be instead looking at pagination.
You could apply pagination to your query like so:
$data = Inspector::latest('id')
->select('id', 'firstname', 'status', 'state', 'phone')
->where('firstname', 'LIKE', '%' . $searchtext . '%')
->paginate();
You can specify the size of each set by passing a number to the paginate method:
$data = Inspector::latest('id')
->select('id', 'firstname', 'status', 'state', 'phone')
->where('firstname', 'LIKE', '%' . $searchtext . '%')
->paginate(25);
If I've misunderstood and you did actually want to do the chunking, I believe you could do the following:
$data = Inspector::latest('id')
->select('id', 'firstname', 'status', 'state', 'phone')
->where('firstname', 'LIKE', '%' . $searchtext . '%')
->chunk(50, function($inspectors) {
foreach ($inspectors as $inspector) {
// apply some action to the chunked results here
}
});
Also, if you're returning an eloquent object it will be automatically cast to json so you don't need to perform json_encode() as far as I'm aware.
I have MySQL query currently selecting and joining 13 tables and finally grouping ~60k rows. The query without grouping takes ~0ms but with grouping the query time increases to ~1.7sec. The field, which is used for grouping is primary field and is indexed. Where could be the issue?
I know group by without aggregate is considered invalid query and bad practise but I need distinct base table rows and can not use DISTINCT syntax.
The query itself looks like this:
SELECT `table_a`.*
FROM `table_a`
LEFT JOIN `table_b`
ON `table_b`.`invoice` = `table_a`.`id`
LEFT JOIN `table_c` AS `r1`
ON `r1`.`invoice_1` = `table_a`.`id`
LEFT JOIN `table_c` AS `r2`
ON `r2`.`invoice_2` = `table_a`.`id`
LEFT JOIN `table_a` AS `i1`
ON `i1`.`id` = `r1`.`invoice_2`
LEFT JOIN `table_a` AS `i2`
ON `i2`.`id` = `r2`.`invoice_1`
JOIN `table_d` AS `_u0`
ON `_u0`.`id` = 1
LEFT JOIN `table_e` AS `_ug0`
ON `_ug0`.`user` = `_u0`.`id`
JOIN `table_f` AS `_p0`
ON ( `_p0`.`enabled` = 1
AND ( ( `_p0`.`role` < 2
AND `_p0`.`who` IS NULL )
OR ( `_p0`.`role` = 2
AND ( `_p0`.`who` = '0'
OR `_p0`.`who` = `_u0`.`id` ) )
OR ( `_p0`.`role` = 3
AND ( `_p0`.`who` = '0'
OR `_p0`.`who` = `_ug0`.`group` ) ) ) )
AND ( `_p0`.`action` = '*'
OR `_p0`.`action` = 'read' )
AND ( `_p0`.`related_table` = '*'
OR `_p0`.`related_table` = 'table_name' )
JOIN `table_a` AS `_e0`
ON ( ( `_p0`.`related_id` = 0
OR `_p0`.`related_id` = `_e0`.`id`
OR `_p0`.`related_user` = `_e0`.`user`
OR `_p0`.`related_group` = `_e0`.`group` )
OR ( `_p0`.`role` = 0
AND `_e0`.`user` = `_u0`.`id` )
OR ( `_p0`.`role` = 1
AND `_e0`.`group` = `_ug0`.`group` ) )
AND `_e0`.`id` = `table_a`.`id`
JOIN `table_d` AS `_u1`
ON `_u1`.`id` = 1
LEFT JOIN `table_e` AS `_ug1`
ON `_ug1`.`user` = `_u1`.`id`
JOIN `table_f` AS `_p1`
ON ( `_p1`.`enabled` = 1
AND ( ( `_p1`.`role` < 2
AND `_p1`.`who` IS NULL )
OR ( `_p1`.`role` = 2
AND ( `_p1`.`who` = '0'
OR `_p1`.`who` = `_u1`.`id` ) )
OR ( `_p1`.`role` = 3
AND ( `_p1`.`who` = '0'
OR `_p1`.`who` = `_ug1`.`group` ) ) ) )
AND ( `_p1`.`action` = '*'
OR `_p1`.`action` = 'read' )
AND ( `_p1`.`related_table` = '*'
OR `_p1`.`related_table` = 'table_name' )
JOIN `table_g` AS `_e1`
ON ( ( `_p1`.`related_id` = 0
OR `_p1`.`related_id` = `_e1`.`id`
OR `_p1`.`related_user` = `_e1`.`user`
OR `_p1`.`related_group` = `_e1`.`group` )
OR ( `_p1`.`role` = 0
AND `_e1`.`user` = `_u1`.`id` )
OR ( `_p1`.`role` = 1
AND `_e1`.`group` = `_ug1`.`group` ) )
AND `_e1`.`id` = `table_a`.`company`
WHERE `table_a`.`date_deleted` IS NULL
AND `table_a`.`company` = 4
AND `table_a`.`type` = 1
AND `table_a`.`date_composed` >= '2016-05-04 14:43:55'
GROUP BY `table_a`.`id`
The ORs kill performance.
This composite index may help: INDEX(company, type, date_deleted, date_composed).
LEFT JOIN table_b ON table_b.invoice = table_a.id seems to do absolutely nothing other than slow down the processing. No fields of table_b are used or SELECTed. Since it is a LEFT join, it does not limit the output. Etc. Get rid if it, or justify it.
Ditto for other joins.
What happens with JOIN and GROUP BY: First, all the joins are performed; this explodes the number of rows in the intermediate 'table'. Then the GROUP BY implodes the set of rows.
One technique for avoiding this explode-implode sluggishness is to do
SELECT ...,
( SELECT ... ) AS ...,
...
instead of a JOIN or LEFT JOIN. However, that works only if there is zero or one row in the subquery. Usually this is beneficial when an aggregate (such as SUM) can be moved into the subquery.
For further discussion, please include SHOW CREATE TABLE.
I have written the following query in mysql for a reporting tool.
It is a inline select.The query gives me correct result but takes a long time to run. Can someone suggest any alternate way or writing the query to make it more efficient.
SELECT z.name1, (
SELECT COUNT( AES_DECRYPT( l.answertext, "aaa" ) )
FROM household_data l
INNER JOIN sms_household m
ON l.prim_key = m.hhid
INNER JOIN sms_psu n
ON n.psu = m.psu
AND n.state = m.state
AND n.district = m.district
INNER JOIN (
SELECT p.prim_key, p.fieldname
FROM household_data p
WHERE p.basicname = 'Q05'
AND AES_DECRYPT( p.answertext, "aaa" ) =2
) women
ON l.prim_key = women.prim_key
AND SUBSTR( l.fieldname, 5, 1 ) = SUBSTR( women.fieldname, 5, 1 )
WHERE l.basicname = 'Q08' AND AES_DECRYPT( l.answertext, "aaa" ) = 14
AND z.psu = n.psu
[AND n.state IN ( {state} )]
[AND n.district IN ( {district} )]
) female14, (
SELECT COUNT( AES_DECRYPT( l.answertext, "aaa" ) )
FROM household_data l
INNER JOIN sms_household m ON l.prim_key = m.hhid
INNER JOIN sms_psu n
ON n.psu = m.psu
AND n.state = m.state
AND n.district = m.district
INNER JOIN (
SELECT p.prim_key, p.fieldname
FROM household_data p
WHERE p.basicname = 'Q05'
AND AES_DECRYPT( p.answertext, "aaa" ) =2
) women
ON l.prim_key = women.prim_key
AND SUBSTR( l.fieldname, 5, 1 ) = SUBSTR( women.fieldname, 5, 1 )
WHERE l.basicname = 'Q08' AND AES_DECRYPT( l.answertext, "aaa" ) = 15
AND z.psu = n.psu
[AND n.state IN ( {state} )]
[AND n.district IN ( {district} )]
) female15, (
SELECT COUNT( AES_DECRYPT( l.answertext, "aaa" ) )
FROM household_data l
INNER JOIN sms_household m
ON l.prim_key = m.hhid
INNER JOIN sms_psu n
ON n.psu = m.psu
AND n.state = m.state
AND n.district = m.district
INNER JOIN (
SELECT p.prim_key, p.fieldname
FROM household_data p
WHERE p.basicname = 'Q05'
AND AES_DECRYPT( p.answertext, "aaa" ) =2
) women
ON l.prim_key = women.prim_key
AND SUBSTR( l.fieldname, 5, 1 ) = SUBSTR( women.fieldname, 5, 1 )
WHERE l.basicname = 'Q08' AND AES_DECRYPT( l.answertext, "aaa" ) =16
AND z.psu = n.psu
[AND n.state IN ( {state} )]
[AND n.district IN ( {district} )]
) female16, (
SELECT count(AES_DECRYPT(household_data.answertext , "aaa")) * 100 / (
SELECT count(AES_DECRYPT(household_data.answertext , "aaa"))
FROM household_data
INNER JOIN sms_household
INNER JOIN sms_psu
ON sms_psu.psu = sms_household.psu
AND sms_psu.state = sms_household.state
AND sms_psu.district = sms_household.district
WHERE basicname = 'Q07_year' AND z.psu= sms_psu.psu
[AND sms_psu.state IN ( {state} )]
[AND sms_psu.district IN ( {district} )]
)
FROM household_data
INNER JOIN sms_household
INNER JOIN sms_psu
ON sms_psu.psu = sms_household.psu
AND sms_psu.state = sms_household.state
AND sms_psu.district = sms_household.district
WHERE AES_DECRYPT(household_data.answertext , "aaa") = 9998
AND basicname = 'Q07_year'
AND z.psu = sms_psu.psu
[AND sms_psu.state IN ( {state} )]
[AND sms_psu.district IN ( {district} )]
) PercYearDontKnow
FROM household_data x
INNER JOIN sms_household y
ON x.prim_key = y.hhid
INNER JOIN sms_psu z
ON y.psu = z.psu
AND z.state = y.state
AND z.district = y.district
WHERE 1=1
[AND y.state IN ( {state} )]
[AND y.district IN ( {district} )
GROUP BY z.psu
I've edited your post to make your query structure more clear. I suggest that you do the same in your own code.
After the restructure, it becomes clear that you are repeating some of your query to find female14, female15 and female16.
Maybe you should make a separate query for that part, like this:
SELECT n.name1,
n.psu,
AES_DECRYPT( l.answertext, "aaa" ) AS answer,
COUNT(*) as count
FROM household_data l
INNER JOIN sms_household m
ON l.prim_key = m.hhid
INNER JOIN sms_psu n
ON n.psu = m.psu
AND n.state = m.state
AND n.district = m.district
INNER JOIN (
SELECT p.prim_key, p.fieldname
FROM household_data p
WHERE p.basicname = 'Q05'
AND AES_DECRYPT( p.answertext, "aaa" ) =2
) women
ON l.prim_key = women.prim_key
AND SUBSTR( l.fieldname, 5, 1 ) = SUBSTR( women.fieldname, 5, 1 )
WHERE l.basicname = 'Q08'
AND AES_DECRYPT( l.answertext, "aaa" ) = 14
[AND n.state IN ( {state} )]
[AND n.district IN ( {district} )]
GROUP BY n.psu, AES_DECRYPT( l.answertext, "aaa" )
That query should give you a summary of Q08 answers, and their numbers.
You can then make a separate query for PercYearDontKnow. I believe that re-assembling the data afterwards will still be faster than the frankenquery. Alternatively, code the above query as a SQL view, and assemble it into the larger query.
By the way, instead of using COUNT( AES_DECRYPT( l.answertext, "aaa" ) ), you can probably get away with COUNT(*). It means that the decrypt function will be called less often.
Another option for you is to do SELECT AES_ENCRYPT( "2", "aaa" ) and to use that value as a constant when comparing for Q08. That way, each field does not need to be repeatedly decrypted.
Afterwards, I'd follow the optimization advice given in another answer here, specifically looking at the execution path to see if you need to add indexes.
Ugly one...my advice:
check indexes and see what can be done to improve performance
check your table structure, be sure is the most close the right one
check your joins and see what can be changed
always always check execution plan for performance monitoring
And of course, do all that on a test enviroment, not production, it can be messy.
Another advice, do one step at a time, check each small modification to be sure the results are still the same. Save results to compare and get the good one at the end.