N1QL choose the latest timestamp records for the conditions - couchbase

Hi I have an application that runs the below N1QL
please observe the letting clause , for the lowest value of the env we are picking other details.
with a change in the application we are getting multiple records for this criteria and we need to pick the one with max(t5.createdTs) . That is we need min(env) record with max time stamp.
How do i change this N1QL
I tried using order by before letting it gave me error
WITH ct3 AS ( SELECT m[1].appName as name , m[1].uuid as id ,m[1].description ,m[1].env , m[1].productStatus ,m[1].fourthParty as dcrFlag,
m[1].createdTs
FROM api_external AS t4
JOIN api_external AS t5 ON t4.uuid = t5.data.partnerAppId
WHERE t4.type = "partnerApp"
AND t4.data.companyId = '70a149da27cc425da86cba890bf5b143'
AND t5.type = "integration"
AND t5.data.partnerAppId IS NOT NULL
GROUP BY t4.uuid
LETTING m = MIN([t5.data.env, {t4.uuid, t4.data.appName, t4.data.description,
t5.data.env, t5.data.productStatus , t4.data.fourthParty , t4.uuid,t5.createdTs }]) )
select ct3.name ,
ct3.id ,
ct3.description ,
ct3.env ,
ct3.dcrFlag,
ct3.createdTs,
(select api_external.data.displayName as productName , uuid as productId
from api_external USE KEYS (ARRAY "product::" || v FOR v IN OBJECT_NAMES(ct3.productStatus) END) ) as ProductDetails
from ct3

WITH ct3 AS ( SELECT m[2].*
FROM api_external AS t4
JOIN api_external AS t5 ON t4.uuid = t5.data.partnerAppId
WHERE t4.type = "partnerApp"
AND t4.data.companyId = '70a149da27cc425da86cba890bf5b143'
AND t5.type = "integration"
AND t5.data.partnerAppId IS NOT NULL
GROUP BY t4.uuid
LETTING m = MIN([t5.data.env, -STR_TO_MILLIS(t5.createdTs), {"id": t4.uuid, "name": t4.data.appName, t4.data.description,
t5.data.env, t5.data.productStatus , "dcrFlag": t4.data.fourthParty , t4.uuid,t5.createdTs }]) )
SELECT c.*,
( SELECT p.data.displayName AS productName, uuid AS productId
FROM api_external AS p USE KEYS (ARRAY "product::" || v FOR v IN OBJECT_NAMES(c.productStatus) END)
) AS ProductDetails
FROM ct3 AS c;

Related

Problems with query speed when using a nested query for item count

When I add the nested query for invCount, my query time goes from .03 sec to 14 sec. The query works and I get correct values, but it is very, very slow in comparison. Is that just because I have to many conditions in that query? When I take it out and still have the second nested query, the time is still .03 secs. There is clearly something about the first nested query the database doesn't like, but I am not seeing what it is. I have a foreign key set for all the inner join lines too. Any help or ideas would be appreciated.
SELECT a.*,
f.name,
f.partNumber,
f.showInAdminStore,
f.showInPublicStore,
f.productImage,
r.mastCatID,
(SELECT COUNT(b.inventoryID)
FROM storeInventory b
INNER JOIN events c ON c.eventID = b.eventID
WHERE b.pluID = a.pluID
AND b.listPrice = a.listPrice
AND b.unlimitedQty = a.unlimitedQty
AND (b.packageID = a.packageID OR (b.packageID IS NULL AND a.packageID IS NULL))
AND b.orderID IS NULL
AND c.isOpen = '1'
AND b.paymentTypeID <= '2'
AND (b.inCart < '$cartTime' OR b.inCart IS NULL) ) AS invCount,
(SELECT COUNT(x.inventoryID)
FROM storeInventory x
WHERE x.packageID = a.inventoryID) AS packageCount
FROM storeInventory a
INNER JOIN storePLUs f ON f.pluID = a.pluID
INNER JOIN storeCategories r ON r.catID = f.catID
INNER JOIN events d ON d.eventID = a.eventID
WHERE a.storeFrontID = '1'
AND a.orderID IS NULL
AND a.paymentTypeID <= '2'
AND d.isOpen = '1'
GROUP BY a.packageID, a.unlimitedQty, a.listPrice, a.pluID
Table from query output
UPDATE: 12/12/2022
I changed the line checking the packageID to "AND (b.packageID <=> a.packageID)" as suggested and that cut my query time down to 7.8 seconds from 14 seconds. Thanks for the pointer. I will definitely use that in the future for NULL comparisons.
using "count(*)" took about half a second off. When I take the first nested query out, it drops down to .05 seconds even with the other nested queries in there, so I feel like there is still something causing issues. I tried running it without the other "AND (b.inCart < '$cartTime' OR b.inCart IS NULL)" line and that did take about a second off, but no where what I was hoping for. Is there an operand that includes NULL on a less than comparison? I also tried running it without the inner join in the nested query and that didn't change much at all. Of course removing any of that, throughs the values off and they become incorrect, so I can't run it that way.
Here is my current query setup that still pulls correct values.
SELECT a.*,
f.name,
f.partNumber,
f.showInAdminStore,
f.showInPublicStore,
f.productImage,
r.mastCatID,
(SELECT COUNT(*)
FROM storeInventory b
INNER JOIN events c ON c.eventID = b.eventID
WHERE b.pluID = a.pluID
AND b.listPrice = a.listPrice
AND b.unlimitedQty = a.unlimitedQty
AND (b.packageID <=> a.packageID)
AND b.orderID IS NULL
AND c.isOpen = '1'
AND b.paymentTypeID <= '2'
AND (b.inCart < '$cartTime' OR b.inCart IS NULL) ) AS invCount,
(SELECT COUNT(x.inventoryID)
FROM storeInventory x
WHERE x.packageID = a.inventoryID) AS packageCount
FROM storeInventory a
INNER JOIN storePLUs f ON f.pluID = a.pluID
INNER JOIN storeCategories r ON r.catID = f.catID
INNER JOIN events d ON d.eventID = a.eventID
WHERE a.storeFrontID = '1'
AND a.orderID IS NULL
AND a.paymentTypeID <= '2'
AND d.isOpen = '1'
GROUP BY a.packageID, a.unlimitedQty, a.listPrice, a.pluID
I am not familiar with the term 'Composite indexes' Is that something different than these?
Screenshot of ForeignKeys on Table a
I think
AND (b.packageID = a.packageID
OR (b.packageID IS NULL
AND a.packageID IS NULL)
)
can be simplified to ( https://dev.mysql.com/doc/refman/8.0/en/comparison-operators.html#operator_equal-to ):
AND ( b.packageID <=> a.packageID )
Use COUNT(*) instead of COUNT(x.inventoryID) unless you check for not-NULL.
The subquery to compute packageCount seems strange; you seem to count inventories but join on packages.
The need to reach into another table to check isOpen is part of the performance problem. If eventID is not the PRIMARY KEYforevents, then add INDEX(eventID, isOpen)`.
Some other indexes that may help:
a: INDEX(storeFrontID, orderID, paymentTypeID)
a: INDEX(packageID, unlimitedQty, listPrice, pluID)
b: INDEX(pluID, listPrice, unlimitedQty, orderID)
f: INDEX(pluID, catID)
r: INDEX(catID, mastCatID)
x: INDEX(packageID, inventoryID)
After OP's Update
There is no way to do (x<y OR x IS NULL) except by switching to a UNION. In your case, it is pretty easy to do the conversion. Replace
( SELECT COUNT(*) ... AND ( b.inCart < '$cartTime'
OR b.inCart IS NULL ) ) AS invCount,
with
( SELECT COUNT(*) ... AND b.inCart < '$cartTime' ) +
( SELECT COUNT(*) ... AND b.inCart IS NULL ) AS invCount,
Revised indexes:
storePLUs:
INDEX(pluID, catID)
storeCategories:
INDEX(catID, mastCatID)
events:
INDEX(isOpen, eventID)
storeInventory:
INDEX(pluID, listPrice, unlimitedQty, orderID, packageID)
INDEX(pluID, listPrice, unlimitedQty, orderID, inCart)
INDEX(packageID, inventoryID)
INDEX(storeFrontID, orderID, paymentTypeID)

I want to improve the speed of query statements, but I want to know how

When executing a query statement, the speed is very slow.
SELECT
T1.APPL_SEQ
, T1.COMP_CD
, (SELECT COMP_NM FROM tb_company WHERE COMP_CD = T1.COMP_CD) AS COMP_NM
, T1.GPROD_CD
, (SELECT GPROD_NM FROM tb_gprod WHERE GPROD_CD = T1.GPROD_CD) AS GPROD_NM
, T1.SITE_CD
, (SELECT SITE_NM FROM tb_site WHERE SITE_CD = T1.SITE_CD) AS SITE_NM
, T1.INFLOW_CD
, T1.INFLOW_URL
, T1.STATUS
, T1.REG_DTM
, DECRYPTO(T1.NAME) AS NAME
, DECRYPTO(T1.HP) AS HP
, ifnull(T1.AGE,T1.`115`) AS AGE
, ifnull(T1.GENDER,T1.`116`) AS GENDER
, ifnull(T1.MEMO,T1.`120`) AS MEMO
, ifnull(T1.`105`,T1.`124`) AS TIME
, T1.`125` AS AGE_CHILD
, T2.API_YN
, T2.API_START_DT
, T2.API_END_DT
, T2.API_CD
, T2.DATA_INFLOWCD
, T2.CONFIRM_YN
, T2.SALE_YN
, T2.SALE_PRICE
, T2.BREAKDOWN
, T2.INPUT_DATE
, T3.DIST_YN
, T3.DIST_DT
,(select ifnull((select timestampdiff(DAY, T11.REG_DTM,T1.REG_DTM) AS DIFF2REGTIME from tb_applicant T11 WHERE T11.HP = T1.HP AND T11.GPROD_CD = T1.GPROD_CD AND T11.REG_DTM < T1.REG_DTM order by T11.REG_DTM desc limit 1),-1)) AS HP2_COUNT
FROM
tb_applicant T1
LEFT JOIN mm_applicant T2
ON T1.APPL_SEQ = T2.APPL_SEQ
LEFT JOIN dist_applicant T3
ON T1.APPL_SEQ = T3.APPL_SEQ
LEFT JOIN tb_site T4
ON T4.site_cd = T1.SITE_CD and T4.comp_cd = T1.COMP_CD and T4.gprod_cd = T1.GPROD_CD
WHERE 1=1
AND T1.APPL_SEQ > 147293
AND T4.is_use = 'Y'
$Sql_Search
ORDER BY
$Sql_OrderBy
) U1
, (SELECT #ROWNUM := 0) U2
) V1";
,(select ifnull((...),-1)) AS HP2_COUNT
This is part of why it's so slow.
This query calculates the number of months difference by comparing REG_DTM when the td_applicant table has the same data for HP, GPROD, and COMP.
I don't need to get the date difference, is there any way to improve the query speed?
The main problem are those subselect in the select. As #Akina suggested, you should move them in FROM and make them as join.
They way you have done implies that each subselect is executed for each row returned by the main select.
You have 4 subselect that mean if you have 100 rows you execute 1 (main select) + (4*100) query so 401 instead of 1.
Using join allow the internal optimization engine to choose the best strategy to perform the query, in your way practically no optimization are applied.
I post a short example of how should be your query, didn't refactor the whole query since without database is a bit difficult to do it and I can easily produce a wrong query.
Notice that you select twice on tb_site with different condition, so is up to you to put the correct one.
SELECT T1.APPL_SEQ, T1.COMP_CD, T1.GPROD_CD, T1.SITE_CD
TC.COMP_NM,
TG.GPROD_NM,
TS.SITE_NM,
......
FROM tb_applicant T1
LEFT JOIN mm_applicant T2
JOIN tb_company TC on TC.COMP_CD = T1.COMP_CD
JOIN tb_gprod TG on GPROD_CD = T1.GPROD_CD
JOIN tb_site TS on TS.SITE_CD = T1.SITE_CD ON T1.APPL_SEQ = T2.APPL_SEQ
.......

How to use the SELECT clause twice in a sql statement

I want to have a table where I can view today's balance as well as yesterday's balance as two different column. Is there any way I can select from two different dates?
Below is the SQL statement I have tried however I am not able to see the yesterday balance.
(SELECT food.food_id, food.food_name, food.food_chi_name, food.food_category, food.chinesechar, SUM(inventory.tmr_input+inventory.final_balance), SUM(inventory.balance), SUM(inventory.input), SUM(inventory.reject), SUM(inventory.final_balance), SUM(inventory.tmr_input), SUM(inventory.sale), SUM(inventory.theoritical), SUM(inventory.yest_theoritical), SUM(inventory.3PMsale), SUM(inventory.3PMbalance), SUM(inventory.wholesale) FROM inventory INNER JOIN food ON inventory.food_id=food.food_id WHERE food.outlet = 'T11' AND inventory.date = '04/30/2021' GROUP BY food.food_id ORDER BY food.food_id ASC); (SELECT SUM(inventory.balance) as yesterday_balance FROM inventory INNER JOIN food ON inventory.food_id=food.food_id WHERE food.outlet = 'T11' AND inventory.date = '04/29/2021' GROUP BY food.food_id ORDER BY food.food_id ASC);
You can use a subselect for that purpose
And using aliases helps to get a better overview
SELECT
f1.food_id
, f1.food_name
, f1.food_chi_name
, f1.food_category
, f1.chinesechar
, SUM(i1.tmr_input+i1.final_balance)
, SUM(i1.balance)
, SUM(i1.input)
, SUM(i1.reject)
,SUM(i1.final_balance)
, SUM(i1.tmr_input)
, SUM(i1.sale)
, SUM(i1.theoritical)
, SUM(i1.yest_theoritical)
, SUM(i1.`3PMsale`)
, SUM(i1.`3PMbalance`)
, SUM(i1.wholesale)
,(SELECT SUM(i2.balance)
FROM
inventory i2 INNER JOIN food f2 ON i2.food_id=f2.food_id
WHERE f2.outlet = 'T11' AND i2.date = '04/29/2021'
AND f2.food_id = f1.food_id
) as yesterday_balance
FROM
inventory i1
INNER JOIN
food f1 ON i1.food_id=f1.food_id
WHERE
f1.outlet = 'T11'
AND i1.date = '04/30/2021'
GROUP BY f1.food_id
ORDER BY f1.food_id ASC;

MySQL Multiple Join Query with Limit on One Join

I have a MYSQL query I'm working on that pulls data from multiple joins.
select students.studentID, students.firstName, students.lastName, userAccounts.userID, userstudentrelationship.userID, userstudentrelationship.studentID, userAccounts.getTexts, reports.pupID, contacts.pfirstName, contacts.plastName, reports.timestamp
from userstudentrelationship
join userAccounts on (userstudentrelationship.userID = userAccounts.userID)
join students on (userstudentrelationship.studentID = students.studentID)
join reports on (students.studentID = reports.studentID)
join contacts on (reports.pupID = contacts.pupID)
where userstudentrelationship.studentID = "10000005" AND userAccounts.getTexts = 1 ORDER BY reports.timestamp DESC LIMIT 1
I have a unique situation where I would like one of the joins (the reports join) to be limited to the latest result only for that table (order by reports.timestamp desc limit 1 is what I use), while not limiting the result quantities for the overall query.
By running the above query I get the data I would expect, but only one record when it should return several.
My question:
How can I modify this query to ensure that I receive all possible records available, while ensuring that only the latest record from the reports join used? I expect that each record will possibly contain different data from the other joins, but all records returned by this query will share the same report record
Provided I understand the issue; one could add a join to a set of data (aliased Z below) that has the max timestamp for each student; thereby limiting to one report record (most recent) for each student.
SELECT students.studentID
, students.firstName
, students.lastName
, userAccounts.userID
, userstudentrelationship.userID
, userstudentrelationship.studentID
, userAccounts.getTexts
, reports.pupID
, contacts.pfirstName
, contacts.plastName
, reports.timestamp
FROM userstudentrelationship
join userAccounts
on userstudentrelationship.userID = userAccounts.userID
join students
on userstudentrelationship.studentID = students.studentID
join reports
on students.studentID = reports.studentID
join contacts
on reports.pupID = contacts.pupID
join (SELECT max(timestamp) mts, studentID
FROM REPORTS
GROUP BY StudentID) Z
on reports.studentID = Z.studentID
and reports.timestamp = Z.mts
WHERE userstudentrelationship.studentID = "10000005"
AND userAccounts.getTexts = 1
ORDER BY reports.timestamp
for get all the records you should avoid limit 1 at the end of the query
for join anly one row from reports table you could use subquery as
select
students.studentID
, students.firstName
, students.lastName
, userAccounts.userID
, userstudentrelationship.userID
, userstudentrelationship.studentID
, userAccounts.getTexts
, t.pupID
, contacts.pfirstName
, contacts.plastName
, t.timestamp
from userstudentrelationship
join userAccounts on userstudentrelationship.userID = userAccounts.userID
join students on userstudentrelationship.studentID = students.studentID
join (
select * from reports
order by reports.timestamp limit 1
) t on students.studentID = t.studentID
join contacts on reports.pupID = contacts.pupID
where userstudentrelationship.studentID = "10000005"
AND userAccounts.getTexts = 1

Not able to get the required ID's from a MySQL GROUP BY + HAVING query

I'm using a query various IN() clauses to get the price of product combinations. I am actually correctly getting the prices, so thats OK. But I also need some other ID's to identify to which ID's the prices belong.
As you can see IN's, they all start with a different bundle_variant_id's but the rest of the bundle_variant_id's are the same. I want to return that first (or all) of those bundle_variant_id's in the result too, but using this query, i always get '1620' as bundle_variant_id. How can i get the first bundle_variant_id in those IN's (1616, 1655, 1677, etc) or ALL of the bundle_variant_id's?
It's not an option to remove the GROUP and the HAVING or any of the WHERE clauses.
SELECT `HardwareProductSubscriptionInstanceLink`.`subscription_instance_id`
,`BundleVariantLink`.`bundle_variant_id`
,`HardwareProductSubscriptionInstanceLink`.`price`
FROM `wax`.`prd_providers` AS `Provider`
INNER JOIN `wax`.`prd_subscription_types` AS `SubscriptionType` ON (`SubscriptionType`.`provider_id` = `Provider`.`id`)
INNER JOIN `wax`.`prd_subscriptions` AS `Subscription` ON (`Subscription`.`subscription_type_id` = `SubscriptionType`.`id`)
INNER JOIN `wax`.`prd_subscription_instances` AS `SubscriptionInstance` ON (`SubscriptionInstance`.`subscription_id` = `Subscription`.`id`)
INNER JOIN `wax`.`prd_bundle_variants_subscription_instances` AS `BundleVariantLink` ON (`BundleVariantLink`.`subscription_instance_id` = `SubscriptionInstance`.`id`)
INNER JOIN `wax`.`prd_hardware_products_subscription_instances` AS `HardwareProductSubscriptionInstanceLink` ON (
`HardwareProductSubscriptionInstanceLink`.`subscription_instance_id` = `SubscriptionInstance`.`id`
AND `HardwareProductSubscriptionInstanceLink`.`hardware_product_id` = 317
)
WHERE (
(
`BundleVariantLink`.`bundle_variant_id` IN (
'1616'
,'1618'
,'1600'
,'1620'
)
)
OR (
`BundleVariantLink`.`bundle_variant_id` IN (
'1655'
,'1618'
,'1600'
,'1620'
)
)
OR (
`BundleVariantLink`.`bundle_variant_id` IN (
'1677'
,'1618'
,'1600'
,'1620'
)
)
OR (
`BundleVariantLink`.`bundle_variant_id` IN (
'1691'
,'1618'
,'1600'
,'1620'
)
)
AND `SubscriptionInstance`.`number_of_bundle_variants` = 4
AND (
(`Provider`.`enabled` = '1')
AND (`Provider`.`visible` = '1')
)
GROUP BY `SubscriptionInstance`.`id`
HAVING count(`BundleVariantLink`.`bundle_variant_id`) = 4
Current results:
(column 1 = subscription_instance_id, 2=bundle_variant_id,3=price)
9213 1620 331.405
9214 1620 311.57
9215 1620 291.736
9219 1620 390.909
Required results:
9213 1616 331.405
9214 1655 311.57
9215 1677 291.736
9219 1691 390.909
Or alternate required results:
9213 1616,1618,1600,1620 331.405
9214 1655,1618,1600,1620 311.57
9215 1677,1618,1600,1620 291.736
9219 1691,1618,1600,1620 390.909