MySQL How To Group by Id that Id contains multiple Ids - mysql

Question

Table qtlsdmx
Id
pid (Receipt Id)
qrrq (Unix Datetime)
sl (QTY)
je (Total Price Don't * QTY)
spmc (Product Name)
zdcx_ids (Promo Id)
1
1
1653839999
3
127.26
Product1
175,167
2
1
1653839999
2
84.84
Product2
175,167
3
1
1653839999
1
183.42
Product3
167
4
1
1653839999
1
165.74
Product4
167
5
1
1653839999
1
165.74
Product4
167
Table zdcxd (Id = Table qtlsdmx Column zdcx_ids)
Id (Promo Id)
hdmc (Promo Name)
167
$500 - $40
175
$25 VOUCHER
177
$50 VOUCHER
179
$75 VOUCHER
Table qtlsd (Id = Table qtlsdmx Column pid)
Id (Receipt Id)
zddm (Shop name)
1
SHOP01
2
SHOP02
3
SHOP03
I need to analyze some promotion using above table.
The main issue is zdcx_ids can contains multiple promotion id. But I need to group it one at a time. I'd tried to create a new column with splitted zdcx_ids (SEE REF1 Output). But this method basically duplicate the record and cause Id not unique and QTY, Price etc. incorrect.
May I ask how to possible to code it? If possible please avoid using IN and LIKE "%175%" method. Because there's over 8mil record in it and loading time is huge.
REF1 Output:
Id
pid (Receipt Id)
qrrq (Unix Datetime)
sl (QTY)
je (Total Price Don't * QTY)
spmc (Product Name)
zdcx_ids (Promo Id)
splitted_zdcx_ids (Splitted Promo Id)
1
1
1653839999
3
127.26
Product1
175,167
175
1
1
1653839999
3
127.26
Product1
175,167
167
2
1
1653839999
2
84.84
Product2
175,167
175
2
1
1653839999
2
84.84
Product2
175,167
167
3
1
1653839999
1
183.42
Product3
167
167
4
1
1653839999
1
165.74
Product4
167
167
5
1
1653839999
1
165.74
Product4
167
167
Here are my 3 expected output.
Expected output 1 :
Id (Promo Id)
Count
167
5
175
2
177
0
179
0
Expected output 2 :
Id (Promo Id)
Price (Only Sum zdcx_ids if exist)
<--- RESULT FORMULA
167
$727
127.26 + 84.84 + 183.42 + 165.7 + 165.74
175
$212.1
127.26 + 84.84
177
$0
179
$0
Expected output 3 :
Id (Promo Id)
Price (If zdcx_ids exist Id, Sum je with same pid)
<--- RESULT FORMULA
167
$727
127.26 + 84.84 + 183.42 + 165.7 + 165.74
175
$727
127.26 + 84.84 + 183.42 + 165.7 + 165.74
177
$0
179
$0
My progress for solution 1:
SET #tDate := '2022-05-29';
SELECT COUNT(*) FROM (
SELECT COUNT(*) FROM qtlsdmx
WHERE FROM_UNIXTIME(qrrq) >= #tDate AND FROM_UNIXTIME(qrrq) < DATE_ADD(#tDate, INTERVAL 1 DAY)
and zdcx_ids like '%175%' group by pid
) a ;
My progress for solution 2:
select sum(je)
from qtlsdmx
where FROM_UNIXTIME(qrrq) >= #tDate and FROM_UNIXTIME(qrrq) < DATE_ADD(#tDate, INTERVAL 1 DAY)
and pid in (select pid from qtlsdmx where zdcx_ids like '%175%');
My progress for solution 3:
SET #tDate := '2022-05-29';
SELECT SUM(je) AS TOTALPRICE
FROM qtlsdmx
WHERE FROM_UNIXTIME(qrrq) >= #tDate AND FROM_UNIXTIME(qrrq) < DATE_ADD(#tDate, INTERVAL 1 DAY)
and pid in (select pid from qtlsdmx where zdcx_ids like '%175%');

Answer 1

Here's an idea using SUBSTRING_INDEX() and UNION ALL to make comma separated values generated by each of their own row. So, we are going to turn this row:
Id
pid
qrrq
sl
je
spmc
zdcx_ids
1
1
1653839999
3
127.26
Product1
175,167
to:
Id
pid
qrrq
sl
je
spmc
zdcx_ids
1
1
1653839999
3
127.26
Product1
175
1
1
1653839999
3
127.26
Product1
167
With this query, we'll get the first in comma separated value of zdcx_ids:
SELECT Id, pid, qrrq, sl, je, spmc,
SUBSTRING_INDEX(SUBSTRING_INDEX(zdcx_ids,',',1),',',-1)
FROM qtlsdmx;
The reason why we're doing SUBSTRING_INDEX() twice it to cater for the next value after the comma. There's another part we need to add in to the query, which is the count of comma separated value. I think I've seen a different method somewhere but the one on top of my head is subtraction of string length between the original value and the value where the comma is being removed and then add 1:
WHERE LENGTH(zdcx_ids)-LENGTH(REPLACE(zdcx_ids,',','')) +1 >= 1
The result of the length subtraction on zdcx_ids=175,167 will return 1 since there's only 1 comma so the count is not correct. Obviously 175,167 should have two value count which is why we add +1 at the end of the subtraction result. The >= 1 is just telling the query to return result if the count value is equal or more than the checked value. We'll add this into the query:
SELECT Id, pid, qrrq, sl, je, spmc,
SUBSTRING_INDEX(SUBSTRING_INDEX(zdcx_ids,',',1),',',-1)
FROM qtlsdmx
WHERE LENGTH(zdcx_ids)-LENGTH(REPLACE(zdcx_ids,',','')) +1 >= 1
Now we'll add the remaining original condition that you've placed and do UNION ALL with the same query structure but with different count or numbering sequence. There are two parts we need to change for the second query:
SET #tDate := '2022-05-29';
SELECT Id, pid, qrrq, sl, je, spmc,
SUBSTRING_INDEX(SUBSTRING_INDEX(zdcx_ids,',',1),',',-1)
FROM qtlsdmx
WHERE LENGTH(zdcx_ids)-LENGTH(REPLACE(zdcx_ids,',','')) +1 >= 1
/*adding the rest of your condition*/
AND FROM_UNIXTIME(qrrq) >= #tDate
AND FROM_UNIXTIME(qrrq) < DATE_ADD(#tDate, INTERVAL 1 DAY)
UNION ALL
SELECT Id, pid, qrrq, sl, je, spmc,
SUBSTRING_INDEX(SUBSTRING_INDEX(zdcx_ids,',',2),',',-1)
FROM qtlsdmx
WHERE LENGTH(zdcx_ids)-LENGTH(REPLACE(zdcx_ids,',','')) +1 >= 2
AND FROM_UNIXTIME(qrrq) >= #tDate
AND FROM_UNIXTIME(qrrq) < DATE_ADD(#tDate, INTERVAL 1 DAY)
Notice the changes in the second query at the first SUBSTRING_INDEX() and the value count. Depending on your data, you might have to repeat the same query for 10 times with incremental value on those two parts. You probably can consider using PREPARED STATEMENT for this. I'll get to that later.
Once the UNION ALL queries done, you can make it as subquery, LEFT JOIN and do the calculation:
For expected output 1:
SELECT t1.Id, COUNT(t2.Id)
FROM zdcxd AS t1
LEFT JOIN
(SELECT Id, pid, qrrq, sl, je, spmc,
SUBSTRING_INDEX(SUBSTRING_INDEX(zdcx_ids,',',1),',',-1) AS val /*assigning alias*/
FROM qtlsdmx
WHERE LENGTH(zdcx_ids)-LENGTH(REPLACE(zdcx_ids,',','')) +1 >= 1
UNION ALL
SELECT Id, pid, qrrq, sl, je, spmc,
SUBSTRING_INDEX(SUBSTRING_INDEX(zdcx_ids,',',2),',',-1)
FROM qtlsdmx
WHERE LENGTH(zdcx_ids)-LENGTH(REPLACE(zdcx_ids,',','')) +1 >= 2) AS t2
ON t1.Id=t2.val
GROUP BY t1.Id;
For expected output 2, you need to just change COUNT(t2.Id) to SUM(je):
SELECT t1.Id, SUM(je)
FROM zdcxd AS t1
LEFT JOIN
...
As for expected output 3, I'm still trying to understand how to make that work so I haven't got any suggestion on that yet. I'll update this post if I've figured it out somehow but hopefully you manage to figure it out yourself too.
Now, since this UNION ALL query depend on how many comma separated value, it will definitely be a long query. So I suggest using MySQL prepare statement to generate the query with some customized sequence numbering.
Here's a fiddle of the full prepare statement queries
Fiddle 1.
Fiddle 2 - in case the first fiddle is not working.