I am working on a MySQL query that needs to go through 1+ million rows in table A and 5+ million in table B. The query selects all unique people from table A and inner joins with the sales for each person on table B. Index is set on both tables where necessary.
The goal here is to add all unique Email addresses from table A with data from table B into table C.
I am looking for the most optimized way to do this. Included is the select part of a REPLACE INTO query being used that will pull records in the specified id's. It is a simple CASE statement with an IN condition.
I also have the exact same query using NOT IN vs IN for the sub query. That one times out.
Looking forward to any help anyone can provide and hopefully a more optimal way of doing this.
SELECT
c.Email,
MAX(c.Birthdate)
UPPER(c.Deleted) AS Deleted,
UPPER(c.Inactive) AS Inactive,
UPPER(c.SendEmail) AS SendEmail,
CONCAT('[',GROUP_CONCAT(DISTINCT c.Site SEPARATOR ']['),']') AS SITEID,
CONCAT('[',GROUP_CONCAT(DISTINCT c.Studio SEPARATOR ']['),']') AS STUDIO,
( SELECT CONCAT('[',GROUP_CONCAT(DISTINCT s2.Service SEPARATOR ']['),']')
FROM my_example c2
INNER JOIN my_example_sales s2 ON s2.idMember = c2.idMember AND s2.Site = c2.Site WHERE c2.Email = c.Email
) as SERVICES,
( SELECT MAX(date(s1.Date)) as Date
FROM my_example_sales s1
WHERE s1.idMember = c.idMember
AND s1.Site = c.Site
AND (CASE WHEN s1.Site = '1' THEN s1.ProductID IN ('1','6','7','12','18','22')
WHEN s1.Site = '2' THEN s1.ProductID = '156'
WHEN s1.Site = '3' THEN s1.ProductID IN ('3','5','6')
WHEN s1.Site = '4' THEN s1.ProductID IN ('11','15')
WHEN s1.Site = '5' THEN s1.ProductID = '23'
WHEN s1.Site = '6' THEN s1.ProductID = '23'
WHEN s1.Site = '7' THEN s1.ProductID = '23'
WHEN s1.Site = '8' THEN s1.ProductID = '23'
WHEN s1.Site = '9' THEN s1.ProductID = '23'
WHEN s1.Site = '10' THEN s1.ProductID IN ('7','11','17','30','31')
WHEN s1.Site = '11' THEN s1.ProductID = '23'
WHEN s1.Site = '12' THEN s1.ProductID IN ('7','11','17','30','31')
WHEN s1.Site = '13' THEN s1.ProductID = '23' END)
WHERE 1
ORDER BY s1.Date DESC
LIMIT 0,1
) as lastPurchaseFreeWeek,
NOW() as dateModified
FROM my_example c
WHERE c.Email !=''
GROUP BY c.Email
ORDER BY c.ModDate DESC
Moving your subselects to joins should yield some performance increase:
SELECT
c.Email,
MAX(c.Birthdate)
UPPER(c.Deleted) AS Deleted,
UPPER(c.Inactive) AS Inactive,
UPPER(c.SendEmail) AS SendEmail,
CONCAT('[',GROUP_CONCAT(DISTINCT c.Site SEPARATOR ']['),']') AS SITEID,
CONCAT('[',GROUP_CONCAT(DISTINCT c.Studio SEPARATOR ']['),']') AS STUDIO,
c2.SERVICES,
s1.`Date` as lastPurchaseFreeWeek,
NOW() as dateModified
FROM my_example c
INNER JOIN (
SELECT CONCAT('[',GROUP_CONCAT(DISTINCT s2.Service SEPARATOR ']['),']') AS SERVICES
FROM my_example c2
INNER JOIN my_example_sales s2 ON s2.idMember = c2.idMember AND s2.Site = c2.Site
) c2 ON c2.Email = c.Email
INNER JOIN (
SELECT MAX(date(s1.Date)) as `Date`
FROM my_example_sales s1
WHERE (CASE WHEN s1.Site = '1' THEN s1.ProductID IN ('1','6','7','12','18','22')
WHEN s1.Site = '2' THEN s1.ProductID = '156'
WHEN s1.Site = '3' THEN s1.ProductID IN ('3','5','6')
WHEN s1.Site = '4' THEN s1.ProductID IN ('11','15')
WHEN s1.Site = '5' THEN s1.ProductID = '23'
WHEN s1.Site = '6' THEN s1.ProductID = '23'
WHEN s1.Site = '7' THEN s1.ProductID = '23'
WHEN s1.Site = '8' THEN s1.ProductID = '23'
WHEN s1.Site = '9' THEN s1.ProductID = '23'
WHEN s1.Site = '10' THEN s1.ProductID IN ('7','11','17','30','31')
WHEN s1.Site = '11' THEN s1.ProductID = '23'
WHEN s1.Site = '12' THEN s1.ProductID IN ('7','11','17','30','31')
WHEN s1.Site = '13' THEN s1.ProductID = '23' END)
) s1 ON ( s1.idMember = c.idMember AND s1.Site = c.Site )
WHERE c.Email !=''
GROUP BY c.Email
ORDER BY c.ModDate DESC
Related
How to make one row in different rows and column
SELECT
internal_id, store_id, user_id, shift_info,
DATE(log_datetime) dates,
(case when log_action = '0' then TIME(log_datetime) end) as time_in,
(case when log_action = '0' then CONCAT(log_lat, ",", log_lng) end) as loc_in,
(case when log_action = '1' then TIME(log_datetime) end) as time_out,
(case when log_action = '1' then CONCAT(log_lat, ",", log_lng) end) as loc_out
FROM
attendance_store_user
WHERE
user_id = "A4CBD64F-D21C-5612-CCF5-497892B62E76"
i want result like this :
You could try using a join of the same table filter for null time_out and time_in
select a.dates, a.store_id, a,time_in, b.time_out
FROM attendance_store_user a
INNER JOIN attendance_store_user b on a.dates = b.dates
and a.user_id = b.user_id
and a.time_out is null
and b.time_in is null
WHERE a.user_id = "A4CBD64F-D21C-5612-CCF5-497892B62E76"
Got this 3 in 1 query:
SELECT * FROM
(
SELECT mesures.date j, AVG(mesures.valeur) maxi
FROM mesures
JOIN plages_horaire ON mesures.id_plage = plages_horaire.id_plage
WHERE MONTH(mesures.date) = '9' AND YEAR(mesures.date) = '2016' AND mesures.code_station = 'P02SE' AND mesures.id_crit = '1' AND mesures.id_type = '1'
GROUP BY mesures.date
) maxi
,
(
SELECT AVG(mesures.valeur) mini
FROM mesures
JOIN plages_horaire ON mesures.id_plage = plages_horaire.id_plage
WHERE MONTH(mesures.date) = '9' AND YEAR(mesures.date) = '2016' AND mesures.code_station = 'P02SE' AND mesures.id_crit = '1' AND mesures.id_type = '2'
GROUP BY mesures.date
) mini
,
(
SELECT AVG(mesures.valeur) moy
FROM mesures
JOIN plages_horaire ON mesures.id_plage = plages_horaire.id_plage
WHERE MONTH(mesures.date) = '9' AND YEAR(mesures.date) = '2016' AND mesures.code_station = 'P02SE' AND mesures.id_crit = '1' AND mesures.id_type = '3'
GROUP BY mesures.date
) moy
GROUP BY j
Problem is that I get what I want excepting values of the 2 last columns are the same at every rows:
query output
I believe it's because of the GROUP BY.
From what I can see from your query, you don't need the plage_horaire table. You can also simplify the logic greatly by using conditional aggregation:
SELECT m.date,
AVG(CASE WHEN m.id_type = 1 THEN m.valeur END) maxi,
AVG(CASE WHEN m.id_type = 2 THEN m.valeur END) mini,
AVG(CASE WHEN m.id_type = 3 THEN m.valeur END) maxmoy,
FROM mesures m
WHERE MONTH(m.date) = 9 AND YEAR(m.date) = 2016 AND
m.code_station = 'P02SE' AND m.id_crit = 1 AND m.id_type IN (1, 2, 3)
GROUP BY m.date ;
Notice that I also removed the quotes from the numeric constants. MONTH() and YEAR() return numbers, so quotes are not appropriate. I am guessing that the ids are numeric as well.
I need to group the sum by different columns:
SUM1 group by Banco_Recarga
SUM2 group by BancoRetiro
SELECT Banco_Recarga, BancoRetiro, bancos_recargas.banco,
SUM(CASE WHEN Tipo='1' AND (Status = '1' OR Status = '2') THEN '1' ELSE 0 END) AS SUM1,
SUM(CASE WHEN Tipo='2' AND (Status = '1' OR Status = '2') THEN '1' ELSE 0 END) AS SUM2
FROM transaccionesrr
INNER JOIN bancos_recargas ON (transaccionesrr.Banco_Recarga = bancos_recargas.id)
WHERE Fecha_Recarga BETWEEN '2017-11-01' AND '2017-11-10' GROUP BY Banco_Recarga ORDER BY SUM1;
I have been looking for hours but i can't find a solution
Try this:
SELECT
CASE
WHEN Tipo = '1' AND (Status = '1' OR Status = '2') THEN Banco_Recarga
WHEN Tipo = '2' AND (Status = '1' OR Status = '2') THEN BancoRetiro
ELSE 'Unexpected'
END AS banco
, SUM(CASE WHEN Tipo = '1' AND (Status = '1' OR Status = '2') THEN '1' ELSE 0 END) AS sum1
, SUM(CASE WHEN Tipo = '2' AND (Status = '1' OR Status = '2') THEN '1' ELSE 0 END) AS sum2
FROM transaccionesrr
INNER JOIN bancos_recargas ON (transaccionesrr.Banco_Recarga = bancos_recargas.id)
WHERE Fecha_Recarga BETWEEN '2017-11-01' AND '2017-11-10'
GROUP BY
CASE
WHEN Tipo = '1' AND (Status = '1' OR Status = '2') THEN Banco_Recarga
WHEN Tipo = '2' AND (Status = '1' OR Status = '2') THEN BancoRetiro
ELSE 'Unexpected'
END
ORDER BY sum1, sum2
You may need to add conditions into the where clause.
So I need to Update table scores and use the updated value of column won to update the second table tbl_users. So far the code updates scores, but uses the old value of won for the second table update:
UPDATE scores a
left join tbl_users b on
a.uid = b.userID
SET a.won = CASE
WHEN a.nright = '0' THEN '0'
WHEN a.nright = '1' THEN '25'
WHEN a.nright = '2' THEN '50'
WHEN a.nright = '3' THEN '100'
WHEN a.nright = '4' THEN '200'
WHEN a.nright = '5' THEN '400'
WHEN a.nright = '6' THEN '700'
WHEN a.nright = '7' THEN '1000'
END,
b.pts=b.pts+a.won,
b.pts_total=b.pts_total+a.won
WHERE a.uid=$user AND b.userID=$user
What you want to do is explicitly documented as correct:
The second assignment in the following statement sets col2 to the
current (updated) col1 value, not the original col1 value. The result
is that col1 and col2 have the same value. This behavior differs from
standard SQL.
UPDATE t1 SET col1 = col1 + 1, col2 = col1;
I assume that the issue is the multi-table update, where the set pulls the value from the earlier table.
You may be able to fix this using variables. I am not 100% sure, but the following is worth a try:
UPDATE scores s JOIN
tbl_users u
ON s.uid = .uuserID
SET s.won = (#w := (CASE WHEN s.nright = '0' THEN '0'
WHEN s.nright = '1' THEN '25'
WHEN s.nright = '2' THEN '50'
WHEN s.nright = '3' THEN '100'
WHEN s.nright = '4' THEN '200'
WHEN s.nright = '5' THEN '400'
WHEN s.nright = '6' THEN '700'
WHEN s.nright = '7' THEN '1000'
END)
),
u.pts = u.pts + #w,
u.pts_total = u.pts_total + #w
WHERE s.uid = $user ;
The documentation strongly suggests that the set clauses are processed in order for a single table. Alas, it is not clear whether this is always true for multiple tables.
If not, you can use two updates.
I want to sump up two or more counted values [pivoting]
My query is as following:
SELECT
client_name.client_name ,
escort ,
count(case mission_status_reason when '4' then mission_status_reason end) as awaiting_upload_at_origin,
count(case mission_status_reason when '6' then mission_status_reason end) as awaiting_military_escort,
count(case mission_status_reason when '3' then mission_status_reason end) as enrouted_to_destination,
count(case mission_status_reason when '9' then mission_status_reason end) as awaiting_download,
( awaiting_military_escort +enrouted_to_destination+ awaiting_download ) as TOTAL
FROM usc_tmr
LEFT JOIN client_name on client_name.id = usc_tmr.client_name
WHERE escort='usg'
and mission_status_ops IN ('1','4','5','6')
GROUP BY client_name
My problem is calculating the TOTAL. Can you guys please help.
SELECT client_name.client_name,
escort,
SUM(mission_status_reason = 4) AS awaiting_upload_at_origin,
SUM(mission_status_reason = 6) AS awaiting_military_escort,
SUM(mission_status_reason = 3) AS enrouted_to_destination,
SUM(mission_status_reason = 9) AS awaiting_download,
SUM(mission_status_reason IN (3,6,9)) AS TOTAL
FROM usc_tmr LEFT JOIN client_name ON client_name.id = usc_tmr.client_name
WHERE escort = 'usg' AND mission_status_ops IN (1,4,5,6)
GROUP BY client_name