GroupBy clause removing all null column values - mysql

I have written the following query wherein I am usig groupby clause on server column
select s.server, MAX(s.ipAddress) as ipAddress,
MAX(r.stacks->>"$[0].name") as stackName,
MAX(a.aMessage) as aMessage
from environments e
inner join servers s
on e.objectId = s.environmentId
inner join resources r
on e.objectId = r.environmentId
inner join audits a
on a.id = (select max(a.id) from audits a where a.logObjId = s.cAudit)
WHERE dateSubmitted BETWEEN NOW() - INTERVAL 90 DAY AND NOW()
Group by s.server
ORDER BY dateSubmitted;
Howerver, server column may have NULL values with a valid ipAddress and stackName.
How to modify the query so that all NULL server column values are not missed out.
Expected Sample Data:
server ipAddress stackName aMessage
NULL NULL Stack A Searching for IP pool
NULL NULL Stack B Message XYZ
NULL NULL Stack A Message ABC

It seems the INNER JOIN used to JOIN the table makes NULL value to removed from the result. So just modified the query. Try this one and see if you are able to see all the data of Server table so that NULL data also will come for Server column.
select s.server, MAX(s.ipAddress) as ipAddress,
MAX(r.stacks->>"$[0].name") as stackName,
MAX(a.aMessage) as aMessage
from servers s
left join environments e
on e.objectId = s.environmentId
left join resources r
on e.objectId = r.environmentId
left join audits a
on a.id = (select max(a.id) from audits a where a.logObjId = s.cAudit)
WHERE dateSubmitted BETWEEN NOW() - INTERVAL 90 DAY AND NOW()
Group by s.server
ORDER BY dateSubmitted;

Related

Select statement can trigger dead lock on table in mysql?

The SQL below is inside a MySQL stored procedure.
The procedure run by a cron job every day once at midnight to populate report table with result.
this procedure take around 2 min to run.
please note that table1 has millions of records.
i put this to run at midnight because there are INSERT/UPDATE transactions during the day but unfortunately there are some few transaction at night also.
when this procedure runs and if there are other transactions running then a deadlock error on table1 occurs.
my question is
why SELECT statement cause deadlock on table1?
how can I avoid deadlock in this kind of situation?
DROP report;
CREATE TABLE IF NOT EXISTS report AS (
SELECT
DISTINCT
companies.id company_id,
(
SELECT
SUM(`message_count`) single_phone
FROM
`table1`
WHERE
`table1`.`company_id` = companies.id
AND
`status` != 'error'
) AS single_phone,
(
SELECT
SUM(`message_count`)
FROM
`table1`
WHERE
`table1`.`company_id` = companies.id
AND
`status` != 'not error'
) AS log,
(
SELECT
SUM(`message_count`)
FROM
`table1`
WHERE
`table1`.`company_id` = companies.id
AND
`status` != 'error'
) AS log_monthly,
(
SELECT
SUM(`number_of_sms`) AS aggregate
FROM
`messages`
WHERE
`messages`.`company_id` = companies.id
) AS p_monthly
FROM
companies
INNER JOIN company_users ON companies.id = company_users.company_id
WHERE
company_users.confirmed = 1
AND
company_users.deleted_at IS NULL
);
thanks you very much for help but i have found the problem. yes this procedure cause the deadlock on table but the actual cause of the issue is that i have put ->everyMinute() in my laravel Kernal for schedule run. and there is also a cron job configured by another developer for the same that run every minute. these will run schedule every minute and that is the real cause of the deadlock problem. i have change my Kernal schedule to ->dailyAt('02:00'); now the problem is solved.
Your field-level queries should be done ONCE in the from clause to get pre-aggregates done ONCE per company ID and left-joined in case a given company may NOT have qualified records in a given category. Additionally, your query to get Single_Phone is the same as your 'log_monthly', but have no criteria showing a
break or filter on the dates of activity to filter out a single month vs overall total of everything. So, I added a where clause for filtering, but only GUESSING if such some date exists.
This query might substantially improve your performance. By moving the COLUMN-based queries for every company ID into its own subquery via left-join, those will be summed() and grouped by company ONCE, then the JOIN for the final result. COALESCE() is used so if no such counts exists, the value returned will be 0 instead of null
DROP report;
CREATE TABLE IF NOT EXISTS report AS (
SELECT
c.id company_id,
coalesce( PhoneSum.Msgs, 0 ) as Single_Phone,
coalesce( PhoneLog.Msgs, 0 ) as Log,
coalesce( MonthLog.Msgs, 0 ) as Log_Monthly,
coalesce( SMSSummary.Aggregate, 0 ) as p_monthly
from
-- this will declare an in-line variable if you do need to filter by a month as a couple of your
-- column result names infer, but have no other indicator of filtering by a given month.
( select #yesterday := date_sub( date(curdate()), interval -1 day ),
#beginOfThatMonth := date_sub( #yesterday, interval dayOfMonth( #yesterday ) -1 day ) sqlvars,
companies c
INNER JOIN company_users cu
ON m.company.id = cu.company_id
AND cu.confirmed = 1
AND cu.deleted_at IS NULL
LEFT JOIN
( SELECT
t.company_id,
SUM( t.message_count ) Msgs
FROM
table1 t
INNER JOIN company_users cu
ON t.company.id = cu.company_id
AND cu.confirmed = 1
AND cu.deleted_at IS NULL
where
t.status != 'error'
GROUP BY
t.company_id ) AS PhoneSum,
on c.id = PhoneSum.company_id
LEFT JOIN
( SELECT
t.company_id,
SUM( t.message_count ) Msgs
FROM
table1 t
INNER JOIN company_users cu
ON t.company.id = cu.company_id
AND cu.confirmed = 1
AND cu.deleted_at IS NULL
where
t.status != 'not error'
GROUP BY
t.company_id ) AS PhoneLog,
on c.id = PhoneLog.company_id
LEFT JOIN
( SELECT
t.company_id,
SUM( t.message_count ) Msgs
FROM
table1 t
INNER JOIN company_users cu
ON t.company.id = cu.company_id
AND cu.confirmed = 1
AND cu.deleted_at IS NULL
where
t.status != 'error'
-- this would only get counts of activity for current month currently active
-- but since you are running at night, you need the day before current
AND t.SomeDateFieldOnTable1 >= #beginOfThatMonth
GROUP BY
t.company_id ) AS MonthLogMsgs,
on c.id = MonthLogMsgs.company_id
LEFT JOIN
( SELECT
m.company_id,
SUM( m.number_of_sms ) aggregate
FROM
messages m
INNER JOIN company_users cu
ON m.company.id = cu.company_id
AND cu.confirmed = 1
AND cu.deleted_at IS NULL
where
m.SomeDateFieldOnMessagesTable >= #beginOfThatMonth
GROUP BY
company_id ) AS SMSSummary,
on c.id = SMSSummary.company_id

Improving the performance of sql joined count query

In my application the users can create campaigns for sending messages. When the campaign tries to send a message, one of the three things can happen:
The message is suppressed and not let through
The message can't reach the recipient and is considered failed
The message is successfully delivered
To keep track of this, I have the following table:
My problem is that when the application has processed a lot of messages (more than 10 million), the query I use for showing campaign statistics for the user slows down by a considerable margin (~ 15 seconds), even when there are only a few (~ 10) campaigns being displayed for the user.
Here is the query I'm using:
select `campaigns`.*, (select count(*) from `processed_messages`
where `campaigns`.`id` = `processed_messages`.`campaign_id` and `status` = 'sent') as `messages_sent`,
(select count(*) from `processed_messages` where `campaigns`.`id` = `processed_messages`.`campaign_id` and `status` = 'failed') as `messages_failed`,
(select count(*) from `processed_messages` where `campaigns`.`id` = `processed_messages`.`campaign_id` and `status` = 'supressed') as `messages_supressed`
from `campaigns` where `user_id` = 1 and `campaigns`.`deleted_at` is null order by `updated_at` desc;
So my question is: how can I make this query run faster? I believe there should be some way of not having to use sub-queries multiple times but I am not very experienced with MySQL syntax yet.
You should write this as a single join, using conditional aggregation:
SELECT
c.*,
COUNT(CASE WHEN pm.status = 'sent' THEN 1 END) AS messages_sent,
COUNT(CASE WHEN pm.status = 'failed' THEN 1 END) AS messages_failed,
COUNT(CASE WHEN pm.status = 'suppressed' THEN 1 END) AS messages_suppressed
FROM campaigns c
LEFT JOIN processed_messages pm
ON c.id = pm.campaign_id
WHERE
c.user_id = 1 AND
c.deleted_at IS NULL
GROUP BY
c.id
ORDER BY
c.updated_at DESC;
It should be noted that at first glance, doing SELECT c.* appears to be a violation of the GROUP BY rules which say that only columns which appear in the GROUP BY clause can be selected. However, assuming that campaigns.id is the primary key column, then there is nothing wrong with selecting all columns from this table, provided that we aggregate by the primary key.
Edit:
If the above answer does not run on your MySQL server version, with an error message complaining about only full group by, then use this version:
SELECT c1.*, c2.messages_sent, c2.messages_failed, c2.message_suppressed
FROM campaigns c1
INNER JOIN
(
SELECT
c.id
COUNT(CASE WHEN pm.status = 'sent' THEN 1 END) AS messages_sent,
COUNT(CASE WHEN pm.status = 'failed' THEN 1 END) AS messages_failed,
COUNT(CASE WHEN pm.status = 'suppressed' THEN 1 END) AS messages_suppressed
FROM campaigns c
LEFT JOIN processed_messages pm
ON c.id = pm.campaign_id
WHERE
c.user_id = 1 AND
c.deleted_at IS NULL
GROUP BY
c.id
) c2
ON c1.id = c2.id
ORDER BY
c2.updated_at DESC;

mysql - sorting data from 2 tables by the value from the 3rd table

I would like to sort data from 2 different tables connected with UNION, sorting parameter has to come from from the 3rd table.(drivers > queno)
If I sort data from 1 table i use following code (works perfectly):
SELECT quotedb.*
FROM quotedb
LEFT JOIN drivers
ON quotedb.driver = drivers.id
ORDER BY IF(queno = ''
OR queno IS NULL, 1, 0)
So when I join 2 tables I tried with this with no succes...
(
SELECT DISTINCT driver
FROM quotedb
LEFT JOIN drivers
ON quotedb.driver=drivers.id)
UNION ALL
(
SELECT DISTINCT driver
FROM packaging
LEFT JOIN drivers
ON packaging.driver=drivers.id )
ORDER BY
order by IF(queno = ''
OR queno IS NULL,1,0)
What i need to do to make it work?. Thank you in advance.
You will need to fetch the queno column also from individual Select queries.
Try the following:
(
SELECT DISTINCT
qdb.driver AS driver,
d.queno AS queno
FROM quotedb AS qdb
LEFT JOIN drivers AS d ON qdb.driver = d.id
)
UNION ALL
(
SELECT DISTINCT
p.driver AS driver,
d.queno AS queno
FROM packaging AS p
LEFT JOIN drivers AS d ON p.driver = d.id
)
ORDER BY
(CASE WHEN queno = '' OR queno IS NULL THEN 1
ELSE 0
END)

Where clause inside joined select

I'm trying to accommodate a similar solution to this one - what I have is a SELECT query inside a JOIN, and the problem is that the query runs at full for all rows (I'm talking 60,000 rows per table - and it runs on 3 tables!).
So what I want to do, is add a WHERE clause to the SELECTs inside the JOIN.
But, I can't access the outer SELECT and get the proper WHERE condition I need.
The query I'm attempting is here:
SELECT c.compete_id AS id,
s.id AS store_id,
c.enabled AS enabled,
s.store_name AS store_name,
s.store_url AS store_url,
c.verified AS verified,
r.rating_total AS rating,
r.positive_percent AS percent,
r.type AS type
FROM compete_settings c
LEFT JOIN stores s
ON c.compete_id = s.id
LEFT JOIN (
(SELECT store_id, rating_total, positive_percent, 'ebay' AS type FROM ebay_sellers WHERE store_id = c.compete_id)
UNION
(SELECT store_id, rating_total, positive_percent, 'amazon' AS type FROM amazon_sellers WHERE store_id = c.compete_id)
UNION
(SELECT store_id, CASE WHEN rank = 0 THEN NULL ELSE (200000 - rank) END AS rating_total, '100' as positive_percent, 'alexa' AS type FROM alexa_ratings WHERE store_id = c.compete_id)
) AS r
ON c.compete_id = r.store_id
WHERE c.store_id = :store_id
Note, :store_id is a variable bound through the framework - let's imagine it's the number 12345.
How can I do this? Any ideas?
We ended up going witha different approach - we just JOINed everything and only selected the right columns with a CASE. Here's the final query:
SELECT c.id AS id,
s.id AS store_id,
c.enabled AS enabled,
s.store_name AS store_name,
s.store_url AS store_url,
c.verified AS verified,
(CASE WHEN eb.rating_total IS NOT NULL THEN eb.rating_total
WHEN am.rating_total IS NOT NULL THEN am.rating_total
WHEN ax.rank IS NOT NULL THEN ax.rank
END) AS rating,
(CASE WHEN eb.positive_percent IS NOT NULL THEN eb.positive_percent
WHEN am.positive_percent IS NOT NULL THEN am.positive_percent
WHEN ax.rank IS NOT NULL THEN '100'
END) AS percent,
(CASE WHEN eb.positive_percent IS NOT NULL THEN 'ebay'
WHEN am.positive_percent IS NOT NULL THEN 'amazon'
WHEN ax.rank IS NOT NULL THEN 'alexa'
END) AS type
FROM compete_settings c
LEFT JOIN stores s
ON c.compete_id = s.id
LEFT JOIN ebay_sellers eb ON c.compete_id = eb.store_id
LEFT JOIN amazon_sellers am ON c.compete_id = am.store_id
LEFT JOIN alexa_ratings ax ON c.compete_id = ax.store_id
WHERE c.store_id = :store_id

Outer Join in mYsql Qyery

I have folowing MYsql Query And Trying to right outer join but unable to understan how to do this
here is query plase any one help
select lp_des.lpname,today.cnt_veh_tdy,todate.cnt_veh_tdate
from
(select distinct registration.lpcode,loadingpoint.lpname
from registration,loadingpoint
where registration.lpcode=loadingpoint.lpcode) lp_des,
(select lpcode,count(vehicleno) cnt_veh_tdate
from registration
where registration.companycode='01'
group by lpcode) todate,
(
select lpcode,count(vehicleno) cnt_veh_tdy
from registration
where registration.companycode='01'
and registration.date=(select max(date) from registration)
group by lpcode) today
right outer join today on lp_des.lpcode = today.lpcode
right outer join todate on lp_des.lpcode = todate.lpcode
I want to make right outer join on this part
where lp_des.lpcode=todate.lpcode
and lp_des.lpcode=today.lpcode
Please help and Thanks in advance
You asked for this:
select
lp_des.lpname,
today.cnt_veh_tdy,
todate.cnt_veh_tdate
from
(select distinct
r.lpcode,
l.lpname
from
registration r
inner join loadingpoint l on l.lpcode = r.lpcode) lp_des
right join
(select
r.lpcode,
count(r.vehicleno) cnt_veh_tdate
from
registration r
where
r.companycode='01'
group by
lpcode) todate on todate.lpcode = lp_des.lpcode
right join
(select
r.lpcode,
count(r.vehicleno) cnt_veh_tdy
from
registration r
where
r.companycode = '01'
and registration.date = (select max(date) from registration)
group by
r.lpcode) today on today.lpcode = lp_des.lpcode
But I think you mean this:
select
r.lpcode,
l.lpname,
count(r.vehicleno) cnt_veh_tdate,
count(case when r.date = md.date then r.vehicleno else null end) cnt_veh_tdy
from
registration r
inner join (select max(rm.date) maxdate from registration rm) md
left join loadingpoint l on l.lpcode = r.lpcode
where
r.companycode = '01'
group by
r.lpcode
and maybe even this:
select
r.lpcode,
l.lpname,
count(r.vehicleno) cnt_veh_tdate,
count(case when r.date = date() then r.vehicleno else null end) cnt_veh_tdy
from
registration r
left join loadingpoint l on l.lpcode = r.lpcode
where
r.companycode = '01'
group by
r.lpcode
If I read it correctly, you want a query that returns the number of vehicles for company 1 assigned to a loading point, overall as well as for today only. And you also want that count for vehicles that do not have loading point assigned yet.
Though it would help if you would have added this description. It will help the ones answering your question, but it will also help you writing the right query in the first place.
The syntax for a right outer join is:
SELECT t1.id, t2.id FROM t1 RIGHT OUTER JOIN t2 ON t1.field1 = t2.field2
If you're joining on the same field you can use USING instead of ON:
SELECT t1.id, t2.id FROM t1 RIGHT OUTER JOIN t2 USING (field)