mysql: please help me get rid of nested clauses - mysql

I have written the following query (the repeated parts put together from string constants) that tries to do the following in a shared calendar app.
search status_relation for all users with status not equal to 0 at current date+time
search default_status_relation for all users with status not equal to 0 at current weekday+time. This gives default weekly statuses when nothing is available in status_relation.
full outer join the two
then join on usernames from userid
this then gets processed by php which will display either the status (or default status, if status doesn't exist) of all users on that day who have status!=0.
the php needs to know whether it was returned a status or default status
Currently the query works (with a UNION to simulate an outer join). However, I would like to optimize it - I understand removing the subqueries might help. How can that be done?
SELECT q.*,
users.username,
users.userid AS uid
FROM (SELECT *
FROM ((SELECT sr.status AS srstatus,
dsr.status AS dsrstatus,
sr.userid AS sruserid,
dsr.userid AS dsruserid
FROM (SELECT *
FROM status_relation
WHERE DATE = '2012-03-19'
AND TIME = '0'
) sr
LEFT JOIN
(SELECT *
FROM default_status_relation
WHERE weekday = '0'
AND TIME = '0')
) dsr
ON sr.userid = dsr.userid)
UNION
(SELECT sr.status AS srstatus,
dsr.status AS dsrstatus,
sr.userid AS sruserid,
dsr.userid AS dsruserid
FROM (SELECT *
FROM status_relation
WHERE DATE = '2012-03-19'
AND TIME = '0') sr
RIGHT JOIN (SELECT *
FROM default_status_relation
WHERE weekday = '0'
AND TIME = '0') dsr
ON sr.userid = dsr.userid)
) myjoin
WHERE ( ( sruserid IS NOT NULL
AND srstatus != '0' )
OR ( sruserid IS NULL
AND dsrstatus != '0' ) )) q
LEFT JOIN users
ON ( q.sruserid = users.userid
OR q.dsruserid = users.userid )

You should be able to replace this part :
(SELECT sr.status AS srstatus,
dsr.status AS dsrstatus,
sr.userid AS sruserid,
dsr.userid AS dsruserid
FROM (SELECT *
FROM status_relation
WHERE DATE = '2012-03-19'
AND TIME = '0'
) sr
LEFT JOIN
(SELECT *
FROM default_status_relation
WHERE weekday = '0'
AND TIME = '0')
) dsr
ON sr.userid = dsr.userid)
by
SELECT sr.status AS srstatus,
dsr.status AS dsrstatus,
sr.userid AS sruserid,
dsr.userid AS dsruserid
FROM status_relation sr
left join default_status_relation dsr
ON sr.userid = dsr.userid and
sr.date = '2012-03-19' and
sr.time = TIME = '0' and
dsr.weekday = 0 and
dsr.time = 0;
You can do the same on the UNION

Related

Subqueries in the same table in mysql

I have a raw SQL query that I'm using in Laravel framework to get last presence of a user in a convocation table.
Basically, the master query is to get all the convocations of a given user that belongs to a specific ministry.
Like:
Select
`conv_pessoas.`id`,
`conv_pessoas`.`pessoa_id`,
`conv_pessoas`.`created_at`,
`conv_pessoas`.`data_chegada`,
`conv_pessoas`.`data_saida`,
`pessoas`.`nome`,
`ministerios`.`nome` as `ministerio_nome`,
`eventos`.`nome` as `evento_nome`,
`evento_data`.`data_inicial` as `data_evento_inicial`,
`evento_data`.`data_final` as `data_evento_final`,
(
SELECT
max(`convocacao_ministerio_pessoas`.`data_chegada`)
FROM
`convocacao_ministerio_pessoas`
WHERE
`convocacao_ministerio_pessoas`.`pessoa_id` = `conv_pessoas`.`pessoa_id`
AND `convocacao_ministerio_pessoas`.`data_chegada` IS NOT NULL
) as ultima_presenca_geral,
(
SELECT
max(`convocacao_ministerio_pessoas`.`data_chegada`)
FROM
`convocacao_ministerio_pessoas`
WHERE
`convocacao_ministerio_pessoas`.`pessoa_id` = `conv_pessoas`.`pessoa_id`
AND `convocacao_ministerio_pessoas`.`ministerio_id` = `conv_pessoas`.`ministerio_id`
AND `convocacao_ministerio_pessoas`.`data_chegada` IS NOT NULL
) as ultima_presenca_no_ministerio,
(
SELECT
count(*)
FROM
`convocacao_ministerio_pessoas`
WHERE
`convocacao_ministerio_pessoas`.`pessoa_id` = `conv_pessoas`.`pessoa_id`
AND `convocacao_ministerio_pessoas`.`ministerio_id` = `conv_pessoas`.`ministerio_id`
) as qtd_convocacoes_no_ministerio,
(
SELECT
count(*)
FROM
`convocacao_ministerio_pessoas`
WHERE
`convocacao_ministerio_pessoas`.`pessoa_id` = `conv_pessoas`.`pessoa_id`
AND `convocacao_ministerio_pessoas`.`ministerio_id` = `conv_pessoas`.`ministerio_id`
AND `convocacao_ministerio_pessoas`.`data_chegada` IS NOT NULL
) as qtd_presenca_no_ministerio
from
`convocacao_ministerio_pessoas` as `conv_pessoas`
inner join `pessoas` on `conv_pessoas`.`pessoa_id` = `pessoas`.`id`
inner join `convocacaos` on `conv_pessoas`.`convocacao_id` = `convocacaos`.`id`
inner join `ministerios` on `conv_pessoas`.`ministerio_id` = `ministerios`.`id`
inner join `eventos` on `convocacaos`.`evento_id` = `eventos`.`id`
inner join `evento_data` on `convocacaos`.`data_evento_id` = `evento_data`.`id`
where
`conv_pessoas`.`ministerio_id` = '2'
and `convocacaos`.`church_id` = 1
and `evento_data`.`data_final` between '2022-02-19 00:00:00'
and '2022-02-22 23:59:59'
But I need to get the last presence of this user in this same convocation table, but It's not necessary to be in the same ministry.
I've created subqueries for that.
But I'm not sure if it's the best approach.
I need some help to ensure I'm using the bests practices. ( I'm not advanced in SQL ).
**notes:**
Translations:
ministerio = ministry
convocacao = convocation
pessoa = person (user)
evento = event
data chegada = arrival date

Combining MySQL queries, with one being an exclusion query

First question on here, so I apologise if I have missed something previously asked, or don't format this well....
My company has a custom CRM + database that I am attempting to improve. We need to find a list of properties that will not have their yearly service renewed. Currently, we do this by first finding properties that will have their service renewed, which is done with the following query:
SELECT DISTINCT
j.`property_id`
FROM
`jobs` AS j
LEFT JOIN `property` AS p
ON j.`property_id` = p.`property_id`
LEFT JOIN `agency` AS a
ON p.`agency_id` = a.`agency_id`
INNER JOIN `property_services` AS ps
ON (
j.`property_id` = ps.`property_id`
AND j.`service` = ps.`alarm_job_type_id`
)
WHERE ps.`service` = 1
AND a.`country_id` = 1
AND (
j.`status` = 'Pending'
OR j.`date` IS NULL
OR j.`date` = '0000-00-00'
OR j.`job_type` = 'Once-off'
OR j.`job_type` = '240v Rebook'
OR (
j.`date` >= '2019-04-22'
AND j.`job_type` = 'Yearly Maintenance'
)
)
Then we find the details we want to display for the user, excluding other items in the process:
SELECT DISTINCT
j.`property_id`,
p.`address_1` AS p_address1,
p.`address_2` AS p_address2,
p.`address_3` AS p_address3,
p.`state` AS p_state,
p.`postcode` AS p_postcode,
a.`agency_id`,
a.`agency_name`
FROM
`jobs` AS j
LEFT JOIN `property` AS p
ON j.`property_id` = p.`property_id`
LEFT JOIN `agency` AS a
ON p.`agency_id` = a.`agency_id`
INNER JOIN `property_services` AS ps
ON (
j.`property_id` = ps.`property_id`
AND j.`service` = ps.`alarm_job_type_id`
)
WHERE p.`property_id` NOT IN (INSERT HERE THE IDS YOU GOT FROM THE FIRST QUERY)
AND ps.`service` = 1
AND p.`deleted` = 0
AND p.`agency_deleted` = 0
AND a.`status` = 'active'
AND a.`country_id` = 1
ORDER BY j.`property_id` DESC
LIMIT 0, 50
Ideally, I'd like to combine these queries, or somehow optimise them, as the page currently takes 2+ minutes to load, even with indexing.
Again, my apologies, I am not a database or query expert, pretty sure the degree only included one or two subjects on the matter!
For the record (in case someone would find it helpful), the combined version is:
SELECT DISTINCT
j.`property_id`,
p.`address_1` AS p_address1,
p.`address_2` AS p_address2,
p.`address_3` AS p_address3,
p.`state` AS p_state,
p.`postcode` AS p_postcode,
a.`agency_id`,
a.`agency_name`
FROM
`jobs` AS j
LEFT JOIN `property` AS p
ON j.`property_id` = p.`property_id`
LEFT JOIN `agency` AS a
ON p.`agency_id` = a.`agency_id`
INNER JOIN `property_services` AS ps
ON (
j.`property_id` = ps.`property_id`
AND j.`service` = ps.`alarm_job_type_id`
)
WHERE p.`property_id` NOT IN
(SELECT DISTINCT
j.`property_id`
FROM
`jobs` AS j
LEFT JOIN `property` AS p
ON j.`property_id` = p.`property_id`
LEFT JOIN `agency` AS a
ON p.`agency_id` = a.`agency_id`
INNER JOIN `property_services` AS ps
ON (
j.`property_id` = ps.`property_id`
AND j.`service` = ps.`alarm_job_type_id`
)
WHERE ps.`service` = 1
AND a.`country_id` = 1
AND (
j.`status` = 'Pending'
OR j.`date` IS NULL
OR j.`date` = '0000-00-00'
OR j.`job_type` = 'Once-off'
OR j.`job_type` = '240v Rebook'
OR (
j.`date` >= '2019-05-08'
AND j.`job_type` = 'Yearly Maintenance'
)
))
AND ps.`service` = 1
AND p.`deleted` = 0
AND p.`agency_deleted` = 0
AND a.`status` = 'active'
AND a.`country_id` = 1
AND (
j.`status` != 'Booked'
AND j.`status` != 'To Be Booked'
AND j.`status` != 'Send Letters'
AND j.`status` != 'On Hold'
AND j.`status` != 'On Hold - COVID'
AND j.`status` != 'Pre Completion'
AND j.`status` != 'Merged Certificates'
)
AND j.`date` > DATE_ADD(NOW(), INTERVAL - 350 DAY)
ORDER BY j.`property_id` DESC
Someone outside of StackOverflow helped combine, but it didn't help...
We ended up having to add a marker and search for that, because this query took 193 seconds to run on our database.
Also, I totally get the requirement to provide a minimum reproducible example, and its my fault for not doing that.

MySQL query taking too much time

query taking 1 minute to fetch results
SELECT
`jp`.`id`,
`jp`.`title` AS game_title,
`jp`.`game_type`,
`jp`.`state_abb` AS game_state,
`jp`.`location` AS game_city,
`jp`.`zipcode` AS game_zipcode,
`jp`.`modified_on`,
`jp`.`posted_on`,
`jp`.`game_referal_amount`,
`jp`.`games_referal_amount_type`,
`jp`.`status`,
`jp`.`is_flaged`,
`u`.`id` AS employer_id,
`u`.`email` AS employer_email,
`u`.`name` AS employer_name,
`jf`.`name` AS game_function,
`jp`.`game_freeze_status`,
`jp`.`game_statistics`,
`jp`.`ats_value`,
`jp`.`integration_id`,
`u`.`account_manager_id`,
`jp`.`model_game`,
`jp`.`group_id`,
(CASE
WHEN jp.group_id != '0' THEN gm.group_name
ELSE 'NA'
END) AS group_name,
`jp`.`priority_game`,
(CASE
WHEN jp.country != 'US' THEN jp.country_name
ELSE ''
END) AS game_country,
IFNULL((CASE
WHEN
`jp`.`account_manager_id` IS NULL
OR `jp`.`account_manager_id` = 0
THEN
(SELECT
(CASE
WHEN
account_manager_id IS NULL
OR account_manager_id = 0
THEN
`u`.`account_manager_id`
ELSE account_manager_id
END) AS account_manager_id
FROM
user_user
WHERE
id = (SELECT
user_id
FROM
game_user_assigned
WHERE
game_id = `jp`.`id`
LIMIT 1))
ELSE `jp`.`account_manager_id`
END),
`u`.`account_manager_id`) AS acc,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '1'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS somewhatgame,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '2'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS verygamecommitted,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '3'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS notgame,
(SELECT
COUNT(joa.id) AS applicationcount
FROM
game_refer_to_member jrmm
INNER JOIN
game_refer jrr ON jrr.id = jrmm.rid
INNER JOIN
game_applied joa ON jrmm.id = joa.referred_by
WHERE
jrmm.STATUS = '1'
AND jrr.referby_user_id IN (SELECT
ab_testing_user_id
FROM
ab_testing)
AND joa.game_post_id = rpr.recach_limit_game_id
AND (rpr.recach_limit = 1
OR rpr.recach_limit = 2)) AS gamecount
FROM
(`game_post` AS jp)
JOIN
`user_info` AS u ON `jp`.`user_user_id` = `u`.`id`
JOIN
`game_functional` jf ON `jp`.`game_functional_id` = `jf`.`id`
LEFT JOIN
`group_musesm` gm ON `gm`.`group_id` = `jp`.`group_id`
LEFT JOIN
`recach_limit` rpr ON `jp`.`id` = `rpr`.`recach_limit_game_id`
WHERE
`jp`.`status` != '3'
GROUP BY `jp`.`id`
ORDER BY `posted_on` DESC
LIMIT 10
I would first suggest not nesting select statements because this will cause an n^x performance hit on every xth level and I see at least 3 levels of selects inside this query.
Add index
INDEX(status, posted_on)
Move LIMIT inside
Then, instead of saying
FROM (`game_post` AS jp)
say
FROM ( SELECT id FROM game_post
WHERE status != 3
ORDER BY posted_on DESC
LIMIT 10 ) AS ids
JOIN game_post AS jp USING(id)
(I am assuming that the PK of jp is (id)?)
That should efficiently use the new index to get the 10 ids needed. Then it will reach back into game_post to get the other columns.
LEFT
Also, don't say LEFT unless you need it. It costs something to generate NULLs that you may not be needing.
Is GROUP BY necessary?
If you remove the GROUP BY, does it show dup ids? The above changes may have eliminated the need.
IN(SELECT) may optimize poorly
Change
AND jrr.referby_user_id IN ( SELECT ab_testing_user_id
FROM ab_testing )
to
AND EXISTS ( SELECT * FROM ab_testing
WHERE ab_testing_user_id = jrr.referby_user_id )
(This change may or may not help, depending on the version you are running.)
More
Please provide EXPLAIN SELECT if you need further assistance.

Adding a subquery to a join

I've been using the following join, to pull rows of users whom have volunteered for various project positions.
SELECT p.id, up.position_id, title, max_vol, current_datetime, IF(up.id IS NULL, "0", "1") volunteered
FROM positions AS p
LEFT JOIN users_positions AS up
ON p.id = up.position_id
AND up.user_id = 1
AND up.calendar_date = '2016-10-03'
WHERE
p.project_id = 1
AND p.day = 1
...but in a change of functionality, I have to now account for the date of the latest edit to a project. In another query, I solved it like so
SELECT *
FROM positions
WHERE
current_datetime = (SELECT MAX(current_datetime)
FROM positions
WHERE
project_id = 1 AND day = 1)
Which works fine, but now I have to also incorporate the return of rows which match the latest datetime in the left join query.
I just can't seem to wrap my head around it. Any suggestions? Thanks.
Use a sub query, like this:
SELECT
p.id,
up.position_id,
title,
max_vol,
current_datetime,
IF(up.id IS NULL,
"0",
"1") volunteered
FROM
( SELECT
*
FROM
positions
WHERE
current_datetime = (
SELECT
MAX(current_datetime)
FROM
positions
WHERE
project_id = 1
AND day = 1
)
) AS p
LEFT JOIN
users_positions AS up
ON p.id = up.position_id
AND up.user_id = 1
AND up.calendar_date = '2016-10-03'
WHERE
p.project_id = 1
AND p.day = 1

Order by clause not behaving correctly after several joins

Here is my query:
SELECT DISTINCT `post_data`. * , pv.`seller_id` , pv.`islimited` , pv.`isquantity` , pv.`isslider`, `price`.`original_price` , `price`.`discount_percentage` , `timelimit`.`start_date` , `timelimit`.`expire_date` , `quantity`.`in_stock`, `currency`.`currency_symbol`, `seller`.`directory`, `post_to_cat`.`cat_id`, count(`sales`.`sales_id`) as sale FROM `post_view` AS pv
INNER JOIN `post_data` ON pv.`post_id` = `post_data`.`post_id` AND pv.`status` = 1
INNER JOIN `price` ON pv.`post_id` = `price`.`post_id`
INNER JOIN `currency` ON `price`.`currency_id` = `currency`.`currency_id`
INNER JOIN `seller` ON pv.`seller_id` = `seller`.`seller_id`
INNER JOIN `post_to_cat` ON `post_to_cat`.`cat_id` = 1 AND `post_to_cat`.`post_id` = `post_data`.`post_id`
LEFT JOIN `timelimit` ON ( CASE WHEN pv.`islimited` = 1 THEN `timelimit`.`post_id` ELSE -1 END ) = pv.`post_id`
LEFT JOIN `quantity` ON ( CASE WHEN pv.`isquantity` = 1 THEN `quantity`.`post_id` ELSE -1 END ) = pv.`post_id`
LEFT JOIN `sales` ON `sales`.`post_id` = pv.`post_id` AND `sales`.`status` = 1
WHERE pv.`status` = 1
ORDER BY pv.`post_id` DESC LIMIT 1
The ORDER BY DESC is not working, it just returns the first row from the table, but I want to get the highest post_id value row. What is the mistake I am making?
AS #Alex said in the comments you've got a LIMIT 1 at the end, you should probably bracket the last LEFT JOIN also for readability.
As #McAdam331 said we need data sample and sql fiddle to investigate what is wrong with you query. But at the moment I have some suggestions how to improve and debug your query.
First off all, what do I see the main and very left table in your query is post_view so all other tables should be LEFT JOIN if you want to get the max id. You should use INNER JOIN only if you think that other table could filter your main table somehow and order or result could be other table dependend. But in your case I see no reason to use INNER JOIN.
Second point is your very weird ON conditions:
LEFT JOIN `timelimit` ON ( CASE WHEN pv.`islimited` = 1 THEN `timelimit`.`post_id` ELSE -1 END ) = pv.`post_id`
LEFT JOIN `quantity` ON ( CASE WHEN pv.`isquantity` = 1 THEN `quantity`.`post_id` ELSE -1 END ) = pv.`post_id`
I have converted them into another one
CASE WHEN pv.`islimited`=1 THEN `timelimit`.`start_date` ELSE NULL END as start_date ,
CASE WHEN pv.`islimited`=1 THEN `timelimit`.`expire_date` ELSE NULL END as expire_date,
CASE WHEN pv.`isquantity`=1 THEN `quantity`.`in_stock` ELSE NULL END as in_stock,
But I still don't like it. It seems very useless to me. And has no sense when I read CASE WHEN pv.islimited=1 THEN timelimit.start_date ELSE NULL END as start_date so if flag pv.islimited=0 you don't need start_date? Are you sure?
And the last thing I can suggest: try to use my or even your query. But add every table by step while debugging. So First query just:
SELECT
pv.`post_id`, pv.`seller_id` , pv.`islimited` , pv.`isquantity` ,
pv.`isslider`
FROM `post_view` AS pv
WHERE pv.`status` = 1
ORDER BY pv.`post_id` DESC
LIMIT 1
If it returns correct post_id add next table:
SELECT
pv.`post_id`, pv.`seller_id` , pv.`islimited` , pv.`isquantity` ,
pv.`isslider`,
`post_data`. *
FROM `post_view` AS pv
LEFT JOIN `post_data`
ON pv.`post_id` = `post_data`.`post_id`
WHERE pv.`status` = 1
AND `post_data`.`slug` = 'abc'
ORDER BY pv.`post_id` DESC
LIMIT 1
Check the result. And continue step by step.
Yes it takes time. But that is debugging process. It could be the fastest way to get that query done. :-)
SELECT `post_data`. * ,
pv.`post_id`, pv.`seller_id` , pv.`islimited` , pv.`isquantity` ,
pv.`isslider`, `price`.`original_price` , `price`.`discount_percentage` ,
CASE WHEN pv.`islimited`=1 THEN `timelimit`.`start_date` ELSE NULL END as start_date ,
CASE WHEN pv.`islimited`=1 THEN `timelimit`.`expire_date` ELSE NULL END as expire_date,
CASE WHEN pv.`isquantity`=1 THEN `quantity`.`in_stock` ELSE NULL END as in_stock,
`currency`.`currency_symbol`, `seller`.`directory`, `post_to_cat`.`cat_id`, count(`sales`.`sales_id`) as sale
FROM `post_view` AS pv
LEFT JOIN `post_data`
ON pv.`post_id` = `post_data`.`post_id`
LEFT JOIN `price`
ON pv.`post_id` = `price`.`post_id`
LEFT JOIN `currency`
ON `price`.`currency_id` = `currency`.`currency_id`
LEFT JOIN `seller`
ON pv.`seller_id` = `seller`.`seller_id`
LEFT JOIN `post_to_cat`
ON `post_to_cat`.`cat_id` = 1
AND `post_to_cat`.`post_id` = pv.`post_id`
LEFT JOIN `timelimit`
ON `timelimit`.`post_id` = pv.`post_id`
LEFT JOIN `quantity`
ON quantity`.`post_id` = pv.`post_id`
LEFT JOIN `sales`
ON `sales`.`post_id` = pv.`post_id`
AND `sales`.`status` = 1
WHERE pv.`status` = 1
AND `post_data`.`slug` = 'abc'
GROUP BY pv.`post_id`
ORDER BY pv.`post_id` DESC
LIMIT 1
EDIT 1 - last GROUP BY pv.post_id was added as per #McAdam331 notice about count() function without GROUP BY
I believe the issue here is mostly as a result of preforming aggregation (using the COUNT()) function, without any group by. Although, it seems like you don't necessarily need it because you want that count only for the post in question.
If you're trying to gather all of that information for a single post, I would adjust your WHERE clause to have a condition to only gather that information for the post with the largest ID.
Instead of ordering by ID and limiting by 1, use a subquery to get the largest id, like this:
...
WHERE pv.status = 1 AND post_data.slug = 'abc' AND pv.post_id = (SELECT MAX(post_id) FROM post_view);