Combining MySQL queries, with one being an exclusion query - mysql

First question on here, so I apologise if I have missed something previously asked, or don't format this well....
My company has a custom CRM + database that I am attempting to improve. We need to find a list of properties that will not have their yearly service renewed. Currently, we do this by first finding properties that will have their service renewed, which is done with the following query:
SELECT DISTINCT
j.`property_id`
FROM
`jobs` AS j
LEFT JOIN `property` AS p
ON j.`property_id` = p.`property_id`
LEFT JOIN `agency` AS a
ON p.`agency_id` = a.`agency_id`
INNER JOIN `property_services` AS ps
ON (
j.`property_id` = ps.`property_id`
AND j.`service` = ps.`alarm_job_type_id`
)
WHERE ps.`service` = 1
AND a.`country_id` = 1
AND (
j.`status` = 'Pending'
OR j.`date` IS NULL
OR j.`date` = '0000-00-00'
OR j.`job_type` = 'Once-off'
OR j.`job_type` = '240v Rebook'
OR (
j.`date` >= '2019-04-22'
AND j.`job_type` = 'Yearly Maintenance'
)
)
Then we find the details we want to display for the user, excluding other items in the process:
SELECT DISTINCT
j.`property_id`,
p.`address_1` AS p_address1,
p.`address_2` AS p_address2,
p.`address_3` AS p_address3,
p.`state` AS p_state,
p.`postcode` AS p_postcode,
a.`agency_id`,
a.`agency_name`
FROM
`jobs` AS j
LEFT JOIN `property` AS p
ON j.`property_id` = p.`property_id`
LEFT JOIN `agency` AS a
ON p.`agency_id` = a.`agency_id`
INNER JOIN `property_services` AS ps
ON (
j.`property_id` = ps.`property_id`
AND j.`service` = ps.`alarm_job_type_id`
)
WHERE p.`property_id` NOT IN (INSERT HERE THE IDS YOU GOT FROM THE FIRST QUERY)
AND ps.`service` = 1
AND p.`deleted` = 0
AND p.`agency_deleted` = 0
AND a.`status` = 'active'
AND a.`country_id` = 1
ORDER BY j.`property_id` DESC
LIMIT 0, 50
Ideally, I'd like to combine these queries, or somehow optimise them, as the page currently takes 2+ minutes to load, even with indexing.
Again, my apologies, I am not a database or query expert, pretty sure the degree only included one or two subjects on the matter!

For the record (in case someone would find it helpful), the combined version is:
SELECT DISTINCT
j.`property_id`,
p.`address_1` AS p_address1,
p.`address_2` AS p_address2,
p.`address_3` AS p_address3,
p.`state` AS p_state,
p.`postcode` AS p_postcode,
a.`agency_id`,
a.`agency_name`
FROM
`jobs` AS j
LEFT JOIN `property` AS p
ON j.`property_id` = p.`property_id`
LEFT JOIN `agency` AS a
ON p.`agency_id` = a.`agency_id`
INNER JOIN `property_services` AS ps
ON (
j.`property_id` = ps.`property_id`
AND j.`service` = ps.`alarm_job_type_id`
)
WHERE p.`property_id` NOT IN
(SELECT DISTINCT
j.`property_id`
FROM
`jobs` AS j
LEFT JOIN `property` AS p
ON j.`property_id` = p.`property_id`
LEFT JOIN `agency` AS a
ON p.`agency_id` = a.`agency_id`
INNER JOIN `property_services` AS ps
ON (
j.`property_id` = ps.`property_id`
AND j.`service` = ps.`alarm_job_type_id`
)
WHERE ps.`service` = 1
AND a.`country_id` = 1
AND (
j.`status` = 'Pending'
OR j.`date` IS NULL
OR j.`date` = '0000-00-00'
OR j.`job_type` = 'Once-off'
OR j.`job_type` = '240v Rebook'
OR (
j.`date` >= '2019-05-08'
AND j.`job_type` = 'Yearly Maintenance'
)
))
AND ps.`service` = 1
AND p.`deleted` = 0
AND p.`agency_deleted` = 0
AND a.`status` = 'active'
AND a.`country_id` = 1
AND (
j.`status` != 'Booked'
AND j.`status` != 'To Be Booked'
AND j.`status` != 'Send Letters'
AND j.`status` != 'On Hold'
AND j.`status` != 'On Hold - COVID'
AND j.`status` != 'Pre Completion'
AND j.`status` != 'Merged Certificates'
)
AND j.`date` > DATE_ADD(NOW(), INTERVAL - 350 DAY)
ORDER BY j.`property_id` DESC
Someone outside of StackOverflow helped combine, but it didn't help...
We ended up having to add a marker and search for that, because this query took 193 seconds to run on our database.
Also, I totally get the requirement to provide a minimum reproducible example, and its my fault for not doing that.

Related

Update sentence with subquery on MySQL

I have the following sentence, that returns the error
Unknown column targets.ID_TARGET in where clause
and I can't find any solution. Could you guys help?
The proposal is update 'sw_automatic' for each row with the value that th subquery provides (0 or 1)
update bt_pry_targets targets
set targets.sw_automatic = (
(
SELECT (CASE WHEN Task.ID_TP_TASKS_GROUPS = 694 THEN '0' ELSE '1' END) AS TYPE_TASK,
Task.ID_TASK FROM bt_tasks AS Task
INNER JOIN bt_pry_cmp_workflows AS BtCmpWorkflows ON (Task.ID_PRY_CMP_WORKFLOW = BtCmpWorkflows.ID_PRY_CMP_WORKFLOW)
INNER JOIN bt_pry_components AS PryComponent ON (PryComponent.ID_PRY_COMPONENT = BtCmpWorkflows.ID_PRY_COMPONENT )
INNER JOIN bt_components AS Component ON (PryComponent.ID_COMPONENT = Component.ID_COMPONENT)
INNER JOIN bt_pry_targets AS PryTarget ON (PryComponent.ID_TARGET = PryTarget.ID_TARGET)
INNER JOIN bt_flows AS Flows ON (Flows.ID_FLOW = Task.ID_FLOW)
WHERE Flows.SW_END_DEPENDENCE = 1
AND PryTarget.ID_TARGET = targets.ID_TARGET
GROUP BY Task.ID_TASK) )
where targets.sw_automatic is null;
In you subquery the column targets.ID_TARGET in not visible
so you could try using you subquery as a join table for updated
update bt_pry_targets targets
inner join (
SELECT (CASE WHEN Task.ID_TP_TASKS_GROUPS = 694 THEN '0' ELSE '1' END) AS TYPE_TASK,
Task.ID_TASK FROM bt_tasks AS Task
INNER JOIN bt_pry_cmp_workflows AS BtCmpWorkflows ON (Task.ID_PRY_CMP_WORKFLOW = BtCmpWorkflows.ID_PRY_CMP_WORKFLOW)
INNER JOIN bt_pry_components AS PryComponent ON (PryComponent.ID_PRY_COMPONENT = BtCmpWorkflows.ID_PRY_COMPONENT )
INNER JOIN bt_components AS Component ON (PryComponent.ID_COMPONENT = Component.ID_COMPONENT)
INNER JOIN bt_pry_targets AS PryTarget ON (PryComponent.ID_TARGET = PryTarget.ID_TARGET)
INNER JOIN bt_flows AS Flows ON (Flows.ID_FLOW = Task.ID_FLOW)
WHERE Flows.SW_END_DEPENDENCE = 1
AND PryTarget.ID_TARGET = targets.ID_TARGET
GROUP BY Task.ID_TASK
) t on t.PryTarget = targets.ID_TARGET
AND targets.sw_automatic is null
set targets.sw_automatic = t.TYPE_TASK

how optimize prestashop category get product for random

this is prestashop 1.7 version category get product query. if use random, it is very slow, how optimize it?
SELECT
cp.id_category,
p.*,
product_shop.*,
stock.out_of_stock,
IFNULL( stock.quantity, 0 ) AS quantity,
IFNULL( product_attribute_shop.id_product_attribute, 0 ) AS id_product_attribute,
product_attribute_shop.minimal_quantity AS product_attribute_minimal_quantity,
pl.`description`,
pl.`description_short`,
pl.`available_now`,
pl.`available_later`,
pl.`link_rewrite`,
pl.`meta_description`,
pl.`meta_keywords`,
pl.`meta_title`,
pl.`name`,
image_shop.`id_image` id_image,
il.`legend` AS legend,
m.`name` AS manufacturer_name,
cl.`name` AS category_default,
DATEDIFF(
product_shop.`date_add`,
DATE_SUB( "2019-11-30 00:00:00", INTERVAL 7 DAY )) > 0 AS new,
product_shop.price AS orderprice
FROM
`ps_category_product` cp
LEFT JOIN `ps_product` p ON p.`id_product` = cp.`id_product`
INNER JOIN ps_product_shop product_shop ON ( product_shop.id_product = p.id_product AND product_shop.id_shop = 1 )
LEFT JOIN `ps_product_attribute_shop` product_attribute_shop ON ( p.`id_product` = product_attribute_shop.`id_product` AND product_attribute_shop.`default_on` = 1 AND product_attribute_shop.id_shop = 1 )
LEFT JOIN ps_stock_available stock ON ( stock.id_product = `p`.id_product AND stock.id_product_attribute = 0 AND stock.id_shop = 1 AND stock.id_shop_group = 0 )
LEFT JOIN `ps_category_lang` cl ON ( product_shop.`id_category_default` = cl.`id_category` AND cl.`id_lang` = 11 AND cl.id_shop = 1 )
LEFT JOIN `ps_product_lang` pl ON ( p.`id_product` = pl.`id_product` AND pl.`id_lang` = 11 AND pl.id_shop = 1 )
LEFT JOIN `ps_image_shop` image_shop ON ( image_shop.`id_product` = p.`id_product` AND image_shop.cover = 1 AND image_shop.id_shop = 1 )
LEFT JOIN `ps_image_lang` il ON ( image_shop.`id_image` = il.`id_image` AND il.`id_lang` = 11 )
LEFT JOIN `ps_manufacturer` m ON m.`id_manufacturer` = p.`id_manufacturer`
WHERE
product_shop.`id_shop` = 1
AND cp.`id_category` = 12
AND product_shop.`active` = 1
AND product_shop.`visibility` IN ( "both", "catalog" )
ORDER BY
RAND()
LIMIT 50
Please provide SHOW CREATE TABLE for each table. Meanwhile, ...
Let's start by optimizing the joins.
LEFT JOIN `ps_product_lang` pl ON ( p.`id_product` = pl.`id_product`
AND pl.`id_lang` = 11
AND pl.id_shop = 1 )
That needs INDEX(id_product, id_lang, id_shop) (The columns may be in any order.)
Don't use LEFT unless you really need to fetch a row from the righthand table as NULLs when it does not exist. In particular,
LEFT JOIN `ps_product` p
is probably getting in the way of optimization.
WHERE product_shop.`id_shop` = 1
AND product_shop.`active` = 1
AND product_shop.`visibility` IN ( "both", "catalog" )
would probably benefit from these indexes
INDEX(id_shop, active, visibility, id_product)
INDEX(id_product, id_shop, active, visibility)
product_category needs
INDEX(id_category, id_product) -- in this order.
In general many-to-many mapping tables need to follow the tips here: http://mysql.rjweb.org/doc.php/index_cookbook_mysql#many_to_many_mapping_table
The query has the "explode-implode" syndrome. This is where it first does a JOINs, collecting a lot of data, then throws away much of it due, in your case, to the LIMIT 10. It can probably be cured by turning the query inside-out. The general ID is to start with a derived table that gets the 10 rows desired, then reaches into the other table for the rest of the desired columns. This "reaching" need happen only 10 times, not however many the JOINs currently require.
SELECT ...
FROM ( SELECT <<primary key columns from cp, p, and product_shop>>
FROM cp
JOIN p ON ...
JOIN product_shop ON ...
ORDER BY RAND()
LIMIT 10 ) AS x
JOIN <<p, product_shop ON their PKs>> -- to get p.*, product_shop.*>>
[LEFT] JOIN << each of the other tables>> -- to get the other tables
You should start by testing the subquery (a "derived" table) to verify that it is noticeably faster than the original query.

MySQL query taking too much time

query taking 1 minute to fetch results
SELECT
`jp`.`id`,
`jp`.`title` AS game_title,
`jp`.`game_type`,
`jp`.`state_abb` AS game_state,
`jp`.`location` AS game_city,
`jp`.`zipcode` AS game_zipcode,
`jp`.`modified_on`,
`jp`.`posted_on`,
`jp`.`game_referal_amount`,
`jp`.`games_referal_amount_type`,
`jp`.`status`,
`jp`.`is_flaged`,
`u`.`id` AS employer_id,
`u`.`email` AS employer_email,
`u`.`name` AS employer_name,
`jf`.`name` AS game_function,
`jp`.`game_freeze_status`,
`jp`.`game_statistics`,
`jp`.`ats_value`,
`jp`.`integration_id`,
`u`.`account_manager_id`,
`jp`.`model_game`,
`jp`.`group_id`,
(CASE
WHEN jp.group_id != '0' THEN gm.group_name
ELSE 'NA'
END) AS group_name,
`jp`.`priority_game`,
(CASE
WHEN jp.country != 'US' THEN jp.country_name
ELSE ''
END) AS game_country,
IFNULL((CASE
WHEN
`jp`.`account_manager_id` IS NULL
OR `jp`.`account_manager_id` = 0
THEN
(SELECT
(CASE
WHEN
account_manager_id IS NULL
OR account_manager_id = 0
THEN
`u`.`account_manager_id`
ELSE account_manager_id
END) AS account_manager_id
FROM
user_user
WHERE
id = (SELECT
user_id
FROM
game_user_assigned
WHERE
game_id = `jp`.`id`
LIMIT 1))
ELSE `jp`.`account_manager_id`
END),
`u`.`account_manager_id`) AS acc,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '1'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS somewhatgame,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '2'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS verygamecommitted,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '3'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS notgame,
(SELECT
COUNT(joa.id) AS applicationcount
FROM
game_refer_to_member jrmm
INNER JOIN
game_refer jrr ON jrr.id = jrmm.rid
INNER JOIN
game_applied joa ON jrmm.id = joa.referred_by
WHERE
jrmm.STATUS = '1'
AND jrr.referby_user_id IN (SELECT
ab_testing_user_id
FROM
ab_testing)
AND joa.game_post_id = rpr.recach_limit_game_id
AND (rpr.recach_limit = 1
OR rpr.recach_limit = 2)) AS gamecount
FROM
(`game_post` AS jp)
JOIN
`user_info` AS u ON `jp`.`user_user_id` = `u`.`id`
JOIN
`game_functional` jf ON `jp`.`game_functional_id` = `jf`.`id`
LEFT JOIN
`group_musesm` gm ON `gm`.`group_id` = `jp`.`group_id`
LEFT JOIN
`recach_limit` rpr ON `jp`.`id` = `rpr`.`recach_limit_game_id`
WHERE
`jp`.`status` != '3'
GROUP BY `jp`.`id`
ORDER BY `posted_on` DESC
LIMIT 10
I would first suggest not nesting select statements because this will cause an n^x performance hit on every xth level and I see at least 3 levels of selects inside this query.
Add index
INDEX(status, posted_on)
Move LIMIT inside
Then, instead of saying
FROM (`game_post` AS jp)
say
FROM ( SELECT id FROM game_post
WHERE status != 3
ORDER BY posted_on DESC
LIMIT 10 ) AS ids
JOIN game_post AS jp USING(id)
(I am assuming that the PK of jp is (id)?)
That should efficiently use the new index to get the 10 ids needed. Then it will reach back into game_post to get the other columns.
LEFT
Also, don't say LEFT unless you need it. It costs something to generate NULLs that you may not be needing.
Is GROUP BY necessary?
If you remove the GROUP BY, does it show dup ids? The above changes may have eliminated the need.
IN(SELECT) may optimize poorly
Change
AND jrr.referby_user_id IN ( SELECT ab_testing_user_id
FROM ab_testing )
to
AND EXISTS ( SELECT * FROM ab_testing
WHERE ab_testing_user_id = jrr.referby_user_id )
(This change may or may not help, depending on the version you are running.)
More
Please provide EXPLAIN SELECT if you need further assistance.

Like best performance in nested query

I have a principal requet with 2 requets in. I have a problem in my second nested query, I have a condition on id and if I made my request id = 10 takes a long time to execute, so if I replace it by id LIKE 10 my request execute in one second.
Here the request:
SELECT SQL_NO_CACHE contact_groupe.id_contact_groupe
FROM toto.contact_groupe
LEFT JOIN toto.`contact` AS `contact`
ON ((toto.contact_groupe.id_contact_groupe = toto.contact.id_contact_groupe))
LEFT JOIN toto.`project` AS `project`
ON ((toto.contact_groupe.id_contact_groupe = toto.project.id_contact_groupe)
AND ( toto.project.id_project
IN (
SELECT MAX(toto.project.id_project)
FROM toto.project
WHERE ( toto.contact_groupe.id_contact_groupe = toto.project.id_contact_groupe )
) ))
LEFT JOIN toto.`phase` AS `phase`
ON ((project.id_phase = toto.phase.id_phase))
LEFT JOIN sql_base.`user` AS `user_suivi`
ON ((toto.contact_groupe.id_user_suivi = user_suivi.id_user))
WHERE ( en_attente = '0' AND contact_groupe.id_contact_groupe
IN (
SELECT DISTINCT(contact_groupe.id_contact_groupe)
FROM toto.contact_groupe
LEFT JOIN toto.`contact` AS `contact`
ON ((toto.contact_groupe.id_contact_groupe = toto.contact.id_contact_groupe)
LEFT JOIN toto.`source_contact_groupe` AS `source_contact_groupe`
ON ((toto.contact_groupe.id_contact_groupe = toto.source_contact_groupe.id_contact_groupe))
LEFT JOIN toto.`project` AS `project`
ON ((toto.contact_groupe.id_contact_groupe = toto.project.id_contact_groupe))
LEFT JOIN toto.`remarque` AS `remarque`
ON ((toto.contact_groupe.id_contact_groupe = toto.remarque.id_contact_groupe))
LEFT JOIN toto.`project_type_construction_options` AS `project_type_construction_options`
ON ((project.id_project = toto.project_type_construction_options.id_project))
LEFT JOIN toto.`project_concurrent` AS `project_concurrent`
ON ((project.id_project = toto.project_concurrent.id_project))
LEFT JOIN toto.`telephone` AS `telephone`
ON ((contact.id_contact = toto.telephone.id_contact))
WHERE ( en_attente = '0' AND ( toto.project.id_project = '10' ) AND toto.contact_groupe.id_entreprise = '2' )
)
AND toto.contact_groupe.id_entreprise = '2' )
ORDER BY toto.contact_groupe.id_contact_groupe ASC
the line is the following problem toto.project.id_project = '10' and I don't understand why the time to execute request is so different between = and LIKE
Let's start with your subquery. Those 17 lines that you've written are functionally identical to this, so why not use this instead?
SELECT DISTINCT g.id_contact_groupe
FROM contact_groupe g
JOIN project p
ON p.id_contact_groupe = g.id_contact_groupe
WHERE g.en_attente = 0
AND p.id_project = 10
AND g.id_entreprise = 2

Same MySQL query runs much slower in 5.6 than in 5.1

I am encountering a strange issue where this particular MySQL query that we have would run almost 50 times slower after we upgraded our database from MySQL 5.1.73 to 5.6.23.
This is the SQL query:
SELECT `companies`.*
FROM `companies`
LEFT OUTER JOIN `company_texts`
ON `company_texts`.`company_id` = `companies`.`id`
AND `company_texts`.`language` = 'en'
AND `company_texts`.`region` = 'US'
INNER JOIN show_texts
ON show_texts.company_id = companies.id
AND `show_texts`.`is_deleted` = 0
AND `show_texts`.`language` = 'en'
AND `show_texts`.`region` = 'US'
INNER JOIN show_region_counts
ON show_region_counts.show_id = show_texts.show_id
AND show_region_counts.region = 'US'
WHERE ( ( `companies`.`id` NOT IN ( '77', '26' ) )
AND ( `company_texts`.is_deleted = 0 )
AND `companies`.id IN (
SELECT DISTINCT show_texts.company_id AS
id
FROM shows
INNER JOIN `show_rollups`
ON
`show_rollups`.`show_id` = `shows`.`id`
AND ( `show_rollups`.`device_id` = 3 )
AND ( `show_rollups`.`package_group_id` = 2 )
AND ( `show_rollups`.`videos_count` > 0 )
LEFT OUTER JOIN `show_texts` ON
`show_texts`.`show_id` = `shows`.`id`
AND
`show_texts`.`is_deleted` = 0
AND
`show_texts`.`language` = 'en'
AND
`show_texts`.`region` = 'US'
AND
shows.is_browseable = 1
AND
show_texts.show_id IS NOT NULL
AND (
`show_rollups`.`episodes_count` > 0
OR `show_rollups`.`clips_count` > 0
OR `show_rollups`.`games_count` > 0
)
) )
GROUP BY companies.id
ORDER BY Sum(show_region_counts.view_count) DESC
LIMIT 30 offset 30;
Now the problem is when I run this query in MySQL 5.1.73 before the upgrade, the query would only take around 1.5 seconds, but after the upgrade to 5.6.23, it now can take upward to 1 minute.
So I did an EXPLAIN of this query in 5.1.73, and I saw this:
Enlarged version : http://i.stack.imgur.com/c4ko0.jpg
And when I did EXPLAIN in 5.6.23 , I saw this :
Enlarged version : http://i.stack.imgur.com/CgBtA.jpg
I can see that in both cases, there is a full scan (type ALL) of the shows table, but is there something else I am not seeing that is causing the massive slowdown in 5.6?
Thanks
IS
Please provide SHOW CREATE TABLE. Without that, I will guess that you are missing this desirable index:
show_rollups: INDEX(device_id, package_group_id, videos_count)
You have LEFT OUTER JOIN show_texts ... ON ... show_texts.show_id IS NOT NULL. This is probably wrong for two reasons: (a) Don't use LEFT if you are not looking for NULL, and (b) the NULL test should be in the missing WHERE clause, not in the ON clause.
Getting rid of the IN ( SELECT ... ) may help it on both machines:
SELECT c.*
FROM
( SELECT DISTINCT st.company_id AS id
FROM shows
INNER JOIN `show_rollups` AS sr ON sr.`show_id` = `shows`.`id`
AND ( sr.`device_id` = 3 )
AND ( sr.`package_group_id` = 2 )
AND ( sr.`videos_count` > 0 )
LEFT OUTER JOIN `show_texts` AS st ON st.`show_id` = `shows`.`id`
AND st.`is_deleted` = 0
AND st.`language` = 'en'
AND st.`region` = 'US'
AND shows.is_browseable = 1
AND st.show_id IS NOT NULL
AND ( sr.`episodes_count` > 0
OR sr.`clips_count` > 0
OR sr.`games_count` > 0 )
) AS x
JOIN `companies` AS c ON x.id = c.id
LEFT OUTER JOIN `company_texts` AS ct ON ct.`company_id` = c.`id`
AND ct.`language` = 'en'
AND ct.`region` = 'US'
INNER JOIN show_texts AS st ON st.company_id = c.id
AND st.`is_deleted` = 0
AND st.`language` = 'en'
AND st.`region` = 'US'
INNER JOIN show_region_counts AS src ON src.show_id = st.show_id
AND src.region = 'US'
WHERE ( c.`id` NOT IN ( '77', '26' ) )
AND ( ct.is_deleted = 0 )
GROUP BY c.id
ORDER BY Sum(src.view_count) DESC
LIMIT 30 offset 30;
There is some chance that the JOINs will mess with the computation in Sum(src.view_count).
I'm sorry, but I can't make out the EXPLAIN screen-grabs on my display.
Without that, I'd wonder if the schemas for the tables changed at all, or of the database engines changed. In particular, companies.id seems to be used as a primary key.