Size:
Campaigns: 3k rows (200 with campaigns.is_active = 1)
Links: 20k rows (4k with links.status = 1 // 500 with links.status = 1 AND campaigns.is_active = 1)
Clicks: 10mln rows (50k with created > '2020-10-25 00:00:00')
This query runs 2 seconds
SELECT links.id, COUNT(clicks.id)
FROM links
INNER JOIN campaigns ON campaigns.id = links.campaign_id
AND campaigns.is_active = 1
LEFT JOIN clicks ON clicks.link_id = links.id
WHERE links.status = 1
AND clicks.created > '2020-10-25 00:00:00'
GROUP BY links.id
When I remove the following line, it runs just 0.13 seconds (15 times faster)
AND campaigns.is_active = 1
There is an INDEX on campaigns.is_active.
Also tried to set an index on 2 columns (campaigns.id + campaigns.is_active) but didn't help.
"campaigns.is_active" contains simply 0 or 1. The campaigns table is small, the campaigns.is_active condition actually reduces the amount of rows. So it should speed up the query instead.
Why does it take so much longer because of this condition and how to fix it?
If I would remove the JOIN to campaigns and instead add links.campaign_id to the SELECT fields and then query every single of the returned campaign_id's in an additional query like "SELECT is_active FROM campaigns WHERE id = ?" it would still be faster, because such a query is 0.000x. From my experience when something is faster in 2 queries, it usually means the first query isn't optimized to its full extent.
Explain-Select
Structure
CREATE TABLE `campaigns` (
`id` int(11) UNSIGNED NOT NULL,
`is_active` tinyint(4) NOT NULL DEFAULT 0
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `clicks` (
`id` int(11) UNSIGNED NOT NULL,
`link_id` int(11) UNSIGNED NOT NULL,
`created` datetime NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `links` (
`id` int(11) UNSIGNED NOT NULL,
`campaign_id` int(8) UNSIGNED NOT NULL,
`status` tinyint(4) NOT NULL DEFAULT 0
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
ALTER TABLE `campaigns`
ADD PRIMARY KEY (`id`),
ADD UNIQUE KEY `id_isactive` (`id`,`is_active`),
ADD KEY `is_active` (`is_active`)
ALTER TABLE `clicks`
ADD PRIMARY KEY (`id`),
ADD KEY `link_id` (`link_id`),
ADD KEY `created` (`created`)
ALTER TABLE `links`
ADD PRIMARY KEY (`id`),
ADD KEY `campaign_id` (`campaign_id`),
How long does this take?
SELECT l.id,
(SELECT COUNT(*)
FROM clicks cl
WHERE cl.link_id = l.id AND
cl.created > '2020-10-25'
)
FROM links l JOIN
campaigns ca
ca.id = l.campaign_id
WHERE l.status = 1 AND ca.is_active = 1;
EDIT:
Hmmm, with an order by, you can try:
SELECT l.id,
(SELECT COUNT(*)
FROM clicks cl
WHERE cl.link_id = l.id AND
cl.created > '2020-10-25'
)
FROM links l
WHERE EXISTS (SELECT 1
FROM campaigns ca
WHERE ca.id = l.campaign_id AND ca.is_active = 1
)
WHERE l.status = 1
ORDER BY l.id;
For this, you want an index on links(status, id) and campaigns(campaign_id, is_active).
Question... If a campaign is not currently active, you don't want any output for it, correct? Furthermore, there won't be any clicks for inactive campaigns, correct? Then why bother checking is_active?
Even if my analysis is wrong, it may be faster to ignore is_active until after the counts have been tallied.
Please don't use LEFT when it is not functional. You have a simple JOIN.
Use COUNT(*); COUNT(x) tests x for being not null.
SELECT links.id, COUNT(*)
FROM links
JOIN clicks ON clicks.link_id = links.id
WHERE links.status = 1
AND clicks.created > '2020-10-25 00:00:00'
GROUP BY links.id
This is redundant:
ADD UNIQUE KEY `id_isactive` (`id`,`is_active`),
since PRIMARY KEY(id) declares id to be an index and unique.
I prefer not to fight the database engine optimizer.
SELECT links.id, campaigns.is_active, COUNT(clicks.id)
FROM links
INNER JOIN campaigns ON campaigns.id = links.campaign_id
LEFT JOIN clicks ON clicks.link_id = links.id
WHERE links.status = 1
AND clicks.created > '2020-10-25 00:00:00'
GROUP BY links.id, campaigns.is_active
HAVING campaigns.is_active = 1;
Second variant!
-- Second Variant
EXPLAIN
SELECT links.id AS LinksId
, COUNT(clicks.id) AS ClickCount
FROM links
LEFT JOIN clicks
ON links.id = clicks.link_id
WHERE links.status = 1
AND clicks.created > '2020-10-25 00:00:00'
AND links.campaign_id IN (SELECT campaign_id
FROM campaigns
WHERE is_active = 1)
GROUP BY links.id;
Third time is the charm! Using CTEs due to the published cardinalities.
-- Third time is the charm
WITH ActiveCampaigns
AS
(SELECT *
FROM campaigns
WHERE is_active = 1)
SELECT links.id, COUNT(clicks.id)
FROM links
INNER JOIN ActiveCampaigns
ON ActiveCampaigns.id = links.campaign_id
LEFT JOIN clicks
ON clicks.link_id = links.id
WHERE links.status = 1
AND clicks.created > '2020-10-25 00:00:00'
GROUP BY links.id;
I have a complex query which takes 700ms to run on my machine. I found that the bottleneck is the ORDER BY at_firstname.value clause, but how can I use indexes to improve this?
SELECT
`e`.*
, `at_default_billing`.`value` AS `default_billing`
, `at_billing_postcode`.`value` AS `billing_postcode`
, `at_billing_city`.`value` AS `billing_city`
, `at_billing_region`.`value` AS `billing_region`
, `at_billing_country_id`.`value` AS `billing_country_id`
, `at_company`.`value` AS `company`
, `at_firstname`.`value` AS `firstname`
, `at_lastname`.`value` AS `lastname`
, CONCAT(at_firstname.value
, " "
, at_lastname.value) AS `full_name`
, `at_phone`.`value` AS `phone`
, IFNULL(at_phone.value,"N/A") AS `telephone`
, `e`.`entity_id` AS `id`
FROM
`customer_entity` AS `e`
LEFT JOIN
`customer_entity_int` AS `at_default_billing`
ON (`at_default_billing`.`entity_id` = `e`.`entity_id`)
AND (`at_default_billing`.`attribute_id` = '13')
LEFT JOIN
`customer_address_entity_varchar` AS `at_billing_postcode`
ON (`at_billing_postcode`.`entity_id` = `at_default_billing`.`value`)
AND (`at_billing_postcode`.`attribute_id` = '30')
LEFT JOIN
`customer_address_entity_varchar` AS `at_billing_city`
ON (`at_billing_city`.`entity_id` = `at_default_billing`.`value`)
AND (`at_billing_city`.`attribute_id` = '26')
LEFT JOIN
`customer_address_entity_varchar` AS `at_billing_region`
ON (`at_billing_region`.`entity_id` = `at_default_billing`.`value`)
AND (`at_billing_region`.`attribute_id` = '28')
LEFT JOIN
`customer_address_entity_varchar` AS `at_billing_country_id`
ON (`at_billing_country_id`.`entity_id` = `at_default_billing`.`value`)
AND (`at_billing_country_id`.`attribute_id` = '27')
LEFT JOIN
`customer_address_entity_varchar` AS `at_company`
ON (`at_company`.`entity_id` = `at_default_billing`.`value`)
AND (`at_company`.`attribute_id` = '24')
LEFT JOIN
`customer_entity_varchar` AS `at_firstname`
ON (`at_firstname`.`entity_id` = `e`.`entity_id`)
AND (`at_firstname`.`attribute_id` = '5')
LEFT JOIN
`customer_entity_varchar` AS `at_lastname`
ON (`at_lastname`.`entity_id` = `e`.`entity_id`)
AND (`at_lastname`.`attribute_id` = '7')
LEFT JOIN
`customer_entity_varchar` AS `at_phone`
ON (`at_phone`.`entity_id` = `e`.`entity_id`)
AND (`at_phone`.`attribute_id` = '136')
ORDER BY
`at_firstname`.`value` ASC LIMIT 20
This is execution plan :
EXPLAIN of Query :
'1','SIMPLE','e',NULL,'ALL',NULL,NULL,NULL,NULL,'19951','100.00','Using temporary; Using filesort'
'1','SIMPLE','at_default_billing',NULL,'eq_ref','UNQ_CUSTOMER_ENTITY_INT_ENTITY_ID_ATTRIBUTE_ID,IDX_CUSTOMER_ENTITY_INT_ATTRIBUTE_ID,IDX_CUSTOMER_ENTITY_INT_ENTITY_ID,IDX_CUSTOMER_ENTITY_INT_ENTITY_ID_ATTRIBUTE_ID_VALUE','UNQ_CUSTOMER_ENTITY_INT_ENTITY_ID_ATTRIBUTE_ID','6','lazurd.e.entity_id,const','1','100.00',NULL
'1','SIMPLE','at_billing_postcode',NULL,'eq_ref','UNQ_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ATTRIBUTE_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID_VALUE','UNQ_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID','6','lazurd.at_default_billing.value,const','1','100.00','Using where'
'1','SIMPLE','at_billing_city',NULL,'eq_ref','UNQ_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ATTRIBUTE_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID_VALUE','UNQ_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID','6','lazurd.at_default_billing.value,const','1','100.00','Using where'
'1','SIMPLE','at_billing_region',NULL,'eq_ref','UNQ_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ATTRIBUTE_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID_VALUE','UNQ_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID','6','lazurd.at_default_billing.value,const','1','100.00','Using where'
'1','SIMPLE','at_billing_country_id',NULL,'eq_ref','UNQ_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ATTRIBUTE_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID_VALUE','UNQ_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID','6','lazurd.at_default_billing.value,const','1','100.00','Using where'
'1','SIMPLE','at_company',NULL,'eq_ref','UNQ_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ATTRIBUTE_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID,IDX_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID_VALUE','UNQ_CUSTOMER_ADDRESS_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID','6','lazurd.at_default_billing.value,const','1','100.00','Using where'
'1','SIMPLE','at_firstname',NULL,'eq_ref','UNQ_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID,IDX_CUSTOMER_ENTITY_VARCHAR_ATTRIBUTE_ID,IDX_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID,IDX_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID_VALUE','UNQ_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID','6','lazurd.e.entity_id,const','1','100.00',NULL
'1','SIMPLE','at_lastname',NULL,'eq_ref','UNQ_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID,IDX_CUSTOMER_ENTITY_VARCHAR_ATTRIBUTE_ID,IDX_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID,IDX_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID_VALUE','UNQ_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID','6','lazurd.e.entity_id,const','1','100.00',NULL
'1','SIMPLE','at_phone',NULL,'eq_ref','UNQ_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID,IDX_CUSTOMER_ENTITY_VARCHAR_ATTRIBUTE_ID,IDX_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID,IDX_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID_VALUE','UNQ_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID','6','lazurd.e.entity_id,const','1','100.00',NULL
Table Structure:
CREATE TABLE `customer_entity_varchar` (
`value_id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'Value Id',
`entity_type_id` smallint(5) unsigned NOT NULL DEFAULT '0' COMMENT 'Entity Type Id',
`attribute_id` smallint(5) unsigned NOT NULL DEFAULT '0' COMMENT 'Attribute Id',
`entity_id` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'Entity Id',
`value` varchar(255) DEFAULT NULL COMMENT 'Value',
PRIMARY KEY (`value_id`),
UNIQUE KEY `UNQ_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID` (`entity_id`,`attribute_id`),
KEY `IDX_CUSTOMER_ENTITY_VARCHAR_ENTITY_TYPE_ID` (`entity_type_id`),
KEY `IDX_CUSTOMER_ENTITY_VARCHAR_ATTRIBUTE_ID` (`attribute_id`),
KEY `IDX_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID` (`entity_id`),
KEY `IDX_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID_ATTRIBUTE_ID_VALUE` (`entity_id`,`attribute_id`,`value`),
CONSTRAINT `FK_CSTR_ENTT_VCHR_ATTR_ID_EAV_ATTR_ATTR_ID` FOREIGN KEY (`attribute_id`) REFERENCES `eav_attribute` (`attribute_id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `FK_CSTR_ENTT_VCHR_ENTT_TYPE_ID_EAV_ENTT_TYPE_ENTT_TYPE_ID` FOREIGN KEY (`entity_type_id`) REFERENCES `eav_entity_type` (`entity_type_id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `FK_CUSTOMER_ENTITY_VARCHAR_ENTITY_ID_CUSTOMER_ENTITY_ENTITY_ID` FOREIGN KEY (`entity_id`) REFERENCES `customer_entity` (`entity_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=131094 DEFAULT CHARSET=utf8 COMMENT='Customer Entity Varchar';
As of now your query is:
Performing ALL left outer joins first.
Then ORDERing the rows.
Then LIMITing the rows.
I would perform the strictly needed outer joins first, then ordering and limiting (to reduce to 20 rows), and finally I would do all the rest of the outer joins. In short I would do:
Performing minimal left outer join first. That is, two tables only.
Then ORDERing the rows.
Then LIMITing the rows. This produce a max of 20 rows.
Perform all the rest of outer joins. At this point this is not thousands of rows anymore, but only 20.
This change should massively reduce the "Unique Key Lookup" executions. The modified query will look like:
select
e.*
, `at_default_billing`.`value` AS `default_billing`
, `at_billing_postcode`.`value` AS `billing_postcode`
, `at_billing_city`.`value` AS `billing_city`
, `at_billing_region`.`value` AS `billing_region`
, `at_billing_country_id`.`value` AS `billing_country_id`
, `at_company`.`value` AS `company`
, `at_lastname`.`value` AS `lastname`
, CONCAT(firstname
, " "
, at_lastname.value) AS `full_name`
, `at_phone`.`value` AS `phone`
, IFNULL(at_phone.value,"N/A") AS `telephone`
from ( -- Step #1: joining customer_entity with customer_entity_varchar
SELECT
`e`.*
, `at_firstname`.`value` AS `firstname`
, `e`.`entity_id` AS `id`
FROM
`customer_entity` AS `e`
LEFT JOIN
`customer_entity_varchar` AS `at_firstname`
ON (`at_firstname`.`entity_id` = `e`.`entity_id`)
AND (`at_firstname`.`attribute_id` = '5')
ORDER BY -- Step #2: Sorting (the bare minimum)
`at_firstname`.`value` ASC
LIMIT 20 -- Step #3: Limiting (to 20 rows)
) e
LEFT JOIN -- Step #4: Performing all the rest of outer joins (only few rows now)
`customer_entity_int` AS `at_default_billing`
ON (`at_default_billing`.`entity_id` = `e`.`entity_id`)
AND (`at_default_billing`.`attribute_id` = '13')
LEFT JOIN
`customer_address_entity_varchar` AS `at_billing_postcode`
ON (`at_billing_postcode`.`entity_id` = `at_default_billing`.`value`)
AND (`at_billing_postcode`.`attribute_id` = '30')
LEFT JOIN
`customer_address_entity_varchar` AS `at_billing_city`
ON (`at_billing_city`.`entity_id` = `at_default_billing`.`value`)
AND (`at_billing_city`.`attribute_id` = '26')
LEFT JOIN
`customer_address_entity_varchar` AS `at_billing_region`
ON (`at_billing_region`.`entity_id` = `at_default_billing`.`value`)
AND (`at_billing_region`.`attribute_id` = '28')
LEFT JOIN
`customer_address_entity_varchar` AS `at_billing_country_id`
ON (`at_billing_country_id`.`entity_id` = `at_default_billing`.`value`)
AND (`at_billing_country_id`.`attribute_id` = '27')
LEFT JOIN
`customer_address_entity_varchar` AS `at_company`
ON (`at_company`.`entity_id` = `at_default_billing`.`value`)
AND (`at_company`.`attribute_id` = '24')
LEFT JOIN
`customer_entity_varchar` AS `at_lastname`
ON (`at_lastname`.`entity_id` = `e`.`entity_id`)
AND (`at_lastname`.`attribute_id` = '7')
LEFT JOIN
`customer_entity_varchar` AS `at_phone`
ON (`at_phone`.`entity_id` = `e`.`entity_id`)
AND (`at_phone`.`attribute_id` = '136')
Unfortunately, SELECT whole_mess_of_rows FROM many_tables ORDER BY one_col LIMIT small_number is a notorious performance antipattern. Why? Because it sorts a big result set, just to discard most of it.
The trick is to cheaply find out which rows are within that LIMIT small_number, then retrieve only those rows from the larger query.
Which rows do you want? It looks to me like this query will retrieve their customer_entity.id values. But it's hard to be sure, so you should test this subquery.
SELECT customer_entity.entity_id
FROM customer_entity
LEFT JOIN customer_entity_varchar AS at_firstname
ON (at_firstname.entity_id = e.entity_id)
AND (at_firstname.attribute_id = '5')
ORDER BY at_firstname.value ASC
LIMIT 20
This should give the twenty relevant entity_id values. Test it. Look at its execution plan. Add an appropriate index to customer_entity if need be. That index might be (firstname_attribute_id, firstname_entity_id, firstname_value) But I am guessing.
Then you can put this at the end of your main query, right before ORDER BY.
WHERE e.entity_id IN (
SELECT customer_entity.entity_id
FROM customer_entity
LEFT JOIN customer_entity_varchar AS at_firstname
ON (at_firstname.entity_id = e.entity_id)
AND (at_firstname.attribute_id = '5')
ORDER BY at_firstname.value ASC
LIMIT 20
)
and things should be a bit faster.
I agree with the previous Answers, but want to emphasize on more antipattern: Over-noramlization.
Your schema is a curious (and inefficient) variant on the already-bad EAV schema pattern.
There is little advantage, and some disadvantage in splitting customer_address_entity_varchar across 5 tables. Similarly for customer_entity_varchar.
An address should (usually) be stored as a few columns in a single table; no JOINs to other tables.
Likewise for firstname+lastname.
Phone could be another issue, since a person/company/entity could have multiple phone numbers (cell, home, work, fax, etc). But that is a different story.
I have a MySQL database table that's quite large with 2.5million rows and growing. To speed up queries I've added an index to one of columns. When I set the index manually, through PHPMyAdmin for example, it's cardinality is around 1500 which seems about right and my queries run without issue.
The problem then comes after a few queries, (especially on an INSERT but not limited to), have been run, the cardinality of that index drops to 17 or 18, and queries run extremely slow. Sometimes it seems to work it's way back to about 1500 or I have to do it through PHPMyAdmin again.
Is there any way to stop this cardinality drop from happening?
CREATE TABLE IF NOT EXISTS `probe_results` (
`probe_result_id` int(11) NOT NULL AUTO_INCREMENT,
`date` date NOT NULL,
`month` int(11) NOT NULL,
`year` int(11) NOT NULL,
`time` time NOT NULL,
`type` varchar(11) NOT NULL,
`probe_id` varchar(50) NOT NULL,
`status` varchar(11) NOT NULL,
`temp_1` decimal(11,0) NOT NULL,
`temp_2` decimal(11,0) NOT NULL,
`crc` varchar(11) NOT NULL,
`raw_data` text NOT NULL,
`txt_file` text NOT NULL,
PRIMARY KEY (`probe_result_id`),
KEY `probe_id` (`probe_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=2527300 ;
The 'probe_result_id' column is the primary key, the probe_id is the column with the index in question.
Example query:
SELECT IF(b.reactive_total IS NULL, 0, b.reactive_total) AS reactive_total, a.* FROM (SELECT COUNT(CASE WHEN asset_testing_results.asset_testing_year = '2016' AND asset_testing_results.asset_testing_month = '7' AND asset_testing_results.asset_stopped = '0' AND asset_testing_results.asset_testing_completed = '0' THEN 1 END) AS due_total, (COUNT(CASE WHEN asset_testing_results.asset_testing_year = '2016' AND asset_testing_results.asset_stopped = '0' AND asset_testing_results.asset_testing_completed = '1' AND asset_testing_results.asset_testing_satisfactory = '1' AND asset_testing_results.asset_testing_actioned = '0' THEN 1 END)+(IF(probes_passed_total IS NULL, 0, probes_passed_total))) AS passed_total, (COUNT(CASE WHEN asset_testing_results.asset_testing_year = '2016' AND asset_testing_results.asset_stopped = '0' AND asset_testing_results.asset_testing_completed = '1' AND asset_testing_results.asset_testing_satisfactory = '0' AND asset_testing_results.asset_testing_actioned = '0' THEN 1 END)+(IF(probes_failed_total IS NULL, 0, probes_failed_total))) AS failed_total, COUNT(CASE WHEN asset_testing_results.asset_testing_year = '2016' AND asset_testing_results.asset_stopped = '0' AND asset_testing_results.asset_testing_completed = '1' AND asset_testing_results.asset_testing_actioned = '1' THEN 1 END) AS actioned_total, COUNT(CASE WHEN asset_testing_results.asset_testing_year = '2016' AND asset_testing_results.asset_testing_month < '7' AND asset_testing_results.asset_testing_completed = '0' AND asset_testing_results.asset_testing_satisfactory = '0' AND asset_testing_results.asset_stopped = '0' THEN 1 END) AS missed_total, site.site_key, site.site_name FROM site LEFT JOIN location ON location.site_key = site.site_key LEFT JOIN sub_location ON sub_location.location_key = location.location_key LEFT JOIN asset ON asset.sub_location_key = sub_location.sub_location_key AND asset.stopped = '0' LEFT JOIN asset_testing ON asset_testing.asset_type_key = asset.asset_type_key AND asset_testing.probe_assessed = '0' LEFT JOIN asset_testing_results ON asset_testing_results.asset_testing_key = asset_testing.asset_testing_key AND asset_testing_results.asset_key = asset.asset_key LEFT JOIN (SELECT site.site_key, COUNT(CASE WHEN p.probe_id IS NOT NULL AND p.asset_testing_key IS NOT NULL THEN 1 END) AS probes_passed_total, COUNT(CASE WHEN p.probe_id IS NOT NULL AND p.asset_testing_key IS NULL AND p.temp_1 IS NOT NULL THEN 1 END) AS probes_failed_total FROM assetsvs_probes LEFT JOIN (SELECT q.probe_id, q.month, q.year, IF(r.temp_1 IS NULL, q.temp_1, r.temp_1) as temp_1, r.asset_testing_key FROM (SELECT DISTINCT probe_results.probe_id, probe_results.month, probe_results.year, probe_results.temp_1 FROM probe_results LEFT JOIN assetsvs_probes ON assetsvs_probes.probe_id = probe_results.probe_id LEFT JOIN asset ON asset.asset_key = assetsvs_probes.asset_key LEFT JOIN sub_location ON sub_location.sub_location_key = asset.sub_location_key LEFT JOIN location ON location.location_key = sub_location.location_key LEFT JOIN site ON site.site_key = location.site_key WHERE site.client_key = '25')q LEFT JOIN (SELECT probe_results.month, probe_results.year, probe_results.probe_id, temp_1, asset_testing.asset_testing_key FROM probe_results LEFT JOIN assetsvs_probes ON assetsvs_probes.probe_id = probe_results.probe_id LEFT JOIN asset_testing ON asset_testing.asset_testing_key = assetsvs_probes.asset_testing_key LEFT JOIN asset ON asset.asset_key = assetsvs_probes.asset_key LEFT JOIN sub_location ON sub_location.sub_location_key = asset.sub_location_key LEFT JOIN location ON location.location_key = sub_location.location_key LEFT JOIN site ON site.site_key = location.site_key WHERE temp_1 != 'invalid' AND ((temp_1 >= test_min AND test_max = '') OR (temp_1 <= test_max AND test_min = '') OR (temp_1 >= test_min AND temp_1 <= test_max)) AND year = '2016' AND site.client_key = '25' GROUP BY probe_results.month, probe_results.year, probe_results.probe_id)r ON r.probe_id = q.probe_id AND r.month = q.month AND r.year = q.year WHERE q.year = '2016' GROUP BY probe_id, month, year) p ON p.probe_id = assetsvs_probes.probe_id LEFT JOIN asset_testing ON asset_testing.asset_testing_key = assetsvs_probes.asset_testing_key LEFT JOIN asset ON asset.asset_key = assetsvs_probes.asset_key LEFT JOIN sub_location ON sub_location.sub_location_key = asset.sub_location_key LEFT JOIN location ON location.location_key = sub_location.location_key LEFT JOIN site ON site.site_key = location.site_key GROUP BY site.site_key) probe_results ON probe_results.site_key = site.site_key WHERE site.client_key = '25' GROUP BY site.site_key)a LEFT JOIN (SELECT COUNT(CASE WHEN jobs.status = '3' THEN 1 END) AS reactive_total, site.site_key FROM jobs LEFT JOIN jobs_meta ON jobs_meta.job_id = jobs.job_id AND jobs_meta.meta_key = 'start_date' LEFT JOIN site ON site.site_key = jobs.site_key WHERE site.client_key = '25' AND jobs_meta.meta_value LIKE '%/2016 %' GROUP BY site.site_key)b ON b.site_key = a.site_key
Thanks
Cardinality (along with other statistics) is calculated and updated by MySQL automatically, so you do not have direct means to prevent it from dropping.
However, you can take a few steps to make this less likely to happen or correct the behaviour.
First of all, MySQL updates index statistics for all supported table engines if you run analyze table command.
For innodb table engine MySQL provides a set of configuration settings that can influence the behaviour of the sampling. The settings and their effect are described in the MySQL documentation:
non-persistent statistics
persistent statistics
The main setting is innodb_stats_transient_sample_pages:
• Small values like 1 or 2 can result in inaccurate estimates of
cardinality.
• Increasing the innodb_stats_transient_sample_pages value might
require more disk reads. Values much larger than 8 (say, 100), can
cause a significant slowdown in the time it takes to open a table or
execute SHOW TABLE STATUS.
• The optimizer might choose very different query plans based on
different estimates of index selectivity
.
For myisam MySQL dos not provide such a variety of settings. myisam_stats_method setting is described in the general index statistics documentation
Lets say i have two stores. Store A(22) and Store B(21). This is the query to fetch the things that matched with the Store ID:
SELECT c . * , s.s_name, s.logo, s.s_slug, cm.c_code, cm.c_shorturl, cm.c_shorturl_id
FROM ci_cptbl c
LEFT JOIN ci_stores s ON s.store_id = c.store_id
LEFT JOIN ci_cptbl_mapper cm ON cm.c_id = c._id
WHERE c.coupon_id <> ''
AND c.store_id in ('22', '21')
AND s.s_status = '1'
AND c.c_status = '1'
AND DATE( c.c_end_date ) >= '2014-10-04'
ORDER BY c.c_id DESC
ci_cptbl Has the collection of product including the store_id. And ci_stores holds the store name, etc.. including s_status(0, 1).
CREATE TABLE `ci_stores` (
`store_id` int(10) NOT NULL AUTO_INCREMENT,
`cat_id` int(10) NOT NULL,
`s_name` varchar(255) NOT NULL,
`s_slug` varchar(255) NOT NULL,
`logo` varchar(255) NOT NULL,
`display_name` varchar(255) NOT NULL,
`s_description` text NOT NULL,
`network_id` int(10) NOT NULL,
`s_status` tinyint(1) NOT NULL DEFAULT '0',
`merged_stores` text NOT NULL,
`stat` bigint(20) NOT NULL,
PRIMARY KEY (`store_id`),
KEY `network_id` (`network_id`),
KEY `cat_id` (`cat_id`),
FULLTEXT KEY `display_name` (`display_name`)
) ENGINE=MyISAM AUTO_INCREMENT=127 DEFAULT CHARSET=utf8
Now condition is i have only first store id( 22 ) enabled and rest is disabled (21, .....) in my stores table and my AND s.s_status = '1' statement works only if all the stores are enabled but i want all stores including disabled.
BUT FIRST STORE ID MUST BE ENABLED
You could try changing you condition to test on store_id and enabled
(c.c_status = '1' or c.store_id <> '22')
so:
SELECT c . * , s.s_name, s.logo, s.s_slug, cm.c_code, cm.c_shorturl, cm.c_shorturl_id
FROM ci_cptbl c
LEFT JOIN ci_stores s ON s.store_id = c.store_id
LEFT JOIN ci_cptbl_mapper cm ON cm.c_id = c._id
WHERE c.coupon_id <> ''
AND c.store_id in ('22', '21')
AND s.s_status = '1'
AND (c.c_status = '1' or c.store_id <> '22')
AND DATE( c.c_end_date ) >= '2014-10-04'
ORDER BY c.c_id DESC
You need to move your "AND s.s_status = 1" clause to the LEFT JOIN part of your query. When you move it to the WHERE clause, it forces to an implied INNER JOIN and thus leaving out of you result set.
LEFT JOIN ci_stores s
ON c.store_id = s.store_id
AND s.s_status = '1'
FROM ci_cptbl c
LEFT JOIN ci_stores s ON s.store_id = c.store_id
The purpose of that LEFT JOIN is to allow unmatched records in the "joined to" table to be listed in the results
e.g. if there is a store 657 in ci_cptbl but no such store in ci_stores 657 would still be listed
However, using the WHERE clause:
if we then ALSO insist EVERY row has s.s_status = '1'
then store 657 would NOT be listed (how can it? it does not exist in ci_stores so ci_stores.s_status has to be NULL and can never ever be equal to 1
So; when using any outer join such as a LEFT OUTER JOIN you have to be very careful how you reference those tables in the WHERE clause. In many cases, such as here, it is better to move the additional condition(s) to the JOIN like this:
FROM ci_cptbl c
LEFT JOIN ci_stores s ON s.store_id = c.store_id
AND s.s_status = '1'
I have got a post table and it's schema is like this:
CREATE TABLE IF NOT EXISTS `post` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`user_id` bigint(20) DEFAULT NULL,
`site_id` bigint(20) DEFAULT NULL,
`parent_id` bigint(20) DEFAULT NULL,
`title` longtext COLLATE utf8_turkish_ci NOT NULL,
`status` varchar(20) COLLATE utf8_turkish_ci NOT NULL,
PRIMARY KEY (`id`),
KEY `IDX_5A8A6C8DA76ED395` (`user_id`),
KEY `IDX_5A8A6C8DF6BD1646` (`site_id`),
KEY `IDX_5A8A6C8D727ACA70` (`parent_id`),
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_turkish_ci AUTO_INCREMENT=16620 ;
I'm using this DQL to fetch a post and it's children:
$post = $this->_diContainer->wordy_app_doctrine->fetch(
"SELECT p,c FROM Wordy\Entity\Post p LEFT JOIN p.children c WHERE p.site = :site AND p.id = :id AND p.language = :language AND p.status != 'trashed' AND c.status != 'trashed' ORDER BY c.title",
array(
'params' => array(
'id' => $id,
'site' => $this->_currentSite['id'],
'language' => $this->_currentLanguage->code,
)
)
);
What i'm trying to do is: Fetch a post and all of it's children. The criteria is, don't include trashed posts or trashed children.
But when i run this query with a post which doesn't even have children, the returned result set is empty.
When i remove the c.status != 'trashed' part from query, everything works fine but i will get trashed posts too.
Thanks, in advance.
edit: here is the SQL output of given DQL:
SELECT p0_.id AS id0, p0_.title AS title5, p0_.status AS status8, p0_.parent_id AS parent_id9, p1_.id AS id15, p1_.title AS title20, p1_.status AS status23, p1_.parent_id AS parent_id24 FROM post p0_ LEFT JOIN post p1_ ON p0_.id = p1_.parent_id WHERE p0_.site_id = ? AND p0_.id = ? AND p0_.language = ? AND p0_.status <> 'trashed' ORDER BY p1_.title ASC
You forgot the ON clause of the left join syntax.
Your long conditional WHERE is being AND'ed together after the join and nothing is passing all the checks. Move the join stuff into the ON clause and leave the c.trashed check in the WHERE clause. Try something like this:
SELECT p,c FROM Wordy\Entity\Post p LEFT JOIN children c ON p.site = :site AND p.id = :id AND p.language = :language AND p.status != 'trashed' AND p.id = c.parent_id WHERE c.status != 'trashed' ORDER BY c.title
I think i solved my own problem.
Just use a with clause on join field instead of where clause, like this:
"SELECT p,c FROM Wordy\Entity\Post p LEFT JOIN p.children c WITH c.status != 'trashed' WHERE p.site = :site........."