Optimize SQL Query with GROUP BY - mysql

I need help with a query running slow when using group by:
SELECT `customers`.`id`
,`customers`.`firstname`
,`customers`.`lastname`
,`customers`.`address`
,`customers`.`address_co`
,`customers`.`country`
,`customers`.`zipcode`
,`customers`.`city`
,`customers`.`phone`
,`customers`.`mobilephone`
,`customers`.`ssn`
,`users`.`email`
,`customer_lists`.`name` AS `customerList`
,COUNT( transactions.id ) AS transactions_count
,SUM( transactions.sum_incl ) AS sum_incl
,SUM( transactions.sum_excl ) AS sum_excl
,`customers`.`created_at`
FROM `customers`
INNER JOIN `users`
ON `customers`.`user_id` = `users`.`id`
LEFT JOIN `transactions`
ON `customers`.`id` = `transactions`.`customer_id`
INNER JOIN `customer_lists`
ON `customers`.`customer_list_id` = `customer_lists`.`id`
WHERE `customer_lists`.`club_id` = '1'
GROUP BY `customers`.`id`
ORDER BY `customers`.`created_at` DESC
When I run a EXPLAIN it says Using temporary; Using filesort.
**id** **select_type** **table** **type** **possible_keys** **key** **key_len** **ref** **rows** **Extra**
1 SIMPLE customer_lists ref PRIMARY,club_id_index club_id_index 8 const 1 Using temporary; Using filesort
1 SIMPLE customers ref customers_user_id_foreign,customers_customer_list_... customers_customer_list_id_index 8 kund_workbox_nu.customer_lists.id 867
1 SIMPLE users eq_ref PRIMARY PRIMARY 8 kund_workbox_nu.customers.user_id 1
1 SIMPLE transactions ref customer_id_index customer_id_index 8 kund_workbox_nu.customers.id 3108
How do I fix that?
UPDATE
Tables:
Customers
CREATE TABLE `customers` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`user_id` bigint(20) unsigned NOT NULL,
`customer_list_id` bigint(20) unsigned NOT NULL,
`firstname` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`lastname` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`address` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`address_co` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`country` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`zipcode` varchar(10) COLLATE utf8_unicode_ci NOT NULL,
`city` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`phone` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
`mobilephone` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
`ssn` varchar(15) COLLATE utf8_unicode_ci NOT NULL,
`last_login` datetime NOT NULL,
`added_information` tinyint(1) NOT NULL DEFAULT '0',
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`id`),
KEY `customers_user_id_foreign` (`user_id`),
KEY `customers_customer_list_id_index` (`customer_list_id`),
CONSTRAINT `customers_customer_list_id_foreign` FOREIGN KEY (`customer_list_id`) REFERENCES `customer_lists` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `customers_user_id_foreign` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=29739 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Transactions
CREATE TABLE `transactions` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`transaction_type_id` bigint(20) unsigned NOT NULL,
`customer_id` bigint(20) unsigned NOT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`sum_incl` decimal(15,4) NOT NULL,
`sum_excl` decimal(15,4) NOT NULL,
PRIMARY KEY (`id`),
KEY `transactions_customer_id_foreign` (`customer_id`),
KEY `transactions_transaction_type_id_foreign` (`transaction_type_id`),
CONSTRAINT `transactions_customer_id_foreign` FOREIGN KEY (`customer_id`) REFERENCES `customers` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `transactions_transaction_type_id_foreign` FOREIGN KEY (`transaction_type_id`) REFERENCES `transaction_types` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=580116 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Customer Lists
CREATE TABLE `customer_lists` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`club_id` bigint(20) unsigned NOT NULL,
`name` varchar(90) COLLATE utf8_unicode_ci NOT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`id`),
KEY `customer_lists_club_id_foreign` (`club_id`),
CONSTRAINT `customer_lists_club_id_foreign` FOREIGN KEY (`club_id`) REFERENCES `clubs` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=11 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

Perhaps try a single table grouping on transactions as a derived table?
SELECT
`customers`.`id`
, `customers`.`firstname`
, `customers`.`lastname`
, `customers`.`address`
, `customers`.`address_co`
, `customers`.`country`
, `customers`.`zipcode`
, `customers`.`city`
, `customers`.`phone`
, `customers`.`mobilephone`
, `customers`.`ssn`
, `users`.`email`
, `customer_lists`.`name` AS `customerList`
, `trans`.transactions_count
, `trans`.sum_incl
, `trans`.sum_excl
, `customers`.`created_at`
FROM `customers`
INNER JOIN `users` ON `customers`.`user_id` = `users`.`id`
INNER JOIN `customer_lists` ON `customers`.`customer_list_id` = `customer_lists`.`id`
LEFT JOIN (
SELECT
`transactions`.`customer_id`
, COUNT(transactions.id) AS transactions_count
, SUM(transactions.sum_incl) AS sum_incl
, SUM(transactions.sum_excl) AS sum_excl
FROM `transactions`
GROUP BY `transactions`.`customer_id`
) `trans` ON `customers`.`id` = `trans`.`customer_id`
WHERE `customer_lists`.`club_id` = '1'
ORDER BY `customers`.`created_at` DESC
A where clause in that subquery may also help.

Have you tried to create indexes on all the JOIN columns, i.e. customers.user_id, users.id, transactions.customer_id, ... ?

Related

How can write the sql for this use case ? I have to execute with recursion

USE CASE
IF strore_id IS > 0 THEN feteh that record along WITH rest of records AND NOT include entity_id = 1425 AND store_id = 0
SQL WHERE criteria
store_id = 1 AND attribute_id = 1 AND entity_id = 1425
CREATE TABLE `eav_categories_varchar` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`store_id` bigint(20) unsigned NOT NULL,
`entity_id` bigint(20) unsigned NOT NULL,
`attribute_id` bigint(20) unsigned NOT NULL,
`value` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
`deleted_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `store_id` (`store_id`),
KEY `entity_id` (`entity_id`),
KEY `attribute_id` (`attribute_id`),
CONSTRAINT `eav_categories_varchar_attribute_id_foreign` FOREIGN KEY (`attribute_id`) REFERENCES `eav_attributes` (`id`) ON DELETE CASCADE,
CONSTRAINT `eav_categories_varchar_entity_id_foreign` FOREIGN KEY (`entity_id`) REFERENCES `categories` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=643 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
SELECT
c2.*
FROM
`categories`
INNER JOIN `eav_categories_varchar` AS `c2`
ON `c2`.`entity_id` = `categories`.`id`
WHERE `c2`.`store_id` IN IF((SELECT COUNT(*) FROM eav_categories_varchar WHERE `store_id` = 0) > 0, 0,0)

MySql unexpected query result

I have such a tables offers and offer_operating_systems related by offer_id. I made two queris and from my understanding they should return same result, but they don't
Query 1:
select
count(*)
from
`offers`
where
(
select count(*)
from `offer_operating_systems`
where
`offer_operating_systems`.`offer_id` = `offers`.`id`
and
`operating_system` = 'android'
) = 1
order by `id` asc
Query 2:
select
count(*)
from offer_operating_systems
where
operating_system = 'android'
order by `id` asc
Can someone explain to me why the results are not the same? Thanks!
EDITED
operating_systemcolumn is unique so each offer can have only one record with adnroid
Table structures
CREATE TABLE `offers` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`type` int(11) NOT NULL DEFAULT '0',
`url` text COLLATE utf8_unicode_ci,
`status` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'pending',
`api_created_at` timestamp NULL DEFAULT NULL,
`api_updated_at` timestamp NULL DEFAULT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`id`),
KEY `offers_status_index` (`status`)
) ENGINE=InnoDB AUTO_INCREMENT=423 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
CREATE TABLE `offer_operating_systems` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`offer_id` int(10) unsigned NOT NULL,
`operating_system` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`id`),
UNIQUE KEY `offer_operating_systems_offer_id_operating_system_unique` (`offer_id`,`operating_system`),
KEY `offer_operating_systems_offer_id_foreign` (`offer_id`),
CONSTRAINT `offer_operating_systems_offer_id_foreign` FOREIGN KEY (`offer_id`) REFERENCES `offers` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=728 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci

DATE_ADD in a subquery slows down execution

Good day.
Why date modification in a subquery reduces performance?
This request is fast (0.015 sec):
SELECT p.id, p.name, s.value
FROM points p
LEFT JOIN settings s ON p.id = s.point_id AND s.type_id = 1
WHERE p.parent_id = 1310
AND NOT EXISTS (SELECT 1 FROM events e WHERE e.point_id = p.id AND e.start_at > '2019-09-09 20:00:00')
UPD Explain:
'1', 'PRIMARY', 'p', 'ref', 'idx_parent_id_type', 'idx_parent_id_type', '5', 'const', '29', 'Using where'
'1', 'PRIMARY', 's', 'ref', 'p_id_idx,sid_idx', 'p_id_idx', '4', 'rm_api.p.id', '1', 'Using where'
'2', 'MATERIALIZED', 'e', 'range', 'idx_start_at_end_at,point', 'idx_start_at_end_at', '5', NULL, '3855', 'Using index condition'
This request is slow (~ 18 sec):
SELECT p.id, p.name, s.value
FROM points p
LEFT JOIN settings s ON p.id = s.point_id AND s.type_id = 1
WHERE p.parent_id = 1310
AND NOT EXISTS (SELECT 1 FROM events e WHERE e.point_id = p.id AND e.start_at > DATE_ADD('2019-09-09 20:00:00', INTERVAL COALESCE(s.value, 0) MINUTE))
UPD Explain:
1 PRIMARY p ref idx_parent_id_type idx_parent_id_type 5 const 29
1 PRIMARY s ref p_id_idx,sid_idx p_id_idx 4 rm_api.p.id 1 Using where
2 DEPENDENT SUBQUERY e ref idx_start_at_end_at,point point 4 rm_api.p.id 478 Using index condition; Using where
What can be done?
I doubt that I am using s.value correctly in the subquery
p.s. this request is also fast:
SELECT p.id, p.name, s.value
FROM points p
LEFT JOIN settins s ON p.id = s.point_id AND s.type_id = 1
WHERE p.parent_id = 1310
AND NOT EXISTS (SELECT 1 FROM events e WHERE e.point_id = p.id AND e.start_at > DATE_ADD('2019-09-09 20:00:00', INTERVAL 15 MINUTE))
UPD
Table points:
CREATE TABLE `points` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(160) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'Не указано',
`parent_id` int(10) unsigned DEFAULT 1,
`type` int(11) NOT NULL DEFAULT 0,
`status` int(11) NOT NULL DEFAULT 1,
`created_at` timestamp NOT NULL DEFAULT '1970-12-31 09:00:00',
`updated_at` timestamp NOT NULL DEFAULT '1970-12-31 09:00:00',
`capacity` int(11) NOT NULL DEFAULT 0,
`is_building` int(11) DEFAULT 0,
`is_ews_linked` int(11) NOT NULL DEFAULT 0,
`ews_resource_name` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`map_name` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`other` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`human_name` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`booking_max_duration` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `idx_parent_id_type` (`parent_id`,`type`)
) ENGINE=InnoDB AUTO_INCREMENT=2958 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Table settings:
CREATE TABLE `points_settins_types_storage` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`point_id` int(11) unsigned NOT NULL,
`settings_prop_type_id` int(11) unsigned NOT NULL,
`value` varchar(255) CHARACTER SET utf8 NOT NULL DEFAULT '',
PRIMARY KEY (`id`,`settings_prop_type_id`,`point_id`),
KEY `p_id_idx` (`point_id`),
KEY `sid_idx` (`settings_prop_type_id`),
CONSTRAINT `pid` FOREIGN KEY (`point_id`) REFERENCES `points` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `sid` FOREIGN KEY (`settings_prop_type_id`) REFERENCES `points_settings_types` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=1062 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Table events:
CREATE TABLE `events` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`start_at` datetime NOT NULL,
`end_at` datetime NOT NULL,
`point_id` int(11) NOT NULL,
`status` int(11) NOT NULL DEFAULT 0,
`owner_id` int(11) NOT NULL,
`approver_id` int(11) DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`date` datetime NOT NULL,
`approver_message` varchar(500) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`user_message` varchar(500) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`linked_vks_id` int(11) DEFAULT NULL,
`ews_event_id` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`is_from_exchange` int(11) NOT NULL DEFAULT 0,
`ews_event_change_key` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`content` varchar(10000) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
PRIMARY KEY (`id`),
UNIQUE KEY `id_UNIQUE` (`id`),
KEY `owner_id_k` (`owner_id`),
KEY `idx_start_at_end_at` (`start_at`,`end_at`),
KEY `end_start` (`end_at`,`start_at`) USING BTREE,
KEY `date` (`date`) USING BTREE,
KEY `status` (`status`) USING BTREE,
KEY `point` (`point_id`)
) ENGINE=InnoDB AUTO_INCREMENT=2338492 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
For your query, start with in index on events(point_id, start_at). Perhaps that will help.
Probably not.
If the interval values are always positive, then try this redundant version (with the above index):
WHERE p.parent_id = 1310 AND
NOT EXISTS (SELECT 1
FROM events e
WHERE e.point_id = p.id AND
e.start_at > '2019-09-09 20:00:00' AND
e.start_at > DATE_ADD('2019-09-09 20:00:00', INTERVAL COALESCE(s.value, 0) MINUTE)
)
Yes, this looks silly. But the additional condition should make it easier to MySQL to use the index correctly.

Optimize MySQL query used to find matches between two tables

On a project I'm working on, I have two tables:
consumption: Contains historical orders from customers with fields specifying the features of the product they have bought (one product per row)
product: Contains current product stock
The database engine is InnoDB.
Goals:
The application must show matches from both sides, I mean:
When I list current products stock, I want to show a column that displays how many historical orders match with a particular product
When I list the historical orders, I want to see how many products match with a particular historical order
Database structure for consumption and product tables plus other related tables:
CREATE TABLE `consumption` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`created_by_id` INT(11) NULL DEFAULT NULL,
`client_id` INT(11) NOT NULL,
`data_import_id` INT(11) NULL DEFAULT NULL,
`tmp_consumption_id` INT(11) NULL DEFAULT NULL,
`material_id` INT(11) NULL DEFAULT NULL,
`quality_id` INT(11) NULL DEFAULT NULL,
`thick` DECIMAL(10,3) NULL DEFAULT NULL,
`thick_max` DECIMAL(10,3) NULL DEFAULT NULL,
`width` DECIMAL(10,2) NULL DEFAULT NULL,
`width_max` DECIMAL(10,2) NULL DEFAULT NULL,
`long` INT(11) NULL DEFAULT NULL,
`long_max` INT(11) NULL DEFAULT NULL,
`purchase_price` DECIMAL(10,2) NULL DEFAULT NULL,
`sale_price` DECIMAL(10,2) NULL DEFAULT NULL,
`comments` VARCHAR(255) NULL DEFAULT NULL,
`annual_consumption` DECIMAL(10,3) NULL DEFAULT NULL,
`type` ENUM('consumption','request') NULL DEFAULT 'consumption',
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
`covering_grammage` VARCHAR(64) NULL DEFAULT NULL,
`asp_sup_acab` VARCHAR(64) NULL DEFAULT NULL,
PRIMARY KEY (`id`),
INDEX `fk_consumption_client1` (`client_id`),
INDEX `created_by_id` (`created_by_id`),
INDEX `material_id` (`material_id`),
INDEX `quality_id` (`quality_id`),
CONSTRAINT `consumption_ibfk_1` FOREIGN KEY (`material_id`) REFERENCES `material` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `consumption_ibfk_2` FOREIGN KEY (`quality_id`) REFERENCES `quality` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `fk_consumption_client1` FOREIGN KEY (`client_id`) REFERENCES `client` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=30673
;
CREATE TABLE `product` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`warehouse_id` INT(11) NULL DEFAULT NULL,
`created_by_id` INT(11) NULL DEFAULT NULL,
`data_import_id` INT(11) NULL DEFAULT NULL,
`tmp_product_id` INT(11) NULL DEFAULT NULL,
`code` VARCHAR(32) NOT NULL,
`material_id` INT(11) NULL DEFAULT NULL,
`quality_id` INT(11) NULL DEFAULT NULL,
`covering_id` INT(11) NULL DEFAULT NULL,
`finish_id` INT(11) NULL DEFAULT NULL,
`source` VARCHAR(128) NULL DEFAULT NULL,
`thickness` DECIMAL(10,3) NULL DEFAULT NULL,
`width` INT(11) NULL DEFAULT NULL,
`tons` DECIMAL(10,3) NULL DEFAULT NULL,
`re` INT(11) NULL DEFAULT NULL,
`rm` INT(11) NULL DEFAULT NULL,
`a_percent` INT(11) NULL DEFAULT NULL,
`comments` VARCHAR(255) NULL DEFAULT NULL,
`price` DECIMAL(10,2) NULL DEFAULT NULL,
`deleted` TINYINT(1) NOT NULL DEFAULT '0',
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `warehouse_id` (`warehouse_id`),
INDEX `material_id` (`material_id`),
INDEX `quality_id` (`quality_id`),
INDEX `covering_id` (`covering_id`),
INDEX `finish_id` (`finish_id`),
CONSTRAINT `product_ibfk_1` FOREIGN KEY (`material_id`) REFERENCES `material` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `product_ibfk_2` FOREIGN KEY (`quality_id`) REFERENCES `quality` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `product_ibfk_3` FOREIGN KEY (`covering_id`) REFERENCES `covering` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `product_ibfk_4` FOREIGN KEY (`finish_id`) REFERENCES `finish` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `product_ibfk_5` FOREIGN KEY (`warehouse_id`) REFERENCES `warehouse` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=740
;
CREATE TABLE `client` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`zone_id` INT(11) NULL DEFAULT NULL,
`zone2_id` INT(11) NULL DEFAULT NULL,
`code` VARCHAR(64) NOT NULL,
`business_name` VARCHAR(255) NULL DEFAULT NULL,
`fiscal_name` VARCHAR(255) NULL DEFAULT NULL,
`nif` VARCHAR(15) NULL DEFAULT NULL,
`contact_short_name` VARCHAR(128) NULL DEFAULT NULL,
`contact_full_name` VARCHAR(128) NULL DEFAULT NULL,
`email` VARCHAR(255) NULL DEFAULT NULL,
`group` VARCHAR(255) NULL DEFAULT NULL,
`status` TINYINT(1) NOT NULL DEFAULT '1',
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`),
INDEX `zone_id` (`zone_id`),
INDEX `zone2_id` (`zone2_id`),
CONSTRAINT `client_ibfk_1` FOREIGN KEY (`zone_id`) REFERENCES `zone` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=443
;
CREATE TABLE `client_group` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(15) NOT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code` (`code`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=49
;
CREATE TABLE `client_has_group` (
`client_id` INT(11) NOT NULL,
`group_id` INT(11) NOT NULL,
INDEX `client_id` (`client_id`),
INDEX `group_id` (`group_id`),
CONSTRAINT `client_has_group_ibfk_1` FOREIGN KEY (`client_id`) REFERENCES `client` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `client_has_group_ibfk_2` FOREIGN KEY (`group_id`) REFERENCES `client_group` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
;
CREATE TABLE `covering` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(128) NOT NULL,
`group` VARCHAR(128) NULL DEFAULT NULL,
`equivalence` VARCHAR(128) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=55
;
CREATE TABLE `finish` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(128) NOT NULL,
`group` VARCHAR(128) NULL DEFAULT NULL,
`equivalence` VARCHAR(128) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=42
;
CREATE TABLE `material` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(128) NOT NULL,
`group` VARCHAR(128) NULL DEFAULT NULL,
`equivalence` VARCHAR(128) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`),
INDEX `group` (`group`),
INDEX `equivalence` (`equivalence`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=46
;
CREATE TABLE `quality` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(128) NOT NULL,
`group` VARCHAR(128) NULL DEFAULT NULL,
`equivalence` VARCHAR(128) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`),
INDEX `group` (`group`),
INDEX `equivalence` (`equivalence`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=980
;
CREATE TABLE `user_filter` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`user_id` INT(11) NOT NULL,
`filter_type` ENUM('consumption','product') NOT NULL DEFAULT 'consumption',
`name` VARCHAR(255) NOT NULL,
`is_default` TINYINT(1) NOT NULL DEFAULT '0',
`client_status` TINYINT(1) NULL DEFAULT NULL,
`client_group` VARCHAR(45) NULL DEFAULT NULL,
`material` VARCHAR(15) NULL DEFAULT NULL,
`quality` VARCHAR(64) NULL DEFAULT NULL,
`thickness` VARCHAR(45) NULL DEFAULT NULL,
`width` VARCHAR(45) NULL DEFAULT NULL,
`tons` VARCHAR(45) NULL DEFAULT NULL,
`covering` VARCHAR(45) NULL DEFAULT NULL,
`finish` VARCHAR(45) NULL DEFAULT NULL,
`re` VARCHAR(45) NULL DEFAULT NULL,
`rm` VARCHAR(45) NULL DEFAULT NULL,
`a_percent` VARCHAR(45) NULL DEFAULT NULL,
`comments` VARCHAR(255) NULL DEFAULT NULL,
`price` VARCHAR(45) NULL DEFAULT NULL,
`warehouse` VARCHAR(45) NULL DEFAULT NULL,
`date` VARCHAR(45) NULL DEFAULT NULL,
`type` ENUM('consumption','request') NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `fk_user_filter_user1` (`user_id`),
INDEX `filter_type` (`filter_type`),
CONSTRAINT `fk_user_filter_user1` FOREIGN KEY (`user_id`) REFERENCES `user` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=5
;
CREATE TABLE `warehouse` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`name` VARCHAR(128) NOT NULL,
`zone_id` INT(11) NULL DEFAULT NULL,
`zone2_id` INT(11) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `zone_id` (`zone_id`),
INDEX `zone2_id` (`zone2_id`),
CONSTRAINT `warehouse_ibfk_1` FOREIGN KEY (`zone_id`) REFERENCES `zone` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=37
;
CREATE TABLE `zone` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`zone2_id` INT(11) NULL DEFAULT NULL,
`name` VARCHAR(128) NOT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `zone2_id` (`zone2_id`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=49
;
What I have done to be able to find matches between the two tables:
I have created a LEFT JOIN query between consumption and product table (that is also joining with additional tables if required).
Looks something like this:
SELECT cons.`id` as `consumption_id`, cons.`client_id` as `consumption_client_id`, cons.`material_id` as `consumption_material_id`, cons.`quality_id` as `consumption_quality_id`, cons.`thick` as `consumption_thick`, cons.`thick_max` as `consumption_thick_max`, cons.`width` as `consumption_width`, cons.`width_max` as `consumption_width_max`, cons.`long` as `consumption_long`, cons.`long_max` as `consumption_long_max`, cons.`type` as `consumption_type`, cons.`date_add` as `consumption_date_add`, prod.`id` as `product_id`, prod.`warehouse_id` as `product_warehouse_id`, prod.`code` as `product_code`, prod.`material_id` as `product_material_id`, prod.`quality_id` as `product_quality_id`, prod.`covering_id` as `product_covering_id`, prod.`finish_id` as `product_finish_id`, prod.`thickness` as `product_thickness`, prod.`width` as `product_width`, prod.`tons` as `product_tons`
FROM consumption cons
INNER JOIN client cli
ON cli.id=cons.client_id
LEFT JOIN client_has_group cli_gr
ON cli_gr.client_id=cons.client_id
LEFT JOIN product prod
ON
(
(cons.material_id=prod.material_id)
OR
prod.material_id IN (
SELECT id FROM material WHERE `equivalence`=(
SELECT `equivalence` FROM material WHERE id=cons.material_id
)
AND `group`=(
SELECT `group` FROM material WHERE id=cons.material_id
)
)
)
AND
(
(cons.quality_id=prod.quality_id)
OR
prod.quality_id IN (
SELECT id FROM quality WHERE `equivalence`=(
SELECT `equivalence` FROM quality WHERE id=cons.quality_id
)
AND `group`=(
SELECT `group` FROM quality WHERE id=cons.quality_id
)
)
)
AND (prod.thickness >= (cons.thick - 0.1) AND prod.thickness <= (cons.thick_max + 0.1))
AND (prod.width >= (cons.width - 1000) AND prod.width <= (cons.width_max + 1000))
WHERE 1 > 0 AND prod.deleted=0 AND cli.status=1 AND cons.date_add >= '2017-10-08 00:00:00'
GROUP BY cons.id, prod.id
When I want to list products and show matches of consumptions per every product, I have a main query that simply lists the products, then I join that query with the previous query from above and count the matches grouping by product id.
SELECT t.*,
count(f.consumption_id) AS matchesCount
FROM `product` t
LEFT JOIN (...previous query here...) f ON f.product_id=t.id
GROUP BY t.id
Other notes/considerations:
The application uses a couple of fields that have the same name in both tables, in order to find matches using ON in the JOIN
The application also uses more complex business logic, for example, product material can be equal or can be inside of an equivalence table or group
The user can save personal filters, that why it is used the user_filter table, so as a user I can have multiple "searches" saved and quickly switch from one to the other
The matches have to be displayed LIVE, I mean, calculated on the fly, not by any cronjob because user filter will always change
The amount of data the application will be working with right now will be about 35k rows in consumption table and about 1.5k rows in the product table
The server where the application is hosted is a dedicated server (64GB RAM) running MySQL
I had good performance with 3k rows of consumptions and 100 products, now with 10k+ consumption and 600 products, starting to get gateway timeout from nginx. Guess queries take too long.
I already know that if the ON cause has a lot of conditions it will work faster because the results sets are smaller, but if the condition is very wide, it will give a timeout, I guess the resulting rows are too many. Maybe the join will produce millions of rows.
What I'd like to ask is:
Am I on the right path in order to do "live matches" of data between both tables? is using the JOIN a good solution? I cannot think of another way to do it.
Apart from trying to optimize queries and indexes, are there any server tweaking I could do to take full advantage of server hardware?
Any other tips or techniques from someone who has done something similar in another project?
Update 1: Adding here full query for listing products with consumption matches:
SELECT t.*,
count(f.consumption_id) AS matchesCount
FROM `product` t
LEFT JOIN (
SELECT cons.`id` as `consumption_id`, cons.`client_id` as `consumption_client_id`, cons.`material_id` as `consumption_material_id`, cons.`quality_id` as `consumption_quality_id`, cons.`thick` as `consumption_thick`, cons.`thick_max` as `consumption_thick_max`, cons.`width` as `consumption_width`, cons.`width_max` as `consumption_width_max`, cons.`long` as `consumption_long`, cons.`long_max` as `consumption_long_max`, cons.`type` as `consumption_type`, cons.`date_add` as `consumption_date_add`, prod.`id` as `product_id`, prod.`warehouse_id` as `product_warehouse_id`, prod.`code` as `product_code`, prod.`material_id` as `product_material_id`, prod.`quality_id` as `product_quality_id`, prod.`covering_id` as `product_covering_id`, prod.`finish_id` as `product_finish_id`, prod.`thickness` as `product_thickness`, prod.`width` as `product_width`, prod.`tons` as `product_tons`
FROM consumption cons
INNER JOIN client cli
ON cli.id=cons.client_id
LEFT JOIN client_has_group cli_gr
ON cli_gr.client_id=cons.client_id
LEFT JOIN product prod
ON
(
(cons.material_id=prod.material_id)
OR
prod.material_id IN (
SELECT id FROM material WHERE `equivalence`=(
SELECT `equivalence` FROM material WHERE id=cons.material_id
)
AND `group`=(
SELECT `group` FROM material WHERE id=cons.material_id
)
)
)
WHERE 1 > 0 AND prod.deleted=0 AND cli.status=1 AND cons.date_add >= '2017-10-08 00:00:00'
GROUP BY cons.id, prod.id
) f ON f.product_id=t.id
GROUP BY t.id
Query time: 00:02:41 (+ 0,078 sec. network).
Note: The subquery JOIN run separately produces 600k rows. I'm thinking to try to group it somehow in order to make it smaller.
Update 2: Major improvement achieved by making the count inside the subquery and so reducing the result set used for the JOIN
Basically the subquery instead of returning 600k+ rows, it only returns as much rows as products or consumptions, depending what you're looking for. For that, the matchesCount has been moved inside the subquery instead of outside, and the group by has been changed, depending what list you want to display.
This is how the final queries look like right now:
List consumption and count products that match each consumption:
SELECT SQL_NO_CACHE `t`.*,
IFNULL(f.matchesCount, 0) AS matchesCount
FROM `consumption` `t`
LEFT JOIN
(SELECT cons.`id` AS `consumption_id`,
cons.`client_id` AS `consumption_client_id`,
cons.`material_id` AS `consumption_material_id`,
cons.`quality_id` AS `consumption_quality_id`,
cons.`thick` AS `consumption_thick`,
cons.`thick_max` AS `consumption_thick_max`,
cons.`width` AS `consumption_width`,
cons.`width_max` AS `consumption_width_max`,
cons.`long` AS `consumption_long`,
cons.`long_max` AS `consumption_long_max`,
cons.`type` AS `consumption_type`,
cons.`date_add` AS `consumption_date_add`,
prod.`id` AS `product_id`,
prod.`warehouse_id` AS `product_warehouse_id`,
prod.`code` AS `product_code`,
prod.`material_id` AS `product_material_id`,
prod.`quality_id` AS `product_quality_id`,
prod.`covering_id` AS `product_covering_id`,
prod.`finish_id` AS `product_finish_id`,
prod.`thickness` AS `product_thickness`,
prod.`width` AS `product_width`,
prod.`tons` AS `product_tons`,
count(prod.`id`) AS matchesCount
FROM consumption cons
INNER JOIN client cli ON cli.id=cons.client_id
LEFT JOIN product prod ON ((cons.material_id=prod.material_id)
OR prod.material_id IN
(SELECT id
FROM material
WHERE `equivalence`=
(SELECT `equivalence`
FROM material
WHERE id=cons.material_id )
AND `group`=
(SELECT `group`
FROM material
WHERE id=cons.material_id ) ))
AND ((cons.quality_id=prod.quality_id)
OR prod.quality_id IN
(SELECT id
FROM quality
WHERE `equivalence`=
(SELECT `equivalence`
FROM quality
WHERE id=cons.quality_id )
AND `group`=
(SELECT `group`
FROM quality
WHERE id=cons.quality_id ) ))
AND (prod.thickness >= (cons.thick - 0.1)
AND prod.thickness <= (cons.thick_max + 0.1))
AND (prod.width >= (cons.width - 1000)
AND prod.width <= (cons.width_max + 1000))
WHERE 1 > 0
AND prod.deleted=0
AND cli.status=1
AND cons.date_add >= '2017-10-08 00:00:00'
GROUP BY cons.id) f ON f.consumption_id=t.id
GROUP BY t.id
List products and count consumptions that match each product:
SELECT SQL_NO_CACHE t.*,
IFNULL(f.matchesCount, 0) AS matchesCount
FROM `product` `t`
LEFT JOIN
(SELECT cons.`id` AS `consumption_id`,
cons.`client_id` AS `consumption_client_id`,
cons.`material_id` AS `consumption_material_id`,
cons.`quality_id` AS `consumption_quality_id`,
cons.`thick` AS `consumption_thick`,
cons.`thick_max` AS `consumption_thick_max`,
cons.`width` AS `consumption_width`,
cons.`width_max` AS `consumption_width_max`,
cons.`long` AS `consumption_long`,
cons.`long_max` AS `consumption_long_max`,
cons.`type` AS `consumption_type`,
cons.`date_add` AS `consumption_date_add`,
prod.`id` AS `product_id`,
prod.`warehouse_id` AS `product_warehouse_id`,
prod.`code` AS `product_code`,
prod.`material_id` AS `product_material_id`,
prod.`quality_id` AS `product_quality_id`,
prod.`covering_id` AS `product_covering_id`,
prod.`finish_id` AS `product_finish_id`,
prod.`thickness` AS `product_thickness`,
prod.`width` AS `product_width`,
prod.`tons` AS `product_tons`,
count(cons.`id`) AS matchesCount
FROM consumption cons
INNER JOIN client cli ON cli.id=cons.client_id
LEFT JOIN product prod ON cons.material_id=prod.material_id
AND cons.quality_id=prod.quality_id
WHERE 1 > 0
AND prod.deleted=0
AND cli.status=1
GROUP BY prod.id) f ON f.product_id=t.id
WHERE deleted=0
GROUP BY t.id
Both queries take less than 1 second to execute (each).
Note: I still use the previous queries in my application, for example, when I want a break down of the list of products that match a single consumption, or the other way around. In that case I already add a filter per consumption id or product id that reduces the size of the result set a lot.
If client_has_group is "many:1", that is the wrong way to do it. You don't need the extra table.
INT is always 4 bytes. Consider smaller datatypes. Eventually the size of the database may add to your problems.
Do you really need date_add and date_upd. They seem like clutter that you will never use.
Avoid IN ( SELECT ... ) where practical. Switch to JOIN or EXISTS.
Why so many tables with code + group + equivalence? Could they be a single group? Do you need all 3 columns? Do you need id since code is UNIQUE? There comes a point where a schema is "over-normalized" and performance suffers without helping space much.
OR is a performance killer in some contexts.
"Correlated subqueries" are useful in some situations, but this one is probably better done via a JOIN:
AND `group` = ( SELECT `group` FROM quality WHERE id=cons.quality_id )
Beware of aggregates (eg, COUNT) with JOIN; you may be getting an inflated value. This is because the JOIN happens first.
why need
LEFT JOIN client_has_group cli_gr ON cli_gr.client_id=cons.client_id
it never used
why need GROUP BY cons.id, prod.id if you select all fields maybe select only what you need
try this select, i think it will be more faster
SELECT count(*), prod.*
FROM consumption cons
INNER JOIN client cli ON cli.id=cons.client_id
INNER JOIN material m ON m.id=cons.material_id
INNER JOIN quality q ON q.id=cons.quality_id
LEFT JOIN product prod
ON
(
(cons.material_id=prod.material_id)
OR
prod.material_id IN (
SELECT id FROM material WHERE `equivalence`=m.equivalence
AND `group`=m.group
)
)
AND
(
(cons.quality_id=prod.quality_id)
OR
prod.quality_id IN (
SELECT id FROM quality WHERE `equivalence`=q.equivalence
AND `group`=q.group
)
)
AND (prod.thickness >= (cons.thick - 0.1) AND prod.thickness <= (cons.thick_max + 0.1))
AND (prod.width >= (cons.width - 1000) AND prod.width <= (cons.width_max + 1000))
WHERE 1 > 0 AND prod.deleted=0 AND cli.status=1 AND cons.date_add >= '2017-10-08 00:00:00'
group by prod.id
maybe better do calculation count in background and add this field in product and consumption table.

MySQL join with Direct table or subfields of table

First query:
SELECT
u.id,
u.first_name,
u.last_name,
u.tazkera_id,
cu.relation_type,
CASE csp.`shift`
WHEN '1' THEN 'Morning'
WHEN '2' THEN 'Afternoon'
else 'Neither'
END AS shift
FROM `course_user` as cu
LEFT JOIN `courses` as c ON c.id = cu.`course_id`
LEFT JOIN (
select users.id,
users.first_name,
users.last_name,
users.tazkera_id
FROM `users`
) as u ON u.id = cu.`user_id`
left JOIN `course_schedule_prefs` as csp ON csp.`user_id` = cu.`user_id`
Where cu.relation_type = 1 group by cu.`user_id`;
Second Query:
SELECT
u.id,
u.first_name,
u.last_name,
u.tazkera_id,
cu.relation_type,
CASE csp.`shift`
WHEN '1' THEN 'Morning'
WHEN '2' THEN 'Afternoon'
else 'Neither'
END AS shift
FROM `course_user` as cu
LEFT JOIN `courses` as c ON c.id = cu.`course_id`
LEFT JOIN users as u ON u.id = cu.`user_id`
left JOIN `course_schedule_prefs` as csp ON csp.`user_id` = cu.`user_id`
Where cu.relation_type = 1 group by cu.`user_id`;
Tables :
users:
CREATE TABLE `users` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`first_name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`last_name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`email` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`password` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`is_password_reset` int(11) NOT NULL DEFAULT '0',
`login_ip` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`login_date` datetime DEFAULT NULL,
`last_login_ip` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`last_login_date` datetime DEFAULT NULL,
`is_email_address_verified` int(11) NOT NULL DEFAULT '0',
`failed_login_attempts` int(11) DEFAULT NULL,
`last_failed_login_date` datetime DEFAULT NULL,
`tazkera_id` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`province_code` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`district_code` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`village` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`home_address` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`promote_unique_user_id` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`remember_token` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`forgot_token` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`email_confirm_token` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`district_other` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`father_name` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`training_provider_id` int(10) unsigned DEFAULT NULL,
`is_profile_completed` int(11) DEFAULT NULL,
`active` int(11) DEFAULT '1',
`picture` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`small_picture` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`last_poll` bigint(20) DEFAULT NULL,
`last_password_reset` date DEFAULT NULL,
`presence_status` int(11) DEFAULT NULL,
`auto_presence` int(11) NOT NULL DEFAULT '0',
`last_activity` timestamp NULL DEFAULT NULL,
`trash` tinyint(4) NOT NULL DEFAULT '0',
`login_approval` tinyint(4) NOT NULL DEFAULT '0',
`lang` varchar(2) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'en',
`cover` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`small_cover` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`forum_reputation` int(11) NOT NULL DEFAULT '0',
`forum_suspension_date` date DEFAULT NULL,
`forum_status` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `users_email_unique` (`email`),
KEY `users_training_provider_id_foreign` (`training_provider_id`),
CONSTRAINT `users_training_provider_id_foreign` FOREIGN KEY (`training_provider_id`) REFERENCES `training_providers` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=1000038 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
course_users:
CREATE TABLE `course_user` (
`course_id` int(10) unsigned DEFAULT NULL,
`user_id` int(10) unsigned DEFAULT NULL,
`relation_type` int(11) DEFAULT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
KEY `course_user_course_id_foreign` (`course_id`),
KEY `course_user_user_id_foreign` (`user_id`),
CONSTRAINT `course_user_course_id_foreign` FOREIGN KEY (`course_id`) REFERENCES `courses` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `course_user_user_id_foreign` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
courses;
CREATE TABLE `courses` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`curriculum_id` int(10) unsigned DEFAULT NULL,
`training_center_id` int(10) unsigned DEFAULT NULL,
`training_coordinator_id` int(10) unsigned DEFAULT NULL,
`focal_point_id` int(10) unsigned DEFAULT NULL,
`start_date` date DEFAULT NULL,
`end_date` date DEFAULT NULL,
`start_time` time DEFAULT NULL,
`end_time` time DEFAULT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`shift` int(11) DEFAULT NULL,
`stage_id` int(10) unsigned DEFAULT NULL,
`stream_id` int(10) unsigned DEFAULT NULL,
`unit_id` int(10) unsigned DEFAULT NULL,
`master_trainer_id` int(10) unsigned DEFAULT NULL,
`status` int(11) DEFAULT NULL,
`pre_test_generated` int(11) NOT NULL DEFAULT '0',
`post_test_generated` int(11) NOT NULL DEFAULT '0',
`conduct_days` int(11) DEFAULT NULL,
`no` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`evaluation_available` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `courses_curriculum_id_foreign` (`curriculum_id`),
KEY `courses_training_coordinator_id_foreign` (`training_coordinator_id`),
KEY `courses_focal_point_id_foreign` (`focal_point_id`),
KEY `courses_stage_id_foreign` (`stage_id`),
KEY `courses_stream_id_foreign` (`stream_id`),
KEY `courses_unit_id_foreign` (`unit_id`),
KEY `courses_master_trainer_id_foreign` (`master_trainer_id`),
KEY `courses_training_center_id_foreign` (`training_center_id`),
CONSTRAINT `courses_curriculum_id_foreign` FOREIGN KEY (`curriculum_id`) REFERENCES `curriculums` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `courses_focal_point_id_foreign` FOREIGN KEY (`focal_point_id`) REFERENCES `users` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `courses_master_trainer_id_foreign` FOREIGN KEY (`master_trainer_id`) REFERENCES `users` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `courses_stage_id_foreign` FOREIGN KEY (`stage_id`) REFERENCES `stages` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `courses_stream_id_foreign` FOREIGN KEY (`stream_id`) REFERENCES `streams` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `courses_training_center_id_foreign` FOREIGN KEY (`training_center_id`) REFERENCES `training_centers` (`id`),
CONSTRAINT `courses_training_coordinator_id_foreign` FOREIGN KEY (`training_coordinator_id`) REFERENCES `users` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `courses_unit_id_foreign` FOREIGN KEY (`unit_id`) REFERENCES `units` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=139 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
course_schedule_prefs;
CREATE TABLE `course_schedule_prefs` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`cunduct_days` int(11) DEFAULT NULL,
`training_centers` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`shift` int(11) NOT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`user_id` int(10) unsigned DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `course_schedule_prefs_user_id_foreign` (`user_id`),
CONSTRAINT `course_schedule_prefs_user_id_foreign` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=24 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
My question is , which one of this query is good in term of performance and accuracy. Why ?
first query: I join directly with table; select all columns of table.
Second query: I used subquery inside of join; select only those element which I need it.
Note : My main question is about users table which is in join
There are too many possibilities here. There is no simple, 'straight', answer.
With LEFT JOIN (instead of JOIN), the Optimizer cannot start with the subquery, which it would usually like to do.
If there is a WHERE clause and it is useful for filtering rows, the Optimizer would like to start with the table that is filtered the most. Your WHERE clause smells like a flag, which is usually not very useful.
If a subquery has a GROUP BY or LIMIT, then it may significantly shrink the number of rows to work with. (Not in your case.) This might make the subquery approach better.
If cu has INDEX(relation_type, user_id), the Optimizer may decide that handling all of the WHERE and GROUP BY leads to the best bet, thereby starting with cu as the 'first' table.
In old versions of MySQL, subqueries in JOIN ( SELECT ... ) had no way of getting an index, thereby leading to inefficient table scans of that tmp table. In new versions (starting with 5.6, I think), the Optimizer takes the extra step (which costs something) to generate the optimal index ((id) in this case) after creating the tmp table and before JOINing it.
More
2 more table definitions needed.
Do you have the indexes I suggested?
InnoDB really needs a PRIMARY KEY - See many:many for advice on course_user.