I am trying to pull the data from two tables by joining the data, but the sql query executed is dead slow. The idea is to pull all users and then combine with the created_at date from points table. Pulling all users or all points is rather quick, but having problems with writing a proper join sql. I did try to add indexes to appropriate columns (points.created_at for example), but those made query even slower.
This is the code that generates the query:
return $this->user
->query()
->select(['users.id', 'users.email', 'users.role', 'users.created_at', 'users.updated_at', 'pt.created_at AS last_transaction'])
->leftJoin(DB::raw('(SELECT points.user_id, points.created_at FROM points ORDER BY points.created_at DESC) AS pt'), 'pt.user_id', '=', 'users.id')
->where('users.role', 'consument')
->groupBy('users.id');
Which generates this query:
select `users`.`id`, `users`.`email`, `users`.`role`, `users`.`created_at`, `users`.`updated_at`, `pt`.`created_at` as `last_transaction`
from `users`
left join (SELECT points.user_id, points.created_at FROM points ORDER BY points.created_at DESC) AS pt on `pt`.`user_id` = `users`.`id`
where `users`.`role` = ? and `users`.`deleted_at` is null
group by `users`.`id`
order by `id` asc
Users table:
CREATE TABLE `users` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`email` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`password` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`remember_token` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`role` varchar(15) COLLATE utf8_unicode_ci DEFAULT 'consument',
`created_at` timestamp NOT NULL DEFAULT current_timestamp(),
`updated_at` timestamp NOT NULL DEFAULT current_timestamp(),
`deleted_at` timestamp NULL DEFAULT NULL,
`email_verified_at` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`email_verify_token` text COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `users_email_unique` (`email`)
) ENGINE=InnoDB AUTO_INCREMENT=84345 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
Points table:
CREATE TABLE `points` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`user_id` int(10) unsigned NOT NULL,
`tablet_id` int(10) unsigned DEFAULT NULL,
`parent_company` int(10) unsigned NOT NULL,
`company_id` int(10) unsigned NOT NULL,
`points` int(10) unsigned NOT NULL,
`mutation_type` tinyint(3) unsigned NOT NULL,
`created_at` timestamp NOT NULL DEFAULT current_timestamp(),
`updated_at` timestamp NOT NULL DEFAULT current_timestamp(),
PRIMARY KEY (`id`),
KEY `points_user_id_foreign` (`user_id`),
KEY `points_company_id_foreign` (`company_id`),
KEY `points_parent_company_index` (`parent_company`),
KEY `points_tablet_id_index` (`tablet_id`),
KEY `points_mutation_type_company_id_created_at_index` (`mutation_type`,`company_id`,`created_at`),
KEY `created_at_user_id` (`created_at`,`user_id`),
CONSTRAINT `points_company_id_foreign` FOREIGN KEY (`company_id`) REFERENCES `companies` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `points_parent_company_foreign` FOREIGN KEY (`parent_company`) REFERENCES `parent_company` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `points_tablet_id_foreign` FOREIGN KEY (`tablet_id`) REFERENCES `tablets` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
CONSTRAINT `points_user_id_foreign` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=1798627 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
Users has 84,263 and points has 1,636,119 rows. If I execute the query manually through phpMyAdmin, it takes about 150 seconds to execute. if I ran it through Laravel, page times out after 180 seconds.
I can add or remove indexes and change the sql query, but I can't change the database structure, so any help with optimized sql query would be greatly appreciated.
If there is is only one row per user in the points table you do the following:
If there is users which does not have a post in points you could use:
select `users`.`id`, `users`.`email`, `users`.`role`, `users`.`created_at`,
`users`.`updated_at`, `pt`.`created_at` as `last_transaction`
from `users`
left join points AS pt on `pt`.`user_id` = `users`.`id`
where `users`.`role` = ? and `users`.`deleted_at` is null
order by `id` ASC
If every user in the user table always has one post in the points table you could skip the left join and just:
select `users`.`id`, `users`.`email`, `users`.`role`, `users`.`created_at`,
`users`.`updated_at`, `pt`.`created_at` as `last_transaction`
from `users`
join points AS pt on `pt`.`user_id` = `users`.`id`
where `users`.`role` = ? and `users`.`deleted_at` is null
order by `id` asc
This will return just one user and the latest point row based on the points column 'created_at'.
SELECT `u`.`id`,
`u`.`email`,
`u`.`role`,
`u`.`created_at`,
`u`.`updated_at`,
`pt`.`created_at` as `last_transaction`
from `users` u
LEFT join points AS pt on `pt`.`user_id` = `u`.`id`
LEFT JOIN (
SELECT user_id, MAX(created_at) AS mm FROM points GROUP BY user_id
) AS m ON m.user_id = pt.user_id
where `u`.`role` = ? and `u`.`deleted_at` is NULL AND m.mm = pt.created_at
order by `id` ASC;
On a project I'm working on, I have two tables:
consumption: Contains historical orders from customers with fields specifying the features of the product they have bought (one product per row)
product: Contains current product stock
The database engine is InnoDB.
Goals:
The application must show matches from both sides, I mean:
When I list current products stock, I want to show a column that displays how many historical orders match with a particular product
When I list the historical orders, I want to see how many products match with a particular historical order
Database structure for consumption and product tables plus other related tables:
CREATE TABLE `consumption` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`created_by_id` INT(11) NULL DEFAULT NULL,
`client_id` INT(11) NOT NULL,
`data_import_id` INT(11) NULL DEFAULT NULL,
`tmp_consumption_id` INT(11) NULL DEFAULT NULL,
`material_id` INT(11) NULL DEFAULT NULL,
`quality_id` INT(11) NULL DEFAULT NULL,
`thick` DECIMAL(10,3) NULL DEFAULT NULL,
`thick_max` DECIMAL(10,3) NULL DEFAULT NULL,
`width` DECIMAL(10,2) NULL DEFAULT NULL,
`width_max` DECIMAL(10,2) NULL DEFAULT NULL,
`long` INT(11) NULL DEFAULT NULL,
`long_max` INT(11) NULL DEFAULT NULL,
`purchase_price` DECIMAL(10,2) NULL DEFAULT NULL,
`sale_price` DECIMAL(10,2) NULL DEFAULT NULL,
`comments` VARCHAR(255) NULL DEFAULT NULL,
`annual_consumption` DECIMAL(10,3) NULL DEFAULT NULL,
`type` ENUM('consumption','request') NULL DEFAULT 'consumption',
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
`covering_grammage` VARCHAR(64) NULL DEFAULT NULL,
`asp_sup_acab` VARCHAR(64) NULL DEFAULT NULL,
PRIMARY KEY (`id`),
INDEX `fk_consumption_client1` (`client_id`),
INDEX `created_by_id` (`created_by_id`),
INDEX `material_id` (`material_id`),
INDEX `quality_id` (`quality_id`),
CONSTRAINT `consumption_ibfk_1` FOREIGN KEY (`material_id`) REFERENCES `material` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `consumption_ibfk_2` FOREIGN KEY (`quality_id`) REFERENCES `quality` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `fk_consumption_client1` FOREIGN KEY (`client_id`) REFERENCES `client` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=30673
;
CREATE TABLE `product` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`warehouse_id` INT(11) NULL DEFAULT NULL,
`created_by_id` INT(11) NULL DEFAULT NULL,
`data_import_id` INT(11) NULL DEFAULT NULL,
`tmp_product_id` INT(11) NULL DEFAULT NULL,
`code` VARCHAR(32) NOT NULL,
`material_id` INT(11) NULL DEFAULT NULL,
`quality_id` INT(11) NULL DEFAULT NULL,
`covering_id` INT(11) NULL DEFAULT NULL,
`finish_id` INT(11) NULL DEFAULT NULL,
`source` VARCHAR(128) NULL DEFAULT NULL,
`thickness` DECIMAL(10,3) NULL DEFAULT NULL,
`width` INT(11) NULL DEFAULT NULL,
`tons` DECIMAL(10,3) NULL DEFAULT NULL,
`re` INT(11) NULL DEFAULT NULL,
`rm` INT(11) NULL DEFAULT NULL,
`a_percent` INT(11) NULL DEFAULT NULL,
`comments` VARCHAR(255) NULL DEFAULT NULL,
`price` DECIMAL(10,2) NULL DEFAULT NULL,
`deleted` TINYINT(1) NOT NULL DEFAULT '0',
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `warehouse_id` (`warehouse_id`),
INDEX `material_id` (`material_id`),
INDEX `quality_id` (`quality_id`),
INDEX `covering_id` (`covering_id`),
INDEX `finish_id` (`finish_id`),
CONSTRAINT `product_ibfk_1` FOREIGN KEY (`material_id`) REFERENCES `material` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `product_ibfk_2` FOREIGN KEY (`quality_id`) REFERENCES `quality` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `product_ibfk_3` FOREIGN KEY (`covering_id`) REFERENCES `covering` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `product_ibfk_4` FOREIGN KEY (`finish_id`) REFERENCES `finish` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `product_ibfk_5` FOREIGN KEY (`warehouse_id`) REFERENCES `warehouse` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=740
;
CREATE TABLE `client` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`zone_id` INT(11) NULL DEFAULT NULL,
`zone2_id` INT(11) NULL DEFAULT NULL,
`code` VARCHAR(64) NOT NULL,
`business_name` VARCHAR(255) NULL DEFAULT NULL,
`fiscal_name` VARCHAR(255) NULL DEFAULT NULL,
`nif` VARCHAR(15) NULL DEFAULT NULL,
`contact_short_name` VARCHAR(128) NULL DEFAULT NULL,
`contact_full_name` VARCHAR(128) NULL DEFAULT NULL,
`email` VARCHAR(255) NULL DEFAULT NULL,
`group` VARCHAR(255) NULL DEFAULT NULL,
`status` TINYINT(1) NOT NULL DEFAULT '1',
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`),
INDEX `zone_id` (`zone_id`),
INDEX `zone2_id` (`zone2_id`),
CONSTRAINT `client_ibfk_1` FOREIGN KEY (`zone_id`) REFERENCES `zone` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=443
;
CREATE TABLE `client_group` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(15) NOT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code` (`code`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=49
;
CREATE TABLE `client_has_group` (
`client_id` INT(11) NOT NULL,
`group_id` INT(11) NOT NULL,
INDEX `client_id` (`client_id`),
INDEX `group_id` (`group_id`),
CONSTRAINT `client_has_group_ibfk_1` FOREIGN KEY (`client_id`) REFERENCES `client` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `client_has_group_ibfk_2` FOREIGN KEY (`group_id`) REFERENCES `client_group` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
;
CREATE TABLE `covering` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(128) NOT NULL,
`group` VARCHAR(128) NULL DEFAULT NULL,
`equivalence` VARCHAR(128) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=55
;
CREATE TABLE `finish` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(128) NOT NULL,
`group` VARCHAR(128) NULL DEFAULT NULL,
`equivalence` VARCHAR(128) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=42
;
CREATE TABLE `material` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(128) NOT NULL,
`group` VARCHAR(128) NULL DEFAULT NULL,
`equivalence` VARCHAR(128) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`),
INDEX `group` (`group`),
INDEX `equivalence` (`equivalence`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=46
;
CREATE TABLE `quality` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(128) NOT NULL,
`group` VARCHAR(128) NULL DEFAULT NULL,
`equivalence` VARCHAR(128) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`),
INDEX `group` (`group`),
INDEX `equivalence` (`equivalence`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=980
;
CREATE TABLE `user_filter` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`user_id` INT(11) NOT NULL,
`filter_type` ENUM('consumption','product') NOT NULL DEFAULT 'consumption',
`name` VARCHAR(255) NOT NULL,
`is_default` TINYINT(1) NOT NULL DEFAULT '0',
`client_status` TINYINT(1) NULL DEFAULT NULL,
`client_group` VARCHAR(45) NULL DEFAULT NULL,
`material` VARCHAR(15) NULL DEFAULT NULL,
`quality` VARCHAR(64) NULL DEFAULT NULL,
`thickness` VARCHAR(45) NULL DEFAULT NULL,
`width` VARCHAR(45) NULL DEFAULT NULL,
`tons` VARCHAR(45) NULL DEFAULT NULL,
`covering` VARCHAR(45) NULL DEFAULT NULL,
`finish` VARCHAR(45) NULL DEFAULT NULL,
`re` VARCHAR(45) NULL DEFAULT NULL,
`rm` VARCHAR(45) NULL DEFAULT NULL,
`a_percent` VARCHAR(45) NULL DEFAULT NULL,
`comments` VARCHAR(255) NULL DEFAULT NULL,
`price` VARCHAR(45) NULL DEFAULT NULL,
`warehouse` VARCHAR(45) NULL DEFAULT NULL,
`date` VARCHAR(45) NULL DEFAULT NULL,
`type` ENUM('consumption','request') NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `fk_user_filter_user1` (`user_id`),
INDEX `filter_type` (`filter_type`),
CONSTRAINT `fk_user_filter_user1` FOREIGN KEY (`user_id`) REFERENCES `user` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=5
;
CREATE TABLE `warehouse` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`name` VARCHAR(128) NOT NULL,
`zone_id` INT(11) NULL DEFAULT NULL,
`zone2_id` INT(11) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `zone_id` (`zone_id`),
INDEX `zone2_id` (`zone2_id`),
CONSTRAINT `warehouse_ibfk_1` FOREIGN KEY (`zone_id`) REFERENCES `zone` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=37
;
CREATE TABLE `zone` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`zone2_id` INT(11) NULL DEFAULT NULL,
`name` VARCHAR(128) NOT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `zone2_id` (`zone2_id`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=49
;
What I have done to be able to find matches between the two tables:
I have created a LEFT JOIN query between consumption and product table (that is also joining with additional tables if required).
Looks something like this:
SELECT cons.`id` as `consumption_id`, cons.`client_id` as `consumption_client_id`, cons.`material_id` as `consumption_material_id`, cons.`quality_id` as `consumption_quality_id`, cons.`thick` as `consumption_thick`, cons.`thick_max` as `consumption_thick_max`, cons.`width` as `consumption_width`, cons.`width_max` as `consumption_width_max`, cons.`long` as `consumption_long`, cons.`long_max` as `consumption_long_max`, cons.`type` as `consumption_type`, cons.`date_add` as `consumption_date_add`, prod.`id` as `product_id`, prod.`warehouse_id` as `product_warehouse_id`, prod.`code` as `product_code`, prod.`material_id` as `product_material_id`, prod.`quality_id` as `product_quality_id`, prod.`covering_id` as `product_covering_id`, prod.`finish_id` as `product_finish_id`, prod.`thickness` as `product_thickness`, prod.`width` as `product_width`, prod.`tons` as `product_tons`
FROM consumption cons
INNER JOIN client cli
ON cli.id=cons.client_id
LEFT JOIN client_has_group cli_gr
ON cli_gr.client_id=cons.client_id
LEFT JOIN product prod
ON
(
(cons.material_id=prod.material_id)
OR
prod.material_id IN (
SELECT id FROM material WHERE `equivalence`=(
SELECT `equivalence` FROM material WHERE id=cons.material_id
)
AND `group`=(
SELECT `group` FROM material WHERE id=cons.material_id
)
)
)
AND
(
(cons.quality_id=prod.quality_id)
OR
prod.quality_id IN (
SELECT id FROM quality WHERE `equivalence`=(
SELECT `equivalence` FROM quality WHERE id=cons.quality_id
)
AND `group`=(
SELECT `group` FROM quality WHERE id=cons.quality_id
)
)
)
AND (prod.thickness >= (cons.thick - 0.1) AND prod.thickness <= (cons.thick_max + 0.1))
AND (prod.width >= (cons.width - 1000) AND prod.width <= (cons.width_max + 1000))
WHERE 1 > 0 AND prod.deleted=0 AND cli.status=1 AND cons.date_add >= '2017-10-08 00:00:00'
GROUP BY cons.id, prod.id
When I want to list products and show matches of consumptions per every product, I have a main query that simply lists the products, then I join that query with the previous query from above and count the matches grouping by product id.
SELECT t.*,
count(f.consumption_id) AS matchesCount
FROM `product` t
LEFT JOIN (...previous query here...) f ON f.product_id=t.id
GROUP BY t.id
Other notes/considerations:
The application uses a couple of fields that have the same name in both tables, in order to find matches using ON in the JOIN
The application also uses more complex business logic, for example, product material can be equal or can be inside of an equivalence table or group
The user can save personal filters, that why it is used the user_filter table, so as a user I can have multiple "searches" saved and quickly switch from one to the other
The matches have to be displayed LIVE, I mean, calculated on the fly, not by any cronjob because user filter will always change
The amount of data the application will be working with right now will be about 35k rows in consumption table and about 1.5k rows in the product table
The server where the application is hosted is a dedicated server (64GB RAM) running MySQL
I had good performance with 3k rows of consumptions and 100 products, now with 10k+ consumption and 600 products, starting to get gateway timeout from nginx. Guess queries take too long.
I already know that if the ON cause has a lot of conditions it will work faster because the results sets are smaller, but if the condition is very wide, it will give a timeout, I guess the resulting rows are too many. Maybe the join will produce millions of rows.
What I'd like to ask is:
Am I on the right path in order to do "live matches" of data between both tables? is using the JOIN a good solution? I cannot think of another way to do it.
Apart from trying to optimize queries and indexes, are there any server tweaking I could do to take full advantage of server hardware?
Any other tips or techniques from someone who has done something similar in another project?
Update 1: Adding here full query for listing products with consumption matches:
SELECT t.*,
count(f.consumption_id) AS matchesCount
FROM `product` t
LEFT JOIN (
SELECT cons.`id` as `consumption_id`, cons.`client_id` as `consumption_client_id`, cons.`material_id` as `consumption_material_id`, cons.`quality_id` as `consumption_quality_id`, cons.`thick` as `consumption_thick`, cons.`thick_max` as `consumption_thick_max`, cons.`width` as `consumption_width`, cons.`width_max` as `consumption_width_max`, cons.`long` as `consumption_long`, cons.`long_max` as `consumption_long_max`, cons.`type` as `consumption_type`, cons.`date_add` as `consumption_date_add`, prod.`id` as `product_id`, prod.`warehouse_id` as `product_warehouse_id`, prod.`code` as `product_code`, prod.`material_id` as `product_material_id`, prod.`quality_id` as `product_quality_id`, prod.`covering_id` as `product_covering_id`, prod.`finish_id` as `product_finish_id`, prod.`thickness` as `product_thickness`, prod.`width` as `product_width`, prod.`tons` as `product_tons`
FROM consumption cons
INNER JOIN client cli
ON cli.id=cons.client_id
LEFT JOIN client_has_group cli_gr
ON cli_gr.client_id=cons.client_id
LEFT JOIN product prod
ON
(
(cons.material_id=prod.material_id)
OR
prod.material_id IN (
SELECT id FROM material WHERE `equivalence`=(
SELECT `equivalence` FROM material WHERE id=cons.material_id
)
AND `group`=(
SELECT `group` FROM material WHERE id=cons.material_id
)
)
)
WHERE 1 > 0 AND prod.deleted=0 AND cli.status=1 AND cons.date_add >= '2017-10-08 00:00:00'
GROUP BY cons.id, prod.id
) f ON f.product_id=t.id
GROUP BY t.id
Query time: 00:02:41 (+ 0,078 sec. network).
Note: The subquery JOIN run separately produces 600k rows. I'm thinking to try to group it somehow in order to make it smaller.
Update 2: Major improvement achieved by making the count inside the subquery and so reducing the result set used for the JOIN
Basically the subquery instead of returning 600k+ rows, it only returns as much rows as products or consumptions, depending what you're looking for. For that, the matchesCount has been moved inside the subquery instead of outside, and the group by has been changed, depending what list you want to display.
This is how the final queries look like right now:
List consumption and count products that match each consumption:
SELECT SQL_NO_CACHE `t`.*,
IFNULL(f.matchesCount, 0) AS matchesCount
FROM `consumption` `t`
LEFT JOIN
(SELECT cons.`id` AS `consumption_id`,
cons.`client_id` AS `consumption_client_id`,
cons.`material_id` AS `consumption_material_id`,
cons.`quality_id` AS `consumption_quality_id`,
cons.`thick` AS `consumption_thick`,
cons.`thick_max` AS `consumption_thick_max`,
cons.`width` AS `consumption_width`,
cons.`width_max` AS `consumption_width_max`,
cons.`long` AS `consumption_long`,
cons.`long_max` AS `consumption_long_max`,
cons.`type` AS `consumption_type`,
cons.`date_add` AS `consumption_date_add`,
prod.`id` AS `product_id`,
prod.`warehouse_id` AS `product_warehouse_id`,
prod.`code` AS `product_code`,
prod.`material_id` AS `product_material_id`,
prod.`quality_id` AS `product_quality_id`,
prod.`covering_id` AS `product_covering_id`,
prod.`finish_id` AS `product_finish_id`,
prod.`thickness` AS `product_thickness`,
prod.`width` AS `product_width`,
prod.`tons` AS `product_tons`,
count(prod.`id`) AS matchesCount
FROM consumption cons
INNER JOIN client cli ON cli.id=cons.client_id
LEFT JOIN product prod ON ((cons.material_id=prod.material_id)
OR prod.material_id IN
(SELECT id
FROM material
WHERE `equivalence`=
(SELECT `equivalence`
FROM material
WHERE id=cons.material_id )
AND `group`=
(SELECT `group`
FROM material
WHERE id=cons.material_id ) ))
AND ((cons.quality_id=prod.quality_id)
OR prod.quality_id IN
(SELECT id
FROM quality
WHERE `equivalence`=
(SELECT `equivalence`
FROM quality
WHERE id=cons.quality_id )
AND `group`=
(SELECT `group`
FROM quality
WHERE id=cons.quality_id ) ))
AND (prod.thickness >= (cons.thick - 0.1)
AND prod.thickness <= (cons.thick_max + 0.1))
AND (prod.width >= (cons.width - 1000)
AND prod.width <= (cons.width_max + 1000))
WHERE 1 > 0
AND prod.deleted=0
AND cli.status=1
AND cons.date_add >= '2017-10-08 00:00:00'
GROUP BY cons.id) f ON f.consumption_id=t.id
GROUP BY t.id
List products and count consumptions that match each product:
SELECT SQL_NO_CACHE t.*,
IFNULL(f.matchesCount, 0) AS matchesCount
FROM `product` `t`
LEFT JOIN
(SELECT cons.`id` AS `consumption_id`,
cons.`client_id` AS `consumption_client_id`,
cons.`material_id` AS `consumption_material_id`,
cons.`quality_id` AS `consumption_quality_id`,
cons.`thick` AS `consumption_thick`,
cons.`thick_max` AS `consumption_thick_max`,
cons.`width` AS `consumption_width`,
cons.`width_max` AS `consumption_width_max`,
cons.`long` AS `consumption_long`,
cons.`long_max` AS `consumption_long_max`,
cons.`type` AS `consumption_type`,
cons.`date_add` AS `consumption_date_add`,
prod.`id` AS `product_id`,
prod.`warehouse_id` AS `product_warehouse_id`,
prod.`code` AS `product_code`,
prod.`material_id` AS `product_material_id`,
prod.`quality_id` AS `product_quality_id`,
prod.`covering_id` AS `product_covering_id`,
prod.`finish_id` AS `product_finish_id`,
prod.`thickness` AS `product_thickness`,
prod.`width` AS `product_width`,
prod.`tons` AS `product_tons`,
count(cons.`id`) AS matchesCount
FROM consumption cons
INNER JOIN client cli ON cli.id=cons.client_id
LEFT JOIN product prod ON cons.material_id=prod.material_id
AND cons.quality_id=prod.quality_id
WHERE 1 > 0
AND prod.deleted=0
AND cli.status=1
GROUP BY prod.id) f ON f.product_id=t.id
WHERE deleted=0
GROUP BY t.id
Both queries take less than 1 second to execute (each).
Note: I still use the previous queries in my application, for example, when I want a break down of the list of products that match a single consumption, or the other way around. In that case I already add a filter per consumption id or product id that reduces the size of the result set a lot.
If client_has_group is "many:1", that is the wrong way to do it. You don't need the extra table.
INT is always 4 bytes. Consider smaller datatypes. Eventually the size of the database may add to your problems.
Do you really need date_add and date_upd. They seem like clutter that you will never use.
Avoid IN ( SELECT ... ) where practical. Switch to JOIN or EXISTS.
Why so many tables with code + group + equivalence? Could they be a single group? Do you need all 3 columns? Do you need id since code is UNIQUE? There comes a point where a schema is "over-normalized" and performance suffers without helping space much.
OR is a performance killer in some contexts.
"Correlated subqueries" are useful in some situations, but this one is probably better done via a JOIN:
AND `group` = ( SELECT `group` FROM quality WHERE id=cons.quality_id )
Beware of aggregates (eg, COUNT) with JOIN; you may be getting an inflated value. This is because the JOIN happens first.
why need
LEFT JOIN client_has_group cli_gr ON cli_gr.client_id=cons.client_id
it never used
why need GROUP BY cons.id, prod.id if you select all fields maybe select only what you need
try this select, i think it will be more faster
SELECT count(*), prod.*
FROM consumption cons
INNER JOIN client cli ON cli.id=cons.client_id
INNER JOIN material m ON m.id=cons.material_id
INNER JOIN quality q ON q.id=cons.quality_id
LEFT JOIN product prod
ON
(
(cons.material_id=prod.material_id)
OR
prod.material_id IN (
SELECT id FROM material WHERE `equivalence`=m.equivalence
AND `group`=m.group
)
)
AND
(
(cons.quality_id=prod.quality_id)
OR
prod.quality_id IN (
SELECT id FROM quality WHERE `equivalence`=q.equivalence
AND `group`=q.group
)
)
AND (prod.thickness >= (cons.thick - 0.1) AND prod.thickness <= (cons.thick_max + 0.1))
AND (prod.width >= (cons.width - 1000) AND prod.width <= (cons.width_max + 1000))
WHERE 1 > 0 AND prod.deleted=0 AND cli.status=1 AND cons.date_add >= '2017-10-08 00:00:00'
group by prod.id
maybe better do calculation count in background and add this field in product and consumption table.
i have table actions (30 rows) and passed_actions(10k rows)
actions table:
CREATE TABLE `actions` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`category_id` int(10) unsigned NOT NULL,
`author_id` int(10) unsigned NOT NULL,
`name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`slug` varchar(255) COLLATE utf8_unicode_ci NOT NULL COMMENT 'need for url',
`about` longtext COLLATE utf8_unicode_ci,
`image` text COLLATE utf8_unicode_ci,
`page_title` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`page_description` varchar(512) COLLATE utf8_unicode_ci DEFAULT NULL,
`active` tinyint(1) NOT NULL DEFAULT '0',
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `actions_slug_unique` (`slug`),
KEY `actions_author_id_foreign` (`author_id`),
KEY `actions_category_id_foreign` (`category_id`),
CONSTRAINT `actions_author_id_foreign` FOREIGN KEY (`author_id`) REFERENCES `users` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `actions_category_id_foreign` FOREIGN KEY (`category_id`) REFERENCES `categories` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=38 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
and passed_actions (~9500 rows)
CREATE TABLE `passed_actions` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`user_id` int(10) unsigned NOT NULL,
`action_id` int(10) unsigned NOT NULL,
`hash` varchar(6) COLLATE utf8_unicode_ci NOT NULL,
`public` tinyint(1) NOT NULL DEFAULT '1',
`successfully_passed` tinyint(1) NOT NULL DEFAULT '0',
`started_at` timestamp NULL DEFAULT NULL,
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `passed_actions_hash_unique` (`hash`),
KEY `passed_actions_user_id_foreign` (`user_id`),
KEY `passed_actions_action_id_foreign` (`action_id`),
CONSTRAINT `passed_actions_action_id_foreign` FOREIGN KEY (`action_id`) REFERENCES `actions` (`id`) ON DELETE CASCADE,
CONSTRAINT `passed_actions_user_id_foreign` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=25733 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
this query takes 0.3-0.5s:
select
actions.*
from actions
left join passed_actions.action_id = actions.id
group by actions.id
order by count(passed_actions.id) DESC
this affects to response time of my api...
why is this happening? i think that 10k rows is not a large table...
i use default mysql config. My server is 1gb ram and 1 cpu (digital ocean droplet)
Your query is actually reasonable fast. Sometimes a correlated subquery can help:
select a.*
from actions a
order by (select count(*) from passed_actions pa where pa.action_id = a.id) desc;
This can use the index you have defined on passed_actions(action_id).
If all you want off the second table is the count for sorting, as appears to be the case, try (untested, sorry):
select
actions.*
from actions
left join (select action_id, count(*) as passed_count from passed_actions group by action_id) p on actions.action_id = p.action_id
order by passed_count DESC
(I can't see where tests.id is coming from, I'm afraid.)
1- Rebuild the index and update statistics
2- Select Only the column you want to use
3- run this query in a new query and hit "Right click" and Click on "Display Estimated Execution Plan" and view the Missing Index Details and build the required index and run the query again
I am trying to improve performance of following query which took 93.2 sec to execute query below:
SELECT year(date), month(date), `country_name_name`,
CEIL(count(res.`user_xmpp_login`) /DAY(LAST_DAY(date))) as avgUser,
CEIL(count(res.user)/DAY(LAST_DAY(date))) as avgPurchase
FROM
( SELECT DATE(`user_registration_timestamp`) as date,
user_country,
NULL as user, `user_xmpp_login`
FROM users
WHERE `user_registration_timestamp` >= "2015-01-01 00:00:00"
AND `user_registration_timestamp` < "2016-01-01 00:00:00"
UNION ALL
SELECT DATE(`ts`) as date, user_country, user, NULL as `user_xmpp_login`
FROM purchase_log p
INNER JOIN users u ON u.`user_xmpp_login` = p.`user`
WHERE `ts` >= "2015-01-01 00:00:00"
AND `ts` < "2016-01-01 00:00:00"
AND result in ('ok', 'cancelled', 'pending')
) AS res
INNER JOIN countries c ON c.`country_id` = res.`user_country`
INNER JOIN country_names cn
ON (cn.`country_name_country` = c.`country_id`
AND cn.`country_name_language` = 'en')
GROUP BY 1,2,3
ORDER BY 4 DESC,5 DESC, 3 ASC;
Explain command shows:
And structure of each table is:
purchase table:
CREATE TABLE `purchase` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`result` varchar(32) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
KEY `iuser` (`user`),
) ENGINE=InnoDB AUTO_INCREMENT=12710221 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
users table:
CREATE TABLE `users` (
`user_id` int(11) NOT NULL AUTO_INCREMENT,
`user_country` int(11) DEFAULT NULL,
`user_xmpp_login` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`user_registration_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`user_id`),
UNIQUE KEY `user_xmpp_login_UNIQUE` (`user_xmpp_login`),
KEY `user_country_FK` (`user_country`),
KEY `user_registration_timestamp` (`user_registration_timestamp`),
CONSTRAINT `users_country_FK` FOREIGN KEY (`user_country`)
REFERENCES `countries` (`country_id`) ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=33504745 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
countries table
CREATE TABLE `countries` (
`country_id` int(11) NOT NULL AUTO_INCREMENT,
`country_code` varchar(2) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`country_id`),
) ENGINE=InnoDB AUTO_INCREMENT=508 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
country names
CREATE TABLE `country_names` (
`country_name_id` int(11) NOT NULL AUTO_INCREMENT,
`country_name_country` int(11) NOT NULL,
`country_name_language` char(2) COLLATE utf8_unicode_ci NOT NULL,
`country_name_name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`country_name_id`),
UNIQUE KEY `country_name_country_language_UNIQUE`
(`country_name_country`,`country_name_language`),
KEY `country_name_language` (`country_name_language`),
CONSTRAINT `country_name_country` FOREIGN KEY (`country_name_country`)
REFERENCES `countries` (`country_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=45793 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Is there any recommendations?
If you time each subquery, I think you will find users is the slowest component.
The purchase_log subquery can probably be improved with this "covering" INDEX(result, ts, user).
Combine the two "country" tables!. Use CHAR(2) CHARACTER SET ascii for the PRIMARY KEY and the JOINs to other tables. It is only 2 bytes, unlike INT, which is 4 bytes and VARCHAR..., which is 3 bytes (in this case).
You mention ts, but I don't see where it is coming from. If it is in purchase_log, then that table needs INDEX(user, ts).
What percentage of the users involved 2015? If it is more than about 20%, the INDEX(user_registration_timestamp) won't help.
Consider: Get rid of PRIMARY KEY (country_name_id), and promote the UNIQUE key to PRIMARY.
The biggest problem seems to be in your users table. Remember, mysql can only use one index per table for most situations. On your users table, the user_xmpp_login_UNIQUE column has been used to join it to the purchase_log table. There fore, the user_registration_timestamp index is not being used on the comparison involving the timestamp column.
One suggestion is to create a composite index on the user_xmpp_login and user_registration_timestamp columns.