MySQL - optimize query - mysql

I don't know how to more optimize this query
SELECT COUNT(*)
FROM `job_offer` AS `o`
WHERE o.`status` = 1
AND EXISTS
( SELECT 1
FROM `job_offer_localitymap` AS lm
WHERE lm.`job_offerid` = o.`job_offerid`
AND lm.`gps_localityid` IN (116, 1)
)
Explain as image here: http://postimg.org/image/52wd06f3f/full/
It's simple select with subquery and it runs about 52 miliseconds on localhost, which is too much. There are many more records server and it runs 350miliseconds there. I also try this query replaced with inner join and group by with same results.
SELECT SQL_NO_CACHE COUNT(DISTINCT(o.job_offerid))
FROM `job_offer` AS `o`
INNER JOIN `job_offer_localitymap` AS lm ON lm.job_offerid = o.`job_offerid`
AND lm.`gps_localityid` IN (116, 1)
WHERE (o.`status` = 1)
Explain as image here: http://postimg.org/image/3renvl3gx/full/
I just don't get it why is that such a problem.
STATS: count returns 1415 rows, job_offer has 65k rows, job_offer_locality_map has 100k rows. All tables are MyISAM.
Thank you for suggestions.
Here is database dump (table job_offer has many rows - about 50-60, so I give here simple version)
DROP TABLE IF EXISTS `job_offer`;
CREATE TABLE `job_offer` (
`job_offerid` int(13) NOT NULL AUTO_INCREMENT,
`name` varchar(255) NOT NULL,
`status` int(13) NOT NULL DEFAULT '1',
`sequence` int(13) NOT NULL DEFAULT '0',
`sequence_desc` tinyint(1) DEFAULT NULL,
`active_updated_when` datetime DEFAULT NULL,
PRIMARY KEY (`job_offerid`),
KEY `active_updated_when` (`active_updated_when`),
KEY `status_sequence_desc_active_updated_when_job_offerid` (`status`,`sequence_desc`,`active_updated_when`,`job_offerid`),
KEY `status_job_offerid` (`status`,`job_offerid`),
FULLTEXT KEY `name` (`name`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
DROP TABLE IF EXISTS `job_offer_localitymap`;
CREATE TABLE `job_offer_localitymap` (
`job_offer_localitymapid` int(13) NOT NULL AUTO_INCREMENT,
`old_id` int(13) NOT NULL DEFAULT '0',
`gps_localityid` int(13) NOT NULL,
`job_offerid` int(13) NOT NULL,
`sequence` int(13) NOT NULL DEFAULT '100',
`created` datetime NOT NULL,
`updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`old_localityid` int(13) NOT NULL DEFAULT '0',
PRIMARY KEY (`job_offer_localitymapid`),
KEY `job_offerid_gps_localityid` (`job_offerid`,`gps_localityid`),
KEY `job_offerid` (`job_offerid`),
KEY `gps_localityid` (`gps_localityid`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

INDEX(job_offered, gps_localityid)
Show us the JOIN version, together with its EXPLAIN and SHOW CREATE TABLE

Related

Getting Group by with left join to work in MySQL 5.7

Im trying to get a query to only return a single row for each unique ID.
given the following:
CREATE TABLE `groups` ( `id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(45) DEFAULT NULL, `type` int(1) DEFAULT NULL,
`date` datetime DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB
AUTO_INCREMENT=5 DEFAULT CHARSET=latin1;
CREATE TABLE `groupmembers` ( `id` int(11) NOT NULL AUTO_INCREMENT,
`username` varchar(255) DEFAULT NULL, `readmessage` int(1) DEFAULT
NULL, `status` int(1) DEFAULT NULL, `group_id` int(20) DEFAULT
NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT
CHARSET=latin1;
CREATE TABLE `messages` ( `id` bigint(10) NOT NULL AUTO_INCREMENT,
`send_to` varchar(50) NOT NULL DEFAULT '', `name` varchar(50) NOT
NULL DEFAULT '', `email` varchar(50) NOT NULL DEFAULT '',
`subject` varchar(100) NOT NULL DEFAULT 'No Subject', `message`
longtext NOT NULL, `timestamp` timestamp NOT NULL DEFAULT
CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `message_read`
enum('Yes','No') NOT NULL DEFAULT 'No', `send_time` datetime NOT
NULL DEFAULT '1970-01-01 00:00:00',
`hide_message` enum('0','1') NOT NULL, `grouper` int(20) DEFAULT
NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT
CHARSET=latin1;
insert into groups (type,date) values ('0',now());
insert into groupmembers (username,group_id) values ('user1',5),('user2',5);
insert into messages (send_to,name,send_time,grouper,message) values
('user1','user2','2019-10-09 19:18:12',5,'hello');
insert into messages (send_to,name,send_time,grouper,message) values
('','user2','2019-10-10 09:18:39',5,'hello');
In mysql 5.6 this query works as anticipated:
select groups.id,send_to,messages.name,max(send_time) from groups
inner join messages on groups.id=messages.grouper left join
groupmembers on groups.id=groupmembers.group_id where
groupmembers.username='user1' group by groups.id ORDER BY
MAX(send_time)
in mysql 5.7 this returns an error
Error Code: 1055. Expression #2 of SELECT list is not in GROUP BY
clause and contains nonaggregated column 'messages.send_to' which is
not functionally dependent on columns in GROUP BY clause; this is
incompatible with sql_mode=only_full_group_by
What do I need to do to get this to work in MySQL 5.7 without adjusting the SQL Mode?
See SQL Fiddle here: http://sqlfiddle.com/#!9/9061da/4
It works as expected on the SQL fiddle as that is running 5.6.
Just a guess...
SELECT DISTINCT g.id
, m.send_to
, m.name
, m.send_time
FROM groups g
JOIN messages m
ON m.grouper = g.id
JOIN
( SELECT send_to
, name
, MAX(send_time) send_time
FROM messages m
GROUP
BY send_to
, name
) x
ON x.send_to = m.send_to
AND x.name = m.name
AND x.send_time = m.send_time
JOIN groupmembers gm
ON gm.group_id = g.id
WHERE gm.username = 'user1'
ORDER
BY send_time;

Optimize MySQL query used to find matches between two tables

On a project I'm working on, I have two tables:
consumption: Contains historical orders from customers with fields specifying the features of the product they have bought (one product per row)
product: Contains current product stock
The database engine is InnoDB.
Goals:
The application must show matches from both sides, I mean:
When I list current products stock, I want to show a column that displays how many historical orders match with a particular product
When I list the historical orders, I want to see how many products match with a particular historical order
Database structure for consumption and product tables plus other related tables:
CREATE TABLE `consumption` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`created_by_id` INT(11) NULL DEFAULT NULL,
`client_id` INT(11) NOT NULL,
`data_import_id` INT(11) NULL DEFAULT NULL,
`tmp_consumption_id` INT(11) NULL DEFAULT NULL,
`material_id` INT(11) NULL DEFAULT NULL,
`quality_id` INT(11) NULL DEFAULT NULL,
`thick` DECIMAL(10,3) NULL DEFAULT NULL,
`thick_max` DECIMAL(10,3) NULL DEFAULT NULL,
`width` DECIMAL(10,2) NULL DEFAULT NULL,
`width_max` DECIMAL(10,2) NULL DEFAULT NULL,
`long` INT(11) NULL DEFAULT NULL,
`long_max` INT(11) NULL DEFAULT NULL,
`purchase_price` DECIMAL(10,2) NULL DEFAULT NULL,
`sale_price` DECIMAL(10,2) NULL DEFAULT NULL,
`comments` VARCHAR(255) NULL DEFAULT NULL,
`annual_consumption` DECIMAL(10,3) NULL DEFAULT NULL,
`type` ENUM('consumption','request') NULL DEFAULT 'consumption',
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
`covering_grammage` VARCHAR(64) NULL DEFAULT NULL,
`asp_sup_acab` VARCHAR(64) NULL DEFAULT NULL,
PRIMARY KEY (`id`),
INDEX `fk_consumption_client1` (`client_id`),
INDEX `created_by_id` (`created_by_id`),
INDEX `material_id` (`material_id`),
INDEX `quality_id` (`quality_id`),
CONSTRAINT `consumption_ibfk_1` FOREIGN KEY (`material_id`) REFERENCES `material` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `consumption_ibfk_2` FOREIGN KEY (`quality_id`) REFERENCES `quality` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `fk_consumption_client1` FOREIGN KEY (`client_id`) REFERENCES `client` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=30673
;
CREATE TABLE `product` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`warehouse_id` INT(11) NULL DEFAULT NULL,
`created_by_id` INT(11) NULL DEFAULT NULL,
`data_import_id` INT(11) NULL DEFAULT NULL,
`tmp_product_id` INT(11) NULL DEFAULT NULL,
`code` VARCHAR(32) NOT NULL,
`material_id` INT(11) NULL DEFAULT NULL,
`quality_id` INT(11) NULL DEFAULT NULL,
`covering_id` INT(11) NULL DEFAULT NULL,
`finish_id` INT(11) NULL DEFAULT NULL,
`source` VARCHAR(128) NULL DEFAULT NULL,
`thickness` DECIMAL(10,3) NULL DEFAULT NULL,
`width` INT(11) NULL DEFAULT NULL,
`tons` DECIMAL(10,3) NULL DEFAULT NULL,
`re` INT(11) NULL DEFAULT NULL,
`rm` INT(11) NULL DEFAULT NULL,
`a_percent` INT(11) NULL DEFAULT NULL,
`comments` VARCHAR(255) NULL DEFAULT NULL,
`price` DECIMAL(10,2) NULL DEFAULT NULL,
`deleted` TINYINT(1) NOT NULL DEFAULT '0',
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `warehouse_id` (`warehouse_id`),
INDEX `material_id` (`material_id`),
INDEX `quality_id` (`quality_id`),
INDEX `covering_id` (`covering_id`),
INDEX `finish_id` (`finish_id`),
CONSTRAINT `product_ibfk_1` FOREIGN KEY (`material_id`) REFERENCES `material` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `product_ibfk_2` FOREIGN KEY (`quality_id`) REFERENCES `quality` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `product_ibfk_3` FOREIGN KEY (`covering_id`) REFERENCES `covering` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `product_ibfk_4` FOREIGN KEY (`finish_id`) REFERENCES `finish` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `product_ibfk_5` FOREIGN KEY (`warehouse_id`) REFERENCES `warehouse` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=740
;
CREATE TABLE `client` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`zone_id` INT(11) NULL DEFAULT NULL,
`zone2_id` INT(11) NULL DEFAULT NULL,
`code` VARCHAR(64) NOT NULL,
`business_name` VARCHAR(255) NULL DEFAULT NULL,
`fiscal_name` VARCHAR(255) NULL DEFAULT NULL,
`nif` VARCHAR(15) NULL DEFAULT NULL,
`contact_short_name` VARCHAR(128) NULL DEFAULT NULL,
`contact_full_name` VARCHAR(128) NULL DEFAULT NULL,
`email` VARCHAR(255) NULL DEFAULT NULL,
`group` VARCHAR(255) NULL DEFAULT NULL,
`status` TINYINT(1) NOT NULL DEFAULT '1',
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`),
INDEX `zone_id` (`zone_id`),
INDEX `zone2_id` (`zone2_id`),
CONSTRAINT `client_ibfk_1` FOREIGN KEY (`zone_id`) REFERENCES `zone` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=443
;
CREATE TABLE `client_group` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(15) NOT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code` (`code`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=49
;
CREATE TABLE `client_has_group` (
`client_id` INT(11) NOT NULL,
`group_id` INT(11) NOT NULL,
INDEX `client_id` (`client_id`),
INDEX `group_id` (`group_id`),
CONSTRAINT `client_has_group_ibfk_1` FOREIGN KEY (`client_id`) REFERENCES `client` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT `client_has_group_ibfk_2` FOREIGN KEY (`group_id`) REFERENCES `client_group` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
;
CREATE TABLE `covering` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(128) NOT NULL,
`group` VARCHAR(128) NULL DEFAULT NULL,
`equivalence` VARCHAR(128) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=55
;
CREATE TABLE `finish` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(128) NOT NULL,
`group` VARCHAR(128) NULL DEFAULT NULL,
`equivalence` VARCHAR(128) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=42
;
CREATE TABLE `material` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(128) NOT NULL,
`group` VARCHAR(128) NULL DEFAULT NULL,
`equivalence` VARCHAR(128) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`),
INDEX `group` (`group`),
INDEX `equivalence` (`equivalence`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=46
;
CREATE TABLE `quality` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`code` VARCHAR(128) NOT NULL,
`group` VARCHAR(128) NULL DEFAULT NULL,
`equivalence` VARCHAR(128) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `code_UNIQUE` (`code`),
INDEX `group` (`group`),
INDEX `equivalence` (`equivalence`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=980
;
CREATE TABLE `user_filter` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`user_id` INT(11) NOT NULL,
`filter_type` ENUM('consumption','product') NOT NULL DEFAULT 'consumption',
`name` VARCHAR(255) NOT NULL,
`is_default` TINYINT(1) NOT NULL DEFAULT '0',
`client_status` TINYINT(1) NULL DEFAULT NULL,
`client_group` VARCHAR(45) NULL DEFAULT NULL,
`material` VARCHAR(15) NULL DEFAULT NULL,
`quality` VARCHAR(64) NULL DEFAULT NULL,
`thickness` VARCHAR(45) NULL DEFAULT NULL,
`width` VARCHAR(45) NULL DEFAULT NULL,
`tons` VARCHAR(45) NULL DEFAULT NULL,
`covering` VARCHAR(45) NULL DEFAULT NULL,
`finish` VARCHAR(45) NULL DEFAULT NULL,
`re` VARCHAR(45) NULL DEFAULT NULL,
`rm` VARCHAR(45) NULL DEFAULT NULL,
`a_percent` VARCHAR(45) NULL DEFAULT NULL,
`comments` VARCHAR(255) NULL DEFAULT NULL,
`price` VARCHAR(45) NULL DEFAULT NULL,
`warehouse` VARCHAR(45) NULL DEFAULT NULL,
`date` VARCHAR(45) NULL DEFAULT NULL,
`type` ENUM('consumption','request') NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `fk_user_filter_user1` (`user_id`),
INDEX `filter_type` (`filter_type`),
CONSTRAINT `fk_user_filter_user1` FOREIGN KEY (`user_id`) REFERENCES `user` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=5
;
CREATE TABLE `warehouse` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`name` VARCHAR(128) NOT NULL,
`zone_id` INT(11) NULL DEFAULT NULL,
`zone2_id` INT(11) NULL DEFAULT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `zone_id` (`zone_id`),
INDEX `zone2_id` (`zone2_id`),
CONSTRAINT `warehouse_ibfk_1` FOREIGN KEY (`zone_id`) REFERENCES `zone` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=37
;
CREATE TABLE `zone` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`zone2_id` INT(11) NULL DEFAULT NULL,
`name` VARCHAR(128) NOT NULL,
`date_add` DATETIME NOT NULL,
`date_upd` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `zone2_id` (`zone2_id`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
AUTO_INCREMENT=49
;
What I have done to be able to find matches between the two tables:
I have created a LEFT JOIN query between consumption and product table (that is also joining with additional tables if required).
Looks something like this:
SELECT cons.`id` as `consumption_id`, cons.`client_id` as `consumption_client_id`, cons.`material_id` as `consumption_material_id`, cons.`quality_id` as `consumption_quality_id`, cons.`thick` as `consumption_thick`, cons.`thick_max` as `consumption_thick_max`, cons.`width` as `consumption_width`, cons.`width_max` as `consumption_width_max`, cons.`long` as `consumption_long`, cons.`long_max` as `consumption_long_max`, cons.`type` as `consumption_type`, cons.`date_add` as `consumption_date_add`, prod.`id` as `product_id`, prod.`warehouse_id` as `product_warehouse_id`, prod.`code` as `product_code`, prod.`material_id` as `product_material_id`, prod.`quality_id` as `product_quality_id`, prod.`covering_id` as `product_covering_id`, prod.`finish_id` as `product_finish_id`, prod.`thickness` as `product_thickness`, prod.`width` as `product_width`, prod.`tons` as `product_tons`
FROM consumption cons
INNER JOIN client cli
ON cli.id=cons.client_id
LEFT JOIN client_has_group cli_gr
ON cli_gr.client_id=cons.client_id
LEFT JOIN product prod
ON
(
(cons.material_id=prod.material_id)
OR
prod.material_id IN (
SELECT id FROM material WHERE `equivalence`=(
SELECT `equivalence` FROM material WHERE id=cons.material_id
)
AND `group`=(
SELECT `group` FROM material WHERE id=cons.material_id
)
)
)
AND
(
(cons.quality_id=prod.quality_id)
OR
prod.quality_id IN (
SELECT id FROM quality WHERE `equivalence`=(
SELECT `equivalence` FROM quality WHERE id=cons.quality_id
)
AND `group`=(
SELECT `group` FROM quality WHERE id=cons.quality_id
)
)
)
AND (prod.thickness >= (cons.thick - 0.1) AND prod.thickness <= (cons.thick_max + 0.1))
AND (prod.width >= (cons.width - 1000) AND prod.width <= (cons.width_max + 1000))
WHERE 1 > 0 AND prod.deleted=0 AND cli.status=1 AND cons.date_add >= '2017-10-08 00:00:00'
GROUP BY cons.id, prod.id
When I want to list products and show matches of consumptions per every product, I have a main query that simply lists the products, then I join that query with the previous query from above and count the matches grouping by product id.
SELECT t.*,
count(f.consumption_id) AS matchesCount
FROM `product` t
LEFT JOIN (...previous query here...) f ON f.product_id=t.id
GROUP BY t.id
Other notes/considerations:
The application uses a couple of fields that have the same name in both tables, in order to find matches using ON in the JOIN
The application also uses more complex business logic, for example, product material can be equal or can be inside of an equivalence table or group
The user can save personal filters, that why it is used the user_filter table, so as a user I can have multiple "searches" saved and quickly switch from one to the other
The matches have to be displayed LIVE, I mean, calculated on the fly, not by any cronjob because user filter will always change
The amount of data the application will be working with right now will be about 35k rows in consumption table and about 1.5k rows in the product table
The server where the application is hosted is a dedicated server (64GB RAM) running MySQL
I had good performance with 3k rows of consumptions and 100 products, now with 10k+ consumption and 600 products, starting to get gateway timeout from nginx. Guess queries take too long.
I already know that if the ON cause has a lot of conditions it will work faster because the results sets are smaller, but if the condition is very wide, it will give a timeout, I guess the resulting rows are too many. Maybe the join will produce millions of rows.
What I'd like to ask is:
Am I on the right path in order to do "live matches" of data between both tables? is using the JOIN a good solution? I cannot think of another way to do it.
Apart from trying to optimize queries and indexes, are there any server tweaking I could do to take full advantage of server hardware?
Any other tips or techniques from someone who has done something similar in another project?
Update 1: Adding here full query for listing products with consumption matches:
SELECT t.*,
count(f.consumption_id) AS matchesCount
FROM `product` t
LEFT JOIN (
SELECT cons.`id` as `consumption_id`, cons.`client_id` as `consumption_client_id`, cons.`material_id` as `consumption_material_id`, cons.`quality_id` as `consumption_quality_id`, cons.`thick` as `consumption_thick`, cons.`thick_max` as `consumption_thick_max`, cons.`width` as `consumption_width`, cons.`width_max` as `consumption_width_max`, cons.`long` as `consumption_long`, cons.`long_max` as `consumption_long_max`, cons.`type` as `consumption_type`, cons.`date_add` as `consumption_date_add`, prod.`id` as `product_id`, prod.`warehouse_id` as `product_warehouse_id`, prod.`code` as `product_code`, prod.`material_id` as `product_material_id`, prod.`quality_id` as `product_quality_id`, prod.`covering_id` as `product_covering_id`, prod.`finish_id` as `product_finish_id`, prod.`thickness` as `product_thickness`, prod.`width` as `product_width`, prod.`tons` as `product_tons`
FROM consumption cons
INNER JOIN client cli
ON cli.id=cons.client_id
LEFT JOIN client_has_group cli_gr
ON cli_gr.client_id=cons.client_id
LEFT JOIN product prod
ON
(
(cons.material_id=prod.material_id)
OR
prod.material_id IN (
SELECT id FROM material WHERE `equivalence`=(
SELECT `equivalence` FROM material WHERE id=cons.material_id
)
AND `group`=(
SELECT `group` FROM material WHERE id=cons.material_id
)
)
)
WHERE 1 > 0 AND prod.deleted=0 AND cli.status=1 AND cons.date_add >= '2017-10-08 00:00:00'
GROUP BY cons.id, prod.id
) f ON f.product_id=t.id
GROUP BY t.id
Query time: 00:02:41 (+ 0,078 sec. network).
Note: The subquery JOIN run separately produces 600k rows. I'm thinking to try to group it somehow in order to make it smaller.
Update 2: Major improvement achieved by making the count inside the subquery and so reducing the result set used for the JOIN
Basically the subquery instead of returning 600k+ rows, it only returns as much rows as products or consumptions, depending what you're looking for. For that, the matchesCount has been moved inside the subquery instead of outside, and the group by has been changed, depending what list you want to display.
This is how the final queries look like right now:
List consumption and count products that match each consumption:
SELECT SQL_NO_CACHE `t`.*,
IFNULL(f.matchesCount, 0) AS matchesCount
FROM `consumption` `t`
LEFT JOIN
(SELECT cons.`id` AS `consumption_id`,
cons.`client_id` AS `consumption_client_id`,
cons.`material_id` AS `consumption_material_id`,
cons.`quality_id` AS `consumption_quality_id`,
cons.`thick` AS `consumption_thick`,
cons.`thick_max` AS `consumption_thick_max`,
cons.`width` AS `consumption_width`,
cons.`width_max` AS `consumption_width_max`,
cons.`long` AS `consumption_long`,
cons.`long_max` AS `consumption_long_max`,
cons.`type` AS `consumption_type`,
cons.`date_add` AS `consumption_date_add`,
prod.`id` AS `product_id`,
prod.`warehouse_id` AS `product_warehouse_id`,
prod.`code` AS `product_code`,
prod.`material_id` AS `product_material_id`,
prod.`quality_id` AS `product_quality_id`,
prod.`covering_id` AS `product_covering_id`,
prod.`finish_id` AS `product_finish_id`,
prod.`thickness` AS `product_thickness`,
prod.`width` AS `product_width`,
prod.`tons` AS `product_tons`,
count(prod.`id`) AS matchesCount
FROM consumption cons
INNER JOIN client cli ON cli.id=cons.client_id
LEFT JOIN product prod ON ((cons.material_id=prod.material_id)
OR prod.material_id IN
(SELECT id
FROM material
WHERE `equivalence`=
(SELECT `equivalence`
FROM material
WHERE id=cons.material_id )
AND `group`=
(SELECT `group`
FROM material
WHERE id=cons.material_id ) ))
AND ((cons.quality_id=prod.quality_id)
OR prod.quality_id IN
(SELECT id
FROM quality
WHERE `equivalence`=
(SELECT `equivalence`
FROM quality
WHERE id=cons.quality_id )
AND `group`=
(SELECT `group`
FROM quality
WHERE id=cons.quality_id ) ))
AND (prod.thickness >= (cons.thick - 0.1)
AND prod.thickness <= (cons.thick_max + 0.1))
AND (prod.width >= (cons.width - 1000)
AND prod.width <= (cons.width_max + 1000))
WHERE 1 > 0
AND prod.deleted=0
AND cli.status=1
AND cons.date_add >= '2017-10-08 00:00:00'
GROUP BY cons.id) f ON f.consumption_id=t.id
GROUP BY t.id
List products and count consumptions that match each product:
SELECT SQL_NO_CACHE t.*,
IFNULL(f.matchesCount, 0) AS matchesCount
FROM `product` `t`
LEFT JOIN
(SELECT cons.`id` AS `consumption_id`,
cons.`client_id` AS `consumption_client_id`,
cons.`material_id` AS `consumption_material_id`,
cons.`quality_id` AS `consumption_quality_id`,
cons.`thick` AS `consumption_thick`,
cons.`thick_max` AS `consumption_thick_max`,
cons.`width` AS `consumption_width`,
cons.`width_max` AS `consumption_width_max`,
cons.`long` AS `consumption_long`,
cons.`long_max` AS `consumption_long_max`,
cons.`type` AS `consumption_type`,
cons.`date_add` AS `consumption_date_add`,
prod.`id` AS `product_id`,
prod.`warehouse_id` AS `product_warehouse_id`,
prod.`code` AS `product_code`,
prod.`material_id` AS `product_material_id`,
prod.`quality_id` AS `product_quality_id`,
prod.`covering_id` AS `product_covering_id`,
prod.`finish_id` AS `product_finish_id`,
prod.`thickness` AS `product_thickness`,
prod.`width` AS `product_width`,
prod.`tons` AS `product_tons`,
count(cons.`id`) AS matchesCount
FROM consumption cons
INNER JOIN client cli ON cli.id=cons.client_id
LEFT JOIN product prod ON cons.material_id=prod.material_id
AND cons.quality_id=prod.quality_id
WHERE 1 > 0
AND prod.deleted=0
AND cli.status=1
GROUP BY prod.id) f ON f.product_id=t.id
WHERE deleted=0
GROUP BY t.id
Both queries take less than 1 second to execute (each).
Note: I still use the previous queries in my application, for example, when I want a break down of the list of products that match a single consumption, or the other way around. In that case I already add a filter per consumption id or product id that reduces the size of the result set a lot.
If client_has_group is "many:1", that is the wrong way to do it. You don't need the extra table.
INT is always 4 bytes. Consider smaller datatypes. Eventually the size of the database may add to your problems.
Do you really need date_add and date_upd. They seem like clutter that you will never use.
Avoid IN ( SELECT ... ) where practical. Switch to JOIN or EXISTS.
Why so many tables with code + group + equivalence? Could they be a single group? Do you need all 3 columns? Do you need id since code is UNIQUE? There comes a point where a schema is "over-normalized" and performance suffers without helping space much.
OR is a performance killer in some contexts.
"Correlated subqueries" are useful in some situations, but this one is probably better done via a JOIN:
AND `group` = ( SELECT `group` FROM quality WHERE id=cons.quality_id )
Beware of aggregates (eg, COUNT) with JOIN; you may be getting an inflated value. This is because the JOIN happens first.
why need
LEFT JOIN client_has_group cli_gr ON cli_gr.client_id=cons.client_id
it never used
why need GROUP BY cons.id, prod.id if you select all fields maybe select only what you need
try this select, i think it will be more faster
SELECT count(*), prod.*
FROM consumption cons
INNER JOIN client cli ON cli.id=cons.client_id
INNER JOIN material m ON m.id=cons.material_id
INNER JOIN quality q ON q.id=cons.quality_id
LEFT JOIN product prod
ON
(
(cons.material_id=prod.material_id)
OR
prod.material_id IN (
SELECT id FROM material WHERE `equivalence`=m.equivalence
AND `group`=m.group
)
)
AND
(
(cons.quality_id=prod.quality_id)
OR
prod.quality_id IN (
SELECT id FROM quality WHERE `equivalence`=q.equivalence
AND `group`=q.group
)
)
AND (prod.thickness >= (cons.thick - 0.1) AND prod.thickness <= (cons.thick_max + 0.1))
AND (prod.width >= (cons.width - 1000) AND prod.width <= (cons.width_max + 1000))
WHERE 1 > 0 AND prod.deleted=0 AND cli.status=1 AND cons.date_add >= '2017-10-08 00:00:00'
group by prod.id
maybe better do calculation count in background and add this field in product and consumption table.

join count per chatroom

i have tried different join examples from w3c site, but could not solve my problem.
All i could get working was:
SELECT *
FROM chatters_online
WHERE TIME_TO_SEC(TIMEDIFF(NOW(),datumtijd )) < 300
But i want is a count of chatters per chatroom: room1(12),room12(2) etc.
CREATE TABLE IF NOT EXISTS `chatters_online` (
`room` int(10) unsigned NOT NULL,
`datumtijd` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`chatterID` int(10) unsigned NOT NULL DEFAULT '0',
`nick` varchar(20) NOT NULL,
UNIQUE KEY `unique_index` (`room`,`nick`),
KEY `room` (`room`),
KEY `nick` (`nick`),
KEY `datumtijd` (`datumtijd`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `chatrooms` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`roomname` varchar(20) NOT NULL,
`moderator` int(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `roomname` (`roomname`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=18 ;
What i am looking for is the needed SQL for a count per chatroom.
JOIN and GROUP BY should give you the required results
select CR.roomname, count(CO.chatterID) as ChattersCount
FROM chatrooms CR
JOIN chatters_online CO
on CO.room = CR.id
GROUP BY CR.roomname

MySql - Create view to read from Multiple Tables

I have archived some old line items for invoices that are no longer current but still need to reference them. I think I need to create a VIEW but not really understanding it. Can someone help so I can run a query to pull the invoice and then the total of all the line items assigned (no matter what table the items are in)?
CREATE TABLE `Invoice` (
`Invoice_ID` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT,
`Invoice_CreatedDateTime` DATETIME DEFAULT NULL,
`Invoice_Status` ENUM('Paid','Sent','Unsent','Hold') DEFAULT NULL,
`LastUpdatedAt` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`ID`),
KEY `LastUpdatedAt` (`LastUpdatedAt`)
) ENGINE=MYISAM DEFAULT CHARSET=latin1
CREATE TABLE `Invoice_LineItem` (
`LineItem_ID` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT,
`LineItem_ChargeType` VARCHAR(64) NOT NULL DEFAULT '',
`LineItem_InvoiceID` INT(11) UNSIGNED DEFAULT NULL,
`LineItem_Amount` DECIMAL(11,4) DEFAULT NULL,
`LastUpdatedAt` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`LineItem_ID`),
KEY `LastUpdatedAt` (`LastUpdatedAt`),
KEY `LineItem_InvoiceID` (`LineItem_InvoiceID`)
) ENGINE=MYISAM AUTO_INCREMENT=1 DEFAULT CHARSET=latin1
CREATE TABLE `Invoice_LineItem_Archived` (
`LineItem_ID` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT,
`LineItem_ChargeType` VARCHAR(64) NOT NULL DEFAULT '',
`LineItem_InvoiceID` INT(11) UNSIGNED DEFAULT NULL,
`LineItem_Amount` DECIMAL(11,4) DEFAULT NULL,
`LastUpdatedAt` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`LineItem_ID`),
KEY `LastUpdatedAt` (`LastUpdatedAt`),
KEY `LineItem_InvoiceID` (`LineItem_InvoiceID`)
) ENGINE=MYISAM AUTO_INCREMENT=1 DEFAULT CHARSET=latin1
Typically I would just run the following query to get the amount due on the invoices
SELECT
Invoice_ID,
Invoice_CreatedDateTime,
Invoice_Status,
(SELECT SUM(LineItem_Amount) AS totAmt FROM Invoice_LineItem WHERE LineItem_InvoiceID=Invoice_ID) AS Invoice_Total
FROM
Invoice
WHERE
Invoice_Status='Sent'
Also how can I select all the line items from both tables in one query?
SELECT
LineItem_ID,
LineItem_ChargeType,
LineItem_Amount
FROM
Invoice_LineItem
WHERE
LineItem_InvoiceID='1234'
You can use the MERGE Storage Engine to create a virtual table that's the union of two real tables:
CREATE TABLE Invoice_LineItem_All
(
`LineItem_ID` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT,
`LineItem_ChargeType` VARCHAR(64) NOT NULL DEFAULT '',
`LineItem_InvoiceID` INT(11) UNSIGNED DEFAULT NULL,
`LineItem_Amount` DECIMAL(11,4) DEFAULT NULL,
`LastUpdatedAt` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
KEY (`LineItem_ID`),
KEY `LastUpdatedAt` (`LastUpdatedAt`),
KEY `LineItem_InvoiceID` (`LineItem_InvoiceID`)
) ENGINE=MERGE UNION=(Invoice_LineItem_Archived, Invoice_LineItem);
You can use UNION :
SELECT a.* FROM a
UNION
SELECT b.* FROM b;
You just need to have the same number and type of column in your different queries.
As far as I remember, you can add test in sub-queries, but I'm not sure you can order on the global result.
http://dev.mysql.com/doc/refman/4.1/en/union.html

Optimize sql query to speed up a search which currently takes around 85 seconds

I have a database with the records near about 2.7 milion . I need to fetch records from that for that i am using the below query
for result
SELECT r3.original_image_title,r3.uuid,r3.original_image_URL FROM `image_attributes` AS r1 INNER JOIN `filenames` as r3 WHERE r1.`uuid` = r3.`uuid` and r3.`status` = 1 and r1.status=1 and (r1.`attribute_name` like "Quvenzhané Wallis%" or r3.original_image_URL like "Quvenzhané Wallis%") group by r3.`uuid` limit 0,20
for total count
SELECT count(DISTINCT(r1.`uuid`)) as count FROM `image_attributes` AS r1 INNER JOIN `filenames` as r3 WHERE r1.`uuid` = r3.`uuid` and r3.`status` = 1 and r1.status=1 and (r1.`attribute_name` like "Quvenzhané Wallis%" or r3.original_image_URL like "Quvenzhané Wallis%")
table structures are as below
CREATE TABLE IF NOT EXISTS `image_attributes` (
`index` int(11) NOT NULL AUTO_INCREMENT,
`attribute_name` text NOT NULL,
`attribute_type` varchar(255) NOT NULL,
`uuid` varchar(255) NOT NULL,
`status` tinyint(1) NOT NULL DEFAULT '1',
PRIMARY KEY (`index`),
KEY `attribute_type` (`attribute_type`),
KEY `uuid` (`uuid`),
KEY `status` (`status`),
KEY `attribute_name` (`attribute_name`(50))
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=2730431 ;
CREATE TABLE IF NOT EXISTS `filenames` (
`index` int(11) NOT NULL AUTO_INCREMENT,
`original_image_title` text NOT NULL,
`original_image_URL` text NOT NULL,
`uuid` varchar(255) NOT NULL,
`status` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`index`),
KEY `uuid` (`uuid`),
KEY `original_image_URL` (`original_image_URL`(50))
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=591967 ;
please suggest me how can i optimize the queries to make the search result faster
I would recommend to you a book called 'High Performance MySql'. There is a section called Optimize databases and queries, or something like that.