MySQL Performance issue: select query takes forever - mysql

This is vitally important query of the project that calculates statistic information for each user, generates almost 300k rows but it takes forever to get answer.
The problem is, this query is needed to be executed almost every 20-30 seconds and the row count always grows.
Despite the fact that the fields that used in join and where are indexed it takes around 2000 seconds to get the answer from query.
One another fact is, the tables used in this query are large. For example, IconKeyword holds almost 95M records.
Please take a look and tell me how can I optimize this query
SELECT
Icon.`user_id` AS user_id,
Keyword.`id` AS keyword_id,
Keyword.`title` AS keyword_title,
Keyword.`demand` AS keyword_demand,
count( IconKeyword.`iconID` ) AS ico_count,
Icon.`type` AS icon_type,
Keyword.`common` AS common
FROM
Icon
INNER JOIN IconKeyword ON IconKeyword.iconID = Icon.id
INNER JOIN Keyword ON Keyword.id = IconKeyword.keywordID
WHERE
Keyword.is_deleted = 0
AND
Keyword.restricted = 0
GROUP BY
Icon.`user_id`,
Icon.type,
Keyword.id;
And here is explain results:
Create tables:
CREATE TABLE `Icon` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`title` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`uri` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`description` text CHARACTER SET utf8 COLLATE utf8_general_ci,
`type` tinyint DEFAULT NULL,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`user_id` int unsigned DEFAULT NULL,
`f_id` int unsigned DEFAULT NULL,
`color` tinyint unsigned DEFAULT '0',
`pack_id` int unsigned DEFAULT NULL,
`pack_bg` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`fi_url` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`is_deleted` tinyint unsigned DEFAULT '0',
`count_of_sets` int unsigned NOT NULL DEFAULT '0',
`p_hash` binary(64) DEFAULT NULL,
`hash_bit_count` int unsigned GENERATED ALWAYS AS (bit_count(`p_hash`)) STORED,
`checked` tinyint unsigned DEFAULT '0',
`md5` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`) USING BTREE,
UNIQUE KEY `f_id` (`f_id`) USING BTREE,
KEY `user_id` (`user_id`) USING BTREE,
KEY `id` (`id`,`type`,`user_id`) USING BTREE,
KEY `type` (`type`) USING BTREE,
KEY `uri` (`uri`) USING BTREE,
KEY `hash_bit_count` (`hash_bit_count`) USING BTREE,
KEY `is_deleted` (`is_deleted`),
KEY `checked` (`checked`),
KEY `user_id_2` (`user_id`,`type`) USING BTREE,
CONSTRAINT `Icon_ibfk_1` FOREIGN KEY (`user_id`) REFERENCES `User` (`id`) ON DELETE SET NULL ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=118691 DEFAULT CHARSET=utf8
CREATE TABLE `Keyword` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`title` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`slug` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`created_at` datetime DEFAULT CURRENT_TIMESTAMP,
`common` tinyint unsigned DEFAULT '0',
`demand` int DEFAULT '0',
`f_id` int unsigned DEFAULT NULL,
`is_deleted` tinyint unsigned DEFAULT '0',
`needs_review` tinyint unsigned DEFAULT '0',
`checked` tinyint unsigned DEFAULT '0',
`restricted` tinyint unsigned DEFAULT '0',
`word_checked` tinyint unsigned DEFAULT '0',
PRIMARY KEY (`id`) USING BTREE,
UNIQUE KEY `slug` (`slug`) USING BTREE,
KEY `title` (`title`) USING BTREE,
KEY `is_deleted` (`is_deleted`) USING BTREE,
KEY `restricted` (`restricted`) USING BTREE,
KEY `restricted_2` (`restricted`,`is_deleted`,`id`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=30000392 DEFAULT CHARSET=utf8
CREATE TABLE `IconKeyword` (
`iconID` int unsigned NOT NULL,
`keywordID` int unsigned NOT NULL,
PRIMARY KEY (`iconID`,`keywordID`) USING BTREE,
UNIQUE KEY `keywordID_2` (`keywordID`,`iconID`),
KEY `keywordID` (`keywordID`) USING BTREE,
KEY `iconID` (`iconID`) USING BTREE,
CONSTRAINT `ik_ibfk_1` FOREIGN KEY (`keywordID`) REFERENCES `Keyword` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `ik_ibfk_2` FOREIGN KEY (`iconID`) REFERENCES `Icon` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8

For your query, you can try adding indexes on:
keyword(is_deleted, is_restricted, id)
IconKeyword(KeywordId, IconId)
Icon(id) (which should already exist because id is a primary key)
The query will still need to do a lot of work for the aggregation, but this might help.

I guess the number of records that have Keyword.is_deleted = 0 AND Keyword.restricted = 0 is quite large, so adding indexes here wouldn't help much.
What you can try is helping the aggregation works faster by adding a Multiple-Column Indexes, specifically (Icon.user_id, Icon.type) because it's what you're using in GROUP BY
Also, can you provide an EXPLAIN result of this query so we can better understand what is causing the performance problem.

Related

Improve count query with join and where statement

I'm running a count query which is very slow, how can improve this?
I've got the following query, but it takes around 1.33 seconds:
select
count(*) as aggregate
from
`tickets`
inner join `orders` on `orders`.`id` = `tickets`.`order_id`
where
`orders`.`status` = 'paid' and
`tickets`.`created_at` > '2023-01-01 00:00:00'
The tickets table has around 650000 rows and the order table has around 320000 rows.
This is the result of SHOW CREATE TABLE tickets:
CREATE TABLE `tickets` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`tickettype_id` int unsigned NOT NULL,
`order_id` int unsigned NOT NULL,
`variant_id` bigint unsigned DEFAULT NULL,
`seat_id` bigint unsigned DEFAULT NULL,
`barcode` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`first_name` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`last_name` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`email` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`telephone` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`birthday` date DEFAULT NULL,
`age` int unsigned DEFAULT NULL,
`gender` enum('m','f') CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`price` double(10,2) DEFAULT NULL,
`extra_info` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `tickets_barcode_unique` (`barcode`),
KEY `tickets_tickettype_id_foreign` (`tickettype_id`),
KEY `tickets_order_id_foreign` (`order_id`),
KEY `tickets_order_id_index` (`order_id`),
KEY `tickets_tickettype_id_index` (`tickettype_id`),
KEY `tickets_seat_id_foreign` (`seat_id`),
KEY `tickets_variant_id_foreign` (`variant_id`),
CONSTRAINT `tickets_ibfk_1` FOREIGN KEY (`order_id`) REFERENCES `orders` (`id`) ON DELETE CASCADE,
CONSTRAINT `tickets_seat_id_foreign` FOREIGN KEY (`seat_id`) REFERENCES `seatplan_seats` (`id`) ON DELETE SET NULL,
CONSTRAINT `tickets_tickettype_id_foreign` FOREIGN KEY (`tickettype_id`) REFERENCES `tickets_types` (`id`) ON DELETE CASCADE,
CONSTRAINT `tickets_variant_id_foreign` FOREIGN KEY (`variant_id`) REFERENCES `ticket_variants` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB AUTO_INCREMENT=2945088 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
How can I improve the speed?
The performance of your query depends on several factors. Such as,
The table size
Performance of your machine
Indexing etc.
If you dont have indices created for status, order_id and created_at, better create them. Which can significantly improve the query performance.
CREATE INDEX order_id_index ON tickets(order_id);
CREATE INDEX status_index ON orders(status);
CREATE INDEX created_at_index ON tickets(created_at);
Additionally, if you are using PostgreSQL, try running VACUUM on your tables which removes the dead tuples and improves performance.
First of all
You need to add two indexes :
CREATE INDEX order_id_idx ON tickets(order_id);
// composite index since you are using both columns in where
CREATE INDEX status_created_at_idx ON tickets(status, created_at);
The query optimizer uses the composite indexes for queries that test all columns in the index, or queries that test the first columns, the first two columns, and so on.
More informations regarding composite can be found here

slow join on table (10k rows)

i have table actions (30 rows) and passed_actions(10k rows)
actions table:
CREATE TABLE `actions` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`category_id` int(10) unsigned NOT NULL,
`author_id` int(10) unsigned NOT NULL,
`name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`slug` varchar(255) COLLATE utf8_unicode_ci NOT NULL COMMENT 'need for url',
`about` longtext COLLATE utf8_unicode_ci,
`image` text COLLATE utf8_unicode_ci,
`page_title` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`page_description` varchar(512) COLLATE utf8_unicode_ci DEFAULT NULL,
`active` tinyint(1) NOT NULL DEFAULT '0',
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `actions_slug_unique` (`slug`),
KEY `actions_author_id_foreign` (`author_id`),
KEY `actions_category_id_foreign` (`category_id`),
CONSTRAINT `actions_author_id_foreign` FOREIGN KEY (`author_id`) REFERENCES `users` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `actions_category_id_foreign` FOREIGN KEY (`category_id`) REFERENCES `categories` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=38 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
and passed_actions (~9500 rows)
CREATE TABLE `passed_actions` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`user_id` int(10) unsigned NOT NULL,
`action_id` int(10) unsigned NOT NULL,
`hash` varchar(6) COLLATE utf8_unicode_ci NOT NULL,
`public` tinyint(1) NOT NULL DEFAULT '1',
`successfully_passed` tinyint(1) NOT NULL DEFAULT '0',
`started_at` timestamp NULL DEFAULT NULL,
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `passed_actions_hash_unique` (`hash`),
KEY `passed_actions_user_id_foreign` (`user_id`),
KEY `passed_actions_action_id_foreign` (`action_id`),
CONSTRAINT `passed_actions_action_id_foreign` FOREIGN KEY (`action_id`) REFERENCES `actions` (`id`) ON DELETE CASCADE,
CONSTRAINT `passed_actions_user_id_foreign` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=25733 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
this query takes 0.3-0.5s:
select
actions.*
from actions
left join passed_actions.action_id = actions.id
group by actions.id
order by count(passed_actions.id) DESC
this affects to response time of my api...
why is this happening? i think that 10k rows is not a large table...
i use default mysql config. My server is 1gb ram and 1 cpu (digital ocean droplet)
Your query is actually reasonable fast. Sometimes a correlated subquery can help:
select a.*
from actions a
order by (select count(*) from passed_actions pa where pa.action_id = a.id) desc;
This can use the index you have defined on passed_actions(action_id).
If all you want off the second table is the count for sorting, as appears to be the case, try (untested, sorry):
select
actions.*
from actions
left join (select action_id, count(*) as passed_count from passed_actions group by action_id) p on actions.action_id = p.action_id
order by passed_count DESC
(I can't see where tests.id is coming from, I'm afraid.)
1- Rebuild the index and update statistics
2- Select Only the column you want to use
3- run this query in a new query and hit "Right click" and Click on "Display Estimated Execution Plan" and view the Missing Index Details and build the required index and run the query again

Index not being used for sort in joined view

I have the following schema:
CREATE TABLE `news` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`news_category_id` int(10) unsigned NOT NULL,
`news_type_id` int(10) unsigned NOT NULL,
`news_pictures_main_id` int(10) unsigned DEFAULT NULL,
`title` tinytext COLLATE latin1_general_ci,
`body` text COLLATE latin1_general_ci,
`tmstp` timestamp NULL DEFAULT NULL,
`subcategory` varchar(64) COLLATE latin1_general_ci DEFAULT NULL,
`source` varchar(128) COLLATE latin1_general_ci DEFAULT NULL,
`old_id` int(10) unsigned DEFAULT NULL,
`tags` text COLLATE latin1_general_ci,
PRIMARY KEY (`id`),
KEY `news_time_idx` (`tmstp`),
KEY `fk_news_news_pictures1` (`news_pictures_main_id`),
KEY `fk_news_news_category1` (`news_category_id`),
KEY `fk_news_news_type1` (`news_type_id`),
CONSTRAINT `fk_news_news_category1` FOREIGN KEY (`news_category_id`) REFERENCES `news_category` (`id`) ON UPDATE CASCADE,
CONSTRAINT `fk_news_news_pictures1` FOREIGN KEY (`news_pictures_main_id`) REFERENCES `news_pictures` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
CONSTRAINT `fk_news_news_type1` FOREIGN KEY (`news_type_id`) REFERENCES `news_type` (`id`) ON UPDATE CASCADE
) ENGINE=InnoDB
CREATE TABLE `news_pictures` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`path` text COLLATE latin1_general_ci,
`description` text COLLATE latin1_general_ci,
`author` varchar(45) COLLATE latin1_general_ci DEFAULT NULL,
`news_id` int(10) unsigned DEFAULT NULL,
`temp_id` varchar(40) COLLATE latin1_general_ci DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `filename_old_id_unq` (`path`(20),`temp_id`(6)),
KEY `fk_news_pictures_news1` (`news_id`),
KEY `temp_id_idx` (`temp_id`(8)),
CONSTRAINT `fk_news_pictures_news1` FOREIGN KEY (`news_id`) REFERENCES `news` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB
CREATE TABLE `news_category` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(45) COLLATE latin1_general_ci DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB
CREATE TABLE `news_type` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(45) COLLATE latin1_general_ci DEFAULT NULL,
`slug` varchar(45) COLLATE latin1_general_ci DEFAULT NULL,
PRIMARY KEY (`id`)
KEY `news_type_slug_idx` (`slug`)
) ENGINE=InnoDB
From that, there is derived the following view:
CREATE OR REPLACE VIEW `news_full` AS select `n`.`id` AS `id`,
`n`.`title` AS `title`,
`n`.`body` AS `body`,
`n`.`tmstp` AS `tmstp`,
`n`.`subcategory` AS `subcategory`,
`n`.`source` AS `source`,
`n`.`old_id` AS `old_id`,
`n`.`news_type_id` AS `news_type_id`,
`n`.`tags` AS `tags`,
`nt`.`name` AS `news_type_name`,
`nt`.`slug` AS `news_type_slug`,
`n`.`news_pictures_main_id` AS `news_pictures_main_id`,
`np`.`path` AS `news_pictures_main_path`,
`np`.`description` AS `news_pictures_main_description`,
`np`.`author` AS `news_pictures_main_author`,
`np`.`temp_id` AS `news_pictures_main_temp_id`,
`n`.`news_category_id` AS `news_category_id`,
`nc`.`name` AS `news_category_name`
from (((`news` `n`
left join `news_pictures` `np` on((`n`.`news_pictures_main_id` = `np`.`id`)))
join `news_category` `nc` on((`n`.`news_category_id` = `nc`.`id`)))
join `news_type` `nt` on((`n`.`news_type_id` = `nt`.`id`)));
However, if I try to run the following query:
select * from news_full order by tmstp limit 100
I get the following execution plan (please click on the image to expand it):
Notice the Using temporary; Using filesort field in the first step. But this is weird, because tmstp field is indexed on the base table.
First I thought this was due the left join on the view, but I've changed it to inner join and I got the same results.
Edit
As #Michael-sqlbot cleverly noticed, the query optimizer is inverting the order of the base tables, putting news_category (nc) first.
If I change the query that creates the view to use only LEFT JOINs it seems to work:
The execution times, as expected, as blatantly different:
Not satisfied, I created another view with the original query, adding the STRAIGHT_JOIN statement. So, the query plan comes as follows:
So, it's not using the index.
However, if I run the plan for the base query adding the same ORDER BY and LIMIT clauses, it does uses the index:
(Not an answer, but some other issues to bring up...)
UNIQUE KEY `filename_old_id_unq` (`path`(20),`temp_id`(6))
That constrains the first 20 characters of path, together with the first 6 characters of temp_id to be unique across the table. Did you really want that?
I suspect the optimizer will never use both columns of that index. (In general, prefixing is useless.)
And...
`title` tinytext COLLATE latin1_general_ci
Change to VARCHAR(255). There are disadvantages of TINYTEXT and perhaps no advantages.

MYSQL INNER JOIN is slow with index

this is my simple inner join:
SELECT
SUM(ASSNZ.assenzeDidattiche) AS TotaleAssenze,
SUM(ASSNZ.ore) AS totale_parziale,
FLOOR(((SUM(ASSNZ.assenzeDidattiche) / SUM(ASSNZ.ore)) * 100)) AS andamento,
MAX(ASSNZ.dataLezione) AS ultima_lezione,
ASSNZ.idServizio,
ASSNZ.idUtente
FROM
ciac_corsi_assenze AS ASSNZ
INNER JOIN
ciac_serviziAcquistati_ITA AS ACQ
ON ACQ.idContatto = ASSNZ.idUtente
AND ACQ.idServizio = ASSNZ.idServizio
AND ACQ.stato_allievo <> 'ritirato'
GROUP BY
ASSNZ.idServizio,
ASSNZ.idUtente
table "ASSNZ" has 213886 rows with index "idUtente", "idServizio"
table "ACQ" has 8950 rows with index "idContatto", "idServizio"
ASSNZ table:
CREATE TABLE `ciac_corsi_assenze` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`idUtente` int(11) DEFAULT NULL,
`idServizio` int(11) DEFAULT NULL,
`idCorso` int(11) DEFAULT NULL,
`idCalendario` int(11) DEFAULT NULL,
`modalita` varchar(255) DEFAULT NULL,
`ore` int(11) DEFAULT NULL,
`assenzeDidattiche` float DEFAULT NULL,
`assenzeAmministrative` float DEFAULT NULL,
`dataLezione` date DEFAULT NULL,
`ora_inizio` varchar(8) DEFAULT NULL,
`ora_fine` varchar(8) DEFAULT NULL,
`dataFineStage` date DEFAULT NULL,
`giustificata` varchar(128) DEFAULT NULL,
`motivazione` longtext,
`grouped` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `idUtente` (`idUtente`) USING BTREE,
KEY `idServizio` (`idServizio`) USING BTREE,
KEY `dataLezione` (`dataLezione`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=574582 DEFAULT CHARSET=utf8;
ACQ table:
CREATE TABLE `ciac_serviziacquistati_ita` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`idServizio` int(11) NOT NULL,
`idContatto` int(11) NOT NULL,
`idAzienda` int(11) NOT NULL,
`idSede` int(11) NOT NULL,
`tipoPersona` int(11) NOT NULL,
`num_registro` int(11) NOT NULL,
`codice` varchar(255) CHARACTER SET latin1 DEFAULT NULL,
`dal` date NOT NULL,
`al` date NOT NULL,
`ore` int(11) NOT NULL,
`costoOrario` decimal(10,0) NOT NULL,
`annoFormativo` varchar(128) CHARACTER SET latin1 NOT NULL,
`stato_attuale` int(11) NOT NULL,
`datore_attuale` int(11) NOT NULL,
`stato_allievo` varchar(64) CHARACTER SET latin1 NOT NULL DEFAULT 'corsista',
`data_ritiro` date DEFAULT NULL,
`crediti_formativi` int(11) NOT NULL,
`note` longtext CHARACTER SET latin1 NOT NULL,
`valore_economico` decimal(10,2) NOT NULL,
`dataInserimento` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idServizio` (`idServizio`) USING BTREE,
KEY `idAzienda` (`idAzienda`) USING BTREE,
KEY `idContatto` (`idContatto`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=9542 DEFAULT CHARSET=utf8;
this is my EXPLAIN of the select
Now because the query is slow, during 1.5s / 2.0s??
Something wrong?
UPDATE
added new index (with the John Bollinger's answer) to the table ciac_corsi_assenze:
PRIMARY KEY (`id`),
KEY `dataLezione` (`dataLezione`) USING BTREE,
KEY `test` (`idUtente`,`idServizio`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=574582 DEFAULT CHARSET=utf8;
added new index to the table ciac_serviziAcquistati_ITA:
PRIMARY KEY (`id`),
KEY `idAzienda` (`idAzienda`) USING BTREE,
KEY `test2` (`idContatto`,`idServizio`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=9542 DEFAULT CHARSET=utf8;
New EXPLAIN:
But it's always slow :(
Your tables have separate indexes on various columns of interest, but MySQL will use at most one index per table to perform your query. This particular query would probably be sped by table ciac_corsi_assenze having an index on (idUtente, idServizio) (and such an index would supersede the existing one on (idUtente) alone). That should allow MySQL to avoid sorting the result rows to perform the grouping, and it will help more in performing the join than any of the existing indexes do.
The query would probably be sped further by table ciac_serviziAcquistati_ITA having an index on (idContatto, idServizio), or even on (idContatto, idServizio, ritirato). Either of those would supersede the existing index on just (idContatto).
John went the right direction. However the order of columns in the composite index needs changing.
For the GROUP BY, this order is needed (on ASSNZ):
INDEX(idServizio, idUtente)
(and that should replace KEY(idServizio), but not KEY(idUtente))
Then ACQ needs, in this order:
INDEX(idContatto, idServizio, stato_allievo)
replacing only KEY(idContatto).

MySQL Long First Query Time

I have a website that has two main queries to the database which are pretty slow the first time they are run, after a bit of testing it appears to be an issue with MySQL. If I run the query directly in Sequal Pro when I run a query the first time it can take up to 4 seconds to run but running the same query again takes ~60ms, the query time is about the same locally as on our server, which make me think its not a server issue.
Not totally sure that increasing the buffer pool size will help too much as the number of potential query combinations is probably around 800K.
The tables in the database are innodb, both queries access the same table that has 52K records, most of the information I need has been grouped together into a 'searchfield' field which is indexed.
Only fields used in queries or are primary/foreign keys are being indexed.
I have tried changing the inner joins to a select statement in the "where" of the main query but this doesn't make the query any faster.
The queries are
Query 1
SELECT
`item_attribute`.`attribute_id` AS `attribute_id`,
`attribute_value_id` AS `attribute_value_id`,
`collection_attribute`.`title` AS `ca_title`,
`collection_attribute`.`type` AS `ca_type`,
`collection_attribute`.`is_collapsible` AS `ca_is_collapsible`,
`collection_attribute`.`orderindex` AS `ca_orderindex`,
`collection_attribute`.`multi_select` AS `ca_multi_select`,
`item_attribute`.`item_id` AS `item_id`,
`product`.`id` AS `product_id`
FROM `item_attribute`
INNER JOIN `item` ON item.id = item_attribute.item_id
INNER JOIN `product` ON product.id = item.product_id
INNER JOIN `collection_attribute` ON item_attribute.attribute_id = collection_attribute.attribute_id
INNER JOIN `attribute_value` ON attribute_value.id = item_attribute.attribute_value_id
WHERE ((`product`.`searchfilter` LIKE '%c:35∆%') AND (`collection_attribute`.`collection_id`='35')) AND (`attribute_value`.`active`=1)
GROUP BY `attribute_value_id`
Query 2
SELECT DISTINCT `item_attribute`.`attribute_id` AS `attribute_id`,
GROUP_CONCAT(item_attribute.attribute_value_id SEPARATOR \"-\") AS `attribute_value`,
GROUP_CONCAT(attribute_value.title SEPARATOR \" - \") AS `title`
FROM `item_attribute`
LEFT JOIN `item` ON item.id = item_attribute.item_id
LEFT JOIN `attribute` ON attribute.id = item_attribute.attribute_id
LEFT JOIN `attribute_value` ON attribute_value.id = attribute_value_id
WHERE (`item`.`product_id`='894') AND (`attribute`.`is_option`=1)
GROUP BY `attribute_id`, `item_id`
Table Structure
CREATE TABLE `product` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(255) NOT NULL,
`slug` varchar(255) NOT NULL,
`sku` varchar(20) NOT NULL,
`active` tinyint(1) DEFAULT '0',
`orderindex` int(2) DEFAULT '-1',
`search` varchar(255) DEFAULT NULL,
`searchfilter` varchar(255) DEFAULT NULL,
`created_at` int(11) DEFAULT NULL,
`updated_at` int(11) DEFAULT NULL,
`protected` tinyint(1) DEFAULT '0',
`description` varchar(512) DEFAULT NULL,
`type_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `product_ibfk_1` (`type_id`),
KEY `searchfilter` (`searchfilter`),
CONSTRAINT `product_ibfk_1` FOREIGN KEY (`type_id`) REFERENCES `attribute_value` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=1882 DEFAULT CHARSET=utf8;
--
CREATE TABLE `collection_attribute` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`collection_id` int(11) DEFAULT NULL,
`attribute_id` int(11) DEFAULT NULL,
`title` varchar(512) NOT NULL,
`slug` varchar(512) NOT NULL,
`created_at` int(11) DEFAULT NULL,
`updated_at` int(11) DEFAULT NULL,
`active` tinyint(1) DEFAULT '0',
`orderindex` int(2) DEFAULT '-1',
`search` varchar(255) DEFAULT NULL,
`searchfilter` varchar(255) DEFAULT NULL,
`protected` tinyint(1) DEFAULT '0',
`is_collapsible` tinyint(1) DEFAULT '0',
`type` enum('icon','checkbox','checkboxIcon','image') DEFAULT NULL,
`multi_select` tinyint(1) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `collection_attribute_ibfk_1` (`collection_id`),
KEY `collection_attribute_ibfk_2` (`attribute_id`),
CONSTRAINT `collection_attribute_ibfk_1` FOREIGN KEY (`collection_id`) REFERENCES `collection` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `collection_attribute_ibfk_2` FOREIGN KEY (`attribute_id`) REFERENCES `attribute` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=140 DEFAULT CHARSET=utf8;
--
CREATE TABLE `item` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`slug` varchar(255) NOT NULL,
`title` varchar(255) NOT NULL,
`pattern_code` varchar(32) NOT NULL,
`tom_code` varchar(32) NOT NULL,
`navision_code` varchar(32) NOT NULL,
`description` varchar(255) DEFAULT NULL,
`active` tinyint(1) DEFAULT '0',
`orderindex` int(2) DEFAULT '-1',
`created_at` int(11) DEFAULT NULL,
`updated_at` int(11) DEFAULT NULL,
`search` varchar(1024) DEFAULT NULL,
`searchfilter` varchar(255) DEFAULT NULL,
`product_id` int(11) DEFAULT NULL,
`protected` tinyint(1) DEFAULT '0',
`pattern_series` varchar(255) DEFAULT NULL,
`pattern_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `item_ibfk_1` (`product_id`),
KEY `searchfilter` (`searchfilter`),
KEY `product_id` (`product_id`),
KEY `pattern_id` (`pattern_id`),
CONSTRAINT `item_ibfk_1` FOREIGN KEY (`product_id`) REFERENCES `product` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `item_ibfk_2` FOREIGN KEY (`pattern_id`) REFERENCES `pattern_series` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=50060 DEFAULT CHARSET=utf8;
--
CREATE TABLE `item_attribute` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_id` int(11) NOT NULL,
`attribute_id` int(11) NOT NULL,
`attribute_value_id` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `attribute_id` (`attribute_id`),
KEY `attribute_value_id` (`attribute_value_id`),
KEY `item_id` (`item_id`),
CONSTRAINT `item_attribute_ibfk_1` FOREIGN KEY (`attribute_id`) REFERENCES `attribute` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `item_attribute_ibfk_2` FOREIGN KEY (`attribute_value_id`) REFERENCES `attribute_value` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `item_attribute_ibfk_3` FOREIGN KEY (`item_id`) REFERENCES `item` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=857111 DEFAULT CHARSET=utf8;
--
CREATE TABLE `attribute_value` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`slug` varchar(255) NOT NULL,
`code` varchar(255) DEFAULT NULL,
`title` varchar(255) NOT NULL,
`active` tinyint(1) DEFAULT '0',
`orderindex` int(2) DEFAULT '-1',
`created_at` int(11) DEFAULT NULL,
`updated_at` int(11) DEFAULT NULL,
`search` varchar(255) DEFAULT NULL,
`searchfilter` varchar(255) DEFAULT NULL,
`attribute_id` int(11) DEFAULT NULL,
`protected` tinyint(1) DEFAULT '0',
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3471 DEFAULT CHARSET=utf8;
--
CREATE TABLE `attribute` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(255) NOT NULL,
`slug` varchar(255) NOT NULL,
`code` varchar(255) DEFAULT NULL,
`is_option` tinyint(1) DEFAULT '0',
`searches` tinyint(1) DEFAULT '0',
`option_type` enum('dropdown','switch','fingersizes') DEFAULT NULL,
`option_label` varchar(32) DEFAULT NULL,
`active` tinyint(1) DEFAULT '0',
`orderindex` int(2) DEFAULT '-1',
`created_at` int(11) DEFAULT NULL,
`updated_at` int(11) DEFAULT NULL,
`search` varchar(255) DEFAULT NULL,
`searchfilter` varchar(255) DEFAULT NULL,
`protected` tinyint(1) DEFAULT '0',
`option_requires` int(11) DEFAULT NULL,
`option_depends` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `is_option` (`is_option`)
) ENGINE=InnoDB AUTO_INCREMENT=47 DEFAULT CHARSET=utf8;
Any suggestions on how to improve the initial query time would be great
Thanks in advance :)
-- EDIT --
EXPLAIN SELECT query 1
EXPLAIN SELECT query 2
The first time you perform a query after starting the server, nothing is cache, so the query needs to fetch stuff from disk. All subsequent queries that access the same parts of the same tables will be much faster because of caching. This is "normal".
If you have the "Query cache" enable (it is probably enabled by default), then the second time you run exactly the same query, it will instantly find the result from the Query cache. By "exactly" I mean that not so much as a blank space has changed. Nearly all "production" servers are better off turning off the Query cache.
innodb_buffer_pool_size should be about 70% of available RAM. Changing the value won't affect a SELECT against a cold cache, but might help/hurt subsequent runs. This does not seem to be relevant in your case, since the second run was quite fast.
Please provide EXPLAIN SELECT ... so we can see how the optimizer decided to execute them.
LIKE '%c:35∆%' -- cannot use an index because of the leading wild card.
What is item_ids?
item_attribute is an EAV schema pattern. It sucks. Both the queries are ugly, and scalability hurts. It may help some to get rid of the id and make a compound PRIMARY KEY from a suitable combination of the other fields. The hope is to use the PRIMARY KEY which is clustered with the data instead of having to bounce from the secondary key. More discussion of EAV.
Assuming this has low cardinality, the index will probably never be used:
KEY is_option (is_option)