i have a three almost identical queries executed one by one in my app. Two of queries are executing fairly fast but one is much slower. Here are queries:
update `supremeshop_product_attributes` set `id_step` = 899 where `id_step` = 1 and `id_product` = 32641
540ms
update `supremeshop_product_attributes` set `id_step` = 1 where `id_step` = 0 and `id_product` = 32641
1.71s
update `supremeshop_product_attributes` set `id_step` = 0 where `id_step` = 899 and `id_product` = 32641
9.75ms
Create table query
CREATE TABLE `supremeshop_product_attributes` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`id_product` int(11) NOT NULL,
`id_attribute` int(11) NOT NULL,
`id_attribute_group` int(11) NOT NULL,
`id_step` int(11) NOT NULL,
`id_alias` int(9) NOT NULL,
`price_retail` int(11) NOT NULL DEFAULT '0',
`behaviour` int(1) NOT NULL DEFAULT '0',
`active` int(1) NOT NULL DEFAULT '1',
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `supremeshop_product_attributes_id_product_index` (`id_product`),
KEY `supremeshop_product_attributes_id_attribute_index` (`id_attribute`),
KEY `supremeshop_product_attributes_id_attribute_group_index` (`id_attribute_group`),
KEY `supremeshop_product_attributes_id_step_index` (`id_step`)
) ENGINE=InnoDB AUTO_INCREMENT=3012991 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
Facts:
table has 1500000 rows
each query updates around 650 rows
each searched/updated field has index (id(primary), id_product, id_step, id_alias)
only second query takes much longer time to execute (everytime - even executed as single query not one by one)
each variable used in query is integer
if i execute queries direcly in phpymysql -> sql - i get the same execution time (like in my laravel app so query is somehow slow)
Question is why? And is there a good explanation/ fix for that problem.
For any help many thanks!
Mark.
Related
I have done some research on the index before posting the question here. So far I believe I have done this correctly, but for some reason, the performance of a query that returns around 2400 records has not been good.
Here is the table schema
CREATE TABLE `tblCheck` (
`id` VARCHAR(50) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
`token` VARCHAR(50) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
`domainId` VARCHAR(50) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
`time` DATETIME NULL DEFAULT NULL,
`responseCode` INT(11) NULL DEFAULT NULL,
`totalTime` DECIMAL(10,2) NULL DEFAULT NULL,
`namelookupTime` INT(11) NULL DEFAULT NULL,
`connectTime` INT(11) NULL DEFAULT NULL,
`pretransferTime` INT(11) NULL DEFAULT NULL,
`startTransferTime` INT(11) NULL DEFAULT NULL,
`redirectTime` INT(11) NULL DEFAULT NULL,
`appconnectTime` INT(11) NULL DEFAULT NULL,
`responseText` TEXT(65535) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
`agentId` VARCHAR(50) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
`isHealthy` CHAR(50) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
`ftp_connect_time` DECIMAL(10,4) NULL DEFAULT NULL,
`ftp_login_time` DECIMAL(10,4) NULL DEFAULT NULL,
`ftp_change_mode_time` DECIMAL(10,4) NULL DEFAULT NULL,
`ftp_list_time` DECIMAL(10,4) NULL DEFAULT NULL,
`syntheticToken` VARCHAR(50) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
UNIQUE INDEX `id` (`id`) USING BTREE,
INDEX `domainId` (`domainId`) USING BTREE,
INDEX `deleteTime` (`time`) USING BTREE,
INDEX `SearchIndex` (`domainId`, `time`, `agentId`) USING BTREE
)
COLLATE='latin1_swedish_ci'
ENGINE=InnoDB
ROW_FORMAT=COMPACT
;
The Query
SELECT *
FROM `tblCheck`
WHERE (`time` BETWEEN '2020-05-04 22:15:04' AND '2020-05-05 22:15:04')
AND `domainId` = '03d4c1ce-8b13-11ea-abf5-124e96b5f417'
AND `agentId` != '145a-f6bb-11e8-983f-1231322cbdb6'
ORDER BY `time` DESC
;
/* Affected rows: 0 Found rows: 2,418 Warnings: 0 Duration for 1 query: 0.109 sec. (+ 10.360 sec. network) */
It returned 2418 rows but took almost 10s.
Running it with EXPLAIN
EXPLAIN SELECT *
FROM `tblCheck`
WHERE (`time` BETWEEN '2020-05-04 22:15:04' AND '2020-05-05 22:15:04')
AND `domainId` = '03d4c1ce-8b13-11ea-abf5-124e96b5f417'
AND `agentId` != '145a-f6bb-11e8-983f-1231322cbdb6'
ORDER BY `time` DESC
Returns this
This looks like it is using the index "SearchIndex". However, I can't figure out why it would take to process 10s for 2k rows
For this query:
SELECT *
FROM `tblCheck`
WHERE
`time` BETWEEN '2020-05-04 22:15:04' AND '2020-05-05 22:15:04'
AND `domainId` = '03d4c1ce-8b13-11ea-abf5-124e96b5f417'
AND `agentId` != '145a-f6bb-11e8-983f-1231322cbdb6'
ORDER BY `time` DESC
The right index would be: (domainId, agentId, time), or (domainId, time, agentId). You have an the second index in place, and the query plan shows that MySQL happily uses it.
Looking at the explain summary, you can see:
Duration for 1 query: 0.109 sec. (+ 10.360 sec. network)
The query runs fast in the database. The bottleneck is the network, that is the time taken to return the 2000+ rows from the database to the client. Not much can be done from database perspective. Speed up your network, or switch to a local database if you can.
As a side note: select * is a not good for performance; you should try and reduce the number of columns that the query returns (this might also reduce the amount that needs to transmitted over the network).
You don't have the primary key.
You only have a UNIQUE INDEX.
If you con't nee that bulky responseText column in the results, don't include it. This may dramatically speed up the query.
(This is because large columns are stored "off-record", thereby taking an extra disk read if the table is huge.)
I have a database with the records near about 2.7 milion . I need to fetch records from that for that i am using the below query
for result
SELECT r3.original_image_title,r3.uuid,r3.original_image_URL FROM `image_attributes` AS r1 INNER JOIN `filenames` as r3 WHERE r1.`uuid` = r3.`uuid` and r3.`status` = 1 and r1.status=1 and (r1.`attribute_name` like "Quvenzhané Wallis%" or r3.original_image_URL like "Quvenzhané Wallis%") group by r3.`uuid` limit 0,20
for total count
SELECT count(DISTINCT(r1.`uuid`)) as count FROM `image_attributes` AS r1 INNER JOIN `filenames` as r3 WHERE r1.`uuid` = r3.`uuid` and r3.`status` = 1 and r1.status=1 and (r1.`attribute_name` like "Quvenzhané Wallis%" or r3.original_image_URL like "Quvenzhané Wallis%")
table structures are as below
CREATE TABLE IF NOT EXISTS `image_attributes` (
`index` int(11) NOT NULL AUTO_INCREMENT,
`attribute_name` text NOT NULL,
`attribute_type` varchar(255) NOT NULL,
`uuid` varchar(255) NOT NULL,
`status` tinyint(1) NOT NULL DEFAULT '1',
PRIMARY KEY (`index`),
KEY `attribute_type` (`attribute_type`),
KEY `uuid` (`uuid`),
KEY `status` (`status`),
KEY `attribute_name` (`attribute_name`(50))
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=2730431 ;
CREATE TABLE IF NOT EXISTS `filenames` (
`index` int(11) NOT NULL AUTO_INCREMENT,
`original_image_title` text NOT NULL,
`original_image_URL` text NOT NULL,
`uuid` varchar(255) NOT NULL,
`status` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`index`),
KEY `uuid` (`uuid`),
KEY `original_image_URL` (`original_image_URL`(50))
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=591967 ;
please suggest me how can i optimize the queries to make the search result faster
I would recommend to you a book called 'High Performance MySql'. There is a section called Optimize databases and queries, or something like that.
I have the following database:
CREATE TABLE IF NOT EXISTS `musics` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`active` tinyint(1) NOT NULL DEFAULT '1',
`slug` varchar(50) NOT NULL,
`movie_id` int(11) NOT NULL,
`added` int(11) NOT NULL,
`updated` int(11) NOT NULL,
`featured` tinyint(1) NOT NULL,
`hits` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `slug` (`slug`),
KEY `active` (`active`),
KEY `featured` (`featured`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=3339 ;
And the query UPDATE musics SET hits = 7 WHERE id = '1770' gets run on each page load of a music album. Is there any way to optimize this query?
example..
set global long_query_time=1;
and, my.cnf
[mysqld]
...
long_query_time=1;
and you can use 'explain'.
explain select * from musics where id = 1770;
like the mysql docs say:
"The slow query log consists of SQL statements that took more than long_query_time seconds to execute" -> long_query_time is a Server System Variable from mysql, which could also be changed.
to the optimization-question:
you could write " = 1770" without ' because thats an integer and not a string
The following query, regardless of environment, takes more than 30 seconds to compute.
SELECT COUNT( r.response_answer )
FROM response r
INNER JOIN (
SELECT G.question_id
FROM question G
INNER JOIN answer_group AG ON G.answer_group_id = AG.answer_group_id
WHERE AG.answer_group_stat = 'statistic'
) AS q ON r.question_id = q.question_id
INNER JOIN org_survey os ON os.org_survey_code = r.org_survey_code
WHERE os.survey_id =42
AND r.response_answer = 5
AND DATEDIFF( NOW( ) , r.added_dt ) <1000000
AND r.uuid IS NOT NULL
When I explain the query,
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 1087
1 PRIMARY r ref question_id,org_survey_code,code_question,uuid,uor question_id 4 q.question_id 1545 Using where
1 PRIMARY os eq_ref org_survey_code,survey_id,org_survey_code_2 org_survey_code 12 survey_2.r.org_survey_code 1 Using where
2 DERIVED G ALL agid NULL NULL NULL 1680
2 DERIVED AG eq_ref PRIMARY PRIMARY 1 survey_2.G.answer_group_id 1 Using where
I have a very basic knowledge of indexing, but I have tried nearly every combination I can think of and cannot seem to improve the speed of this query. The responses table is right around 2 million rows, question is about 1500 rows, answer_group is about 50, and org_survey is about 8,000.
Here is the basic structure for each:
CREATE TABLE `response` (
`response_id` int(10) unsigned NOT NULL auto_increment,
`response_answer` text NOT NULL,
`question_id` int(10) unsigned NOT NULL default '0',
`org_survey_code` varchar(7) NOT NULL,
`uuid` varchar(40) default NULL,
`added_dt` datetime default NULL,
PRIMARY KEY (`response_id`),
KEY `question_id` (`question_id`),
KEY `org_survey_code` (`org_survey_code`),
KEY `code_question` (`org_survey_code`,`question_id`),
KEY `IDX_ADDED_DT` (`added_dt`),
KEY `uuid` (`uuid`),
KEY `response_answer` (`response_answer`(1)),
KEY `response_question` (`response_answer`(1),`question_id`),
) ENGINE=MyISAM AUTO_INCREMENT=2298109 DEFAULT CHARSET=latin1
CREATE TABLE `question` (
`question_id` int(10) unsigned NOT NULL auto_increment,
`question_text` varchar(250) NOT NULL default '',
`question_group` varchar(250) default NULL,
`question_position` tinyint(3) unsigned NOT NULL default '0',
`survey_id` tinyint(3) unsigned NOT NULL default '0',
`answer_group_id` mediumint(8) unsigned NOT NULL default '0',
`seq_id` int(11) NOT NULL default '0',
PRIMARY KEY (`question_id`),
KEY `question_group` (`question_group`(10)),
KEY `survey_id` (`survey_id`),
KEY `agid` (`answer_group_id`)
) ENGINE=MyISAM AUTO_INCREMENT=1860 DEFAULT CHARSET=latin1
CREATE TABLE `org_survey` (
`org_survey_id` int(11) NOT NULL auto_increment,
`org_survey_code` varchar(10) NOT NULL default '',
`org_id` int(11) NOT NULL default '0',
`org_manager_id` int(11) NOT NULL default '0',
`org_url_id` int(11) default '0',
`division_id` int(11) default '0',
`sector_id` int(11) default NULL,
`survey_id` int(11) NOT NULL default '0',
`process_batch` tinyint(4) default '0',
`added_dt` datetime default NULL,
PRIMARY KEY (`org_survey_id`),
UNIQUE KEY `org_survey_code` (`org_survey_code`),
KEY `org_id` (`org_id`),
KEY `survey_id` (`survey_id`),
KEY `org_survey_code_2` (`org_survey_code`,`total_taken`),
KEY `org_manager_id` (`org_manager_id`),
KEY `sector_id` (`sector_id`)
) ENGINE=MyISAM AUTO_INCREMENT=9268 DEFAULT CHARSET=latin1
CREATE TABLE `answer_group` (
`answer_group_id` tinyint(3) unsigned NOT NULL auto_increment,
`answer_group_name` varchar(50) NOT NULL default '',
`answer_group_type` varchar(20) NOT NULL default '',
`answer_group_stat` varchar(20) NOT NULL default 'demographic',
PRIMARY KEY (`answer_group_id`)
) ENGINE=MyISAM AUTO_INCREMENT=53 DEFAULT CHARSET=latin1
I know there are small things I can probably do to improve the efficiency of the database, such as reducing the size of integers where it's unnecessary. However, those are fairly trivial considering the ridiculous time it takes just to produce a result here. How can I properly index these tables, based on what explain has shown me? It seems that I have tried a large variety of combinations to no avail. Also, is there anything else that anyone can see that will optimize the table and reduce the query? I need it to be computed in less than a second. Thanks in advance!
1.If you want the index of r.added_dt to be used, instead of:
DATEDIFF(NOW(), r.added_dt) < 1000000
use:
CURDATE() - INTERVAL 1000000 DAY < r.added_dt
Anyway, the above condition is checking if added_at is a million days old or not. Do you really store so old dates? If not, you can simply remove this condition.
If you want this condition, an index on added_at would help a lot. Your query as it is now, checks all rows for this condition, calling the DATEDIFF() function as many times as the rows of the response table.
2.Since r.response_answer cannot be NULL, instead of:
SELECT COUNT( r.response_answer )
use:
SELECT COUNT( * )
COUNT(*) is faster than COUNT(field).
3.Two of the three fields that you use for joining tables have different datatypes:
ON question . answer_group_id
= answer_group . answer_group_id
CREATE TABLE question (
...
answer_group_id mediumint(8) ..., <--- mediumint
CREATE TABLE answer_group (
answer_group_id` tinyint(3) ..., <--- tinyint
-------------------------------
ON org_survey . org_survey_code
= response . org_survey_code
CREATE TABLE response (
...
org_survey_code varchar(7) NOT NULL, <--- 7
CREATE TABLE org_survey (
...
org_survey_code varchar(10) NOT NULL default '', <--- 10
Datatype mediumint is not the same as tinyint and the same goes for varchar(7) and varchar(10). When they are used for join, MySQL has to lose time doing conversion from one type to another. Convert one of them so they have identical datatypes. This is not the main issue of the query but this change will also help all other queries that use these joins.
And after making this change do a 'Analyze Table ' for the table. It will help mysql making better execution plans.
You have a response_answer = 5 condition, where response_answer is text. It's not an error, but it's better to use response_answer = '5' (the conversion of 5 to '5' will be done by MySQL anyway, if you don't do that).
Real issue is that you don't have a compound index on the 3 fields that are used in the WHERE conditions. Try adding this one:
ALTER TABLE response
ADD INDEX ind_u1_ra1_aa
(uuid(1), response_answer(1), added_at) ;
(this may take a while as your table is not small)
Can you try the following query? I've removed the sub-query from your original one. This may let the optimiser produce a better execution plan.
SELECT COUNT(r.response_answer)
FROM response r
INNER JOIN question q ON r.question_id = q.question_id
INNER JOIN answer_group ag ON q.answer_group_id = ag.answer_group_id
INNER JOIN org_survey os ON os.org_survey_code = r.org_survey_code
WHERE
ag.answer_group_stat = 'statistic'
AND os.survey_id = 42
AND r.response_answer = 5
AND DATEDIFF(NOW(), r.added_dt) < 1000000
AND r.uuid IS NOT NULL
I have the following two tables:
CREATE TABLE `temporal_expressions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`dated_obj_type` varchar(255) DEFAULT NULL,
`dated_obj_id` int(11) DEFAULT NULL,
`start_date` datetime DEFAULT NULL,
`end_date` datetime DEFAULT NULL,
`start_time` int(11) DEFAULT NULL,
`end_time` int(11) DEFAULT NULL,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`lock_version` int(11) NOT NULL DEFAULT '0',
`wday` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `te_search` (`dated_obj_type`,`dated_obj_id`,`start_date`,`end_date`),
KEY `te_calendar` (`dated_obj_type`,`dated_obj_id`,`start_date`,`end_date`,`start_time`,`end_time`),
KEY `te_search_wday` (`dated_obj_type`,`dated_obj_id`,`start_date`,`end_date`,`wday`),
KEY `te_calendar_wday` (`dated_obj_type`,`dated_obj_id`,`start_date`,`end_date`,`start_time`,`end_time`,`wday`),
KEY `te_index` (`wday`,`dated_obj_type`,`start_date`,`end_date`,`start_time`,`end_time`,`dated_obj_id`)
) ENGINE=InnoDB AUTO_INCREMENT=8162445 DEFAULT CHARSET=latin1
CREATE TABLE `asset_blocks` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`block_type` int(11) DEFAULT '0',
`spaces_left` int(11) DEFAULT NULL,
`provider_note` varchar(255) DEFAULT NULL,
`extra_data` text,
`lock_version` int(11) DEFAULT '0',
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`type` varchar(255) DEFAULT NULL,
`service_provider_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `type` (`type`,`id`),
KEY `service_provider_id` (`service_provider_id`,`type`,`id`),
) ENGINE=InnoDB AUTO_INCREMENT=516867 DEFAULT CHARSET=latin1
If I run explain on this query (note that I am only selecting fields in the te_calendar_wday index from temporal_expressions) it uses the index for the join as expected
EXPLAIN SELECT asset_blocks.*, temporal_expressions.id,
temporal_expressions.dated_obj_type, temporal_expressions.dated_obj_id,
temporal_expressions.start_date, temporal_expressions.end_date,
temporal_expressions.start_time
FROM `asset_blocks`
LEFT OUTER JOIN `temporal_expressions`
ON `temporal_expressions`.dated_obj_id = `asset_blocks`.id
AND `temporal_expressions`.dated_obj_type = 'AssetBlock'
WHERE ( temporal_expressions.start_date <= '2010-11-25'
AND temporal_expressions.end_date >= '2010-11-01'
AND temporal_expressions.start_time < 1000 AND temporal_expressions.end_time > 1200
AND temporal_expressions.wday IN (1,2,3,4,5,6)
AND asset_blocks.id IN (1,2,3,4,5,6,7,8,9) )
1 SIMPLE temporal_expressions range te_search,te_calendar,te_search_wday,te_calendar_wday,te_index te_calendar_wday 272 NULL 9 Using where; Using index
1 SIMPLE asset_blocks eq_ref PRIMARY PRIMARY 4 lb_production.temporal_expressions.dated_obj_id 1
However, if I run this query (note that I have added a non-indexed field to the field list) it no longer uses the index (it uses a join buffer). Is this intentional or am I missing something?
EXPLAIN SELECT asset_blocks.*, temporal_expressions.id,
temporal_expressions.dated_obj_type, temporal_expressions.dated_obj_id,
temporal_expressions.start_date, temporal_expressions.end_date,
temporal_expressions.start_time, temporal_expressions.created_at
FROM `asset_blocks`
LEFT OUTER JOIN `temporal_expressions`
ON `temporal_expressions`.dated_obj_id = `asset_blocks`.id
AND `temporal_expressions`.dated_obj_type = 'AssetBlock'
WHERE ( temporal_expressions.start_date <= '2010-11-25'
AND temporal_expressions.end_date >= '2010-11-01'
AND temporal_expressions.start_time < 1000 AND temporal_expressions.end_time > 1200
AND temporal_expressions.wday IN (1,2,3,4,5,6)
AND asset_blocks.id IN (1,2,3,4,5,6,7,8,9) )
1 SIMPLE asset_blocks range PRIMARY PRIMARY 4 NULL 9 Using where
1 SIMPLE temporal_expressions range te_search,te_calendar,te_search_wday,te_calendar_wday,new_te_index te_search 272 NULL 9 Using where; Using join buffer
I cannot be sure if this is the case here, but:
If you select only indexed fields, MySQL can answer the whole query out of the index and does not even load the table data file.
If you select a field that is not indexed, it has to load the table data.
When making its execution plan, in certain cases (see comment) MySQL decides to do a full table scan although an index is present. This is because it's much quicker to read all data blindly than to look up every entry in the index and then read the data.