How can I optimize sql for price and timestamp fields? - mysql

checking explain (mysqlnd 8.1):
explain analyze SELECT *
FROM `ts_products` WHERE `ts_products`.`creator_id` = '2' AND
`ts_products`.`title` like '%no%' AND
`ts_products`.`status` = 'A' AND
ts_products.published_at >= '2022-04-01' AND
ts_products.published_at < '2022-04-21' AND
ts_products.sale_price > '261'
ORDER BY `sale_price` asc
on table :
CREATE TABLE `ts_products` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`creator_id` bigint unsigned NOT NULL,
`title` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`status` enum('D','P','A','I') COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'D' COMMENT ' D => Draft, P=>Pending Review, A=>Active, I=>Inactive',
`slug` varchar(260) COLLATE utf8mb4_unicode_ci NOT NULL,
`sku` varchar(100) COLLATE utf8mb4_unicode_ci NOT NULL,
`regular_price` decimal(9,2) DEFAULT NULL,
`sale_price` decimal(9,2) DEFAULT NULL,
`in_stock` tinyint(1) NOT NULL DEFAULT '0',
`stock_qty` mediumint unsigned NOT NULL DEFAULT '0',
`has_discount_price` tinyint(1) NOT NULL DEFAULT '0',
`is_featured` tinyint(1) NOT NULL DEFAULT '0',
`short_description` mediumtext COLLATE utf8mb4_unicode_ci,
`description` longtext COLLATE utf8mb4_unicode_ci,
`published_at` datetime DEFAULT NULL,
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `ts_products_slug_index` (`slug`),
UNIQUE KEY `ts_products_sku_index` (`sku`),
UNIQUE KEY `ts_products_creator_id_3fields_index` (`creator_id`,`sale_price`,`status`,`title`,`published_at`),
KEY `ts_products_status_title_regular_price_published_at_index` (`status`,`title`,`regular_price`,`published_at`),
KEY `ts_products_status_5fields_index` (`status`,`in_stock`,`has_discount_price`,`is_featured`,`title`,`published_at`,`stock_qty`,`sale_price`),
CONSTRAINT `ts_products_creator_id_foreign` FOREIGN KEY (`creator_id`) REFERENCES `users` (`id`) ON DELETE RESTRICT ON UPDATE RESTRICT
) ENGINE=InnoDB AUTO_INCREMENT=801 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
I see that ts_products_creator_id_3fields_index index is selected and seems good, but looking at output :
-> Index range scan on ts_products using ts_products_creator_id_3fields_index over (creator_id = 2 AND 261.00 < sale_price), with index condition: ((ts_products.creator_id = 2) and (ts_products.title like '%no%') and (ts_products.`status` = 'A') and (ts_products.published_at >= TIMESTAMP'2022-04-01 00:00:00') and (ts_products.published_at < TIMESTAMP'2022-04-21 00:00:00') and (ts_products.sale_price > 261.00)) (cost=20.06 rows=44) (actual time=0.070..0.179 rows=2 loops=1)
I wonder if how signs ">=" / "<" / ">" work where. AS far as I know only '=' works for indices.
If there is sense to add sale_price and published_at in ts_products_creator_id_3fields_index index ?
How can I optimize this request ?
Thanks!

This might help:
INDEX(creator_id, status, sale_price)
There is one column that is unindexable (title like '%no%') because of the leading wildcard, and 2 ranges: published_at and sale_price. I picked the latter because it might be useful both in WHERE and in ORDER BY.
Please provide the EXPLAIN.
As discussed in Index Cookbook , columns tested by = should come first in a composite index, followed by one 'range' column.

Related

MySQL COUNT(Joining table) is taking too long to fetch the data inside large tables

Description:
We have two tables as below:
table_1 ("question" main table)
table_2 ("question_attempted" joining table)
Cases:
In "table_2" we have a column that has a column "is_correct" (holds 1,0) for right or wrong answers.
In "table_1" we have 1 m records and in "table_2" we have 10m records
We want to sort our listing data by below columns/values:
Total number of times questions were attempted
Total number of times questions were answered correctly
The percentages questions were answered correctly (based on above two values)
Issue:
As soon as we join the table_1 and table_2 to get the count of total_questions_attempted, total_questiones_give_correct_answer, perntage_corrected_given_answers. The query starts taking around 8-10 minutes to run. Table structures are given below. Thanks in advance.
Table structures:
CREATE TABLE IF NOT EXISTS `question` (
`id` bigint(20) UNSIGNED NOT NULL AUTO_INCREMENT,
`category` bigint(20) NOT NULL DEFAULT 0,
`parent` bigint(20) UNSIGNED NOT NULL DEFAULT 0,
`name` text COLLATE utf8mb4_unicode_ci NOT NULL,
`questiontext` text COLLATE utf8mb4_unicode_ci NOT NULL,
`questiontextformat` tinyint(4) NOT NULL DEFAULT 0,
`generalfeedback` text COLLATE utf8mb4_unicode_ci NOT NULL,
`generalfeedbackformat` tinyint(4) NOT NULL DEFAULT 0,
`defaultmark` decimal(12,7) NOT NULL DEFAULT 1.0000000,
`penalty` decimal(12,7) NOT NULL DEFAULT 0.3333333,
`qtype` varchar(20) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '' COMMENT '''1''',
`length` bigint(20) UNSIGNED NOT NULL DEFAULT 1,
`stamp` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
`version` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
`hidden` tinyint(3) UNSIGNED NOT NULL DEFAULT 0,
`timecreated` bigint(20) UNSIGNED NOT NULL DEFAULT 0,
`timemodified` bigint(20) UNSIGNED NOT NULL DEFAULT 0,
`createdby` bigint(20) UNSIGNED DEFAULT NULL,
`modifiedby` bigint(20) UNSIGNED DEFAULT NULL,
`type_data_id` bigint(20) NOT NULL,
`img_id` bigint(20) DEFAULT NULL,
`qimg_gallary_text` text COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`qrimg_gallary_text` text COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`qimg_gallary_ids` text COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`qrimg_gallary_ids` text COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`case_id` bigint(20) NOT NULL DEFAULT 0,
`ques_type_id` bigint(20) DEFAULT NULL,
`year` bigint(20) DEFAULT NULL,
`spec` bigint(20) DEFAULT NULL,
`sub_speciality_id` int(11) DEFAULT NULL,
`sub_sub_speciality_id` int(11) DEFAULT NULL,
`spec_level` bigint(20) DEFAULT 1,
`is_deleted` int(11) NOT NULL DEFAULT 0,
`sequence` int(11) NOT NULL DEFAULT 0,
`sort_order` bigint(20) NOT NULL DEFAULT 0 COMMENT 'Question order in list',
`idnumber` varchar(100) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`addendum` text COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`text_for_search` longtext COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'this is for the text based searching, this will store the text of the question without html tags',
`text_for_search_ans` longtext COLLATE utf8mb4_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `type_data_id` (`type_data_id`),
UNIQUE KEY `mdl_ques_catidn_uix` (`category`,`idnumber`),
KEY `mdl_ques_cat_ix` (`category`),
KEY `mdl_ques_par_ix` (`parent`),
KEY `mdl_ques_cre_ix` (`createdby`),
KEY `mdl_ques_mod_ix` (`modifiedby`),
KEY `id` (`id`),
KEY `mq_spec_ix` (`spec`),
KEY `sort_order` (`sort_order`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci COMMENT='The questions themselves';
CREATE TABLE IF NOT EXISTS `question_attempted` (
`id` bigint(20) UNSIGNED NOT NULL AUTO_INCREMENT,
`questionusageid` bigint(20) UNSIGNED NOT NULL,
`slot` bigint(20) UNSIGNED NOT NULL,
`behaviour` varchar(32) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
`questionid` bigint(20) UNSIGNED NOT NULL,
`variant` bigint(20) UNSIGNED NOT NULL DEFAULT 1,
`maxmark` decimal(12,7) NOT NULL,
`minfraction` decimal(12,7) NOT NULL,
`flagged` tinyint(3) UNSIGNED NOT NULL DEFAULT 2,
`questionsummary` text COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`rightanswer` text COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`responsesummary` text COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`timemodified` bigint(20) UNSIGNED NOT NULL,
`maxfraction` decimal(12,7) DEFAULT 1.0000000,
`in_remind_state` int(11) NOT NULL DEFAULT 0,
`is_correct` tinyint(1) DEFAULT 1,
PRIMARY KEY (`id`),
UNIQUE KEY `mdl_quesatte_queslo_uix` (`questionusageid`,`slot`),
KEY `mdl_quesatte_que_ix` (`questionid`),
KEY `mdl_quesatte_que2_ix` (`questionusageid`),
KEY `mdl_quesatte_beh_ix` (`behaviour`),
KEY `questionid` (`questionid`),
KEY `is_correct` (`is_correct`)
) ENGINE=InnoDB AUTO_INCREMENT=151176 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci COMMENT='Each row here corresponds to an attempt at one question, as ';
I tried with the below query:
SELECT mq.id, mq.name, COUNT(is_correct)
FROM mdl_question_attempts as mqa
LEFT JOIN mdl_question mq on mq.id = mqa.questionid where mq.id IS NOT NULL and mq.is_deleted = '0'
GROUP by mqa.questionid
ORDER by mq.sort_order desc, mq.id DESC
LIMIT 50
https://i.stack.imgur.com/mHK6W.png
The correct query is
SELECT mq.id, mq.name, COUNT(mqa.questionid)
FROM mdl_question mq
LEFT JOIN mdl_question_attempts mqa ON mq.id = mqa.questionid AND mqa.is_correct
WHERE NOT mq.is_deleted
GROUP by mq.id
ORDER by mq.sort_order DESC, mq.id DESC
LIMIT 50;
Now let's see, how fast this can get. There is just one criteria on the question table (WHERE NOT mq.is_deleted). We can probably assume that very many if not most questions are not deleted, so using an index here makes no sense on first glance; reading the full table seems quicker.
Then we outer join the answers on the question ID and the is_correct flag. This means we should at least have an index on the ID, better even on the ID and the flag:
CREATE INDEX idx1 ON mdl_question_attempts (questionid, is_correct);
Now we must order all rows by the question's sort_order and ID to get the first 50 rows. It would be great to have an index that is already sorted, so we could just take the first 50 entries from there. But then, we are only looking at rows matching NOT mq.is_deleted, so the index must include that flag:
CREATE INDEX idx2 ON mdl_question (is_deleted, sort_order DESC, id DESC);
We could even include the name, so all data is available from the index and the table must not be read anymore (covering index).
CREATE INDEX idx2 ON mdl_question (is_deleted, sort_order DESC, id DESC, name);
It is still up to the DBMS to use these indexes or not. We are just providing them to give the DBMS the option. With this query it depends on how well MySQL's optimizer works. Does it see that it can just read the 50 first entries from the question index and then use the answer index for the simple counting?
add index in your table
CREATE INDEX index_name ON table_name (column_name);
references : https://www.w3schools.com/sql/sql_create_index.asp

Optimizing MySQL query to reduced runtime

Below is the query that will going to run on two tables with 60+ million and 400+ million records. Only the table name will be different, otherwise query is same for both the tables.
SELECT * FROM
(
SELECT A.CUSIP, A.ISIN, A.SEDOL, A.LocalCode, A.MIC, A.ExchgCD, A.PrimaryExchgCD, A.Currency, A.Open, A.High, A.Low, A.Close, A.Mid, A.Ask, A.Last,
A.Bid, A.Bidsize, A.Asksize, A.TradedVolume, A.SecID, A.PriceDate, A.MktCloseDate, A.VolFlag, A.IssuerName, A.TotalTrades, A.CloseType, A.SectyCD,
row_number() OVER (partition by A.CUSIP order by A.MktCloseDate desc) as 'rank'
from EDI_Price04 A
WHERE A.CUSIP IN (
"91879Q109", "583840509", "583840608", "59001A102", "552848103") AND (A.PrimaryExchgCD = A.ExchgCD) AND A.CloseType='CC'
) t WHERE t.rank <= 3;
When A.CUSIP IN () condition have 10-15 values, the query complete in 2-3sec. With 400 values it took 28sec. But I want to make A.CUSIP IN () take 2k-3k value at a time.
This is my table structure.
CREATE TABLE `EDI_Price04` (
`MIC` varchar(6) NOT NULL DEFAULT '',
`LocalCode` varchar(60) NOT NULL DEFAULT '' COMMENT 'PricefileSymbol',
`ISIN` varchar(12) DEFAULT NULL,
`Currency` varchar(3) NOT NULL DEFAULT '',
`PriceDate` date DEFAULT NULL,
`Open` double DEFAULT NULL,
`High` double DEFAULT NULL,
`Low` double DEFAULT NULL,
`Close` double DEFAULT NULL,
`Mid` double DEFAULT NULL,
`Ask` double DEFAULT NULL,
`Last` double DEFAULT NULL,
`Bid` double DEFAULT NULL,
`BidSize` int(11) DEFAULT NULL,
`AskSize` int(11) DEFAULT NULL,
`TradedVolume` bigint(20) DEFAULT NULL,
`SecID` int(11) NOT NULL DEFAULT '0',
`MktCloseDate` date NOT NULL DEFAULT '0000-00-00',
`Volflag` char(1) DEFAULT NULL,
`IssuerName` varchar(255) DEFAULT NULL,
`SectyCD` varchar(3) DEFAULT NULL,
`SecurityDesc` varchar(255) DEFAULT NULL,
`SEDOL` varchar(7) DEFAULT NULL,
`CUSIP` varchar(9) DEFAULT NULL COMMENT 'USCode',
`PrimaryExchgCD` varchar(6) DEFAULT NULL,
`ExchgCD` varchar(6) NOT NULL DEFAULT '',
`TradedValue` double DEFAULT NULL,
`TotalTrades` int(11) DEFAULT NULL,
`Comment` varchar(255) DEFAULT NULL,
`Repush` tinyint(4) NOT NULL DEFAULT '0',
`CloseType` varchar(2) NOT NULL DEFAULT '',
PRIMARY KEY (`MIC`,`LocalCode`,`Currency`,`SecID`,`MktCloseDate`,`ExchgCD`,`Repush`,`CloseType`),
KEY `idx_EDI_Price04_0` (`MIC`),
KEY `idx_EDI_Price04_1` (`LocalCode`),
KEY `idx_EDI_Price04_2` (`ISIN`),
KEY `idx_EDI_Price04_3` (`PriceDate`),
KEY `idx_EDI_Price04_4` (`SEDOL`),
KEY `idx_EDI_Price04_5` (`CUSIP`),
KEY `idx_EDI_Price04_6` (`PrimaryExchgCD`),
KEY `idx_EDI_Price04_7` (`ExchgCD`),
KEY `idx_EDI_Price04_8` (`CloseType`),
KEY `idx_EDI_Price04_9` (`MktCloseDate`),
KEY `idx_EDI_Price04_CUSIP_ExchgCD_CloseType_MktCloseDate` (`CUSIP`,`ExchgCD`,`CloseType`,`MktCloseDate`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
For this query:
SELECT *
FROM (SELECT a.*
ROW_NUMBER() OVER (PARTITION BY A.CUSIP ORDER BY A.MktCloseDate DESC) as rank
FROM EDI_Price04 A
WHERE A.CUSIP IN ('91879Q109', '583840509', '583840608', '59001A102', '552848103') AND
A.PrimaryExchgCD = A.ExchgCD AND
A.CloseType = 'CC'
) t
WHERE t.rank <= 3;
The place to start is with an index. For this query, you want an index on EDI_Price04(CloseType, CUSIP, ExchgCD, MktCloseDate).
Unfortunately, the condition A.PrimaryExchgCD = A.ExchgCD prevents index seeks. If you were to make changes to the query/data, then one approach would be to add a flag when these are the same, rather then looking at the values separately. That would allow an index on:
EDI_Price04(CloseType, IsPrimary, CUSIP, PrimaryExchgCD, ExchgCD, MktCloseDate)
PRIMARY KEY (id),
UNIQUE(`MIC`,`LocalCode`,`Currency`,`SecID`,`MktCloseDate`,
`ExchgCD`,`Repush`,`CloseType`),
-- KEY `idx_EDI_Price04_0` (`MIC`),
KEY `idx_EDI_Price04_1` (`LocalCode`),
KEY `idx_EDI_Price04_2` (`ISIN`),
KEY `idx_EDI_Price04_3` (`PriceDate`),
KEY `idx_EDI_Price04_4` (`SEDOL`),
-- KEY `idx_EDI_Price04_5` (`CUSIP`),
KEY `idx_EDI_Price04_6` (`PrimaryExchgCD`),
KEY `idx_EDI_Price04_7` (`ExchgCD`),
KEY `idx_EDI_Price04_8` (`CloseType`),
KEY `idx_EDI_Price04_9` (`MktCloseDate`),
KEY `idx_EDI_Price04_CUSIP_ExchgCD_CloseType_MktCloseDate` (`CUSIP`,
`ExchgCD`, `CloseType`, `MktCloseDate`)
KEY (CUSIP, MktCloseDate)
Having so many columns in the PK costs in space and insert time. So, I added an id, which needs to be AUTO_INCREMENT.
Keys 0 and 5 are redundant because of the rule "If you have INDEX(a,b), INDEX(a) redundant.
I added (CUSIP, MktCloseDate) in hopes that it will optimize the RANK expression.

Why do this query takes more than 180 secs to run?

CREATE TABLE `tvnotif` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`pingId` int(11) DEFAULT NULL,
`token` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`summary` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`startTime` int(11) DEFAULT NULL,
`endTime` int(11) DEFAULT NULL,
`processed` int(1) DEFAULT '0',
`created` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`modified` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `processedIndex` (`processed`),
KEY `summaryIndex` (`summary`),
KEY `tokenIndex` (`token`)
) ENGINE=InnoDB AUTO_INCREMENT=18297898 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
CREATE TABLE `vv_us` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`athleteid` int(11) DEFAULT NULL,
`token` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`secret` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`active` int(1) DEFAULT '1',
`created` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`modified` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`type` varchar(45) COLLATE utf8_bin DEFAULT 'mc',
`step` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`host` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`server` mediumblob,
`tempcreds` mediumblob,
PRIMARY KEY (`id`),
KEY `activeIndex` (`active`),
KEY `typeIndex` (`type`)
) ENGINE=InnoDB AUTO_INCREMENT=33888 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
I am running a query which is mostly based on the table tvnotif which has at least 2 million rows of data in it, all the other tables are having less amount only. I added the index for the tables before that the query took 20 minutes to run and now its taking 160 secs.
EXPLAIN SELECT tvu.secret,COALESCE(php_timezone,"America/Los_Angeles") AS userTz,tn.*,tvu.athleteid,tvu.type FROM tvnotif AS tn
LEFT JOIN vv_us AS tvu ON tvu.token = tn.token
LEFT JOIN tbl_ath_pro AS tap ON tap.athleteid = tvu.athleteid
LEFT JOIN timezones AS tz ON tz.tz_id = tap.tz_id
WHERE tvu.active = 1 AND tn.summary = 'dailies' AND tn.processed = 0
LIMIT 300
The problem is probably your indexes... You have indexes on each field individually. What you need is a composite index on ALL 3 parts as a single index. Without, it can't pick the best one as you have 3 parts of the where clause.
Build a SINGLE index on ( processed, summary, token )
This way the query can jump directly to the processed records, directly to the summary value and then get those records and be done.
Additionally, your VV_US table should have an index on ( token, active ) so the join will be optimized on BOTH parts.

Mysql make 3 queries in 1 and improve performance

I'm trying to create a report and running 4 queries, but performance is so terrible.
I'm using 2 tables
This one has 2500 items in it
CREATE TABLE `bolt_accounts` (
`id` int(11) NOT NULL,
`slug` varchar(128) COLLATE utf8_unicode_ci NOT NULL,
`datecreated` datetime NOT NULL,
`datechanged` datetime NOT NULL,
`datepublish` datetime DEFAULT NULL,
`datedepublish` datetime DEFAULT NULL,
`username` varchar(32) COLLATE utf8_unicode_ci DEFAULT '',
`ownerid` int(11) DEFAULT NULL,
`status` varchar(32) COLLATE utf8_unicode_ci NOT NULL,
`templatefields` longtext COLLATE utf8_unicode_ci COMMENT '(DC2Type:json_array)',
`managerid` varchar(128) COLLATE utf8_unicode_ci DEFAULT '',
`parentid` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`name` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`qualify` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`regdate` date DEFAULT NULL,
`city` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`phone` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`passhash` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`cookie` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`resettoken` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`block` tinyint(1) NOT NULL DEFAULT '0',
`blocksms` tinyint(1) NOT NULL DEFAULT '0',
`birthday` date DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
ALTER TABLE `bolt_accounts`
ADD PRIMARY KEY (`id`),
ADD KEY `IDX_9C703491989D9B62` (`slug`),
ADD KEY `IDX_9C703491AFBA6FD8` (`datecreated`),
ADD KEY `IDX_9C703491BE74E59A` (`datechanged`),
ADD KEY `IDX_9C703491A5131421` (`datepublish`),
ADD KEY `IDX_9C703491B7805520` (`datedepublish`),
ADD KEY `IDX_9C7034917B00651C` (`status`),
ADD KEY `IDX_9C703491C13A5CC2` (`managerid`),
ADD KEY `IDX_9C703491856A684C` (`parentid`(255)),
ADD KEY `IDX_9C7034911E6AC3AE` (`regdate`),
ADD KEY `IDX_9C7034914709B432` (`birthday`);
and another one with all statistics, it has more than 1 400 000 items in it
CREATE TABLE `bolt_statistics` (
`id` int(11) NOT NULL,
`slug` varchar(128) COLLATE utf8_unicode_ci NOT NULL,
`datecreated` datetime NOT NULL,
`datechanged` datetime NOT NULL,
`datepublish` datetime DEFAULT NULL,
`datedepublish` datetime DEFAULT NULL,
`username` varchar(32) COLLATE utf8_unicode_ci DEFAULT '',
`ownerid` int(11) DEFAULT NULL,
`status` varchar(32) COLLATE utf8_unicode_ci NOT NULL,
`templatefields` longtext COLLATE utf8_unicode_ci COMMENT '(DC2Type:json_array)',
`managerid` varchar(256) COLLATE utf8_unicode_ci DEFAULT '',
`statdate` datetime DEFAULT NULL,
`lopv` double NOT NULL DEFAULT '0',
`gope` double NOT NULL DEFAULT '0',
`gopv` double NOT NULL DEFAULT '0'
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
ALTER TABLE `bolt_statistics`
ADD PRIMARY KEY (`id`),
ADD KEY `IDX_BE38DFD2989D9B62` (`slug`),
ADD KEY `IDX_BE38DFD2AFBA6FD8` (`datecreated`),
ADD KEY `IDX_BE38DFD2BE74E59A` (`datechanged`),
ADD KEY `IDX_BE38DFD2A5131421` (`datepublish`),
ADD KEY `IDX_BE38DFD2B7805520` (`datedepublish`),
ADD KEY `IDX_BE38DFD27B00651C` (`status`),
ADD KEY `IDX_BE38DFD2C13A5CC2` (`managerid`(255));
So the problem is, when I join this tables together, performance become low...
SELECT ba.managerid,name,replace(phone,'+','') as phone,passhash, date_format(ba.datepublish,'%d.%m.%Y %H:%i') as datepublish, max(bs.lopv) as lopv, max(bs.gopv) as gopv
FROM bolt_accounts ba
LEFT JOIN bolt_statistics bs ON ba.managerid=bs.managerid
WHERE (parentid='007-645930')
AND (date(ba.datechanged)=('2018-01-06'))
AND (date(bs.datecreated)=('2018-01-06'))
GROUP BY ba.managerid
ORDER BY gopv desc
this query will run for 360-450ms ~0,3 of a sec..
It will return all managerids that has parentid=007-645930
some thing like that:
managerid
007-663360
007-677590
007-697191
007-1526400
007-1155884
007-1842169
077-1564660
007-1883072
007-777143
007-1865946
007-1875083
007-1753407
007-1322124
007-1100631
007-1603795
007-1171656
007-1890892
007-1166247
007-1564611
007-1882959
007-1145375
007-1878383
007-1128857
007-1762655
007-1346877
007-1714252
007-1709538
007-1319044
007-1698517
007-1316756
007-1679094
007-1298984
007-1905146
007-1675451
007-1287166
007-1899632
007-1629224
007-1190862
007-1894824
007-1616741
007-1171665
007-1894330
Than I take 1 id from that list, and run 3 queries
SELECT max(s.lopv) as lopv, max(s.gopv) as gopv
FROM bolt_statistics s WHERE (managerid='007-663360')
AND (datecreated between DATE_FORMAT('2018-01-06' - INTERVAL 1 MONTH,'%Y-%m-28 23:00:00') and DATE_FORMAT(LAST_DAY('2018-01-06' - INTERVAL 1 MONTH),'%Y-%m-%d 23:59:59'))
execution time 20-25ms
SELECT max(s.lopv) as lopv, max(s.gopv) as gopv
FROM bolt_statistics s
WHERE (managerid='007-663360')
AND (date(datecreated) = date('2018-01-06' -INTERVAL 1 day))
execution time 15-20ms
SELECT max(s.lopv) as lopv, max(s.gopv) as gopv
FROM bolt_statistics s
WHERE (managerid='007-663360')
AND (date(datecreated) = date('2018-01-06' -INTERVAL 2 day))
execution time 15-20ms
When all executions are over, it took 1,5 sec (1500ms) to render the php report.
I know, that I'm not quite good at mysql querying ;)) but I wonder, how can I improve performance on that queries?
Will it be much faster if I union all this queries in 1?
Do those fields really need a full 256 characters? Change them to a reasonable number, then get rid of the prefixing on ADD KEY IDX_BE38DFD2C13A5CC2 (managerid(255)), etc. (Prefix indexes are often useless.)
Don't hid columns inside functions (date(ba.datechanged)). Instead:
AND ba.datechanged >= '2018-01-06' - INTERVAL 2 DAY
AND ba.datechanged < '2018-01-06' - INTERVAL 3 DAY
Note: The above pattern works fine regardless of what datatype datechanged is -- DATE, DATETIME, DATETIME(6), TIMESTAMP. And the Optimizer can make use of an index such as ...
After that, have the composite INDEX(managerid, datecreated) for significant performance improvement.
Use a derived table instead of LEFT JOIN plus GROUP BY. This is likely to improve speed a bunch.
What is status? Why VARCHAR(32)? If it is a simple, low-cardinality, value, don't index it by itself; the index won't be used.
(There may be more tips, but this should get you started.)

Fast FullText search for one occurrence of exact phrase in MySQL sorted by id

I'm using MySQL/InnoDb. I have this table News which has about 3,000,000 records and I hope it gets 7,000,000 more. Currently I use the following query to fetch records from News table with exact phrase inside their title or content:
SELECT * FROM News WHERE MATCH (title, content) AGAINST ('"My Phrase"' in Boolean Mode)
But the results aren't satisfying. For me, when searching for an exact phrase, it's logical to see the latest news with that phrase first. I don't need the results to be sorted by relevance. I don't mind if the first record found has 1 occurrence of my phrase and the second one has 10. All I care is ORDER BY time_added or ORDER BY id. So I tried:
SELECT * FROM News WHERE MATCH (title, content) AGAINST ('"My Phrase"' in Boolean Mode) ORDER BY id DESC limit 40
But when I try to EXPLAIN this query, I see that it is Using filesort as I excpected.
Is it the only way MySQL works? Can I get the results sorted by time in an optimal way?
Edit:
MySQL version: 5.6.34
News table structure is as follows:
CREATE TABLE `News` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`title` varchar(400) COLLATE utf8_bin NOT NULL,
`url_title` varchar(80) COLLATE utf8_bin NOT NULL,
`link` varchar(2000) COLLATE utf8_bin NOT NULL,
`link_unique_index` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`source` smallint(5) unsigned NOT NULL COMMENT,
`category` smallint(5) unsigned NOT NULL DEFAULT '0',
`description` varchar(600) COLLATE utf8_bin NOT NULL,
`has_content` tinyint(1) NOT NULL DEFAULT '0',
`content` text COLLATE utf8_bin NOT NULL,
`has_image` tinyint(1) NOT NULL,
`image` varchar(14) COLLATE utf8_bin NOT NULL,
`image_orientation` char(1) COLLATE utf8_bin NOT NULL,
`original_image` varchar(2000) COLLATE utf8_bin NOT NULL,
`keywords` varchar(300) COLLATE utf8_bin NOT NULL,
`year_added` smallint(4) NOT NULL,
`date_added` date NOT NULL,
`time_added` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`time_updated` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`number_of_views` int(11) NOT NULL DEFAULT '0',
`last_view_time` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`number_of_positives` int(11) NOT NULL DEFAULT '0',
`number_of_negatives` int(11) NOT NULL DEFAULT '0',
`number_of_votes` int(11) NOT NULL DEFAULT '0',
`number_of_suggestions` int(11) NOT NULL DEFAULT '0',
`last_suggestion_time` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`id`),
UNIQUE KEY `lui` (`link_unique_index`) USING BTREE,
KEY `st` (`source`,`time_added`),
KEY `snv` (`source`,`number_of_views`),
KEY `snvo` (`source`,`number_of_votes`),
KEY `sct` (`source`,`category`,`time_added`),
KEY `scnv` (`source`,`category`,`number_of_views`),
KEY `scnvo` (`source`,`category`,`number_of_votes`),
KEY `ct` (`category`,`time_added`),
KEY `cnv` (`category`,`number_of_views`),
KEY `cnvo` (`category`,`number_of_votes`),
KEY `hicdnv` (`has_image`,`category`,`date_added`,`number_of_views`),
KEY `sdnv` (`source`,`date_added`,`number_of_views`),
KEY `sdnvo` (`source`,`date_added`,`number_of_votes`),
KEY `scdnv` (`source`,`category`,`date_added`,`number_of_views`),
KEY `scdnvo` (`source`,`category`,`date_added`,`number_of_votes`),
KEY `cdnv` (`category`,`date_added`,`number_of_views`),
KEY `cdnvo` (`category`,`date_added`,`number_of_votes`),
KEY `dnv` (`date_added`,`number_of_views`),
KEY `dnvo` (`date_added`,`number_of_votes`),
KEY `clst` (`category`,`last_suggestion_time`) USING BTREE,
KEY `slst` (`source`,`last_suggestion_time`) USING BTREE,
KEY `nv` (`number_of_views`) USING BTREE,
KEY `nvo` (`number_of_votes`) USING BTREE,
KEY `t` (`time_added`) USING BTREE,
KEY `lvt` (`last_view_time`) USING BTREE,
FULLTEXT KEY `title_content` (`title`,`content`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT
Try this, using a tip I found from Rolando on DBA.stackexchange:
SELECT News.* FROM (
SELECT id FROM News
WHERE MATCH (title, content) AGAINST ('"My Phrase"' in Boolean Mode)
) matches
INNER JOIN News
ON News.id = matches.id
ORDER BY News.id DESC
limit 40