Optimizing MySQL query to reduced runtime - mysql

Question

Below is the query that will going to run on two tables with 60+ million and 400+ million records. Only the table name will be different, otherwise query is same for both the tables.
SELECT * FROM
(
SELECT A.CUSIP, A.ISIN, A.SEDOL, A.LocalCode, A.MIC, A.ExchgCD, A.PrimaryExchgCD, A.Currency, A.Open, A.High, A.Low, A.Close, A.Mid, A.Ask, A.Last,
A.Bid, A.Bidsize, A.Asksize, A.TradedVolume, A.SecID, A.PriceDate, A.MktCloseDate, A.VolFlag, A.IssuerName, A.TotalTrades, A.CloseType, A.SectyCD,
row_number() OVER (partition by A.CUSIP order by A.MktCloseDate desc) as 'rank'
from EDI_Price04 A
WHERE A.CUSIP IN (
"91879Q109", "583840509", "583840608", "59001A102", "552848103") AND (A.PrimaryExchgCD = A.ExchgCD) AND A.CloseType='CC'
) t WHERE t.rank <= 3;
When A.CUSIP IN () condition have 10-15 values, the query complete in 2-3sec. With 400 values it took 28sec. But I want to make A.CUSIP IN () take 2k-3k value at a time.
This is my table structure.
CREATE TABLE `EDI_Price04` (
`MIC` varchar(6) NOT NULL DEFAULT '',
`LocalCode` varchar(60) NOT NULL DEFAULT '' COMMENT 'PricefileSymbol',
`ISIN` varchar(12) DEFAULT NULL,
`Currency` varchar(3) NOT NULL DEFAULT '',
`PriceDate` date DEFAULT NULL,
`Open` double DEFAULT NULL,
`High` double DEFAULT NULL,
`Low` double DEFAULT NULL,
`Close` double DEFAULT NULL,
`Mid` double DEFAULT NULL,
`Ask` double DEFAULT NULL,
`Last` double DEFAULT NULL,
`Bid` double DEFAULT NULL,
`BidSize` int(11) DEFAULT NULL,
`AskSize` int(11) DEFAULT NULL,
`TradedVolume` bigint(20) DEFAULT NULL,
`SecID` int(11) NOT NULL DEFAULT '0',
`MktCloseDate` date NOT NULL DEFAULT '0000-00-00',
`Volflag` char(1) DEFAULT NULL,
`IssuerName` varchar(255) DEFAULT NULL,
`SectyCD` varchar(3) DEFAULT NULL,
`SecurityDesc` varchar(255) DEFAULT NULL,
`SEDOL` varchar(7) DEFAULT NULL,
`CUSIP` varchar(9) DEFAULT NULL COMMENT 'USCode',
`PrimaryExchgCD` varchar(6) DEFAULT NULL,
`ExchgCD` varchar(6) NOT NULL DEFAULT '',
`TradedValue` double DEFAULT NULL,
`TotalTrades` int(11) DEFAULT NULL,
`Comment` varchar(255) DEFAULT NULL,
`Repush` tinyint(4) NOT NULL DEFAULT '0',
`CloseType` varchar(2) NOT NULL DEFAULT '',
PRIMARY KEY (`MIC`,`LocalCode`,`Currency`,`SecID`,`MktCloseDate`,`ExchgCD`,`Repush`,`CloseType`),
KEY `idx_EDI_Price04_0` (`MIC`),
KEY `idx_EDI_Price04_1` (`LocalCode`),
KEY `idx_EDI_Price04_2` (`ISIN`),
KEY `idx_EDI_Price04_3` (`PriceDate`),
KEY `idx_EDI_Price04_4` (`SEDOL`),
KEY `idx_EDI_Price04_5` (`CUSIP`),
KEY `idx_EDI_Price04_6` (`PrimaryExchgCD`),
KEY `idx_EDI_Price04_7` (`ExchgCD`),
KEY `idx_EDI_Price04_8` (`CloseType`),
KEY `idx_EDI_Price04_9` (`MktCloseDate`),
KEY `idx_EDI_Price04_CUSIP_ExchgCD_CloseType_MktCloseDate` (`CUSIP`,`ExchgCD`,`CloseType`,`MktCloseDate`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci

Answer 1

For this query:
SELECT *
FROM (SELECT a.*
ROW_NUMBER() OVER (PARTITION BY A.CUSIP ORDER BY A.MktCloseDate DESC) as rank
FROM EDI_Price04 A
WHERE A.CUSIP IN ('91879Q109', '583840509', '583840608', '59001A102', '552848103') AND
A.PrimaryExchgCD = A.ExchgCD AND
A.CloseType = 'CC'
) t
WHERE t.rank <= 3;
The place to start is with an index. For this query, you want an index on EDI_Price04(CloseType, CUSIP, ExchgCD, MktCloseDate).
Unfortunately, the condition A.PrimaryExchgCD = A.ExchgCD prevents index seeks. If you were to make changes to the query/data, then one approach would be to add a flag when these are the same, rather then looking at the values separately. That would allow an index on:
EDI_Price04(CloseType, IsPrimary, CUSIP, PrimaryExchgCD, ExchgCD, MktCloseDate)

Answer 2

PRIMARY KEY (id),
UNIQUE(`MIC`,`LocalCode`,`Currency`,`SecID`,`MktCloseDate`,
`ExchgCD`,`Repush`,`CloseType`),
-- KEY `idx_EDI_Price04_0` (`MIC`),
KEY `idx_EDI_Price04_1` (`LocalCode`),
KEY `idx_EDI_Price04_2` (`ISIN`),
KEY `idx_EDI_Price04_3` (`PriceDate`),
KEY `idx_EDI_Price04_4` (`SEDOL`),
-- KEY `idx_EDI_Price04_5` (`CUSIP`),
KEY `idx_EDI_Price04_6` (`PrimaryExchgCD`),
KEY `idx_EDI_Price04_7` (`ExchgCD`),
KEY `idx_EDI_Price04_8` (`CloseType`),
KEY `idx_EDI_Price04_9` (`MktCloseDate`),
KEY `idx_EDI_Price04_CUSIP_ExchgCD_CloseType_MktCloseDate` (`CUSIP`,
`ExchgCD`, `CloseType`, `MktCloseDate`)
KEY (CUSIP, MktCloseDate)
Having so many columns in the PK costs in space and insert time. So, I added an id, which needs to be AUTO_INCREMENT.
Keys 0 and 5 are redundant because of the rule "If you have INDEX(a,b), INDEX(a) redundant.
I added (CUSIP, MktCloseDate) in hopes that it will optimize the RANK expression.