Mysql primary key association does not use index - mysql

I have a SQL problem.
When the user, organization, and organization are associated with the table, if the user status is used to filter the table, the index user_id cannot be used. If the condition is removed, the index user_id will be used.
Why is that?
MSYQL VERSION:5.7.32-log
Below is the specific SQL and table structure.
sql 1 :
SELECT DISTINCT USER
.user_id,
USER.NAME,
USER.nickname,
USER.position,
USER.first_line_id,
USER.second_line_id,
USER.org_id,
user.state
FROM
USER INNER JOIN user_org ON USER.user_id = user_org.user_id
INNER JOIN org ON user_org.org_id = org.id
WHERE
( org.end_time IS NULL OR org.end_time > NOW( ) )
AND USER.state = 1
AND ( full_id LIKE 'H_ROOT.00000001.00000002.50060182.50091585.50095679.50092012.10148706.50092333.10161139%' )
explain:user_id index not sufficient
sql2 :
SELECT DISTINCT USER
.user_id,
USER.NAME,
USER.nickname,
USER.position,
USER.first_line_id,
USER.second_line_id,
USER.org_id,
user.state
FROM
USER INNER JOIN user_org ON USER.user_id = user_org.user_id
INNER JOIN org ON user_org.org_id = org.id
WHERE
( org.end_time IS NULL OR org.end_time > NOW( ) )
-- AND USER.state = 1
AND ( full_id LIKE 'H_ROOT.00000001.00000002.50060182.50091585.50095679.50092012.10148706.50092333.10161139%' )
explain:user_id index sufficient
table count
USER:356007
ORG:142713
USER_ORG:353088
table schema
SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;
DROP TABLE IF EXISTS `user_org`;
CREATE TABLE `user_org` (
`user_id` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`org_id` varchar(128) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`created_at` datetime(0) NULL DEFAULT NULL,
`updated_at` datetime(0) NULL DEFAULT NULL,
PRIMARY KEY (`user_id`, `org_id`) USING BTREE,
INDEX `org_id`(`org_id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;
SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;
DROP TABLE IF EXISTS `user`;
CREATE TABLE `user` (
`user_id` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '工号',
`name` varchar(128) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '姓名',
`email` varchar(64) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '邮箱',
`email_private` varchar(64) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '个人邮箱',
`mobile` varchar(64) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '手机号',
`position` varchar(256) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT '' COMMENT '岗位',
`state` tinyint(4) NOT NULL DEFAULT 1 COMMENT '状态(1:启用;0:禁用)',
`org_id` varchar(128) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '部门编码',
PRIMARY KEY (`user_id`) USING BTREE,
INDEX `user_email_index`(`email`) USING BTREE,
INDEX `user_mobile_index`(`mobile`) USING BTREE,
INDEX `user_name_index`(`name`) USING BTREE,
INDEX `user_org_id_index`(`org_id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8 COLLATE = utf8_general_ci COMMENT = '用户表' ROW_FORMAT = Dynamic;
SET FOREIGN_KEY_CHECKS = 1;
SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;
DROP TABLE IF EXISTS `org`;
CREATE TABLE `org` (
`id` varchar(128) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`name` varchar(256) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`parent_id` varchar(128) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`full_id` varchar(512) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`end_time` datetime(0) NULL DEFAULT NULL COMMENT '部门过期时间',
`created_at` datetime(0) NOT NULL DEFAULT CURRENT_TIMESTAMP(0) COMMENT '创建时间',
`updated_at` datetime(0) NOT NULL DEFAULT CURRENT_TIMESTAMP(0) COMMENT '更新时间',
`customer_code` varchar(40) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT '',
`org_type` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '组织类别',
`state` tinyint(4) NULL DEFAULT NULL COMMENT ' 1 正常 2 停用\r\n冗余目前还是用endtime来识别有效性',
PRIMARY KEY (`id`) USING BTREE,
INDEX `org_full_id_index`(`full_id`(255)) USING BTREE,
INDEX `org_name_index`(`name`(255)) USING BTREE,
INDEX `org_parent_id_index`(`parent_id`) USING BTREE,
INDEX `end_time`(`end_time`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8 COLLATE = utf8_general_ci COMMENT = '组织表' ROW_FORMAT = Dynamic;
SET FOREIGN_KEY_CHECKS = 1;
STRAIGHT_JOIN not sufficient
STRAIGHT_JOIN not sufficien v2
FORCE INDEX not sufficient
FORCE INDEX not sufficient v2

What version of MySQL are you using? There have been Optimization and Index-limit changes that are relevant to your query and schema.
If you set end_time to some date in the distant future, you could avoid the OR by changing to simply end_time > NOW(). (OR used to be bad for performance.)
The indexes you have for the many-to-many table (user_org) are optimal.
Index "prefixing" (full_id(255)) is problematic. It can be eliminated in newer versions. INDEX(full_id) would let the query start with `full_id LIKE '...%' be much more usable.
Perhaps you should change to utf8mb4? It is needed for the more obscure Chinese characters, plus some Emoji.
This index may be picked by the Optimizer; suggest you add it:
USER: INDEX(state, user_id)
If you don't actually need user.name to be a full 256 characters, lower it to 255. That way you can eliminate the prefixing:
USER: INDEX(name)
See other options here: http://mysql.rjweb.org/doc.php/limits#767_limit_in_innodb_indexes

Related

mysql query tree struct with mptt slower than 2D relational table

My business scenario is shown in the figure above. A user can create multiple products, a product can have multiple modules, and a module can have multiple parameters. The parameters include variable types, variable names, and variable values.
Before starting I thought the query speed of mptt was better than 2D relational table, but the result is completely opposite.
I now have two data table designs.
Option One:
CREATE TABLE `products` (
`product_id` bigint(20) NOT NULL AUTO_INCREMENT,
`product_name` varchar(30) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL,
`cfdversion` bigint(20) NULL DEFAULT NULL,
`product_info` varchar(30) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL,
`is_activated` tinyint(1) NULL DEFAULT NULL,
PRIMARY KEY (`product_id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 226 CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;
CREATE TABLE `person_param` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`product_id` bigint(20) NULL DEFAULT NULL,
`param_name` varchar(20) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL,
`var_type` varchar(10) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL,
`var_value` varchar(30) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL,
`var_name` varchar(30) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL,
`is_activated` tinyint(1) NULL DEFAULT 1,
`compute_value` varchar(20) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL,
`module_name` varchar(10) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL,
PRIMARY KEY (`id`) USING BTREE,
INDEX `product_id`(`product_id`) USING BTREE,
CONSTRAINT `person_param_ibfk_1` FOREIGN KEY (`product_id`) REFERENCES `products` (`product_id`) ON DELETE RESTRICT ON UPDATE RESTRICT
) ENGINE = InnoDB AUTO_INCREMENT = 19 CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;
I connect the id of the product information with the parameter table.
Option two:
products table is same just table name is different.
person_paramlike this:
CREATE TABLE `mptt_param` (
`node_id` bigint(20) NOT NULL AUTO_INCREMENT,
`node_name` varchar(30) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL,
`lft` bigint(20) NOT NULL,
`rgt` bigint(20) NOT NULL,
`node_level` bigint(20) NOT NULL,
PRIMARY KEY (`node_id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;
I added 200 products with 3 modules per product and 10 parameters per module.
option 1's sql statement:SELECT * from `products` a RIGHT JOIN `person_param` b ON a.product_id=b.product_id WHERE a.product_id=246;
option 2's sql statement:SELECT * FROM mptt_param WHERE lft>=(SELECT lft FROM mptt_param WHERE node_name='246') AND rgt<=(SELECT rgt FROM mptt_param WHERE node_name='246') ;
I don't know what the problem is, hope you can give me some advice

I get the error: incorrect table definition [duplicate]

This question already has answers here:
There can be only one auto column
(6 answers)
Closed 2 years ago.
Im using phpmyadmin for the first time and I get this error: incorrect table definition. there can be only one auto column and it must be defined as key. What am I doing wrong?
This is my code:
CREATE TABLE `database_reservering`.`formData` (
`nameTeacher` VARCHAR(50) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`nameChild` VARCHAR(50) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL , `email` VARCHAR(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`age` INT(11) NOT NULL ,
`date` DATE NOT NULL ,
`comment` VARCHAR(300) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`id` INT(30) UNSIGNED NOT NULL AUTO_INCREMENT ,
PRIMARY KEY (`nameTeacher`)
) ENGINE = InnoDB;
The actual error message is:
Incorrect table definition; there can be only one auto column and it must be defined as a key
The problem is that you have id as auto-increment, but the primary key is on nameTeacher. This is not allowed. You can change the statement to make id the primary key, and put a unique constraint on nameTeacher. This implements the same logic, but is valid MySQL syntax:
CREATE TABLE `formData` (
...
PRIMARY KEY (`id`),
UNIQUE (`nameTeacher`)
) ENGINE = InnoDB;
Demo on DB Fiddle
Change
PRIMARY KEY (nameTeacher)
to
PRIMARY KEY (id)
Full statement:
CREATE TABLE formData (
nameTeacher VARCHAR(50) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
nameChild VARCHAR(50) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
email VARCHAR(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
age INT(11) NOT NULL ,
`date` DATE NOT NULL ,
`comment` VARCHAR(300) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
id INT UNSIGNED NOT NULL AUTO_INCREMENT ,
PRIMARY KEY (id)) ENGINE = InnoDB;
What do you want to achieve when PK is set to the column other than AUTO_INCREMENT?
If you want to have separate independent autoincremented sequence for each nameTeacher value then alter the engine to MyISAM and define PK like (nameTeacher, id):
CREATE TABLE `formData` (
`nameTeacher` VARCHAR(50) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`nameChild` VARCHAR(50) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`email` VARCHAR(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`age` INT(11) NOT NULL , `date` DATE NOT NULL ,
`comment` VARCHAR(300) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`id` INT(30) UNSIGNED NOT NULL AUTO_INCREMENT ,
PRIMARY KEY (`nameTeacher`, `id`)
) ENGINE = MyISAM;
fiddle
See Using AUTO_INCREMENT for details.

Invalid default value null for timestamp

my database is mySql 5.6.48. Invalid default value for 'modify_time' Error occur during I excute the following SQL:
CREATE TABLE `check_wave_status` (
`com_uid` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`modify_time` timestamp(0) DEFAULT NULL,
PRIMARY KEY (`com_uid`, `wave_uid`, `trade_uid`) USING BTREE,
INDEX `idx_modify_time`(`com_uid`, `modify_time`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Compact;
,My sql_mode = NO_ENGINE_SUBSTITUTION,How can I correct this error?
I think you mean NOT NULL instead of DEFAULT NULL. The first TIMESTAMP column is auto-populated.

Mysql left join take too much time

Can anyone help me to know why is my query take too much time to excute.
I have 2 table, transaction_log and purchasing_receipt.
Here is the DDL:
-- payment.purchasing_receipt definition
CREATE TABLE `purchasing_receipt` (
`purchasing_receipt_id` bigint NOT NULL AUTO_INCREMENT COMMENT 'Purchasing Receipt Id',
`processing_id` bigint DEFAULT NULL COMMENT 'Processing Id',
`merchant_no` varchar(4) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'Merchant Number',
`receipt_no` varchar(25) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'Receipt No',
`ext_store_cd` varchar(25) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'External Store Code',
`total_amount` decimal(12,0) DEFAULT NULL COMMENT 'Total Amount',
`purchasing_date` datetime DEFAULT NULL COMMENT 'Purchasing Date',
`created_date` datetime DEFAULT NULL COMMENT 'Created Date',
`created_by` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'Created By',
`updated_date` datetime DEFAULT NULL COMMENT 'Updated Date',
`updated_by` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'Updated By',
`ext_store_name` varchar(200) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT 'External Store Name',
`pos_number` varchar(25) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT 'POS Number',
`management_no` varchar(89) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT 'Management Number',
PRIMARY KEY (`purchasing_receipt_id`),
KEY `purchasing_receipt_processing_idx` (`processing_id`),
KEY `ix_batch_p` (`processing_id`,`purchasing_date`,`total_amount`,`ext_store_cd`,`merchant_no`),
CONSTRAINT `purchasing_receipt_processing` FOREIGN KEY (`processing_id`) REFERENCES `batch_processing` (`processing_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=2019121320000000015 DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='Purchasing Receipt';
and:
-- payment.transaction_log definition
CREATE TABLE `transaction_log` (
`transaction_id` char(20) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'Transaction Id',
`transaction_date` datetime DEFAULT NULL COMMENT 'Transaction Date',
`transaction_status` varchar(1) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL COMMENT 'Transaction Status',
`transaction_amount` decimal(8,0) DEFAULT NULL COMMENT 'Transaction Amount',
`consumer_user_profile_id` bigint DEFAULT NULL COMMENT 'Consumer User Profile Id',
`consumer_pay_profile_id` bigint DEFAULT NULL COMMENT 'Consumer Pay Profile Id',
`merchant_no` varchar(4) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT 'Merchant Number',
`merchant_name_kanji` varchar(64) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT 'Merchant Name Kanji',
`merchant_name_kana` varchar(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT 'Merchant Name Kana',
`affiliated_store_no` varchar(9) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT 'Affiliated Store Number',
`affiliated_store_name_kanji` varchar(64) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT 'Affiliated Store Name Kanji',
`affiliated_store_name_kana` varchar(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT 'Affiliated Store Name Kana',
`terminal_no` varchar(13) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT 'Terminal Number',
`cancelled_flag` char(1) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL COMMENT 'Cancelled Flag',
`created_date` datetime DEFAULT NULL COMMENT 'Created Date',
`created_by` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL COMMENT 'Created By',
`updated_date` datetime DEFAULT NULL COMMENT 'Updated Date',
`updated_by` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL COMMENT 'Updated By',
`matched_flag` char(1) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`report_year_month` char(6) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`app_user_id` varchar(64) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`merchant_pay_name` varchar(64) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`installed_store_no` varchar(9) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT 'Installed Store Number',
`installed_store_name_kanji` varchar(64) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`installed_store_name_kana` varchar(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`usage_item_flag` char(1) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`processing_id` bigint DEFAULT NULL,
PRIMARY KEY (`transaction_id`),
KEY `transaction_log_sequence_idx` (`transaction_id`),
KEY `con_idx` (`consumer_user_profile_id`),
KEY `con_idx1` (`consumer_pay_profile_id`),
KEY `FK_idx` (`terminal_no`),
KEY `FK_idx1` (`processing_id`),
KEY `optimize_batch_t` (`transaction_id`,`transaction_date`,`transaction_amount`,`installed_store_no`,`merchant_no`,`matched_flag`),
CONSTRAINT `FK_con_pay` FOREIGN KEY (`consumer_pay_profile_id`) REFERENCES `consumer_pay_profile` (`consumer_pay_profile_id`),
CONSTRAINT `FK_con_user` FOREIGN KEY (`consumer_user_profile_id`) REFERENCES `consumer_user_profile` (`consumer_user_profile_id`),
CONSTRAINT `FK_process` FOREIGN KEY (`processing_id`) REFERENCES `batch_processing` (`processing_id`),
CONSTRAINT `FK_terminal` FOREIGN KEY (`terminal_no`) REFERENCES `pos_terminal_profile` (`terminal_no`),
CONSTRAINT `FK_tran_sequence` FOREIGN KEY (`transaction_id`) REFERENCES `transaction_id_sequence` (`transaction_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='Transaction Log';
I've indexed the column for two table:
CREATE INDEX optimize_batch_t
ON transaction_log(transaction_date, transaction_amount, merchant_no, matched_flag, processing_id);
CREATE INDEX ix_batch_p
ON purchasing_receipt(purchasing_date, total_amount, merchant_no, processing_id);
And here is the query:
SELECT p.purchasing_receipt_id
, CASE WHEN count(t.transaction_id) = 1 THEN min(t.transaction_id) ELSE NULL END
, CASE WHEN count(t.transaction_id) = 1 THEN 'A' ELSE 'U' END
FROM purchasing_receipt p
LEFT JOIN transaction_log t
ON t.transaction_date BETWEEN DATE_SUB(p.purchasing_date, INTERVAL 2 MINUTE) AND DATE_ADD(p.purchasing_date, INTERVAL 2 MINUTE)
AND t.transaction_amount = p.total_amount
AND p.merchant_no = t.merchant_no
AND t.matched_flag = 'N'
WHERE p.processing_id = 783 GROUP BY p.purchasing_receipt_id ;
I've check and it show that the query use correct index.
The transaction_log table has 261690 record. And Purchasing_receipt has 100063 record.
And the query take 128 minutes to complete.
I've stucked with this issue a few days and still dont understand why the query execute is too slow like that.
Start by making sure to use the same COLLATION in JOINs:
AND p.merchant_no = t.merchant_no
`merchant_no` varchar(4) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci
DEFAULT NULL COMMENT 'Merchant Number',
`merchant_no` varchar(4) CHARACTER SET utf8 COLLATE utf8_bin
DEFAULT NULL COMMENT 'Merchant Number',
Fixing that may not be the whole solution.
What do you think this means? count(t.transaction_id) = 1 ? It means "count the number of rows (in each GROUP) where t.transaction_id is NOT NULL. Can there be 2 such rows? If so, that =1 will be 'false'.
Ranges with the BETWEEN being variables is hard to hard to optimizer. (Alas, I have no remedy.)
ON t.transaction_date
BETWEEN DATE_SUB(p.purchasing_date, INTERVAL 2 MINUTE)
AND DATE_ADD(p.purchasing_date, INTERVAL 2 MINUTE)
But, we can probably get rid of the GROUP BY by changing the LEFT JOIN into
EXISTS ( SELECT 1
FROM transaction_log t
WHERE t.transaction_date
BETWEEN DATE_SUB(p.purchasing_date, INTERVAL 2 MINUTE)
AND DATE_ADD(p.purchasing_date, INTERVAL 2 MINUTE)
AND t.transaction_amount = p.total_amount
AND p.merchant_no = t.merchant_no
AND t.matched_flag = 'N'
)
Now, let's make an index for t:
INDEX(matched_flag, merchant_no, transaction_amount, -- in any order
transaction_date) -- put the range last
You have two indexes with transaction_date before the end. Usually the utility of an index terminates with a "range". This makes your indexes not as useful as they could be.
OK, I hit a roadblock. You need the EXISTS twice. That takes you back to LEFT JOIN. But I don't want to get into the "inflate-deflate" problem. This where a JOIN leads to multiple rows (in a temp table), only to have a GROUP BY deflate back to what you started with. I'll assume the worst, namely that there can be multiple transactions in the 4 minute period for some receipts.
SELECT purchasing_receipt_id,
min_tid,
IF(min_tid IS NULL, 'U', 'A')
FROM (
SELECT p.purchasing_receipt_id,
( SELECT min(t.transaction_id)
FROM transaction_log t
WHERE t.transaction_amount = p.total_amount
AND t.merchant_no = p.merchant_no
AND t.matched_flag = 'N'
AND t.transaction_date
BETWEEN p.purchasing_date - INTERVAL 2 MINUTE
AND p.purchasing_date + INTERVAL 2 MINUTE
) AS min_tid
WHERE p.processing_id = 783
) AS x ;
(I don't want to ask what this is about: AUTO_INCREMENT=2019121320000000015 .)

MySQL 8.0.15 Dam slow select query

Motto of the query is very simple, to find out the last entry on a foreign key column.
the pseudo code I can say is
select vehicleid , last_journey_point , last_journey_time from journeyTable.
here is my SQL statement
-- loconumber is a indexed column
-- journeyserla is a autonumber primary key int(11)
-- the table locojourney contains 400,000 records
-- the below block of code executes in 19 secs
with LocomotiveLastRun AS(
-- this block of code runs in 0.016 sec
SELECT locojourney.loconumber , MAX(locojourney.journeyserla) as lastrunid
FROM locojourney GROUP BY loconumber)
SELECT locojourney.CurrentCombiners , locojourney.JourneySerla ,
locojourney.From_RunPoint , locojourney.NEXT_RunPoint
FROM LocomotiveLastRun FORCE INDEX(lastrunid)
JOIN locojourney FORCE INDEX(PRIMARY) ON x.lastrunid = locojourney.journeyserla
WHERE locojourney.ishoc = 'n'
the EXPLAIN command shows a derived table which is using no index and using where and type ALL
This is the table definition:
-- SHOW CREATE TABLE locojourney
CREATE TABLE `locojourney` (
`trainID` smallint(5) NOT NULL,
`LocoNumber` varchar(5) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL,
`CurrentLocoBase` varchar(10) CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT NULL,
`CurrentDuedate` date DEFAULT NULL,
`LocoConsist` varchar(10) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL,
`CurrentLocoDomain` varchar(10) CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT NULL,
`DomainChange` varchar(10) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL,
`FEDR` enum('N','Y') CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT 'N',
`LADR` enum('N','Y') CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT 'N',
`ISBANKER` enum('N','Y') CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT 'N',
`TrainName` varchar(10) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL,
`WithOutLoad` enum('N','Y') CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL DEFAULT 'N',
`runRoute` varchar(50) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL,
`From_RunPoint` varchar(10) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL,
`From_RunTime` datetime NOT NULL,
`NEXT_RunPoint` varchar(10) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL,
`NEXT_RunTime` datetime NOT NULL,
`Affects_Outage` enum('N','Y') CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT 'N',
`Affects_Mileage` enum('N','Y') CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT 'N',
`GroundDistance` double(5,2) DEFAULT '0.00',
`SHGallowance` int(11) DEFAULT '0',
`Outage` double(5,4) DEFAULT '0.0000',
`UnderServiceType` enum('FHT','CHG','DEP','MIX','DETN') CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL DEFAULT 'FHT',
`SubServiceHead` varchar(25) CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL DEFAULT 'RUN',
`IShoc` enum('N','Y') CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT 'N',
`CurrentCombiners` varchar(28) CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT NULL,
`RunSetSerla` varchar(25) CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT NULL,
`JourneySerla` int(11) NOT NULL AUTO_INCREMENT,
`NominationSerla` varchar(50) CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT NULL,
`Traction` enum('DSL','AC') CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL DEFAULT 'DSL',
`Trainload` smallint(4) NOT NULL DEFAULT '0',
`LeadAssist` enum('Y','N') CHARACTER SET latin1 COLLATE latin1_swedish_ci NOT NULL DEFAULT 'N',
`DEO` varchar(100) CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT NULL,
`DEOtime` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`JourneySerla`),
KEY `trainID` (`trainID`) USING BTREE,
KEY `routesection_idx` (`runRoute`) USING BTREE,
KEY `loconumber_idx` (`LocoNumber`) USING BTREE,
KEY `runsetserla_idx` (`RunSetSerla`) USING BTREE,
KEY `subservicehead_idx` (`SubServiceHead`) USING BTREE,
CONSTRAINT `locojourney_ibfk_1` FOREIGN KEY (`SubServiceHead`) REFERENCES `ineffective` (`IneffectiveHead`) ON UPDATE CASCADE,
CONSTRAINT `locojourney_ibfk_3` FOREIGN KEY (`runRoute`) REFERENCES `routesections` (`Sectionname`) ON DELETE RESTRICT ON UPDATE CASCADE,
CONSTRAINT `loconumber_fk` FOREIGN KEY (`LocoNumber`) REFERENCES `lococontainer` (`LocoNumber`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=345719 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
with LocomotiveLastRun AS(
-- this block of code runs in 0.016 sec
SELECT locojourney.loconumber , MAX(locojourney.journeyserla) as lastrunid
FROM locojourney
GROUP BY loconumber)
Why is this CTE subquery fast? Because your table already has an index on (loconumber, journeyserla). (InnoDb automatically appends the primary key to every index.) This query can be satisfied with a loose index scan on that index, and those are fast.
Now for your main query:
Get rid of FORCE INDEX(). Don't even dream of using that unless you have at least a decade of SQL experience or you have read the source code for the InnoDB indexing stuff in MySQL. Notably, it's completely useless on the CTE because CTEs don't have indexes.
For clarity put your main (detail) table first and your CTE second.
For clarity recast the JOIN as a WHERE...IN...
Those three suggestions give us this:
WITH LocomotiveLastRun AS (...)
SELECT locojourney.CurrentCombiners , locojourney.JourneySerla ,
locojourney.From_RunPoint , locojourney.NEXT_RunPoint
FROM locojourney
WHERE journeyserla IN (SELECT lastrunid FROM LocomotiveLastRun)
AND locojourney.ishoc = 'n'
Now, it's plain what index can help this query.
An index on (ishoc) will help a bit. (It's actually an index, because InnoDB, on (ishoc, journeyserla) so it helps with both WHERE conditions.) The query planner uses BTREE random access to find the first index row with the ishoc value 'n', then scans the values of the primary key to match them with the IN clause.
Instead of that index, a compound index that covers the query will help even more. Such a covering index helps especially because each row of your table is large, with many columns. That index mentions the columns in the WHERE clause and those you want to select, like this:
(ishoc, journeyserla, CurrentCombiners, From_RunPoint, NEXT_RunPoint)
The query planner can satisfy your query entirely from the index, which saves on disk reading time to satisfy the query. If you use your query a lot, this index is a good idea. But, it does consume disk space and slow down INSERT and UPDATE operations a bit.
Read https://use-the-index-luke.com/
Give this a try:
SELECT lj.CurrentCombiners , lj.JourneySerla , lj.From_RunPoint , lj.NEXT_RunPoint
FROM ( SELECT MAX(journeyserla) as lastrunid
FROM locojourney
GROUP BY loconumber
) AS llr
JOIN locojourney AS lj ON llr.lastrunid = lj.journeyserla
WHERE lj.ishoc = 'n'
(time it and provide EXPLAIN for it)