How to optimize the low performance MySQL query - mysql
The below query finds one id by using the below query format but it seems I have indexed all columns, but it took time to execute between 54.3 seconds - 1.2 mins. What can I try next?
Query:
select
op.ord_prod_id
FROM
ordered_product op,
payment_type pt,
account a,
service s,
payment_method pm,
acct_order ao,
acct_order_item aoi
WHERE
(
(
op.type = 1
OR op.type = 2
)
AND (
pt.description = 'Levis'
AND op.validity_end_date < '2022-09-18 11:24:08 Etc/GMT'
AND op.validity_end_date >= '2022-09-18 00:00:00 Etc/GMT'
AND op.acct_status_id = 1
AND op.is_Renewable = 'T'
AND a.bu_id = 103
)
)
AND pm.type_id = pt.id
AND s.acct_id = a.acct_id
AND op.serv_id = s.serv_id
AND ao.pay_meth_id = pm.pay_method_id
AND aoi.order_id = ao.order_id
AND aoi.ord_prod_id = op.ord_prod_id
ORDER BY
op.validity_end_date ASC
explain plan
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: pt
partitions: NULL
type: ref
possible_keys: PRIMARY,idx_pmttype_descrip,idx_description
key: idx_pmttype_descrip
key_len: 32
ref: const
rows: 1
filtered: 100.00
Extra: Using index; Using temporary; Using filesort
*************************** 2. row ***************************
id: 1
select_type: SIMPLE
table: pm
partitions: NULL
type: ref
possible_keys: PRIMARY,payment_method_type_id,Idx_type_id,idx_type_id_zip_code_last_digits
key: payment_method_type_id
key_len: 10
ref: ccbuser.pt.id
rows: 3297
filtered: 100.00
Extra: Using index
*************************** 3. row ***************************
id: 1
select_type: SIMPLE
table: ao
partitions: NULL
type: ref
possible_keys: PRIMARY,idx_pay_meth_id
key: idx_pay_meth_id
key_len: 11
ref: ccbuser.pm.pay_method_id
rows: 2
filtered: 100.00
Extra: Using index
*************************** 4. row ***************************
id: 1
select_type: SIMPLE
table: aoi
partitions: NULL
type: ref
possible_keys: idx_accttorderitem_orderid,idx_ord_prod_id
key: idx_accttorderitem_orderid
key_len: 10
ref: ccbuser.ao.order_id
rows: 1
filtered: 100.00
Extra: NULL
*************************** 5. row ***************************
id: 1
select_type: SIMPLE
table: op
partitions: NULL
type: eq_ref
possible_keys: PRIMARY,idx_ord_prod_servid,idx_ord_prod_statprod,idx_ordprod_type,idx_ord_prod_ve_date,idx_ord_prod_acct_status_id,idx_combo,idx_is_renewable,idx_validity_package_ind_acct_status
key: PRIMARY
key_len: 10
ref: ccbuser.aoi.ord_prod_id
rows: 1
filtered: 5.00
Extra: Using where
*************************** 6. row ***************************
id: 1
select_type: SIMPLE
table: s
partitions: NULL
type: eq_ref
possible_keys: PRIMARY,idx_service_acctid
key: PRIMARY
key_len: 10
ref: ccbuser.op.serv_id
rows: 1
filtered: 100.00
Extra: NULL
*************************** 7. row ***************************
id: 1
select_type: SIMPLE
table: a
partitions: NULL
type: eq_ref
possible_keys: PRIMARY,idx_account_buidasid,idx_combo,idx_bu_id,idx_1
key: PRIMARY
key_len: 10
ref: ccbuser.s.acct_id
rows: 1
filtered: 5.00
Extra: Using where
7 rows in set, 11 warnings (0.03 sec)
i have given show warnings;
*************************** 1. row ***************************
Level: Warning
Code: 1292
Message: Incorrect datetime value: '2022-09-18 11:24:08 Etc/GMT' for column 'validity_end_date' at row 1
*************************** 2. row ***************************
Level: Warning
Code: 1292
Message: Incorrect datetime value: '2022-09-18 00:00:00 Etc/GMT' for column 'validity_end_date' at row 1
*************************** 3. row ***************************
Level: Warning
Code: 1292
Message: Incorrect datetime value: '2022-09-18 11:24:08 Etc/GMT' for column 'validity_end_date' at row 1
*************************** 4. row ***************************
Level: Warning
Code: 1292
Message: Incorrect datetime value: '2022-09-18 11:24:08 Etc/GMT' for column 'validity_end_date' at row 1
*************************** 5. row ***************************
Level: Warning
Code: 1292
Message: Incorrect datetime value: '2022-09-18 11:24:08 Etc/GMT' for column 'validity_end_date' at row 1
*************************** 6. row ***************************
Level: Warning
Code: 1292
Message: Incorrect datetime value: '2022-09-18 11:24:08 Etc/GMT' for column 'validity_end_date' at row 1
*************************** 7. row ***************************
Level: Warning
Code: 1292
Message: Incorrect datetime value: '2022-09-18 00:00:00 Etc/GMT' for column 'validity_end_date' at row 1
*************************** 8. row ***************************
Level: Warning
Code: 1292
Message: Incorrect datetime value: '2022-09-18 00:00:00 Etc/GMT' for column 'validity_end_date' at row 1
*************************** 9. row ***************************
Level: Warning
Code: 1292
Message: Incorrect datetime value: '2022-09-18 00:00:00 Etc/GMT' for column 'validity_end_date' at row 1
*************************** 10. row ***************************
Level: Warning
Code: 1292
Message: Incorrect datetime value: '2022-09-18 00:00:00 Etc/GMT' for column 'validity_end_date' at row 1
*************************** 11. row ***************************
Level: Note
Code: 1003
Message: /* select#1 */ select
.`op`.`ord_prod_id` AS `ord_prod_id`
from
`ordered_product` `op`
join `payment_type` `pt`
join `account` `a`
join `service` `s`
join `payment_method` `pm`
join `acct_order` `ao`
join `acct_order_item` `aoi`
where
(
(
`op`.`ord_prod_id` = `aoi`.`ord_prod_id`
)
and (
`aoi`.`order_id` = `ao`.`order_id`
)
and (
`ao`.`pay_meth_id` = `pm`.`pay_method_id`
)
and (
`s`.`serv_id` = `op`.`serv_id`
)
and (
`a`.`acct_id` = `s`.`acct_id`
)
and (
`pm`.`type_id` = `pt`.`id`
)
and (
`op`.`is_Renewable` = 'T'
)
and (
`pt`.`description` = 'Credit Card'
)
and (
(`op`.`type` = 1)
or (`op`.`type` = 2)
)
and (
`op`.`validity_end_date` < '2022-09-18 11:24:08 Etc/GMT'
)
and (
`op`.`validity_end_date` >= '2022-09-18 00:00:00 Etc/GMT'
)
and (
`op`.`acct_status_id` = 1
)
and (`a`.`bu_id` = 103)
)
order by
`op`.`validity_end_date`
All the rows scan one row, some of them scan 3k for one row - do I have to use str_to_date functions for the above one, are there any ways to improve performance?
Table structure:
Table: ordered_product
Create Table: CREATE TABLE `ordered_product` (
`acct_status_id` decimal(22,0) NOT NULL,
`anual_prepay_ind` varchar(1) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`bill_end_date` datetime DEFAULT NULL,
`bill_start_date` datetime DEFAULT NULL,
`equip_id` decimal(22,0) DEFAULT NULL,
`ord_prod_id` decimal(22,0) NOT NULL,
`package_id` decimal(22,0) DEFAULT NULL,
`package_ind` varchar(1) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`parent_ord_prod_id` decimal(22,0) DEFAULT NULL,
`prod_id` decimal(22,0) DEFAULT NULL,
`quantity` decimal(38,0) DEFAULT NULL,
`remove_flag` varchar(1) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`serv_id` decimal(22,0) NOT NULL,
`srv_end_date` datetime DEFAULT NULL,
`srv_start_date` datetime DEFAULT NULL,
`split_prod_id` decimal(22,0) DEFAULT NULL,
`term` decimal(22,0) DEFAULT NULL,
`type` decimal(22,0) DEFAULT NULL,
`tax_charged` decimal(20,6) DEFAULT NULL,
`comments` varchar(4000) DEFAULT NULL,
`reason_code_id` decimal(20,0) DEFAULT NULL,
`price_charged` decimal(20,2) DEFAULT NULL,
`validity_period` varchar(20) DEFAULT NULL,
`validity_duration` decimal(38,0) DEFAULT NULL,
`validity_end_date` datetime DEFAULT NULL,
`coupon_amount` decimal(20,2) DEFAULT NULL,
`coupon_code` varchar(100) DEFAULT NULL,
`applied_wallet_amount` decimal(20,2) DEFAULT NULL,
`applied_rate_type` varchar(100) DEFAULT NULL,
`partner_domain` varchar(100) DEFAULT NULL,
`is_Renewable` varchar(1) DEFAULT NULL,
`coupon_external_id` varchar(256) DEFAULT NULL,
`validity_extended_date` datetime DEFAULT NULL,
`cancellation_date` datetime DEFAULT NULL,
`rec_indicator` varchar(1) DEFAULT NULL,
`suspend_start_date` datetime DEFAULT NULL,
`suspend_end_date` datetime DEFAULT NULL,
`suspend_duration` decimal(20,0) DEFAULT NULL,
`commitment_end_date` datetime DEFAULT NULL,
`modified_ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`contact_id` varchar(30) DEFAULT NULL,
`subscription_status_id` decimal(22,0) DEFAULT NULL,
`bu_charged_currency_id` decimal(38,0) DEFAULT NULL,
`currency_code` varchar(200) DEFAULT NULL,
`legacy_id` varchar(200) DEFAULT NULL,
`app_service_id` varchar(200) DEFAULT NULL,
`app_channel_name` varchar(200) DEFAULT NULL,
`applied_prod_rate_id` decimal(22,0) DEFAULT NULL,
`applied_prod_rate_in_charged_currency_id` decimal(22,0) DEFAULT NULL,
`applied_prod_price` decimal(22,0) DEFAULT NULL,
`bill_to_parent` varchar(1) DEFAULT NULL,
PRIMARY KEY (`ord_prod_id`),
KEY `idx_ord_prod_servid` (`serv_id`),
KEY `idx_ord_prod_statprod` (`acct_status_id`,`prod_id`,`package_id`),
KEY `idx_ord_prod_bs_date` (`bill_start_date`),
KEY `idx_ord_prod_be_date` (`bill_end_date`),
KEY `ordered_product_modified_ts` (`modified_ts`),
KEY `idx_ordprod_type` (`type`),
KEY `idx_ord_prod_ve_date` (`validity_end_date`),
KEY `idx_ord_prod_prod_id` (`prod_id`),
KEY `idx_ord_prod_srv_start_date` (`srv_start_date`),
KEY `idx_ord_prod_pack_id` (`package_id`),
KEY `idx_ord_prod_acct_status_id` (`acct_status_id`),
KEY `idx_ord_prod_validity_extend_dt` (`validity_extended_date`),
KEY `idx_remove_flag` (`remove_flag`),
KEY `idx_ord_prod_srv_end_date` (`srv_end_date`),
KEY `idx_subscription_status_id` (`subscription_status_id`),
KEY `idx_coupon_code` (`coupon_code`),
KEY `idx_parent_ord_prod_id` (`parent_ord_prod_id`),
KEY `idx_combo` (`type`,`serv_id`),
KEY `idx_is_renewable` (`is_Renewable`),
KEY `idx_package_ind` (`package_ind`),
KEY `idx_validity_package_ind_acct_status` (`validity_end_date`,`package_ind`,`acct_status_id`),
KEY `idx_legacyid` (`legacy_id`),
KEY `idx_cancellation_date` (`cancellation_date`),
KEY `idx_suspend_start_date` (`suspend_start_date`),
KEY `idx_suspend_end_date_validity_end_date_combo` (`suspend_end_date`,`validity_end_date`,`acct_status_id`,`is_Renewable`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
Table: payment_type
Create Table: CREATE TABLE `payment_type` (
`auto_pay_capable_ind` varchar(1) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`can_post_credit_ind` varchar(1) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`check_number_ind` varchar(1) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`description` varchar(30) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`display_order` decimal(22,0) NOT NULL,
`id` decimal(22,0) NOT NULL,
`requires_gateway_ind` varchar(1) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`display_name` varchar(30) DEFAULT NULL,
`modified_ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`image_url` varchar(4096) DEFAULT NULL,
`short_code` varchar(20) DEFAULT NULL,
`payment_group` varchar(100) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `idx_pmttype_descrip` (`description`),
KEY `idx_display_name` (`display_name`),
KEY `idx_description` (`description`),
KEY `idx_payment_group` (`payment_group`),
KEY `idx_modified_ts` (`modified_ts`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
Table: account
Create Table: CREATE TABLE `account` (
`acct_id` decimal(22,0) NOT NULL,
`acct_status_id` decimal(22,0) NOT NULL,
`acct_type_id` decimal(22,0) DEFAULT NULL,
`attach_bill_cycle` char(1) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`bb_acct_id` varchar(256) DEFAULT NULL,
`bill_del_type_id` decimal(22,0) DEFAULT NULL,
`acct_bill_mode` varchar(20) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`bu_id` decimal(22,0) NOT NULL,
`acct_comment` varchar(500) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`compliment_acct` varchar(1) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`contact_id` decimal(22,0) NOT NULL,
`corporate_id` decimal(22,0) DEFAULT NULL,
`create_date` datetime NOT NULL,
`customer_id` varchar(100) DEFAULT NULL,
`direct_acct` varchar(1) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`do_not_auto_script` char(5) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`email_inv_format` varchar(25) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`external_id_prefix` varchar(6) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`external_id_suffix` varchar(25) DEFAULT NULL,
`inter_pic_id` varchar(4) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`intra_pic_id` varchar(4) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`legacy_id` decimal(22,0) DEFAULT NULL,
`parent_id` decimal(22,0) DEFAULT NULL,
`partner_id` decimal(22,0) DEFAULT NULL,
`psa_acct_id` varchar(20) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`acct_reg_num` varchar(25) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`sales_method` varchar(100) DEFAULT NULL,
`shipping_add_id` decimal(22,0) DEFAULT NULL,
`tax_exempt` varchar(1) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`tax_number` varchar(20) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`tv_dma_id` varchar(30) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`status_change_date` datetime DEFAULT NULL,
`external_creation_date` date DEFAULT NULL,
`sales_person` varchar(200) DEFAULT NULL,
`offer_type` varchar(50) DEFAULT NULL,
`cancel_reasons_id` int(38) DEFAULT NULL,
`acct_addl_info_id` decimal(38,0) DEFAULT NULL,
`vpn_id` decimal(22,0) DEFAULT NULL,
`verified` varchar(1) DEFAULT NULL,
`verified_channel` varchar(50) DEFAULT NULL,
`active_logins_count` decimal(20,0) DEFAULT NULL,
`acct_role_id` decimal(20,0) DEFAULT NULL,
`mso_id` varchar(200) DEFAULT NULL,
`mso_name` varchar(200) DEFAULT NULL,
`anonymized` varchar(1) DEFAULT NULL,
`parental_control` varchar(1) DEFAULT NULL,
`act_state_id` decimal(38,0) DEFAULT NULL,
`verified_date` datetime DEFAULT NULL,
`complaintmode_acct` varchar(1) DEFAULT NULL,
`modified_ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`mobile_verified` varchar(1) DEFAULT NULL,
`acct_sub_type_id` decimal(22,0) DEFAULT NULL,
`created_by` decimal(22,0) DEFAULT NULL,
`ip_address` varchar(50) DEFAULT NULL,
PRIMARY KEY (`acct_id`),
KEY `idx_account_customer_id` (`customer_id`),
KEY `idx_account_buidasid` (`bu_id`,`acct_status_id`),
KEY `idx_account_bbaccountid` (`bb_acct_id`),
KEY `idx_create_date` (`create_date`),
KEY `idx_contact_id` (`contact_id`),
KEY `idx_attach_bill_cycle` (`attach_bill_cycle`),
KEY `idx_act_state` (`act_state_id`),
KEY `idx_accountt_legacyid` (`legacy_id`),
KEY `account_modified_ts` (`modified_ts`),
KEY `idx_account_acctroleid` (`acct_role_id`),
KEY `idx_extrnl_suffix` (`external_id_suffix`),
KEY `idx_extrnl_prefix` (`external_id_prefix`),
KEY `idx_combo` (`bu_id`,`psa_acct_id`),
KEY `idx_bu_id` (`bu_id`),
KEY `idx_acct_reg_num` (`acct_reg_num`),
KEY `idx_acct_stat_id` (`acct_status_id`),
KEY `account_idx_combo4` (`tax_number`,`acct_id`,`contact_id`,`bu_id`),
KEY `idx_1` (`bu_id`,`customer_id`),
KEY `idx_acct_type_id` (`acct_type_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
Table: service
Create Table: CREATE TABLE `service` (
`acct_id` decimal(22,0) NOT NULL,
`acct_status_id` decimal(22,0) DEFAULT NULL,
`do_not_auto_script` char(5) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`end_date` datetime DEFAULT NULL,
`last_billed` datetime DEFAULT NULL,
`loc_id` decimal(22,0) DEFAULT NULL,
`serv_id` decimal(22,0) NOT NULL,
`start_date` datetime NOT NULL,
`type` decimal(22,0) DEFAULT NULL,
`suspend_start_date` date DEFAULT NULL,
`suspend_end_date` date DEFAULT NULL,
`pending_suspension_date` date DEFAULT NULL,
`reactivation_date` date DEFAULT NULL,
`suspend_duration` decimal(20,0) DEFAULT NULL,
`modified_ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`serv_id`),
KEY `idx_service_acctid` (`acct_id`),
KEY `idx_service_locid` (`loc_id`),
KEY `service_modified_ts` (`modified_ts`),
KEY `idx_service_modified_ts` (`modified_ts`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
Table: payment_method
Create Table: CREATE TABLE `payment_method` (
`account_number` varchar(200) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
`bob_id` decimal(22,0) DEFAULT NULL,
`closed_date` datetime DEFAULT NULL,
`pay_method_id` decimal(22,0) NOT NULL,
`start_date` datetime NOT NULL,
`type_id` decimal(22,0) NOT NULL,
`credit_card_type_id` decimal(22,0) DEFAULT NULL,
`expiration_date` datetime DEFAULT NULL,
`name` varchar(100) DEFAULT NULL,
`pay_meth_type_id` decimal(22,0) NOT NULL,
`security_code` varchar(100) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`str_addr` varchar(250) DEFAULT NULL,
`zip_code` varchar(10) DEFAULT NULL,
`acct_type_id` decimal(22,0) DEFAULT NULL,
`bank_name` varchar(512) DEFAULT NULL,
`bank_phone` varchar(15) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`rtn` varchar(20) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`city` varchar(100) DEFAULT NULL,
`drivers_license` varchar(20) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`email` varchar(256) DEFAULT NULL,
`sent_out` varchar(1) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`ssn` varchar(30) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`state` varchar(4) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`country` varchar(30) DEFAULT NULL,
`storage_token` varchar(200) DEFAULT NULL,
`payer_id` varchar(50) DEFAULT NULL,
`last_digits` varchar(4) DEFAULT NULL,
`ref_number` varchar(50) DEFAULT NULL,
`phone_number` varchar(20) DEFAULT NULL,
`dc_code` varchar(20) DEFAULT NULL,
`xid` varchar(250) DEFAULT NULL,
`eci` varchar(5) DEFAULT NULL,
`cavv` varchar(250) DEFAULT NULL,
`ucaf_authentication_data` varchar(250) DEFAULT NULL,
`ucaf_collection_ind` varchar(5) DEFAULT NULL,
`subscription_id` varchar(50) DEFAULT NULL,
`effort_id` decimal(10,0) DEFAULT NULL,
`receipt_data` longtext,
`transaction_id` varchar(512) DEFAULT NULL,
`org_transaction_id` varchar(512) DEFAULT NULL,
`tax_area_id` varchar(30) DEFAULT NULL,
`modified_ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`nick_name` varchar(100) DEFAULT NULL,
`iin` varchar(8) DEFAULT NULL,
`allow_unlimited` tinyint(1) DEFAULT NULL,
`name_of_bank` varchar(100) DEFAULT NULL,
`card_number` varchar(100) DEFAULT NULL,
`pin` varchar(20) DEFAULT NULL,
`card_finger_print` varchar(200) DEFAULT NULL,
`card_issuer_country` varchar(30) DEFAULT NULL,
`card_service_provider` varchar(512) DEFAULT NULL,
PRIMARY KEY (`pay_method_id`),
KEY `payment_method_modified_ts` (`modified_ts`),
KEY `idx_pmtmethod_transid` (`transaction_id`),
KEY `idx_payment_method_start_dt` (`start_date`),
KEY `payment_method_start_date` (`start_date`),
KEY `payment_method_bob_id` (`bob_id`),
KEY `payment_method_type_id` (`type_id`),
KEY `idx_pm_accountno` (`account_number`),
KEY `idx_exp_date` (`expiration_date`),
KEY `idx_org_transaction_id` (`org_transaction_id`),
KEY `idx_transaction_id` (`transaction_id`),
KEY `Idx_bob_id` (`bob_id`),
KEY `idx_account_number` (`account_number`),
KEY `Idx_type_id` (`type_id`),
KEY `i_modified_ts` (`modified_ts`),
KEY `idx_last_digits` (`last_digits`),
KEY `idx_ref_number` (`ref_number`),
KEY `ix_refer_num` (`ref_number`),
KEY `idx_zipcode` (`zip_code`),
KEY `idx_card_fingerprint` (`card_finger_print`),
KEY `idx_phone_number` (`phone_number`),
KEY `idx_type_id_zip_code_last_digits` (`type_id`,`zip_code`,`last_digits`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
Table: acct_order
Create Table: CREATE TABLE `acct_order` (
`acct_id` decimal(22,0) NOT NULL,
`ao_wf_instance_id` decimal(22,0) DEFAULT NULL,
`comm` varchar(4000) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`comment_id` decimal(38,0) DEFAULT NULL,
`external_status` varchar(50) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`house_hold_user` varchar(60) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`legacy_id` varchar(200) DEFAULT NULL,
`modify_date` datetime NOT NULL,
`order_id` decimal(22,0) NOT NULL,
`promise_date` datetime NOT NULL,
`remove_flag` varchar(1) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
`serv_id` decimal(22,0) DEFAULT NULL,
`txn_id` decimal(22,0) DEFAULT NULL,
`workflow_instance_id` decimal(22,0) DEFAULT NULL,
`external_order_id` varchar(200) DEFAULT NULL,
`ref_number` varchar(200) DEFAULT NULL,
`pmt_profile_id` varchar(2048) DEFAULT NULL,
`pay_meth_id` decimal(22,0) DEFAULT NULL,
`modified_ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`txn_summary_id` int(38) DEFAULT NULL,
`country` varchar(10) DEFAULT NULL,
`recon_required` varchar(50) DEFAULT NULL,
PRIMARY KEY (`order_id`),
KEY `idx_acctorder_acctid` (`acct_id`),
KEY `idx_acctorder_wfiid` (`workflow_instance_id`),
KEY `idx_acctorder_servid` (`serv_id`),
KEY `idx_ao_mod_dt` (`modify_date`),
KEY `idx_ext_id` (`external_order_id`),
KEY `acct_order_modified_ts` (`modified_ts`),
KEY `idx_pmt_profile_id` (`pmt_profile_id`),
KEY `idx_ref_number` (`ref_number`),
KEY `idx_pay_meth_id` (`pay_meth_id`),
KEY `idx_txn_id` (`txn_id`),
KEY `idx_promise_date` (`promise_date`),
KEY `idx_modified_ts` (`modified_ts`),
KEY `idx_ao_wf_instance_id` (`ao_wf_instance_id`),
KEY `idx_acct_id_legacy_id` (`acct_id`,`legacy_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
Table: acct_order_item
Create Table: CREATE TABLE `acct_order_item` (
`modify_date` datetime NOT NULL,
`ord_prod_id` decimal(22,0) NOT NULL,
`order_id` decimal(22,0) NOT NULL,
`order_item_id` decimal(22,0) NOT NULL,
`modified_ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`order_item_id`),
KEY `idx_accttorderitem_orderid` (`order_id`),
KEY `idx_ord_prod_id` (`ord_prod_id`),
KEY `acct_order_item_modified_ts` (`modified_ts`),
KEY `idx_acct_order_item_modified_ts` (`modified_ts`),
KEY `idx_modified_ts` (`modified_ts`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
Re-arranging your query, you get this...
SELECT
op.ord_prod_id
FROM
account AS a
INNER JOIN
service AS s
ON s.acct_id = a.acct_id
INNER JOIN
ordered_product AS op
ON op.serv_id = s.serv_id
INNER JOIN
acct_order_item AS aoi
ON aoi.ord_prod_id = op.ord_prod_id
INNER JOIN
acct_order AS ao
ON aoi.order_id = ao.order_id
INNER JOIN
payment_method AS pm
ON ao.pay_meth_id = pm.pay_method_id
INNER JOIN
payment_type AS pt
ON pm.type_id = pt.id
WHERE
a.bu_id = 103
AND op.type IN (1,2)
AND op.validity_end_date < '2022-09-18 11:24:08 Etc/GMT'
AND op.validity_end_date >= '2022-09-18 00:00:00 Etc/GMT'
AND op.acct_status_id = 1
AND op.is_Renewable = 'T'
AND pt.description = 'Levis'
ORDER BY
op.validity_end_date ASC
This looks wrong to me, but I can't tell without the database schema.
Here's why I think it's wrong...
Get all accounts with bu_id = 103
Get all of those accounts' services (the same service may appear multiple times)
Get all ordered products associated to those services regardless of account
I strongly suspect that acct_order has an acct_id column, which would likely enable something like this...
SELECT
op.ord_prod_id
FROM
account AS a
INNER JOIN
acct_order AS ao
ON ao.acct_id = a.acct_id
INNER JOIN
payment_method AS pm
ON pm.pay_method_id = ao.pay_meth_id
INNER JOIN
payment_type AS pt
ON pt.id = pm.type_id
INNER JOIN
acct_order_item AS aoi
ON aoi.order_id = ao.order_id
INNER JOIN
ordered_product AS op
ON op.ord_prod_id = aoi.ord_prod_id
--
-- No need to join on service now
--
WHERE
a.bu_id = 103
AND pt.description = 'Levis'
AND op.type IN (1,2)
AND op.acct_status_id = 1
AND op.is_Renewable = 'T'
AND op.validity_end_date >= '2022-09-18 00:00:00 Etc/GMT'
AND op.validity_end_date < '2022-09-18 11:24:08 Etc/GMT'
ORDER BY
op.validity_end_date ASC
That alone will likely reduce the number of rows being processed, and give insights into what indexes you want, potentially being...
account (bu_id, acct_id)
acct_order (acct_id, pay_meth_id, order_id)
payment_method (pay_method_id, type_id) or (type_id, pay_method_id)
payment_type (id, description) or (description, id)
acct_order_item (order_id, ord_prod_id)
ordered_product (type, acct_status_id, is_renewable, validity_end_date, ord_prod_id)
You should also fix the date literal being used in the WHERE clause.
Either way, if you want a more categorical response, you need to include the full schema for those tables, including indexes and foreign keys (to validate your joins) as well as row counts, cardinality, etc (to direct you on index creation, performance tuning, etc).
Edited to remove description per Rick James's comment.
An index designed match your query will make it faster. (But it won't make it correct if it's wrong, just sayin'.)
You filter your ordered_product table on:
type (two values)
account_status_id (equality)
is_Renewable (equality)
validity_end_date (date range)
So try an index with all the equality match columns followed by the range column. MySql can random-access it to the first eligible column and then scan it sequentially. Fast.
CREATE INDEX desc_stat_renew_date ON ordered_product
(acct_status_id, is_Renewable, validity_end_date);
To learn about this, see Marcus Winad's excellent https://use-the-index-luke.com/ .
Pro tip Using mixed case in column or table names can cause confusion if/when you migrate to another server machine. Avoid!
There are several orders in which the Optimizer might evaluate the query. Here are some additional indexes that the optimizer might use, especially if it chooses to start with op.
op: INDEX(acct_status_id, is_Renewable, type, validity_end_date, serv_id, ord_prod_id)
op: INDEX(acct_status_id, is_Renewable, validity_end_date, type, serv_id, ord_prod_id)
Related
Why using inner join itself is more faster than where
Here is my table: CREATE TABLE `cerp_oms_order` ( `id` bigint NOT NULL, `company_id` bigint NOT NULL, `order_no` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL, `source_type` int NOT NULL, `shop_id` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL, `outer_shop` varchar(64) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL, `origin_status` int NOT NULL, `system_status` int NOT NULL, `created_time` datetime DEFAULT NULL, `paid_time` datetime DEFAULT NULL, `sent_time` datetime DEFAULT NULL, `end_time` datetime DEFAULT NULL, `modify_time` datetime DEFAULT NULL, `delivery_deadline_time` datetime DEFAULT NULL, `amount` text CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL , `spu_kind` int NOT NULL, `sku_kind` int NOT NULL, `total_quantity` decimal(16,4) NOT NULL, `buyer_id` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL, `outer_buyer_identifier` varchar(64) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL DEFAULT '', `tax_info` varchar(256) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL, `warehouse_owner` int DEFAULT NULL, `warehouse_id` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL, `logistics_type` int NOT NULL, `logistics_outer_info` varchar(256) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL, `delivery_id` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL, `delivery_no` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL, `wave_no` varchar(64) DEFAULT '', `is_deleted` tinyint NOT NULL DEFAULT '0', `backend_processing_type` tinyint NOT NULL, `create_type` tinyint NOT NULL, `is_hang_up` tinyint NOT NULL, `hang_up_case_type` smallint DEFAULT NULL, `hang_up_case_id` bigint DEFAULT NULL, `rc_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, `rm_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `vat` varchar(128) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL, `weight` decimal(16,4) NOT NULL DEFAULT '0.0000', `volume` decimal(16,4) NOT NULL DEFAULT '0.0000', `is_abnormal` tinyint NOT NULL DEFAULT '0', `estimate_profit` decimal(16,4) NOT NULL DEFAULT '0.0000', `business_man_id` varchar(256) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL, `business_man` varchar(256) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL, `currency` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL, `net_amount_summary` decimal(19,4) NOT NULL, `domestic_amount` decimal(19,4) NOT NULL, `secret_key` varchar(64) DEFAULT '', `secretKey` varchar(255) DEFAULT NULL, `sale_id` bigint DEFAULT NULL, `total_refund_include_tax` decimal(16,4) NOT NULL DEFAULT '0.0000', `total_refund_money` decimal(16,4) NOT NULL DEFAULT '0.0000', `total_refund_tax` decimal(16,4) NOT NULL DEFAULT '0.0000', `total_return_goods` decimal(16,2) NOT NULL DEFAULT '0.00', PRIMARY KEY (`id`), UNIQUE KEY `key_order_no` (`order_no`), KEY `idx_order_company_id` (`company_id`,`created_time`), KEY `IDX_RM_TIME` (`rm_time`), KEY `IDX_IS_ABNORMAL` (`is_abnormal`), KEY `cerp_oms_order_company_id_index` (`company_id`), KEY `idx_order_company_status_deleted` (`company_id`,`is_deleted`,`system_status`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci COMMENT='system order table' explain select * from cerp_oms_order inner join (select id from cerp_oms_order where source_type = 43 order by created_time) as tmp using (id); id select_type table partitions type possible_keys key key_len ref rows filtered Extra 1 SIMPLE cerp_oms_order null ALL PRIMARY null null null 60787 10 Using where 1 SIMPLE cerp_oms_order null eq_ref PRIMARY PRIMARY 8 cerp_oms_1.cerp_oms_order.id 1 100 null using inner join execution: 7 ms, fetching: 109 ms VS explain select * from cerp_oms_order where source_type = 43 order by created_time; id select_type table partitions type possible_keys key key_len ref rows filtered Extra 1 SIMPLE cerp_oms_order null ALL null null null null 60787 10 Using where; Using filesort using simple where clause execution: 80 ms, fetching: 138 ms I don't understand why use inner join can speed up my sql?
If you had INDEX(source_type, created_time) both formulations would run faster. And, I think, the second would be faster. As for "why". Look at what the second does: Scan the entire table ("table scan") looking for rows with source_type = 43. For each matching row, gather all the columns (*) from the row, into a temp table. Sort that temp table. Note that this is bulkier than with the subquery that hauls around only id. Deliver the results. With the INDEX I suggest: Look in the index's B+Tree for all 'rows' with source_type = 43. Since those rows are sorted by created_time, not "sort" pass will be needed now or later. Each of those index rows will contain id. (That's the way InnoDB gets to the rest of the columns), so Look up the desired rows in the data's B+Tree and pull out all (*) the columns. Deliver them. Note that the JOIN version works essentially the same, but, technically speaking, the ORDER BY may get lost. If it does get lost, you would need to add a second ORDER BY, thereby forcing a sort. (Unrelated) Your "KEY cerp_oms_order_company_id_index (company_id)," can be dropped since there are two other indexes starting with company_id. More on optimizing indexes: Index Cookbook
how to rewrite a join query with CTE?
here I have tried to rewrite the query with cte cuz of good readability but when I try to rewrite the data is mismatched how to solve the problem for this? Query; select count(1) as rage_tap from ue_summary.summary_funnel_1066 s join user_tasks_metadata utm on utm.asi = s.asi join user_tasks ut on ut.id = utm.user_task_id where s.seq_no = 1 and s.created_at between '2022-09-27 00:00:00' and '2022-10-27 00:00:00' and ut.is_ragetap = 1 Explain plan ; *************************** 1. row *************************** id: 1 select_type: SIMPLE table: ut partitions: NULL type: ref possible_keys: PRIMARY,idx_ir key: idx_ir key_len: 1 ref: const rows: 8413412 filtered: 100.00 Extra: Using index *************************** 2. row *************************** id: 1 select_type: SIMPLE table: utm partitions: NULL type: ref possible_keys: id_asi,asi key: id_asi key_len: 8 ref: ue_stage.ut.id rows: 1 filtered: 100.00 Extra: Using index *************************** 3. row *************************** id: 1 select_type: SIMPLE table: s partitions: NULL type: eq_ref possible_keys: PRIMARY,unique_asi_seq_no,seq_no_date,created_at,idx_combo,idx_seq_created_asi key: unique_asi_seq_no key_len: 12 ref: ue_stage.utm.asi,const rows: 1 filtered: 50.00 Extra: Using where; Using index Table structure; Create Table: CREATE TABLE `summary_funnel_1066` ( `funnel_id` int DEFAULT NULL, `app_id` int DEFAULT NULL, `platform` int DEFAULT NULL, `app_version_id` int NOT NULL, `seq_no` int NOT NULL, `property_id` bigint DEFAULT NULL, `property_name` varchar(255) DEFAULT NULL, `property_type` varchar(50) DEFAULT NULL, `asi` bigint NOT NULL, `created_at` datetime NOT NULL, `capture_time_relative` decimal(15,4) DEFAULT NULL, `last_event_id` bigint DEFAULT NULL, `last_event_name` varchar(100) DEFAULT NULL, `last_message_id` bigint DEFAULT NULL, `last_message_name` varchar(100) DEFAULT NULL, `last_tag_id` bigint DEFAULT NULL, `last_tag_name` varchar(100) DEFAULT NULL, `is_crash` tinyint DEFAULT NULL, `is_anr` tinyint DEFAULT NULL, `is_ragetap` tinyint DEFAULT NULL, `last_error_type_id` bigint DEFAULT NULL, `last_error_type` varchar(100) DEFAULT NULL, `screen_id` bigint DEFAULT NULL, `screen_name` varchar(100) DEFAULT NULL, `last_screen_id` bigint DEFAULT NULL, `last_screen_name` varchar(100) DEFAULT NULL, `user_task_id` bigint DEFAULT NULL, `ue_id` bigint DEFAULT NULL, PRIMARY KEY (`asi`,`seq_no`,`created_at`,`app_version_id`), UNIQUE KEY `unique_asi_seq_no` (`asi`,`seq_no`), KEY `seq_no_date` (`seq_no`,`created_at`), KEY `last_ids` (`last_screen_id`,`last_event_id`), KEY `idx_seq_created_asi`(seq_no,created_at,asi), KEY `created_at` (`created_at`), KEY `idx_combo` (`seq_no`,`property_id`,`property_name`,`created_at`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 Table: user_tasks_metadata Create Table: CREATE TABLE `user_tasks_metadata` ( `id` bigint NOT NULL AUTO_INCREMENT, `user_task_id` bigint NOT NULL, `device_id` bigint NOT NULL, `custom_user_id` bigint DEFAULT NULL, `asi` bigint NOT NULL DEFAULT '0', `session_id` varchar(300) DEFAULT NULL, `model` bigint DEFAULT NULL, `api_level` varchar(300) DEFAULT NULL, `app_version_id` bigint NOT NULL DEFAULT '0', `os_version` bigint DEFAULT NULL, `location` bigint DEFAULT NULL, `connection_speed` varchar(10) DEFAULT NULL, `network_operator` varchar(100) CHARACTER SET utf8mb3 COLLATE utf8_general_ci DEFAULT NULL, `config_response` tinyint DEFAULT '1', `total_internal_memory` double(12,5) DEFAULT NULL, `available_internal_memory` double(12,5) DEFAULT NULL, `total_ram` double(12,5) DEFAULT NULL, `available_ram` double(12,5) DEFAULT NULL, `framework` varchar(45) DEFAULT '', `ue_sdk_version` mediumint DEFAULT NULL, `crash_type` bigint DEFAULT NULL, `created_at` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, `user_profile_id` bigint DEFAULT NULL, `associated_custom_user_id` bigint DEFAULT NULL, `first_usr_interaction` bigint DEFAULT NULL, `app_launch_type` varchar(45) DEFAULT '', `app_launch_time` bigint DEFAULT '0', PRIMARY KEY (`id`), KEY `session_metadata_filter_idx` (`custom_user_id`,`device_id`), KEY `usertask_fk_idx` (`user_task_id`), KEY `idx_app_version` (`app_version_id`), KEY `asi_idx` (`asi`), KEY `device_id` (`device_id`), KEY `user_profile_id` (`user_profile_id`), KEY `id_asi` (`user_task_id`,`asi`), KEY `asi` (`asi`) ) ENGINE=InnoDB AUTO_INCREMENT=2252872743 DEFAULT CHARSET=latin1 Table: user_tasks Create Table: CREATE TABLE `user_tasks` ( `id` bigint NOT NULL AUTO_INCREMENT, `app_id` bigint NOT NULL, `status` tinyint NOT NULL DEFAULT '0', `app_version` varchar(100) DEFAULT NULL, `platform` tinyint NOT NULL DEFAULT '1', `exception_type` tinyint NOT NULL DEFAULT '0', `error_count` smallint NOT NULL DEFAULT '0', `crash_type` varchar(300) DEFAULT NULL, `crash_log` varchar(300) DEFAULT NULL, `avg_signal_level` int DEFAULT '0', `is_read` tinyint(1) NOT NULL DEFAULT '0', `is_important` tinyint(1) NOT NULL DEFAULT '0', `is_video_available` tinyint(1) NOT NULL DEFAULT '0', `is_video_played` tinyint(1) NOT NULL DEFAULT '0', `is_ex` tinyint(1) NOT NULL DEFAULT '0', `is_ragetap` tinyint(1) NOT NULL DEFAULT '0', `session_start_time` datetime DEFAULT NULL, `network_type` tinyint NOT NULL DEFAULT '0', `s3_video_url` varchar(255) DEFAULT NULL, `image_format` tinyint DEFAULT '0', `ue_release_version` smallint NOT NULL DEFAULT '0', `created_at` datetime NOT NULL, `updated_at` datetime DEFAULT NULL, `batch_created_at` datetime DEFAULT NULL, `sys_creation_date` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `session_filter_idx_2` (`app_id`,`platform`,`created_at`,`exception_type`,`app_version`), KEY `batch_created_idx` (`app_id`,`platform`,`batch_created_at`), KEY `app_id_created_at` (`app_id`,`created_at`), KEY `id_app_id` (`app_id`), KEY `idx_ir` (`is_ragetap`) ) ENGINE=InnoDB AUTO_INCREMENT=1648177712 DEFAULT CHARSET=latin1 rewritten query; with cte1 as ( select asi,count(1) as rage_tap from ue_summary.summary_funnel_1066 where s.seq_no = 1 and s.created_at between '2022-09-27 00:00:00' and '2022-10-27 00:00:00' ), cte2 as ( select id, count(*) 'rage_tap1' from user_tasks ut where is_ragetap = 1 ) select cte1.*,cte2.* from cte1 inner join user_tasks_metadata utm on utm.asi = cte1.asi inner join cte2 on b.id = utm.user_task_id I need like below output; +----------+ | rage_tap | +----------+ | 1812564 | +----------+ It takes time to search so I choose cte, I have tried with subquery but it does not work and it takes around 30 sec - 1.14 min. as per this, I have indexed the column but also takes time : slow performance of query and scanning many rows is there any other way to optimize it?
This is your query with two CTEs. The aggregation takes place after the tables after the joins, just as in the original query. with s as ( select * from ue_summary.summary_funnel_1066 where seq_no = 1 and created_at >= date '2022-09-27' and created_at < date '2022-10-27' ) , ut as ( select * from user_tasks where is_ragetap = 1 ) select count(*) as rage_tap from s join user_tasks_metadata utm on utm.asi = s.asi join ut on ut.id = utm.user_task_id; As created_at is a datetime, you should not use BETWEEN, but >= and <. Please check if the date range that I put in my query matches your requirements. It excludes 2022-10-27. If you want to include it, change this to and created_at < date '2022-10-28'.
select id, count(*) 'rage_tap1' from user_tasks ut where is_ragetap = 1 does not make sense. There is an aggregate (COUNT(*)) but no GROUP BY. Were you showing to get one row? If so, which row? GROUP BY id does not make since id is Unique. DOUBLE(m,n) is worse than simply DOUBLE. In fact, (m,n) is going away in 8.0 for FLOAT and DOUBLE. When you have INDEX(a,b), you don't need INDEX(a). count(1) as rage_tap is done after the JOINs, so it may have an inflated value. Did you do a sanity check? utm has two indexes on asi. Toss both and add INDEX(asi, user_task_id) As for turning the Joins into Ctes, go back to when you envisioned the query. You probably said "I need this stuff from this table", then "that stuff from that table", and finally "put things together this way". If you can go back to that thought process, you have the CTEs. (I don't have any idea what the data means or what the goal is, so I cannot reproduce that thought process.)
slow performance of query and scanning many rows
The below query is taking 6.18 min to execute for one row and the cardinality value for the exception_type = 1 is 3, I don't know how to improve the performance. Query; select count(1) as rage_tap from summary_funnel_1066 s join user_tasks_metadata utm on utm.asi = s.asi join user_tasks ut on ut.id = utm.user_task_id where s.seq_no = 1 and s.created_at between '2022-09-27 00:00:00' and '2022-10-27 00:00:00' and ut.exception_type = 1 explain plan; *************************** 1. row *************************** id: 1 select_type: SIMPLE table: ut partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 129554700 filtered: 10.00 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: utm partitions: NULL type: ref possible_keys: usertask_fk_idx,asi_idx,id_asi,asi key: usertask_fk_idx key_len: 8 ref: ue_stage.ut.id rows: 1 filtered: 100.00 Extra: NULL *************************** 3. row *************************** id: 1 select_type: SIMPLE table: s partitions: NULL type: eq_ref possible_keys: PRIMARY,unique_asi_seq_no,seq_no_date,created_at,idx_combo key: unique_asi_seq_no key_len: 12 ref: ue_stage.utm.asi,const rows: 1 filtered: 50.00 Extra: Using where; Using index table structure; Create Table: CREATE TABLE `summary_funnel_1066` ( `funnel_id` int DEFAULT NULL, `app_id` int DEFAULT NULL, `platform` int DEFAULT NULL, `app_version_id` int NOT NULL, `seq_no` int NOT NULL, `property_id` bigint DEFAULT NULL, `property_name` varchar(255) DEFAULT NULL, `property_type` varchar(50) DEFAULT NULL, `asi` bigint NOT NULL, `created_at` datetime NOT NULL, `capture_time_relative` decimal(15,4) DEFAULT NULL, `last_event_id` bigint DEFAULT NULL, `last_event_name` varchar(100) DEFAULT NULL, `last_message_id` bigint DEFAULT NULL, `last_message_name` varchar(100) DEFAULT NULL, `last_tag_id` bigint DEFAULT NULL, `last_tag_name` varchar(100) DEFAULT NULL, `is_crash` tinyint DEFAULT NULL, `is_anr` tinyint DEFAULT NULL, `is_ragetap` tinyint DEFAULT NULL, `last_error_type_id` bigint DEFAULT NULL, `last_error_type` varchar(100) DEFAULT NULL, `screen_id` bigint DEFAULT NULL, `screen_name` varchar(100) DEFAULT NULL, `last_screen_id` bigint DEFAULT NULL, `last_screen_name` varchar(100) DEFAULT NULL, `user_task_id` bigint DEFAULT NULL, `ue_id` bigint DEFAULT NULL, PRIMARY KEY (`asi`,`seq_no`,`created_at`,`app_version_id`), UNIQUE KEY `unique_asi_seq_no` (`asi`,`seq_no`), KEY `seq_no_date` (`seq_no`,`created_at`), KEY `last_ids` (`last_screen_id`,`last_event_id`), KEY `created_at` (`created_at`), KEY `idx_combo` (`seq_no`,`property_id`,`property_name`,`created_at`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 Table: user_tasks_metadata Create Table: CREATE TABLE `user_tasks_metadata` ( `id` bigint NOT NULL AUTO_INCREMENT, `user_task_id` bigint NOT NULL, `device_id` bigint NOT NULL, `custom_user_id` bigint DEFAULT NULL, `asi` bigint NOT NULL DEFAULT '0', `session_id` varchar(300) DEFAULT NULL, `model` bigint DEFAULT NULL, `api_level` varchar(300) DEFAULT NULL, `app_version_id` bigint NOT NULL DEFAULT '0', `os_version` bigint DEFAULT NULL, `location` bigint DEFAULT NULL, `connection_speed` varchar(10) DEFAULT NULL, `network_operator` varchar(100) CHARACTER SET utf8mb3 COLLATE utf8_general_ci DEFAULT NULL, `config_response` tinyint DEFAULT '1', `total_internal_memory` double(12,5) DEFAULT NULL, `available_internal_memory` double(12,5) DEFAULT NULL, `total_ram` double(12,5) DEFAULT NULL, `available_ram` double(12,5) DEFAULT NULL, `framework` varchar(45) DEFAULT '', `ue_sdk_version` mediumint DEFAULT NULL, `crash_type` bigint DEFAULT NULL, `created_at` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, `user_profile_id` bigint DEFAULT NULL, `associated_custom_user_id` bigint DEFAULT NULL, `first_usr_interaction` bigint DEFAULT NULL, `app_launch_type` varchar(45) DEFAULT '', `app_launch_time` bigint DEFAULT '0', PRIMARY KEY (`id`), KEY `session_metadata_filter_idx` (`custom_user_id`,`device_id`), KEY `usertask_fk_idx` (`user_task_id`), KEY `idx_app_version` (`app_version_id`), KEY `asi_idx` (`asi`), KEY `device_id` (`device_id`), KEY `user_profile_id` (`user_profile_id`), KEY `id_asi` (`user_task_id`,`asi`), KEY `asi` (`asi`) ) ENGINE=InnoDB AUTO_INCREMENT=2252872743 DEFAULT CHARSET=latin1 Table: user_tasks Create Table: CREATE TABLE `user_tasks` ( `id` bigint NOT NULL AUTO_INCREMENT, `app_id` bigint NOT NULL, `status` tinyint NOT NULL DEFAULT '0', `app_version` varchar(100) DEFAULT NULL, `platform` tinyint NOT NULL DEFAULT '1', `exception_type` tinyint NOT NULL DEFAULT '0', `error_count` smallint NOT NULL DEFAULT '0', `crash_type` varchar(300) DEFAULT NULL, `crash_log` varchar(300) DEFAULT NULL, `avg_signal_level` int DEFAULT '0', `is_read` tinyint(1) NOT NULL DEFAULT '0', `is_important` tinyint(1) NOT NULL DEFAULT '0', `is_video_available` tinyint(1) NOT NULL DEFAULT '0', `is_video_played` tinyint(1) NOT NULL DEFAULT '0', `is_ex` tinyint(1) NOT NULL DEFAULT '0', `is_ragetap` tinyint(1) NOT NULL DEFAULT '0', `session_start_time` datetime DEFAULT NULL, `network_type` tinyint NOT NULL DEFAULT '0', `s3_video_url` varchar(255) DEFAULT NULL, `image_format` tinyint DEFAULT '0', `ue_release_version` smallint NOT NULL DEFAULT '0', `created_at` datetime NOT NULL, `updated_at` datetime DEFAULT NULL, `batch_created_at` datetime DEFAULT NULL, `sys_creation_date` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `session_filter_idx_2` (`app_id`,`platform`,`created_at`,`exception_type`,`app_version`), KEY `batch_created_idx` (`app_id`,`platform`,`batch_created_at`), KEY `app_id_created_at` (`app_id`,`created_at`), KEY `id_app_id` (`app_id`,`id`) ) ENGINE=InnoDB AUTO_INCREMENT=1636224717 DEFAULT CHARSET=latin1 is there any way possible to improve the performance of the query? fdsfdfjdnfkjdkjfnkjdnjkfndkjnfjkndjkfnkjdnsfkjndsjknf
Indexes: utm: INDEX(asi, user_task_id) s: INDEX(seq_no, created_at, asi)
how to optimize the below query without adding index
the below query has some problems so can we able to optimize the query with the current index format cuz the table size is huge so I don't know what to do. Query: select `sale_amount`, `provider_transaction_id`, `id` as `acc_id`, `status`, `service_id`, `provider_status`, `is_sale_settled`, CASE WHEN status = 1 and is_sale_settled = 1 then 4 WHEN status = 1 and is_sale_settled = 0 then 3 ELSE 5 END AS acc_ics_status, `system_transaction_id`, `engine_transaction_id`, `transaction_date`, `created_at` as `transaction_date_time`, HOUR(created_at) as transaction_hour from `st_ret_txn_aeps` where `created_at` >= '2022-10-03 14:45:01' and `created_at` <= '2022-10-03 14:48:00' and `service_id` = 14 and `transaction_date` = '2022-10-03' Row scans; *************************** 1. row *************************** id: 1 select_type: SIMPLE table: st_ret_txn_aeps partitions: NULL type: index_merge possible_keys: transaction_id_sevrice_id_unique,st_ret_txn_aeps_service_id_index,idx_txn_date key: idx_txn_date,st_ret_txn_aeps_service_id_index key_len: 3,1 ref: NULL rows: 122328 filtered: 11.11 Extra: Using intersect(idx_txn_date,st_ret_txn_aeps_service_id_index); Using where 1 row in set, 1 warning (0.00 sec) Table structure; *************************** 1. row *************************** Table: st_ret_txn_aeps Create Table: CREATE TABLE `st_ret_txn_aeps` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `service_id` tinyint(4) NOT NULL, `transaction_type_id` tinyint(4) NOT NULL, `business_org_id` int(10) unsigned NOT NULL, `sub_agent_id` int(11) DEFAULT NULL, `agent_id` int(10) unsigned DEFAULT NULL, `business_org_user_id` int(10) unsigned NOT NULL, `business_org_sub_user_id` int(10) unsigned DEFAULT NULL, `retailer_category_code` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL, `rate_profile_id` int(11) DEFAULT '0', `wallet_type` varchar(20) COLLATE utf8mb4_unicode_ci NOT NULL, `business_org_pan` varchar(12) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `business_org_gstin` varchar(25) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `operator_id` int(10) unsigned NOT NULL, `operator_name` varchar(100) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `product_id` int(11) DEFAULT NULL, `sub_product_id` int(11) DEFAULT NULL, `provider_id` int(10) unsigned NOT NULL, `provider_name` varchar(100) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `branch_id` int(11) NOT NULL DEFAULT '0', `branch_name` varchar(100) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `sale_amount` double NOT NULL, `total_b_org_charges` double NOT NULL, `total_b_org_comm_tds` double DEFAULT '0', `total_b_org_commission` double NOT NULL, `final_debit_value` double NOT NULL, `balance_after_txn` double NOT NULL, `system_transaction_id` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL, `engine_transaction_id` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL, `provider_transaction_id` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `operator_transaction_id` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `reference_1` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `reference_2` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `reference_3` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `reference_4` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `reference_5` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `business_org_markup` double NOT NULL DEFAULT '0', `refund_date` date DEFAULT NULL, `remark` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `is_invoice_data_generated` tinyint(1) NOT NULL, `status` tinyint(4) NOT NULL COMMENT '1: success ,2: failed', `provider_status` tinyint(4) DEFAULT NULL, `provider_status_old` tinyint(4) DEFAULT NULL, `transaction_date` date NOT NULL, `customer_id` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `customer_mobile_no` varchar(15) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `app_type` varchar(40) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT '1:WEBPOS 2:ANDROID', `is_cashout_sale` tinyint(1) NOT NULL, `is_sale_settled` tinyint(1) NOT NULL, `settlement_date` date DEFAULT NULL, `analytics_sync_status` tinyint(1) DEFAULT NULL, `analytics_sync_status_refund` tinyint(1) DEFAULT NULL, `invoice_data_id` int(11) DEFAULT NULL, `credit_note_data_id` int(11) DEFAULT NULL, `status_updated_at` bigint(20) DEFAULT NULL, `created_at` timestamp NULL DEFAULT NULL, `updated_at` timestamp NULL DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `transaction_id_sevrice_id_unique` (`service_id`,`engine_transaction_id`), UNIQUE KEY `system_transaction_id_unique` (`system_transaction_id`), KEY `st_ret_txn_aeps_service_id_index` (`service_id`), KEY `st_ret_txn_aeps_business_org_id_index` (`business_org_id`), KEY `st_ret_txn_aeps_agent_id_index` (`agent_id`), KEY `st_ret_txn_aeps_business_org_user_id_index` (`business_org_user_id`), KEY `st_ret_txn_aeps_system_transaction_id_index` (`system_transaction_id`), KEY `st_ret_txn_aeps_refund_date_index` (`refund_date`), KEY `idx_txn_date` (`transaction_date`), KEY `st_ret_txn_aeps_settlement_date_index` (`settlement_date`), KEY `status_updated_at_idx` (`status_updated_at`) ) ENGINE=InnoDB AUTO_INCREMENT=122756253 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci yeah I know if we create an index for the created_at column means the problem will solve does this have any other way to optimize?
Well, you could delete all the data, then the query would be very quick (just kidding)! Honestly, you need to create the index to optimize this query. The best index for this query would be a compound index on (service_id, transaction_date, created_at) in that order. There are at least three ways to create the index without much downtime: pt-online-schema-change gh-ost Create the index on a replica database, then when the index is ready, switch the app to use the replica At my last job we used pt-online-schema-change. While I worked there we executed over 75,000 ALTER TABLEs in production during peak hours, without downtime. Some of the tables were larger than yours.
mysql optimizer confusion explain plan
I am hoping someone can help decipher an explain plan and information in the optimizer trace. I have the following query: select doc_id, group_concat(distinct concat(a.event_time, id mod 10) order by id) as times, count(distinct id) as count_of_items FROM doc_audit a, rwfd1213310051e2821b404f29bc453ce7386f764e b WHERE b.rptid = a.doc_id AND a.property = 'status' group by 1; And get this execution plan. *************************** 1. row *************************** id: 1 select_type: SIMPLE table: b partitions: NULL type: index possible_keys: idx_tmp_rptid key: idx_tmp_rptid key_len: 8 ref: NULL rows: 7153 filtered: 100.00 Extra: Using index; Using temporary; Using filesort *************************** 2. row *************************** id: 1 select_type: SIMPLE table: a partitions: NULL type: ref possible_keys: DOC_ID_PROPERTY_IDX,event_time,event_time2 key: DOC_ID_PROPERTY_IDX key_len: 776 ref: database.b.rptid,const rows: 2 filtered: 43.75 Extra: Using where My limited understanding is the order in the explain plan is the order the plan is executed. If this is the case I don't understand where the "Using filsort" comes from on table b? If I enable the optimzer profile trace it suggests the filsort is related to table a. I reached this conclusion because doc_id is in table a. Furthermore I believe filsort simply means "Sort"? doc_id is indexed so I don't understand why it needs sorting further. "filesort_information": [ { "direction": "asc", "table": "intermediate_tmp_table", "field": "**doc_id**" } ], "filesort_priority_queue_optimization": { "usable": false, "cause": "not applicable (no LIMIT)" }, "filesort_execution": [ ], "filesort_summary": { "rows": 17620, "examined_rows": 17620, "number_of_tmp_files": 2, "sort_buffer_size": 262128, "sort_mode": "<sort_key, rowid>" I would really appreciate any help on this. Apologies if my understanding is way off. doc_audit PRIMARY KEY (id), KEY DOC_ID_PROPERTY_IDX (doc_id,property), KEY event (instance_id,event,event_time), KEY event_time (doc_id,event,event_time), KEY event_user (instance_id,user_id,event,event_time), KEY event_time2 (event_time,doc_id,event), KEY user (user_id,event_time,instance_id), KEY event_time_user_event (event_time,user_id,event(80)) Adding the table defs: CREATE TABLE doc_audit ( id bigint(20) NOT NULL AUTO_INCREMENT, doc_id bigint(20) NOT NULL, event varchar(255) , event_time datetime NOT NULL, instance_id bigint(20) NOT NULL, major_version int(11) NOT NULL, minor_version int(11) NOT NULL, new_value varchar(1500) DEFAULT NULL, old_value varchar(1500) DEFAULT NULL, property varchar(255) DEFAULT NULL, signature_meaning varchar(255) DEFAULT NULL, task_name varchar(255) DEFAULT NULL, user_id bigint(20) NOT NULL, version int(11) DEFAULT NULL, workflow_name varchar(255) DEFAULT NULL, user_on_behalf_of bigint(20) DEFAULT NULL, xml longtext, user_name varchar(255) DEFAULT NULL, user_display_name varchar(255) DEFAULT NULL, document_number varchar(255) DEFAULT NULL, property_public_key varchar(100) DEFAULT NULL, property_label varchar(500) DEFAULT NULL, old_value_public_key varchar(100) DEFAULT NULL, old_value_label varchar(1500) DEFAULT NULL, new_value_public_key varchar(100) DEFAULT NULL, new_value_label varchar(1500) DEFAULT NULL, workflow_label varchar(500) DEFAULT NULL, task_label varchar(500) DEFAULT NULL, precalculated bit(1) DEFAULT b'0', PRIMARY KEY (id), KEY DOC_ID_PROPERTY_IDX (doc_id,property), KEY event (instance_id,event,event_time), KEY event_time (doc_id,event,event_time), KEY event_user (instance_id,user_id,event,event_time), KEY event_time2 (event_time,doc_id,event), KEY user (user_id,event_time,instance_id), KEY event_time_user_event (event_time,user_id,event(80)) ) ENGINE=InnoDB AUTO_INCREMENT=1209095 DEFAULT CHARSET=utf8 and.. CREATE TEMPORARY TABLE rwfd1213310051e2821b404f29bc453ce7386f764e ( wfrptid bigint(20) NOT NULL DEFAULT '0', workflow_status__v varchar(255) COLLATE utf8_bin DEFAULT NULL, workflow_id__v bigint(20) DEFAULT NULL, task_capacity_key__v varchar(255) COLLATE utf8_bin DEFAULT NULL, workflow_process_version__v bigint(20) DEFAULT NULL, workflow_name__v varchar(255) COLLATE utf8_bin DEFAULT NULL, workflow_initiator_name__v varchar(255) COLLATE utf8_bin DEFAULT NULL, task_delegate__v varchar(255) COLLATE utf8_bin DEFAULT NULL, workflow_completionDate__v datetime DEFAULT NULL, workflow_initiator__v varchar(255) COLLATE utf8_bin DEFAULT NULL, workflow_cancelationDate__v datetime DEFAULT NULL, workflow_document_id__v bigint(20) DEFAULT NULL, task_verdict_key__v varchar(255) COLLATE utf8_bin DEFAULT NULL, workflow_lifecycle__v varchar(255) COLLATE utf8_bin DEFAULT NULL, task_assignee_name__v varchar(255) COLLATE utf8_bin DEFAULT NULL, task_meta_data_id__v varchar(255) COLLATE utf8_bin DEFAULT NULL, task_completionDate__v datetime DEFAULT NULL, workflow_startDate__v datetime DEFAULT NULL, task_iteration_count__v bigint(20) DEFAULT NULL, rptid bigint(20) NOT NULL DEFAULT '0', major_version_number__v bigint(20) DEFAULT NULL, minor_version_number__v bigint(20) DEFAULT NULL, study__v varchar(1500) COLLATE utf8_bin DEFAULT NULL, document_number__v varchar(255) COLLATE utf8_bin DEFAULT NULL, status__v varchar(1500) COLLATE utf8_bin DEFAULT NULL, name__v varchar(100) COLLATE utf8_bin DEFAULT NULL, days_in_qc__c bigint(20) DEFAULT NULL, durationInValue15f07c5893cf4ba98a3441aab045c460 varchar(255) COLLATE utf8_bin DEFAULT NULL, 0PERIOD int(6) DEFAULT NULL, KEY idx_tmp_rptid (rptid) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin
Two exceptions to the order of tables in EXPLAIN: Subqueries tend to be at the end of the EXPLAIN. Using temporary and Using filesort tend to be on the first line of the EXPLAIN, giving you no clue of when they are actually used. Also, there may be more than one sort, again without a clue. The EXPLAIN FORMAT=JSON is better at saying what part needed the sort. Multiple filesorts show up clearly. "filesort" is a generic term. It might involve disk (slowest), it might be a 'quicksort' in RAM, or it might be "filesort_priority_queue_optimization" (handy with LIMIT). There may be other options. (Please provide SHOW CREATE TABLE.) event(80) -- Index prefixes are rarely useful. Was this a TEXT column? The optimal index for this query for a is INDEX(property, doc_id, event_time, id)