Reduce MySQL Query Runtime - mysql

Question

I have a query like this:
SELECT DISTINCT `cr`.`idCustomer`, `rbase`.`id`
FROM `customers` `t`
JOIN `customersregion` `cr` ON t.idCustomer = cr.idCustomer
and cr.isDeleted = 0
JOIN `calendaritems` `rbase` ON rbase.idAgentsRegion = cr.idRegion
and rbase.isDeleted = 0
where (
(rbase.startDate <= '2020-07-06 00:00:00' and rbase.endDate >= '2020-07-06 00:00:00') or
(rbase.startDate <= '2020-07-28 00:00:00' and rbase.endDate >= '2020-07-28 00:00:00') or
(rbase.startDate >= '2020-07-06 00:00:00' and rbase.startDate <= '2020-07-28 23:59:59') or
(rbase.endDate >= '2020-07-06 00:00:00' and rbase.endDate <= '2020-07-28 23:59:59')
)
Database: MySQL
Customers: 132,000 row
CustomersRegion: 1,754,000 row
CalendarItems: 3,838,000 row (with conditions reduce to 555,000 row)
t.idCustomer & cr.idCustomer & cr.isDeleted & rbase.idAgentsRegion & cr.idRegion & rbase.isDeleted are indexes
This query runtime is about 100 seconds and i want to reduce the runtime of this query
I can't have limit on rows or have another condition in tables
Can you help me?
Thank you
Explain Query:
Customers DDL:
create table customers
(
idCustomer int auto_increment
primary key,
CustomerName varchar(255) not null comment 'نام فروشگاه',
FirstName varchar(60) null comment 'نام رابط',
LastName varchar(60) null comment 'نام مشتري',
idUser int null comment '!#dont show',
idPayment int null,
idCompany int default 0 not null,
LatitudePoint decimal(18, 12) default 0.000000000000 null comment 'gpslat',
LongitudePoint decimal(18, 12) default 0.000000000000 null comment 'gpslongs',
LastOrderDate datetime default '0000-00-00 00:00:00' null comment 'lastorderdate',
VisitPeriod int default 0 null comment 'visitperiod',
LastVisit datetime default '0000-00-00 00:00:00' null comment 'LastVisitDate',
LastNoOrderDate datetime default '0000-00-00 00:00:00' null,
Credit decimal(20, 4) default 0.0000 null comment 'credit',
RemainCredit decimal(20, 4) default 0.0000 null comment 'remaincredit',
Balance decimal(20, 4) default 0.0000 null comment '!#dont show',
RFID varchar(60) null comment 'rfid',
ReturnCheck tinyint(1) default 0 null comment '!#dont show',
AccountStatus tinyint(1) default 0 null comment 'accountstatus',
FaxNumber varchar(20) null,
LiquidationDate date default '0000-00-00' null comment '!#dont show',
EldestDue date default '0000-00-00' null comment '!#dont show',
MaturityDate date default '0000-00-00' null comment '!#dont show',
PriceKind int null,
isDefault tinyint(1) default 0 not null comment '!#dont show',
TimeStamp timestamp default current_timestamp() not null on update current_timestamp(),
isDeleted tinyint(1) default 0 not null,
Address varchar(255) null,
PhoneNumber varchar(60) null,
MobileNumber varchar(60) null,
CustomerErpCode varchar(60) null comment '!#dont show',
StoreType int null,
country varchar(255) null,
state varchar(255) null,
City varchar(30) null,
Region varchar(30) null,
idUserCreator int null,
idBranche int null,
idTagsinfo int null,
shop_id int null,
shop_id_address int null,
lastActivityDate datetime null,
lastActivityType tinyint(1) null,
duplicateOf int null,
isConfirmed tinyint(1) default 2 not null comment '0:rejected - 1:confirmed - 2:notChecked',
Status tinyint(1) default 1 not null,
createDate datetime null,
idProcess int null comment 'نیازی نیست به اینکه حتما پروسه داشته باشد',
idUserConfirmer int null comment 'this is refered to agents table',
nextDate datetime null,
prevDate datetime null,
idImage int null,
idColor int null,
idRate int null,
LastImageDate datetime null,
LastOrderAgentName varchar(255) null,
LastVisitAgentName varchar(255) null,
LastNoOrderAgentName varchar(255) null,
LastImageAgentName varchar(255) null,
LastOrderIdAgent int null,
LastVisitIdAgent int null,
LastNoOrderIdAgent int null,
LastImageIdAgent int null,
isSaleActive tinyint(1) default 1 null,
isReturnActive tinyint(1) default 1 null,
alley varchar(256) null,
street varchar(256) null,
plaque varchar(256) null,
secondAddress varchar(255) null,
description varchar(255) null,
appType varchar(50) default 'iorder' not null,
idPipeline varchar(255) default '0' null,
constraint shop_id
unique (shop_id),
constraint shop_id_address
unique (shop_id_address),
constraint ux_customererp
unique (CustomerErpCode),
constraint customers_ibfk_1
foreign key (idBranche) references branches (idBranche)
on update set null on delete set null,
constraint customers_ibfk_2
foreign key (idTagsinfo) references tagsinfo (idTag)
on update set null on delete set null,
constraint customers_ibfk_3
foreign key (idRate) references rates (idRate)
on update set null on delete set null,
constraint customers_ibfk_4
foreign key (idColor) references colors (idColor)
on update set null on delete set null,
constraint customers_ibfk_5
foreign key (idRate) references rates (idRate)
on update set null on delete set null,
constraint customers_ibfk_6
foreign key (idColor) references colors (idColor)
on update set null on delete set null,
constraint fk_customer_agents
foreign key (idUser) references agents (idAgents)
on update set null on delete set null,
constraint fk_customer_paymant
foreign key (idPayment) references payment (idPayment),
constraint fk_customer_pricelist
foreign key (PriceKind) references pricelist (idPriceList),
constraint fk_customer_storeinfo
foreign key (StoreType) references storesinfo (idStore)
)
charset = utf8;
create index fk_customer_agents_idx
on customers (idUser);
create index fk_customer_paymant_idx
on customers (idPayment);
create index fk_customer_pricelist_idx
on customers (PriceKind);
create index fk_customer_storeinfo_idx
on customers (StoreType);
create index idBranche
on customers (idBranche);
create index idColor
on customers (idColor);
create index idProcess
on customers (idProcess);
create index idRate
on customers (idRate);
create index idTagsinfo
on customers (idTagsinfo);
create index idx_isdeleted_customername
on customers (isDeleted, CustomerName);
create index isdeleted_lat_lng
on customers (isDeleted, LatitudePoint, LongitudePoint);
create index isdeleted_status_isconfirmed
on customers (isDeleted, Status, isConfirmed);
create index lat_lng
on customers (LatitudePoint, LongitudePoint);
CalendarItems DDL:
create table calendaritems
(
id int auto_increment
primary key,
TimeStamp timestamp default current_timestamp() not null on update current_timestamp(),
isDone tinyint(1) null,
isDeleted tinyint(1) default 0 not null,
subject varchar(255) null,
startDate datetime not null,
endDate datetime not null,
isAllDayEvent tinyint(1) default 1 null,
message varchar(255) null,
color varchar(200) null,
rMessage varchar(255) null,
rTime datetime null,
rLocationLat decimal(18, 12) null,
rLocationLong decimal(18, 12) null,
idAgent int not null,
idCustomer int null,
idVisitPath int null,
isFinal tinyint(1) null,
idUserCreator int not null,
idAgentsRegion int null,
type int(5) default 1 not null,
systemFill tinyint(1) default 0 not null,
createDate datetime null,
reqUp tinyint(1) default 0 not null,
dependOn int null,
idPlan int null comment 'to keep track of customer types of a region inside a plan',
idPlanTour int null,
startTime time null,
endTime time null,
constraint calendaritems_ibfk_agents
foreign key (idAgent) references agents (idAgents),
constraint calendaritems_ibfk_agents2
foreign key (idUserCreator) references agents (idAgents),
constraint calendaritems_ibfk_customers
foreign key (idCustomer) references customers (idCustomer)
on delete set null
)
charset = utf8;
create index `Index 10`
on calendaritems (isDeleted, idAgent, startDate, idCustomer);
create index `Index 14`
on calendaritems (isDeleted, idAgent, idAgentsRegion, idPlan, startDate, endDate);
create index `Index 7`
on calendaritems (startDate);
create index `Index 8`
on calendaritems (isDeleted, idAgent, startDate, idVisitPath);
create index `Index 9`
on calendaritems (isDeleted, idAgent, startDate, idAgentsRegion);
create index createDate
on calendaritems (createDate);
create index idAgent
on calendaritems (idAgent);
create index idAgentsRegion
on calendaritems (idAgentsRegion);
create index idCustomer
on calendaritems (idCustomer);
create index idUserCreator
on calendaritems (idUserCreator);
create index idVisitPath
on calendaritems (idVisitPath);
create index reqUp
on calendaritems (reqUp);
create index `systemFill-startDate-idAgent-idPlan`
on calendaritems (systemFill, startDate, idAgent, idPlan);
CustomersRegion DDL:
create table customersregion
(
idCustomer int not null,
idRegion int not null,
idCompany int default 0 null,
isDeleted tinyint(1) default 0 null,
TimeStamp timestamp default current_timestamp() null on update current_timestamp(),
ERPCode varchar(255) default '' null,
createDate datetime null,
primary key (idCustomer, idRegion),
constraint customersregion_ibfk_1
foreign key (idCustomer) references customers (idCustomer)
on update cascade on delete cascade,
constraint customersregion_ibfk_2
foreign key (idRegion) references region (idRegion)
on update cascade on delete cascade
)
charset = utf8;
create index idRegion
on customersregion (idRegion);
create index isdeleted_idregion_idcustomer
on customersregion (isDeleted, idRegion, idCustomer);

Answer 1

The EXPLAIN plan shows that the first step taken is to scan the calendaritems table ("rbase"), scanning a total of estimated 1.6 million rows.
There is an index being used, but it's not really fitting as it has too many extra columns not really used. A better index would be one consisting of (isDeleted, startDate, endDate, idAgentsRegion), in that order, the first three columns of that would be perfect for the first three OR parts of the WHERE condition, but unfortunately not for the last one.
The idAgentsRegion column is not needed for the WHERE or JOIN conditions at all, by adding it you make the index a "covering" one though, so that all data needed can be retrieved from the index alone, without extra lookup steps needed for actual table rows.
What I would do in this case would be to have two indexes, one on (isDeleted, startDate, endDate, idAgentsRegion) and one on just (isDeleted, startDate, endDate, idAgentsRegion), and then split the query into two separate ones combined by UNION:
SELECT DISTINCT `cr`.`idCustomer`, `rbase`.`id`
FROM `customers` `t`
JOIN `customersregion` `cr` ON t.idCustomer = cr.idCustomer and cr.isDeleted = 0
JOIN `calendaritems` `rbase` ON rbase.idAgentsRegion = cr.idRegion and rbase.isDeleted = 0
where (
(rbase.startDate <= '2020-07-06 00:00:00' and rbase.endDate >= '2020-07-06 00:00:00') or
(rbase.startDate <= '2020-07-28 00:00:00' and rbase.endDate >= '2020-07-28 00:00:00') or
(rbase.startDate >= '2020-07-06 00:00:00' and rbase.startDate <= '2020-07-28 23:59:59')
)
UNION
SELECT DISTINCT `cr`.`idCustomer`, `rbase`.`id`
FROM `customers` `t`
JOIN `customersregion` `cr` ON t.idCustomer = cr.idCustomer and cr.isDeleted = 0
JOIN `calendaritems` `rbase` ON rbase.idAgentsRegion = cr.idRegion and rbase.isDeleted = 0
where (rbase.endDate >= '2020-07-06 00:00:00' and rbase.endDate <= '2020-07-28 23:59:59')
For the first part the first index is perfect, for the 2nd part the 2nd index is perfect, leading to much smaller index range scans, and in the end the results just need to be combined and duplicates removed.

Answer 2

First of all, there is a foreign key relation from customersreqion to customers, so you don't need the customers table in your query. You don't select anything from it and the foreign key relation already ensures that you won't select any customerid's that are not in the customers table. This doesn't reduce your 100 seconds significantly, but every bit helps.
To get full gain of indexes, you will need two extra indexes:
CREATE INDEX firstindextoadd ON calendaritems(idAgentsRegion, isDeleted, startDate, endDate);
CREATE INDEX secondindextoadd ON calendaritems(idAgentsRegion, isDeleted, endDate);
The first index will be used for your first 3 conditions:
(rbase.startDate <= '2020-07-06 00:00:00' and rbase.endDate >= '2020-07-06 00:00:00') or
(rbase.startDate <= '2020-07-28 00:00:00' and rbase.endDate >= '2020-07-28 00:00:00') or
(rbase.startDate >= '2020-07-06 00:00:00' and rbase.startDate <= '2020-07-28 23:59:59')
The second will be your for the forth condition:
(rbase.endDate >= '2020-07-06 00:00:00' and rbase.endDate <= '2020-07-28 23:59:59')
It depends on the number of deleted records if you should include the isDeleted, but I added them 'just in case'.
It didn't test it on a huge dataset, so you need to tell me if this worked for you.
In addtion you can simplify your conditions to:
SELECT DISTINCT `cr`.`idCustomer`, `rbase`.`id`
FROM `customersregion` `cr` ON t.idCustomer = cr.idCustomer and cr.isDeleted = 0
JOIN `calendaritems` `rbase` ON rbase.idAgentsRegion = cr.idRegion and rbase.isDeleted = 0
where
rbase.startDate <= '2020-07-06 00:00:00' and rbase.endDate >= '2020-07-28 00:00:00' OR
rbase.startDate BETWEEN '2020-07-06 00:00:00' and '2020-07-28 00:00:00' OR
rbase.endDate BETWEEN '2020-07-06 00:00:00' and '2020-07-28 00:00:00'

Answer 3

rbase: INDEX(isDeleted, startDate, endDate, idAgentsRegion, id)
rbase: INDEX(isDeleted, endDate, startDate, idAgentsRegion, id)
Those have these qualities:
First two columns are useful in ON and WHERE.
Optimizer will pick between them based on whether startDate or endDate is more selective.
Covering
That assumes that the Optimizer will start with rbase. If, instead, it starts with cr, then have both of these for the Optimizer to choose between:
rbase: INDEX(idAgentsRegion, isDeleted, startDate, endDate, id)
rbase: INDEX(idAgentsRegion, isDeleted, endDate, startDate, id)
cr is the only other table that the Optimizer might start with. (There is a WHERE clause to filter by.)
cr: INDEX(isDeleted, idRegion, -- first, (in either order)
idCustomer) -- last
Assuming that start <= end, the range test can probably be simplified to only this:
WHERE rbase.startDate < '2020-07-28'
AND rbase.endDate >= '2020-07-06'
(I don't recognize the funnybusiness with '2020-07-28' versus '2020-07-28 23:59:59'.)
I recommend using "< midnight" and ">= midnight" consistently. A plain date is equivalent to midnight for that morning. Another way to specify '2020-07-28' is '2020-07-06' + INTERVAL 22 DAY. The latter is convenient when you know the span (22 days) and don't want to fuss with leap days, etc.
It is "proper" for the ON to specify how the tables are 'related', and the WHERE to be used for filtering. That is, the isDeleted tests belong in the WHERE clause. (The execution is unaffected for JOIN, but important for LEFT JOIN.)
The Last...Id... and Last...Name columns seem to be redundant? Somewhere else there is a mapping from id to name?
Rates and Colors -- Those seem like things that are not worth normalizing? If you ever need to search on either, undoing this normalization will help performance, possibly a lot.
This combo seems 'wrong'; is there a reason for it:
startDate DATETIME
startTime TIME
When you have both of these,
INDEX(a) -- drop
INDEX(a,b) -- keep (it takes care of the other case)
LatitudePoint decimal(18, 12) takes 9 bytes; gross overkill. Suggested alternatives: http://mysql.rjweb.org/doc.php/latlng#representation_choices

Answer 4

Unless you can assume startDate is no more than e.g. 30 days before endDate, there is no practical way to index for this query to avoid having to check all rows. You can try a composite index on (startDate,endDate) and that may help some.
You can try doing a union with some of your where conditions using a start date index and some using an end date index, but if you really are expecting half a million of your 3.8 million rows to get selected, it may not help at all.