Summary Join query taking too long - mysql

Question

I am trying to get the max record of each 'telephone_number' where process_status='0' and for that I am using the below query.
SELECT ID, CUSTID, telephone_number, TOTAL_USAGE, ACCOUNT_STATUS
FROM SPRINTABLE_DATA t
JOIN (SELECT MAX( id ) AS maxid FROM SPRINTABLE_DATA GROUP BY telephone_number)dt
ON t.id = dt.maxid WHERE process_status = '0'
AND RESET_FLAG = '0'
ORDER BY id DESC limit 0,700
The above query is giving me the desired result but the problem is this is too slow..
My table has about 20 million rows and this query is taking about 15-20 mins at times.
What can be done to improve this?
This is the structure:.
CREATE TABLE `SPRINTABLE_DATA` (
`ID` bigint(11) NOT NULL AUTO_INCREMENT,
`CUSTID` int(11) DEFAULT NULL,
`telephone_number` varchar(20) DEFAULT NULL,
`TOTAL_USAGE` int(11) DEFAULT NULL,
`PROCESS_STATUS` tinyint(4) DEFAULT '0',
`RESET_FLAG` tinyint(4) DEFAULT '0',
`RESET_REASON` varchar(10) DEFAULT NULL,
`PLAN_ID` varchar(20) DEFAULT NULL,
`ACCOUNT_STATUS` varchar(30) DEFAULT NULL,
PRIMARY KEY (`ID`),
KEY `telephone_number` (`telephone_number`),
KEY `CALL_CUST` (`CALL_START_TIME`,`CUSTID`),
KEY `telephone_number1` (`telephone_number `,`PROCESS_STATUS`,`SOC_ADDED`),
KEY `CURRENT_USAGE` (`CURRENT_USAGE`),
KEY `TOTAL_USAGE` (`TOTAL_USAGE`)
) ENGINE=InnoDB AUTO_INCREMENT=36392272 DEFAULT CHARSET=latin1

Answer 1

It seems that you are looking for the 700 most recently called numbers. (If that is not correct, please edit your question.
Your query follows a good practice for retrieving the latest log row for each item (telephone number in your case), as follows, in your subquery.
SELECT MAX( id ) AS id
FROM SPRINTABLE_DATA
GROUP BY telephone_number
To optimize the performance of this subquery, you need a compound index on two fields: (telephone_number, id), in that order. If you don't have that index, add it in. This is to allow a so-called loose index scan, an extraordinarily efficient way of satisfying a query.
Secondly, you're looking for (I presume) a small subset of your data. Presumably you have plenty more than 700 distinct telephone_number values. This means you're sorting a lot of data with ORDER BY only to discard it with LIMIT. So, let's do a deferred join, sorting a minimal number of columns, and then retrieving all the information you need.
Here's how to get the ID values of the 700 rows you need
SELECT q.ID /* get our 700 records */
FROM SPRINTABLE_DATA q
JOIN (
SELECT MAX( id ) AS id
FROM SPRINTABLE_DATA
GROUP BY telephone_number
) r ON q.id = r.id
WHERE q.process_status = '0'
AND q.RESET_FLAG = '0'
ORDER BY q.ID DESC
LIMIT 0,700
This pulls out 700 id numbers. You need to do some experimenting with indexes to find out what helps the most to optimize this. It's possible that an index on
process_status, RESET_FLAG, id
will help. It's also possible that changing the order of columns in the index will help, like this:
id, process_status, RESET_FLAG
Try them both.
Finally, we'll use this as a subquery to carry out the join (the so-called deferred join) to fetch the actual detail records. This technique gets rid of the need for sorting all that data.
SELECT t.ID, t.CUSTID, t.telephone_number, t.TOTAL_USAGE, t.ACCOUNT_STATUS
FROM SPRINTABLE_DATA t
JOIN (
SELECT q.ID /* get our 700 records */
FROM SPRINTABLE_DATA q
JOIN (
SELECT MAX( id ) AS id
FROM SPRINTABLE_DATA
GROUP BY telephone_number
) r ON q.id = r.id
WHERE q.process_status = '0'
AND q.RESET_FLAG = '0'
ORDER BY q.ID DESC
LIMIT 0,700
) s ON t.ID = s.ID
ORDER BY t.ID DESC
This will yield the same results, but will be faster.
Now, finally, if it's possible to select the latest calls from the 700 numbers that meet your criteria, you can simplify this query a lot. This will change your result set in a subtle way, though. In that case your call-selection subquery will look like this:
SELECT MAX( id ) AS id /* 700 matching numbers */
FROM SPRINTABLE_DATA
WHERE process_status = '0'
AND reset_flag = '0'
GROUP BY telephone_number
ORDER BY ID desc
LIMIT 0,700
With a compound covering index on
reset_flag, process_status, telephone_number, ID
this query will be quite fast. Your final query in this case would be
SELECT t.ID, t.CUSTID, t.telephone_number, t.TOTAL_USAGE, t.ACCOUNT_STATUS
FROM SPRINTABLE_DATA t
JOIN (
SELECT MAX( id ) AS id /* 700 matching numbers */
FROM SPRINTABLE_DATA
WHERE process_status = '0'
AND reset_flag = '0'
GROUP BY telephone_number
ORDER BY ID desc
LIMIT 0,700
) s ON t.ID = s.ID
ORDER BY t.ID DESC

Answer 2

Made a slight modification to your query
SELECT ID, CUSTID, telephone_number, TOTAL_USAGE, ACCOUNT_STATUS
FROM SPRINTABLE_DATA t
JOIN (SELECT telephone_number,MAX( id ) AS maxid FROM SPRINTABLE_DATA GROUP BY telephone_number)dt
ON t.id = dt.maxid WHERE process_status = '0'
AND RESET_FLAG = '0'
ORDER BY id DESC limit 0,700
Add these indexes if are not there already
ALTER TABLE SPRINTABLE_DATA ADD KEY (telephone_number,id)
ALTER TABLE SPRINTABLE_DATA ADD KEY (process_status,reset_flag,id)
Another option which is probably the fastest is to use a correlated subquery
SELECT ID, CUSTID, telephone_number, TOTAL_USAGE, ACCOUNT_STATUS
FROM SPRINTABLE_DATA t WHERE EXISTS
(SELECT MAX( id ) FROM SPRINTABLE_DATA tt WHERE t.id=tt.id AND tt.process_status = '0'
AND tt.RESET_FLAG = '0' )
ORDER BY id DESC limit 0,700
For this you need
ALTER TABLE SPRINTABLE_DATA ADD KEY (id,process_status,reset_flag)