How to retrieve 500k records DB data faster? - mysql

Question

I have two tables T1 1 000 records and T2 with 500 000 records. I have a query where I run a join between them and fetch data by performing some aggregations. My page seems to be loading slow. Are there any approaches to make this query faster?
I have created indexes on columns for which aggregations are being performed. I think it is a generic statement.
$query = Mymodel::selectRaw("supplier_data.name as distributor,supplier_data.name as name, supplier_data.group_id as group_id, supplier_data.pay,supplier_data.group_id as submitted_group_plan,supplier_data.group_id as group_id_string,
(SELECT sum(t.net_claim) AS trans_number
FROM transactions_data_new as t
JOIN `supplier_data` AS d ON `t`.`member_id` = `d`.`group_id`
WHERE
(
(
t.`submit_date`>= '$date_from' and t.`submit_date`<= '$date_to'
AND t.`member_id` = supplier_data.group_id
)
OR
(
(t.claim_status IS NULL)
AND
(t.submit_date is NULL)
)
)
AND d.id = supplier_data.id
) as trans_number,
(SELECT sum(t.claim) AS trans_number
FROM transactions_data_new as t
JOIN `supplier_data` AS d ON `t`.`member_id` = `d`.`group_id`
WHERE
(
(
t.`submit_date`>= '$date_from' and t.`submit_date`<= '$date_to'
AND t.`member_id` = supplier_data.group_id
)
OR
(
(t.claim_status IS NULL)
AND
(t.submit_date is NULL)
)
)
AND d.id = supplier_data.id
) as claim,
(SELECT sum(t.reversed) AS trans_number
FROM transactions_data_new as t
JOIN `supplier_data` AS d ON `t`.`member_id` = `d`.`group_id`
WHERE
(
(
t.`submit_date`>= '$date_from' and t.`submit_date`<= '$date_to'
AND t.`member_id` = supplier_data.group_id
)
OR
(
(t.claim_status IS NULL)
AND
(t.submit_date is NULL)
)
)
AND d.id = supplier_data.id
) as reversed,
(SELECT sum(t.reversal) AS trans_number
FROM transactions_data_new as t
JOIN `supplier_data` AS d ON `t`.`member_id` = `d`.`group_id`
WHERE
(
(
t.`submit_date`>= '$date_from' and t.`submit_date`<= '$date_to'
AND t.`member_id` = supplier_data.group_id
)
OR
(
(t.claim_status IS NULL)
AND
(t.submit_date is NULL)
)
)
AND d.id = supplier_data.id
) as reversal
");

Answer 1

I don't see the need of this too complex/repeated with same clauses and multiple sub selects for same table which can done using a single left join
SELECT
s.name AS distributor,
s.name AS name,
s.group_id AS group_id,
s.pay,
s.group_id AS submitted_group_plan,
s.group_id AS group_id_string,
SUM(t.net_claim) AS trans_number,
SUM(t.claim) AS claim,
SUM(t.reversed) reversed,
SUM(t.reversal) reversal
FROM
supplier_data s
LEFT JOIN transactions_data_new t
ON `t`.`member_id` = s.`group_id`
AND (
(
t.`submit_date` >= '$date_from'
AND t.`submit_date` <= '$date_to'
)
OR (
t.claim_status IS NULL
AND t.submit_date IS NULL
)
)
GROUP BY s.name,
s.group_id,
s.pay

Answer 2

As I understand it the chunk() method is for use when you need to work with a large dataset and take an action on that data chunk by chunk.
From your question, it sounds like you're performing a query then returning the data as JSON so to me, it doesn't sound like you're taking an action on your dataset that requires chunking.
If you want to break up the returned JSON data you should be instead looking at pagination.
You could apply pagination to your query like so:
$data = Inspector::latest('id')
->select('id', 'firstname', 'status', 'state', 'phone')
->where('firstname', 'LIKE', '%' . $searchtext . '%')
->paginate();
You can specify the size of each set by passing a number to the paginate method:
$data = Inspector::latest('id')
->select('id', 'firstname', 'status', 'state', 'phone')
->where('firstname', 'LIKE', '%' . $searchtext . '%')
->paginate(25);
If I've misunderstood and you did actually want to do the chunking, I believe you could do the following:
$data = Inspector::latest('id')
->select('id', 'firstname', 'status', 'state', 'phone')
->where('firstname', 'LIKE', '%' . $searchtext . '%')
->chunk(50, function($inspectors) {
foreach ($inspectors as $inspector) {
// apply some action to the chunked results here
}
});
Also, if you're returning an eloquent object it will be automatically cast to json so you don't need to perform json_encode() as far as I'm aware.