How can optimize these instruction with MYSQL (subqueries for each row)? - mysql

Question

My problem is that we make a select, and then, for each row, we run 4 differents request SQL (is madness), as you can guess we make a lot of requests, and the system using this is very slow.
SELECT
deal_source.id,
deal_source.source_name,
deal_source.spider_status,
spider.last_success_date
FROM deal_source
JOIN spider
ON deal_source.id = spider.deal_source_id
Then for each row of this query we make:
$total_query = "SELECT count(id) as total
FROM spider_log
WHERE deal_source_id = '$deal_source_id'
AND date_format(date_created, '%Y-%m-%d') = '$lastdate' ";
$added_query = "SELECT count(id) as added
FROM spider_log
WHERE deal_source_id = '$deal_source_id'
AND action = 'added'
AND date_format(date_created, '%Y-%m-%d') = '$lastdate' ";
$extended_query = "SELECT count(id) as extended
FROM spider_log
WHERE deal_source_id = '$deal_source_id'
AND action = 'extended'
AND date_format(date_created, '%Y-%m-%d') = '$lastdate' ";
$duplicate_query = "SELECT count(id) as duplicate
FROM spider_log
WHERE deal_source_id = '$deal_source_id'
AND action = 'duplicate'
AND date_format(date_created, '%Y-%m-%d') = '$lastdate' ";

Answer 1

SELECT d.id,
d.source_name,
d.spider_status,
s.last_success_date,
COUNT(l.id) AS total,
SUM(l.id IS NOT NULL AND l.action='added' ) AS added,
SUM(l.id IS NOT NULL AND l.action='extended' ) AS extended,
SUM(l.id IS NOT NULL AND l.action='duplicate') AS duplicate
FROM deal_source d
JOIN spider s
ON s.deal_source_id = d.id
JOIN spider_log l
ON l.deal_source_id = d.id
ON l.date_created >= s.last_success_date
AND l.date_created < s.last_success_date + INTERVAL 1 DAY
GROUP BY d.id

Answer 2

Some points:
You can optimize performance of each query, using EXPLAIN and the careful adding of indexes.
You can combine all the queries to a big one, so you don't have to hit the database with a lot of queries.
Besides the lots of queries, The date_format(date_created, '%Y-%m-%d') = '$lastdate' is a performance killer because it apples a function (DATE_FORMAT()) to a column (date_created) so no index can be used and the function is called thosuand or million of times (as many rows are examined). Change such conditions - wherever they are in your code - to:
( date_created >= DATE('$lastdate')
AND date_created < DATE('$lastdate') + INTERVAL 1 DAY
)
or even better, if that $lastdate is a date, to:
( date_created >= '$lastdate'
AND date_created < '$lastdate' + INTERVAL 1 DAY
)
and even more better, if date_created is a DATE column, to:
date_created = '$lastdate'