I have an SQL request that take 100% of my VM CPU while it's working. I wanna know how to optimize it :
SELECT g.name AS hostgroup
, h.name AS hostname
, a.host_id
, s.display_name AS servicename
, a.service_id
, a.entry_time AS ack_time
, ( SELECT ctime
FROM logs
WHERE logs.host_id = a.host_id
AND logs.service_id = a.service_id
AND logs.ctime < a.entry_time
AND logs.status IN (1, 2, 3)
AND logs.type = 1
ORDER BY logs.log_id DESC
LIMIT 1) AS start_time
, ar.acl_res_name AS timeperiod
, a.state AS state
, a.author
, a.acknowledgement_id AS ack_id
FROM centstorage.acknowledgements a
LEFT JOIN centstorage.hosts h ON a.host_id = h.host_id
LEFT JOIN centstorage.services s ON a.service_id = s.service_id
LEFT JOIN centstorage.hosts_hostgroups p ON a.host_id = p.host_id
LEFT JOIN centstorage.hostgroups g ON g.hostgroup_id = p.hostgroup_id
LEFT JOIN centreon.hostgroup_relation hg ON a.host_id = hg.host_host_id
LEFT JOIN centreon.acl_resources_hg_relations hh ON hg.hostgroup_hg_id = hh.hg_hg_id
LEFT JOIN centreon.acl_resources ar ON hh.acl_res_id = ar.acl_res_id
WHERE ar.acl_res_name != 'All Resources'
AND YEAR(FROM_UNIXTIME( a.entry_time )) = YEAR(CURDATE())
AND MONTH(FROM_UNIXTIME( a.entry_time )) = MONTH(CURDATE())
AND a.service_id is not null
ORDER BY a.acknowledgement_id ASC
The problem is at this part :
(SELECT ctime FROM logs
WHERE logs.host_id = a.host_id
AND logs.service_id = a.service_id
AND logs.ctime < a.entry_time
AND logs.status IN (1, 2, 3)
AND logs.type = 1
ORDER BY logs.log_id DESC
LIMIT 1) AS start_time
The table logs is really huge and some friends told me to use a buffer table/database but i pretty knew to this things and i don't know how to do it.
There is an EXPLAIN EXTENDED of the query :
It seems that he will examined only 2 row of the table logs so why it takes so much time ? (There is 560000 row in the table logs).
Here is all indexes of those tables :
centstorage.acknowledgements :
centstorage.hosts :
centstorage.services :
centstorage.hosts_hostgroups :
centstorage.hostgroups :
centreon.hostgroup_relation :
centreon.acl_resources_hg_relations :
centreon.acl_resources :
For SQL Server there is the possibility to define the maximum degree of parallelism of your query using MAXDOP
For example you can define at the end of your query
option (maxdop 2)
I'm pretty sure there's an equivalent in MySql.
You can try to approach this situation if the execution time is not relevant.
Create a Temporary Table from where condition for acknowledgements, schema will have column required in final result and used in JOIN with all your 7 tables
CREATE TEMPORARY TABLE __tempacknowledgements AS SELECT g.name AS hostgroup
, '' AS hostname
, a.host_id
, s.display_name AS servicename
, a.service_id
, a.entry_time AS ack_time
, '' AS AS start_time
, '' AS timeperiod
, a.state AS state
, a.author
, a.acknowledgement_id AS ack_id
FROM centstorage.acknowledgements a
WHERE YEAR(FROM_UNIXTIME( a.entry_time )) = YEAR(CURDATE())
AND MONTH(FROM_UNIXTIME( a.entry_time )) = MONTH(CURDATE())
AND a.service_id IS NOT NULL
ORDER BY a.acknowledgement_id ASC;
Or create using proper column definition
Update fields from all tables having left join, you can use Inner Join in update. You should write 7 different update statements. 2 examples are given below.
UPDATE __tempacknowledgements a JOIN centstorage.hosts h USING(host_id)
SET a.name=h.name;
UPDATE __tempacknowledgements s JOIN centstorage.services h USING(service_id)
SET a.acl_res_name=s.acl_res_name;
similar way update ctime from logs using Join with Logs, this is 8th update statement.
pick select from temp table.
drop temp table
a sp can be written for this.
Turn LEFT JOIN into JOIN unless you have a real need for LEFT.
AND YEAR(FROM_UNIXTIME( a.entry_time )) = YEAR(CURDATE())
AND MONTH(FROM_UNIXTIME( a.entry_time )) = MONTH(CURDATE())
AND a.service_id is not null
Do you have any rows with a.service_id is not null? If not, get rid of it.
As already mentioned, that date comparison does not optimize. Here is what to use instead:
AND a.entry_time >= CONCAT(LEFT(CURDATE(), 7), '-01')
AND a.entry_time < CONCAT(LEFT(CURDATE(), 7), '-01') + INTERVAL 1 MONTH
And add one of these (depending on my above comment):
INDEX(entry_time)
INDEX(service_id, entry_time)
The correlated subquery is hard to optimize. This index (on logs) may help:
INDEX(type, host_id, service_id, status)
WHERE IN is time killer!
Instead of
logs.status IN (1, 2, 3)
use
logs.status=1 or logs.status=2 or logs.status=3
I have SLIGHTLY reformatted the query for my readability reference and better seeing the relations between the tables... otherwise ignore that part.
SELECT
g.name AS hostgroup,
h.name AS hostname,
a.host_id,
s.display_name AS servicename,
a.service_id,
a.entry_time AS ack_time,
( SELECT
ctime
FROM
logs
WHERE
logs.host_id = a.host_id
AND logs.service_id = a.service_id
AND logs.ctime < a.entry_time
AND logs.status IN (1, 2, 3)
AND logs.type = 1
ORDER BY
logs.log_id DESC
LIMIT 1) AS start_time,
ar.acl_res_name AS timeperiod,
a.state AS state,
a.author,
a.acknowledgement_id AS ack_id
FROM
centstorage.acknowledgements a
LEFT JOIN centstorage.hosts h
ON a.host_id = h.host_id
LEFT JOIN centstorage.services s
ON a.service_id = s.service_id
LEFT JOIN centstorage.hosts_hostgroups p
ON a.host_id = p.host_id
LEFT JOIN centstorage.hostgroups g
ON p.hostgroup_id = g.hostgroup_id
LEFT JOIN centreon.hostgroup_relation hg
ON a.host_id = hg.host_host_id
LEFT JOIN centreon.acl_resources_hg_relations hh
ON hg.hostgroup_hg_id = hh.hg_hg_id
LEFT JOIN centreon.acl_resources ar
ON hh.acl_res_id = ar.acl_res_id
WHERE
ar.acl_res_name != 'All Resources'
AND YEAR(FROM_UNIXTIME( a.entry_time )) = YEAR(CURDATE())
AND MONTH(FROM_UNIXTIME( a.entry_time )) = MONTH(CURDATE())
AND a.service_id is not null
ORDER BY
a.acknowledgement_id ASC
I would first recommend starting with your "acknowledgements" table and have an index at a minimum of ( entry_time, acknowledgement_id ). Next, update your WHERE clause. Because you are running a function to convert the unix timestamp to a date and grabbing the YEAR (and month) respectively, I don't believe it is utilizing the index as it has to compute that for every row. To eleviate that, a unix timestamp is nothing but a number representing seconds from a specifc point in time. If you are looking for a specific month, then pre-compute the starting and ending unix times and run for that range. Something like...
and a.entry_time >= UNIX_TIMESTAMP( '2015-10-01' )
and a.entry_time < UNIX_TIMESTAMP( '2015-11-01' )
This way, it accounts for all seconds within the month up to 11:59:59 on Oct 31, just before November 1st.
Then, without my glasses to see all the images more clearly, and short time this morning, I would ensure you have at least the following indexes on each table respectively
table index
logs ( host_id, service_id, type, status, ctime, log_id )
acknowledgements ( entry_time, acknowledgement_id, host_id, service_id )
hosts ( host_id, name )
services ( service_id, display_name )
hosts_hostgroups ( host_id, hostgroup_id )
hostgroups ( hostgroup_id, name )
hostgroup_relation ( host_host_id, hostgroup_hg_id )
acl_resources_hg_relations ( hh_hg_id, acl_res_id )
acl_resources ar ( acl_res_id, acl_res_name )
Finally, your correlated sub-query field is going to be a killer as it is processed for every row, but hopefully the other index optimization ideas will help performance.
Related
The SQL below is inside a MySQL stored procedure.
The procedure run by a cron job every day once at midnight to populate report table with result.
this procedure take around 2 min to run.
please note that table1 has millions of records.
i put this to run at midnight because there are INSERT/UPDATE transactions during the day but unfortunately there are some few transaction at night also.
when this procedure runs and if there are other transactions running then a deadlock error on table1 occurs.
my question is
why SELECT statement cause deadlock on table1?
how can I avoid deadlock in this kind of situation?
DROP report;
CREATE TABLE IF NOT EXISTS report AS (
SELECT
DISTINCT
companies.id company_id,
(
SELECT
SUM(`message_count`) single_phone
FROM
`table1`
WHERE
`table1`.`company_id` = companies.id
AND
`status` != 'error'
) AS single_phone,
(
SELECT
SUM(`message_count`)
FROM
`table1`
WHERE
`table1`.`company_id` = companies.id
AND
`status` != 'not error'
) AS log,
(
SELECT
SUM(`message_count`)
FROM
`table1`
WHERE
`table1`.`company_id` = companies.id
AND
`status` != 'error'
) AS log_monthly,
(
SELECT
SUM(`number_of_sms`) AS aggregate
FROM
`messages`
WHERE
`messages`.`company_id` = companies.id
) AS p_monthly
FROM
companies
INNER JOIN company_users ON companies.id = company_users.company_id
WHERE
company_users.confirmed = 1
AND
company_users.deleted_at IS NULL
);
thanks you very much for help but i have found the problem. yes this procedure cause the deadlock on table but the actual cause of the issue is that i have put ->everyMinute() in my laravel Kernal for schedule run. and there is also a cron job configured by another developer for the same that run every minute. these will run schedule every minute and that is the real cause of the deadlock problem. i have change my Kernal schedule to ->dailyAt('02:00'); now the problem is solved.
Your field-level queries should be done ONCE in the from clause to get pre-aggregates done ONCE per company ID and left-joined in case a given company may NOT have qualified records in a given category. Additionally, your query to get Single_Phone is the same as your 'log_monthly', but have no criteria showing a
break or filter on the dates of activity to filter out a single month vs overall total of everything. So, I added a where clause for filtering, but only GUESSING if such some date exists.
This query might substantially improve your performance. By moving the COLUMN-based queries for every company ID into its own subquery via left-join, those will be summed() and grouped by company ONCE, then the JOIN for the final result. COALESCE() is used so if no such counts exists, the value returned will be 0 instead of null
DROP report;
CREATE TABLE IF NOT EXISTS report AS (
SELECT
c.id company_id,
coalesce( PhoneSum.Msgs, 0 ) as Single_Phone,
coalesce( PhoneLog.Msgs, 0 ) as Log,
coalesce( MonthLog.Msgs, 0 ) as Log_Monthly,
coalesce( SMSSummary.Aggregate, 0 ) as p_monthly
from
-- this will declare an in-line variable if you do need to filter by a month as a couple of your
-- column result names infer, but have no other indicator of filtering by a given month.
( select #yesterday := date_sub( date(curdate()), interval -1 day ),
#beginOfThatMonth := date_sub( #yesterday, interval dayOfMonth( #yesterday ) -1 day ) sqlvars,
companies c
INNER JOIN company_users cu
ON m.company.id = cu.company_id
AND cu.confirmed = 1
AND cu.deleted_at IS NULL
LEFT JOIN
( SELECT
t.company_id,
SUM( t.message_count ) Msgs
FROM
table1 t
INNER JOIN company_users cu
ON t.company.id = cu.company_id
AND cu.confirmed = 1
AND cu.deleted_at IS NULL
where
t.status != 'error'
GROUP BY
t.company_id ) AS PhoneSum,
on c.id = PhoneSum.company_id
LEFT JOIN
( SELECT
t.company_id,
SUM( t.message_count ) Msgs
FROM
table1 t
INNER JOIN company_users cu
ON t.company.id = cu.company_id
AND cu.confirmed = 1
AND cu.deleted_at IS NULL
where
t.status != 'not error'
GROUP BY
t.company_id ) AS PhoneLog,
on c.id = PhoneLog.company_id
LEFT JOIN
( SELECT
t.company_id,
SUM( t.message_count ) Msgs
FROM
table1 t
INNER JOIN company_users cu
ON t.company.id = cu.company_id
AND cu.confirmed = 1
AND cu.deleted_at IS NULL
where
t.status != 'error'
-- this would only get counts of activity for current month currently active
-- but since you are running at night, you need the day before current
AND t.SomeDateFieldOnTable1 >= #beginOfThatMonth
GROUP BY
t.company_id ) AS MonthLogMsgs,
on c.id = MonthLogMsgs.company_id
LEFT JOIN
( SELECT
m.company_id,
SUM( m.number_of_sms ) aggregate
FROM
messages m
INNER JOIN company_users cu
ON m.company.id = cu.company_id
AND cu.confirmed = 1
AND cu.deleted_at IS NULL
where
m.SomeDateFieldOnMessagesTable >= #beginOfThatMonth
GROUP BY
company_id ) AS SMSSummary,
on c.id = SMSSummary.company_id
I am trying to retrieve some data from MySql database using Excel Vba. Everything is working fine...but the MySql query is taking too much time to execute.
Here is my code:
SELECT
d.DATE,
c.name,
c.address,
c.state_name,
c.contact_no,
d.AMOUNT,
d.BY_NAME,
d.NARATION,
t.REMARK
FROM
database1.data d
JOIN
(
SELECT DISTINCT
cust_id,
OR_NO
FROM
database1.ordbill
) o ON SUBSTRING_INDEX(database1.d.NARATION,
':',
-1) = o.OR_NO
JOIN
database1.contact c ON o.cust_id = c.id
JOIN
database1.total t ON t.VCH_NO = d.VCH_NO
WHERE
d.PARTY_NAME = 'advance' AND(
d.`BY_NAME` = 'Bank1' OR d.`BY_NAME` = 'CASH' OR d.`BY_NAME` = 'Bank2'
) AND d.DATE BETWEEN '2019-09-01' AND '2019-09-30'
ORDER BY
d.DATE ASC `
Assuming you have already index on table contact pk id an index
SELECT *
FROM `loans`
WHERE `date` >= '2019-11-25'
AND `date`<='2019-11-28'
AND `designation` LIKE '%sdf%'
why does this happen ?
SELECT d.DATE
,c.name
,c.address
,c.state_name
,c.contact_no
, d.AMOUNT
, d.BY_NAME
, d.NARATION
,t.REMARK
FROM database1.data d JOIN (
SELECT DISTINCT cust_id, OR_NO FROM database1.ordbill
) o ON SUBSTRING_INDEX(database1.d.NARATION,':',-1)=o.OR_NO
JOIN database1.contact c on o.cust_id=c.id
JOIN database1.total t on t.VCH_NO=d.VCH_NO
WHERE d.PARTY_NAME = 'advance'
AND (d.`BY_NAME` = 'Bank1' OR d.`BY_NAME` = 'CASH' OR d.`BY_NAME` = 'Bank2')
AND d.DATE BETWEEN '2019-09-01' AND '2019-09-30'
ORDER BY d.DATE ASC
be sure you have also proper composite index on
table data columns(PARTY_NAME, BY_NAME, DATE, VCH_NO )
and a index also
table total column (VCH_NO)
I have two requests
UPDATE :
I need to do something like that :
SELECT poste_nom, ups_type_contrat,
(SELECT `entpro_date`
FROM ENT_PRO
WHERE entpro_user_id = 2
ORDER BY `entpro_id` DESC
LIMIT 1) ,
serv_nom,
serv_id_resp,
user_credit_cpf,
user_indice_salarial,
FLOOR( DATEDIFF( CURDATE( ) , user_dateentree ) /365 ) AS dateEntree
FROM USER
INNER JOIN USER_POSTE_SERVICE
ON USER.user_id= USER_POSTE_SERVICE.ups_poste_id
INNER JOIN POSTE
ON USER_POSTE_SERVICE. ups_poste_id = POSTE.poste_id
INNER JOIN SERVICE
ON USER_POSTE_SERVICE.ups_id_serv = SERVICE.serv_id
WHERE user_id = 2
ORDER BY user_nom ASC
Is it possible to gather two requests in only one ?
From what I understood you want to simple merge the result of your sub-query to your main SELECT, if so you could try it this way:
SELECT poste_nom,
ups_type_contrat,
ENT_PRO_RESULT.entpro_date,
serv_nom,
serv_id_resp,
user_credit_cpf,
user_indice_salarial,
FLOOR( DATEDIFF( CURDATE( ) , user_dateentree ) /365 ) AS dateEntree
FROM USER
LEFT JOIN (SELECT entpro_date,
entpro_user_id
FROM ENT_PRO
ORDER BY entpro_id DESC
LIMIT 1) ENT_PRO_RESULT
ON USER.user_id = ENT_PRO_RESULT.entpro_user_id
INNER JOIN USER_POSTE_SERVICE
ON USER.user_id = USER_POSTE_SERVICE.ups_poste_id
INNER JOIN POSTE
ON USER_POSTE_SERVICE.ups_poste_id = POSTE.poste_id
INNER JOIN SERVICE
ON USER_POSTE_SERVICE.ups_id_serv = SERVICE.serv_id
WHERE user_id = 2
ORDER BY user_nom ASC
I've joined it on:
ON USER.user_id = ENT_PRO_RESULT.entpro_user_id
So you only need to specify the:
WHERE user_id = 2
And the sub-query will use the current row user id for the LEFT JOIN.
I am trying to bring back a string based on an IF statement but it is extremely slow.
It has something to do with the first subquery but I am unsure of how to rearrange this as to bring back the same results but faster.
Here is my SQL:
SELECT IF
(
(
SELECT COUNT(*)
FROM
(
SELECT DISTINCT enquiryId, type
FROM parts_enquiries, parts_service_types AS pst
WHERE parts_enquiries.serviceTypeId = pst.id
) AS parts
WHERE parts.enquiryId = enquiries.id
) > 1, 'Mixed',
(
SELECT DISTINCT type
FROM parts_enquiries, parts_service_types AS pst
WHERE parts_enquiries.serviceTypeId = pst.id AND enquiryId = enquiries.id
)
) AS partTypes
FROM enquiries,
entities
WHERE enquiries.entityId = entities.id
How can I make it faster?
I have modified my original query below, but I am getting the error that subquery returns more than one row:
SELECT
(SELECT
CASE WHEN COUNT(DISTINCT type) > 1 THEN 'Mixed' ELSE `type` END AS type
FROM parts_enquiries
INNER JOIN parts_service_types AS pst ON parts_enquiries.serviceTypeId = pst.id
INNER JOIN enquiries ON parts_enquiries.enquiryId = enquiries.id
INNER JOIN entities ON enquiries.entityId = entities.id
GROUP BY enquiryId) AS partTypes
FROM enquiries,
entities
WHERE enquiries.entityId = entities.id
Please have a look if this query yields the same results:
SELECT
enquiryId,
CASE WHEN COUNT(DISTINCT type) > 1 THEN 'Mixed' ELSE `type` END AS type
FROM parts_enquiries
INNER JOIN parts_service_types AS pst ON parts_enquiries.serviceTypeId = pst.id
INNER JOIN enquiries ON parts_enquiries.enquiryId = enquiries.id
INNER JOIN entities ON enquiries.entityId = entities.id
GROUP BY enquiryId
But N.B.'s comment is still valid. To see if and index is used and other information we need to see the EXPLAIN and the table definitions.
This should get you what you want.
I would first pre-query your parts enquiries and parts service types looking for both the count and MINIMUM of the part 'type', grouped by the enquiry ID.
then, run your IF() against that result. If the distinct count is > 0, then 'Mixed'. If only one, since I did the MIN(), it would only have the description of that one value that you desire anyhow.
SELECT
E.ID
IF ( PreQuery.DistTypes > 1, 'Mixed', PreQuery.FirstType ) as PartType
from
Enquiries E
JOIN ( SELECT
PE.EnquiryID,
COUNT( DISTINCT PE.ServiceTypeID ) as DistTypes,
MIN( PST.Type ) as FirstType
from
Parts_Enquiries PE
JOIN Parts_Service_Types PST
ON PE.ServiceTypeID = PST.ID
group by
PE.EnquiryID ) as PreQuery
ON E.ID = PreQuery.EnquiryID
I have a SQL statement in which I do this
... group by date having date between '2010-07-01' and '2010-07-10';
The result looks like:
sum(test) day
--------------------
20 2010-07-03
120 2010-07-07
33 2010-07-09
42 2010-07-10
So I have these results, but is it possible, that I can write a statement that returns me for every day in the "between" condition a result row in this kind:
sum(test) day
--------------------
0 2010-07-01
0 2010-07-02
20 2010-07-03
0 2010-07-04
0 2010-07-05
0 2010-07-06
120 2010-07-07
... ...
42 2010-07-10
Otherwise, if this is not possible, I have to do it in my program logic.
Thanks a lot in advance & Best Regards.
Update: Perhaps it will be better if I will show you the full SQL statement:
select COALESCE(sum(DUR), 0) AS "r", 0 AS "opt", DATE_FORMAT(date, '%d.%m.%Y') AS "day" from (
select a.id as ID, a.dur as DUR, DATE(FROM_UNIXTIME(REVTSTMP / 1000)) as date,
a_au.re as RE, a_au.stat as STAT from b_c
join c on b_c.c_id = c.id
join a on c.id = a.c_id
join a_au on a.id = a_au.id
join revi on a_au.re = revi.re
join (
select a.id as ID, DATE(FROM_UNIXTIME(REVTSTMP / 1000)) as date,
max(a_au.re) as MAX_RE from b_c
join c on b_c.c_id = c.id
join a on c.id = a.c_id
join a_au on a.id = a_au.id
join revi on a_au.re = revi.re
where b_c.b_id = 30 group by ID, date) x on
x.id = a.id and x.date = date and x.MAX_RE = a_au.rev
where a_au.stat != 7
group by ID, x.date)
AS SubSelTable where date between '2010-07-01' and '2010-07-15' group by date;
Update:
My new SQL statement (-> Dave Rix):
select coalesce(`theData`.`real`, 0) as 'real', 0 as 'opt', DATE_FORMAT(`DT`.`ddDate`, '%d.%m.%Y') as 'date'
from `dimdates` as DT
left join (
select coalesce(sum(DUR), 0) AS 'real', 0 AS 'opt', date
from (
select a.id as ID, a.dur as DUR, DATE(FROM_UNIXTIME(REVTSTMP / 1000)) as date, a_au.RE as RE, a_au.stat as STAT
from b_c
join c on b_c.c_id = c.id
join a on c.id = a.c_id
join a_au on a.id = a_au.id
join revi on a_au.RE = revi.RE
join (
select a.id as ID, DATE(FROM_UNIXTIME(REVTSTMP / 1000)) as date, max(a_au.RE) as MAX_RE
from b_c
join c on b_c.c_id = c.id
join a on c.id = a.c_id
join a_au on a.id = a_au.id
join revi on a_au.RE = revi.RE
where b_c.b_id = 30 GROUP BY ID, date
) x
on x.id = a.id and x.date = date and x.MAX_RE = a_au.RE
where a_au.stat != 20
group by ID, x.date
) AS SubTable
where date between '2010-07-01' and '2010-07-10' group by date) AS theData
ON `DT`.`ddDate` = `theData`.`date` where `DT`.`ddDate` between '2010-07-01' and '2010-07-15';
Put the Between Logic in a Where Clause
Select Sum(day), day
From Table
Where day Between date1 and date2
Group By day
EDIT:
Having should only be used to filter data in the aggregates... i.e.
Having Sum(day) > 10
Check out my answer to the following question;
Select all months within given date span, including the ones with 0 values
This may be just what you are looking for :)
You can modify your query above as follows (you could integrate this, but this way is simpler!);
SELECT COALESCE(`theData`.`opt`, 0), `DT`.`myDate`
FROM `dateTable` AS DT
LEFT JOIN (
... INSERT YOUR QUERY HERE ...
) AS theData
ON `DT`.`myDate` = `theData`.`date`
and you will also need to change the DATE_FORMAT(date, '%d.%m.%Y') AS "day" in your query to just date
E.g.
select COALESCE(sum(DUR), 0) AS "r", 0 AS "opt", `date` from
As for #OMG Ponies answer, you will need to pre-populate the dateTable with plenty of rows of data!
Does anyone know how I can post my SQL dump of this table as a file which can be attached? It's quite big, but can be useful...
Assuming that your date column is a DATETIME column, you need to use something to change time values to be the same for proper grouping to happen. IE:
SELECT SUM(t.test),
DATE_FORMAT(t.date, '%Y-%m-%d') AS day
FROM TABLE t
WHERE t.date BETWEEN #start AND #end
GROUP BY DATE_FORMAT(t.date, '%Y-%m-%d')
But if there's no record for a given date, the date will not appear in the result set. In other words, no dates with zero will appear in your output.
To solve that, you need to LEFT JOIN to a table of dates, which MySQL doesn't have the ability to generate. It can't even generate a list of numbers, so you have to create a table with a single column:
DROP TABLE IF EXISTS `example`.`numbers`;
CREATE TABLE `example`.`numbers` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
...and populate it:
INSERT INTO numbers (id) VALUES (NULL)
...before you can use the number value to generate a list of dates using the DATE_ADD function:
SELECT COALESCE(SUM(t.test), 0),
x.the_date AS day
FROM (SELECT DATE_FORMAT(DATE_ADD(NOW(), INTERVAL n.id-1 DAY), '%Y-%m-%d') AS the_date
FROM NUMBERS n) x
LEFT JOIN your_table yt ON DATE_FORMAT(yt.date, '%Y-%m-%d') = x.the_date
WHERE x.the_date BETWEEN #start AND #end
GROUP BY x.the_date