I have a log table with a date field called logTime. I need to show the number of rows within a date range and the number of records per day. The issue is that i still want to show days that do not have records.
Is it possible to do this only with SQL?
Example:
SELECT logTime, COUNT(*) FROM logs WHERE logTime >= '2011-02-01' AND logTime <= '2011-02-04' GROUP BY DATE(logTime);
It returns something like this:
+---------------------+----------+
| logTime | COUNT(*) |
+---------------------+----------+
| 2011-02-01 | 2 |
| 2011-02-02 | 1 |
| 2011-02-04 | 5 |
+---------------------+----------+
3 rows in set (0,00 sec)
I would like to show the day 2011-02-03 too.
MySQL will not invent rows for you, so if the data is not there, they will naturally not be shown.
You can create a calendar table, and join in that,
create table calendar (
day date primary key,
);
Fill this table with dates (easy with a stored procedure, or just some general scripting), up till around 2038 and something else will likely break unitl that becomes a problem.
Your query then becomes e.g.
SELECT logTime, COUNT(*)
FROM calendar cal left join logs l on cal.day = l.logTime
WHERE day >= '2011-02-01' AND day <= '2011-02-04' GROUP BY day;
Now, you could extend the calendar table with other columns that tells you the month,year, week etc. so you can easily produce statistics for other time units. (and purists might argue the calendar table would have an id integer primary key that the logs table references instead of a date)
In order to accomplish this, you need to have a table (or derived table) which contains the dates that you can then join from, using a LEFT JOIN.
SQL operates on the concept of mathematical sets, and if you don't have a set of data, there is nothing to SELECT.
If you want more details, please comment accordingly.
I'm not sure if this is a problem that should be solved by SQL. As others have shown, this requires maintaining a second table that contains the all of the individual dates of a given time span, which must be updated every time that time span grows (which presumably is "always" if that time span is the current time.
Instead, you should use to inspect the results of the query and inject dates as necessary. It's completely dynamic and requires no intermediate table. Since you specified no language, here's pseudo code:
EXECUTE QUERY `SELECT logTime, COUNT(*) FROM logs WHERE logTime >= '2011-02-01' AND logTime <= '2011-02-04' GROUP BY DATE(logTime);`
FOREACH row IN query result
WHILE (date in next row) - (date in this row) > 1 day THEN
CREATE new row with date = `date in this row + 1 day`, count = `0`
INSERT new row IN query result AFTER this row
ADVANCE LOOP INDEX TO new row (`this row` is now the `new row`)
END WHILE
END FOREACH
Or something like that
DECLARE #TOTALCount INT
DECLARE #FromDate DateTime = GetDate() - 5
DECLARE #ToDate DateTime = GetDate()
SET #FromDate = DATEADD(DAY,-1,#FromDate)
Select #TOTALCount= DATEDIFF(DD,#FromDate,#ToDate);
WITH d AS
(
SELECT top (#TOTALCount) AllDays = DATEADD(DAY, ROW_NUMBER()
OVER (ORDER BY object_id), REPLACE(#FromDate,'-',''))
FROM sys.all_objects
)
SELECT AllDays From d
Related
I have three columns User_ID, New_Status and DATETIME.
New_Status contains 0(inactive) and 1(active) for users.
Every user starts from active status - ie. 1.
Subsequently table stores their status and datetime at which they got activated/inactivated.
How to calculate number of active users at the end of each date, including dates when no records were generated into the table.
Sample data:
| ID | New_Status | DATETIME |
+----+------------+---------------------+
| 1 | 1 | 2019-01-01 21:00:00 |
| 1 | 0 | 2019-02-05 17:00:00 |
| 1 | 1 | 2019-03-06 18:00:00 |
| 2 | 1 | 2019-01-02 01:00:00 |
| 2 | 0 | 2019-02-03 13:00:00 |
Format the date time value to a date only string and group by it
SELECT DATE_FORMAT(DATETIME, '%Y-%m-%d') as day, COUNT(*) as active
FROM test
WHERE New_Status = 1
GROUP BY day
ORDER BY day
In MySQL 8 you can use the row_number() window function to get the last status of a user per day. Then filter for the one that indicate the user was active GROUP BY the day and count them.
SELECT date(x.datetime),
count(*)
FROM (SELECT date(t.datetime) datetime,
t.new_status,
row_number() OVER (PARTITION BY date(t.datetime)
ORDER BY t.datetime DESC) rn
FROM elbat t) x
WHERE x.rn = 1
AND x.new_status = 1
GROUP BY x.datetime;
If not all days are in the table you need to create a (possibly derived) table with all days and cross join it.
Find out the last activity status of users whose activity was changed for each day
select User_ID, New_Status, DATE_FORMAT(DATETIME, '%Y-%m-%d')
from activity_table
where not exists
(
select 1
from activity_table at
where at.User_ID = activity_table.User_ID and
DATE_FORMAT(at.DATETIME, '%Y-%m-%d') = DATE_FORMAT(activity_table.DATETIME, '%Y-%m-%d') and
at.DATETIME > activity_table.DATETIME
)
order by DATE_FORMAT(activity_table.DATETIME, '%Y-%m-%d');
This is not the solution yet, but a very very useful information before solution. Note that here not all dates are covered yet and the values are individual records, more precisely their last values on each day, ordered by the date.
Let's get aggregate numbers
Using the query above as a subselect and aliasing it into a table, you can group by DATETIME and do a select sum(new_Status) as activity, count(*) total, DATETIME so you will know that activity - (total - activity) is the difference in comparison to the previous day.
Knowing the delta for each day present in the result
At the previous section we have seen how the delta can be calculated. If the whole query in the previous section is aliased, then you can self join it using a left join, with pairs of (previous date, current date), still having the gaps of dates, but not worrying about that just yet. In the case of the first date, its activity is the delta. For subsequent records, adding the previous day's delta to their delta yields the result you need. To achieve this you can use a recursive query, supported by MySQL 8, or, alternatively, you can just have a subquery which sums the delta of previous days (with special attention to the first date, as described earlier) will and adding the current date's delta yields the result we need.
Fill the gaps
The previous section would already perfectly work (assuming the lack of integrity problems), assuming that there were activity changes for each day, but we will not continue with the assumption. Here we know that the figures are correct for each date where a figure is present and we will need to just add the missing dates into the result. If the results are properly ordered, as they should be, then one can use a cursor and loop the results. At each record after the first one, we can determine the dates that are missing. There might be 0 such dates between two consequent dates or more. What we do know about the gaps is that their values are exactly the same as the previous record, that do has data. If there were no activity changes on a given date, then the number of active users is exactly the same as in the previous day. Using some structure, like a table you can generate the results you have with the knowledge described here.
Solving possible integrity problems
There are several possibilities for such problems:
First, a data item might exist prior to the introduction of this table's records were started to be spawned.
Second, bugs or any other causes might have made a pause in creating records for this activity table.
Third, the addition of user is or was not necessarily generating an activity change, since its popping into existence renders its previous state of activity undefined and subject to human standards, which might change over time.
Fourth, the removal of user is or was not necessarily generating an activity change, since its popping out of existence renders is current state of activity undefined and subject to human standards, which might change over time.
Fifth, there is an infinity of other issues which might cause data integrity issues.
To cope with these you will need to comprehensively analyze whatever you can from the source-code and the history of the project, including database records, logs and humanly available information to detect such anomalies, the time they were effective and figure out what their solution is if they exist.
EDIT
In the meantime I was thinking about the possibility of a user, who was active at the start of the day being deactivated and then activated again by the end of the day. Similarly, an inactive user during a day might be activated and then finally deactivated by the end of the day. For users that have more than an activation at the start of the day, we need to compare their activity status at the start and the end of the day to find out what the difference was.
SELECT
DATE(DATETIME),
COUNT(*)
FROM your_table
WHERE New_Status = 1
GROUP BY User_ID,
DATE(DATETIME)
For MySQL
WITH RECURSIVE
cte AS (
SELECT MIN(DATE(DT)) dt
FROM src
UNION ALL
SELECT dt + INTERVAL 1 DAY
FROM cte
WHERE dt < ( SELECT MAX(DATE(DT)) dt
FROM src )
),
cte2 AS
(
SELECT users.id,
cte.dt,
SUM( CASE src.New_Status WHEN 1 THEN 1
WHEN 0 THEN -1
ELSE 0
END ) OVER ( PARTITION BY users.id
ORDER BY cte.dt ) status
FROM cte
CROSS JOIN ( SELECT DISTINCT id
FROM src ) users
LEFT JOIN src ON src.id = users.id
AND DATE(src.dt) = cte.dt
)
SELECT dt, SUM(status)
FROM cte2
GROUP BY dt;
fiddle
Do not forget to adjust max recursion depth.
Here is what I believe is a good solution for this problem of yours:
SELECT SUM(New_Status) "Number of active users"
, DATE_FORMAT(DATEC, '%Y-%m-%d') "Date"
FROM TEST T1
WHERE DATE_FORMAT(DATEC,'%H:%i:%s') =
(SELECT MAX(DATE_FORMAT(T2.DATEC,'%H:%i:%s'))
FROM TEST T2
WHERE T2.ID = T1.ID
AND DATE_FORMAT(T1.DATEC, '%Y-%m-%d') = DATE_FORMAT(T2.DATEC, '%Y-%m-%d')
GROUP BY ID
, DATE_FORMAT(DATEC, '%Y-%m-%d'))
GROUP BY DATE_FORMAT(DATEC, '%Y-%m-%d');
Here is the DEMO
I am using the Graph Reports for the select below. The MySQL database only has the active records in the database, so if no records are in the database from X hours till Y hours that select does not return anything. So in my case, I need that select return Paypal zero values as well even the no activity was in the database. And I do not understand how to use the UNION function or re-create select in order to get the zero values if nothing was recorded in the database in time interval. Could you please help?
select STR_TO_DATE ( DATE_FORMAT(`acctstarttime`,'%y-%m-%d %H'),'%y-%m-%d %H')
as '#date', count(*) as `Active Paid Accounts`
from radacct_history where `paymentmethod` = 'PayPal'
group by DATE_FORMAT(`#date`,'%y-%m-%d %H')
When I run the select the output is:
Current Output
But I need if there are no values between 2016-07-27 07:00:00 and 2016-07-28 11:00:00, then in every hour it should show zero active accounts Like that:
Needed output with no values every hour
I have created such select below , but it not put to every hour the zero value like i need. showing the big gap between the 12 Sep and 13 Sep anyway, but there should be the zero values every hour
(select STR_TO_DATE ( DATE_FORMAT(acctstarttime,'%y-%m-%d %H'),'%y-%m-%d %H')
as '#date', count(paymentmethod) as Active Paid Accounts
from radacct_history where paymentmethod <> 'PayPal'
group by DATE_FORMAT(#date,'%y-%m-%d %H'))
union ALL
(select STR_TO_DATE ( DATE_FORMAT(acctstarttime,'%y-%m-%d %H'),'%y-%m-%d %H')
as '#date', 0 as Active Paid Accounts
from radacct_history where paymentmethod <> 'PayPal'
group by DATE_FORMAT(#date,'%y-%m-%d %H')) ;
I guess, you want to return 0 if there is no matching rows in MySQL. Here is an example:
(SELECT Col1,Col2,Col3 FROM ExampleTable WHERE ID='1234')
UNION (SELECT 'Def Val' AS Col1,'none' AS Col2,'' AS Col3) LIMIT 1;
Updated the post: You are trying to retrieve data that aren't present in the table, I guess in reference to the output provided. So in this case, you have to maintain a date table to show the date that aren't in the table. Please refer to this and it's little bit tricky - SQL query that returns all dates not used in a table
You need an artificial table with all necessary time intervals. E.g. if you need daily data create a table and add all day dates e.g. start from 1970 till 2100.
Then you can use the table and LEFT JOIN your radacct_history. So for each desired interval you will have group item (group by should be based on the intervals table.
I have a MySQL table department_members that contains rows with a string field (member_name) and an int field (recent_actions) for every person in a single department. Recent_actions is currently NULL for all rows.
I have another, much larger table company_actions that contains a row for every time someone in the whole company has performed that type of action in the past year. Each row has a member_name, timestamp, and a unique action_id.
I want to update department_members.recent_actions with a count of how many times that member has performed that type of action within the past two weeks. If they haven't performed any actions recently, I want to update department_members.recent_actions with 0.
I've tried various CASE and IF approaches, but I can't get the syntax right.
In pseudocode, this is what I'm trying to do:
UPDATE department_members AS d,
(SELECT COUNT(action_id) AS recent, member_name
FROM company_actions
WHERE timestamp > DATE_SUB(NOW(), INTERVAL 14 DAY) AS tmp
/* then do something like this, only for real: */
IF d.member_name IN tmp(member_names) THEN d.recent_actions = tmp.recent
WHERE d.member_name = tmp.member_name
ELSE IF d.member_name NOT IN tmp(member_names) THEN d.recent_actions = 0
Hopefully that gets across what I'm going for? Any help would be appreciated! Been beating my head against this problem all day.
Join department_members with a subquery that calculates the total number of action_id in table company_actions using LEFT JOIN.
The COALESCE() returns the first non-null value in the params list.
UPDATE department_members a
LEFT JOIN
(
SELECT member_name, COUNT(*) TotalAction
FROM company_actions
WHERE timestamp > DATE_SUB(NOW(), INTERVAL 14 DAY)
GROUP BY member_name
) b ON a.member_name = b.member_name
SET a.action_id = COALESCE(b.TotalAction, 0)
I have a report that displays a graph. The X axis uses the date from the below query. Where the query returns no date, I am getting gaps and would prefer to return a value. Is there any way to force a date where there are no records?
SELECT
DATE(instime),
CASE
WHEN direction = 1 AND duration > 0 THEN 'Incoming'
WHEN direction = 2 THEN 'Outgoing'
WHEN direction = 1 AND duration = 0 THEN 'Missed'
END AS type,
COUNT(*)
FROM taxticketitem
GROUP BY
DATE(instime),
CASE
WHEN direction = 1 AND duration > 0 THEN 'Incoming'
WHEN direction = 2 THEN 'Outgoing'
WHEN direction = 1 AND duration = 0 THEN 'Missed'
END
ORDER BY DATE(instime)
One possible way is to create a table of dates and LEFT JOIN your table with them. The table could look something like this:
CREATE TABLE `datelist` (
`date` DATE NOT NULL,
PRIMARY KEY (`date`)
);
and filled with all dates between, say Jan-01-2000 through Dec-31-2050 (here is my Date Generator script).
Next, write your query like this:
SELECT datelist.date, COUNT(taxticketitem.id) AS c
FROM datelist
LEFT JOIN taxticketitem ON datelist.date = DATE(taxticketitem.instime)
WHERE datelist.date BETWEEN `2012-01-01` AND `2012-12-31`
GROUP BY datelist.date
ORDER BY datelist.date
LEFT JOIN and counting not null values from right table's ensures that the count is correct (0 if no row exists for a given date).
You would need to have a set of dates to LEFT JOIN your table to it. Unfortunately, MySQL lacks a way to generate it on the fly.
You would need to prepare a table with, say, 100000 consecutive integers from 0 to 99999 (or how long you think your maximum report range would be):
CREATE TABLE series (number INT NOT NULL PRIMARY KEY);
and use it like this:
SELECT DATE(instime) AS r_date, CASE ... END AS type, COUNT(instime)
FROM series s
LEFT JOIN
taxticketitems ti
ON ti.instime >= '2013-01-01' + INTERVAL number DAY
AND ti.instime < '2013-01-01' + INTERVAL number + 1 DAY
WHERE s.number <= DATEDIFF('2013-02-01', '2013-01-01')
GROUP BY
r_date, type
Had to do something similar before.
You need to have a subselect to generate a range of dates. All the dates you want. Easiest with a start date added to a number:-
SELECT DATE_ADD(SomeStartDate, INTERVAL (a.I + b.1 * 10) DAY)
FROM integers a, integers b
Given a table called integers with a single column called i with 10 rows containing 0 to 9 that SQL will give you a range of 100 days starting at SomeStartDate
You can then left join your actual data against that to get the full range.
I have a table that tracks emails sent. It is pretty simple.
ID | DATETIME | E-MAIL | SUBJECT | MESSAGE
I have been collecting data for several years. Some days I don't have any entries in the table.
query1:
SELECT COUNT(ID) FROM emails
WHERE DATE(datetime) >= 'XXXX-XX-XX'
AND DATE(datetime) is <= 'ZZZZ-ZZ-ZZ'
GROUP BY DATE(datetime)
I then use a some php to get one year prior for both XXXX and YYYY and run the second query which is the same as the first...
query2:
SELECT COUNT(ID) from emails
WHERE DATE(datetime) >= 'XXXX-XX-XX'
AND DATE(datetime) is <= 'ZZZZ-ZZ-ZZ'
GROUP BY DATE(datetime)
I am using a charting package to compare how many emails I got for a date range and then I overlay how many emails I got for the same range only one year prior. This is two queries right now and I chart the results.
The issue is where mysql does not have any emails for 2011 for a day in question, but has a few in 2012 for the same day.
Combining the results and graphing them skews the results since I am missing a date and a 0 value for last year for that day, effectively making all my values no longer match up.
2011-03-01 10 2012-03-01 4
2011-03-02 4 2012-03-02 2
2011-03-03 6 2012-03-04 1 <---- see where the two queries
end up diverging? (I had nothing
logged for 2012-03-03 so naturally
it was not in the results.
Is there a way I can get mysql to output the data I need including dates where value appear in one year but not another OR if no values appear in either year (still need date and 0) so my chart works?
I cannot seem to figure out how to do this...
Thanks!
There are a few different ways to get the results for a contiguous set of dates. My favourite one is to create the full set that is required using a dummy table or an existing contiguous set of ids from an AI PK. Something like this -
SELECT '2011-01-01' + INTERVAL (id -1) DAY
FROM dummy
WHERE id BETWEEN 1 AND 365
This will return a full set of days for 2011 which can then be LEFT JOINed to your emails table to get the counts -
SELECT `dates`.`date`, COUNT(emails.id)
FROM (
SELECT '2011-01-01' + INTERVAL (id - 1) DAY AS `date`, '2011-01-01 23:59:59' + INTERVAL (id - 1) DAY AS `end_of_day`
FROM dummy
WHERE id BETWEEN 1 AND 365
) `dates`
LEFT JOIN emails
ON `emails`.`datetime` BETWEEN `dates`.`date` AND `dates`.`end_of_day`
GROUP BY `dates`.`date`
To populate your dummy / seq table you can insert the first ten values manually and then use INSERT ... SELECT to add the rest -
CREATE TABLE dummy (id INTEGER NOT NULL PRIMARY KEY);
INSERT INTO dummy VALUES (0),(1),(2),(3),(4),(5),(6),(7),(8),(9),(10);
SET #tmp := (SELECT MAX(id) FROM dummy) + 1;
INSERT INTO dummy
SELECT #tmp + id
FROM dummy;
You need to execute the SET query before each run of the INSERT ... SELECT query.