Related
I have 3 columns, employee_id, start_time and end_time I want to make bucks of 1 hour to show me how many employees were working in each hour. For example, employee A worked from 12 pm to 3 pm and employee B worked from 2 pm to 4 pm so, at 12 pm (1 employee was working) 1 pm (1 employee) 2 pm (2 employees were working) 3 pm (2 employees) and 4 pm (1 employee), how can I make this in SQL? Let me show you a picture of the start and end time columns.
Sample input would be:
Expected outcome would be something like
I want to create a bucket in order to know how many people were working in each hour of the day.
SELECT
Employee_id,
TIME(shift_start_at,timezone) AS shift_start,
TIME(shift_end_at,timezone) AS shift_end,
FROM
`employee_shifts` AS shifts
WHERE
DATE(shifts.shift_start_at_local) >= "2022-05-01"
GROUP BY
1,
2,
3
Assuming you are on mysql version 8 or above generate all the buckets , left join to shifts to infill times in start-endtime ranges , filter out those that are not applicable then count eg:-
DROP TABLE IF EXISTS t;
create table t (id int, startts datetime, endts datetime);
insert into t values
(1,'2022-06-19 08:30:00','2022-06-19 10:00:00'),
(2,'2022-06-19 08:30:00','2022-06-19 08:45:00'),
(3,'2022-06-19 07:00:00','2022-06-19 07:59:00');
with cte as
(select 7 as bucket union select 8 union select 9 union select 10 union select 11),
cte1 as
(select bucket,t.*,
floor(hour(startts)) starthour, floor(hour(endts)) endhour
from cte
left join t on cte.bucket between floor(hour(startts)) and floor(hour(endts))
)
select bucket,count(id) nof from cte1 group by bucket
;
+--------+-----+
| bucket | nof |
+--------+-----+
| 7 | 1 |
| 8 | 2 |
| 9 | 1 |
| 10 | 1 |
| 11 | 0 |
+--------+-----+
5 rows in set (0.001 sec)
If you have a limited number of time bucket maybe you can use it this way
WITH CTE AS
(SELECT
COUNTRY,
MONTH,
TIMESTAMP_DIFF(time_b, time_a, MINUTE) dt,
METRIC_a,
METRIC_b
FROM
TABLE_NAME)
SELECT
CASE
WHEN dt BETWEEN 0 AND 10 THEN "0-10"
WHEN dt BETWEEN 10 AND 20 THEN "11-20"
WHEN dt BETWEEN 20 AND 30 THEN "21-30"
WHEN dt BETWEEN 30 AND 40 THEN "31-40"
WHEN dt > 40 THEN ">40"
END as time_bucket,
AVG(METRIC_a),
SUM(METRIC_b)
FROM CTE
Althought, I should emphasize that this solution works if you have a limited bucket. If you have a lot of buckets, you can create a base table with your buckets then LEFT JOIN it to get your results.
Just use a subquery for each column mentioning the required timestamp in between, also make sure your start_time and end_time columns are timestamp types. For more information, please share the table structure, sample data, and expected output
If I understood well, this would be
SELECT HOUR, (SELECT COUNT(*)
FROM employee
WHERE start_time <= HOUR
AND end_time >= HOUR) AS working
FROM schedule HOUR
Where schedule is a table with employee schedules.
We have a MySQL database containing bookings on different courts. Table properties (shortened):
CREATE TABLE `booking` (
`startDate` datetime NOT NULL,
`endDate` datetime NOT NULL,
`courtId` varchar(36),
FOREIGN KEY (`courtId`) REFERENCES `court` (`id`) ON DELETE CASCADE
)
Usually, bookings are paid, but under certain conditions (which I can check in the WHERE-part of a query), bookings can be free.
Given a court and booking duration, I want to query the next datetime at which the booking can be created for free. The conditions are not the problem, the problem is how to query not for entities but for datetime values.
How to realize this efficiently in MySQL?
EDIT: Maybe it helps to outline the conditions under which bookings are free:
The conditions under which bookings are free are dependent on how many courts are offered at the startDate by someone (courts are always offered except if there are special "not-offered"-bookings on that court) and how many other bookings overlapping the startDate are already free. This means bookings can be (and probably are) free even if there are no bookings at all in the database.
Solution
Finding available slot before the last booking :
Find the difference between each booking with it's following one. If the difference is greater than the number of days of the new booking, you can use that slot.
Finding available slot after the last booking :
If there is no such slot, you can assign a day after the end date of the last booking.
If this query returns null, it means there is no booking for the court. You can handle that in the client side.
Code
SET #c := 1; # Court id
SET #n := 2; # Number of days
/*
Previous booking
*/
SET #i := 0;
CREATE TEMPORARY TABLE bp AS
SELECT #i := #i + 1 AS id, startDate, endDate FROM booking
WHERE courtId = #c
ORDER BY startDate;
/*
Next booking
*/
SET #i := -1;
CREATE TEMPORARY TABLE bn AS
SELECT #i := #i + 1 AS id, startDate, endDate FROM booking
WHERE courtId = #c
ORDER BY startDate;
/*
Finding available slot before the last booking (Intermediate slot).
*/
SELECT DATE_ADD(MIN(bp.endDate), INTERVAL 1 DAY) INTO #si FROM
bp
JOIN
bn
ON bn.id = bp.id
WHERE DATEDIFF(bn.startDate, bp.endDate) > #n;
/*
Finding available slot after the last booking
*/
SELECT DATE_ADD(MAX(endDate), INTERVAL 1 DAY) INTO #sa FROM bn;
SELECT IFNULL(#si, #sa);
Using the code
Just replace the values of the variables #c and #n.
An idea to solve this is to rephrase it as: for the given :court_id parameter, give me the smallest future end_time for which no other booking starts within the given :duration parameter.
This can be expressed in different ways in SQL.
With a not exists condition and a correlated subquery that ensures that no further booking on the same court starts within :duration minutes.
select min(b.end_date) next_possible_start_date
from bookings b
where
b.court_id = :court_id
and b.end_date > now()
and not exists (
select 1
from bookings b1
where
b.court_id = :court_id
and b1.start_date > b.end_date
and b1.start_date < DATE_ADD(b.end_date, interval :duration minute)
)
Note: if you have additional conditions, they must be repeated in the where clause of the query and of the subquery.
The same logic as not exists can be impemented with a left join antipattern
select min(b.end_date) next_possible_start_date
from bookings b
left join booking b1
on b1.court_id = b.court_id
and bi1.start_date > b.end_date
and b1.start < DATE_ADD(b.end_date, interval :duration minute)
where
b.court_id = :court_id
and b.end_date > now()
and b1.court_id is null
In MySQL 8.0, it is also possible to use window functions: lag() retrieves the start_date of the next booking, which can then be compared with the end_date of the current booking.
select min(end_date) next_possible_start_date
from (
select
end_date,
lead(start_date) over(partition by court_id order by start_date) next_start_date
from booking b
where court_id = :court_id
)
where
next_start_date is null
or next_start_date >= DATE_ADD(end_date, interval :duration minute)
Edit
Here is a new version of the query that adresses the use case when the court is immediatly free at the time when the search is performed:
select
court_id,
greatest(min(b.end_date), now()) next_possible_start_date
from bookings b
where
-- b.court_id = :court_id and
not exists (
select 1
from bookings b1
where
b1.court_id = b.court_id
and b1.start_date > b.end_date
and b1.start_date < date_add(greatest(b.end_date, now()), interval ::duration minute)
)
group by court_id
Note: this searches for all available courts at once; you can uncomment the where clause to filter on a specific court.
Given this sample data:
court_id | start_date | end_date
-------: | :------------------ | :------------------
1 | 2019-10-29 13:00:00 | 2019-10-29 13:30:00
1 | 2019-10-29 14:00:00 | 2019-10-29 15:00:00
2 | 2019-10-29 23:14:05 | 2019-10-30 00:14:05
2 | 2019-10-30 01:14:05 | 2019-10-30 02:14:05
Court 1 is immedialty free. Court 2 is booked for next hour, then there is a 60 minutes vacancy before the next booking.
If we run the query for a duration of 60 minutes, we get:
court_id | next_possible_start_date
-------: | :-----------------------
1 | 2019-10-29 23:14:05 -- available right now
2 | 2019-10-30 00:14:05 -- available in 1 hour
While for 90 minutes, we get:
court_id | next_possible_start_date
-------: | :-----------------------
1 | 2019-10-29 23:14:05 -- available right now
2 | 2019-10-30 02:14:05 -- available in 3 hours
Demo on DB Fiddle
While preparing for an interview, I have come across an SQL question and I hope to get some insight as to how to better answer it.
Given timestamps, userid, how to determine the number of users who are active everyday in a week?
There's very little to it, but that's the question in front of me.
I'm going to demonstrate such an idea based on what makes most sense to me and the way I would reply if the question was presented same as here:
First, let's assume a data set as such, we will name the table logins:
+---------+---------------------+
| user_id | login_timestamp |
+---------+---------------------+
| 1 | 2015-09-29 14:05:05 |
| 2 | 2015-09-29 14:05:08 |
| 1 | 2015-09-29 14:05:12 |
| 4 | 2015-09-22 14:05:18 |
| ... | ... |
+---------+---------------------+
There may be other columns, but we don't mind those.
First of all we should determine the borders of that week, for that we can use ADDDATE(). Combined with the idea that today's date-today's week-day (MySQL's DAYOFWEEK()), is sunday's date.
For instance: If today is Wednesday the 10th, Wed - 3 = Sun, thus 10 - 3 = 7, and we can expect Sunday to be the 7th.
We can get WeekStart and WeekEnd timestamps this way:
SELECT
DATE_FORMAT(ADDDATE(CURDATE(), INTERVAL 1-DAYOFWEEK(CURDATE()) DAY), "%Y-%m-%d 00:00:00") WeekStart,
DATE_FORMAT(ADDDATE(CURDATE(), INTERVAL 7-DAYOFWEEK(CURDATE()) DAY), "%Y-%m-%d 23:59:59") WeekEnd;
Note: in PostgreSQL there's a DATE_TRUNC() function which returns the beginning of a specified time unit, given a date, such as week start, month, hour, and so on. But that's not available in MySQL.
Next, let's utilize WeekStart and weekEnd in order to clice our data set, in this example I'll just show how to filter, using hard coded dates:
SELECT *
FROM `logins`
WHERE login_timestamp BETWEEN '2015-09-29 14:05:07' AND '2015-09-29 14:05:13'
This should return our data set sliced, with only relevant results:
+---------+---------------------+
| user_id | login_timestamp |
+---------+---------------------+
| 2 | 2015-09-29 14:05:08 |
| 1 | 2015-09-29 14:05:12 |
+---------+---------------------+
We can then reduce our result set to only the user_ids, and filter out duplicates. then count, this way:
SELECT COUNT(DISTINCT user_id)
FROM `logins`
WHERE login_timestamp BETWEEN '2015-09-29 14:05:07' AND '2015-09-29 14:05:13'
DISTINCT will filter out duplicates, and count will return just the amount.
Combined, this becomes:
SELECT COUNT(DISTINCT user_id)
FROM `logins`
WHERE login_timestamp
BETWEEN DATE_FORMAT(ADDDATE(CURDATE(), INTERVAL 1- DAYOFWEEK(CURDATE()) DAY), "%Y-%m-%d 00:00:00")
AND DATE_FORMAT(ADDDATE(CURDATE(), INTERVAL 7- DAYOFWEEK(CURDATE()) DAY), "%Y-%m-%d 23:59:59")
Replace CURDATE() with any timestamp in order to get that week's user login count.
But I need to break this down to days, I hear you cry. Of course! and this is how:
First, let's translate our over-informative timestamps to just the date data. We add DISTINCT because we don't mind the same user logging in twice the same day. we count users, not logins, right? (note we step back here):
SELECT DISTINCT user_id, DATE_FORMAT(login_timestamp, "%Y-%m-%d")
FROM `logins`
This yields:
+---------+-----------------+
| user_id | login_timestamp |
+---------+-----------------+
| 1 | 2015-09-29 |
| 2 | 2015-09-29 |
| 4 | 2015-09-22 |
| ... | ... |
+---------+-----------------+
This query, we will wrap with a second, in order to count appearances of every date:
SELECT `login_timestamp`, count(*) AS 'count'
FROM (SELECT DISTINCT user_id, DATE_FORMAT(login_timestamp, "%Y-%m-%d") AS `login_timestamp` FROM `logins`) `loginsMod`
GROUP BY `login_timestamp`
We use count and a grouping in order to get the list by date, which returns:
+-----------------+-------+
| login_timestamp | count |
+-----------------+-------+
| 2015-09-29 | 1 +
| 2015-09-22 | 2 +
+-----------------+-------+
And after all the hard work, both combined:
SELECT `login_timestamp`, COUNT(*)
FROM (
SELECT DISTINCT user_id, DATE_FORMAT(login_timestamp, "%Y-%m-%d") AS `login_timestamp`
FROM `logins`
WHERE login_timestamp BETWEEN DATE_FORMAT(ADDDATE(CURDATE(), INTERVAL 1- DAYOFWEEK(CURDATE()) DAY), "%Y-%m-%d 00:00:00") AND DATE_FORMAT(ADDDATE(CURDATE(), INTERVAL 7- DAYOFWEEK(CURDATE()) DAY), "%Y-%m-%d 23:59:59")) `loginsMod`
GROUP BY `login_timestamp`;
Will give you a daily breakdown of logins per-day in this week. Again, replace CURDATE() to get a different week.
As for the users themselves who logged in, let's combine the same stuff in a different order:
SELECT `user_id`
FROM (
SELECT `user_id`, COUNT(*) AS `login_count`
FROM (
SELECT DISTINCT `user_id`, DATE_FORMAT(`login_timestamp`, "%Y-%m-%d")
FROM `logins`) `logins`
GROUP BY `user_id`) `logincounts`
WHERE `login_count` > 6
I have two inner queries, the first is logins:
SELECT DISTINCT `user_id`, DATE_FORMAT(`login_timestamp`, "%Y-%m-%d")
FROM `logins`
Will provide the list of users, and the days when they logged in on, without duplicates.
Then we have logincounts:
SELECT `user_id`, COUNT(*) AS `login_count`
FROM `logins` -- See previous subquery.
GROUP BY `user_id`) `logincounts`
Will return the same list, with a count of how many logins each user had.
And lastly:
SELECT user_id
FROM logincounts -- See previous subquery.
WHERE login_count > 6
Filtering our those who didn't login 7 times, and dropping the date column.
This kinda got long, but I think it's rife with ideas and I think it may definitely help answering in an interesting way in a work interview. :)
create table fbuser(id integer, date date);
insert into fbuser(id,date)values(1,'2012-01-01');
insert into fbuser(id,date)values(1,'2012-01-02');
insert into fbuser(id,date)values(1,'2012-01-01');
insert into fbuser(id,date)values(1,'2012-01-01');
insert into fbuser(id,date)values(1,'2012-01-01');
insert into fbuser(id,date)values(1,'2012-01-01');
insert into fbuser(id,date)values(1,'2012-01-02');
insert into fbuser(id,date)values(1,'2012-01-03');
insert into fbuser(id,date)values(1,'2012-01-04');
insert into fbuser(id,date)values(1,'2012-01-05');
insert into fbuser(id,date)values(1,'2012-01-06');
insert into fbuser(id,date)values(1,'2012-01-07');
insert into fbuser(id,date)values(4,'2012-01-08');
insert into fbuser(id,date)values(4,'2012-01-08');
insert into fbuser(id,date)values(1,'2012-01-08');
insert into fbuser(id,date)values(1,'2012-01-09');
select * from fbuser;
id | date
----+------------
1 | 2012-01-01
1 | 2012-01-02
1 | 2012-01-01
1 | 2012-01-01
1 | 2012-01-01
1 | 2012-01-01
1 | 2012-01-02
1 | 2012-01-03
1 | 2012-01-04
1 | 2012-01-05
1 | 2012-01-06
1 | 2012-01-07
2 | 2012-01-07
3 | 2012-01-07
4 | 2012-01-07
4 | 2012-01-08
4 | 2012-01-08
1 | 2012-01-08
1 | 2012-01-09
select id,count(DISTINCT date) from fbuser
where date BETWEEN '2012-01-01' and '2012-01-07'
group by id having count(DISTINCT date)=7
id | count
----+-------
1 | 7
(1 row)
Query counts unique dates logged in by user for the given period and returns id with 7 occurrences. If you have time also in your date you can use date_format.
With given data of: userid and timestamp; How does one calculate the number of "active users" on each day in a week?
The problem of course is that there might be no logins at all, or none on certain days in a week, so the basic solution to such a requirement is that you must have a series of dates to compare the logins against.
There are a wide variety of ways to generate the dates of a week and the method one chooses would depend on 2 main factors:
How often do I need these (or similar) results?
the platform I am using. (For example it is very easy to "generate a series" using Postgres but MySQL does not offer such a feature whereas recently MariaDB has introduced series tables to help solve such needs. So knowing your platform's capabilities will affect how you solve this.)
IF I need to do this regularly (which I assume will be true) then I would create a "calendar table" of one row per day for a reasonable extensive period (say 10 years) which is only approx 3652 rows, with its primary key as the date column. In this table we can also store the "week_number" using the week() function which makes week by week reporting simpler (and we can add other columns in this table as well).
So, assuming I have built the calendar table containing each date and a week number then we can take the week number from today's date, subtract 1, and gather the needed login data like this:
select
c.caldate, count(distinct l.userid) as user_logins
from calendar_table as c
left join login_table l on l.timestamp >= c.caldate and l.timestamp < date_add(c.caldate,INTERVAL 1 DAY)
where c.week_number = WEEK(curdate())-1
group by c.caldate
How did I create the calendar table?
Well as said earlier there are a variety of methods, and for MySQL there are options available here: How to populate a table with a range of dates?
I tried this in Teradata and here is the SQL. First, get the User unique to a date, then check, if the user is present for 7 days.
SELECT src.USER_ID
,COUNT(*) CNT
FROM (SELECT USER_ID
,CAST(LOGIN_TIMESTAMP AS DATE FORMAT 'YYYY-MM-DD') AS LOGIN_DT
FROM src_table
WHERE LOGIN_TIMESTAMP BETWEEN '2017-11-12 00:00:00' AND '2017-11-18 23:59:59'
GROUP BY 1,2
)src GROUP BY 1 HAVING CNT = 7;
INSERT INTO src_table VALUES (1,'2017-11-12 10:10:10');
INSERT INTO src_table VALUES (1,'2017-11-13 10:10:10');
INSERT INTO src_table VALUES (1,'2017-11-13 11:10:10');
INSERT INTO src_table VALUES (1,'2017-11-13 12:10:10');
INSERT INTO src_table VALUES (1,'2017-11-14 10:10:10');
INSERT INTO src_table VALUES (1,'2017-11-15 10:10:10');
INSERT INTO src_table VALUES (1,'2017-11-16 10:10:10');
INSERT INTO src_table VALUES (1,'2017-11-17 10:10:10');
INSERT INTO src_table VALUES (1,'2017-11-18 10:10:10');
INSERT INTO src_table VALUES (2,'2017-11-12 01:10:10');
INSERT INTO src_table VALUES (2,'2017-11-13 13:10:10');
INSERT INTO src_table VALUES (2,'2017-11-14 14:10:10');
INSERT INTO src_table VALUES (2,'2017-11-15 12:10:10');
INSERT INTO src_table VALUES (5,'2017-11-12 01:10:10');
INSERT INTO src_table VALUES (5,'2017-11-13 02:10:10');
INSERT INTO src_table VALUES (5,'2017-11-14 03:10:10');
INSERT INTO src_table VALUES (5,'2017-11-15 04:10:10');
INSERT INTO src_table VALUES (5,'2017-11-16 05:10:10');
INSERT INTO src_table VALUES (5,'2017-11-17 06:10:10');
INSERT INTO src_table VALUES (8,'2017-11-12 04:10:10');
INSERT INTO src_table VALUES (8,'2017-11-13 05:10:10');
INSERT INTO src_table VALUES (8,'2017-11-14 06:10:10');
INSERT INTO src_table VALUES (8,'2017-11-15 01:10:10');
INSERT INTO src_table VALUES (8,'2017-11-16 02:10:10');
INSERT INTO src_table VALUES (8,'2017-11-17 03:10:10');
INSERT INTO src_table VALUES (8,'2017-11-18 03:10:10');
This works for me
select a.user_id, count(a.user_id) as active_time_in_days
from
(
select user_id, login_time, lead(login_time) over (partition by user_id order by login_time asc ) as next_day
from dev.login_info
group by 1,2
order by user_id, login_time asc
)a where a.login_time + interval '1 day' = next_day
group by 1;
How about this? I tried it and it works.
select yearweek(ts) as yearwk, user_id,
count(user_id) as counts
from log
group by 1,2
having count(user_id) =7;
I am trying to create a SELECT to return information from a couple of tables. I had it working but then received an additional requirement and now I am having trouble figuring out how to get what I want.
I have a table with information on programs that could be included in the report (based on further requirements)...this file is called milestones.
I have another table with projects in it that relate to the programs - if the IDs match
I have a new table that has a manually entered override end date - this is the new requirement. There is a system end date in the milestones table, but if this override date is entered then it takes precedence over the system end date. If an override date has been entered, the exception file will have the same program ID and two dates which match dates in the milestones table.
dates are yyyy-mm-dd formatted
Example data:
Milestones:
prgId | startDate | endDate
------------------------------
123 | 2014-03-09 | 2014-11-10
123 | 2014-07-10 | 2014-11-10
324 | 2014-05-09 | 2014-11-12
exceptions:
prgId | startDate | overEnd
-------------------------------
123 | 2014-03-09 | 2014-05-31
projects:
prgId | cust
-------------
123 | 12121
123 | 4323
what I currently have being returned is:
prgId prjCnt startDate endDate overEnd
123 2 2014-03-09 2014-11-10 2014-05-31
123 2 2014-07-10 2014-11-10
324 0 2014-05-09 2014-11-12
I do realize that right now the two projects for program 123 will show for both lines - we will be looking for a way to associate them with the right ones but do not have that yet.
We added the override date requirement so that a report of current programs would not show both the '123' lines but only the one that is current (the second one).
My current SELECT is like this (sorry, I can't get this to show easier it is really long):
SELECT milestones.*, newtbl.prjcnt, exceptions.overEnd
FROM milestones
LEFT JOIN ((
SELECT prgGuid, count( prgGuid ) AS prjcnt
FROM projects
GROUP BY prgGuid
) AS newtbl )
ON milestones.prgId = newtbl.prgId
LEFT JOIN exceptions
ON (milestones.prgId = exceptions.prgId
AND milestones.startDate = exceptions.startDate)
WHERE <(milestones.startDate > '2013-00-00')
AND (milestones.startDate <= CURDATE() AND milestones.endDate >= CURDATE())
ORDER BY milestones.endDate, milestones.startDate DESC
Now what I want is to change this to only grab programs, project counts, start and end dates, and the override end date for programs where the start date is anything from 2013 to the current date and that have not ended yet. Now....if a program has an override end date and that end date is current (>= the current date) it should be included but if the override date is NULL or <= the current date, I do not want to include it.
What I want to have returned is:
prgId prjCnt startDate endDate overEnd
123 2 2014-07-10 2014-11-10
324 0 2014-05-09 2014-11-12
The first line before has expired so shouldn't show.
I've tried a few things but I either end up with no results or I get everything that I am currently getting.
Can someone help me figure out what the SELECT should be?
So if I follow you, your DDL might look like this:
CREATE TABLE MILESTONES
(`prgId` int, `startDate` varchar(10), `endDate` varchar(10))
;
INSERT INTO MILESTONES
(`prgId`, `startDate`, `endDate`)
VALUES
(123, '2014-03-09', '2014-11-10'),
(123, '2014-07-10', '2014-11-10'),
(324, '2014-05-09', '2014-11-12')
;
CREATE TABLE EXCEPTIONS
(`prgId` int, `startDate` varchar(10), `overEnd` varchar(10))
;
INSERT INTO EXCEPTIONS
(`prgId`, `startDate`, `overEnd`)
VALUES
(123, '2014-03-09', '2014-05-31')
;
CREATE TABLE PROJECTS
(`prgId` int, `cust` int)
;
INSERT INTO PROJECTS
(`prgId`, `cust`)
VALUES
(123, 12121),
(123, 4323)
;
And your current query which isn't working is this (note I've corrected what I presume are typos in your query from your question):
SELECT milestones.*, newtbl.prjcnt, exceptions.overEnd
FROM milestones
LEFT JOIN ((
SELECT prgId, count( prgId ) AS prjcnt
FROM projects
GROUP BY prgId
) AS newtbl )
ON milestones.prgId = newtbl.prgId
LEFT JOIN exceptions
ON (milestones.prgId = exceptions.prgId
AND milestones.startDate = exceptions.startDate)
WHERE (milestones.startDate > '2013-00-00')
AND (milestones.startDate <= CURDATE() AND milestones.endDate >= CURDATE())
ORDER BY milestones.endDate, milestones.startDate DESC
A working solution looks like this:
SELECT DISTINCT
M.prgId as PRGID
, ( SELECT COUNT(X.prgID)
FROM PROJECTS X
WHERE X.prgID = M.prgID ) as PRJCNT
, M.startDate as STARTDATE
, M.endDate as ENDDATE
, COALESCE(E.overEnd,'') as OVEREND
FROM MILESTONES M
LEFT OUTER JOIN PROJECTS P
ON M.prgId = P.prgId
LEFT JOIN EXCEPTIONS E
ON M.prgId = E.prgId
AND M.startDate = E.startDate
WHERE M.startDate > '2013-01-01'
AND M.startDate <= CURDATE()
AND M.endDate >= CURDATE()
AND ( E.overEnd IS NULL
OR E.overEnd > CURDATE() )
You can see it in action here: SQLFiddle.
Note that the solution relies on the COALESCE function for clean output and more of your business rules being put in place in the WHERE clause.
I'm logging statistics of the gamers in my community. For both their online and in-game states I'm registering when they "begin" and when they "end". In order to show the most active day and hour of the day I'd like to use an SQL statement that measures the most active moments based on the "begin" and "end" datetime values.
Looking at SQL - select most 'active' time from db I can see similarities, but I need to also include the moments between the start and end time.
Perhaps the easiest way is to write a cron that does the calculations, but I hope this question might teach me how to address this issue in SQL instead.
I've been searching for an SQL statement that allows to create a datetime period and use that to substract single hours and days. But to no avail.
--- update
As I'm thinking more about this, I'm wondering whether it might be wise to run 24 queries based on each hour of the day (for most active hour) and several queries for the most active day. But that seems like a waste of performance. But this solution might make a query possible like:
SELECT COUNT(`userID`), DATE_FORMAT("%H",started) AS starthour,
DATE_FORMAT("%H",ended) AS endhour
FROM gameactivity
WHERE starthour >= $hour
AND endhour <= $hour GROUP BY `userID`
($hour is added for example purposes, of course I'm using PDO. Columns are also just for example purposes, whatever you think is easy for you to use in explaining that is identifiable as start and end is ok with me)
Additional information; PHP 5.5+, PDO, MySQL 5+
Table layout for ingame would be: gameactivity: activityid, userid, gameid, started, ended
DDL:
CREATE TABLE IF NOT EXISTS `steamonlineactivity` (
`activityID` int(13) NOT NULL AUTO_INCREMENT,
`userID` varchar(255) NOT NULL,
`online` datetime DEFAULT NULL,
`offline` datetime DEFAULT NULL,
PRIMARY KEY (`activityID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1;
If I understood your requirements correctly, if this graph represents user activity:
Day
12/1 12/2 12/3 12/4 ...
Hour 0 xx x x xx
1 x xx xx
2 xxx x x xx
3 x x
4 x x
5 x x
6 x
...
You want to know that 02:00 is the time of the day with the highest average activity (a row with 7 x), and 12/4 was most active day (a column with 10 x). Note that this doesn't imply that 02:00 of 12/4 was the most active hour ever, as you can see in the example. If this is not what you want please clarify with concrete examples of input and desired result.
We make a couple assumptions:
An activity record can start on one date and finish on the next one. For instance: online 2013-12-02 23:35, offline 2013-12-03 00:13.
No activity record has a duration longer than 23 hours, or the number of such records is negligible.
And we need to define what does 'activity' mean. I picked the criteria that were easier to compute in each case. Both can be made more accurate if needed, at the cost of having more complex queries.
The most active time of day will be the hour with which more activity records overlap. Note that if a user starts and stops more than once during the hour it will be counted more than once.
The most active day will be the one for which there were more unique users that were active at any time of the day.
For the most active time of day we'll use a small auxiliary table holding the 24 possible hours. It can also be generated and joined on the fly with the techniques described in other answers.
CREATE TABLE hour ( hour tinyint not null, primary key(hour) );
INSERT hour (hour)
VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)
, (11), (12), (13), (14), (15), (16), (17), (18), (19), (20)
, (21), (22), (23);
Then the following queries give the required results:
SELECT hour, count(*) AS activity
FROM steamonlineactivity, hour
WHERE ( hour BETWEEN hour(online) AND hour(offline)
OR hour(online) BETWEEN hour(offline) AND hour
OR hour(offline) BETWEEN hour AND hour(online) )
GROUP BY hour
ORDER BY activity DESC;
SELECT date, count(DISTINCT userID) AS activity
FROM (
SELECT userID, date(online) AS date
FROM steamonlineactivity
UNION
SELECT userID, date(offline) AS date
FROM steamonlineactivity
) AS x
GROUP BY date
ORDER BY activity DESC;
You need a sequence to get values for hours where there was no activity (e.g. hours where nobody starting or finishing, but there were people on-line who had started but had not finished in that time). Unfortunately there is no nice way to create a sequence in MySQL so you will have to create the sequence manually;
CREATE TABLE `hour_sequence` (
`ID` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`hour` datetime NOT NULL,
KEY (`hour`),
PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
# this is not great
INSERT INTO `hour_sequence` (`hour`) VALUES
("2013-12-01 00:00:00"),
("2013-12-01 01:00:00"),
("2013-12-01 02:00:00"),
("2013-12-01 03:00:00"),
("2013-12-01 04:00:00"),
("2013-12-01 05:00:00"),
("2013-12-01 06:00:00"),
("2013-12-01 07:00:00"),
("2013-12-01 08:00:00"),
("2013-12-01 09:00:00"),
("2013-12-01 10:00:00"),
("2013-12-01 11:00:00"),
("2013-12-01 12:00:00");
Now create some test data
CREATE TABLE `log_table` (
`ID` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`userID` bigint(20) unsigned NOT NULL,
`started` datetime NOT NULL,
`finished` datetime NOT NULL,
KEY (`started`),
KEY (`finished`),
PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET latin1;
INSERT INTO `log_table` (`userID`,`started`,`finished`) VALUES
(1, "2013-12-01 00:00:12", "2013-12-01 02:25:00"),
(2, "2013-12-01 07:25:00", "2013-12-01 08:23:00"),
(1, "2013-12-01 04:25:00", "2013-12-01 07:23:00");
Now the query - for every hour we keep a tally (accumulation/running total/integral etc) of how many people started a session hour-on-hour
SELECT
HS.hour as period_starting,
COUNT(LT.userID) AS starts
FROM `hour_sequence` HS
LEFT JOIN `log_table` LT ON HS.hour > LT.started
GROUP BY
HS.hour
And also how many people went off-line likewise
SELECT
HS.hour as period_starting,
COUNT(LT.userID) AS finishes
FROM `hour_sequence` HS
LEFT JOIN `log_table` LT ON HS.hour > LT.finished
GROUP BY
HS.hour
By subtracting the accumulation of people that had gone off-line at a point in time from the accumulation of people that have come on-line at that point in time we get the number of people who were on-line at that point in time (presuming there were zero people on-line when the data starts, of course).
SELECT
starts.period_starting,
starts.starts as users_started,
finishes.finishes as users_finished,
starts.starts - finishes.finishes as users_online
FROM
(
SELECT
HS.hour as period_starting,
COUNT(LT.userID) AS starts
FROM `hour_sequence` HS
LEFT JOIN `log_table` LT ON HS.hour > LT.started
GROUP BY
HS.hour
) starts
LEFT JOIN (
SELECT
HS.hour as period_starting,
COUNT(LT.userID) AS finishes
FROM `hour_sequence` HS
LEFT JOIN `log_table` LT ON HS.hour > LT.finished
GROUP BY
HS.hour
) finishes ON starts.period_starting = finishes.period_starting;
Now a few caveats. First of all you will need a process to keep your sequence table populated with the hourly timestamps as time progresses. Additionally the accumulators do not scale well with large amounts of log data due to the tenuous join - it would be wise to constrain access to the log table by timestamp in both the starts and finishes subquery, and the sequence table while you are at it.
SELECT
HS.hour as period_starting,
COUNT(LT.userID) AS finishes
FROM `hour_sequence` HS
LEFT JOIN `log_table` LT ON HS.hour > LT.finished
WHERE
LT.finished BETWEEN ? AND ? AND HS.hour BETWEEN ? AND ?
GROUP BY
HS.hour
If you start constraining your log_table data to specific time ranges bear in mind you will have an offset issue if, at the point you start looking at the log data, there were already people on-line. If there were 1000 people on-line at the point where you start looking at your log data then you threw them all off the server from the query it would look like we went from 0 people on-line to -1000 people on-line!
#rsanchez had an amazing answer, but the query for most active time of day has a weird behaviour when handling session times that started and ended on the same hour (a short session). The query seems to calculate them to last for 24 hours.
With trial and error I corrected his query from that part to be following
SELECT hour, count(*) AS activity
FROM steamonlineactivity, hour
WHERE ( hour >= HOUR(online) AND hour <= HOUR(offline)
OR HOUR(online) > HOUR(offline) AND HOUR(online) <= hour
OR HOUR(offline) >= hour AND HOUR(offline) < HOUR(online) )
GROUP BY hour
ORDER BY activity DESC;
So with following structure:
CREATE TABLE hour ( hour tinyint not null, primary key(hour) );
INSERT hour (hour)
VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)
, (11), (12), (13), (14), (15), (16), (17), (18), (19), (20)
, (21), (22), (23);
CREATE TABLE `steamonlineactivity` (
`activityID` int(13) NOT NULL AUTO_INCREMENT,
`userID` varchar(255) NOT NULL,
`online` datetime DEFAULT NULL,
`offline` datetime DEFAULT NULL,
PRIMARY KEY (`activityID`)
);
INSERT INTO `steamonlineactivity` (`activityID`, `userID`, `online`, `offline`) VALUES
(1, '1', '2014-01-01 16:01:00', '2014-01-01 19:01:00'),
(2, '2', '2014-01-02 16:01:00', '2014-01-02 19:01:00'),
(3, '3', '2014-01-01 22:01:00', '2014-01-02 02:01:00'),
(4, '4', '2014-01-01 16:01:00', '2014-01-01 16:05:00');
The top query to get the most active times output following:
+------+----------+
| hour | activity |
+------+----------+
| 16 | 3 |
| 17 | 2 |
| 18 | 2 |
| 19 | 2 |
| 22 | 1 |
| 23 | 1 |
| 0 | 1 |
| 1 | 1 |
| 2 | 1 |
+------+----------+
Instead of the original query which gives following erronous result:
+------+----------+
| hour | activity |
+------+----------+
| 16 | 3 |
| 17 | 3 |
| 18 | 3 |
| 19 | 3 |
| 0 | 2 |
| 1 | 2 |
| 2 | 2 |
| 22 | 2 |
| 23 | 2 |
| 11 | 1 |
| 12 | 1 |
| 13 | 1 |
| 14 | 1 |
| 15 | 1 |
| 3 | 1 |
| 4 | 1 |
| 20 | 1 |
| 5 | 1 |
| 21 | 1 |
| 6 | 1 |
| 7 | 1 |
| 8 | 1 |
| 9 | 1 |
| 10 | 1 |
+------+----------+
This query is for oracle, but you can get idea from it:
SELECT
H, M,
COUNT(BEGIN)
FROM
-- temporary table that should return numbers from 0 to 1439
-- each number represents minute of the day, for example 0 represents 0:00, 100 represents 1:40, etc.
-- in oracle you can use CONNECT BY clause which is designated to do recursive queries
(SELECT LEVEL - 1 DAYMIN, FLOOR((LEVEL - 1) / 60) H, MOD((LEVEL - 1), 60) M FROM dual CONNECT BY LEVEL <= 1440) T LEFT JOIN
-- join stats to each row from T by converting discarding date and converting time to minute of a day
STATS S ON 60 * TO_NUMBER(TO_CHAR(S.BEGIN, 'HH24')) + TO_NUMBER(TO_CHAR(S.BEGIN, 'MI')) <= T.DAYMIN AND
60 * TO_NUMBER(TO_CHAR(S.END, 'HH24')) + TO_NUMBER(TO_CHAR(S.END, 'MI')) > T.DAYMIN
GROUP BY H, M
HAVING COUNT(BEGIN) > 0
ORDER BY H, M
GROUP BY H, M
HAVING COUNT(BEGIN) > 0
ORDER BY H, M
Fiddle: http://sqlfiddle.com/#!4/e5e31/9
The idea is to have some temp table or view with one row for time point, and left join to it. In my example there is one row for every minute in day. In mysql you can use variables to create such view on-the-fly.
MySQL version:
SELECT
FLOOR(T.DAYMIN / 60), -- hour
MOD(T.DAYMIN, 60), -- minute
-- T.DAYMIN, -- minute of the day
COUNT(S.BEGIN) -- count not null stats
FROM
-- temporary table that should return numbers from 0 to 1439
-- each number represents minute of the day, for example 0 represents 0:00, 100 represents 1:40, etc.
-- in mysql you must have some table which has at least 1440 rows;
-- I use (INFORMATION_SCHEMA.COLLATIONSxINFORMATION_SCHEMA.COLLATIONS) for that purpose - it should be
-- in every database
(
SELECT
#counter := #counter + 1 AS DAYMIN
FROM
INFORMATION_SCHEMA.COLLATIONS A CROSS JOIN
INFORMATION_SCHEMA.COLLATIONS B CROSS JOIN
(SELECT #counter := -1) C
LIMIT 1440
) T LEFT JOIN
-- join stats to each row from T by converting discarding date and converting time to minute of a day
STATS S ON (
(60 * DATE_FORMAT(S.BEGIN, '%H')) + (1 * DATE_FORMAT(S.BEGIN, '%i')) <= T.DAYMIN AND
(60 * DATE_FORMAT(S.END, '%H')) + (1 * DATE_FORMAT(S.END, '%i')) > T.DAYMIN
)
GROUP BY T.DAYMIN
HAVING COUNT(S.BEGIN) > 0 -- filter empty counters
ORDER BY T.DAYMIN
Fiddle: http://sqlfiddle.com/#!2/de01c/1
I've been overthinking this question myself and based on everyone's answers I think it's obvious to conclude with the following;
In general it's probably easy to implement some kind of separate table that has the hours of the day and do inner selects from that separate table. Other examples without a separate table have many sub selects, even with four tiers, which makes me believe they will probably not scale. Cron solutions have come to my mind as well, but the question was asked - out of curiosity - to focus on SQL queries and not other solutions.
In my own case and completely outside the scope of my own question, I believe the best solution is to create a separate table with two fields (hour [Y-m-d H], onlinecount, playingcount) that counts the number of people online at a certain hour and the people playing at a certain hour. When a player stops playing or goes offline we update the count (+1) based on the start and end times. Thus I can easily deduce tables and graphs from this separate table.
Please, let me know whether you come to the same conclusion. My thanks to #lolo, #rsanchez and #abasterfield. I wish I could split the bounty :)
sqlFiddle, this query will give you the period that has the most userCount, the period could be between anytime, it just gives you the start time and end time that has the most userCount
SELECT StartTime,EndTime,COUNT(*)as UserCount FROM
(
SELECT T3.StartTime,T3.EndTime,GA.Started,GA.Ended FROM
(SELECT starttime,(SELECT MIN(endtime) FROM
(SELECT DISTINCT started as endtime FROM gameactivity WHERE started BETWEEN '1970-01-01 00:00:00' AND '1970-01-01 23:59:59'
UNION
SELECT DISTINCT ended as endtime FROM gameactivity WHERE ended BETWEEN '1970-01-01 00:00:00' AND '1970-01-01 23:59:59'
)T1
WHERE T1.endtime > T2.starttime
)as endtime
FROM
(SELECT DISTINCT started as starttime FROM gameactivity WHERE started BETWEEN '1970-01-01 00:00:00' AND '1970-01-01 23:59:59'
UNION
SELECT DISTINCT ended as starttime FROM gameactivity WHERE ended BETWEEN '1970-01-01 00:00:00' AND '1970-01-01 23:59:59'
)T2
)T3,
GameActivity GA
WHERE T3.StartTime BETWEEN GA.Started AND GA.Ended
AND T3.EndTime BETWEEN GA.Started AND GA.Ended
)FinalTable
GROUP BY StartTime,EndTime
ORDER BY UserCount DESC
LIMIT 1
just change the date of '1970-01-01' occurences to the date you're trying to get data from.
What the query does it selects all the times in the inner queries and then create intervals out of them, then join with GameActivity and count occurrences of users within those intervals and return the interval with the most userCount(most activity).
here's an sqlFiddle with one less tier
SELECT StartTime,EndTime,COUNT(*)as UserCount FROM
(
SELECT T3.StartTime,T3.EndTime,GA.Started,GA.Ended FROM
(SELECT DISTINCT started as starttime,(SELECT MIN(ended)as endtime FROM
gameactivity T1 WHERE ended BETWEEN '1970-01-01 00:00:00' AND '1970-01-01 23:59:59'
AND T1.ended > T2.started
)as endtime
FROM
gameactivity T2
WHERE started BETWEEN '1970-01-01 00:00:00' AND '1970-01-01 23:59:59'
)T3,
GameActivity GA
WHERE T3.StartTime BETWEEN GA.Started AND GA.Ended
AND T3.EndTime BETWEEN GA.Started AND GA.Ended
)FinalTable
GROUP BY StartTime,EndTime
ORDER BY UserCount DESC
LIMIT 1
or according to your query in your question above, you don't seem to care about dates, but only hour statistics across all dates then the below query might do it (your query just looks at the HOUR of started and ended and ignore users that play longer than 1 hour.
the below query might do it for you sqlFiddle
SELECT COUNT(*) as UserCount,
HOURSTABLE.StartHour,
HOURSTABLE.EndHour
FROM
(SELECT #hour as StartHour,
#hour:=#hour + 1 as EndHour
FROM
gameActivity as OrAnyTableWith24RowsOrMore,
(SELECT #hour:=0)as InitialValue
LIMIT 24) as HOURSTABLE,
gameActivity GA
WHERE HOUR(GA.started) >= HOURSTABLE.StartHour
AND HOUR(GA.ended) <= HOURSTABLE.EndHour
GROUP BY HOURSTABLE.StartHour,HOURSTABLE.EndHour
ORDER BY UserCount DESC
LIMIT 1
just delete the LIMIT 1 if you want to see userCount for other hours as well.
The easiest solution is to run a cron at the top of each hour of who has a start time but no end time (null end time? if you reset it when they login) and log that count. This will give you a count of currently logged in at each hour without needing to do funky schema changes or wild queries.
Now when you check the next hour and they had logged out they would fall out of your results. This query would work if you reset end time when they login.
SELECT CONCAT(CURDATE(), ' ', HOUR(NOW()), ' ', COUNT(*)) FROM activity WHERE DATE(start) = CURDATE() AND end IS NULL;
Then you can log this at your hearts content to a file or to another table (Of course you might need to adjust the select per your log table). For example you can have a table that gets one entry per day and only gets updated once.
Assume a log table like:
current_date | peak_hour | peak_count
SELECT IF(peak_count< $peak_count, true, false) FROM log where DATE(current_date) = NOW();
where $peak_count is a variable coming from your cron. If you find that you have a new bigger peak count you do an update, if the record does not exist for the day do an insert into log. Otherwise, no you have not beat a peak_hour from earlier in the day, don't do an update. This means each day will give you only 1 row in your table. Then you don't need to do any aggregation, it is all right there for you to see the date and hour over the course of a week or month or whatever.