SQL for time periods - mysql

Question

I have statistic table for internet radio (MySQL), there are such columns:
ip_address
time_start (datetime of listening start)
time_end (datetime of listening finish)
I need to select the listeners peak for each day, I mean maximum number of simultaneous unique ip listeners.
And it would be great also to have start and finish time for that peak.
For example:
2011-30-01 | 4 listeners peak | from 10:30 | till 11:25

Answer 1

IMHO it's simpler to load these 35'000 rows in memory, enumerate them, and maintain a count of the concurrent listener at a given moment.
This would be simpler if you load the row in the following format:
IP, Time, flag_That_Indicate_StartOrStop_Listening_For_This_Given_IP
so you'll be able to load the data ordered by time, and the you should simply enumerate all rows maintaining a list of listening IP.
Anyway, how do you consider multiple connections from the same IP?
There can be 10 different listeners behind a NAT using the same IP address.
Update:
You don't really need to change the DB structure, it's enough use a different SQL to load the data
SELECT ip_address, Time_Start AS MyTime, 1 As StartStop
FROM MyTable
ORDER BY Time_Start
UNION ALL
SELECT ip_address, Time_Stop AS MyTime, 0 As StartStop
FROM MyTable
Using this SQL you should be able to load all the data, and then enumerate all the rows.
It's important that the rows are sorted correctly.
if StartStop = 1 it's somone that start listening --> Add it's IP to the list of listeners, and increment the listeners count by 1
if StartStop = 0 it's someone that stop listening --> remove it's IP from the list of listeners, and decrement the listeners count by 1
and in the enumeration loop check when you reach the maximum number of concurrent listeners

Answer 2

Let go to find for an algorithm to get results with best performance.
Spliting time: Time is a continuous dimension, we need some points to mark as checkpoint where do a listener recount. How to find intervals or when check for total radio listener. I thing that the best strategy is to get different time_start and time_end.
This is my approach to split time. I create a view to simplify post:
create view time_split as
select p_time from (
Select
time_start
from
your_table
union
Select
time_end
from
your_table
) as T
I suggest to you 2 database index:
your_table( time_start, time_end) <--(1) explained below
your_table( time_end)
to avoid tablescan.
Count listeners peak: Join previous table with your table to do a recount of peak at each time checkpoint:
This is my approach for count listeners by check point time:
create view peak_by_time as
select p_time, count(*) as peak
from
your_table t
inner join
time_split
on time_split.p_time between t.time_start and t.time_end
group by
p_time
order by
p_time, peak
Remember to make a database index on your_table( time_start, time_end) <--(1) Here
Looking for max peak: Unfortunately MySQL don't has analytic functions, then over partition is not available and is not a way to take max peak over a day in previous view. Then you should get max peak of previous views. This is a performance killer operation. I suggest to you make this operation and next on in application logic and not in data base.
This is my approach for get max_peak by day (performance killer):
create view max_peak_by_day as
select
cast(p_time as date) as p_day ,
max(peak) as max_peak
from peak_by_time
group by cast(p_time as date)
Looking for slot times: at this moment you have max_peak for each day, now you need to look for continuous check times with same max_peak. Also MySQL don't has statistical functions neither CTE. I suggest to you that this code will be wrote on app layer. But, if you want to do this in database solution this is a way (warning performance killer):
First, extend peak_by_time view to get previous peak for p_time and for previous p_time:
create view time_split_extended as
select c.p_time, max( p.p_time) as previous_ptime
from
time_split c
inner join
time_split p
on p.p_time < c.p_time
group by c.p_time
create view peak_by_time_and_previous as
select
te.p_time,
te.previous_ptime,
pc.peak as peak,
pp.peak as previous_peak
from
time_split_extended te
inner join
peak_by_time pc on te.p_time = pc.p_time
inner join
peak_by_time pp on te.previous_ptime = pp.p_time
Now check that previous slot and current one have a max_peak:
select
cast(p_time as date) as p_day,
min( p_time ) as slot_from,
max( p_time) as slot_to,
peak
from
peak_by_time_and_previous p
inner join
max_peak_by_day m
on cast(p.p_time as date) = m.p_day and
p.peak = m.max_peak
where
p.peak = p.previous_peak
group by cast(p_time as date)
Disclaimer:
This is not tested. Sure that they are mistakes with table aliases or columns.
The last steps are performance killers. Perhaps someone can suggest best approach for this steps.
Also, I suggest to you that create temporary tables and materialize each view of this answer. This will improve performance and also you can know how many time takes each step.

Answer 3

This is essentially an implementation of the answer given by Max above. For simplicity I'll represent each listening episode as a start time and length as integer values (they could be changed to actual datetimes, and then the queries would need to be modified to use date arithmetic.)
> select * from episodes;
+--------+------+
| start | len |
+--------+------+
| 50621 | 480 |
| 24145 | 546 |
| 93943 | 361 |
| 67668 | 622 |
| 64681 | 328 |
| 110786 | 411 |
...
The following query combines the start and end times with a UNION, flagging end times to distinguish from start times, and keeping a running accumulator of the number of listeners:
SET #idx=0;
SET #n=0;
SELECT (#idx := #idx + 1) as idx,
t,
(#n := #n + delta) as n
FROM
(SELECT start AS t,
1 AS delta
FROM episodes
UNION ALL
SELECT start + len AS t,
-1 AS delta FROM episodes
ORDER BY t) stage
+------+--------+------+
| idx | t | n |
+------+--------+------+
| 1 | 8 | 1 |
| 2 | 106 | 2 |
| 3 | 203 | 3 |
| 4 | 274 | 2 |
| 5 | 533 | 3 |
| 6 | 586 | 2 |
...
where 't' is the start of each interval (it's a new "interval" whenever the number of listeners, "n", changes). In a version where "t" is an actual datetime, you could easily group by day to obtain a peak episode for each day, or other such summaries. To get the end time of each interval - you could take the table above and join it to itself on right.idx = left.idx + 1 (i.e. join each row with the succeeding one).

Answer 4

SELECT
COUNT(*) AS listeners,
current.time_start, AS peak_start,
MIN(overlap.time_end) AS peak_end
FROM
yourTable AS current
INNER JOIN
yourTable AS overlap
ON overlap.time_start <= current.time_start
AND overlap.time_end > current.time_start
GROUP BY
current.time_start,
current.time_end
HAVING
MIN(overlap.time_end) < COALESCE((SELECT MIN(time_start) FROM yourTable WHERE timeStart > current.timeStart), current.time_end+1)
For each record, join on everything that overlaps.
The MIN() of the overlapping records' time_end is when the first current listener stops listening.
If that time is less than next occurance of a time_start, it's a peak. (Peak = start immediately followed by a stop)