I have written a query that returns time series data for a graph. The data represents requests coming over a network, we get the data in 30 second intervals, and we are aggregating by 5 minute intervals. Some of these intervals have no info, so the result (of time intervals) will look like the following for some domains when a full set would look like the second list.
2017-11-01 11:05:00
2017-11-01 11:15:00
2017-11-01 11:35:00
2017-11-01 11:05:00
2017-11-01 11:10:00
2017-11-01 11:15:00
2017-11-01 11:20:00
2017-11-01 11:25:00
2017-11-01 11:30:00
2017-11-01 11:35:00
My question is, could we add to the following query so all of the missing five minute intervals would be present, but the number of requests would be zero. Essentially replacing the missing data with the proper timestamp and 0 corresponding requests?
The batch_id just represents the most recent set of data that we received.
select
DATE_FORMAT(FROM_UNIXTIME(`timestamp`)
, '%Y-%m-%d %H:00:00') INTERVAL(MINUTE(FROM_UNIXTIME(`timestamp`))
- MINUTE(FROM_UNIXTIME(`timestamp`)) MOD 5) MINUTE as `time`
, SUM(requests) as requests
from `raw_graphs`
where (`batch_id` = 25) group by `time`
Related
I have two tables in my schema. The first contains a list of recurring appointments - default_appointments. The second table is actual_appointments - these can be generated from the defaults or individually created so not linked to any default entry.
Example:
default_appointments
id
day_of_week
user_id
appointment_start_time
appointment_end_time
1
1
1
10:00:00
16:00:00
2
4
1
11:30:00
17:30:00
3
6
5
09:00:00
17:00:00
actual_appointments
id
default_appointment_id
user_id
appointment_start
appointment_end
1
1
1
2021-09-13 10:00:00
2021-09-13 16:00:00
2
NULL
1
2021-09-13 11:30:00
2021-09-13 13:30:00
3
6
5
2021-09-18 09:00:00
2021-09-18 17:00:00
I'm looking to calculate the total minutes that were scheduled in against the total that were actually created/generated. So ultimately I'd end up with a query result with this data:
user_id
appointment_date
total_planned_minutes
total_actual_minutes
1
2021-09-13
360
480
1
2021-09-16
360
0
5
2021-09-18
480
480
What would be the best approach here? Hopefully the above makes sense.
Edit
OK so the default_appointments table contains all appointments that are "standard" and are automatically generated. These are what appointments "should" happen every week. So e.g. ID 1, this appointment should occur between 10am and 4pm every Monday. ID 2 should occur between 11:30am an 5:30pm every Thursday.
The actual_appointments table contains a list of all of the appointments which did actually occur. Basically what happens is a default_appointment will automatically generate itself an instance in the actual_appointments table when initially set up. The corresponding default_appointment_id indicates that it links to a default and has not been changed - therefore the times on both will remain the same. The user is free to change these appointments that have been generated by a default, resulting in setting the default_appointment_id to NULL * - or -* can add new appointments unrelated to a default.
So, if on a Monday (day_of_week = 1) I should normally have a default appointment at 10am - 4pm, the total minutes I should have planned based on the defaults are 360 minutes, regardless of what's in the actual_appointments table, I should be planned for those 360 minutes every Monday without fail. If in the system I say - well actually, I didn't have an appointment from 10am - 4pm and instead change it to 10am - 2pm, actual_appointments table will then contain the actual time for the day, and the actual minutes appointed would be 240 minutes.
What I need is to group each of these by the date and user to understand how much time the user had planned for appointments in the default_appointments table vs how much they actually appointed.
Adjusted based on new detail in the question.
Note: I used day_of_week values compatible with default MySQL behavior, where Monday = 2.
The first CTE term (args) provides the search parameters, start date and number of days. The second CTE term (drange) calculates the dates in the range to allow generation of the scheduled appointments within that range.
allrows combines the scheduled and actual appointments via UNION to prepare for aggregation. There are other ways to set this up.
Finally, we aggregate the results per user_id and date.
The test case:
Working Test Case (Updated)
WITH RECURSIVE args (startdate, days) AS (
SELECT DATE('2021-09-13'), 7
)
, drange (adate, days) AS (
SELECT startdate, days-1 FROM args UNION ALL
SELECT adate + INTERVAL '1' DAY, days-1 FROM drange WHERE days > 0
)
, allrows AS (
SELECT da.user_id
, dr.adate
, ROUND(TIME_TO_SEC(TIMEDIFF(da.appointment_end_time, da.appointment_start_time))/60, 0) AS planned
, 0 AS actual
FROM drange AS dr
JOIN default_appointments AS da
ON da.day_of_week = dayofweek(adate)
UNION
SELECT user_id
, DATE(appointment_start) AS xdate
, 0 AS planned
, TIMESTAMPDIFF(MINUTE, appointment_start, appointment_end)
FROM drange AS dr
JOIN actual_appointments aa
ON DATE(appointment_start) = dr.adate
)
SELECT user_id, adate
, SUM(planned) AS planned
, SUM(actual) AS actual
FROM allrows
GROUP BY adate, user_id
;
Result:
+---------+------------+---------+--------+
| user_id | adate | planned | actual |
+---------+------------+---------+--------+
| 1 | 2021-09-13 | 360 | 480 |
| 1 | 2021-09-16 | 360 | 0 |
| 5 | 2021-09-18 | 480 | 480 |
+---------+------------+---------+--------+
I have a table with three date fields (YYYY-MM-DD HH-MM-SS format): in, out and near. in and out always go together, while they might be along with near or not; near might exists by itself alone. There's no chance of all of them being NULL; something like this:
in_time out_time near_time
2019-10-02 10:46:31 2019-10-02 12:34:43 2019-10-02 09:51:57
NULL NULL 2019-10-02 11:51:08
2019-10-02 12:02:40 2019-10-02 12:08:56 NULL
I need to turn it into another table with the sum of every date field grouped by 30-minute intervals, this way:
interval qt_in qt_out qt_near
2019-10-02 08:30:00 1 0 2
2019-10-02 09:00:00 1 1 5
2019-10-02 09:30:00 2 0 3
I mean, I need to know the amount of entries, leavings and "being arounds" per day and per 30-min periods within that day; any possible interval might have data, it doesn't mind it's [00:00 - 00:30], [12:00 - 12:30] or [23:30 - 24:00].
Right now I only have this:
SELECT
from_unixtime((unix_timestamp(in_time) DIV (60 * 30)) * (30 * 60)) AS interval_in,
sum(CASE WHEN in_time IS NOT NULL THEN 1 ELSE 0 END) AS qt_in
FROM count_config_module.ap_summary
GROUP BY
interval_in
ORDER BY
interval_in
;
Analogous queries for out and near return analogous results, but I'd need to mix all up in the same table with a unique and shared 30-min intervals field that might come from... some of them? All of them? Any of them?
Thanks in advance!
I have the following table, I am using MYSQL
BayNo FixDateTime FixType
1 04/05/2015 16:15:00 tyre change
1 12/05/2015 00:15:00 oil change
1 12/05/2015 08:15:00 engine tuning
1 04/05/2016 08:11:00 car tuning
2 13/05/2015 19:30:00 puncture
2 14/05/2015 08:00:00 light repair
2 15/05/2015 10:30:00 super op
2 20/05/2015 12:30:00 wiper change
2 12/05/2016 09:30:00 denting
2 12/05/2016 10:30:00 wiper repair
2 12/06/2016 10:30:00 exhaust repair
4 12/05/2016 05:30:00 stereo unlock
4 17/05/2016 15:05:00 door handle repair
on any given day need do find the highest number of fixes made on a given bay number, and if that calculated number is repeated then it should also appear in the resultset
so would like to see the result set as follows
BayNo FixDateTime noOfFixes
1 12/05/2015 00:15:00 2
2 12/05/2016 09:30:00 2
4 12/05/2016 05:30:00 1
4 17/05/2016 15:05:00 1
I manage to get the counts of each but struggling to get the max and keep the highest calculated repeated value. can someone help please
Calculate the fixes per day per BayNo
Find the max daily fixes per BayNo
Use the result from 2 to filter out the result from 1
Something like this:
SELECT fixes.*
FROM (
#1
SELECT BayNo,DATE(FixDateTime) as day,count(*) as noOfFixes
FROM yourTable
GROUP BY BayNo,day
) as fixes
JOIN (
#2
SELECT MAX(noOfFixes) as maxNoOfFixes,BayNo
FROM (
#1
SELECT BayNo,DATE(FixDateTime) as day,count(*) as noOfFixes
FROM yourTable
GROUP BY BayNo,day
) as t
GROUP BY BayNo
) as maxfixes ON fixes.BayNo = maxfixes.BayNo
#3
WHERE fixes.noOfFixes = maxfixes.maxNoOfFixes
You can run the repeated query (1) separately and store the result in a temporary table if needed.
I'm assuming the FixDateTime column is a an actual datetime or timestamp column. If it's not, you will need to use a different method to get the date from it.
The final result of this will be used for a graphing application where sometimes we would not want the detailed granularity of data at the level it is stored in the table. This may be hard to phrase in a single question so I will give an example:
Example table:
DateTime AddressID Amount
1/1/2015 10:00:00 1 10
1/1/2015 10:00:00 2 8
1/1/2015 10:01:00 1 7
1/1/2015 10:01:00 2 12
1/1/2015 10:02:00 1 21
1/1/2015 10:02:00 2 15
etc...
Note: The times will always have 00 for the seconds - if that helps.
Note: The entries may NOT always have an entry for every minute, but they generally should. So it is possible some might times might be skipped. But there will always be an entry for both addressIDs (1 & 2) every time without fail.
I need to return the above 3 fields, in a period of time requested (for example past 24 hours), but only for certain increments of time FOR EACH OF THE ADDRESS ID's. For example, records for every 5 minutes, or every 10 minutes.
so in the case of 5 minutes it would return:
DateTime AddressID Amount
1/1/2015 10:**00**:00 1 10
1/1/2015 10:**00**:00 2 8
1/1/2015 10:**05**:00 1 11
1/1/2015 10:**05**:00 2 17
1/1/2015 10:**10**:00 1 28
1/1/2015 10:**10**:00 2 5
etc...
Performance is very important. I hope I explained that well enough for someone to get the idea of what I need and I thank you in advance for your suggestions.
EDIT: For clarification, the 5 minutes in the above example should be the minimum time BETWEEN each row. So, if in the above example, on the rare chance that there was a missing time entry for 10:05:00 it should not simply select the 10:10:00 row, it should select the 10:06:00 record and then the next row selected would be 10:11:00, etc.
I have a table that contains Following entries:
completed_time|| BOOK_CNT
*********************************************
2013-07-23 | 2
2013-07-22 | 1
2013-07-19 | 3
2013-07 16 |5
2013-07-12 |4
2013-07-11 |2
2013-07-02 |9
2013-06-30 |5
Now, I want to use above entries for data analysis.
Lets say DAYS_FROM, DAYS_TO and PERIOD are three variables.
I need to fire following sort of queries:
"Total book from DAYS_FROM to DAYS_TO in interval of PERIOD."
DAYS_FROM is a date in format YYYY-MM-DD
,DAYS_TO is a date in format YYYY-MM-DD
PERIOD is {1W,2W,1M,2M,1Y}
where W,M,Y represents WEEK,MONTH and YEAR.
Example: The queries DAYS_FROM=2013-07-23 , DAYS_TO=2013-07-03 and PERIOD=1W should return:
ith week - total
1 - 3
2- 8
3- 6
4- 14
Explanation:
1-3 means (The total book from 2013-07-21(sun) to 2013-07-23(tue) is 3 )
2-8 means (The total book from 2013-07-14(sun) to 2013-07-21(sun) is 8 )
3-16 means (The total book from 2013-07-07(sun) to 2013-07-14(sun) is 6 )
4-14 means (The total book from 2013-07-03(wed) to 2013-07-07(sun) is 14 )
Please refer the calendar image for better understanding.
How to fire such query?
What I tried?
SELECT DAY(completed_time), COUNT(total) AS Total
FROM my_tab
WHERE completed_time BETWEEN '2013-07-23' - INTERVAL 1 WEEK AND '2013-07-03'
GROUP BY DAY(completed_time);
The above queries subtracted 7 days from 2013-07-23 and thus considered 2013-07-16 to 2013-07-23 as first week, 2013-07-09 to 2013-07-16 as second week and so on.
A simple starting point would be something like below, of course you may want to adjust the ith value to suit your needs;
SET #period='1M';
SELECT CASE WHEN #period='1Y' THEN YEAR(completed_time)
WHEN #period='1M' THEN YEAR(completed_time)*100+MONTH(completed_time)
WHEN #period='2M' THEN FLOOR((YEAR(completed_time)*100+MONTH(completed_time))/2)*2
WHEN #period='1W' THEN YEARWEEK(completed_time)
WHEN #period='2W' THEN FLOOR(YEARWEEK(completed_time)/2)*2
END ith,
SUM(BOOK_CNT) Total
FROM my_tab
GROUP BY ith
ORDER BY ith DESC;
An SQLfiddle to test with.