Best way to create on the fly, date ranges, for use with report.
So I can avoid empty rows on my report if there's no activity for a given day.
Mostly to avoid this issue: What is the most straightforward way to pad empty dates in sql results (on either mysql or perl end)?
My advice is: don't make your life harder, make it easier. Just create a table with one row for each calendar day, having as many rows as you think you reasonably need to last. In datawarehousing, this is the common solution, and it is so widely implemented this way that a dwh that doesn't have it, has a code smell.
Many people used to dealing with more traditional oltp/data entry apps feel a natural revulsion against this idea, because the feel the can generate the data anyway, and therefore it shouldn't be stored. But if you do create a table like that, you can adorn it with many useful attributes, such as whether it's a holdiday or a weekend, and you can store many common date representations (iso, european, us format etc) inside it, which can save you a ton of time when creating reports (since you don't have to bother figuring out how the date formatting works in each reporting tool you come by. Or you can go a step further and update your date table everyday to mark flags for the current day, current week, current month, current year, etc - all kinds of useful tools that make it much, much easier to build reports that need to work against some date range.
MySQL sample code as per request in comment:
delimiter //
DROP PROCEDURE IF EXISTS p_load_dim_date
//
CREATE PROCEDURE p_load_dim_date (
p_from_date DATE
, p_to_date DATE
)
BEGIN
DECLARE v_date DATE DEFAULT p_from_date;
DECLARE v_month tinyint;
CREATE TABLE IF NOT EXISTS dim_date (
date_key int primary key
, date_value date
, date_iso char(10)
, year smallint
, quarter tinyint
, quarter_name char(2)
, month tinyint
, month_name varchar(10)
, month_abbreviation varchar(10)
, week char(2)
, day_of_month tinyint
, day_of_year smallint
, day_of_week smallint
, day_name varchar(10)
, day_abbreviation varchar(10)
, is_weekend tinyint
, is_weekday tinyint
, is_today tinyint
, is_yesterday tinyint
, is_this_week tinyint
, is_last_week tinyint
, is_this_month tinyint
, is_last_month tinyint
, is_this_year tinyint
, is_last_year tinyint
);
WHILE v_date < p_to_date DO
SET v_month := month(v_date);
INSERT INTO dim_date(
date_key
, date_value
, date_iso
, year
, quarter
, quarter_name
, month
, month_name
, month_abbreviation
, week
, day_of_month
, day_of_year
, day_of_week
, day_name
, day_abbreviation
, is_weekend
, is_weekday
) VALUES (
v_date + 0
, v_date
, DATE_FORMAT(v_date, '%y-%c-%d')
, year(v_date)
, ((v_month - 1) DIV 3) + 1
, CONCAT('Q', ((v_month - 1) DIV 3) + 1)
, v_month
, DATE_FORMAT(v_date, '%M')
, DATE_FORMAT(v_date, '%b')
, DATE_FORMAT(v_date, '%u')
, DATE_FORMAT(v_date, '%d')
, DATE_FORMAT(v_date, '%j')
, DATE_FORMAT(v_date, '%w') + 1
, DATE_FORMAT(v_date, '%W')
, DATE_FORMAT(v_date, '%a')
, IF(DATE_FORMAT(v_date, '%w') IN (0,6), 1, 0)
, IF(DATE_FORMAT(v_date, '%w') IN (0,6), 0, 1)
);
SET v_date := v_date + INTERVAL 1 DAY;
END WHILE;
CALL p_update_dim_date();
END;
//
DROP PROCEDURE IF EXISTS p_update_dim_date;
//
CREATE PROCEDURE p_update_dim_date()
UPDATE dim_date
SET is_today = IF(date_value = current_date, 1, 0)
, is_yesterday = IF(date_value = current_date - INTERVAL 1 DAY, 1, 0)
, is_this_week = IF(year = year(current_date) AND week = DATE_FORMAT(current_date, '%u'), 1, 0)
, is_last_week = IF(year = year(current_date - INTERVAL 7 DAY) AND week = DATE_FORMAT(current_date - INTERVAL 7 DAY, '%u'), 1, 0)
, is_this_month = IF(year = year(current_date) AND month = month(current_date), 1, 0)
, is_last_month = IF(year = year(current_date - INTERVAL 1 MONTH) AND month = month(current_date - INTERVAL 1 MONTH), 1, 0)
, is_this_year = IF(year = year(current_date), 1, 0)
, is_last_year = IF(year = year(current_date - INTERVAL 1 YEAR), 1, 0)
WHERE is_today
OR is_yesterday
OR is_this_week
OR is_last_week
OR is_this_month
OR is_last_month
OR is_this_year
OR is_last_year
OR IF(date_value = current_date, 1, 0)
OR IF(date_value = current_date - INTERVAL 1 DAY, 1, 0)
OR IF(year = year(current_date) AND week = DATE_FORMAT(current_date, '%u'), 1, 0)
OR IF(year = year(current_date - INTERVAL 7 DAY) AND week = DATE_FORMAT(current_date - INTERVAL 7 DAY, '%u'), 1, 0)
OR IF(year = year(current_date) AND month = month(current_date), 1, 0)
OR IF(year = year(current_date - INTERVAL 1 MONTH) AND month = month(current_date - INTERVAL 1 MONTH), 1, 0)
OR IF(year = year(current_date), 1, 0)
OR IF(year = year(current_date - INTERVAL 1 YEAR), 1, 0)
;
//
delimiter ;
Using p_load_dim_date you uinitially load the dim_date table with say 25 years of data. And daily, prefereabluy round midnight, you run p_update_dim_date. Then you can use the flag fields is_today, is_yesterday, is_this_week, is_last_week and so on to select common ranges. Of course, you should amend this code to suit your particular needs but this is the idea. So no generaging ranges on the fly, you just preload for a long enough period of time ahead. For the time of day, a similar design can be set up - you should be able to manage that yourself going by this code.
For even fancier date dimensions that take care of holidays, and localized names for month and days, you can take a look at:
http://rpbouman.blogspot.com/2007/04/kettle-tip-using-java-locales-for-date.html
and
http://rpbouman.blogspot.com/2010/01/easter-eggs-for-mysql-and-kettle.html
I've recently done some research to find and evaluate possible options. http://www.freeportmetrics.com/devblog/2012/11/02/how-to-quickly-add-date-dimension-to-pentaho-mondrian-olap-cube/.
You can use:
kettle
degenerated dimensions
lucidb build-in function
up-coming Mondrian built-in function
your own custom script to generate SQL
mysql script mentioned earlier
Please check the blog post for more details. It also contains improved version of Roland's sql script that will automatically calculate date range for given column and join it with date dimension.
There is no straightforward way to do that in MySQL. Your best bet is to generate a daterange array in your server-side language of choice, and then pull data from the database and merge the resulting array with your daterange array using the date as a key.
Which server side language are you using?
Edit:
Basically what you would do is (pseudocode):
// Create an array with all dates for a given range
dates = makeRange(startDate, endDate);
getData = mysqlQuery('SELECT date, x, y, z FROM a WHERE a AND b AND c');
while (r = fetchRowArray(getData)) {
dates[ date(r['date']) ] = Array ( x, y, z);
}
You end up with an array of dates you can loop through, with the dates that have or don't have activity data associated to them.
Can easily be modified to group / filter data by hours.
Try using a loop in a MySQL stored routine to create date ranges:
declare iterDate date;
set iterDate = startDate;
DROP TABLE IF EXISTS MyDates;
create temporary table MyDates (
theDate date
);
label1: LOOP
insert into MyDates(theDate) values (iterDate);
SET iterDate = DATE_ADD(iterDate, INTERVAL 1 DAY);
IF iterDate <= endDate THEN
ITERATE label1;
END IF;
LEAVE label1;
END LOOP label1;
select * from MyDates;
DROP TABLE IF EXISTS MyDates;
startDate and endDate constitute the endpoints of the range and are supplied as parameters to the routine.
I realise this is an old post but, to keep Stack Overflow a bit up-to-date, I feel the urge to respond.
With the new SEQUENCE engine in MariaDB, this is possible within a SELECT statement without any stored routine or temporary table:
SELECT
DATE_ADD(
CAST('2022-06-01' AS DATE),
INTERVAL `s1`.`seq` DAY
) AS `dates`
FROM `seq_0_to_364` AS `s1`;
Any interval will work as long as it is within the limits of BIGINT(20) UNSIGNED, as this is the limit of the SEQUENCE engine.
Related
I'm new to MySQL, I've a date variable inside my MySQL procedure and I'm trying to find week start date for that IN date variable. My procedure goes as below,
CREATE PROCEDURE my_proc (IN week_start_num INT, IN my_date DATE)
BEGIN
DECLARE my_new_date DATE;
#I know what I'm trying here is wrong
SET my_new_date=startdate - (INTERVAL WEEKDAY( startdate ) - week_start_num + IF( WEEKDAY( startdate ) > week_start_num, 0, 7 ))
#rest of my codes goes here
END
I know this is wrong 'SET my_new_date =startdate - (INTERVAL WEEKDAY( startdate ) - week_start_num + IF( WEEKDAY( startdate ) > week_start_num, 0, 7 ))', what is the correct way to accomplish it?
Try this:
SET my_new_date = startdate
- INTERVAL
(
WEEKDAY(startdate)
- week_start_num
+ IF(WEEKDAY(startdate) > week_start_num, 0, 7)
)
DAY;
I'm currently using DATE_ADD(date,INTERVAL expr type) to set a due date as a trigger in a mySQL Database.
What I'm wanting to know is if it is possible to skip weekends (Saturday, Sunday) as part of the trigger.
You'd have to create an own function for doing that. You can look how to do that in this answer, for example (just use function instead of procedure). As for how to write such a function, here's a working algorithm. The code is quite straightforward: it loops through days and skips weekends.
CREATE FUNCTION `DAYSADDNOWK`(addDate DATE, numDays INT) RETURNS date
BEGIN
IF (WEEKDAY(addDate)=5) THEN
SET addDate=DATE_ADD(addDate, INTERVAL 1 DAY);
END IF;
IF (WEEKDAY(addDate)=6) THEN
SET addDate=DATE_ADD(addDate, INTERVAL 1 DAY);
END IF;
WHILE numDays>0 DO
SET addDate=DATE_ADD(addDate, INTERVAL 1 DAY);
IF (WEEKDAY(addDate)=5) THEN
SET addDate=DATE_ADD(addDate, INTERVAL 1 DAY);
END IF;
IF (WEEKDAY(addDate)=6) THEN
SET addDate=DATE_ADD(addDate, INTERVAL 1 DAY);
END IF;
SET numDays=numDays-1;
END WHILE;
RETURN addDate;
END
Currently SELECT DAYSADDNOWK(CURDATE(), 5) yields 2016-03-07, which is correct.
Of course you only can use it with days, so no arbitrary interval, but your question mentioned date datatype, and I don't quite see how one could add a month not counting working days.
This function simply creates a list of dates starting at the date given in the arguments, and then figures out which date is x number of days (the interval) out while disregarding days 1 and 7 (which are Sunday and Saturday respectively on SQL Server).
CREATE FUNCTION [dbo].[udf_days_add_no_wknd]
(
#start_date date
, #interval int
)
RETURNS date
AS
BEGIN
declare #answer date
; with dates as
(
select #start_date as date_val
union all
select dateadd(d, 1, date_val) as date_val
from dates
where date_val < dateadd(d, #interval * 10, #start_date)
)
, final as
(
select top 1 lead(ld.date_val, #interval, NULL) over (order by ld.date_val asc) as new_date_val
from dates as ld
where 1=1
and datepart(dw, ld.date_val) not in (1,7) --eliminating weekends
)
select #answer = (select new_date_val from final)
return #answer
END
It is worth nothing that this solution is dependent on having SQL Server 2012 or later, considering the use of the lead() function.
I am developing a Java application using MySQL. I need to know which is the week of each month, of the stored dates. Is there any MySQL function for that ? Basically , if i was to use this for the current date (13.09) it would show me its in week number 2 and tomorrow it will be week number 3.
You can play with the WEEK() function, and see if it suits your needs. Here I'm using WEEK(date, 3) that will return the week of the year from 1 to 53, starting from Mondays:
set #my_date = '2015-09-13';
SELECT
WEEK(#my_date, 3) -
WEEK(#my_date - INTERVAL DAY(#my_date)-1 DAY, 3) + 1
AS week_number;
WEEK(date, 3) will return the week of the year of the selected date
WEEK(date - INTERVAL DAY(#my_date)-1 DAY, 3) will return the week of the year of the first day of the month of the selected date
It will return 1 for 01-March-2015 (because it's the first day of the month so it's week 1) and 2 for 02-March-2015 (because weeks starts from Mondays, so it's a new week). If this is not the desidered behaviour you should specify your requirements more precisely.
Please see a fiddle here.
Unfortunately, there isn't a "weekofmonth" function, but you could use dayofmonth, and manipulate the result a bit:
SELECT CURRENT_DATE(),
FLOOR((DAYOFMONTH(CURRENT_DATE()) - 1) / 7) + 1 AS week_of_month
Create a mysql function.
CREATE FUNCTION `WEEK_OF_MONTH`(
datee DATE
) RETURNS INT(11)
BEGIN
DECLARE DayNamee VARCHAR(20);
DECLARE StartDatee DATE;
DECLARE DayNumber INT DEFAULT 0;
SET DayNamee = (SELECT DAYNAME(datee));
SET StartDatee = (SELECT FIRST_DAY(datee));
WHILE StartDatee <= datee DO
IF DayNamee = DAYNAME(StartDatee) THEN
SET DayNumber = DayNumber + 1;
END IF;
SET StartDatee = DATE_ADD( StartDatee, INTERVAL 1 DAY);
END WHILE;
RETURN DayNumber;
END$$
DELIMITER ;
Call as --
SELECT `WEEK_OF_MONTH`('2018-12-31');
Result : 5
I'd like to write a query or stored procedure to retrieve less and less records over time from a relational database.
Think of this like populating the Google Finance stock chart: The past few days will have all ticks fit the day, and the further you go back, less and less ticks are displayed on each date. All ticks will show for today, 50% of ticks will show for one week ago, 30% for one month ago, and 10% for one year ago. Think of this like a gradient.
Is it possible to achieve this with one query? Or perhaps it would be necessary to use multiple queries? What might this look like?
Note that record ids are non-contiguous (there are gaps), but each record has a timestamp for determining order.
Also note that I am using MySQL.
Here is the structure of my table:
quotes
id
security_id
last_price
bid_price
ask_price
date
timestamp
trade_volume
cumulative_volume
average_volume
created_at
Sounds like you are looking for a constant set of records that represent the time-span. You can do so by defining a control date set.
Here's a sample query (doesn't account for weekends and holidays but that can be added):
POPULATE:
CREATE TABLE #quotes
(
id int identity(1,1)
,security_id VARCHAR(50)
,last_price FLOAT
,bid_price FLOAT
,ask_price FLOAT
,[date] DATETIME
,[timestamp] DATETIME
,trade_volume FLOAT
,cumulative_volume FLOAT
,average_volume FLOAT
,created_at DATETIME
)
DECLARE #i int
set #i = 100000
WHILE #i > 0
BEGIN
INSERT INTO #quotes (
security_id
,last_price
,bid_price
,ask_price
,[date]
,[timestamp]
,trade_volume
,cumulative_volume
,average_volume
,created_at
)
values( 'IBM US'
, 100.00 + RAND()
, 100.00 + RAND()
, 100.00 + RAND()
, DATEADD(MINUTE, -1* #i, GETDATE())
, DATEADD(MINUTE, -1* #i, GETDATE())
, 10000000.00 + RAND()*1000000.00
, 10000000.00 + RAND()*1000000.00
, 10000000.00 + RAND()*1000000.00
,getdate())
set #i= #i-1
END
You can change around the time span, but the following will give you around 1000 records that represent the set from start to finish.
DECLARE #StartDate DATETIME,
#EndDate DATETIME,
#j FLOAT,
#step FLOAT
set #StartDate = GETDATE()-20
SET #EndDAte = GETDATE()
set #j = 0.0
CREATE TABLE #TimeTable
(
IntervalDate DATETIME
)
--say you always want 1000 measures
--use the datediff value to define the step size:
select #step = DATEDIFF(MINUTE, #StartDate, #EndDate)/1000.0
WHILE #j < DATEDIFF(MINUTE, #StartDate, #EndDate)
BEGIN
INSERT #TimeTable (IntervalDate) VALUES (DATEADD(minute, #j, #StartDate))
SET #j = #j+#step
print #j
END
select security_id
,last_price
,bid_price
,ask_price
,[date]
,[timestamp]
,trade_volume
,cumulative_volume
,average_volume
,created_at
from #Quotes q
join #TimeTable t on dateadd(mi, datediff(mi, 0, q.date), 0) = dateadd(mi, datediff(mi, 0, t.IntervalDate), 0)
I have a SQL Server table that has a "Time" column. The table is a log table the houses status messages and timestamps for each message. The log table is inserted into via a batch file. There is an ID column that groups rows together. Each time the batch file runs it initializes the ID and writes records. What I need to do is get the elapsed time from the first record in an ID set to the last record of the same ID set. I started toying with select Max(Time) - Min(Time) from logTable where id = but couldn't figure out how to format it correctly. I need it in HH:MM:SS.
SQL Server doesn't support the SQL standard interval data type. Your best bet is to calculate the difference in seconds, and use a function to format the result. The native function CONVERT() might appear to work fine as long as your interval is less than 24 hours. But CONVERT() isn't a good solution for this.
create table test (
id integer not null,
ts datetime not null
);
insert into test values (1, '2012-01-01 08:00');
insert into test values (1, '2012-01-01 09:00');
insert into test values (1, '2012-01-01 08:30');
insert into test values (2, '2012-01-01 08:30');
insert into test values (2, '2012-01-01 10:30');
insert into test values (2, '2012-01-01 09:00');
insert into test values (3, '2012-01-01 09:00');
insert into test values (3, '2012-01-02 12:00');
Values were chosen in such a way that for
id = 1, elapsed time is 1 hour
id = 2, elapsed time is 2 hours, and
id = 3, elapsed time is 3 hours.
This SELECT statement includes one column that calculates seconds, and one that uses CONVERT() with subtraction.
select t.id,
min(ts) start_time,
max(ts) end_time,
datediff(second, min(ts),max(ts)) elapsed_sec,
convert(varchar, max(ts) - min(ts), 108) do_not_use
from test t
group by t.id;
ID START_TIME END_TIME ELAPSED_SEC DO_NOT_USE
1 January, 01 2012 08:00:00 January, 01 2012 09:00:00 3600 01:00:00
2 January, 01 2012 08:30:00 January, 01 2012 10:30:00 7200 02:00:00
3 January, 01 2012 09:00:00 January, 02 2012 12:00:00 97200 03:00:00
Note the misleading "03:00:00" for the 27-hour difference on id number 3.
Function to format elapsed time in SQL Server
UPDATED:
Correctly calculate a timespan in SQL Server, even if more than 24 hours:
-- Setup test data
declare #minDate datetime = '2012-12-12 20:16:47.160'
declare #maxDate datetime = '2012-12-13 15:10:12.050'
-- Get timespan in hh:mi:ss
select cast(
(cast(cast(#maxDate as float) - cast(#minDate as float) as int) * 24) /* hours over 24 */
+ datepart(hh, #maxDate - #minDate) /* hours */
as varchar(10))
+ ':' + right('0' + cast(datepart(mi, #maxDate - #minDate) as varchar(2)), 2) /* minutes */
+ ':' + right('0' + cast(datepart(ss, #maxDate - #minDate) as varchar(2)), 2) /* seconds */
-- Returns 18:53:24
Edge cases that show inaccuracy are especially welcome!
DECLARE #EndTime AS DATETIME, #StartTime AS DATETIME
SELECT #StartTime = '2013-03-08 08:00:00', #EndTime = '2013-03-08 08:30:00'
SELECT CAST(#EndTime - #StartTime AS TIME)
Result: 00:30:00.0000000
Format result as you see fit.
The best and simple way:
Convert(varchar, {EndTime} - {StartTime}, 108)
Just like Anri noted.
Use the DATEDIFF to return value in milliseconds, seconds, minutes, hours, ...
DATEDIFF(interval, date1, date2)
interval REQUIRED - The time/date part to return. Can be one of the following values:
year, yyyy, yy = Year
quarter, qq, q = Quarter
month, mm, m = month
dayofyear = Day of the year
day, dy, y = Day
week, ww, wk = Week
weekday, dw, w = Weekday
hour, hh = hour
minute, mi, n = Minute
second, ss, s = Second
millisecond, ms = Millisecond
date1, date2 REQUIRED - The two dates to calculate the difference between
select convert(varchar, Max(Time) - Min(Time) , 108) from logTable where id=...
See if this helps. I can set variables for Elapsed Days, Hours, Minutes, Seconds.
You can format this to your liking or include in a user defined function.
Note: Don't use DateDiff(hh,#Date1,#Date2). It is not reliable! It rounds in unpredictable ways
Given two dates...
(Sample Dates: two days, three hours, 10 minutes, 30 seconds difference)
declare #Date1 datetime = '2013-03-08 08:00:00'
declare #Date2 datetime = '2013-03-10 11:10:30'
declare #Days decimal
declare #Hours decimal
declare #Minutes decimal
declare #Seconds decimal
select #Days = DATEDIFF(ss,#Date1,#Date2)/60/60/24 --Days
declare #RemainderDate as datetime = #Date2 - #Days
select #Hours = datediff(ss, #Date1, #RemainderDate)/60/60 --Hours
set #RemainderDate = #RemainderDate - (#Hours/24.0)
select #Minutes = datediff(ss, #Date1, #RemainderDate)/60 --Minutes
set #RemainderDate = #RemainderDate - (#Minutes/24.0/60)
select #Seconds = DATEDIFF(SS, #Date1, #RemainderDate)
select #Days as ElapsedDays, #Hours as ElapsedHours, #Minutes as ElapsedMinutes, #Seconds as ElapsedSeconds
Hope this helps you in getting the exact time between two time stamps
Create PROC TimeDurationbetween2times(#iTime as time,#oTime as time)
As
Begin
DECLARE #Dh int, #Dm int, #Ds int ,#Im int, #Om int, #Is int,#Os int
SET #Im=DATEPART(MI,#iTime)
SET #Om=DATEPART(MI,#oTime)
SET #Is=DATEPART(SS,#iTime)
SET #Os=DATEPART(SS,#oTime)
SET #Dh=DATEDIFF(hh,#iTime,#oTime)
SET #Dm = DATEDIFF(mi,#iTime,#oTime)
SET #Ds = DATEDIFF(ss,#iTime,#oTime)
DECLARE #HH as int, #MI as int, #SS as int
if(#Im>#Om)
begin
SET #Dh=#Dh-1
end
if(#Is>#Os)
begin
SET #Dm=#Dm-1
end
SET #HH = #Dh
SET #MI = #Dm-(60*#HH)
SET #SS = #Ds-(60*#Dm)
DECLARE #hrsWkd as varchar(8)
SET #hrsWkd = cast(#HH as char(2))+':'+cast(#MI as char(2))+':'+cast(#SS as char(2))
select #hrsWkd as TimeDuration
End