I am working with MySQL database and I have to generate date intervals for specified period(specified by start and stop date) with specified step(for example one day).
I have written a stored procedure to generate intervals, to create a temporary table and to populate this table with intervals.
DELIMITER $$
CREATE PROCEDURE showu(IN start date, IN stop date)
BEGIN
CREATE TEMPORARY TABLE intervals(single_day DATE);
next_date: LOOP
IF start>stop THEN
LEAVE next_date;
END IF;
INSERT INTO intervals(single_day) VALUES(start);
SET start = DATE_ADD(start, INTERVAL 1 DAY);
END LOOP next_date;
END$$
DELIMITER ;
I want to use this temporary table in join queries. However I faced with a problem. When I call procedure call showu('2008-01-09', '2010-02-09'); it is executing approximately 30 seconds. The question why it is executing so long? Is it possible to improve it? If this solution is wrong how can I resolve my problem in different way?
From comments:
2 big problems: 1. I don't know exactly value of step(one day or one month or one hour).
Create one big table like this once (not temporary):
full_date | year | month | day | full_time | hour | minute | is_weekend | whatever
----------------------------------------------------------------------------------
...
Create as much indexes as needed and you will have a very performant a powerful swiss knife for all sorts of reports.
Note: You might consider not having time and date in the same table. This is just to simplify the example.
Your second problem
I will clog my database with not model data.
is no problem. Databases are there to hold data. That's it. If you have problems with space or whatever, the solution is to get more space, not to limit your ability to work efficiently.
That being said, here's some examples how to use this table.
You need dates:
SELECT
*
FROM
(
SELECT full_date AS your_step
FROM your_new_swiss_army_knife
WHERE `year` = 2012
GROUP BY full_date
) dates
LEFT JOIN your_tables_that_you_want_to_build_a_report_on y ON dates.your_step = y.date
Same with months:
SELECT
*
FROM
(
SELECT CONCAT(year, '-', month) AS your_step
FROM your_new_swiss_army_knife
WHERE full_date BETWEEN this AND that
GROUP BY year, month
) dates
LEFT JOIN your_tables_that_you_want_to_build_a_report_on y ON dates.your_step = CONCAT(YEAR(y.date), '-', MONTH(y.date))
Related
I'm trying create an SQL query to resolve my problem.
I use mysqk5.7.
My Table:
|project_id|start |end |cost(per month)|
|1 |2018-05-01|2018-06-30|1000 |
|2 |2018-06-01|2018-07-31|2000 |
I want to generate date-columns by start and end columns.
like this:
|date |project_id|cost|
|2018-05|1 |1000|
|2018-06|1 |1000|
|2018-06|2 |2000|
|2018-07|2 |2000|
Create a table and populate it with first day of each month. You can programmatically do that or even use Excel to generate data and port it to MySQL.
create table dates (
start_date date
);
insert into dates values
('2018-04-01'),
('2018-05-01'),
('2018-06-01'),
('2018-07-01'),
('2018-8-01');
Then, you can run a query like so:
Query
select
date_format(start_date, '%Y-%m') as `Date`,
a.project_id,
a.cost
from projects a
inner join dates b on b.start_date between a.start and a.end;
Result
Date project_id cost
2018-05 1 1000
2018-06 1 1000
2018-06 2 2000
2018-07 2 2000
Example
http://rextester.com/JRIUZ98116
Alternative
The other alternative is to create a stored procedure that creates a temporary table containing dates so that you don't have to generate a table. Minimum start date and maximum end date from the table can be extracted to create the temporary table of dates.
Then, the stored procedure can do the same join as above to generate a resultset.
This is one of those places where a separate date table will make life much easier. If you have a table with something like this:
create table DateTable(ThisDate date, Month varchar(7))
adding whatever other columns you might need (isWeekday etc) and populate it in a loop then you will be able to re-use this for all sorts of things, including this query. For example you can create a view on it to get month, startdate, enddate, and then join from that back into your table looking for dates that are between the start and end date.
This and lots of other queries will become simple.
create table DateTable(ThisDate date, Month varchar(7))
--- populate the table just once, re-use for all future queries
create view MonthView as
select Month,
min(ThisDate) as StartOfMonth,
max(ThisDate) as EndOfMonth
from DateTable
select Month, ProjectID, Cost
from MonthView
join MyTab on myTab.Start<=EndOfmonth and myTab.End>=StartofMonth
I need to get results month by month for the last 3 years. This is what I have:
CREATE DEFINER=`root`#`localhost` PROCEDURE `by_year`()
BEGIN
DROP TEMPORARY TABLE IF EXISTS aaa;
SET #year = 2014;
WHILE #year <= YEAR(CURDATE()) DO
CREATE TEMPORARY TABLE IF NOT EXISTS aaa (Año int, Registrados int, Activos int, Compraron int, Recurrentes int);
INSERT INTO aaa
SELECT #year, registrados.count, activos.count, compraron.count, recurrentes.count FROM
# Here is my query to get the data. With a WHERE that limits the creation date by #year. [YEAR(AU.created) <= #year] For example
END WHILE;
SELECT * FROM aaa;
END
It works for years. But I need to get the result by months or even weeks and days.
My problem is if I replace the YEAR function with MONTH or DAY These functions will return a number taken from the date literally (relative to the bigger divider)
For example from
MONTH('2015-11-25')
I get
11
So MONTH('2015-11-25') is equal to MONTH('2016-11-02')
I see two ways... Either I find a function or something like:
MONTH_FROM_THE_BEGGINING_OF_TIMES('2016-11-02')
That similarly to dates handled by TIME, returns the month number in general in this case from year 1 that should be somenthing like (2015 * 12) + 11 = 24191
The other way could be playing around with double WHILE. And then for days triple WHILE. This will produce a complexity of O(n^3) which is not good.
Any ideas are very welcome.
Thank you
You might like MySQL's EXTRACT() function:
mysql> SELECT EXTRACT(YEAR_MONTH FROM CURDATE()) AS YM;
+--------+
| YM |
+--------+
| 201608 |
+--------+
If you want days, you might like TO_DAYS():
mysql> SELECT TO_DAYS(CURDATE()) AS days;
+--------+
| days |
+--------+
| 736564 |
+--------+
You cant not replace the function YEAR, you just need to add the function MONTH to get the result by months with the logical AND operator. Also in your query to get the data you have to add the AND part for the month at the group by.
It is the only way to get the month number relative to the year.
I have a log table with a date field called logTime. I need to show the number of rows within a date range and the number of records per day. The issue is that i still want to show days that do not have records.
Is it possible to do this only with SQL?
Example:
SELECT logTime, COUNT(*) FROM logs WHERE logTime >= '2011-02-01' AND logTime <= '2011-02-04' GROUP BY DATE(logTime);
It returns something like this:
+---------------------+----------+
| logTime | COUNT(*) |
+---------------------+----------+
| 2011-02-01 | 2 |
| 2011-02-02 | 1 |
| 2011-02-04 | 5 |
+---------------------+----------+
3 rows in set (0,00 sec)
I would like to show the day 2011-02-03 too.
MySQL will not invent rows for you, so if the data is not there, they will naturally not be shown.
You can create a calendar table, and join in that,
create table calendar (
day date primary key,
);
Fill this table with dates (easy with a stored procedure, or just some general scripting), up till around 2038 and something else will likely break unitl that becomes a problem.
Your query then becomes e.g.
SELECT logTime, COUNT(*)
FROM calendar cal left join logs l on cal.day = l.logTime
WHERE day >= '2011-02-01' AND day <= '2011-02-04' GROUP BY day;
Now, you could extend the calendar table with other columns that tells you the month,year, week etc. so you can easily produce statistics for other time units. (and purists might argue the calendar table would have an id integer primary key that the logs table references instead of a date)
In order to accomplish this, you need to have a table (or derived table) which contains the dates that you can then join from, using a LEFT JOIN.
SQL operates on the concept of mathematical sets, and if you don't have a set of data, there is nothing to SELECT.
If you want more details, please comment accordingly.
I'm not sure if this is a problem that should be solved by SQL. As others have shown, this requires maintaining a second table that contains the all of the individual dates of a given time span, which must be updated every time that time span grows (which presumably is "always" if that time span is the current time.
Instead, you should use to inspect the results of the query and inject dates as necessary. It's completely dynamic and requires no intermediate table. Since you specified no language, here's pseudo code:
EXECUTE QUERY `SELECT logTime, COUNT(*) FROM logs WHERE logTime >= '2011-02-01' AND logTime <= '2011-02-04' GROUP BY DATE(logTime);`
FOREACH row IN query result
WHILE (date in next row) - (date in this row) > 1 day THEN
CREATE new row with date = `date in this row + 1 day`, count = `0`
INSERT new row IN query result AFTER this row
ADVANCE LOOP INDEX TO new row (`this row` is now the `new row`)
END WHILE
END FOREACH
Or something like that
DECLARE #TOTALCount INT
DECLARE #FromDate DateTime = GetDate() - 5
DECLARE #ToDate DateTime = GetDate()
SET #FromDate = DATEADD(DAY,-1,#FromDate)
Select #TOTALCount= DATEDIFF(DD,#FromDate,#ToDate);
WITH d AS
(
SELECT top (#TOTALCount) AllDays = DATEADD(DAY, ROW_NUMBER()
OVER (ORDER BY object_id), REPLACE(#FromDate,'-',''))
FROM sys.all_objects
)
SELECT AllDays From d
So lets say I have some records that look like:
2011-01-01 Cat
2011-01-02 Dog
2011-01-04 Horse
2011-01-06 Lion
How can I construct a query that will return 2011-01-03 and 2011-01-05, ie the unused dates. I postdate blogs into the future and I want a query that will show me the days I don't have anything posted yet. It would look from the current date to 2 weeks into the future.
Update:
I am not too excited about building a permanent table of dates. After thinking about it though it seems like the solution might be to make a small stored procedure that creates a temp table. Something like:
CREATE PROCEDURE MISSING_DATES()
BEGIN
CREATE TABLE TEMPORARY DATES (FUTURE DATETIME NULL)
INSERT INTO DATES (FUTURE) VALUES (CURDATE())
INSERT INTO DATES (FUTURE) VALUES (ADDDATE(CURDATE(), INTERVAL 1 DAY))
...
INSERT INTO DATES (FUTURE) VALUES (ADDDATE(CURDATE(), INTERVAL 14 DAY))
SELECT FUTURE FROM DATES WHERE FUTURE NOT IN (SELECT POSTDATE FROM POSTS)
DROP TABLE TEMPORARY DATES
END
I guess it just isn't possible to select the absence of data.
You're right — SQL does not make it easy to identify missing data. The usual technique is to join your sequence (with gaps) against a complete sequence, and select those elements in the latter sequence without a corresponding partner in your data.
So, #BenHoffstein's suggestion to maintain a permanent date table is a good one.
Short of that, you can dynamically create that date range with an integers table. Assuming the integers table has a column i with numbers at least 0 – 13, and that your table has its date column named datestamp:
SELECT candidate_date AS missing
FROM (SELECT CURRENT_DATE + INTERVAL i DAY AS candidate_date
FROM integers
WHERE i < 14) AS next_two_weeks
LEFT JOIN my_table ON candidate_date = datestamp
WHERE datestamp is NULL;
One solution would be to create a separate table with one column to hold all dates from now until eternity (or whenever you expect to stop blogging). For example:
CREATE TABLE Dates (dt DATE);
INSERT INTO Dates VALUES ('2011-01-01');
INSERT INTO Dates VALUES ('2011-01-02');
...etc...
INSERT INTO Dates VALUES ('2099-12-31');
Once this reference table is set up, you can simply outer join to determine the unused dates like so:
SELECT d.dt
FROM Dates d LEFT JOIN Blogs b ON d.dt = b.dt
WHERE b.dt IS NULL
If you want to limit the search to two weeks in the future, you could add this to the WHERE clause:
AND d.dt BETWEEN NOW() AND ADDDATE(NOW(), INTERVAL 14 DAY)
The way to extract rows from the mysql database is via SELECT. Thus you cannot select rows that do not exist.
What I would do is fill my blog table with all possible dates (for a year, then repeat the process)
create table blog (
thedate date not null,
thetext text null,
primary key (thedate));
doing a loop to create all dates entries for 2011 (using a program, eg $mydate is the date you want to insert)
insert IGNORE into blog (thedate,thetext) values ($mydate, null);
(the IGNORE keyword to not create an error (thedate is a primary key) if thedate exists already).
Then you insert the values normally
insert into blog (thedate,thetext) values ($mydate, "newtext")
on duplicate key update thetext="newtext";
Finally to select empty entries, you just have to
select thedate from blog where thetext is null;
You probably not going to like this:
select '2011-01-03', count(*) from TABLE where postdate='2011-01-03'
having count(*)=0 union
select '2011-01-04', count(*) from TABLE where postdate='2011-01-04'
having count(*)=0 union
select '2011-01-05', count(*) from TABLE where postdate='2011-01-05'
having count(*)=0 union
... repeat for 2 weeks
OR
create a table with all days in 2011, then do a left join, like
select a.days_2011
from all_days_2011
left join TABLE on a.days_2011=TABLE.postdate
where a.days_2011 between date(now()) and date(date_add(now(), interval 2 week))
and TABLE.postdate is null;
Is there an easy way to do a GROUP BY DATE(timestamp) that includes all days in a period of time, regardless of whether there are any records associated with that date?
Basically, I need to generate a report like this:
24 Dec - 0 orders
23 Dec - 10 orders
22 Dec - 8 orders
21 Dec - 2 orders
20 Dec - 0 orders
Assuming you have more orders than dates something like this could work:
select date, count(id) as orders
from
(
SELECT DATE_ADD('2008-01-01', INTERVAL #rn:=#rn+1 DAY) as date from (select #rn:=-1)t, `order` limit 365
) d left outer join `order` using (date)
group by date
One method is to create a calendar table and join against it.
I would create it permanently, and then create a task that will insert new dates, it could be done weekly, daily, monthly, etc.
Note, that I am assuming that you are converting your timestamp into a date.
Instead of using GROUP BY, make a table (perhaps a temporary table) which contains the specific dates you want, for example:
24 Dec
23 Dec
22 Dec
21 Dec
20 Dec
Then, join that table to the Orders table.
you need to generate an intermediate result set with all the dates in it that you want included in the output...
if you're doing this in a stored proc, then you could create a temp table or table variable (I don't knoiw MySQL's capabilities), but once you have all the dates in a table or resultset of some kind
Just join to the real dataa from the temp table, using an outer join
In SQL Server it would be like this
Declare #Dates Table (aDate DateTime Not Null)
Declare #StartDt DateTime Set #StartDt = 'Dec 1 2008'
Declare #EndDt DateTime Set #EndDt = 'Dec 31 2008'
While #StartDt < #EndDt Begin
Insert #Dates(aDate) Values(#StartDt)
Set #StartDt = DateAdd(Day, 1, #StartDt)
End
Select D.aDate, Count(O.*) Orders
From #Dates D Left Join
OrderTable O On O.OrderDate = D.aDate
Group By D.aDate
In a data warehouse, the method taken is to create a table that contains all dates and create a foreign key between your data and the date table. I'm not saying that this is the best way to go in your case, just that it is the best practice in cases where large amounts of data need to be rolled up in numerous ways for reporting purposes.
If you are using a reporting layer over SQL Server, you could just write some logic to insert the missing dates within the range of interest after the data returns and before rendering your report.
If you are creating your reports directly from SQL Server and you do not already have a data warehouse and there isn't the time or need to create one right now, I would create a date table and join to it. The formatting necessary to do the join and get the output you want may be a bit wonky, but it will get the job done.
There's a pretty straightforward way to do this… except that I can't remember it. But I adapted this query from this thread:
SELECT
DISTINCT(LEFT(date_field,11)) AS `Date`,
COUNT(LEFT(date_field,11)) AS `Number of events`
FROM events_table
GROUP BY `Date`
It works in MySQL too