I have some MAU data with monthly granularity like following:
I want to get yoy date like:
2016-04 yoy MAU = (2016-04 MAU - 2015-04 MAU)/2015-04 MAU
2016-03 yoy MAU = (2016-03 MAU - 2015-03 MAU)/2015-03 MAU
...
I am using MySQL so I don't have window function.
Is there any easier way to achieve what I want?
*The data type:
Period | varchar(16)
MAU | double
Ideally, periodic data would be stored as a date or as its components, in this case, year and month as separate integer columns. Since they are represented together as a string here, we need to parse them using SUBSTRING. The general idea behind the solution is that we need to join the table to itself using an ON clause that "lines up" each row with it's counterpart from one year prior.
This ought to do the trick:
select a.Period, a.MAU, b.Period as prior_year, b.MAU as prior_mau,
(a.MAU - b.MAU) / b.MAU as yoy
from tbl1 a
left join tbl1 b on
cast(substring(b.Period, 1, 4) as int) =
cast(substring(a.Period, 1, 4) as int) - 1
and substring(b.Period, 5, 3) = substring(a.Period, 5, 3)
order by a.Period asc
Here's a second way that's slightly less clear to read but is more efficient (especially if speed is an issue, the table is large, and it's indexed on Period) because it avoids a calculation in the where clause (at least on one side):
select a.Period, a.MAU, b.Period as prior_year, b.MAU as prior_mau,
(a.MAU - b.MAU) / b.MAU as yoy
from tbl1 a
left join tbl1 b on
a.Period
= cast(cast(substring(b.Period, 1, 4) as int) - 1 as varchar(4))
+ substring(b.Period, 5, 3)
order by a.Period asc
I hope this helps.
Related
We have a statistics database of which we would like to group some results. Every entry has a timestamp 'tstarted'.
We would like to group by every quarter of the day. For each quarter, we would like to know the day count where we have > 0 results (for that quarter).
We could resolve this by using a subquery:
select quarter, sum(q), count(quarter), sum(q) / count(quarter) as average
from (
select SEC_TO_TIME((TIME_TO_SEC(tstarted) DIV 900) * 900) as quarter, sum(qdelivered) as q
from statistics
where stat_field = 1
group by SEC_TO_TIME((TIME_TO_SEC(tstarted) DIV 900) * 900), date(tstarted)
order by SEC_TO_TIME((TIME_TO_SEC(tstarted) DIV 900) * 900) asc
) as sub
group by quarter
My question: is there a more efficient way to retrieve this result (e.g. join or other way)?
Efficiency could be improved by eliminating the inline view (derived table aliased as sub), and doing all the work in a single query. (This is because of the way that MySQL processes the inline view, creating and populating a temporary MyISAM table.)
I don't understand why the expression date(tstarted) needs to be included in the GROUP BY clause; I don't see that removing that would change the result set returned by the query.
I do now see the effect of including the date(tstarted) in the GROUP BY of the inline view query.
I think this query returns the same result as the original:
SELECT SEC_TO_TIME((TIME_TO_SEC(s.tstarted) DIV 900) * 900) AS `quarter`
, SUM(s.qdelivered) AS `q`
, COUNT(DISTINCT DATE(s.tstarted)) AS `day_count`
, SUM(s.qdelivered) / COUNT(DISTINCT DATE(s.tstarted)) AS `average`
FROM statistics s
WHERE s.stat_field = 1
GROUP BY SEC_TO_TIME((TIME_TO_SEC(s.tstarted) DIV 900) * 900)
This should be more efficient since it avoids materializing an intermediate derived table.
Your question said you wanted a "day count"; that sounds like you want a count of the each day that had a row within a particular quarter hour.
To get that, you could just add an aggregate expression to the SELECT list,
, COUNT(DISTINCT DATE(s.tstarted)) AS `day_count`
I would be tempted to set up a table of quarters in the day. Use this table and LEFT JOIN your statistics table it.
CREATE TABLE quarters
(
id INT,
start_qtr INT,
end_qtr INT
);
INSERT INTO quarters (id, start_qtr, end_qtr) VALUES
(1,0,899),
(2,900,1799),
(3,1800,2699),
(4,2700,3599),
(5,3600,4499),
(6,4500,5399),
(7,5400,6299),
(8,6300,7199),
etc;
Your query can then be:-
SELECT SEC_TO_TIME(quarters.start_qtr) AS quarter,
sum(statistics.qdelivered),
count(statistics.qdelivered),
sum(statistics.qdelivered) / count(statistics.qdelivered) as average
FROM quarters
LEFT OUTER JOIN statistics
ON TIME_TO_SEC(statistics.tstarted) BETWEEN quarters.start_qtr AND quarters.end_qtr
AND statistics.stat_field = 1
AND DATE(statistics.tstarted) = '2014-06-30'
GROUP BY quarter
ORDER BY quarter;
Advantage of this is that it will give you entries with a count of 0 (and an average of NULL) for quarters where there are no statistics, and it saves some of the calculations.
You could save more calculations by adding time columns to the quarters table:-
CREATE TABLE quarters
(
id INT,
start_qtr INT,
end_qtr INT
start_qtr_time TIME,
end_qtr_time TIME,
);
INSERT INTO quarters (id, start_qtr, end_qtr, start_qtr_time, end_qtr_time) VALUES
(1,0,899, '00:00:00', '00:14:59'),
(2,900,1799, '00:15:00', '00:29:59'),
(3,1800,2699, '00:30:00', '00:44:59'),
(4,2700,3599, '00:45:00', '00:59:59'),
(5,3600,4499, '01:00:00', '01:14:59'),
(6,4500,5399, '01:15:00', '01:29:59'),
(7,5400,6299, '01:30:00', '01:44:59'),
(8,6300,7199, '01:45:00', '01:59:59'),
etc
Then this saves the use of a function on the JOIN:-
SELECT start_qtr_time AS quarter,
sum(statistics.qdelivered),
count(statistics.qdelivered),
sum(statistics.qdelivered) / count(statistics.qdelivered) as average
FROM quarters
LEFT OUTER JOIN statistics
ON TIME(statistics.tstarted) BETWEEN quarters.start_qtr_time AND quarters.end_qtr_time
AND statistics.stat_field = 1
AND DATE(statistics.tstarted) = '2014-06-30'
GROUP BY quarter
ORDER BY quarter;
These both assume you are interested in a particular day.
SELECT c.siteno, a.sitename, a.location, Count(a.status) AS ChargeablePermit
FROM (PermitStatus AS a LEFT JOIN states AS b ON a.status = b.statusheading)
LEFT JOIN Sitedetails AS c ON a.zone = c.compexzone
WHERE b.statusheading like "Chargeable" and a.loaded_date between
(select monthstart from ChargeDate) and (select Monthend from ChargeDate)
GROUP BY a.sitename, c.siteno, a.location;
This query returns me the count of chargeable permits by site
Mar14
Siteno (1) Sitename (site1) Location (location1) Chargeablepermit (30)
these calculations are based on the period determined by the two sub selects (i.e. for the month of March 14)
i was wondering if i could change the date range covered by the subselects (i.e.to April 14) and do math on (subtract one count from the other) the counts of chargeable permits from the two different result sets and have that result displayed on the on one table
for instance if April 14 was
April
Siteno (1) Sitename (Site1) Location (Location1) ChargeablePermit (40) Difference (10)
Not in the way it seems you are proposing, you would simply double-up your SQL within a UNION query to return the data sets for the 2 periods, and then perform an aggregate on the results:
SELECT SUM(CP) FROM (
SELECT (ChargeablePermit * -1) AS CP FROM ... WHERE dates = Date1
UNION ALL
SELECT ChargeablePermit AS CP FROM ... WHERE dates = Date2
)
Depending on how many records you're dealing with, a UNION like this could be quite slow however. So the other approach would be to turn your SQL into an Append query which inserts the output into a temp table. You would run the query for each period, before running a 2nd query to aggregate the results from the temp table.
Also you should consider using joins to filter your results rather than subqueries.
I have a report that displays a graph. The X axis uses the date from the below query. Where the query returns no date, I am getting gaps and would prefer to return a value. Is there any way to force a date where there are no records?
SELECT
DATE(instime),
CASE
WHEN direction = 1 AND duration > 0 THEN 'Incoming'
WHEN direction = 2 THEN 'Outgoing'
WHEN direction = 1 AND duration = 0 THEN 'Missed'
END AS type,
COUNT(*)
FROM taxticketitem
GROUP BY
DATE(instime),
CASE
WHEN direction = 1 AND duration > 0 THEN 'Incoming'
WHEN direction = 2 THEN 'Outgoing'
WHEN direction = 1 AND duration = 0 THEN 'Missed'
END
ORDER BY DATE(instime)
One possible way is to create a table of dates and LEFT JOIN your table with them. The table could look something like this:
CREATE TABLE `datelist` (
`date` DATE NOT NULL,
PRIMARY KEY (`date`)
);
and filled with all dates between, say Jan-01-2000 through Dec-31-2050 (here is my Date Generator script).
Next, write your query like this:
SELECT datelist.date, COUNT(taxticketitem.id) AS c
FROM datelist
LEFT JOIN taxticketitem ON datelist.date = DATE(taxticketitem.instime)
WHERE datelist.date BETWEEN `2012-01-01` AND `2012-12-31`
GROUP BY datelist.date
ORDER BY datelist.date
LEFT JOIN and counting not null values from right table's ensures that the count is correct (0 if no row exists for a given date).
You would need to have a set of dates to LEFT JOIN your table to it. Unfortunately, MySQL lacks a way to generate it on the fly.
You would need to prepare a table with, say, 100000 consecutive integers from 0 to 99999 (or how long you think your maximum report range would be):
CREATE TABLE series (number INT NOT NULL PRIMARY KEY);
and use it like this:
SELECT DATE(instime) AS r_date, CASE ... END AS type, COUNT(instime)
FROM series s
LEFT JOIN
taxticketitems ti
ON ti.instime >= '2013-01-01' + INTERVAL number DAY
AND ti.instime < '2013-01-01' + INTERVAL number + 1 DAY
WHERE s.number <= DATEDIFF('2013-02-01', '2013-01-01')
GROUP BY
r_date, type
Had to do something similar before.
You need to have a subselect to generate a range of dates. All the dates you want. Easiest with a start date added to a number:-
SELECT DATE_ADD(SomeStartDate, INTERVAL (a.I + b.1 * 10) DAY)
FROM integers a, integers b
Given a table called integers with a single column called i with 10 rows containing 0 to 9 that SQL will give you a range of 100 days starting at SomeStartDate
You can then left join your actual data against that to get the full range.
I have a table recording the accumulative total visit numbers of some web pages every day. I want to fetch the real visit numbers in a specific day for all these pages. the table is like
- record_id page_id date addup_number
- 1 1 2012-9-20 2110
- 2 2 2012-9-20 1160
- ... ... ... ...
- n 1 2012-9-21 2543
- n+1 2 2012-9-21 1784
the result I'd like to fetch is like:
- page_id date increment_num(the real visit numbers on this date)
- 1 2012-9-21 X
- 2 2012-9-21 X
- ... ... ...
- N 2012-9-21 X
but I don't want to do this in php, cause it's time consuming. Can I get what I want with SQL directives or with some mysql functions?
Ok. You need to join the table on itself by joining on the date column and adding a day to one side of the join.
Assuming:
date column is a legitimate DATE Type and not a string
Every day is accounted for each page (no gaps)
addup_number is an INT of some type (BIGINT, INT, SMALLINT, etc...)
table_name is substituted for your actual table name which you don't indicate
Only one record per day for each page... i.e. no pages have multiple counts on the same day
You can do this:
SELECT t2.page_id, t2.date, t2.addup_number - t1.addup_number AS increment_num
FROM table_name t1
JOIN table_name t2 ON t1.date + INTERVAL 1 DAY = t2.date
WHERE t1.page_id = t2.page_id
One thing to note is if this is a huge table and date is an indexed column, you'll suffer on the join by having to transform it by adding a day in the ON clause, but you'll get your data.
UPDATED:
SELECT today.page_id, today.date, (today.addup_number - yesterday.addup_number) as increment
FROM myvisits_table today, myvisits_table yesterday
WHERE today.page_id = yesterday.page_id
AND today.date='2012-9-21'
AND yesterday.date='2012-9-20'
GROUP BY today.page_id, today.date, yesterday.page_id, yesterday.date
ORDER BY page_id
Something like this:
SELECT date, SUM(addup_number)
FROM your_table
GROUP BY date
Assume this table:
id date
----------------
1 2010-12-12
2 2010-12-13
3 2010-12-18
4 2010-12-22
5 2010-12-23
How do I find the average intervals between these dates, using MySQL queries only?
For instance, the calculation on this table will be
(
( 2010-12-13 - 2010-12-12 )
+ ( 2010-12-18 - 2010-12-13 )
+ ( 2010-12-22 - 2010-12-18 )
+ ( 2010-12-23 - 2010-12-22 )
) / 4
----------------------------------
= ( 1 DAY + 5 DAY + 4 DAY + 1 DAY ) / 4
= 2.75 DAY
Intuitively, what you are asking should be equivalent to the interval between the first and last dates, divided by the number of dates minus 1.
Let me explain more thoroughly. Imagine the dates are points on a line (+ are dates present, - are dates missing, the first date is the 12th, and I changed the last date to Dec 24th for illustration purposes):
++----+---+-+
Now, what you really want to do, is evenly space your dates out between these lines, and find how long it is between each of them:
+--+--+--+--+
To do that, you simply take the number of days between the last and first days, in this case 24 - 12 = 12, and divide it by the number of intervals you have to space out, in this case 4: 12 / 4 = 3.
With a MySQL query
SELECT DATEDIFF(MAX(dt), MIN(dt)) / (COUNT(dt) - 1) FROM a;
This works on this table (with your values it returns 2.75):
CREATE TABLE IF NOT EXISTS `a` (
`dt` date NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
INSERT INTO `a` (`dt`) VALUES
('2010-12-12'),
('2010-12-13'),
('2010-12-18'),
('2010-12-22'),
('2010-12-24');
If the ids are uniformly incremented without gaps, join the table to itself on id+1:
SELECT d.id, d.date, n.date, datediff(d.date, n.date)
FROM dates d
JOIN dates n ON(n.id = d.id + 1)
Then GROUP BY and average as needed.
If the ids are not uniform, do an inner query to assign ordered ids first.
I guess you'll also need to add a subquery to get the total number of rows.
Alternatively
Create an aggregate function that keeps track of the previous date, and a running sum and count. You'll still need to select from a subquery to force the ordering by date (actually, I'm not sure if that's guaranteed in MySQL).
Come to think of it, this is a much better way of doing it.
And Even Simpler
Just noting that Vegard's solution is much better.
The following query returns correct result
SELECT AVG(
DATEDIFF(i.date, (SELECT MAX(date)
FROM intervals WHERE date < i.date)
)
)
FROM intervals i
but it runs a dependent subquery which might be really inefficient with no index and on a larger number of rows.
You need to do self join and get differences using DATEDIFF function and get average.