MySQL SELECT Query to Turn History into Weekly Summary Over Time - mysql

Question

I have a history table ('property_histories') that logs events in our property management system. These events can be used to determine whether a given property was available to rent and I am trying to build a (weekly) summary of 'live' properties.
The 4 events in question are 'published', 'unpublished', 'hidden_from_search' and 'unhidden_from_search.
For a property to be live it must have been:
Published.
If it has ever been unpublished a subsequent published event mush be the most recent.
If it has ever been hidden_from_search a subsequent 'unhidden_from_search' event must have taken place more recently.
Most properties will have a simple history that most likely consists of a single 'Published' event but some are more complicated an example is here:
property_histories
----------------------------
id | property_id | City | status | date
1 | 325407 | Paris | published | 2014-01-01
2 | 325407 | Paris | hidden_from_search | 2014-01-24
3 | 325407 | Paris | unhidden_from_search | 2014-02-05
4 | 325407 | Paris | unpublished | 2014-02-15
5 | 410008 | London | published | 2014-01-01
6 | 410008 | London | unpublished | 2014-01-10
7 | 410008 | London | published | 2014-01-18
My aim is to be able to count 'live' properties by week:
weekly_count
----------------------------
Year | Week | City | Live_Count
2014 | 1 | Paris | 0
2014 | 1 | London | 0
2014 | 2 | Paris | 1
2014 | 2 | London | 1
2014 | 3 | Paris | 1
2014 | 3 | London | 0
2014 | 4 | Paris | 1
2014 | 4 | London | 1
2014 | 5 | Paris | 0
2014 | 5 | London | 1
2014 | 6 | Paris | 0
2014 | 6 | London | 1
2014 | 7 | Paris | 1
2014 | 7 | London | 0
2014 | 8 | Paris | 0
2014 | 8 | London | 1
2014 | 9 | Paris | 0
2014 | 9 | London | 1
----------------------------
Help appreciated!!

Answer 1

Your own test results don't match what you're asking for. You state the live count is by week, which means London should be live in week #1 as it was published in week #1 and then unpublished in week #2.
Assuming week starts on a Sunday (sql default) then this will work. Just put in your own date range, and replace my numbers table with yours.
If you need Monday to be your start date, use this at the top of your query
SET DATEFIRST 1
Emulating your test:
-- Create dummy data
CREATE TABLE #property_histories
(
id int, property_id int, City varchar(50), status varchar(50), date date
)
INSERT INTO #property_histories
SELECT 1 , 325407 , 'Paris' , 'published' , '2014-01-01' UNION ALL
SELECT 2 , 325407 , 'Paris' , 'hidden_from_search' , '2014-01-24' UNION ALL
SELECT 3 , 325407 , 'Paris' , 'unhidden_from_search' , '2014-02-05' UNION ALL
SELECT 4 , 325407 , 'Paris' , 'unpublished' , '2014-02-15' UNION ALL
SELECT 5 , 410008 , 'London' , 'published' , '2014-01-01' UNION ALL
SELECT 6 , 410008 , 'London' , 'unpublished' , '2014-01-10' UNION ALL
SELECT 7 , 410008 , 'London' , 'published' , '2014-01-18'
Now the code:
-- TODO: Set your date range
DECLARE #SD Datetime = '2014-01-01'
DECLARE #ED Datetime = '2014-12-31'
DECLARE #Wks INT = Datediff(week,#SD,#ED) -- Don't change this
-- Generate dates table
SELECT NumberID as 'Week',
DATEADD(DAY, 1-DATEPART(WEEKDAY, DateAdd(week,NumberID-1,#SD)), DateAdd(week,NumberID-1,#SD)) as 'WeekStart',
DATEADD(DAY, 7-DATEPART(WEEKDAY, DateAdd(week,NumberID-1,#SD)), DateAdd(week,NumberID-1,#SD)) as 'WeekEnd'
INTO #Dates
FROM Generic.tblNumbers -- TODO: use your own Numbers table here
WHERE NumberID BETWEEN 1 AND #Wks
-- Now generate report
SELECT T.Year, T.Week, T.City,
SUM(CASE WHEN PH1.status = 'published' THEN 1
WHEN PH1.status = 'unhidden_from_search' THEN 1
ELSE 0 END) as 'Live_Count'
FROM #Dates D1
LEFT JOIN
-- Get latest date per week
(SELECT YEAR(D.WeekStart) as 'Year',
D.Week,
PH.City,
PH.property_ID,
MAX(PH.date) as MaxDate
FROM #Dates D
LEFT JOIN #property_histories PH
ON PH.date BETWEEN #SD AND D.WeekEnd
GROUP BY D.WeekStart, D.Week, D.WeekStart, D.WeekEnd, PH.City, PH.property_id
) T
ON T.Week = D1.Week
LEFT JOIN #property_histories PH1
ON PH1.City = T.City AND PH1.property_id = T.property_id AND PH1.date = T.MaxDate
GROUP BY T.Year, T.Week, T.City
To break down the logic: Firstly I'm creating a helper table with week number, week start and week end dates. Week start is largely redundant but might come in handy for reporting.
I then subquery to get the latest date relevant for each week / city / property. For this "max" date, city and property I get the status, and if it's live, I sum it. So in layman terms ; get the latest status per city per property per week and SUM(if live).
Unlike the other answers posted, this solution caters for gaps in data. If the latest status recorded for a city and property was actually all the way back to week 1, it still works in any subsequent week.

Answer 2

I have a feeling I have missed a simpler way to do this.
However the following query uses 2 sub queries. The first gets all the published / unpublished ranges for a property (ie, the smallest unpublished date following a published date), while the 2nd does the same for properties being hidden from search.
These are then joined to properties on the property id, where the current date is within the range returned by the sub queries. The WHERE clause then checks that a record is matched for published and not found for the hidden sub queries
Had to use DISTINCT as otherwise the multiple published dates for a single unpublish would trigger duplicate property rows being returned.
SELECT DISTINCT properties.*
FROM properties
INNER JOIN
(
SELECT a.property_id, a.created_at AS start_date, IFNULL(MIN(b.created_at), NOW()) AS end_date
FROM property_histories a
LEFT OUTER JOIN property_histories b
ON a.property_id = b.propert_id
AND a.created_at < b.created_at
WHERE a.status = 'published'
AND b.status = 'unpublished'
GROUP BY a.property_id, a.created_at
) published
ON properties.property_id = published.property_id
AND NOW() BETWEEN published.start_date AND published.end_date
LEFT OUTER JOIN
(
SELECT a.property_id, a.created_at AS start_date, MIN(b.created_at) AS end_date
FROM property_histories a
LEFT OUTER JOIN property_histories b
ON a.property_id = b.propert_id
AND a.created_at < b.created_at
WHERE a.status = 'hidden_from_search'
AND b.status = 'unhidden_from_search'
GROUP BY a.property_id, a.created_at
) hidden
ON properties.property_id = hidden.property_id
AND NOW() BETWEEN hidden.start_date AND hidden.end_date
WHERE published.property_id IS NOT NULL
AND hidden.property_id IS NULL

Answer 3

I used a numbers table as a handy shortcut. Essentially, your question revolved around wanting to know a running sum of published or unhidden versus unpublished or hidden. At this point, the paper IDs become a moot point in the view (provided their uniqueness is properly constrained elsewhere), and all we need is a custom sum. I have the example on SQLFiddle. Here's the query:
select years.n + 2013 as year, weeks.n as week
, c.City
,
(select
sum(case
when status in ('published','unhidden_from_research') then 1
when status in ('unpublished','hidden_from_research') then -1
else 0
end)
from property_histories p2
where weekofyear(p2.date) <= weeks.n
and p2.city=c.city
) AS Live_Count
from numbers weeks
inner join numbers years on weeks.n <= 52
cross join (select City from property_histories group by city) c
where years.n + 2013 <= (select max(year(date)) from property_histories)
group by years.n + 2013, weeks.n
, c.City
;