MySQL: How to combine partitions of ranges into largest possible contiguous ranges - mysql

Question

I've been trying to pull off a fairly complex SQL query (maybe simple?) to compress a table with repetitive information. I'm using MySQL 5.7.14 in SequelPro. I'm a novice SQL user with a basic understanding of joins, unions etc. I'm thinking a subquery with some group bys is needed for this one, but I don't know how to do it best.
A simple example of what I'm trying to do is illustrated by the table below:
table
For every col_1 repeated entry, I want to compress into a single entry when the range set by col_2 and 3 (start and end of a range, respectively) overlap. For col_4 and 5, the max value among entries falling in this range should be reported. With the example above, in col_1, there are three ranges for a that overlap and I want to compress this to the min for col_1 and max for col_2 with the max for col_4 and 5. For 'b' in col_2, there are two ranges (31-50, 12-15) that do not overlap, so it would return both rows as is. For c, it would return one row with range 100-300 and values 3, 2 for col_4 and col_5, respectively. The full result desired from this example is shown below:
query output
I should add that there are 'null' values in some places that should be treated as zeros.
Does anybody have anybody know the best, and simplest way to do this?
Thank you in advance!!
Update: I've tried using the range setting query suggested but I get an error. The query is as follows:
WITH a AS (SELECT range
, lower(col_2) AS startdate
, max(upper(col_3)) OVER (ORDER BY range) AS `end`
FROM `combine`
)
, b AS (
SELECT *, lag(`end`) OVER (ORDER BY range) < `start` OR NULL AS step
FROM a
)
, c AS (
SELECT *, count(step) OVER (ORDER BY range) AS grp
FROM b
)
SELECT daterange(min(`start`), max(`end`)) AS range
FROM c
GROUP BY grp
ORDER BY 1;
The error I receive is:
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'a AS (SELECT range
, lower(col_2) AS startdate
, max(upper(col_3)) OVE' at line 1

Answer 1

This is not trivial, but it can be done in one single query.
The hard part is combining a set of intervals into the largest possible contiguous intervals. Solutions are detailed in this post.
To get the result you are after, you now need to:
Calculate the largest possible contiguous intervals for each value in col1, using the query given in the link.
The result, based on your example values, would be:
col_1 lower_bound upper_bound
a 20 60
b 12 15
b 31 50
c 100 300
Associate one of those large intervals to each row in your_table. There can only be one such interval per row, so let's INNER JOIN:
SELECT my_table.*, large_intervals.lower_bound, large_intervals.upper_bound
FROM my_table
INNER JOIN (my_awesome_query(your_table)) large_intervals
ON large_intervals.col1 = my_table.col1
AND large_intervals.lower_bound <= my_table.col2
AND large_intervals.upper_bound >= my_table.col3
You would get:
col1 col2 col3 col4 col5 lower_bound upper_bound
a 45 50 1 0 20 60
a 50 61 6 0 20 60
a 20 45 0 5 20 60
b 31 50 0 1 31 50
b 12 15 5 0 12 15
c 100 200 3 2 100 300
c 150 300 1 2 100 300
Then it's easy, just group by col1, lower_bound, upper bound:
SELECT col1, lower_bound AS col2, upper_bound AS col3, MAX(col4) AS col4, MAX(col5) AS col5 FROM (query above) decorated_table GROUP BY col1, lower_bound, upper_bound
And you get exactly the result you're after.
To get back on the hard part: The post mentioned above exposes solutions for PostgreSQL. MySQL doesn't have range types, but the solution can be adapted. For instance, instead of lower(range), use the lower bound directly col2. The solution also makes use of window functions, namely lag and lead, but that is supported by MySQL, with the same syntax, so no problem here. Also note that they use COALESCE(upper(range), 'infinity') to guard against unbound ranges. Since your ranges are finite, you don't need to care about this, you can simply use the upper range directly, i.e. col3. Here's the adaptation:
WITH a AS (
SELECT
col2,
col3,
col2 AS lower_bound,
MAX(col3) OVER (ORDER BY col2, col3) AS upper_bound
FROM combine
)
, b AS (
SELECT *, lag(upper_bound) OVER (ORDER BY col2, col3) < lower_bound OR NULL AS step
FROM a
)
, c AS (
SELECT *, count(step) OVER (ORDER BY col2, col3) AS grp
FROM b
)
SELECT
MIN(lower_bound) AS lower_bound,
MAX(upper_bound) AS range
FROM c
GROUP BY grp
ORDER BY 1;
This works for a single group. If you want to get the ranges by col1, you can tweak it like this:
WITH a AS (
SELECT
col1,
col2,
col3,
col2 AS lower_bound,
MAX(col3) OVER (PARTITION BY col1 ORDER BY col2, col3) AS upper_bound
FROM combine
)
, b AS (
SELECT *, lag(upper_bound) OVER (PARTITION BY col1 ORDER BY col2, col3) < lower_bound OR NULL AS step
FROM a
)
, c AS (
SELECT *, count(step) OVER (PARTITION BY col1 ORDER BY col2, col3) AS grp
FROM b
)
SELECT
MIN(lower_bound) AS lower_bound,
MAX(upper_bound) AS range
FROM c
GROUP BY col1, grp
ORDER BY 1;
Combining everything, we get the following, which (tested on the example you provided), returns exactly the output you expected:
WITH a AS (
SELECT
col1,
col2,
col3,
col2 AS lower_bound,
MAX(col3) OVER (PARTITION BY col1 ORDER BY col2, col3) AS upper_bound
FROM combine
)
, b AS (
SELECT *, lag(upper_bound) OVER (PARTITION BY col1 ORDER BY col2, col3) < lower_bound OR NULL AS step
FROM a
)
, c AS (
SELECT *, count(step) OVER (PARTITION BY col1 ORDER BY col2, col3) AS grp
FROM b
)
, large_intervals AS (
SELECT
col1,
MIN(lower_bound) AS lower_bound,
MAX(upper_bound) AS upper_bound
FROM c
GROUP BY col1, grp
ORDER BY 1
)
, combine_with_large_interval AS (
SELECT
combine.*,
large_intervals.lower_bound,
large_intervals.upper_bound
FROM combine
INNER JOIN large_intervals
ON large_intervals.col1 = combine.col1
AND large_intervals.lower_bound <= combine.col2
AND large_intervals.upper_bound >= combine.col3
)
SELECT
col1,
lower_bound AS col2,
upper_bound AS col3,
MAX(col4) AS col4,
MAX(col5) AS col5
FROM combine_with_large_interval
GROUP BY col1, lower_bound, upper_bound
ORDER BY col1, col2, col3;
Voilà !