SQL Strip Longest Common Prefix - mysql

Question

I have a table tbl1 with two columns col1 and col2 containing strings:
col1 | col2
--------+--------
bar | foo
foo | foobar
bar1foo | bar2foo
Corresponding SQL dump:
CREATE TABLE `tbl1` (
`col1` varchar(20) COLLATE latin1_general_ci NOT NULL,
`col2` varchar(20) COLLATE latin1_general_ci NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci;
INSERT INTO `tbl1` (`col1`, `col2`) VALUES
('bar', 'foo'),
('foo', 'foobar'),
('bar1foo', 'bar2foo');
The strings of an entry share a common prefix in most cases. I need a query that strips those common prefixes. Expected result:
bar | foo
| bar
1foo | 2foo
My approach, so far:
SELECT
SUBSTR(`col1`, 1+GREATEST(LENGTH(`col1`), LENGTH(`col2`)) - CEIL(LENGTH(TRIM(TRAILING '0' FROM HEX(ABS(CONV(HEX(REVERSE(`col1`)),16,10) - CONV(HEX(REVERSE(`col2`)),16,10)))))/2)),
SUBSTR(`col2`, 1+GREATEST(LENGTH(`col1`), LENGTH(`col2`)) - CEIL(LENGTH(TRIM(TRAILING '0' FROM HEX(ABS(CONV(HEX(REVERSE(`col1`)),16,10) - CONV(HEX(REVERSE(`col2`)),16,10)))))/2))
FROM tbl1
Short explanation: The strings are reversed (REVERSE), converted into integers (HEX and CONV), subtracted from each other (- and ABS), converted into hexadecimal representation (HEX), 0's are trimmed from the end (TRIM), the length of this result is subtracted from the length of the longest string (-, LENGTH and GREATEST) and then used by SUBSTR to get the result.
Problems with my approach:
Does not work with strings longer than 64bit.
Does not work with strings containing multi-byte characters
Very lengthy and ugly
Does not have good performance.

Answer 1

Sadly, the most general and performance method is probably a giant case expression. However, this works only up to a certain length:
select substr(col1, prefix_length + 1),
substr(col2, prefix_length + 1)
from (select tbl1.*,
(case when left(col1, 10) = left(col2, 10) then 10
when left(col1, 9) = left(col2, 9) then 9
. . .
else 0
end) as prefix_length
from tbl1
) t;
Actually, you can do this with a recursive CTE, which is the most general approach:
with recursive cte as (
select col1, col2, 1 as lev, col1 as orig_col1, col2 as orig_col2
from tbl1
union all
select substr(col1, 2), substr(col2, 2), lev + 1, orig_col1, orig_col2
from cte
where left(col1, 1) = left(col2, 1)
)
select col1, col2
from (select cte.*,
dense_rank() over (partition by orig_col1, orig_col2 order by lev desc) as seqnum
from cte
) x
where seqnum = 1;
Although the performance will definitely be worse than your solution or the massive case expression, it is probably not that bad, and you might find it sufficient for your purposes.
Here is a db<>fiddle with both solutions.

Answer 2

This code works, although it's lengthy and ugly and (maybe) unperformant:
select
substring(t.col1, g.maxlen + 1) col1,
substring(t.col2, g.maxlen + 1) col2
from tbl1 t inner join (
select t.col1, t.col2,
max(case when left(col1, tt.n) = left(col2, tt.n) then tt.n else 0 end) maxlen
from tbl1 t inner join (
select 1 n union all select 2 union all select 3 union all select 4 union all
select 5 union all select 6 union all select 7 union all select 8 union all
select 9 union all select 10 union all select 11 union all select 12 union all
select 13 union all select 14 union all select 15 union all select 16 union all
select 17 union all select 18 union all select 19 union all select 20
) tt on least(length(t.col1), length(t.col2)) >= tt.n
group by t.col1, t.col2
) g on g.col1 = t.col1 and g.col2 = t.col2
See the demo.
For MySql 8.0+ you can use a recursive CTE and in this case there is no need of prior knowledge of the length of the columns:
with
recursive lengths as (
select 1 n
union all
select n + 1
from lengths
where n < (select max(least(length(col1), length(col2))) from tbl1)
),
cte as (
select t.col1, t.col2,
max(case when left(col1, l.n) = left(col2, l.n) then l.n else 0 end) maxlen
from tbl1 t inner join lengths l
on least(length(t.col1), length(t.col2)) >= l.n
group by t.col1, t.col2
)
select
substring(t.col1, c.maxlen + 1) col1,
substring(t.col2, c.maxlen + 1) col2
from tbl1 t inner join cte c
on c.col1 = t.col1 and c.col2 = t.col2
See the demo.
Results:
| col1 | col2 |
| ---- | ---- |
| | bar |
| bar | foo |
| 1foo | 2foo |