optimizing short text comparisons - mysql

Question

i have 2 tables qs and local.
qs has 2 columns (actually built from several other columns) that are part of the comparison i need to do:
f1 | t1
abcdaa | abcdbb
local just has one column that's part of the comparison:
rangeA
abcd
I am trying to find the entries in qs that do not have a matching substring in local
I've tried this in about a dozen different ways, and i must be missing something , since it's taking an unusual amount of time.
here is the fastest method I've found so far:
CREATE TEMPORARY TABLE `tempB` SELECT f1, t1,
LEFT(f1,2) AS l2,LEFT(f1,3) AS l3,LEFT(f1,4) AS l4,LEFT(f1,5) AS l5,LEFT(f1,6) AS l6,LEFT(f1,7) AS l7,LEFT(f1,8) AS l8,
LEFT(f1,9) AS l9,LEFT(f1,10) AS l10,LEFT(f1,11) AS l11,LEFT(f1,12) AS l12,LEFT(f1,13) AS l13,
LEFT(t1,2) AS lt2,LEFT(t1,3) AS lt3,LEFT(t1,4) AS lt4,LEFT(t1,5) AS lt5,LEFT(t1,6) AS lt6,LEFT(t1,7) AS lt7,LEFT(t1,8) AS lt8,
LEFT(t1,9) AS lt9,LEFT(t1,10) AS lt10,LEFT(t1,11) AS lt11,LEFT(t1,12) AS lt12,LEFT(t1,13) AS lt13 FROM
(SELECT CONCAT(c1,n1,s1) AS f1, CONCAT(c1,n1,s2) AS t1 FROM qs WHERE c1 ='a')tab0 ORDER BY f1 ASC;
CREATE TEMPORARY TABLE `tempB2` SELECT rangeA FROM local WHERE rangeA LIKE 'a%' ORDER BY rangeA ASC;
CREATE TEMPORARY TABLE `tempB3` SELECT rangeA AS rangeAA FROM local WHERE rangeA LIKE 'a%' ORDER BY rangeA ASC;
SELECT f1,t1, rangeA, rangeAA FROM tempB
LEFT JOIN tempB2 ON rangeA IN(l2,l3,l4,l5,l6,l7,l8,l9,l10,l11,l12,l13)
LEFT JOIN tempB3 ON rangeAA IN(lt2,lt3,lt4,lt5,lt6,lt7,lt8,lt9,lt10,lt11,lt12,lt13)
WHERE rangeA IS NULL OR rangeAA IS NULL
creating the temp tables is fast and starting with one character at a time (in this case 'a') significantly reduces the size of the datasets, but this is still very very slow even with only a few hundred thousand rows in each temp table.
I've tried using just f1 and t1 with a
ON f1 LIKE CONCAT (rangeA,'%')
but that seemed to be even slower.
Any other ideas?
Note that rangeA is at least 2 characters long and at most 13 characters long. hence the LEFTs.
example data:
qs :
c1 | n1 | s1 | s2
ab | cd | aa | bb
bb | bbb | bb | bc
cbc | cc | cdd | ddd
ddd | e | ddf | def
local :
rangeA
abcd
bdddd
cbcccdd
dddedd
expected result:
f1 | t1 | f1match | t1match
bbbbbbb | bbbbbbc | NULL | NULL
cbccccdd | cbcccddd | NULL | cbcccdd
dddeddf | dddedef | dddedd | NULL

Answer 1

Thank you Paul Spiegel for making this work.
Let's set up some test data.
mysql> select * from qs;
+----+---------------+-------------------+
| id | f1 | t1 |
+----+---------------+-------------------+
| 6 | match1 | no match |
| 7 | match1 | match2 |
| 8 | foo match1 | match1 bar |
| 9 | no match | abc match2 123 |
| 10 | no match | no match |
| 11 | also no match | again not a match |
+----+---------------+-------------------+
mysql> select * from local;
+--------+
| rangeA |
+--------+
| match1 |
| match2 |
+--------+
And we expect only those rows which neither f1 nor t1 match any row in local.
+----+---------------+-------------------+
| id | f1 | t1 |
+----+---------------+-------------------+
| 10 | no match | no match |
| 11 | also no match | again not a match |
+----+---------------+-------------------+
UPDATE: Indexing qs(f1,t1) and local(rangeA) will help performance.
create index index_qs_fields on qs(f1,t1);
create index index_local_rangeA on local(rangeA);
instr finds a substring in a string, that simplifies many things.
We can do this with a left excluding join. That is to get only the rows on the left side (qs) which have no match on the right (local).
We do a normal left join to check for matches.
select qs.*, rangeA
from qs
left join local on
instr(f1,rangeA) or
instr(t1,rangeA)
+----+---------------+-------------------+--------+
| id | f1 | t1 | rangeA |
+----+---------------+-------------------+--------+
| 1 | match1 | no match | match1 |
| 2 | match1 | match2 | match1 |
| 3 | foo match1 | match1 bar | match1 |
| 2 | match1 | match2 | match2 |
| 4 | no match | abc match2 123 | match2 |
| 5 | no match | no match | NULL |
| 6 | also no match | again not a match | NULL |
+----+---------------+-------------------+--------+
And turn it into an excluding join by filtering for only those which don't match at all.
select qs.*, rangeA
from qs
left join local on
instr(f1,rangeA) or
instr(t1,rangeA)
where rangeA is null
+----+---------------+-------------------+
| id | f1 | t1 |
+----+---------------+-------------------+
| 5 | no match | no match |
| 6 | also no match | again not a match |
+----+---------------+-------------------+
dbfiddle
UPDATE: Lots of entries in local can make this slow. We can try optimizing it by joining all the matches together into one regular expression. This might be faster.
We can construct our regex using group_concating all the matches together as a single regex.
select group_concat(rangeA separator '|')
into #range_re
from local;
select qs.*
from qs
where not f1 regexp(#range_re) and not t1 regexp(#range_re);
Note that you'll need to be careful to escape regex characters in your matches.
Original way too complicated answer follows.
That tells us which entries in qs don't match entries in local.
select qs.id, f1, t1, rangeA
from qs
left join local on 1=1
where instr(f1,rangeA) = 0 and instr(t1,rangeA) = 0;
+----+---------------+-------------------+--------+
| id | f1 | t1 | rangeA |
+----+---------------+-------------------+--------+
| 6 | match1 | no match | match2 |
| 8 | foo match1 | match1 bar | match2 |
| 9 | no match | abc match2 123 | match1 |
| 10 | no match | no match | match1 |
| 10 | no match | no match | match2 |
| 11 | also no match | again not a match | match1 |
| 11 | also no match | again not a match | match2 |
+----+---------------+-------------------+--------+
But we want those which don't match all of local. We can do that by counting up how many times a row appears in our list of not matches.
select qs.id, f1, t1, count(id)
from qs
left join local on 1=1
where instr(f1,rangeA) = 0
and instr(t1,rangeA) = 0
group by qs.id;
+----+---------------+-------------------+-----------+
| id | f1 | t1 | count(id) |
+----+---------------+-------------------+-----------+
| 6 | match1 | no match | 1 |
| 8 | foo match1 | match1 bar | 1 |
| 9 | no match | abc match2 123 | 1 |
| 10 | no match | no match | 2 |
| 11 | also no match | again not a match | 2 |
+----+---------------+-------------------+-----------+
And then select only those whose count is the same as the number of matches.
mysql> select qs.id, f1, t1
from qs
left join local on 1=1
where instr(f1,rangeA) = 0
and instr(t1,rangeA) = 0
group by qs.id
having count(id) = (select count(*) from local);
+----+---------------+-------------------+
| id | f1 | t1 |
+----+---------------+-------------------+
| 10 | no match | no match |
| 11 | also no match | again not a match |
+----+---------------+-------------------+
dbfiddle

Answer 2

here's what i have so far, which works pretty well for <50k rows. Thank you to Schwern for the helpful discussion about INSTR().
CREATE TEMPORARY TABLE `tempB` SELECT f1, t1 FROM
(SELECT LEFT(CONCAT(c1,n1,s1),17) AS f1, LEFT(CONCAT(c1,n1,s2),17) AS t1 FROM qs WHERE c1 ='a')tab0 ORDER BY f1 ASC;
CREATE TEMPORARY TABLE `tempB2` SELECT rangeA FROM local WHERE rangeA LIKE 'a%' ORDER BY rangeA ASC;
CREATE TEMPORARY TABLE `tempB3` SELECT rangeA AS rangeAA FROM local WHERE rangeA LIKE 'a%' ORDER BY rangeA ASC;
SELECT f1,t1, rangeA, rangeAA FROM tempB
LEFT JOIN tempB2 ON INSTR(f1,rangeA) =1
LEFT JOIN tempB3 ON INSTR(t1,rangeAA) =1
WHERE rangeA IS NULL OR rangeAA IS NULL

Answer 3

If I correctly understand your question, I think you should look into using LOCATE() or POSITION(). I don't really get the need to using all those LEFT().
A overly simplified version of what I think you want is this:
CREATE TEMPORARY TABLE `tempB`
SELECT CONCAT(c1,n1,s1) AS f1, CONCAT(c1,n1,s2) AS t1 FROM qs ORDER BY f1 ASC;
CREATE TEMPORARY TABLE `tempB2` SELECT rangeA FROM local ;
SELECT tempB.f1, tempB.t1
from tempB
WHERE (SELECT COUNT(*) from tempB2
WHERE POSITION(rangeA IN tempB.f1) != 0 AND POSITION(rangeA IN tempB.t1) != 0) = 0;