How would you compare 3 tables to 1 in MySQL? - mysql

Question

so I have a possibly silly question, but I'm looking for a basic approach or strategy for the following problem.
I have 1 master file and 3 source files, lets call them master, src1, src2, and src3. The master file is SUPPOSED to have the same records as the 3 source files combined, however, the master file has more records than the sum of all 3 sources. My goal is to validate that all records in said src1-3 are inside the master file AND also extract the records from the master that aren't in any 1 of the 3 sources. Additionally, each of the 4 files have different (but similar) headers
I have been able to find the distinct records from src1 (and subsequent sources) and mapped it to the matching records in the master file by using the following :
WITH tmp1 AS (
SELECT src1.*
FROM src1 as s1
LEFT JOIN master as mstr
ON (
s1.name = mstr.fname
s1.quant = mstr.qty
s1.item = mstr.obj
s1.price = mstr.prc
s1.age = mstr.time_since_dob
)
) SELECT DISTINCT primaryKey from tmp1;
Using this, I can get a count of distinct matches between the two files that are present in src1 and if that matches the count from select distinct PK from src1 then I'm in decent shape. Albeit, I know that using the criteria above I could easily get many collision since several records could have the same name, quantity, item, price, etc... But suffice it to say, using the above criteria I can get unique matches since there are no matching ID's between the two tables or anything like that. Additionally, the join criteria for each source is slightly different so I had to do the above 3 separate times and validate each source independently.
Having done the above along with some other analysis, I have been able to validate that each distinct record from src1-3 has at least 1 distinct match in the master file. I'm having issue, however, with the second half of this challenge where I have to select the records from the master file that did NOT have a corresponding match.
How can I select those records from the master file that were not matched? Can I do a simple
select * from master not in newView1 where newView1 is the combination of the 3 selects for the 3 sources? Again, I'm using different columns for each join condition so putting 3 sources under the same header might be difficult (but worth pursuing?). Another thing worth mentioning is that each file is ~1gb and the master file is ~3gb so time complexity is worth considering.
Thanks for all and any help.

Answer 1

First, using UNION ALL to get all matching rows and rows contained only in src1-3 tables.
Next, getting also rows of the master table that is contained only in the master table by joining with the tmp1 table.
Refer to the following query:
with tmp1(tbl,name,quant,item,price,age,fname,qty,obj,prc,time_since_dob) as (
select 'src1',s1.*,m.* from src1 s1 left join master1 m on
s1.name=m.fname and
s1.quant=m.qty and
s1.item=m.obj and
s1.price=m.prc and
s1.age=m.time_since_dob
union all
select 'src2',s2.*,m.* from src2 s2 left join master1 m on
s2.name=m.fname and
s2.quant=m.qty and
s2.item=m.obj and
s2.price=m.prc and
s2.age=m.time_since_dob
union all
select 'src3',s3.*,m.* from src3 s3 left join master1 m on
s3.name=m.fname and
s3.quant=m.qty and
s3.item=m.obj and
s3.price=m.prc and
s3.age=m.time_since_dob
)
select 'master',m.fname,m.qty,m.obj,m.prc,m.time_since_dob from master1 m left join tmp1 t on
m.fname=t.name and
m.qty=t.quant and
m.obj=t.item and
m.prc=t.price and
m.time_since_dob=t.age
where t.name is null
union all
select t.tbl,t.name,t.quant,t.item,t.price,t.age from tmp1 t
where t.fname is null
db fiddle