I have 2 tables (t1 with 581978 rows,t2 with 581196 rows) containing some common fields (f1,f2,f3). The tables normally have the same number of rows, but can get out of sync. When this happens I'd like to identify the rows in t1 that are not in t2. The following python code (just use the 'select stmt' for mysql without python) identified the additional 782 rows in t1 but on my ageing iMac took over 12hrs! Is there a more efficient method?
orphans = ("select t1.f1,t1.f2,t1.f3 from t1 where not exists " \
"(select 1 from t2 where t2.f1=t1.f1 and t2.f2=t1.f2 and " \
"t2.f3=t1.f3)")
cursor.execute(orphans)
for i,(f1,f2,f3) in enumerate(cursor):
print(i+1,"orphans {:<10} {:<10} {<:8}".format(f1,f2,f3))
db-fiddle code (Is there a free db-fiddle, seems I can't save without upgrading to pro?)
CREATE DATABASE if not exists SO_61403216;
use SO_61403216;
drop table if exists t1,t2;
create table t1 (f1 varchar(1),f2 int(2),f3 int(2),f4 int(2));
create table t2 (f1 varchar(1),f2 int(2),f3 int(2),f4 int(2),f5 int(2));
insert into t1 (f1,f2,f3,f4) values ("A",2,3,4);
insert into t1 (f1,f2,f3,f4) values ("B",2,3,4);
insert into t1 (f1,f2,f3,f4) values ("C",2,3,4);
insert into t1 (f1,f2,f3,f4) values ("D",2,3,4);
insert into t2 (f1,f2,f3,f4,f5) values ("A",2,3,44,55);
insert into t2 (f1,f2,f3,f4,f5) values ("B",2,3,99,99);
insert into t2 (f1,f2,f3,f4,f5) values ("C",2,3,99,99);
Query, which exposes the extra row in table t1 using both the "not exist" (preferred output) and "null" methods, which both take 0ms for these small tables, but same query takes hours with 500K+ rows - is there a better/quicker method?
use SO_61403216;
desc t1;
desc t2;
select * from t1;
select * from t2;
select t1.f1,t1.f2,t1.f3 from t1 where not exists (select 1 from t2 where t2.f1=t1.f1 and t2.f2=t1.f2 and t2.f3=t1.f3)
select * from t1 left join t2 on t1.f1=t2.f1 and t1.f2=t2.f2 and t1.f3=t2.f3 where t2.f1 is null;
Related
I have two tables (say t1 and t2). I want to insert all records from t2 to t1. I can do this by using the query -
Insert into t1 select * from t2
While inserting, there might be duplicates. How to update all columns in table t1 from the t2 table in case of duplicates. We can do it this way -
Insert into t1 select * from t2 on duplicate update t1.a=t2.a...
But there are more than 60 columns in table t2. Is there any way to update all columns at once?
I am trying to compare multiple column such as part, quantity, and type with another table. I tried with compare part id without problem but when I want to compare part id with quantity it will cause error such as return nothing.
Below coding is for compare part id only.
INSERT INTO `table3`
SELECT *
FROM table1
WHERE NOT EXISTS (SELECT *
FROM table2
WHERE table2.part = table1.part)
How can I compare part and quantity together to find unmatched record.
A handy method in MySQL is in with tuples:
INSERT INTO `table3` ( . . . )
SELECT . . .
FROM table1
WHERE (part, quantity) NOT IN (SELECT part, quantity FROM table2);
More importantly, you should explicitly list all the columns when using an insert. This helps prevent unexpected problems when inserting data.
There are 2 solutions -
1)
INSERT INTO `table3`
SELECT *
FROM table1
WHERE NOT EXISTS (SELECT *
FROM table2
WHERE table2.part = table1.part And table2.quantity=table1.quantity)
2)
INSERT INTO table2 (part, quantity, type)
SELECT part, quantity, type
FROM table1
ON DUPLICATE KEY UPDATE quantity= VALUES(quantity), type= VALUES(type)
Not sure if it's the best approach, but I'd be tempted to try a null-left-join on either a set of values or a concatenation - e.g.
select t1.*
from mytable1 as t1
left join mytable2 as t2 on t1.foo = t2.foo and t1.bar = t2.bar
where t1.id is null;
The only thing to note is that t1.id (and t2.id) are assumed to be the primary key - i.e. can never be null. The end result of the above should be all the rows in t1 that cannot be joined to a row in t2.
Can you nest INSERT statements in MySQL?
I have this schema:
T1(ids)
T2(id, x, ...)
T3(id, y, ...)
What I need to do is copy data from table T2 to table T3 while modifying some fields and adding new data to T3, then updating the parent table T1 to reference the new records.
I know we can run a command like this to copy records between tables:
INSERT INTO T3 SELECT id+1000, X, 'new', ... FROM T2
However I then need to pipe the new records that have been created into T1.
Something like this:
INSERT INTO T1 SELECT id FROM
(INSERT INTO T3 SELECT id+1000, X, 'new', ... FROM T2)
Is this possible? My ids are not auto-increment, T1 is a list of id's of T2 and T3 with some additional data per records. T2 and T3 are not identical tables.
I just need to copy data from T3 to T2 with new id's and then update T1 to reflect the new records that were created, all in 1 transaction if possible.
I have two tables T1 and T2. I want to merge only those rows of T2 which is currently not there in T1 .
`Insert into T1 select * from (select * from T2 where id not in (select id from T1))`
Is there a better and faste way of achieving the same. ID column is unique across the table
Insert into T1
select * from T2
where id not in (select id from T1)
You could also join but then you'd need another subselect since MySQL does not want to select from a table it inserts at the same time without using a subselect.
I ran into a situation where I have two tables that store employee information, Table1 are the employees and table2 are 3rd party temps. These tables use a different convention for the IDs (not something I have control over).
The problem is that oftentimes these 3rd party temps become employed and there is no link between these tables. When this happens, I need to make sure they don't exists in Table2 before I create them. Right now the I just want to identify matches on DOB & Last 4, although I'm probably going to add at least first name to the criteria but right now I'd like to start somewhere.
The columns although name differently are the same (DOB = Birth Date, Code = Last 4)
CREATE TABLE Table1
([Emp_ID] int, [DOB] date, [code] varchar(10))
;
INSERT INTO Table1
([Emp_ID], [DOB], [code])
VALUES
(55556, '1966-01-15', '5454'),
(55557, '1980-03-21', '6868'),
(55558, '1985-04-26', '7979'),
(55559, '1990-10-17', '1212'),
(55560, '1992-12-30', '6767')
;
CREATE TABLE Table2
([USer_ID] int, [Birth_Date] date, [last4] varchar(10))
;
INSERT INTO Table2
([User_ID], [Birth_Date], [last4])
VALUES
(22223, '1966-01-15', '5454'),
(22224, '1980-03-21', '6868'),
(22225, '1975-07-19', '4545'),
(22226, '1988-05-24', '3434')
;
Here is what I came up with, It seems to work but I need to return the user_id from table2 that is producing this match?
SELECT *
FROM table1 t1
WHERE EXISTS (SELECT 1 FROM table2 t2 WHERE t1.DOB = t2.Birth_date)
AND EXISTS (SELECT 1 FROM table2 t2 WHERE t1.code = t2.last4)
Thanks!
Try this
Without JOINS:
SELECT t1.*,
(SELECT user_id FROM table2 t2
WHERE t1.DOB = t2.Birth_date and t1.code = t2.last4) user_id
FROM table1 t1
WHERE EXISTS (SELECT 1 FROM table2 t2
WHERE t1.DOB = t2.Birth_date and t1.code = t2.last4)
With JOINS
SELECT t1.*, t2.user_id
FROM table1 t1
inner join table2 t2 on t1.DOB = t2.Birth_date and t1.code = t2.last4
SQL DEMO