Left Join with Null Script Efficiency Explanation Needed - mysql

Why would I use a LEFT JOIN in SQL in a FROM clause and attach a WHERE clause where the entity "is null"? I was told this is a very efficient script and I should learn the methodology behind it.
For example:
FROM
something
LEFT JOIN aRow a AND bRow b AND cRow c AND dRow d
WHERE
bRow.b IS NULL;

This kind of construct is used when you specifically want to know something like "a list of all customers who have never ordered anything" :
SELECT
customer.*
FROM
customers
LEFT JOIN
orders
ON
orders.customerid = customers.id
WHERE
orders.id IS NULL
Or to quote an old manager of mine: "Can you get the database to give me a list of everything that isn't in the database?"
Me> "Sure, can you give me a list of what things the database should tell you it doesn't have?"
Him> "How am I supposed to know that?"

This really is a fairly generic, non-RDBMS-specific question. The logic will apply to pretty much any flavor of SQL. And this is a technique that anyone who works with data queries should be familiar with.
For all intents and purposes (and moving past the flawed syntax in the OP), this is the same query as:
SELECT *
FROM table1
WHERE table1.col1 NOT IN (
SELECT table2.col1 FROM table2 WHERE table2.col2 = <filterHere>
)
When you are dealing with a couple of hundred rows, you may not see a significant difference in performance. But when you're working with just a few million rows in both tables, you will most definitely see a significant performance increase in
SELECT table1.*
FROM table1
LEFT OUTER JOIN table2 ON table1.col1 = table2.col1
AND table2.col2 = 42
WHERE table2.id IS NULL
Let's illustrate what is happening with these queries.
Create test tables.
CREATE TABLE table1 (col1 int, col2 varchar(10)) ;
INSERT INTO table1 ( col1, col2 )
VALUES (1,'a')
, (2,'b')
, (3,'c')
, (4,'d')
CREATE TABLE table2 (col1 int, col2 varchar(10)) ;
INSERT INTO table2 ( col1, col2 )
VALUES (1,'a')
, (3,'c')
This gives us
table1
col1 col2
1 a
2 b
3 c
4 d
table2
col1 col2
1 a
3 c
Now we want the columns that are in table1 but not in table2.
SELECT t1.col1, t1.col2
FROM table1 t1
WHERE t1.col1 NOT IN (
SELECT t2.col1 FROM table2 t2
)
We can't SELECT anything from table2, because that table is just a sub-query and not part of the whole query. It's not available to us.
This breaks down to
SELECT t1.col1, t1.col2
FROM table1 t1
WHERE t1.col1 NOT IN ( 1,3 )
Which further breaks down to
SELECT t1.col1, t1.col2
FROM table1 t1
WHERE t1.col1 <> 1
OR t1.col1 <> 3
These queries give us
col1 col2
2 b
4 d
That's a subquery broken down into 2 different OR statements to filter our results.
So lets look at a JOIN. We want all of the records on the left side, and only include those on the right side that match. So
SELECT t1.col1 AS t1_col1, t1.col2 AS t1_col2, t2.col1 AS t2_col1, t2.col2 AS t2_col2
FROM table1 t1
LEFT OUTER JOIN table2 t2 ON t1.col1 = t2.col1
With a JOIN, both tables are available to our SELECT, so we can see which records in tablel2 match up to those in table1. The above gives us
t1_col1 t1_col2 t2_col1 t2_col2
1 a 1 a
2 b NULL NULL
3 c 3 c
4 d NULL NULL
With the extra data, we can see that col1 for 2 and 4 don't match in the two tables. We can now filter those out with a simple WHERE statement.
SELECT t1.col1, t1.col2
FROM table1 t1
LEFT OUTER JOIN table2 t2 ON t1.col1 = t2.col1
WHERE t2.col1 IS NULL
Giving us
col1 col2
2 b
4 d
There's no subquery and just one statement in the filter. Plus, this allows the engine's optimizer to make a more efficient query plan.
It's impossible to see a difference in performance when we're only dealing with a couple of rows, but multiply these tables by a few million rows, and you will definitely see how much faster a JOIN can be.

Related

MYSQL select same column name as alias in union not working

I have a simple MYSQL query that unions two tables:
SELECT * FROM (
SELECT col1 AS col1A FROM table1
UNION
SELECT col1 AS col1B FROM table2
) AS t WHERE col1A <> col1B
I have a column called col1 in both tables and I need to select only rows that have a different value of that column so I select them as aliases. When I run this query I got:
Unknown column 'col1B' in 'where clause'
Table1 data:
col1
----
test
Table2 data:
col1
----
test
The query should return no rows as each value in col1 in table1 is equal to each value in col1 in table2 instead it returns that col1 in table2 is unknown although I select it as an alias
I think you need to look up the appropriate usage of UNION. It will return all results from first query combined with all results from the second query. This results in a single dataset, with a single column (not col1 and col2), just col1 in this case.
Assuming you're trying to get all records in table1 that don't exist in table2, you can use NOT EXISTS:
SELECT col1
FROM table1 t1
WHERE NOT EXISTS (
SELECT 1
FROM table2 t2
WHERE t1.col1 = t2.col1
)
Why Error 1054 is being returned by OP query
The error that's being returned is because the name assigned to a column from the result of a UNION is taken from the first SELECT.
You can observe this by running a simple example:
SELECT 1 AS one
UNION
SELECT 2 AS two
The resultset returned by that query will contain a single column, the name assigned to the column will be one, the column name from the first SELECT. This explains why you are getting the error from your query.
One way to return rows with no match
To return values of col1 from table1 which do not match any value in the col1 column from table2...
one option to use an anti-join pattern...
SELECT t1.col1
FROM table1 t1
LEFT
JOIN table2 t2
ON t2.col1 = t1.col1
WHERE t2.col1 IS NULL
The LEFT JOIN operation returns all rows from table1, along with any "matching" rows found in table2. The "trick" is the predicate in the WHERE clause... any "matching" rows from table2 will have a non-NULL value in col1. So, if we exclude all of the rows where we found a match, we're left with rows from table1 that didn't have a match.
If we want to get rows from table2 that don't have a "matching" row in table1, we can do the same thing, just flipping the order of the tables.
If we combine the two sets, but only want a "distinct" list of "not matched" values, we can use the UNION set operator:
SELECT t1.col1
FROM table1 t1
LEFT
JOIN table2 t2
ON t2.col1 = t1.col1
WHERE t2.col1 IS NULL
UNION
SELECT s2.col1
FROM table2 s2
LEFT
JOIN table1 s1
ON s1.col1 = s2.col1
WHERE s1.col1 IS NULL
--
Finding out which table the non-matched value is from
Sometimes, we want to know which query returned the value; we can get that by including a literal value as a discriminator in each query.
SELECT 'table1' AS src
, t1.col1
FROM table1 t1
LEFT
JOIN table2 t2
ON t2.col1 = t1.col1
WHERE t2.col1 IS NULL
UNION
SELECT 'table2' AS src
, s2.col1
FROM table2 s2
LEFT
JOIN table1 s1
ON s1.col1 = s2.col1
WHERE s1.col1 IS NULL
ORDER BY 2
A different (usually less performant) approach to finding non-matching rows
An entirely different approach, to returning an equivalent result, would be do something like this:
SELECT q.col1
FROM ( SELECT 't1' AS src, t1.col1 FROM table1 t1
UNION
SELECT 't2' AS src, t2.col1 FROM table2 t2
) q
GROUP BY q.col1
HAVING COUNT(DISTINCT q.src) < 2
ORDER BY q.col1
(The inline view q will be "materialized" as a derived table, so this approach can be expensive for large sets, and this approach won't take advantage of indexes on col1 to perform the matching.) One other small difference between this and the anti-join approach: this will omit a col1 value of NULL if a NULL exists in both tables. Aside from that, the resultset is equivalent.

Merging two SQL Server tables conditionally into a third table

Clearly, I am not a SQL guy, so I have to ask for help on the following rather simple task.
I have two SQL Server 2008 tables: t1 and t2 with many identical columns and a key column (entry_ID). T2 has rows that do not exist in t1 but should.
I want to merge those rows from t2 that do not exist in t1 but I also do not want any rows from t2 that already exist in t1. I would like the result set to fill a new t3.
I have looked at many solutions online but can't find the solution to the above scenario.
Thank you.
There are a number of ways to do it you could use UNION ALL or OUTER JOIN.
Assuming you are using Entry_ID to find identical records, and Entry_ID is unique within each table, here is a OUTER JOIN method:
This gets you your recordset: T1 and T2 merged:
SELECT
CASE
WHEN T1.Entry_ID IS NULL THEN 'T2'
WHEN T2.Entry_ID IS NULL THEN 'T1'
ELSE 'Both'
END SourceTable,
COALESCE(T1.Entry_ID,T2.Entry_ID) As Entry_ID,
COALESCE(T1.Col1, T2.Col1) As Col1,
COALESCE(T1.Col2, T2.Col2) As Col2,
COALESCE(T1.Col3, T2.Col3) As Col3,
COALESCE(T1.Col4, T2.Col4) As Col4
FROM T1 FULL OUTER JOIN T2
ON T1.Entry_DI = T2.Entry_ID
ORDER BY COALESCE(T1.Entry_DI,T2.Entry_ID)
This inserts it into T3:
INSERT INTO T3 (Entry_ID,Col1, COl2,Col3,Col4)
SELECT
COALESCE(T1.Entry_DI,T2.Entry_ID) As Entry_ID,
COALESCE(T1.Col1, T2.Col1) As Col1,
COALESCE(T1.Col2, T2.Col2) As Col2,
COALESCE(T1.Col3, T2.Col3) As Col3,
COALESCE(T1.Col4, T2.Col4) As Col4
FROM T1 FULL OUTER JOIN T2
ON T1.Entry_DI = T2.Entry_ID
Again you must note that Entry_ID needs to be unique within their tables, and it uses this to match between the tables.
Also note the columns from the select line up with the column list in the insert statement - the order of the columns in the physical table doesn't matter, the INSERT and SELECT just have to line up.

How can I compare if 2 tables have the same data?

If I have 2 tables and want to find if they have the same data, what is the most straightforward way to do it in MySQL?
I have read about doing a correlated subquery and UNION ALL but this query is about 2 pages (!) and can not really follow what it is doing. There must be an easier way.
Even if it is e.g. make MySQL copy the table data to files and do a vimdiff (I am not sure that this is even possible -is it?- just thinking out loud).
UPDATE
I am interested only in the table data and not structure. This is to clarify due to an ambiguous comment I made
If you just want to tell whether the tables are identical or not as efficiently as possible, use this query:
SELECT 1 FROM (
SELECT * FROM table1
UNION ALL
SELECT * FROM table2
) t
GROUP BY col1, col2, col3
HAVING count(*) = 1
LIMIT 1
List all the columns in GROUP BY to compare the entire table.
If the result is an empty set, the two tables are identical.
If you want to see the differences, use this query:
SELECT * FROM (
SELECT 'table1' tname, col1, col2, col3 FROM table1
UNION ALL
SELECT 'table2' tname, col1, col2, col3 FROM table2
) t
GROUP BY col1, col2, col3
HAVING count(*) = 1
List the same columns in the inner SELECT as in the GROUP BY, plus a column to distinguish the two tables.
Just throwing this out there, you could emulate a full outer join and then return the rows where just the right or the left side is null.
select t1.*
from table1 t1
LEFT OUTER JOIN table2 t2
ON t1.col1 = t2.col1
AND t1.col2 = t2.col2
AND ...
WHERE t2.id is null
UNION
select t2.*
from table2 t2
LEFT OUTER JOIN table1 t1
ON t2.col1 = t1.col1
AND t2.col2 = t1.col2
AND ...
WHERE t1.id is null
With the FULL OUTER JOIN you can show all rows where the other row is not available in the other table.
Use the following query:
SELECT c1 = cjoin AND c2 = cjoin equiv
FROM (SELECT COUNT(*) c1 FROM Table1) t1,
(SELECT COUNT(*) c2 FROM Table2) t2,
(SELECT COUNT(*) cjoin
FROM Table1 t1
JOIN Table2 t2
ON t1.col1 = t2.col1 AND t1.col2 = t2.col2 AND t1.col3 = t2.col3 ...) tjoin
Assuming the tables have a unique key, this will return equiv = 1 if the tables are equal. It doesn't show the differences, it's just a binary test.
I was reading SQL Cookbook from A.Molinaro, when I came across a solution.
It is based on to tables
emp(empno,ename,job,mgr,hiredate,sal,comm,deptno)
and a view
V
which has the same columns but different rows. The columns mgr and comm might be NULL, other columns not.
The solution in the book is very long and it does not show all differences, although this was the stated problem in 3.7.
I made up my solution which is shorter and shows all differences (means all rows which have different counts in the two tables).
select * from
# those which are contained in the (distinct) union of (col1,col2,...,coln, count) of both tables:
( select empno,ename,job,mgr,hiredate,comm,deptno, count(*) cnt from emp group by empno,ename,job,mgr,hiredate,comm,deptno
union
select empno,ename,job,mgr,hiredate,comm,deptno, count(*) cnt from V group by empno,ename,job,mgr,hiredate,comm,deptno
) as unionOfBoth
where (empno,ename,job,mgr,hiredate,comm,deptno,cnt)
not in
# those which are contained in the intersection of both tables with the equal number of counts:
( select e.empno,e.ename,e.job,e.mgr,e.hiredate,e.comm,e.deptno,e.cnt
from
(select empno, ename,job,mgr,hiredate,comm,deptno, count(*) cnt from emp group by empno,ename,job,mgr,hiredate,comm,deptno) e,
(select empno, ename,job,mgr,hiredate,comm,deptno, count(*) cnt from V group by empno,ename,job,mgr,hiredate,comm,deptno) v
where
e.empno = v.empno
and e.ename = v.ename
and e.job = v.job
and ifnull(e.mgr,0) = ifnull(v.mgr,0)
and e.hiredate = v.mgr
and e.deptno = v.deptno
and ifnull(e.comm,0) = ifnull(v.comm,0)
and e.cnt = v.cnt
);
Basically you count the distinct rows in both tables and do a union (not union all) to get the tmp.table unionBoth. Then you remove those rows, which both tables have in common.
Here two rows r1 from table t1 and r2 from table t2 are considered the same, if
(r1,count of r1 in t1) = (r2, count of r2 in t2), which is equivalent to r1=r2 (on all columns) and (count of r1 in t1) = (count of r2 in t2).
If the tables are small enough, you can export both tables as csv files and then copy one of the tables and paste them side-by-side with the other table. You can just go row by row and see if the outputs are the same that way.

Select value if it appears elsewhere in table

I am using mysql and I'm interested in rows where reciprocals appear in a different row in the table.
Imagine 2 columns, each with letters a through z.
Lets say row1 has a,b, row2 has a,c, and row 3 has c,a. I am interested in the pair a,c because it appears both as c,a and a,c in different rows in the table.
Do I have to use a nested select? Or perhaps an exists clause?
I believe this is what you're after, a self-join:
SELECT t1.*
FROM table1 t1
JOIN table1 t2
ON t1.col1 = t2.col2
AND t1.col2 = t2.col1
Here is a SQL fiddle demo: SQL Fiddle
also use SELECT REVERSE('abc') see http://dev.mysql.com/doc/refman/5.0/en/string-functions.html#function_reverse with above #spencer7593 said
`SELECT t1.*
FROM table1 t1
JOIN table1 t2
ON t1.col1 = t2.col2
AND t1.col2 = t2.col1`

Select from table1 where similar rows do NOT appear in table2?

I've been struggling with this for a while, and haven't been able to find any examples to point me in the right direction.
I have 2 MySQL tables that are virtually identical in structure. I'm trying to perform a query that returns results from Table 1 where the same data isn't present in table 2. For example, imagine both tables have 3 fields - fieldA, fieldB and fieldC. I need to exclude results where the data is identical in all 3 fields.
Is it even possible?
There are several ways to do it (assuming the fields don't allow NULLs):
SELECT a, b, c FROM Table1 T1 WHERE NOT EXISTS
(SELECT * FROM Table2 T2 WHERE T2.a = T1.a AND T2.b = T1.b AND T2.c = T1.c)
or
SELECT T1.a, T1.b, T1.c FROM Table1 T1
LEFT OUTER JOIN Table2 T2 ON T2.a = T1.a AND T2.b = T1.b AND T2.c = T1.c
WHERE T2.a IS NULL
select
t1.*
from
table1 t1
left join table2 t2 on
t1.fieldA = t2.fieldA and
t1.fieldB = t2.fieldB and
t1.fieldC = t2.fieldC
where
t2.fieldA is null
Note that this will not work if any of the fields is NULL in both tables. The expression NULL = NULL returns false, so these records are excluded as well.
This is a perfect use of EXCEPT (the key word/phase is "set difference"). However, MySQL lacks it. But no fear, a work-around is here:
Intersection and Set-Difference in MySQL (A workaround for EXCEPT)
Please not that approaches using NOT EXISTS in MySQL (as per above link) are actually less than ideal although they are semantically correct. For an explanation of the performance differences with the above (and alternative) approaches as handled my MySQL, complete with examples, see NOT IN vs. NOT EXISTS vs. LEFT JOIN / IS NULL: MySQL:
That’s why the best way to search for missing values in MySQL is using a LEFT JOIN / IS NULL or NOT IN rather than NOT EXISTS.
Happy coding.
The 'left join' is very slow in MYSQL. The gifford algorithm shown below speeds it many orders of magnitude.
select * from t1
inner join
(select fieldA from
(select distinct fieldA, 1 as flag from t1
union all
select distinct fieldA, 2 as flag from t2) a
group by fieldA
having sum(flag) = 1) b on b.fieldA = t1.fieldA;