I am facing some difficulty when using 'OR' queries with multi-table joins with MySql 5.7.
For example, let's say I want a list of Animals that are either currently at a location or are being shipped there.
Select * from animals
left outer join animal_locations on animals.id = animal_locations.animal_id
left outer join animal_shipments on animals.id = animal_shipments.animal_id
where animal_locations.location_id = 5 OR animal_shipments.location_id = 5
The queries are very fast when using "AND" as well as using individual columns without any operator. But when "OR" queries are used, there is a big performance hit, the right indexes are not being hit.
Any way I can achieve this without splitting it up into multiple queries?
SELECT *
FROM animals
LEFT OUTER JOIN animal_locations ON animals.id = animal_locations.animal_id
AND animal_locations.location_id = 5
LEFT OUTER JOIN animal_shipments ON animals.id = animal_shipments.animal_id
AND animal_shipments.location_id = 5
WHERE COALESCE(animal_locations.animal_id, animal_shipments.animal_id) IS NOT NULL
The logic - previously the rows in slave tables with location_id = 5 are filtered, and then filtered rows are joined to main table.
Related
I have a three tables.
Alan1, Alan2, Alan3 are the same in all tables.I want to merge tables. The difference of the tables is the rightmost column. How should I write a SQL Query?
You can join. As commented by Barmar, the idea is to use the first 3 columns as join keys;
select a.*, b.alan4 as alan4b, c.alan4 as alan4c
from a
inner join b
on b.alan1 = a.alan1
and b.alan2 = a.alan2
and b.alan3 = a.alan3
inner join c
on c.alan1 = a.alan1
and c.alan2 = a.alan2
and c.alan3 = a.alan3
This gives you rows that are available in all 3 tables. Say you want to allow "missing" rows in b and/or c, then you need to change the two inner joins to left joins.
I am pretty new in SQL and I have the following problem with a simple query.
I have to perform a query that JOIN 2 tables. The join operation is not on a single field but have to be performed on 2 differents field having the same name on these tables.
So I have done something like this:
SELECT count(*)
FROM TID003_ANAGEDIFICIO anagraficaEdificio
INNER JOIN TID002_CANDIDATURA candidatura ON(candidatura.PRG_PAR = anagraficaEdificio.PRG_PAR, candidatura.PRG_CAN = anagraficaEdificio.PRG_CAN)
WHERE anagraficaEdificio.FLG_GRA=1;
But, performing this query, I obaint the following error message:
Operand should contain 1 column(s)
As you can see I am trying to join these 2 tables and the join is based on 2 fields, by:
ON(candidatura.PRG_PAR = anagraficaEdificio.PRG_PAR, candidatura.PRG_CAN = anagraficaEdificio.PRG_CAN)
What is wrong? What am I missing? How can I fix this issue?
Use AND to specify the additional join conditions if you want to use more than one condition together:
SELECT count(*)
FROM TID003_ANAGEDIFICIO anagraficaEdificio
JOIN TID002_CANDIDATURA candidatura
ON candidatura.PRG_PAR = anagraficaEdificio.PRG_PAR
AND candidatura.PRG_CAN = anagraficaEdificio.PRG_CAN
WHERE anagraficaEdificio.FLG_GRA = 1;
If the column names of the joined columns are the same in both tables you could use a join with the USING predicate instead:
JOIN TID002_CANDIDATURA candidatura USING (PRG_PAR, PRG_CAN)
This is not supported in all databases, but it is in MySQL (see the documentation for more information).
I'm querying my database for a list of locations that are either a teacher's primary location or locations that the teacher has provided availability information for.
If I do two joins like this:
SELECT tea.*, avail_loc.*, pri_loc.*
FROM teachers AS tea
LEFT JOIN availability AS ava
ON(tea.teacher_id = ava.teacher_id AND ava.end_date > 1398706428)
LEFT JOIN locations AS avail_loc
ON(ava.location_id = avail_loc.location_id)
LEFT JOIN locations AS pri_loc
ON(tea.location_id = pri_loc.location_id)
WHERE tea.active = 1
My query takes .05 seconds. The problem is that I have to clean up the output in php because my locations are split between avail_loc (the alternate locations) and pri_loc (the primary location).
So, if I combine them into one join with an OR conditional, the query takes .8 seconds.
SELECT tea.*, loc.*
FROM teachers AS tea
LEFT JOIN availability AS ava
ON(tea.teacher_id = ava.teacher_id AND ava.end_date > 1398706428)
LEFT JOIN locations AS loc
ON(ava.location_id = loc.location_id OR tea.location_id = loc.location_id)
WHERE tea.active = 1
When I use EXPLAIN, the first query has indexes that match everything. When I run the second query, it's missing a join for my OR join.
Why are the two joins faster than the one with an OR? The resulting data is the same.
First, these two queries are not equivalent. If there is one match in each of the tables, then the first will return one row and the second will return two rows.
The answer to your question, though, is a matter of optimization. MySQL does a poor job of optimizing across or conditions. As you note, it misses the fact that different indexes could be used to satisfy each condition. To be honest, this is a problem with most database engines. If you want the effect of an or with better performance, then union all often works better:
SELECT tea.*, loc.*
FROM teachers AS tea
LEFT JOIN availability AS ava
ON(tea.teacher_id = ava.teacher_id AND ava.end_date > 1398706428)
LEFT JOIN locations AS loc
ON(tea.location_id = loc.location_id)
WHERE tea.active = 1
union all
SELECT tea.*, loc.*
FROM teachers AS tea
LEFT JOIN availability AS ava
ON(tea.teacher_id = ava.teacher_id AND ava.end_date > 1398706428)
LEFT JOIN locations AS loc
ON(ava.location_id = loc.location_id)
WHERE tea.active = 1;
I'm reasonably new to MySQL and I'm trying to select a distinct set of rows using this statement:
SELECT DISTINCT sp.atcoCode, sp.name, sp.longitude, sp.latitude
FROM `transportdata`.stoppoints as sp
INNER JOIN `vehicledata`.gtfsstop_times as st ON sp.atcoCode = st.fk_atco_code
INNER JOIN `vehicledata`.gtfstrips as trip ON st.trip_id = trip.trip_id
INNER JOIN `vehicledata`.gtfsroutes as route ON trip.route_id = route.route_id
INNER JOIN `vehicledata`.gtfsagencys as agency ON route.agency_id = agency.agency_id
WHERE agency.agency_id IN (1,2,3,4);
However, the select statement is taking around 10 minutes, so something is clearly afoot.
One significant factor is that the table gtfsstop_times is huge. (~250 million records)
Indexes seem to be set up properly; all the above joins are using indexed columns. Table sizes are, roughly:
gtfsagencys - 4 rows
gtfsroutes - 56,000 rows
gtfstrips - 5,500,000 rows
gtfsstop_times - 250,000,000 rows
`transportdata`.stoppoints - 400,000 rows
The server has 22Gb of memory, I've set the InnoDB buffer pool to 8G and I'm using MySQL 5.6.
Can anybody see a way of making this run faster? Or indeed, at all!
Does it matter that the stoppoints table is in a different schema?
EDIT:
EXPLAIN SELECT... returns this:
It looks like you are trying to find a collection of stop points, based on certain criteria. And, you're using SELECT DISTINCT to avoid duplicate stop points. Is that right?
It looks like atcoCode is a unique key for your stoppoints table. Is that right?
If so, try this:
SELECT sp.name, sp.longitude, sp.latitude, sp.atcoCode
FROM `transportdata`.stoppoints` AS sp
JOIN (
SELECT DISTINCT st.fk_atco_code AS atcoCode
FROM `vehicledata`.gtfsroutes AS route
JOIN `vehicledata`.gtfstrips AS trip ON trip.route_id = route.route_id
JOIN `vehicledata`.gtfsstop_times AS st ON trip.trip_id = st.trip_id
WHERE route.agency_id BETWEEN 1 AND 4
) ids ON sp.atcoCode = ids.atcoCode
This does a few things: It eliminates a table (agency) which you don't seem to need. It changes the search on agency_id from IN(a,b,c) to a range search, which may or may not help. And finally it relocates the DISTINCT processing from a situation where it has to handle a whole ton of data to a subquery situation where it only has to handle the ID values.
(JOIN and INNER JOIN are the same. I used JOIN to make the query a bit easier to read.)
This should speed you up a bit. But, it has to be said, a quarter gigarow table is a big table.
Having 250M records, I would shard the gtfsstop_times table on one column. Then each sharded table can be joined in a separate query that can run parallel in separate threads, you'll only need to merge the result sets.
The trick is to reduce how many rows of gtfsstop_times SQL has to evaluate. In this case SQL first evaluates every row in the inner join of gtfsstop_times and transportdata.stoppoints, right? How many rows does transportdata.stoppoints have? Then SQL evaluates the WHERE clause, then it evaluates DISTINCT. How does it do DISTINCT? By looking at every single row multiple times to determine if there are other rows like it. That would take forever, right?
However, GROUP BY quickly squishes all the matching rows together, without evaluating each one. I normally use joins to quickly reduce the number of rows the query needs to evaluate, then I look at my grouping.
In this case you want to replace DISTINCT with grouping.
Try this;
SELECT sp.name, sp.longitude, sp.latitude, sp.atcoCode
FROM `transportdata`.stoppoints as sp
INNER JOIN `vehicledata`.gtfsstop_times as st ON sp.atcoCode = st.fk_atco_code
INNER JOIN `vehicledata`.gtfstrips as trip ON st.trip_id = trip.trip_id
INNER JOIN `vehicledata`.gtfsroutes as route ON trip.route_id = route.route_id
INNER JOIN `vehicledata`.gtfsagencys as agency ON route.agency_id = agency.agency_id
WHERE agency.agency_id IN (1,2,3,4)
GROUP BY sp.name
, sp.longitude
, sp.latitude
, sp.atcoCode
There other valuable answers to your question and mine is an addition to it. I assume sp.atcoCode and st.fk_atco_code are indexed columns in their table.
If you can validate and make sure that agency ids in the WHERE clause are valid, you can eliminate joining `vehicledata.gtfsagencys` in the JOINS as you are not fetching any records from the table.
SELECT DISTINCT sp.atcoCode, sp.name, sp.longitude, sp.latitude
FROM `transportdata`.stoppoints as sp
INNER JOIN `vehicledata`.gtfsstop_times as st ON sp.atcoCode = st.fk_atco_code
INNER JOIN `vehicledata`.gtfstrips as trip ON st.trip_id = trip.trip_id
INNER JOIN `vehicledata`.gtfsroutes as route ON trip.route_id = route.route_id
WHERE route.agency_id IN (1,2,3,4);
I have the following query:
SELECT c.*
FROM companies AS c
JOIN users AS u USING(companyid)
JOIN jobs AS j USING(userid)
JOIN useraccounts AS us USING(userid)
WHERE j.jobid = 123;
I have the following questions:
Is the USING syntax synonymous with ON syntax?
Are these joins evaluated left to right? In other words, does this query say: x = companies JOIN users; y = x JOIN jobs; z = y JOIN useraccounts;
If the answer to question 2 is yes, is it safe to assume that the companies table has companyid, userid and jobid columns?
I don't understand how the WHERE clause can be used to pick rows on the companies table when it is referring to the alias "j"
Any help would be appreciated!
USING (fieldname) is a shorthand way of saying ON table1.fieldname = table2.fieldname.
SQL doesn't define the 'order' in which JOINS are done because it is not the nature of the language. Obviously an order has to be specified in the statement, but an INNER JOIN can be considered commutative: you can list them in any order and you will get the same results.
That said, when constructing a SELECT ... JOIN, particularly one that includes LEFT JOINs, I've found it makes sense to regard the third JOIN as joining the new table to the results of the first JOIN, the fourth JOIN as joining the results of the second JOIN, and so on.
More rarely, the specified order can influence the behaviour of the query optimizer, due to the way it influences the heuristics.
No. The way the query is assembled, it requires that companies and users both have a companyid, jobs has a userid and a jobid and useraccounts has a userid. However, only one of companies or user needs a userid for the JOIN to work.
The WHERE clause is filtering the whole result -- i.e. all JOINed columns -- using a column provided by the jobs table.
I can't answer the bit about the USING syntax. That's weird. I've never seen it before, having always used an ON clause instead.
But what I can tell you is that the order of JOIN operations is determined dynamically by the query optimizer when it constructs its query plan, based on a system of optimization heuristics, some of which are:
Is the JOIN performed on a primary key field? If so, this gets high priority in the query plan.
Is the JOIN performed on a foreign key field? This also gets high priority.
Does an index exist on the joined field? If so, bump the priority.
Is a JOIN operation performed on a field in WHERE clause? Can the WHERE clause expression be evaluated by examining the index (rather than by performing a table scan)? This is a major optimization opportunity, so it gets a major priority bump.
What is the cardinality of the joined column? Columns with high cardinality give the optimizer more opportunities to discriminate against false matches (those that don't satisfy the WHERE clause or the ON clause), so high-cardinality joins are usually processed before low-cardinality joins.
How many actual rows are in the joined table? Joining against a table with only 100 values is going to create less of a data explosion than joining against a table with ten million rows.
Anyhow... the point is... there are a LOT of variables that go into the query execution plan. If you want to see how MySQL optimizes its queries, use the EXPLAIN syntax.
And here's a good article to read:
http://www.informit.com/articles/article.aspx?p=377652
ON EDIT:
To answer your 4th question: You aren't querying the "companies" table. You're querying the joined cross-product of ALL four tables in your FROM and USING clauses.
The "j.jobid" alias is just the fully-qualified name of one of the columns in that joined collection of tables.
In MySQL, it's often interesting to ask the query optimizer what it plans to do, with:
EXPLAIN SELECT [...]
See "7.2.1 Optimizing Queries with EXPLAIN"
Here is a more detailed answer on JOIN precedence. In your case, the JOINs are all commutative. Let's try one where they aren't.
Build schema:
CREATE TABLE users (
name text
);
CREATE TABLE orders (
order_id text,
user_name text
);
CREATE TABLE shipments (
order_id text,
fulfiller text
);
Add data:
INSERT INTO users VALUES ('Bob'), ('Mary');
INSERT INTO orders VALUES ('order1', 'Bob');
INSERT INTO shipments VALUES ('order1', 'Fulfilling Mary');
Run query:
SELECT *
FROM users
LEFT OUTER JOIN orders
ON orders.user_name = users.name
JOIN shipments
ON shipments.order_id = orders.order_id
Result:
Only the Bob row is returned
Analysis:
In this query the LEFT OUTER JOIN was evaluated first and the JOIN was evaluated on the composite result of the LEFT OUTER JOIN.
Second query:
SELECT *
FROM users
LEFT OUTER JOIN (
orders
JOIN shipments
ON shipments.order_id = orders.order_id)
ON orders.user_name = users.name
Result:
One row for Bob (with the fulfillment data) and one row for Mary with NULLs for fulfillment data.
Analysis:
The parenthesis changed the evaluation order.
Further MySQL documentation is at https://dev.mysql.com/doc/refman/5.5/en/nested-join-optimization.html
SEE http://dev.mysql.com/doc/refman/5.0/en/join.html
AND start reading here:
Join Processing Changes in MySQL 5.0.12
Beginning with MySQL 5.0.12, natural joins and joins with USING, including outer join variants, are processed according to the SQL:2003 standard. The goal was to align the syntax and semantics of MySQL with respect to NATURAL JOIN and JOIN ... USING according to SQL:2003. However, these changes in join processing can result in different output columns for some joins. Also, some queries that appeared to work correctly in older versions must be rewritten to comply with the standard.
These changes have five main aspects:
The way that MySQL determines the result columns of NATURAL or USING join operations (and thus the result of the entire FROM clause).
Expansion of SELECT * and SELECT tbl_name.* into a list of selected columns.
Resolution of column names in NATURAL or USING joins.
Transformation of NATURAL or USING joins into JOIN ... ON.
Resolution of column names in the ON condition of a JOIN ... ON.
Im not sure about the ON vs USING part (though this website says they are the same)
As for the ordering question, its entirely implementation (and probably query) specific. MYSQL most likely picks an order when compiling the request. If you do want to enforce a particular order you would have to 'nest' your queries:
SELECT c.*
FROM companies AS c
JOIN (SELECT * FROM users AS u
JOIN (SELECT * FROM jobs AS j USING(userid)
JOIN useraccounts AS us USING(userid)
WHERE j.jobid = 123)
)
as for part 4: the where clause limits what rows from the jobs table are eligible to be JOINed on. So if there are rows which would join due to the matching userids but don't have the correct jobid then they will be omitted.
1) Using is not exactly the same as on, but it is short hand where both tables have a column with the same name you are joining on... see: http://www.java2s.com/Tutorial/MySQL/0100__Table-Join/ThekeywordUSINGcanbeusedasareplacementfortheONkeywordduringthetableJoins.htm
It is more difficult to read in my opinion, so I'd go spelling out the joins.
3) It is not clear from this query, but I would guess it does not.
2) Assuming you are joining through the other tables (not all directly on companyies) the order in this query does matter... see comparisons below:
Origional:
SELECT c.*
FROM companies AS c
JOIN users AS u USING(companyid)
JOIN jobs AS j USING(userid)
JOIN useraccounts AS us USING(userid)
WHERE j.jobid = 123
What I think it is likely suggesting:
SELECT c.*
FROM companies AS c
JOIN users AS u on u.companyid = c.companyid
JOIN jobs AS j on j.userid = u.userid
JOIN useraccounts AS us on us.userid = u.userid
WHERE j.jobid = 123
You could switch you lines joining jobs & usersaccounts here.
What it would look like if everything joined on company:
SELECT c.*
FROM companies AS c
JOIN users AS u on u.companyid = c.companyid
JOIN jobs AS j on j.userid = c.userid
JOIN useraccounts AS us on us.userid = c.userid
WHERE j.jobid = 123
This doesn't really make logical sense... unless each user has their own company.
4.) The magic of sql is that you can only show certain columns but all of them are their for sorting and filtering...
if you returned
SELECT c.*, j.jobid....
you could clearly see what it was filtering on, but the database server doesn't care if you output a row or not for filtering.