Correct format for Select in SQL Server - mysql

I have what should be a simple query for any database and which always runs in MySQL but not in SQL Server
select
tagalerts.id,
ts,
assetid,
node.zonename,
battlevel
from tagalerts, node
where
ack=0 and
tagalerts.nodeid=node.id
group by assetid
order by ts desc
The error is:
column tagalerts.id is invalid in the select list because it is not contained in either an aggregate function or the group by clause.
It is not a simple case of adding tagalerts.id to the group by clause because the error repeats for ts and for assetid etc, implying that all the selects need to be in a group or in aggregate functions... either of which will result in a meaningless and inaccurate result.
Splitting the select into a subquery to sort and group correctly (which again works fine with MySQL, as you would expect) makes matters worse
SELECT * from
(select
tagalerts.id,
ts,
assetid,
node.zonename,
battlevel
from tagalerts, node
where
ack=0 and
tagalerts.nodeid=node.id
order by ts desc
)T1
group by assetid
the order by clause is invalid in views, inline functions, derived tables and expressions unless TOP etc is used
the 'correct output' should be
id ts assetid zonename battlevel
1234 a datetime 1569 Reception 0
3182 another datetime 1572 Reception 0
Either I am reading SQL Server's rules entirely wrong or this is a major flaw with that database.
How can I write this to work on both systems?

In most databases you can't just include columns that aren't in the GROUP BY without using an aggregate function.
MySql is an exception to that. But MS SQL Server isn't.
So you could keep that GROUP BY with only the "assetid".
But then use the appropriate aggregate functions for all the other columns.
Also, use the JOIN syntax for heaven's pudding sake.
A SQL like select * from table1, table2 where table1.id2 = table2.id is using a syntax from the previous century.
SELECT
MAX(node.id) AS id,
MAX(ta.ts) AS ts,
ta.assetid,
MAX(node.zonename) AS zonename,
MAX(ta.battlevel) AS battlevel
FROM tagalerts AS ta
JOIN node ON node.id = ta.nodeid
WHERE ta.ack = 0
GROUP BY ta.assetid
ORDER BY ta.ts DESC;
Another trick to use in MS SQL Server is the window function ROW_NUMBER.
But this is probably not what you need.
Example:
SELECT id, ts, assetid, zonename, battlevel
FROM
(
SELECT
node.id,
ta.ts,
ta.assetid,
node.zonename,
ta.battlevel,
ROW_NUMBER() OVER (PARTITION BY ta.assetid ORDER BY ta.ts DESC) AS rn
FROM tagalerts AS ta
JOIN node ON node.id = ta.nodeid
WHERE ta.ack = 0
) q
WHERE rn = 1
ORDER BY ts DESC;

I strongly suspect this query is WRONG even in MySql.
We're missing a lot of details (sample data, and we don't know which table all of the columns belong to), but what I do know is you're grouping by assetid, where it looks like one assetid value could have more than one ts (timestamp) value in the group. It also looks like you're counting on the order by ts desc to ensure both that you see recent timestamps in the results first and that each assetid group uses the most recent possible ts timestamp for that group.
MySql only guarantees the former, not the latter. Nothing in this query guarantees that each assetid is using the most recent timestamp available. You could be seeing the wrong timestamps, and then also using those wrong timestamps for the order by. This is the problem the Sql Server rule is there to stop. MySql violates the SQL standard to allow you to write that wrong query.
Instead, you need to look at each column and either add it to the group by (best when all of the values are known to be the same, anyway) or wrap it in an aggregrate function like MAX(), MIN(), AVG(), etc, so there is a deterministic result for which value from the group is used.
If all of the values for a column in a group are the same, then there's no problem adding it to the group by. If the values are different, you want to be precise about which one is chosen for the result set.
While I'm here, the tagalerts, node join syntax has been obsolete for more than 20 years now. It's also good practice to use an alias with every table and prefix every column with the alias. I mention these to explain why I changed it for my code sample below, though I only prefix columns where I am confident in which table the column belongs to.
This query should run on both databases:
SELECT ta.assetid, MAX(ta.id) "id", MAX(ta.ts) "ts",
MAX(n.zonename) "zonename", MAX(battlevel) "battlevel"
FROM tagalerts ta
INNER JOIN node n ON ta.nodeid = n.id
WHERE ack = 0
GROUP BY ta.assetid
ORDER BY ts DESC
There is also a concern here the results may be choosing values from different records in the joined node table. So if battlevel is part of the node table, you might see a result that matches a zonename with a battlevel that never occurs in any record in the data. In Sql Server, this is easily fixed by using APPLY to match only one node record to each tagalert. MySql doesn't support this (APPLY or an equivalent has been in every other major database since at least 2012), but you can simulate with it in this case with two JOINs, where the first join is a subquery that uses GROUP BY to determine values will uniquely identify the needed node record, and second join is to the node table to actually produce that record. Unfortunately, we need to know more about the tables in question to actually write this code for you.

Related

Order of JOIN and WHERE [duplicate]

I am confused with the order of execution of this query, please explain me this.
I am confused with when the join is applied, function is called, a new column is added with the Case and when the serial number is added. Please explain the order of execution of all this.
select Row_number() OVER(ORDER BY (SELECT 1)) AS 'Serial Number',
EP.FirstName,Ep.LastName,[dbo].[GetBookingRoleName](ES.UserId,EP.BookingRole) as RoleName,
(select top 1 convert(varchar(10),eventDate,103)from [3rdi_EventDates] where EventId=13) as EventDate,
(CASE [dbo].[GetBookingRoleName](ES.UserId,EP.BookingRole)
WHEN '90 Day Client' THEN 'DC'
WHEN 'Association Client' THEN 'DC'
WHEN 'Autism Whisperer' THEN 'DC'
WHEN 'CampII' THEN 'AD'
WHEN 'Captain' THEN 'AD'
WHEN 'Chiropractic Assistant' THEN 'AD'
WHEN 'Coaches' THEN 'AD'
END) as Category from [3rdi_EventParticipants] as EP
inner join [3rdi_EventSignup] as ES on EP.SignUpId = ES.SignUpId
where EP.EventId = 13
and userid in (
select distinct userid from userroles
--where roleid not in(6,7,61,64) and roleid not in(1,2))
where roleid not in(19, 20, 21, 22) and roleid not in(1,2))
This is the function which is called from the above query.
CREATE function [dbo].[GetBookingRoleName]
(
#UserId as integer,
#BookingId as integer
)
RETURNS varchar(20)
as
begin
declare #RoleName varchar(20)
if #BookingId = -1
Select Top 1 #RoleName=R.RoleName From UserRoles UR inner join Roles R on UR.RoleId=R.RoleId Where UR.UserId=#UserId and R.RoleId not in(1,2)
else
Select #RoleName= RoleName From Roles where RoleId = #BookingId
return #RoleName
end
Queries are generally processed in the follow order (SQL Server). I have no idea if other RDBMS's do it this way.
FROM [MyTable]
ON [MyCondition]
JOIN [MyJoinedTable]
WHERE [...]
GROUP BY [...]
HAVING [...]
SELECT [...]
ORDER BY [...]
SQL is a declarative language. The result of a query must be what you would get if you evaluated as follows (from Microsoft):
Logical Processing Order of the SELECT statement
The following steps show the logical
processing order, or binding order,
for a SELECT statement. This order
determines when the objects defined in
one step are made available to the
clauses in subsequent steps. For
example, if the query processor can
bind to (access) the tables or views
defined in the FROM clause, these
objects and their columns are made
available to all subsequent steps.
Conversely, because the SELECT clause
is step 8, any column aliases or
derived columns defined in that clause
cannot be referenced by preceding
clauses. However, they can be
referenced by subsequent clauses such
as the ORDER BY clause. Note that the
actual physical execution of the
statement is determined by the query
processor and the order may vary from
this list.
FROM
ON
JOIN
WHERE
GROUP BY
WITH CUBE or WITH ROLLUP
HAVING
SELECT
DISTINCT
ORDER BY
TOP
The optimizer is free to choose any order it feels appropriate to produce the best execution time. Given any SQL query, is basically impossible to anybody to pretend it knows the execution order. If you add detailed information about the schema involved (exact tables and indexes definition) and the estimated cardinalities (size of data and selectivity of keys) then one can take a guess at the probable execution order.
Ultimately, the only correct 'order' is the one described ion the actual execution plan. See Displaying Execution Plans by Using SQL Server Profiler Event Classes and Displaying Graphical Execution Plans (SQL Server Management Studio).
A completely different thing though is how do queries, subqueries and expressions project themselves into 'validity'. For instance if you have an aliased expression in the SELECT projection list, can you use the alias in the WHERE clause? Like this:
SELECT a+b as c
FROM t
WHERE c=...;
Is the use of c alias valid in the where clause? The answer is NO. Queries form a syntax tree, and a lower branch of the tree cannot be reference something defined higher in the tree. This is not necessarily an order of 'execution', is more of a syntax parsing issue. It is equivalent to writing this code in C#:
void Select (int a, int b)
{
if (c = ...) then {...}
int c = a+b;
}
Just as in C# this code won't compile because the variable c is used before is defined, the SELECT above won't compile properly because the alias c is referenced lower in the tree than is actually defined.
Unfortunately, unlike the well known rules of C/C# language parsing, the SQL rules of how the query tree is built are somehow esoteric. There is a brief mention of them in Single SQL Statement Processing but a detailed discussion of how they are created, and what order is valid and what not, I don't know of any source. I'm not saying there aren't good sources, I'm sure some of the good SQL books out there cover this topic.
Note that the syntax tree order does not match the visual order of the SQL text. For example the ORDER BY clause is usually the last in the SQL text, but as a syntax tree it sits above everything else (it sorts the output of the SELECT, so it sits above the SELECTed columns so to speak) and as such is is valid to reference the c alias:
SELECT a+b as c
FROM t
ORDER BY c;
SQL query is not imperative but declarative, so you have no idea which the statement is executed first, but since SQL is evaluated by SQL query engines, most of the SQL engines follows similar process to obtain the results. You may have to understand how the query engine works internally to understand some SQL execution behavior.
Julia Evens has a great post explaining this, it is worth to check it out:
https://jvns.ca/blog/2019/10/03/sql-queries-don-t-start-with-select/
SQL is a declarative language, meaning that it tells the SQL engine what to do, not how. This is in contrast to an imperative language such as C, in which how to do something is clearly laid out.
This means that not all statements will execute as expected. Of particular note are boolean expressions, which may not evaluate from left-to-right as written. For example, the following code is not guaranteed to execute without a divide by zero error:
SELECT 'null' WHERE 1 = 1 OR 1 / 0 = 0
The reason for this is the query optimizer chooses the best (most efficient) way to execute a statement. This means that, for example, a value may be loaded and filtered before a transforming predicate is applied, causing an error. See the second link above for an example
See: here and here.
"Order of execution" is probably a bad mental model for SQL queries. Its hard to actually write a single query that would actually depend on order of execution (this is a good thing). Instead you should think of all join and where clauses happening simultaneously (almost like a template)
That said you could run display the Execution Plans which should give you insight into it.
However since its's not clear why you want to know the order of execution, I'm guessing your trying to get a mental model for this query so you can fix it in some way. This is how I would "translate" your query, although I've done well with this kind of analysis there's some grey area with how precise it is.
FROM AND WHERE CLAUSE
Give me all the Event Participants rows. from [3rdi_EventParticipants
Also give me all the Event Signup rows that match the Event Participants rows on SignUpID inner join 3rdi_EventSignup] as ES on EP.SignUpId = ES.SignUpId
But Only for Event 13 EP.EventId = 13
And only if the user id has a record in the user roles table where the role id is not in 1,2,19,20,21,22
userid in (
select distinct userid from userroles
--where roleid not in(6,7,61,64) and roleid not in(1,2))
where roleid not in(19, 20, 21, 22) and roleid not in(1,2))
SELECT CLAUSE
For each of the rows give me a unique ID
Row_number() OVER(ORDER BY (SELECT 1)) AS 'Serial Number',
The participants First Name EP.FirstName
The participants Last Name Ep.LastName
The Booking Role name GetBookingRoleName
Go look in the Event Dates and find out what the first eventDate where the EventId = 13 that you find
(select top 1 convert(varchar(10),eventDate,103)from [3rdi_EventDates] where EventId=13) as EventDate
Finally translate the GetBookingRoleName in Category. I don't have a table for this so I'll map it manually (CASE [dbo].[GetBookingRoleName](ES.UserId,EP.BookingRole)
WHEN '90 Day Client' THEN 'DC'
WHEN 'Association Client' THEN 'DC'
WHEN 'Autism Whisperer' THEN 'DC'
WHEN 'CampII' THEN 'AD'
WHEN 'Captain' THEN 'AD'
WHEN 'Chiropractic Assistant' THEN 'AD'
WHEN 'Coaches' THEN 'AD'
END) as Category
So a couple of notes here. You're not ordering by anything when you select TOP. You should probably have na order by there. You could also just as easily put that in your from clause e.g.
from [3rdi_EventParticipants] as EP
inner join [3rdi_EventSignup] as ES on EP.SignUpId = ES.SignUpId,
(select top 1 convert(varchar(10),eventDate,103)
from [3rdi_EventDates] where EventId=13
Order by eventDate) dates
There is a logical order to evaluation of the query text, but the database engine can choose what order execute the query components based upon what is most optimal. The logical text parsing ordering is listed below. That is, for example, why you can't use an alias from SELECT clause in a WHERE clause. As far as the query parsing process is concerned, the alias doesn't exist yet.
FROM
ON
OUTER
WHERE
GROUP BY
CUBE | ROLLUP
HAVING
SELECT
DISTINCT
ORDER BY
TOP
See the Microsoft documentation (see "Logical Processing Order of the SELECT statement") for more information on this.
Simplified order for T-SQL -> SELECT statement:
1) FROM
2) Cartesian product
3) ON
4) Outer rows
5) WHERE
6) GROUP BY
7) HAVING
8) SELECT
9) Evaluation phase in SELECT
10) DISTINCT
11) ORDER BY
12) TOP
as I had done so far - same order was applicable in SQLite.
Source => SELECT (Transact-SQL)
... of course there are (rare) exceptions.

Order by showing bad results in MariaDB

In my query, I need to get the previous row with the current row and then join a few tables. I got the previous row by using SQL variables in my development server(MySQL 5.7), everything works fine, but in my production(MariaDB 10) server that previous row results are bad just mixed, bad part is only that previous row with SQL variables other query parts works good. Before it i thought, that problem is in sql variables part, but now i realized the problem is in "order by" keyword.
My query:
SELECT
customers.title,
calendar.start_time,
calendar.hours_per_time,
calendar.self_certification,
calendar.bulletin_certification,
calendar.extra,
DATE_FORMAT(calendar.date, '%d-%m') AS day_month,
TIME_FORMAT(calendar.start_time, '%H:%i') AS hours_min,
#previous_start AS previous_start,
#previous_start := calendar.start_time,
#previous_end AS previous_end,
#previous_end := calendar.hours_per_time
FROM
(SELECT #previous_start := '00:00', #previous_end := '0.00') AS calendar_prev, calendar
INNER JOIN relationships ON calendar.relation_id = relationships.relation_id
INNER JOIN customers ON customers.customer_id = relationships.customer_id
WHERE relationships.user_id = '$user_id'
AND DATE_FORMAT(calendar.date, '%m-%Y') = '$date'
ORDER BY calendar.date, hours_min ASC
If i remove hours_min from order by part everything works fine in both servers, but then i lose my ordering.
This is my result from development server green part is is from sql variables and here works fine:
And here is from production server with bad results in red part
So how can I keep my order and still have good results? Order is only needed by date(Dato) and hours_min(Fra kl.) columns.
Wrap the query as an inline view, and specify a different ORDER BY on the outer query.
As a simple demonstration of the pattern:
SELECT v.fee
, v.fo
, v.fi
FROM (
SELECT t.fee
, t.fi
, t.fo
JOIN t
ORDER BY t.fi ASC, t.fo ASC
) v
ORDER
BY v.fee DESC
We can process rows "in order" in the query inside the inline view, using an ORDER BY clause on the inner SELECT statement.
The ORDER BY on the outer query can reorder the results returned by the inner query.
NOTES:
The MySQL Reference manual cautions that the behavior of user-defined variables that are set and read within the same statement is not guaranteed. With that said, we do observe a consistent behavior.
It's an order of operations issue. That is, we are carefully constructing our SQL in such a way that MySQL execution plan gets us a predictable order of operations.
What we have discovered is that the ORDER BY is being processed before the expressions in the SELECT list are evaluated.
So, if we need to process rows "in order" such that the user defined variables contain values from the "previous" row when the expressions in the SELECT list are evaluated, then we need to have an ORDER BY that gets the rows in the desired order.
If we want the resulting rows in a different order, we need another ORDER BY operation to be processed later. And we can get that using an inline view (what MySQL refers to as a "derived table"). That's because MySQL materializes that derived table "v" before the outer query is processed.
The SELECT list of the outer query can specify a different order of columns or omit columns. The order of expressions in the SELECT list of the inner query can be dictated by the order of operations required when working with the user-defined variables: the assignments that save the current row to the user-defined variables has to happen AFTER the user-defined variables are evaluated.
Also, I would recommend ditching the comma syntax for the join operation, and replace that with the JOIN keyword. The CROSS keyword is optional, but it does serve as an indication to the future reader that the omission of the ON clause is intentional, and not an oversight.
The INNER keyword is also optional; it has no effect, and my preference is to omit that.
FROM (SELECT #previous_start := '00:00', #previous_end := '0.00') calendar_prev
CROSS
JOIN calendar
JOIN relationships
ON calendar.relation_id = relationships.relation_id
Combining ORDER BY with variables in MariaDB is a bit tricky. Can you make a copy of the table already ordered by calendar.date, hours_min ASC and run the query on that copy without the ORDER BY?

MySQL ORDER BY from subquery lost by GROUP BY

I have a table x :
id lang externalid
1 nl 10
2 nl 11
3 fr 10
From this table I want al the rows for a certain lang and externalid, if the externalid doesn't exist for this lang, I want the row with any other lang.
The subquery sorts the table correct, but when I add the group by, the sort of the subquery is lost. This works in older mysql versions but not in 5.7.
(
SELECT
*
FROM
x
ORDER BY FIELD(lang, "fr") DESC, id
)
as y
group by externalid
I want the query to return the records with id 2 & 3. So for each distinct external id, if possible the lang = 'fr', else any other lang.
How can i solve this problem?
You are talking of given externalid and land. No need to group by externalid hence; use a mere where clause instead.
Combined with ORDER BY and LIMIT you get the record you want (i.e. the desired language if such a record exists, else another one).
select *
from mytable
where externalid = 10
order by lang = 'fr' desc
limit 1;
UPDATE: Okay, according to your comment you want to get the "best" record per externalid. In standard SQL you'd use ROW_NUMBER for this. Other DBMS have further solutions, e.g. Oracle's KEEP FIRST or Postgre's DISTINCT ON. MySQL doesn't support any of these. One way would be to emulate ROW_NUMBER with variables. Another would be to use above query as a subquery per externalid to find the best records:
select *
from mytable
where id in
(
select
(
select m.id
from mytable m
where m.externalid = e.externalid
order by m.lang = 'fr' desc
limit 1
) as best_id
from (select distinct externalid from mytable) e
);
Your subquery generates a result set (a virtual table) that's passed to your outer query.
All SQL queries, without exception, generate their results in unpredictable order unless you specify the order completely in an ORDER BY clause.
Unpredictable is like random, except worse. Random implies you'll get a different order every time you run the query. Unpredictable means you'll get the same order every time, until you don't.
MySQL ordinarily ignores ORDER BY clauses in subqueries (there are a few exceptions, mostly related to subquery LIMIT clauses). Move your ORDER BY to the top level query.
Edit. You are also misusing MySQL's notorious nonstandard extension to GROUP BY.

MySQL alias replaced by column name when creating view involving subquery

Why is a column alias being replaced by the original column name when I create a view from a script? The script works, the view fails.
The script selects records using an outer query / inner query a.k.a. query / subquery. The subquery is used in the SELECT clause. Each subquery is itself a SELECT clause which becomes a column in the result set. See http://www.techonthenet.com/mysql/subqueries.php.
The alias used inside the subquery's SELECT clauses is replaced with its original column name. The alias used to give the subquery a short name is not replaced.
Here is a meta version so you can see the structure.
select `t1`.`Date` as **`*When*`**,
( select avg(t1.column)
from t1
where `t1.`Date` = `***When***`
) as `Short column name`,
from t1
group by `Date`
order by `Date`
In the View version, with aliases replaced, the subquery becomes;
(
select avg(t1.column)
from t1
where ***`t1.`Date` = `t1.`Date`***
) as `Short column name`,
The effect of this is that the average is calculated across all dates rather than just for the date specified as When in the outer query.
Another script built the same way translates into a view without a problem. The alias is kept.
There is a difference between the clauses used in the bad and good views but it is not obvious to me that it should cause the problem.
The bad view ends with;
group by `Date`
order by `Date`
while the good one ends only with a group by clause.
Another difference is that the column being aliased in the bad view is probably of field type DATETIME, while the one in the good view ia probably one of the INT types. (it's actually week(t1.Date).
Using:
MySQL 5.5
MySQL Workbench 6.0.8
Ubuntu 14.04
The aliases in the SELECT refer to the output of the query block, not to the processing of the query block.
The correct way to do what you want is to use a correlated subquery with table aliases:
select touter.`Date` as **`*When*`**,
(select avg(tinner.column)
from t1 tinner
where `tinner.`Date` = touter.date
) as `Short column name`,
from t1 as touter
group by `Date`
order by `Date`;
I have no idea why the average would be calculated for all the dates. I would expect it to return an error, or perhaps a NULL value. Perhaps your real where clause is t1.Date = Date and you expect MySQL to magically know what the second Date refers to. Don't depend on magic. Use table aliases and be explicit.

wrapping inside aggregate function in SQL query

I have 2 tables called Orders and Salesperson shown below:
And I want to retrieve the names of all salespeople that have more than 1 order from the tables above.
Then firing following query shows an error:
SELECT Name
FROM Orders, Salesperson
WHERE Orders.salesperson_id = Salesperson.ID
GROUP BY salesperson_id
HAVING COUNT( salesperson_id ) >1
The error is:
Column 'Name' is invalid in the select list because it is
not contained in either an aggregate function or
the GROUP BY clause.
From the error and searching it on google, I could understand that the error is because of Name column must be either a part of the group by statement or aggregate function.
Also I tried to understand why does the selected column have to be in the group by clause or art of an aggregate function? But didn't understand clearly.
So, how to fix this error?
SELECT max(Name) as Name
FROM Orders, Salesperson
WHERE Orders.salesperson_id = Salesperson.ID
GROUP BY salesperson_id
HAVING COUNT( salesperson_id ) >1
The basic idea is that columns that are not in the group by clause need to be in an aggregate function now here due to the fact that the name is probably the same for every salesperson_id min or max make no real difference (the result is the same)
example
Looking at your data you have 3 entry's for Dan(7) now when a join is created the with row Dan (Name) gets multiplied by 3 (For every number 1 Dan) and then the server does not now witch "Dan" to pick cos to the server that are 3 lines even doh they are semantically the same
also try this so that you see what I am talking about:
SELECT Orders.Number, Salesperson.Name
FROM Orders, Salesperson
WHERE Orders.salesperson_id = Salesperson.ID
As far as the query goes INNER JOIN is a better solution since its kinda the standard for this simple query it should not matter but in some cases can happen that INNER JOIN produces better results but as far as I know this is more of a legacy thing since this days the server should pretty much produce the same execution plan.
For code clarity I would stick with INNER JOIN
Assuming the name is unique to the salesperson.id then simply add it to your group by clause
GROUP BY salesperson_id, salesperson.Name
Otherwise use any Agg function
Select Min(Name)
The reason for this is that SQL doesn't know whether there are multiple name per salesperson.id
For readability and correctness, I usually split aggregate queries into two parts:
The aggregate query
Any additional queries to support fields not contained in aggregate functions
So:
1.Aggregate query - salespeople with more than 1 order
SELECT salesperson_id
FROM ORDERS
GROUP BY salespersonId
HAVING COUNT(Number) > 1
2.Use aggregate as subquery (basically a select joining onto another select) to join on any additional fields:
SELECT *
FROM Salesperson SP
INNER JOIN
(
SELECT salesperson_id
FROM ORDERS
GROUP BY salespersonId
HAVING COUNT(Number) > 1
) AGG_QUERY
ON AGG_QUERY.salesperson_id = SP.ID
There are other approaches, such as selecting the additional fields via aggregation functions (as shown by the other answers). These get the code written quickly so if you are writing the query under time pressure you may prefer that approach. If the query needs to be maintained (and hence readable) I would favour subqueries.