I used the following query with MySQL 5.5 (or previous versions) for years without any problems:
SELECT t2.Code from (select Country.Code from Country order by Country.Code desc ) AS t2;
The order of the result was always descending as I needed.
Last week, I just migrated to a new MySQL Version (In fact, I migrated to MariaDB 10.0.14) and now the same query with the same database is not sorted descending anymore. It is sorted ascending (or sorted using the natural order, not sure in fact).
So, can somebody could tell me if this is a bug or if this is a change of the behaviour in recent versions of MySQL/MariaDB?
After a bit of digging, I can confirm both your scenarios:
MySQL 5.1 does apply the ORDER BY inside the subquery.
MariaDB 5.5.39 on Linux does not apply the ORDER BY inside the subquery when no LIMIT is supplied. It does however correctly apply the order when a corresponding LIMIT is given:
SELECT t2.Code
FROM (
SELECT Country.Code FROM Country ORDER BY Country.Code DESC LIMIT 2
) AS t2;
Without that LIMIT, there isn't a good reason to apply the sort inside the subquery. It can be equivalently applied to the outer query.
Documented behavior:
As it turns out, MariaDB has documented this behavior and it is not regarded as a bug:
A "table" (and subquery in the FROM clause too) is - according to the SQL standard - an unordered set of rows. Rows in a table (or in a subquery in the FROM clause) do not come in any specific order. That's why the optimizer can ignore the ORDER BY clause that you have specified. In fact, SQL standard does not even allow the ORDER BY clause to appear in this subquery (we allow it, because ORDER BY ... LIMIT ... changes the result, the set of rows, not only their order).
You need to treat the subquery in the FROM clause, as a set of rows in some unspecified and undefined order, and put the ORDER BY on the top-level SELECT.
So MariaDB also recommends applying the ORDER BY in the outermost query, or a LIMIT if necessary.
Note: I don't currently have access to a proper MySQL 5.5 or 5.6 to confirm if the behavior is the same there (and SQLFiddle.com is malfunctioning). Comments on the original bug report (closed as not-a-bug) suggest that MySQL 5.6 probably behaves the same way as MariaDB.
In newer versions of MySQL and MariaDB you can force the ORDER BY in a sub query by applying a LIMIT. If you don't want to limit the rows, use the biggest number of BIGINT as a LIMIT.
This may come in handy at times, when the sub query needs to be generated in a desired order, for applying line numbers, for example.
Related
I need you help to resolve a problem..
Before, my website used MySql 5.5
Now, it seems to use MariaDB 10.0
I found no difference but...
This request (I have simplified the request for a better understanding)
select * from ( select * from MYTABLE ) tmpTable ORDER BY tmpTable.id DESC
This request WORKS on Mysql and MariaDB
BUT...
select * from ( select * from MYTABLE ORDER BY tmpTable.id DESC) tmpTable
I think if my order by is inside my seconde select, he is not concidered
This request DOESN'T WORK ! Result is good, but ORDER BY doesn't work ... It's order by ASCENDING and not DESCENDING like I specified in my second select ...
Someone understand why ? Is it a difference between mysql and Maria DB?
Thanks a lot !
Have a good day
In SQL, rows of a table have no pre-defined order. You need order by to sort a record set.
What happens with the second query is that the subquery creates a derived table that is then used in the outer query. The fact that you order the rows in the subquery does not make a difference: from the perspective of the outer query, rows of the derived table have no inherent ordering.
In other words there is no guarantee that the inner sort propagates to the outer scope. If you want the resultset to be consistently sorted, use order by in the outer scope.
This is more of an academic question, because in my particular case I can create an easy workaround, but I would like to understand the reason behind this anyway.
Using an InnoDB table (MariaDB 10.0.31) with (among others) columns customer and uri, I wanted to select the distinct uris for a specific customer. Now, the table is quite large (around 50M entries), so there is a composite index on customer and uri.
Basically what I don't understand is why the order of the columns in the group by clause matters.
explain select customer, uri from `tableName` group by customer,uri;
tells me it will use the existing index for group by, but
explain select customer, uri from `tableName` group by uri,customer;
won't do so.
Could someone explain why this is the case? I always thought of the group by clause as declarative.
Maybe it's because it's Friday, but I can't think of a case, where the order of the group by columns would affect the result.
Your observation is correct. Results would be different as the "prefix" order of columns mentioned in the composite index declaration is used for decision making by the Cost based optimizer. This behavior is due to the usage of B-TREE index
GROUP BY clause is used for ordering the result and hence if
the correct order of index is used or
only leftmost columns are used in group by
leftmost column is used in WHERE clause and rest in correct order in GROUP BY clause index would be used.
More on this and topic of Loose/Tight Index Scan can be found here
https://dev.mysql.com/doc/refman/5.7/en/group-by-optimization.html
In index is basically an ordered table. In you case it is ordered according like ORDER BY customer, uri (because this is how your index is defined).
MySQL executes group by by first ordering the result according to the group by clause and then collapsing the rows with the same values (that happen to follow each other after sorting).
Apparently, MySQL is not smart enough to recognize that the different group by clause could also be executed when the result is ordered the other way.
More about this:
http://use-the-index-luke.com/sql/sorting-grouping
In particular: http://use-the-index-luke.com/sql/sorting-grouping/indexed-group-by
Write a feature request at bugs.mysql.com .
On the one hand, GROUP BY is (or was) defined to imply ORDER BY with the same columns in the same order.
On the other hand, if you ignore that non-standard feature, even by saying ORDER BY NULL, MySQL fails to shuffle the columns in order to use the index.
5.7 (and before) says
GROUP BY implicitly sorts by default (that is, in the absence of ASC
or DESC designators), but relying on implicit GROUP BY sorting is
deprecated. To produce a given sort order, use explicit ASC or DESC
designators for GROUP BY columns or provide an ORDER BY clause. GROUP
BY sorting is a MySQL extension that may change in a future release;
for example, to make it possible for the optimizer to order groupings
in whatever manner it deems most efficient and to avoid the sorting
overhead.
and
If a query includes GROUP BY but you want to avoid the overhead of
sorting the result, you can suppress sorting by specifying ORDER BY
NULL.
But, watch out; 8.0 says
Previously, relying on implicit GROUP BY sorting was deprecated but
GROUP BY did sort by default (that is, in the absence of ASC or DESC
designators). In MySQL 8.0, GROUP BY no longer sorts by default, so
query results may differ from previous MySQL versions. To produce a
given sort order, use explicit ASC or DESC designators for GROUP BY
columns or provide an ORDER BY clause.
I have this query.
SELECT * FROM (SELECT * FROM private_messages ORDER BY id DESC) a
The "DESC" works on my local server but doesn't work on another server.
However if i just write it like this:
SELECT * FROM private_messages ORDER BY id DESC
it works on both servers. What would cause this?
This is going to be too long for a comment..
It's not a bug. It's documented here:
As of MySQL 5.7.6, the optimizer handles propagation of an ORDER BY
clause in a derived table or view reference to the outer query block
by propagating the ORDER BY clause if the following conditions apply:
The outer query is not grouped or aggregated; does not specify
DISTINCT, HAVING, or ORDER BY; and has this derived table or view
reference as the only source in the FROM clause. Otherwise, the
optimizer ignores the ORDER BY clause. Before MySQL 5.7.6, the
optimizer always propagated ORDER BY, even if it was irrelevant or
resulted in an invalid query.
However it's not the case for your query. So i guess your second server is running MariaDB which seams to ingnore any ORDER BY in a subquery without LIMIT
A "table" (and subquery in the FROM clause too) is - according to the
SQL standard - an unordered set of rows. Rows in a table (or in a
subquery in the FROM clause) do not come in any specific order. That's
why the optimizer can ignore the ORDER BY clause that you have
specified. In fact, SQL standard does not even allow the ORDER BY
clause to appear in this subquery (we allow it, because ORDER BY ...
LIMIT ... changes the result, the set of rows, not only their order).
You need to treat the subquery in the FROM clause, as a set of rows in
some unspecified and undefined order, and put the ORDER BY on the
top-level SELECT.
Why is ORDER BY in a FROM Subquery Ignored?
So best you can do is just to move the ORDER BY clause to the outer query. Or don't use a subquery at all.
It seems like in version 5.7 of MySQL, they added one nasty thing which was (or still is) a real headache for those who deal with SQL Server.
The thing is: MySQL throws an error, when you try to SELECT DISTINCT rows for one set of columns and want to ORDER BY another set of columns. Previously, in version 5.6 and even in some builds of version 5.7 you could do this, but now it is prohibited (at least by default).
I hope there exists some configuration, some variable that we could set to make it work. But unfortunately I do not know that nasty variable. I hope someone knows that.
EDIT
This is some typical query in my case that worked literally for years (until the last build of MySQL 5.7):
SELECT DISTINCT a.attr_one, a.attr_two, a.attr_three, b.attr_four FROM table_one a
LEFT JOIN table_two b ON b.some_idx = a.idx
ORDER BY b.id_order
And, indeed, if I now include b.id_order to the SELECT part (as MySQL suggests doing), then what I will get, will be rubbish.
In most cases, a DISTINCT clause can be considered as a special case of GROUP BY. For example,
ONLY_FULL_GROUP_BY
MySQL 5.7.5 and up implements detection of functional dependence. If the ONLY_FULL_GROUP_BY SQL mode is enabled (which it is by default), MySQL rejects queries for which the select list, HAVING condition, or ORDER BY list refer to nonaggregated columns that are neither named in the GROUP BY clause nor are functionally dependent on them. (Before 5.7.5, MySQL does not detect functional dependency and ONLY_FULL_GROUP_BY is not enabled by default. For a description of pre-5.7.5 behavior )
If ONLY_FULL_GROUP_BY is disabled, a MySQL extension to the standard SQL use of GROUP BY permits the select list, HAVING condition, or ORDER BY list to refer to nonaggregated columns even if the columns are not functionally dependent on GROUP BY columns. This causes MySQL to accept the preceding query. In this case, the server is free to choose any value from each group, so unless they are the same, the values chosen are indeterminate, which is probably not what you want. Furthermore, the selection of values from each group cannot be influenced by adding an ORDER BY clause. Result set sorting occurs after values have been chosen, and ORDER BY does not affect which value within each group the server chooses. Disabling ONLY_FULL_GROUP_BY is useful primarily when you know that, due to some property of the data, all values in each nonaggregated column not named in the GROUP BY are the same for each group.
for more http://dev.mysql.com/doc/refman/5.7/en/sql-mode.html#sqlmode_only_full_group_by
for particular answer
SELECT DISTINCT attr_one,
attr_two,
attr_three,
attr_four
FROM
(SELECT a.attr_one,
a.attr_two,
a.attr_three,
b.attr_four
FROM table_one a
LEFT JOIN table_two b ON b.some_idx = a.idx
ORDER BY b.id_order) tmp
I have read the post on the link you mentioned, and looks like been given the clear explanation of why the error is thrown and how to avoid it.
In your case you may want to try the following (not tested of course):
SELECT a.attr_one, a.attr_two, a.attr_three, b.attr_four
FROM table_one a
LEFT JOIN table_two b ON b.some_idx = a.idx
GROUP BY a.attr_one, a.attr_two, a.attr_three, b.attr_four
ORDER BY max(b.id_order)
You should choose whether to use ORDER BY max(b.id_order), or ORDER BY min(b.id_order) or other aggregate function
I used the following query with MySQL 5.5 (or previous versions) for years without any problems:
SELECT t2.Code from (select Country.Code from Country order by Country.Code desc ) AS t2;
The order of the result was always descending as I needed.
Last week, I just migrated to a new MySQL Version (In fact, I migrated to MariaDB 10.0.14) and now the same query with the same database is not sorted descending anymore. It is sorted ascending (or sorted using the natural order, not sure in fact).
So, can somebody could tell me if this is a bug or if this is a change of the behaviour in recent versions of MySQL/MariaDB?
After a bit of digging, I can confirm both your scenarios:
MySQL 5.1 does apply the ORDER BY inside the subquery.
MariaDB 5.5.39 on Linux does not apply the ORDER BY inside the subquery when no LIMIT is supplied. It does however correctly apply the order when a corresponding LIMIT is given:
SELECT t2.Code
FROM (
SELECT Country.Code FROM Country ORDER BY Country.Code DESC LIMIT 2
) AS t2;
Without that LIMIT, there isn't a good reason to apply the sort inside the subquery. It can be equivalently applied to the outer query.
Documented behavior:
As it turns out, MariaDB has documented this behavior and it is not regarded as a bug:
A "table" (and subquery in the FROM clause too) is - according to the SQL standard - an unordered set of rows. Rows in a table (or in a subquery in the FROM clause) do not come in any specific order. That's why the optimizer can ignore the ORDER BY clause that you have specified. In fact, SQL standard does not even allow the ORDER BY clause to appear in this subquery (we allow it, because ORDER BY ... LIMIT ... changes the result, the set of rows, not only their order).
You need to treat the subquery in the FROM clause, as a set of rows in some unspecified and undefined order, and put the ORDER BY on the top-level SELECT.
So MariaDB also recommends applying the ORDER BY in the outermost query, or a LIMIT if necessary.
Note: I don't currently have access to a proper MySQL 5.5 or 5.6 to confirm if the behavior is the same there (and SQLFiddle.com is malfunctioning). Comments on the original bug report (closed as not-a-bug) suggest that MySQL 5.6 probably behaves the same way as MariaDB.
In newer versions of MySQL and MariaDB you can force the ORDER BY in a sub query by applying a LIMIT. If you don't want to limit the rows, use the biggest number of BIGINT as a LIMIT.
This may come in handy at times, when the sub query needs to be generated in a desired order, for applying line numbers, for example.