Hello im going through a tutorial on SQL joins and this statement below is throwing "sql_mode=only_full_group_by" error. I copied this directly from the tutorial and its working for him. Do I just need to disable this mode in the terminal?
SELECT *
FROM customers
JOIN orders
ON customers.id = orders.customer_id
GROUP BY orders.customer_id;
If the ONLY_FULL_GROUP_BY SQL mode is enabled (which it is by default), MySQL rejects queries for which the select list, HAVING condition, or ORDER BY list refer to nonaggregated columns that are neither named in the GROUP BY clause nor are functionally dependent on them. [1]
You can disable this, but it really makes sense to leave it enabled. Think about the following case:
id | type
---|-----
1 | foo
2 | foo
3 | bar
If you used a similar query to yours, select * from table group by type, what would you expect the output to be? Perhaps you expect two rows, foo and bar, but what do you expect to be in the id column, namely for the foo record? Should we just choose the earliest record? The latest record? A random record? This is what the error is telling you - instead of having the database engine choose for you, you must be explicit.
So, in your example, you need to think about what the resulting data should look like. Maybe instead of select * you want select customers.name, count(*) ..., etc.
The reason this works for the tutorial you're following is either they don't have only_full_group_by enabled or they're using a MySql version older than 5.7.5 where this was introduced. In this case, the server is free to choose any value from each group, which is probably not what you want.
Finally, if you want your console to behave in the same way your tutorial is behaving, you can disable ONLY_FULL_GROUP_BY in the console by:
mysql > SET GLOBAL sql_mode=(SELECT REPLACE(##sql_mode,'ONLY_FULL_GROUP_BY',''));
[1] https://dev.mysql.com/doc/refman/5.7/en/group-by-handling.html
Related
In a ColdFusion application I have a query with several joins and I need one MAX number from one of the tables. I am getting all tangled up in 'only_full_group_by'. I cannot disable it in etc or in the Administrator, because I am on a shared system and have no access to those. I'd like to disable it at run time. Here's a simplified version of the query:
<cfquery name = 'test' datasource = 'whatever'>
select PersActExt,PersActPaid,PersActMdate,PersActbl,
PersTrId,PersTrMas,PersTrSub,PersTrCode,
MAX(PersTrPaid), MAX(PersTrDate)
from PersActiv
left join PersTrans on
PersTrId = PersActId and
PersTrMas = PersActMas and
PersTrSub = PersActSub and
PersTrCode = PersActCode
where PersActMas = 'bill'
group by PersTrId,PersTrMas,PersTrSub,PersTrCode
</cfquery>
The fact is the condition PersActMas = 'bill' reduces this to one record per Person in the PersAct table. I suppose I could put a MAX on all those Persact type fields, but there are other tables that also have to be joined in; and frankly I don't want a MAX on everything just to pick up one Max number in PersTr.
Is there a way to disable the ONLY_FULL_GROUP_BY mode within the query? Or better yet within the application? Or can anyone suggest another way around this problem. I know I can split this up into several queries and then stitch everything back together in ColdFusion, but that's a lot of code to get around this bad MySql mode.
Thinking laterally about the problem, could you restructure your query to join the PersAct table to a pre-aggregated subquery? This removes the need to perform a grouping on the outer query and running up against the limitation of your setup.
SELECT
PersActExt,
PersActPaid,
PersActMdate,
PersActbl,
PersTrId,
PersTrMas,
PersTrSub,
PersTrCode,
MaxPersTrPaid,
MaxPersTrDate
FROM
PersActiv
LEFT JOIN
(
SELECT
PersTrId,
PersTrMas,
PersTrSub,
PersTrCode,
MAX(PersTrPaid) AS MaxPersTrPaid,
MAX(PersTrDate) AS MaxPersTrDate
FROM
PersTrans
GROUP BY
PersTrId,
PersTrMas,
PersTrSub,
PersTrCode
) AS PersTr ON
PersTrId = PersActId and
PersTrMas = PersActMas and
PersTrSub = PersActSub and
PersTrCode = PersActCode
WHERE PersActMas = 'bill'
Edit:
For those wanting to know how to remove the 'ONLY_FULL_GROUP_BY' on an ad hoc basis, you need to add the below snippet to your query.
SET SESSION sql_mode = sys.list_drop(##session.sql_mode, 'ONLY_FULL_GROUP_BY');
This will only work if the DB user used in ColdFusion has the correct permissions and the option to allow multiple queries in a cfquery call is set.
Tested the query on MySQL 5.7 here: https://dbfiddle.uk/E1em1_k2
I tested with your query, with the last line as you wrote it:
...
group by PersTrId,PersTrMas,PersTrSub,PersTrCode;
This resulted in the error due to the ONLY_FULL_GROUP_BY:
Expression #1 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'test.PersActiv.PersActExt' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by
Then I tested with an alternative last line:
...
group by PersActId,PersActMas,PersActSub,PersActCode;
This works with the ONLY_FULL_GROUP_BY at the default value. See the demo for proof. Note that I had to do my alternative query first, because the dbfiddle aborts when it gets an error.
The reason is that MySQL 5.7 introduced the capability to infer functional dependencies when using GROUP BY on the primary key.
https://dev.mysql.com/doc/refman/5.7/en/group-by-handling.html
SQL:1999 and later permits such nonaggregates per optional feature T301 if they are functionally dependent on GROUP BY columns: If such a relationship exists between name and custid, the query is legal. This would be the case, for example, were custid a primary key of customers.
MySQL 5.7.5 and later implements detection of functional dependence.
Since the four columns in the GROUP BY are in fact the primary key of the PersActive table, MySQL can tell that other columns of that table are guaranteed to have one value per grouping.
There are multiple rows in PersTrans, but the functional dependencies are still analyzed, so the columns that are known to be equal to primary key columns of PersActiv are still okay to use in the query without aggregation. The other columns of PersTrans that are not functionally dependent are already in MAX() expressions in your query, which also satisfies the requirement.
I conclude that functional dependency analysis works in MySQL 5.7, and this would be an effective workaround in the case you show.
You wrote:
I'm used to a system where if there is an aggregated field and some fields are not aggregated, the first or last value of that the non-aggregated field is automatically used.
In my experience, this includes only MySQL prior to 5.7, or SQLite. It's illegitimate in the ANSI/ISO SQL standard, and other SQL implementations conform to the standard.
MySQL 5.7 does not have a "bad sql mode" — it has been corrected to enforce the rule per the SQL standard, like most other brands of SQL also do.
Back to your original question:
Is there a way to disable the ONLY_FULL_GROUP_BY mode within the query? Or better yet within the application?
You can change the sql mode globally, either in the my.cnf options file, or using SET GLOBAL sql_mode='...';. I would do both, because if you change options dynamically with SET GLOBAL, they are reset to whatever is in the options file the next time you restart the MySQL Server.
You can also change the sql mode in a session with SET SESSION sql_mode='...'; and it takes effect for the remainder of the current session. But SET SESSION is a separate statement that you would have to execute in the session prior to your query. It can't be combined with a query.
There is an option in MySQL 8.0 to use the "hint" syntax to set certain options for the scope of a single SQL query. But in your case, you're using MySQL 5.7, so you'd have to upgrade to get this feature.
A similar feature exists in Percona Server 5.6 and later to set a variable for one statement. This does not apply if you use stock MySQL. You'd have to use the Percona branch of MySQL.
Regardless, I recommend you do not change the sql mode. It's proper that it behaves the way it does, because it prevents queries that may inadvertently produce wrong results.
I have two tables and I need to create a view. The tables are:
credit_orders(id, client_id, number_of_credits, payment_status)
credit_usage(id, client_id, credits_used, date)
I use the following query to do this. The query without the "create view" part works well but with "create view", it shows the error "View's SELECT contains a subquery in the FROM clause". What could be the issue & possible solution:
create view view_credit_status as
(select credit_orders.client_id,
sum(credit_orders.number_of_credits) as purchased,
ifnull(t1.credits_used,0) as used
from credit_orders
left outer join (select * from (select credit_usage.client_id,
sum(credits_used) as credits_used
from credit_usage
group by credit_usage.client_id) as t0
) as t1 on t1.client_id = credit_orders.client_id
where credit_orders.payment_status='Paid'
group by credit_orders.client_id)
As per documentation:
MySQL Docs
The SELECT statement cannot contain a subquery in the FROM clause.
Your workaround would be to create a view for each of your subqueries.
Then access those views from within your view view_credit_status
As the more recent MySQL documentation on view restrictions says:
Before MySQL 5.7.7, subqueries cannot be used in the FROM clause of a view.
This means, that choosing a MySQL v5.7.7 or newer or upgrading the existing MySQL instance to such a version, would remove this restriction on views completely.
However, if you have a current production MySQL version that is earlier than v5.7.7, then the removal of this restriction on views should only be one of the criteria being assessed while making a decision as to upgrade or not. Using the workaround techniques described in the other answers may be a more viable solution - at least on the shorter run.
create view view_clients_credit_usage as
select client_id, sum(credits_used) as credits_used
from credit_usage
group by client_id
create view view_credit_status as
select
credit_orders.client_id,
sum(credit_orders.number_of_credits) as purchased,
ifnull(t1.credits_used,0) as used
from credit_orders
left outer join view_clients_credit_usage as t1 on t1.client_id = credit_orders.client_id
where credit_orders.payment_status='Paid'
group by credit_orders.client_id)
Looks to me as MySQL 3.6 gives the following error while MySQL 3.7 no longer errors out. I am yet to find anything in the documentation regarding this fix.
After MySQL 8.0.25 (InnoDB engine on Windows) upgrade from MariaDB (10.1.26, and I don't think we were using InnoDB engine on Linux), every time our DB server restarts, we run into this issue:
SELECT * INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = {insert table name here};
returns the column names alphabetically, though the values in the ORDINAL_POSITION column are correct.
SELECT * FROM {insert table name here} WHERE 1=0;
This returns columns ordered alphabetically rather than by ORDINAL_POSITION in some clients (LabView queries seem to be mostly affected by this).
After a few days, everything goes back to normal, with SELECT statements always ordering columns by ORDINAL_POSITION, and INFORMATION_SCHEMA.COLUMNS being sorted by ORDINAL_POSITION as expected.
Unfortunately, due to some of our programs depending on column order being consistent in SQL query results, this has been causing a lot of trouble, and we seem to be going in circles trying to figure it out.
Any thoughts as to what would cause this delayed sorting of INFORMATION_SCHEMA.COLUMNS, and why some clients would return the column order based on the order of INFORMATION_SCHEMA.COLUMNS rather than by ORDINAL_POSITION?
Any known server/DB settings to help resolve this?
Thanks, and apologies, this is way out of my knowledge base, so this is not a great post for content or question style.
======= UPDATE/EDIT =======
Apologies, I think I accidentally lumped in a different issue/behavior that was due to a separate thing that was already fixed. The column order in the SELECT queries may not have been incorrect this time, as that was a different issue that was previously resolved, and was due to a configuration issue during the upgrade.
Essentially the issue seems to be due to the assumption that the INFORMATION_SCHEMA.COLUMNS view would always have the same sort order, and I think #Tangenially Perpendicular's answer verifies that cannot be assumed.
Side note, it looks like FlySpeed SQL Query (issue submitted) was also assuming this, as did Heidi SQL (fixed) at one point.
In SQL order is not an inherent property of a data set. The wording that appears in the SQL-92 spec is
If an <order by clause> is not specified, then the ordering of the rows ... is implementation-dependent.
You'll find that in some form in a number of places.
There's no server-wide or database-wide option to set it. You need to specify the required order in your query. Your example above becomes
SELECT * from INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = <your table name> order by ORDINAL_POSITION;
You'll need to update your queries.
Following is a simple SQL query:
SELECT * FROM *table_name*
GROUP BY *column_name*
In my system I have MySQL 5.5. It is working absolutely fine.
Whereas in my friend's system he have MySQL 5.7, and he is getting the following error:
ERROR 1055 (42000): Expression #1 of SELECT list is not in GROUP BY
clause and contains nonaggregated column 'testdb.assetentry.entryId'
which is not functionally dependent on columns in GROUP BY clause;
this is incompatible with sql_mode=only_full_group_by
It is clearly visible this is happening because the versions are different.
But what I want to know is the reason behind this.
Can any one please explain.
First of all, please read Group by clause in mySQL and postgreSQL, why the error in postgreSQL?
It is not SQL Standard behaviour.
12.16.3 MySQL Handling of GROUP BY
To disable the MySQL GROUP BY extension and enable standard SQL behavior, enable the ONLY_FULL_GROUP_BY SQL mode. In this case, columns not named in the GROUP BY clause cannot be used in the select list or HAVING clause unless enclosed in an aggregate function.
It looks like on second server you have acitivated ONLY_FULL_GROUP_BY mode.
SELECT ##sql_mode;
You could simulate this behaviour on your MySQL 5.5:
SET SESSION sql_mode = 'ONLY_FULL_GROUP_BY';
SELECT *
FROM tab
GROUP BY col;
-- tab.col2' isn't in GROUP BY
SqlFiddleDemo
From MySQL 5.7:
Implementation of the ONLY_FULL_GROUP_BY SQL mode has been made more
sophisticated, to no longer reject deterministic queries that
previously were rejected. In consequence, ONLY_FULL_GROUP_BY is now
enabled by default, to prohibit nondeterministic queries containing
expressions not guaranteed to be uniquely determined within a group.
This question is to settle an argument between me and a coworker.
Let's say we have the following query, executed on a standard LAMP server.
SELECT field1, field2, field3
FROM some_table
WHERE some_table.field1 = 123
ORDER BY field2 DESC
LIMIT 0, 15
Now let's assume the limit clause is vulnerable to SQL injection.
LIMIT [insert anything here], [also insert anything here]
The point of my coworker is that there is no way to exploit this injection, so there's no need to escape it (since it take more processing power and stuff).
I think her reasoning is stupid, but I can't figure out how to prove her wrong by finding an example.
I can't use UNION since the query is using an ORDER BY clause, and the MySQL user running the query doesn't have the FILE priviledge so using INTO OUTFILE is also out of the question.
So, can anyone tell us who is right on this case?
Edit: the query is executed using PHP, so adding a second query using a semicolon won't work.
The LIMIT clause is vulnerable to SQL injection, even when it follows an ORDER BY, as Maurycy Prodeus demonstrated earlier this year:
mysql> SELECT field FROM user WHERE id >0 ORDER BY id LIMIT 1,1
procedure analyse(extractvalue(rand(),concat(0x3a,version())),1);
ERROR 1105 (HY000): XPATH syntax error: ':5.5.41-0ubuntu0.14.04.1'
Voilà! The above solution is based on handy known technique of so-called error based injection. If, therefore, our vulnerable web application discloses the errors of the database engine (this is a real chance, such bad practices are common), we solve the problem. What if our target doesn’t display errors? Are we still able to exploit it successfully?
It turns out that we can combine the above method with another well-known technique – time based injection. In this case, our solution will be as follows:
SELECT field FROM table WHERE id > 0 ORDER BY id LIMIT 1,1
PROCEDURE analyse((select extractvalue(rand(),
concat(0x3a,(IF(MID(version(),1,1) LIKE 5, BENCHMARK(5000000,SHA1(1)),1))))),1)
It works. What is interesting that using SLEEP is not possible in this case. That’s why there must be a BENCHMARK instead.
I would insert this:
1; DELETE FROM some_table WHERE 1; --
Just after the limit, that will select 1 row from some_table, then DELETE all some_table rows. then the rest will be considered as a comment.
SQL Injection occurs if “externally-influenced input […] could modify the intended SQL command”. And in this case it’s clear that user input can modify the intended SQL command.
However, exploitability is another question. You may not be able to exploit it today. But maybe someday someone is able to exploit it because:
The database connection layer has changed and it is possible to execute multiple statements at once.
The statement has changed and it is possible to use UNION.
The user privileges have changed and it is possible to use INTO OUTFILE/INTO DUMPFILE.
Someone finds a way that you may not have thought of. Have you noticed you can store the result in variables and/or execute a stored procedure?
If you are creating the SQL query from user input, then it will be vulnerable, unless you are using strong typing way before you get anywhere near generating the query. In such a case, there's no way you could make an integer become text saying to drop a table, for example.
Never trust user input, and always perform bounds checking. For example, don't even bother doing the query if the limit values are negative.
You should come with a coding example, not a query example as you do now. She is right on the fact that you cannot really alter a statement since the order by function is always "last". So yes, when you are manually inputting those query's on a sql server, you simply cannot alter the query to output something different rather than a error message. Yet, when you simply add an other query.. you can :)
Finish the last user input as a true value to let the first query run successfully, and add a ';'. Now you can start your own query.
I just have done this on my local mysql server:
SELECT * FROM city order by ID desc limit 0,15; SELECT * FROM city
Even so, in a strong case where there is absolute 0% chance someone could alter the statement, you simply do not even want to receive possible altered data. There should be a reason you use the LIMIT dynamically. Once a person can change your reason, you already have failed. Its not even about risking damage, losing data or what ever. You do not want any manipulation in any way.