My PHP/mySQL backend hosted on an external site has been running fine since 2014. Recently, it started throwing up "field has no default value" errors.
I checked the config and found STRICT_TRANS_TABLES, which gives these errors for fields with no default in some cases.
My question is whether or not it is safe for me to remove this config value. It's mySQL 5.5.5-10.3.12-MariaDB.
Alternatively I could give everything default values, but I don't know which of these solutions is more likely to cause the existing codebase to stop working properly.
I encourage use of strict mode in MySQL because if you disable strict mode, you risk causing some unwanted effects, such as:
If you insert a value to a column where it can't fit, the value is silently truncated. Like inserting a long string into a shorter varchar column, or inserting a big integer into an INT column. This leads you to potentially have bogus values in your database. I prefer these cases to be errors, to prevent such bogus values.
Non-strict mode allows nonsensical dates, like 0000-00-00. There is no such date in the calendar. I'd rather this value not be allowed. If I need to symbolize an absence of a value, I'll use NULL.
Will these cases affect your app? There's no way I or anyone else on Stack Overflow can predict that. You need to test it yourself.
Related
My server is using a MySQL DB, connecting to it via the C++ connector. I'm nearing production and I've been spending some time trying to break things as part of hardening the server.
One action item I had was to see what would happen if I execute a statement with a string that is longer than VARCHAR. For example, if I have a column defined as VARCHAR(4) and then set it to the string "hello".
This of course throws an exception with the error code 1406 (Data too long for column).
What I was wondering was if there was a good or standard way to defend against this? Obviously one thing is to check against the string length and truncate manually. I can do this, however there are many tables and several columns with VARCHAR. So my worry is updating server code if one of the columns using VARCHAR has its length increased (i.e. code maintainability)
Note that the server does do some validation up front. I'm just trying to defend against a subtle bug or corner case that lets something slip through.
A couple of other options on the table are to disable strict so it will give a warning and truncate or to convert VARCHAR to TEXT.
I was wondering a few things.
Is there a recommended method to handle this situation?
What are the disadvantages of disabling strict?
Is it worth (and is it possible) to query the DB at runtime the VARCHAR lengths? Note that I'm using the C++ connector. I suppose I could also write a tool that is run before compiling which would extract out VARCHAR lengths from the SQL code used to generate tables. But that then makes me wonder is I'm over engineering this.
I'm just sorting through the possible approaches now and thought I'd seek advice from those with more experience with MySQL.
As an experience database engineer I would recommend a combination of the follow two strategies:
1) If you that know that a there is a chance, however small, that data for your varchar(4) could go higher than 4 then make the varchar field larger than 4. For example, if you expect that the field can go as high as 8 then set the field to varchar(10). The beauty of using a varchar field instead of a char is that a varchar will only use whatever storage it needs.
2) If there is a real issue with data constantly being larger than the varchar field length then you should right your own exception handler to trap for the 1406 error. For the exception to work properly you will need to come up with some type of strategy on exactly how you want to handle the exception. For example, you could send an error to the user and ask them to fix the problem, you could accept the data but truncated it so it fits into the field, or you could send the error to a log file to get fixed at a later time.
I have been using MySQL to create a database and sometimes warnings come up rather than errors. An example is when I entered a page of full stops as a value. I do not really understand the point of these are why they appear instead of errors. Surely every incorrect value entered into a database should come up with an error, not just some. Please could somebody explain?
If you declared a column to be an integer but are attempting to insert a string, MySQL may perform implicit type conversion and emit an warning. See this question for a detailed example.
If you want to treat all warnings as errors, you should try strict mode.
I was quite surprised when MySQL allowed me to insert a NULL into a field that was created with NOT NULL. I did some research and discovered how to enable strict mode. However, I am not quite sure what validation MySQL does when STRICT_ALL_TABLES is enabled.
The manual says:
Strict mode controls how MySQL handles input values that are invalid or missing. A value can be invalid for several reasons. (emphasis mine) For example, it might have the wrong data type for the column, or it might be out of range.
I understand what it considers missing and how it handles that. I am unclear what it considers invalid. I've done some testing and discovered the following:
strings that are too long are invalid
numbers that are out of range are invalid
NULLs for a non-NULL column are invalid
TRUE and FALSE always seem to be valid (they become 1 and 0 respectively)
invalid dates are invalid
zero dates are valid (additional modes can be enabled to change this behaviour)
floats in an integer field are valid (they get rounded)
letters in a number field are invalid
Does MySQL do any other validation checks other than what is mentioned above?
The manual says 'wrong data type for the column', but the only situation I see where this actually comes into play is letters in a number field. Are there other examples of data type errors?
Is there a list somewhere of exactly what checks MySQL performs?
EDIT: For the record, my application already does extensive validation. I am using strict mode as a last-chance, just-in-case check. If I forget to check something, I want it to fail fast rather than 'silently mangle my data'.
A good resource is to check the MySQL source, and read mysql-test/t/strict.test to see all the cases they test for after setting STRICT mode or TRADITIONAL mode (which is a superset of STRICT).
You did excellent research and testing, but there are a few more cases, such as:
Trying to insert an undefined ENUM value.
Trying to insert a default value for a NOT NULL column with no DEFAULT defined.
Trying to use CAST() to convert strings to integers, etc.
Conversion of VARCHAR to MEDIUMTEXT or LONGTEXT if you give a length greater than 65536.
Truncation of COMMENT strings for tables and columns.
Conversion of string to YEAR type.
Defining a SET or ENUM column with duplicate entries.
Also mysql-test/include/strict_autoinc.inc, because it tests for overflow when an auto-inc value grows too large.
There are a few other test files that use STRICT mode for specific tests, but I didn't examine them.
I grepped the source code of MySQL 5.5.28 (Community Server Edition) to find instances of STRICT_ALL_TABLES being used.
From what I saw, it looks like your list is already pretty complete, but below are a few more things I learned about the use of STRICT_ALL_TABLES. The first one is another validation, while the rest all deal with errors and warnings.
Geometry columns can't have a default value. (sql/field.cc:9394)
If a table comment is longer than 2048 characters, MySQL produces an error rather than truncating the comment with a warning. (sql/unireg.cc:231)
If a field comment is longer than 1024 characters, MySQL produces an error rather than truncating the comment with a warning. (sql/unireg.cc:739)
On an insert or update, if a duplicate value exists in a SET or ENUM MySQL produces an error rather than a warning. (sql/sql_table.cc:2503)
There are a bunch of operations that will abort on warning with STRICT_ALL_TABLES rather than proceeding with just a warning. These are too numerous for me to list, and the setting of the abort_on_warning flag is too far removed from the code that creates the warnings for me to easily document (or even understand) all of them. But if anybody wants to get their hands dirty in the source code, a few places to start would be sql/sql_update.cc:630 and sql/sql_insert.cc:841
If you try to create a TEXT column on a table, and give it a default value in MySQL, you get an error (on Windows at least). I cannot see any reason why a text column should not have a default value. No explanation is given by the MySQL documentation. It seems illogical to me (and somewhat frustrating, as I want a default value!). Anybody know why this is not allowed?
Windows MySQL v5 throws an error but Linux and other versions only raise a warning. This needs to be fixed. WTF?
Also see an attempt to fix this as bug #19498 in the MySQL Bugtracker:
Bryce Nesbitt on April 4 2008 4:36pm:
On MS Windows the "no DEFAULT" rule is an error, while on other platforms it is often a warning. While not a bug, it's possible to get trapped by this if you write code on a lenient platform, and later run it on a strict platform:
Personally, I do view this as a bug. Searching for "BLOB/TEXT column can't have a default value" returns about 2,940 results on Google. Most of them are reports of incompatibilities when trying to install DB scripts that worked on one system but not others.
I am running into the same problem now on a webapp I'm modifying for one of my clients, originally deployed on Linux MySQL v5.0.83-log. I'm running Windows MySQL v5.1.41. Even trying to use the latest version of phpMyAdmin to extract the database, it doesn't report a default for the text column in question. Yet, when I try running an insert on Windows (that works fine on the Linux deployment) I receive an error of no default on ABC column. I try to recreate the table locally with the obvious default (based on a select of unique values for that column) and end up receiving the oh-so-useful BLOB/TEXT column can't have a default value.
Again, not maintaining basic compatability across platforms is unacceptable and is a bug.
How to disable strict mode in MySQL 5 (Windows):
Edit /my.ini and look for line
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
Replace it with
sql_mode='MYSQL40'
Restart the MySQL service (assuming that it is mysql5)
net stop mysql5
net start mysql5
If you have root/admin access you might be able to execute
mysql_query("SET ##global.sql_mode='MYSQL40'");
Without any deep knowledge of the mySQL engine, I'd say this sounds like a memory saving strategy. I assume the reason is behind this paragraph from the docs:
Each BLOB or TEXT value is represented internally by a separately allocated object. This is in contrast to all other data types, for which storage is allocated once per column when the table is opened.
It seems like pre-filling these column types would lead to memory usage and performance penalties.
As the main question:
Anybody know why this is not allowed?
is still not answered, I did a quick search and found a relatively new addition from a MySQL developer at MySQL Bugs:
[17 Mar 2017 15:11] Ståle Deraas
Posted by developer:
This is indeed a valid feature request, and at first glance it might seem trivial to add. But TEXT/BLOBS values are not stored directly in the record buffer used for reading/updating tables. So it is a bit more complex to assign default values for them.
This is no definite answer, but at least a starting point for the why question.
In the mean time, I'll just code around it and either make the column nullable or explicitly assign a (default '') value for each insert from the application code...
"Support for DEFAULT in TEXT/BLOB columns"
is a
feature request in the MySQL Bugtracker (Bug #21532).
I see I'm not the only one who would like to put a default value in a TEXT column.
I think this feature should be supported in a later version of MySQL.
This can't be fixed in the version 5.0 of MySQL,
because apparently it would cause incompatibility and dataloss if anyone tried to transfer a database back and forth between the (current) databases that don't support that feature and any databases that did support that feature.
You can get the same effect as a default value by using a trigger
create table my_text
(
abc text
);
delimiter //
create trigger mytext_trigger before insert on my_text
for each row
begin
if (NEW.abc is null ) then
set NEW.abc = 'default text';
end if;
end
//
delimiter ;
Support for using expression as default values was added to MySQL 8.0.13, released 2018-10-22, and works for TEXT, JSON, BLOB and GEOMETRY.
You still cannot write :
create table foo(bar text default 'baz')
But you can now write:
create table foo(bar text default ('baz'))
Which achieve the same thing.
I normally run sites on Linux, but I also develop on a local Windows machine. I've run into this problem many times and just fixed the tables when I encountered the problems. I installed an app yesterday to help someone out and of course ran into the problem again. So, I decided it was time to figure out what was going on - and found this thread. I really don't like the idea of changing the sql_mode of the server to an earlier mode (by default), so I came up with a simple (me thinks) solution.
This solution would of course require developers to wrap their table creation scripts to compensate for the MySQL issue running on Windows. You'll see similar concepts in dump files. One BIG caveat is that this could/will cause problems if partitioning is used.
// Store the current sql_mode
mysql_query("set #orig_mode = ##global.sql_mode");
// Set sql_mode to one that won't trigger errors...
mysql_query('set ##global.sql_mode = "MYSQL40"');
/**
* Do table creations here...
*/
// Change it back to original sql_mode
mysql_query('set ##global.sql_mode = #orig_mode');
That's about it.
For Ubuntu 16.04:
How to disable strict mode in MySQL 5.7:
Edit file /etc/mysql/mysql.conf.d/mysqld.cnf
If below line exists in mysql.cnf
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
Then Replace it with
sql_mode='MYSQL40'
Otherwise
Just add below line in mysqld.cnf
sql_mode='MYSQL40'
This resolved problem.
This is a very old question but still it doesn't seems to have been answered properly. And, my this answer isn't actual answer to the question - "WHY can't a text column have a default value", but as it isn't possible to write long text in comment, and as my comment could help someone to prevent the error, here it is as a separate answer:
Some are saying that the error is occurring because of OS - Windows-Linux; but this isn't directly related to OS. (However, there may be differences in default settings of MySQL within different installers for different OSes, I am not sure.)
The main reason is the flag STRICT_TRANS_TABLES for sql_mode setting. if a value is not specified in INSERT statement for TEXT datatype column and if the flag exist in the sql_mode setting then MySQL is reporting an error; and if the flag doesn't exist then MySQL is only reporting a warning and inserts the record.
So, to prevent this error, one can remove the STRICT_TRANS_TABLES from sql_mode setting of MySQL. (He my need to reset the mode to the previous value if it can affect other operations on the database.)
According to the documentation of SQL mode in MySQL ...
For STRICT_TRANS_TABLES, MySQL converts an invalid value to the closest valid value for the column and inserts the adjusted value. If a value is missing, MySQL inserts the implicit default value for the column data type. In either case, MySQL generates a warning rather than an error and continues processing the statement. Implicit defaults are described in Section 11.6, “Data Type Default Values”.
... and documentation of Data Type Default Values ...
The BLOB, TEXT, GEOMETRY, and JSON data types cannot be assigned a default value.
... TEXT column can not have a default value, but if STRICT_TRANS_TABLES is removed from sql_mode then MySQL inserts empty string '' if no value is specified for TEXT column in INSERT statement.
I get an SqlDateTime overflow error (Must be between 1/1/1753 12:00:00 AM and 12/31/9999 11:59:59 PM.) when doing an INSERT using an Linq DataContext connected to SQL Server database when I do the SubmitChanges().
When I use the debugger the date value is correct. Even if I temporary update the code to set the date value to DateTime.Now it will not do the insert.
Did anybody found a work-around for this behaviour? Maybe there is a way to check what SQL the datacontext submits to the database.
Do you have the field set as autogenerated in the designer? If that's not the problem, I'd suggest setting up logging of the data context actions to the console and checking the actual SQL generated to make sure that it's inserting that column, then trace backward to find the problem.
context.Log = Console.Out;
FWIW, I often set my "CreatedTime" and "LastUpdatedTime" columns up as autogenerated (and readonly) in the designer and give them a suitable default or use a DB trigger to set the value on insert or update. When you set it up as autogenerated, it won't include it in the insert/update even if modified. If the column doesn't allow nulls, then you need to supply an alternate means of setting the value, thus the default constraint and/or trigger.
Are you sure you're looking at the right Date column? Happened to me once, and the error turned out to be caused by another non-nullable Date column that wasn't set before submitting.
I came across this recently.
The error may as well say "something's preventing the save!". Because in my case, it was not the DateTime value that was the problem.
I thought I was passing a value in for the primary key, and what was arriving was "null". Being the key, it can't be null - and so my problem was completely somewhere else. By resolving the null, the problem disappeared.
We all hate misleading errors - and this is one of them.
Lastly, as a suggestion... If you do find conversion of dates a problem, then don't use dates at all! .NET's DateTime class supports the "Ticks" value. It can also instantiate a new DateTime(ticks); too. The only Gotcha with that one, is the implementation of ticks in Javascript has a different starting point in history. So you might want a conversion between ticks if you ever tried getting DateTimes from c# to Javascript.
I suggest you change your project's Target Framework. Maybe SQL Server is newer than .Net Framework. I see the same your issue:
My project's Target Framework is 3.5.
SQL Server is 2012
And then I change to 4.0. The issue is solved.
Bottom line: watch the order of your calls to SubmitChanges() and ensure that all objects that would be "submitted" are actually ready to be submitted. This often happens to me when I'm in the middle of setting the attributes of new LINQ object (e.g, the ".FirstName" of new "tblContact"), and then some conditional logic requires the creation of a separate, related record (e.g., a new "tblAddress" record), so the code goes to create the "tblAddress" and tries to SubmitChanges() on saving that record, but that SubmitChanges() then also tries to insert the unfinished "tblContact" record, which maybe doesn't yet have a required "BirthDate" field value set. Thus, the exception looks to occur when I'm inserting the "tblAddress" object/record, but actually refers to the lack of "BirthDate" for the "tblContact" object/record.