I recently moved my SQL Database to another Amazon RDS server with version 5.7.
Before that, the application was working fine but now I started logging errors:
"ER_BAD_NULL_ERROR: Column xyz cannot be null" - The column already has a default value CURRENT_TIMESTAMP
I checked online and people suggested to have the sql_mode equal to NO_ENGINE_SUBSTITUTION
I checked the existing settings and it is already like that.
Any other reason I am getting this error? Any tricks?
Thanks.
After searching more, the problem was only in timestamp fields with current_timestamp default value. I searched in the parameters and found explicit_defaults_for_timestamp that was enabled (value 1) and with a bit more research, I had to disable this parameter as per the documentation here
https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_explicit_defaults_for_timestamp
in order to get the required result and fix the problem.
Simply deactivate explicit_defaults_for_timestamp
SET GLOBAL explicit_defaults_for_timestamp = 0;
I have no idea why it works like that in this particular case, so I would concentrate on fixing a problem.
According to the docs NO_ENGINE_SUBSTITUTION has nothing to do with the error, during application run. I would select rows with column "xyz" of NULL value and update it to something - not null.
Default is applied when row is created. Let's say you have a table with some millions of rows, and want to add column with not null. That would block your table for significant amount of time. So you can create column without not null, but with default. That operation deals only with metadata, so is fast. Default will deal with all new rows. After that you can slowly update all rows. At the end not null constraint can be added. Not sure if DB is checking constraint when adding it at last step. Or maybe prev. version had problem with it? With MySQL things like that happens.
Related
The webhosting I use has enabled StrictMode for the Databases. All my php scripts now stopped working because they report I haven't defined a default value for some columns.
As I have a lot of columns in a lot of tables, is there a way to set all the columns with "default value = none" with "default value = NULL" ?
In this way it won't report me the error anymore.
Of course, If there's another (better) way, I am available for it.
I tried looking on the net, but I couldn't find anything suitable for this case.
you can alter column
ALTER TABLE table_name
MODIFY COLUMN col datatype DEFAULT null
A general approach here which should work for each column causing an error would be to set a default value, and then maybe do an update to backfill records missing a value.
ALTER TABLE yourTable ALTER some_text_column SET DEFAULT 'None';
And here is the update:
UPDATE yourTable SET some_text_column = 'None' WHERE some_text_column IS NULL;
You are not required to do this update, but it might make sense to bring older records missing values in line with what newer records would look like.
After upgrading a client from MySQL 5.4 to 5.7 I started getting "BIGINT UNSIGNED value is out of range..." errors. I edited my.cnf and set sql_mode = "NO_UNSIGNED_SUBTRACTION" and used SHOW VARIABLES to ensure it was indeed using that mode. That did fix the initial problem in the application but today an unsigned int column is throwing the same error (BTW inventory is 2)...
UPDATE table SET inventory = inventory - 1
I don't understand why since I thought the new sql_mode would revert behavior to pre 5.5? I don't want to start casting everything in my statements just to make simple arithmetic work so the logical solution in my mind is just to convert all of my unsigned int cols to signed ints with the exception of primary auto increment keys.
Is this the best solution to escape these out of range errors?
I did some more digging and found my answer and wanted to share here for anyone stuck on this in the future. The SQL mode NO_UNSIGNED_SUBTRACTION was indeed working and had not reverted for this table/column. In short, the problem was that there was a trigger on that table and I discovered from a somewhat related MySQL bug report from 2009 that as stated in user comments by Davi Arnaut that triggers and prepared statements use the SQL mode that was in use when they were created... aaah! So, I redefined the trigger and everything works as expected.
I know the title is weird, but I am on that same bug for HOURS.
I have this query
UPDATE tournaments SET password_req_count = password_req_count + 1 WHERE id = 20;
(You can replace 20 by anything, it really does not matter)
This query modifies a timestamp field called start_timestamp by ALWAYS setting it the computers current hour.
And this query is perfectly fine, there's no bug with this :
UPDATE tournaments SET password_req_count = 0 WHERE id = 20;
This was happening in the PHP code until I removed that one query, then it stopped. Then I decided to try it by directly executing the query myself, without PHP, and the bug is still here.
password_req_count is an int (I mean, I checked it, the problem isn't here)
This query does not appear in the query history of MySQL (the one you can get by pressing the "UP" key to remake a query quickly...), and this bug doesn't appear locally (it only appears on my server). Note that I exported my local database to the server's one, so everything is exactly the same there and here.
The MySQL server version was 5.5 on my server and 5.7 at home, I thought this was the problem so I updated it, but absolutely nothing changed. I also googled a lot about this, but I found no topic talking about this subject.
I do have query logs, so I am SURE that there is NOTHING that edits the start_timestamp (except this weird bug obviously). It is not supposed to be edited anyway.
Edit : I just edited the field name to password_request_count because password_req_count already exists in another table. But the bug is still here.
RECAP HERE
Edit 2 : Here is a video because apparently the post is not clear enough. Notice that I can't do the UPDATE query again by pressing the "up" touch, and please also notice that start_timestamp gets edited if I increment password_req_count.
http://www.nx-lab.com/bug.mp4
Edit 3 : Apparently this also happens if I edit other fields (such as top_prize)
There could be a couple of things doing this, one is a trigger on the table. This code will show you if there are any...
SHOW TRIGGERS FROM tournaments;
The other thing, which is the correct answer in this case, is an auto update on the datetime column. This causes the date and time in the column to be updated automatically when there is an update to the table.
You can read more about it here:
https://dev.mysql.com/doc/refman/5.7/en/timestamp-initialization.html
If you want to remove to auto update then an ALTER table is required to remove it from the column, from Timestamp without change on update...
ALTER TABLE leads MODIFY added_on TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
I did a dump of a mysql 5.5 database and loaded it into a 5.6 server.
The dump added ON UPDATE CURRENT_TIMESTAMP to a bunch of columns that didn't have it previously.
I'm searching for an ALTER TABLE statement that will remove the ON UPDATE CURRENT_TIMESTAMP rule without making any other changes. In my imagination it should be something like ON UPDATE NOOP or ON UPDATE NO_CURRENT_TIMESTAMP.
ON UPDATE JUST_BE_A_NORMAL_COLUMN?
I tried using the "Clear default" option in mysql workbench and it did the opposite of what it should have done - it gave the column a default!
I was able to get rid of the default with ALTER TABLE t ALTER COLUMN c DROP DEFAULT, so the column is mandatory in INSERTs (just like it was before the dump/reload, as I wanted it) but the unwanted behavior on UPDATEs remains.
I have not enabled the explicit_defaults_for_timestamp option. If I was starting fresh I'd definitely use that option since it seems a lot more sane. But since I already had the columns configured the way I wanted them in 5.5, I expected them to keep the same semantics when transferred to 5.6. Apparently mysqldump just wasn't smart enough.
At this point I'm not sure I understand what effects would result from enabling explicit_defaults_for_timestamp. Would that option change the behavior of existing tables, or does it only change the interpretation of future CREATE TABLE commands? Would turning it on somehow help me fix the broken columns?
UPDATE:
A similar question is here but that one is about creating a new table, not altering an existing column. In fact that question is the one I used as a guide when creating the tables on the 5.5 server. I used the 2-step procedure: create with default 0 to suppress ON UPDATE CURRENT_TIMESTAMP, then drop default.
The 2-step procedure definitely doesn't produce the correct result on the 5.6 server without explicit_defaults_for_timestamp; this is a sign that either 5.6 doesn't perfectly imitate the old behavior in this mode, or the old server never did what I thought it was doing. I can't be sure which.
ALTER TABLE mytable
CHANGE mycolumn
mycolumn TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
I believe this will reset and void the ON UPDATE. This would effectively make this definition:
CREATE TABLE mytable (
# Other Columns
mycolumn timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP
)
Change into this one:
CREATE TABLE mytable (
# Other Columns
mycolumn timestamp NOT NULL default CURRENT_TIMESTAMP
)
If you wanted to reset the column entirely, you should be able to simply redefine it like:
ALTER TABLE mytable
CHANGE mycolumn
mycolumn TIMESTAMP NOT NULL;
Using the ideas from the other answers, and a couple of freshly installed mysql server instances, I have done a comparison of the behavior of several different CREATE and ALTER commands on 3 different server configurations:
mysql 5.5.45
mysql 5.6.26 without explicit_defaults_for_timestamp
mysql 5.6.26 with explicit_defaults_for_timestamp
The easiest one to explain is 5.6 with explicit_defaults_for_timestamp. Everything is sane. The timestamp type is not noticeably different from any other type. Columns created before the explicit_defaults_for_timestamp flag was turned on retain their old defaults and magic update.
In 5.5, the implicit defaults happen when a timestamp column is created (if it is the first timestamp column in the table). These are well documented already. The magic update behavior can be avoided by setting an explicit default, and then the default can be removed, leaving the column with the 3 desired attributes: non-nullable, no default, no magic update. This is the result of CREATE TABLE t (TIMESTAMP c NOT NULL DEFAULT 0) and ALTER TABLE t ALTER COLUMN c DROP DEFAULT.
This state can't be recreated with a single CREATE TABLE command, and it doesn't survive a mysqldump.
5.6 without explicit_defaults_for_timestamp is the most interesting case. It's almost the same as 5.5, but the DROP DEFAULT command is different. If you try the "create with default 0 then drop default" sequence, the magic update attribute appears as a side effect of the drop. But if you make the default CURRENT_TIMESTAMP instead of 0, then the DROP DEFAULT works without the side effect. (Must be a bug. I can't imagine any reason it would intentionally behave this way.)
Therefore this pair of commands will have the same result on all of the server configurations I tested:
ALTER TABLE t CHANGE COLUMN c c TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP;
ALTER TABLE t ALTER COLUMN c DROP DEFAULT;
The column now has no default and no magic update.
For your use case I think you would be better served with DATETIME, eg:
ALTER TABLE `my_table`
CHANGE `my_col` `my_col` DATETIME NOT NULL DEFAULT NOW();
This will default to NOW() on insert, but remain unaffected on update.
See this question for a good explanation of the difference:
Should I use field 'datetime' or 'timestamp'?
Try enabling the explicit_defaults_for_timestamp system variable and then redefine the columns with:
ALTER TABLE `table` CHANGE COLUMN `col` `col` TIMESTAMP NOT NULL;
If I understand the documentation correctly enabling explicit_defaults_for_timestamp is mandatory to be able to define TIMESTAMP columns declared as NOT NULL and without an explicit DEFAULT.
If you want to remove both the DEFAULT value and ON UPDATE value, nothing but the following helped me
ALTER TABLE `your_table` CHANGE `your_column` `your_column` TIMESTAMP NOT NULL DEFAULT '0000-00-00 00:00:00';
I have a database where old code likes to insert '0000-00-00' in Date and DateTime columns instead of a real date. So I have the following two questions:
Is there anything that I could do on the db level to block this? I know that I can set a column to be not-null, but that does not seem to be blocking these zero values.
What is the best way to detect the existing zero values in date fields? I have about a hundred tables with 2-3 date columns each and I don't want to query them individually.
Followup:
The default is already set to null. A long time ago, the default was '0000-00-00'. Some code still explicitly places '0000-00-00'. I would prefer to force that code to throw an error so I could isolate and remove it.
Is there anything that I could do on the db level to block this?
Yes, enable the NO_ZERO_DATE mode:
SET sql_mode = 'NO_ZERO_DATE';
The behaviour is documented. Additionally, you might want to also set the mode to include NO_ZERO_IN_DATE...
Also make sure the sql_mode includes either STRICT_ALL_TABLES or STRICT_TRANS_TABLES; without these NO_ZERO_IN_DATE only give a warning, but insert still succeeds.
What is the best way to detect the existing zero values in date fields? I have about a hundred tables with 2-3 date columns each and I don't want to query them individually.
Separate columns means they have to be checked individually--nothing you can do about that.
Assuming you can't easily fix the data and "SET sql_mode = 'NO_ZERO_DATE';", you could create a view on the table...
CREATE VIEW filter AS
SELECT other_column,
CASE
WHEN realtable.dodgy_date = 0 THEN NULL
ELSE realtable.dodgy_date
END AS dodgy_date
FROM realtable;
SET SQL_MODE='ALLOW_INVALID_DATES';
SET GLOBAL sql_mode=(SELECT REPLACE(##sql_mode,'NO_ZERO_DATE',''));
SET GLOBAL sql_mode=(SELECT REPLACE(##sql_mode,'NO_ZERO_IN_DATE',''));
Just run above queries on database. It will solve the issue permanently.
If it doesn't matter what date goes in there (ie as long as it's non-zero) you can change the column definition to use NOW() as the default. Probably not an ideal solution, but it does satisfy the criteria :
1) Not-null
2) Non-zero
I'm actually really not proud of that suggestion
You could make the columns nullable and have NULL as the default value, but it sounds like you already have that and it's not working. ALTHOUGH... it could be the tool you're using to display the data doesn't like displaying NULL dates... what tool are you using? Or is the '0000-00-00' showing up in data retreived by code?
You could set a default value that is non-null and also easily recognizable as a default such as 1900-01-01 (assuming you don't normally deal with dates that are close to this date).
A trigger can be used to enforce values for columns.
set the timestamp by default is maybe an option for you, use table change statement for that:
ALTER TABLE mytable CHANGE date_update timestamp NOT NULL DEFAULT CURRENT_DATE()
MySQL CURRENT_DATE() documentation at w3c resource
To remove Zero Dates and replace them by e.g. the current date do this:
UPDATE mytable SET date_update = CURRENT_DATE() where date_update = "0000-00-00"
This is the trigger I use:
delimiter //
CREATE TRIGGER bad_date BEFORE INSERT ON some_table
FOR EACH ROW
BEGIN
IF NEW.the_date='0000-00-00' THEN
SET NEW.the_date= CURDATE();
END IF;
END;//
If updates are a concern, add a separate trigger (BEFORE UPDATE) to do the same thing.