I have a .NET 6 application that is currently backed by MSSQL database, that is maintained by Entity Framework using a Model First approach. I am trying to migrate it to use a MySQL database backend, for a variety of reasons.
I have installed MySQL Locally (Windows) to start exploring and getting it working. I can migrate the schema easily enough (With either MySQL Workbench or using EF) but migrating the data is proving to be a little tricky.
Around half of the tables migrated fine, but the other half, relating to string data, are failing due to errors which look a little like this - the column obviously differs from table to table. The source data is nvarchar in SQL, and the destination is type `varchar'
Statement execution failed: Incorrect string value: '\xF0\x9F\x8E\xB1' for column 'AwayNote'
Does anyone know how I can get the Migration to run successfully?
The research I have read has said to ensure server and table character sets are aligned as per the below.
I have set up my Source as SQL using the ODBC FreeTDS
The data import screen is set up like this - the check box doesn;t seem to affect things especially.
I have MySQL setup with this too, which I have also read is important.
[mysql]
default-character-set = utf8mb4
Background:
Legacy code running in MS Access 2003.
Sqls run by CurrentDB in Access.
Currently running on Windows 7 32-bit machine.
Connecting to MySQL Server 5.5 through a ODBC 5.1 Driver.
Problem:
Trying to migrate to Windows server 2012 64 bit.
ODBC 5.3 Unicode Driver (32 bit).
Don't want to use time rewriting everything as there is a lot of code, and it will in a not to distant future be removed.
Issue:
Several sql statements fail when running on the new servers. Worked on old servers.
All the failed sqls have now() in the statement.
Error description says ODBC call failed. While the more detailed description says date overflow - "[MySQL][ODBC 5.3(w) Driver][mysqlid-5.5.28-log]Date overflow".
Happens randomly, and when it happens and Access stops one can usually just choose continue and the sql will then work. It fails less than 1% of the times it runs (of thousands).
The only dates in the sql are in the where clause: "and fieldA > now()", where fieldA is a datetime column. This is a get recordset sql. Another error during an insert is the same, but where an integer was subtracted from now() before compared to a datetime.
I don't understand why it says date overflow when there doesn't seem to be a reason for the time of either datetime or "now" to be removed? And since the datetime field is already in the database and now() will get current date and time there shouldn't be any invalid dates?
Any help in what the problem might be or how to debug/log anything that might help would be highly appreciated.
Turning trace on in the ODBC driver is not an option because it happens so randomly, there is so much traffic therefore this would slow down everything such that nothing happens.
Note that I also did encounter one sql where the date overflow error message was correct. It seems that prior to 5.3 when inserting a datetime into a date field it was automatically truncated, because a sql which had been successfull 3000 times started failing. Therefore this sql has been fixed by extracting the date from the field first. But the other errors must be something different.
New version containing a bugfix has been released by Oracle: 5.3.8
This error was a bug that seems to have been introduced in version 5.1.11.
In advanced options there is now a Date overflow check box which has to be ticked for the code to continue when there is an error.
Reply from Oracle about the fix:
"For your information the fix approach was that in C or C++ it is possible to read or write DATE type using SQL_TIMESTAMP_STRUCT.
This struct can hold both date and time.
The error (Date overflow) was generated when with operations that are supposed to be DATE-only this struct got non-zero values for time.
That is the canonical approach as ODBC API requires, however, it causes inconveniences sometimes when for instance the app did not bother
to initialize the whole structure with 0 values because it knows it will only need the DATE part but the random values for TIME fraction could cause the errors despite of being truncated.
A new option was introduced to continue with the query execution rather then return error.
The server will ignore the TIME part and the result is the same as if there were zeroes."
This appears to be an issue with MySQL Connector/ODBC versions 5.2 and later. A web search led to this thread on another site, which in turn led to this unresolved bug report. Note that this is a broader MySQL Connector/ODBC issue; it is not specific to Access applications.
MySQL Connector/ODBC 5.1.13 is still available for download, so your most expedient solution would probably be to simply use that version until the code in question is taken out of service. Your other alternatives might include:
using a newer version of MySQL Server (with [better?] support for fractional seconds), or
tweaking the Access SQL queries to use something other than the Now() function (which you, understandably, would like to avoid).
I had exactly the same problem - "Date overflow" error when saving the data in an Access form.
I changed the data type from "datetime" to "timestamp" in the linked MySQL table and this solved the issue. The datetime data type seems to be too short to accommodate the value of Now() in MS Access.
Remember to refresh the linked tables in Access afterwards.
I also noticed that there was a "Not Null" tick mark next to the ModifyTime timestamp field in the MySQL table. I unticked this.
After these two changes I no longer get the Date Overflow error when Me.Dirty=False executes in Access.
I use Access365 (x64), MySQL ODBC Connector 8.0.31 (x64) and MySQL 5.6
Ok, had a problem when I was trying to insert an string that was believed came in UTF-8 format when truly was latin-1. Problem was, behaviour between different servers varied like:
Windows will give an error when confronted when a non-UTF8 char: "Incorrect string value: '\xD1OL S....' for column [...]"
Linux will quietly insert whatever I give it until it finds the first char it doesn't like, and then truncate the rest of the string.
For once, I find that the windows behaviour is the one I'd rather have. I've been looking and haven't found some kind of option to make the mysql linux server more strict.
Do you now of one?.
Thanks!.
Well, that was quick. Problem was, we had a 5.6 Linux server that was not in strict mode, though our DBA assured me that it was, making us think that it was a Linux problem.
So, just activate STRICT_ALL_TABLES. MySQL modes and how to activate them are here:
http://dev.mysql.com/doc/refman/5.6/en/sql-mode.html
If you try to create a TEXT column on a table, and give it a default value in MySQL, you get an error (on Windows at least). I cannot see any reason why a text column should not have a default value. No explanation is given by the MySQL documentation. It seems illogical to me (and somewhat frustrating, as I want a default value!). Anybody know why this is not allowed?
Windows MySQL v5 throws an error but Linux and other versions only raise a warning. This needs to be fixed. WTF?
Also see an attempt to fix this as bug #19498 in the MySQL Bugtracker:
Bryce Nesbitt on April 4 2008 4:36pm:
On MS Windows the "no DEFAULT" rule is an error, while on other platforms it is often a warning. While not a bug, it's possible to get trapped by this if you write code on a lenient platform, and later run it on a strict platform:
Personally, I do view this as a bug. Searching for "BLOB/TEXT column can't have a default value" returns about 2,940 results on Google. Most of them are reports of incompatibilities when trying to install DB scripts that worked on one system but not others.
I am running into the same problem now on a webapp I'm modifying for one of my clients, originally deployed on Linux MySQL v5.0.83-log. I'm running Windows MySQL v5.1.41. Even trying to use the latest version of phpMyAdmin to extract the database, it doesn't report a default for the text column in question. Yet, when I try running an insert on Windows (that works fine on the Linux deployment) I receive an error of no default on ABC column. I try to recreate the table locally with the obvious default (based on a select of unique values for that column) and end up receiving the oh-so-useful BLOB/TEXT column can't have a default value.
Again, not maintaining basic compatability across platforms is unacceptable and is a bug.
How to disable strict mode in MySQL 5 (Windows):
Edit /my.ini and look for line
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
Replace it with
sql_mode='MYSQL40'
Restart the MySQL service (assuming that it is mysql5)
net stop mysql5
net start mysql5
If you have root/admin access you might be able to execute
mysql_query("SET ##global.sql_mode='MYSQL40'");
Without any deep knowledge of the mySQL engine, I'd say this sounds like a memory saving strategy. I assume the reason is behind this paragraph from the docs:
Each BLOB or TEXT value is represented internally by a separately allocated object. This is in contrast to all other data types, for which storage is allocated once per column when the table is opened.
It seems like pre-filling these column types would lead to memory usage and performance penalties.
As the main question:
Anybody know why this is not allowed?
is still not answered, I did a quick search and found a relatively new addition from a MySQL developer at MySQL Bugs:
[17 Mar 2017 15:11] Ståle Deraas
Posted by developer:
This is indeed a valid feature request, and at first glance it might seem trivial to add. But TEXT/BLOBS values are not stored directly in the record buffer used for reading/updating tables. So it is a bit more complex to assign default values for them.
This is no definite answer, but at least a starting point for the why question.
In the mean time, I'll just code around it and either make the column nullable or explicitly assign a (default '') value for each insert from the application code...
"Support for DEFAULT in TEXT/BLOB columns"
is a
feature request in the MySQL Bugtracker (Bug #21532).
I see I'm not the only one who would like to put a default value in a TEXT column.
I think this feature should be supported in a later version of MySQL.
This can't be fixed in the version 5.0 of MySQL,
because apparently it would cause incompatibility and dataloss if anyone tried to transfer a database back and forth between the (current) databases that don't support that feature and any databases that did support that feature.
You can get the same effect as a default value by using a trigger
create table my_text
(
abc text
);
delimiter //
create trigger mytext_trigger before insert on my_text
for each row
begin
if (NEW.abc is null ) then
set NEW.abc = 'default text';
end if;
end
//
delimiter ;
Support for using expression as default values was added to MySQL 8.0.13, released 2018-10-22, and works for TEXT, JSON, BLOB and GEOMETRY.
You still cannot write :
create table foo(bar text default 'baz')
But you can now write:
create table foo(bar text default ('baz'))
Which achieve the same thing.
I normally run sites on Linux, but I also develop on a local Windows machine. I've run into this problem many times and just fixed the tables when I encountered the problems. I installed an app yesterday to help someone out and of course ran into the problem again. So, I decided it was time to figure out what was going on - and found this thread. I really don't like the idea of changing the sql_mode of the server to an earlier mode (by default), so I came up with a simple (me thinks) solution.
This solution would of course require developers to wrap their table creation scripts to compensate for the MySQL issue running on Windows. You'll see similar concepts in dump files. One BIG caveat is that this could/will cause problems if partitioning is used.
// Store the current sql_mode
mysql_query("set #orig_mode = ##global.sql_mode");
// Set sql_mode to one that won't trigger errors...
mysql_query('set ##global.sql_mode = "MYSQL40"');
/**
* Do table creations here...
*/
// Change it back to original sql_mode
mysql_query('set ##global.sql_mode = #orig_mode');
That's about it.
For Ubuntu 16.04:
How to disable strict mode in MySQL 5.7:
Edit file /etc/mysql/mysql.conf.d/mysqld.cnf
If below line exists in mysql.cnf
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
Then Replace it with
sql_mode='MYSQL40'
Otherwise
Just add below line in mysqld.cnf
sql_mode='MYSQL40'
This resolved problem.
This is a very old question but still it doesn't seems to have been answered properly. And, my this answer isn't actual answer to the question - "WHY can't a text column have a default value", but as it isn't possible to write long text in comment, and as my comment could help someone to prevent the error, here it is as a separate answer:
Some are saying that the error is occurring because of OS - Windows-Linux; but this isn't directly related to OS. (However, there may be differences in default settings of MySQL within different installers for different OSes, I am not sure.)
The main reason is the flag STRICT_TRANS_TABLES for sql_mode setting. if a value is not specified in INSERT statement for TEXT datatype column and if the flag exist in the sql_mode setting then MySQL is reporting an error; and if the flag doesn't exist then MySQL is only reporting a warning and inserts the record.
So, to prevent this error, one can remove the STRICT_TRANS_TABLES from sql_mode setting of MySQL. (He my need to reset the mode to the previous value if it can affect other operations on the database.)
According to the documentation of SQL mode in MySQL ...
For STRICT_TRANS_TABLES, MySQL converts an invalid value to the closest valid value for the column and inserts the adjusted value. If a value is missing, MySQL inserts the implicit default value for the column data type. In either case, MySQL generates a warning rather than an error and continues processing the statement. Implicit defaults are described in Section 11.6, “Data Type Default Values”.
... and documentation of Data Type Default Values ...
The BLOB, TEXT, GEOMETRY, and JSON data types cannot be assigned a default value.
... TEXT column can not have a default value, but if STRICT_TRANS_TABLES is removed from sql_mode then MySQL inserts empty string '' if no value is specified for TEXT column in INSERT statement.
I have been charged with maintaining a legacy classic ASP application. The application uses an ODBC system DSN to connect to a MySQL database.
We had to recently update the servers to satisfy some licencing requirements. We were on Windows, with MySQL 4.x and the 3.51 ODBC driver. We moved to a Linux machine running MySQL 5.1.43 and are running the 5.1.6 ODBC driver on the new IIS server.
Users almost instantly started reporting errors as such:
Row cannot be located for updating.
Some values may have been changed
since it was last read.
This is a ghost error, and the same data changes, on the same record, at a different time won't always produce the error. It is also intermittent between different records, as in, sometimes, no matter what values I plug in, I haven't been able to repro the defect on all records.
It is happening across 70 of about 120 scripts, many over 1,000 lines long.
The only consistency I can find is that on all of the scripts that fail, they are all reading/writing floats to the DB. Fields that have a null value don't seem to crash, but if there is a value like '19' in the database (note the no decimal places) that seems to fail, whereas, '19.00' does not. Most floats are defined as 11,2.
The scripts are using ADODB and recordsets. Updates are done with the following pattern:
select * from table where ID =
udpdated recordID
update properties of the record from the form
call RecordSet.Update and RecordSet.Close
The error is generated from the RecordSet.Update command.
I have created a workaround, where rather than select/copy/update I generate an SQL statement that I execute. This works flawlessly (obviously, an UPDATE statement with a where clause is more focused and doesn't consider fields not updated), so I have a pretty good feeling that it is a rounding issue with the floats that is causing a mis-match with the re-retrieval of the record on the update call.
I really would prefer NOT re-writing 100's of these instances (a grep across the source directly finds 280+ update calls).
Can anyone confirm that the issue here is related to floats/rounding?
And if so, is there a global fix I can apply?
Thanks in advance,
-jc
Have a look at MySQL Forums :: ODBC :: Row cannot be located for updating.
They seem to have found some workaround and some explanations as well..
I ran into a similar issue with a VBA macro utilizing 4.1. When upgraded to 5 errors started popping up.
For me the issue was that values being returned to VBA from MySQL was in a unhandled (by VBA) decimal format.
A CAST on the numbers when querying helped to fix the issue.
So for your issue perhaps the ODBC/ASP combination is recording/reading values differently then what you might expect them to be.