I'm trying to create a simple AWS Glue script that would load the data from one MySQL database to another.
I'm stuck at the beginning, because my source DB is configured to use "0000-00-00 00:00:00" as default values for timestamp columns.
I've tried using resolveChoice to convert the column to string, I've also tried (for a test) to only select columns that are not timestamp, but I always get the same error:
java.sql.SQLException: Value '0000-00-00' can not be represented as java.sql.Timestamp
I can't modify the DB to replace those values with NULLs and at this point I can't modify any part of Glue preferences (to include zeroDateTimeBehaviour in the jdbc string)
Upgrading to MySQL5.7 or later will solve this issue.
Changes in MySQL 5.7
Related
I am getting errors in Mac OS Coldfusion 2016 reading a mySQL 5.6.41 database with field type of datetime. A simple cfquery select * with cfdump produces java class error "java.time.LocalDateTime" on the datetime fields while producing expected data output in all other fields.
Attempting to output the field value as text, it returns the date/time with a T separator '2021-02-07T15:32:54' (which could be parsed).
But no ColdFusion date/time functions work due to this format.
The data was exported from mySQL 5.6.19 via SQL export using Sequel Pro and imported into the new 5.6.41 instance. All code runs fine on the previous server. I have attempted using the installed mySQL 5 datasource in ColdFusion and a JDBC driver. Both connect fine, but produce same DATETIME format.
Changing the field type to DATE or TIMESTAMP allows the CFDUMP to display without error in the DATETIME fields (obviously minus TIME if DATE).
There is a large amount of labor/overhead involved to not be able to keep DATETIME working as built (plus I believe its the correct field type). I have run out of google options and hoping someone can explain the difference and reason and solution the Coldfusion 2016 will not query data in the same manner as similar code/server.
The only way I solved this was to remove mysql-connector-java-8.0.28.jar and replace it with an older version - mysql-connector-java-5.1.38-bin.jar in my case. So, the problem comes from the mySQL connector.
There are similar questions, but I am beginning to think mine is related to Spark jdbc APIs since both components seem to be working correctly on their own. I am using Spark 2.4 (which has ms support for timestamps) and have a MySQL 5.7.x version that supports fractional seconds.
I created a simple Dataset, with a TimestampType column, and when I show() it, here is what I get:
+-----------------------+
|my_timestamp |
+-----------------------+
|2021-02-06 12:11:45.335|
+-----------------------+
When I write this to MySQL (using dataset.write()), it creates the table automatically, with SQL TIMESTAMP type for the column, and the milliseconds part is lost upon insert.
For a second test, I created the table manually and defined the colon as TIMESTAMP(3). When I manually insert timestamps with ms part to it, everything works correctly. But when I write using Spark jdbc APIs, once again the ms part is truncated and it becomes 2021-02-06 12:11:45.0.
The only workaround that comes to mind is to keep the column as a long/BIGINT and convert it to DATETIME/TIMESTAMP when querying.
Am I doing something wrong here?
Well, StringType to the rescue. Apparently if I keep the Spark column as String with a value formatted the way MySQL expects, e.g. "2020-11-20 23:06:41.745", I can insert to a MySQL TIMESTAMP(3) column without any truncation or other problems.
This feels more like a workaround, so I still want to learn if there is a way to do this correctly.
I've been using the connector for a few years now to push and pull data between a MS SQL Server database and a couple different MySQL databases. I set up a Linked Server, then using OPENQUERY I create the views I need and I write my selects, updates, and inserts against those views. Works like a dream.
However, I'm trying to integrate with a new MySQL database built by a vendor which uses the ENUM datatype, which is causing me trouble.
When I try the OPENQUERY I get a weird error:
OLE DB provider 'MSDASQL' for linked server 'MYSQL_DATABASE' returned data that does not match expected data length for column '[MSDASQL].EnumDataField'. The (maximum) expected data length is 10, while the returned data length is 8.
I can fix this by converting the ENUM field to a CHAR in the query and it works ok.
But now I need to insert or update that ENUM field, and I cannot figure out how to do it. If I convert the datatype on the view, I can't use that view to insert or update.
Is there a way for me to work with ENUM fields through the connector? Especially a way for me to do INSERT or UPDATE of an ENUM value?
I have a table with DATE column. All the dates are valid i.e. no 0000-00-00. But we were in fact using only the year part of these dates. I tried changing the type of this column to YEAR(4), I found following scenarios:
On my local system, MySQL version 5.5.37 via MySQL CLI, changing type retains the year.
On my local system, MySQL version 5.5.37 via Adminer, changing type retains the year.
Our internal DB server, MySQL version 5.0.46 via MySQL CLI, changing type retains the year.
Our internal DB server, MySQL version 5.0.46 via PhpMyAdmin, changing type retains the year.
Staging DB server, MySQL version 5.6.13 via PhpMyAdmin, dates in columns get converted to 0000.
Staging DB server, MySQL version 5.6.13 via MySQL CLI, dates in columns get converted to 0000.
What could cause these issues and how can I solve this? Currently we created a Rake task where we first create an additional column, copy existing column dates to new one, alter the column and copy just years back.
Edit: http://dev.mysql.com/doc/refman/5.6/en/upgrading-from-previous-series.html
Try this
ALTER TABLE `tablename`
CHANGE `columnName` `columnName` YEAR
This returns you 0000 in phpmyadmin. You should create a new column and rename it and then delete the old one.
The YEAR(2) data type has certain issues that you should consider before choosing to use it.
As of MySQL 5.6.6, YEAR(2) is deprecated.
YEAR(2) columns in existing tables are treated as before, but YEAR(2) in new or altered tables are converted to YEAR(4).
Read more here:
http://dev.mysql.com/doc/relnotes/mysql/5.6/en/news-5-6-13.html
When trying to migrate my data from MS SQL to mySQL, essentially converting one database to another I get the following error:
[WRN][copytable]: Invalid timestamp literal detected;"
I've attached a screenshot of what I am getting when using WorkBench. Why would this be occuring? Size of database is 10438MB
configured at my local system with SQL SERVER 2008 R2. I had tried many
freeware utilities for this purposes but failed to do so. Last time i
tried using MySQL workbench to convert this database but again failed. I
tried selection based table conversion as well as the complete database.
Quote:
Datetime - In MYSQL datetime does not contain milliseconds. The SQL
Server datetime datatype contains milliseconds. Error: “Invalid
timestamp literal detected” error. Solution: Convert the datetime
type to a smalldatetime in SQL Server if you do not mind losing the
milliseconds.
from here.