There are similar questions, but I am beginning to think mine is related to Spark jdbc APIs since both components seem to be working correctly on their own. I am using Spark 2.4 (which has ms support for timestamps) and have a MySQL 5.7.x version that supports fractional seconds.
I created a simple Dataset, with a TimestampType column, and when I show() it, here is what I get:
+-----------------------+
|my_timestamp |
+-----------------------+
|2021-02-06 12:11:45.335|
+-----------------------+
When I write this to MySQL (using dataset.write()), it creates the table automatically, with SQL TIMESTAMP type for the column, and the milliseconds part is lost upon insert.
For a second test, I created the table manually and defined the colon as TIMESTAMP(3). When I manually insert timestamps with ms part to it, everything works correctly. But when I write using Spark jdbc APIs, once again the ms part is truncated and it becomes 2021-02-06 12:11:45.0.
The only workaround that comes to mind is to keep the column as a long/BIGINT and convert it to DATETIME/TIMESTAMP when querying.
Am I doing something wrong here?
Well, StringType to the rescue. Apparently if I keep the Spark column as String with a value formatted the way MySQL expects, e.g. "2020-11-20 23:06:41.745", I can insert to a MySQL TIMESTAMP(3) column without any truncation or other problems.
This feels more like a workaround, so I still want to learn if there is a way to do this correctly.
Related
I am getting errors in Mac OS Coldfusion 2016 reading a mySQL 5.6.41 database with field type of datetime. A simple cfquery select * with cfdump produces java class error "java.time.LocalDateTime" on the datetime fields while producing expected data output in all other fields.
Attempting to output the field value as text, it returns the date/time with a T separator '2021-02-07T15:32:54' (which could be parsed).
But no ColdFusion date/time functions work due to this format.
The data was exported from mySQL 5.6.19 via SQL export using Sequel Pro and imported into the new 5.6.41 instance. All code runs fine on the previous server. I have attempted using the installed mySQL 5 datasource in ColdFusion and a JDBC driver. Both connect fine, but produce same DATETIME format.
Changing the field type to DATE or TIMESTAMP allows the CFDUMP to display without error in the DATETIME fields (obviously minus TIME if DATE).
There is a large amount of labor/overhead involved to not be able to keep DATETIME working as built (plus I believe its the correct field type). I have run out of google options and hoping someone can explain the difference and reason and solution the Coldfusion 2016 will not query data in the same manner as similar code/server.
The only way I solved this was to remove mysql-connector-java-8.0.28.jar and replace it with an older version - mysql-connector-java-5.1.38-bin.jar in my case. So, the problem comes from the mySQL connector.
I use Laravel 5.6.x framework, i have a connection on a Mysql database of which one of the tables contains a Time(6) field type.
sample value : '00:01:02.154120'
I want my model to receive the value with same precision but i always get a truncated value corresponding to 'H:i:s' format even i cast my column to datetime or string.
I created my table from migration and i saw Blueprint support well precision option for time type since version 5.5. So i guess there is a way to retrieve the whole value ?
How can i achieve this ?
This is a known problem: https://github.com/laravel/framework/issues/24214
It's caused by a bug in PHP that will be fixed in 7.3: https://github.com/php/php-src/pull/3257
I'm migrating and changing some data from one Mysql db to another Mysql db with a slightly different structure. The main difference is that on the first database dates are expressed in local timezone (Europe/Rome), instead on the target db they are UTC.
I'm sharing my db connection in all transformations.
I already made my transformations and everything works fine, but I didn't figure out a way to convert automatically all my dates in the right timezone. I was hoping to have something at connection level in order that all dates are automatically converted.
Otherwise I've add some extra transformation for every table and field (and they are many)!
I tried with the option serverTimezone at database level but it didn't work.
Does exist a smart way to do this conversion avoiding to add new transformations?
Eventually I found the solution of my problem using the approach described from adamnyc here.
I've got a legacy database that was running on CakePHP and MySQL that's being migrated to a new Rails App on a Postgres database with a wildly different structure. I have one small piece of migration that's giving me fits, and I'm hoping someone here can point me in the right direction.
Essentially, there are date columns (of MySQL type Date) that contain malformed dates. Most of the malformed dates are of the form '2012-08-00', and the MySQL2 adapter chokes on these (as obviously 00 is not a valid day of the month). If I could just get them into the model I could do the necessary conversions to turn them into the much more complete new format. Even getting them out of the Database as a string would be sufficient, I could do the necessary manipulation that way.
I get the following error:
Mysql2::Error: Invalid date: 2011-12-00
whenever I try to select one of the invalid dates from the system. There are 3800 rows in the table, I would estimate that about half are so attempting to go through and modify them all by hand would take a great deal (but not inordinate, if that's the way it needs to be done) amount of time.
Any suggestions would be highly appreciated!
Something like this should work (not tested):
update [table] set [field] = DATE_ADD([field],INTERVAL 1 day) where day([field]) = '0'
while inserting date mysql server takes the year and day correctly but the month is defaultly taking januavary into the database.
i am using mysql server version 5.0.22
i am inserting the date though application.
the application was developed by using Springs web mvc frame-work and Hibernate.
Can you display the mysql INSERT statement which is being used to insert the data into the database? This would indicate whether it is a malformed INSERT statement or whether it is a valid statement using incorrect data.
Once you know what is happening, you can track down where the problem really is. I would guess it is a faulty date-parsing function.
Another possibility is that the software is expecting the date in European format (dd-mm-yyyy) and getting the entry in US format (mm-dd-yyyy) or vice-versa. And this is causing parsing errors.