I've got a legacy database that was running on CakePHP and MySQL that's being migrated to a new Rails App on a Postgres database with a wildly different structure. I have one small piece of migration that's giving me fits, and I'm hoping someone here can point me in the right direction.
Essentially, there are date columns (of MySQL type Date) that contain malformed dates. Most of the malformed dates are of the form '2012-08-00', and the MySQL2 adapter chokes on these (as obviously 00 is not a valid day of the month). If I could just get them into the model I could do the necessary conversions to turn them into the much more complete new format. Even getting them out of the Database as a string would be sufficient, I could do the necessary manipulation that way.
I get the following error:
Mysql2::Error: Invalid date: 2011-12-00
whenever I try to select one of the invalid dates from the system. There are 3800 rows in the table, I would estimate that about half are so attempting to go through and modify them all by hand would take a great deal (but not inordinate, if that's the way it needs to be done) amount of time.
Any suggestions would be highly appreciated!
Something like this should work (not tested):
update [table] set [field] = DATE_ADD([field],INTERVAL 1 day) where day([field]) = '0'
Related
I have a problem where I try to read some data from QuickBooks Desktop file in SSIS through ADO.NET Source using QODBC driver. Driver is set up correctly, because I am able to preview table contents of the file like so
I am also successful in using SQL Command access mode when I do a simple query like SELECT * FROM Item.
But the moment I try to use SQL Command to incorporate a WHERE clause against TimeModified field like so: SELECT * FROM Item WHERE TimeModified >= '12/21/2022 1:16:34 PM' and try to preview I get an error saying this:
Now, I sent that row by using table access mode into a derived column and looked up what type TimeModified column uses and saw that it uses DBTIMESTAMP. So I suspected that you can't use the >= operator on that type, so what I also tried in my query is utilizing the DATE() function on the WHERE clause, omitting the time piece like so (Although ideally I would like to keep the hours to be able to use them in my query as time is important but for test sake):
But as you can see I got the Expected Lexical Element Not Found error.
I originally intended for this to work through an expression in which I obtain a value for a variable of most recent date by querying from a database prior this step and then get the items from the Item table that are older than said date. However, now I completely cut the expression idea just for the sake of trying to get this to work but it doesn't seem like it wants to. Am I doing something wrong? I made some querying in SQL against a date field by using strings of various formats like I showed earlier and those queries in SQL Server all succeeded but here nothing works.
If you are interested how the connection to QuickBooks Desktop is made I am using 32bit QODBC Driver which you can see there as 'QuickBooks Data' in the ADO.NET Source.
Thank you in advance for your help!
So I figured it out. Apparently with QODBC when it comes to timestamps you have to query them by doing something like this:
SELECT * FROM Item WHERE TimeModified >= {ts '2022-12-16 15:05:55.000'} ORDER BY TimeModified DESC
Here is some more info: https://support.flexquarters.com/esupport/index.php?/Knowledgebase/Article/View/2407/0/qodbc-desktop-troubleshooting-problems-with-date-format
There are similar questions, but I am beginning to think mine is related to Spark jdbc APIs since both components seem to be working correctly on their own. I am using Spark 2.4 (which has ms support for timestamps) and have a MySQL 5.7.x version that supports fractional seconds.
I created a simple Dataset, with a TimestampType column, and when I show() it, here is what I get:
+-----------------------+
|my_timestamp |
+-----------------------+
|2021-02-06 12:11:45.335|
+-----------------------+
When I write this to MySQL (using dataset.write()), it creates the table automatically, with SQL TIMESTAMP type for the column, and the milliseconds part is lost upon insert.
For a second test, I created the table manually and defined the colon as TIMESTAMP(3). When I manually insert timestamps with ms part to it, everything works correctly. But when I write using Spark jdbc APIs, once again the ms part is truncated and it becomes 2021-02-06 12:11:45.0.
The only workaround that comes to mind is to keep the column as a long/BIGINT and convert it to DATETIME/TIMESTAMP when querying.
Am I doing something wrong here?
Well, StringType to the rescue. Apparently if I keep the Spark column as String with a value formatted the way MySQL expects, e.g. "2020-11-20 23:06:41.745", I can insert to a MySQL TIMESTAMP(3) column without any truncation or other problems.
This feels more like a workaround, so I still want to learn if there is a way to do this correctly.
I am developing an application in which user schedules his date, time and event.
I was wondering if there is any possible way that I can store time in hh:mm:ss AM/PM format rather than 24 hour.
I think my question wasn't clear enough , adding some stuff
Problem Definition : Migration from PHP-MSSQL (Windows) web service to PHP-MYSQL (Linux)
Back End was written before y2k its an old program launced as an single platform , prior to me developers ported this program on the web but did not ported the DMS (database management system aka utility for data entry guys) i am not sure about the reason behind this.
Old procedure to enter data , data entry guy used to log in on windows server start the application and enter data.
After migration we can no longer use old DMS program hence i have to write new DMS program.
I asked few question about migration from mssql to mysql before you all can have look here https://stackoverflow.com/questions/22603868/converting-porting-mssql-table-to-mysql-table
Biggest problem that i have facing is data entry guys want their dms just like before not a inch less or more (cant blame them for this).
old dms view
new dms
I am trying my level best to give them old look feel and functionality back as well as wanted to reduce their work since most of the times they have to update an old entry with new dates only , before that they used to do it by deleting whore record and recreation it again.
front end view of date added:
mssql db structure
mysql db structure
You are probably looking for DATE_FORMAT(date,format)
%r Time, 12-hour (hh:mm:ss followed by AM or PM)
You should store dates and times in a database as either a Date, Time or DateTime datatype (depending on what types your db provider supports (MySql reference)). Never store these as a string.
The way the user inputs the value should be determined by their culture settings on the machine:
dateTimePicker1.Format = DateTimePickerFormat.Custom
dateTimePicker1.CustomFormat = Globalization.CultureInfo.CurrentCulture.DateTimeFormat.ShortTimePattern
That way if they prefer 12h format, they just set that in control panel
You can get the inputted value like so:
Dim ts As TimeSpan = New TimeSpan(DateTimePicker1.Value.Hour, DateTimePicker1.Value.Minute, DateTimePicker1.Value.Second)
Storing dates and time in the database instead of string makes life a lot easier when you come to read them because you can just format the date or time in any way you wish.
You can then use the same code in your application to show that date or time in the users preferred culture. (Formatting Date and Time for a Specific Culture)
It also allows you to perform queries on the actual date or time which would not be possible (or at least very inefficient) on a string
.ToString("HH:mm") Solved all my problems for a while.
Whats the best way to accomplish changing the timezone of an app? The way I see it the following must occur:
Server TZ is changed by sys admin
mysql must be restarted.
every time based column in the database must have all values updated, using convert_tz or equivalent. So either a mysql script must be written or a grails script that loads every row for each class, updating all the time fields.
Obviously the server should be taken down while this is happening, and backups must be in place incase of an error.
Is there a better/easier way to do this?
Java does not use time zones when using Dates; it stores everything as UTC and only uses time zones when displaying dates. see the following link for a discussion of java date/time.
http://www.odi.ch/prog/design/datetime.php
If you're using the Date, Time, or DateTime column types in MySQL, time zone does not matter.
If you’re using the TIMESTAMP column type, time zones may matter since the TIMESTAMP is stored as a UTC but has conversion done when both retrieving and storing the values. For a discussion of MySQL time zone behavior see
http://dev.mysql.com/doc/refman/5.1/en/time-zone-support.html .
If you’re worried about synchronizing objects across multiple servers in different time zones things get more complicated, see the following thread for a discussion of this.
http://www.pubbs.net/201006/grails/2500-grails-user-how-to-get-gorm-to-store-dates-as-timestamp-in-utc-by-default-without-a-custom-hibernate-mapping-or-joda-time-plu.html
I know this is an old question but I think it's also pretty timeless... at least, I have stumbled upon it a fair number of times recently... so I thought I would contribute my solution.
First, I am using Grails 2.5.1 and PostgreSQL 9.4 as the backend.
Second, Date fields in Groovy/Grails are stored as timestamp without time zone in PostgreSQL. So it seems to me the first answer above is not actually fully correct - the date is not stored in UTC. This observation got me thinking... along the lines of "well if the database doesn't know what the timezone is, who does"? And the first answer that came to mind was "maybe it's Spring".
Third, the specifics of my problem is that I have a lot of dates that I bootstrapped into the database via BootStrap.groovy and new ThisClass().save(). And because these were dates, not dates + times, they all look like 2005-11-03 00:00:00 as PostgreSQL timestamps (without timezones).
Fourth, what really made the penny drop was when I edited one of my GSPs to include the timezone in the date format string, which showed up as PST (where my server is); and when I included timeZone="Asia/Kolkata" in the g:formatDate of the field in question, the time advanced by 12h30. So pretty clearly my server was running in PST8PDT and since that wasn't PostgreSQL I came back to Spring as the potential place to change things.
Fifth, after reading a few comments about setting the locale in grails-app/conf/spring/resources.groovy I decided to try setting the locale and timezone there, as per:
// Place your Spring DSL code here
beans = {
// from http://stackoverflow.com/questions/1569446/grails-how-to-change-the-current-locale
localeResolver(org.springframework.web.servlet.i18n.SessionLocaleResolver) {
defaultLocale = new Locale("en","IN")
java.util.Locale.setDefault(defaultLocale)
println "configure spring/resources.groovy defaultLocale $defaultLocale"
defaultTimeZone = TimeZone.getTimeZone("Asia/Kolkata")
java.util.TimeZone.setDefault(defaultTimeZone)
println "configure spring/resources.groovy defaultTimeZone $defaultTimeZone"
}
}
I also used g:format timezone="Asia/Kolkata" format="dd MMM, yyyy a z" for all my date fields. And that seems to interpret all data in PostgreSQL timestamp fields in the correct timezone and at the anticipated hour (ie the hour that was entered), even though the dates were first entered "in the wrong time zone".
Sixth, g:datePicker - I read a number of posts about making this "time zone sensitive", but I found that its dates are interpreted as in the timezone used by Spring and so in my case, this is exactly what I need. Conversely, if someone wanted to enter dates in their locale and have Spring convert them on the fly to the server's time zone, I guess that would require some extra effort.
Personally I think it would be really cool if g:datePicker accepted timeZone as a parameter and used it in the same way g:formatDate does.
We had problems with time differences between using GORM and using groovy.sql.Sql (for quicker data import).
GORM was using the grails config timezone (UTC) that we set in the Bootstrap, but groovy sql was using the default system timezone (GMT).
The problem was solved by setting the timezone in the $JAVA_OPTS, although you could add the switch to grails opts or to the run-app command.
grails -Duser.timezone=UTC run-app
while inserting date mysql server takes the year and day correctly but the month is defaultly taking januavary into the database.
i am using mysql server version 5.0.22
i am inserting the date though application.
the application was developed by using Springs web mvc frame-work and Hibernate.
Can you display the mysql INSERT statement which is being used to insert the data into the database? This would indicate whether it is a malformed INSERT statement or whether it is a valid statement using incorrect data.
Once you know what is happening, you can track down where the problem really is. I would guess it is a faulty date-parsing function.
Another possibility is that the software is expecting the date in European format (dd-mm-yyyy) and getting the entry in US format (mm-dd-yyyy) or vice-versa. And this is causing parsing errors.