when inserting data into MySQL and dates are not in an agreeable SQL format like yyyy-mm-dd, what approach do you take to correct this column?
usually i write a custom ETL script to handle date formatting as data is being inserted, but i'm wondering if there a quicker and easier method? like something i could just do in mysql after the data is loaded?
for example, i have different files each with different date formats...
2013 Mar 05 23:26:32
01/03/2015
01/03/2015 23:26:32
annoying right?
when inserting data into MySQL and dates are not in an agreeable SQL format like yyyy-mm-dd, what approach do you take to correct this column?
Parse the dates on the client.
Pass the date as an object of type date.
Repeat for all the values I'm adding according to the required type in each case.
This has two benefits:
I don't need to know what format the database engine requires (which could be dependent on the locale the server happens to be using).
I don't need to worry about quoting of those values to avoid SQL injection attacks.
Summary: database clients understand how to send typed data to their database: make use of that functionality rather than re-creating it.
Related
Trying to do sqoop export from HDFS to MYSQL. Getting mapper error because of different date format between input file vs MySQL. Input file have data in mm/dd/yyyy format where in SQL it is date. I guess MySQL is yyyy-mm-dd.
Because of same getting an error as:
caused by: java.lang.RuntimeException: Can't parse input data: '2/18/2019'
My limitation as the source is from different provider and we can not request them to change it. So in this situation what options do i have? Any suggestions
edit
Unfortunately this answer may not be for you. If you are using a program that you don't have control over the source for, this won't help you.
I'll leave it up only because it is a common question that I see with people new to rdbms programming.
Original answer
Why are you treating dates and times as strings? For that matter why are you building SQL for each row? On the MySql side there is a better way to handle that.
Most RDBMS support the concept of a Prepared Statement, although the implementation differs by vendor. Java had support through jdbc for all of the major vendors flavor of prepared statement, so you don't need to worry about the implementation details.
Every time you execute SQL the database engine goes through several phases before the data is applied or returned. The first and most time consuming phase, called the "prepare" phase, is to analyze the SQL string and computer the ideal access path to complete it with. 50 to 80 percent of the SQL "execution" time is spent in this "Prepare" phase.
A simple optimization is to recognize that the ideal access path in a mature database rarely varies, which allows the programmer to prepare the statement once, return a handle to the access path, then pass only the handle and it's parameters across the wire from the application to the database. This minimizes overheads of access path computation, data type conversions, and network communication while automatically protecting from SQL injection attacks and taking care of such administrivia as date formatting.
In Java, this is represented with the PreparedStatement class.
Always use prepared statements. If used properly, they will eliminate 50 to 80% of the overheads of each database call. They also allow you to choose more simply by using native java types and simply passing the value into the execution with the PS.
Using PreparedStatement also eliminates much of the need to sanitize inputs. By it's nature, you don't need to worry about special characters, apart from those the target will reject (example: dropping a character with a codeine greater than 127 into a database that was built for ASCII only on a platform that enforces character set).
If you need to take input as String, and convert to Date, use java's DateFormat class.
Is it possible to apply some function (user defined / system) to selected columns automatically, may be binding it at schema level.
My scenario is I am saving timestamps of record saving in each table automatically, for which I have used getdate() as default value of those columns, It was working fine till we had our own hosting. But since now we are moving to shared hosting and don't know in which timezone the servers shall be placed in future, I am using GETUTCDATE() to get GMT time.
Since a lot of procedures / functions are already in place, I am looking for something where I don't need to convert this UTC time to my local time explicitly.
So that my Select * from MyTable shall give me time in my fixed timezone using the function I've created.
Let me know if its possible by any way.
Thanks.
It's not exactly clear what you want to do, but there's no way to replace what the SELECT statement asks for with something else: what you ask for in a query is what you get. Unless you replace a table with a view with the same name, but that probably isn't the best approach.
Using a view or function would still mean you have to change your code anyway, so why not just UPDATE all data to UTC time and then do the conversion in your application code? SQL Server has no idea what time zone a client is in anyway, so it isn't possible to do the conversion reliably on the server side. Unless perhaps the client sends the local time zone to the server as a parameter or in CONTEXT_INFO, but there wouldn't be much point because doing it in the client would be simpler anyway.
And of course handling it all in the application will give you a much more flexible, robust solution.
Need some ideas on how to convert an entire database TIMESTAMP columns from one timezone to another.
The current server is UTC, MySQL is also UTC so everything is good in that area. All time related columns are TIMESTAMPs. The problem is that when the time information was being entered, they were in EST/EDT. For example, enter start time: data is 1/1/2011 08:00:00 AM (EST/EDT). Because timezone wasn't implemented at the start, the database stored this as 08:00:00 UTC. So all the data in the database is stored like this. Once we get data that requires timezone info, this model will break.
The question is: how do you convert all these TIMESTAMP columns into the correct UTC time? The code will be changed to deal with this on the display side on a go-forward basis but what about historic data?
The simplest way seems to do a mysqldump --tz-utc of some sort and then import the data back, then release the code. However, I can't seem to find a good example of how to do this properly or if there are other ways to do this in the most efficient way possible.
Thanks!
Could you use the MySQL AddTime function to update the existing data?
UPDATE MyTable SET MyTimeColumn = ADDTIME (MyTimeColumn, -8:00:00) WHERE <the data is bad>
Is there MySQL equivalent to Oracle's TIMESTAMP WITH TIME ZONE?
I need to map a Oracle table, which has some columns with that datatype, into a MySQL table but I can't seem to find an easy way to do this without resorting to some MySQL functions.
Thanks and best regards.
No, you'll need to split the data into 2 columns, one a datetime, and the other holding the timezone information. But what you put in the latter field is dependant on what you've got stored in Oracle - the TIMESTAMP WITH TIME ZONE Datatype can contain the TZ offset and (optionally) the time zone region. Obviously the latter is a requirement for the date time to be semantically correct, but IIRC Oracle does not enforce this data being populated.
without resorting to some MySQL functions
Since MySQL doesn't have the datatype, it'll be very difficult to write MySQL function to process it - it's a lot simpler to create a MySQL compatible representation in Oracle where the datatype is supported. You just need to work out what data you've actually got and decide how you want to represent it in MySQL. By convention that means storing it in UTC along with the TZ in a seperate column, then convert it on selection with the convert_tz function (always from UTC)
MySQL always store timestamps as utc. Dates are always stored without timezone information.
You can configure mysql to return values from now() in different timezones.
To store the current offset you need to add this to some column on your own.
Whats the best way to accomplish changing the timezone of an app? The way I see it the following must occur:
Server TZ is changed by sys admin
mysql must be restarted.
every time based column in the database must have all values updated, using convert_tz or equivalent. So either a mysql script must be written or a grails script that loads every row for each class, updating all the time fields.
Obviously the server should be taken down while this is happening, and backups must be in place incase of an error.
Is there a better/easier way to do this?
Java does not use time zones when using Dates; it stores everything as UTC and only uses time zones when displaying dates. see the following link for a discussion of java date/time.
http://www.odi.ch/prog/design/datetime.php
If you're using the Date, Time, or DateTime column types in MySQL, time zone does not matter.
If you’re using the TIMESTAMP column type, time zones may matter since the TIMESTAMP is stored as a UTC but has conversion done when both retrieving and storing the values. For a discussion of MySQL time zone behavior see
http://dev.mysql.com/doc/refman/5.1/en/time-zone-support.html .
If you’re worried about synchronizing objects across multiple servers in different time zones things get more complicated, see the following thread for a discussion of this.
http://www.pubbs.net/201006/grails/2500-grails-user-how-to-get-gorm-to-store-dates-as-timestamp-in-utc-by-default-without-a-custom-hibernate-mapping-or-joda-time-plu.html
I know this is an old question but I think it's also pretty timeless... at least, I have stumbled upon it a fair number of times recently... so I thought I would contribute my solution.
First, I am using Grails 2.5.1 and PostgreSQL 9.4 as the backend.
Second, Date fields in Groovy/Grails are stored as timestamp without time zone in PostgreSQL. So it seems to me the first answer above is not actually fully correct - the date is not stored in UTC. This observation got me thinking... along the lines of "well if the database doesn't know what the timezone is, who does"? And the first answer that came to mind was "maybe it's Spring".
Third, the specifics of my problem is that I have a lot of dates that I bootstrapped into the database via BootStrap.groovy and new ThisClass().save(). And because these were dates, not dates + times, they all look like 2005-11-03 00:00:00 as PostgreSQL timestamps (without timezones).
Fourth, what really made the penny drop was when I edited one of my GSPs to include the timezone in the date format string, which showed up as PST (where my server is); and when I included timeZone="Asia/Kolkata" in the g:formatDate of the field in question, the time advanced by 12h30. So pretty clearly my server was running in PST8PDT and since that wasn't PostgreSQL I came back to Spring as the potential place to change things.
Fifth, after reading a few comments about setting the locale in grails-app/conf/spring/resources.groovy I decided to try setting the locale and timezone there, as per:
// Place your Spring DSL code here
beans = {
// from http://stackoverflow.com/questions/1569446/grails-how-to-change-the-current-locale
localeResolver(org.springframework.web.servlet.i18n.SessionLocaleResolver) {
defaultLocale = new Locale("en","IN")
java.util.Locale.setDefault(defaultLocale)
println "configure spring/resources.groovy defaultLocale $defaultLocale"
defaultTimeZone = TimeZone.getTimeZone("Asia/Kolkata")
java.util.TimeZone.setDefault(defaultTimeZone)
println "configure spring/resources.groovy defaultTimeZone $defaultTimeZone"
}
}
I also used g:format timezone="Asia/Kolkata" format="dd MMM, yyyy a z" for all my date fields. And that seems to interpret all data in PostgreSQL timestamp fields in the correct timezone and at the anticipated hour (ie the hour that was entered), even though the dates were first entered "in the wrong time zone".
Sixth, g:datePicker - I read a number of posts about making this "time zone sensitive", but I found that its dates are interpreted as in the timezone used by Spring and so in my case, this is exactly what I need. Conversely, if someone wanted to enter dates in their locale and have Spring convert them on the fly to the server's time zone, I guess that would require some extra effort.
Personally I think it would be really cool if g:datePicker accepted timeZone as a parameter and used it in the same way g:formatDate does.
We had problems with time differences between using GORM and using groovy.sql.Sql (for quicker data import).
GORM was using the grails config timezone (UTC) that we set in the Bootstrap, but groovy sql was using the default system timezone (GMT).
The problem was solved by setting the timezone in the $JAVA_OPTS, although you could add the switch to grails opts or to the run-app command.
grails -Duser.timezone=UTC run-app