I'm migrating and changing some data from one Mysql db to another Mysql db with a slightly different structure. The main difference is that on the first database dates are expressed in local timezone (Europe/Rome), instead on the target db they are UTC.
I'm sharing my db connection in all transformations.
I already made my transformations and everything works fine, but I didn't figure out a way to convert automatically all my dates in the right timezone. I was hoping to have something at connection level in order that all dates are automatically converted.
Otherwise I've add some extra transformation for every table and field (and they are many)!
I tried with the option serverTimezone at database level but it didn't work.
Does exist a smart way to do this conversion avoiding to add new transformations?
Eventually I found the solution of my problem using the approach described from adamnyc here.
Related
I want to implement Kafka CDC to migate data from mySQL to MongoDB
According to the document, mySQL needs to set up before that
- --gtid-mode=ON
- --enforce-gtid-consistency=ON
- --binlog_rows_query_log_events=ON
However, my mySQL does not set that at the beginning, and it already has data inside it. I just wonder if it still works in my case or not. Before investing time and effort to research and do it ?
I have a requirement to add a rule to a legacy MySQL 5.1.73 database whenever a specified date is about to be inserted or updated into any of the tables. (Each field already had a default date setting so using that in some way is not a viable solution).
For example:
IF NewDate = *TheSpecifiedDate* THEN SET NewDate = *ConstantDate*
The logic for this will be identical for all tables that have one or more DateTime fields.
My only solution at the moment is manually add triggers to each and every table. This will be a lot of work to do, and a lot of hassle to maintain if the requirement ever changes.
I therefore wondered if I can somehow make this a globalized rule/trigger for the entire database whenever an insert or update is attempted on any DateTime field?
Or is there a more elegant/preferred way of implementing this kind of global rule that may not even involve using triggers?
Just to close this off, for anyone trying to do something similar...
The main restriction in my scenario was that I am slowly migrating 200+ legacy applications and could therefore not manipulate the legacy database behavior or structure until all applications are converted. I also needed to replicate some existing behavior, whether I liked it or not!
When I posted this question I was using EF6 for my data access layer. Having changed to EF Core 1.1.0 I can now utilise the 'HasDefaultValueSql' setting in my table mappings which resolves my particular issue by allowing me to set a value whenever the DB is updated.
I am developing an application in which user schedules his date, time and event.
I was wondering if there is any possible way that I can store time in hh:mm:ss AM/PM format rather than 24 hour.
I think my question wasn't clear enough , adding some stuff
Problem Definition : Migration from PHP-MSSQL (Windows) web service to PHP-MYSQL (Linux)
Back End was written before y2k its an old program launced as an single platform , prior to me developers ported this program on the web but did not ported the DMS (database management system aka utility for data entry guys) i am not sure about the reason behind this.
Old procedure to enter data , data entry guy used to log in on windows server start the application and enter data.
After migration we can no longer use old DMS program hence i have to write new DMS program.
I asked few question about migration from mssql to mysql before you all can have look here https://stackoverflow.com/questions/22603868/converting-porting-mssql-table-to-mysql-table
Biggest problem that i have facing is data entry guys want their dms just like before not a inch less or more (cant blame them for this).
old dms view
new dms
I am trying my level best to give them old look feel and functionality back as well as wanted to reduce their work since most of the times they have to update an old entry with new dates only , before that they used to do it by deleting whore record and recreation it again.
front end view of date added:
mssql db structure
mysql db structure
You are probably looking for DATE_FORMAT(date,format)
%r Time, 12-hour (hh:mm:ss followed by AM or PM)
You should store dates and times in a database as either a Date, Time or DateTime datatype (depending on what types your db provider supports (MySql reference)). Never store these as a string.
The way the user inputs the value should be determined by their culture settings on the machine:
dateTimePicker1.Format = DateTimePickerFormat.Custom
dateTimePicker1.CustomFormat = Globalization.CultureInfo.CurrentCulture.DateTimeFormat.ShortTimePattern
That way if they prefer 12h format, they just set that in control panel
You can get the inputted value like so:
Dim ts As TimeSpan = New TimeSpan(DateTimePicker1.Value.Hour, DateTimePicker1.Value.Minute, DateTimePicker1.Value.Second)
Storing dates and time in the database instead of string makes life a lot easier when you come to read them because you can just format the date or time in any way you wish.
You can then use the same code in your application to show that date or time in the users preferred culture. (Formatting Date and Time for a Specific Culture)
It also allows you to perform queries on the actual date or time which would not be possible (or at least very inefficient) on a string
.ToString("HH:mm") Solved all my problems for a while.
Whats the best way to accomplish changing the timezone of an app? The way I see it the following must occur:
Server TZ is changed by sys admin
mysql must be restarted.
every time based column in the database must have all values updated, using convert_tz or equivalent. So either a mysql script must be written or a grails script that loads every row for each class, updating all the time fields.
Obviously the server should be taken down while this is happening, and backups must be in place incase of an error.
Is there a better/easier way to do this?
Java does not use time zones when using Dates; it stores everything as UTC and only uses time zones when displaying dates. see the following link for a discussion of java date/time.
http://www.odi.ch/prog/design/datetime.php
If you're using the Date, Time, or DateTime column types in MySQL, time zone does not matter.
If you’re using the TIMESTAMP column type, time zones may matter since the TIMESTAMP is stored as a UTC but has conversion done when both retrieving and storing the values. For a discussion of MySQL time zone behavior see
http://dev.mysql.com/doc/refman/5.1/en/time-zone-support.html .
If you’re worried about synchronizing objects across multiple servers in different time zones things get more complicated, see the following thread for a discussion of this.
http://www.pubbs.net/201006/grails/2500-grails-user-how-to-get-gorm-to-store-dates-as-timestamp-in-utc-by-default-without-a-custom-hibernate-mapping-or-joda-time-plu.html
I know this is an old question but I think it's also pretty timeless... at least, I have stumbled upon it a fair number of times recently... so I thought I would contribute my solution.
First, I am using Grails 2.5.1 and PostgreSQL 9.4 as the backend.
Second, Date fields in Groovy/Grails are stored as timestamp without time zone in PostgreSQL. So it seems to me the first answer above is not actually fully correct - the date is not stored in UTC. This observation got me thinking... along the lines of "well if the database doesn't know what the timezone is, who does"? And the first answer that came to mind was "maybe it's Spring".
Third, the specifics of my problem is that I have a lot of dates that I bootstrapped into the database via BootStrap.groovy and new ThisClass().save(). And because these were dates, not dates + times, they all look like 2005-11-03 00:00:00 as PostgreSQL timestamps (without timezones).
Fourth, what really made the penny drop was when I edited one of my GSPs to include the timezone in the date format string, which showed up as PST (where my server is); and when I included timeZone="Asia/Kolkata" in the g:formatDate of the field in question, the time advanced by 12h30. So pretty clearly my server was running in PST8PDT and since that wasn't PostgreSQL I came back to Spring as the potential place to change things.
Fifth, after reading a few comments about setting the locale in grails-app/conf/spring/resources.groovy I decided to try setting the locale and timezone there, as per:
// Place your Spring DSL code here
beans = {
// from http://stackoverflow.com/questions/1569446/grails-how-to-change-the-current-locale
localeResolver(org.springframework.web.servlet.i18n.SessionLocaleResolver) {
defaultLocale = new Locale("en","IN")
java.util.Locale.setDefault(defaultLocale)
println "configure spring/resources.groovy defaultLocale $defaultLocale"
defaultTimeZone = TimeZone.getTimeZone("Asia/Kolkata")
java.util.TimeZone.setDefault(defaultTimeZone)
println "configure spring/resources.groovy defaultTimeZone $defaultTimeZone"
}
}
I also used g:format timezone="Asia/Kolkata" format="dd MMM, yyyy a z" for all my date fields. And that seems to interpret all data in PostgreSQL timestamp fields in the correct timezone and at the anticipated hour (ie the hour that was entered), even though the dates were first entered "in the wrong time zone".
Sixth, g:datePicker - I read a number of posts about making this "time zone sensitive", but I found that its dates are interpreted as in the timezone used by Spring and so in my case, this is exactly what I need. Conversely, if someone wanted to enter dates in their locale and have Spring convert them on the fly to the server's time zone, I guess that would require some extra effort.
Personally I think it would be really cool if g:datePicker accepted timeZone as a parameter and used it in the same way g:formatDate does.
We had problems with time differences between using GORM and using groovy.sql.Sql (for quicker data import).
GORM was using the grails config timezone (UTC) that we set in the Bootstrap, but groovy sql was using the default system timezone (GMT).
The problem was solved by setting the timezone in the $JAVA_OPTS, although you could add the switch to grails opts or to the run-app command.
grails -Duser.timezone=UTC run-app
I need to convert data that already exists in a MySQL database, to a SQL Server database.
The caveat here is that the old database was poorly designed, but the new one is in a proper 3N form. Does any one have any tips on how to go about doing this? I have SSMS 2005.
Can I use this to connect to the MySQL DB and create a DTS? Or do I need to use SSIS?
Do I need to script out the MySQL DB and alter every statement to "insert" into the SQL Server DB?
Has anyone gone through this before? Please HELP!!!
See this link. The idea is to add your MySQL database as a linked server in SQL Server via the MySQL ODBC driver. Then you can perform any operations you like on the MySQL database via SSMS, including copying data into SQL Server.
Congrats on moving up in the RDBMS world!
SSIS is designed to do this kind of thing. The first step is to map out manually where each piece of data will go in the new structure. So your old table had four fields, in your new structure fileds1 and 2 go to table a and field three and four go to table b, but you also need to have the autogenerated id from table a. Make notes as to where data types have changed and you may need to make adjustments or where you have required fileds where the data was not required before etc.
What I usually do is create staging tables. Put the data in the denormalized form in one staging table and then move to normalized staging tables and do the clean up there and add the new ids as soon as you have them to the staging tables. One thing you will need to do if you are moving from a denormalized database to a normalized one is that you will need to eliminate the duplicates from the parent tables before inserting them into the actual production tables. You may also need to do dataclean up as there may be required fileds in the new structure that were not required in the old or data converstion issues becasue of moving to better datatypes (for instance if you stored dates in the old database in varchar fields but properly move to datetime in the new db, you may have some records which don't have valid dates.
ANother issue you need to think about is how you will convert from the old record ids to the new ones.
This is not a an easy task, but it is doable if you take your time and work methodically. Now is not the time to try shortcuts.
What you need is an ETL (extract, transform, load) tool.
http://en.wikipedia.org/wiki/Extract,_transform,_load#Tools
I don't really know how far an 'ETL' tool will get you depending on the original and new database designs. In my career I've had to do more than a few data migrations and we usually always had to design a special utility which would update a fresh database with records from the old database, and yes we coded it complete with all the update/insert statements that would transform data.
I don't know how many tables your database has, but if they are not too many then you could consider going the grunt root. That's one technique that's guaranteed to work after all.
If you go to your database in SSMS and right-click, under tasks should be an option for "Import Data". You can try to use that. It's basically just a wizard that creates an SSIS package for you, which it can then either run for you automatically or which you can save and then alter as needed.
The big issue is how you need to transform the data. This goes into a lot of specifics which you don't include (and which are probably too numerous for you to include here anyway).
I'm certain that SSIS can handle whatever transformations you need to do to change it from the old format to the new. An alternative though would be to just import the tables into MS SQL as-is into staging tables, then use SQL code to transform the data into the 3NF tables. It's all a matter of what your most comfortable with. If you go the second route, then the import process that I mentioned above in SSMS could be used. It will even create the destination tables for you. Just be sure that you give them unique names, maybe prefixing them with "STG_" or something.
Davud mentioned linked servers. That's definitely another way that you can go (and got my upvote). Personally, I prefer to copy the tables over into MS SQL first since linked servers can sometimes have weirdness, especially when it comes to data types not mapping between different providers. Having the tables all in MS SQL will also probably be a bit faster and saves time if you have to rerun or correct portions of the data. As I said though, the linked server method would probably be fine too.
I have done this going the other direction and SSIS works fine, although I might have needed to use a script task to deal with slight data type weirdness. SSIS does ETL.