In my DB I am storing the users last login date/time using NOW(). I realize this stores using my servers timezone which I have no problem with. While I plan to use this for my own purposes (they say they never logged in or yada yada) I also want to be able to display this last login time on their profile for them to see as well.
What would be the best practice for this? At the moment I am thinking I should just let them choose/edit the timezone from their profile and then this in turn will convert the NOW() value stored in the DB to their time. This would require another column in the DB, but would allow me to do the proper calculation for their time as well as list the current timezone they have selected for their profile.
For instance, I'm in est so I could show 2013-09-04 02:46:05 EST to them on their account profile, but also let them edit their timezone to correct it for their display while keeping the original est date in the DB (my servers config tz).
I realize the giving the user their last login date/time seems kind of useless, but for this website it actually will have some value to them so I want to make sure it reflects 'their' time and makes sense to them.
Is this the best practice or better ideas out there?
You cannot find out the timezone of the user from the server. Period.
I strongly recommend you store the login time using UTC (from getutcdate()) as it is unambiguous. If you use local time then daylight savings transitions result in "repeated hours" where the same time will be recorded for events which occur an hour apart. This makes it impossible to work out what time the event actually occurred.
If you don't know the timezone of the user, just display UTC and tell the user it is UTC so they are not confused.
However if you want to know the timezone of the user, you must get the client software to tell you, perhaps by asking the user, and storing this in his preferences. Alternatively you may be able to guess from JavaScript.
See also this question, which discusses the timezone issue in more detail:
Storing DateTime (UTC) vs. storing DateTimeOffset
Related
For instance, I have a blog where users can comment and I want everyone can see how long ago the comment was posted, for example: 5 minutes ago OR 3 hours ago.
So if a guy in London posts a comment and a guy in India visits the page, they both should see "1 minute ago" and on hover should see the time relative to their timezone. (10pm in London, 3.30am in India).
My current solution in mind is to use varchar(25) data type and store the time as ISO-8601 (e.g. 2019-12-12T21:46:42+00:00)
Using this I can get the timezone of the commenter and convert the time to the current user's timezone. It works perfectly.
But I wonder if there is a better / more elegant way to do it?
So far I tried using DATETIME and TIMESTAMP data types but they do not seem to be useful in this scenario. I read online that TIMESTAMP is supposed to store time in UTC timezone and send it back in user's timezone but that did not happen for me, it got saved in my local time instead. And yes, I did not specify any time while saving data, MySQL used the CURRENT_TIMESTAMP.
Any thoughts or ideas?
I'd recommend storing all your dates/times in one universal format in your database and UTC would be the best candidate for this.
That way, regardless of their location, it's easy for you to say 1 minute ago...
If you need to display the full date/time on the front-end, you'd need to convert the time from UTC to that user's location, which you can do via PHP's handy DateTime functions:
https://www.php.net/manual/en/datetime.settimezone.php
This question is specifically about future dates and times (for past values UTC is undoubtedly the first choice).
I'd wonder if anybody had a suggestion as to the "best" way to save a future date and time in a MySQL database (or for that matter also generally), particularly in a context where the column can hold times from different timezones. Considering that timezone rules may change UTC might not be the best option.
The most feasible option I found so far would be the save it as text in the location's local time, together with the location (eg. "America/*"). Unfortunately this solution might be more prone to data corruption and is definitely less convenient for calculations.
Could anybody think of something better?
Thanks in advance
First, I've written about this in extensive detail before, so please read my answers here and here, as well as this blog post by Lau Taarnskov.
With specific regard to MySQL, you generally don't want to use a TIMESTAMP field for the local time of a future event, as it will convert from the session's time zone to UTC at write time, and convert back from UTC to the session's time zone at read time. Even if these are the same time zone ids (which they don't have to be), there's no guarantee that the time zone data won't change for one or both of the time zones between when you write the data and when the event takes place.
Instead, use a DATETIME field, which does no implicit time zone conversions. You get the exact value out that you wrote. Store the local time of the event, and store a VARCHAR field containing the time zone identifier for the event. This is the only way to retain the user's original intent.
Rationale and edge cases are all described in the answers I gave previously.
The considerations for saving future datestamps are pretty much the same as for past datestamps.
(I call them datestamps because both DATETIME and TIMESTAMP are reserved words in MySQL. For the sake of discussion I want a word that doesn't imply either data type.)
If you're building a system to be used by people in multiple time zones, it's a good idea to ask each user for her time zone preference, and store it in her user profile. Then, when she logs in you can retrieve it, then do
SET time_zone = 'America/Halifax'
or whatever, naming the user's time zone preference.
If your MySQL server is running on a Linux, BSD, or other *nix system, these time zones come from the zoneinfo subsystem on that machine. zoneinfo gets updated when various national jurisdictions change time zone rules. The people who maintain popular distros routinely push updates to zoneinfo, so you'll be reasonably up to date. (If your MySQL server is running on a Windows host, do some reading about MySQL time zone stuff on that OS. It's more of a hassle to keep things up to date.)
Then, if you use TIMESTAMP data types for your datestamps, any time you retrieve a value, it is automatically translated from UTC to the local timezone before display. Any time you store a value it is automatically translated to UTC. The NOW() value is timestamp-like in this respect. So if you, for example, do
UPDATE appointment
SET datestamp = NOW() + INTERVAL 7 DAY
WHERE id = something
you'll store a UTC time that's a week after this moment. Then if you do
SELECT datestamp
FROM appointment
WHERE id = something
the user will see the time in her local timezone as set with SET timezone.
If you use DATETIME data types for your datestamps, you can offset them yourself when you store and retrieve them. When you store them, offset them from the local timezone to UTC. When you retrieve them, go the other way. Use CONVERT_TZ() for that.
UPDATE appointment
SET datestamp = CONVERT_TZ(NOW(), 'America/Halifax', 'UTC') + INTERVAL 7 DAY
WHERE id = something
SELECT CONVERT_TZ(datestamp, 'UTC', 'America/Halifax') datestamp
FROM appointment
WHERE id = something
Obviously, substitute your user's choice of timezone for 'America/Halifax' in these queries.
IF YOU POSSIBLY CAN AVOID IT don't store your datestamps with reference to a local time that changes from daylight savings to standard time. If you do that, you will have ongoing glitches on the changeover days for the lifetime of your application. I know this because I've inherited a couple of systems that worked that way. Bad idea. UTC: good idea.
Here's the scenario. I'm using a MySQL/NodeJS/Sequelize stack on the server, and I have a request I want to perform.
There are anywhere from 1000-2000 entries that are retrieved from the request, but I don't need the full list of entries. I want the summary of the entries after they have been grouped by the day the entry was made, which condenses down to about 5-10 objects in an array.
If I group by day on the server, it may group them differently from the time zone of the client. And if I send it to the client, then I have to send 1-2k entries for analysis, but it will group them correctly.
How would you handle this scenario?
Use UTC timestamps for everything. For example open the Chrome console and enter this:
// Milliseconds since 1/1/1970
new Date().getTime()
1438263135084
// Human friendly without timezone
new Date().toISOString()
"2015-07-30T13:32:15.715Z"
// Human friendly with timezone
new Date().toString()
"Thu Jul 30 2015 09:32:21 GMT-0400 (EDT)"
All of these are the same (time since 1/1/1970), but simply formatted differently. Using either of the first two will allow you to send dates that can be interpreted and formatted correctly regardless of timezone.
Interesting reading: https://en.wikipedia.org/wiki/Coordinated_Universal_Time
Perhaps you could convert the dates to the user's timezone Django MySQL group by day with timezone - but that of course requires you to know the user's timezone, and caching will be harder
In sequelize, it would be expressed something like
sequelize.fn('CONVERT_TZ', sequelize.col('date'), 'UTC', user_tz)
First off, thanks all for your suggestions on how to handle this. Ultimately, I resolved this issue by doing the following.
First, in the request from the client, I used the moment js method .utcOffset() to get the offset value.
Then, once I had the offset value, I included it into the API query as ?offset=(value here)
Finally, on the server side, I used this value to interpret the local timezone offset of the client, and then grouped all the data there, and send the formatted data back to the client. It resulted in a much faster, much, much smaller response query.
It was a fairly simple solution - I'm bugged it took me to long to come up with it.
This is a HARD question. In fact it is so hard it seems the SQL standard and most of the major databases out there don't have a clue in their implementation.
Converting all datetimes to UTC allows for easy comparison between records but throws away the timezone information, which means you can't do calculations with them (e.g. add 8 months to a stored datetime) nor retrieve them in the time zone they were stored in. So the naive approach is out.
Storing the timezone offset from UTC in addition to the timestamp (e.g. timestamp with time zone in postgres) would seem to be enough, but different timezones can have the same offset at one point in the year and a different one 6 months later due to DST. For example you could have New York and Chile both at UTC-4 now (August) but after the 4th of November New York will be UTC-5 and Chile (after the 2nd of September) will be UTC-3. So storing just the offset will not allow you to do accurate calculations either. Like the above naive approach it also discards information.
What if you store the timezone identifier (e.g. America/Santiago) with the timestamp instead? This would allow you to distinguish between a Chilean datetime and a New York datetime. But this still isn't enough. If you are storing an expiration date, say midnight 6 months into the future, and the DST rules change (as unfortunately politicians like to do) then your timestamp will be wrong and expiration could happen at 11 pm or 1 am instead. Which might or might not be a big deal to your application. So using a timestamp also discards information.
It seems that to truly be accurate you need to store the local datetime (e.g. using a non timezone aware timestamp type) with the timezone identifier. To support faster comparisons you could cache the utc version of it until the timezone db you use is updated, and then update the cached value if it has changed. So that would be 2 naive timestamp types plus a timezone identifier and some kind of external cron job that checks if the timezone db has changed and runs the appropriate update queries for the cached timestamp.
Is that an accurate solution? Or am I still missing something? Could it be done better?
I'm interested in solutions for MySQL, SQL Server, Oracle, PostgreSQL and other DBMS that handle TIMESTAMP WITH TIME ZONE.
You've summarized the problem well. Sadly the answer is to do what you've described.
The correct format to use does depend the pragmatics of what the timestamp is supposed to represent. It can in general be divided between past and future events (though there are exceptions):
Past events can and usually should be stored as something which can never be reinterpreted differently. (eg: a UTC time stamp with a numeric time zone). If the named time zone should be kept (to be informative to the user) then this should be separate.
Future events need the solution you've described. Local timestamp and named time zone. This is because you want to change the "actual" (UTC) time of that event when the time zone rules change.
I would question if time zone conversion is such an overhead? It's usually pretty quick. I'd only go through the pain of caching if you are seeing a really significant performance hit. There are (as you pointed out) some big operations which will require caching (such as sorting billions of rows based on the actual (UTC) time.
If you require future events to be cached in UTC for performance reasons then yes, you need to put a process in place to update the cached values. Depending of the type of DB it is possible that this could be done by the sysadmins as TZ rules change rarely.
If you care about the offset, you should store the actual offset. Storing the timezone identifier is not that same thing as timezones can, and do, change over time. By storing the timezone offset, you can calculate the correct local time at the time of the event, rather than the local time based on the current offset. You may still want to store the timezone identifier, if it's important to know what actual timezone event was considered to have happened in.
Remember, time is a physical attribute, but a timezone is a political one.
If you convert to UTC you can order and compare the records
If you add the name of the timezone it originated from you can represent it in it's original tz and be able to add/substract timeperiods like weeks, months etc (instead of elapsed time).
In your question you state that this is not enough because DST might be changed. DST makes calculating with dates (other than elapsed time) complicated and quite code intensive. Just like you need code to deal with leap years you need to take into account if for a given data / period you need to apply a DST correction or not. For some years the answer will be yes for others no.
See this wiki page for how complex those rules have become.
Storing the offset is basically storing the result of those calculations. That calculated offset is only valid for that given point in time and can't be applied as is to later or earlier points like you suggest in your question. You do the calculation on the UTC time and then convert the resulting time to the required timezone based on the rules that are active at that time in that timezone.
Note that there wasn't any DST before the first world war anywhere and date/time systems in databases handle those cases perfectly.
I'm interested in solutions for MySQL, SQL Server, Oracle, PostgreSQL and other DBMS that handle TIMESTAMP WITH TIME ZONE.
Oracle converts with instant in time to UTC but keeps the time zone or UTC offset depending on what you pass. Oracle (correctly) makes a difference between the time zone and UTC offset and returns what you passed to you. This only costs two additional bytes.
Oracle does all calculations on TIMESTAMP WITH TIME ZONE in UTC. This is does not make a difference for adding months, but makes a difference for adding days as there is no daylight savings time. Note that the result of a calculation must always be a valid timestamp, e.g. adding one month to January 31st will throw an exception in Oracle as February 31st does not exist.
Should I store all datetime's in MySQL database in UTC?
(Using UTC_TIMESTAMP())
How can I display datetime to user, if I don't know user's timezone?
Using UTC is usually a good bet, particularly if you're dealing with logical "instants in time" (such as timestamps for creating entries etc).
As for how you display them to your user: if you don't know the time zone, you can't display a date/time to the user sensibly however you store it, so this is really separate from the UTC decision. One option could be to use Javascript to convert from UTC to local time. Another possibility is to allow the user to specify the time zone themselves.