Distilling this project down to the simplest of terms;
Users click a button, a record is made with a timestamp of NOW().
NOW() of course equals the time on the server of record creation.
I need to show them stats based on their timezone, not mine.
What is the best method for dealign with time zone offsets in MySql? Is there a specific field format that is designed to deal with an offset?
I will need to run things along the lines of:
SELECT DATE_FORMAT(b_stamp, '%W') AS week_day, count(*) AS b_total, b_stamp
FROM table
WHERE
(b_stamp >= DATE_SUB(NOW(), INTERVAL 7 DAY))
AND
(user_id = '$user_id') GROUP BY week_day ORDER BY b_stamp DESC
I would rather not ask the user what time zone they are in, I assume JS is the only way to pull this data out of the browser. Maybe if they are on a mobile device, and this is not a web based app, I could get it there, but that may not be the direction this goes in.
I am considering the best way may be to determine their offset, and set a variable to "server_time" +/- their_offset. This makes it appear as if the server is in a different location. I believe this would be best, as there would be no additional +/- logic I need to add to the code, muddying it and making it ugly.
On the other hand, that puts the data in the database with time stamps that are all over the board.
Suggestions?
You can use javascript to get timezone from client as follows:
var timeZone=(new Date().gettimezoneOffset()/60)*(-1);
print the variable out and test before using it. I think this will be your simplest bet.
Other than using JS, you could get the time zone of their IP address (using something like ip2location) then use MySQL's CONVERT_TZ() function.
Related
I have an attribute in MYSQL database called "dueDate". I want to update the record on the due date at 11:59PM.
Is there a way to create an event or cronjob that could do that?
You can set an event, or cronjob, for any particular time, if your system administration allows you to use either facility. It's easy to look up how to create a MySQL repeating event.
But this is a brittle way of dealing with time dependencies in your business rules. What do I mean "brittle?" For one thing, if something goes wrong and the job doesn't run, your business rules are fouled up and need to be repaired. For another thing, cronjobs and events don't run at precise times of day, they run on or after that time of day. They can take awhile to start.
So, I suggest you use rules in a query to enforce your business rules. Suppose, for example, your original desire is to set a column called is_overdue to 1 at the end of the due date. Instead, use a query like this to compute your is_overdue column.
SELECT whatever, dueDate,
IF(dueDate >= CURDATE() + INTERVAL 1 DAY, 1, 0) is_overdue
FROM table ...
This has the advantage that it will always be correct, down to the millisecond, and won't depend on the running of a brittle background job.
Events and cronjobs are better used for purging of stale records. For example, you can get rid of any records that have been expired for 30 days or more by using this kind of query in them.
DELETE FROM table WHERE dueDate <= CURDATE() - INTERVAL 30 DAY
If your cronjob / event fails to run on a particular day, the next day's run will still do the cleanup correctly.
Edit. The point of this suggestion is to compute the time-dependent column (is_expired in the example). If you follow this suggestion, you won't update the table at all. Instead, you'll use the suggested query whenever you retrieve the is_expired value.
Pro tip. When you want to do something at a time <= the last moment of a particular day, you're better off doing it at a time < the first moment of the next day. That is, for best results use
WHERE dueDate < '2017-11-17' + INTERVAL 1 DAY
in place of
WHERE dueDate <= '2017-11-17 23:59:59'
Why? the last moment of a day is hard to express precisely, especially if your system's timing using subsecond precision. But the first moment of a day is easy to express precisely.
This question is specifically about future dates and times (for past values UTC is undoubtedly the first choice).
I'd wonder if anybody had a suggestion as to the "best" way to save a future date and time in a MySQL database (or for that matter also generally), particularly in a context where the column can hold times from different timezones. Considering that timezone rules may change UTC might not be the best option.
The most feasible option I found so far would be the save it as text in the location's local time, together with the location (eg. "America/*"). Unfortunately this solution might be more prone to data corruption and is definitely less convenient for calculations.
Could anybody think of something better?
Thanks in advance
First, I've written about this in extensive detail before, so please read my answers here and here, as well as this blog post by Lau Taarnskov.
With specific regard to MySQL, you generally don't want to use a TIMESTAMP field for the local time of a future event, as it will convert from the session's time zone to UTC at write time, and convert back from UTC to the session's time zone at read time. Even if these are the same time zone ids (which they don't have to be), there's no guarantee that the time zone data won't change for one or both of the time zones between when you write the data and when the event takes place.
Instead, use a DATETIME field, which does no implicit time zone conversions. You get the exact value out that you wrote. Store the local time of the event, and store a VARCHAR field containing the time zone identifier for the event. This is the only way to retain the user's original intent.
Rationale and edge cases are all described in the answers I gave previously.
The considerations for saving future datestamps are pretty much the same as for past datestamps.
(I call them datestamps because both DATETIME and TIMESTAMP are reserved words in MySQL. For the sake of discussion I want a word that doesn't imply either data type.)
If you're building a system to be used by people in multiple time zones, it's a good idea to ask each user for her time zone preference, and store it in her user profile. Then, when she logs in you can retrieve it, then do
SET time_zone = 'America/Halifax'
or whatever, naming the user's time zone preference.
If your MySQL server is running on a Linux, BSD, or other *nix system, these time zones come from the zoneinfo subsystem on that machine. zoneinfo gets updated when various national jurisdictions change time zone rules. The people who maintain popular distros routinely push updates to zoneinfo, so you'll be reasonably up to date. (If your MySQL server is running on a Windows host, do some reading about MySQL time zone stuff on that OS. It's more of a hassle to keep things up to date.)
Then, if you use TIMESTAMP data types for your datestamps, any time you retrieve a value, it is automatically translated from UTC to the local timezone before display. Any time you store a value it is automatically translated to UTC. The NOW() value is timestamp-like in this respect. So if you, for example, do
UPDATE appointment
SET datestamp = NOW() + INTERVAL 7 DAY
WHERE id = something
you'll store a UTC time that's a week after this moment. Then if you do
SELECT datestamp
FROM appointment
WHERE id = something
the user will see the time in her local timezone as set with SET timezone.
If you use DATETIME data types for your datestamps, you can offset them yourself when you store and retrieve them. When you store them, offset them from the local timezone to UTC. When you retrieve them, go the other way. Use CONVERT_TZ() for that.
UPDATE appointment
SET datestamp = CONVERT_TZ(NOW(), 'America/Halifax', 'UTC') + INTERVAL 7 DAY
WHERE id = something
SELECT CONVERT_TZ(datestamp, 'UTC', 'America/Halifax') datestamp
FROM appointment
WHERE id = something
Obviously, substitute your user's choice of timezone for 'America/Halifax' in these queries.
IF YOU POSSIBLY CAN AVOID IT don't store your datestamps with reference to a local time that changes from daylight savings to standard time. If you do that, you will have ongoing glitches on the changeover days for the lifetime of your application. I know this because I've inherited a couple of systems that worked that way. Bad idea. UTC: good idea.
Last year I was working on a project for university where one feature necessitated the expiry of records in the database with almost to-the-second precision (i.e. exactly x minutes/hours after creation). I say 'almost' because a few seconds probably wouldn't have meant the end of the world for me, although I can imagine that in something like an auction site, this probably would be important (I'm sure these types of sites use different measures, but just as an example).
I did research on MySQL events and did end up using them, although now that I think back on it I'm wondering if there is a better way to do what I did (which wasn't all that precise or efficient). There's three methods I can think of using events to achieve this - I want to know if these methods would be effective and efficient, or if there is some better way:
Schedule an event to run every second and update expired records. I
imagine that this would cause issues as the number of records
increases and takes longer than a second to execute, and might even
interfere with normal database operations. Correct me if I'm wrong.
Schedule an event that runs every half-hour or so (could be any
time interval, really), updating expired records. At the same time, impose
selection criteria when querying the database to only return records
whose expiration date has not yet passed, so that any records that
expired since the last event execution are not retrieved. While this
would be accurate at the time of retrieval, it defeats the purpose
of having the event in the first place, and I'd assume the extra
selection criteria would slow down the select query. In my project
last year, I used this method, and the event updating the records
was really only for backend logging purposes.
At insert, have a trigger that creates a dynamic event specific to
the record that will expire it precisely when it should expire.
After the expiry, delete the event. I feel like this would be a
great method of doing it, but I'm not too sure if having so many
events running at once would impact on the performance of the
database (imagine a database that has even 60 inserts an hour -
that's 60 events all running simultaneously for just one hour. Over
time, depending on how long the expiration is, this would add up).
I'm sure there's more ways that you could do this - maybe using a separate script that runs externally to the RDBMS is an option - but these are the ones I was thinking about. If anyone has any insight as to how you might expire a record with precision, please let me know.
Also, despite the fact that I actually did use it in the past, I don't really like method 2 because while this works for the expiration of records, it doesn't really help me if instead of expiring a record at a precise time, I wanted to make it active at a certain time (i.e. a scheduled post in a blog site). So for this reason, if you have a method that would work to update a record at a precise time, regardless of what that that update does (expire or post), I'd be happy to hear it.
Option 3:
At insert, have a trigger that creates a dynamic event specific to the record that will expire it precisely when it should expire. After the expiry, delete the event. I feel like this would be a great method of doing it, but I'm not too sure if having so many events running at once would impact on the performance of the database (imagine a database that has even 60 inserts an hour - that's 60 events all running simultaneously for just one hour. Over time, depending on how long the expiration is, this would add up).
If you know the expiry time on insert just put it in the table..
library_record - id, ..., create_at, expire_at
And query live records with the condition:
expire_at > NOW()
Same with publishing:
library_record - id, ..., create_at, publish_at, expire_at
Where:
publish_at <= NOW() AND expire_at > NOW()
You can set publish_at = create_at for immediate publication or just drop create_at if you don't need it.
Each of these, with the correct indexing, will have performance comparable to an is_live = 1 flag in the table and save you a lot of event related headache.
Also you will be able to see exactly why a record isn't live and when it expired/should be published easily. You can also query things such as records that expire soon and send reminders with ease.
I've got a dataset that I want to be able to slice up by date interval. It's a bunch of scraped web data and each item has a unix-style milisecond timestamp as well as a standard UTC datetime.
I'd like to be able to query the dataset, picking out the rows that are closest to various time intervals:
e.g.: Every hour, once a day, once a week, etc.
There is no guarantee that the timestamps are going to fall evenly on the interval times, otherwise I'd just do a mod query on the timestamp.
Is there a way to do this with SQL commands that doesn't involve stored procs or some sort of pre-computed support tables?
I use the latest MariaDB.
EDIT:
The marked answer doesn't quite answer my specific question but it is a decent answer to the more generalized problem so I went ahead and marked it.
I was specifically looking for a way to query a set of data where the timestamp is highly variable and to grab out rows that are reasonably close to periodic time intervals. E.g.: get all the rows that are the closest to being on 24 hour intervals from right now.
I ended up using a modulus query to solve the problem: timestamp % interval < average spacing between data points. This occasionally grabs extra points and misses a few but was good enough for my graphing application.
And them I got sick of the node-mysql library crashing all the time so I moved to MongoDB.
You say you want 'closest to various time intervals' but then say 'every hour/day/week', so the actual implementation will depend on what you really want, but you can use a host of standard date/time functions to group records, for example count by day:
SELECT DATE(your_DateTime) AS Dt, COUNT(something) AS CT
FROM yourTable
GROUP BY DATE(your_DateTime)
Count by Hour:
SELECT DATE(your_DateTime) AS Dt,HOUR(your_DateTime) AS Hr, COUNT(something) AS CT
FROM yourTable
GROUP BY DATE(your_DateTime), HOUR(your_DateTime)
See the full list of supported date and time functions here:
https://mariadb.com/kb/en/date-and-time-functions/
In my DB I am storing the users last login date/time using NOW(). I realize this stores using my servers timezone which I have no problem with. While I plan to use this for my own purposes (they say they never logged in or yada yada) I also want to be able to display this last login time on their profile for them to see as well.
What would be the best practice for this? At the moment I am thinking I should just let them choose/edit the timezone from their profile and then this in turn will convert the NOW() value stored in the DB to their time. This would require another column in the DB, but would allow me to do the proper calculation for their time as well as list the current timezone they have selected for their profile.
For instance, I'm in est so I could show 2013-09-04 02:46:05 EST to them on their account profile, but also let them edit their timezone to correct it for their display while keeping the original est date in the DB (my servers config tz).
I realize the giving the user their last login date/time seems kind of useless, but for this website it actually will have some value to them so I want to make sure it reflects 'their' time and makes sense to them.
Is this the best practice or better ideas out there?
You cannot find out the timezone of the user from the server. Period.
I strongly recommend you store the login time using UTC (from getutcdate()) as it is unambiguous. If you use local time then daylight savings transitions result in "repeated hours" where the same time will be recorded for events which occur an hour apart. This makes it impossible to work out what time the event actually occurred.
If you don't know the timezone of the user, just display UTC and tell the user it is UTC so they are not confused.
However if you want to know the timezone of the user, you must get the client software to tell you, perhaps by asking the user, and storing this in his preferences. Alternatively you may be able to guess from JavaScript.
See also this question, which discusses the timezone issue in more detail:
Storing DateTime (UTC) vs. storing DateTimeOffset