My application wants to insert a timestamp value into a TIMESTAMP variable in a mySQL database. The timestamp value is a UTC time in the usual "YYYY-MM-DD HH:MM:SS" format. The problem is that my SQL server is set to use SYSTEM time (SELECT ##global.time_zone says SYSTEM), and the system timezone is Europe/London (the server is running Ubuntu 14.04), so mySQL does a daylight-saving conversion and stores the value one hour off from what it ought to be. (I guess if I was in another timezone e.g. CST then I'd have an unwanted timezone offset as well as the daylight saving hour).
To get mySQL to do The Right Thing it appears that I need to covert the UTC timestamp into system time before I insert it so that mySQL can convert it from system time to UTC time before it stores it internally. This bit of code has the desired effect:-
mysql> select timestamp("2015-05-06 12:34:56")+CURRENT_TIMESTAMP-UTC_TIMESTAMP;
+------------------------------------------------------------------+
| timestamp("2015-05-06 12:34:56")+CURRENT_TIMESTAMP-UTC_TIMESTAMP |
+------------------------------------------------------------------+
| 20150506133456.000000 |
+------------------------------------------------------------------+
I'm going to run with this for now, but it seems like a bit of a palaver, so is there a better way?
[edit] Another round of RTFM'ing gave me this idea...
mysql> select CONVERT_TZ("2015-05-06 12:34:56", "+00:00", "SYSTEM");
+-------------------------------------------------------+
| CONVERT_TZ("2015-05-06 12:34:56", "+00:00", "SYSTEM") |
+-------------------------------------------------------+
| 2015-05-06 13:34:56 |
+-------------------------------------------------------+
That looks a lot cleaner. Any better ideas?
You can probably use the CONVERT_TZ function which uses the specified timezone to parse the date instead of system timezone:
SELECT CONVERT_TZ('2015-05-06 12:34:56','+00:00','SYSTEM') AS `Local Time`,
UNIX_TIMESTAMP(CONVERT_TZ('2015-05-06 12:34:56', '+00:00', 'SYSTEM')) AS `UNIX Timestamp`;
+---------------------+----------------+
| Local Time | UNIX Timestamp |
+---------------------+----------------+
| 2015-05-06 17:34:56 | 1430915696 |
+---------------------+----------------+
The local time value will differ depending on which system the query is run. However, the UNIX timestamp value will be same.
You can then insert the local time value in the timestamp column; MySQL will store the value after conversion.
Related
I am using a table with:
CREATE TABLE tv (
datetime datetime NOT NULL,
value int(4),
metric varchar(25),
PRIMARY KEY (datetime)
);
My system is in CET so I get this from a select:
select * from tv;
+---------------------+-------+----------+
| datetime | value | metric |
+---------------------+-------+----------+
| 2017-08-09 14:17:27 | 0 | TV power |
| 2017-08-09 14:20:04 | 0 | TV power |
| 2017-08-09 14:40:04 | 0 | TV power |
| 2017-08-09 14:45:03 | 0 | TV power |
When I try to graph it (in grafana), I use:
SELECT
UNIX_TIMESTAMP(datetime) as time_sec,
value,
metric
FROM tv
ORDER BY datetime
And it works fine except that it writes everything with 2 hours more in the future. So I guess this is because the UNIX_TIMESTAMP is thinking that the datetime is in UTC and looking at the system value, as it is CET, it is doing a +02:00 before converting to epoch or something like this.
I get this also:
SELECT ##global.time_zone;
+--------------------+
| ##global.time_zone |
+--------------------+
| SYSTEM |
+--------------------+
1 row in set (0.00 sec)
Meaning it is using the system timezone which is CET.
How can I get my UNIX_TIMESTAMP to convert correctly to the CET?
Thanks.
Here is what the documentation has to say about UNIX_TIMESTAMP:
If UNIX_TIMESTAMP() is called with a date argument, it returns the value of the argument as seconds since '1970-01-01 00:00:00' UTC.
From what I understand, MySQL is shifting the input timestamp to UTC time, doing the calculation, and then returning the offset still in UTC time. One option to correct for this would be to shift your timestamps from "UTC" time to "CET" time before calling UNIX_TIMESTAMP. I use quotes here, because your timestamps are already in the desired CET timezone, but we have to spoof MySQL by countering its own efforts to be smart and convert everything to UTC.
Something along these lines should work:
SELECT
UNIX_TIMESTAMP(CONVERT_TZ(datetime, 'UTC', 'CET')) AS time_sec,
value,
metric
FROM tv
ORDER BY datetime
Once again, we are shifting your timestamps from UTC to CET time, to offset UNIX_TIMESTAMP() which will be doing the opposite of that.
If the call above to CONVERT_TZ() does not work for you, you may have to spend some time configuring your MySQL timezone tables. Here is a SO question which will get you started:
Database returned an invalid value in QuerySet.dates()
As a final comment, I think the best long term solution for you would be to just store your timestamps in UTC time. Then, you only need to convert for incoming and outgoing dates, but not for internal MySQL calculations. Storing all timestamp information in UTC time is a common database practice.
Is there a way that Timestamp can insert UTC time rather than the local time(System_time_Zone).
As per MySQL documentation -
"MySQL converts TIMESTAMP values from the current time zone to UTC for storage, and back from UTC to the current time zone for retrieval."
When I insert a now() in a timestamp datatype, the value getting saved is the local time which is the same with Datetime datatype. The date value seems to be static for Datetime but TimeZone keeps updating when Session timezone is updated.
MySQL Workbench 6.3 queries - I am Currently in EDT time zone.
desc datedemo ;
--------------------------------------------------
| mydatetime | 'datetime' |
--------------------------------------------------
| mytimestamp |'timestamp' |
--------------------------------------------------
| utcdate | datetime |
--------------------------------------------------
SELECT ##global.time_zone, ##session.time_zone;
--------------------------------------------------
| ##global.time_zone|'##session.time_zone'
--------------------------------------------------
| SYSTEM | SYSTEM |
--------------------------------------------------
insert into datedemo values (now(), now(),UTC_TIMESTAMP());
select * from datedemo
----------------------------------------------------------------------
| mydatetime | mytimestamp | utcdate
----------------------------------------------------------------------
| 2017-06-21 16:20:28| 2017-06-21 16:20:28 |2017-06-21 20:20:28
----------------------------------------------------------------------
SET SESSION TIME_ZONE = "+00:00"; // UTC time ZONE
select * from datedemo
----------------------------------------------------------------------
| mydatetime | mytimestamp | utcdate
----------------------------------------------------------------------
| 2017-06-21 16:20:28| 2017-06-21 20:20:28 |2017-06-21 20:20:28
----------------------------------------------------------------------
Where is timeStamp datatype saving the UTC time? As per my understanding it is just using the session timeZone and converting the time accordingly, which is done by Datetime also, the only difference being datetime is static but TimeStamp keeps shifting using the client timeZone.
I could be completely wrong but I would greatly appreciate if this question can be answered.
From the MySQL docs:
MySQL converts TIMESTAMP values from the current time zone to UTC for storage, and back from UTC to the current time zone for retrieval. (This does not occur for other types such as DATETIME.)
Whatever value you put into a DATETIME type will remain exactly how you set it.
The value you put into a TIMESTAMP type will be converted when you store it, from the session time zone to UTC. Then when you read it back it converts from UTC to the session time zone.
The global time zone only defines what the session time zone is set to if you don't set it explicitly during your session.
In general, if you know you want conversions across time zones, then use a TIMESTAMP type. On the other hand, if you want the value to always be exactly what you put in, regardless of session time zone, then use a DATETIME type.
It is perfectly valid to create a field like CreatedUTC that is always written and read in UTC and is a DATETIME type. However you might instead prefer just Created as a TIMESTAMP, if you trust the session time zone will be used properly.
MySQL has two data types that are purpose built for storing date & time values - DATETIME and TIMESTAMP. Neither type stores timezone information, and both have different rules.
A DATETIME column will store the exact date & time value provided in the insertion query. (No converstions and no affordances for time zone)
A TIMESTAMP column will convert the date & time value provided at insertion from the timezone of the connection at insertion time to UTC. On retrieval it will convert the date & time value stored from UTC to the timezone of the retrieval connection.
The timezone of both connections can be explicitly or implicitly set according to these rules.
Now before I get to my question let's look at some of the nuances of handling dates & times when daylight savings is involved. Summarizing the answers on another Stack Overflow question as well as what I understand from the MySQL documentation regarding date/time:
When using a DATETIME column and explicitly specifying a value (ie/ 2009-11-01 01:30:00), the value can be ambiguous. DATETIME performs no conversation and simply stores this exact date/time. Say I'm in New York (which follows a daylight savings time). Both at insertion and retrieval I have no way of indicating/knowing if this value refers to 1:30AM at the with-daylight-savings moment (UTC-4) or 1:30AM at the without-daylight-savings moment (UTC-5).
When using a DATETIME column along with NOW(), NOW() evaluates to the date & time value at the start of query execution (ie/ 2009-11-01 01:30:00) and this value is inserted, with no converstions, into the DATETIME field causing the same exact ambiguity as mentioned above.
When using a TIMESTAMP column and explicitly specifying a value (ie/ 2009-11-01 01:30:00), I again have the same problem as mentioned above. There's no way to specify and no way to know which 1:30am I'm referring to.
Now, here's my question:
Given a MySQL connection that is set to a timezone that includes daylight savings (say America/New York), can I be certain that inserting NOW() into a TIMESTAMP column will cause the correct UTC date & time value to be stored? UTC of course does not observe daylight savings, so the UTC time at the 1:30am New York timezone with-daylight-savings moment is different from the UTC time at the 1:30AM New York timezone without-daylight-savings moment.
More specifically: Is the UTC offset of the connection timezone at the start of query execution what is used to perform the to-UTC/from-UTC conversion when I insert/select from a TIMESTAMP column? Going back to my example, at the 1:30am with-daylight-savings moment (America\New York timezone) I'm at UTC-4 and at the 1:30am without-daylight-savings moment (America\New York timezone) I'm at UTC-5 - so in both these moments, would a different value be stored in a TIMESTAMP field when I explicitly insert 2009-11-01 01:30:00 or implicitly insert this same value by using NOW()? Finally, if I'm within a single MySQL connection that spans both these moments, and I execute two queries (one in the first moment, and a separate one in the second moment), will both queries cause the correct (different) UTC value to be stored?
You might want this for a server:
mysql> SHOW VARIABLES LIKE '%zone%';
+------------------+--------+
| Variable_name | Value |
+------------------+--------+
| system_time_zone | UTC | -- Comes from OS
| time_zone | SYSTEM | -- Probably the 'right' setting
+------------------+--------+
With those settings, SELECT NOW() will deliver UTC time, not local time.
For your personal computer, it is probably better to have system_time_zone equal to something like Pacific Daylight Time (or whatever), to reflect your current location.
No conversion happens when INSERTing or SELECTing a DATE or DATETIME. Think of it as being a picture of the clock.
For TIMESTAMP, whatever you give it is converted to/from UTC. That is, the bits stored in the table are UTC, but you can't see that; you only see the converted date&time based on the two settings above.
I suggest that the best way to get the answer is to create a table with a DATETIME and a TIMESTAMP, set the two settings, then see what happens when storing. Then change the settings and do a SELECT.
As far as I know, you're best off with doing this from your code. If you really have to do it via DB, you could use UTC_TIMESTAMP() which will always give you the current time based on UTC. Relying on your server timezone however is not a good idea in my opinion because the servers are bound to undergo changes as you grow/scale.
On the other hand, specifying the time from code will tend to be more important/accurate for you than letting it hit the DB. (Think latency, delayed inserts and what not).
You can test the time_zone change by using the SET time_zone function :
mysql> create table demo (test timestamp);
mysql> SET time_zone='-06:00';
mysql> insert into demo VALUES(NOW());
mysql> SELECT * FROM demo;
+---------------------+
| test |
+---------------------+
| 2017-05-23 08:55:16 |
+---------------------+
mysql> SET time_zone='+02:00';
mysql> insert into demo VALUES(NOW());
mysql> SELECT * FROM demo;
+---------------------+
| test |
+---------------------+
| 2017-05-23 16:55:16 |
| 2017-05-23 16:55:32 |
+---------------------+
According to those results changing the timezone gives consistent results which prove that times are stored UTC (when using timestamp).
I used this one Not sure if it will helpful or not. Please add the below text in your my.cnf file.
[mysqld_safe]
timezone = UTC
Now I am developing a rails project.
My database is MySQL.
I check my MySQL's timezone:
mysql> SELECT ##global.time_zone, ##session.time_zone;
+--------------------+---------------------+
| ##global.time_zone | ##session.time_zone |
+--------------------+---------------------+
| SYSTEM | SYSTEM |
+--------------------+---------------------+
And the SYSTEM timezone is:
mysql> SELECT EXTRACT(HOUR FROM (TIMEDIFF(NOW(), UTC_TIMESTAMP))) AS `timezone`;
+----------+
| timezone |
+----------+
| 9 |
+----------+
Also, my linux system's timezone is:
$ date +'%:z %Z'
+09:00 JST
Now, when I insert a record to database, the created_at value will be utc time. For example:
# Now is 14:25:59
Product.create(name: 'Best')
# select created_at from products;
# 5:25:59
So if I test the feature with rspec in my spec source, I should get that data from database and plus 9 to equal the current system time. Is it good?
Why the data been saved as a utc time with active_record? If use this configuration, is there a way to test the database record in a right way?
The (maybe) best approach is to save all at UTC-Time. So Then you can use the Time.in_time_zone to get the time inside a specified zone.
My 20:00 PM is UTC 13:00 and maybe your 06:00 AM - but in the database its just the same time - UTC.
If it then comes to Rendering you use the rails helper for timing and always give the created_at.in_time_zone.
If it comes to rendering you just throw out the UTC timestamp and let JavaScript to the rest, formatting the timestamp to the users local time.
I am trying to set up a database that stores daily alert times as specified by users. For example, the user wants to receive an alert if some criterion is met each day between 7:00 AM and 7:30 AM. In trying to implement this, I need to accommodate daylight saving time. Here's my attempted solution:
Store the users local time zone (in long form, e.g. "US/Eastern") information in one table (say userInfo), and the alarm times in another table (say userAlarms).
When querying the userAlarms table, convert UTC time into the users local time as specified by the tz column stored in the userInfo table via CONVERT_TZ(UTC_TIME(), 'UTC', userInfo.tz).
Question 1. From my understanding, specifying the time zone name (like US/Eastern) should take daylight saving time into account. For example, calling CONVERT_TZ('00:00:00', 'UTC', 'US/EASTERN') on January 1 should yield '19:00:00', but on July 1 the call should yield '20:00:00'. Am I correct?
Question 2. If Q1 is correct, do I need to constantly update MySQL's time zone table to keep the time zone UTC offsets up to date?
Question 3. The sample given in the MySQL documentation SELECT CONVERT_TZ('2004-01-01 12:00:00','GMT','MET') yields "NULL" when run on my server. Could this be caused by not having the time zone tables set-up?
How can I check this?
If this yields null then the TZ tables have not been set up:
SELECT CONVERT_TZ(now(),'US/Eastern','US/Central');
If you do not have the time zone tables set up you could update the hour
offset in the user table and then do:
select utc_timezone() - interval user_timezone_offset_in_hours hour
from userinfo a
where user_id = 999;
You'd still need a way to update the user's time zone however.
If you are writing this for a web application you can get the time zone via javascript, here's an article that describes how (haven't tried this but it looks like it'll work).
A bit of an explanation with respect to 'interval' above...
One of the more trick constructs in MySQL is the use of the INTERVAL
keyword, best shown by example the (numeric value can be an expression or the field value)
select now() today, now() - interval 1 day yesterday;
+---------------------+---------------------+
| today | yesterday |
+---------------------+---------------------+
| 2011-05-26 13:20:55 | 2011-05-25 13:20:55 |
+---------------------+---------------------+
You can add them and subtract them anyway you like, this is why I never
bother with the date/time add/subtract/convert functions
select now() a, now() - interval 1 day + interval 4 hour + interval 8 minute b;
+---------------------+---------------------+
| a | b |
+---------------------+---------------------+
| 2011-05-26 13:24:16 | 2011-05-25 17:32:16 |
+---------------------+---------------------+
You can use negative numbers (should be good for negative time zone offsets)
these are the same:
select now() - interval 1 month a, now() + interval -1 month b;
+---------------------+---------------------+
| a | b |
+---------------------+---------------------+
| 2011-04-26 13:38:05 | 2011-04-26 13:38:05 |
+---------------------+---------------------+
I found this thread helpful and decided to share the doc page for importing this information. I did exactly as instructed below in CentOS and in RHEL and it worked flawlessly. I am now able to use the CONVERT_TZ function with arguments like "GMT" and "US/Eastern".
From the MySQL documentation this is how to import the MySQL time zone table information:
http://dev.mysql.com/tech-resources/articles/4.1/time.html
For all users running MySQL 4.1.3 or later on a Unix-based system (recall this doesn't work on Windows systems yet):
Populate the time zone tables.
To do so, run the mysql_tzinfo_to_sql program, provided with the MySQL distribution. mysql_tzinfo_to_sql reads the operating system time zone files and generates SQL statements from them. The SQL statements are then processed by mysql, to load the time zone tables.
To run mysql_tzinfo_to_sql successfully, one needs to know where the server machine's operating system time zone files are stored; check for a directory with a name similar to /usr/share/zoneinfo. Pass the directory name on the command line to mysql_tzinfo_to_sql, and send the output into the mysql program. Here's an example.
shell> mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql
Note: The above command assumes that "mysql_tzinfo_to_sql" and "mysql" are in your path. If they're not, you'll need to supply the full path to both when running the command, or switch into the mysql bin folder and use a command like so:
shell> ./mysql_tzinfo_to_sql /usr/share/zoneinfo | ./mysql -u root mysql
Q1: Yes, the CONVERT_TZ takes daylight savings time into account for you. This information and the time that DST starts/ends for each time zone is stored in the time_zone_* tables.
Q2: Yes, as stated in the mysql docs, the time zone information changes per the politics of each area. You'll have to update the time_zone_* tables every time a change occurs. Sucks to be IT sometimes, this is one of them.
Q3: These are the 5 timezone tables, query them to see if they have anything in them:
select * from mysql.time_zone_transition_type;
select * from mysql.time_zone_transition;
select * from mysql.time_zone_name;
select * from mysql.time_zone_leap_second;
select * from mysql.time_zone;