Related
My data contains hour & minute of time and I want to load this data into MySql database.
Sample Data: 305 -- This means Hours=03 & Minutes=05
But when I upload the data into MySql column of type TIME, it is stored as 00:03:05 (HH:MM:SS).
Whereas, I want it to be stored as 03:05:00.
I can make changes to my data using Python & then load into MySql. However, I was wondering if there is a way to do it using MySql itself.
It is not up to MySQL to guess what you mean by "305". It's up to you to convert/format yor data to "03:05:00", then MySQL will undertand it properly.
insert into try (elasp) values ("03:05:00")
See MySQL documentation: https://dev.mysql.com/doc/refman/8.0/en/time.html
While converting to TIME datatype the numeric value is assumed to have HHMMSS format.
MySQL 8.0 Reference Manual / ... / The TIME Type:
Be careful about assigning abbreviated values to a TIME column. MySQL interprets abbreviated TIME values with colons as time of the day. That is, '11:12' means '11:12:00', not '00:11:12'. MySQL interprets abbreviated values without colons using the assumption that the two rightmost digits represent seconds (that is, as elapsed time rather than as time of day). For example, you might think of '1112' and 1112 as meaning '11:12:00' (12 minutes after 11 o'clock), but MySQL interprets them as '00:11:12' (11 minutes, 12 seconds). Similarly, '12' and 12 are interpreted as '00:00:12'.
If you want to use HHMM format during convertion then simply multiply the value by 100.
CREATE TABLE test (int_source INT, time_destination TIME);
INSERT INTO test (int_source) VALUES ('305');
SELECT * FROM test;
UPDATE test SET time_destination = int_source * 100;
SELECT * FROM test;
✓
✓
int_source | time_destination
---------: | :---------------
305 | null
✓
int_source | time_destination
---------: | :---------------
305 | 03:05:00
db<>fiddle here
Or, if the value to be converted to TIME datatype has some string type you may concatenate '00' to it:
SET time_destination = CONCAT(int_source, '00')
Lets say, I have a table:
+------------+-----------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+------------+-----------+------+-----+-------------------+-----------------------------+
| id | int(10) | NO | PRI | | AUTOINCREMENT |
| id_action | int(10) | NO | IDX | | |
| a_date | date | NO | IDX | | |
| a_datetime | datetime | NO | IDX | | |
+------------+-----------+------+-----+-------------------+-----------------------------+
Each row has some id_action, and the a_date and a_datetime when it was executed on the website.
My question is, when I want to return COUNT() of each id_action grouped by a_date, is it same, when I use this two selects, or they are different in speed? Thanks for any explanation.
SELECT COUNT(id_action), id_action, a_date
FROM my_table
GROUP BY a_date
ORDER BY a_date DESC
and
SELECT COUNT(id_action), id_action, DATE_FORMAT(a_datetime, '%Y-%m-%d') AS `a_date`
FROM my_table
GROUP BY DATE_FORMAT(a_datetime, '%Y-%m-%d')
ORDER BY a_date DESC
In other words, my question is, that each action has its datetime, and if I really need column a_date, or it is the same using DATE_FORMAT function and column a_datetime and I dont need column a_date?
I ran both the queries on similar table on MySQL 5.5.
The table has 10634079 rows.
First one took 10.66 initially and always takes approx 10 secs on further attempts.
Seconds Query takes 1.25 mins to execute first time, on second, 3rd.... attempts its taking 22.091 secs
So in my view, if your are looking for performance, then you must have column a_date, as its taking half of the time when executed without Date_Format.
If performance is not the primay concern (like data redundancy can be) then a_datetime column will serve all other date/datetime related purposes.
DATE : The DATE type is used for values with a date part but no time part.
DATETIME: The DATETIME type is used for values that contain both date and time parts.
so if you have DATETIME you can always derive DATE from it but from DATE you can not get DATETIME.
And as per your sql there will not be a major difference.
It will be better not to have a_date because you already have a_datetime.
but in general if you can use TIMESTAMP you should, because it is more space-efficient than DATETIME.
Using a_date to group by day will be more efficient than a_datetime because of your conversion. In T-SQL I use a combination of DATEADD() and DATEDIFF() to get the date only from DATETIME since math is more efficient than data conversion. For example (again, using T-SQL though I'm sure there's something similar for MySQL):
SELECT COUNT(id_action), id_action,
DATEADD(DD,DATEDIFF(DD,0,a_datetime),0) AS [a_date]
FROM my_table
GROUP BY DATEADD(DD,DATEDIFF(DD,0,a_datetime),0) AS [a_date]
ORDER BY a_date DESC
This will find the number of days between day 0 and a_datetime then add that number of days to day 0 again. (Day 0 is just an arbitrary date chosen for it's simplicity.)
Perhaps the MySQL version of that would be:
DATE_ADD('2014-01-01', INTERVAL DATEDIFF('2014-01-01',a_datetime) DAY)
Sorry I don't have MySQL installed or I would try that myself. I'd expect it to be more efficient than casting/formatting but less efficient than using a_date.
If you are doing a function in your group by clause: "GROUP BY DATE_FORMAT(a_datetime, '%Y-%m-%d')", you will not be leveraging your index: "a_datetime".
As for speed, I believe there will be no noticeable difference between indexing on datetime vs date (but it's always easy to test with 'explain')
Lastly, you can always read a datetime as a date (using cast functions if need be). Your schema is not normalized if you have both a a_date and a_datetime. You should consider removing one of them. If date provides enough granularity for your application, then get rid of datetime. Otherwise, get rid of a_date and cast as required
As already mentioned, the performance of any function(o_datetime) will be worse than just a_date. The choice depends on on your needs, if there is no need to DATETIME, take a DATE and that is.
If you still need to find a function to convert, then I advise you to take a date().
See also How to cast DATETIME as a DATE in mysql?
Put the two statements in editor SQL and execute (CTRL-L) statistics.
https://technet.microsoft.com/en-us/library/ms178071%28v=sql.105%29.aspx
https://msdn.microsoft.com/pt-br/library/ms190287.aspx?f=255&MSPPError=-2147217396
I have thousands of photos that were taken in Tanzania and I want to store the date and time each photo was taken in a MySQL database. The server, however, is located in the U.S. and I run into problems when I try to store a Tanzanian date-time that falls within the "invalid" hour during spring Daylight Savings time (in the U.S.). Tanzania doesn't do DST, so the time is an actually valid time.
Additional complications are that there are collaborators from many different timezones who will need to access the date-time values stored in the database. I want them to always come out as Tanzanian time and not in the local times that various collaborator are in.
I'm reluctant to set session times because I know that there will be problems when someone sometime forgets to set a session time and gets the times out all wrong. And I do not have authority to change anything about the server.
I've read:
Daylight saving time and time zone best practices and
MySQL datetime fields and daylight savings time -- how do I reference the "extra" hour? and
Storing datetime as UTC in PHP/MySQL
But none of them seems to address my particular problem. I'm not an SQL expert; is there a way to specify timezone when setting DATETIMEs? I haven't seen one. Otherwise, any suggestions on how to approach this issue is greatly appreciated.
Edit:
Here's an example of the problem I'm running into. I send the command:
INSERT INTO Images (CaptureEvent, SequenceNum, PathFilename, TimestampJPG)
VALUES (122,1,"S2/B04/B04_R1/IMAG0148.JPG","2011-03-13 02:49:10")
And I get the error:
Error 1292: Incorrect datetime value: '2011-03-13 02:49:10' for column 'TimestampJPG'
This date and time exists in Tanzania, but not in the U.S., where the database is.
You said:
I want them to always come out as Tanzanian time and not in the local times that various collaborator are in.
If this is the case, then you should not use UTC. All you need to do is to use a DATETIME type in MySQL instead of a TIMESTAMP type.
From the MySQL documentation:
MySQL converts TIMESTAMP values from the current time zone to UTC for storage, and back from UTC to the current time zone for retrieval. (This does not occur for other types such as DATETIME.)
If you are already using a DATETIME type, then you must be not setting it by the local time to begin with. You'll need to focus less on the database, and more on your application code - which you didn't show here. The problem, and the solution, will vary drastically depending on language, so be sure to tag the question with the appropriate language of your application code.
None of the answers here quite hit the nail on the head.
How to store a datetime in MySQL with timezone info
Use two columns: DATETIME, and a VARCHAR to hold the time zone information, which may be in several forms:
A timezone or location such as America/New_York is the highest data fidelity.
A timezone abbreviation such as PST is the next highest fidelity.
A time offset such as -2:00 is the smallest amount of data in this regard.
Some key points:
Avoid TIMESTAMP because it's limited to the year 2038, and MySQL relates it to the server timezone, which is probably undesired.
A time offset should not be stored naively in an INT field, because there are half-hour and quarter-hour offsets.
If it's important for your use case to have MySQL compare or sort these dates chronologically, DATETIME has a problem:
'2009-11-10 11:00:00 -0500' is before '2009-11-10 10:00:00 -0700' in terms of "instant in time", but they would sort the other way when inserted into a DATETIME.
You can do your own conversion to UTC. In the above example, you would then have '2009-11-10 16:00:00' and '2009-11-10 17:00:00' respectively, which would sort correctly. When retrieving the data, you would then use the timezone info to revert it to its original form.
One recommendation which I quite like is to have three columns:
local_time DATETIME
utc_time DATETIME
time_zone VARCHAR(X) where X is appropriate for what kind of data you're storing there. (I would choose 64 characters for timezone/location.)
An advantage to the 3-column approach is that it's explicit: with a single DATETIME column, you can't tell at a glance if it's been converted to UTC before insertion.
Regarding the descent of accuracy through timezone/abbreviation/offset:
If you have the user's timezone/location such as America/Juneau, you can know accurately what the wall clock time is for them at any point in the past or future (barring changes to the way Daylight Savings is handled in that location). The start/end points of DST, and whether it's used at all, are dependent upon location, so this is the only reliable way.
If you have a timezone abbreviation such as MST, (Mountain Standard Time) or a plain offset such as -0700, you will be unable to predict a wall clock time in the past or future. For example, in the United States, Colorado and Arizona both use MST, but Arizona doesn't observe DST. So if the user uploads his cat photo at 14:00 -0700 during the winter months, was he in Arizona or California? If you added six months exactly to that date, would it be 14:00 or 13:00 for the user?
These things are important to consider when your application has time, dates, or scheduling as core function.
References:
MySQL Date/Time Reference
The Proper Way to Handle Multiple Time Zones in MySQL (Disclosure: I did not read this whole article.)
MySQL stores DATETIME without timezone information. Let's say you store '2019-01-01 20:00:00' into a DATETIME field, when you retrieve that value you're expected to know what timezone it belongs to.
So in your case, when you store a value into a DATETIME field, make sure it is Tanzania time. Then when you get it out, it will be Tanzania time. Yay!
Now, the hairy question is: When I do an INSERT/UPDATE, how do I make sure the value is Tanzania time? Two cases:
You do INSERT INTO table (dateCreated) VALUES (CURRENT_TIMESTAMP or NOW()).
You do INSERT INTO table (dateCreated) VALUES (?), and specify the current time from your application code.
CASE #1
MySQL will take the current time, let's say that is '2019-01-01 20:00:00' Tanzania time. Then MySQL will convert it to UTC, which comes out to '2019-01-01 17:00:00', and store that value into the field.
So how do you get the Tanzania time, which is '20:00:00', to store into the field? It's not possible. Your code will need to expect UTC time when reading from this field.
CASE #2
It depends on what type of value you pass as ?. If you pass the string '2019-01-01 20:00:00', then good for you, that's exactly what will be stored to the DB. If you pass a Date object of some kind, then it'll depend on how the db driver interprets that Date object, and ultimate what 'YYYY-MM-DD HH:mm:ss' string it provides to MySQL for storage. The db driver's documentation should tell you.
All the symptoms you describe suggest that you never tell MySQL what time zone to use so it defaults to system's zone. Think about it: if all it has is '2011-03-13 02:49:10', how can it guess that it's a local Tanzanian date?
As far as I know, MySQL doesn't provide any syntax to specify time zone information in dates. You have to change it a per-connection basis; something like:
SET time_zone = 'EAT';
If this doesn't work (to use named zones you need that the server has been configured to do so and it's often not the case) you can use UTC offsets because Tanzania does not observe daylight saving time at the time of writing but of course it isn't the best option:
SET time_zone = '+03:00';
Beginning with MySQL 8.0.19, you can specify a time zone offset when inserting TIMESTAMP and DATETIME values into a table.
The offset is appended to the time part of a datetime literal, with no intervening spaces, and uses the same format used for setting the time_zone system variable, with the following exceptions:
For hour values less than 10, a leading zero is required.
The value '-00:00' is rejected.
Time zone names such as 'EET' and 'Asia/Shanghai' cannot be used; 'SYSTEM' also cannot be used in this context.
The value inserted must not have a zero for the month part, the day part, or both parts. This is enforced beginning with MySQL 8.0.22, regardless of the server SQL mode setting.
EXAMPLE:
This example illustrates inserting datetime values with time zone offsets into TIMESTAMP and DATETIME columns using different time_zone settings, and then retrieving them:
mysql> CREATE TABLE ts (
-> id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY,
-> col TIMESTAMP NOT NULL
-> ) AUTO_INCREMENT = 1;
mysql> CREATE TABLE dt (
-> id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
-> col DATETIME NOT NULL
-> ) AUTO_INCREMENT = 1;
mysql> SET ##time_zone = 'SYSTEM';
mysql> INSERT INTO ts (col) VALUES ('2020-01-01 10:10:10'),
-> ('2020-01-01 10:10:10+05:30'), ('2020-01-01 10:10:10-08:00');
mysql> SET ##time_zone = '+00:00';
mysql> INSERT INTO ts (col) VALUES ('2020-01-01 10:10:10'),
-> ('2020-01-01 10:10:10+05:30'), ('2020-01-01 10:10:10-08:00');
mysql> SET ##time_zone = 'SYSTEM';
mysql> INSERT INTO dt (col) VALUES ('2020-01-01 10:10:10'),
-> ('2020-01-01 10:10:10+05:30'), ('2020-01-01 10:10:10-08:00');
mysql> SET ##time_zone = '+00:00';
mysql> INSERT INTO dt (col) VALUES ('2020-01-01 10:10:10'),
-> ('2020-01-01 10:10:10+05:30'), ('2020-01-01 10:10:10-08:00');
mysql> SET ##time_zone = 'SYSTEM';
mysql> SELECT ##system_time_zone;
+--------------------+
| ##system_time_zone |
+--------------------+
| EST |
+--------------------+
mysql> SELECT col, UNIX_TIMESTAMP(col) FROM dt ORDER BY id;
+---------------------+---------------------+
| col | UNIX_TIMESTAMP(col) |
+---------------------+---------------------+
| 2020-01-01 10:10:10 | 1577891410 |
| 2019-12-31 23:40:10 | 1577853610 |
| 2020-01-01 13:10:10 | 1577902210 |
| 2020-01-01 10:10:10 | 1577891410 |
| 2020-01-01 04:40:10 | 1577871610 |
| 2020-01-01 18:10:10 | 1577920210 |
+---------------------+---------------------+
mysql> SELECT col, UNIX_TIMESTAMP(col) FROM ts ORDER BY id;
+---------------------+---------------------+
| col | UNIX_TIMESTAMP(col) |
+---------------------+---------------------+
| 2020-01-01 10:10:10 | 1577891410 |
| 2019-12-31 23:40:10 | 1577853610 |
| 2020-01-01 13:10:10 | 1577902210 |
| 2020-01-01 05:10:10 | 1577873410 |
| 2019-12-31 23:40:10 | 1577853610 |
| 2020-01-01 13:10:10 | 1577902210 |
+---------------------+---------------------+
Note:
Sadly, the offset is not displayed when selecting a datetime value, even if one was used when inserting it.
The range of supported offset values is -13:59 to +14:00, inclusive.
Datetime literals that include time zone offsets are accepted as parameter values by prepared statements.
MySQL converts TIMESTAMP values from the current time zone to UTC for storage, and back from UTC to the current time zone for retrieval. (This does not occur for other types such as DATETIME.)
You can't... you will find a lot of answers saying you "it is not necessary, store using UTC", but it is: you need to store datetimes with the timezone and MySQL can't...
I worked last 10 years in Postgres and all this kind of problems doesn't exist (date times and timezones are managed with no friction, you can store and compare datetimes expressed in different time zones transparently, the ISOString format is managed naturally,etc...).
I actually work in MariaDB and I can't understand why in 2022, in a globalized world, MySQL continues not supporting per value timezones.
I once also faced such an issue where i needed to save data which was used by different collaborators and i ended up storing the time in unix timestamp form which represents the number of seconds since january 1970 which is an integer format.
Example todays date and time in tanzania is Friday, September 13, 2019 9:44:01 PM which when store in unix timestamp would be 1568400241
Now when reading the data simply use something like php or any other language and extract the date from the unix timestamp. An example with php will be
echo date('m/d/Y', 1568400241);
This makes it easier even to store data with other collaborators in different locations. They can simply convert the date to unix timestamp with their own gmt offset and store it in a integer format and when outputting this simply convert with a
Running the following statement, MySQL seems to mix things up:
select now(), if(false, date(now()), time(now()));
| 2013-07-24 10:06:21 | 2010-06-21 00:00:00 |
If replacing the second argument of the if with a literal string, the statement behaves correctly:
select now(), if(false, 'Banana', time(now()));
| 2013-07-24 10:06:21 | 10:06:21 |
Is this a bug or some really strange quirk?
The return type of IF has to be a datatype that includes the types of both arguments. So if one of the arguments is a DATE and the other is a TIME, the type of IF will be DATETIME.
This doesn't seem necessary in the trivial example query, but consider something like:
SELECT IF(col1, date(col2), time(col2)) AS dt
FROM Table
All the rows of the result have to have the same datatype in the dt column, even though the specific data will depend on what's in that row.
If you want just the date or time, convert it to a string.
I'm trying to figure out what MySQL is doing during the math operation of timestamps.
Picture of resulting problem:
You'll see on the left I have two timestamps, start and end, and I need to find the duration from start to end so I just do this:
end - start
I was getting some really weird results. You can see for a duration of only 3 hours I was getting result back that indicated 2 to 3 times that amount.
When I convert to UTC first, the math works out fine.
Can anyone explain what SQL is doing with the timestamps on the left? I've always been under the impression that all timestamps are UTC under the hood, which is why things like min, max, less than, etc work without converting.
Thanks!
Code:
select
min(timestamp) start,
max(timestamp) end,
max(timestamp) - min(timestamp) start_to_end,
unix_timestamp(min(timestamp)) startUTC,
unix_timestamp(max(timestamp)) endUTC,
unix_timestamp(max(timestamp)) - unix_timestamp(min(timestamp)) start_to_end_UTC
from post_snapshots group by permalink;
These examples have nothing to do with timezone conversions -- when you subtract one date directly from the other, MySQL generates a integer from all existing date parts and then makes the math operations. For example, this query:
select now()+1;
returns (it was '2013-02-26 14:38:31' + 1):
+----------------+
| now()+1 |
+----------------+
| 20130226143832 |
+----------------+
So the difference between "2013-02-19 16:49:21" and "2013-02-19 19:07:31" turns out to be:
20130219190731 - 20130219164921 = 25810
The correct way for getting this subtraction is to either convert the dates to timestamps (like you did) or to use TIMESTAMPDIFF(SECOND, start_date, end_date), which would return 8290.
This isn't a DATETIME vs. TIMESTAMP or a time zone problem.
MySQL handles datetimes as operands to a subtraction (or other mathematical operation) by converting each value to a number, but it's not the number of seconds, its just the datetime digits crunched together. Take an example from your data:
2013-02-19 16:49:21 becomes 20130219164921
2013-02-19 19:07:31 becomes 20130219190731
The difference between those two numbers is... 25810, which is the value you're seeing as the result of your subtraction operation. That's not a result in seconds, as you noted. It really doesn't mean much useful at all.
In contrast, TIMESTAMPDIFF() (or pre-converting to Unix timestamps as you did) actually performs the difference using time-appropriate math if you're looking for the difference to be significant for much beyond sorting:
SELECT TIMESTAMPDIFF(SECOND, '2013-02-19 16:49:21', '2013-02-19 19:07:31')
>>> 8290
What happens is you cannot substract dates/datetimes in mysql. For all math operations, the mysql timestamp data type behaves like datetime data type.
You could use instead
select
TIMESTAMPDIFF(SECOND,min(timestamp),max(timestamp))
from post_snapshots group by permalink;