I recently met with a weird problem about SQL timestamp.
I created a table and the column was like
`time` TIMESTAMP(3) DEFAULT '1970-01-01 08:00:01.000'
And I manually inserted 2021-03-18 17:00:32.123
And what I read through python mysql.connector is 2021-03-18 17:00:32.000123 ?????
Seems like I found the rule, then I changed it to TIMESTAMP(1). Guess what I got 2021-03-18 17:00:32.000001
Obviously, it's the way in contrast, what could be the problem, thanks
--- Update---
For the python code, there's nothing special
cursor.execute("select time from table")
times = list(cursor)
And from the debug console, I can see the time is incorrect, as well as the timestamp in UNIX, like
unixTime = times[0].timestamp()
The Unix time will be something like XXXX.000123 instead of XXXX.123
But I can get the correct result from UNIX_TIMESTAMP(), like
cursor.execute("select UNIX_TIMESTAMP(time) from table")
So I think it seems like python mysql lib seems didn't get or convert the format correctly.
Related
I just noticed something odd in my Rails application.
When I update a DateTime column with a random value like below code, it automatically generates a wrong DateTime and saves it instead of NULL.
user = AdminUser.last
data = {last_sign_in_at: "1234568!"}
user.update_attributes(data)
And it generates a query like below:
UPDATE `admin_users` SET `last_sign_in_at` = '1234-01-01 04:56:02', `updated_at` = '2018-01-24 21:27:50' WHERE `admin_users`.`id` = 3
Where is that odd DateTime value coming from ? And interestingly it works with that specific random value. Not with something like "invalid!" and "1234568!" .
So is this something to do with MySql or Rails query generation ?
Maybe I am not aware of such things in database, as it is not something I work with regularly .
It looks like rails use Time library to convert your 1234568! and if you try that in rails console like Time.zone.parse('1234568!') you get back a Time object Sun, 01 Jan 1234 00:00:00 UTC +00:00 but if you try to do a Date.parse(1234568!) it throws you an ArgumentError: invalid date error so maybe you can parse using date to avoid that.
Rails Active Record save uses datetime in insert query although the datatype is time in mysql and time is set in model before saving
mysql schema:
rtime time DEFAULT NULL
ActiveRecord model: Abc
abc = Abc.new {'rtime'=> '18:23 PM'}
abc.save!
corresponding mysql query generated by active record:
insert into abces (rtime) values('2000-01-01 18:23:00');
Later in mysql only the time is stored and date is sliced off, and a warning is also generated.
+-----------------------+
| rtime |
+-----------------------+
| 18:23:00 |
+-----------------------+`
Why is the date appended with time while mysql insertion?
Rails version: 3.2.16
Looks like ActiveRecord often falls back to a DateTime object when assigning a Time in Rails 3+: https://github.com/rails/rails/blob/3-2-stable/activerecord/lib/active_record/attribute_assignment.rb#L130
There's some documentation on the conversion method here: http://apidock.com/rails/Time/time_with_datetime_fallback/class. Also, there's a comment that hints at this on line 180 of the same file: # we instantiate Time object and convert it back to a date thus using Time's logic in handling invalid dates
Thus it seems Rails 3 often treats Time objects as though they were DateTimes, which explains the odd insert query you're seeing. The behavior seems to be changed in Rails 4 so upgrading may resolve the issue.
In any case the date values seem to simply get truncated by MySQL hence the warning, but the stored time seems correct to me.
Finally, do you need a timezone for this time?
I retrieve some data from my MySQL database. This data has the date (not datetime) in one column and some other random data in the other columns. Let's say dtf is my dataframe. There is no index yet so I set one
dtf.set_index('date', inplace=True)
Now I would like to get data from a specific date so I write for example
dtf.loc['2000-01-03']
or just
dtf['2000-01-03']
This gives me a KeyError:
KeyError: '2000-01-03'
But I know its in there. dtf.head() shows me that.
So I did take a look at the type of the index of the first row:
type(dtf.index[0])
and it tells me: datetime.date. All good, now what happens if I just type
dtf.index
Index([2000-01-03, 2000-01-04, 2000-01-05, 2000-01-06, 2000-01-07, 2000-01-10,
2000-01-11, 2000-01-12, 2000-01-13, 2000-01-14,
...
2015-09-09, 2015-09-10, 2015-09-11, 2015-09-14, 2015-09-15, 2015-09-16,
2015-09-17, 2015-09-18, 2015-09-21, 2015-09-22],
dtype='object', name='date', length=2763)
I am a bit confused about the dtype='object'. Shouldn't this read datetime.date?
If I use datetime in my mysql table instead of date everything works like a charm. Is this a bug or a feature? I really would like to use datetime.date because it describes my data best.
My pandas version is 0.17.0
I am using python 3.5.0
My os is arch linux
You should use datetime64/Timestamp rather than datetime.datetime:
dtf.index = pd.to_datetime(dtf.index)
will mean you have a DatetimeIndex and can do nifty things like loc by strings.
dtf.loc['2000-01-03']
You won't be able to do that with datetime.datetime.
I have a small Django project that imports data dumps from MongoDB into MySQL. Inside these Mongo dumps are dates stored in epoch time. I would expect epoch time to be the same regardless of time zone but what I am seeing is that the Django TIME_ZONE setting has an effect on the data created in MySQL.
I have been testing my database output with the MySQL UNIX_TIMESTAMP function. If I insert a date with the epoch of 1371131402880 (this includes milliseconds) I have my timezone set to 'America/New_York', UNIX_TIMESTAMP gives me 1371131402, which is the same epoch time excluding milliseconds. However if I set my timezone to 'America/Chicago' I get 1371127802.
This is my code to convert the epoch times into Python datetime objects,
from datetime import datetime
from django.utils.timezone import utc
secs = float(epochtime) / 1000.0
dt = datetime.fromtimestamp(secs)
I tried to fix the issue by putting an explict timezone on the datetime object,
# epoch time is in UTC by default
dt = dt.replace(tzinfo=utc)
PythonFiddle for the code
I've tested this Python code in isolation and it gives me the expected results. However it does not give the correct results after inserting these object into MySQL through a Django model DateTimeField field.
Here is my MySQL query,
SELECT id, `date`, UNIX_TIMESTAMP(`date`) FROM table
I test this by comparing the unix timestamp column in the result of this query against the MongoDB JSON dumps to see if the epoch matches.
What exactly is going on here? Why should timezone have any effect on epoch times?
Just for reference, I am using Django 1.5.1 and MySQL-python 1.2.4. I also have the Django USE_TZ flag set to true.
I am no python or Django guru, so perhaps someone can answer better than me. But I will take a guess at it anyway.
You said that you were storing it in a Django DateTimeField, which according to the documents you referenced, stores it as a Python datetime.
Looking at the docs for datetime, I think the key is understanding the difference between "naive" and "aware" values.
And then researching further, I came across this excellent reference. Be sure the read the second section, "Naive and aware datetime objects". That gives a bit of context to how much of this is being controlled by Django. Basically, by setting USE_TZ = true, you are asking Django to use aware datetimes instead of naive ones.
So then I looked back at you question. You said you were doing the following:
dt = datetime.fromtimestamp(secs)
dt = dt.replace(tzinfo=utc)
Looking at the fromtimestamp function documentation, I found this bit of text:
If optional argument tz is None or not specified, the timestamp is converted to the platform’s local date and time, and the returned datetime object is naive.
So I think you could do this:
dt = datetime.fromtimestamp(secs, tz=utc)
Then again, right below that function, the docs show utcfromtimestamp function, so maybe it should be:
dt = datetime.utcfromtimestamp(secs)
I don't know enough about python to know if these are equivalent or not, but you could try and see if either makes a difference.
Hopefully one of these will make a difference. If not, please let me know. I'm intimately familiar with date/time in JavaScript and in .Net, but I'm always interested in how these nuances play out differently in other platforms, such as Python.
Update
Regarding the MySQL portion of the question, take a look at this fiddle.
CREATE TABLE foo (`date` DATETIME);
INSERT INTO foo (`date`) VALUES (FROM_UNIXTIME(1371131402));
SET TIME_ZONE="+00:00";
select `date`, UNIX_TIMESTAMP(`date`) from foo;
SET TIME_ZONE="+01:00";
select `date`, UNIX_TIMESTAMP(`date`) from foo;
Results:
DATE UNIX_TIMESTAMP(`DATE`)
June, 13 2013 13:50:02+0000 1371131402
June, 13 2013 13:50:02+0000 1371127802
It would seem that the behavior of UNIX_TIMESTAMP function is indeed affected by the MySQL TIME_ZONE setting. That's not so surprising, since it's in the documentation. What's surprising is that the string output of the datetime has the same UTC value regardless of the setting.
Here's what I think is happening. In the docs for the UNIX_TIMESTAMP function, it says:
date may be a DATE string, a DATETIME string, a TIMESTAMP, or a number in the format YYMMDD or YYYYMMDD.
Note that it doesn't say that it can be a DATETIME - it says it can be a DATETIME string. So I think the actual value being implicitly converted to a string before being passed into the function.
So now look at this updated fiddle that converts explicitly.
SET TIME_ZONE="+00:00";
select `date`, convert(`date`, char), UNIX_TIMESTAMP(convert(`date`, char)) from foo;
SET TIME_ZONE="+01:00";
select `date`, convert(`date`, char), UNIX_TIMESTAMP(convert(`date`, char)) from foo;
Results:
DATE CONVERT(`DATE`, CHAR) UNIX_TIMESTAMP(CONVERT(`DATE`, CHAR))
June, 13 2013 13:50:02+0000 2013-06-13 13:50:02 1371131402
June, 13 2013 13:50:02+0000 2013-06-13 13:50:02 1371127802
You can see that when it converts to character data, it strips away the offset. So of course, it makes sense now that when UNIX_TIMESTAMP takes this value as input, it is assuming the local time zone setting and thus getting a different UTC timestamp.
Not sure if this will help you or not. You need to dig more into exactly how Django is calling MySQL for both the read and the write. Does it actually use the UNIX_TIMESTAMP function? Or was that just what you did in testing?
I using phpMyAdmin 4.4.14 in Win7+Chrome and MySQL 5.6 in Linux.
My timezone is +8
The date command in Linux returns a correct date and time.
When I issue select now() inside the phpMyAdmin, the date and time is correct.
But, when I print the result, the time value in the Generation Time is wrong.
Look like that the Generation Time does not do a +8 to the hour.
How to fix ?
Cheers,
Alvin SIU
Print view is done via PHP script, so the issue is in PHP, not in MySQL. In order to change this timestamp, you need to open php.ini and to change/add date.timezone variable with desired value:
[Date]
; Defines the default timezone used by the date functions
; http://php.net/date.timezone
date.timezone = "Europe/Paris"
All available timezones can be found here:
http://www.php.net/manual/en/timezones.php