I am using spark and my task is to write the last day of month into mysql. But I had some problem with the timezone. My system is in UTC+8 timezone and mysql is in UTC+0 timezone.
The spark code is fairly straightforward
val randomDay = args(0)
val temp = spark.sql(s"""select last_day('$randomDay') as lastday from .....""")
But when I looked at the mysql table, it shows 2019/3/30, which is one day off. I tried temp.show(), and it shows the correct date 2019/3/31.
I know it has something to do with timezone but I don't know how to fix it.
Also, the column type of mysql table is date, which shouldn't depend on timezone.
Related
I developed node js services with Sequelize with MySQL Everything working good. I am storing data in DB with date and time with service but if I get the same data with service, node js automatically converting time (a wrong format which is not in the database), I want same time which is present in the database. to solve this issue I set the timezone in services and in the database but none of them is working. can anyone provide me with the solution.
suggestion
you could include time format function in mysql
for example
SELECT TIME_FORMAT("19:30:10", "%H %i %s");
result
TIME_FORMAT("19:30:10", "%H %i %s")
19 30 10
ref
That's not the issue of node js , Sequelize by default gives the time in UTC , so that date time will be correct
Check : Just compare your date time with UTC , and check is that you are getting.
Solution : So what you can do is , just convert returned date and time to your timezone and you are good to go.
And that's a good practice if you are using UTC everywhere , you can get benefits when you are getting / showing in many countries.
My purpose is since the time I login my page, I want my web to show how many updated data in the database. My code is like this
$current = $_SESSION['date'];
$query2 = "SELECT * FROM gmaptracker1 WHERE datetime >= '$current'";
When I echo the $current, it showed 27/09/14 : 06:53:24, so the $current is correct, however, when I request the number of database where date>='$current', I get zero, although I have inserted to the database the data with datetime 28/09/14 : 06:53:24 and 29/09/14 : 06:53:24.
Can anyone help me to get out of this, please?
Few things,
It seems like your code is vulnerable to SQL Injection. Just because you retrieve the content of the date from a session, it doesn't mean that it's safe.
Also, why do you need it to be in a session variable? If you always want to retrieve dates bigger than NOW() you can just write your query this way:
SELECT * FROM gmaptracker1 WHERE datetime >= NOW()
The part that caught my attention was the format you're storing the dates.
You said that when you echo'ed $_SESSION['date'] the value was: 27/09/14 : 06:53:24
Now, that does not look like the date format at all. Is your column actually a datetime or timestampcolumn?
If it's a VARCHAR or any other type other than datetime or timestamp, then there's no way for MySQL to know that you're trying to retrieve dates that occur in the future.
If you already have data stored, then it isn't going to be as easy as changing the data type because you already have data, and your data is in the wrong format. The format that MySQL stores datetime information is as follows:
YYYY-MM-DD HH:MM:SS
Based on the comments you left, you don't need the time > NOW(), you need the time when you log in. Now it makes sense why you're storing that time in a variable.
The problem is the format you're storing it.
Since you're using PHP, then you have to store the time this way:
$time = new DateTime();
$_SESSION['date'] = $time->format("Y-m-d H:i:s");
I have a very simple code for a cron job that makes a date entry into an SQL DB:
$qry_cron_test = "INSERT INTO ".$tblprefix."cron_test SET
create_datetime = '".date("Y-d-m H:i:s")."'";
$rs_cron_test = $db -> Execute($qry_cron_test);
The problem is the following:
Between 1st and 12th of every month the date entry is like this - 2014-10-03 07:30:39, which is what i want.
However, when the current date is between 13th and the end of the month, the date entry looks like this - 0000-00-00 00:00:00. Then when 1st comes the entires are all ok again.
I tested this on couple of servers and also locally on Xampp always with the same result.
Any suggestions? What could be possibly wrong?
You have month and day the wrong way around.
$qry_cron_test = "INSERT INTO ".$tblprefix."cron_test SET
create_datetime = '".date("Y-m-d H:i:s")."'";
$rs_cron_test = $db -> Execute($qry_cron_test);
date("Y-m-d H:i:s")
I recommend that, unless you need milisecond information, you always store date information in Unix Timestamp. It is lighter to store, since it is only a integer value, is faster to retrieve and is universal, since it is always based on UTC.
Specially in PHP, converting date information to (time() and strtotime) and from (date()) a unix timestamp is pretty easy. This way no matter where your user is, you can always show correct information in local time with almost no effort.
Wouldn't it be simpler to just do this:
insert into cron_test
create_datetime
values
(current_timestamp)
I have a small Django project that imports data dumps from MongoDB into MySQL. Inside these Mongo dumps are dates stored in epoch time. I would expect epoch time to be the same regardless of time zone but what I am seeing is that the Django TIME_ZONE setting has an effect on the data created in MySQL.
I have been testing my database output with the MySQL UNIX_TIMESTAMP function. If I insert a date with the epoch of 1371131402880 (this includes milliseconds) I have my timezone set to 'America/New_York', UNIX_TIMESTAMP gives me 1371131402, which is the same epoch time excluding milliseconds. However if I set my timezone to 'America/Chicago' I get 1371127802.
This is my code to convert the epoch times into Python datetime objects,
from datetime import datetime
from django.utils.timezone import utc
secs = float(epochtime) / 1000.0
dt = datetime.fromtimestamp(secs)
I tried to fix the issue by putting an explict timezone on the datetime object,
# epoch time is in UTC by default
dt = dt.replace(tzinfo=utc)
PythonFiddle for the code
I've tested this Python code in isolation and it gives me the expected results. However it does not give the correct results after inserting these object into MySQL through a Django model DateTimeField field.
Here is my MySQL query,
SELECT id, `date`, UNIX_TIMESTAMP(`date`) FROM table
I test this by comparing the unix timestamp column in the result of this query against the MongoDB JSON dumps to see if the epoch matches.
What exactly is going on here? Why should timezone have any effect on epoch times?
Just for reference, I am using Django 1.5.1 and MySQL-python 1.2.4. I also have the Django USE_TZ flag set to true.
I am no python or Django guru, so perhaps someone can answer better than me. But I will take a guess at it anyway.
You said that you were storing it in a Django DateTimeField, which according to the documents you referenced, stores it as a Python datetime.
Looking at the docs for datetime, I think the key is understanding the difference between "naive" and "aware" values.
And then researching further, I came across this excellent reference. Be sure the read the second section, "Naive and aware datetime objects". That gives a bit of context to how much of this is being controlled by Django. Basically, by setting USE_TZ = true, you are asking Django to use aware datetimes instead of naive ones.
So then I looked back at you question. You said you were doing the following:
dt = datetime.fromtimestamp(secs)
dt = dt.replace(tzinfo=utc)
Looking at the fromtimestamp function documentation, I found this bit of text:
If optional argument tz is None or not specified, the timestamp is converted to the platform’s local date and time, and the returned datetime object is naive.
So I think you could do this:
dt = datetime.fromtimestamp(secs, tz=utc)
Then again, right below that function, the docs show utcfromtimestamp function, so maybe it should be:
dt = datetime.utcfromtimestamp(secs)
I don't know enough about python to know if these are equivalent or not, but you could try and see if either makes a difference.
Hopefully one of these will make a difference. If not, please let me know. I'm intimately familiar with date/time in JavaScript and in .Net, but I'm always interested in how these nuances play out differently in other platforms, such as Python.
Update
Regarding the MySQL portion of the question, take a look at this fiddle.
CREATE TABLE foo (`date` DATETIME);
INSERT INTO foo (`date`) VALUES (FROM_UNIXTIME(1371131402));
SET TIME_ZONE="+00:00";
select `date`, UNIX_TIMESTAMP(`date`) from foo;
SET TIME_ZONE="+01:00";
select `date`, UNIX_TIMESTAMP(`date`) from foo;
Results:
DATE UNIX_TIMESTAMP(`DATE`)
June, 13 2013 13:50:02+0000 1371131402
June, 13 2013 13:50:02+0000 1371127802
It would seem that the behavior of UNIX_TIMESTAMP function is indeed affected by the MySQL TIME_ZONE setting. That's not so surprising, since it's in the documentation. What's surprising is that the string output of the datetime has the same UTC value regardless of the setting.
Here's what I think is happening. In the docs for the UNIX_TIMESTAMP function, it says:
date may be a DATE string, a DATETIME string, a TIMESTAMP, or a number in the format YYMMDD or YYYYMMDD.
Note that it doesn't say that it can be a DATETIME - it says it can be a DATETIME string. So I think the actual value being implicitly converted to a string before being passed into the function.
So now look at this updated fiddle that converts explicitly.
SET TIME_ZONE="+00:00";
select `date`, convert(`date`, char), UNIX_TIMESTAMP(convert(`date`, char)) from foo;
SET TIME_ZONE="+01:00";
select `date`, convert(`date`, char), UNIX_TIMESTAMP(convert(`date`, char)) from foo;
Results:
DATE CONVERT(`DATE`, CHAR) UNIX_TIMESTAMP(CONVERT(`DATE`, CHAR))
June, 13 2013 13:50:02+0000 2013-06-13 13:50:02 1371131402
June, 13 2013 13:50:02+0000 2013-06-13 13:50:02 1371127802
You can see that when it converts to character data, it strips away the offset. So of course, it makes sense now that when UNIX_TIMESTAMP takes this value as input, it is assuming the local time zone setting and thus getting a different UTC timestamp.
Not sure if this will help you or not. You need to dig more into exactly how Django is calling MySQL for both the read and the write. Does it actually use the UNIX_TIMESTAMP function? Or was that just what you did in testing?
I have a MySQL Server set to UTC (##global.time_zone = '+00:00') and a table with a DATETIME column in which I store dates in UTC. I'm having problems getting UTC dates to come through when I call a stored procedure via JDBC. Example:
java.util.Date now = new java.util.Date();
sproc = conn.prepareCall("{call TzTestInsert(?)}");
sproc.setTimestamp(1, new java.sql.Timestamp(now.getTime()), Calendar.getInstance(TimeZone.getTimeZone("GMT+00:00")));
sproc.execute();
The TzTestInsert sproc simply takes a DATETIME and inserts it into the table.
I'd expect the database to now hold my current time in UTC, but in fact it holds the current time for my timezone.
If I change the sproc to take a string it works...
DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
dateFormat.setTimeZone(TimeZone.getTimeZone("UTC"));
...
sproc.setString(1, dateFormat.format(now));
but I'd rather use the correct type in the sproc.
Also works if I bypass the sproc, but again not my preferred solution...
String sql = "INSERT INTO TzTest VALUES('" + dateFormat.format(now) + "') ;
With the original sproc I have the same issue if I use a TIMESTAMP datatype in the sproc and table, which isn't surprising with the server in UTC since any timezone conversions specific to MySQL TIMESTAMP should be noops.
Calling the sproc from a MySQL Workbench connection works fine, e.g.
CALL TzTestInsert(UTC_TIMESTAMP());
Seems like the problem is in JDBC. I've looked at the various timezone connection parameters and haven't found any that make a difference.
I must be missing something basic - lots of people do this, right?
Solution was to pass the JDBC driver "useLegacyDatetimeCode=false". See mysql bug http://bugs.mysql.com/bug.php?id=15604
Looks like they left the old code in the driver for backwards compatibility.