I need to get the timestamp of interval of 7 days from the current time in milliseconds. I tried date_sub using now() but didn't work for me. How do we do this in hive. I need exactly the interval current_timestamp(unix) and interval of 7 days from the current in my query. Also is there any provision to select the time zone like UTC + 5:30 hrs like that?
I could not find information about millisecond based time calculations in HIVE.
unix_timestamp() is the current timestamp, but it does not have milliseconds.
The offset is 7 days*24 hours/day*3600 secs/hour = 604800 milliseconds
So the timestamp of the current time plus 7 days would be unix_timestamp() + 604800
The UTC part is trickier; you can use to_utc_timestamp, giving it your calculated timestamp, and the timezone it is coming from (as a date). It will return a date string, which you will pass through unix_timestamp()
In other words, assuming it is coming from PST, you should use:
select unix_timestamp(to_utc_timestamp(from_unixtime(unix_timestamp() + 604800), 'PST')) from dual;
See the documentation here:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF
Related
I am facing an odd issue. The global timezone setting in the mySQL is in UTC. There are multiple tables (sub databases) within a single instance of mySQL (I am using database.NET as the manager); so I can't change the global timezone.
The following is my query. All I need is to display the times in EST. I have seen a few solutions with ##sessionTimeZone but they didn't work. Furthermore, I am running into the issue of data being collected till 9PM EST, but in UTC that's 1AM on the next day.
My main confusion is that the data is already stored in the table in UTC; however, I would like to see those datetime fields displayed in EST.
cartId is a numerical field
createDate is a datetime field
I am just using http://fishcodelib.com/database.htm and connecting to the mySQL database.
SELECT DATE(createDate)
,DATE_FORMAT(createDate, '%l%p') as HourOfDay
,count(cartId) as numCarts_ALL
FROM carts
WHERE createDate >= '2014-09-24'
AND createDate < '2014-10-01'
AND HOUR(createDate) >= 10 AND HOUR(createDate) <21
GROUP by DATE(createDate),HOUR(createDate)
;
I will really appreciate any help.
Thanks again
DATETIME data types are not affected by either the global or connection-local timezone setting. That only works for TIMESTAMP data types. That explains why your attempts to muck around with ##session.TimeZone don't have any effect. NOW() and CURDATE() are affected, though.
You say your DATETIME data is stored in UTC. That's excellent. Life is much easier when you store your data that way.
Before you use the advice I'm about to give you, please make sure your MySQL server has its timezone tables loaded correctly. Do this command and make sure you don't get a NULL result.
SELECT CONVERT_TZ(NOW(), 'America/New_York', 'UTC')
If this doesn't work -- if you get a NULL or an error -- you need to get your server people to load the time zone tables. They should do this. They know how. (If they don't, you should get a new service provider.)
We need to use the timezone named 'America/New_York' because presumably you want to switch automatically between EDT and EST on the appropriate dates each year.
Now, to retrieve a correctly converted UTC DATETIME value from a table, you do this:
SELECT CONVERT_TZ(createDate, 'UTC', 'America/New_York') AS createDate
FROM yourTable
This is cool because you can make the timezone setting a user preference if you have users in various time zones.
To store a local time value as UTC just do it in reverse. For example.
INSERT INTO yourTable (createDate) VALUES (CONVERT_TZ(?, 'America/New_York', 'UTC'))
Now do
SELECT NOW()
and look to see whether NOW() is in local time or in UTC. If it is in UTC, then you should start your session by doing SET TIME_ZONE='America/New_York' . This will get your timezone set right so NOW() and CURDATE() do what you want.
Then, to fetch yesterday's (local time) rows from your table do this:
WHERE createDate >= CONVERT_TZ(CURDATE(),'America/New_York','UTC') - INTERVAL 1 DAY
AND createDate < CONVERT_TZ(CURDATE(),'America/New_York','UTC')
This will convert midnight local time to UTC and fetch the range of items.
To get all the data from 4pm today until 2am tomorrow, you could do this:
4pm today (localtime) is CURDATE() + INTERVAL 16 HOUR . 2am tomorrow is CURDATE() + INTERVAL 26 HOUR or you could write it CURDATE() + INTERVAL 1 DAY + INTERVAL 2 HOUR
So fetching that range would need this:
WHERE createDate >= CONVERT_TZ(CURDATE() + INTERVAL 16 HOUR,'America/New_York','UTC')
AND createDate < CONVERT_TZ(CURDATE() + INTERVAL 26 HOUR,'America/New_York','UTC')
You could also use ADDTIME(CURDATE(),'16:00') to obtain a DATETIME value for 4pm today, if you'd rather combine ordinary date objects and ordinary time objects. Similarly, you could get 2am tomorrow like this:
ADDTIME(CURDATE(),'02:00') + INTERVAL 1 DAY
Notice that this form of WHERE clause allows a range scan on an index on the createDate column. This is very good for performance.
To convert datetime from a timezone to another timezone:
SELECT DATE(CONVERT_TZ(createDate, '+00:00', '-04:00')) createdDate,
HOUR(CONVERT_TZ(createDate, '+00:00', '-04:00')) hourOfDay
FROM carts
WHERE
createDate BETWEEN CURRENT_DATE + INTERVAL 16 HOUR AND CURRENT_DATE + INTERVAL 26 HOUR
GROUP BY createdDate, hourOfDay
For different dates
SELECT DATE(CONVERT_TZ(createDate, '+00:00', '-04:00')) createdDate,
HOUR(CONVERT_TZ(createDate, '+00:00', '-04:00')) hourOfDay
FROM carts
WHERE
createDate BETWEEN '2014-09-24' AND '2014-10-01'
GROUP BY createdDate, hourOfDay
HAVING hourOfDay <= 2 OR hourOfDay >= 20
You might think "why not in WHERE clause?"
If functions are used in the where clause, indexes can't be used (which results in a full table scan). So just filter out the records within the daterange, and the hours in the HAVING clause
FYI
CURRENT_DATE + INTERVAL 16 HOUR
Is short for
DATE_ADD(CURRENT_DATE, INTERVAL 16 HOUR)
I'm trying to get an mysql query similar to date_trunc in psql.
Unfortunate Mysql do not have date_trunc function and I found I can use extract instead in Mysql.
What I want to do is write a script which i will run let say 10 minutes past each hour but I want to only select data from begin of an hour till end of this hour.
For example I will run script 12:10 and I want to display data from 11:00:00 till 11:59:59.
In PSQL query would look like that:
SELECT *
FROM data
WHERE time > ( date_trunc('hour',now()) - interval '1 hour' )
AND time <= ( date_trunc('hour',now()) ) ORDER BY time;
I was trying to use extract in similar fashion but I have no rows returned or error :/
Query below returns for example some narrowed data but it's like 2 hours each day from day one when database was started not last hour only:
SELECT *
FROM data
WHERE extract(hour from cr_date) between extract(hour from now()) - interval 1 hour)
AND extract(hour from now())
ORDER BY cr_date;
Any ideas how this can be achieved? or what I'm doing wrong in this query?
Hour is only an integer, so it's finding any matches between , for example, 9 and 10, regardless of the date.
I would recommend
select * FROM data
where cr_date >= date(now()) + INTERVAL hour(now())-1 HOUR
and cr_date <= date(now()) + INTERVAL hour(now()) HOUR
date(now()) returns midnight, and hour(now()) returns the number of hours since midnight
so, at 11:10 am, it should result in a results between midnight + 10 hours (10 am) and midnight + 11 hours (11 am)
first of all, I know that my question is very similar to that one:
MySQL select rows from exactly 7 days ago
the difference is that my dates are stored in the database as a timestamp.
I know that I can use FROM_UNIXTIME to get the date from the timestamp, the thing is, in another answer I read that was very resource consuming (because the timestamp field has to be converted to date in all the records before comparing).
DATE(from_unixtime(timestamp)) = CURRENT_DATE()
Is there any optimized way to do this?
Turn it around: calculate the unix timestamp of the target date first and use that.
WHERE timestamp = UNIX_TIMESTAMP(NOW() - INTERVAL 7 DAY)
MySQL should calculate that value once and use it all the time (needs testing though). If it doesn't, use a variable or code.
I have a table 't' with date(yyyy-mm-dd), hour(1-12), minute(00-59), ampm(a/p), and timezone(pst/est) fields.
How can I select the rows that are <= now()? (ie. already happened)
Thank you for your suggestions!
edit: this does it without attention to the hour/minute/ap/tz fields:
SELECT * FROM t.date WHERE date <= now()
Here's one way to do it - combine all your seconds, minutes, etc into a date and compare to NOW(), making sure you do the comparison in the same time-zone. (Untested):
SELECT *
FROM t
LEFT JOIN y ON t.constant=y.constant
WHERE CONVERT_TZ(STR_TO_DATE(CONCAT(date,' ',hour,':',minute,' 'ampm),
'%Y-%m-%d %l:%i %p' ),
timezone,"SYSTEM") < NOW();
If your hour is 01 - 12 not 1-12 then use %h instead of %l in the STR_TO_DATE.
The STR_TO_DATE tries to stick your date and time columns together and convert them into a date.
The CONVERT_TZ(...,timezone,"SYSTEM") converts this date from whatever timezone is specified in the timezone column to system time.
This is then compared to NOW(), which is always in system time.
As an aside, perhaps you should make a single column date using MySQL's date datatype, as it's a lot easier to do arithmetic on that!
For reference, here is a summary of very useful mysql date functions where you can read up on those featuring in this answer.
Good luck!
SELECT * FROM t
WHERE `date`<=DATE_SUB(curdate(), INTERVAL 1 DAY)
OR (
`date`<=DATE_ADD(curdate(), INTERVAL 1 DAY)
AND
CONVERT_TZ(CAST(CONCAT(`date`,' ',IF(`hour`=12 AND ampm='a',0,if(ampm='a',`hour`,`hour`+12)),':',`minute`,':00') AS DATETIME),'GMT',`timezone`)<=NOW()
)
Rationale for date<=DATE_[ADD|SUB](curdate(), INTERVAL 1 DAY):
The fancy conversion is quite an expensive operation, so we don't want it to run on the complete table. This is why we pre-select against an UNCHANGED date field (possibly using an index). In no timezone can an event being more than a day in current timezone's past be in the future, and in no timezone can an event more than a day in the curent timezone's future be in the past.
I need to pull a variable amount of days of data from a MySQL database that stores the date as a UNIX timestamp in an int column.
For example, if I need the last 5 days of data, what would my query be?
(All queries would start from current date and would go back x amount of days).
Timestamp is considered one of the Date and Time types and therefore any of the Date Time Functions can be used on it.
SELECT * FROM your_table
WHERE Ftimestamp_column > UNIX_TIMESTAMP(DATE_SUB(NOW(), INTERVAL 5 DAY));
I've never tried it but there's a MySQL function to convert unix timestamps into MySQL dates and then you can use DATE_SUB() or whatever. http://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_from-unixtime