I have big table (100M+ rows) with Timestamps (GMT) in microseconds as PK and values for that Timestamp.
I would like to get MAX values for days (in period), but "Day" must consider clients local TZ offset.
I tried:
CREATE TEMPORARY TABLE TBASE
SELECT
FLOOR(Timestamp / (86400 * 1000)) * 86400 as ts,
value
FROM mytable
WHERE Timestamp BETWEEN [GmtTimestampStart] and [GmtTimestampEnd]
And then:
SELECT
ts * 1000 as start_of_the_day_timestamp,
MAX(value)
FROM TBASE
GROUP BY start_of_the_day_timestamp
This works ofcourse only for +00:00 TZ.
But if client is in let's say -03:00 TZ:
Client sends correct Timestamp for TEMPORARY TABLE query, for example, if i want to query from start of the day 2021-01-04 to end of the day 2021-01-07, TS will be:
start: 1609729200000 (GMT: 2021-01-04 3:00:00.000)
end: 1610074799999 (GMT: 2021-01-08 2:59:99.999)
How can i get Max values for "local" days?
Above query gives results grouped by GMT time, not local; It rounds to 2021-01-04 00:00:00 GMT to 2021-01-08 00:00:00.
I tried to add TZ hours to TEMPORARY TABLE:
CREATE TEMPORARY TABLE TBASE
SELECT
FLOOR(Timestamp / (86400 * 1000)) * 86400 + 3*3600 as ts,
value
FROM mytable
WHERE Timestamp BETWEEN [GmtTimestampStart] and [GmtTimestampEnd]
but i gives me wrong results because, for example, if MAX value for local date 2021-01-04 is at 23:00:00 in GMT it is 2021-01-05 02:00:00 so i'm missing that value
Excuse my english, but i tried to explain as much as possible.
Related
I need to write SQL query to output the Maximum number and Minimum number of movies produces by diffrent actors and actresses between year 1991 and 2001
query written . When I tried this, I got error enter image description here
The expected result is to output the maximum numer each actor or atress produces within that year range
The result should look like this
When I tried this, I got error what i tried
The expected result is to output the maximum number each actor and atress produces within that year range
The result should look like this
Data type Description
DATE A date. Format: YYYY-MM-DD.
DATETIME(fsp) A date and time combination. Format: YYYY-MM-DD hh:mm:ss.. Adding DEFAULT and ON UPDATE in the column definition to get automatic initialization and updating to the current date and time
TIMESTAMP(fsp) A hh:mm:ss. The supported range is from '1970-01-01 00:00:01' UTC to '2038-01-09 03:14:07' UTC. Automatic initialization and updating to the current date and time can be specified using DEFAULT CURRENT_TIMESTAMP and ON UPDATE CURRENT_TIMESTAMP in the column definition
TIME(fsp) A time. Format: hh:mm:ss. The supported range is from '-838:59:59' to '838:59:59'
YEAR A year in four-digit format. Values four-digit format: 1901 to 2155, and 0000.
MySQL 8.0 does not support year in two-digit format.
Oracle dates contain both date and time so if you're looking to test years only you need to extract that part of the date
ALTER SESSION SET NLS_DATE_FORMAT = 'DD-MON-YYYY HH24:MI:SS';
CREATE TABLE t1 (datestamp) AS
SELECT DATE '2022-10-17' FROM DUAL UNION ALL
SELECT DATE '2022-12-03' FROM DUAL UNION ALL
SELECT DATE '2021-04-03' FROM DUAL UNION ALL
SELECT DATE '2021-05-10' FROM DUAL;
select * from t1
where extract(YEAR from datestamp) = 2021
DATESTAMP
03-APR-2021 00:00:00
10-MAY-2021 00:00:00
select * from t1
where extract(MONTH from datestamp) = 04
DATESTAMP
03-APR-2021 00:00:00
select * from t1
where extract(DAY from datestamp) = 10
DATESTAMP
10-MAY-2021 00:00:00
If you don't want to use the extract method you can do the following. Note since dates contain time you need the +1 to include any dates on 2021-04-30 that fall between the times 00:00:00 and 23:59:59
select * from t1
WHERE datestamp >= DATE'2021-04-01' AND
datestamp <DATE'2021-04-30' +1
DATESTAMP
03-APR-2021 00:00:00
Whenever you have a question, please post a little sample data (CREATE TABLE and INSERT statements for all tables involved, relevant columns only) so the people who want to help you can recreate the problem and test their ideas. Also post the exact results you want from that data, and explain why you want those results from that data. Lastly, don't post any images as they can't be cut and pasted.
I have a network created MYSQL table with following fields:
IP_SRC, IP_DST, BYTES_IN, BYTES_OUT, START_TIME, STOP_TIME
1.1.1.1 8.8.8.8 1080 540 1580684018 1580684100
8.8.4.4 1.1.1.1 2000 4000 1580597618 1580597800
The TIME values are epoch time ticks and each record is basically one TCP session.
I would like formulate a query (or procedure) to return a table with the following fields:
DayOfMonth, TotalOutBytes
1 12345
2 83747
3 2389
where DayOfMonth is the last 14 days or a range of last "n" days (or to keep the focus on the main problem assume the values are 1, 2, 3 of Feb 2020). The challenge is to grab all rows from the network table where STOP_TIME falls within the timeticks for DayOfMonth value for that row and sum up the BYTES_OUT to report as TotalOutBytes for that date.
I'm afraid I'm somewhat new to MYSQL and hopelessly lost.
Consider:
select
day(from_unixtime(stop_time)) dayOfMonth,
sum(bytes_out) TotalOutBytes
from mytable
where stop_time >= unix_timestamp(current_date - interval 14 day)
group by day(from_unixtime(stop_time))
Rationale:
the where clause uses unix_timestamp() to generate the unix timstamp corresponding to 14 days before the current date, which we can use to filter the table
from_unixtime() turns an epoch timestam pto a datetime value, then day() gives you the day number for that point in time
you can then aggregate with that value and sum bytes_out per group
This question already has answers here:
Can MySQL convert a stored UTC time to local timezone?
(6 answers)
Closed 6 years ago.
How do I SELECT all records from yesterday, from Midnight to Midnight, for a specific location's timezone (timestamps all in UTC and locations have named timezones - e.g. America/Vancouver)?
Please note that I'm not asking how to convert a date and time to another timezone. What I'm asking is how [best] to localize a date range comparison.
Assuming the timezone for your MySQL database session is UTC (i.e. time_zone='+00:00')...
And assuming that you want to use the same timezone for all of the rows (i.e. not different timezones based on contents of the row ...
Take the value for "midnight" in the user's specified timezone, and convert that to UTC. e.g. 2016-04-16 00:00 CST6CDT (i.e. America/Chicago) converts to 2016-04-16 05:00 UTC.
Assuming that your table column is named utc_timestamp_col, and is datatype TIMESTAMP, your query would look look something like this:
SELECT ...
FROM mytable t
WHERE t.utc_timestamp_col >= '2016-04-16 05:00' + INTERVAL -1 DAY
AND t.utc_timestamp_col < '2016-04-16 05:00' + INTERVAL 0 DAY
If you have populated the MySQL timezone tables, you can make use of the MySQL support for named timezones. http://dev.mysql.com/doc/refman/5.7/en/time-zone-support.html
You can build an expression that gets you previous "midnight" in a named timezone.
Below is a demonstration query:
verify session time_zone is UTC
return current date and time in UTC
convert to local time zone CST6CDT
truncate to midnight
convert back to UTC
e.g.
SELECT ##session.time_zone AS `time_zone`
, NOW() AS `now_utc`
, CONVERT_TZ(NOW(),'UTC','CST6CDT') AS `now_CST6CDT`
, DATE(CONVERT_TZ(NOW(),'UTC','CST6CDT')) AS `midnight_CST6CDT`
, CONVERT_TZ(DATE(CONVERT_TZ(NOW(),'UTC','CST6CDT')),'CST6CDT','UTC')
AS midnight_CST6CDT_utc
Returns:
time_zone now_utc now_CST6CDT midnight_CST6CDT midnight_CST6CDT_utc
--------- ------------------- ------------------- ---------------- --------------------
UTC 2016-04-17 01:53:31 2016-04-16 20:53:31 2016-04-16 2016-04-16 05:00:00
Demonstrating the same thing, using time zone named 'America/Chicago'
SELECT ##session.time_zone AS `time_zone`
, NOW() AS now_utc
, CONVERT_TZ(NOW(),'UTC','America/Chicago') AS `now_America/Chicago`
, DATE(CONVERT_TZ(NOW(),'UTC','America/Chicago')) AS `midnight_America/Chicago`
, CONVERT_TZ(DATE(CONVERT_TZ(NOW(),'UTC','America/Chicago')),'America/Chicago','UTC')
AS `midnight_America/Chicago_utc`
returns the same result:
time_zone now_utc now_America/Chicago midnight_America/Chicago midnight_America/Chicago_utc
--------- ------------------- ------------------- ------------------------ ----------------------------
UTC 2016-04-17 01:57:19 2016-04-16 20:57:19 2016-04-16 2016-04-16 05:00:00
FOLLOWUP
If I needed to do that kind of timezone conversion in the query, I would use an inline view to return the value from that fairly complicated expression, to make the outer statement simpler. For example, the inline view aliased as "d" returns the value as column named "dt", which can be referenced in the outer query.
Since that inline view returns exactly one row, we can use a JOIN to mytable without duplicating any rows. We can move the predicates from the WHERE clause to an ON clause. e.g.
SELECT ...
FROM ( SELECT CONVERT_TZ(DATE(CONVERT_TZ(NOW(),'UTC','CST6CDT')),'CST6CDT','UTC') AS dt
) d
JOIN mytable t
ON t.utc_timestamp_col >= d.dt + INTERVAL -1 DAY
AND t.utc_timestamp_col < d.dt + INTERVAL 0 DAY
I'm trying to group data on 1 day interval using GROUP BY DIV as mentioned on this post:
Grouping into interval of 5 minutes within a time range
It looks fine on first glance.
But I notice inconsistency when comparing queries on 2 different date interval (but intersected).
First I use date range from Feb 01 00:00 to Feb 26 00:00,
second I use date range from Feb 20 00:00 to Feb 26 00:00
The values on Feb 20 are different between those 2 queries. But the rest (21 - 25) are matched.
Any idea what's going on & how to fix it?
Update:
Here's the stored procedure to generate dummy data on February on each minute:
DELIMITER $$
CREATE DEFINER=`root`#`127.0.0.1` PROCEDURE `testdata`()
BEGIN
DECLARE gap int;
DECLARE x bigint;
SET gap = 60000;
SET x = 1454265000000;
CREATE TABLE IF NOT EXISTS testdata (
timestamp bigint(20) default NULL,
value int(20) default NULL
)
ENGINE=MyISAM DEFAULT CHARSET=utf8;
WHILE x <= 1456770599000 DO
INSERT INTO testdata(timestamp, value) VALUES (x, FLOOR(RAND() * (270 + 1)) + 30);
SET x = x + gap;
END WHILE;
select x;
END
And here're 2 queries to compare 2 interval:
select from_unixtime(timestamp / 1000), count(value) from testdata where timestamp >= 1454265000000 and timestamp <= 1456770599000 group by timestamp div 86400000;
select from_unixtime(timestamp / 1000), count(value) from testdata where timestamp >= 1455906600000 and timestamp <= 1456770599000 group by timestamp div 86400000;
First query at 2016-02-20 return 1440. Second query at 2016-02-20 return 2 records at 2016-02-20 00:00:00 = 330 and at 2016-02-20 05:30:00 = 1440.
The duplication is because your server's timezone isn't the same as UTC. Unix timestamps are based on the time in UTC, so timestamp DIV 86400000 is grouping by UTC dates. But FROM_UNIXTIME() will return a time in the database's timezone. Since you're selecting FROM_UNIXTIME(timestamp/1000), you're selecting an arbitrary row within the group, and the date of that in the server's timezone may be different from its UTC date. As a result, two different UTC date groups will show the same timestamp date.
What you should do is select the date in UTC, so you're displaying the same date that you're grouping by.
SELECT FROM_UNIXTIME((TIMESTAMP DIV 86400000) * 86400), COUNT(*)
FROM testdata
WHERE timestamp BETWEEN 1455906600000 and 1456770599000
GROUP BY TIMESTAMP DIV 86400000
I have the following data:
Table Name: NODE
NID | Name | Date
001 | One | 1252587739
Date is a unix timestamp. I need a query, whereby I can select only the nodes who's "Date" is older than 24 hours. Something like this:
SELECT * FROM NODE WHERE Date < NOW() - SOMETHING
Anybody know how to do this?
Does the NOW() - SOMETHING part take into account that the date is stored as a unix timestamp?
Unix timestamp is in seconds. This works with MySQL:
SELECT * FROM NODE WHERE Date < (UNIX_TIMESTAMP(NOW()) - 24*60*60)
where datediff(hh, date, now()) < 24
Going by the definition of "Unix Timestamp = number of seconds from Jan 1, 1970", and based on MS SQL Server (7.0 and up compatible):
SELECT *
from NODE
where datediff(ss, dateadd(ss, Date, 'Jan 1, 1970'), getdate()) < 86400
The innermost parenthesis adds the number of seconds to Jan 1 1970 to get the row's datetime in SQL server format, the outer parenthesis gets the number of seconds between that date and "now", and 86400 is the number of seconds in 24 hours. (But double-check this--I can't debug this just now, and I might have the function paramater order mixed.)