I've a table of measurements with timestamp. One data each 5 minutes.
I've created a view to extract average/min/max values for every 30minutes. Problem is I found this query very slow (+/- 5 s for total of 13290 rows... which is very few..
Any idea of optimisation ?
My code:
SELECT t_mesures.sonde_id AS sonde_id
,min(t_mesures.timestamp) AS start_period
,max(t_mesures.timestamp) AS end_period
,from_unixtime(
unix_timestamp(min(t_mesures.timestamp))+
floor( (unix_timestamp(max(t_mesures.timestamp))-unix_timestamp(min(t_mesures.timestamp)))/2)
) AS mid_period
,timediff(max(t_mesures.timestamp), min(t_mesures.timestamp)) AS dur_period
,avg(t_mesures.Mesure) AS avg_mesure
,min(t_mesures.Mesure) AS min_mesure
,max(t_mesures.Mesure) AS max_mesure
,count(t_mesures.Mesure) as nb_mesure
FROM t_mesures
GROUP BY t_mesures.sonde_id
,(floor((unix_timestamp(t_mesures.timestamp) / 1800)) * 1800)
As you've already noticed, calculating these things live tends to get pretty heavy.
The best way to make sure you don't have this problem is by simply not calculating it. In addition to storing the 5 minute interval data you should also store the 30 minutes min/max/avg/count data (and/or any other interval you're going to use).
Alternatively, you could also try something like Whisper, RRD or OpenTSDB
Related
We have dates stored as a unix timestamp. To allow a user to search for a certain date - based on his timezone-setting, we used to convert that timestamp inside the query, to make sure a search for "2012-05-03" will not find results of the prior / next day depending on which timezone the user has setup.
i.e. if a date is stored as 2012-05-03 23:00 (UTC) A user with the proper timezone offset searching for 2012-05-04 should find this entry.
This is done like this at the moment:
CONVERT_TZ(FROM_UNIXTIME(`javaTimeStampColumn`/1000),'+00:00','+00:00')
where ofc. the offsets are set depending on the users timezone.
The problem we are facing at the moment: Java successfully stores dates after the year 2038 as a unix-timestamp. The MySQL method from_unixtime however does not support any conversion of values greater than 2147483647 due to it's integer type limitation:
SELECT FROM_UNIXTIME(2147483647); //2038-01-19 04:14:07
SELECT FROM_UNIXTIME(2147483648); //null
The MySQL server itself is 64bit, but ofc. FROM_UNIXTIME would need to accept a long as argument.
I could not find a proper replacement by now, any hints?
We could ofc. load the timestamp as a Long and handle it in the application - But for lazylaoding we need to be able to convert it correctly during the query as well.
A workaround might be to use DATE_ADD, but I'm not sure how it behaves performance-wise:
SELECT DATE_ADD(FROM_UNIXTIME(0), INTERVAL 2147483647 SECOND); //2038-01-19 04:14:07
SELECT DATE_ADD(FROM_UNIXTIME(0), INTERVAL 2147483648 SECOND); //2038-01-19 04:14:08
...
SELECT DATE_ADD(FROM_UNIXTIME(0), INTERVAL 4147483647 SECOND); //2101-06-06 07:47:27
So for now, I'm using
...
CASE
WHEN `javaTimeStampColumn` > 2147483647 THEN
CONVERT_TZ(DATE_ADD(FROM_UNIXTIME(0), INTERVAL `javaTimeStampColumn`/1000 SECOND),'+00:00','+00:00')
ELSE
CONVERT_TZ(FROM_UNIXTIME(`javaTimeStampColumn`/1000), '+00:00','+00:00')
END as ts
FROM table
...
which should minimize the impact on performance if there is any.
I googled a lot for this, and don't see anyone talking about it, so it must be a simple issue, but still it has me stumped.
This performance_schema table - http://dev.mysql.com/doc/refman/5.6/en/events-statements-current-table.htm has timer_start and TIMER_END columns. Accordintg to the documentation " The TIMER_START and TIMER_END values indicate when event timing started and ended" .
One small problem. It's a bigint and not a date. How do I convert it to a date?
I saw one blogger suggest that it's the number of time units since the server was started. In my case statements are supposed to be measured to a nanosecond (10^9). So if I have a timer_start value of 3723676792253626000 that would mean 3723676792 s which would be unlikely since the server uptime is 3723716 s. a simple comparison of the number of digits in these two numbers would lead me to think that the unit of time is really picoseconds (10^12).
so the question is :
1. is timer_start really the number of units from the last restart?
2. if so, why is it in picoseconds when setup_timers indicates nanoseconds?
TIA
Here is a corrected version for MySQL 5.7:
SELECT
DATE_SUB(NOW(), INTERVAL (SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME='UPTIME') - TIMER_START*10e-13 second) AS `start_time`,
ROUND(timer_wait*10E-10, 3) AS `wait in (ms)`,
sql_text,
current_schema
FROM performance_schema.events_statements_history_long;
The fixes are:
get global_status from performance_schema (introduced in MySQL 5.7)
fix wait in (ms) (10E-8 vs. 10E-10) + formatting; example test is the following SQL query: SELECT SLEEP(0.5)
the returned columns
uses the "long" history table (events_statements_history_long)
ok, I was able to partially figure it out. Answer to the first question is yes. Here is a query that converts the timer_start value to a time stamp that a human can recognize:
select
date_sub(now(),INTERVAL (select VARIABLE_VALUE from information_schema.global_status where variable_name='UPTIME')-TIMER_START*10e-13 second) `start_time`
,timer_wait/10E-8 `wait in (ms)`
,timer_wait
,sql_text
,digest_text
from performance_schema.events_statements_history
I would like to filter out old records and present only newer ones. The idea is currentDate minus 2 years. One record looks like this (table Symposium):
Attributes:
ID;Day;Month;Year;Firstname;Lastname;Symposium_title;Speakers;
Records
1;26;10;2012;Markus;Meier;Topic MySQL;5;
3;26;10;2011;Markus;Meier;Topic PHP;5;
6;26;01;2010;Markus;Meier;Topic CSS;5;
Wished output
1;26;10;2012;Markus;Meier;Topic MySQL;5;
3;26;10;2011;Markus;Meier;Topic PHP;5;
I could easily do a small php to filter out older data, but I'm sure there is a way to do it direct with mysql in one command.
some query like this (this is just pseudo code)
SELECT * FROM Symposium WHERE (current_data() - 2 Years);
remark: The attributes day;month;year will not be changed. There are more then 50 tables, that are build up by this attributes. So don't recommend change of the table.
Can someone help me ?
You can use STR_TO_DATE function and DATE_ADD for this task:
syntax:
STR_TO_DATE(string, format)
So in your code:
SELECT * from Symposium
WHERE DATE_ADD(STR_TO_DATE(concat(Day,'-',Month,'-',Year),'%d-%m-%Y'), INTERVAL 2 YEAR) > NOW();
(sadly, I don't have a MySQL at hand right now, so I can't test it, but unless there is some very basic typo in it, it should work)
I have a regular DATETIME row, eg: 2012-06-19 13:56:56
I am running the following to see how much the time difference is:
DATEDIFF(end_time, NOW()) * 24
It returns 48
Edit: How do I get the Minutes/Seconds? I have tried UNIXTIME(field) - UNIXTIME(NOW()) but i cant get beyond it.
Im trying to convert this into 48:00:00 (Or however that timestamp works)
I keep looking up time functions but they have to do with EXTRACT, and Im not sure thats the way to go about it.
Since datediff only ever returns a difference in days, you might as well just use a string operation to do your formating:
SELECT CONCAT(DATEDIFF(end_time, NOW()) * 24), ':00:00')
I set up my MySQL database with the field 'time'
It is not HH:MM in the traditional sense, it is the time an event occurred, so an event with the value of 5:45 occurred with 5 minutes 45 seconds left in a game. 12:25 occurred with 12 minutes and 25 seconds left, etc.
I would like to be able to find out the total time elapsed, so if I have an event that occurred at 12:25 and the next event occurred at 5:45 I want to be able to get the difference, which would be equal to 6:40. Then, I would like to express this as a decimal, in this case 6.67.
Is this possible?
For me this worked:
TIMESTAMPDIFF(MINUTE, T0.created, T0.modified)
Reference: https://dev.mysql.com/doc/refman/5.7/en/date-and-time-functions.html#function_timestampdiff
Just use
TIMEDIFF(fromtime, totime):
SELECT TIMEDIFF("12:25", "5:45");
If you need a decimal, use TIME_TO_SEC()/3600 (it assumes you passed it seconds, but the format you're storing the times in is vague and SQL probably will interpret them as HH:MM - you can fix this with CONCAT("00:",MinuteSecondField) maybe?) - then you can use TIME_TO_SEC()/60, which is more correct)
I needed similar. Two useful functions: TIME_TO_SEC and SUBTIME.
e.g. if your time fields were normal HH:MM times, then
SELECT (TIME_TO_SEC(end_time) - TIME_TO_SEC(start_time))/60 AS `minutes`
In your case, as I understand it, the times are backwards, ie. 12:00<6:00 so your end_time and start_time would need swapping.
If you wanted the output in HH:MM:SS you could do
SELECT SUBTIME(end_time, start_time)
I used UNIX_TIMESTAMP(event1)-UNIX_TIMESTAMP(event2), which gives you seconds between events. Divide it by 60.0 and you will get a decimal as per your requirement. This solution assumes, your columns are date-compatible values. It will work really fast if they are TIMESTAMPs.
UPDATE: This solution only works with timestamp data types, not time datatypes as in original question.