Prune mysql table - averaging records - mysql

I have a MySQL table which stores real-time data from different devices. Data is recorded about every 20 seconds.
The table looks like this:
report_dt device value1 value2
2015-10-16 10:32:15 solar 34.4 67.8
2015-10-16 10:32:15 grid 56.9 23.5
2015-10-16 10:32:35 solar 45.6 34.3
Queries to get the recorded values per device are pretty CPU consuming. I have (report_dt,device) as primary key. Besides, after a few days I do not need the 20 second data anymore, but just the 10 minute average.
So either the same table setup or a table per device would be satisfactory, with this contents:
report_dt device value1
2015-10-16 10:00:00 solar avg(value1 over 10 mins)
2015-10-16 10:10:10 grid avg(value1 over 10 mins)
2015-10-16 10:20:20 solar avg(value1 over 10 mins)
The idea is to run the pruning action once every 24 hours.
The reason is that data containing averages must be retrieved fast. And because the data in a 20 second interval is extremely fluctuating, I must use averaging or the data is difficult to assess.
I could do this in an application. Retrieve the average values for 10 minutes at each 10 minute interval and write it back to a different table or different tables, one for each device.
But would something also be possible with a database function?

You can do this by Events in mysql. Suppose i have 2 tables in mysql. Table_1 store the data continiously per second or miunte as your application generate. Table_2 is your main table which contains the 10 min avg data. Now you need to create a stored procedure which calls once in 10 minutes (this can be done by Events or CRON in linux ). So whats contains the stored procedure ? These steps perform in Stored Procedure
BEGIN
SELECT avg(value) of last 10 min from Table_1
INSERT into Table_2 the avg record.
DELETE avg(value) of last 10 min from Table_1
END
Caution- I am not developing mysql code, i am just show the approach

Related

mysql: taking many recurring averages with different intervals using events in

i have around 500 tables and each table is for a specific device. every table is getting full with data that comes every 3 minutes . now i need to store averages of different timeframes of one same column(data) in tables .
timeframes are 10 minutes, 1 hour , 12 hour , 24 hour ,1 month.
is mysql events good for this purpose? if yes how many of them is required , i mean should i create a table called 10 minute averages with 500 columns with one event or divide them into 5 tables with 100 column and with 5 events? in second case there would be 25 events
if not what is the better solution? as you see these are datacentric jobs.
thanks a lot.

How can I add a row for Unpivoted data that is at the hourly level and I want to display it at the 15 min interval

I am very new to SSIS so the answer to this might be very basic.
I have data that is coming in at an hourly level, meaning that a single record would have 25 columns (hour 1 - 25 with the 25th being null for DST) I have unpivoted the data to show a single column with the hour. I now need to display this data at the 15 min interval.
My plan is to duplicate the row of data for hour 1, say it is 8 and divide by 4 and now each 15 min interval would be 2
What can I do to accomplish this goal?
I have no clue where to start, looking for ideas.
I strongly suspect that this is a terrible approach to whatever it is you really want to do, but since you haven't asked about what you really want to do, I'll treat your question as academic.
If you have to do this in the dataflow, you could write a script transformation with a loop that creates 4 output rows for every input row and does the division.
Personally I would do this in a stored procedure that gets called after the data is initially loaded into a staging table on the destination server:
SELECT from the staging table, cross joined to a CTE with 4 rows, and dividing the value by 4.

MySQL Limit amount of records returned between 2 dates

I'm afraid I with this situation:
I have a MySQL table with just 3 columns: ID, CREATED, TOTAL_VALUE.
A new TOTAL_VALUE is recorded roughly every 60 seconds, so about 1440 times a day.
I am using PHP to generate some CanvasJS code that plots the MySQL records into line graph - this so that I can see how TOTAL_VALUE changes over time.
it works great for displaying 1 day worth of data, but when doing 1 week(7*1440=10080 plot points) things get really slow.
And a date range of for example 1-JAN-2016 and 1-SEP-2016 just leads to time outs in the PHP script.
How can I write some MySQL that still selects records between a date range but limit the rows returned to ie max 1000 rows?
I need to optimize this by limiting the number of data points that need to be plotted.
Can MySQL do some clever stuff where it decides to skip 1 every so many rows and return 1000 averaged values - this so that my line graph would by approximation still be correct- but using fewer data points?

get average interarrival time for service requests by timestamp

I have partly the following MySQL schema
ServiceRequests
----------
id int
RequestDateTime datetime
This is what a typical collection of records might look like.
1 | 2009-10-11 14:34:22
2 | 2009-10-11 14:34:56
3 | 2009-10-11 14:35:01
In this case the average request time is (34+5)/2 = 19.5 seconds, being
14:34:22 ---> (34 seconds) ----> 14:34:56 ------> (5 seconds) -----> 14:35:01
Basically I need to work out the difference in time between consecutive records, sum that up and divide by the number of records.
The closest thing I can think of is to convert the timestamp to epoch time and start there. I can add a field to the table to precalculate the epoch time if necessary.
How do I determine 19.5 using a sql statement(s)?
You don't really need to know the time difference of each record to get the average. You have x data points ranging from some point t0 to t1. Notice that the the last time - first time is also 39 sec. (max-min)/(count-1) should work for you
select max(RequestDateTime)-min(RequestDateTime) / (count(id)-1) from ServiceRequests;
Note: This will not work if the table is empty, due to a divide by zero.
Note2: Different databases handle subtraction of dates differently so you may need to turn that difference into seconds.
Hint: maybe using TIMEDIFF(expr1,expr2) and/or TIME_TO_SEC(expr3)

Returning rows of aggregate results from single SQL query

I have a MySQL table containing a column to store time and another to store a value associated with that time.
time | value
------------
1 | 0.5
3 | 1.0
4 | 1.5
.... | .....
The events are not periodic, i.e., the time values do not increment by fix interval.
As there are large number of rows (> 100000), for the purpose of showing the values in a graph I would like to be able to aggregate (mean) the values for an interval of fixed size over the entire length of time for which the data is available. So basically the output should consist of pairs of interval and mean values.
Currently, I am splitting the total time interval into fixed chunks of time, executing individual aggregate queries for that interval and collecting the results in application code (Java). Is there a way to do all of these steps in SQL. Also, I am currently using MySQL but am open to other databases that might support an efficient solution.
SELECT FLOOR(time / x) AS Inter, AVG(value) AS Mean
FROM `table`
GROUP BY Inter;
Where x is your interval of fixed size.
I've usually solved this through a "period" table, with all the valid times in it, and an association with the period on which I report.
For instance:
time day week month year
1 1 1 1 2001
2 1 1 1 2001
....
999 7 52 12 2010
You can then join your time to the "period" table time, and use AVG.