I am running a mysql - 10.1.39-MariaDB - mariadb.org binary- database.
I am having the following table:
| id | date | api_endpoint | ticker | open | high | low | close | volume |
|------|---------------------|--------------|--------|-----------|-----------|-----------|-----------|-----------|
| 18 | 2019-08-07 00:00:00 | daily | AAPL | 195.41000 | 199.56000 | 193.82000 | 199.04000 | 33364400 |
| 19 | 2019-08-06 00:00:00 | daily | AAPL | 196.31000 | 198.07000 | 194.04000 | 197.00000 | 35824800 |
| 20 | 2019-08-05 00:00:00 | daily | AAPL | 197.99000 | 198.65000 | 192.58000 | 193.34000 | 52393000 |
| 21 | 2019-08-02 00:00:00 | daily | AAPL | 205.53000 | 206.43000 | 201.62470 | 204.02000 | 40862100 |
| 44 | 2019-08-01 00:00:00 | monthly | AAPL | 213.90000 | 218.03000 | 206.74000 | 208.43000 | 54017900 |
| 5273 | 1999-09-07 00:00:00 | monthly | AAPL | 73.75000 | 77.93800 | 73.50000 | 76.37500 | 246198400 |
I am calculating returns using mysql:
SELECT *
,(CLOSE - (SELECT (t2.close)
FROM prices t2
WHERE t2.date < t1.date
ORDER BY t2.date DESC
LIMIT 1 ) ) / (SELECT (t2.close)
FROM prices t2
WHERE t2.date < t1.date
ORDER BY t2.date DESC
LIMIT 1 ) AS daily_returns
FROM prices
The above query adds a column daily_returns to my table.
I would like to get the top 5 highest daily_returns. I tried to use ORDER BY, however, this does not work with a calculated column.
Any suggestions how to get the top 5 highest daily_returns?
Update: MySQL 8
SELECT
prices.*,
prices.close - LAG(prices.close) OVER w AS daily_return
FROM prices
WHERE api_endpoint = 'daily'
WINDOW w AS (ORDER BY prices.`date` ASC)
ORDER BY daily_return DESC
LIMIT 5;
MySQL 5.7 & Lower
Use MySQL variable to store close value of last day. Compare it with close value to the current row to do the calculation.
SELECT
*
FROM (
SELECT
prices.*,
(`close` - #old_close) / #old_close AS daily_return, -- Use #old_case, currently it has value of old row, next column will set it to current close value.
#old_close:= `close` -- Set #old_close to close value of this row, so it can be used in next row
FROM prices,
(SELECT #old_close:= 0 as o_c) AS t -- Initialize old_close as 0
WHERE api_endpoint = 'daily'
ORDER BY `date` ASC -- return is calculated based on last day close, so keep it sorted based on ascending order of date
) AS tt
ORDER BY daily_return DESC
LIMIT 5;
Reference: How to get diff between two consecutive rows
Related
This is the table I am working with:
+---------------------+-----------
| Field | Type |
+---------------------+--------------+
| ID | binary(17) |
| MiscSensor_ID | binary(17) |
| rawValue | varchar(100) |
| RawValueUnitType_ID | int |
| timestamp | timestamp |
+---------------------+--------------+
Now my goal is to implement an event which deletes all entries older than a month BUT for each week I want to leave one entry per MiscSensor_ID (the one with the lowest rawValue).
I am this far:
CREATE EVENT delete_old_miscsensordatahistory
ON SCHEDULE EVERY 1 DAY
STARTS CURRENT_DATE + INTERVAL 1 DAY
DO
DELETE
FROM history
WHERE TIMESTAMPDIFF(DAY, timestamp,NOW()) > 31;
I need to do something like: delete if (value > minvalue) and group it in by MiscSensor_ID and 7 day periods but i am stuck right now on how to do that.
Any help would be much appreciated.
You can try using the ROW_NUMBER window function to match the rows which you don't want to delete. Records having row number equal to 1 will be those rows with the minimum "rawValue" for each combination of (week, sensorId).
WITH cte AS (
SELECT *, ROW_NUMBER() OVER(
PARTITION BY MiscSensorId, WEEK(timestamp)
ORDER BY rawValue ) AS rn
FROM history
WHERE TIMESTAMPDIFF(DAY, timestamp,NOW()) > 31
)
DELETE
FROM history
INNER JOIN cte
ON history.ID = cte.ID
WHERE rn > 1;
This is how i implemented the event right now:
CREATE EVENT delete_old_miscsensordatahistory
ON SCHEDULE EVERY 1 DAY
STARTS CURRENT_DATE + INTERVAL 1 DAY
DO
WITH cte AS (
SELECT *, ROW_NUMBER() OVER(
PARTITION BY MiscSensor_ID, WEEK(timestamp)
ORDER BY CAST(rawValue AS SIGNED) ) AS rn
FROM MiscSensorDataHistory
WHERE TIMESTAMPDIFF(DAY, timestamp,NOW()) > 31
)
DELETE MiscSensorDataHistory
FROM MiscSensorDataHistory
INNER JOIN cte
ON cte.ID = MiscSensorDataHistory.ID
WHERE rn > 1
Testing my method I found out that there are still entries with the same MiscSensor_ID and less than 7 days apart:
| 0x3939333133303037343939353436393032 | 0x3439303031303031303730303030303535 | 554 | 30 | 2022-02-17 23:09:21 |
| 0x3939333133303037343939313631333039 | 0x3439303031303031303730303030303535 | 554 | 30 | 2022-02-06 16:52:48 |
| 0x3939333133303037343938383835353239 | 0x3439303031303031303730303030303535 | 553 | 30 | 2022-01-30 08:21:55 |
| 0x3939333133303037343938383639333436 | 0x3439303031303031303730303030303535 | 554 | 30 | 2022-01-29 22:48:06 |
| 0x3939333133303037343937303734353537 | 0x3439303031303031303730303030303535 | 444 | 30 | 2021-12-26 06:12:07 |
| 0x3939333133303037343937303530363738 | 0x3439303031303031303730303030303535 | 446 | 30 | 2021-12-25 21:53:03 |
| 0x3939333133303037343936333034343238 | 0x3439303031303031303730303030303535 | 0 | 30 | 2021-12-14 13:08:04 |
| 0x3939333133303037343935393934303832 | 0x3439303031303031303730303030303535 | 415 | 30 | 2021-12-08 12:56:43
Any suggestions would be much appreciated.
I am running a mysql - 10.1.39-MariaDB - mariadb.org binary- database.
I am having the following table:
| id | date | product_name | close |
|----|---------------------|--------------|-------|
| 1 | 2019-08-07 00:00:00 | Product 1 | 806 |
| 2 | 2019-08-06 00:00:00 | Product 1 | 982 |
| 3 | 2019-08-05 00:00:00 | Product 1 | 64 |
| 4 | 2019-08-07 00:00:00 | Product 2 | 874 |
| 5 | 2019-08-06 00:00:00 | Product 2 | 739 |
| 6 | 2019-08-05 00:00:00 | Product 2 | 555 |
| 7 | 2019-08-07 00:00:00 | Product 3 | 762 |
| 8 | 2019-08-06 00:00:00 | Product 3 | 955 |
| 9 | 2019-08-05 00:00:00 | Product 3 | 573 |
I want to get the following output:
| id | date | product_name | close | daily_return |
|----|---------------------|--------------|-------|--------------|
| 4 | 2019-08-07 00:00:00 | Product 2 | 874 | 0,182679296 |
| 1 | 2019-08-07 00:00:00 | Product 1 | 806 | -0,179226069 |
Basically I want ot get the TOP 2 products with the highest return. Whereas return is calculated by (close_currentDay - close_previousDay)/close_previousDay for each product.
I tried the following:
SELECT
*,
(
CLOSE -(
SELECT
(t2.close)
FROM
prices t2
WHERE
t2.date < t1.date
ORDER BY
t2.date
DESC
LIMIT 1
)
) /(
SELECT
(t2.close)
FROM
prices t2
WHERE
t2.date < t1.date
ORDER BY
t2.date
DESC
LIMIT 1
) AS daily_return
FROM
prices t1
WHERE DATE >= DATE(NOW()) - INTERVAL 1 DAY
Which gives me the return for each product_name.
How to get the last product_name and sort this by the highest daily_return?
Problem Statement: Find the top 2 products with the highest returns on the latest date i.e. max date in the table.
Solution:
If you have an index on date field, it would be super fast.
Scans table only once and also uses date filter(index would allow MySQL to only process rows of given date range only.
A user-defined variable #old_close is used to find the return. Note here we need sorted data based on product and date.
SELECT *
FROM (
SELECT
prices.*,
CAST((`close` - #old_close) / #old_close AS DECIMAL(20, 10)) AS daily_return, -- Use #old_case, currently it has value of old row, next column will set it to current close value.
#old_close:= `close` -- Set #old_close to close value of this row, so it can be used in next row
FROM prices
INNER JOIN (
SELECT
DATE(MAX(`date`)) - INTERVAL 1 DAY AS date_from, -- if you're not sure whether you have date before latest date or not, can keep date before 1/2/3 day.
#old_close:= 0 as o_c
FROM prices
) AS t ON prices.date >= t.date_from
ORDER BY product_name, `date` ASC
) AS tt
ORDER BY `date` DESC, daily_return DESC
LIMIT 2;
Another version which doesn't depend on this date parameter.
SELECT *
FROM (
SELECT
prices.*,
CAST((`close` - #old_close) / #old_close AS DECIMAL(20, 10)) AS daily_return, -- Use #old_case, currently it has value of old row, next column will set it to current close value.
#old_close:= `close` -- Set #old_close to close value of this row, so it can be used in next row
FROM prices,
(SELECT #old_close:= 0 as o_c) AS t
ORDER BY product_name, `date` ASC
) AS tt
ORDER BY `date` DESC, daily_return DESC
LIMIT 2
You can do it with a self join:
select
p.*,
cast((p.close - pp.close) / pp.close as decimal(20, 10)) as daily_return
from prices p left join prices pp
on p.product_name = pp.product_name
and pp.date = date_add(p.date, interval -1 day)
order by p.date desc, daily_return desc, p.product_name
limit 2
See the demo.
Results:
| id | date | product_name | close | daily_return |
| --- | ------------------- | ------------ | ----- | ------------ |
| 4 | 2019-08-07 00:00:00 | Product 2 | 874 | 0.182679296 |
| 1 | 2019-08-07 00:00:00 | Product 1 | 806 | -0.179226069 |
I have a table with 'ON' and 'OFF' values in column activity and another column datetime.
id(AUTOINCREMENT) id_device activity datetime
1 a ON 2017-05-26 22:00:00
2 b ON 2017-05-26 05:00:00
3 a OFF 2017-05-27 04:00:00
4 b OFF 2017-05-26 08:00:00
5 a ON 2017-05-28 12:00:00
6 a OFF 2017-05-28 15:00:00
I need to get total ON time by day
day id_device total_minutes_on
2017-05-26 a 120
2017-05-26 b 180
2017-05-27 a 240
2017-05-27 b 0
2017-05-28 a 180
2017-05-28 b 0
i have searched and tried answers for another posts, i tried TimeDifference and i get correct total time.
I don't find the way to get total time grouped by date
i appreciate your help
I'm not posting this as a definite answer rather it's an experiment for me and hopefully you'll find is useful in your case. Also I would like to mention that the MySQL database version I'm working with is quite old so the method I'm using is also very manual to say the least.
First of all lets extract your expected output:
The date value in day need to be repeated twice fro each of id_device a and b.
Minutes are calculated based on the activity; if activity is 'ON' until tomorrow, it needs to be calculated until the day end at 24:00:00 while the next day will calculate minutes until the activity is OFF.
What I come up with is this:
Creating condition (1):
SELECT * FROM
(SELECT DATE(datetime) dtt FROM mytable GROUP BY DATE(datetime)) a,
(SELECT id_device FROM mytable GROUP BY id_device) b
ORDER BY dtt,id_device;
The query above will return the following result:
+------------+-----------+
| dtt | id_device |
+------------+-----------+
| 2017-05-26 | a |
| 2017-05-26 | b |
| 2017-05-27 | a |
| 2017-05-27 | b |
| 2017-05-28 | a |
| 2017-05-28 | b |
+------------+-----------+
*Above will only work with all the dates you have in the table. If you want all date regardless if there's activity or not, I suggest you create a calendar table (refer: Generating a series of dates).
So this become the base query. Then I've added an outer query to left join the query above with the original data table:
SELECT v.*,
GROUP_CONCAT(w.activity ORDER BY w.datetime SEPARATOR ' ') activity,
GROUP_CONCAT(TIME_TO_SEC(TIME(w.datetime)) ORDER BY w.datetime SEPARATOR ' ') tr
FROM
-- this was the first query
(SELECT * FROM
(SELECT DATE(datetime) dtt FROM mytable GROUP BY DATE(datetime)) a,
(SELECT id_device FROM mytable GROUP BY id_device) b
ORDER BY a.dtt,b.id_device) v
--
LEFT JOIN
mytable w
ON v.dtt=DATE(w.datetime) AND v.id_device=w.id_device
GROUP BY DATE(v.dtt),v.id_device
What's new in the query is the addition of GROUP_CONCAT operation on both activity and time value extracted from datetime column which is converted into seconds value. You notice that in both of the GROUP_CONCAT there's a similar ORDER BY condition which is important in order to get the exact corresponding value.
The query above will return the following result:
+------------+-----------+----------+-------------+
| dtt | id_device | activity | tr |
+------------+-----------+----------+-------------+
| 2017-05-26 | a | ON | 79200 |
| 2017-05-26 | b | ON OFF | 18000 28800 |
| 2017-05-27 | a | OFF | 14400 |
| 2017-05-27 | b | (NULL) | (NULL) |
| 2017-05-28 | a | ON OFF | 43200 54000 |
| 2017-05-28 | b | (NULL) | (NULL) |
+------------+-----------+----------+-------------+
From here, I've added another query outside to calculate how many minutes and attempt to get the expected result:
SELECT dtt,id_device,
CASE
WHEN SUBSTRING_INDEX(activity,' ',1)='ON' AND SUBSTRING_INDEX(activity,' ',-1)='OFF'
THEN (SUBSTRING_INDEX(tr,' ',-1)-SUBSTRING_INDEX(tr,' ',1))/60
WHEN activity='ON' THEN 1440-(tr/60)
WHEN activity='OFF' THEN tr/60
WHEN activity IS NULL AND tr IS NULL THEN 0
END AS 'total_minutes_on'
FROM
-- from the last query
(SELECT v.*,
GROUP_CONCAT(w.activity ORDER BY w.datetime SEPARATOR ' ') activity,
GROUP_CONCAT(TIME_TO_SEC(TIME(w.datetime)) ORDER BY w.datetime SEPARATOR ' ') tr
FROM
-- this was the first query
(SELECT * FROM
(SELECT DATE(datetime) dtt FROM mytable GROUP BY DATE(datetime)) a,
(SELECT id_device FROM mytable GROUP BY id_device) b
ORDER BY a.dtt,b.id_device) v
--
LEFT JOIN
mytable w
ON v.dtt=DATE(w.datetime) AND v.id_device=w.id_device
GROUP BY DATE(v.dtt),v.id_device
--
) z
The last part I do is if the activity value have both ON and OFF on the same day then (OFF-ON)/60secs=total minutes. If activity value is only ON then minutes value for '24:00:00' > 24 hr*60 min= 1440-(ON/60secs)= total minutes, and if activity only OFF, I just convert seconds to minutes because the day starts at 00:00:00 anyhow.
+------------+-----------+------------------+
| dtt | id_device | total_minutes_on |
+------------+-----------+------------------+
| 2017-05-26 | a | 120 |
| 2017-05-26 | b | 180 |
| 2017-05-27 | a | 240 |
| 2017-05-27 | b | 0 |
| 2017-05-28 | a | 180 |
| 2017-05-28 | b | 0 |
+------------+-----------+------------------+
Hopefully this will give you some ideas. ;)
Using table below, How would get a column for 5 period moving average, 10 period moving average, 5 period exponential moving average.
+--------+------------+
| price | data_date |
+--------+------------+
| 122.29 | 2009-10-08 |
| 122.78 | 2009-10-07 |
| 121.35 | 2009-10-06 |
| 119.75 | 2009-10-05 |
| 119.02 | 2009-10-02 |
| 117.90 | 2009-10-01 |
| 119.61 | 2009-09-30 |
| 118.81 | 2009-09-29 |
| 119.33 | 2009-09-28 |
| 121.08 | 2009-09-25 |
+--------+------------+
The 5-row moving average in your example won't work. The LIMIT operator applies to the return set, not the rows being considered for the aggregates, so changing it makes no difference to the aggregate values.
SELECT AVG(a.price) FROM (SELECT price FROM t1 WHERE data_date <= ? ORDER BY data_date DESC LIMIT 5) AS a;
Replace ? with the date whose MA you need.
SELECT t1.data_date,
( SELECT SUM(t2.price) / COUNT(t2.price) as MA5 FROM mytable AS t2 WHERE DATEDIFF(t1.data_date, t2.data_date) BETWEEN 0 AND 6 )
FROM mytable AS t1 ORDER BY t1.data_date;
Change 6 to 13 for 10-day MA
I want to join two tables in a special way, first table is devices which has a list of devices.
The second table is datalog which is where abit of data is stored for everytime a device in devices gets polled.
Devices Table:
+----------+------------+----------------------------+---------------------+
| deviceId | deviceName | deviceDescription | timeCreated |
+----------+------------+----------------------------+---------------------+
| 1 | System 1 | Main System in Server Room | 2010-01-01 00:00:00 |
| 2 | System 2 | Outdoor System | 2010-01-01 00:00:00 |
+----------+------------+----------------------------+---------------------+
DataLog Table:
+----+---------------------+----------+-----------+---------+
| id | time_stamp | DeviceId | FuelLevel | Voltage |
+----+---------------------+----------+-----------+---------+
| 1 | 2010-01-01 00:00:00 | 1 | 60 | 220 |
| 2 | 2010-01-01 00:00:00 | 2 | 20 | 221 |
| 3 | 2010-01-02 00:00:00 | 1 | 100 | 219 |
| 4 | 2010-01-02 00:00:00 | 2 | 100 | 222 |
| 5 | 2010-01-03 00:00:00 | 1 | 80 | 219 |
| 6 | 2010-01-03 00:00:00 | 2 | 99 | 220 |
+----+---------------------+----------+-----------+---------+
Currently I am getting the latest data for each device using a query on the DataLog table with:
Where DeviceId = 1 ORDER BY timestamp DESC LIMIT 1
What I would like is one query to return a list of all devices, with the columns joined with the latest data for each device like this:
+----------+------------+----------------------------+---------------------+-----------+---------+
| deviceId | deviceName | deviceDescription | time_stamp |FuelLevel | Voltage |
+----------+------------+----------------------------+---------------------+-----------+---------+
| 1 | System 1 | Main System in Server Room | 2010-01-03 00:00:00 | 80 | 219 |
| 2 | System 2 | Outdoor System | 2010-01-03 00:00:00 | 99 | 220 |
+----------+------------+----------------------------+---------------------+-----------+---------+
You can't do the "limit 1" at the outer level, you loose what you are looking for... ALL devices last entry. Use a pre-query for the last ID of each device, then join back...
select
Devices.*,
DataLog.Time_Stamp,
DataLog.FuelLevel,
DataLog.Voltage
from
( select DeviceID,
max( ID ) LastActionID
from
DataLog
group by
1 ) LastInstance
join DataLog
on LastInstance.LastActionID = DataLog.ID
join Devices
on LastInstance.DeviceID = Devices.DeviceID
order by
Devices.DeviceName
Per your last comment, I would actually change to something like...
Update your device table with a "LastLogID". Then, via a trigger an insert into your DataLog table, update the Device table immediately with that new ID... This way, you never need to pre-query the data log directly.. You'll already HAVE the last ID and run from that directly to the data log joined by that ID.
I know it's horrible, not elegant and time consuming, but this query works:
SELECT deviceId,deviceName,deviceDescription,
(SELECT time_stamp FROM datalog
WHERE datalog.DeviceId=devices.deviceId
ORDER BY time_stamp DESC LIMIT 0,1) time_stamp,
(SELECT FuelLevel FROM datalog
WHERE datalog.DeviceId=devices.deviceId
ORDER BY time_stamp DESC LIMIT 0,1) FuelLevel,
(SELECT Voltage FROM datalog
WHERE datalog.DeviceId=devices.deviceId
ORDER BY time_stamp DESC LIMIT 0,1) Voltage
FROM devices
I tried to have a single subquery retrieving multiple columns, but MySql complains because it wants only one column.
try
by the way if u want only latest row then u can search it by auto increment field (datalog_table.id)
SELECT dvc.deviceId,dvc.deviceName,dvc.deviceDescription,
dtl.time_stamp,dtl.FuelLevel,dtl.Voltage
FROM device_table dvc
INNER JOIN datalog_table dtl
ON dtl.DeviceId=dvc.deviceId
ORDER BY dtl.id LIMIT 1
SELECT
d.deviceId, d.deviceName, d.deviceDescription,
dl.time_stamp, dl.FuelLevel, dl.Voltage
FROM Device d, DataLog dl
WHERE d.deviceId=dl.deviceID
ORDER BY time_stamp DESC
LIMIT 1