The general question is how to update column_A on row n, which depend on column_A on row n-1.
The specific question is compute and persist EMA (exponential moving average) is a MYSQL table.
for EMA calculation (for 10% or 19-day EMA), EMA(n) = 0.1 * price + 0.9 EMA(n-1)
So, we have a recursive equation where the value of row n depend on the value row (n-1)
In any programming language, this is easy to compute.
But if EMA is stored as a column in a MYSQL table. How one can compute and update EMA column efficiently in MYSQL script? Right now, I can only do a sequential update in a loop. My ugly solution is listed here. It requires order(n) updates.
Any smart ideas that only need order(1) update?
Thank you very much
DELIMITER $$
DROP PROCEDURE IF EXISTS update_ema$$
CREATE PROCEDURE update_ema(
IN series_id INT
)
BEGIN
DECLARE counter INT;
SET counter = 2;
WHILE counter <= 5000 DO
update price_table as x
inner join price_table y
on x.id = y.id and x.row_num = y.row_num+1
and x.Id=series_id and x.row_num = counter
set x.EMA19 = func_ema(19, y.EMA19, x.price);
SET counter = counter + 1;
END WHILE;
END$$
DELIMITER ;
If the value of column_a, row(n) depends on the value of column_a, row(n-1), then changing it will change the value in all following rows:
(not code)
if
x = row number
row(x) col(a) = f(row(x-1) col(a))
then
where x = n
row(n) col(a) = f(row(n-1) col(a))
where x = n + 1
row((n+1)) col(a) = f(row((n+1)-1) col(a))
row(n+1) col(a) = f(row(n) col(a))
When you change n, you have to update n+1, which goes on until the end of the table. I don't believe you can get much better than what you have.
Related
Hi Can any one please help me with a procedure for inserting 7 million records for 7 years (million records each year) in loop in MYSQL,
I need to insert in a batch of 500,000 for each batch .
Data is there in table "Archive_data", need to insert in "Stg_table"
Archive data has yearly sales, i want to write a loop in MYSQL looping on Year and insert in a batch of 500K each
i tried
insert into SDL_Stg_Bill_Details
select SDL_Id, Rec_Is_Processed, concat(Bill_Header_Key,'_',Row_Num), Bill_Header_Key,Row_Num from (
SELECT SDL_Id, Rec_Is_Processed, Bill_Details_Key, Bill_Header_Key,
ROW_NUMBER() OVER(partition by Bill_Header_Key order by SDL_Id ) Row_Num
FROM PANTALOONS_SOLUS_PROD.SDL_Stg_Bill_Details_Archive
where EXTRACT(YEAR_MONTH FROM Bill_Date) in ('201406',
'201407',
'201408',
'201409',
'201410') ff
i am getting lock wait time out exceeded error if i am trying 7 M records
at once
Thanks in advance
Not sure if this is what you want, but I hope with some tweaks it might help you:
DELIMITER $
CREATE PROCEDURE CopyDataInBatch()
BEGIN
DECLARE x INT;
DECLARE y INT;
SET x = 1;
SET y = x + 10000;
the_loop : LOOP
IF x > 1000000 THEN /* Use the number of rows you wanted to copy*/
LEAVE the_loop;
END IF;
INSERT INTO table_in_which_copying (col_name_1, col_name_2)
SELECT (col_name_1, col_name_2) FROM table_from_which_copying
ORDER BY (col_name_1)
LIMIT x, y;
SET x = x + 10000;
SET y = y + 10000;
SELECT "Copied 10000 rows"; /* Only for testing... better remove it(?)*/
END LOOP;
END $
DELIMITER ;
SQL gurus, I'm stumped on how I'm going to complete this task. I have a MySQL database table with 40k records that I need to update the group column with an identifier (round robin style). The identifiers are predefined (2, 5, 9).
How could I update this table accordingly? Should look something like the example below:
record group
-----------------
record A 2
record B 5
record C 9
record D 2
record E 5
record F 9
record G 2
Any help is greatly appreciated!
Well after researching dozens of articles I formulated a two-step approach to achieve what I needed. For others who may come across this here is what I did:
Step 1: created a stored procedure to loop through and assign a number to each record. The numbers where 1-3 to represent the three round robin values I had (2, 5, 9). Below is the procedure:
DROP PROCEDURE IF EXISTS ezloop;
DELIMITER ;;
CREATE PROCEDURE ezloop()
BEGIN
DECLARE n, i, z INT DEFAULT 0;
SELECT COUNT(*) FROM `table` INTO n;
SET i = 1;
SET z = 1;
WHILE i < n DO
UPDATE `table` SET `group` = z WHERE `id` = i;
SET i = i + 1;
SET z = z + 1;
IF z > 3 THEN
SET z = 1;
END IF;
END WHILE;
End;
;;
DELIMITER ;
CALL ezloop();
Step 2: created a simple UPDATE statement to update each of the values to my actual round robin values and ran it once for each group:
UPDATE `table` SET `group` = 9 WHERE `group` = 3;
UPDATE `table` SET `group` = 5 WHERE `group` = 2;
UPDATE `table` SET `group` = 2 WHERE `group` = 1;
We have a SQL Server Scalar Function and part of the process is to take one of the input values and do the following
'inputvalue'
Create a table variable and populate with the following rows
inputvalue
inputvalu
inputval
inputva
inputv
input
inpu
inp
Then this table is joined to a query, ordered by len of the inputvalue desc and returns the top 1. The actual code is here
DECLARE #Result NVARCHAR(20);
DECLARE #tempDialCodes TABLE (tempDialCode NVARCHAR(20));
DECLARE #counter INT = LEN(#PhoneNumber);
WHILE #counter > 2
BEGIN
INSERT INTO #tempDialCodes(tempDialCode) VALUES(#PhoneNumber);
SET #PhoneNumber = SUBSTRING(#PhoneNumber, 1, #counter - 1);
SET #counter = #counter - 1;
END
SET #Result = (SELECT TOP 1 [DialCodeID]
FROM DialCodes dc JOIN #tempDialCodes s
ON dc.DialCode = s.tempDialCode
ORDER BY LEN(DialCode) DESC);
RETURN #Result
It works fine but I am asking if there is a way to replace the while loop and somehow joining to the inputvalue to get the same result. When I say it works fine, it's too dam slow but it does work.
I'm stumped on how to break up this string without using a loop and to a table variable but my warning light tells me this is not efficient for running against a table with a million rows.
Are you familiar with tally tables? The speed difference can be incredible. I try to replace every loop with a tally table if possible. The only time I haven't been able to so far is when calling a proc from within a cursor. If using this solution I would recommend a permanent dbo.Tally table with a sufficiently large size rather than recreating every time in the function. You will find other uses for it!
declare #PhoneNumber nvarchar(20) = 'inputvalue';
declare #tempDialCodes table (tempDialCode nvarchar(20));
--create and populate tally table if you don't already a permanent one
--arbitrary 1000 rows for demo...you should figure out if that is enough
--this a 1-based tally table - you will need to tweak if you make it 0-based
declare #Tally table (N int primary key);
insert #Tally
select top (1000) row_number() over (order by o1.object_id) from sys.columns o1, sys.columns o2 order by 1;
--select * from #Tally order by N;
insert #tempDialCodes
select substring(#PhoneNumber, 1, t.N)
from #Tally t
where t.N between 3 and len(#PhoneNumber)
order by t.N desc;
select *
from #tempDialCodes
order by len(tempDialCode) desc;
I understand that I cannot do a transaction inside a trigger, but I CAN inside a procedure. And I should be able to call a procedure from a trigger. Which is exactly what I'm doing.
TABLE A has an "after update trigger".
I only want to call the procedure if one of two columns has been changed.
TABLE A TRIGGER:
IF (
NEW.colA != OLD.colA
OR
NEW.colB != OLD.colB
) THEN
call setBonuses( NEW.colA, NEW.colB );
END IF
The setBonuses SP effects TABLE B by doing some calculations using TABLE_A.colA,TABLE_A.colB (I don't think it's relevant, but TABLE B does have an "after update trigger", but it has a conditional to only do its update if fields NOT effected by the below procedure are changed, so it does 'nothing' when the bonus field is updated.)
THE STORED PROCEDURE:
begin
IF A > B THEN
SET n = A;
...
ELSEIF A < B THEN
SET n = B;
...
END IF;
start transaction;
-- Bonuses
UPDATE TABLE_B
SET bonus = 0
WHERE... ;
UPDATE TABLE_B
SET bonus = bonus + n
WHERE...;
UPDATE TABLE_B
SET bonus = bonus + n
WHERE...;
UPDATE TABLE_B
SET bonus = bonus + n
WHERE...;
UPDATE TABLE_B
SET bonus = bonus + n
WHERE...;
UPDATE TABLE_B
SET bonus = bonus + n
WHERE...;
commit;
end
The SP works fine if called directly by the application, however, I don't want to have to explicitly call it as a chain in some event, I really just need it to do its calcs and update TABLE B - whenever someone changed TABLE A (colA or colB).
Any reason why this isn't possible.
I have a table which contains relative large data,
so that it takes too long for the statements below:
SELECT MIN(column) FROM table WHERE ...
SELECT MAX(column) FROM table WHERE ...
I tried index the column, but the performance still does not suffice my need.
I also thought of caching min and max value in another table by using trigger or event.
But my MySQL version is 5.0.51a which requires SUPER privilege for trigger and does not support event.
It is IMPOSSIBLE for me to have SUPER privilege or to upgrade MySQL.
(If possible, then no need to ask!)
How to solve this problem just inside MySQL?
That is, without the help of OS.
If your column is indexed, you should find min(column) near instantly, because that is the first value MySQL will find.
Same goes for max(column) on an indexed column.
If you cannot add an index for some reason the following triggers will cache the MIN and MAX value in a separate table.
Note that TRUE = 1 and FALSE = 0.
DELIMITER $$
CREATE TRIGGER ai_table1_each AFTER INSERT ON table1 FOR EACH ROW
BEGIN
UPDATE db_info i
SET i.minimum = LEAST(i.minimum, NEW.col)
,i.maximum = GREATEST(i.maximum, NEW.col)
,i.min_count = (i.min_count * (new.col < i.minumum))
+ (i.minimum = new.col) + (i.minimum < new.col)
,i.max_count = (i.max_count * (new.col > i.maximum))
+ (i.maximum = new.col) + (new.col > i.maximum)
WHERE i.tablename = 'table1';
END $$
CREATE TRIGGER ad_table1_each AFTER DELETE ON table1 FOR EACH ROW
BEGIN
DECLARE new_min_count INTEGER;
DECLARE new_max_count INTEGER;
UPDATE db_info i
SET i.min_count = i.min_count - (i.minimum = old.col)
,i.max_count = i.max_count - (i.maximum = old.col)
WHERE i.tablename = 'table1';
SELECT i.min_count INTO new_min_count, i.max_count INTO new_max_count
FROM db_info i
WHERE i.tablename = 'table1';
IF new_max_count = 0 THEN
UPDATE db_info i
CROSS JOIN (SELECT MAX(col) as new_max FROM table1) m
SET i.max_count = 1
,i.maximum = m.new_max;
END IF;
IF new_min_count = 0 THEN
UPDATE db_info i
CROSS JOIN (SELECT MIN(col) as new_min FROM table1) m
SET i.min_count = 1
,i.minimum = m.new_min;
END IF;
END $$
DELIMITER ;
The after update trigger will be some mix of the insert and delete triggers.