SQL gurus, I'm stumped on how I'm going to complete this task. I have a MySQL database table with 40k records that I need to update the group column with an identifier (round robin style). The identifiers are predefined (2, 5, 9).
How could I update this table accordingly? Should look something like the example below:
record group
-----------------
record A 2
record B 5
record C 9
record D 2
record E 5
record F 9
record G 2
Any help is greatly appreciated!
Well after researching dozens of articles I formulated a two-step approach to achieve what I needed. For others who may come across this here is what I did:
Step 1: created a stored procedure to loop through and assign a number to each record. The numbers where 1-3 to represent the three round robin values I had (2, 5, 9). Below is the procedure:
DROP PROCEDURE IF EXISTS ezloop;
DELIMITER ;;
CREATE PROCEDURE ezloop()
BEGIN
DECLARE n, i, z INT DEFAULT 0;
SELECT COUNT(*) FROM `table` INTO n;
SET i = 1;
SET z = 1;
WHILE i < n DO
UPDATE `table` SET `group` = z WHERE `id` = i;
SET i = i + 1;
SET z = z + 1;
IF z > 3 THEN
SET z = 1;
END IF;
END WHILE;
End;
;;
DELIMITER ;
CALL ezloop();
Step 2: created a simple UPDATE statement to update each of the values to my actual round robin values and ran it once for each group:
UPDATE `table` SET `group` = 9 WHERE `group` = 3;
UPDATE `table` SET `group` = 5 WHERE `group` = 2;
UPDATE `table` SET `group` = 2 WHERE `group` = 1;
Related
Hi Can any one please help me with a procedure for inserting 7 million records for 7 years (million records each year) in loop in MYSQL,
I need to insert in a batch of 500,000 for each batch .
Data is there in table "Archive_data", need to insert in "Stg_table"
Archive data has yearly sales, i want to write a loop in MYSQL looping on Year and insert in a batch of 500K each
i tried
insert into SDL_Stg_Bill_Details
select SDL_Id, Rec_Is_Processed, concat(Bill_Header_Key,'_',Row_Num), Bill_Header_Key,Row_Num from (
SELECT SDL_Id, Rec_Is_Processed, Bill_Details_Key, Bill_Header_Key,
ROW_NUMBER() OVER(partition by Bill_Header_Key order by SDL_Id ) Row_Num
FROM PANTALOONS_SOLUS_PROD.SDL_Stg_Bill_Details_Archive
where EXTRACT(YEAR_MONTH FROM Bill_Date) in ('201406',
'201407',
'201408',
'201409',
'201410') ff
i am getting lock wait time out exceeded error if i am trying 7 M records
at once
Thanks in advance
Not sure if this is what you want, but I hope with some tweaks it might help you:
DELIMITER $
CREATE PROCEDURE CopyDataInBatch()
BEGIN
DECLARE x INT;
DECLARE y INT;
SET x = 1;
SET y = x + 10000;
the_loop : LOOP
IF x > 1000000 THEN /* Use the number of rows you wanted to copy*/
LEAVE the_loop;
END IF;
INSERT INTO table_in_which_copying (col_name_1, col_name_2)
SELECT (col_name_1, col_name_2) FROM table_from_which_copying
ORDER BY (col_name_1)
LIMIT x, y;
SET x = x + 10000;
SET y = y + 10000;
SELECT "Copied 10000 rows"; /* Only for testing... better remove it(?)*/
END LOOP;
END $
DELIMITER ;
I have a database table like this one:
group | detailsID | price
EK | 1 | 1.40
EK | 2 | 1.50
EK | 3 | 1.60
H | 1 | 2.40
H | 2 | 2.50
Now I want to copy the data from group "EK" to the group "H", so the prices for the detailsID's must be adjusted for the detailsIDs 1 and 2, and the entry for detailsID 3 must be inserted for group "H".
How can I do that with one or two MySQL query's?
Thanks!
We can try doing an INSERT INTO ... SELECT with ON DUPLICATE KEY UPDATE:
INSERT INTO yourTable (`group`, detailsID, price)
SELECT 'H', detailsID, price
FROM yourTable t
WHERE `group` = 'EK'
ON DUPLICATE KEY UPDATE price = t.price;
But this assumes that there exists a unique key on (group, detailsID). If this would not be possible, then this approach would not work.
As an alternative, I might do this in two steps. First, remove the H group records, then insert the updated H records you expect.
DELETE
FROM yourTable
WHERE `group` = 'H';
INSERT INTO yourTable (`group`, detailsID, price)
SELECT 'H', detailsID, price
FROM yourTable
WHERE `group` = 'EK';
I use the above approach because a single update can't handle your requirement, since new records also need to be inserted.
Note that you should avoid naming your columns and tables using reserved MySQL keywords such as GROUP.
You can try this as well, Following code implemented using stored procedures. Very simple not that difficult to understand. You may need to modify data type and optimize the code as per the requirement.
DELIMITER $$;
DROP PROCEDURE IF EXISTS update_H $$;
CREATE PROCEDURE update_H()
BEGIN
DECLARE finished INTEGER DEFAULT 0;
DECLARE `group_col` varchar(255) DEFAULT "";
DECLARE `detaildid_col` varchar(255) DEFAULT "";
DECLARE `price_col` varchar(255) DEFAULT "";
DECLARE H_FOUND INTEGER DEFAULT 0;
DECLARE pull_data CURSOR FOR select `group`, `detaildid`, `price` from test.newtab WHERE `group` = 'EK';
DECLARE CONTINUE HANDLER FOR NOT FOUND SET finished = 1;
OPEN pull_data;
traverse_data: LOOP
FETCH pull_data INTO group_col, detaildid_col, price_col;
IF finished = 1 THEN
LEAVE traverse_data;
END IF;
SET H_FOUND = (SELECT count(*) from test.newtab where `group` = 'H' AND `detaildid` = detaildid_col);
IF ( H_FOUND = 1 ) THEN
UPDATE test.newtab SET `price` = price_col where `group` = 'H' AND `detaildid` = detaildid_col;
ELSE
INSERT INTO test.newtab (`group`, `detaildid`, `price`) VALUES ('H', detaildid_col, price_col);
END IF;
END LOOP traverse_data;
CLOSE pull_data;
END $$;
DELIMITER ;
You can call this procedure by executing, call update_H();
I have a table with different zone names with Id's and their parent id's. I want to generate report with the hierarchies. Like below.
Table: Groups
ID Name ParentID
1 Corporate NULL
2 Zone 1 1
3 Zone 2 1
4 Zone 3 1
5 Zone 4 1
6 Telangana 2
7 Hyderabad 6
8 Khammam 6
9 Odisha 3
10 Bhubaneshwar 9
Using above table now I want to generate report. If I select corporate then I need to get all data. If I select Zone 1 I need to get all child relations as well.Like Below
Zone 1, Telangana, Hyderabad, Khammam
Please help me on writing query for this.
Working with hierarchical data in SQL is tricky. I suggest you to use nested sets model and modify your table: add left and right columns. Update them when adding, updating and deleting the data (this is the price for easy SELECTs). When you do it, getting all the children of record #2 will be easy:
SELECT `left`, `right` FROM `table` WHERE id=2;
-- here we get $left and $right
SELECT * FROM `table` WHERE `left`>=$left AND `right` <= $right;
I've found the answer for my above question. This can be achieved by using Stored Procedure.
DELIMITER $$
DROP PROCEDURE IF EXISTS getHierarchy_proc $$
CREATE PROCEDURE getHierarchy_proc (IN GivenID INT, OUT ids VARCHAR(10000))
BEGIN
DECLARE int_check VARCHAR(1000);
DECLARE is_exit TINYINT(1) DEFAULT 0;
DROP TABLE IF EXISTS bu_tmp;
CREATE TEMPORARY TABLE bu_tmp(
bu_id INT(11) NOT NULL,
is_upd TINYINT(1) NOT NULL DEFAULT 0);
SET SESSION GROUP_CONCAT_MAX_LEN = 100000;
INSERT INTO bu_tmp (bu_id)
SELECT GivenID;
SET int_check = (SELECT bu_id FROM bu_tmp WHERE bu_id = GivenID AND is_upd = 0);
SET is_exit = 1;
REPEAT
IF is_exit > 0 THEN
INSERT INTO bu_tmp (bu_id,is_upd)
SELECT ID,0 FROM Groups WHERE FIND_IN_SET(parent_id , int_check);
UPDATE bu_tmp SET is_upd = 1 WHERE FIND_IN_SET(bu_id,int_check);
SET is_exit = (SELECT COUNT(*) FROM bu_tmp WHERE is_upd = 0);
SET int_check = (SELECT GROUP_CONCAT(bu_id) FROM bu_tmp WHERE is_upd = 0);
END IF;
UNTIL is_exit = 0 END REPEAT;
SET ids = (SELECT GROUP_CONCAT(lew.le_wh_id)
FROM bu_tmp bu JOIN legalentity_warehouses lew
WHERE bu.bu_id = lew.bu_id
AND lew.dc_type = 118001
AND lew.status = 1);
END$$
DELIMITER ;
The general question is how to update column_A on row n, which depend on column_A on row n-1.
The specific question is compute and persist EMA (exponential moving average) is a MYSQL table.
for EMA calculation (for 10% or 19-day EMA), EMA(n) = 0.1 * price + 0.9 EMA(n-1)
So, we have a recursive equation where the value of row n depend on the value row (n-1)
In any programming language, this is easy to compute.
But if EMA is stored as a column in a MYSQL table. How one can compute and update EMA column efficiently in MYSQL script? Right now, I can only do a sequential update in a loop. My ugly solution is listed here. It requires order(n) updates.
Any smart ideas that only need order(1) update?
Thank you very much
DELIMITER $$
DROP PROCEDURE IF EXISTS update_ema$$
CREATE PROCEDURE update_ema(
IN series_id INT
)
BEGIN
DECLARE counter INT;
SET counter = 2;
WHILE counter <= 5000 DO
update price_table as x
inner join price_table y
on x.id = y.id and x.row_num = y.row_num+1
and x.Id=series_id and x.row_num = counter
set x.EMA19 = func_ema(19, y.EMA19, x.price);
SET counter = counter + 1;
END WHILE;
END$$
DELIMITER ;
If the value of column_a, row(n) depends on the value of column_a, row(n-1), then changing it will change the value in all following rows:
(not code)
if
x = row number
row(x) col(a) = f(row(x-1) col(a))
then
where x = n
row(n) col(a) = f(row(n-1) col(a))
where x = n + 1
row((n+1)) col(a) = f(row((n+1)-1) col(a))
row(n+1) col(a) = f(row(n) col(a))
When you change n, you have to update n+1, which goes on until the end of the table. I don't believe you can get much better than what you have.
I've read that this can be done without issue using MyISAM as it is the default behavior , but I'm using InnoDB so need a trigger for such.
The two PK fields are batch and lineItem. If a record is deleted I want the numbering to start from the largest integer for batch. Not fill in the holes.
This is to set up a testing environment for a legacy system. So the schema is the way it is, I thought I'd mention that to avoid any discussion about whether it is good or not.
Edit: I want something like the following insert statement as a trigger
INSERT INTO payroll(`batch`,`lineItem`)
(select 'T105',t1.lineItem + 1 from payroll as t1 where batch = 'T105' order by lineItem desc limit 1);
But where 'T105' (the batch id) is hard coded I want the trigger to pick that up from the insert.
So I want to be able to say something like:
INSERT INTO payroll(`batch`)VALUES('T001','T001','T001', 'T002', 'T002', 'T002');
and I would expect to see in the table:
batch lineItem
T001 1
T001 2
T001 3
T002 1
T002 2
T002 3
Getting further:
In trying to implement this I've come up with:
DELIMITER $$
CREATE TRIGGER `co05_test`.`ins_lineItem`
BEFORE INSERT ON `co05_test`.`my_table`
FOR EACH ROW
BEGIN
select lineItem + 1 into #newLineItem from my_table where batch = NEW.batch order by lineItem desc limit 1;
set NEW.lineItem = #newLineItem;
END$$
However when I try...
INSERT INTO `co05_test`.`my_table`(`batch`)VALUES('T001');
I get this error: Column 'lineItem' cannot be null
Which is defined as not being nullable but I though the trigger should set the value!
Solution which I used:
-- Trigger DDL Statements
DELIMITER $$
USE `co05_test`$$
CREATE TRIGGER `co05_test`.`ins_lineItem`
BEFORE INSERT ON `co05_test`.`my_table`
FOR EACH ROW
BEGIN
select count(*) into #batchCount from my_table where batch = NEW.batch;
select lineItem + 1 into #newLineItem from my_table where batch = NEW.batch order by lineItem desc limit 1;
if #batchCount > 0 then
set NEW.lineItem = #newLineItem;
else
set NEW.lineItem = 1;
end if;
END;
$$
Have you tried declaring the variable instead?
DELIMITER $$
CREATE TRIGGER `co05_test`.`ins_lineItem`
BEFORE INSERT ON `co05_test`.`my_table`
FOR EACH ROW
BEGIN
DECLARE newLineItem INT;
SELECT
lineItem + 1 into newLineItem
FROM my_table
WHERE batch = NEW.batch
ORDER BY lineItem DESC
LIMIT 1;
SET NEW.lineItem = newLineItem;
END$$