mysql optimize stored procedure insert - mysql
This is my first stored procedure so I am not sure I am doing this correctly. I have tried to optimize this as much as I can but still end up with a query timeout at 10 minutes of running. I really need this to scale even higher than what I am working with currently. Any assistance would be great.
I have a decent sized data set (108K rows) and one of the fields contains a comma delimited list (I wish the engineers hadn't done this). I need to break apart that field so each entry is on it's own row AND all other fields are assigned to that row as well. I have developed a stored procedure that loops through the table row by row then breaks apart the field and inserts it into a second table.
Here is the code I have used:
DROP TABLE IF EXISTS dwh_inventory.nas_share_temp;
CREATE TABLE dwh_inventory.nas_share_temp (
share_id int(11) NOT NULL,
fileShareId int(11) NOT NULL,
storageId int(11) NOT NULL,
identifier varchar(1024) NOT NULL,
name varchar(255) NOT NULL,
protocol enum('CIFS','NFS') NOT NULL,
ipInterfaces VARCHAR(100) NOT NULL
) ENGINE=INNODB DEFAULT CHARSET=utf8;
DROP PROCEDURE IF EXISTS dwh_inventory.share_step;
DELIMITER $$
CREATE PROCEDURE dwh_inventory.share_step()
BEGIN
DECLARE n INT DEFAULT 0;
DECLARE i INT DEFAULT 0;
DECLARE strLen INT DEFAULT 0;
DECLARE SubStrLen INT DEFAULT 0;
DECLARE ip VARCHAR(20);
SET autocommit = 0;
SELECT COUNT(*) FROM dwh_inventory.nas_share INTO n;
SET i=0;
WHILE i<n DO
SELECT id, fileShareId, storageId, identifier, name, protocol, ipInterfaces
INTO #share_id, #fileShareId, #storageId, #identifier, #name, #protocol, #ipInterfaces
FROM dwh_inventory.nas_share LIMIT i,1;
IF #ipInterfaces IS NULL THEN
SET #ipInterfaces = '';
END IF;
do_this:
LOOP
SET strLen = CHAR_LENGTH(#ipInterfaces);
SET ip = SUBSTRING_INDEX(#ipInterfaces, ',', 1);
INSERT INTO dwh_inventory.nas_share_temp
(share_id, fileShareId, storageId, identifier,name,protocol,ipInterfaces)
VALUES (#share_id,
#fileShareId,
#storageId,
#identifier,
#name,
#protocol,
ip
);
SET SubStrLen = CHAR_LENGTH(SUBSTRING_INDEX(#ipInterfaces, ',', 1)) + 2;
SET #ipInterfaces = MID(#ipInterfaces, SubStrLen, strLen);
IF #ipInterfaces = '' THEN
LEAVE do_this;
END IF;
END LOOP do_this;
COMMIT;
SET i = i + 1;
END WHILE;
SET autocommit = 1;
END;
$$
DELIMITER ;
CALL dwh_inventory.share_step();
Example of the data:
id,fileShareId,storageId,identifier,name,protocol,ipInterfaces
1325548,1128971,33309,/vol/vol0/:NFS,/vol/vol0/,NFS,"10.66.213.118,10.68.208.76"
1325549,1128991,33309,/vol/vol0/:NFS,/vol/vol0/,NFS,"10.66.213.119,10.68.208.77"
1325550,1128992,33325,/vol/aggr2_64_hs2032/EPS_ROOT/:NFS,/vol/aggr2_64_hs2032/EPS_ROOT/,NFS,10.17.124.10
1325551,1128993,33325,/vol/aggr2_64_hs2032/GCO_Report/:NFS,/vol/aggr2_64_hs2032/GCO_Report/,NFS,10.17.124.10
1325552,1128995,33325,/vol/aggr2_64_hs2032/PI/:NFS,/vol/aggr2_64_hs2032/PI/,NFS,10.17.124.10
1325553,1128996,33325,/vol/aggr2_64_hs2032/a/:NFS,/vol/aggr2_64_hs2032/a/,NFS,10.17.124.10
1325554,1128997,33325,/vol/aggr1_64_sapserv/:NFS,/vol/aggr1_64_sapserv/,NFS,147.204.2.13
1325555,1128999,33325,/vol/aggr2_64_hs2032/:NFS,/vol/aggr2_64_hs2032/,NFS,10.17.124.10
1325556,1129001,33325,/vol/aggr2_64_hs2032/central/:NFS,/vol/aggr2_64_hs2032/central/,NFS,10.17.124.10
1325557,1129004,33325,/vol/nsvfm0079b_E5V/db_clients/:NFS,/vol/nsvfm0079b_E5V/db_clients/,NFS,"10.21.188.161,10.70.151.93"
1325558,1129006,33325,/vol/aggr2_64_hs2032/istrans/:NFS,/vol/aggr2_64_hs2032/istrans/,NFS,10.17.124.10
1325559,1129008,33325,/vol/nsvfm0017_DEWDFGLD00603/:NFS,/vol/nsvfm0017_DEWDFGLD00603/,NFS,"10.21.188.115,10.70.151.138"
1325560,1129009,33325,/vol/nsvfm0017_vol0/:NFS,/vol/nsvfm0017_vol0/,NFS,"10.21.188.115,10.70.151.138"
1325561,1129011,33325,/vol/nsvfm0017a_ls2278/:NFS,/vol/nsvfm0017a_ls2278/,NFS,"10.21.188.115,10.70.151.138"
1325562,1129015,33325,/vol/nsvfm0051passive_vol0/:NFS,/vol/nsvfm0051passive_vol0/,NFS,10.17.144.249
1325563,1129017,33325,/vol/nsvfm0053_vol0/:NFS,/vol/nsvfm0053_vol0/,NFS,"10.21.189.251,10.70.151.109"
InnoDB tables must have a PRIMARY KEY.
LIMIT i,1 will get slower and slower as you go through the table -- it hast to skip over i rows before finding the one you need.
Don't try to split comma-separated text in SQL; use a real language (PHP / Perl / etc). Or, as Lew suggests, write out that column, then use LOAD DATA to bring it into another table.
LIMIT should be preceded by an ORDER BY.
Related
Update a value and set a local variable in a case statement
I am trying to update a value in the database but also want to set a local variable VAR_IS_RATE_LIMITED. The reason for this is because I do not want to use a select statement and want to execute it within one. How can I set the VAR_IS_RATE_LIMITED? I looked at other questions but their CASE statements weren't embedded in an update statement. DELIMITER // CREATE FUNCTION F_RATE_LIMITED(P_IP varchar(45), P_MAX_RATE int unsigned ) RETURNS INT UNSIGNED BEGIN DECLARE VAR_IS_RATE_LIMITED INT UNSIGNED DEFAULT 0; INSERT INTO rate_limit (ip, rate) VALUES (P_IP, 1) ON DUPLICATE KEY UPDATE rate = CASE WHEN (rate + 1) > P_MAX_RATE THEN SET VAR_IS_RATE_LIMITED = 1; rate ELSE rate + 1 END; RETURN VAR_IS_RATE_LIMITED; END; // DELIMITER ;
To set something during a data modification statement, it has to be part of an expression, not a statement. And as far as I know you can't set a stored function variable in an expression. But you can set a user variable: DELIMITER // CREATE FUNCTION F_RATE_LIMITED(P_IP varchar(45), P_MAX_RATE int unsigned ) RETURNS INT UNSIGNED BEGIN SET #VAR_IS_RATE_LIMITED = 0; INSERT INTO rate_limit (ip, rate) VALUES (P_IP, 1) ON DUPLICATE KEY UPDATE rate = CASE WHEN (rate + 1) > P_MAX_RATE THEN CASE WHEN #VAR_IS_RATE_LIMITED := 1 THEN rate END ELSE rate + 1 END; RETURN #VAR_IS_RATE_LIMITED; END; // DELIMITER ; Here the variable is set in a CASE test (you can also use an IF expression) that is always true so rate is always returned from the CASE.
Mysql stored procedure to update and return a list of matching rows
I built this stored procedure. It's supposed to select all the 'non-started' rows from a table we're using as a queue. Update those rows to mark them as 'STARTING' so they won't be grabbed by another processor and then return those rows back to the requestor for processing. It works but it seems to be putting an unacceptably high load on our servers. The main reason for this stored procedure is to select and update the rows so we don't get the same row selected by multiple processors at the same time. I'm not a DBA and I had to learn some stored procedure and cursor stuff on the fly so maybe I'm missing something obvious to someone else. I'm guessing there is a way to do this without completely overloading our db servers. Any help is appreciated. We use a MySql db 5.6.xx with a Java/Tomcat web-app. CREATE DEFINER=`admin`#`%` PROCEDURE `select_and_start_non_started`( IN p_companyId INT(11), IN p_howMany INT, IN p_instance varchar(50), IN p_status varchar(50), IN p_updateBy varchar(50) ) BEGIN DECLARE v_currentId INT; DECLARE v_loopDone INT DEFAULT 0; DECLARE v_loopCounter INT DEFAULT 0; DECLARE v_idList VARCHAR(1024) DEFAULT NULL; DECLARE queue_csr CURSOR FOR SELECT id FROM queue WHERE (status in (_utf8'NEW' COLLATE utf8_unicode_ci, _utf8'RESTARTED' COLLATE utf8_unicode_ci, _utf8'WAITING' COLLATE utf8_unicode_ci, _utf8'QUEUED' COLLATE utf8_unicode_ci)) AND if(LENGTH(p_companyId) > 0, companyid=p_companyId, true) LIMIT p_howMany FOR UPDATE; DECLARE CONTINUE HANDLER FOR NOT FOUND SET v_loopDone=1; DECLARE CONTINUE HANDLER FOR SQLEXCEPTION BEGIN ROLLBACK; SET autocommit = TRUE; RESIGNAL; END; SET v_idList = ""; SET autocommit = FALSE; START TRANSACTION; OPEN queue_csr; iq_loop:LOOP FETCH queue_csr INTO v_currentId; IF v_loopDone THEN LEAVE iq_loop; END IF; UPDATE queue SET status = p_status COLLATE utf8_unicode_ci, updatedDate=NOW(), updatedBy=p_updateBy, recordStatus=p_instance WHERE id = v_currentId; SET v_idList = CONCAT(v_idList, ",", v_currentId); SET v_loopCounter=v_loopCounter+1; IF v_loopCounter > p_howMany THEN LEAVE iq_loop; END IF; END LOOP iq_loop; CLOSE queue_csr; SET v_loopDone=0; COMMIT; SET autocommit = TRUE; SELECT * FROM queue q WHERE FIND_IN_SET(id, v_idList); END
It works but it seems to be putting an unacceptably high load on our servers. As 2 first steps, It might be helpful to have "indexes" on the columns: status and companyid. If you don't have the indexes, you can create them using: alter table select_and_start_non_started add index (status); alter table select_and_start_non_started add index(companyid); p_companyId is an integer, so I am not sure why you are taking its length. Anyway, since LENGTH(p_companyId) is a constant, instead of calling it for every row, you may want to save it in a variable: declare v_companyid_len int; set v_companyid_len=LENGTH(p_companyId); BEGIN DECLARE queue_csr CURSOR FOR .... AND if(v_companyid_len>0 ...
Foreach Data in Field Insert Selected Field from One Database to Another in MySQL
I have two (2) databases of dissimilar Schematics, db1 migrated from MSSQL to MYSQL and db2 created from Laravel Migration. Here's the challenge: The tables of db1 do not have id columns (Primary Key) like is easily found on db2 tables. So I kept getting the warning message: Current selection does not contain a unique column. Grid edit, checkbox, Edit, Copy and Delete features are not available. So I had to inject the id columns on the tables in the db1 I need to extract fields [level_name, class_name] from stdlist in db1, Create levels (id,level_name,X,Y) on db2 classes (id,class_name,level_id) on db2 To throw more light: The level_id should come from the already created levels table I have already succeeded in extracting the first instance using the following snippet: First Query to Create Levels INSERT INTO db2.levels(level_name,X,Y) SELECT class_name as level_name,1 as X,ClassAdmitted as Y FROM db1.stdlist GROUP BY ClassAdmitted; This was successful. Now, I need to use the newly created ids in levels table to fill up level_id column in the classes table. For that to be possible, must I re-run the above selection schematics? Is there no better way to maybe join the table column from db1.levels to db2.stdlist and extract the required fields for the new insert schematics. I'll appreciate any help. Thanks in advance.
Try adding a column for Processed and then do a while exists loop INSERT INTO db2.levels(level_name,X,Y) SELECT class_name as level_name,1 as X,ClassAdmitted as Y, 0 as Processed FROM db1.stdlist GROUP BY ClassAdmitted; WHILE EXISTS(SELECT * FROM db2.levels WHERE Processed = 0) BEGIN DECLARE #level_name AS VARCHAR(MAX) SELECT TOP 1 #level_name=level_name FROM db2.levels WHERE Processed = 0 --YOUR CODE UPDATE db2.levels SET Processed=1 WHERE level_name=#level_name END You may need to dump into a temp table first and then insert into your real table (db2.levels) when you're done processing. Then you wouldn't need the Unnecessary column of processed on the final table.
This is what worked for me eventually:
First, I picked up the levels from the initial database thus:
INSERT INTO db2.levels(`name`,`school_id`,`short_code`)
SELECT name ,school_id,short_code
FROM db1.levels
GROUP BY name
ORDER BY CAST(IF(REPLACE(name,' ','')='','0',REPLACE(name,' ','')) AS UNSIGNED
INTEGER) ASC;
Then I created a PROCEDURE for the classes insertion
CREATE PROCEDURE dowhileClasses()
BEGIN
SET #Level = 1;
SET #Max = SELECT count(`id`) FROM db2.levels;
START TRANSACTION;
WHILE #Level <= #Max DO
BEGIN
DECLARE val1 VARCHAR(255) DEFAULT NULL;
DECLARE val2 VARCHAR(255) DEFAULT NULL;
DECLARE bDone TINYINT DEFAULT 0;
DECLARE curs CURSOR FOR
SELECT trim(`Class1`)
FROM db1.dbo_tblstudent
WHERE CAST(IF(REPLACE(name,' ','')='','0',REPLACE(name,' ','')) AS UNSIGNED INTEGER) =#Level
GROUP BY `Class1`;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET bDone = 1;
OPEN curs;
SET bDone = 0;
REPEAT
FETCH curs INTO val1;
IF bDone = 0 THEN
SET #classname = val1;
SET #levelID = (SELECT id FROM db2.levels WHERE short_code=#Level limit 1);
SET #schoolId = 1;
SET #classId = (SELECT `id` FROM db2.classes where class_name = #classname and level_id= #levelID limit 1);
IF #classId is null and #classname is not null THEN
INSERT INTO db2.classes(class_name,school_id,level_id)
VALUES(#classname,#schoolId,#levelID);
END IF;
END IF;
UNTIL bDone END REPEAT;
CLOSE curs;
END;
SELECT CONCAT('lEVEL: ',#Level,' Done');
SET #Level = #Level + 1;
END WHILE;
END;
//
delimiter ;
CALL dowhileClasses();
With this, I was able to dump The classes profile matching the previously created level_ids.
The whole magic relies on the CURSOR protocol.
For further details here is one of the documentations I used.
Function that generate Code returns the same things
There is a MySQL function in our web system to generate Code. The structure of the code is
district_cd(length:2) + date(length:8) + sequence no(length:5,start at 1).<like : ab2016090800001>
The sequence no was saved in table and will be updated (+1) when generate a new code.
But sometimes it returned two same codes and makes us fall in trouble. Here are the captures to replicate this problem, I will attach the DDL after this.
Step 1.Client1->change to manual commit then generate a code, but do not commit.
SET autocommit = 0;
select * from applies;
select * from sequence where apply_date = "2016-09-08";
select nextval("ab");
insert into applies (apply_id,apply_no,created,district_cd) values (2,"ab2016090800002","ab",now());
select * from sequence where apply_date = "2016-09-08";
Step2.Client2->change to manual commit then generate a code, stuck as Client1 locked
SET autocommit = 0;
select * from applies;
select * from sequence where apply_date = "2016-09-08";
insert into applies (apply_id,apply_no,created,district_cd) values (3,"ab20160908123456780","ab",now());
Step3.Client1->commit;
commit;
select * from sequence where apply_date = "2016-09-08";
Step4.Client2->code was generated and two records appeared in sequence table
select * from sequence where apply_date = "2016-09-08";
capture of Step4
Step5.Client2->commit;one of the two records that appeared in sequence table was deleted.The codes generated are duplicated.
commit;
select * from sequence where apply_date = "2016-09-08";
select * from applies;
capture of Step5
※DDL
Table:applies (apply_no:save the code)
CREATE TABLE `applies` (
`apply_id` varchar(100) NOT NULL DEFAULT '',
`apply_no` varchar(100) NOT NULL DEFAULT '',
`district_cd` varchar(100) DEFAULT NULL,
`created` datetime DEFAULT NULL,
PRIMARY KEY (`apply_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Table:sequence (current_value:save current sequnce value)
CREATE TABLE `sequence` (
`district_cd` varchar(3) NOT NULL DEFAULT '',
`current_value` int(11) NOT NULL DEFAULT '0',
`apply_date` date NOT NULL DEFAULT '0000-00-00',
PRIMARY KEY (`district_cd`,`current_value`,`apply_date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Function:currval->get current sequence value by district_cd
DELIMITER ;;
CREATE DEFINER=`usr`#`%` FUNCTION `currval`(d VARCHAR(3)) RETURNS int(11)
DETERMINISTIC
BEGIN
DECLARE value INTEGER;
DECLARE needInitSequence INTEGER;
DECLARE today DATE;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET needInitSequence = 1;
SET value = 0;
SET today = current_date();
SELECT `current_value` INTO value
FROM `sequence`
WHERE `district_cd` = d AND `apply_date` = today limit 1;
IF needInitSequence = 1 THEN
INSERT INTO `sequence` (`district_cd`, `current_value`, `apply_date`) VALUES (d, value, today);
END IF;
RETURN value;
END
;;
DELIMITER ;
Function:nextval->generate code by district_cd
DELIMITER ;;
CREATE DEFINER=`usr`#`%` FUNCTION `nextval`(d VARCHAR(3)) RETURNS varchar(16) CHARSET utf8
DETERMINISTIC
BEGIN
DECLARE value INTEGER;
SET value = currval(d);
UPDATE `sequence`
SET `current_value` = `current_value` + 1
WHERE `district_cd` = d AND `apply_date` = current_date();
RETURN concat(d, date_format(now(), '%Y%m%d'), LPAD(currval(d), 5, '0'));
END
;;
DELIMITER ;
Triggers of applies->a business logic,if the length of apply_no is greater than 18,it will call the function:nextval to generate a new code
DELIMITER ;;
CREATE TRIGGER `convert_long_no` BEFORE INSERT ON `applies` FOR EACH ROW BEGIN
IF ((SELECT LENGTH(NEW.apply_no)) >= 18) THEN
SET NEW.apply_no = (SELECT nextval(NEW.district_cd));
END IF;
END
;;
DELIMITER ;
My Questions:
Why did the function:nextval returns two same codes?
Why did two records appear in sequnce when update the record.
Match two mysql cols on alpha chars (ignoring numbers in same field)
I was wondering if you know of a way I could filter a mysql query to only show the ‘alpha’ characters from a specific field So something like SELECT col1, col2, **alpha_chars_only(col3)** FROM table I am not looking to update only select. I have been looking at some regex but without much luck most of what turned up was searching for fields that only contain ‘alpha’ chars. In a much watered down context... I have col1 which contains abc and col two contains abc123 and I want to match them on alpha chars only. There can be any number of letters or numbers. Any help very much wel come
You probably need to write a custom function for this. If you want to do it in MySQL, you could create a stored function like this: DELIMITER $$ drop function if exists alpha_chars_only $$ create function alpha_chars_only (p_string text) returns text begin declare v_return_val text default ''; declare v_iter int unsigned default 1; declare v_length int unsigned default 0; declare v_char char(1) default null; set v_length = char_length(p_string); while (v_iter <= v_length) do set v_char = substring(p_string,v_iter,1); if (v_char REGEXP '[a-z]') then set v_return_val = concat(v_return_val,v_char); end if; set v_iter = v_iter + 1; end while; return v_return_val; end $$ DELIMITER ;