Related
This is my first stored procedure so I am not sure I am doing this correctly. I have tried to optimize this as much as I can but still end up with a query timeout at 10 minutes of running. I really need this to scale even higher than what I am working with currently. Any assistance would be great.
I have a decent sized data set (108K rows) and one of the fields contains a comma delimited list (I wish the engineers hadn't done this). I need to break apart that field so each entry is on it's own row AND all other fields are assigned to that row as well. I have developed a stored procedure that loops through the table row by row then breaks apart the field and inserts it into a second table.
Here is the code I have used:
DROP TABLE IF EXISTS dwh_inventory.nas_share_temp;
CREATE TABLE dwh_inventory.nas_share_temp (
share_id int(11) NOT NULL,
fileShareId int(11) NOT NULL,
storageId int(11) NOT NULL,
identifier varchar(1024) NOT NULL,
name varchar(255) NOT NULL,
protocol enum('CIFS','NFS') NOT NULL,
ipInterfaces VARCHAR(100) NOT NULL
) ENGINE=INNODB DEFAULT CHARSET=utf8;
DROP PROCEDURE IF EXISTS dwh_inventory.share_step;
DELIMITER $$
CREATE PROCEDURE dwh_inventory.share_step()
BEGIN
DECLARE n INT DEFAULT 0;
DECLARE i INT DEFAULT 0;
DECLARE strLen INT DEFAULT 0;
DECLARE SubStrLen INT DEFAULT 0;
DECLARE ip VARCHAR(20);
SET autocommit = 0;
SELECT COUNT(*) FROM dwh_inventory.nas_share INTO n;
SET i=0;
WHILE i<n DO
SELECT id, fileShareId, storageId, identifier, name, protocol, ipInterfaces
INTO #share_id, #fileShareId, #storageId, #identifier, #name, #protocol, #ipInterfaces
FROM dwh_inventory.nas_share LIMIT i,1;
IF #ipInterfaces IS NULL THEN
SET #ipInterfaces = '';
END IF;
do_this:
LOOP
SET strLen = CHAR_LENGTH(#ipInterfaces);
SET ip = SUBSTRING_INDEX(#ipInterfaces, ',', 1);
INSERT INTO dwh_inventory.nas_share_temp
(share_id, fileShareId, storageId, identifier,name,protocol,ipInterfaces)
VALUES (#share_id,
#fileShareId,
#storageId,
#identifier,
#name,
#protocol,
ip
);
SET SubStrLen = CHAR_LENGTH(SUBSTRING_INDEX(#ipInterfaces, ',', 1)) + 2;
SET #ipInterfaces = MID(#ipInterfaces, SubStrLen, strLen);
IF #ipInterfaces = '' THEN
LEAVE do_this;
END IF;
END LOOP do_this;
COMMIT;
SET i = i + 1;
END WHILE;
SET autocommit = 1;
END;
$$
DELIMITER ;
CALL dwh_inventory.share_step();
Example of the data:
id,fileShareId,storageId,identifier,name,protocol,ipInterfaces
1325548,1128971,33309,/vol/vol0/:NFS,/vol/vol0/,NFS,"10.66.213.118,10.68.208.76"
1325549,1128991,33309,/vol/vol0/:NFS,/vol/vol0/,NFS,"10.66.213.119,10.68.208.77"
1325550,1128992,33325,/vol/aggr2_64_hs2032/EPS_ROOT/:NFS,/vol/aggr2_64_hs2032/EPS_ROOT/,NFS,10.17.124.10
1325551,1128993,33325,/vol/aggr2_64_hs2032/GCO_Report/:NFS,/vol/aggr2_64_hs2032/GCO_Report/,NFS,10.17.124.10
1325552,1128995,33325,/vol/aggr2_64_hs2032/PI/:NFS,/vol/aggr2_64_hs2032/PI/,NFS,10.17.124.10
1325553,1128996,33325,/vol/aggr2_64_hs2032/a/:NFS,/vol/aggr2_64_hs2032/a/,NFS,10.17.124.10
1325554,1128997,33325,/vol/aggr1_64_sapserv/:NFS,/vol/aggr1_64_sapserv/,NFS,147.204.2.13
1325555,1128999,33325,/vol/aggr2_64_hs2032/:NFS,/vol/aggr2_64_hs2032/,NFS,10.17.124.10
1325556,1129001,33325,/vol/aggr2_64_hs2032/central/:NFS,/vol/aggr2_64_hs2032/central/,NFS,10.17.124.10
1325557,1129004,33325,/vol/nsvfm0079b_E5V/db_clients/:NFS,/vol/nsvfm0079b_E5V/db_clients/,NFS,"10.21.188.161,10.70.151.93"
1325558,1129006,33325,/vol/aggr2_64_hs2032/istrans/:NFS,/vol/aggr2_64_hs2032/istrans/,NFS,10.17.124.10
1325559,1129008,33325,/vol/nsvfm0017_DEWDFGLD00603/:NFS,/vol/nsvfm0017_DEWDFGLD00603/,NFS,"10.21.188.115,10.70.151.138"
1325560,1129009,33325,/vol/nsvfm0017_vol0/:NFS,/vol/nsvfm0017_vol0/,NFS,"10.21.188.115,10.70.151.138"
1325561,1129011,33325,/vol/nsvfm0017a_ls2278/:NFS,/vol/nsvfm0017a_ls2278/,NFS,"10.21.188.115,10.70.151.138"
1325562,1129015,33325,/vol/nsvfm0051passive_vol0/:NFS,/vol/nsvfm0051passive_vol0/,NFS,10.17.144.249
1325563,1129017,33325,/vol/nsvfm0053_vol0/:NFS,/vol/nsvfm0053_vol0/,NFS,"10.21.189.251,10.70.151.109"
InnoDB tables must have a PRIMARY KEY.
LIMIT i,1 will get slower and slower as you go through the table -- it hast to skip over i rows before finding the one you need.
Don't try to split comma-separated text in SQL; use a real language (PHP / Perl / etc). Or, as Lew suggests, write out that column, then use LOAD DATA to bring it into another table.
LIMIT should be preceded by an ORDER BY.
I have developed a function for split string in tsql but mysql don't have some built in functions. I needed to function in MYSQL as i am new in mysql. Function should accept 2 parameters
1. String to be split
2. separator (',' or whatever)
Kindly reply me.
i had found solution on the internet you can into that.
DELIMITER //
DROP FUNCTION IF EXISTS `splitAndTranslate` //
CREATE FUNCTION splitAndTranslate(str TEXT, delim VARCHAR(124))
RETURNS TEXT
DETERMINISTIC
BEGIN
DECLARE i INT DEFAULT 0; -- total number of delimiters
DECLARE ctr INT DEFAULT 0; -- counter for the loop
DECLARE str_len INT; -- string length,self explanatory
DECLARE out_str text DEFAULT ''; -- return string holder
DECLARE temp_str text DEFAULT ''; -- temporary string holder
DECLARE temp_val VARCHAR(255) DEFAULT ''; -- temporary string holder for query
-- get length
SET str_len=LENGTH(str);
SET i = (LENGTH(str)-LENGTH(REPLACE(str, delim, '')))/LENGTH(delim) + 1;
-- get total number delimeters and add 1
-- add 1 since total separated values are 1 more than the number of delimiters
-- start of while loop
WHILE(ctr<i) DO
-- add 1 to the counter, which will also be used to get the value of the string
SET ctr=ctr+1;
-- get value separated by delimiter using ctr as the index
SET temp_str = REPLACE(SUBSTRING(SUBSTRING_INDEX(str, delim, ctr), LENGTH(SUBSTRING_INDEX(str, delim,ctr - 1)) + 1), delim, '');
-- query real value and insert into temporary value holder, temp_str contains the exploded ID
SELECT <real_value_column> INTO temp_val FROM <my_table> WHERE <table_id>=temp_str;
-- concat real value into output string separated by delimiter
SET out_str=CONCAT(out_str, temp_val, ',');
END WHILE;
-- end of while loop
-- trim delimiter from end of string
SET out_str=TRIM(TRAILING delim FROM out_str);
RETURN(out_str); -- return
END//
reference http://www.slickdev.com/2008/09/15/mysql-query-real-values-from-delimiter-separated-string-ids/
In mysql they they dont support some functionality like sqlserver. so spliting will be difficult in mysql
SELECT e.`studentId`, SPLIT(",", c.`courseNames`)[e.`courseId`]
FROM ..
SELECT TRIM(SUBSTRING_INDEX(yourcolumn,',',1)), TRIM(SUBSTRING_INDEX(yourcolumn,',',-1)) FROM yourtable
CREATE FUNCTION [dbo].[SplitString]
(
#RowData nvarchar(2000),
#SplitOn nvarchar(5)
)
RETURNS #RtnValue table
(
--Id int identity(1,1),
Data nvarchar(100)
)
AS
BEGIN
Declare #Cnt int
Set #Cnt = 1
While (Charindex(#SplitOn,#RowData)>0)
Begin
Insert Into #RtnValue (data)
Select
Data = ltrim(rtrim(Substring(#RowData,1,Charindex(#SplitOn,#RowData)-1)))
Set #RowData = Substring(#RowData,Charindex(#SplitOn,#RowData)+1,len(#RowData))
Set #Cnt = #Cnt + 1
End
Insert Into #RtnValue (data)
Select Data = ltrim(rtrim(#RowData))
Return
END
I have a table call lp_upload and it contain plate number of car and other related information:
CREATE TABLE `lp_upload` (
`date` date NULL ,
`plate` char(10) NULL ,
`site` int NULL ,
`dateid` char(20) NULL
)
;
this table is getting information from a traffic camera. however, sometime letter in the plate is not recognized and it will be replace by $. So if a plate is really abc123, but the camera didnt recognized c and 1, it will be ac$$23 that get enter into the table.
im suppose to make it so when a new plate is entered and 6 of its letters match an existing plate, it will become that plate. EX: 123$5678 is entered and 12345678 already exist, then 123$5678 will be replace by 12345678.
so i first wrote a match function:
CREATE DEFINER = CURRENT_USER FUNCTION `matchingfun`(`str1` char(10),`str2` char(10))
RETURNS int
BEGIN
DECLARE myindex int DEFAULT 0;
DECLARE count int DEFAULT 0;
DECLARE maxlength int;
SET maxlength = length(str1);
for_loop: LOOP
SET myindex = myindex + 1;
IF maxlength < myindex then
RETURN 0;
END IF;
IF SUBSTRING(str1,myindex,1)= SUBSTRING(str2,myindex,1)then
SET count = count +1;
END IF;
IF count > 6 then
RETURN 1;
END IF;
IF SUBSTRING(str1,myindex,1)!= SUBSTRING(str2,myindex,1) and SUBSTRING(str1,myindex,1)!= '$' and SUBSTRING(str2,myindex,1)!= '$'then
RETRUN 0;
END IF;
END LOOP for_loop;
RETURN 0;
END
and I added a trigger function to the table
CREATE TRIGGER `trigger1` AFTER INSERT ON `lpr_opt_upload`
BEGIN
declare old_site_id int;
declare old_text char(10);
select lpr_text into old_text from lpr_opt_upload where matchingfun(new.lpr_text, lpr_text) = 1;
if(old_text is not null) then
set new.lpr_text = old_text;
end if;
END
when i run this, the database crashes. can you help fix this problem or suggest a better way to do this. thanks.
I suspect that the problem you're running into is multiple matches. For example, if you have abcd01234 and abcde1234 in the database and attempt to insert abcd$1234, you'll get an error.
Now, I'm going to assume that this application is supposed to match OCR'd license plates from speed cameras or red-light cameras in order to facilitate ticketing of the vehicle owner. If that's the case, then you want to err on the side of caution and not have the system automatically try to pick from multiple candidates and instead have a real human look at the result and confirm the plate number.
So, operating on that assumption:
DELIMITER //
CREATE TRIGGER `attempt_match_existing_plate`
BEFORE INSERT
ON `lp_upload`
FOR EACH ROW BEGIN
DECLARE exist_plate CHAR(10);
DECLARE rowcount INT;
SELECT COUNT(*), plate INTO rowcount, exist_plate FROM lp_upload WHERE platematch(NEW.plate, plate) = 1;
IF (1 = rowcount) AND (exist_plate IS NOT NULL) THEN
SET NEW.plate = exist_plate;
END IF;
END
//
DELIMITER ;
DELIMITER //
CREATE DEFINER = CURRENT_USER
FUNCTION `platematch`(`plate_new` char(10), `plate_exist` char(10))
RETURNS INT
BEGIN
DECLARE myindex INT DEFAULT 0;
DECLARE match_count INT DEFAULT 0;
DECLARE maxlength INT;
SET maxlength = length(plate_new);
for_loop: LOOP
SET myindex = myindex + 1;
IF maxlength < myindex THEN
RETURN 0;
END IF;
IF SUBSTRING(plate_new, myindex, 1) = SUBSTRING(plate_exist, myindex, 1)
THEN
SET match_count = match_count +1;
END IF;
IF match_count >= 6 THEN
RETURN 1;
END IF;
IF SUBSTRING(plate_new, myindex, 1) != SUBSTRING(plate_exist, myindex, 1)
AND SUBSTRING(plate_new, myindex, 1) != '$'
AND SUBSTRING(plate_exist, myindex, 1) != '$'
THEN
RETURN 0;
END IF;
END LOOP for_loop;
RETURN 0;
END
//
DELIMITER ;
In the scenario I described above, abcd$1234 will be inserted into the database as-is instead of just being matched to one of multiple potential results automatically.
I have a table which contains header information for transactions. The transactions belong to different projects.
In the header I have columns:
rhguid - uniqueidentifier
rhserial - int
rh_projectID - int
First I insert the row (there's more columns)
Then I calculate the serial number for that project:
update responseheader
set rhSerial = 1 + (select isnull(max(rhSerial), 0)
from responseheader
where (rhstatus = 0) AND (rh_projectID = 1234))
where
(rhGUID = <preassignedGUID>);
However when there are many transactions happening at the same time for a project I am finding duplicate rhserial values.
I'm doing this in classic ASP with SQL Server 2008.
Is there a better way?
From your example, it doesn't look like you're using a transaction. My guess is that the SELECT portion of the statement is running as READ UNCOMMITTED, otherwise you would not see duplicates. There are ways to start transactions with ADO, but I prefer using stored procedures instead.
Try implementing something like this:
CREATE PROC dbo.ResponseHeader_Insert
<more data to insert>,
#ProjectID INT,
#Status SMALLINT
as
insert responseheader (column names here)
select <param values here>, isnull(max(rhSerial), 0) + 1
from responseheader
where (rhstatus = #Status) AND (rh_projectID = #ProjectID))
If this doesn't work for ya, try creating sequence tables (one for each sequence).
create table <tablename> (
SeqID int identity(1,1) primary key,
SeqVal varchar(1)
)
Create a procedure to get the next identity:
create procedure GetNewSeqVal_<tablename>
as
begin
declare #NewSeqValue int
set NOCOUNT ON
insert into <tablename> (SeqVal) values ('a')
set #NewSeqValue = scope_identity()
delete from <tablename> WITH (READPAST)
return #NewSeqValue
end
If there are too many sequence tables that need to be created or you want to create sequences on the fly, try this approach:
Create table AllSequences (
SeqName nvarchar(255) primary key, -- name of the sequence
Seed int not null default(1), -- seed value
Incr int not null default(1), -- incremental
Currval int
)
Go
create procedure usp_CreateNewSeq
#SeqName nvarchar(255),
#seed int = 0,
#incr int = 1
as
begin
declare #currval int
if exists (
select 1 from AllSequences
where SeqName = #SeqName )
begin
print 'Sequence already exists.'
return 1
end
if #seed is null set #seed = 1
if #incr is null set #incr = 1
set #currval = #seed
insert into AllSequences (SeqName, Seed, Incr, CurrVal)
values (#SeqName, #Seed, #Incr, #CurrVal)
end
go
create procedure usp_GetNewSeqVal
#SeqName nvarchar(255)
as
begin
declare #NewSeqVal int
set NOCOUNT ON
update AllSequences
set #NewSeqVal = CurrVal = CurrVal+Incr
where SeqName = #SeqName
if ##rowcount = 0 begin
print 'Sequence does not exist'
return
end
return #NewSeqVal
end
go
I have a table with two columns: price (int) and price_display (varchar).
price is the actual numerical price, e.g. "9990"
price_display is the visual representation, e.g. "$9.99" or "9.99Fr"
I've been able to confirm the two columns match via regexp:
price_display not regexp
format(price/1000, 2)
But in the case of a mismatch, I want to extract the value from the price_display column and set it into the price column, all within the context of an update statement. I've not been able to figure out how.
Thanks.
This function does the job of only returning the digits 0-9 from the string, which does the job nicely to solve your issue, regardless of what prefixes or postfixes you have.
http://www.artfulsoftware.com/infotree/queries.php?&bw=1280#815
Copied here for reference:
SET GLOBAL log_bin_trust_function_creators=1;
DROP FUNCTION IF EXISTS digits;
DELIMITER |
CREATE FUNCTION digits( str CHAR(32) ) RETURNS CHAR(32)
BEGIN
DECLARE i, len SMALLINT DEFAULT 1;
DECLARE ret CHAR(32) DEFAULT '';
DECLARE c CHAR(1);
IF str IS NULL
THEN
RETURN "";
END IF;
SET len = CHAR_LENGTH( str );
REPEAT
BEGIN
SET c = MID( str, i, 1 );
IF c BETWEEN '0' AND '9' THEN
SET ret=CONCAT(ret,c);
END IF;
SET i = i + 1;
END;
UNTIL i > len END REPEAT;
RETURN ret;
END |
DELIMITER ;
SELECT digits('$10.00Fr');
#returns 1000
One approach would be to use REPLACE() function:
UPDATE my_table
SET price = replace(replace(replace(price_display,'Fr',''),'$',''),'.','')
WHERE price_display not regexp format(price/1000, 2);
This works for the examples data you gave:
'$9.99'
'9.99Fr'
Both result in 999 in my test. With an update like this, it's important to be sure to back up the database first, and be cognizant of the formats of the items. You can see all the "baddies" by doing this query:
SELECT DISTINCT price_display
FROM my_table
WHERE price_display not regexp format(price/1000, 2)
ORDER BY price_display;
For me CASTING the field did the trick:
CAST( price AS UNSIGNED ) // For positive integer
CAST( price AS SIGNED ) // For negative and positive integer
IF(CAST(price AS UNSIGNED)=0,REVERSE(CAST(REVERSE(price) AS UNSIGNED)),CAST(price AS UNSIGNED)) // Fix when price starts with something else then a digit
For more details see:
https://dev.mysql.com/doc/refman/5.0/en/cast-functions.html
This is a "coding horror", relational database schemas should NOT be written like this!
Your having to write complex and unnecessary code to validate the data.
Try something like this:
SELECT CONCAT('$',(price/1000)) AS Price FROM ...
In addition, you can use a float, double or real instead of a integer.
If you need to store currency data, you might consider adding a currency field or use the systems locale functions to display it in the correct format.
I create a procedure that detect the first number in a string and return this, if not return 0.
DROP FUNCTION IF EXISTS extractNumber;
DELIMITER //
CREATE FUNCTION extractNumber (string1 VARCHAR(255)) RETURNS INT(11)
BEGIN
DECLARE position, result, longitude INT(11) DEFAULT 0;
DECLARE string2 VARCHAR(255);
SET longitude = LENGTH(string1);
SET result = CONVERT(string1, SIGNED);
IF result = 0 THEN
IF string1 REGEXP('[0-9]') THEN
SET position = 2;
checkString:WHILE position <= longitude DO
SET string2 = SUBSTR(string1 FROM position);
IF CONVERT(string2, SIGNED) != 0 THEN
SET result = CONVERT(string2, SIGNED);
LEAVE checkString;
END IF;
SET position = position + 1;
END WHILE;
END IF;
END IF;
RETURN result;
END //
DELIMITER ;
Return last number from the string:
CREATE FUNCTION getLastNumber(str VARCHAR(255)) RETURNS INT(11)
DELIMETER //
BEGIN
DECLARE last_number, str_length, position INT(11) DEFAULT 0;
DECLARE temp_char VARCHAR(1);
DECLARE temp_char_before VARCHAR(1);
IF str IS NULL THEN
RETURN -1;
END IF;
SET str_length = LENGTH(str);
WHILE position <= str_length DO
SET temp_char = MID(str, position, 1);
IF position > 0 THEN
SET temp_char_before = MID(str, position - 1, 1);
END IF;
IF temp_char BETWEEN '0' AND '9' THEN
SET last_number = last_number * 10 + temp_char;
END IF;
IF (temp_char_before NOT BETWEEN '0' AND '9') AND
(temp_char BETWEEN '0' AND '9') THEN
SET last_number = temp_char;
END IF;
SET position = position + 1;
END WHILE;
RETURN last_number;
END//
DELIMETER;
Then call this functions:
select getLastNumber("ssss111www222w");
print 222
select getLastNumber("ssss111www222www3332");
print 3332