Data comes in via a zip file in a group of csv files which are used to populate holding tables. Some can include new data or updates of existing entries. The holding tables are used for intermediate processing then are used for import to the working tables.
In an effort to speed up the process I have been writing stored procedures to update the working tables. On my laptop one such file with only 730 records is taking about a minute to do its stuff.
I did consider making the 'insert ... on duplicate key update' into a prepared statement inside the SP but some of the other tables have many more fields and I could find no good guide on how to write such a complex one.
Here is my stored procedure:
DROP PROCEDURE IF EXISTS impLuRacodes;
DELIMITER $$
CREATE DEFINER=`root`#`localhost` PROCEDURE impLuRacodes()
BEGIN
DECLARE done INT DEFAULT 0;
DECLARE v_ATID varchar(10);
DECLARE v_Code varchar(10);
DECLARE v_IssType varchar(10);
DECLARE v_Category varchar(10);
DECLARE v_CNumber int(4);
DECLARE v_CDesc varchar(255);
DECLARE v_ColCode varchar(6);
DECLARE v_LLike int(3);
DECLARE v_LifeS int(3);
DECLARE v_PropS int(3);
DECLARE v_BusS int(3);
DECLARE c_1 CURSOR FOR
SELECT `ATID`, `Code`, `IssType`, `Category`, `CNumber`, `CDesc`, `ColCode`, `LLike`, `LifeS`, `PropS`, `BusS`
FROM lu_racodes_temp;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done=1;
OPEN c_1;
REPEAT
FETCH c_1 INTO v_ATID, v_Code, v_IssType, v_Category, v_CNumber,
v_CDesc, v_ColCode, v_LLike, v_LifeS, v_PropS, v_BusS;
INSERT INTO lu_racodes (ATID, `Code`, IssType, Category, CNumber,
CDesc, ColCode, LLike, LifeS, PropS, BusS)
VALUES(v_ATID, v_Code, v_IssType, v_Category, v_CNumber, v_CDesc,
v_ColCode, v_LLike, v_LifeS, v_PropS, v_BusS)
ON DUPLICATE KEY UPDATE
ATID= v_ATID, `Code`= v_Code, IssType= v_IssType, Category= v_Category,
CNumber= v_CNumber, CDesc= v_CDesc, ColCode= v_ColCode, LLike=v_LLike,
LifeS= v_LifeS, PropS= v_PropS, BusS= v_BusS;
UNTIL done END REPEAT;
CLOSE c_1;
END $$
Alternatively, is it possible to put the Select inside the 'Insert ... On Duplicate Key' instead of using a cursor - again something I have had no joy in finding a clear answer to.
Thanks to the guidance I rewrote the code to put the select inside and execution time dropped to 0.156 seconds for a first population. I was concerned about the syntax of the update section with the reference to VALUES as it was a Select statement bringing in the data but changing a couple of rows in the source table did get updated and still with only a 0.235 second time. So here is the code:
DROP PROCEDURE IF EXISTS impLuRacodes;
DELIMITER $$
CREATE DEFINER=`root`#`localhost` PROCEDURE impLuRacodes()
BEGIN
INSERT INTO lu_racodes (`ATID`, `Code`, `IssType`, `Category`, `CNumber`, `CDesc`, `ColCode`, `LLike`, `LifeS`, `PropS`, `BusS`) SELECT `ATID`, `Code`, `IssType`, `Category`, `CNumber`, `CDesc`, `ColCode`, `LLike`, `LifeS`, `PropS`, `BusS` FROM lu_racodes_temp ON DUPLICATE KEY UPDATE `ATID`= VALUES(`ATID`), `Code`= VALUES(`Code`), `IssType`= VALUES(`IssType`), `Category`= VALUES(`Category`), `CNumber`= VALUES(`CNumber`), `CDesc`= VALUES(`CDesc`), `ColCode`= VALUES(`ColCode`), `LLike`= VALUES(`LLike`), `LifeS`= VALUES(`LifeS`), `PropS`= VALUES(`PropS`), `BusS`= VALUES(`BusS`);
END
$$
DELIMITER ;
Related
I want to add the salary from two tables in stored procedure on the basis of id column:
DDl:
create table salary1 (id varchar(20), salary varchar(20));
create table salary2 (id varchar(20), salary varchar(20));
DML:
insert into salary1 values('1', '100');
insert into salary1 values('2', '200');
insert into salary2 values('1', '10');
insert into salary2 values('2', '10');
Database: mysql
Output should like this:
id total_sal
1 110
2 210
My stored procedure look like:
CREATE PROCEDURE totalSal()
BEGIN
DECLARE tbl1_id varchar(30);
DECLARE tbl1_sal varchar(30);
DECLARE tbl2_id varchar(30);
DECLARE tbl2_sal varchar(30);
DECLARE total_sal varchar(30);
DECLARE c1 CURSOR FOR SELECT * FROM salary1;
DECLARE c2 CURSOR FOR SELECT * FROM salary2;
-- Open first cursor
OPEN c1;
LOOP
FETCH c1 INTO tbl1_id, tbl1_sal;
-- Open second cursor
OPEN c2;
LOOP
FETCH c2 INTO tbl2_id, tbl2_sal;
IF tbl1_id = tbl2_id THEN
set total_sal := tbl1_sal + tbl2_sal;
ELSE
set total_sal := tbl_sal;
END IF;
END LOOP;
CLOSE c2;
END LOOP;
CLOSE c1;
end $$
It got's successfully compiled, but when i am running the procedure i am getting the below error:
ERROR 1329 (02000): No data - zero rows fetched, selected, or processed
I have also used the DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1; in my procedure. but still my problem is unresolved.
If someone can solve this problem in oracle, that would also help me.
Note : I cannot perform join operation on these tables. Because of a few performance issues.
Thanks in advance !!!
Solution 1:
Using collection and only one iteration of 2 loop
You should consider to fix your performance issue on join. Performing loop is slower than a set base approach in most case.
If I follow your logic, what you realy want is to loop trough all the salary2 table for each salary1 row in order to found the right ID => millions of loop.
You can consider doing 2 separated loop and store data inside and indexed array. ( the key will be the tlb1_id).
If the key exist : sum the salary values, if not exist insert it inside the array.
At the end of the procedure, just select the array as table.
Solution 2:
Using a join on integer indexed columns
you can add a new integer column on each table
Populate this column with the casted value of the ID column
Add an index on these columns on each tables
After that you will be able to perform a join
Have a look at this fiddle http://sqlfiddle.com/#!9/c445de/1 , it can be time consuming to perform theses step and disk space consumuming to add a new columns and indexes but the join operation may be faster than before.
You can do something like this... I have moved the second cursor inside the loop so that it only goes over the id's from table 1. This should help the logic for the procedure but still I would recommend trying to figure out how to fix the join to get the results as that seems like an easier way and should be much faster if done correctly.
CREATE PROCEDURE totalSal()
BEGIN
DECLARE tbl1_id varchar(30);
DECLARE tbl1_sal varchar(30);
DECLARE tbl2_id varchar(30);
DECLARE tbl2_sal varchar(30);
DECLARE total_sal varchar(30);
DECLARE c1 CURSOR FOR SELECT * FROM salary1;
-- Open first cursor
OPEN c1;
LOOP
FETCH c1 INTO tbl1_id, tbl1_sal;
SELECT COUNT(*) INTO v_rowcount FROM salary2 WHERE id = tbl1_id;
IF v_rowcount > 0 THEN
Begin
DECLARE c2 CURSOR FOR SELECT * FROM salary2 WHERE id = tbl1_id;
-- Open second cursor
OPEN c2;
LOOP
FETCH c2 INTO tbl2_id, tbl2_sal;
IF tbl1_id = tbl2_id THEN
set total_sal := tbl1_sal + tbl2_sal;
ELSE
set total_sal := tbl_sal;
END IF;
END LOOP;
CLOSE c2;
END IF;
END
END LOOP;
CLOSE c1;
end $$
Well you asked for an answer without JOIN, but that seemed arbitrary, so here's an answer with JOIN.
SELECT
sums1.id
, S1Sum + S2Sum AS SalarySum
FROM (SELECT id, SUM(CAST(salary AS int)) AS S1Sum
FROM salary1
GROUP BY id) sums1
JOIN (SELECT id, SUM(CAST(salary AS int)) AS S2Sum
FROM salary2
GROUP BY id) sums2
ON sums1.id = sums2.id
I am guessing your performance is bad because all of your columns are varchar when they should be int or numeric. But we don't have much to go on so hopefully this helps you come to a solid solution.
Also the post was edited to add both MySQL and Oracle tags so it's difficult to determine what the syntax should be...
I tried to replace heavy Java method which runs multiple requests to the DB with stored SQL procedure.
It's doing its work but I expected much higher performance improvement.
The logic of the procedure(as well as Java method):
Get list of IDs from table1(purpose)
Iterate the list and get average value of a field from table2(record) for each id
Return list
of pairs id/average_value
Are there any efficiency issues in the procedure?
DROP PROCEDURE IF EXISTS test1.getGeneralAverage;
CREATE DEFINER=root#localhost PROCEDURE getGeneralAverage()
BEGIN
DECLARE p_id BIGINT(20);
DECLARE exit_loop BOOLEAN;
DECLARE cur CURSOR FOR
SELECT purpose_id FROM purpose
WHERE purpose.type = 'GENERAL'
AND (SELECT COUNT(*) > 0 FROM record
WHERE record.purpose_id=purpose.purpose_id) is true;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET exit_loop = TRUE;
CREATE TEMPORARY TABLE IF NOT EXISTS general_average
(id BIGINT(20), average DOUBLE) ENGINE=memory;
TRUNCATE TABLE general_average;
OPEN cur;
average_loop: LOOP
FETCH cur INTO p_id;
INSERT INTO test1.general_average (id, average)
VALUES (p_id, (SELECT AVG(amount) FROM record
WHERE record.purpose_id=p_id));
IF exit_loop THEN
CLOSE cur;
LEAVE average_loop;
END IF;
END LOOP average_loop;
INSERT INTO test1.general_average (id, average)
VALUES (0,
(select avg(amount) from record where purpose_type='CUSTOM'));
SELECT * FROM general_average;
END
Several patterns to change...
Try to avoid CURSORs; usually the entire operation can be done with a single SQL statement. That will be much faster.
INSERT ... VALUES (0, ( SELECT ... ) ) --> INSERT ... SELECT 0, ...
Don't use a TEMP table when you can simply deliver the results. In your case, you may need a UNION ALL to deliver the two chunks at once.
I did a trigger in mysql to shoot alerts always an input value was less than the set value. But now I need it is done in SQL SERVER.
I would be grateful if someone could help me transform mysql trigger to a SQL Server trigger.
Thanks to all at once.
My trigger is:
DELIMITER $$
create TRIGGER alert
AFTER INSERT ON records
FOR EACH ROW
begin
Set #comp=0;
Set #tempmax=0;
Set #tempmin=0;
select lim_inf_temp into #tempmin from sensores where idSensor=NEW.idSensor;
Set #maxidAlarme=0;
if (CAST(NEW.Temperatura AS UNSIGNED)<#tempmin) then
SELECT MAX(idAlarme) into #maxidAlarme FROM alarmes;
SET #maxidAlarme=#maxidAlarme+1;
INSERT INTO alarmes(idAlarme,descricao_alarme, idRegisto) VALUES (#maxidAlarme,"inserted below the normal temperature",New.idRegisto);
INSERT INTO sensores_tem_alarmes(idSensor,idAlarme,dataAlarme) VALUES (NEW.idSensor,#maxidAlarme,NOW());
set #comp=+1;
end if;
set #id_sensores_em_alerta=1;
SELECT MAX(id_sensores_em_alerta) into #id_sensores_em_alerta FROM sensores_em_alerta;
INSERT INTO sensores_em_alerta(id_sensores_em_alerta, idSensor, idAlarme, data_registo, numerosensoresdisparados) VALUES (id_sensores_em_alerta,NEW.idSensor, #maxidAlarme, NOW(), #comp);
end $$;
DELIMITER ;
I've tried to make the trigger in SQL Server, but as the script is different and I'm getting many difficulties to do the right way.
My attempt that was not going at all well:
CREATE TRIGGER Alert ON registos AFTER INSERT AS
BEGIN
DECLARE #comp decimal= 0
DECLARE #tempmax decimal= 0
DECLARE #tempmin decimal= 0
DECLARE #current_max_idAlarme int = (SELECT MAX(IdAlarme) FROM alarmes)
-- Insert into alarmes from the inserted rows if temperature less than tempmin
INSERT alarmes (IdAlarme, descricao_alarme, idRegisto)
SELECT
ROW_NUMBER() OVER (ORDER BY i.idRegisto) + #current_max_idAlarme,
'temp Error',
i.idRegisto
FROM
inserted AS i
WHERE
i.Temperatura < #tempmin
END
But dont do anything.
Dont create data on table alarmes :S
Does anyone could help me please. I would be eternally grateful.
Many Greetings and thank you all.
First of all, MSSQL doesn't have the option FOR EACH ROW, so it treats multiple inserted rows at once as a set. You will therefore have to insert the values into a table variable.
Unfortunately I do not know much MySQL actually, but I believe this is a starting point?
CREATE TRIGGER ALERT
ON records
AFTER INSERT
AS
BEGIN
DECLARE #comp INT;
DECLARE #tempmax INT;
DECLARE TABLE #tempmin (tempmin INT);
INSERT INTO #tempmin
SELECT s.lim_inf_temp FROM sensores s WHERE s.idSensor IN (inserted.idSensor);
--rest of the code
I'm going to post this code against my better judgement - redesign the tables is better than this hack.
This uses a ROW_number() to virtualise a surrogate identity key for the alarmes table. This is a 'bad plan' (tm).
Also the answer is partial - it doesn't do everything your question asked for -- I hope it gets your further along the road. Use it as a guide for how to interact with the virtual INSERTED table. Good luck
CREATE TRIGGER Alert ON records AFTER INSERT AS
BEGIN
DECLARE #comp INT = 0
DECLARE #tempmax INT = 0
DECLARE #tempmin INT = 0
-- get the max current id.
-- note that this is EXTREMELY unsafe as if two pieces of code are executing
-- at the same time then you *will* end up with key conflicts.
-- you could use SERIALIZABLE.... but better would be to redisn the schema
DECLARE #current_max_idAlarme = (SELECT MAX(IdAlarme) FROM alarmes)
-- Insert into alarmes from the inserted rows if temperature less than tempmin
INSERT alarmes (IdAlarme, descricao_alarme, idRegisto)
SELECT
ROW_NUMBER() OVER (ORDER BY i.idRegisto) + #current_max_idAlarme,
'temp Error',
i.idRegisto
FROM
inserted AS i
WHERE
i.Temperatura < #tempmin
END
I am in the process of converting a SQL Server 2005 database to MySQL and having problems with a Stored procedure. I'm new to MySQL stored procedures so I'm sure it is a problem with my conversion but I'm not seeing it.
The stored procedure is supposed to generate a temporary table which is used to populate a Data Grid View in a vb.net application. However, I'm getting the error "Data No Data - Zero rows fetched, selected or processed.". Seems simple enough but the select procedure in the stored procedure will get data if I just run it as a query which is why I don't understand why the error.
I'm really hoping someone can tell me why because I have several hundred stored procedures to convert and I'm having this problem on the very first one.
Here's the Stored Procedure:
DELIMITER $$
DROP PROCEDURE IF EXISTS `usp_get_unassigned_media`$$
CREATE DEFINER=`showxx`#`67.111.11.110` PROCEDURE `usp_get_unassigned_media`()
BEGIN
/* GET CURSOR WITH LOCAL LOCATIONS */
DECLARE intKey INT;
DECLARE dteDateInserted DATETIME;
DECLARE vchIdField VARCHAR(200);
DECLARE vchValueField VARCHAR(200);
DECLARE intLastKey INT;
/*TAKE OUT SPECIFIC PLAYLIST ITEMS IF TOO SLOW*/
DECLARE csrMediaToBeAssigned CURSOR FOR
SELECT
`media`.`key` AS `key`,
`media`.`date_inserted` AS `date_inserted`,
`media_detail_types`.`id` AS `id`,
`media_details`.`value` AS `value`
FROM (`media`
LEFT JOIN (`media_detail_types`
JOIN `media_details`
ON ((`media_detail_types`.`key` = `media_details`.`detail_key`)))
ON ((`media_details`.`media_key` = `media`.`key`)))
WHERE ((`media`.`is_assigned` = 0)
AND ((`media_detail_types`.`id` = 'Volume Name')
OR (`media_detail_types`.`id` = 'Drive Id')))
ORDER BY `media`.`key`,`media`.`date_inserted`,`media_detail_types`.`id`;
OPEN csrMediaToBeAssigned;
DROP TEMPORARY TABLE IF EXISTS temp_unassigned_media;
CREATE TEMPORARY TABLE temp_unnassigned_media
(temp_key INT, DateInserted DATETIME, IdField VARCHAR(200), ValueField VARCHAR (200))
ENGINE=MEMORY;
SET intLastKey = 0;
/*--GET FIRST RECORD */
FETCH FROM csrMediaToBeAssigned
INTO intKey, dteDateInserted, vchIdField, vchValueField;
/*--LOOP THROUGH CURSOR */
WHILE intLastKey = 0 DO
/*--DATA SHOULD BE IN DRIVE ID THEN VOLUME NAME */
INSERT INTO temp_unnassigned_media
VALUES (intKey, dteDateInserted, vchValueField, '');
FETCH NEXT FROM csrMediaToBeAssigned
INTO intKey, dteDateInserted, vchIdField, vchValueField;
UPDATE temp_unnassigned_media
SET IdField = vchValueField
WHERE temp_key = temp_key;
FETCH NEXT FROM csrMediaToBeAssigned
INTO intKey, dteDateInserted, vchIdField, vchValueField;
END WHILE;
SELECT *
FROM temp_unnassigned_media
ORDER BY date_inserted;
CLOSE csrMediaToBeAssigned;
/*DEALLOCATE csrMediaToBeAssigned */
/*DROP TABLE #temp_unnassigned_media */
END$$
DELIMITER ;
You never hit a condition where that WHILE loop will exit; you initialize intLastKey variable, but it never changes, so you fetch through the entire resultset. The exception is thrown when you fetch again, after the last record.
The normative pattern is to declare a CONTINUE HANDLER, which MySQL will execute when the NOT FOUND condition is triggered. The handler is normally used to set a variable, which you can then test, so you know when to exit the loop.
In your case, it looks like just adding this line, after your DECLARE CURSOR statement and before the OPEN statement, would be sufficient:
DECLARE CONTINUE HANDLER FOR NOT FOUND SET intLastKey = 1;
Am new to Stored procedures.I wrote the stored procedure to copy table from one dtabase to another database.On executing my stored procedures everytime My datas are added in the destination table .My row counts was increasing on every execution.
Please help to resolve the issue.Hope the problem In the loops.
My SP is:
--exec mall
alter procedure mall
as
begin
declare #mallid int
declare #mallname nvarchar(40)
declare #mallstatus nvarchar(40)
declare #malludsuomid nchar(2)
declare #malludsassetcode nvarchar(6)
declare #malludsassettype nvarchar(15)
declare #malludsremarks nvarchar(max)
declare #malludsdwdb int
declare #mallsecterr int
declare #mallassetid int
declare #secterr int
declare #Maxmallid int
declare #mallentityid int
Select #mallentityid = customtable.Bord_TableId From CRM.dbo.Custom_Tables as customtable With (NoLock) Where Upper(Bord_Caption) = Upper('Mall') And Bord_Deleted Is Null
DECLARE cur_address CURSOR FOR
SELECT
udsasset.Asset_ID,udsasset.Asset_Name,udsasset.Asset_Status,udsasset.UOM_ID, udsasset.Asset_Code,udsasset.Asset_Type,udsasset.Remarks,udsasset.DW_Key_Source_DB --,crmterr.TPro_SecTerr
from
CMA_UDS.dbo.Dim_Asset as udsasset
OPEN cur_address
FETCH NEXT FROM cur_address INTO #mallid,#mallname,#mallstatus,#malludsuomid,#malludsassetcode,#malludsassettype,#malludsremarks,#malludsdwdb --,#mallsecterr
WHILE ##FETCH_STATUS = 0
BEGIN
if not exists (select crmmall.mall_MallID from CRM.dbo.Mall as crmmall where crmmall.mall_MallID = #mallid)
begin
exec #Maxmallid = CRM.dbo.crm_next_id #Table_Id=#mallentityid
insert into
CRM.dbo.Mall
(mall_MallID,mall_Name,mall_Status,mall_uds_UOMID,mall_uds_asset_code,mall_uds_asset_type,
mall_uds_remarks,mall_uds_dw_db,mall_CreatedBy,mall_CreatedDate,mall_Secterr,mall_AMOSUploaded,mall_asset_id)
values(#Maxmallid,#mallname,#mallstatus,#malludsuomid,#malludsassetcode,#malludsassettype,#malludsremarks,#malludsdwdb,1,GETDATE(),
#mallsecterr,GETDATE(),#mallid)
end
else
begin
update
CRM.dbo.Mall
set
mall_asset_id=#mallid,mall_Name = #mallname,mall_Status=#mallstatus,mall_uds_UOMID =#malludsuomid,mall_uds_asset_code=#malludsassetcode,
mall_uds_asset_type=#malludsassettype,mall_uds_remarks=#malludsremarks,mall_uds_dw_db=#malludsdwdb,mall_UpdatedBy=1,
mall_UpdatedDate=GETDATE(),mall_Secterr=#mallsecterr,mall_AMOSUploaded=GETDATE()
where
mall_MallID=#mallid
end
FETCH NEXT FROM cur_address INTO #mallid,#mallname,#mallstatus,#malludsuomid,#malludsassetcode,#malludsassettype,#malludsremarks,#malludsdwdb--,#mallsecterr
end
CLOSE cur_address
DEALLOCATE cur_address
End
Why are you inserting crm_next_id as the value in mall_MallID, but using that same id to compare with #mallid to see if the record is already inserted? For example, if you have id 5, and you insert a new record with id 150, it's not going to see that the record is already inserted when you run the SP again. Next run, it will add record with id 151, then 152, and so forth forever. You shouldn't use the same field as both an auto-increment identity and a foreign key reference at the same time...
You either need to use the same #mallid when you insert the new records so they match, or after you generate a new id and insert into the table, update the original record CMA_UDS.dbo.Dim_Asset to have Asset_ID = #mallid so they are linked up properly. Which method you use depends on the meanings of those id's and what constraints you have in your particular application.