Background - I have a DB created from a single large flat file. Instead of creating a single large table with 106 columns. I created a "columns" table which stores the column names and the id of the table that holds that data, plus 106 other tables to store the data for each column. Since not all the records have data in all columns, I thought this might be a more efficient way to load the data (maybe a bad idea).
The difficulty with this was rebuilding a single record from this structure. To facilitate this I created the following procedure:
DROP PROCEDURE IF EXISTS `col_val`;
delimiter $$
CREATE PROCEDURE `col_val`(IN id INT)
BEGIN
DROP TEMPORARY TABLE IF EXISTS tmp_record;
CREATE TEMPORARY TABLE tmp_record (id INT(11), val varchar(100)) ENGINE=MEMORY;
SET #ctr = 1;
SET #valsql = '';
WHILE (#ctr < 107) DO
SET #valsql = CONCAT('INSERT INTO tmp_record SELECT ',#ctr,', value FROM col',#ctr,' WHERE recordID = ',#id,';');
PREPARE s1 FROM #valsql;
EXECUTE s1;
DEALLOCATE PREPARE s1;
SET #ctr = #ctr+1;
END WHILE;
END$$
DELIMITER ;
Then I use the following SQL where the stored procedure parameter is the id of the record I want.
CALL col_val(10);
SELECT c.`name`, t.`val`
FROM `columns` c INNER JOIN tmp_record t ON c.ID = t.id
Problem - The first time I run this it works great. However, each subsequent run returns the exact same record even though the parameter is changed. How does this persist even when the stored procedure should be dropping and re-creating the temp table?
I might be re-thinking the whole design and going back to a single table, but the problem illustrates something I would like to understand.
Unsure if it matters but I'm running MySQL 5.6 (64 bit) on Windows 7 and executing the SQL via MySQL Workbench v5.2.47 CE.
Thanks,
In MySQL stored procedures, don't put an # symbol in front of local variables (input parameters or locally declared variables). The #id you used refers to a user variable, which is kind of like a global variable for the session you're invoking the procedure from.
In other words, #id is a different variable from id.
That's the explanation of the immediate problem you're having. However, I would not design the tables as you have done.
Since not all the records have data in all columns, I thought this might be a more efficient way to load the data
I recommend using a conventional single table, and use NULL to signify missing data.
Related
I have a Stored Procedure in mySQL that takes a subset of data from a table and performs some analysis on that subset within a temp table. Here is my code:
CREATE PROCEDURE GetPortfolioStats
(
InIdx_i INT,
InStart_i INT,
InEnd_i INT
)
BEGIN
DECLARE myLimit INT;
DECLARE myOffset INT;
SET myLimit = InEnd_i - InStart_i + 1;
SET myOffset = InStart_i - 1;
CREATE TEMPORARY TABLE IF NOT EXISTS myTmpTable AS (SELECT * FROM Leases WHERE Portfolio_i = InIdx_i ORDER BY Index_i LIMIT myLimit OFFSET myOffset);
SET #Additive_i := (SELECT COUNT(Index_i) FROM myTmpTable WHERE ReviewType_vc = 'Additive');
DROP TABLE myTmpTable;
SELECT #Additive_i;
END; GO
This works fine. However, the problem I have is that this is a multi-threaded application and when multiple threads are calling this stored proc, they start sharing the same temp table, which messes up the Stats I'm trying to compile.
Is there a way to either apply a unique table name to each call of the stored proc or limit the scope of the temp table to just that instance of the stored proc?
To answer the specific question: the easiest solution would be to use a different database connection per thread because temporary tables are session (connection) specific:
You can use the TEMPORARY keyword when creating a table. A TEMPORARY table is visible only to the current session, and is dropped automatically when the session is closed. This means that two different sessions can use the same temporary table name without conflicting with each other or with an existing non-TEMPORARY table of the same name.
However, after checking out the actual code, I would suggest not to use a temporary table at all, use a single query with a subquery:
SELECT COUNT(Index_i)
FROM
(SELECT Index_i, ReviewType_vc
FROM Leases
WHERE Portfolio_i = InIdx_i
ORDER BY Index_i
LIMIT myLimit OFFSET myOffset) t
WHERE ReviewType_vc = 'Additive'
I have a stored procedure, internally I want to call another procedure that returns a record set, how do I get an navigate the record set returned by the stored procedure via the 'CALL' ?
[edit] I've been trying to use a TEMPORARY TABLE as suggested, but having problems:
DROP TEMPORARY TABLE IF EXISTS tbl_HeadOfDepts;
CREATE TEMPORARY TABLE tbl_HeadOfDepts (biDept_id tinyint(4))
INSERT INTO tbl_HeadOfDepts CALL rsHeadOfAnyDepartments(vcCompKey, biWho_id);
I need to use CALL because 'rsHeadOfAnyDepartments' is not a function, but this will not be accepted.
Work in progress, but what I have so far that is not accepted by editor:
BEGIN
#--
# Procedure:
# rsWhoCanIaccess
#
# Parameters:
# vcCompKey, the key corresponding to the company
# biWho_id, the id of the person to check access for
#
# Returns:
# recordset containing all the people this person can access
#--
DECLARE tiSuperUser tinyint(4);
DECLARE EXIT HANDLER FOR SQLEXCEPTION
BEGIN
GET DIAGNOSTICS CONDITION 1 #sqlstate = RETURNED_SQLSTATE,
#errno = MYSQL_ERRNO, #text = MESSAGE_TEXT;
CALL procLogError(vcCompKey, CONCAT("rsWhoCanIaccess: "
,#errno, " (", #sqlstate, "): ", #text));
END;
#Is this user a super user?
SELECT tiIsSuperUser(vcCompKey, biWho_id) INTO tiSuperUser;
SET tiSuperUser = 0;#Hack for testing
IF (tiSuperUser = 1) THEN
#The user is a superuser, return everyone in the company
SELECT
t1.biPerson_id
FROM
tbl_people t1
INNER JOIN
tbl_companies t2
ON
t1.biCompany_id=t2.biCompany_id
AND
t2.vcKey=vcCompKey;
ELSE
#User is not a superuser, is the user head of any departments?
DROP TEMPORARY TABLE IF EXISTS tbl_HeadOfDepts;
CREATE TEMPORARY TABLE tbl_HeadOfDepts (biDept_id tinyint(4))
INSERT INTO tbl_HeadOfDepts CALL rsHeadOfAnyDepartments(vcCompKey, biWho_id);
SELECT * FROM tbl_HeadOfDepts;
END IF;
END
No, stored procedures can produce result sets, but not consume them directly as output from inner calls to other stored procedures. The best that you can do performance-wise is to populate a non-temporary work table and use the results.
Depending on your software and the reality of multiple callers concurrently, you might need to include a session id concept with an auto_increment (AI) column in some control table. This would ensure that with concurrency, multiple callers are not stomping on each other's rows, thus making it non-viable.
How that session would work, at a high level, is the following. The inner stored proc would be handed an AI value (theSession) from the control table, use it to populate a safely segmented session in the work table, and return as an out parameter to the outer (calling) stored proc. That outer one could then safely use those rows, and clean up at the end (delete from workTable where sessionId=theSession).
Why do I suggest a non-temporary work table? To be clear, the work table would be non-temporary. First of all there is the hassle of getting the if exists drop to work. Most importantly, though, it is about performance. DDL calls for temporary table creation are not inexpensive. You will only believe this when you do performance testing to see what I mean. It may seem trivial, but in trivial operations, those DDL calls for creation could very well account for the lion share of the time necessary for the inner stored proc to complete.
internally I want to call another procedure that returns a record set,
In your inner procedure create a TEMPORARY TABLE and populate that temp table saying insert into your_temp_table select query. then you can use that same temp table in your outer query anywhere.
It can even be a normal table as well and need not be temporary table. Also make sure to DROP the table once your procedure computation done as clean-up.
That's wrong per your comment. You should do it like below (a sample code)
create procedure rsHeadOfAnyDepartments(vcCompKey varchar(10), biWho_id int)
as
begin
DROP TEMPORARY TABLE IF EXISTS tbl_HeadOfDepts;
CREATE TEMPORARY TABLE tbl_HeadOfDepts(col1 int, col2 varchar(10), col3 varchar(30));
INSERT INTO tbl_HeadOfDepts
SELECT col1, col2, col3
FROM tblTest;
end
I have a query like
Select id,name,... FROM table WHERE id IN (?)
The query is actually more complicated with joins and another (?) with the same ids. I've been told to put it on a stored procedure to make it easier to manage and safer, but mysql doesn't support arrays, so I'm not sure what is the best way to do it, considering that I don't know how many id I will have to pass. I have seen people passing a string. Is it the only way?
You can use temporary tables. Though this answer is not highly voted so I suppose that using a string parameter is the prefered way (17 votes vs 2 votes) (see the link). The answer with temporary table is taken from another question: Pass array to MySQL stored routine
Use a join with a temporary table. You don't need to pass temporary
tables to functions, they are global.
create temporary table ids( id int ) ;
insert into ids values (1),(2),(3) ;
delimiter //
drop procedure if exists tsel //
create procedure tsel() -- uses temporary table named ids. no params
READS SQL DATA
BEGIN
-- use the temporary table `ids` in the SELECT statement or
-- whatever query you have
select * from Users INNER JOIN ids on userId=ids.id ;
END //
DELIMITER ;
CALL tsel() ; -- call the procedure
I noticed a very interesting (and unexpected as well) thing yesterday. I was given a task (on production environment) to update three columns of TableA (I am changing the table and column names due to some obvious reasons) by getting all the values present in dummytable. The primary key of both the tables is column A. I know that this task was very simple and could be accomplished in several ways but I chose to write a stored procedure (given below) for that.
When the stored procedure was finished executing then it was noticed that columns B, C & statusCode were having the same values (i.e. thousands of records were having identical values in these three columns). Can someone tell me what went wrong?
1) What's wrong (or missing) in this stored procedure? (Dummy table had thousands of records as well)
2) What could be the best possible way of doing this task other than creating a stored procedure?
PS: I created (executed as well) this stored procedure on production environment using MySQL workbench and I got an exception during the execution of the procedure which stated something "Lost connection to MySQL server" but I guess since I was running this procedure on the remote machine then there was no interruption on the server while the procedure was executing.
Here is my stored procedure.
DELIMITER $$
CREATE DEFINER=`ABC`#`%` PROCEDURE `RetrieveExtractionData`()
BEGIN
DECLARE claimlisttraversed BOOLEAN DEFAULT FALSE;
DECLARE a VARCHAR(20);
DECLARE b INTEGER;
DECLARE c INTEGER;
DECLARE claimlist CURSOR FOR SELECT
`dummytable`.`A`,
`dummytable`.`B`,
`dummytable`.`C`
FROM `ABC`.`dummytable`;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET claimlisttraversed = TRUE;
OPEN claimlist;
claimlistloop: LOOP
FETCH claimlist INTO a, b, c;
IF claimlisttraversed THEN
CLOSE claimlist;
LEAVE claimlistloop;
END IF;
UPDATE `ABC`.`TableA`
SET
`B` = b,
`C` = c,
`statuscode` = 'Sent'
WHERE `A` = a;
END LOOP claimlistloop;
END
For your first question:
1) What's wrong (or missing) in this stored procedure? (Dummy table had
thousands of records as well)
I guess you forgot to CLOSE the CURSOR. Right after you end the LOOP, you should CLOSE the CURSOR.
END LOOP claimlistloop;
CLOSE claimlist;
END
2) What could be the best possible way of doing this task other than
creating a stored procedure?
Doing that in the STORED PROCEDURE should be fine. And also using CURSOR would be fine since you will just execute the procedure once (I guess because this is a production fix).
But from your question, you just want to update TableA based from the provided DummyTable. I assume that these tables have the same columns.
So I think this query is better than the CURSOR:
UPDATE TableA A
INNER JOIN DummyTable D ON D.A = A.A
SET A.B = D.B
, A.C = D.C
, A.statuscode = 'Sent';
But please try it first on a backup or dummy table. I haven't tested it yet.
Forget the cursor. In fact you should never use a cursor if it's avoidable. Cursors are incredibly slow.
Simply do
UPDATE
yourTable yt
INNER JOIN dummyTable dt ON yt.A = dt.A
SET
yt.B = dt.B,
yt.C = dt.C;
and you're fine.
1) What's wrong (or missing) in this stored procedure? (Dummy table
had thousands of records as well)
2) What could be the best possible
way of doing this task other than creating a stored procedure?
IMHO the most important thing that you're currently missing is that you don't need any cursors for that. Your whole stored procedure is one UPDATE statement. Execute it alone or wrap it in a stored procedure
CREATE PROCEDURE RetrieveExtractionData()
UPDATE TableA a JOIN dummytable d
ON a.a = d.a
SET a.b = d.b, a.c = d.c, a.statuscode = 'Sent';
You don't even need to change a delimiter and use BEGIN ... END block
Here is SQLFiddle demo.
I am trying to create a procedure which will fill up the table until certain amount of elements.
At the current moment I have
CREATE PROCEDURE PopulateTable(
IN dbName tinytext,
IN tableName tinytext,
IN amount INT)
BEGIN
DECLARE current_amount INT DEFAULT 0;
SET current_amount = SELECT COUNT(*) FROM dbName,'.',tableName;
WHILE current_amount <= amount DO
set #dll=CONCAT('INSERT INTO `',dbName,'`.`',tableName,'`(',
'`',tableName,'_name`) ',
'VALUES(\'',tableName,'_',current_amount,'\')');
prepare stmt from #ddl;
execute stmt;
SET current_amount = current_amount + 1;
END WHILE;
END;
So what I am trying to do is, once user calls the procedure, the function will check and see how many current elements exist, and fill up the remaining elements.
First problem I have is that I do not know how to count the elements, so my SELECT COUNT(*) FROM dbName,'.',tableName; does not work.
I also want to a suggestion since I am kind of new to databases if what I am doing is correct or if there is a better way to do this?
Also just if this is of any help the table I am trying to do this to only has 2 fields, one of them being id, which is auto incremented and is primary and the other being profile_name which I am populating.
Thanks to anyone for their help!
Firstly, I think you'll have a delimiter problem if you try to execute the code you pasted. The procedure declaration delimiter has to be different from the one you use into the procedure code (here ';'). You will have to use DELIMITER statement;
Your procedure belongs to a schema; I'm not sure you can query tables from other shemas (especially without USE statement);
This is not a good idea if your database contains tables whitch are not supposed to be populated through your procedure;
If you have a limited number of concerned tables, I think it will be a better idea to define one procedure for each table. In this way, table name will be explicit in each procedure code and it will avoid the use of prepared statements;
Be careful about you 'amount' parameter : are you sure that your server can handle requests if I pass the maximum value for INT as amount ?
I think you should use '<' instead of '<=' in your WHILE condition
If you want to insert a large number of lines, you'll obtain better performances performing "grouped" inserts, or generating a temporary table (for example with MEMORY engine) containing all your lines and performing a unique INSERT selecting your temporary table's content.