It's not like that I am having trouble executing my cursors which are enclosed in a stored procedure. But I want to find more efficient way to achieve the same.
Here it goes.
Stored procedure : RawFeed.sql (runs every 5 minutes)
Set #GetATM = Cursor For
Select DeviceCode,ReceivedOn
From RawStatusFeed
Where CRWR=2 AND Processed=0
Order By ReceivedOn Desc
Open #GetATM
Fetch Next
From #GetATM Into #ATM,#ReceivedOn
While ##FETCH_STATUS = 0
Begin
Set #RawFeed=#ATM+' '+Convert(VarChar,#ReceivedOn,121)+' '+'002307'+' '+#ATM+' : Card Reader/Writer - FAULTY '
Exec usp_pushRawDataAndProcess 1,#RawFeed
Fetch Next
From #GetATM Into #ATM,#ReceivedOn
End
Set #GetATM = Cursor For
Select DeviceCode,ReceivedOn
From RawStatusFeed
Where CRWR=0 AND Processed=0
Order By ReceivedOn Desc
Open #GetATM
Fetch Next
From #GetATM Into #ATM,#ReceivedOn
While ##FETCH_STATUS = 0
Begin
Set #RawFeed=#ATM+' '+Convert(Varchar,#ReceivedOn,121)+' '+'002222'+' '+#ATM+' : Card Reader/Writer - OK '
Exec usp_pushRawDataAndProcess 1,#RawFeed
Fetch Next
From #GetATM Into #ATM,#ReceivedOn
End
Likewise I have 10 more SET statements which differ on WHERE condition parameter & string enclosed in #RawFeed variable.
For each row I get I execute another stored procedure on that particular row.
My question is
Is there any better way to achieve the same without using cursors?
Variable #RawFeed Contains following string which is input to usp_pushRawDataAndProcess stored procedure. now this will divide whole string and do some operation like INSERT,UPDATE,DELETE on some tables.
WE JUST CAN NOT PROCESS MORE THAN 1 STRING IN usp_pushRawDataAndProcess
NMAAO226 2012-09-22 16:10:06.123 002073 NMAAO226 : Journal Printer - OK
WMUAO485 2012-09-22 16:10:06.123 002222 WMUAO485 : Card Reader/Writer - OK
SQL Server, like other relational databases, is desgined to, and is pretty good at, working on sets of data.
Databases are not good at procedural code where all the opportunities for optimization are obscured from the query processing engine.
Using RawStatusFeed to store some proprietry request string and then processing a list of those one by one, is going to be ineffiencnt for database code. This might make the inserts very fast for the client, and this might be very important, but it comes at a cost.
If you break the request string down on insert, or better still, before insert via a specialised SP call, then you can store the required changes in some intermediate relational model, rather than a list of strings. Then, every so often, you can process all the changes at once with one call to a stored procedure. Admittedly, it would probably make sense for that stored procedure to contain several query statements. However, with the right indexes and statistics the query processing engine will able to make an efficient execution plan for this new stored procedure.
The exact details of how this should be achieved depend on the exact details of the RawStatusFeed table and the implementation of usp_pushRawDataAndProcess. Although this seems like a rewrite, I don't imagine the DeviceCode column is that complicated.
So, the short answer is certainly yes but, I'd need to know what usp_pushRawDataAndProcess does in detail.
The signature of the usp_pushRawDataAndProcess SP is acting as a bottle neck.
If you can't change usp_pushRawDataAndProcess and and won't create a set based alternative then you are stuck with the bottle neck.
So, rather than removing the bottle neck you could take another tack. Why not make more concurrent instances of the bottle neck to feed the data through.
If you are using SQL Server 2005 or above you could use some CLR to perform numerous instances of usp_pushRawDataAndProcess in parallel.
Here is a link to a project I used before to do something similar.
I had always disliked cursors because of their slow performance. However, I found I didn't fully understand the different types of cursors and that in certain instances, cursors are a viable solution.
When you have a business problem that can only be solved by processing one row at a time, then a cursor is appropriate.
So to improve performance with the cursor, change the type of cursor you are using. Something I didn't know was, if you don't specify which type of cursor you are declaring, you get the Dynamic Optimistic type by default, which is the one that is the slowest for performance because it's doing lots of work under the hood. However, by declaring your cursor as a different type, say a static cursor, it has very good performance.
See these articles for a fuller explanation:
The Truth About Cursors: Part I
The Truth About Cursors: Part II
The Truth About Cursors: Part III
I think the biggest con against cursors is performance, however, not laying out a task in a set based approach would probably rank second. Third would be readability and layout of the tasks as they usually don't have a lot of helpful comments.
The best alternative to a cursor I've found is to rework the logic to take a set based approach.
SQL Server is optimized to run the set based approach. You write the query to return a result set of data, like a join on tables for example, but the SQL Server execution engine determines which join to use: Merge Join, Nested Loop Join, or Hash Join. SQL Server determines the best possible joining algorithm based upon the participating columns, data volume, indexing structure, and the set of values in the participating columns. So it generally the best approach in performance over the procedural cursor approach.
Here is an article on Cursors and how to avoid them. It also discusses the alternatives to cursors.
Alernates for CURSOR in SQL server
1.While loop
2.Recursive CTE
Alernates for CURSOR in SQL server
1. Use temp table. create any column ID as identity column.
2. Use while loop to perform the operation.
Related
I'm declaring a cursor in a stored procedure with following;
declare cur1 cursor for select * from tmp_01;
Here, my temporary table source is tmp_01.
The source table name is dynamically generated.
I'm wondering if there is a way that I could define the same cursor with different source for each instance when the stored procedure called.
For example,
on first run,
declare cur1 cursor for select * from tmp_01;
on second run,
declare cur1 cursor for select * from tmp_02;
The main problem I'm having is, I'm experiencing some strange behavior with the cursor when called with multiple queries using mysqli_multiquery, that is not clear to me. when I run each query separately, everything works fine. I'm not sure whether it's because something like parallel query processing.
All I'm trying to achieve is, declaring a unique source name for the cursor, on each procedure call.
Can anyone please point me in a right direction to achieve this?
No, the DECLARE CURSOR statement must take a fixed SQL query as its argument, and therefore the table name must be fixed. If your table name is variable, you cannot use a cursor in a stored routine.
It's not clear from your question what purpose you have for using multiquery, or what is the "strange behavior" you have seen. I can guess that it has to do with the fact that each call to a stored procedure returns multiple result sets, so it gets confusing if you try to call multiple procedures in a multiquery. If you are looping over multiple result sets, it becomes unclear when one procedure is done with its result sets and the next procedure starts returning its result sets.
Regardless, I don't recommend using multiquery in any case. There is hardly ever a good reason to use it. There's no performance or functionality advantage of using multiquery. I recommend you just run each call individually, and do not use multiquery.
For that matter, I also avoid using MySQL stored procedures. They have poor performance and scalability, the code is harder to write than any other programming languages, there is no compiler, no debugger, no support for packages, no standard library of utility procedures, the documentation is thin, etc. I understand that in the Oracle or Microsoft SQL Server community, it is customary to write lots of stored procedures, but in MySQL, I write my application logic in a client programming language such as Java, Go, or Python.
I'm working on an order system that can have anywhere from 1 to 200+ orders in it. Right now we have a loop that loops through each order and inserts it into the database. Would it make sense to use SqlBulkCopy? Would using it degrade performance on small orders? Are there any other techniques to speed up inserts?
Thanks!
Basically there are several things you can do.
using SqlBulkCopy.WriteToServer which doesn't work great together with EF, however there are several attempts to create extensions
using a stored procedure which will take one big record and split it according to some logic
using table-typed stored procedure parameters and do one call to the stored procedure (and several insert ... select inside the stored procedure)
Overall, I prefer third option.
Check this question Entity Framework Stored Procedure Table Value Parameter
I've been trying to figure out what's wrong with a set of queries I've got and I'm just confused at this point.
It's supposed to be in a stored procedure which gets called by a GUI application.
There's only one "tiny" problem, it's first a simple UPDATE, then an INSERT using a SELECT with a subselect and finally another UPDATE. Running these queries together by hand I get a total execution time of 0.057s, not too shabby.
Now, I try creating a stored procedure with these queries in it and five input variables, I run this procedure and on the first attempt it took 47.096s with subsequent calls to it showing similar execution times (35 to 50s). Running the individual queries from the MySQL Workbench still show execution times of less than 0.1s
There really isn't anything fancy about these queries, so why is the stored procedure taking an eternity to execute while the queries by themselves only take a fraction of a second? Is there some kind of MySQL peculiarity that I'm missing here?
Additional testing results:
It seems that if I run the queries in MySQL Workbench but use variables instead of just putting the values of the variables in the queries it runs just as slow as the stored procedure. So I tried changing the stored procedure to just use static values instead of variables and suddenly it ran blazingly fast. Apparently for some reason using a variable makes it run extremely slow (for example, the first UPDATE query goes from taking approximately 0.98s with three variables to 0.04-0.05s when I use the values of variables directly in the query, regardless of if it's in the stored procedure or running the query directly).
So, the problem isn't the stored procedure, it's something related to my use of variables (which is unavoidable).
I had the same problem. After researching for a while, I found out the problem was the collation issue while MySQL was comparing text.
TL;DR: the table was created in one collation while MySQL "thought" the variable was in another collation. Therefore, MySQL cannot use the index intended for the query.
In my case, the table was created with (latin1, latin1_swedish_ci) collation. To make MySQL to use the index, I had to change the where clause in the stored procedure from
UPDATE ... WHERE mycolumn = myvariable
to
UPDATE ... WHERE mycolumn =
convert(myvariable using latin1) collate latin1_swedish_ci
After the change, the stored procedure looked something like this:
CREATE PROCEDURE foo.'bar'()
BEGIN
UPDATE mytable SET mycolumn1 = variable1
WHERE mycolumn2 =
convert(variable2 using latin1) collate latin1_swedish_ci
END;
where (latin1, latin1_swedish_ci) is the same collation that my tableA was created with.
To check if MySQL uses the index or not, you can change the stored procedure to run an explain statement as followed:
CREATE PROCEDURE foo.'bar'()
BEGIN
EXPLAIN SELECT * FROM table WHERE mycolumn2 = variable2
END;
In my case, the explain result showed that no index was used during the execution of the query.
Note that MySQL may use the index when you run the query alone, but still won't use the index for the same query inside a stored procedure, which maybe because somehow MySQL sees the variable in another collation.
More information on the collation issue can be found here:
http://lowleveldesign.wordpress.com/2013/07/19/diagnosing-collation-issue-mysql-stored-procedure/
Back up link:
http://www.codeproject.com/Articles/623272/Diagnosing-a-collation-issue-in-a-MySQL-stored-pro
I had a similar problem.
Running a mysql routine was horrible slow.
But a colleague helped me.
The problem was that AUTOCOMMIT was true;
So every insert into and select was creating a complete transaction.
Then I run my routine with
SET autocommit=0;
at the beginning and
SET autocommit=1;
at the end. The performance went from nearly 500s to 4s
Since I didn't want to waste too much time trying to figure out why using variables in my stored procedures made them extremely slow I decided to employ a fix some people would consider quite ugly. I simply executed each query directly from the data access layer of my application. Not the prettiest way to do it (since a lot of other stuff for this app uses stored procedures) but it works and now the user won't have to wait 40+ seconds for certain actions as they happen almost instantly.
So, not really a solution or explanation of what was happening, but at least it works.
Upvote for a very interesting and important question. I found this discussion of some of the reasons that a stored procedure might be slow. I'd be interested to see readers' reactions to it.
The main recommendation that I took from the interchange: it helps to add more indexes.
Something that we ran across today that makes procedures slow, even when they run very fast as direct queries, is having parameter (or, presumably, variable) names that are the same as column names. The short version is, don't use a parameter name that is the same as one of the columns in the query in which it will be used. For example, if you had a field called account_id and a parameter named the same, change it to something like in_account_id and your run time can go from multiple seconds to hundredths of a second.
I have a complicated SELECT query that filters on a time range, and I want this time range (start and end dates) to be specifiable using user-supplied parameters. So I can use a stored procedure to do this, and the return is a multiple-row result set. The problem I'm having is how to deal with this result set afterwards. I can't do something like:
SELECT * FROM (CALL stored_procedure(start_time, end_time))
even though the stored procedure is just a SELECT that takes parameters. Server-side prepared statement also don't work (and they're not persistent either).
Some suggest using temporary tables; the reason that's not an ideal solution is that 1) I don't want to specify the table schema and it seems that you have to, and 2) the lifetime of the temporary table would only be limited to a invocation of the query, it doesn't need to persist beyond that.
So to recap, I want something like a persistent prepared statement server-side, whose return is a result set that MySQL can manipulate as if it was a subquery. Any ideas? Thanks.
By the way, I'm using MySQL 5.0. I know it's a pretty old version, but this feature doesn't seem to exist in any more recent version. I'm not sure whether SELECT-ing from a stored procedure is possible in other SQL engines; switching is not an option at the moment, but I'd like to know whether it's possible anyway, in case we decide to switch in the future.
Selecting from functions is possible in other engines. For instance, Oracle allows you to write a function that returns a table of user defined type. You can define result sets in the function, fill them by using queries or even using a combination of selects and code. Eventually, the result set can be returned from the function, and you can continue to query on that by using:
select * from table(FunctionToBeCalls(parameters));
The only disadvantage, is that this result set is not indexed, so it might be slow if the function is used within a complex query.
In MySQL nothing like this is possible. There is no way to use a result set from a procedure directly in a select query. You can return single values from a function and you can use OUT or INOUT parameters to you procedure to return values from.
But entire result sets is not possible. Filling a temporary table within you procedure is the closest you will get.
Is there any way to debug a SQL Server 2008 query?
Yes. You can use the T-SQL debugger:
http://technet.microsoft.com/en-us/library/cc646008.aspx
Here is how you step through a query: http://technet.microsoft.com/en-us/library/cc646018.aspx
What exactly do you mean by debug?
Are you seeing incorrect data?
Are you getting unexpected duplicate rows?
What I usually do is start with a known set of data, usually one or two rows if possible, and comment out all joins and where conditions.
Introduce each additional element in your query one at a time starting with joins.
At each step, you should know how many records you are expecting.
As soon as you introduce something like a join or where condition that does not match your prediction, you know you have found the problem statement.
If it is a stored procedure with variables and such, you can always PRINT the values of your variables at different points.
If you want to only execute to a particular point in a stored procedure, then you may RETURN at any time and halt processing.
If you have temp tables that must get destroyed between executions while debugging your procedure, a common trick I use is to create a label like-
cleanup:
then at whatever point I want to bail, I can goto cleanup (I know goto is horrible, but it works great when debugging sprocs)
Yes, you need to set a breakpoint.
Frankly I find the debugging to be virtually useless. I often don't want to see the variables, but rather the records I would be inserting to a table or updating or deleting.
What I do is this when I havea complex sp to debug.
First I create a test variable. I set it =1 when I want to test. This will ensure that All actions iteh transaction are rolled back at the end (don't want to actually change the datbase until you are sure the proc is doing what you want.) by making sure the commit statement requires the test variable to be set to 0.
At the end of the proc I generally have a if test = 1 begin
END statement and between the begin and end, I put the select statments for all the things I want to see the values of. This might include any table variables or temp tables, the records in a particular table after the insert, the records I deleted or whatever else I feel I need to see. If I am testing mulitple times, I might comment out references to tables that I know are right and concentrate only on the ones I've changed that go around.
Now I can see what the effect of my proc is and the changes are rolled back in case they weren't right. To actually commit the changes (and not see the intermediate steps) , I simply change the value of the test variable.
If I use dynamic SQL, I also havea debug variable that instead of executing the dynamic SQl simply prints the results out to the screen. I find all this far more useful in debugging a complex script than breakpoints that show me the value of variables.