I'm working on an order system that can have anywhere from 1 to 200+ orders in it. Right now we have a loop that loops through each order and inserts it into the database. Would it make sense to use SqlBulkCopy? Would using it degrade performance on small orders? Are there any other techniques to speed up inserts?
Thanks!
Basically there are several things you can do.
using SqlBulkCopy.WriteToServer which doesn't work great together with EF, however there are several attempts to create extensions
using a stored procedure which will take one big record and split it according to some logic
using table-typed stored procedure parameters and do one call to the stored procedure (and several insert ... select inside the stored procedure)
Overall, I prefer third option.
Check this question Entity Framework Stored Procedure Table Value Parameter
Related
I'm declaring a cursor in a stored procedure with following;
declare cur1 cursor for select * from tmp_01;
Here, my temporary table source is tmp_01.
The source table name is dynamically generated.
I'm wondering if there is a way that I could define the same cursor with different source for each instance when the stored procedure called.
For example,
on first run,
declare cur1 cursor for select * from tmp_01;
on second run,
declare cur1 cursor for select * from tmp_02;
The main problem I'm having is, I'm experiencing some strange behavior with the cursor when called with multiple queries using mysqli_multiquery, that is not clear to me. when I run each query separately, everything works fine. I'm not sure whether it's because something like parallel query processing.
All I'm trying to achieve is, declaring a unique source name for the cursor, on each procedure call.
Can anyone please point me in a right direction to achieve this?
No, the DECLARE CURSOR statement must take a fixed SQL query as its argument, and therefore the table name must be fixed. If your table name is variable, you cannot use a cursor in a stored routine.
It's not clear from your question what purpose you have for using multiquery, or what is the "strange behavior" you have seen. I can guess that it has to do with the fact that each call to a stored procedure returns multiple result sets, so it gets confusing if you try to call multiple procedures in a multiquery. If you are looping over multiple result sets, it becomes unclear when one procedure is done with its result sets and the next procedure starts returning its result sets.
Regardless, I don't recommend using multiquery in any case. There is hardly ever a good reason to use it. There's no performance or functionality advantage of using multiquery. I recommend you just run each call individually, and do not use multiquery.
For that matter, I also avoid using MySQL stored procedures. They have poor performance and scalability, the code is harder to write than any other programming languages, there is no compiler, no debugger, no support for packages, no standard library of utility procedures, the documentation is thin, etc. I understand that in the Oracle or Microsoft SQL Server community, it is customary to write lots of stored procedures, but in MySQL, I write my application logic in a client programming language such as Java, Go, or Python.
We have a large database and we do manipulations on it ever day by using the basic mysql queries.
Can anyone please tell me, what is the use of Mysql Stored Procedures?
The real use of the Stored Procedures comes into picture when have any application accessing database.
For example: Imagine that you have written all your database operations in the form of queries in your data access code.
Suppose, that you need to make any change to query , then you need to rebuild and redeploy the entire application in order see your changes.
But, if you are using stored procs and refering them in application, you can just make changes in your database with out need for redeploying the application.
So, obviously better security , maintainability and much more
Note: This is one scenario where stored procs are better than normal queries.
Usage of Stored Procs also avoids SQL Injection Attacks
In very simple words, stored procedures allow you to store your quires along with database, you can combine multiple quires in single procedure. now whenever you want to execute those quires just "CALL yourProcedure;"
Need to perform specific query daily ?
Read about MySQL events = stored procedures with scheduling capability !
https://dev.mysql.com/doc/refman/5.1/en/events.html
I am trying to work with EF, for the first time ever. I'm not sure I fully understand EF yet.
I already have a database with data in it, so I've generated my models from DB.
Our current setup runs EVERYTHING through stored procedures, even selects.
However, unless I'm mistaking, the models select directly into the tables, when I have generated them.
Can I change this behaviour, so it calls the select procedure instead?
No - at least with EF4. I can't speak for EF5
You can use stored procedures to insert and update, but those stored procedures must have all the parameters EF expects, so you're probably going to have to wrap your existing procedures in new procedures.
For select, you can use a FunctionImport and ExecuteFunction to populate an Entity.
It's not like that I am having trouble executing my cursors which are enclosed in a stored procedure. But I want to find more efficient way to achieve the same.
Here it goes.
Stored procedure : RawFeed.sql (runs every 5 minutes)
Set #GetATM = Cursor For
Select DeviceCode,ReceivedOn
From RawStatusFeed
Where CRWR=2 AND Processed=0
Order By ReceivedOn Desc
Open #GetATM
Fetch Next
From #GetATM Into #ATM,#ReceivedOn
While ##FETCH_STATUS = 0
Begin
Set #RawFeed=#ATM+' '+Convert(VarChar,#ReceivedOn,121)+' '+'002307'+' '+#ATM+' : Card Reader/Writer - FAULTY '
Exec usp_pushRawDataAndProcess 1,#RawFeed
Fetch Next
From #GetATM Into #ATM,#ReceivedOn
End
Set #GetATM = Cursor For
Select DeviceCode,ReceivedOn
From RawStatusFeed
Where CRWR=0 AND Processed=0
Order By ReceivedOn Desc
Open #GetATM
Fetch Next
From #GetATM Into #ATM,#ReceivedOn
While ##FETCH_STATUS = 0
Begin
Set #RawFeed=#ATM+' '+Convert(Varchar,#ReceivedOn,121)+' '+'002222'+' '+#ATM+' : Card Reader/Writer - OK '
Exec usp_pushRawDataAndProcess 1,#RawFeed
Fetch Next
From #GetATM Into #ATM,#ReceivedOn
End
Likewise I have 10 more SET statements which differ on WHERE condition parameter & string enclosed in #RawFeed variable.
For each row I get I execute another stored procedure on that particular row.
My question is
Is there any better way to achieve the same without using cursors?
Variable #RawFeed Contains following string which is input to usp_pushRawDataAndProcess stored procedure. now this will divide whole string and do some operation like INSERT,UPDATE,DELETE on some tables.
WE JUST CAN NOT PROCESS MORE THAN 1 STRING IN usp_pushRawDataAndProcess
NMAAO226 2012-09-22 16:10:06.123 002073 NMAAO226 : Journal Printer - OK
WMUAO485 2012-09-22 16:10:06.123 002222 WMUAO485 : Card Reader/Writer - OK
SQL Server, like other relational databases, is desgined to, and is pretty good at, working on sets of data.
Databases are not good at procedural code where all the opportunities for optimization are obscured from the query processing engine.
Using RawStatusFeed to store some proprietry request string and then processing a list of those one by one, is going to be ineffiencnt for database code. This might make the inserts very fast for the client, and this might be very important, but it comes at a cost.
If you break the request string down on insert, or better still, before insert via a specialised SP call, then you can store the required changes in some intermediate relational model, rather than a list of strings. Then, every so often, you can process all the changes at once with one call to a stored procedure. Admittedly, it would probably make sense for that stored procedure to contain several query statements. However, with the right indexes and statistics the query processing engine will able to make an efficient execution plan for this new stored procedure.
The exact details of how this should be achieved depend on the exact details of the RawStatusFeed table and the implementation of usp_pushRawDataAndProcess. Although this seems like a rewrite, I don't imagine the DeviceCode column is that complicated.
So, the short answer is certainly yes but, I'd need to know what usp_pushRawDataAndProcess does in detail.
The signature of the usp_pushRawDataAndProcess SP is acting as a bottle neck.
If you can't change usp_pushRawDataAndProcess and and won't create a set based alternative then you are stuck with the bottle neck.
So, rather than removing the bottle neck you could take another tack. Why not make more concurrent instances of the bottle neck to feed the data through.
If you are using SQL Server 2005 or above you could use some CLR to perform numerous instances of usp_pushRawDataAndProcess in parallel.
Here is a link to a project I used before to do something similar.
I had always disliked cursors because of their slow performance. However, I found I didn't fully understand the different types of cursors and that in certain instances, cursors are a viable solution.
When you have a business problem that can only be solved by processing one row at a time, then a cursor is appropriate.
So to improve performance with the cursor, change the type of cursor you are using. Something I didn't know was, if you don't specify which type of cursor you are declaring, you get the Dynamic Optimistic type by default, which is the one that is the slowest for performance because it's doing lots of work under the hood. However, by declaring your cursor as a different type, say a static cursor, it has very good performance.
See these articles for a fuller explanation:
The Truth About Cursors: Part I
The Truth About Cursors: Part II
The Truth About Cursors: Part III
I think the biggest con against cursors is performance, however, not laying out a task in a set based approach would probably rank second. Third would be readability and layout of the tasks as they usually don't have a lot of helpful comments.
The best alternative to a cursor I've found is to rework the logic to take a set based approach.
SQL Server is optimized to run the set based approach. You write the query to return a result set of data, like a join on tables for example, but the SQL Server execution engine determines which join to use: Merge Join, Nested Loop Join, or Hash Join. SQL Server determines the best possible joining algorithm based upon the participating columns, data volume, indexing structure, and the set of values in the participating columns. So it generally the best approach in performance over the procedural cursor approach.
Here is an article on Cursors and how to avoid them. It also discusses the alternatives to cursors.
Alernates for CURSOR in SQL server
1.While loop
2.Recursive CTE
Alernates for CURSOR in SQL server
1. Use temp table. create any column ID as identity column.
2. Use while loop to perform the operation.
Sometimes an application requires quite a few SQL queries before it can do anything useful. I was wondering if there is a way to send those as a batch to the database, to avoid the overhead of going back and forth between the client and the server?
If there is no standard way to do it, I'm using the python bindings of MySQL.
PS: I know MySQL has an executemany() function, but that's only for the same query executed many times with different parameters, right?
This process works best on inserts
Make all you SQL queries into Stored Procedures. These eventually will become child stored procedures
Create Master Store procedure to run all other Stored Procedures.
Modify master Stored procedure to accept values required by child Stored Procedures
Modify master Stored procedure to accept commands using "if" statements to know which
child stored procedures to run
If you need return data from Database use 1 stored procedure at the time.