I'm going to rebuilt an existing moderate-scale web-app to be used for a supply-chain-management web-solution. The core would be around some Item\Site combination-records, Organization\User management, display Organization specific data (a dashboard with 2 levels of Grid) and a Drilldown which has some complex calculations for Item transactions and a robust Grid to show multiple level of totals, sub-totals, etc...
In past, I had developed it using
ASP.Net 2.0 and ADO.Net, the backend
is SQL-Server. I'm suppose to migrate
this web-solution to a full fiedge
ASP.Net v3.5 based MVC architecture
and integrate other pieces like
Membership API, Validation-framework,
etc... its going to be a complete
re-design. The performance and
scalability (i.e. handle millions of
records, perform quick calculations,
quick-response-time, etc..) are the
two main priorities. In addition
simplicity and long-term maintenance
and periodic upgrades at any level are
also to be considered.
I was wondering which of the following two would make a robust DAL which meets my above mentioned requirements:
L2S (LINQ to SQL) or EF (Entity
Framework)
I've been searching a lot and based on that I'd like to verify my understanding:
Is L2S like Win98 - strong, stable, simple and performing and -
Is EF something like WinXP - better then win98, also stable in a way but
somewhat more layers so might have a
diff in performance and simplicity
compared to L2S
I don't think we're going to involve 'multiple-databases' in our development, SQL-Server will stay. Also, I don't think we'll need to actually map multiple-tables to an Entity, etc... Generally, we'll have one to one mapping for our DB-tables to Entities (i.e. User, Org, etc..) and for some complex cases - like the dashboard fetches records from multiple tables (most probably we'll need to use a stored-procedure and/or a DB VIEW).
The Drilldown page fetches data from a single table but has many calculations so again we're planning to usa a stored-procedure which will provide us a formatted table (but this table might not look like the ones already present in DB)
So, thats some tricky mapping but apart from that we need to keep things simple, consider performance and scalability. Oh! and last but not the least - we've a tight deadline so its a kind of 'fast-track' development.
Simple, fast, scalable & performing - thats what we need!
Thanks in advance - please guide me.
PS: Ref links:
ORM-LINQ-Entity-Framework-Eric-Nelson
Discuss - LINQ to SQL vs. ADO.NET Entity Framework
Short - ADO.NET Entity Framework When To Use It Vs Linq To SQL
Details - Choosing between ADO.NET Entity Framework and LINQ to SQL
List - Choosing between LINQ to SQL and Entity Framework
I like the comparrison of L2Sql as Win98, but I would compare the EF to Vista ... it's got lots of power potentially but bits left undone until they came out with the next version make it like a death of a thousand cuts.
I am looking foward to the EF 4 ... I am hoping it is Windows 7 to stay with your analogy.
I don't think you can go wrong with Linq to sql. It has it's quirks but it works. We decided to go with L2S even though there are issues with using multiple data contexts.
For fast DAL generation check out plinqo It requires Codesmith to generate the code but it can do a lot and addresses some of the quirks linq to sql has. Also there is Oleg Sych T4 templates which is free and built into VS2008.
If you can't wait until entity v2 is out, which is .net 4.0, I would say go with linq 2 sql.
You can switch to entity later on if you need it.
I did play a little with entity v1, and I think it is still a baby that need to grow up a little.
Another reason for linq 2 sql is the fact that you will work with sql server only and it's a fast track project.
In my personal opinion, having used all of the things you mentioned, i would use NHibernate over either Linq2sql or Entity. If you are after a simple CRUD site, then the decision doesn't matter too much, but the ability to separate the business layer from the data layer is quite difficult to accomplish in ADO.Net, where as it's a lot simpler (in my opinion with Nhibernate).
As I say, opinions vary hugely in this area.
Related
I am working on a project using ZK Framework, Hibernate, Spring and Mysql.
I need to generate some charts from Mysql database, but after I calculate the number of objects that I need to calculate the values of those charts I found it more than 1400 objects and same numbers of queries and transactions.
So i thought if using stored procedures in Mysql to calculate those values and save them in a separate tables (using an architecture close to Data Warehouse), and then use my web application to just read the values of those tables and display them as charts.
I want to know in terms of speed and performance, which of those methods is better?
And thank you
No way to tell, really, without many more details. However:
What you want to do is called Denormalisation. This is a recognised technique for speeding up reporting and making it easier. (If it doesn't, your denormalisation has failed!) When it works it has the following advantages:
Reports run faster
Report code is easier to write
On the other hand:
Report Data is out of date, containing only data as at the time you
last did the calculations
An extreme form of doing this is to take the OLTP database (a standard database) and export it into an Analysis database (aka a Cube or an OLAP database).
One of the problems of Denormalisation is that a) it is usually a significant effort, b) it adds extra code which adds complexity and thus increases support costs, and c) it might not make enough (or any) difference. Because of this, it is usual not to do it until you know you have a problem. This will happen when you have done your reports on the basic database and have found that they either are too difficult to write and/or run too slowly. I would strongly suggest that only when you reach that point do you go for Denormalisation.
There can be times when you don't need to do that, but I've only seen 1 such example in over 25 years of development; and that decision was helped by a desire to use an OLAP database by Management for political purposes.
Problem scope: I want to use EF4.1 without any trade offs to the speed and reliability of the Enterprise Library Data Access Block that I know and trust.
Thanks to lots of Stackoverflow links and blogs about EF performance tuning I'm posting this way , among many, to use EF4.1 that matches the performance of ADO/ Enterprise Lib Data Access Block (SqlDataReader).
The Project:
1. No linq to Entities/ dynamic sql. I love linq, I just try to use it against objects mostly.
2. 100% stored procedures and no tracking, no merge, and most importantly, never call .SaveChanges(). I just call the insert/ update/ delete proc DbContext.StoredProcName(params). At this point we have eliminated several of the rapid dev elements of EF but the way it auto creates a complex type for your stored proc is enough for me.
The GetString and similar methods are an AbstractMapper that just goes through the expected types and casts the datareader into the type.
So this is the mark to beat as far as I'm concerned. It would be hard to adopt something I know to be slower.
That is SLOWER!!! A lot slower!
That is more like it!!
Performance Pie
Based on my results that performance pie should increase the tracking overhead by a lot more than 1%
I tried pre compiling the views and nothing got as big of a boost as no tracking! Why?? Maybe somebody can chime in on that.
So, this one is not really fair to compare to Enterprise Lib, but I'm making one untimed call to the database to load the the meta data that I understand is loaded once per IIS app pool. Essentially once in the life of your app.
I'm using EF this way with auto stored procedure generation and I used Linq to Edmx to auto import all these edmx function nodes to map up to the entities. Then I auto gen a repository for each entity and an engine.
Since I never call SaveChanges, I don't bother taking the time to map to stored procs in the designer. It takes too long and it is way to easy to break it and not know it. So I just call the procs from the context.
Before I actually implement this in my new mission critical medical equipment delivery web application I would appreciate any observations and critiques.
Thanks!
Just a few remarks:
Performance Pie Based on my results that performance pie should
increase the tracking overhead by a lot more than 1% I tried pre
compiling the views and nothing got as big of a boost as no tracking!
Why??
The blog post is from 2008 and therefore based on EF Version 1 and on EntityObject derived entities. With EF 4.1 you are using POCOs. Change tracking behaves very differently with POCOs. Especially when a POCO object is loaded from the database into the object context Entity Framework creates a snapshot of the original property values and stores is in the context. Change tracking relies on comparison between current entity values and original snapshop values. Creating this snapshot is apparently expensive in terms of performance and memory consumption as well. My observations are that it costs at least 50 percent (query time without change tracking is the half of the query time with change tracking). You seem to have measured an even bigger impact.
The Project: 1. No linq to Entities/ dynamic sql. I love linq, I just
try to use it against objects mostly. 2. 100% stored procedures and no
tracking, no merge, and most importantly, never call .SaveChanges(). I
just call the insert/ update/ delete proc
DbContext.StoredProcName(params). At this point we have eliminated
several of the rapid dev elements of EF but the way it auto creates a
complex type for your stored proc is enough for me.
For me this looks like you are ignoring essentially some of the main features why Entity Framework exists and it is questionable why you want to use EF for your purpose at all. If your main goal is to have a tool which helps to materialize query results into complex objects you could take a look at Dapper which focusses on this task with high performance in mind. (Dapper is the underlying ORM used here at Stackoverflow.)
A few days ago here was a question with great answers about EF performance. It has been migrated to "Programmers" now:
https://softwareengineering.stackexchange.com/questions/117357/is-entity-framework-suitable-for-high-traffic-websites
I'm a long-time user of the DevExpress XPO library. It has many great features, but there are a few weaknesses:
When saving an existing object, all properties are sent in an update query; changes are tracked on a per-object basis, not per-property.
Optimistic locking is done on a per-object basis, rather than per-column.
When an optimistic locking exception occurs, no context is provided describing the nature of the conflict; your only real response is to fail the operation or reproduce it and try again in a loop.
LINQ support for XPQuery is very weak (at least in 8.1, which we're using). Thus, you're often forced to use XPView, which is not type-safe, or XPCollection, which can be returning columns you don't necessarily need.
After reading about how LINQ to SQL implements optimisting locking and handling update conflicts, I was sold! I like how it implements column-level optimistic locking and doesn't need to add a column to the table. Being able to inspect and handle the exact nature of conflicts is great. And the fact that they track per-column changes should make its update queries much more efficient.
Of course, I haven't yet used LINQ to SQL in real applications, so I don't know it compares in reality. Also, I'm unclear on if it has analogs for some of the features we enjoy with XPO, such as:
Automatic schema updates (we believe in object design driving database structure rather than the reverse, and this greatly simplifies software deployment)
Two options for how inheritance is implemented (same-table or one-to-one table relationships)
Support for in-memory storage (though I suppose that we could substitute LINQ to Objects in our unit tests)
Storage provider customization (that allowed us to add NOLOCK support to our XPO queries)
We're going to be doing an exploratory partial migration where we will be temporarily using the two ORMs for different parts of our code. Have any of you had real-world experience with both XPO and LINQ to SQL? How do they compare in practice? Specifically, do you know of any features that LINQ to SQL lacks that would provide challenges to a code migration?
Oh, and should I even care about LINQ to Entities? It looks far more complicated than anything we need.
I'm sad that I didn't get any answers from the community, but here's my thoughts so far. I've had a chance to try out LINQ to SQL and ADO.NET Entity Framework for a while on different projects, and I feel that ADO.NET Entity Framework would better fill our needs. As far as the XPO-specific features I was hoping to keep:
Automatic schema updates will have to go once we convert. It's a minor annoyance, but there are a few benefits to maintaining this separately.
ADO.NET Entity Framework has a lot of data mapping options; the different inheritance models appear to be supported.
For in-memory storage, I'm still unsure how well-supported this is. There appears to be a SQLite ADO.NET provider that is compatible with the Entity Framework, and SQLite can do in-memory storage, so in theory the unit tests could use a different connection string specifying the in-memory database. Hopefully it's that easy; otherwise, writing unit tests will be pretty hard to do without a lot of work (abstracting out a repository interface, etc).
I haven't looked into provider customization yet. I've tried to architect the system such that we won't have as much data shared among services as before, so maybe we won't need all those WITH (NO LOCK) statements that I needed in previous systems. Or maybe SQL Server 2008 has improved its locking mechanisms so that we won't encounter the same locking issues.
So you did migrate your application from XPO to Linq2Sql, didn't you? I've been playing with XPO as part of XAF too, honestly I prefer Linq2Sql/EF to XPO but since it is tightly coupled in XAF so I don't have other choice. We're going to use XAF to speed up UI implementation of a our product, I think XAF does its work quite well, but I'm really worried about XPO.
Thanks,
Why should we use LINQtoSQL when ADO.NET is giving us better performance?
Checkout this link which shows the comparison reports
I am looking for Speed /Performance of my application rather than deployment easiness. Is there anything else which should we considered before i leave LINQ to SQL and go back to my OLD ADO.NET DataAcces Layer which user stored procedures ?
How often do you execute 2000 identical queries? This is a purely synthetic test which does not show a thing.
Generally speaking, it's much easier (and cost-effective) to throw more hardware at a problem (add more dirt-cheap RAM to an SQL Server machine) than to spend thousands of dollars and hundreds of man-hours trying to eliminate all bugs in
var dataReader = command.ExecuteReader();
while(dataReader.Read())
var id = dataReader.GetInt32("ID");
Granted, sometimes one needs to resort to plain ADO.NET to increase performance, but this should be done only where profiler shows a problem.
One of an ORMs biggest advantages over ADO.NET is in initial development speed (not application speed). Maintenance can also be much easier with an ORM than with ADO.NET. ORMs offer many features that would need to be coded by hand with ADO.NET. Linq does not work with ADO.NET and Linq currently requires an ORM.
Most ORMs allow you to drop down to SQL/ADO.NET when necessary for performance reasons.
How you choose to implement your data layer should completely depend on your requirements and the answer to that question is always situational. It's not possible to say ORMs are always superior to ADO.NET or vice versa.
IMO ORM is so good for a project which is not distributed. Sometimes your client use shared server or different database server that is even in the another country. In this case, I'm using SP. But ORM has another benefit. For instance, When you change the table structure, in SP, you've to change all of dependent SP. But in ORM, everything is ok after a few changes.
In what domains do each of these software architectures shine or fail?
Which key requirements would prompt you to choose one over the other?
Please assume that you have developers available who can do good object oriented code as well as good database development.
Also, please avoid holy wars :) all three technologies have pros and cons, I'm interested in where is most appropriate to use which.
Every one of these tools provides differing layers of abstraction, along with differing points to override behavior. These are architecture choices, and all architectural choices depend on trade-offs between technology, control, and organization, both of the application itself and the environment where it will be deployed.
If you're dealing with a culture where DBAs 'rule the roost', then a stored-procedure-based architecture will be easier to deploy. On the other hand, it can be very difficult to manage and version stored procedures.
Code generators shine when you use statically-typed languages, because you can catch errors at compile-time instead of at run-time.
ORMs are ideal for integration tools, where you may need to deal with different RDBMSes and schemas on an installation-to-installation basis. Change one map and your application goes from working with PeopleSoft on Oracle to working with Microsoft Dynamics on SQL Server.
I've seen applications where Generated Code is used to interface with Stored Procedures, because the stored procedures could be tweaked to get around limitations in the code generator.
Ultimately the only correct answer will depend upon the problem you're trying to solve and the environment where the solution needs to execute. Anything else is arguing the correct pronunciation of 'potato'.
I'll add my two cents:
Stored procedures
Can be easily optimized
Abstract fundamental business rules, enhancing data integrity
Provide a good security model (no need to grant read or write permissions to a front facing db user)
Shine when you have many applications accessing the same data
ORMs
Let you concentrate only on the domain and have a more "pure" object oriented approach to development
Shine when your application must be cross db compatible
Shine when your application is mostly driven by behaviour instead of data
Code Generators
Provide you similar benefits as ORMs, with higher maintenance costs, but with better customizability.
Are generally superior to ORMs in that ORMs tend to trade compile-time errors for runtime errors, which is generally to be avoided
I agree that there are pros and cons to everything and a lot depends on your architecture. That being said, I try to use ORM's where it makes sense. A lot of the functionality is already there and usually they help prevent SQL Injection (plus it helps avoid re-inventing the wheel).
Please see these other two posts on the topic (dynamic SQL vs
stored procedures vs ORM) for more information
Dynamic SQL vs. stored procedures
Which is better: Ad hoc queries, or stored procedures?
ORMs vs. stored procedures
Why is parameterized SQL generated by NHibernate just as fast as a stored procedure?
ORMs and code generators are kind of on one side of the field, and stored procedures are on another. Typically, it's easier to use ORMs and code generators in greenfield projects, because you can tailor your database schema to match the domain model you create. It's much more difficult to use them with legacy projects, because once software is written with a "data-first" mindset, it's difficult to wrap it with a domain model.
That being said, all three of the approaches have value. Stored procedures can be easier to optimize, but it can be tempting to put business logic in them that may be repeated in the application itself. ORMs work well if your schema matches the concept of the ORM, but can be difficult to customize if not. Code generators can be a nice middle ground, because they provide some of the benefits of an ORM but allow customization of the generated code -- however, if you get into the habit of altering the generated code, you then have two problems, because you will have to alter it each time you re-generate it.
There is no one true answer, but I tend more towards the ORM side because I believe it makes more sense to think with an object-first mindset.
Stored Procedures
Pros: Encapsulates data access code and is application-independent
Cons: Can be RDBMS-specific and increase development time
ORM
At least some ORMs allow mapping to stored procedures
Pros: Abstracts data access code and allows entity objects to be written in domain-specific way
Cons: Possible performance overhead and limited mapping capability
Code generation
Pros: Can be used to generate stored-proc based code or an ORM or a mix of both
Cons: Code generator layer may have to be maintained in addition to understanding generated code
You forgot a significant option that deserves a category of its own: a hybrid data mapping framework such as iBatis.
I have been pleased with iBatis because it lets your OO code remain OO in nature, and your database remain relational in nature, and solves the impedance mismatch by adding a third abstraction (the mapping layer between the objects and the relations) that is responsible for mapping the two, rather than trying to force fit one paradigm into the other.