I have a fairly simple composite one to many relationship defined using POCO/Fluent API, one column of which is a string.
I've discovered that the data in this column in our database is inconsistent in terms of case ie 'abb', 'ABB' - this is our main ERP system and is fed by a variety of sources which are mainly beyond our control.
This is leading to problems using EF code first when joining to related tables as the join is silently ignored by EF when the case of PK/FK is different even though SQL Profiler shows the correct SQL being executed and results returned.
I'm using WCF so have lazy loading and proxy creation turned off and am eager loading required related entities using Include. eg.
var member = context.Member.Include(m => m.Audits).First(m => m.Id == id);
Are there any solutions to this outside of amending the database schema?
EF Insensitive join comparison
Hi I'm having the same problem (although not wit code first, but with a generated model)
The cause is that EF makes a case-sensitive comparison of the key fields, and it doesn'n find the related objects.
I'm guessing the problem lies in the "EDM Relationship Manager" and maybe there's a possibility of overriding this behavior.
I found a simple workaround for this: lower casing the related properties:
[EdmScalarPropertyAttribute(EntityKeyProperty=true, IsNullable=false)]
[DataMemberAttribute()]
public global::System.String id
{
get
{
return _id.ToLower(); // **<- here**
}
set
{
if (_id != value)
{
OnidChanging(value);
ReportPropertyChanging("id");
_id = StructuralObject.SetValidValue(value, false);
ReportPropertyChanged("id");
OnidChanged();
}
}
}
private global::System.String _id;
partial void OnidChanging(global::System.String value);
partial void OnidChanged();
It actually works, but, of course, it's a lame workoround.
I'm sticking to it for a while util I (or somebody) comes out with a better solution.
Good Luck!
I came up with a workaround that manually "stitches up" the association after the context has retrieved the appropriate rows from the database. Translated to your problem it would be along these lines:
//Your original query
var members = context.Member.Include(m => m.Audits).First(m => m.Id == id);
//The "stitch up" code that should probably be moved to a method of the data context.
var membersWithoutAudits = context.Members.Local.Where(m => !m.Audits.Any()).ToList();
foreach (var nextMember in membersWithoutAudits) {
//Now we can populate the association using whatever logic we like
nextMember.Audits = context.Audits.Local.Where(a => a.MemberId.ToLower() == nextMember.Id.ToLower()).ToList();
}
Notice how we use the context.[DbSet].Local property to ensure that we do all the "stitch up" in memory without making any further SQL calls. I also fetch the members without audits as a performance optimization so we are not re-doing the work of EF's association (in the cases where it did work). But you could just as easily remap every "member" instance.
Related
I'm retrieving data from a Mysql database, and this is done via mapping my entity properties to database fields using Doctrine with annotations as shown in this
image. Retrieving the data is not the problem, but my property name is used as key for the returned data, which is not what i want.
Example:
I have a property gridId (camelCase), which is mapped to a database field called grid_id (snake_case) as shown in the image above. The key for the rows returned for this property will be gridId, and not grid_id, which is desired in my case.
An example of the current situation:
image.
The desired situation:
$grid =
[
grid_block_id = 18
width = 50
width_pixels = 50
]
Changing either the database fields or the property names is not an option in my case.
EDIT
I'm using JMS as serializer. I guess the 4th option of #Jakumi's answer is probably the best option for my use case. Somehow use the serializer to return the property's column name as key for returned values.
My struggle when retrieving data from doctrine is that i have to manually adjust the keys fetched by doctrine as shown in this image as this is how the front-end expects it to be.
so what you're saying is ... you want to actively ignore the object relational mapping (ORM) that the doctrine ORM you're using is providing you and which actually provide the keys exactly the way your entity names them?
I assume there is some good (or not so good) reason for this. Depending on your use case, different solutions may be the best. Options including (but probably very incomplete):
option 1: add a function to the entity
class Grid {
/** all your fields **/
public function toArray() {
return [
'grid_id' => $this->gridId,
// ...
];
}
}
option 2: don't use the orm
add a function in your GridRepository like:
function getRawAll() {
$conn = $this->getEntityManager()->getConnection();
$_result = $conn->query('SELECT * FROM grid');
$results = [];
while($row = $_result->fetch()) {
$results[] = $row;
}
return $results;
}
and similar. Those are obviously circumventing the ORM. The ORM is for retrieving objects, and those have specified fields.
option 3: return the given array back to the unmapped state
use the EntityManager's getClassMetadata (ClassMetadataInfo->fieldMapping) to fetch the mapping for your fields, and "revert" the renaming.
This is probably slightly better than option 1 to some degree, since it doesn't have to be updated.
option 4: use a serializer that does your property name translation.
since you're apparently already using the jms serializer, the SerializedName annotation can probably help, although the default appears to be snake case, but maybe somewhere it's overridden? you might want to check your configuration ...
This is related (but fairly independent) to my question here: Why SELECT N + 1 with no foreign keys and LINQ?
I've tried using DataLoadOptions to force eager loading, but I'm not getting it to work.
I'm manually writing my LinqToSQL mappings and was first following this tutorial: http://www.codeproject.com/Articles/43025/A-LINQ-Tutorial-Mapping-Tables-to-Objects
Now I've found this tutorial: http://msdn.microsoft.com/en-us/library/bb386950.aspx
There's at least one major difference that I can spot. The first tutorial suggest returning ICollection's and the second EntitySet's. Since I'm having issues I tried to switch my code to return EntitySet's, but then I got issue with needing to reference System.Data.Linq in my Views and Controllers. I tried to do that, but didn't get it to work. I'm also not sure it's a good idea.
At this point, I just want to know which return type I'm supposed to use for a good design? Can I have a good design and still be able to force eager loading in specific cases?
A lot of trial and error finally lead to the solution. It's fine to return ICollection or IList, or in some cases IEnumerable. Some think returning EntitySet or IQueryable is a bad idea, and I agree because it exposes to much of the datasource/technology. Some thing returning IEnumerable is a bad idea and it seems like it depends. The problem beeing that it can be used for lazy loading, which may or may not be a good thing.
One reoccuring issue is that of returning paged results with a count for the total items outside the page. This can be solved by creating a CollectionPage<T> ( http://www.codetunnel.com/blog/post/104/how-to-properly-return-a-paged-result-set-from-your-repository )
More on what to return from repositories here:
http://www.codetunnel.com/blog/post/103/should-you-return-iqueryablet-from-your-repositories
http://www.shawnmclean.com/blog/2011/06/iqueryable-vs-ienumerable-in-the-repository-pattern/
IEnumerable vs IQueryable for Business Logic or DAL return Types
List, IList, IEnumerable, IQueryable, ICollection, which is most flexible return type?
Even more important, DataLoadOptions can do the eager loading! I've now restructured my code so much I'm not 100% sure what I did wrong to cause DataLoadOptions not to work. As far as I've gathered I should get an exception if I tried to add it to the DataContext after the DataContext has been used, which it didn't. What I've found out though is to think in the Unit of Work-pattern. However, for my needs (and because I don't want to return EntitySet or IQueryable from my repositories) I'm not going to implement a cross-repository Unit of Work. Instead I'm just thinking about my repository methods as their own small Unit of Work. I'm sure there's bad things about this (for instance it might cause more round-trips to the database in some update scenarios), and in the future I might reconcider. However it's a simple clean solution.
More info here:
https://stackoverflow.com/a/7941017/1312533
http://www.asp.net/mvc/tutorials/getting-started-with-ef-using-mvc/implementing-the-repository-and-unit-of-work-patterns-in-an-asp-net-mvc-application
This is what I ended up with in my repository:
public class SqlLocalizedCategoriesRepository : ILocalizedCategoriesRepository
{
private string connectionString;
private HttpContextBase httpContext;
public SqlLocalizedCategoriesRepository(string connectionString, HttpContextBase httpContext) // Injected with Inversion of Control
{
this.connectionString = connectionString;
this.httpContext = httpContext;
}
public CollectionPage<Product> GetProductsByLocalizedCategory(string category, int countryId, int page, int pageSize)
{
// Setup a DataContext
using (var context = new DataContext(connectionString)) // Because DataContext implements IDisposable it should be disposed of
{
var dlo = new System.Data.Linq.DataLoadOptions();
dlo.LoadWith<Product>(p => p.ProductSubs); // In this case I want all ProductSubs for the Products, so I eager load them with LoadWith. There's also AssociateWith which can filter what is eager loaded.
context.LoadOptions = dlo;
context.Log = (StringWriter)httpContext.Items["linqToSqlLog"]; // For logging queries, a must so you can see what LINQ to SQL generates
// Query the DataContext
var cat = (from lc in context.GetTable<LocalizedCategory>()
where lc.CountryID == countryId && lc.Name == category
select lc.Category).First(); // Gets the category into memory. Might be some way to not get it into memory by combining with the next query, but in my case my next step is that I'm also going to need the Category anyway so it's not worth doing because I'm going to restructure this code to take a categoryId parameter instead of the category parameter.
var products = (from p in context.GetTable<Product>()
where p.ProductCategories.Any(pm => pm.Category.CategoryID == cat.CategoryID)
select p); // Generates a single query to get the the relevant products, which with DataLoadOptions loads related ProductSubs. It's important that this is just a query and not loaded into memory since we're going to split it into pages.
// Return the results
var pageOfItems = new CollectionPage<Product>
{
Items = products.Skip(pageSize * (page - 1)).Take(pageSize).ToList(), // Gets the page of products into memory
TotalItems = products.Count(), // Get to total count of items belonging to the Category
CurrentPage = page
};
return pageOfItems;
}
}
}
I am porting an existing application from Linq to SQL to Entity Framework 4 (default code generation).
One difference I noticed between the two is that a foreign key property is not updated when resetting the object reference. Now I need to decide how to deal with this.
For example supposing you have two entity types, Company and Employee. One Company has many Employees.
In Linq To SQL, setting the company also sets the company id:
var company=new Company(ID=1);
var employee=new Employee();
Debug.Assert(employee.CompanyID==0);
employee.Company=company;
Debug.Assert(employee.CompanyID==1); //Works fine!
In Entity Framework (and without using any code template customization) this does not work:
var company=new Company(ID=1);
var employee=new Employee();
Debug.Assert(employee.CompanyID==0);
employee.Company=company;
Debug.Assert(employee.CompanyID==1); //Throws, since CompanyID was not updated!
How can I make EF behave the same way as LinqToSQL? I had a look at the default code generation T4 template, but I could not figure out how to make the necessary changes. It seems like a one-liner should do the trick, but I could not figure out how to get the ID property for a given reference.
From what I can see in the default T4 template, the foreign key properties of entities are not directly linked to the entity reference associated with the key.
Theres a couples to approach to your issue regarding migration from Linq to SQL to EF4. One of them would be to register to the AssociationChanged event of your associations so that it updates your field automatically. In your context, one approach could be something like like this :
// Extends Employee entity
public partial class Employee
{
private void CompanyChanged(Object sender, CollectionChangeEventArgs e)
{
// Apply reactive changes; aka set CompanyID
// here
}
// Create a default constructor that registers your event handler
public Employee()
{
this.CompanyReference.AssociationChanged += CompanyChanged;
}
}
Personally, if you want to limit the maintenance required to maintain this sort of logic, I'd suggest changing your T4 template (either change it yourself or find one) so that it sets the CompanyId when Company is changed as shown previously.
Gil Fink wrote a pretty good introdution to T4 templates with EF4, and you can look up Scott Hanselman wrapped a good bunch of useful links and ressources to work with T4 templates.
On a last note, unless I'm mistaken, accessing foreign keys directly as propeties of an entity is something new from EF3.5 to 4. In 3.5, only way you could access it was through the associated entity (Employee.Company.CompanyID). I believe the feature was added in EF4 so that you didn't have to load associations (using "include") in order to get the foreign key when selecting from the data store.
Perhaps EF's take on this would be, if you got the association, go through the association to get the ID, first and foremost. But that's just speculation as I got no quotes to back it up.
[EDIT 2010-06-16]:
After a quick readthrough and analysis of the edmx xml elements, I found one called ReferentialConstraint which appears to contain foreign key fields to a specfic FK_Relation.
Heres the code snippet to modify inside a default T4 edmx template, section Write Navigation Properties. (Template_RegionNavigationProperties), around line 388 of an unmodified template. Try to ignore the horrible formatting...
<#=code.SpaceAfter(NewModifier(navProperty))#><#=Accessibility.ForProperty(navProperty)#> <#=MultiSchemaEscape(navProperty.ToEndMember.GetEntityType(), code)#> <#=code.Escape(navProperty)#>
{
<#=code.SpaceAfter(Accessibility.ForGetter(navProperty))#>get
{
return ((IEntityWithRelationships)this).RelationshipManager.GetRelatedReference<<#=MultiSchemaEscape(navProperty.ToEndMember.GetEntityType(), code)#>>("<#=navProperty.RelationshipType.FullName#>", "<#=navProperty.ToEndMember.Name#>").Value;
}
<#=code.SpaceAfter(Accessibility.ForSetter(navProperty))#>set
{
// edit begins here
if(value != null)
{
// Automatically sets the foreign key attributes according to linked entity
<#
AssociationType association = GetSourceSchemaTypes<AssociationType>().FirstOrDefault(_ => _.FullName == navProperty.RelationshipType.FullName);
foreach(var cons in association.ReferentialConstraints)
{
foreach(var metadataProperty in cons.FromProperties)
{
#>
this.<#=metadataProperty.Name#> = value.<#=metadataProperty.Name#>;
//this._<#=metadataProperty.Name#> = value._<#=metadataProperty.Name#>; // use private field to bypass the OnChanged events, property validation and the likes..
<#
}
}
#>
}
else
{
// what usually happens in Linq-to-SQL when an association is set to null
// here
}
// edit ends here
((IEntityWithRelationships)this).RelationshipManager.GetRelatedReference<<#=MultiSchemaEscape(navProperty.ToEndMember.GetEntityType(), code)#>>("<#=navProperty.RelationshipType.FullName#>", "<#=navProperty.ToEndMember.Name#>").Value = value;
}
}
I roughly tested it, but it's a given that theres some validation and such missing. Perhaps it could give you a tip towards a solution regardless.
Thanks for this solution. I've enhanced it (does not depend on specific naming conventions anymore) and encluded in a fix that also fixes an other issue with the Entity Framework template.
Check here for my solution and fixed code generation template
I'm trying to decide on the best pattern for data access in my MVC application.
Currently, having followed the MVC storefront series, I am using repositories, exposing IQueryable to a service layer, which then applies filters. Initially I have been using LINQtoSQL e.g.
public interface IMyRepository
{
IQueryable<MyClass> GetAll();
}
Implemented in:
public class LINQtoSQLRepository : IMyRepository
{
public IQueryable<MyClass> GetAll()
{
return from table in dbContext.table
select new MyClass
{
Field1 = table.field1,
... etc.
}
}
}
Filter for IDs:
public static class TableFilters
{
public static MyClass WithID(this IQueryable<MyClass> qry, string id)
{
return (from t in qry
where t.ID == id
select t).SingleOrDefault();
}
}
Called from service:
public class TableService
{
public MyClass RecordsByID(string id)
{
return _repository.GetAll()
.WithID(id);
}
}
I ran into a problem when I experimented with implementing the repository using Entity Framework with LINQ to Entities. The filters class in my project contains some more complex operations than the "WHERE ... == ..." in the example above, which I believe require different implementations depending on the LINQ provider. Specifically I have a requirement to perform a SQL "WHERE ... IN ..." clause. I am able to implement this in the filter class using:
string[] aParams = // array of IDs
qry = qry.Where(t => aParams.Contains(t.ID));
However, in order to perform this against Entity Framework, I need to provide a solution such as the BuildContainsExpression which is tied to the Entity Framework. This means I have to have 2 different implementations of this particular filter, depending on the underlying provider.
I'd appreciate any advice on how I should proceed from here.
It seemed to me that exposing an IQueryable from my repository, would allow me to perform filters on it regardless of the underlying provider, enabling me to switch between providers if and when required. However the problem I describe above makes me think I should be performing all my filtering within the repositories and returning IEnumerable, IList or single classes.
Many thanks,
Matt
This is a very popular question. One that I constantly ask myself. I've always felt it best to return IEnumerable rather than IQueryable from a repository.
The purpose of a repository is to encapsulate the database infrastructure so the client need not worry about the data source. However, if you return IQueryable you are at the mercy of the consumer as to what kind of query will get run against your db, and whether they will do something that the LINQ provider doesn't support.
Take paging for example. Lets say you have a Customer entity and your database could have hundreds of thousands of customers. Which code would you rather have your client write?
var customers = repos.GetCustomers().Skip(skipCount).Take(pageSize).ToList();
OR
var customers = repos.GetCustomers(pageIndex, pageSize);
In the first approach you make it impossible for the repository to restrict the number of records retrieved from the data source. Also, your consumer has to calculate the skipCount.
In the second approach you provide a more coarse grained interface to your client. Now your repository can enforce some constraints on the pageSize in order to optimize the query. You also encapsulate the calculation of the skipCount.
However, that being said, in your situation your client is your service. So I suppose the question really comes down to a separation of concerns. Where is it better to perform such validation logic? Well that answer may very well be "in the service". But what about the answer to "Where is it better to contain query logic?". To me the answer is clearly "The Repository". That is its intended area of expertise.
I am very interested in Linq to SQL with Lazy load feature. And in my project I used AutoMapper to map DB Model to Domain Model (from DB_RoleInfo to DO_RoleInfo). In my repository code as below:
public DO_RoleInfo SelectByKey(Guid Key)
{
return SelectAll().Where(x => x.Id == Key).SingleOrDefault();
}
public IQueryable<DO_RoleInfo> SelectAll()
{
Mapper.CreateMap<DB_RoleInfo, DO_RoleInfo>();
return from role in _ctx.DB_RoleInfo
select Mapper.Map<DB_RoleInfo, DO_RoleInfo>(role);
}
SelectAll method is run well, but when I call SelectByKey, I get the error:
Method “RealMVC.Data.DO_RoleInfo MapDB_RoleInfo,DO_RoleInfo” could not translate to SQL.
Is it that Automapper doesn't support Linq completely?
Instead of Automapper, I tried the manual mapping code below:
public IQueryable<DO_RoleInfo> SelectAll()
{
return from role in _ctx.DB_RoleInfo
select new DO_RoleInfo
{
Id = role.id,
name = role.name,
code = role.code
};
}
This method works the way I want it to.
While #Aaronaught's answer was correct at the time of writing, as often the world has changed and AutoMapper with it. In the mean time, QueryableExtensions were added to the code base which added support for projections that get translated into expressions and, finally, SQL.
The core extension method is ProjectTo1. This is what your code could look like:
using AutoMapper.QueryableExtensions;
public IQueryable<DO_RoleInfo> SelectAll()
{
Mapper.CreateMap<DB_RoleInfo, DO_RoleInfo>();
return _ctx.DB_RoleInfo.ProjectTo<DO_RoleInfo>();
}
and it would behave like the manual mapping. (The CreateMap statement is here for demonstration purposes. Normally, you'd define mappings once at application startup).
Thus, only the columns that are required for the mapping are queried and the result is an IQueryable that still has the original query provider (linq-to-sql, linq-to-entities, whatever). So it is still composable and this will translate into a WHERE clause in SQL:
SelectAll().Where(x => x.Id == Key).SingleOrDefault();
1 Project().To<T>() prior to v. 4.1.0
Change your second function to this:
public IEnumerable<DO_RoleInfo> SelectAll()
{
Mapper.CreateMap<DB_RoleInfo, DO_RoleInfo>();
return from role in _ctx.DB_RoleInfo.ToList()
select Mapper.Map<DB_RoleInfo, DO_RoleInfo>(role);
}
AutoMapper works just fine with Linq to SQL, but it can't be executed as part of the deferred query. Adding ToList() at the end of your Linq query causes it to immediately evaluate the results, instead of trying to translate the AutoMapper segment as part of the query.
Clarification
The notion of deferred execution (not "lazy load") does not make any sense once you've changed the resulting type to something that's not a data entity. Consider these two classes:
public class DB_RoleInfo
{
public int ID { get; set; }
public string Name { get; set; }
}
public class DO_RoleInfo
{
public Role Role { get; set; } // Enumeration type
}
Now consider the following mapping:
Mapper.CreateMap<DB_RoleInfo, DO_RoleInfo>
.ForMember(dest => dest.Role, opt => opt.MapFrom(src =>
(Role)Enum.Parse(typeof(Role), src.Name)));
This mapping is completely fine (unless I made a typo), but let's say you write the SelectAll method in your original post instead of my revised one:
public IQueryable<DO_RoleInfo> SelectAll()
{
Mapper.CreateMap<DB_RoleInfo, DO_RoleInfo>();
return from role in _ctx.DB_RoleInfo
select Mapper.Map<DB_RoleInfo, DO_RoleInfo>(role);
}
This actually kind of works, but by calling itself a "queryable", it lies. What happens if I try to write this against it:
public IEnumerable<DO_RoleInfo> SelectSome()
{
return from ri in SelectAll()
where (ri.Role == Role.Administrator) ||
(ri.Role == Role.Executive)
select ri;
}
Think really hard about this. How could Linq to SQL possibly be able to successfully turn your where into an actual database query?
Linq knows nothing about the DO_RoleInfo class. It doesn't know how to do the mapping backward - in some cases, that may not even possible. Sure, you may look at this code and go "Oh, that's easy, just search for 'Administrator' or 'Executive' in the Name column", but you're the only one who knows that. As far as Linq to SQL is concerned, the query is pure nonsense.
Imagine that somebody gave you these instructions:
Go to the supermarket and bring back the ingredients for making Morton Thompson Turkey.
Unless you've made it before, and most people haven't, your response to that instruction is most likely going to be:
What the hell is that?
You can go to the market, and you can get specific ingredients by name, but you can't evaluate the condition I've given you while you're over there. I have to "un-map" the criteria first. I have to tell you, here are the ingredients we need for this recipe - now go and get them.
To summarize, this is not some simple incompatibility between Linq to SQL and AutoMapper. It is not unique to either of those two libraries. It doesn't matter how you actually do the mapping to a non-entity type - you could just as easily do the mapping manually, and you'd still get the same error, because you are now giving Linq to SQL a set of instructions that are no longer comprehensible, dealing with mysterious classes that don't have an intrinsic mapping to any particular entity type.
This issue is fundamental to the concept of O/R Mapping and deferred query execution. A projection is a one-way operation. Once you project, you can no longer go back to the query engine and say oh by the way, here are some more conditions for you. It's too late. The best you can do is take what it already gave you and evaluate the extra conditions yourself.
Last but not least, I'll leave you with a workaround. If the only thing you want to be able to do from your mapping is filter the rows, you can write this:
public IEnumerable<DO_RoleInfo> SelectRoles(Func<DB_RoleInfo, bool> selector)
{
Mapper.CreateMap<DB_RoleInfo, DO_RoleInfo>();
return _ctx.DB_RoleInfo
.Where(selector)
.Select(dbr => Mapper.Map<DB_RoleInfo, DO_RoleInfo>(dbr));
}
This is a utility method that handles the mapping for you and accepts a filter on the original entity, and not the mapped entity. It might be useful if you have many different kinds of filters but always need to do the same mapping.
Personally, I think you will be better off just writing out the queries properly, by first determining what you need to retrieve from the database, then doing any projections/mappings, and then, finally, if you need to do further filtering (which you shouldn't), then materialize the results with ToList() or ToArray() and write more conditions against the local list.
Don't try to use AutoMapper or any other tool to hide the real entities exposed by Linq to SQL. The domain model is your public interface. The queries you write are an aspect of your private implementation. It's important to understand the difference and maintain a good separation of concerns.