Here is some code:
//all possible search terms of interest
searchTerms = from s in dc.SearchTerms
select s.term;
//all possible results
var results = from r in dc.Data
select r.hyperlinks;
I want to perform an operation where I get all "r.hyperlinks" that contains s.term.
It is something like r.hyperlinks.Contains(s.term). How can I do this?
It's almost as you wrote it in english:
var results = from r in dc.Data
where searchTerms.Any(x => r.hyperlinks.Contains(x))
select r.hyperlinks;
That's all!
You can put any condition you might come up inside a where clause. Actually, you can put whatever returns a boolean.
Local sequences cannot be used in many LinqToSql operators. But your original question didn't require a local sequence.
var results =
from r in dc.Data
where dc.SearchTerms.Any(s => r.hyperlinks.Contains(s.Term))
select r.hyperlinks;
Related
I've been stuck on a GORM issue for about a full day now. I need to be able to filter a messages table on any of 4 things: sender, recipient, keyword, and date range. It also has to paginate. Filtering by sender and recipient is working, and so is pagination. So far this is the query that I have come up with, but it does not seem to work for date ranges or keywords.
Here is how I am selecting from MySQL
db.Preload("Thread").Where(query).Scopes(Paginate(r)).Find(&threadMessages)
I am creating the query like this:
var query map[string]interface{}
Then based on which parameters I am passed, I update the query like this by adding new key values to the map:
query = map[string]interface{}{"user_id": sender, "recipient_id": recipient}
For dates it does not seem to work if I try something like this:
query = map[string]interface{}{"created_at > ?": fromDate}
And for a LIKE condition is also does not seem to work:
query = map[string]interface{}{"contents LIKE ?": keyword}
The reason I chose this approach is that I could not seem to get optional inputs to work in .Where since it takes a string with positional parameters and null positional parameters seem to cause MySQL to return an empty array. Has anyone else dealt with a complicated GORM issue like this? Any help is appreciated at this point.
Passing the map[string]interface{} into Where() only appears to work for Equals operations, or IN operations (if a slice is provided as the value instead).
One way to achieve what you want, is to construct a slice of clause.Expression, and append clauses to the slice when you need to. Then, you can simply pass in all of the clauses (using the ... operator to pass in the whole slice) into db.Clauses().
clauses := make([]clause.Expression, 0)
if mustFilterCreatedAt {
clauses = append(clauses, clause.Gt{Column: "created_at", fromDate})
}
if mustFilterContents {
clauses = append(clauses, clause.Like{Column: "contents", Value: keyword})
}
db.Preload("Thread").Clauses(clauses...).Scopes(Paginate(r)).Find(&threadMessages)
Note: If you're trying to search for content that contains keyword, then you should concatenate the wildcard % onto the ends of keyword, otherwise LIKE behaves essentially the same as =:
clause.Like{Column: "contents", Value: "%" + keyword + "%"}
My final solution to this was to create dynamic Where clauses based on which query params were sent from the client like this:
fields := []string{""}
values := []interface{}{}
If, for example, there is a keyword param:
fields = []string{"thread_messages.contents LIKE ?"}
values = []interface{}{"%" + keyword + "%"}
And to use the dynamic clauses in the below query:
db.Preload("Thread", "agency_id = ?", agencyID).Preload("Thread.ThreadUsers", "agency_id = ?", agencyID).Joins("JOIN threads on thread_messages.thread_id = threads.id").Where("threads.agency_id = ?", agencyID).Where(strings.Join(fields, " AND "), values...).Scopes(PaginateMessages(r)).Find(&threadMessages)
Currently when I need to run a query that will be used w/ paging I do it something like this:
//Setup query (Typically much more complex)
var q = ctx.People.Where(p=>p.Name.StartsWith("A"));
//Get total result count prior to sorting
int total = q.Count();
//Apply sort to query
q = q.OrderBy(p => p.Name);
q.Select(p => new PersonResult
{
Name = p.Name
}.Skip(skipRows).Take(pageSize).ToArray();
This works, but I wondered if it is possible to improve this to be more efficient while still using linq? I couldn't think of a way to combine the count w/ the data retrieval in a single trip to the DB w/o using a stored proc.
The following query will get the count and page results in one trip to the database, but if you check the SQL in LINQPad, you'll see that it's not very pretty. I can only imagine what it would look like for a more complex query.
var query = ctx.People.Where (p => p.Name.StartsWith("A"));
var page = query.OrderBy (p => p.Name)
.Select (p => new PersonResult { Name = p.Name } )
.Skip(skipRows).Take(pageSize)
.GroupBy (p => new { Total = query.Count() })
.First();
int total = page.Key.Total;
var people = page.Select(p => p);
For a simple query like this, you could probably use either method (2 trips to the database, or using GroupBy to do it in 1 trip) and not notice much difference. For anything complex, I think a stored procedure would be the best solution.
Jeff Ogata's answer can be optimized a little bit.
var results = query.OrderBy(p => p.Name)
.Select(p => new
{
Person = new PersonResult { Name = p.Name },
TotalCount = query.Count()
})
.Skip(skipRows).Take(pageSize)
.ToArray(); // query is executed once, here
var totalCount = results.First().TotalCount;
var people = results.Select(r => r.Person).ToArray();
This does pretty much the same thing except it won't bother the database with an unnecessary GROUP BY. When you are not certain your query will contain at least one result, and don't want it to ever throw an exception, you can get totalCount in the following (albeit less cleaner) way:
var totalCount = results.FirstOrDefault()?.TotalCount ?? query.Count();
Important Note for People using EF Core >= 1.1.x && < 3.0.0:
At the time I was looking for solution to this and this page is/was Rank 1 for the google term "EF Core Paging Total Count".
Having checked the SQL profiler I have found EF generates a SELECT COUNT(*) for every row that is returned. I have tired every solution provided on this page.
This was tested using EF Core 2.1.4 & SQL Server 2014. In the end I had to perform them as two separate queries like so. Which, for me at least, isn't the end of the world.
var query = _db.Foo.AsQueryable(); // Add Where Filters Here.
var resultsTask = query.OrderBy(p => p.ID).Skip(request.Offset).Take(request.Limit).ToArrayAsync();
var countTask = query.CountAsync();
await Task.WhenAll(resultsTask, countTask);
return new Result()
{
TotalCount = await countTask,
Data = await resultsTask,
Limit = request.Limit,
Offset = request.Offset
};
It looks like the EF Core team are aware of this:
https://github.com/aspnet/EntityFrameworkCore/issues/13739
https://github.com/aspnet/EntityFrameworkCore/issues/11186
I suggest making two queries for the first page, one for the total count and one for the first page or results.
Cache the total count for use as you move beyond the first page.
I have 2 tables that I need to load together all the time, the both must exist together in the database. However I am wondering why Linq to Sql demands that I have to load in a collection and then do a join, I only want to join 2 single tables where a record where paramid say = 5, example...
var data = _repo.All<TheData>(); //why do I need a collection/IQueryable like this?
var _workflow = _repo.All<WorkFlow>()
.Where(x => x.WFID== paramid)
.Join(data, x => x.ID, y => y.WFID, (x, y) => new
{
data = x,
workflow = y
});
I gues then I need to do a SingleOrDefault()? If the record is not null pass it back?
I Understand the Sql query comes out correctly, is there a better way to write this?
NOTE: I need to search a table called Participants to see if the loggedonuser can actually view this record, so I guess I should leave it as this? (this is main requirement)
var participant = _repo.All<Participants>();
.Any(x=> x.ParticipantID == loggedonuser.ID); //add this to above query...
The line var data = _repo.All<TheData>(); is something like saying 'start building query against the TheData table'.
This function returns you an IQueryable which will contain a definition of the query against your database.
So this doesn't mean you load the whole TheData table data with this line!
The query will be executed the moment you do something like .Count(), .Any(), First(), Single(), or ToList(). This is called deferred execution.
If you would end your query with SingleOrDefault() this will create a sql query that joins the two tables, add the filter and select the top most record or null(or throw an error if there are more!).
You could also use Linq instead of query extension methods.
It would look like:
var data = _repo.All<TheData>();
var _workflow = from w in _repo.All<WorkFlow>()
join t in _repo.All<TheData> on w.Id equals t.WFID
where x.WIFD = paramid
select new
{
data = t,
workflow = x
});
I've stumbled upon a very strange LINQ to SQL behaviour / bug, that I just can't understand.
Let's take the following tables as an example: Customers -> Orders -> Details.
Each table is a subtable of the previous table, with a regular Primary-Foreign key relationship (1 to many).
If I execute the follow query:
var q = from c in context.Customers
select (c.Orders.FirstOrDefault() ?? new Order()).Details.Count();
Then I get an exception: Could not format node 'Value' for execution as SQL.
But the following queries do not throw an exception:
var q = from c in context.Customers
select (c.Orders.FirstOrDefault() ?? new Order()).OrderDateTime;
var q = from c in context.Customers
select (new Order()).Details.Count();
If I change my primary query as follows, I don't get an exception:
var q = from r in context.Customers.ToList()
select (c.Orders.FirstOrDefault() ?? new Order()).Details.Count();
Now I could understand that the last query works, because of the following logic:
Since there is no mapping of "new Order()" to SQL (I'm guessing here), I need to work on a local list instead.
But what I can't understand is why do the other two queries work?!?
I could potentially accept working with the "local" version of context.Customers.ToList(), but how to speed up the query?
For instance in the last query example, I'm pretty sure that each select will cause a new SQL query to be executed to retrieve the Orders. Now I could avoid lazy loading by using DataLoadOptions, but then I would be retrieving thousands of Order rows for no reason what so ever (I only need the first row)...
If I could execute the entire query in one SQL statement as I would like (my first query example), then the SQL engine itself would be smart enough to only retrieve one Order row for each Customer...
Is there perhaps a way to rewrite my original query in such a way that it will work as intended and be executed in one swoop by the SQL server?
EDIT:
(longer answer for Arturo)
The queries I provided are purely for example purposes. I know they are pointless in their own right, I just wanted to show a simplistic example.
The reason your example works is because you have avoided using "new Order()" all together. If I slightly modify your query to still use it, then I still get an exception:
var results = from e in (from c in db.Customers
select new { c.CustomerID, FirstOrder = c.Orders.FirstOrDefault() })
select new { e.CustomerID, Count = (e.FirstOrder != null ? e.FirstOrder : new Order()).Details().Count() }
Although this time the exception is slightly different - Could not format node 'ClientQuery' for execution as SQL.
If I use the ?? syntax instead of (x ? y : z) in that query, I get the same exception as I originaly got.
In my real-life query I don't need Count(), I need to select a couple of properties from the last table (which in my previous examples would be Details). Essentially I need to merge values of all the rows in each table. Inorder to give a more hefty example I'll first have to restate my tabels:
Models -> ModelCategoryVariations <- CategoryVariations -> CategoryVariationItems -> ModelModuleCategoryVariationItemAmounts -> ModelModuleCategoryVariationItemAmountValueChanges
The -> sign represents a 1 -> many relationship. Do notice that there is one sign that is the other way round...
My real query would go something like this:
var q = from m in context.Models
from mcv in m.ModelCategoryVariations
... // select some more tables
select new
{
ModelId = m.Id,
ModelName = m.Name,
CategoryVariationName = mcv.CategoryVariation.Name,
..., // values from other tables
Categories = (from cvi in mcv.CategoryVariation.CategoryVariationItems
let mmcvia = cvi.ModelModuleCategoryVariationItemAmounts.SingleOrDefault(mmcvia2 => mmcvia2.ModelModuleId == m.ModelModuleId) ?? new ModelModuleCategoryVariationItemAmount()
select new
{
cvi.Id,
Amount = (mmcvia.ModelModuleCategoryVariationItemAmountValueChanges.FirstOrDefault() ?? new ModelModuleCategoryVariationItemAmountValueChange()).Amount
... // select some more properties
}
}
This query blows up at the line let mmcvia =.
If I recall correctly, by using let mmcvia = new ModelModuleCategoryVariationItemAmount(), the query would blow up at the next ?? operand, which is at Amount =.
If I start the query with from m in context.Models.ToList() then everything works...
Why are you looking into only the individual count without selecting anything related to the customer.
You can do the following.
var results = from e in
(from c in db.Customers
select new { c.CustomerID, FirstOrder = c.Orders.FirstOrDefault() })
select new { e.CustomerID, DetailCount = e.FirstOrder != null ? e.FirstOrder.Details.Count() : 0 };
EDIT:
OK, I think you are over complicating your query.
The problem is that you are using the new WhateverObject() in your query, T-SQL doesnt know anyting about that; T-SQL knows about records in your hard drive, your are throwing something that doesn't exist. Only C# knows about that. DON'T USE new IN YOUR QUERIES OTHER THAN IN THE OUTER MOST SELECT STATEMENT because that is what C# will receive, and C# knows about creating new instances of objects.
Of course is going to work if you use ToList() method, but performance is affected because now you have your application host and sql server working together to give you the results and it might take many calls to your database instead of one.
Try this instead:
Categories = (from cvi in mcv.CategoryVariation.CategoryVariationItems
let mmcvia =
cvi.ModelModuleCategoryVariationItemAmounts.SingleOrDefault(
mmcvia2 => mmcvia2.ModelModuleId == m.ModelModuleId)
select new
{
cvi.Id,
Amount = mmcvia != null ?
(mmcvia.ModelModuleCategoryVariationItemAmountValueChanges.Select(
x => x.Amount).FirstOrDefault() : 0
... // select some more properties
}
Using the Select() method allows you to get the first Amount or its default value. I used "0" as an example only, I dont know what is your default value for Amount.
So, I'm trying to return a collection of People whose ID is contained within a locally created collection of ids ( IQueryable)
When I specify "locally created collection", I mean that the Ids collection hasnt come from a LinqToSql query and has been programatically created (based upon user input).
My query looks like this:
var qry = from p in DBContext.People
where Ids.Contains(p.ID)
select p.ID;
This causes the following exception...
"queries with local collections are not supported"
How can I find all the People with an id that is contained within my locally created Ids collection?
Is it possible using LinqToSql?
If Ids is a List, array or similar, L2S will translate into a contains.
If Ids is a IQueryable, just turn it into a list before using it in the query. E.g.:
List<int> listOfIDs = IDs.ToList();
var query =
from st in dc.SomeTable
where listOfIDs.Contains(st.ID)
select .....
I was struggling with this problem also. Solved my problem with using Any() instead
people.Where(x => ids.Any(id => id == x.ID))
As the guys mentioned above, converting the ids, which is of type IQueryable to List or Array will solve the issue, this will be translated to "IN" operator in SQL.But be careful because if the count of ids >= 2100 this will cause another issue which is "The server supports a maximum of 2100 parameters" and that is the maximum number of parameters(values) you can pass to "IN" in SQL server.
Another alternative would be keeping ids as IQueryable and using LINQ "Any" operator instead of "Contains", this will be translated to "EXISTS" in SQL server.
I'm sorry but the answers here didn't work for me as I'm doing dynamic types further along.
What I did was to use "UNION" in a loop which works great. Here's how:
var firstID = cityList.First().id;
var cities = dc.zs_Cities.Where(c => c.id == firstID);
foreach(var c in cityList)
{
var tempCity = c;
cities = cities.Union(dc.zs_Cities.Where(cty => cty.id == tempCity.id));
}