EF Core Full-text search: parameterized keySelector could not be translated into SQL - linq-to-sql

I'd like to create a generic extension method which allow me to use Full-text search.
● The code below works:
IQueryable<MyEntity> query = Repository.AsQueryable();
if (!string.IsNullOrEmpty(searchCondition.Name))
query = query.Where(e => EF.Functions.Contains(e.Name, searchCondition.Name));
return query.ToList();
● But I want a more-generic-way so I create the following extension method
public static IQueryable<T> FullTextContains<T>(this IQueryable<T> query, Func<T, string> keySelector, string value)
{
return query.Where(e => EF.Functions.Contains(keySelector(e), value));
}
When I call the exxtension method like below, I got an exception
IQueryable<MyEntity> query = Repository.AsQueryable();
if (!string.IsNullOrEmpty(searchCondition.Name))
query = query.FullTextContains(e => e.Name, searchCondition.Name);
return query.ToList();
> System.InvalidOperationException: 'The LINQ expression 'DbSet
> .Where(c => __Functions_0
> .Contains(
> _: Invoke(__keySelector_1, c[MyEntity])
> ,
> propertyReference: __value_2))' could not be translated. Either rewrite the query in a form that
> can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(),
> AsAsyncEnumerable(), ToList(), or ToListAsync().
> See https://go.microsoft.com/fwlink/?linkid=2101038 for more information
>
How do I "rewrite the query in a form that can be translated" as the Exception suggested?

Your are falling into the typical IQueryable<> trap by using delegates (Func<>) instead of expressions (Expression<Func<>). You can see that difference in the lambda arguments of every Queryable extension method vs corresponding Enumerable method. The difference is that the delegates cannot be translated (they are like unknown methods), while the expressions can.
So in order to do what you want, you have to change the signature of the custom method to use expression(s):
public static IQueryable<T> FullTextContains<T>(
this IQueryable<T> query,
Expression<Func<T, string>> keySelector, // <--
string value)
But now you have implementation problem because C# does not support syntax for "invoking" expressions similar to delegates, so the following
keySelector(e)
does not compile.
In order to do that you need at minimum a small utility for composing expressions like this:
public static partial class ExpressionUtils
{
public static Expression<Func<TOuter, TResult>> Apply<TOuter, TInner, TResult>(this Expression<Func<TOuter, TInner>> outer, Expression<Func<TInner, TResult>> inner)
=> Expression.Lambda<Func<TOuter, TResult>>(inner.Body.ReplaceParameter(inner.Parameters[0], outer.Body), outer.Parameters);
public static Expression<Func<TOuter, TResult>> ApplyTo<TInner, TResult, TOuter>(this Expression<Func<TInner, TResult>> inner, Expression<Func<TOuter, TInner>> outer)
=> outer.Apply(inner);
public static Expression ReplaceParameter(this Expression expression, ParameterExpression source, Expression target)
=> new ParameterReplacer { source = source, target = target }.Visit(expression);
class ParameterReplacer : ExpressionVisitor
{
public ParameterExpression source;
public Expression target;
protected override Expression VisitParameter(ParameterExpression node)
=> node == source ? target : node;
}
}
Use Apply or ApplyTo depending of what type of expression you have. Other than that they do the same.
In your case, the implementation of the method with Expression<Func<T, string>> keySelector would be
return query.Where(keySelector.Apply(key => EF.Functions.Contains(key, value)));

Related

EF Core 7 can't deserialize dynamic-members in JSON column

I am trying to map my Name column to a dynamic object. This is how the raw JSON data looks (note that this is SQL-morphed from our old relational data and I am not able to generate or interact with this column via EF Core):
{ "en": "Water", "fa": "آب", "ja": "水", ... }
Just to note, available languages are stored in a separate table and thus are dynamically defined.
Through T-SQL I can perfectly interact with these objects eg
SELECT *
FROM [MyObjects]
WHERE JSON_VALUE(Name, '$.' + #languageCode) = #searchQuery
But it seems EF Core doesn't want to even deserialize these objects as whole, let alone query them.
What I get in a simple GetAll query is an empty Name. Other columns are not affected though.
I have tried so far
Using an empty class with a [JsonExtensionData] dictionary inside
Using a : DynamicObject inheritance and implementing GetDynamicMembers, TryGetMember, TrySetMember, TryCreateInstance
Directly mapping to a string dictionary.
Combining 1 & 2 and adding an indexer operator on top.
All yield the same results: an empty Name.
I have other options like going back to a junction table relational which I have many issues with, hardcoding languages which is not really intuitive and might cause problems in the future, using HasJsonConversion which basically destroys the performance on any search action... so I'm basically stuck here with this.
I think currently it's not fully supported:
You can not use dynamic operations on an expression tree like a Select statement because it needs to be translated.
JsonValue and JsonQuery requires a path to be resolved.
If you specify OwnsOne(entity = >entity.owned, owned => owned.ToJson()) and the Json could not be parsed you will get an error.
I suggest this workaround while the EF team improves the functionality.
Create a static class with static methods to be used as decoys in the expression tree. This will be mapped to the server built-in functions.
public static class DBF
{
public static string JsonValue(this string column, [NotParameterized] string path)
=> throw new NotSupportedException();
public static string JsonQuery(this string column, [NotParameterized] string path) => throw new NotSupportedException();
}
Include the database functions on your OnModelCreating method.
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
base.OnModelCreating(modelBuilder);
modelBuilder.HasDbFunction(
typeof(DBF).GetMethod(nameof(DBF.JsonValue))!
).HasName("JSON_VALUE").IsBuiltIn();
modelBuilder.HasDbFunction(
typeof(DBF).GetMethod(nameof(DBF.JsonQuery))!
).HasName("JSON_QUERY").IsBuiltIn();
/// ...
modelBuilder.Entity(entity => {
//treat entity as text
entity.Property(x => x.Metadata)
.HasColumnType("varchar")
.HasMaxLength(8000);
});
}
Call them dynamically with LINQ.
var a = await _context.FileInformation
.AsNoTracking()
.Where(x => x.Metadata!.JsonValue("$.Property1") == "some value")
.Select(x => x.Metadata!.JsonValue("$.Property2"))
.ToListAsync();
You can add casts or even build anonymous types with this method.
My solution was I added a new class which has KEY and VALUE , which will represent the dictionary i needed :
public class DictionaryObject
{
public string Key { set; get; }
public string Value { set; get; }
}
and instead of having this line in the JSON class :
public Dictionary<string, string> Name { get; set; }
I changed to :
public List<DictionaryObject> Name { get; set; }
Hope it helps.

any() vs any(Class.class) Mockito

I am not able to understand why below two tests are not giving the same result.
#Service
public class SomeManager{
private final SomeDependency someDependency;
#Autowired
public SomeManager(SomeDependency someDependency){
this.someDependency = someDependency;
}
public List<returnType> methodToTest(Arg arg){
List<JsonObject> jo = someDependency.search(arg);
return jo.stream().map(returnType::parse).collect(Collectors.toList());
}
}
Test with any(). This Test pass.
#RunWith(SpringJUnit4ClassRunner.class)
public class TestMethodToTest(){
#Test
public void TestMethod(){
SomeDependency someDependency = mock(SomeDependency.class);
List<JsonObject> expected := \some valid list of JsonObject\
// Here I used any() method.
when(someDependency.search(any())).thenReturn(expected);
SomeManager someManager = new SomeManager(someDependency);
List<returnType> actual = someManager.methodToTest(any(Arg.class));
assertArrayEquals(acutal.toArray(), expected.stream().map(returnType::parse).toArray());
}
}
But since search(Arg arg) method of SomeDependency takes parameter of class Arg so I changed above test like this:
#RunWith(SpringJUnit4ClassRunner.class)
public class TestMethodToTest(){
#Test
public void TestMethod(){
SomeDependency someDependency = mock(SomeDependency.class);
List<JsonObject> expected := \some valid list of JsonObject\
// Here I used any(Arg.class) method.
when(someDependency.search(any(Arg.class))).thenReturn(expected);
SomeManager someManager = new SomeManager(someDependency);
List<returnType> actual = someManager.methodToTest(any(Arg.class));
assertArrayEquals(acutal.toArray(), expected.stream().map(returnType::parse).toArray());
}
}
This second test fails with output java.lang.AssertionError: array lengths differed, expected.length=1 actual.length=0.What's the possible reason behind this?
Note: The value expected.length=1 in output depends on what value is provided by the user as valid list of json objects in the test.
The difference stems from the fact that any matches null, while anyClass does not match null. See ArgumentMatchers javadoc:
any() Matches anything, including nulls and varargs.
any​(Class<T> type) Matches any object of given type, excluding nulls.
You are passing null to your method under test here:
List<returnType> actual = someManager.methodToTest(any(Arg.class));
any() returns null which you pass to method under test.
Note that using argument matchers this way is illegal - you should only call them inside calls to when and verify. You should pass a real instance of Arg to method under test.
See Mockito javadoc
Matcher methods like any(), eq() do not return matchers. Internally, they record a matcher on a stack and return a dummy value (usually null). This implementation is due to static type safety imposed by the java compiler. The consequence is that you cannot use any(), eq() methods outside of verified/stubbed method.

Spring Jpa Projection interface occur error with Boolean type

Q. Why JPA Projection can't convert Mysql bit(1) to Java Boolean?
Spring Jpa Projection occur error Projection type must be an interface! when the Mysql bit(1) type maps to the Java Boolean type.
Jpa converts a Boolean column in Entity class to bit(1) column in Mysql Table.
If I change getIsBasic's type in PlanInfoProjection interface Integer to Boolean, It doesn't work. Why does it occur error?
JPA Repository
#Query(nativeQuery=true, value="select true as isBasic from dual")
ProductOrderDto.PlanInfoProjection findPlanInfoById(Long id);
Projection interface
public class ProductOrderDto {
#Getter
public static class PlanInfo {
private Boolean isBasic;
public PlanInfo(PlanInfoProjection projection) {
// this.isBasic = projection.getIsBasic(); //<-- I want to use like this.
if (projection.getIsBasic() == null) {
this.isBasic = null;
} else {
this.isBasic = projection.getIsBasic() == 0 ? false : true; // <-- I have to convert
}
}
}
public interface PlanInfoProjection {
Integer getIsBasic(); // It works, but I have to convert Integer to Boolean to use.
//Boolean getIsBasic(); // doesn't work, but why???
//Boolean isBasic(); // also doesn't work
//boolean isBasic(); // also doesn't work
}
}
It seems like this doesn't work out of the box. What works for me (although I'm using DB2 so my datatype is different but this shouldn't be a problem) is to annotate it and use SpEL like this:
#Value("#{target.isBasic == 1}")
boolean getIsBasic();
This just takes your int value (0 for false, 1 for true) and creturns a boolean value. Should also work with Boolean but I didn't test it.
Another option is to use #Value("#{T(Boolean).valueOf(target.isBasic)}") but this only works for String values, so you would have to store 'true' or 'false' in your database. With T() you can import Static classes into Spring Expression Language, and then just call the valueOf method which returns a boolean (either Boolean or boolean)

Hamcrest: dump current type and value

I am writing test with usage of Java library Hamcrest and it's non-fluent API makes it impossible to reason about expression types when complex expression evolves, like:
.andExpect(JsonUnitResultMatchers.json()
.matches(CoreMatchers.anyOf(CoreMatchers.allOf(
JsonMatchers.jsonPartEquals("id", "123"),
JsonMatchers.jsonPartEquals("name", "test")))))
Is there always TRUE matcher that dumps type & value of currently active expression? Like:
.andExpect(JsonUnitResultMatchers.json()
.matches(CoreMatchers.anyOf(CoreMatchers.allOf(
Slf4jMatcher.logType(),
Slf4jMatcher.logTypeAndToString(),
ConsumerMatcher.apply(System.out::println),
JsonMatchers.jsonPartEquals("id", "123"),
JsonMatchers.jsonPartEquals("name", "test")))))
I don't like to step into Hamcrest code with debugger. It is unproductive to delve into someones internals.
I came up with ugly:
.andExpect(JsonUnitResultMatchers.json()
.matches(Matchers.hasItem(CoreMatchers.allOf(
new BaseMatcher() {
#Override
public boolean matches(Object item) {
log.info("type: {}", item.getClass());
log.info("toString: {}", item.toString());
return true;
}
#Override
public void describeTo(Description description) {}
},
JsonMatchers.jsonPartEquals("id", "123"),
JsonMatchers.jsonPartEquals("name", "test")))))
I hope there are some funny DSL...

LINQ to SQL extension method for sorting and paging

I have found an extension method that handles sorting and paging for LINQ. While this works well, I am trying to see if there are some other ways I can use this.
At present, the code for the extension method is as follows:
public static IQueryable<T> Page<T, TResult>(
this IQueryable<T> obj,
int page,
int pageSize,
System.Linq.Expressions.Expression<Func<T, TResult>> keySelector,
bool asc,
out int rowsCount)
{
rowsCount = obj.Count();
int innerRows = (page - 1) * pageSize;
if (asc)
return obj.OrderBy(keySelector).Skip(innerRows).Take(pageSize).AsQueryable();
else
return obj.OrderByDescending(keySelector).Skip(innerRows).Take(pageSize).AsQueryable();
}
The method takes in an expression, which is based off the type.
In my Dealer class, I have a method GetDealers, which essentially calls this,
i.e.
db.User.Page(1, 2, p => p.User.UserProperty.Name, true, out rowCount)
From the presentation side of things though, I do not know or can access the expression as above, e.g.
ListView1.DataSource = users.GetDealers("SortColumn", pageNo, pageSize, out rowCount, bool asc);
ListView1.DataBind();
The only way is to have a switch statement in my GetDealers method that would then convert to the expression. Is there a way to bypass this, or is this method OK?
I'm not exactly sure what you're asking, but I believe it's something that I have looked into myself. If you would like to know how to dynamically sort your results based on a string, rather than a proper LINQ expression, then you're in luck.
Scott Guthrie published a great article on that very topic. It references a Microsoft file which extends any IQueryable object to support dynamic sorting.
C# Dynamic Query Library (included in the \LinqSamples\DynamicQuery directory). Just add the page to your App_Code folder and include "Using System.Linq.Dynamic" in your project and you will be able to use the following syntax:
myUsers = myUsers.OrderBy("LastName");
I hope this helps!
If you are looking for extension method to work on all types
public static class SortingAndPagingHelper
{
/// <summary>
/// Returns the list of items of type on which method called
/// </summary>
/// <typeparam name="TSource">This helper can be invoked on IEnumerable type.</typeparam>
/// <param name="source">instance on which this helper is invoked.</param>
/// <param name="sortingModal">Page no</param>
/// <returns>List of items after query being executed on</returns>
public static IEnumerable<TSource> SortingAndPaging<TSource>(this IEnumerable<TSource> source, SortingAndPagingInfo sortingModal)
{
// Gets the coloumn name that sorting to be done o`enter code here`n
PropertyInfo propertyInfo = source.GetType().GetGenericArguments()[0].GetProperty(sortingModal.SortColumnName);
// sorts by ascending if sort criteria is Ascending otherwise sorts descending
return sortingModal.SortOrder == "Ascending" ? source.OrderByDescending(x => propertyInfo.GetValue(x, null)).Skip(sortingModal.PageSelected * sortingModal.PageSize).Take(sortingModal.PageSize)
: source.OrderBy(x => propertyInfo.GetValue(x, null)).Skip(sortingModal.PageSelected * sortingModal.PageSize).Take(sortingModal.PageSize);
}
}
DbContext dbContext = new DbContext();
dbContext.rainingSessions.Where(x => x.RegistrationDeadline > DateTime.Now) .SortingAndPaging(sortAndPagingInfo).ToList()