How can I query substring of nullable varchar with linq to sql? - linq-to-sql

I'm trying to import address data from some a database where there was not much data integrity. So there are a number of addresses (even in the US) that don't have postal codes and they're being read in as NULL.
I'm trying to do some matching of these addresses against an existing clean address database. I'm determining matches based on Addressee (company name), State (District), City (Locality) and either Street1 OR the first 5 of the postal code.
I tried this:
//This is just coded for the example -- In my routine, potentialAddress
//is coming from a data source where Postal Code may or may not be null.
Address potentialAddress = new Address() {
Street1 = "2324 Lakeview Drive",
PostalCode = null,
CountryId = 234, //US
Locality = "Sanford",
District = "FL"
};
//What I want here is Country & District have to match and either
//Street matches OR (if this is a US address) the first 5 of the postal code matches
_context.Addresses.Where(a => ((a.Street1 == potentialAddress.Street1)
|| (a.PostalCode != null && potentialAddress.PostalCode != null && potentialAddress.PostalCode.SubString(0,5) == a.PostalCode.SubString(0,5))
&& a.CountryId == potentialAddress.CountryId
&& a.District == potentialAddress.District).Select(a => a.Id).ToList();
I'm constantly getting an error message whenever potentialAddress is null. I'm getting:
Object reference not set to an instance of an object
when the query generator tries to parse potentialAddress.SubString(..).
I don't want to call it a match by postal code if one or the other (or both) are null.
Any ideas?

It seems as though the issue had to do with evaluating the potentialAddress object within the query. IOW, a.PostalCode != null was evaluated and a false value would stop any further processing of the condition whereas potentialAddress.PostalCode != null seemed to be ignored and the evaluation of the statement continued -- causing an error when it hit potentialAddress.PostalCode.SubString(0,5).
The only way I got around this was to move the conditional code dealing with potentialAddress outside of the query.
So, I now have something like this:
if(potentialAddress.PostalCode == null || potentialAddress.PostalCode.Length < 5)
//Perform query that ignores postal code
else
//Perform query that performs substring of postal code comparison
endif
Sure would've been nice if I could do that all in the query itself. Maybe someone could explain why the evaluation of potentialAddress.PostalCode in the query doesn't seem to work.

Related

Linq to EF not returning all data

I have following query against EF whereby mysql was used:
var query = from r in context.myContext
where r.clmn1.CompareTo("2015-11-19 00:00:00") > 0)
orderby r.someColumn
select r;
return query;
The number of returned rows is as expected. however some values of the property r.clmn2 repeat itself in the result of the query. For example I could not find clmn2 == 220011 because it was "overwritten" by the value 220033 (The value 220033 is correct and expected but should not "overwrite" other values). Strangely enough, when I add this condition to the query I get it in the result (of course then only and only this value) which means that the first condition is also valid for clmn2:
var query = from r in context.myContext
where r.clmn1.CompareTo("2015-11-19 00:00:00") > 0) && r.clmn2.Equals("220011")
orderby r.someColumn
select r;
return query;
The same query (the first one) works at DB-level and returns all values (will not be overwritten)
SELECT * FROM myContext.myTable
WHERE r.clmn1 > ("2015-11-19 00:00:00")
ORDER BY r.someColumn
It should be a problem of EF. I hope someone could help me!
Thanks in Advance.
I have prefixed the column/property clmn2 with [key] atribute in the generated entity class so that it is now a part of the multiple key, i.e., with other columns/properties. It works and i get all values from DB. Maybe cus this property comes from a DB-view, Visual Studio could not recognize it as a primary key as done by other properties.

How to avoid inserting a field that is not null from csv to mysql using Perl?

I am inserting a csv file into MySQL using Perl.
I have successfully inserted some data into MySQL.
However, some fields in the csv file are null and the code cannot continue to run.
What is the correct way to express if statement to check if $rttjitter is not null, else print that row number, then execute?
i.e. 2 criteria: $rtt is not equal to 0 and $rttjitter is not null.
my $empty = q{};
my #nulls = ();
if ($rtt !=0 && $rttjitter ne undef)
^
These do not work quite well.
You can use the built-in defined function to explicitly check if a variable is defined, so e.g.
if( defined( $rttjitter ) and $rttjitter ) { ...
Will check that it's defined and has a value that is not 0 or an empty string.
It seems likely though you should take another look at how you're reading in your CSV (you are using a module like Text::CSV, right?), and/or how you're doing the DB insertion, as generally with this sort of thing the undef value will cleanly insert as a NULL into the DB and should not cause any problems with your code, unless you're doing something strange.
if (defined($rttjitter) && defined($rtt) && $rtt != 0)
{
... # $rttjitter is not null (undef) and $rtt is not null (undef) and nonzero
}
else
{
... # at least one column fails the test
}

Alternative way to do FirstOrDefault in LINQ

I've got a membership table that records whether a user is a member of a list. When an update to a user's membership occurs a new record is written and the previous records are left as is which allows a history of their memberships to be maintained. To get a user's membership status involves selecting their most recent entry.
An example of some user list membership data is below. The aim is to find a LINQ expression that groups by list and user but only returns the row with most recently inserted record.
List Name, Username, Comment, ExpiresOn, Inserted
Test List, joeb, second update, 2012-03-13 16:55:03, 2012-01-31 22:28:40
Test List, joeb, first update, 2012-02-13 16:55:01, 2012-01-31 22:28:39
Test List, joeb, initial, 2012-01-13 16:55:02, 2012-01-31 22:28:38
An SQL query illustrates how the current list membership status can be extracted.
select ulm2.ID, ulm2.ExpiresOn, ulm2.Comment, ulm2.Inserted
from UserListMembership as ulm1
left outer join UserListMembership ulm2 on ulm1.id = ulm2.id
group by ulm1.userlistid, ulm1.userid;
The question is how to write a LINQ expression for the query that doesn't use a nested FirstOrDefault call which causes my MySQL entity framework provider to throw a "System.NotSupportedException: Specified method is not supported." exception?
Update:
Below is my failing LINQ expression, it throws the "Specified method is not supported" once the FirstOrDefault call is added.
var query = from mem in db.UserListMemberships
where
mem.User.UserUsernames.Any(y => y.Username.ToLower() == username.ToLower())
&& mem.UserList.Account.Subscriptions.Any(x => x.ID == subscriptionID)
&& mem.ExpiresOn > utcNow
group mem by new { mem.UserListID, mem.UserID } into g
select new { UserListMembership = (from mem2 in db.UserListMemberships where mem2.UserListID == g.Key.UserListID && mem2.UserID == g.Key.UserID orderby mem2.Inserted descending select mem2).FirstOrDefault() };
return query.Select(a => a.UserListMembership).ToList();
Without posting code it's hard for anyone to see what your issue is.
Are you not able to just order the results by your date field and then select the first record? Something like users.OrderByDescending(u => u.Inserted).FirstOrDefault()?

Use of custom expression in LINQ leads to a query for each use

I have the following problem: In our database we record helpdesk tickets and we book hours under tickets. Between those is a visit report. So it is: ticket => visitreport => hours.
Hours have a certain 'kind' which is not determined by a type indicator in the hour record, but compiled by checking various properties of an hour. For example, an hour which has a customer but is not a service hour is always an invoice hour.
Last thing I want is that the definitions of those 'kinds' roam everywhere in the code. They must be at one place. Second, I want to be able to calculate totals of hours from various collections of hours. For example, a flattened collection of tickets with a certain date and a certain customer. Or all registrations which are marked as 'solution'.
I have decided to use a 'layered' database access approach. The same functions may provide data for screen representation but also for a report in .pdf . So the first step gathers all relevant data. That can be used for .pdf creation, but also for screen representation. In that case, it must be paged and ordered in a second step. That way I don't need separate queries which basically use the same data.
The amount of data may be large, like the creation of year totals. So the data from the first step should be queryable, not enumerable. To ensure I stay queryable even when I add the summation of hours in the results, I made the following function:
public static decimal TreeHours(this IEnumerable<Uren> h, FactHourType ht)
{
IQueryable<Uren> hours = h.AsQueryable();
ParameterExpression pe = Expression.Parameter(typeof(Uren), "Uren");
Expression left = Expression.Property(pe, typeof(Uren).GetProperty("IsOsab"));
Expression right = Expression.Constant(true, typeof(Boolean));
Expression isOsab = Expression.Equal(Expression.Convert(left, typeof(Boolean)), Expression.Convert(right, typeof(Boolean)));
left = Expression.Property(pe, typeof(Uren).GetProperty("IsKlant"));
right = Expression.Constant(true, typeof(Boolean));
Expression isCustomer = Expression.Equal(Expression.Convert(left, typeof(Boolean)), Expression.Convert(right, typeof(Boolean)));
Expression notOsab;
Expression notCustomer;
Expression final;
switch (ht)
{
case FactHourType.Invoice:
notOsab = Expression.Not(isOsab);
final = Expression.And(notOsab, isCustomer);
break;
case FactHourType.NotInvoice:
notOsab = Expression.Not(isOsab);
notCustomer = Expression.Not(isCustomer);
final = Expression.And(notOsab, notCustomer);
break;
case FactHourType.OSAB:
final = Expression.And(isOsab, isCustomer);
break;
case FactHourType.OsabInvoice:
final = Expression.Equal(isCustomer, Expression.Constant(true, typeof(Boolean)));
break;
case FactHourType.Total:
final = Expression.Constant(true, typeof(Boolean));
break;
default:
throw new Exception("");
}
MethodCallExpression whereCallExpression = Expression.Call(
typeof(Queryable),
"Where",
new Type[] { hours.ElementType },
hours.Expression,
Expression.Lambda<Func<Uren, bool>>(final, new ParameterExpression[] { pe })
);
IQueryable<Uren> result = hours.Provider.CreateQuery<Uren>(whereCallExpression);
return result.Sum(u => u.Uren1);
}
The idea behind this function is that it should remain queryable so that I don't switch a shipload of data to enumerable.
I managed to stay queryable until the end. In step 1 I gather the raw data. In step 2 I order the data and subsequently I page it. In step 3 the data is converted to JSon and sent to the client. It totals hours by ticket.
The problem is: I get one query for the hours for each ticket. That's hundreds of queries! That's too much...
I tried the following approach:
DataLoadOptions options = new DataLoadOptions();
options.LoadWith<Ticket>(t => t.Bezoekrapport);
options.LoadWith<Bezoekrapport>(b => b.Urens);
dc.LoadOptions = options;
Bezoekrapport is simply Dutch for 'visitreport'. When I look at the query which retrieves the tickets, I see it joins the Bezoekrapport/visitreport but not the hours which are attached to it.
A second approach I have used is manually joining the hours in LINQ, but that does not work as well.
I must do something wrong. What is the best approach here?
The following code snippets are how I retrieve the data. Upon calling toList() on strHours in the last step, I get a hailstorm of queries. I've been trying for two days to work around it but it just doesn't work... Something must be wrong in my approach or in the function TreeHours.
Step 1:
IQueryable<RelationHoursTicketItem> HoursByTicket =
from Ticket t in allTickets
let RemarkSolved = t.TicketOpmerkings.SingleOrDefault(tr => tr.IsOplossing)
let hours = t.Bezoekrapport.Urens.
Where(h =>
(dateFrom == null || h.Datum >= dateFrom)
&& (dateTo == null || h.Datum <= dateTo)
&& h.Uren1 > 0)
select new RelationHoursTicketItem
{
Date = t.DatumCreatie,
DateSolved = RemarkSolved == null ? (DateTime?)null : RemarkSolved.Datum,
Ticket = t,
Relatie = t.Relatie,
HoursOsab = hours.TreeHours(FactHourType.OSAB),
HoursInvoice = hours.TreeHours(FactHourType.Invoice),
HoursNonInvoice = hours.TreeHours(FactHourType.NotInvoice),
HoursOsabInvoice = hours.TreeHours(FactHourType.OsabInvoice),
TicketNr = t.Id,
TicketName = t.Titel,
TicketCategorie = t.TicketCategorie,
TicketPriority = t.TicketPrioriteit,
TicketRemark = RemarkSolved
};
Step 2
sort = sort ?? "TicketNr";
IQueryable<RelationHoursTicketItem> hoursByTicket = GetRelationHours(relation, dateFrom, dateTo, withBranches);
IOrderedQueryable<RelationHoursTicketItem> orderedResults;
if (dir == "ASC")
{
orderedResults = hoursByTicket.OrderBy(sort);
}
else
{
orderedResults = hoursByTicket.OrderByDescending(sort);
}
IEnumerable<RelationHoursTicketItem> pagedResults = orderedResults.Skip(start ?? 0).Take(limit ?? 25);
records = hoursByTicket.Count();
return pagedResults;
Step 3:
IEnumerable<RelationHoursTicketItem> hours = _hourReportService.GetRelationReportHours(relation, dateFrom, dateTo, metFilialen, start, limit, dir, sort, out records);
var strHours = hours.Select(h => new
{
h.TicketNr,
h.TicketName,
RelationName = h.Relatie.Naam,
h.Date,
TicketPriority = h.TicketPriority.Naam,
h.DateSolved,
TicketCategorie = h.TicketCategorie == null ? "" : h.TicketCategorie.Naam,
TicketRemark = h.TicketRemark == null ? "" : h.TicketRemark.Opmerking,
h.HoursOsab,
h.HoursInvoice,
h.HoursNonInvoice,
h.HoursOsabInvoice
});
I don't think your TreeHours extension method can be converted to SQL by LINQ in one go. So are evaluated on execution of each constructor of the row, causing a 4 calls to the database in this case per row.
I would simplfy your LINQ query to return you the raw data from SQL, using a simple JOIN to get all tickets and there hours. I would then group and filter the Hours by type in memory. Otherwise, if you really need to perform your operations in SQL then look at the CompiledQuery.Compile method. This should be able to handle not making a query per row. I'm not sure you'd get the switch in there but you may be able to convert it using the ?: operator.

Insert using linq templates not returning the id - MySQL

I'm using the latest subsonic dll and the latest linq templates from github. The db i'm inserting into is MySQL. Id column on table is primary key auto increment.
Versions:
Subsonic.Core.dll - 3.0.0.3 - (November 18, 2009 Merged pulls from Github).
LinqTemplates - July 29, 2009.
MySQL.Data.CF.dll - 6.1.2.0.
The row is inserted but the id is returned as 0.
Example of the insert:
mysqldb db = new mysqldb.mysqldbDB();
int ID = db.Insert.Into<db.myTable>(
r => r.message,
r => r.name,
r => r.status).Values(
message,
name,
status).Execute();
Am I doing something wrong? Shouldn't the new id be returned, not zero?
Found the bug in subsonic core.
It's in Subsonic.Core.Query.Insert.cs
The Execute method does not have a condition for id's returned that are of type long.
I've rewritten the method in my local version to:
public int Execute()
{
int returner = 0;
object result = _provider.ExecuteScalar(GetCommand());
if(result != null)
{
if(result.GetType() == typeof(decimal))
returner = Convert.ToInt32(result);
else if (result.GetType() == typeof(int))
returner = Convert.ToInt32(result);
else if (result.GetType() == typeof(long))
returner = Convert.ToInt32(result);
else
returner = Convert.ToInt32(result);
}
return returner;
}
I've changed the multiple if statements to else if's and added the type comparison of long. Also I've added the final else condition which does a convert to int. Not sure if that's such a good idea but it works for me.
If someone wants to update the source great. If i find time sometime soon i'll update it myself.