I have the following query
#initial_matches = Listing.find_by_sql(["SELECT * FROM listings WHERE industry = ?", current_user.industry])
Is there a way I can run another SQL query on the selection from the above query using a each do? I want to run geokit calculations to eliminate certain listings that are outside of a specified distance...
Your question is slightly confusing. Do you want to use each..do (ruby) to do the filtering. Or do you want to use a sql query. Here is how you can let the ruby process do the filtering
refined list = #initial_matches.map { |listing|
listing.out_of_bounds? ? nil : listing
}.comact
If you wanted to use sql you could simply add additional sql (maybe a sub-select) it into your Listing.find_by_sql call.
If you want to do as you say in your comment.
WHERE location1.distance_from(location2, :units=>:miles)
You are mixing ruby (location1.distance_from(location2, :units=>:miles)) and sql (WHERE X > 50). This is difficult, but not impossible.
However, if you have to do the distance calculation in ruby already, why not do the filtering there as well. So in the spirit of my first example.
listing2 = some_location_to_filter_by
#refined_list = #initial_matches.map { |listing|
listing.distance_from(listing2) > 50 ? nil : listing
}.compact
This will iterate over all listings, keeping only those that are further than 50 from some predetermined listing.
EDIT: If this logic is done in the controller you need to assign to #refined_list instead of refined_list since only controller instance variables (as opposed to local ones) are accessible to the view.
In short, no. This is because after the initial query, you are not left with a relational table or view, you are left with an array of activerecord objects. So any processing to be done after the initial query has to be in the format of ruby and activerecord, not sql.
When you query an EntitySet property on a model object in Linq-to-SQL, it returns all rows from the entityset and does any further querying client-side.
This is confirmed in a few places online and I've observed the behavior myself. The EntitySet does not implement IQueryable.
What I've had to do is convert code like:
var myChild = ... ;
// Where clause performed client-side.
var query = myChild.Parents().Where(...) ;
to:
var myChild = ... ;
// Where clause performed in DB and only minimal set of rows returned.
var query = MyDataContext.Parents().Where(p => p.Child() == myChild) ;
Does anyone know a better solution?
A secondary question: is this fixed in the Entity Framework?
An EntitySet is just a collection of entities. It implements IEnumerable, not IQueryable. The Active Record pattern specifies that entities be directly responsible for their own persistence. OR mapper entities don't have any direct knowledge of the persistence layer. OR Mappers place this responsibility, along with Unit Of Work, and Identity Map responsibilities into the Data Context. So if you need to query the data source, you gotta use the context (or a Table object). To change this would bend the patterns in use.
I had a similar problem: How can I make this SelectMany use a join. After messing with LINQPad for a good amount of time I found a decent workaround. The key is to push the EntitySet you are looking at inside a SelectMany, Select, Where, etc. Once it's inside that it becomes an Expression and then the provider can turn it into a proper query.
Using your example try this:
var query = from c in Children
where c == myChild
from p in c.Parents
where p.Age > 35
select p;
I'm not able to 100% verify this query as I don't know the rest of your model. But the first two lines of the query cause the rest of it to become an Expression that the provider turns into a join. This does work with my own example that is on the question linked to above.
I am getting an IQueryable from my database and then I am getting another IQueryable from that first one -that is, I am filtering the first one.
My question is -does this affect performance? How many times will the code call the database? Thank you.
Code:
DataContext _dc = new DataContext();
IQueryable offers =
(from o in _dc.Offers
select o);
IQueryable filtered =
(from o in offers
select new { ... } );
return View(filtered);
The code you have given will never call the database since you're never using the results of the query in any code.
IQueryable collections aren't filled until you iterate through them...and you're not iterating through anything in that code sample (ah, the beauty of lazy initialization).
That also means that each of those statements will be executed as its own query against the database which results in no performance cost over doing two completely independent queries.
SO is not a replacement for developer tools. There are many good free tools able to tell you exactly what this code translates into and how it works. Use Reflector on this method and look at what code is generated and reason for yourself what is going on from there.
Our win32 application assembles objects from the data in a number of tables in a MySQL relational database. Of such an object, multiple revisions are stored in the database.
When storing multiple revisions of something, sooner or later you'll ask yourself the question if you can visualize the differences between two revisions :) So my question is: what would be a good way to "diff" two such database objects?
Would you do the comparison at the database level? (Doesn't sound like a good idea: too low-level, and too sensitive to the schema).
Would you compare the objects?
Would you write a function that "manually" compares the properties and fields of two objects?
How would you store the diff? In a separate, generic "TDiff" object?
Any general recommendations on how to visualize such things in a user interface?
Advice, or stories about your own experiences with this, are very welcome; thanks a bunch!
Extra info on use case (20090515)
In reply to Antony's comment: this specific application is used to schedule training courses, run by teams of teachers. The schedule of a teacher is stored in various tables in the database, and contains info such as "where does she have to go on which day", "who are her colleagues in the team", etc. This information is spread out over multiple tables.
Once in a while, we "publish" the schedule, so the teachers can see it on a webpage. Each "publication" is a revision, and we'd like to be able to show the users (and later also the teachers) what's changed between two publications --- if anything.
Hope that makes the scenario a bit more tangible :)
Some final remarks
Well, the bounty has come to an end, so I've accepted an answer. If it'd somehow be possible to slice a couple of extra 100's off of my rep and give it to some of the other answers, I would do so without hesitation. All your guys' help has been great, and I am very grateful! ~ Onno 20090519
Just an idea, but would it be worthwhile for you to convert the two object versions being compared to some text format and then comparing these text objects using an existing diff program - like diff for example? There are lots of nice diff programs out there that can offer nice visual representations, etc.
So for example
Text version of Object 1:
first_name: Harry
last_name: Lime
address: Wien
version: 0.1
Text version of Object 2:
first_name: Harry
last_name: Lime
address: Vienna
version: 0.2
The diff would be something like:
3,4c3,4
< address: Wien
< version: 0.1
---
> address: Vienna
> version: 0.2
Assume that a class has 5 known properties - date, time, subject, outline, location. When I look at my schedule, I'm most interested in the most recent (ie current/accurate) version of these properties. It would also be useful for me to know what, if anything, has changed. (As a side note, if the date, time or location changed, I'd also expect to get an email/sms advising me in case I don't check for an updated schedule :-))
I would suggest that the 'diff' is performed at the time the schedule is amended. So, when version 2 of the class is created, record which values have changed, and store this in two 'changelog' fields on the version 2 object (there must already be one parent table that sits atop all your tables - use that one!). One changelog field is 'human readable text' eg 'Date changed from Mon 1 May to Tues 2 May, Time changed from 10:00am to 10:30am'. The second changelog field is a delimted list of changed fields eg 'date,time' To do this, before saving you would loop over the values submitted by the user, compare to current database values, and concatenate 2 strings, one human readable, one a list of field names. Then, update the data and set your concatenated strings as the 'changelog' values.
When displaying the schedule load the current version by default. Loop through the fields in the changelog field list, and annotate the display to show that the value has changed (a * or a highlight, etc). Then, in a separate panel display the human readable change log.
If a schedule is amended more than once, you would probably want to combine the changelogs between version 1 & 2, and 2 & 3. Say in version 3 only the course outline changed - if that was the only changelog you had when displaying the schedule, the change to date and time wouldn't be displayed.
Note that this denormalised approach won't be great for analysis - eg working out which specific location always has classes changed out of it - but you could extend it using an E-A-V model to store the change log.
Doing a comparison at the database level would be good if what you cared about was changes to the database. That makes the most sense if you're trying to design a layer of generic functionality on top of the database itself.
Doing a comparison at the object level would be good if you care about changes to the data. For example, if the data was the input to a program and you were interested in looking at changes in the input to verify that changes to the output were correct.
Your use case doesn't appear to be either of these. You appear to care about the output and want differences from that perspective. If that's the case, I would do differences on the output report (or a pure-text version of it) instead of on the underlying data. You can do that with any off-the-shelf diff tool. To make things easier for your end-users you could parse the diff results and render them as HTML. There are lots of options here: side-by-side with color coding to indicate changes, one document with markup for changes (e.g. red strikethrough for deletions and green for additions), maybe just highlight areas that have changed and use balloons to show the previous/current values on demand.
I've thought about doing database comparisons but never tried to implement it. As you noted, any such attempts are intimately intertwined with the schema.
I have done object-level comparisons. The general algorithm was this:
Do a set comparison on the lists of object IDs. This creates three result groupings: added objects, deleted objects, and objects that live in both sets.
Report the deletions.
Report the additions.
For the things in both sets, do an attribute-by-attribute comparison.
If any differences are found, report the object ID, the attributes that differ, and the respective values. If appropriate, highlight the portion of the attribute value that has changed.
In my case, the comparison algorithms were hand-written to match the object attributes. This gave me control over which attributes were compared and how. A generic comparator might be possible for some cases but would depend on the situation and at least partially on the implementation language.
I've looked into MysQL Diffing a number of times. Unfortunately, there aren't any really good solutions available.
One tool I've tried was mysqldiff (www.mysqldiff.org). mysqldiff is a tool written in PHP which is capable of diffing mysql schemas. Unfortunately, it doesn't do a great job a lot of the time.
MySQL Workbench, MySQLs own SQL IDE provides the option to generate an alter script and I would imagine it does this by performing some kind of diff operation internally.
Aqua Data Studio is another tool that is capable of comparing schemas and outputing a diff of the two. While the ADS diff is quite nice, it does not provide a tool to create an alter script.
If I were writing my own I guess I would write code capable of comparing structure of two tables. Such code could be tuned to be highly sensitive (Ig if column order differs from from version to the next, it's a difference) or more moderately sensitive (Eg Column order is not a major issue, datatypes and lengths are important, as are indices and constraints).
Storage, I'm not to sure. I would look into how a version control system such as Mercurial stores its diff information for revisions and use that to elaborate a method appropriate for the DB.
Finally, for visual output I recommend you take a look at the Aqua Data Stduio compare feature (You can use the Trial version to test this...). Its diff output is pretty good.
My application dbscript compares hierarchical data (database schemas) in a stored procedure, which of course has to compare each field/property of every object with its counterpart. I guess you won't get around that step (unless you have a generic object description model)
As for the UI part of your question, have a look at screenshots to view and select differences.
I would think about some sort of common text representation of the objects and let the texts compare with an existing diffing tool like WinMerge.
I see no need to invent diffing by myself since there are already plenty of nice tools I can use.
In your situation in PostgreSQL I used a difference tables with the schema:
history_columns (
column_id smallint primary key,
column_name text not null,
table_name text not null,
unique (table_name, column_name)
);
create temporary sequence column_id_seq;
insert into history_columns
select nextval('column_id_seq'), column_name, table_name
from information_schema.columns
where
table_name in ('table1','table2','table3')
and table_schema=current_schema() and table_catalog=current_database();
create table history (
column_id smallint not null references history_columns,
id int not null,
change_time timestamp with time zone not null
constraint change_time_full_second -- only one change allowed per second
check (date_trunc('second',change_time)=change_time),
primary key (column_id,id,change_time),
value text
);
And on the tables I used a trigger like this:
create or replace function save_history() returns trigger as
$$
if (tg_op = 'DELETE') then
insert into historia values (
find_column_id('id',tg_relname), OLD.id,
date_trunc('second',current_timestamp),
OLD.id );
[for each column_name] {
if (char_length(OLD.column_name)>0) then
insert into history values (
find_column_id(column_name,tg_relname), OLD.id,
OLD.change_time, OLD.column_name
)
}
elsif (tg_op = 'UPDATE') then
[for each column_name] {
if (OLD.column_name is distinct from NEW.column_name) then
insert into history values (
find_column_id(column_name,tg_relname), OLD.id,
OLD.change_time, OLD.column_name
);
end if;
}
end if;
$$ language plpgsql volatile;
create trigger save_history_table1
before update or delete on table1
for each row execute procedure save_history();
This isn't really an answer to the question you asked rather an attempt to re-imagine the problem. Would you consider altering your database and object model to store the aggregate root and a series of deltas? That is, model and store RevisionSets that are collections of Revisions; a Revision is an entity property paired with a value. In a sense this is internalizing the revision structure into your architecture that the other posters are suggesting that you bolt-on to what you already have via "logs".
It's trivial to display the aggregate from the deltas, and even easier to display the deltas as a change history. The fact that you are using a rich client with state and local memory makes this even more compelling. You could very easily display "all the changes since date xxxx" without revisiting the database.
Credit for the basic idea goes to Greg Young and his work with financial data streams, but it is imminently applicable to your problem.
I'm riffing off of what Harry Lime suggested: Output your properties to text format, then hash the results. That way you can compare the hash values and easily flag the data that has been altered. This way you get the best of both worlds as you can visually see differences but programmatically identify differences. With the has you'll have a good source for an index should you want to store and retrieve the deltas.
Given you want to create a UI for this and need to indicate where the differences are, it seems to me you can either go custom or create a generic object comparer - the latter being dependent on the language you are using.
For the custom method, you need to create a class that takes to two instances of the classes to be comparied. It then returns differences;
public class Person
{
public string name;
}
public class PersonComparer
{
public PersonComparer(Person old, Person new)
{
....
}
public bool NameIsDifferent() { return old.Name != new.Name; }
public string NameDifferentText() { return NameIsDifferent() ? "Name changed from " + old.Name + " to " + new.Name : ""; }
}
This way you can use the NameComparer object to create your GUI.
The gereric approach would be much the same, just that you generalize the calls, and use object insepection (getObjectProperty call below) to find differences;
public class ObjectComparer()
{
public ObjectComparer(object old, object new)
{
...
}
public bool PropertyIsDifferent(string propertyName) { return getObjectProperty(old, propertyName) != getObjectProperty(new, propertyName) };
public string PropertyDifferentText(string propertyName) { return PropertyIsDifferent(propertyName) ? propertyName + " " + changed from " + getObjectProperty(old, propertyName) + " to " + getObjectProperty(new, propertyName): ""; }
}
}
I would go for the second, as it makes things really easy to change GUI on needs. The GUI I would try 'yellowing' the differences to make them easy to see - but that depends on how you want to show the differences.
Getting the object to compare would be loading your object with the initial revision and latest revision.
My 2 cents... Not as techy as the database compare stuff already here.
Have you looked at Open Source DiffKit?
www.diffkit.org
I think it does what you want.
Example with Oracle.
Export ordered objects to text with dbms_metadata
Export ordered tables data into CSV or query format
Make big text file
Diff
Now that LINQ to SQL is a little more mature, I'd like to know of any techniques people are using to create an n-tiered solution using the technology, because it does not seem that obvious to me.
LINQ to SQL doesn't really have a n-tier story that I've seen, since the objects that it creates are created in the class with the rest of it, you don't really have an assembly that you can nicely reference through something like Web Services, etc.
The only way I'd really consider it is using the datacontext to fetch data, then fill an intermediary data model, passing that through, and referencing it on both sides, and using that in your client side - then passing them back and pushing the data back into a new Datacontext or intellgently updating rows after you refetch them.
That's if I'm understanding what you're trying to get at :\
I asked ScottGu the same question on his blog when I first started looking at it - but I haven't seen a single scenario or app in the wild that uses LINQ to SQL in this way. Websites like Rob Connery's Storefront are closer to the provider.
Hm, Rockford Lhotka sad, that LINQ to SQL is wonderful technology for fetching data from database. He suggests that afterwards they'll must to be bind to "reach domain objects" (aka. CSLA objetcs).
Seriously speaking, LINQ to SQL had it's support for n-tier architecture see DataContext.Update method.
You might want to look into the ADO .Net Entity Framework as an alternative to LINQ to SQL, although it does support LINQ as well. I believe LINQ to SQL is designed to be fairly lightweight and simple, whereas the Entity Framework is more heavy duty and probably more suitable in large Enterprise applications.
OK, I am going to give myself one possible solution.
Inserts/Updates were never an issue; you can wrap the business logic in a Save/Update method; e.g.
public class EmployeesDAL
{
...
SaveEmployee(Employee employee)
{
//data formatting
employee.FirstName = employee.FirstName.Trim();
employee.LastName = employee.LastName.Trim();
//business rules
if(employee.FirstName.Length > 0 && employee.LastName.Length > 0)
{
MyCompanyContext context = new MyCompanyContext();
//insert
if(employee.empid == 0)
context.Employees.InsertOnSubmit(employee);
else
{
//update goes here
}
context.SubmitChanges();
}
else
throw new BusinessRuleException("Employees must have first and last names");
}
}
For fetching data, or at least the fetching of data that is coming from more than one table you can use stored procedures or views because the results will not be anonymous so you can return them from an outside method. For instance, using a stored proc:
public ISingleResult<GetEmployeesAndManagersResult> LoadEmployeesAndManagers()
{
MyCompanyContext context = new MyCompanyContext();
var emps = context.GetEmployeesAndManagers();
return emps;
}
Seriously speaking, LINQ to SQL had it's support for n-tier architecture see DataContext.Update method
Some of what I've read suggests that the business logic wraps the DataContext - in other words you wrap the update in the way that you suggest.
The way i traditionally write business objects i usually encapsulate the "Load methods" in the BO as well; so I might have a method named LoadEmployeesAndManagers that returns a list of employees and their immediate managers (this is a contrived example) . Maybe its just me, but in my front end I'd rather see e.LoadEmployeesAndManagers() than some long LINQ statement.
Anyway, using LINQ it would probably look something like this (not checked for syntax correctness):
var emps = from e in Employees
join m in Employees
on e.ManagerEmpID equals m.EmpID
select new
{ e,
m.FullName
};
Now if I understand things correctly, if I put this in say a class library and call it from my front end, the only way I can return this is as an IEnumerable, so I lose my strong typed goodness. The only way I'd be able to return a strongly typed object would be to create my own Employees class (plus a string field for manager name) and fill it from the results of my LINQ to SQL statement and then return that. But this seems counter intuitive... what exactly did LINQ to SQL buy me if I have to do all that?
I think that I might be looking at things the wrong way; any enlightenment would be appreciated.
"the only way I can return this is as an IEnumerable, so I lose my strong typed goodness"
that is incorrect. In fact your query is strongly typed, it is just an anonymous type. I think the query you want is more like:
var emps = from e in Employees
join m in Employees
on e.ManagerEmpID equals m.EmpID
select new Employee
{ e,
m.FullName
};
Which will return IEnumerable.
Here is an article I wrote on the topic.
Linq-to-sql is an ORM. It does not affect the way that you design an N-tiered application. You use it the same way you would use any other ORM.
#liammclennan
Which will return IEnumerable. ... Linq-to-sql is an ORM. It does not affect the way that you design an N-tiered application. You use it the same way you would use any other ORM.
Then I guess I am still confused. Yes, Linq-to-Sql is an ORM; but as far as I can tell I am still littering my front end code with inline sql type statements (linq, not sql.... but still I feel that this should be abstracted away from the front end).
Suppose I wrap the LINQ statement we've been using as an example in a method. As far as I can tell, the only way I can return it is this way:
public class EmployeesDAL
{
public IEnumerable LoadEmployeesAndManagers()
{
MyCompanyContext context = new MyCompanyContext();
var emps = from e in context.Employees
join m in context.Employees
on e.ManagerEmpID equals m.EmpID
select new
{ e,
m.FullName
};
return emps;
}
}
From my front end code I would do something like this:
EmployeesDAL dal = new EmployeesDAL;
var emps = dal.LoadEmployeesAndManagers();
This of course returns an IEnumerable; but I cannot use this like any other ORM like you say (unless of course I misunderstand), because I cannot do this (again, this is a contrived example):
txtEmployeeName.Text = emps[0].FullName
This is what I meant by "I lose strong typed goodness." I think that I am starting to agree with Crucible; that LINQ-to-SQL was not designed to be used in this way. Again, if I am not seeing things correctly, someone show me the way :)