Using Postgraphile, let's say I am querying all Foos, ie. allFoosList. Let's also say my Foo table has a many-to-many relationship with a Bar table in my database (ie. my Foos have Bars).
In my results I will get back an array of Foo objects. Those objects will have a property that is an array of Bars. However, that field won't be bars or barsByBarIdList ... it will have a name like barFoosByFooIdList. The objects inside that array will have a barByBarId property, which will have the actual record values.
Is there a way for me to correct/simplify this?
There's a number of approaches, a couple of which are documented in PostGraphile's documentation under Relations > Many-to-many relations.
Computed column
create function foos_bars(f foos) returns setof bars as $$
select bars.*
from bars
inner join foo_bars
on foo_bars.bar_id = bars.id
where foo_bars.foo_id = f.id
$$ language sql stable;
Schema extension
const { makeExtendSchemaPlugin, gql, embed } = require("graphile-utils");
module.exports = makeExtendSchemaPlugin(build => {
const { pgSql: sql } = build;
return {
typeDefs: gql`
extend type Foo {
bars: BarsConnection #pgQuery(
source: ${embed(sql.fragment`bars`)}
withQueryBuilder: ${embed((queryBuilder, args) => {
queryBuilder.where(
sql.fragment`exists(select 1 from foo_bars where foo_bars.bar_id = ${queryBuilder.getTableAlias()}.id and foo_bars.foo_id = ${queryBuilder.parentQueryBuilder.getTableAlias()}.id`
);
})}
)
}
`,
};
});
(This is more performant, but somewhat harder to read - we're hoping to optimise this syntax in future).
Contrib many-to-many plugin
yarn add #graphile-contrib/pg-many-to-many
postgraphile --append-plugins #graphile-contrib/pg-many-to-many
You can also optimize the naming
We provide a pg-simplify-inflector plugin that can be used to simplify names in your schema if you're pretty sure there'll be no conflicts; I highly recommend it's usage if your database schema is well defined:
https://github.com/graphile/pg-simplify-inflector
Related
I'm currently working on my first API with the Perfect framework. It's been a while since I made an API myself so I must admit my SQL and API logic is a little rusty.
I'm using a MySQL database for my implementation.
For sake of example I'll explain my database structure below;
I have a table which resembles an Object, let's call this Table A. Table A has a Varchar based id as primary key.
There are 2 other tables let's call them Table B and Table C. Table A has a one to many relation to both Table B and C. Where the id of table A is the foreign key.
What I'm trying to do is obtain everything with one query and cast it to an object in my backend.
By using outer joins I'm making the call to retrieve all the required data.
SELECT control.id, control.type, control.description, control.address, control.city, control.created, control.updated, control.latitude, control.longitude, images.id AS image_id, images.image, images.description AS image_description, updates.id AS update_id, updates.still_present, updates.created_at AS update_created
FROM Control control left outer join control_images images
ON control.id=images.control_id
left outer join Control_Updates updates
ON control.id=updates.control_id
Now is my question what would be the best way to store this data in an object that holds an array of updates and an array of images.
Before writing the join query I only attempted to get the values from Table A I used the following code to cast the results to my desired object.
let result = mysql.storeResults()
let checkResult = self.checkResult(result: result, response: response)
response = checkResult.response
var controls: [Control] = []
while let row = result?.next() {
let type = Types(rawValue: row[1].unwrap)!
let control = Control(id: row[0].unwrap, type: type, description: row[2].unwrap, address: row[3].unwrap, city: row[4].unwrap, latitude: Double(row[7].unwrap).unwrap, longitude: Double(row[8].unwrap).unwrap)
controls.append(control)
}
obviously this will just return duplicate objects apart from the images and updates of course.
I'm wondering if this is the best way to do it or if I should call a new query in the while loop
The best way to resolve this issue, by still only using one query and one loop is by using 'hashmaps'. I'm not familiar with Perfect framework, but in PHP it would look something like:
// Get results from the db:
$results = $db->execute($query, $params);
// Define map for controls:
$map = [];
// Loop over results/rows
foreach($results as $row){
// Get unique identifier for the Control model:
$controlId = $row['id'];
// Check if control is NOT already in map:
if(!isset($map[$controlId]){
// Add control to map:
$control = [
'id' => $controlId,
'description' => $row['description'],
'images' => []
// other fields
];
// Add control to map:
$map[$controlId] = $control;
}
else{
// Control exists, retrieve it from the map:
$control = $map[$controlId];
}
// Retrieve unique identifier of the image:
$imageId = $row['image_id'];
// Same tactic with hasmap, check if control already has the image, if not add it
if(!isset($control['images'][$imageId]){
// add the image to the hashmap:
}
else{
// Image is already added, the content from the 'update' data is not added yet, handle that part (also with a hash map)
}
}
Hope that helps you figure it out in Perfect framework
I do want to join two table in HTML5 indexed db.
I found lot samples to add, Update, Delete and listing record but can't found any samples to join multiple table.
Sample URL:
*http://users.telenet.be/kristofdegrave/*
Was wondering the same question when just generally looking at these technologies. Did not check your particular example, but here is an example from Mozilla on how to do a join if you need it:
https://hacks.mozilla.org/2010/06/comparing-indexeddb-and-webdatabase/
It is quite a lot of code compared to SQL, but I think the idea is that abstracting these using JavaScript later is quite easy.
There is no join, but you can open multiple object stores and join during cursor iterations.
Using my own ydn-db library, you can query SELECT * FROM Supplier, Part WHERE Supplier.CITY = Part.CITY by
var iter_supplier = new ydn.db.IndexValueIterator('Supplier', 'CITY');
var iter_part = new ydn.db.IndexValueIterator('Part', 'CITY');
var req = db.scan(function(keys, values) {
var SID = keys[0];
var PID = keys[1];
console.log(SID, PID);
if (!SID || !PID) {
return []; // done
}
var cmp = ydn.db.cmp(SID, PID); // compare keys
if (cmp == 0) {
console.log(values[0], values[1]);
return [true, true]; // advance both
} else if (cmp == 1) {
return [undefined, SID]; // jump PID cursor to match SID
} else {
return [PID, undefined]; // jump SID cursor to match PID
}
}, [iter_supplier, iter_part]);
See more detail on Join query article.
As far as I know, IndexedDB does not have an API for doing JOINs, yet. The solutions I adopted involved opening a cursor, looping through the results and doing the join manually. It's the dreadful RBAR approach, but I couldn't find a better solution.
Instead of trying to join, redesign how you store the data so that a join is not required. You do not need to follow the same normalization constraints as SQL when you are using a no-sql approach. No-sql supports storing data redundantly when it is appropriate.
IndexedDB is in fact an object store and normaly in an object store there is less need for joins because you can just save the nested structure.
In the case you show, data from a codetable gets joined with real data. The case to solve this without a join is just fetch your code table into memory (normally this data never changes) and make the join in code.
any hint on what's wrong with the below query?
return new ItemPricesViewModel()
{
Source = (from o in XpoSession.Query<PRICE>()
select new ItemPriceViewModel()
{
ID = o.ITEM_ID.ITEM_ID,
ItemCod = o.ITEM_ID.ITEM_COD,
ItemModifier = o.ITEM_MODIFIER_ID.ITEM_MODIFIER_COD,
ItemName = o.ITEM_ID.ITEM_COD,
ItemID = o.ITEM_ID.ITEM_ID,
ItemModifierID = o.ITEM_MODIFIER_ID.ITEM_MODIFIER_ID,
ItemPrices = (from d in o
where d.ITEM_ID.ITEM_ID == o.ITEM_ID.ITEM_ID && d.ITEM_MODIFIER_ID.ITEM_MODIFIER_ID == o.ITEM_MODIFIER_ID.ITEM_MODIFIER_ID
select new Price()
{
ID = o.PRICE_ID,
PriceList = o.PRICELIST_ID.PRICELIST_,
Price = o.PRICE_
}).ToList()
}).ToList()
};
o in subquery is in read and I got the message "Could not find an implementation of the query pattern for source type . 'Where' not found."
I would like to have distinct ItemID, ItemModifier: should I create a custom IEqualityComparer to do it?
Thank you!
It seems like XPO it's not able to respond to this scenario. For reference this is what you could do with DbContext.
It sounds like maybe you want a GroupBy. Try something like this.
var result = dbContext.Prices
.GroupBy(p => new {p.ItemName, p.ItemTypeName)
.Select(g => new Item
{
ItemName = g.Key.ItemName,
ItemTypeName = g.Key.ItemTypeName,
Prices = g.Select(p => new Price
{
Price = p.Price
}
).ToList()
})
.Skip(x)
.Take(y)
.ToList();
Probable cause
In general, XPO does not support "free joins" in most of the cases. It was explicitelly written somewhere in their knowledgebase or Q/A site. If I hit that article again, I'll include a link to it.
In your original code example, you were trying to perform a "free join" in the INNER query. The 'WHERE' clause was doing a join-by-key, probably navigational, but also it contained an extra filter by "modifier" which probably is not a part of the definition of the relation.
Moreover, the query tried to reuse the IQueryable<PRICE> o in the inner query - what actually seems somewhat supported by XPO - but if you ever add any prefiltering ('where') to the toplevel 'o', it would have high odds of breaking again.
The docs state that XPO supports only navigational joins, along paths formed by properties and/or xpcollections defined in your XPObjects. This applies to XPO as whole, so XPQuery too. All other kinds of joins are called "free joins" and either:
are silently emulated by XPO by fetching related objects, extracting key values from them and rewriting the query into a multiple roundtrips with a series of partial queries that fetch full objects with WHERE-id-IN-(#p0,#p1,#p2,...) - but this happens only in the some simpliest cases
or are "not fully supported", meaning they throw exceptions and require you to manually split the query or rephrase it
Possible direct solution schemes
If ITEM_ID is a relation and XPCollection in PRICE class, then you could rewrite your query so that it fetches a PRICE object then builds up a result object and initializes its fields with PRICE object's properties. Something like:
return new ItemPricesViewModel()
{
Source = (from o in XpoSession.Query<PRICE>().AsEnumerable()
select new ItemPriceViewModel()
{
ID = o.ITEM_ID.ITEM_ID,
ItemCod = o.ITEM_ID.ITEM_COD,
....
ItemModifierID = o.ITEM_MODIFIER_ID.ITEM_MODIFIER_ID,
ItemPrices = (from d in o
where d.ITEM_ID.ITEM_ID == ....
select new Price()
.... .... ....
};
Note the 'AsEnumerable' that breaks the query and ensures that PRICE objects are first fetched instead of just trying to translate the query. Very probable that this would "just work".
Also, splitting the query into explicit stages sometimes help the XPO to analyze it:
return new ItemPricesViewModel()
{
Source = (from o in XpoSession.Query<PRICE>()
select new
{
id = o.ITEM_ID.ITEM_ID,
itemcod = o.ITEM_ID.ITEM_COD,
....
}
).AsEnumerable()
.Select(temp =>
select new ItemPriceViewModel()
{
ID = temp.id
ItemCod = temp.itemcod,
....
ItemPrices = (from d in XpoSession.Query<PRICE>()
where d.ITEM_ID.ITEM_ID == ....
select new Price()
.... .... ....
};
Here, note that I first fetch the item-data from server, and then conctruct the item on the 'client', and then build the required groupings. Note that I could not refer to the variable o anymore. In these precise case and examples, unsuprisingly, the second one (splitted) would be probably even slower than the first one, since it would fetch all PRICEs and then refetch the groupings through additional queries, while the first one would just fetch all PRICEs and then would calculate the groups in-memory basing on the PRICEs already fetched. This is not an a sideeffect of my laziness, but it is a common pitfall when rewriting the LINQ queries, so I included it as a warning :)
Both of these code examples are NOT RECOMMENDED for your case, as they would probably have very poor performace, especially if you have many PRICEs in the table, which is highly likely. I included them to present as only an example of how you could rewrite the query to siplify its structure so the XPO can eat it without choking. However, you have to be really careful and pay attention to little details, as you can very easily spoil the performance.
observations and real solution
However, it is worth noting that they are not that much worse than your original query. It was itself quite poor, since it tried to perform something near O(N^2) row-fetches from the table just to perform to group te rows by "ITEM_ID" and then formatting the results as separate objects. Properly done, it would be something like O(N lg N)+O(N), so regardless of being supported or not, your alternate attempt with GroupBy is surely a much better approach, and I'm really glad you found it yourself.
Very often when you are trying to split/simplify the XPQuery expressions as I did above, you implicitely rethink the problem and find an easier and simplier way to express the query that was initially not-supported or just were crashing.
Unfortunatelly, your query was in fact quite simple. For a really complex queries that cannot be "just rephrased", splitting into stages and making some of the join-filter work at 'client' is unavoidable.. But again, doing them on XPCollections or XPViews with CritieriaOperators is impossible too, so either we have to bear with it or use plain direct handcrafted SQL..
Sidenote:
Whole XPO has problems with "free joins", they are "not fully supported" not only in XPQuery, but also there's not much for them in XPCollection, XPView, CriteriaOperators, etc, too. But, it is worth noting that at least in "my version" of DX11, the XPQuery has very poor LINQ support at all.
I've hit many cases where a proper LINQ query was:
throwing "NotSupportedException", mostly in FreeJoins, but also very often with complex GroupBy or Select-with-Projection, GroupJoin, and many others - sometimes even Distinct(!) seemed to malfunction
throwing "NullReferenceExceptions" at some proper type conversions (XPO tried to interprete a column that held INT/NULL as an object..), often I had to write some completely odd and artificial expressions like foo!=null && foo.bar!=123 instead of foo = 123 despite the 'foo' being an public int Foo {get;set;}, all because the DX could not cope properly with NULLs in the database (because XPO created nullable-INT column for this property.. but that's another story)
throwing other random ArgumentException/InvalidOperation exceptions from other constructs
or even analyzing the query structure improperly, for example this one is usually valid:
session.Query<ABC>()
.Where( abc => abc.foo == "somefilter" )
.Select( abc => new { first = abc, b = abc } )
.ToArray();
but things like this one usually throws:
session.Query<ABC>()
.Select( abc => new { first = abc, b = abc } )
.Where ( temp => temp.first.foo == "somefilter" )
.ToArray();
but this one is valid:
session.Query<ABC>()
.Select( abc => new { first = abc, b = abc } )
.ToArray()
.Where ( temp => temp.first.foo == "somefilter" )
.ToArray();
The middle code example usually throws with an error that reveals that XPO layer were trying to find ".first.foo" path inside the ABC class, which is obviously wrong since at that point the element type isn't ABC anymore but instead a' anonymous class.
disclaimer
I've already noted that, but let me repeat: these observations are related to DX11 and most probably also earlier. I do not know what of that was fixed in DX12 and above (if anything at all was!).
We have followed the approach below to get the data from multiple results using LINQ To SQL
CREATE PROCEDURE dbo.GetPostByID
(
#PostID int
)
AS
SELECT *
FROM Posts AS p
WHERE p.PostID = #PostID
SELECT c.*
FROM Categories AS c
JOIN PostCategories AS pc
ON (pc.CategoryID = c.CategoryID)
WHERE pc.PostID = #PostID
The calling method in the class the inherits from DataContext should look like:
[Database(Name = "Blog")]
public class BlogContext : DataContext
{
...
[Function(Name = "dbo.GetPostByID")]
[ResultType(typeof(Post))]
[ResultType(typeof(Category))]
public IMultipleResults GetPostByID(int postID)
{
IExecuteResult result =
this.ExecuteMethodCall(this,
((MethodInfo)(MethodInfo.GetCurrentMethod())),
postID);
return (IMultipleResults)(result.ReturnValue);
}
}
Notice that the method is decorated not only with the Function attribute that maps to the stored procedure name, but also with the ReturnType attributes with the types of the result sets that the stored procedure returns. Additionally, the method returns an untyped interface of IMultipleResults:
public interface IMultipleResults : IFunctionResult, IDisposable
{
IEnumerable<TElement> GetResult<TElement>();
}
so the program can use this interface in order to retrieve the results:
BlogContext ctx = new BlogContext(...);
IMultipleResults results = ctx.GetPostByID(...);
IEnumerable<Post> posts = results.GetResult<Post>();
IEnumerable<Category> categories = results.GetResult<Category>();
In the above stored procedures we had two select queries
1. Select query without join
2. Select query with Join
But in the above second select query the data which is displayed is from one of the table i.e. from Categories table. But we have used join and want to display the data table with the results from both the tables i.e. from Categories as well as PostCategories.
Please if anybody can let me know how to achieve this using LINQ to SQL
What is the performance trade-off if we use the above approach vis-à-vis implement the above approach with simple SQL
Scott Guthrie (the guy who runs the .Net dev teams at MS) covered how to do this on his blog some months ago much better than I ever could, link here. On that page there is a section titled "Handling Multiple Result Shapes from SPROCs". That explains how to handle multiple results from stored procs of different shapes (or the same shape).
I highly recommend subscribing to his RSS feed. He is pretty much THE authoritative source on all things .Net.
Heya dude - does this work?
IEnumerable<Post> posts;
IEnumerable<Category> categories;
using (BlogContext ctx = new BlogContext(...))
{
ctx.DeferredLoadingEnabled = false; // THIS IS IMPORTANT.
IMultipleResults results = ctx.GetPostByID(...);
posts = results.GetResult<Post>().ToList();
categories = results.GetResult<Category>().ToList();
}
// Now we need to associate each category to the post.
// ASSUMPTION: Each post has only one category (1-1 mapping).
if (posts != null)
{
foreach(var post in posts)
{
int postId = post.PostId;
post.Category = categories
.Where(p => p.PostId == postId)
.SingleOrDefault();
}
}
Ok. lets break this down.
First up, a nice connection inside a using block (so it's disposed of nicely).
Next, we make sure DEFERRED LOADING is off. Otherwise, when u try and do the set (eg. post.Category == blah) it will see that it's null, lazy-load the data (eg. do a rountrip the database) set the data and THEN override the what was just dragged down from the db, with the result of there Where(..) method. phew! Summary: make sure deferred loading is off for the scope of the query.
Last, for each post, iterate and set the category from the second list.
does that help?
EDIT
Fixed it so that it doesn't throw an enumeration error by calling the ToList() methods.
Just curious, if a Post have have one or many Categories, is it possible to instead of using the for loop, to load the Post.PostCategories with the list of Categories (one to many), all in one shot, using a JOIN?
var rslt = from p in results.GetResult<Post>()
join c in results.GetResult<Category>() on p.PostId = c.PostID
...
p.Categories.Add(c)
To simplify my current situation lets say I have 2 tables (legacy so can't touch the schema to play better with Linq)
node
Columns: key_field1, key_field2, [lots of other fields]
node_children
Columns: parent_key_field1, parent_key_field2, child_key_field1, child_key_field2
node_children is similar to the has_and_belongs_to_many join table from Rails... except that both ids refer to the same table.
Ideally I would like to have a Node Class which has a Property Children, which returns the right List?
Results should be equivalent to
select * from node
join node_children
on node.key_field1 = node_children.child_key_field1 and node.key_field2 = node_children.child_key_field2
where node_children.parent_key_field1 = #paramX and node_children.parent_key_field2 = #paramY
Whats the best way to model this in Linq to Sql?
This is what I came up with (works too!)
Open up the ORM generated Node class MyDatabase.cs
partial class Node
{
public IEnumerable<Node> Children
{
get
{
MyDatabaseDataContext dc = new MyDatabaseDataContext();
return from link in this.ChildLinks
join node in dc.Nodes on new { Site = link.child_key_field1, ID = link.child_key_field2 } equals new { Site = node.key_field1, ID = node.key_field2 }
select node;
}
}
}
The ORM designer automatically adds an association between the two tables. I renamed the property on the Node class to ChildLinks.
(This drove me up the wall for a couple of hours) For the LINQ join to work with composite keys, you need the Name as well as Order of each part to match. See How to: Join by Using Composite Keys
One of these days, I'm gonna complete the LINQ Getting Started set of articles :)