Warning:
this is an exercise to understand better JSON database design in Firebase
it is not necessarily realistic
I have got a two ways relationship between users and door keys. I would like to understand:
how to represent this relationship visually (I can imagine it only as two separate trees)
how this would work on Firebase, would both users and door-keys be child of a parent node "myparentnodename"?
If I model the database in this way it feels highly inefficient because every time I would query the child node "users" I would get all the users back. Or am I wrong? Is it possible to only get back the data matching to a specific user? E.g. get user where "user = user1"? Can we do nested queries? e.g. combine the previous condition with some condition on the door keys so the JSON object returned is only relevant to the door-keys contained in the "user1" node?
This is a very long answer as your question was actually about 5 different questions.
root node is: myparentnodename
Your users
users
uid_0
name: "William"
door_keys:
key_0: true
key_3: true
uid_2
name: "Leonard"
door_keys:
key_3: true
key_5: true
and your keys
keys
key_0
uid_0: true
key_3
uid_0: true
uid_2: true
key_5
uid_5: true
With this structure, all of the elements 'point' at each other.
If you query uid_0 you can see that they use keys 0 and 3
If you query key_3 you can see they belong to users 0 and 2
Your question was
every time I would query the child node "users" I would get all the
users back
That's slightly incomplete. When a query is done, you usually query for something specific. With Firebase however, there are two ways to retrieve data: observing a node and a query.
If you want back all users in the users node you would observe that node by .Value (or .ChildAdded for 1 at a time).
ref = myParentNodeName
let usersRef = myParentNodeName.childByAppendingPath("users")
usersRef.observeEventType(.Value, withBlock: { snapshot in
//.Value can return multiple nodes within the snapshot so iterate over them
for child in snapshot.children {
let name = child.value.objectForKey("name") as! String
print(name) //prints each users name
}
})
note that the above attaches an observer to the users node so any future changes within that node will notify the app and re-send the entire node to the app
If you want just one user's info, and don't want to continue watching for changes
ref = myParentNodeName
let usersRef = myParentNodeName.childByAppendingPath("users")
let thisUserRef = usersRef.childByAppendingPath("uid_2")
thisUserRef.observeSingleEventOfType(.Value, withBlock: { snapshot in
let name = child.value.objectForKey("name") as! String
print(name) //prints each users name
})
Finally, to query for all keys that belong to uid_0 (which is a little redundant in this example since we already know which keys they have from their node). If the keys ref also contained other info like the door name, the building the door was in, or the door location, it would be more appropriate and would require a different structure, so assume that's the case:
ref = myParentNodeName
let keysRef = myParentNodeName.childByAppendingPath("keys")
keysRef.queryOrderedByChild("uid_0").queryEqualToValue(true)
.observeSingleEventOfType(.Value, withBlock: { snapshot in
let doorLocation = child.value.objectForKey("door_location") as! String
print(doorLocation) //prints each users name
})
note this code is Swift since the platform was not specified in the question.
The other question:
Can we do nested queries? e.g. combine the previous condition with
some condition on the door keys so the JSON object returned is only
relevant to the door-keys contained in the "user1" node?
I think you mean can you query for uid_2, see which keys they have and then load in the info from those specific keys.
Yes! But... (there's always a but)
Firebase is asynchronous so you have to take that into account when nesting queries i.e. you need to ensure all of the data is returned before getting more data. So for example, if you wanted uid_2 key data, you could observeSingleEventOfType on node uid_2. You would then have their keys and could then observeSingleEventOfType on each key.
Technically this will work but with asynchronous data flying around, you could end up with code stomping on other code and processing data before it's actually been returned.
The better option (per my example) is to just avoid that entirely and query the keys node for uid_2's keys.
As a side note, observing a node has a lot less overhead than a query so if you want to load a single node and you know the path, use observe.
Related
I have phonorgraph object with billions of rows and we are querying it through object set service
for example, I want to get all DriverLicences from certain city.
#Function()
public getDriverLicences(city: string): ObjectSet<DriverLicences> {
let drivers = Objects.search().DriverLicences().filter(row => row.city.exactMatch(city));
return drivers ;
}
I am facing this error when I am trying query it from slate:
ERROR 400: {"errorCode":"INVALID_ARGUMENT","errorName":"ObjectSet:PagingAboveConfiguredLimitNotAllowed","errorInstanceId":"0000-000","parameters":{}}
I understand that I am probably retrieving more than 100 000 results but I need all the results because of the implemented logic in the front is a complex slate dashboard built by another team that we cannot re-factor.
The issue here is that, specifically in the Slate <> Function connector, there is a "translation layer" that serializes the contents of the object set and provides a response data structure that materializes the property:value pairs for each object in the set.
This clearly doesn't work for large object sets where throwing so much data into the browser is likely to overwhelm the resources allocated to the tab.
From context it seems like you might be migrating an existing Slate app over to Functions; in the current version, how is the query limiting the number of results returned? It certainly must not be returning several 100 thousand results for further processing on the front end? (And if so, that might be an anti-pattern to consider addressing).
As for options that you could currently explore, you can sort your object set and then specify a smaller limit to return:
Objects.search().DriverLicences().filter(row => row.city.exactMatch(city)).orderBy(date_of_issue).take(100)
You'll find a few more details in the Functions documentation Reference entry on Ontology API: Object Sets in the section on Ordering and limiting.
You can even make a work around for the (current) lack of paging when return an ObjectSet to Slate by using the last value from the property ordered on (i.e. date_of_issue) as a filter in the subsequent request and return the next N objects.
This can work if you need a Slate table or HTML widget that renders on set of results then, on a user action, gets the next page.
I'm implementing a searchbar in IONIC 2 that search a JSON in one view so it can send its details to another view.
I have this JSON:
{
"Alphaville I": { //FIRST KEY
"ida": [{ //SECOND KEYS
"hora": "05:40",
"local": "AV. FERNÃO DIAS PAES LEME (Pref. Várzea Paulista)"
},... ],
"volta": [{ //SECOND KEYS
"hora": "05:40",
"local": "AV. FERNÃO DIAS PAES LEME (Pref. Várzea Paulista)"
},... ]
}, ... //MULTIPLE ITENS
}
So, in one view i create a list with the first keys (like Alphaville I), but i need to search the local inside of it.
But the Angular 2 *ngFor requires an array, so i iterate through my object and push it to an array, doing this it excludes my first key, so what i'm doing now (without searching, of course) is saving the keys in one array, geting the index and passing the jsonResultExample[index] to another page.
i'm using the basic searchbar example like the one in Seachbar Component Docs.
So what i need is: Search by the local key and return the first key (Alphaville I) of the nodes the contain the input text, the same local can appear in other first keys.
How can i do this? I can't post a better code because i haven't tried anything.
Is there a better way to structure my JSON for this? (i'm using firebase btw);
Any help or ideas is welcome, thanks.
EDIT
So i saved the first key value along with ida and volta so i can simply iterate through it, get the key value and everything without many problems, but since i need to filter by local it appears inside idaand volta as another array (cause i have many of these values), so it's looking like this now:
So now how can i access the local? Is it better to create another object only with all local and a key for every linha so i can return the values?
Remembering this is the searchbar code for Ionic 2 and my JSON has over 4k lines:
getItems(ev: any) {
// Reset items back to all of the items
this.initializeItems();
// set val to the value of the searchbar
let val = ev.target.value;
// if the value is an empty string don't filter the items
if (val && val.trim() != '') {
this.items = this.items.filter((item) => {
return (item.toLowerCase().indexOf(val.toLowerCase()) > -1);
})
}
}
Thanks in advance :)
It comes to personal experience on how to design data structure. Therefore I can't say the follow method is the best way.
First, in the case that we have complicated data structure, I don't prefer using map (a.k.a. object as data structure) in javascript. The main reason is pretty related to what you are facing, object by design cannot be iterated. Yes you can use Object.keys() or Object.values() but they are so ugly and hard to fit on every cases.
It is a nice move to put your first key as a property. That comes to the second problem. There seems to be an assumption in your structure that, one linha is mapped only to one local or one local is only related to one linha. If so, I suggest building another separated map only for the linha and local relationship.
Another approach is to normalize your data structure in to multiple separated javascript objects like what you do on database. By doing so, you can maximize the data flexibility that you can query whatever you want by Array.prototype.filter(), Array.prototype.map() or even directly access by its index. However, this approach may increase the lines of code as you need to manage multiple maps.
I have a set of data for a family tree in Neo4J and am trying to build a Cypher query that produces a JSON data set similar to the following:
{Name: "Bob",
parents: [
{Name: "Roger",
parents: [
Name: "Robert",
Name: "Jessica"
]},
{Name: "Susan",
parents: [
Name: "George",
Name: "Susan"
]}
]}
My graph has a relationship of PARENT between MEMBER nodes (i.e. MATCH (p.Member)-[:PARENT]->(c.Member) ). I found Nested has_many relationships in cypher and neo4j cypher nested collect which ends up grouping all parents together for the main child node I am searching for.
Adding some clarity based on feedback:
Every member has a unique identifier. The unions are currently all associated with the PARENT relationship. Everything is indexed so that performance will not suffer. When I run a query to just get back the node graph I get the results I expect. I'm trying to return an output that I can use for visualization purposes with D3. Ideally this will be done with a Cypher query as I'm using the API to access neo4j from the frontend being built.
Adding a sample query:
MATCH (p:Person)-[:PARENT*1..5]->(c:Person)
WHERE c.FirstName = 'Bob'
RETURN p.FirstName, c.FirstName
This query returns a list of each parent for five generations, but instead of showing the hierarchy, it's listing 'Bob' as the child for each relationship. Is there a Cypher query that would show each relationship in the data at least? I can format it as I need to from there...
Genealogical data might comply with the GEDCOM standard and include two types of nodes: Person and Union. The Person node has its identifier and the usual demographic facts. The Union nodes have a union_id and the facts about the union. In GEDCOM, Family is a third element bringing these two together. But in Neo4j, I found it suitable to also include the union_id in Person nodes. I used 5 relationships: father, mother, husband, wife and child. The family is then two parents with an inward vector and each child with an outward vector. The image illustrates this. This is very handy for visualizing connections and generating hypotheses. For example, consider the attached picture and my ancestor Edward G Campbell, the product of union 1917 where three brothers married three Vaught sisters from union 8944 and two married Gaither sisters from union 2945. Also, in the upper left, how Mahala Campbell married her step-brother John Greer Armstrong. Next to Mahala is an Elizabeth Campbell who is connected by marriage to other Campbell, but is likely directly related to them. Similarly, you can hypothesize about Rachael Jacobs in the upper right and how she might relate to the other Jacobs.
I use bulk inserts which can populate ~30000 Person nodes and ~100,000 relationships in just over a minute. I have a small .NET function that returns the JSon from a dataview; this generic solution works with any dataview so it is scalable. I'm now working on adding other data, such as locations (lat/long), documentation (particularly that linking folks, such as a census), etc.
You might also have a look at Rik van Bruggens Blog on his family data:
Regarding your query
You already create a path pattern here: (p:Person)-[:PARENT*1..5]->(c:Person) you can assign it to a variable tree and then operate on that variable, e.g. returning the tree, or nodes(tree) or rels(tree) or operate on that collection in other ways:
MATCH tree = (p:Person)-[:PARENT*1..5]->(c:Person)
WHERE c.FirstName = 'Bob'
RETURN nodes(tree), rels(tree), tree, length(tree),
[n in nodes(tree) | n.FirstName] as names
See also the cypher reference card: http://neo4j.com/docs/stable/cypher-refcard and the online training http://neo4j.com/online-training to learn more about Cypher.
Don't forget to
create index on :Person(FirstName);
I'd suggest building a method to flatten out your data into an array. If they objects don't have UUIDs you would probably want to give them IDs as you flatten and then have a parent_id key for each record.
You can then run it as a set of cypher queries (either making multiple requests to the query REST API, or using the batch REST API) or alternatively dump the data to CSV and use cypher's LOAD CSV command to load the objects.
An example cypher command with params would be:
CREATE (:Member {uuid: {uuid}, name: {name}}
And then running through the list again with the parent and child IDs:
MATCH (m1:Member {uuid: {uuid1}}), (m2:Member {uuid: {uuid2}})
CREATE m1<-[:PARENT]-m2
Make sure to have an index on the ID for members!
The only way I have found thus far to get the data I am looking for is to actually return the relationship information, like so:
MATCH ft = (person {firstName: 'Bob'})<-[:PARENT]-(p:Person)
RETURN EXTRACT(n in nodes(ft) | {firstName: n.firstName}) as parentage
ORDER BY length(ft);
Which will return a dataset I am then able to morph:
["Bob", "Roger"]
["Bob", "Susan"]
["Bob", "Roger", "Robert"]
["Bob", "Susan", "George"]
["Bob", "Roger", "Jessica"]
["Bob", "Susan", "Susan"]
I have imported a few nodes with properties via the Cypher CSV import (command below) and the nodes seem to have loaded correctly as I can view them in the REST-API (the data browser). When I execute a MATCH (n) RETURN n query, all of the nodes are displayed in the Results pane, and when I click on one of the nodes the properties are displayed in the left pane of the browser (I would attach a screen shot showing what I am trying to refer to here which would make this issue A LOT more clear and easy to understand, but apparently us neophytes are prohibited from providing such useful information).
However, when I try to query any of the nodes directly, I get no rows returned. By "query the nodes directly" I am referring to querying with a WHERE condition where I ask for a specific property:
MATCH (n)
WHERE n:Type="Idea"
RETURN n
Type is one of the properties of the node. No rows are returned from the query. I can click on the node in the Stream pane to open the properties dialog, and I can see the Type property is clearly "Idea."
Am I missing something? The nodes and properties seemed to have loaded into the DB correctly, but I can't seem to query anything. Is "ID" a restricted term? Do I even need an "ID" property (i thought I read somewhere you shouldn't trust the auto-generated IDs as they aren't guaranteed to be unique over time)?
Import statement used to load the nodes is below:
$ auto-index name, ID
$ import-cypher -i ProjectNodesCSV.csv -o ProjectOut.csv CREATE (n:Project {ID:{ID},Name: {Name}, Type: {Type}, ProjectGroupName: {ProjectGroupName}, ProjectCategoryName: {ProjectCategoryName}, UnifierID: {UnifierID}, StartDate: {StartDate}, EndDate: {EndDate}, CapitalCosts: {CapitalCosts}, OandMCosts: {OandMCosts}}) RETURN ID(n) as ID, n.Name as Name
Using sqlalchemy 0.7.2
Is there a way to find the table class from the query object? For example:
q = session.query(Customers)
how can I find Customers in q? Possible? Not Possible?
Yes. You need column_descriptions.
It's a long road to the table, though. sqlalchemy.orm.Query.column_descriptions returns a list of dicts, describing each query entity as it was given to query. in your example, there's only one entity, so you need the first item from that list. And since you're interested in the type of the query entity, rather than its' structure, you want the "type" key from that list:
q_entity = q.column_descriptions[0]['type']
assert q_entity == Customer
Accessing the table for the mapped class requires snooping around in the mapper subsystem. for that, you should use manager_of_class. The table is accessible from the manager through the mapper.mapped_table attribute:
from sqlalchemy.orm.attribute import manager_of_class
q_table = manager_of_class(q_entity).mapper.mapped_table
Resist the urge to skip strait to the mapper through Customer.__mapper__, or even Customer.__table__; That's specific to sqlalchemy.ext.declarative, and won't work with classes that are mapped by other means.