I'm starting to dig into graph databases, but i have no idea, how these graphs are stored internally. Let's say i have this graph (taken from Wikipedia):
How do i serialize this graph as a key-value object? (a Python dict, for example)
I imagine two dicts, one for vertices and one for edges:
{'vertices':
{'1': {'Name': 'Alice', 'Age': 18},
'2': {'Name': 'Bob', 'Age': 22},
'3': {'Type': 'Group', 'Name': 'Chess'}},
'edges':
{'100': {'Label': 'knows', 'Since': '2001/10/03'},
'101': {'Label': 'knows', 'Since': '2001/10/04'},
'102': {'Label': 'is_member', 'Since': '2005/7/01'},
'103': {'Label': 'Members'},
'104': {'Label': 'Members'},
'105': {'Label': 'is_member', 'Since': '2011/02/14'}},
'connections': [['1', '2', '100'], ['2', '1', '101'],
['1', '3', '102'], ['3', '1', '103'],
['3', '2', '104'], ['2', '3', '105']]}
But i'm not sure, whether this is the most practical implementation. Maybe the "connections" should be inside "vertices" dict. So, what is the best way to implement graph datastore using key-value objects? What and where can i read more about it?
Possibly related, but not a duplicate: How to represent a strange graph in some data structure
The normal pattern is to not have a separate connections structure but to put that information in the edges structure. This gives something like:
{
'vertices': {
'1': {'Name': 'Alice', 'Age': 18},
'2': {'Name': 'Bob', 'Age': 22},
'3': {'Type': 'Group', 'Name': 'Chess'} },
'edges': [
{'from': '1', 'to': '2', 'Label': 'knows', 'Since': '2001/10/03'},
{'from': '2', 'to': '1', 'Label': 'knows', 'Since': '2001/10/04'},
{'from': '1', 'to': '3', 'Label': 'is_member', 'Since': '2005/7/01'},
{'from': '3', 'to': '1', 'Label': 'Members'},
{'from': '3', 'to': '2', 'Label': 'Members'},
{'from': '2', 'to': '3', 'Label': 'is_member', 'Since': '2011/02/14'} ] }
seems ok - each object has its it, there is no duplications. it's good for 'read and process purpose'. but there is no 'best' representation. it always depends on your purpose. do you want to be able to quickly find vertices by name? or edges by date? or maybe you want to quickly test if two vertices are connected? or the opposite - you want to quickly modify some parts of the graph? each purpose requires different data structures of database tables
how these graphs are stored internally
how do I serialize this graph as a key-value object
These questions are different and they need different answers.
In the former case, the main requirement is probably to perform complex queries efficiently.
I'd suggest to investigate existing industrial-strength solutions.
In NoSQL terms, these nested key-value objects are documents. Hence, one could look into how graphs are stored in "layered" multi-model databases that:
support graph data model, and
use underlying document data model.
Examples of such databases are ArangoDB, OrientDB, Azure CosmosDB.
You could also replace "document data model" with "wide column data model", because wide column data model can be conidered as two-dimensional key-value model.
Examples of such databases are DataStax Enterprise Graph and perhaps Grakn.
For instance, in ArangoDB, edges are stored as regular documents, but in special collections.
Obviously, data structures used may be accompanied with additional indexes etc. (or not).
So, what is the best way to implement graph datastore using key-value objects?
What and where can i read more about it?
I'd suggest another one article from ArangoDB:
Storing a graph in a pure document store
I'd make few changes in Eamonn's answer.
Every vertex and edge has 3 things.. id, Label and Properties
{
'vertices': {
'1': {'Label' : Person, 'Properties' : { 'Name': 'Alice', 'Age': 18}},
'2': {'Label' : Person, 'Properties' : {'Name': 'Bob', 'Age': 22}},
'3': {'Label': 'Group', 'Properties' : { 'Name': 'Chess'} },
'edges': [
'4' : {'from': '1', 'to': '2', 'Label': 'knows', 'Properties':{'Since': '2001/10/03' , 'Until' : '2001/10/03'}},
'5' : {'from': '2', 'to': '1', 'Label': 'knows', 'Properties':{'Since': '2001/10/04', 'Until' : '2001/10/05'}}
]
}
This way you can do query by vertex/edge, and their Labels and their properties.
I would serialize it like this, except you should choose the keys based on what you are looking up by. I assumed you are using the id, but perhaps using the name could be better.
{
'members': {
'1': {
'id': '1',
'name': 'Alice',
'age': 18,
'groups': {
'3': {
'path': 'groups.3',
'since': '2005-07-01'
}
},
'knows': {
'2': {
'path': 'members.2',
'since': '2001-10-03'
}
}
},
'2': {
'id': '2',
'name': 'Bob',
'age': 22,
'groups': {
'3': {
'path': 'groups.3',
'since': '2011-02-14'
}
},
'knows': {
'1': {
'path': 'members.1',
'since': '2001-10-04'
}
}
}
},
'groups': {
'3': {
'id': '3',
'name': 'Chess',
'members': {
'1': { 'path': 'members.1' },
'2': { 'path': 'members.2' }
}
}
}
}
You can serialize graphs directly into key-value pairs if you have a way of serializing references to other parts of the graph, which is what I use 'path' for. If I was deserializing it into a dict, I may consider replacing the path values with the actual dictionaries they refer to. Keep in mind that this may cause circular references which could cause problems if you were serializing it into json or something.
I would add an adjacency to the structure too. My take would be like this,
{
'vertices': {
'1': {'Name': 'Alice', 'Age': 18},
'2': {'Name': 'Bob', 'Age': 22},
'3': {'Type': 'Group', 'Name': 'Chess'}
},
'edges': {
'100' : {'from': '1', 'to': '2', 'Label': 'knows', 'Since': '2001/10/03'},
'101': {'from': '2', 'to': '1', 'Label': 'knows', 'Since': '2001/10/04'},
....
},
'adjacency': {
'1': ['101', '102'],
...
}
}
This way I can easily find which edges are adjacent to my vertices instead of iterating through all the edges.
Related
Python version: 3.10
Running a function returns this:
[{'type': 1, 'components': [{'type': 2, 'style': 1, 'label': 'She/Her', 'custom_id': 'She/Her'}, {'style': 1, 'label': 'He/Him', 'custom_id': 'He/Him', 'type': 2}]}]
How can I get all values of 'custom_id' within what is returned? Thank you!
You can do it like so:
myList = [{'type': 1, 'components': [{'type': 2, 'style': 1, 'label': 'She/Her', 'custom_id': 'She/Her'}, {'style': 1, 'label': 'He/Him', 'custom_id': 'He/Him', 'type': 2}]}]
for user in list(myList[0]["components"]):
print(user["custom_id"])
You can format your json here
https://jsonformatter.curiousconcept.com/
to see, wich list is in wich :)
#infinity wrote it similar.
[
{
"type":1,
"components":[
{
"type":2,
"style":1,
"label":"She/Her",
"custom_id":"She/Her"
},
{
"style":1,
"label":"He/Him",
"custom_id":"He/Him",
"type":2
}
]
}
]
myList = [{'type': 1, 'components': [{'type': 2, 'style': 1, 'label': 'She/Her', 'custom_id': 'She/Her'}, {'style': 1, 'label': 'He/Him', 'custom_id': 'He/Him', 'type': 2}]}]
for user in myList[0]['components']:
print(user['custom_id'])
I am new to python and am struggling with remove a key and value from a json return by an http request. When querying a task I get the following back.
data = requests.get(url,headers=hed).json()['data']
[{
'gid': '12011553977',
'due_on': None,
'name': 'do something',
'notes': 'blalbla,
'projects': [{
'gid': '120067502445',
'name': 'Project1'
}]
}, {
'gid': '12002408815',
'due_on': '2021-10-21',
'name': 'Proposal',
'notes': 'bla',
'projects': [{
'gid': '12314323523',
'name': 'Project1'
}, {
'gid': '12314323523',
'name': 'Project2'
}, {
'gid': '12314323523',
'name': 'Project3'
}]
I am trying to remove 'gid' from all projects so projects look like this
'projects': [{
'name': 'Company'
}]
What is the best way to do this with python3?
You can use recursion to make a simpler function to handle all elements and sub-elements. I haven't done extensive testing, or included any error checking or exception handling; but this should be close to what you want:
def rec_pop(top_level_list,key_to_pop='gid'):
for item in top_level_list:
item.pop(key_to_pop)
for v in item.values():
if isinstance(v,list):
rec_pop(v)
# call recursive fn
rec_pop(data)
Result:
In [25]: data
Out[25]:
[{'due_on': None,
'name': 'do something',
'notes': 'blalbla',
'projects': [{'name': 'Project1'}]},
{'due_on': '2021-10-21',
'name': 'Proposal',
'notes': 'bla',
'projects': [{'name': 'project2'}]}]
I am working on a React project, where I am trying to pass jsondata from one folder to another folder but it's not working. It is showing an error like this
./src/Pages/Dashboard/Dashboard.js Module not found: Can't resolve
'./API/jsondata'
This is is my code
This is jsondata.js
{
user: [
{
'id': '1',
'name': 'test1',
'age': '11',
'gender': 'male',
'email': 'test1#gmail.com'
},
{
'id': '2',
'name': 'test2',
'age': '12',
'gender': 'male',
'email': 'test2#gmail.com'
}, {
'id': '3',
'name': 'test3',
'age': '13',
'gender': 'male',
'email': 'test3#gmail.com'
}, {
'id': '4',
'name': 'test4',
'age': '14',
'gender': 'male',
'email': 'test4#gmail.com'
}, {
'id': '5',
'name': 'test5',
'age': '15',
'gender': 'male',
'email': 'test5#gmail.com'
},
{
'id': '6',
'name': 'test6',
'age': '16',
'gender': 'male',
'email': 'test6#gmail.com'
},
]
}
This is Dashboard.js
import React from 'react';
import Jsondata from './API/jsondata'
import './Dashboard.css';
const Dashboard = () => {
console.log(Jsondata, 'data')
return (
<div className='container'>
<div className='row'>
<div className='col-12'>
</div>
</div>
</div>
)
}
export default Dashboard
Any ideas to what may be causing the problem?
There can be many reasons:
First one you didn't exported your data from json file.
export const Jsondata = [
{
'id': '1',
'name': 'test1',
'age': '11',
'gender': 'male',
'email': 'test1#gmail.com'
},
{
'id': '2',
'name': 'test2',
'age': '12',
'gender': 'male',
'email': 'test2#gmail.com'
},
];
Second maybe your file path is wrong.Just double check it.Here you have given the name Jsondata but there is no such const in json file.
import {Jsondata} from './API/jsondata'
You can define a variable inside jsondata.js file.
Like;
const users = [
...
];
export { users };
And from Dashboard.js, you can import as
import { users } from './API/jsondata';
Rename the jsondata file to data.json (the file should have .json extension) and then it works. Your import can omit .json extension like this
import mydata from "./data";
Note: You don't have to export anything in data.json file
I have a Map<Datetime, List> in Dart that stores events needed to fill up a calendar in my Flutter app:
final Map<DateTime, List> _events = {
DateTime(2020, 7, 7): [
{'name': 'Event A', 'isDone': true},
],
DateTime(2020, 7, 11): [
{'name': 'Event A', 'isDone': true}
],
DateTime(2020, 7, 9): [
{'name': 'Event A', 'isDone': true},
{'name': 'Event B', 'isDone': true},
],
DateTime(2020, 7, 10): [
{'name': 'Event A', 'isDone': true},
{'name': 'Event B', 'isDone': true},
],
DateTime(2020, 7, 13): [
{'name': 'Event A', 'isDone': true},
{'name': 'Event B', 'isDone': true},
{'name': 'Event C', 'isDone': false},
],
DateTime(2020, 7, 25): [
{'name': 'Event A', 'isDone': true},
{'name': 'Event B', 'isDone': true},
{'name': 'Event C', 'isDone': false},
],
DateTime(2020, 7, 6): [
{'name': 'Event A', 'isDone': false},
],
};
I want to convert this into JSON but I'm not entirely sure how to model the class. Anyone got any ideas?
If you simply want to convert this to a JSON you can use the jsonEncode function and use the toEncodable named parameter to allow your object to be encoded. I made an example here that just cast the keys of your Map to a String so that it can be encoded.
jsonEncode(
_events,
toEncodable: (input) {
return _events.map((key, value) {
return MapEntry(key.toString(), value);
});
}
);
When you want to decode this back to your Map<DateTime, List> you can just do the reverse.
var objectTemp = jsonDecode(
json,
);
var output = objectTemp.map((key, value) {
return MapEntry(DateTime.parse(key), value);
})
Create a model class for Event and store the event data in it (name, isDone and date). Then you put them in a List<Event>. You can search on this list for a spesific date and get the according events if any. Then you can make that data class serializable.
I need to request a PG Database thru an API, I am using Flask requests package :
payload = {'key':'**', 'schema':'**', 'table':'testh','where_clause':'0', 'liste_fields':'*'}
r = requests.get('https://myapi/', params=payload, verify=False)
I need to get all the content of my testh table store in a JSON, but the function r.json() gets me this invalid JSON :
{'id': {'0': '1', '1': '2', '2': '3', '3': '4'}, 'brand': {'0': 'apple', '1': 'microsoft', '2': 'google', '3': 'amazon'}}
I need a JSON as : {0: {id:'2', brand:'apple}, 1:{id:'2', brand:'microsoft}, ....}
You can use pandas for this:
import pandas as pd
d = {'id': {'0': '1', '1': '2', '2': '3', '3': '4'}, 'brand': {'0': 'apple', '1': 'microsoft', '2': 'google', '3': 'amazon'}}
d = pd.DataFrame(d).to_dict(orient='index')
Output:
{'0': {'id': '1', 'brand': 'apple'},
'1': {'id': '2', 'brand': 'microsoft'},
'2': {'id': '3', 'brand': 'google'},
'3': {'id': '4', 'brand': 'amazon'}}
It is just a guess, but maybe the API used pandas as well and if you call to_dict() without parameter you get exactly the output you are describing.