Related
I am working on a React project, where I am trying to pass jsondata from one folder to another folder but it's not working. It is showing an error like this
./src/Pages/Dashboard/Dashboard.js Module not found: Can't resolve
'./API/jsondata'
This is is my code
This is jsondata.js
{
user: [
{
'id': '1',
'name': 'test1',
'age': '11',
'gender': 'male',
'email': 'test1#gmail.com'
},
{
'id': '2',
'name': 'test2',
'age': '12',
'gender': 'male',
'email': 'test2#gmail.com'
}, {
'id': '3',
'name': 'test3',
'age': '13',
'gender': 'male',
'email': 'test3#gmail.com'
}, {
'id': '4',
'name': 'test4',
'age': '14',
'gender': 'male',
'email': 'test4#gmail.com'
}, {
'id': '5',
'name': 'test5',
'age': '15',
'gender': 'male',
'email': 'test5#gmail.com'
},
{
'id': '6',
'name': 'test6',
'age': '16',
'gender': 'male',
'email': 'test6#gmail.com'
},
]
}
This is Dashboard.js
import React from 'react';
import Jsondata from './API/jsondata'
import './Dashboard.css';
const Dashboard = () => {
console.log(Jsondata, 'data')
return (
<div className='container'>
<div className='row'>
<div className='col-12'>
</div>
</div>
</div>
)
}
export default Dashboard
Any ideas to what may be causing the problem?
There can be many reasons:
First one you didn't exported your data from json file.
export const Jsondata = [
{
'id': '1',
'name': 'test1',
'age': '11',
'gender': 'male',
'email': 'test1#gmail.com'
},
{
'id': '2',
'name': 'test2',
'age': '12',
'gender': 'male',
'email': 'test2#gmail.com'
},
];
Second maybe your file path is wrong.Just double check it.Here you have given the name Jsondata but there is no such const in json file.
import {Jsondata} from './API/jsondata'
You can define a variable inside jsondata.js file.
Like;
const users = [
...
];
export { users };
And from Dashboard.js, you can import as
import { users } from './API/jsondata';
Rename the jsondata file to data.json (the file should have .json extension) and then it works. Your import can omit .json extension like this
import mydata from "./data";
Note: You don't have to export anything in data.json file
I'm completing this IBM Data Science certification on Coursera and one of the assignments require us to replicate this link- https://rawnote.dinhanhthi.com/files/ibm/neighborhoods_in_toronto.
I'm fairly new to this so I was going through the link to understand it and I couldn't understand some parts of the code.
So the objective of this assignment is to:
Extract a table from wikipedia and store it in a dataframe
Create a map of toronto city and explore the boroughs that contain "Toronto"
Explore any random neighborhood in Toronto using the FourSqaure API ("The Beaches" have been chosen here)
Get the top 100 venues that are in "The Beaches" within a radius of 500 meters.
They've done the 4th point using the FourSqaure API as shown below:
LIMIT = 100 # limit of number of venues returned by Foursquare API
radius = 500 # define radius
url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
CLIENT_ID,
CLIENT_SECRET,
VERSION,
neighborhood_latitude,
neighborhood_longitude,
radius,
LIMIT)
# get the result to a json file
results = requests.get(url).json()
The "results" variable looks like this:
{'meta': {'code': 200, 'requestId': '5eda4fb9aba297001b2f6207'},
'response': {'headerLocation': 'The Beaches',
'headerFullLocation': 'The Beaches, Toronto',
'headerLocationGranularity': 'neighborhood',
'totalResults': 4,
'suggestedBounds': {'ne': {'lat': 43.680857404499996,
'lng': -79.28682091449052},
'sw': {'lat': 43.67185739549999, 'lng': -79.29924148550948}},
'groups': [{'type': 'Recommended Places',
'name': 'recommended',
'items': [{'reasons': {'count': 0,
'items': [{'summary': 'This spot is popular',
'type': 'general',
'reasonName': 'globalInteractionReason'}]},
'venue': {'id': '4bd461bc77b29c74a07d9282',
'name': 'Glen Manor Ravine',
'location': {'address': 'Glen Manor',
'crossStreet': 'Queen St.',
'lat': 43.67682094413784,
'lng': -79.29394208780985,
'labeledLatLngs': [{'label': 'display',
'lat': 43.67682094413784,
'lng': -79.29394208780985}],
'distance': 89,
'cc': 'CA',
'city': 'Toronto',
'state': 'ON',
'country': 'Canada',
'formattedAddress': ['Glen Manor (Queen St.)',
'Toronto ON',
'Canada']},
'categories': [{'id': '4bf58dd8d48988d159941735',
'name': 'Trail',
'pluralName': 'Trails',
'shortName': 'Trail',
'icon': {'prefix': 'https://ss3.4sqi.net/img/categories_v2/parks_outdoors/hikingtrail_',
'suffix': '.png'},
'primary': True}],
'photos': {'count': 0, 'groups': []}},
'referralId': 'e-0-4bd461bc77b29c74a07d9282-0'},
{'reasons': {'count': 0,
'items': [{'summary': 'This spot is popular',
'type': 'general',
'reasonName': 'globalInteractionReason'}]},
'venue': {'id': '4ad4c062f964a52011f820e3',
'name': 'The Big Carrot Natural Food Market',
'location': {'address': '125 Southwood Dr',
'lat': 43.678879,
'lng': -79.297734,
'labeledLatLngs': [{'label': 'display',
'lat': 43.678879,
'lng': -79.297734}],
'distance': 471,
'postalCode': 'M4E 0B8',
'cc': 'CA',
'city': 'Toronto',
'state': 'ON',
'country': 'Canada',
'formattedAddress': ['125 Southwood Dr',
'Toronto ON M4E 0B8',
'Canada']},
'categories': [{'id': '50aa9e744b90af0d42d5de0e',
'name': 'Health Food Store',
'pluralName': 'Health Food Stores',
'shortName': 'Health Food Store',
'icon': {'prefix': 'https://ss3.4sqi.net/img/categories_v2/shops/food_grocery_',
'suffix': '.png'},
'primary': True}],
'photos': {'count': 0, 'groups': []},
'venuePage': {'id': '75150878'}},
'referralId': 'e-0-4ad4c062f964a52011f820e3-1'},
{'reasons': {'count': 0,
'items': [{'summary': 'This spot is popular',
'type': 'general',
'reasonName': 'globalInteractionReason'}]},
'venue': {'id': '4b8daea1f964a520480833e3',
'name': 'Grover Pub and Grub',
'location': {'address': '676 Kingston Rd.',
'crossStreet': 'at Main St.',
'lat': 43.679181434941015,
'lng': -79.29721535878515,
'labeledLatLngs': [{'label': 'display',
'lat': 43.679181434941015,
'lng': -79.29721535878515}],
'distance': 460,
'postalCode': 'M4E 1R4',
'cc': 'CA',
'city': 'Toronto',
'state': 'ON',
'country': 'Canada',
'formattedAddress': ['676 Kingston Rd. (at Main St.)',
'Toronto ON M4E 1R4',
'Canada']},
'categories': [{'id': '4bf58dd8d48988d11b941735',
'name': 'Pub',
'pluralName': 'Pubs',
'shortName': 'Pub',
'icon': {'prefix': 'https://ss3.4sqi.net/img/categories_v2/nightlife/pub_',
'suffix': '.png'},
'primary': True}],
'photos': {'count': 0, 'groups': []}},
'referralId': 'e-0-4b8daea1f964a520480833e3-2'},
{'reasons': {'count': 0,
'items': [{'summary': 'This spot is popular',
'type': 'general',
'reasonName': 'globalInteractionReason'}]},
'venue': {'id': '4df91c4bae60f95f82229ad5',
'name': 'Upper Beaches',
'location': {'lat': 43.68056321147582,
'lng': -79.2928688743688,
'labeledLatLngs': [{'label': 'display',
'lat': 43.68056321147582,
'lng': -79.2928688743688}],
'distance': 468,
'cc': 'CA',
'city': 'Toronto',
'state': 'ON',
'country': 'Canada',
'formattedAddress': ['Toronto ON', 'Canada']},
'categories': [{'id': '4f2a25ac4b909258e854f55f',
'name': 'Neighborhood',
'pluralName': 'Neighborhoods',
'shortName': 'Neighborhood',
'icon': {'prefix': 'https://ss3.4sqi.net/img/categories_v2/parks_outdoors/neighborhood_',
'suffix': '.png'},
'primary': True}],
'photos': {'count': 0, 'groups': []}},
'referralId': 'e-0-4df91c4bae60f95f82229ad5-3'}]}]}}
I'm not sure how to proceed. The below image is what is mentioned in the link but:
I don't understand why they've created a function get_category_row?
Why are we writing venues = results['response']['groups'][0]['items']? Isn't json_normalize()
supposed to convert a json file to a datframe? So why cant we
directly do json_normalize(results)?
I'm pretty much lost from section 4.6 onwards in the link.
if anyone could help me out or guide me that would be amazing! :)
No, you are completely wrong json_normalize() normalize semi-structured JSON data into a flat table not to a DataFrame. That's why they use venues = results['response']['groups'][0]['items'] to get the venues. They used the function get_category_type() to get the category of the venue.
If you want to know more about json_normalize() please refer this link
json_normalize will only flatten the records in one path, for example in your json, you can flatten each path separately:
meta
response -> suggestedBounds
response -> groups -> items
And then you'd have to merge them together
df1 = pd.json_normalize(d['response'], record_path=['groups', 'items'], meta=[])
print(df1)
df2 = pd.json_normalize(d['response'])
print(df2)
df3 = pd.json_normalize(d['meta'])
print(df3)
referralId reasons.count ... venue.location.postalCode venue.venuePage.id
0 e-0-4bd461bc77b29c74a07d9282-0 0 ... NaN NaN
1 e-0-4ad4c062f964a52011f820e3-1 0 ... M4E 0B8 75150878
2 e-0-4b8daea1f964a520480833e3-2 0 ... M4E 1R4 NaN
3 e-0-4df91c4bae60f95f82229ad5-3 0 ... NaN NaN
[4 rows x 21 columns]
headerLocation headerFullLocation headerLocationGranularity ... suggestedBounds.ne.lng suggestedBounds.sw.lat suggestedBounds.sw.lng
0 The Beaches The Beaches, Toronto neighborhood ... -79.286821 43.671857 -79.299241
[1 rows x 9 columns]
code requestId
0 200 5eda4fb9aba297001b2f6207
If you want to flatten the full json, you can try flatten_json. Documentation: Flatten JSON
I have a list of nested dictionary as below. There is list inside the dictionary as well. I don't want to change the inner list but to turn the entire list into dictionary. The list is as below:
[{"id" : "A", "data" : [{"key_1" : 012},{"key_2" : 123}, {"key_3" : 234}]}]
I only want to change the list itself to dictionary without changing the inner list as below:
{{"id" : "A", "data" : [{"key_1" : 012},{"key_2" : 123}, {"key_3" : 234}]}}
Appreciate your help in this.
Use a dictionary comprehension and zip the list of dictionaries up with whatever list of keys you want. Here is an example using a subset of ascii lowercase letters:
{ pair[0]: pair[1] for pair in list(zip([*string.ascii_lowercase[0:len(input)]], input))}
>>> foo = [ { 'Age': 32,
'Emp_id': 'A',
'Gender': 'M',
'Result': [ {'Incentive': 3000, 'Month': 'Aug'},
{'Incentive': 3500, 'Month': 'Sep'},
{'Incentive': 2000, 'Month': 'Oct'}]},
{ 'Age': 35,
'Emp_id': 'B',
'Gender': 'M',
'Result': [{'Incentive': 1500, 'Month': 'Aug'}]},
{ 'Age': 31,
'Emp_id': 'C',
'Gender': 'F',
'Result': [ {'Incentive': 5000, 'Month': 'Aug'},
{'Incentive': 2400, 'Month': 'Sep'}]}]
>>> { pair[0]: pair[1] for pair in list(zip([*string.ascii_lowercase[0:len(input)]], input))}
>>> { 'a': { 'Age': 32,
'Emp_id': 'A',
'Gender': 'M',
'Result': [ {'Incentive': 3000, 'Month': 'Aug'},
{'Incentive': 3500, 'Month': 'Sep'},
{'Incentive': 2000, 'Month': 'Oct'}]},
'b': { 'Age': 35,
'Emp_id': 'B',
'Gender': 'M',
'Result': [{'Incentive': 1500, 'Month': 'Aug'}]},
'c': { 'Age': 31,
'Emp_id': 'C',
'Gender': 'F',
'Result': [ {'Incentive': 5000, 'Month': 'Aug'},
{'Incentive': 2400, 'Month': 'Sep'}]}}
note: you can pretty print code as follows:
import pp = pprint.PrettyPrinter(indent=4)
pp.pprint(data)
I am making a API call and getting my JSON data like so:
import requests
import jmespath
import pandas as pd
import json
url = 'a.com'
r = requests.get(url).json()
The object returned looks like this:
{'question': [{
'response': {'firstname': {'value': 'John'},
'lastname': {'value': 'Bob'}},
'profile_question': [{
'identities': [{'type': 'ID,
'value': '1'},
{'type': 'EMAIL',
'value': 'test#test.com'}]}]}]}
I tried putting this into json.fr but I get error that it is not correctly formed json. How every I can crawl this object as is but not successfully for what I need.
I am trying to use jmespath library to crawl and want to pull out four pieces of information firstname, lastname, ID, EMAIL like so and appending the data into list:
lst =[]
fname = jmespath.search('question[*].response.{firstname:firstname.value}',my_dict)
lst.append(fname)
lname = jmespath.search('question[*].response.{lastname:lastname.value}',my_dict)
lst.append(lname)
email_path = jmespath.search("question[*].profile_question[].identities.{email:[?type=='EMAIL'].value}",my_dict)
lst.append(email)
ID = jmespath.search("question[*].profile_question[].identities.{email:[?type=='ID'].value}",my_dict)
lst.append(ID)
I append into a list in hopes of creating tuples per iteration that I can push into a dataframe.
The list looks like this:
[[{'firstname': 'John'}],
[{'lastname': 'Bob'}],
[{'email': ['test#test.com']}],
[{'ID': ['1']}]]
However when I crawl the dictionary with missing values like so:
{'question': [{
'response': {'firstname': {'value': 'John'},
'lastname': {'value': 'Bob'}},
'profile-question': [{
'identities': [{'type': 'ID,
'value': '1'},
{'type': 'EMAIL',
'value': 'test#test.com'}]}]}],
'response': {'firstname': {'value': 'John1'},
'lastname': {'value': 'Bob1'}},
'profile-question': [{
'identities': [{'type': 'ID,
'value': '2'}]}]}
causes my list to behave like this (I can not tell why):
[[{'firstname': 'John'}], [{'email': ['test#test.com']}], [{'email': ['1']},[{'firstname': 'John'}],
[{'lastname': 'Bob'}],
[{'email': [][][]}],
[{'ID': ['1']}]]]
which causes the df to look like this:
firstname lastname email ID
john bob test#test.com 1
john1 bob1 test#test.com 1
How do I crawl a JSON dict object as it comes in from the API, pulling out four pieces of data firstname, lastname, email, ID and appending into a dataframe like so? :
firstname lastname email ID
john bob test#test.com 1
john1 bob1 2
More than willing to get away from jmespath library, and to add, the above dictionary has many more fields, I have shortened so only the key points and their indentation is listed.
Well before anything the reason for the error is because the json object is missing quotes after ID.
{'question': [{
'response': {'firstname': {'value': 'John'},
'lastname': {'value': 'Bob'}},
'profile_question': [{
'identities': [{'type': 'ID,
'value': '1'},
{'type': 'EMAIL',
'value': 'test#test.com'}]}]}]}
It should look like this:
{'question': [{
'response': {'firstname': {'value': 'John'},
'lastname': {'value': 'Bob'}},
'profile_question': [{
'identities': [{'type': 'ID',
'value': '1'},
{'type': 'EMAIL',
'value': 'test#test.com'}]}]}]}
From here you can use the json library to turn the json object into a python dictionary object with json.loads(). Once you fixed the json object, your code can look something like this.
import jmespath as jp
import pandas as pd
jon = {'question':
[{'response': {'firstname': {'value': 'John'},
'lastname': {'value': 'Bob'}},
'profile_question': [{'identities': [{'type': 'ID',
'value': '1'},
{'type': 'EMAIL', 'value': 'test#test.com'}]}]}]}
jsons = [jon] # list of all json objects
df_list = []
for json in jsons:
try:
fname = jp.search('question[*].response.firstname.value', jon)[0]
except IndexError:
fname = None
try:
lname = jp.search('question[*].response.lastname.value', jon)[0]
except IndexError:
lname = None
try:
email = jp.search("question[*].profile_question[].identities.{email:[?type=='EMAIL'].value}", jon)[0]['email'][0]
except IndexError:
email = None
try:
user_id = jp.search("question[*].profile_question[].identities.{email:[?type=='ID'].value}", jon)[0]['email'][0]
except IndexError:
user_id = None
df_list.append(pd.DataFrame({'firstname': fname, 'lastname': lname, 'email': email, 'id': user_id}, index=[0]))
df = pd.concat(df_list, ignore_index=True, sort=False)
print(df)
I'm starting to dig into graph databases, but i have no idea, how these graphs are stored internally. Let's say i have this graph (taken from Wikipedia):
How do i serialize this graph as a key-value object? (a Python dict, for example)
I imagine two dicts, one for vertices and one for edges:
{'vertices':
{'1': {'Name': 'Alice', 'Age': 18},
'2': {'Name': 'Bob', 'Age': 22},
'3': {'Type': 'Group', 'Name': 'Chess'}},
'edges':
{'100': {'Label': 'knows', 'Since': '2001/10/03'},
'101': {'Label': 'knows', 'Since': '2001/10/04'},
'102': {'Label': 'is_member', 'Since': '2005/7/01'},
'103': {'Label': 'Members'},
'104': {'Label': 'Members'},
'105': {'Label': 'is_member', 'Since': '2011/02/14'}},
'connections': [['1', '2', '100'], ['2', '1', '101'],
['1', '3', '102'], ['3', '1', '103'],
['3', '2', '104'], ['2', '3', '105']]}
But i'm not sure, whether this is the most practical implementation. Maybe the "connections" should be inside "vertices" dict. So, what is the best way to implement graph datastore using key-value objects? What and where can i read more about it?
Possibly related, but not a duplicate: How to represent a strange graph in some data structure
The normal pattern is to not have a separate connections structure but to put that information in the edges structure. This gives something like:
{
'vertices': {
'1': {'Name': 'Alice', 'Age': 18},
'2': {'Name': 'Bob', 'Age': 22},
'3': {'Type': 'Group', 'Name': 'Chess'} },
'edges': [
{'from': '1', 'to': '2', 'Label': 'knows', 'Since': '2001/10/03'},
{'from': '2', 'to': '1', 'Label': 'knows', 'Since': '2001/10/04'},
{'from': '1', 'to': '3', 'Label': 'is_member', 'Since': '2005/7/01'},
{'from': '3', 'to': '1', 'Label': 'Members'},
{'from': '3', 'to': '2', 'Label': 'Members'},
{'from': '2', 'to': '3', 'Label': 'is_member', 'Since': '2011/02/14'} ] }
seems ok - each object has its it, there is no duplications. it's good for 'read and process purpose'. but there is no 'best' representation. it always depends on your purpose. do you want to be able to quickly find vertices by name? or edges by date? or maybe you want to quickly test if two vertices are connected? or the opposite - you want to quickly modify some parts of the graph? each purpose requires different data structures of database tables
how these graphs are stored internally
how do I serialize this graph as a key-value object
These questions are different and they need different answers.
In the former case, the main requirement is probably to perform complex queries efficiently.
I'd suggest to investigate existing industrial-strength solutions.
In NoSQL terms, these nested key-value objects are documents. Hence, one could look into how graphs are stored in "layered" multi-model databases that:
support graph data model, and
use underlying document data model.
Examples of such databases are ArangoDB, OrientDB, Azure CosmosDB.
You could also replace "document data model" with "wide column data model", because wide column data model can be conidered as two-dimensional key-value model.
Examples of such databases are DataStax Enterprise Graph and perhaps Grakn.
For instance, in ArangoDB, edges are stored as regular documents, but in special collections.
Obviously, data structures used may be accompanied with additional indexes etc. (or not).
So, what is the best way to implement graph datastore using key-value objects?
What and where can i read more about it?
I'd suggest another one article from ArangoDB:
Storing a graph in a pure document store
I'd make few changes in Eamonn's answer.
Every vertex and edge has 3 things.. id, Label and Properties
{
'vertices': {
'1': {'Label' : Person, 'Properties' : { 'Name': 'Alice', 'Age': 18}},
'2': {'Label' : Person, 'Properties' : {'Name': 'Bob', 'Age': 22}},
'3': {'Label': 'Group', 'Properties' : { 'Name': 'Chess'} },
'edges': [
'4' : {'from': '1', 'to': '2', 'Label': 'knows', 'Properties':{'Since': '2001/10/03' , 'Until' : '2001/10/03'}},
'5' : {'from': '2', 'to': '1', 'Label': 'knows', 'Properties':{'Since': '2001/10/04', 'Until' : '2001/10/05'}}
]
}
This way you can do query by vertex/edge, and their Labels and their properties.
I would serialize it like this, except you should choose the keys based on what you are looking up by. I assumed you are using the id, but perhaps using the name could be better.
{
'members': {
'1': {
'id': '1',
'name': 'Alice',
'age': 18,
'groups': {
'3': {
'path': 'groups.3',
'since': '2005-07-01'
}
},
'knows': {
'2': {
'path': 'members.2',
'since': '2001-10-03'
}
}
},
'2': {
'id': '2',
'name': 'Bob',
'age': 22,
'groups': {
'3': {
'path': 'groups.3',
'since': '2011-02-14'
}
},
'knows': {
'1': {
'path': 'members.1',
'since': '2001-10-04'
}
}
}
},
'groups': {
'3': {
'id': '3',
'name': 'Chess',
'members': {
'1': { 'path': 'members.1' },
'2': { 'path': 'members.2' }
}
}
}
}
You can serialize graphs directly into key-value pairs if you have a way of serializing references to other parts of the graph, which is what I use 'path' for. If I was deserializing it into a dict, I may consider replacing the path values with the actual dictionaries they refer to. Keep in mind that this may cause circular references which could cause problems if you were serializing it into json or something.
I would add an adjacency to the structure too. My take would be like this,
{
'vertices': {
'1': {'Name': 'Alice', 'Age': 18},
'2': {'Name': 'Bob', 'Age': 22},
'3': {'Type': 'Group', 'Name': 'Chess'}
},
'edges': {
'100' : {'from': '1', 'to': '2', 'Label': 'knows', 'Since': '2001/10/03'},
'101': {'from': '2', 'to': '1', 'Label': 'knows', 'Since': '2001/10/04'},
....
},
'adjacency': {
'1': ['101', '102'],
...
}
}
This way I can easily find which edges are adjacent to my vertices instead of iterating through all the edges.