2 Nested Dictionaries Json format using Json_norminalize using python - json

Question

Hi I am trying to retrieve this data, there are 2 nested dictionaries within it :
{
"metadata": {
"stations": [
{
"id": "S108",
"device_id": "S108",
"name": "Kuala Lumpur",
"location": {
"latitude": 3.1390,
"longitude": 101.6869
}
},
{
"id": "S118",
"device_id": "S118",
"name": "Bukit Bintang",
"location": {
"latitude":3.1468,
"longitude": 101.7113
}
}
],
"reading_type": "DBT 1M F",
"reading_unit": "deg C"
},
"items": [
{
"timestamp": "2021-06-20T15:05:00+08:00",
"readings": [
{
"station_id": "S108",
"value": 32.6
},
{
"station_id": "S118",
"value": 30.3
}
]
}
]
I wanted to get the result like this :
Result
I have tried a few ways :
data = airtemp.json()
df = pd.json_normalize(data,record_path=['metadata', 'stations'])
df
data = airtemp.json()
df1 = pd.json_normalize(data,record_path=['items','readings'])
df1
Is there a way that I can use json_norminalize to form one table with station_id, name, latitude, longtitude, timestamp and value without breaking into 2 tables ?
Many thanks!

Answer 1

You can merge the two dataframes you created:
import pandas as pd
data = {
"metadata": {
"stations": [
{
"id": "S108",
"device_id": "S108",
"name": "Kuala Lumpur",
"location": {
"latitude": 3.1390,
"longitude": 101.6869
}
},
{
"id": "S118",
"device_id": "S118",
"name": "Bukit Bintang",
"location": {
"latitude": 3.1468,
"longitude": 101.7113
}
}
],
"reading_type": "DBT 1M F",
"reading_unit": "deg C"
},
"items": [
{
"timestamp": "2021-06-20T15:05:00+08:00",
"readings": [
{
"station_id": "S108",
"value": 32.6
},
{
"station_id": "S118",
"value": 30.3
}
]
}
]
}
stations = pd.json_normalize(data, record_path=['metadata', 'stations'])
readings = pd.json_normalize(data, record_path=['items', 'readings'])
result = stations.merge(readings, left_on='id', right_on='station_id')
with pd.option_context('display.max_rows', None, 'display.max_columns', None):
print(result)
Outputs:
id device_id name location.latitude location.longitude \
0 S108 S108 Kuala Lumpur 3.1390 101.6869
1 S118 S118 Bukit Bintang 3.1468 101.7113
station_id value
0 S108 32.6
1 S118 30.3
There is only one timestamp in the data you provided so you will have to fetch that separately.