Turning a multi-dictionary JSON into a Pandas dataframe - json

Question

I would like to turn a JSON data structure into a pandas dataframe. My data is retrieved from OpenWeatherMap. The resulting JSON contain multiple nested dictionaries which contain weather data on cities, divided by the URL where the data are retrieved from. Here the last two lines of the JSON, called json_data:
http://api.openweathermap.org/data/2.5/weather?units=Imperial&APPID=a13759df887d2de294c2c7adef912758&q=new norfolk:
{'coord': {'lon': 147.0587, 'lat': -42.7826}, 'weather': [{'id': 803, 'main': 'Clouds', 'description': 'broken clouds', 'icon': '04d'}], 'base': 'stations', 'main': {'temp': 46.04, 'feels_like': 44.29, 'temp_min': 43.92, 'temp_max': 51.93, 'pressure': 1000, 'humidity': 74}, 'visibility': 10000, 'wind': {'speed': 4, 'deg': 319, 'gust': 15.99}, 'clouds': {'all': 77}, 'dt': 1652657623, 'sys': {'type': 2, 'id': 2031307, 'country': 'AU', 'sunrise': 1652649449, 'sunset': 1652684337}, 'timezone': 36000, 'id': 2155415, 'name': 'New Norfolk', 'cod': 200}
http://api.openweathermap.org/data/2.5/weather?units=Imperial&APPID=a13759df887d2de294c2c7adef912758&q=fortuna:
{'coord': {'lon': -124.1573, 'lat': 40.5982}, 'weather': [{'id': 801, 'main': 'Clouds', 'description': 'few clouds', 'icon': '02d'}], 'base': 'stations', 'main': {'temp': 66.67, 'feels_like': 66.18, 'temp_min': 64.98, 'temp_max': 67.93, 'pressure': 1017, 'humidity': 67}, 'visibility': 10000, 'wind': {'speed': 17.27, 'deg': 360}, 'clouds': {'all': 20}, 'dt': 1652657623, 'sys': {'type': 2, 'id': 2040243, 'country': 'US', 'sunrise': 1652619589, 'sunset': 1652671580}, 'timezone': -25200, 'id': 5563839, 'name': 'Fortuna', 'cod': 200}
However, when I turn the JSON into a Pandas Dataframe, only the last dictionary goes into the dataframe.
Here is my code:
pd.set_option('display.max_columns', None)
pd.json_normalize(json_data)
Here is the result (I cannot copy the panda dataframe directly without losing formatting).
Why is only the last dictionary turned into a dataframe? How can I get a multiple-line dataframe?

Answer 1

If you're only seeing one line in your dataframe, you are probably overwriting your json_data with the last value.
Besides, you can normalize the weather column separately and join it to the rest:
json_data = [
{'coord': {'lon': 147.0587, 'lat': -42.7826}, 'weather': [{'id': 803, 'main': 'Clouds', 'description': 'broken clouds', 'icon': '04d'}], 'base': 'stations', 'main': {'temp': 46.04, 'feels_like': 44.29, 'temp_min': 43.92, 'temp_max': 51.93, 'pressure': 1000, 'humidity': 74}, 'visibility': 10000, 'wind': {'speed': 4, 'deg': 319, 'gust': 15.99}, 'clouds': {'all': 77}, 'dt': 1652657623, 'sys': {'type': 2, 'id': 2031307, 'country': 'AU', 'sunrise': 1652649449, 'sunset': 1652684337}, 'timezone': 36000, 'id': 2155415, 'name': 'New Norfolk', 'cod': 200},
{'coord': {'lon': -124.1573, 'lat': 40.5982}, 'weather': [{'id': 801, 'main': 'Clouds', 'description': 'few clouds', 'icon': '02d'}], 'base': 'stations', 'main': {'temp': 66.67, 'feels_like': 66.18, 'temp_min': 64.98, 'temp_max': 67.93, 'pressure': 1017, 'humidity': 67}, 'visibility': 10000, 'wind': {'speed': 17.27, 'deg': 360}, 'clouds': {'all': 20}, 'dt': 1652657623, 'sys': {'type': 2, 'id': 2040243, 'country': 'US', 'sunrise': 1652619589, 'sunset': 1652671580}, 'timezone': -25200, 'id': 5563839, 'name': 'Fortuna', 'cod': 200}
]
pd.set_option('display.max_columns', None)
df = pd.json_normalize(json_data)
df = df.loc[:, df.columns!='weather'].join(pd.json_normalize(json_data, record_path='weather', record_prefix='weather.'))
print(df)
Output:
base visibility dt timezone id name cod \
0 stations 10000 1652657623 36000 2155415 New Norfolk 200
1 stations 10000 1652657623 -25200 5563839 Fortuna 200
coord.lon coord.lat main.temp main.feels_like main.temp_min \
0 147.0587 -42.7826 46.04 44.29 43.92
1 -124.1573 40.5982 66.67 66.18 64.98
main.temp_max main.pressure main.humidity wind.speed wind.deg \
0 51.93 1000 74 4.00 319
1 67.93 1017 67 17.27 360
wind.gust clouds.all sys.type sys.id sys.country sys.sunrise \
0 15.99 77 2 2031307 AU 1652649449
1 NaN 20 2 2040243 US 1652619589
sys.sunset weather.id weather.main weather.description weather.icon
0 1652684337 803 Clouds broken clouds 04d
1 1652671580 801 Clouds few clouds 02d
As you can see, both lines are in the dataframe