Accessing pandas dataframe columns that are not in Index - json

Question

I am trying to pull a few fields from the following nested json and write to a separate csv file:
{
"AccountID": "00000000-0000-0000-0000-000000000000",
"LocationID": "00000000-0000-0000-0000-000000000000",
"CreatedBy": "string",
"ModifiedBy": "string",
"Created": "string",
"Modified": "string",
"LocationData": {
"KeyFields": {},
"DisplayPoint": {
"Type": "Calculated",
"Latitude": 0.0,
"Longitude": 0.0,
"VerificationType": "Client"
},
"BusinessStatus": "Open",
"Status": "Active",
"BusinessName": {
"Name": "string",
"LongName": "string",
"Locale": "Not_set"
},
"BusinessDescription": {
"Description": "string",
"ShortDescription": "string",
"LongDescription": "string"
},
"PrimaryAddress": {
"AddressLine1": "string",
"AddressLine2": "string",
"AddressLine3": "string",
"AddressLine4": "string",
"AddressLine5": "string",
"Neighborhood": "string",
"Locality": "string",
"Region": "string",
"PostalCode": "string",
"CountryCode": "string"
},
"PhoneNumbers": {
"PrimaryPhoneNumber": "string",
"Landline": "string",
"Mobile": "string",
"Fax": "string",
"TollFree": "string"
},
"HoursOfOperationStructured": {
"Su": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"Mo": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"Tu": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"We": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"Th": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"Fr": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"Sa": {
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
},
"SpecialHours": [
{
"Date": "string",
"Ranges": [
{
"StartTime": "string",
"EndTime": "string"
}
],
"State": "Open",
"AdditionalInfo": "string"
}
]
}
}
I am able to use pandas and json_normalize to flatten the data. Then I am able to pull fields by referencing the fields I want like df['LocationData.PrimaryAddress.Locality']. This works for all the fields I need except the 'StartTime' and 'EndTime' ranges which throw a KeyError
When I try to extract the 'StartTime' or 'Endtime' ranges of any specific day by referencing it like so: df['LocationData.HoursOfOperationStructured.Su.Ranges.StartTime'] ---- it returns with a
KeyError: "['LocationData.HoursOfOperationStructured.Su.Ranges.StartTime'] not in index"
How can I access the 'StartTime'/'EndTime' columns for all of the days from this file using pandas?

Answer 1

The column df['LocationData.HoursOfOperationStructured.Su.Ranges'] and all other similar columns have been "undernormalized": they contain single-element lists of dictionaries with the keys "StartTime" and "EndTime". You can convert these dictionary columns to "real" columns in a loop and then concatenate with the original dataframe:
ranges = df.columns[df.columns.str.match('.*Ranges.*')]
missing = [df[r].str[0].apply(pd.Series)\
.rename(columns={'StartTime' : f"{r}.StartTime",
'EndTime' : f"{r}.EndTime"})
for r in ranges]
df = df.join(pd.concat(missing, axis=1))
It's ugly, but it works.