need struct type but got string: Cannot extract value - json

Question

I have a tricky scenario that I need help with, I have to extract information from a field.
The Data set can EITHER look like this:
"orderDelivery": {
"id":123,
"deliveryFee": 3500,
"deliveryFeeExcludingVAT": 3500,
"deliveryFeeRefunded": true,
"immediatedeliveryselected": true,
"deliverySlotId": null,
"deliverySlotChanged": false,
"driver": null,
"storeCollectionSLATime": 1635759479391,
"storeArrivalTimeEstimate": null,
"storeArrivalTimeActual": null,
"timeDelivered": null,
"timeCollected": null,
"DeliveryEstimates": [
{
"estimatedDeliveryTime": 1635762179390,
"createdOn": 1635758579390
}
],
"instructions": null,
"deliveryInstructions": null,
"deliveryPartnerCommunication": {
"informedOfNewOrder": true,
"serviceMessage": null
},
"deliveryIssueIds": null
}
Or it can look like this:
"orderDelivery": {
"id":1234,
"deliveryFee": 3500,
"deliveryFeeExcludingVAT": 3500,
"deliveryFeeRefunded": true,
"immediatedeliveryselected": true,
"deliverySlotId": null,
"deliverySlotChanged": false,
"driver": null,
"storeCollectionSLATime": 1635759479391,
"storeArrivalTimeEstimate": null,
"storeArrivalTimeActual": null,
"timeDelivered": null,
"timeCollected": null,
"DeliveryEstimates": [
{
"estimatedDeliveryTime": 1635762179390,
"createdOn": 1635758579390
}
],
"instructions": null,
"deliveryInstructions": [
{
"createdOn": 1635762179390,
"deliveryMessage": "delivered"
},
{
"createdOn": 1735762179390,
"deliveryMessage": "not door"
}
],
"deliveryPartnerCommunication": {
"informedOfNewOrder": true,
"serviceMessage": null
},
"deliveryIssueIds": null
}
I need to take into account when deliveryInstructions is null or when it is an array.
I am extracting the data like this:
df["orderDelivery"]["deliveryInstructions"]["createdOn"].cast(StringType()).alias("created_on")
And I am also extracting like this :
df["orderDelivery"]["deliveryInstructions"]["deliveryMessage"].cast(StringType()).alias("delivery_message")
When I query it when is null, I get the following error:
Can't extract value from orderDelivery#27.deliveryInstructions: need struct type but got
string
Can anyone help on how to handle this?
also when when delivery instruction repeats I want it look like this
id
createdOn
deliveryMessage
1234
1635762179390
delivered
1234
1735762179390
Not door
There can be one or more delivery instructions in addition to the null value.
If it's null then I just want null for both CreatedOn and DeliveryMessage