Extract fields from json in Pyspark - json

Question

I am trying to extract only itineraries.element and validatingAirlineCodes and then form a json containing only these two fields in Pyspark
|-- id: string (nullable = true)
|-- instantTicketingRequired: boolean (nullable = true)
|-- itineraries: array (nullable = true)
| |-- element: map (containsNull = true)
| | |-- key: string
| | |-- value: string (valueContainsNull = true)
|-- lastTicketingDate: string (nullable = true)
|-- nonHomogeneous: boolean (nullable = true)
|-- numberOfBookableSeats: long (nullable = true)
|-- oneWay: boolean (nullable = true)
|-- price: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)
|-- pricingOptions: map (nullable = true)
| |-- key: string
| |-- value: array (valueContainsNull = true)
| | |-- element: string (containsNull = true)
|-- source: string (nullable = true)
|-- travelerPricings: array (nullable = true)
| |-- element: map (containsNull = true)
| | |-- key: string
| | |-- value: string (valueContainsNull = true)
|-- type: string (nullable = true)
|-- validatingAirlineCodes: array (nullable = true)
| |-- element: string (containsNull = true)
I tried using the df.select() but cant select the fields I want. What should I do ?

Answer 1

Something to bear in mind is that what you are trying to extract (element) is an Array of Maps. There is no logical way to extract this without first exploding the array and proceeding to extract element.
Assuming you already have your dataframe stored as df you can explode itineraries using the below example. This will create a row in your dataframe for each element of your array and should allow you to extract your element quite simply thereafter.
df.select(explode(df.itineraries))
Extra reading for context:
Documentation page on Spark By Examples.
https://sparkbyexamples.com/pyspark/select-columns-from-pyspark-dataframe/
Documentation on explode in pySpark is here: https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.functions.explode.html
Hope this helps!