Extract and explode embedded json fields in apache spark - json

Question

I'm completely new to spark, but don't mind if the answer is in python or Scala. I can't show the actual data for privacy reasons, but basically I am reading json files with a structure like this:
{
"EnqueuedTimeUtc": 'some date time',
"Properties": {},
"SystemProperties": {
"connectionDeviceId": "an id",
"some other fields that we don't care about": "data"
},
"Body": {
"device_id": "an id",
"tabs": [
{
"selected": false,
"title": "some title",
"url": "https:...."
},
{"same again, for multiple tabs"}
]
}
}
Most of the data is of no interest. What I want is a Dataframe consisting of the time, device_id, and url. There can be multiple urls for the same device and time, so I'm looking to explode these into one row per url.
| timestamp | device_id | url |
My immediate problem is that when I read this, although it can work out the structure of SystemProperties, Body is just a string, probably because of variation. Perhaps I need to specify the schema, would that help?
root
|-- Body: string (nullable = true)
|-- EnqueuedTimeUtc: string (nullable = true)
|-- SystemProperties: struct (nullable = true)
| |-- connectionAuthMethod: string (nullable = true)
| |-- connectionDeviceGenerationId: string (nullable = true)
| |-- connectionDeviceId: string (nullable = true)
| |-- contentEncoding: string (nullable = true)
| |-- contentType: string (nullable = true)
| |-- enqueuedTime: string (nullable = true)
Any idea of an efficient (there are lots and lots of these records) way to extract urls and associate with the time and device_id? Thanks in advance.

Answer 1

Here's an example for extraction. Basically you can use from_json to convert the Body to something that is more structured, and use explode(transform()) to get the URLs and expand to different rows.
# Sample dataframe
df.show(truncate=False)
+----------------------------------------------------------------------------------------------------------------------------------------------+---------------+----------------+
|Body |EnqueuedTimeUtc|SystemProperties|
+----------------------------------------------------------------------------------------------------------------------------------------------+---------------+----------------+
|{"device_id":"an id","tabs":[{"selected":false,"title":"some title","url":"https:1"},{"selected":false,"title":"some title","url":"https:2"}]}|some date time |[an id] |
+----------------------------------------------------------------------------------------------------------------------------------------------+---------------+----------------+
df.printSchema()
root
|-- Body: string (nullable = true)
|-- EnqueuedTimeUtc: string (nullable = true)
|-- SystemProperties: struct (nullable = true)
| |-- connectionDeviceId: string (nullable = true)
# Extract desired properties
df2 = df.selectExpr(
"EnqueuedTimeUtc as timestamp",
"from_json(Body, 'device_id string, tabs array<map<string,string>>') as Body"
).selectExpr(
"timestamp",
"Body.device_id",
"explode(transform(Body.tabs, x -> x.url)) as url"
)
df2.show()
+--------------+---------+-------+
| timestamp|device_id| url|
+--------------+---------+-------+
|some date time| an id|https:1|
|some date time| an id|https:2|
+--------------+---------+-------+