Aggregate JSON files in Spark RDD - json

Question

I have a series of files that look similar to this:
[
{
'id':1,
'transactions': [
{
'date': '2019-01-01',
'amount': 50.50
},
{
'date': '2019-01-02',
'amount': 10.20
},
]
},
{
'id':2,
'transactions': [
{
'date': '2019-01-01',
'amount': 10.20
},
{
'date': '2019-01-02',
'amount': 0.50
},
]
}
]
I load these files to Spark using the following code
users= spark.read.option("multiline", "true").json(file_location)
The result is a SparkData Frame with two columns id and transactions where transactions is a StructType.
I want to be able to "map" the transactions per user to aggregate them.
Currently I am using rdd and a function that looks like this:
users.rdd.map(lambda a: summarize_transactions(a.transactions))
The summarize function can be of two types:
a) Turn the list of objects into a Pandas Dataframe to summarize it.
b) Iterate over the list of objects to summarize it.
However I find out that a.transactions is a list of pyspark.sql.types.Row. Instead of actual dictionaries.
1) Is this the best way to accomplish my goal?
2) How can I turn the list of Spark Rows into the original list of Dictionaries?

Answer 1

I found a way to solve my own problem:
STEP 1: LOAD DATA AS TEXTFILE:
step1= sc.textFile(file_location)
STEP 2: READ AS JSON AND FLATMAP
import json
step2 = step1.map(lambda a: json.loads(a)).flatMap(lambda a: a)
STEP 3: KEY MAP REDUCE
setp3 = (
step2
.map(lambda line: [line['id'], line['transactions']])
.reduceByKey(lambda a, b: a + b)
.mapValues(lambda a: summarize_transactions(a))
)