Create a JSON by joining dataframes (parent & child) - json

Question

I would like to create a json out of 2 dataframes (one is Parent and the other is child). Child records should be an array forming a nested JSON
Df1 (Department):
+----------+------------+
| dept_Id | dept_name |
+----------+------------+
| 10 | Sales |
+----------+------------+
Df2 (Employee):
+----------+--------+----------+
| dept_Id | emp_id | emp_name |
+----------+--------+----------+
| 10 | 1001 | John |
| 10 | 1002 | Rich |
+----------+--------+----------+
I want the JSON to be created as follows:
{
"dept_id":"10",
"dept_name":"Sales",
"employee":[
{ "emp_id":"1001","emp_name":"John" },
{ "emp_id":"1002","emp_name":"Rich" }
]
}
Appreciate your thoughts. Thanks

Answer 1

First join the two dataframes together:
val df = df1.join(df2, Seq("dept_Id"))
Then use groupBy and collect_list. Two case classes is used here to get the correct names in the final json. These should be placed outside of the main method.
case class Department(dept_Id: Int, dept_name: String, employee: Seq[Employee])
case class Employee(emp_id: Int, emp_name: String)
val dfDept = df.groupBy("dept_id", "dept_name")
.agg(collect_list(struct($"emp_id", $"emp_name")).as("employee"))
.as[Department]
Resulting dataframe:
+-------+---------+--------------------------+
|dept_id|dept_name|employee |
+-------+---------+--------------------------+
|10 |Sales |[[1002,Rich], [1001,John]]|
+-------+---------+--------------------------+
Finally, save it as a json file:
dfDept .coalesce(1).write.json("department.json")