Transform JSON Dataframe/RDD schema in Spark - json

Question

I am seeking help in transforming a JSON dataframe/Rdd from one schema structure to another. I am looking at more than 100,000 rows of json data being read and would like to transform that data before inserting into a document store. I am performing the transformations in Spark using scala. Currently i am using json4s framework to parse one json row at a time and transforming it before inserting into the document store, which is not leveraging the power of Spark processing. I would like to transform all the data together, which I believe will speed up the processing.
Following is a code snippet to illustrate my problem.
My input data looks like the inpString string, with multiple rows of JSON.
val inpString= """[{
"data": {
"id": "1234",
"name": "abc",
"time": "2015-01-01 13-44-21",
"x": [
50,
10
],
"y": [
100,
20
],
"z": [
150,
30
],
"x_limit": [
70,
90,
15,
20
],
"y_limit": [
70,
90,
15,
20
],
"z_limit": [
70,
90,
15,
20
]
}},
{
"data": {
"id": "1235",
"name": "cde",
"time": "2015-01-01 3-21-01",
"x": [
50,
10
],
"y": [
100,
20
],
"z": [
150,
30
],
"x_limit": [
70,
90,
15,
20
],
"y_limit": [
70,
90,
15,
20
],
"z_limit": [
70,
90,
15,
20
]
}}]"""
I read it into a dataframe and am able to select, groupby and do all other operations using Sparksql.
val inputRDD = sc.parallelize(inpString::Nil)
val inputDf = sqlContext.read.json(inputRDD)
inputDf.printSchema()
Schema looks like below
root
|-- data: struct (nullable = true)
| |-- id: string (nullable = true)
| |-- name: string (nullable = true)
| |-- time: string (nullable = true)
| |-- x: array (nullable = true)
| | |-- element: long (containsNull = true)
| |-- x_limit: array (nullable = true)
| | |-- element: long (containsNull = true)
| |-- y: array (nullable = true)
| | |-- element: long (containsNull = true)
| |-- y_limit: array (nullable = true)
| | |-- element: long (containsNull = true)
| |-- z: array (nullable = true)
| | |-- element: long (containsNull = true)
| |-- z_limit: array (nullable = true)
| | |-- element: long (containsNull = true)
The x array in my input data has two types of readings(say type1 and type2) and the x_limit array has the lower and upper limits for the two types of readings in x array. x_limit[0] is the lower limit for type1, x_limit[1] is the upper limit for type1, x_limit[2] is the lower limit for type2 and x_limit[3] is the upper limit for type2. I need to group all the data for type1 together in one struct and all the data for type2 in one struct.
Following code snippet will give us the output schema
val outString= """[{
"data": {
"id": "1234",
"name": "abc",
"time": "2015-01-01 13-44-21",
"type1": {
"x_axis" : 50,
"y_axis" : 100,
"z_axis" : 150,
"x_lower_limit" : 70,
"x_upper_limit" : 90,
"y_lower_limit" : 70,
"y_upper_limit" : 90,
"z_lower_limit" : 70,
"z_upper_limit" : 90
},
"type2": {
"x_axis" : 10,
"y_axis" : 20,
"z_axis" : 30,
"x_lower_limit" : 15,
"x_upper_limit" : 20,
"y_lower_limit" : 15,
"y_upper_limit" : 20,
"z_lower_limit" : 15,
"z_upper_limit" : 20
}
}
},
{
"data": {
"id": "1235",
"name": "cde",
"time": "2015-01-01 3-21-01",
"type1": {
"x_axis" : 50,
"y_axis" : 100,
"z_axis" : 150,
"x_lower_limit" : 70,
"x_upper_limit" : 90,
"y_lower_limit" : 70,
"y_upper_limit" : 90,
"z_lower_limit" : 70,
"z_upper_limit" : 90
},
"type2": {
"x_axis" : 10,
"y_axis" : 20,
"z_axis" : 30,
"x_lower_limit" : 15,
"x_upper_limit" : 20,
"y_lower_limit" : 15,
"y_upper_limit" : 20,
"z_lower_limit" : 15,
"z_upper_limit" : 20
}
}
}
]"""
val outputRDD = sc.parallelize(outString::Nil)
val outputDf = sqlContext.read.json(outputRDD)
outputDf.printSchema()
Output Schema
root
|-- data: struct (nullable = true)
| |-- id: string (nullable = true)
| |-- name: string (nullable = true)
| |-- time: string (nullable = true)
| |-- type1: struct (nullable = true)
| | |-- x_axis: long (nullable = true)
| | |-- x_lower_limit: long (nullable = true)
| | |-- x_upper_limit: long (nullable = true)
| | |-- y_axis: long (nullable = true)
| | |-- y_lower_limit: long (nullable = true)
| | |-- y_upper_limit: long (nullable = true)
| | |-- z_axis: long (nullable = true)
| | |-- z_lower_limit: long (nullable = true)
| | |-- z_upper_limit: long (nullable = true)
| |-- type2: struct (nullable = true)
| | |-- x_axis: long (nullable = true)
| | |-- x_lower_limit: long (nullable = true)
| | |-- x_upper_limit: long (nullable = true)
| | |-- y_axis: long (nullable = true)
| | |-- y_lower_limit: long (nullable = true)
| | |-- y_upper_limit: long (nullable = true)
| | |-- z_axis: long (nullable = true)
| | |-- z_lower_limit: long (nullable = true)
| | |-- z_upper_limit: long (nullable = true)
I did some research to find for a similar scenario but could not. I appreciate your inputs.