Spark Scala creating schema based on target JSON structure - json

Question

I'm hopelessly stuck trying to generate my Spark Schema based on the JSON structure I know I want.
I have a JSON structure that looks like this:
{
"key1": "value1",
"key2": "value2",
"key3": "value3",
"key4": "value4",
"key5": [
"key6": "value6",
"key7": [
"key8": "value8",
"key8": "value9"
]
]
}
I tried to recreate the structure by creating the following Schema in Spark 2.4.8, running in Scala:
val targetSchemaSO = StructType(
List(
StructField("key1", StringType, true),
StructField("key2", StringType, true),
StructField("key3", StringType, true),
StructField("key4", StringType, true),
StructField("key5", StructType(
List(
StructField("key6", StringType, true),
StructField("key7", ArrayType(StructType(
List(
StructField("key8", StringType, true)
))), true)
)), true)
)
)
However, when trying to format each row as a Spark Row using this code:
val outputDictSO = scala.collection.mutable.LinkedHashMap[String, Any](
"key1" -> "value1",
"key2" -> "value2",
"key3" -> "value3",
"key4" -> "value4",
"key5" -> (
"key6" -> "value6",
"key7" -> (
"key8" -> "value8",
"key8" -> "value9"
)
)
)
return Row.fromSeq(output_dict.values.toSeq)
I get the following error when mapping it to the provided Schema:
Error while encoding: java.lang.RuntimeException: scala.Tuple2 is not a valid external
type for schema of struct<key6:string,key7:array<struct<key8:string>>>
The program I'm basing this off of uses this exact Schema in PySpark and the DataFrame is created just fine; do the StructTypes work differently between PySpark and Spark Scala? What would be the correct Schema to make so that nested arrays in the Schema are possible?

Answer 1

You can use from_json function ,pass the json string column as input and printSchema to retrieve valid schema. (Use one set of json string from the data and create dataframe if needed)
val df1 = df.withColumn("json",from_json(json_string_column))
df1.printSchema()

Answer 2

There is no difference in creating the DataFrame schema in both languages. If you want to create a dataframe with a specified schema, the correct way to build the Rows, according to that provided schema, could be something like:
val outputDictSO =
Row("value1", "value2", "value3", "value4",
Row("value6", Array(
Row("value8"))))
val df0 =
spark.createDataFrame(
spark.sparkContext.parallelize(Seq(outputDictSO)), targetSchemaSO)
df0.printSchema()
The dataframe´s schema is as expected:
root
|-- key1: string (nullable = true)
|-- key2: string (nullable = true)
|-- key3: string (nullable = true)
|-- key4: string (nullable = true)
|-- key5: struct (nullable = true)
| |-- key6: string (nullable = true)
| |-- key7: array (nullable = true)
| | |-- element: struct (containsNull = true)
| | | |-- key8: string (nullable = true)
If you see the content of the dataframe:
df0.show()
Gives:
+------+------+------+------+--------------------+
| key1| key2| key3| key4| key5|
+------+------+------+------+--------------------+
|value1|value2|value3|value4|[value6, [[value8]]]|
+------+------+------+------+--------------------+
You can select for a nested key:
df0.select("key5.key6").show()
Spark returns:
+------+
| key6|
+------+
|value6|
+------+
The error is because the specified schema and the data do not match. If you take your data:
val outputDictSOMap = Map(
"key1" -> "value1",
"key2" -> "value2",
"key3" -> "value3",
"key4" -> "value4",
"key5" -> (
"key6" -> "value6",
"key7" -> (
"key8" -> "value8",
"key9" -> "value9"
)
))
And convert it to json:
import org.json4s.jackson.Serialization
implicit val formats = org.json4s.DefaultFormats
val json = Serialization.write(outputDictSOMap)
val df1 = spark.read.json(Seq(json).toDS)
df1.printSchema()
The resulting schema is:
root
|-- key1: string (nullable = true)
|-- key2: string (nullable = true)
|-- key3: string (nullable = true)
|-- key4: string (nullable = true)
|-- key5: struct (nullable = true)
| |-- _1: struct (nullable = true)
| | |-- key6: string (nullable = true)
| |-- _2: struct (nullable = true)
| | |-- key7: struct (nullable = true)
| | | |-- _1: struct (nullable = true)
| | | | |-- key8: string (nullable = true)
| | | |-- _2: struct (nullable = true)
| | | | |-- key9: string (nullable = true)
And that is the reason of your error.