How to convert WrappedArray to string in spark? - json

Question

I have a json file which contain nested array as like below,
| | |-- coordinates: array (nullable = true)
| | | |-- element: array (containsNull = true)
| | | | |-- element: array (containsNull = true)
| | | | | |-- element: array (containsNull = true)
| | | | | | |-- element: long (containsNull = true)
I have used Spark to read json and exploded the array.
explode(col("list_of_features.geometry.coordinates"))
which returns values as below,
WrappedArray(WrappedArray(WrappedArray(1271700, 6404100), WrappedArray(1271700, 6404200), WrappedArray(1271600, 6404200), WrappedArray(1271600, 6404300),....
But the original input looks like without WrappedArray.
something like,
[[[[1271700,6404100],[1271700, 6404200],[1271600, 6404200]
The ultimate aim is to store the coordinates without WrappedArray (may be as String) in csv file for Hive to read the data.
After explode is there any way to just the coordinates enclosed with proper square brackets.
Or can I use replace to replace the WrappedArray string value in RDD?

Answer 1

You can use UDF to flatten the WrappedArray and make it String value as
//udf
val concatArray = udf((value: Seq[Seq[Seq[Seq[Long]]]]) => {
value.flatten.flatten.flatten.mkString(",")
})
Now use udf to create/replace the column as
df1.withColumn("coordinates", concatArray($"coordinates") )
This should give you a string separated with "," replacing the WrappedArray
UPDATE: If you wan in the same format as string with brackets then you can do as
val concatArray = udf((value: Seq[Seq[Seq[Seq[Long]]]]) => {
value.map(_.map(_.map(_.mkString("[", ",", "]")).mkString("[", "", "]")).mkString("[", "", "]"))
})
Output:
[[[[1271700,6404100][1271700,6404200][1271600,6404200]]]]
Hope this helps!