How to load csv with nested columns using Apache Spark - csv

Question

I have a csv file:
name,age,phonenumbers
Tom,20,"[{number:100200, area_code:555},{number:100300, area_code:444}]"
Harry,20,"[{number:100400, area_code:555},{number:100500, area_code:666}]"
How can I load this file in Spark to a RDD/Dataset of Person where Person object looks like:
class Person {
String name;
Integer age;
List<Phone> phonenumbers;
class Phone {
int number;
int area_code;
}
}

Answer 1

Unfortunately, the column names for the nested object don't have quotes in your example. Is that truly the case? Because if they DO have quotes (e.g. well-formed JSON) then you could very easily use the from_json function as below:
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
val schema = new ArrayType(new StructType()
.add("number", IntegerType)
.add("area_code", IntegerType), false)
val converted = input.withColumn("phones", from_json('phonenumbers, schema))
If that's not the case, then you'll need to use your own logic to convert the string into the actual nested object, such as:
import org.apache.spark.sql.functions._
case class Phone(number: Int, area_code:Int)
case class Person(name: String, age: Int, phonenumbers: Array[Phone])
val converted = input.map {
case Row(name: String, age: Int, phonenumbers: String) => {
import scala.util.matching.Regex
val phoneFormat = raw"\{number:(\d{6}), area_code:(\d{3})\}".r
val phones = for (m <- phoneFormat.findAllMatchIn(phonenumbers)) yield Phone(m.group(1).toInt, m.group(2).toInt)
Person(name, age, phones.toArray)
}
}