Parse Nested JSON in Spark - json

Question

I have a JSON file whose schema is like this--
root
|-- errorcode: string (nullable = true)
|-- errormessage: string (nullable = true)
|-- ip: string (nullable = true)
|-- label: string (nullable = true)
|-- status: string (nullable = true)
|-- storageidlist: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- errorcode: string (nullable = true)
| | |-- errormessage: string (nullable = true)
| | |-- fedirectorList: array (nullable = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- directorId: string (nullable = true)
| | | | |-- errorcode: string (nullable = true)
| | | | |-- errordesc: string (nullable = true)
| | | | |-- metrics: string (nullable = true)
| | | | |-- portMetricDataList: array (nullable = true)
| | | | | |-- element: array (containsNull = true)
| | | | | | |-- element: struct (containsNull = true)
| | | | | | | |-- data: array (nullable = true)
| | | | | | | | |-- element: struct (containsNull = true)
| | | | | | | | | |-- ts: string (nullable = true)
| | | | | | | | | |-- value: string (nullable = true)
| | | | | | | |-- errorcode: string (nullable = true)
| | | | | | | |-- errordesc: string (nullable = true)
| | | | | | | |-- metricid: string (nullable = true)
| | | | | | | |-- portid: string (nullable = true)
| | | | | | | |-- status: string (nullable = true)
| | | | |-- status: string (nullable = true)
| | |-- metrics: string (nullable = true)
| | |-- status: string (nullable = true)
| | |-- storageGroupList: string (nullable = true)
| | |-- storageid: string (nullable = true)
|-- sublabel: string (nullable = true)
|-- ts: string (nullable = true)
I am supposed to extract ip,storageid,directorid,metricid,value and ts. In the storageidlist, there is just 1 item, but in the fedirectorList, there are 56 items. But I am unable to parse the JSON beyond storageidlist.
scala> val ip_df = spark.read.option("multiline",true).json("FEDirector_port_data.txt")
ip_df: org.apache.spark.sql.DataFrame = [errorcode: string, errormessage: string ... 6 more fields]
scala> ip_df.select($"storageidlist.storageid").show()
+--------------+
| storageid|
+--------------+
|[000295700670]|
+--------------+
scala> ip_df.select($"storageidlist.fedirectorList.directorId").show()
org.apache.spark.sql.AnalysisException: cannot resolve '`storageidlist`.`fedirectorList`['directorId']' due to data type mismatch: argument 2 requires integral type, however, ''directorId'' is of string type.;;

Answer 1

storageidlist is an array column, so you need to select the first array element and do further selections from that:
ip_df.selectExpr("storageidlist[0].fedirectorList.directorId")
or
ip_df.select($"storageidlist"(0).getField("fedirectorList").getField("directorId"))
It's better to specify an array index whenever you work with array type columns. If you don't specify an array index, you can go 1 level deeper and fetch all the corresponding struct elements in the next level, but you can't go further, as shown in your question.

Answer 2

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
file = "<s3path>/<json_file_name.json>"
schema_path = "<s3path>/<json_schame_name.json>"
json_schema = spark.read.json(schema_path, multiLine=True)
df = sqlContext.read.json(file,json_schema.json_schema,multiLine=True)
#display(df)
df.createOrReplaceTempView("temptable")
#example UDF
def parse_nested_list(nested_list):
parsed_str = []
if nested_list:
for item_list in nested_list:
if item_list:
for item in item_list:
if item:
parsed_str.append(item)
return "|".join(parsed_str)
def parse_arrs(x):
if x:
return "| ".join(
", ".join(i for i in e if i is not None) for e in x if e is not None
)
else:
""
sqlContext.udf.register("parse_nested_list", parse_nested_list)
sqlContext.udf.register("parse_arrs", parse_arrs)
structured_df =sqlContext.sql("select parse_nested_list(column1.column2) as column3, parse_arrs(column1) as column2 from temptable")
display(structured_df)
To fetch the nested array, list, dictionaries. You will have to write a UDF to get the nested values and register it to pyspark so that you can use them in sparksql coding.