convert a Nested Json to a dataframe in Pyspark - json

Question

I'm trying to create a dataframe from a json with nested feilds and dates feilds that i'd like to concatenate :
root
|-- MODEL: string (nullable = true)
|-- CODE: string (nullable = true)
|-- START_Time: struct (nullable = true)
| |-- day: string (nullable = true)
| |-- hour: string (nullable = true)
| |-- minute: string (nullable = true)
| |-- month: string (nullable = true)
| |-- second: string (nullable = true)
| |-- year: string (nullable = true)
|-- WEIGHT: string (nullable = true)
|-- REGISTED: struct (nullable = true)
| |-- day: string (nullable = true)
| |-- hour: string (nullable = true)
| |-- minute: string (nullable = true)
| |-- month: string (nullable = true)
| |-- second: string (nullable = true)
| |-- year: string (nullable = true)
|-- TOTAL: string (nullable = true)
|-- SCHEDULED: struct (nullable = true)
| |-- day: long (nullable = true)
| |-- hour: long (nullable = true)
| |-- minute: long (nullable = true)
| |-- month: long (nullable = true)
| |-- second: long (nullable = true)
| |-- year: long (nullable = true)
|-- PACKAGE: string (nullable = true)
objective is to get a result more like :
+---------+------------------+----------+-----------------+---------+-----------------+
|MODEL | START_Time | WEIGHT |REGISTED |TOTAL |SCHEDULED |
+---------+------------------+----------+-----------------+---------+-----------------+
|.........| yy-mm-dd-hh-mm-ss| WEIGHT |yy-mm-dd-hh-mm-ss|TOTAL |yy-mm-dd-hh-mm-ss|
where yy-mm-dd-hh-mm-ss are the conactenation of: day, hour, minute.... in the json
|-- example: struct (nullable = true)
| |-- day: string (nullable = true)
| |-- hour: string (nullable = true)
| |-- minute: string (nullable = true)
| |-- month: string (nullable = true)
| |-- second: string (nullable = true)
| |-- year: string (nullable = true)
i have tried explode function may be didn't use it as it should but didn't work
can anyone inspire me for a solution
Thank you

Answer 1

You can do it in below simple steps.
Lets we have the data as below in the data.json file
{"MODEL": "abc", "CODE": "CODE1", "START_Time": {"day": "05", "hour": "08", "minute": "30", "month": "08", "second": "30", "year": "21"}, "WEIGHT": "231", "REGISTED": {"day": "05", "hour": "08", "minute": "30", "month": "08", "second": "30", "year": "21"}, "TOTAL": "1", "SCHEDULED": {"day": "05", "hour": "08", "minute": "30", "month": "08", "second": "30", "year": "21"},"PACKAGE": "CAR"}
This data has the same schema as you shared.
Read this json file in pyspark as below.
from pyspark.sql.functions import *
df = spark.read.json('data.json')
Now you can read the nested values and modify the column values as below.
df.withColumn('START_Time', concat(col('START_Time.year'), lit('-'), col('START_Time.month'), lit('-'), col('START_Time.day'), lit('-'), col('START_Time.hour'), lit('-'), col('START_Time.minute'), lit('-'), col('START_Time.second'))).withColumn('REGISTED',concat(col('REGISTED.year'), lit('-'), col('REGISTED.month'), lit('-'), col('REGISTED.day'), lit('-'), col('REGISTED.hour'), lit('-'), col('REGISTED.minute'), lit('-'), col('REGISTED.second'))).withColumn('SCHEDULED',concat(col('SCHEDULED.year'), lit('-'), col('SCHEDULED.month'), lit('-'), col('SCHEDULED.day'), lit('-'), col('SCHEDULED.hour'), lit('-'), col('SCHEDULED.minute'), lit('-'), col('SCHEDULED.second'))).show()
The output would be
CODE
MODEL
PACKAGE
REGISTED
SCHEDULED
START_Time
TOTAL
WEIGHT
CODE1
abc
CAR
21-08-05-08-30-30
21-08-05-08-30-30
21-08-05-08-30-30
1
231