Spark importing split JSON payload - json

Question

I have a json structure which contains some top-level metadata and a payload which is equivalent to what pandas would define as a split orientation json payload. This was done to reduce duplication as we ingest a lot of these files.
Usually in pandas I would load the json file and pass the required parts (index, column names and data) to a DataFrame constructor, which would give me a flat table which is easy to work with and can then be exported to influxdb or sql.
obj = load_json('file.json')
df = pd.DataFrame(index=obj['payload']['Time'], columns=obj['Names']['Time'], data=obj['payload']['Data'])
df['Machine_ID'] = obj['Machine_ID']
df['TimeSend'] = obj['TimeSend']
df['Version'] = obj['Version']
It seems that this schema is not easy to flatten with spark, as the data isn't record based and so the column names and data aren't associated. Is there anyway I can process this to flat schema with spark or should I add an extra pandas processing step in my pipeline.
root
|-- Machine_ID: string (nullable = true)
|-- TimeSend: string (nullable = true)
|-- Version: long (nullable = true)
|-- Payload: struct (nullable = true)
| |-- Data: array (nullable = true)
| | |-- element: array (containsNull = true)
| | | |-- element: double (containsNull = true)
| |-- Names: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- Time: array (nullable = true)
| | |-- element: string (containsNull = true)
Edit: I found a way that works, however I'm curious whether the ordering can be relied on, because I split the dataframe before adding id's.
Probably its better to zip Time and Data to enable them both to be exploded, and work from there.
# Make flattened dataframe
df_ = df.select(col('Payload.Time').alias('Time'), col('Payload.Names').alias('Names'), col('Payload.Data').alias('Data'), col('Machine_ID'), col('TimeSend'), col('Version'))
# Make exploded `Data` table
columns = df_.rdd.flatMap(lambda x: x.Names).collect()
df_a = df_.select(explode(col('Data')))
df_a = df_a.select([df_a.col[x] for x in range(len(columns))])
df_a = rdd_data.toDF(columns)
df_a = df_a.withColumn("id", monotonically_increasing_id())
# Make exploded `Metadata` table
df_b = df_.select(explode(col('Time')), col('Machine_ID'), col('TimeSend'), col('Version'))
df_b = df_b.withColumn("id", monotonically_increasing_id())
# Join tables
df_c = df_a.join(df_b, "id")
# Schema is now flattened and joined
df_c.printSchema()
root
|-- id: long (nullable = false)
|-- Machine_ID: string (nullable = true)
|-- TimeSend: string (nullable = true)
|-- Version: long (nullable = true)
|-- Index: string (nullable = true) <- From Payload.Time
|-- TagA: double (nullable = true) <- From Payload.Names & Data
|-- TagB: double (nullable = true) <- || -
|-- TagC: double (nullable = true) <- || -
|-- TagD: double (nullable = true) <- || -