How to bring json format to relational form? - json

Question

this is my code:
%spark.pyspark
df_principalBody = spark.sql("""
SELECT
gtin
, principalBodyConstituents
--, principalBodyConstituents.coatings.materialType.value
FROM
v_df_source""")
df_principalBody.createOrReplaceTempView("v_df_principalBody")
df_principalBody.collect();
And this is the output:
[Row(gtin='7617014161936', principalBodyConstituents=[Row(coatings=[Row(materialType=Row(value='003', valueRange='405')
How can I read the value and valueRange fields in relational format?
I tried with explode and flatten, but it will not work.
Part of my json:
{
"gtin": "7617014161936",
"timePeriods": [
{
"fractionData": {
"principalBody": {
"constituents": [
{
"coatings": [
{
"materialType": {
"value": "003",
"valueRange": "405"
},
"percentage": 0.1
}
],
...

Answer 1

You can use data_dict.items() to list key/value pairs.
I used part of your json as below -
str1 = """{"gtin": "7617014161936","timePeriods": [{"fractionData": {"principalBody": {"constituents": [{"coatings": [
{
"materialType": {
"value": "003",
"valueRange": "405"
},
"percentage": 0.1
}
]}]}}}]}"""
import json
res = json.loads(str1)
res_dict = res['timePeriods'][0]['fractionData']['principalBody']['constituents'][0]['coatings'][0]['materialType']
df = spark.createDataFrame(data=res_dict.items())
Output :
+----------+---+
| _1| _2|
+----------+---+
| value|003|
|valueRange|405|
+----------+---+
You can even specify your schema:
from pyspark.sql.types import *
df = spark.createDataFrame(res_dict.items(),
schema=StructType(fields=[
StructField("key", StringType()),
StructField("value", StringType())])).show()
Resulting in
+----------+-----+
| key|value|
+----------+-----+
| value| 003|
|valueRange| 405|
+----------+-----+