Exclude column header when writing DataFrame to json - json

Question

I have the following dataframe df1
SomeJson
=================
[{
"Number": "1234",
"Color": "blue",
"size": "Medium"
}, {
"Number": "2222",
"Color": "red",
"size": "Small"
}
]
and I am trying to write just the contents of this column to blob storage as a json.
df1.select("SomeJson")
.write
.option("header", false)
.mode("append")
.json(blobStorageOutput)
This Code works but it creates the following json in blob storage.
{
"SomeJson": [{
"Number": "1234",
"Color": "blue",
"size": "Medium"
}, {
"Number": "2222",
"Color": "red",
"size": "Small"
}
]
}
But I just want the contents of the column not the column Header as well, I dont want the "SomeJson" in my final Json. Any Suggestions?

Answer 1

If you don't want dataframe column to get appended, write your dataframe as text and not as json. It will only write the content of your column.
df1.select("SomeJson")
.write
.option("header", false)
.mode("append")
.text(blobStorageOutput)

Answer 2

Just an additional assumption to this question,
We derive JSON structure itself from the dataset and then we encounter this header scenario like here. We can follow the below approach.
spark.sql("SELECT COLLECT_SET(STRUCT(<field_name>)) AS `` FROM <table_name> LIMIT 1").coalesce(1).write.format("org.apache.spark.sql.json").mode("overwrite").save(<Blob Path1/ ADLS Path1>)
Output will be like,
{"":[{<field_name>:<field_value>}]}
Here the header can be avoided by following 3 lines (Assumption No Tilda in data),
jsonToCsvDF=spark.read.format("com.databricks.spark.csv").option("delimiter", "~").load(<Blob Path1/ ADLS Path1>)
jsonToCsvDF.createOrReplaceTempView("json_to_csv")
spark.sql("SELECT SUBSTR(`_c0`,5,length(`_c0`)-5) FROM json_to_csv").coalesce(1).write.option("header",false).mode("overwrite").text(<Blob Path2/ ADLS Path2>)