Pyspark adding additional double quotes in array at saving time - json

Question

I have a small cluster setup of Spark 3.x. I have read some data and after transformations, I have to save it as JSON. But the problem I am facing is that, in array type of columns, Spark is adding extra double quotes when written as json file.
Sample data-frame data
I am saving this data frame as JSON with following command
df.write.json("Documents/abc")
The saved output is as follows
Finally, the schema info is as follows

Answer 1

The elements of the string array contain double quotes within the data, e.g. the first element is "Saddar Cantt, Lahore Punjab Pakistan" instead of Saddar Cantt, Lahore Punjab Pakistan. You can remove the extra double quotes from the strings before writing the json with transform and replace:
df.withColumn("ADDRESS", F.expr("""transform(ADDRESS, a -> replace(a, '"'))""")) \
.write.json("Documents/abc")

Answer 2

If we enforce schema before writing the dataframe as json, I believe we can work around such issues without having to replace/change any characters.
Without schema ->
df.show()
id | address
1 | ['Saddar Cantt, Lahore Punjab Pakistan', 'Shahpur']
df.write.json("path")
{"id":"1","address":["'Saddar Cantt, Lahore Punjab Pakistan'","'Shahpur'"]}
With Schema ->
df = df.withColumn('address', F.col('address').cast(StringType()))
df = df.withColumn('address', F.from_json(F.col('address'), ArrayType(StringType())))
df.write.json("path")
{"id":"1","address":["Saddar Cantt, Lahore Punjab Pakistan","Shahpur"]}
from_json only takes string as input. Hence we need to first cast array to string.