How to write single CSV file using pyspark in Databricks - csv

Question

Good morning all!!
Yesterday I was looking for a function that was able to write a single file CSV with pyspark in Azure Databricks but I did not find anything. So I've built my own function and I wanted to share my solution with the community and if it's possible create like a thread with different solutions for the same problem.
Sorry, because I commented the code in Spanish but basically the function does:
Save the dataframe you've created into a new directory (which is allocated inside the path you've defined and begin with 'temp_') and save the partition there using "coalesce(1)"
Rename the CSV file as you want and moves it to the desired path
Delete de temporary file
That's all! You have your unique CSV file
def escribe_fichero_unico(dataframe, path, file_name, file_format = 'csv'):
"""
Definición: (1) Genera carpeta temporal para guardar particiones que Spark genera por defecto
a la hora de guardar archivos, (2) Une todas las particiones en un único archivo, (3) Mueve este
archivo al directorio anterior y (4) Borra la carpeta temporal
Parámetros:
dataframe: dataframe que quieras guardar como fichero único
file_name: en formato string escribe nombre del archivo
file_format: en formato string escribe 'csv' o 'parquet'
path: en formato string escribe el path donde quieres guardar el csv
"""
import os
# 1) Guardamos el dataframe creando una carpeta temporal que guarda todas las particiones
path_temp = path + 'temp_' + file_name + '_trash'
if file_format == 'csv':
dataframe.coalesce(1).write.format('csv').mode('overwrite') \
.options(header="true", schema="true", delimiter=";") \
.save(path_temp)
else if file_format == 'parquet':
dataframe.coalesce(1).write.format("parquet").mode("overwrite") \
.save(path_temp)
# 2)Une todas las particiones en un único archivo
file_part = [file.path for file in dbutils.fs.ls(path_temp) if os.path.basename(file.path).startswith('part')][0]
# 3) Mueve este archivo al directorio anterior
dbutils.fs.mv(file_part, path + file_name + '.' + file_format)
# 4) Borra la carpeta temporal
dbutils.fs.rm(path_temp, True)
I hope this work for you as well :)