Mulitline headline for csv in Spark - csv

Question

I need Spark to write a compressed csv file to HDFS, but I need it to start with a few lines of version information.
Example of file content
version=2
date=2020-01-31
id,name,age
1,Alice,21
2,Bob,23
Three ideas of ways to do this
First write out to hdfs://data/tmp/file1.csv.gz, then use hadoop fs -cat
to stream it to hdfs://data/real/file1.csv.gz
Convert output
datafram to text format / RDD[String] and union real file with the
extra header lines
Change the first column name to multi line
So for approach 3:
column1 ="version=2\ndate=2020-01-31\n\nid"
Let me know if you know a more elegant way to do this.

Answer 1

I tried all the approaches. Here is simplified code:
Approach 1
Approach 1 is using Hadoop commands from a bash script etc.
This works, but requires a double HDFS write and a cleanup. Also it doesn't fit so well in a Scala Spark project.
(echo -e "version=2\ndate=2020-01-31\n\nid,name,age" | gzip -vc ; hadoop fs -cat "$INPUT_DIR/*" ) | hadoop fs -put - "$OUTPUT_PATH"
What is happening here is that it will
echo multi line header out to stdout
pipe that into gzip and to stdout
pipe other HDSF dir to stdout
pipe into hadoop fs -put that will combine everything
Approach 2
Code is a little more complicated there are not bad quote characters in headline, but headlines sometimes come after csv part.
import org.apache.hadoop.io.compress.GzipCodec
val heading = """version=2
date=2020-01-31
id,name,age""".split("\n", -1).toSeq
val headingRdd: RDD[String] = sc.parallelize(heading)
val mediamathRdd: RDD[String] = df.rdd.map(row => row.mkString(","))
val combinedResult: RDD[String] = (headingRdd union mediamathRdd)
combinedResult.repartition(1).saveAsTextFile(path, classOf[GzipCodec])
Approach 3
The simplest approach, but the output is slightly off
df.repartition(1)
.withColumnRenamed("id", "version=2\ndate=2020-01-31\n\nid")
.option("header", true)
.option("delimiter", ",")
.option("quoteMode", "NONE")
.option("quote", " ")
.option("codec", "gzip")
.csv(path)
Result will look like this, which might or might not be acceptable
version=2
date=2020-01-31
id ,name,age
1,Alice,21
2,Bob,23
I also tried with:
.option("quote", "\u0000")
It actually prints the ascii charter zero and while this did not show up in my HDFS viewer this was not part of the spec.
Best Approach
None of them are perfect for what seems like a very simple task. Maybe there is a small fix to make approach 2 work perfectly.