F# merge CSV files with different columns - csv

Question

I'm fairly new to F# but I'm fascinated about it and want to apply it to some applications. Currently, I have multiple csv files which is just timestamp and some sensor's values, the timestamp is unique but the columns values' are different.
For example I have two csv file
csv1:
timestamp, sensor1
time1, 1.0
csv2:
timestamp, sensor1, sensor2
time2, 2.0, 3.0
The result I want is
timestamp, sensor1, sensor2
time1, 1.0,
time2, 2.0, 3.0
I wonder if any easy way to do it in F#. Thanks
UPDATE 1:
Here my current solution which involves using LumenWorks.Framework.IO.Csv (https://www.nuget.org/packages/LumenWorksCsvReader) to parse csv to Data.DataTable and Deedle (https://www.nuget.org/packages/Deedle) to convert Data.DataTable to Frame and use the SaveCsv method to save to csv files.
open System.IO
open System
open LumenWorks.Framework.IO.Csv
open Deedle
// get list of csv files
let filelist = expression_to_get_list_of_csv_file_path
// func to readCsv from path and return Data.DataTable
let funcReadCSVtoDataTable (path:string) =
use csv = new CachedCsvReader(new StreamReader(path), true)
let tmpdata = new Data.DataTable()
tmpdata.Load(csv)
tmpdata
// map list of file paths to get list of datatable
let allTables = List.map funcReadCSVtoDataTable filelist
// create allData table to iterate over the list
let allData = new Data.DataTable()
List.iter (fun (x:Data.DataTable) -> allData.Merge(x)) allTables
//convert datatable to Deedle Frame and save to csv file
let df = Frame.ReadReader (allData.CreateDataReader())
df.SaveCsv("./final_csv.csv")
The reason for using LumenWorks.Framework.IO.Csv is because I need to parse a few thousands of files at the same time, and according to this article (https://www.codeproject.com/Articles/11698/A-Portable-and-Efficient-Generic-Parser-for-Flat-F) LumenWorks.Framework.IO.Csv is the fastest.
UPDATE 2: FINAL SOLUTION
Thanks to Tomas about the RowsKey map solution (see his comment below), I re-twisted his code for the case of list of files
// get list of csv files
let filelist = expression_to_get_list_of_csv_file_path
// function to merge two Frames
let domerge (df0:Frame<int,string>) (df1:Frame<int,string>) =
df1
|> Frame.mapRowKeys (fun k-> k+df0.Rows.KeyCount)
|> Frame.merge df0
// read filelist to Frame list
let dflist = filelist |> List.map (fun (x:string)-> Frame.ReadCsv x)
// using List.fold to "fold" through the list with dflist.[0] is the intial state
let dffinal = List.tail dflist |> List.fold domerge (List.head dflist)
dffinal.SaveCsv("./final_csv.csv")
Now the code looks "functional", however, I get a small warning of Frame.ReadCsv that the method is not meant for F#, but it works anyway.

Answer 1

If you are happy to use an external library, then you can do this very easily using the data frame manipulation library called Deedle. Deedle lets you read data frames from CSV files and when you merge data frames, it makes sure to align column and row keys for you:
open Deedle
let f1 = Frame.ReadCsv("c:/temp/f1.csv")
let f2 = Frame.ReadCsv("c:/temp/f2.csv")
let merged =
f2
|> Frame.mapRowKeys (fun k -> k + f1.Rows.KeyCount)
|> Frame.merge f1
merged.SaveCsv("c:/temp/merged.csv")
The one tricky thing that we have to do here is to use mapRowKeys. When you read the frames, Deedle automatically generates ordinal row keys for your data and so merging would fail because you have two rows with a key 0. The mapRowKeys function lets us transform the keys so that they are unique and the frames can be merged. (Saving the CSV file does not automatically write the row keys to the output, so the result of this is exactly what you wanted.)

Answer 2

If yo do a lot of processing like this you should look into the CSV TypeProvider and Parser or my favorite FileHelpers.
If you don't want to use any third party libraries, here's a quick step-by-step process to read, re-assemble and write out the file:
open System.IO
open System
let csv1path = #"E:\tmp\csv1.csv"
let csv2path = #"E:\tmp\csv2.csv"
/// Read the file, split it up, and remove the header from the first csv file
let csv1 =
File.ReadAllLines(csv1path)
|> Array.map (fun x -> x.Split(','))
|> Array.tail
let csv2 =
File.ReadAllLines(csv2path)
|> Array.map (fun x -> x.Split(','))
///Split the header and data in the second csv file
let header', data = (csv2.[0], Array.tail csv2)
let header = String.Join(",", header')
///put back the data together, this is an array of arrays
let csv3 =
Array.append(csv1) data
///Sort the combined file, put it back together as a csv and add back the header
let csv4 =
csv3
|> Array.sort
|> Array.map (fun x -> String.Join(",", x))
|> Array.append [|header|]
///Write it out
File.WriteAllLines(#"E:\tmp\combined.csv",csv4)