How can we merge multiple csv files? - csv

Question

Each CSV file can have more than 1000 common fields,
We cannot use indexing as fields may not be in same order,
A field can have no values in one file but can have values in other file.

Answer 1

You haven't indicated what type of application you are working on ... but I would do it in .NET and use a Dictionary object. The key on the dictionary object would be the field name (this solves your not in same order problem) and the value for each dictionary item would be a list of strings (CSV truly originates as string).
Then you can enumerate the dictionary and join all of the strings per key into whatever format you desire.
This isn't fully tested ... but should get you going:
private void ReadCSVFiles (string[] filenames)
{
Dictionary<string, List<string>> data = new Dictionary<string, List<string>>();
foreach (string filename in filenames)
{
string[] content = System.IO.File.ReadAllLines(filename);
string[] fieldNames = content[0].Split(',');
for (int i = 1; i < content.Length; i++)
{
string[] tokens = content[i].Split(',');
for (int j = 0; j < fieldNames.Length; j++ )
{
List<string> values = null;
if (!data.TryGetValue(fieldNames[j], out values))
{
values = new List<string>();
data.Add(fieldNames[j], values);
}
values.Add(tokens[j]);
}
}
}
}