Remove columns with same values from a large csv file [closed] - csv

Question

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I have a large csv file with thousands of columns (50,000 rows and 25,000 columns).
I want to;
Obtain the list of all columns with same values
Remove all column obtained in step 1
Sample input
F1,F2,F3,F4
0,2,1,4
0,1,1,3
0,3,1,3
0,2,1,3
Sample output
Columns with same values: F1 F2
F2,F4
2,4
1,3
3,3
2,3
I have implemented python based solutions which works fine for small files but are too slow for large files (more than 8 GB).
A solution in any programming language (but robust and fast) will be appreciated.

Answer 1

another awk, using subset of #Luuk's sample input
$ awk -F, 'NR==FNR && NR>1 {if(NR>2) for(i=1;i<=NF;i++) if($i!=p[i]) c[i];
split($0,p); next}
FNR==1 {n=asorti(c,m); max=m[n]}
{for(i in c) printf "%s",$i (i==max?RS:FS)}' file{,}
F2,F4
1,3
2,2
0,1
2,1
2,0
1,2
1,3
The trick is it's enough to check if any of the consecutive values are different.
some extra work to find the max included column index (required for clean printing).
If you have thousands of columns you can speed up the filtering by changing i<=NF to i<=NF && !(i in c)
Perhaps this alternative will be the fastest, which quickly removes the columns if there are different values from the search set.
$ awk -F, 'NR==FNR && NR==1 {for(i=1;i<=NF;i++) a[i]; next}
NR==FNR {if(NR>2) for(i in a) if($i!=p[i]) {delete a[i]; c[i]};
split($0,p); next}
FNR==1 {n=asorti(c,m); max=m[n]}
{for(i in c) printf "%s",$i (i==max?RS:FS)}' file{,}
All solutions double scans the file so it can't be very fast but I expect it to complete in couple minutes. Please post the timings if you can test.

Answer 2

$ cat tst.awk
BEGIN { FS=OFS="," }
NR==FNR {
if ( NR > 1 ) {
for (inFldNr=1; inFldNr<=NF; inFldNr++) {
vals[inFldNr][$inFldNr]
}
}
next
}
FNR==1 {
for (inFldNr=1; inFldNr<=NF; inFldNr++) {
if ( length(vals[inFldNr]) > 1 ) {
out2in[++numOutFlds] = inFldNr
}
}
}
{
for (outFldNr=1; outFldNr<=numOutFlds; outFldNr++) {
printf "%s%s", $(out2in[outFldNr]), (outFldNr<numOutFlds ? OFS : ORS)
}
}
$ awk -f tst.awk file file
F2,F4
2,4
1,3
3,3
2,3
EDIT: tested against a file that's 50,000 lines of 25,000 fields:
$ time awk -f tst.awk file.csv file.csv > out.csv
real 17m42.591s
user 17m29.421s
sys 0m3.858s
The above input file was created by running this script:
$ awk 'BEGIN{OFS=","; x=50000; y=25000; for (i=1;i<=x;i++) for (j=1;j<=y;j++) printf "%s%s", (i>1?substr(rand(),3,1):"F"j), (j<y?OFS:ORS)}' > file.csv
$ wc file.csv
50000 50000 2500113283 file.csv
$ ls -lh file.csv
-rw-r--r-- 1 MyLogin None 2.4G Nov 22 12:45 file.csv

Answer 3

I created a piece of code in .NET 5.0, which has following output:
(EDIT: oops, i noticed i have compiled it for .NET 3.1 🙄)
WHICH COLUMNS HAVE UNIQUE VALUES
0: True
1: False
2: True
3: False
TIME AFTER CHECKING UNIQUE COLUMNS: 72,9047581 secs
TIME AFTER WRITING NON UNIQUE COLUMNS TO NEW FILE: 221,751107 secs
This is under 4 minutes. My PC (running Windows) might be quicker then yours, but the difference, compared to 1 hour, is to large... 😉
And i created an 'input file with 4 columns as your sample, like this:
D:\Temp>type random.txt
0,1,1,3
0,2,1,2
0,0,1,1
0,2,1,1
0,2,1,0
0,1,1,2
0,1,1,3
0,0,1,3
0,2,1,4
0,3,1,1
....
My example file has 2756616192 bytes, or 344577024 lines.
The output looks like this:
D:\Temp>type randomNEW.txt
1,3
2,2
0,1
2,1
2,0
1,2
1,3
0,3
2,4
3,1
this file has 1722885120 bytes, and has also 344577024 lines.
The source looks like this (note: I am not very experienced in C#, just learning, so there might be things that can be improved!):
using System;
using System.IO;
namespace ConsoleApp59
{
class Program
{
static void Main(string[] args)
{
DateTime start = DateTime.Now;
bool[] a = new bool[4];
int[] first = new int[4];
int t = 0;
for(int i=0; i<4; i++ ) { a[i] = true; }
foreach (string line in File.ReadLines(#"d:\temp\random.txt"))
{
string[] s = line.Split(",");
int[] b = new int[4]{ Convert.ToInt32(s[0]), Convert.ToInt32(s[1]), Convert.ToInt32(s[2]), Convert.ToInt32(s[3]) };
if (t==0)
{
first = b;
}
else
{
for (int i = 0; i < 4; i++)
{
if (a[i]) a[i] = b[i] == first[i];
}
}
t++;
}
Console.WriteLine("WHICH COLUMNS HAVE UNIQUE VALUES");
for(int i=0; i<4; i++)
{
Console.WriteLine($"{i}: {a[i]}");
}
Console.WriteLine($"TIME AFTER CHECKING UNIQUE COLUMNS: { ((DateTime.Now-start).TotalSeconds)} secs");
StreamWriter n = new StreamWriter(#"D:\temp\randomNEW.txt");
foreach (string line in File.ReadLines(#"d:\temp\random.txt"))
{
string[] s = line.Split(",");
string output = "";
for (int i=0; i<4; i++ )
{
if (!a[i]) output += String.Format($"{s[i]},");
}
if (output.EndsWith(",")) output = output.Substring(0, output.Length - 1);
output += "\r\n";
n.Write(output.ToCharArray());
}
n.Close();
Console.WriteLine($"TIME AFTER WRITING NON UNIQUE COLUMNS TO NEW FILE: { ((DateTime.Now - start).TotalSeconds)} secs");
//Console.ReadLine();
}
}
}
EDIT: I just did a test on a Mac-MINI (Late 2012; macOS Catalina 10.15.7; 2.3 Ghz; 16Gb RAM)
TIME AFTER CHECKING UNIQUE COLUMNS: 104.473532 secs
TIME AFTER WRITING NON UNIQUE COLUMNS TO NEW FILE: 300.272936 secs