finding patterns in csv file - csv

Question

trying to sort a csv file based on the repeating rows
awk -F, 'NR>1{arr[$4,",",$5,",",$6,,",",$7,",",$8,",",$9]++}END{for (a in arr) printf "%s\n", arr[a] "-->" a}' test.txt
Input file
a,b,d,1,2,3,4,5,6,y,x,z
k,s,t,1,2,3,4,5,6,t,z,s
a,b,k,1,4,5,5,5,6,k,r,s
Create a file with
a,b,d,1,2,3,4,5,6,y,x,z-->2
k,s,t,1,2,3,4,5,6,2,t,z,s-->2
a,b,k,1,4,5,5,5,6,1,k,r,s-->1
where the last column contains the number of occurrences of the pattern of numbers that start in the 4th place till the 9th place.
Count and sort the duplicate lines
I got to the point that I have the patterns with the count - but I don't know how to add the rest of the columns to the line:
thank you for the support.

Answer 1

A solution where the data is read twice, on the first go the duplicates are counted and on the second outputed:
$ awk -F, '
NR==FNR {
a[$4 ORS $5 ORS $6 ORS $7 ORS $8 ORS $9]++ # count
next
}
{
print $0 "-->" a[$4 ORS $5 ORS $6 ORS $7 ORS $8 ORS $9] # output
}' file file
a,b,d,1,2,3,4,5,6,y,x,z-->2
k,s,t,1,2,3,4,5,6,t,z,s-->2
a,b,k,1,4,5,5,5,6,k,r,s-->1

Answer 2

Could you please try following, reading Input_file single time only.
awk '
BEGIN{
FS=OFS=","
}
{
a[FNR]=$0
b[FNR]=$4 FS $5 FS $6 FS $7 FS $8 FS $9
c[$4 FS $5 FS $6 FS $7 FS $8 FS $9]++
}
END{
for(i=1;i<=FNR;i++){
print a[i]" ---->" c[b[i]]
}
}' Input_file

Answer 3

The answer of James Brown is a very simple double-pass solution which has the advantage that you don't need to store the file into memory but the disadvantage of having to read a file twice. The following solution will just do the inverse, read the file only ones but have to keep it into memory. To this end we need 3 arrays. Array c to keep track of the count, array b to act as a buffer and array a to keep track of the original order.
Furthermore, we will make use of multidimensional array indices:
A valid array index shall consist of one or more <comma>-separated expressions, similar to the way in which multi-dimensional arrays are indexed in some programming languages. Because awk arrays are really one-dimensional, such a <comma>-separated list shall be converted to a single string by concatenating the string values of the separate expressions, each separated from the other by the value of the SUBSEP variable. Thus, the following two index operations shall be equivalent:
var[expr1, expr2, ... exprn]
var[expr1 SUBSEP expr2 SUBSEP... SUBSEP exprn]
The solution now reads:
{ a[NR] = $4 SUBSEP $5 SUBSEP $6 SUBSEP $7 SUBSEP $8 SUBSEP $9
b[$4,$5,$6,$7,$8,$9] = $0
c[$4,$5,$6,$7,$8,$9]++ }
END { for(i=1;i<=NR;++i) print b[a[i]],"-->",c[a[i]] }

Answer 4

Since the problem resembles SQL pattern, you can use sqlite also. Check this out.
$ cat shimon.txt
a,b,d,1,2,3,4,5,6,y,x,z
k,s,t,1,2,3,4,5,6,t,z,s
a,b,k,1,4,5,5,5,6,k,r,s
$ cat sqllite_cols4_to_9.sh
#!/bin/sh
sqlite3 <<EOF
create table data(c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12);
.separator ','
.import "$1" data
select t1.*, " --> " || t2.cw from data t1, ( select c4,c5,c6,c7,c8,c9, count(*) as cw from data group by c4,c5,c6,c7,c8,c9 ) t2
where t1.c4=t2.c4 and t1.c5=t2.c5 and t1.c6=t2.c6 and t1.c7=t2.c7 and t1.c8=t2.c8 and t1.c9=t2.c9;
EOF
$ ./sqllite_cols4_to_9.sh shimon.txt
a,b,d,1,2,3,4,5,6,y,x,z, --> 2
k,s,t,1,2,3,4,5,6,t,z,s, --> 2
a,b,k,1,4,5,5,5,6,k,r,s, --> 1
$

Answer 5

You can try Perl also. The file is read only once, so it will be faster. Check this out:
$ cat shimon.txt
a,b,d,1,2,3,4,5,6,y,x,z
k,s,t,1,2,3,4,5,6,t,z,s
a,b,k,1,4,5,5,5,6,k,r,s
$ perl -F, -lane ' $v=join(",",#F[3..8]);$kv{$_}{$v}=$kv2{$v}++; END { while(($x,$y)=each (%kv)){ while(($p,$q)=each (%{$y})) { print "$x --> $kv2{$p}" }}}' shimon.txt
a,b,k,1,4,5,5,5,6,k,r,s --> 1
a,b,d,1,2,3,4,5,6,y,x,z --> 2
k,s,t,1,2,3,4,5,6,t,z,s --> 2
$
Another Perl - shorter code
$ perl -F, -lane ' $kv{$_}=$kv2{join(",",#F[3..8])}++; END { for(keys %kv) { $t=join(",",(split /,/)[3..8]); print "$_ --> $kv2{$t}" } } ' shimon.txt
a,b,k,1,4,5,5,5,6,k,r,s --> 1
a,b,d,1,2,3,4,5,6,y,x,z --> 2
k,s,t,1,2,3,4,5,6,t,z,s --> 2
or
$ perl -F, -lane ' $kv{$_}=$kv2{join(",",#F[3..8])}++; END { for(keys %kv) { print "$_ --> ",$kv2{join(",",(split /,/)[3..8])} } } ' shimon.txt
a,b,k,1,4,5,5,5,6,k,r,s --> 1
a,b,d,1,2,3,4,5,6,y,x,z --> 2
k,s,t,1,2,3,4,5,6,t,z,s --> 2
$