How to create seperate variables in a .csv file derived from values in text files? - csv

Question

I would be grateful for any advice you could provide on how to run the following in the UNIX command line. Essentially, I have text files for each of my subjects, which look like the following (simulated data).
2.97 3.61 -1.88
-0.38 2.33 -0.22
0.76 -0.71 -0.97
The subject ID is contained in the textfile heading (e.g. '100012_var.txt')
I would like to write a .csv file where each value (for each subject) in a row appears under a new variable heading. For instance:
ID Var1 Var2 Var3 Var4 Var5 Var6 Var7 Var8 Var9
100012 2.97 3.61 -1.88 -0.38 2.33 -0.22 0.76 -0.71 -0.97
100013 -1.21 1.79 -0.88 -0.91 2.01 2.88 0.32 -1.15 2.70
I would also like to ensure this is consistent across all subjects, i.e. value 1 in row 1 is always coded VAR 1.
I would really appreciate any suggestions!

Answer 1

Using awk:
$ awk -v RS="" -v OFS="\t" ' # using whole file as a record *
NR==1 { # first record, build the header
printf "ID" OFS
for(i=1;i<=NF;i++)
printf "Var%d%s",i,(i<NF?OFS:ORS)
}
{
split(FILENAME,f,"_") # split filename by _ to get the number
$1=$1 # rebuild the record to use tabs (OFS)
print f[1],$0 # print number part and the values
}' 100012_var.txt 100013_var.txt # them files
Output:
ID Var1 Var2 Var3 Var4 Var5 Var6 Var7 Var8 Var9
100012 2.97 3.61 -1.88 -0.38 2.33 -0.22 0.76 -0.71 -0.97
100013 -1.21 1.79 -0.88 -0.91 2.01 2.88 0.32 -1.15 2.70
* -v RS="" explained here.

Answer 2

Using Miller (https://github.com/johnkerl/miller) and perl
mlr --n2x --ifs ' ' --repifs put '$file=FILENAME' then reorder -f file input.tsv | \
perl -p -e 's/^\r\n$//g' | \
mlr --n2c --ifs ' ' --repifs uniq -a then cut -f 2 then cat -n then reshape -s n,2 \
then rename 1,ID then rename -r '([0-9]+),VAR\1'
you will have (it's a CSV)
ID,VAR2,VAR3,VAR4,VAR5,VAR6,VAR7,VAR8,VAR9,VAR10
input.tsv,2.97,3.61,-1.88,-0.38,2.33,-0.22,0.76,-0.71,-0.97
Then you can do a for loop for all files.