Write partly tab-delimited data to MySQL database - mysql

Question

I have a MySQL-Database with 7 Columns (chr, pos, num, iA, iB, iC, iD) and a file that contains 40 million lines each containing a dataset. Each line has 4 tab delimited columns, whereas the first three columns always contain data, and the fourth column can contain up to three different key=value pairs separated by a semicolon
chr pos num info
1 10203 3 iA=0.34;iB=nerv;iC=45;iD=dskf12586
1 10203 4 iA=0.44;iC=45;iD=dsf12586;iB=nerv
1 10203 5
1 10213 1 iB=nerv;iC=49;iA=0.14;iD=dskf12586
1 10213 2 iA=0.34;iB=nerv;iD=cap1486
1 10225 1 iD=dscf12586
The key=value pairs in the column info have no specific order. I'm also not sure if a key can occur twice (I hope not).
I'd like to write the data into the database. The first three columns are no problem, but extractiong the values from the info-columns puzzles me, since the key=value pairs are unordered and not every key has to be in the line.
For a similar dataset (with ordered info-Column) I used a java-Programm in connection with regular expressions, which allowed me to (1) check and (2) extract data, but now I'm stranded.
How can I resolve this task, preferably with a bash-script or directly in MySQL?

Answer 1

You did not mention exactly how you want to write the data. But the below example with awk shows how you can get each individual id and key in each line. instead of the printf, you can use your own logic to write data
[[bash_prompt$]]$ cat test.sh; echo "###########"; awk -f test.sh log
{
if(length($4)) {
split($4,array,";");
print "In " $1, $2, $3;
for(element in array) {
key=substr(array[element],0,index(array[element],"="));
value=substr(array[element],index(array[element],"=")+1);
printf("found %s key and %s value for %d line from %s\n",key,value,NR,array[element]);
}
}
}
###########
In 1 10203 3
found iD= key and dskf12586 value for 1 line from iD=dskf12586
found iA= key and 0.34 value for 1 line from iA=0.34
found iB= key and nerv value for 1 line from iB=nerv
found iC= key and 45 value for 1 line from iC=45
In 1 10203 4
found iB= key and nerv value for 2 line from iB=nerv
found iA= key and 0.44 value for 2 line from iA=0.44
found iC= key and 45 value for 2 line from iC=45
found iD= key and dsf12586 value for 2 line from iD=dsf12586
In 1 10213 1
found iD= key and dskf12586 value for 4 line from iD=dskf12586
found iB= key and nerv value for 4 line from iB=nerv
found iC= key and 49 value for 4 line from iC=49
found iA= key and 0.14 value for 4 line from iA=0.14
In 1 10213 2
found iA= key and 0.34 value for 5 line from iA=0.34
found iB= key and nerv value for 5 line from iB=nerv
found iD= key and cap1486 value for 5 line from iD=cap1486
In 1 10225 1
found iD= key and dscf12586 value for 6 line from iD=dscf12586

Answer 2

Awk solution from #abasu with inserts that also solves the unordered key-value pairs.
parse.awk :
NR>1 {
col["iA"]=col["iB"]=col["iC"]=col["iD"]="null";
if(length($4)) {
split($4,array,";");
for(element in array) {
split(array[element],keyval,"=");
col[keyval[1]] = "'" keyval[2] "'";
}
}
print "INSERT INTO tbl VALUES (" $1 "," $2 "," $3 "," col["iA"] "," col["iB"] "," col["iC"] "," col["iD"] ");";
}
Test/run :
$ awk -f parse.awk file
INSERT INTO tbl VALUES (1,10203,3,'0.34','nerv','45','dskf12586');
INSERT INTO tbl VALUES (1,10203,4,'0.44','nerv','45','dsf12586');
INSERT INTO tbl VALUES (1,10203,5,null,null,null,null);
INSERT INTO tbl VALUES (1,10213,1,'0.14','nerv','49','dskf12586');
INSERT INTO tbl VALUES (1,10213,2,'0.34','nerv',null,'cap1486');
INSERT INTO tbl VALUES (1,10225,1,null,null,null,'dscf12586');