Read file, parse by tabs and insert into mysql database with Bash Script - mysql

Question

I am doing a bash script to read a file line by line, parse and insert it in mysql database.
The script is the following:
#!/bin/bash
echo "Start!"
while IFS=' ' read -ra ADDR;
do
for line in $(cat filename)
do
regex='(\d\d)-(\d\d)-(\d\d)\s(\d\d:\d\d:\d\d)'
if [[$line=~$regex]]
then
$line='20$3-$2-$1 $4';
fi
echo "insert into table (time, total, caracas, anzoategui) values('$line', '$line', '$line', $
done | mysql -uuser -ppassword database;
done < filename
The file 'filename' with data is something like this:
15/08/13 09:34:38 17528 5240 399 89 460 159 1107 33240
15/08/13 09:42:57 17528 5240 399 89 460 159 1107 33240
15/08/13 10:20:03 17492 5217 394 89 459 159 1101 33245
15/08/13 11:20:02 17521 5210 402 90 462 158 1112 33249
15/08/13 12:20:04 17540 5209 396 90 459 160 1105 33258
And its dropping this:

Answer 1

Use the LOAD DATA statement. Do any transformations you have to on your file first, then
LOAD DATA LOCAL INFILE 'filename' INTO table (time, total, caracas, anzoategui)
Tab separated fields is the default.

Answer 2

I would recommend creating your sql file using awk and then sourcing it from mysql. Re-direct the output once it looks ok to you into a new file (say insert.sql) and source it from mysql command line. Something like this:
$ cat file
15/08/13 09:34:38 17528 5240 399 89 460 159 1107 33240
15/08/13 09:42:57 17528 5240 399 89 460 159 1107 33240
15/08/13 10:20:03 17492 5217 394 89 459 159 1101 33245
15/08/13 11:20:02 17521 5210 402 90 462 158 1112 33249
15/08/13 12:20:04 17540 5209 396 90 459 160 1105 33258
$ awk -F'[/ ]+' -v q="'" '{print "insert into table (time, total, caracas, anzoategui) values ("q"20"$3"-"$2"-"$1" "$4q","q$5q","q$6q","q$7q");"}' file
insert into table (time, total, caracas, anzoategui) values ('2013-08-15 09:34:38','17528','5240','399');
insert into table (time, total, caracas, anzoategui) values ('2013-08-15 09:42:57','17528','5240','399');
insert into table (time, total, caracas, anzoategui) values ('2013-08-15 10:20:03','17492','5217','394');
insert into table (time, total, caracas, anzoategui) values ('2013-08-15 11:20:02','17521','5210','402');
insert into table (time, total, caracas, anzoategui) values ('2013-08-15 12:20:04','17540','5209','396');

Answer 3

Bash uses POSIX character classes. So instead of \d you could use [0-9] or [[:digit:]] to match a digit.
Also, there must be spaces in between the brackets and operator in your regex test. So [[$line=~$regex]] would be fixed by doing [[ $line =~ $regex ]].
In order to access the regular expressions enclosed in parentheses, you must access the builtin array variable BASH_REMATCH. So, to access the first match, you'd reference ${BASH_REMATCH[1]}, the second ${BASH_REMATCH[2]}, and so on.