Perl many insert statments into mysql database - mysql

Question

I currently have code for perl that looks like this:
#valid = grep { defined($column_mapping{ $headers[$_] }) } 0 .. $#headers;
...
my $sql = sprintf 'INSERT INTO tablename ( %s ) VALUES ( %s )',
join( ',', map { $column_mapping{$_} } #headers[#valid] ),
join( ',', ('?') x scalar #valid);
my $sth = $dbh->prepare($sql);
...
my #row = split /,/, <INPUT>;
$sth->execute( #row[#valid] );
(Taken from mob's answer to a previous question.)
That is basically dynamically building a sql insert statement from csv data, and only allowing the csv data with proper headers from my column mapping to be picked.
I have been looking for examples on how to do an insert statment with multiple rows of data at once.
My perl script needs to run around a few hundred million insert statments, and doing it one at a time seems really slow, especially since the server I am running it on only has 6gb of ram and a slowish internet connection.
Is there a way I can upload more than 1 row at a time of data? So one insert statment uploads maybe 50 rows, or 100 rows at once? I cant find out how with perl DBI.

Answer 1

my $sql_values = join( ' ', ('(?, ?, ?)') x scalar(#array) );
As said before, then you can just flatten it.

Answer 2

You can insert multiple rows at once with the same syntax as in regular SQL, but you need to build your INSERT statemtent properly with Perl. Perl's slice() may help you:
Suppose you have 7 rows of data and want to insert them in chunks of 3 rows. "Regular" SQL would be like so:
insert into T (col1, col2) values ( 1, 2), ( 3, 4), ( 5, 6);
insert into T (col1, col2) values ( 7, 8), ( 9,10), (11,12);
insert into T (col1, col2) values (13,14);
Let's suppose your perl structure is like this:
my $values = [ [1,2], [3,4], ..., [13,14] ];
If it's not, bring it into this shape. Now:
use constant CHUNKSIZE => 3;
my $stmt = sprintf( 'insert into T (col1, col2) values %s',
join(',', '(?,?)' x CHUNKSIZE) );
# $stmt is now 'insert into T (col1, col2) values (?,?),(?,?),(?,?)'
my $sth = $dbh->prepare($stmt);
while( my #chunk = splice( #{$values}, 0, CHUNKSIZE ) ) {
# #chunk has 3 elements (rows), or less for the last chunk
if (scalar #chunk == CHUNKSIZE) {
$sth->execute( #chunk ); # inserts 3 rows at once
} else {
# build and prepare a new statement for the remaining rows.
# in our sample there is only 1 remaining row.
$stmt = sprintf( 'insert into T (col1, col2) values %s',
join(',', '(?,?)' x scalar #chunk) );
$sth = $dbh->prepare($stmt);
$sth->execute( #chunk ); # inserts the last row
}
}