Dynamic Multi Insert with DBI placeholders for many sets of VALUES - mysql

Question

I'm building a dynamic SQL statement, that will insert one or more sets of VALUES via a prepared DBI statement, my question is this:
Since I have a dynamic number of VALUES sets, and I will need to add as many ( ?, ?, ?),( ?, ?, ?) etc as necessary to extend the statement INSERT INTO `tblname` ( $columnsString ) VALUES in order to submit only one query using placeholders and bind values- is this the preferred method(most efficient, etc., - reasoning behind efficiency would be helpful in your answer if possible) or should I just be building this as a query string with sprintf and dbh->quote()?
(As a little extra information: I'm actually using AnyEvent::DBI right now, which only exposes placeholders & bind values and not the quote() method so this wouldn't be easy for me to accomplish without creating another straight DBI $dbh and using another db server connection just to use the quote() method, or without altering the AnyEvent::DBI module myself.)
Normally I would just execute the statements as necessary but in this heavy workload case I'm trying to batch inserts together for some DB efficiency.
Also, if anyone could answer if it is possible( and then how to ) insert an sql DEFAULT value using placeholders and bind values that'd be awesome. Typically if I ever needed to do that I'd append the DEFAULTs to the string directly and use sprintf and $dbh->quote() only for the non DEFAULT values.
UPDATE:
Worked out the misunderstanding in a quick chat. User ikegami suggested that instead of building the query string myself without placeholders, that I just intermingle VALUES and placeholders such as:
$queryString .= '(DEFAULT,?,?),(DEFAULT,DEFAULT,DEFAULT)';
Some of the reasoning behind my first asking of this question on SO was because I was somewhat against this intermingling due to my thought that it made the code less readable, though after being assured that sql 'DEFAULT' couldn't be in a placeholder bind value, this was the method I had begun implementing.
Using placeholders where possible does seem to be the more accepted method of building queries, and if you want an SQL DEFAULT you just need to include it in the same query building as the placeholders. This does not apply to NULL values, as those CAN be inserted with placeholders and a bind value of undef.
Update 2:
The reasoning I asked about performance, the 'acceptance' of building your own query with quote() vs building with placeholders, and why I've gone with a solution that involves using all columns for the SQL INSERT INTO tblname (cols) is because I have roughly 2-4 million rows a day going into a terrible db server, and my code is running on an equally terrible server. With my requirements of needing DEFAULT sql values, and these terrible performance constraints, I've chosen a solution for now.
For future devs who stumble upon this - take a look at #emazep's solution of using SQL::Abstract, or if for some reason you need to build your own, you might consider either using #Schwern's subroutine solution or possibly incorporating some of #ikegami's answer into it as these are all great answers as to the 'Current state of affairs' regarding the usage of DBI and building dynamic queries.

Answer 1

Unless there is a specific reason to reinvent the wheel (there could be some), SQL::Abstract (among others) has already solved the problem of dynamic SQL generation for all of us:
my %data = (
name => 'Jimbo Bobson',
phone => '123-456-7890',
address => '42 Sister Lane',
city => 'St. Louis',
state => 'Louisiana'
);
use SQL::Abstract;
my ($stmt, #bind)
= SQL::Abstract->new->insert('people', \%data);
print $stmt, "\n";
print join ', ', #bind;
which prints:
INSERT INTO people ( address, city, name, phone, state)
VALUES ( ?, ?, ?, ?, ? )
42 Sister Lane, St. Louis, Jimbo Bobson, 123-456-7890, Louisiana
SQL::Abstract then offers a nice trick to iterate over many rows to insert without regenerating the SQL every time, but for bulk inserts there is also SQL::Abstract::Plugin::InsertMulti
use SQL::Abstract;
use SQL::Abstract::Plugin::InsertMulti;
my ($stmt, #bind)
= SQL::Abstract->new->insert_multi( 'people', [
{ name => 'foo', age => 23 },
{ name => 'bar', age => 40 },
]);
# INSERT INTO people ( age, name ) VALUES ( ?, ? ), ( ?, ? )
# 23, foo, 40, bar

Answer 2

I have, on occasion, used a construct like:
#!/usr/bin/env perl
use strict; use warnings;
# ...
my #columns = ('a' .. 'z');
my $sql = sprintf(q{INSERT INTO sometable (%s) VALUES (%s)},
join(',', map $dbh->quote($_), #columns),
join(',', ('?') x #columns),
);
As for handling DEFAULT, wouldn't leaving that column out ensure that the DB sets it to the default value?

Answer 3

If you would use placeholders for "static" queries, you should use them for "dynamic" queries too. A query is a query.
my $stmt = 'UPDATE Widget SET foo=?'
my #params = $foo;
if ($set_far) {
$stmt .= ', far=?';
push #params, $far;
}
{
my #where;
if ($check_boo) {
push #where, 'boo=?';
push #params, $boo;
}
if ($check_bar) {
push #where, 'bar=?';
push #params, $bar;
}
$stmt .= ' WHERE ' . join ' AND ', map "($_)", #where
if #where;
}
$dbh->do($stmt, undef, #params);
I used an UPDATE since it allowed me to demonstrate more, but everything applies to INSERT too.
my #fields = ('foo');
my #params = ($foo);
if ($set_far) {
push #fields, 'bar';
push #params, $far;
}
$stmt = 'INSERT INTO Widget ('
. join(',', #fields)
. ') VALUES ('
. join(',', ('?')x#fields)
. ')';
$dbh->do($stmt, undef, #params);

Answer 4

You've expressed concerns about the readability of the code and also being able to pass in a DEFAULT. I'll take #ikegami's answer one step further...
sub insert {
my($dbh, $table, $fields, $values) = #_;
my $q_table = $dbh->quote($table);
my #q_fields = map { $dbh->quote($_) } #$fields;
my #placeholders = map { "?" } #q_fields;
my $sql = qq{
INSERT INTO $q_table
( #{[ join(', ', #q_fields) ]} )
VALUES ( #{[ join(', ', #placeholders ]} )
};
return $dbh->do($sql, undef, #$values);
}
Now you have a generic multi value insert routine.
# INSERT INTO foo ('bar', 'baz') VALUES ( 23, 42 )
insert( $dbh, "foo", ['bar', 'baz'], [23, 43] );
To indicate a default value, don't pass in that column.
# INSERT INTO foo ('bar') VALUES ( 23 )
# 'baz' will use its default
insert( $dbh, "foo", ['bar'], [23] );
You can optimize this to make your subroutine do multiple inserts with one subroutine call and one prepared statement saving CPU on the client side (and maybe some on the database side if it supports prepared handles).
sub insert {
my($dbh, $table, $fields, #rows) = #_;
my $q_table = $dbh->quote($table);
my #q_fields = map { $dbh->quote($_) } #$fields;
my #placeholders = map { "?" } #q_fields;
my $sql = qq{
INSERT INTO $q_table
( #{[ join(', ', #q_fields) ]} )
VALUES ( #{[ join(', ', #placeholders ]} )
};
my $sth = $dbh->prepare_cached($sql);
for my $values (#rows) {
$sth->execute(#$values);
}
}
# INSERT INTO foo ('bar', 'baz') VALUES ( 23, 42 )
# INSERT INTO foo ('bar', 'baz') VALUES ( 99, 12 )
insert( $dbh, "foo", ['bar', 'baz'], [23, 43], [99, 12] );
Finally, you can write a bulk insert passing in multiple values in a single statement. This is probably the most efficient way to do large groups of inserts. This is where having a fixed set of columns and passing in a DEFAULT marker comes in handy. I've employed the idiom where values passed as scalar references are treated as raw SQL values. Now you have the flexibility to pass in whatever you like.
sub insert {
my($dbh, $table, $fields, #rows) = #_;
my $q_table = $dbh->quote($table);
my #q_fields = map { $dbh->quote($_) } #$fields;
my $sql = qq{
INSERT INTO $q_table
( #{[ join(', ', #q_fields) ]} )
VALUES
};
# This would be more elegant building an array and then joining it together
# on ",\n", but that would double the memory usage and there might be
# a lot of values.
for my $values (#rows) {
$sql .= "( ";
# Scalar refs are treated as bare SQL.
$sql .= join ", ", map { ref $value ? $$_ : $dbh->quote($_) } #$values;
$sql .= "),\n";
}
$sql =~ s{,\n$}{};
return $dbh->do($sql);
}
# INSERT INTO foo ('bar', 'baz') VALUES ( 23, NOW ), ( DEFAULT, 12 )
insert( $dbh, "foo", ['bar', 'baz'], [23, \"NOW"], [\"DEFAULT", 12] );
The down side is this builds a string in memory, possibly very large. To get around that you have to involve database specific bulk insert from file syntax.
Rather than writing all this SQL generation stuff yourself, go with #emazep's answer and use SQL::Abstract and SQL::Abstract::Plugin::InsertMulti.
Just make sure you profile.