Has anyone seen a DBI-type module for Perl which capitalizes, easily, on MySQL's multi-insert syntax
insert into TBL (col1, col2, col3) values (1,2,3),(4,5,6),...?
I've not yet found an interface which allows me to do that. The only thing I HAVE found is looping through my array. This method seems a lot less optimal vs throwing everything into a single line and letting MySQL handle it. I've not found any documentation out there IE google which sheds light on this short of rolling my own code to do it.
TIA
There are two approaches. You can insert (?, ?, ?) a number of times based on the size of the array. The text manipulation would be something like:
my $sql_values = join( ' ', ('(?, ?, ?)') x scalar(#array) );
Then flatten the array for calling execute(). I would avoid this way because of the thorny string and array manipulation that needs to be done.
The other way is to begin a transaction, then run a single insert statement multiple times.
my $sql = 'INSERT INTO tbl (col1, col2, col3)';
$dbh->{AutoCommit} = 0;
my $sth = $dbh->prepare_cached( $sql );
$sth->execute( #$_ ) for #array;
$sth->finish;
$dbh->{AutoCommit} = 1;
This is a bit slower than the first method, but it still avoids reparsing the statement. It also avoids the subtle manipulations of the first solution, while still being atomic and allowing disk I/O to be optimized.
If DBD::mysql supported DBI's execute_for_fetch (see DBI's execute_array and execute_for_fetch) this is the typical usage scenario i.e., you have multiple rows of inserts/updates/deletes available now and want to send them in one go (or in batches). I've no idea if the mysql client libs support sending multiple rows of bound parameters in one go but most other database client libs do and can take advantage of DBI's execute_array/execute_for_fetch. Unfortunately few DBDs actually implement execute_array/execute_for_fetch and rely on DBI implementing it one row at a time.
Jim,
Frezik has it. That is probably the most optimal:
my $sth = $dbh->prepare( 'INSERT INTO tbl (?, ?, ?)' );
foreach(#array) { $sth->execute( #{$_} ); }
$sth->finish;
Related
I am dissecting some code and came across this,
$sql = 'SELECT page.*, author.name AS author, updator.name AS updator '
. 'FROM '.TABLE_PREFIX.'page AS page '
. 'LEFT JOIN '.TABLE_PREFIX.'user AS author ON author.id = page.created_by_id '
. 'LEFT JOIN '.TABLE_PREFIX.'user AS updator ON updator.id = page.updated_by_id '
. 'WHERE slug = ? AND parent_id = ? AND (status_id='.Page::STATUS_REVIEWED.' OR status_id='.Page::STATUS_PUBLISHED.' OR status_id='.Page::STATUS_HIDDEN.')';
I am wondering what the "?" does in the WHERE statement. Is it some sort of parameter holder?
Prepared statments use the '?' in MySQL to allow for binding params to the statement. Highly regarded as more secure against SQL injections if used properly. This also allows for quicker SQL queries as the request only has to be compiled once and can be reused.
The question mark represents a parameter that will later be replaced. Using parameterized queries is more secure than embedding the parameters right into the query.
SQL Server calls this parameterize queries, and Oracle calls it bind variables.
The usage varies with the language that you are executing the query from.
Here is an example of how it is used from PHP.
assuming that $mysqli is a database connection and people is a table with 4 columns.
$stmt = $mysqli->prepare("INSERT INTO People VALUES (?, ?, ?, ?)");
$stmt->bind_param('sssd', $firstName, $lastName, $email, $age);
The 'sssd' is a flag identifying the rest of the parameters, where s represents string and d represents digits.
? has no special meaning in MySQL WHERE = statements, only in prepared statements
The most common case where we see it is due to special meaning given to ? by several web frameworks like PHP and Rails.
? is just a syntax error at:
CREATE TABLE t (s CHAR(1));
SELECT * FROM t WHERE s = ?;
because it is unquoted, and in:
INSERT INTO t VALUES ('a');
INSERT INTO t VALUES ("?");
SELECT * FROM t WHERE s = '?';
it returns:
s
?
thus apparently without special meaning.
MySQL 5.0 prepared statements
MySQL 5.0 added the prepared statement feature which has similar semantics to the question mark in web frameworks.
Example from the docs:
PREPARE stmt1 FROM 'SELECT SQRT(POW(?,2) + POW(?,2)) AS hypotenuse';
SET #a = 3;
SET #b = 4;
EXECUTE stmt1 USING #a, #b;
Output:
hypotenuse
5
These also escape special characters as expected:
PREPARE stmt1 FROM 'SELECT ? AS s';
SET #a = "'";
EXECUTE stmt1 USING #a;
Output:
s
'
Rails example
In Rails for example, the question mark is replaced by an argument given by a variable of the library's programming language (Ruby), e.g.:
Table.where("column = ?", "value")
and it automatically quotes arguments to avoid bugs and SQL injection, generating a statement like:
SELECT * FROM Table WHERE column = 'value';
The quoting would save us in case of something like:
Table.where("column = ?", "; INJECTION")
These are prepared statements ,prepared statements offer two major benefits:
The query only needs to be parsed (or prepared) once, but can be
executed multiple times with the same or different parameters. When
the query is prepared, the database will analyze, compile and optimize
its plan for executing the query. For complex queries this process can
take up enough time that it will noticeably slow down an application
if there is a need to repeat the same query many times with different
parameters. By using a prepared statement the application avoids
repeating the analyze/compile/optimize cycle. This means that prepared
statements use fewer resources and thus run faster.
The parameters to prepared statements don't need to be quoted; the
driver automatically handles this. If an application exclusively uses
prepared statements, the developer can be sure that no SQL injection
will occur (however, if other portions of the query are being built up
with unescaped input, SQL injection is still possible).
http://php.net/manual/en/pdo.prepared-statements.php
I have perl script as following my $tb = 'rajeev';
$query = 'select * from table where name = ?'
$sth = $dbh->prepare($query);
$sth->execute($tb);
Does $tb replaced by rajeev or 'rajeev' when query executes ? means does query executs as select * from table where name = rajeevorselect * from table where name = 'rajeev'
DBI handles all the escaping for you. In the case of a string, it will be 'rajeev'. Calling select * from table where name = rajeev will give you an error.
If you provide a number, it will not add quotation marks because they are not needed.
See the DBI Doc. It also says:
The quote() method should not be used with "Placeholders and Bind Values".
Using placeholders sometimes takes care of the quoting for you, depending on which DBD you are using. In your case the DBD::mysql calls $dbh->quote() as mentioned in the doc:
An alternative approach is
$dbh->do("INSERT INTO foo VALUES (?, ?)", undef, $number, $name);
in which case the quote method is executed automatically.
If you have access to the query log you can check what the queries look like. If you have queries that take a long time you can also open a mysql console and say SHOW FULL PROCESSLIST; to see a list of the running queries. That will also hold the complete SQL statements for you to look at. On Windows you could use HeidiSQL to do it.
Seems that it may not be possible, but hey I might as well ask, I could be wrong. Was wondering if there's anyway for perl to update multiple rows using one MySQL call, I'm using DBI.
Any help or feedback would be greatly appreciated, this is possible in MSSQL through ASP and ASP.net so was wondering if also possible through perl on MySQL.
Thank you for your feedback!
First and most important, you absolutely should not interpolate variables directly into your SQL strings. That leaves open the possibility of SQL injection attacks. Even if those variables don't come from user input, it leaves open the possibility of dangerous bugs that can screw up your data.
The MySQL DBD driver does support multiple statements, though it's turned off by default as a safety feature. See mysql_multi_statements under the Class Methods section in the DBD::mysql documentation.
But a much better solution, which solves both problems at once and is more portable, is to use prepared statements and placeholder values.
my $sth = $dbh->prepare("UPDATE LOW_PRIORITY TableName SET E1=?,F1=? WHERE X=?");
Then, get your data in a loop of some sort:
while( $whatever) {
my ( $EC, $MR, $EM ) = get_the_data();
$sth->execute( $EC, $MR, $EM );
}
You only need to prepare the statement once, and the placeholder values are replaced (and guaranteed to be properly quoted) by the DBD driver.
Read more about placeholders in the DBI docs.
You don't need mysql_multi_statements, as friedo suggests.
You need turn off AutoCommit mode before you call the loop containing your UPDATE command:
**$dbh->{AutoCommit} = 0;**
while( $condition ) {
my $myParam = something();
...
$sth->execute( $myParam ); #your prepared UPDATE statement
...
}
**$dbh->commit();**
I am dissecting some code and came across this,
$sql = 'SELECT page.*, author.name AS author, updator.name AS updator '
. 'FROM '.TABLE_PREFIX.'page AS page '
. 'LEFT JOIN '.TABLE_PREFIX.'user AS author ON author.id = page.created_by_id '
. 'LEFT JOIN '.TABLE_PREFIX.'user AS updator ON updator.id = page.updated_by_id '
. 'WHERE slug = ? AND parent_id = ? AND (status_id='.Page::STATUS_REVIEWED.' OR status_id='.Page::STATUS_PUBLISHED.' OR status_id='.Page::STATUS_HIDDEN.')';
I am wondering what the "?" does in the WHERE statement. Is it some sort of parameter holder?
Prepared statments use the '?' in MySQL to allow for binding params to the statement. Highly regarded as more secure against SQL injections if used properly. This also allows for quicker SQL queries as the request only has to be compiled once and can be reused.
The question mark represents a parameter that will later be replaced. Using parameterized queries is more secure than embedding the parameters right into the query.
SQL Server calls this parameterize queries, and Oracle calls it bind variables.
The usage varies with the language that you are executing the query from.
Here is an example of how it is used from PHP.
assuming that $mysqli is a database connection and people is a table with 4 columns.
$stmt = $mysqli->prepare("INSERT INTO People VALUES (?, ?, ?, ?)");
$stmt->bind_param('sssd', $firstName, $lastName, $email, $age);
The 'sssd' is a flag identifying the rest of the parameters, where s represents string and d represents digits.
? has no special meaning in MySQL WHERE = statements, only in prepared statements
The most common case where we see it is due to special meaning given to ? by several web frameworks like PHP and Rails.
? is just a syntax error at:
CREATE TABLE t (s CHAR(1));
SELECT * FROM t WHERE s = ?;
because it is unquoted, and in:
INSERT INTO t VALUES ('a');
INSERT INTO t VALUES ("?");
SELECT * FROM t WHERE s = '?';
it returns:
s
?
thus apparently without special meaning.
MySQL 5.0 prepared statements
MySQL 5.0 added the prepared statement feature which has similar semantics to the question mark in web frameworks.
Example from the docs:
PREPARE stmt1 FROM 'SELECT SQRT(POW(?,2) + POW(?,2)) AS hypotenuse';
SET #a = 3;
SET #b = 4;
EXECUTE stmt1 USING #a, #b;
Output:
hypotenuse
5
These also escape special characters as expected:
PREPARE stmt1 FROM 'SELECT ? AS s';
SET #a = "'";
EXECUTE stmt1 USING #a;
Output:
s
'
Rails example
In Rails for example, the question mark is replaced by an argument given by a variable of the library's programming language (Ruby), e.g.:
Table.where("column = ?", "value")
and it automatically quotes arguments to avoid bugs and SQL injection, generating a statement like:
SELECT * FROM Table WHERE column = 'value';
The quoting would save us in case of something like:
Table.where("column = ?", "; INJECTION")
These are prepared statements ,prepared statements offer two major benefits:
The query only needs to be parsed (or prepared) once, but can be
executed multiple times with the same or different parameters. When
the query is prepared, the database will analyze, compile and optimize
its plan for executing the query. For complex queries this process can
take up enough time that it will noticeably slow down an application
if there is a need to repeat the same query many times with different
parameters. By using a prepared statement the application avoids
repeating the analyze/compile/optimize cycle. This means that prepared
statements use fewer resources and thus run faster.
The parameters to prepared statements don't need to be quoted; the
driver automatically handles this. If an application exclusively uses
prepared statements, the developer can be sure that no SQL injection
will occur (however, if other portions of the query are being built up
with unescaped input, SQL injection is still possible).
http://php.net/manual/en/pdo.prepared-statements.php
I've been using the following snippet in developements for years. Now all of a sudden I get a DB Error: no such field warning
$process = "process";
$create = $connection->query
(
"INSERT INTO summery (process) VALUES($process)"
);
if (DB::isError($create)) die($create->getMessage($create));
but it's fine if I use numerics
$process = "12345";
$create = $connection->query
(
"INSERT INTO summery (process) VALUES($process)"
);
if (DB::isError($create)) die($create->getMessage($create));
or write the value directly into the expression
$create = $connection->query
(
"INSERT INTO summery (process) VALUES('process')"
);
if (DB::isError($create)) die($create->getMessage($create));
I'm really confused ... any suggestions?
It's always better to use prepared queries and parameter placeholders. Like this in Perl DBI:
my $process=1234;
my $ins_process = $dbh->prepare("INSERT INTO summary (process) values(?)");
$ins_process->execute($process);
For best performance, prepare all your often-used queries right after opening the database connection. Many database engines will store them on the server during the session, much like small temporary stored procedures.
Its also very good for security. Writing the value into an insert string yourself means that you must write the correct escape code at each SQL statement. Using a prepare and execute style means that only one place (execute) needs to know about escaping, if escaping is even necessary.
Ditto what Zan Lynx said about placeholders. But you may still be wondering why your code failed.
It appears that you forgot a crucial detail from the previous code that worked for you for years: quotes.
This (tested) code works fine:
my $thing = 'abcde';
my $sth = $dbh->prepare("INSERT INTO table1 (id,field1)
VALUES (3,'$thing')");
$sth->execute;
But this next code (lacking the quotation marks in the VALUES field just as your first example does) produces the error you report because VALUES (3,$thing) resolves to VALUES (3,abcde) causing your SQL server to look for a field called abcde and there is no field by that name.
my $thing = 'abcde';
my $sth = $dbh->prepare("INSERT INTO table1 (id,field1)
VALUES (3,$thing)");
$sth->execute;
All of this assumes that your first example is not a direct quote of code that failed as you describe and therefore not what you intended. It resolves to:
"INSERT INTO summery (process) VALUES(process)"
which, as mentioned above causes your SQL server to read the item in the VALUES set as another field name. As given, this actually runs on MySQL without complaint and will fill the field called 'process' with NULL because that's what the field called 'process' contained when MySQL looked there for a value as it created the new record.
I do use this style for quick throw-away hacks involving known, secure data (e.g. a value supplied within the program itself). But for anything involving data that comes from outside the program or that might possibly contain other than [0-9a-zA-Z] it will save you grief to use placeholders.