I am using perl dbi connection to pull data remotely. I am looking to compress the data if possible.
Is there a way to do a file compression over the net with perl dbi for mysql?
Here is my snippet for getting data:
my $sth = $dbh->prepare("SELECT UUID(), '$node', 1, 2, 3, 4, vts FROM $tblist");
$sth->execute();
while (my($uid, $hostnm,$1,$2,$3,$upd,$vts) = $sth->fetchrow_array() ) {
print $gzip_fh "rec^A$uid^Ehost^A$hostnm^E1^A$1^E2^A$2^E3^A$3^E4^A$upd^Evts^A$vts^D";
}
$sth->finish;
Best option will be to use prepared statements; with this, once you have compiled the statement once you just have to send the arguments over the wire each time you need to run the queries.
The example here shows how to perform a simple prepare, if you are going to utilize the same query over and over again you should keep your $sth which you can continue to call $sth->execute() on with new variables. The reason this cuts down on your network traversal is because you're not sending the query each time you run $sth->execute($var), instead you're just passing an ID for the prepared statement and the variables that go into the ? placeholders.
$sth = $dbh->prepare("select * from table where column=?");
$sth->execute($var);
Related
Throughout some testings; a little question popped up. When I usually code database updates; I usually do this via callbacks which I code in PHP; to which I simply pass a given mysqli connection object as function argument. Executing all queries of for example three queries across the same single connection proved to be much faster than if closing and reopening a DB connection for each query of a given query sequence. This also works easily with SQL transactions, the connection can be passed along to callbacks without any issues.
My question is; can you also do this with prepared statement objects ? What I mean is, considering we successfully established a $conn object, representing the mysqli connection, is stuff like this legit? :
function select_users( $users_id, $stmt ) {
$sql = "SELECT username FROM users where ID = ?";
mysqli_stmt_prepare( $stmt, $sql );
mysqli_stmt_bind_param( $stmt, "i", $users_id );
mysqli_stmt_execute( $stmt );
return mysqli_stmt_get_result( $stmt );
}
function select_labels( $artist, $stmt ) {
$sql = "SELECT label FROM labels where artist = ?";
mysqli_stmt_prepare( $stmt, $sql );
mysqli_stmt_bind_param( $stmt, "s", $artist );
mysqli_stmt_execute( $stmt );
return mysqli_stmt_get_result( $stmt );
}
$stmt = mysqli_stmt_init( $conn );
$users = select_users( 1, $stmt );
$rappers = select_labels( "rapperxyz", $stmt );
or is it bad practice; and you should rather use:
$stmt_users = mysqli_stmt_init( $conn );
$stmt_rappers = mysqli_stmt_init( $conn );
$users = select_users( 1, $stmt_users );
$rappers = select_labels( "rapperxyz", $stmt_rappers );
During the testing; I noticed that the method by using a single statement object passed along callbacks works for server calls where I call like 4 not too complicated DB queries via the 4 according callbacks in a row.
When I however do a server call with like 10 different queries, sometimes (yes, only sometimes; for pretty much the same data used across the different executions; so this seems to be weird behavior to me) I get the error "Commands out of sync; you can't run this command now" and some other weird errors I've never experienced, like the amount of variables not matching the amount of parameters; although they prefectly do after checking them all. The only way to fix this I found after some research was indeed by using different statement objects for each callback. So, I just wondered; should you actually ALWAYS use ONE prepared statement object for ONE query, which you then may execute N times in a row?
Yes.
The "commands out of sync" error is because MySQL protocol is not like http. You can't send requests any time you want. There is state on the server-side (i.e. mysqld) that is expecting a certain sequence of requests. This is what's known as a stateful protocol.
Compare with a protocol like ftp. You can do an ls in an ftp client, but the list of files you get back depends on the current working directory. If you were sharing that ftp client connection among multiple functions in your app, you don't know that another function hasn't changed the working directory. So you can't be sure the file list you get from ls represents the directory you thought you were in.
In MySQL too, there's state on the server-side. You can only have one transaction open at a time. You can only have one query executing at a time. The MySQL client does not allow you to execute a new query where there are still rows to be fetched from an in-progress query. See Commands out of sync in the MySQL doc on common errors.
So if you pass your statement handle around to some callback functions, how can that function know it's safe to execute the statement?
IMO, the only safe way to use a statement is to use it immediately.
I have the following perl code that will eventually be a webpage:
my($dbh) = DBI->connect("DBI:mysql:host=dbsrv;database=database","my_sqlu","my_sqlp") or die "Canny Connect";
my($sql) = "SELECT * FROM hardware where srv_name = \"$srv_name\"";
my($sth) = $dbh->prepare($sql);
$sth->execute();
$sth->bind_col( 1, \my($db_id));
$sth->bind_col( 2, \my($db_srv_name));
$sth->bind_col( 5, \my($db_site));
$sth->fetchrow();
$sth->finish ();
my($sql) = "SELECT sites.\`site_code\`, sites.\`long_name\` FROM \`hardware\` JOIN \`sites\` ON \`sites\`.id=\`hardware\`.\`site\` where \`hardware\`.\`id\`=\'$db_id\'";
my($sth) = $dbh->prepare($sql);
$sth->execute();
$sth->bind_col( 1, \my($db_site_code));
$sth->bind_col( 2, \my($db_long_name));
$sth->fetchrow();
$sth->finish ();
$dbh->disconnect;
print "$db_site_code<br>$db_long_name";
The query above does work however what I'm trying to find out is there any way I can run one SQL query and get the db_site_code and db_long_name from the sites DB without running the 2nd query? The hardware DB has the foreign key 'id' in the sites Db.
When you read anything about relational DBs they all say it's by far the most efficient method of getting data from your database but I just can't see how this is any quicker than just running 2 select queries. What I've done above would surely take longer than "select from hardware where srv_name = $srv_name" then "select from sites where id = db_site_id"? Any comments are greatly appreciated.
Here's an example of how to do this with placeholders as well as a combined query. If I understand your DB correctly, you can just omit the first query and add the server name instead of the ID in the second query. I might be mistaken there, but my example will still be of value for the Perl suggestions.
use strict;
use warnings;
use DBI;
# Create DB connection
my $dbh = DBI->connect("DBI:mysql:host=dbsrv;database=database","my_sqlu","my_sqlp")
or die "Cannot connect to database";
# Create the statement handle
my $sth = $dbh->prepare(<<'SQLQUERY') or die $dbh->errstr;
SELECT s.site_code, s.long_name
FROM hardware h
JOIN sites s ON s.id=h.site
WHERE h.srv_name=?
SQLQUERY
$sth->execute('Server Name'); # There's the parameter
my $res = $sth->fetchrow_hashref; # $res now has a hash ref with the first row
print "$res->{'site_code'}<br>$res->{'long_name'}";
There were a few issues with your code I'd like to point out to you:
You should always use strict and use warnings. They make your life easier!
You can leave the parens ( and ) out with my. Saves you keystrokes and makes your code more readable.
You can (but do not have to, this is preference!) leave out the parens after method calls that do not have arguments. Decide this for yourself.
As was already pointed out, always use placeholders with DBI. They are very simple. Now you don't have to escape the " with backslashes. Instead, just use ?.
Once you've combined your query, you can put it in a heredoc (<<'SQLQUERY'). It's a string that lasts from the next line to the delimiter (SQLQUERY). That way, your query is easier to read.
You can use one of the ref-fetchrow-methods to get all your result's columns into one hash. I used $sth->fetchrow_hashref because I find it most convenient. You've got the complete row and all the columns are named hash keys.
If called in a small scope (like a short sub), you don't need to finish a statement handle. It will be finished and destroyed by Perl automatically once it goes out of scope.
Another thing about performance: If this is just run occasionally, don't worry about it. You can profile your queries with DBI::Profile to see which way it is faster, but you should only do that if you really need to.
In my experience, especially with very huge queries and a very busy database, two or three queries are a lot better than a single big one because they do not take over the servers resources. But again, that is something you need to profile and benchmark (if the need arises).
Aside from #tadman's recommendation to use placeholders, I'd tag this as a sql question as well, but your solution is to simply add
srv_name = \"$srv_name\"
to your second where clause, so that your statement is:
"SELECT sites.\`site_code\`, sites.\`long_name\` FROM \`hardware\` JOIN \`sites\` ON \`sites\`.id=\`hardware\`.\`site\` where \`hardware\`.\`id\`=\'$db_id\'";
I strongly second #tadman's suggestion though -- use prepared statements and/or placeholders whenever possible.
I have perl script as following my $tb = 'rajeev';
$query = 'select * from table where name = ?'
$sth = $dbh->prepare($query);
$sth->execute($tb);
Does $tb replaced by rajeev or 'rajeev' when query executes ? means does query executs as select * from table where name = rajeevorselect * from table where name = 'rajeev'
DBI handles all the escaping for you. In the case of a string, it will be 'rajeev'. Calling select * from table where name = rajeev will give you an error.
If you provide a number, it will not add quotation marks because they are not needed.
See the DBI Doc. It also says:
The quote() method should not be used with "Placeholders and Bind Values".
Using placeholders sometimes takes care of the quoting for you, depending on which DBD you are using. In your case the DBD::mysql calls $dbh->quote() as mentioned in the doc:
An alternative approach is
$dbh->do("INSERT INTO foo VALUES (?, ?)", undef, $number, $name);
in which case the quote method is executed automatically.
If you have access to the query log you can check what the queries look like. If you have queries that take a long time you can also open a mysql console and say SHOW FULL PROCESSLIST; to see a list of the running queries. That will also hold the complete SQL statements for you to look at. On Windows you could use HeidiSQL to do it.
I have couple of mysql queries in perl but some of the values of the where clause contain space between words e.g. the gambia. When my scripts runs with the where clause arguments containing a space it ignore the second word.
I want to know how can I solve this problem i.e. if I type the gambia it should be treated the gambia not the.
If you are using DBI, you can use placeholders to send arbitrary data to database without need to care about escaping. The placeholder is question mark in prepare statement, actual value is given to execute:
use DBI;
$dbh = DBI->connect("DBI:mysql:....",$user,$pass)
or die("Connect error: $DBI::errstr");
my $sth = $dbh->prepare(qq{ SELECT something FROM table WHERE name = ? });
$sth->execute('the gambia');
# fetch data from $sth
$dbh->disconnect();
Edit: If you are composing the query (as you suggested in comments), you can utilize quote method:
my $country = "AND country = " . $dbh->quote('the gambia');
my $sth = $dbh->prepare(qq{ SELECT something FROM table WHERE name = ? $country});
Well, firstly, you should look at using something like DBIx::Class instead of raw SQL in your application.
But if you're stuck with raw SQL, then (assuming that you're, at least, using DBI) you should use bind points in your SQL statements. This will handle all of your quoting problems for you.
$sth = $dbh->prepare('select something from somewhere where country = ?');
$sth->execute('The Gambia');
See the DBI docs for more information about binding.
Seems that it may not be possible, but hey I might as well ask, I could be wrong. Was wondering if there's anyway for perl to update multiple rows using one MySQL call, I'm using DBI.
Any help or feedback would be greatly appreciated, this is possible in MSSQL through ASP and ASP.net so was wondering if also possible through perl on MySQL.
Thank you for your feedback!
First and most important, you absolutely should not interpolate variables directly into your SQL strings. That leaves open the possibility of SQL injection attacks. Even if those variables don't come from user input, it leaves open the possibility of dangerous bugs that can screw up your data.
The MySQL DBD driver does support multiple statements, though it's turned off by default as a safety feature. See mysql_multi_statements under the Class Methods section in the DBD::mysql documentation.
But a much better solution, which solves both problems at once and is more portable, is to use prepared statements and placeholder values.
my $sth = $dbh->prepare("UPDATE LOW_PRIORITY TableName SET E1=?,F1=? WHERE X=?");
Then, get your data in a loop of some sort:
while( $whatever) {
my ( $EC, $MR, $EM ) = get_the_data();
$sth->execute( $EC, $MR, $EM );
}
You only need to prepare the statement once, and the placeholder values are replaced (and guaranteed to be properly quoted) by the DBD driver.
Read more about placeholders in the DBI docs.
You don't need mysql_multi_statements, as friedo suggests.
You need turn off AutoCommit mode before you call the loop containing your UPDATE command:
**$dbh->{AutoCommit} = 0;**
while( $condition ) {
my $myParam = something();
...
$sth->execute( $myParam ); #your prepared UPDATE statement
...
}
**$dbh->commit();**