Update MySQL within Perl loop failing (fetchrow_array) - mysql

I've created a Perl script which is meant to loop through an array (a shortlist of customers who meet certain criteria), execute an external command using system() , then update a field within each row once the operation has completed.
It works on the first record (ie external command executes, customer record updates), however when it gets to the second record I receive this error:
DBD::mysql::st fetchrow_array failed: fetch() without execute() at customer_update.pl
Through some googling I added the $sth->finish(); command, however whether I include it or not (either inside the loop as shown, or straight afterward) I still get the same error.
Can anyone shed any light for me as to what I am doing wrong here?
Here's an extract:
# PERL MYSQL CONNECT()
$dbh = DBI->connect('dbi:mysql:signups', $user, $pw)
or die "Connection Error: $DBI::errstr\n";
# DEFINE A MySQL QUERY
$myquery = "SELECT * FROM accounts WHERE field3 = false";
$sth = $dbh->prepare($myquery);
# EXECUTE THE QUERY
$sth->execute
or die "SQL Error: $DBI::errstr\n";
#records = $sth->rows;
print "Amount of new customers: #records\n\n";
while ( my ($field1, $field2, $field3) = $sth->fetchrow_array() ) {
#execute external command via system();
$update_customer_status = "UPDATE accounts SET field3=true WHERE id=$id";
$sth = $dbh->prepare($update_customer_status);
$sth->execute
or die "SQL Error: $DBI::errstr\n";
print "Customer record modified & MySQL updated accordingly\n\n";
$sth->finish();
}

Building a SQL statement with variables and then prepare()ing it defeats the purpose of the prepare. You should build the SQL statement with a placeholder ? instead of $id, prepare() it, and then execute($id) it. As it is, you are leaving yourself open to SQL injection attacks.
Also, it seems that you are not using the warnings and strict pragmas. These two lines should be at the top of every program you write:
use warnings;
use strict;
They will save you much heartache and frustration in the future.

In your loop, you overwrite the handle over from which you are fetching. Use a different variable. (Changing $sth = ...; to my $sth = ...; will do.) While we're at it, let's move the prepare out of the loop.
my $sth_get = $dbh->prepare("SELECT * FROM accounts WHERE field3 = false");
my $sth_upd = $dbh->prepare("UPDATE accounts SET field3=true WHERE id = ?");
$sth_get->execute();
while ( my ($field1, $field2, $field3) = $sth_get->fetchrow_array() ) {
...
$sth_upd->execute($id);
}

You are stomping on your $sth variable when you execute this line ...
$sth = $dbh->prepare($update_customer_status);
Why not save off the result of $sth->fetchrow_array() to an array variable.
Something like ...
my #select_results_AoA = $sth->fetchrow_array();
... and then iterate over the array ...
for my #row ( #select_resilts_AoA ) {
... instead of ...
while ( my ($field1, $field2, $field3) = $sth->fetchrow_array() ) {

Related

Inserting Data From Perl To MysQL

I have already connected MySQL to the database but the problem is that
I have some problems with inserting Data from Perl to Mysql.
The error that pops out is:
Use of uninitialized value in concatenation (.) or string at ./etl_server_info.pl line 204, (Which is the connection of the database )
DBD::mysql::st execute failed: called with 1 bind variables when 0 are needed at ($stmt->execute($sql);)
sub insert_record(){
my($data,$db_config)=#_;
my $result = -1; # -1 fail; 0 - succ
# connect to db
# connect to MySQL database
my $dsn = "DBI:mysql:database=".$db_config->{'Top_Data'}.";host=".$db_config->{'127.0.0.1'}.";port=".$db_config->{'3306'};
my $username = $db_config->{'username'};
my $password = $db_config->{'password'};
my %attr = (PrintError=>0,RaiseError=>1 );
my $dbh = DBI->connect($dsn,$username,$password,\%attr) or die $DBI::errstr;
print "We Have Successfully Connected To The Database \n";
# prepare sql statement
# execute insert
my $sql = 'insert into Top(Load_Average, CPU_us, CPU_id, CPU-wa, CPU_hi, CPU_si, CPU_st, Swap_Total, Swap_Free, Swap_Used, Memory_Total, Memeory_Free, Memory_Used, Memory_Buff, Date) values(float,float,float,float,float,float,float,float,varchar,varchar,varchar,varchar,varchar,varchar,date)';
my $stmt =$dbh->prepare($sql) or die "Its A Fail" . $dbh->errstr();
$stmt->execute($sql);
$stmt->finish();
$dbh->disconnect();
$result = 0;
return($result);
Your use of the $db_config variable looks suspicious to me. Either your config hash is strange, or you're using values instead of keys.
You haven't shown us where $db_config is set up, but I'd guess it looks something like this:
$db_config = {
name => 'Top_Data',
host => '127.0.0.1',
port => 3306,
username => 'someone',
password => 'a secret',
};
And then you would use it like this:
my $dsn = "DBI:mysql:database=".$db_config->{name}.";host=".$db_config->{host}.";port=".$db_config->{port};
Notice that I've used the key names (name, host and port) instead of the values (Top_Data, 127.0.0.1 and 3306).
I'll also point out that you can simplify this slightly by using Perl's ability to expand variables inside a double-quoted string.
my $dsn = "DBI:mysql:database=$db_config->{name};host=$db_config->{host};port=$db_config->{port}";
There's another problem later on, with your SQL statement.
my $sql = 'insert into Top(Load_Average, CPU_us, CPU_id, CPU-wa, CPU_hi,
CPU_si, CPU_st, Swap_Total, Swap_Free, Swap_Used, Memory_Total,
Memeory_Free, Memory_Used, Memory_Buff, Date)
values(float,float,float,float,float,float,float,float,
varchar,varchar,varchar,varchar,varchar,varchar,date)';
The values that you should be inserting are the actual data items. So where you have the strings "float", "varchar" or "date", you should actually have data items (a floating-point number, a string or a date).
Finally, having prepared your statement, you don't need to pass it to the execute() method. You should, however, look at using bind points in your SQL and passing your actual data items to the execute() call

perl script to connect two databases at same time

I am trying to connect to 2 databases on the same instance of MySQL from 1 Perl script.
I am using this in a migration script where I am grabbing data from the original database and inserting it into the new one.
Connecting to 1 database and then trying to initiate a second connection with the same user just changes the current database to the new one.
#!/usr/bin/perl
use DBI;
use strict;
my $driver = "mysql";
my $database1 = "db1";
my $dsn1 = "DBI:$driver:database=$database1";
my $userid = "userhead";
my $password = "pwdhead";
my $database2 = "db2";
my $dsn2 = "DBI:$driver:database=$database2";
my $dbh1 = DBI->connect($dsn1, $userid, $password ) or die $DBI::errstr;
my $dbh2 = DBI->connect($dsn2, $userid, $password ) or die $DBI::errstr;
my $sth = $dbh2->prepare("INSERT INTO Persons") $dbh1->prepare("SELECT *FROM Persons");
$sth->execute() or die $DBI::errstr;
print "Number of rows found :" + $sth->rows;
In the above example i am trying to copy from one database table to another datbase table. but i am getting error while running the script. Please help me out
At a guess, you're trying to use the same database handle to connect to both databases. If you need to operate two separate connections then you need two separate handles
This program uses the data_sources class method to discover all of the available MySQL databases and creates a connection to each of them, putting the handles in the array #dbh. You can use each element of that array as normal, for instance
my $stmt = $dbh[0]->prepare('SELECT * FROM table)
It may be that you prefer to set up the #databases array manually, or the username and password may be different for the two data sources, so some variation on this may be necessary
use strict;
use warnings 'all';
use DBI;
my $user = 'username';
my $pass = 'password';
my #databases = DBI->data_sources('mysql');
my #dbh = map { DBI->connect($_, $user, $pass) } #databases;
Update
You need to select data from the source table, fetch it one row at a time, and insert each row into the destination table
Here's an idea how that might work, but you need to adjust the number of question marks in the VALUES of the INSERT statement to match the number of columns
Note that, if you're just intending to copy the whole dataset, there aree better ways to go about this. In particular, if you have any foreign key constraints then you won't be able to add data until the table it it is dependent on is populated
#!/usr/bin/perl
use strict;
use warnings 'all';
use DBI;
my $userid = "userhead";
my $password = "pwdhead";
my ($dbase1, $dbase2) = qw/ db1 db2 /;
my $dsn1 = "DBI:mysql:database=$dbase1";
my $dsn2 = "DBI:mysql:database=$dbase2";
my $dbh1 = DBI->connect($dsn1, $userid, $password ) or die $DBI::errstr;
my $dbh2 = DBI->connect($dsn2, $userid, $password ) or die $DBI::errstr;
my $select = $dbh1->prepare("SELECT * FROM Persons");
my $insert = $dbh2->prepare("INSERT INTO Persons VALUES (?, ?, ?, ?, ?)");
$select->execute;
while ( my #row = $select->fetchrow_array ) {
$insert->execute(#row);
}
If you need to handle the columns from the source data separately then you can use named scalars instead of the array #row. Like this
while ( my ($id, $name) = $select->fetchrow_array ) {
my $lastname = '';
$insert->execute($id, $name, $lastname);
}
Plan A (especially if one-time task):
Run mysqldump on the source machine; feed the output to mysql on the target machine. This will be much faster and simpler. If you are on a Unix machine, do it with an exec() from Perl (if you like).
Plan B (especially if repeated task):
If the table is not "too big", do one SELECT to fetch all the rows into an array in Perl. Then INSERT the rows into the target machine. This can be sped up (with some effort) if you build a multi-row INSERT or create a CSV file and use LOAD DATA instead of INSERT.

MySQL query 2 tables, 1 insert

I'm trying to pull data from 2 tables and then insert result into a third table. My code follows but only does 1 correct entry, and the rest are blank. There are 348 entries total. What am I missing here?
$dbh = DBI->connect(
"DBI:mysql:$mysqldatabase:$mysqlhostname",
"$mysqlusername",
"$mysqlpassword"
);
if(!$dbh) { die("Error: could not get DBI handle\n"); }
$sqlConnect = 1;
$SQL =<<SQL;
SELECT * FROM oscmax2.info2
SQL
$sth = $dbh->prepare($SQL);
if(!$sth) { die("Error: " . $dbh->errstr . "\n"); }
if(!$sth->execute) { die("Error4: " . $sth->errstr . "\n"); }
while (my #row = $sth->fetchrow_array) {
$products_id = $FORM{'product_id'};
$affiliate_id = $FORM{'affiliate_id'};
$demo = $FORM{'demo'};
}
if($sth->rows != 0) {
$total_rows = $sth->rows;
for ($counter = 0; $counter < $total_rows; $counter++) {
$SQL =<<SQL;
SELECT products_attributes_id FROM oscmax2.products_attributes
WHERE products_id = '$products_id'
SQL
$sth = $dbh->prepare($SQL);
if(!$sth) { die("Error: " . $dbh->errstr . "\n"); }
if(!$sth->execute) { die("Error: " . $sth->errstr . "\n");}
while (my #row = $sth->fetchrow_array) {
$products_attributes_id = $FORM2{'products_attributes_id'};
}
$SQL =<<SQL;
INSERT INTO oscmax2.products_attributes_download(
products_attributes_id, products_attributes_filename,
products_attributes_maxdays, products_attributes_maxcount
)
VALUES
('$products_attributes_id', '$affiliate_id/$demo', '7', '1')
SQL
$dbh->do($SQL) || die("Error5: " . $dbh->errstr . "\n");
}
}
$sth->finish();
if($sqlConnect) { $dbh->disconnect();
The blocks
while (my #row = $sth->fetchrow_array) {
$products_id = $FORM{'product_id'};
$affiliate_id = $FORM{'affiliate_id'};
$demo = $FORM{'demo'};
}
and
while (my #row = $sth->fetchrow_array) {
$products_attributes_id = $FORM2{'products_attributes_id'};
}
are wrong. You use #row to accept the data from each row of the result, but never use it. The database fetch won't affect %FORM and %FORM2 so you are just collecting the same data from them several times over
Update
At a guess you want something like this. Please study it and use the techniques rather than copying and testing it as it is, as I have no way of knowing what the structure of your database is and I have made several guesses
You should note the following points
There is no need to test the status of each DBI operation and die if it failed. By default the PrintError option is enabled and DBI will raise a warning if there are any errors. If you want your program to die instead, which is wise, then you can enable RaiseError and disable PrintError and DBI will do it all for you
There is no need to fetch all of the data from a table into memory (which I think is what you are trying to do with your while loops. You should fetch each row into an array and process the data row by row unless you have a reason to do otherwise
You should always prepare your statement and use placeholders. Then you can pass the actual parameters to the execute call and DBI will correctly quote them for you. Furthermore you can move all the prepare calls top the top of the program, making your logic much clearer to read
There is almost never a reason to call finish or disconnect. Perl will do the right thing for you when your database or statement handles go out of scope or your program ends
I have named the statement handles $select1 and $select2. These are very poor names, but I don't know the structure of your database so I couldn't write anything better. That shouyldn't stop you from improving them
I have had to guess at the columns returned by the first SELECT statement. If the three variables don't correspond to the first three elements of #row then you need to correct that
You should avoid using capital letters in Perl lexical identifiers. They are reserved for globals like package names, and nasty clashes of purpose can be caused if you don't abide by this rule
use strict;
use warnings;
my ($mysqldatabase, $mysqlhostname, $mysqlusername, $mysqlpassword) = qw/ dbase host user pass /;
my $dbh = DBI->connect(
"DBI:mysql:$mysqldatabase:$mysqlhostname",
"$mysqlusername",
"$mysqlpassword",
{RaiseError => 1, PrintError => 0}
) or die "Unable to connect to database: $DBI::errstr";
my $select1 = $dbh->prepare('SELECT * FROM oscmax2.info2');
my $select2 = $dbh->prepare(<<__END_SQL__);
SELECT products_attributes_id FROM oscmax2.products_attributes\
WHERE products_id = ?
__END_SQL__
my $insert = $dbh->prepare(<<__END_SQL__);
INSERT INTO oscmax2.products_attributes_download (
products_attributes_id,
products_attributes_filename,
products_attributes_maxdays,
products_attributes_maxcount
)
VALUES ( ?, ?, ?, ? )
__END_SQL__
$select1->execute;
while ( my #row = $select1->fetchrow_array ) {
my ($products_id, $affiliate_id, $demo) = #row;
$select2->execute($products_id);
while ( my #row = $select2->fetchrow_array ) {
my ($products_attributes_id) = #row;
$insert->execute($products_attributes_id, "$affiliate_id/$demo", 7, 1 );
}
}

Executing two queries in one perl script

i'm a newbie on perl scripting and i found a problem while trying to execute two sqls, here you have the code, for sure not the best one.
use DBI;
use DBD::mysql;
use Socket;
use strict;
use warnings;
# CONFIG VARIABLES
my $platform = 'mysql';
my $database = 'database_name';
my $host = 'hostname';
my $port = '3306';
my $user ='user';
my $pw ='password';
# DATA SOURCE NAME
my $dsn = "dbi:mysql:$database:$host:3306";
# PERL DBI CONNECT
my $dbh = DBI->connect($dsn,$user,$pw,{RaiseError=>1,PrintError=>1}) or die "Could not connect to database: $DBI::errstr";
# READ THE LASTID OF THE DATABASE
my $queryID = "SELECT event.id from snorby.event order by event.id desc limit 1";
my $lastid = $dbh->selectrow_array($queryID);
#HIGH
while ( 1 == 1 )
{
my $query = "SELECT event.id, inet_ntoa(iphdr.ip_src) as 'src', tcp_sport, inet_ntoa(iphdr.ip_dst) as 'dst', tcp_dport, signature.sig_name, event.timestamp, unhex(data.data_payload) from snorby.event join snorby.signature on signature.sig_id = event.signature join snorby.iphdr on event.cid=iphdr.cid and event.sid=iphdr.sid join snorby.data on event.cid=data.cid and event.sid=data.sid join snorby.tcphdr on event.cid=tcphdr.cid and event.sid=tcphdr.sid where event.id > $lastid and signature.sig_priority = '1' order by event.id";
my $sth = $dbh->prepare($query);
$sth->execute() or die "SQL Error: $DBI::errstr\n";
# BIND TABLE COLUMNS TO VARIABLES
my($eventid,$src,$sport,$dst,$dport,$signature,$timestamp,$payload);
$sth->bind_columns(undef, \$eventid, \$src, \$sport, \$dst, \$dport, \$signature, \$timestamp, \$payload);
# LOOP THROUGH RESULTS
while($sth->fetch) {
my $src_temp = inet_aton($src);
my $dst_temp = inet_aton($dst);
print "IT WORKS!";
}
So, if i comment this part of the code
# READ THE LASTID OF THE DATABASE
my $queryID = "SELECT event.id from snorby.event order by event.id desc limit 1";
my $lastid = $dbh->selectrow_array($queryID);
Everything works fine, but when i try to execute first this one, script stops responding exactly on this line:
while($sth->fetch) {
I tried to debug the code, look for tutorials, read a lot of pages and cannot figure where is the problem :(
Regards.
**** UPDATE ********
I think i found the problem after some more debug but not the solution. On the second sql named $query i passed the variable $lastid that i get on the first sql, see:
my $query = "SELECT stuff from table join things where event.id > **$lastid** and blablabla
If i change the $lastid for, as an example, 13330506, everything works, so seems to be that there is an issue about how this variable is passed. The strange thing is that when i print the $query with $lastid inside the content of $lastid is correct, the number appears... strange, at least for me.
If you read the documentation http://search.cpan.org/dist/DBI/DBI.pm
you'll see there is no ->fetch function, but there are various fetch methods:
#row_ary = $sth->fetchrow_array;
$ary_ref = $sth->fetchrow_arrayref;
$hash_ref = $sth->fetchrow_hashref;
$ary_ref = $sth->fetchall_arrayref;
$ary_ref = $sth->fetchall_arrayref( $slice, $max_rows );
$hash_ref = $sth->fetchall_hashref( $key_field );
Each one returns a reference you should store in variable for later use, for example:
while ( #row = $sth->fetchrow_array ) { ... }
while (my $data = $sth->fetchrow_hashref) { ... }
Then, you can use #row or $data inside the loop to retrieve the data you need.

Why am I not able to query a database from a forked child in Perl?

I have tried with Perl fork manager and DBI . But i got the error DBD::mysql::st execute failed: Lost connection to MySQL server during query .
Here the sample code: I want make query between low to high value (i have spitted int 10k records)
use Parallel::ForkManager;
my $pm = new Parallel::ForkManager(50);
my $db = krish::DB->new or die $!; # its has all connection details
while ( $low < $high ) {
# Some value manipulation
my $pid = $pm->start and next;
#db_execution returns execution
while ( my $sth = db_execution ( $db, $low , $high ) ) {
...
#fetch row operation
...
}
$pm->finish;
}
sub db_execution {
...
my $dbh = $db->connect( 'students' ) or die $!;
my $sth = $dbh->prepare( $sql ) or die "$!:" . $dbh->errstr;
$sth->execute or die "$!:" . $sth->errstr;
...
}
The same code is executing with out parallel processing. What is the issue?
How to resolve is this?
When you share database connections between processes (which is what you're doing with a fork) you need to make sure that one process doesn't close it out from under the other one. Because the connections are also variables, when the Perl interpreter shuts down it will call the DESTROY method of that object which in this case will close the connection.
So if any of the children close the db connection (which will happen when they finish and shutdown) it will kill it out from under the parent process. The way to prevent this is to set InactiveDestroy to true in the parent process before the fork and then to close the connection explicitly in the parent when done.
https://metacpan.org/pod/DBI#InactiveDestroy
You are asking for trouble by using the same db handle simultaneously in all of the child processes. You should be creating a new connection in each child.
Never mind...I read the rest of the code.