perl script to connect two databases at same time - mysql

I am trying to connect to 2 databases on the same instance of MySQL from 1 Perl script.
I am using this in a migration script where I am grabbing data from the original database and inserting it into the new one.
Connecting to 1 database and then trying to initiate a second connection with the same user just changes the current database to the new one.
#!/usr/bin/perl
use DBI;
use strict;
my $driver = "mysql";
my $database1 = "db1";
my $dsn1 = "DBI:$driver:database=$database1";
my $userid = "userhead";
my $password = "pwdhead";
my $database2 = "db2";
my $dsn2 = "DBI:$driver:database=$database2";
my $dbh1 = DBI->connect($dsn1, $userid, $password ) or die $DBI::errstr;
my $dbh2 = DBI->connect($dsn2, $userid, $password ) or die $DBI::errstr;
my $sth = $dbh2->prepare("INSERT INTO Persons") $dbh1->prepare("SELECT *FROM Persons");
$sth->execute() or die $DBI::errstr;
print "Number of rows found :" + $sth->rows;
In the above example i am trying to copy from one database table to another datbase table. but i am getting error while running the script. Please help me out

At a guess, you're trying to use the same database handle to connect to both databases. If you need to operate two separate connections then you need two separate handles
This program uses the data_sources class method to discover all of the available MySQL databases and creates a connection to each of them, putting the handles in the array #dbh. You can use each element of that array as normal, for instance
my $stmt = $dbh[0]->prepare('SELECT * FROM table)
It may be that you prefer to set up the #databases array manually, or the username and password may be different for the two data sources, so some variation on this may be necessary
use strict;
use warnings 'all';
use DBI;
my $user = 'username';
my $pass = 'password';
my #databases = DBI->data_sources('mysql');
my #dbh = map { DBI->connect($_, $user, $pass) } #databases;
Update
You need to select data from the source table, fetch it one row at a time, and insert each row into the destination table
Here's an idea how that might work, but you need to adjust the number of question marks in the VALUES of the INSERT statement to match the number of columns
Note that, if you're just intending to copy the whole dataset, there aree better ways to go about this. In particular, if you have any foreign key constraints then you won't be able to add data until the table it it is dependent on is populated
#!/usr/bin/perl
use strict;
use warnings 'all';
use DBI;
my $userid = "userhead";
my $password = "pwdhead";
my ($dbase1, $dbase2) = qw/ db1 db2 /;
my $dsn1 = "DBI:mysql:database=$dbase1";
my $dsn2 = "DBI:mysql:database=$dbase2";
my $dbh1 = DBI->connect($dsn1, $userid, $password ) or die $DBI::errstr;
my $dbh2 = DBI->connect($dsn2, $userid, $password ) or die $DBI::errstr;
my $select = $dbh1->prepare("SELECT * FROM Persons");
my $insert = $dbh2->prepare("INSERT INTO Persons VALUES (?, ?, ?, ?, ?)");
$select->execute;
while ( my #row = $select->fetchrow_array ) {
$insert->execute(#row);
}
If you need to handle the columns from the source data separately then you can use named scalars instead of the array #row. Like this
while ( my ($id, $name) = $select->fetchrow_array ) {
my $lastname = '';
$insert->execute($id, $name, $lastname);
}

Plan A (especially if one-time task):
Run mysqldump on the source machine; feed the output to mysql on the target machine. This will be much faster and simpler. If you are on a Unix machine, do it with an exec() from Perl (if you like).
Plan B (especially if repeated task):
If the table is not "too big", do one SELECT to fetch all the rows into an array in Perl. Then INSERT the rows into the target machine. This can be sped up (with some effort) if you build a multi-row INSERT or create a CSV file and use LOAD DATA instead of INSERT.

Related

Inserting Data From Perl To MysQL

I have already connected MySQL to the database but the problem is that
I have some problems with inserting Data from Perl to Mysql.
The error that pops out is:
Use of uninitialized value in concatenation (.) or string at ./etl_server_info.pl line 204, (Which is the connection of the database )
DBD::mysql::st execute failed: called with 1 bind variables when 0 are needed at ($stmt->execute($sql);)
sub insert_record(){
my($data,$db_config)=#_;
my $result = -1; # -1 fail; 0 - succ
# connect to db
# connect to MySQL database
my $dsn = "DBI:mysql:database=".$db_config->{'Top_Data'}.";host=".$db_config->{'127.0.0.1'}.";port=".$db_config->{'3306'};
my $username = $db_config->{'username'};
my $password = $db_config->{'password'};
my %attr = (PrintError=>0,RaiseError=>1 );
my $dbh = DBI->connect($dsn,$username,$password,\%attr) or die $DBI::errstr;
print "We Have Successfully Connected To The Database \n";
# prepare sql statement
# execute insert
my $sql = 'insert into Top(Load_Average, CPU_us, CPU_id, CPU-wa, CPU_hi, CPU_si, CPU_st, Swap_Total, Swap_Free, Swap_Used, Memory_Total, Memeory_Free, Memory_Used, Memory_Buff, Date) values(float,float,float,float,float,float,float,float,varchar,varchar,varchar,varchar,varchar,varchar,date)';
my $stmt =$dbh->prepare($sql) or die "Its A Fail" . $dbh->errstr();
$stmt->execute($sql);
$stmt->finish();
$dbh->disconnect();
$result = 0;
return($result);
Your use of the $db_config variable looks suspicious to me. Either your config hash is strange, or you're using values instead of keys.
You haven't shown us where $db_config is set up, but I'd guess it looks something like this:
$db_config = {
name => 'Top_Data',
host => '127.0.0.1',
port => 3306,
username => 'someone',
password => 'a secret',
};
And then you would use it like this:
my $dsn = "DBI:mysql:database=".$db_config->{name}.";host=".$db_config->{host}.";port=".$db_config->{port};
Notice that I've used the key names (name, host and port) instead of the values (Top_Data, 127.0.0.1 and 3306).
I'll also point out that you can simplify this slightly by using Perl's ability to expand variables inside a double-quoted string.
my $dsn = "DBI:mysql:database=$db_config->{name};host=$db_config->{host};port=$db_config->{port}";
There's another problem later on, with your SQL statement.
my $sql = 'insert into Top(Load_Average, CPU_us, CPU_id, CPU-wa, CPU_hi,
CPU_si, CPU_st, Swap_Total, Swap_Free, Swap_Used, Memory_Total,
Memeory_Free, Memory_Used, Memory_Buff, Date)
values(float,float,float,float,float,float,float,float,
varchar,varchar,varchar,varchar,varchar,varchar,date)';
The values that you should be inserting are the actual data items. So where you have the strings "float", "varchar" or "date", you should actually have data items (a floating-point number, a string or a date).
Finally, having prepared your statement, you don't need to pass it to the execute() method. You should, however, look at using bind points in your SQL and passing your actual data items to the execute() call

Why does DBI implicitly change integers to strings?

I have a MySQL table with following structure.
alid bigint(20),
ndip varchar(20),
ndregion varchar(20),
occ_num int(3),
Delta_Flag int(1)
After selecting data from the table, I am getting all the data quoted and as a string value.
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
use FindBin;
use lib $FindBin::Bin;
use Database;
my $pwd = $FindBin::Bin;
my $db = Database->new( 'mysql', "$pwd/config.ini" );
my $db1 = Database->new( 'mysql', "$pwd/config2.ini" );
my #tables = qw( AutoTT_AlarmStatus_Major1 );
for my $table ( #tables ) {
my $query_select = "SELECT alid, ndip, ndregion, occ_num, Delta_Flag FROM $table LIMIT 1";
my $result = $db->db_get_results( $query_select );
print Dumper( $result );
for my $item ( #{$result} ) {
# Here I want to prepare, bind and insert this data
# into other table with same structure
}
}
Database.pm
sub db_get_results {
my $self = shift;
my $qry = shift;
my $sth = $self->{dbh}->prepare( $qry );
$sth->execute();
my #return = ();
while ( my #line = $sth->fetchrow_array ) {
push #return, \#line;
}
return \#return;
}
Output:
$VAR1 = [
[
'1788353',
'10.34.38.12',
'North Central',
'1',
'1'
]
];
Why is DBI implicitly converting all integers to strings?
As #choroba notes in his answer, it's not the DBI that's doing anything with the data. It's just passing through what the driver module (DBD::mysql in your case) returned.
In the General Interface Rules & Caveats section of the DBI docs it says:
Most data is returned to the Perl script as strings. (Null values are returned as undef.) This allows arbitrary precision numeric data to be handled without loss of accuracy. Beware that Perl may not preserve the same accuracy when the string is used as a number.
I wrote that back in the days before it was common to configure perl to support 64-bit integers, and long-double floating point types were unusual. These days I recommend that drivers return values in the most 'natural' Perl type that doesn't risk data loss.
For some drivers that can be tricky to implement, especially those that support returning multiple result sets, with different numbers of columns, from a single handle, as DBD::mysql does.
I skimmed the DBD::mysql docs but didn't see any mention of this topic, so I looked at the relevant code where I can see that the current DBD::mysql is returning numbers as numbers. There's also lots of references to recent changes in this area in the Change log.
Perhaps you're using an old version of DBD::mysql and should upgrade.
That's how the DBD driver for MySQL works. Other databases might behave differently. For example, in SQLite, numbers remain numeric:
#!/usr/bin/perl
use warnings;
use strict;
use DBI;
use Data::Dumper;
my $dbh = 'DBI'->connect('dbi:SQLite:dbname=:memory:', q(), q());
$dbh->do('CREATE TABLE t (id INT, val VARCHAR(10))');
my $insert = $dbh->prepare('INSERT INTO t VALUES (?, ?)');
$insert->execute(#$_) for [ 1, 'foo' ], [ 2, 'bar' ];
my $query = $dbh->prepare('SELECT id, val FROM t');
$query->execute;
while (my $row = $query->fetchrow_arrayref) {
print Dumper($row);
}
__END__
$VAR1 = [
1,
'foo'
];
$VAR1 = [
2,
'bar'
];
This is nothing DBI does in general. As it already was pointed out many database-drivers (DBD::xy) of the DBI system convert numbers to strings. AFAIK its not possible to avoid that.
What you can do is ask the statement handle for corresponding native type or (easier in your case) wether the column of your resultset is numeric in the mysql-DB or not. Here is an example
Given this basic database:
mysql> create table test (id INT,text VARCHAR(20));
Query OK, 0 rows affected (0.01 sec)
mysql> INSERT INTO test VALUES (1,'lalala');
Query OK, 1 row affected (0.00 sec)
you can lookup wether the column is numeric or not by using the driver-specific field 'mysql_is_num':
Reference to an array of boolean values; TRUE indicates, that the respective column contains numeric values.
(from DBD::mysql)
#!/usr/bin/env perl
use strict;
use warnings;
use utf8;
use DBI;
use DBD::mysql;
use Data::Dumper;
my $dsn = "DBI:mysql:database=test;host=localhost";
my $dbh = DBI->connect($dsn,'user','pass') or die "$!";
my $sql = "SELECT * FROM test WHERE id = ?";
my $sth = $dbh->prepare($sql);
$sth->execute(1);
my $num_fields = $sth->{'NUM_OF_FIELDS'};
my $num_mask = $sth->{'mysql_is_num'};
my $result;
my $cnt = 0;
while (my $line = $sth->fetchrow_arrayref){
for (my $i = 0; $i < $num_fields; $i++){
if ($num_mask->[$i]){
$line->[$i] + 0;
}
$result->[$cnt] = $line;
}
$cnt++;
}
print Dumper($result);
I hope this helps. As it was written in a hurry, please excuse the style. Of course i'm open to any suggestions.

MySQL disconnect from server for new "prepare"

I'm working on a Perl script which executes few queries on SQL server.
For example:
my $sth = $dbh->prepare ( "select x y z .. );
$sth->execute();
# do some code
my $sth = $dbh->prepare ( "select a b c .. );
$sth->execute();
# do some code
# etc..
It is recommended to disconnect from the server explicitly $dbh->disconnect.
Do I need to disconnect after each execution ? Or it is enough to disconnect only once when script is finished ?
From the documentation:
Disconnects the database from the database handle. disconnect is typically only used before exiting the program. The handle is of little use after disconnecting.
So no.
Disconnecting just means you have to reconnect. It would be a waste of resources.
No, you do not need to disconnect every time. That would be very contra-productive.
You can just fire as many queries as you want. But take into account that you are redeclaring $sth if you use my every time.
Typically if there are several tables that you need to fill, it would look something like this.
use strict;
use warnings;
use DBI;
my $dsn = '...';
my $dbh = DBI->connect($dsn);
my $sth_employee =
$dbh->prepare('select name, title, department_id from employee');
my $sth_department =
$dbh->prepare('select name from department where department_id=?');
$sth_employee->execute;
while (my $res_employee = $sth_employee->fetchrow_hashref) {
$sth_department->execute($res_employee->{department_id});
( my $department ) = $sth_department->fetchrow_array;
say $res_employee->{name};
say "$res_employee->{title}, $department";
say '=' x 10;
}
# no need to explicitly finnish handles or disconnect, DBI does that for us

Update MySQL within Perl loop failing (fetchrow_array)

I've created a Perl script which is meant to loop through an array (a shortlist of customers who meet certain criteria), execute an external command using system() , then update a field within each row once the operation has completed.
It works on the first record (ie external command executes, customer record updates), however when it gets to the second record I receive this error:
DBD::mysql::st fetchrow_array failed: fetch() without execute() at customer_update.pl
Through some googling I added the $sth->finish(); command, however whether I include it or not (either inside the loop as shown, or straight afterward) I still get the same error.
Can anyone shed any light for me as to what I am doing wrong here?
Here's an extract:
# PERL MYSQL CONNECT()
$dbh = DBI->connect('dbi:mysql:signups', $user, $pw)
or die "Connection Error: $DBI::errstr\n";
# DEFINE A MySQL QUERY
$myquery = "SELECT * FROM accounts WHERE field3 = false";
$sth = $dbh->prepare($myquery);
# EXECUTE THE QUERY
$sth->execute
or die "SQL Error: $DBI::errstr\n";
#records = $sth->rows;
print "Amount of new customers: #records\n\n";
while ( my ($field1, $field2, $field3) = $sth->fetchrow_array() ) {
#execute external command via system();
$update_customer_status = "UPDATE accounts SET field3=true WHERE id=$id";
$sth = $dbh->prepare($update_customer_status);
$sth->execute
or die "SQL Error: $DBI::errstr\n";
print "Customer record modified & MySQL updated accordingly\n\n";
$sth->finish();
}
Building a SQL statement with variables and then prepare()ing it defeats the purpose of the prepare. You should build the SQL statement with a placeholder ? instead of $id, prepare() it, and then execute($id) it. As it is, you are leaving yourself open to SQL injection attacks.
Also, it seems that you are not using the warnings and strict pragmas. These two lines should be at the top of every program you write:
use warnings;
use strict;
They will save you much heartache and frustration in the future.
In your loop, you overwrite the handle over from which you are fetching. Use a different variable. (Changing $sth = ...; to my $sth = ...; will do.) While we're at it, let's move the prepare out of the loop.
my $sth_get = $dbh->prepare("SELECT * FROM accounts WHERE field3 = false");
my $sth_upd = $dbh->prepare("UPDATE accounts SET field3=true WHERE id = ?");
$sth_get->execute();
while ( my ($field1, $field2, $field3) = $sth_get->fetchrow_array() ) {
...
$sth_upd->execute($id);
}
You are stomping on your $sth variable when you execute this line ...
$sth = $dbh->prepare($update_customer_status);
Why not save off the result of $sth->fetchrow_array() to an array variable.
Something like ...
my #select_results_AoA = $sth->fetchrow_array();
... and then iterate over the array ...
for my #row ( #select_resilts_AoA ) {
... instead of ...
while ( my ($field1, $field2, $field3) = $sth->fetchrow_array() ) {

Perl Import large .csv to MySQL, don't repeat data

I am trying to import several .csv files into a mysql database, the script below works except that it only imports the first row of my csv data into the database. Both my tables are populated with exactly one data entry.
Any help would be appreciated.
Thank you
#!/usr/bin/perl
use DBI;
use DBD::mysql;
use strict;
use warnings;
# MySQL CONFIG VARIABLES
my $host = "localhost";
my $user = "someuser";
my $pw = "somepassword";
my $database = "test";
my $dsn = "DBI:mysql:database=" . $database . ";host=" . $host;
my $dbh = DBI->connect($dsn, $user, $pw)
or die "Can't connect to the DB: $DBI::errstr\n";
print "Connected to DB!\n";
# enter the file name that you want import
my $filename = "/home/jonathan/dep/csv/linux_datetime_test_4.26.13_.csv";
open FILE, "<", $filename or die $!;
$_ = <FILE>;
$_ = <FILE>;
while (<FILE>) {
my #f = split(/,/,$_);
if (length($f[4]) < 10) {
print "No Weight\n";
}
else {
#insert the data into the db
print "insert into datetime_stamp\n";
}
my $sql = "INSERT INTO datetime_stamp (subject, date, time, weight)
VALUES('$f[1]', '$f[2]', '$f[3]', '$f[4]')";
print "$sql\n";
my $query = $dbh->do($sql);
my $sql = "INSERT INTO subj_weight (subject, weight) VALUES('$f[1]', '$f[2]')";
my $query = $dbh->do($sql);
close(FILE);
}
As has been commented, you close the input file after reading the first data entry, and so only populate your database with a single record.
However there are a few problems with your code you may want to consider:
You should set autoflush on the STDOUT file handle if you are printing diagnostics as the program runs. Otherwise perl won't print the output until either it has a buffer full of text to print or the file handle is closed when the program exits. That means you may not see the messages you have coded until long after the event
You should use Text::CSV to parse CSV data instead of relying on split
You can interpolate variables into a double-quoted string. That avoids the use of several concatenation operators and makes the intention clearer
Your open is near-perfect - an unusual thing - because you correctly use the three-parameter form of open as well as testing whether it succeeded and putting $! in the die string. However you should also always use a lexical file handle as well instead of the old-fashioned global ones
You don't chomp the lines you read from the input, so the last field will have a trailing newline. Using Text::CSV avoids the need for this
You use indices 1 through 4 of the data split from the input record. Perl indices start at zero, so that means you are droppping the first field. Is that correct?
Similarly you are inserting fields 1 and 2, which appear to be subject and date, into fields called subject and weight. It seems unlikely that this can be right
You should prepare your SQL statements, use placeholders, and provide the actual data in an execute call
You seem to diagnose the data read from the file ("No Weight") but insert the data into the database anyway. This may be correct but it seems unlikely
Here is a version of your program that includes these amendments. I hope it is of use to you.
#!/usr/bin/perl
use strict;
use warnings;
use DBI;
use Text::CSV;
use IO::Handle;
STDOUT->autoflush;
# MySQL config variables
my $host = "localhost";
my $user = "someuser";
my $pw = "somepassword";
my $database = "test";
my $dsn = "DBI:mysql:database=$database;host=$host";
my $dbh = DBI->connect($dsn, $user, $pw)
or die "Can't connect to the DB: $DBI::errstr\n";
print "Connected to DB!\n";
my $filename = "/home/jonathan/dep/csv/linux_datetime_test_4.26.13_.csv";
open my $fh, '<', $filename
or die qq{Unable to open "$filename" for input: $!};
my $csv = Text::CSV->new;
$csv->getline($fh) for 1, 2; # Drop header lines
my $insert_datetime_stamp = $dbh->prepare( 'INSERT INTO datetime_stamp (subject, date, time, weight) VALUES(?, ?, ?, ?)' );
my $insert_subj_weight = $dbh->prepare( 'INSERT INTO subj_weight (subject, weight) VALUES(?, ?)' );
while (my $row = $csv->getline($fh)) {
if (length($row->[4]) < 10) {
print qq{Invalid weight: "$row->[4]"\n};
}
else {
#insert the data into the db
print "insert into datetime_stamp\n";
$insert_datetime_stamp->execute(#$row[1..4]);
$insert_subj_weight->execute(#$row[1,4]);
}
}