[Perl]read text file with delimiter and insert into mysql table in perl 5.10 - mysql

I have text file(employeedata.txt) with values like this :
ABC#VVV#JHY#ABC#VVV#JHY#ABC#VVV#JHY#ABC#VVV
BBN#NJU#NULL#ABC#VVV#JHY#ABC#VVV#JHY#ABC#OLJ
ABC#BYR#MPL#ABC#VVV#JHY#ABC#TGB#JHY#ABC#NULL
NMP#JDC#NULL#ABC#VVV#JHY#ABC#XCD#JHY#ABC#NULL
UJK#SAB#NULL#ABC#VVV#JHY#ABC#NBG#JHY#ABC#MPL
my text file contains 5,000 lines and I have a Table called Employee with values like this:
id|EmployeLastName|EmployeFirstName|EmployeeAddress
In my file text, in each line, i have EmployeLastName in the first position, EmployeFirstName in the fourth position, EmployeeAddress in the last position
Example :
EmployeLastName#VVV#JHY#EmployeFirstName#VVV#JHY#ABC#VVV#JHY#ABC#EmployeeAddress
Now I want to read text file line by line and insert into table Employee by using perl 5.10.
I am a novice in perl. How can do it?

Well you need to do some reading on DBI driver to get a grip on the code provided bellow -- it is best invested time to work with DB.
NOTE: in this piece of code I read data from internal block __DATA__
use strict;
use warnings;
use Data::Dumper;
use DBI;
my $debug = 0;
my #fields = qw(id last first address); # Field names in Database
my(%record,$rv);
my $hostname = 'db_server_1'; # Database server name
my $database = 'db_employees'; # Database name
my $table = 'db_table'; # Table name
my $port = '3306'; # Database port [default]
# Define DSN
my $dsn = "DBI:mysql:database=$database;host=$hostname;port=$port";
# Connect to Database
my $dbh = DBI->connect($dsn, $user, $password, {RaiseError => 1});
# Define query
my $stq = qq(INSERT INTO $table (id,last,first,address) VALUES(?,?,?,?););
# Prepare query
my $sth = $dbh->prepare($stq);
$dbh->begin_work(); # Ok, we will do insert in one transaction
my $skip = <DATA>; # We skip header in data block
while( <DATA> ) {
#record{#fields} = split /#/; # Fill the hash with record data
print Dumper(\%row) if $debug; # Look at hash in debug mode
$rv = $sth->execute(#record{#fields}); # Execute query with data
print $DBI::errstr if $rv < 0; # If error lets see ERROR message
}
$dbh->commit(); # Commit the transaction
$dbh->disconnect(); # Disconnect from DataBase
__DATA__
id#EmployeLastName#EmployeFirstName#EmployeeAddress
1#Alexander#Makedonsky#267 Mozarella st., Pizza, Italy
2#Vladimir#Lenin#12 Glinka st., Moscow, Italy
3#Donald#Trump#765 Tower ave., Florida, USA
4#Angela#Merkel#789 Schulstrafe st., Berlin, Germany
You can read data from a file with following code
use strict;
use warnings;
my $debug = 1;
my #fields = qw(id last first address); # Field names in Database
my(%record);
my $filename = shift
or die "Provide filename on command line";
open DATA, "< $filename"
or die "Could not open $filename";
while( <DATA> ) {
#record{#fields} = split /#/;
print Dumper(\%record) if $debug;
}
close DATA;
As you very fresh in Perl programming then you probably should start from Learning Perl, then move on Programming Perl, when you get in trouble visit Perl Cookbook and if you decided to dive into database programming Programming the Perl DBI

Well it is nice to know Perl and DBI driver programming. But in your particular case you could load data from a file directly by utilizing MySQL commands.
Loading Data into a Table
Sometimes it is much much easier than it looks at first sight.

Related

perl script to connect two databases at same time

I am trying to connect to 2 databases on the same instance of MySQL from 1 Perl script.
I am using this in a migration script where I am grabbing data from the original database and inserting it into the new one.
Connecting to 1 database and then trying to initiate a second connection with the same user just changes the current database to the new one.
#!/usr/bin/perl
use DBI;
use strict;
my $driver = "mysql";
my $database1 = "db1";
my $dsn1 = "DBI:$driver:database=$database1";
my $userid = "userhead";
my $password = "pwdhead";
my $database2 = "db2";
my $dsn2 = "DBI:$driver:database=$database2";
my $dbh1 = DBI->connect($dsn1, $userid, $password ) or die $DBI::errstr;
my $dbh2 = DBI->connect($dsn2, $userid, $password ) or die $DBI::errstr;
my $sth = $dbh2->prepare("INSERT INTO Persons") $dbh1->prepare("SELECT *FROM Persons");
$sth->execute() or die $DBI::errstr;
print "Number of rows found :" + $sth->rows;
In the above example i am trying to copy from one database table to another datbase table. but i am getting error while running the script. Please help me out
At a guess, you're trying to use the same database handle to connect to both databases. If you need to operate two separate connections then you need two separate handles
This program uses the data_sources class method to discover all of the available MySQL databases and creates a connection to each of them, putting the handles in the array #dbh. You can use each element of that array as normal, for instance
my $stmt = $dbh[0]->prepare('SELECT * FROM table)
It may be that you prefer to set up the #databases array manually, or the username and password may be different for the two data sources, so some variation on this may be necessary
use strict;
use warnings 'all';
use DBI;
my $user = 'username';
my $pass = 'password';
my #databases = DBI->data_sources('mysql');
my #dbh = map { DBI->connect($_, $user, $pass) } #databases;
Update
You need to select data from the source table, fetch it one row at a time, and insert each row into the destination table
Here's an idea how that might work, but you need to adjust the number of question marks in the VALUES of the INSERT statement to match the number of columns
Note that, if you're just intending to copy the whole dataset, there aree better ways to go about this. In particular, if you have any foreign key constraints then you won't be able to add data until the table it it is dependent on is populated
#!/usr/bin/perl
use strict;
use warnings 'all';
use DBI;
my $userid = "userhead";
my $password = "pwdhead";
my ($dbase1, $dbase2) = qw/ db1 db2 /;
my $dsn1 = "DBI:mysql:database=$dbase1";
my $dsn2 = "DBI:mysql:database=$dbase2";
my $dbh1 = DBI->connect($dsn1, $userid, $password ) or die $DBI::errstr;
my $dbh2 = DBI->connect($dsn2, $userid, $password ) or die $DBI::errstr;
my $select = $dbh1->prepare("SELECT * FROM Persons");
my $insert = $dbh2->prepare("INSERT INTO Persons VALUES (?, ?, ?, ?, ?)");
$select->execute;
while ( my #row = $select->fetchrow_array ) {
$insert->execute(#row);
}
If you need to handle the columns from the source data separately then you can use named scalars instead of the array #row. Like this
while ( my ($id, $name) = $select->fetchrow_array ) {
my $lastname = '';
$insert->execute($id, $name, $lastname);
}
Plan A (especially if one-time task):
Run mysqldump on the source machine; feed the output to mysql on the target machine. This will be much faster and simpler. If you are on a Unix machine, do it with an exec() from Perl (if you like).
Plan B (especially if repeated task):
If the table is not "too big", do one SELECT to fetch all the rows into an array in Perl. Then INSERT the rows into the target machine. This can be sped up (with some effort) if you build a multi-row INSERT or create a CSV file and use LOAD DATA instead of INSERT.

How can I load a text file into MySQL using Perl?

Can anyone tell me how to read data from a text file and store it in a MySQL database using Perl?
For example:
StudentID Name Dept
1 Chellappa IT
2 Vijay CSE
3 AAA ECE
99.9% of the time, the best approach for this kind of thing is to use the native bulk load tools that are specific to the target database. In this case, that would be LOAD DATA INFILE or its command line equivalent, mysqlimport.
We can use Perl to massage the data into the correct format:
$ perl -wlne 'next if $. == 1; s/\s+/\t/g; print;' input.txt > output.txt
And then use mysqlimport to load it:
$ mysqlimport [options] db_name output.txt
If your example really is as simple as what you posted, you could actually load your file as-is just by specifying some additional options for mysqlimport:
$ mysqlimport --ignore-lines=1 --fields-terminated-by=' ' [options ...] db_name input.txt
Use DBI module to connect the database. Read input file and split the value and store into an array. Now insert the array values into database.
Here is a way (please note this code is not tested):
#!/usr/bin/perl
use warnings;
use strict;
use DBI;
my $dsn = "DBI:mysql:host=hostname; database=dbname; port=portname";
# database connection
my $dbh = DBI->connect($dsn,"username","password") or die "Couldn't connect to MySQL server: $!";
my $query = 'INSERT INTO tableName (StudentID,Name,Dept) VALUES (?,?,?)';
my $sth = $dbh->prepare($query) or die "Prepare failed: " . $dbh->errstr();
open my $fh, "<", "file.txt" or die $!;
<$fh>; #skip header
while (<$fh>)
{
chomp;
my #vals = split;
$sth->execute(#vals);
}
close $fh;

perl tcp socket server to mysql

i have to create a little perl script which is creating an tcp socket and put the input from this socket into an mysql table.
Background: I get Call Data Records from a Siemens Hipath phone system via this TCP Socket.
I have to write the data, which is CSV in an DB for future use.
The format i get is following: 13.05.14;15:01:14;3;10;00:26;00:00:45;0123456789;;1;
Im new to perl, an have a few "noob" questions :)
1: How is it possible to run that script in background (as deamon) ?
2: The Script is only handling the last line of an deliverd csv line. If there are two lines, it ignores the first line. How can i fix that ?
3: Today i got this output from my script: DBD::mysql::st execute failed: MySQL server has gone away at ./hipath-CDR-Server.pl line 41. What happend there?
Can anyone help me with that maybe easy questions.
This is what i got till now:
#!/usr/bin/perl
use strict;
use warnings;
use DBI;
use IO::Socket::INET;
my $dbh = DBI->connect('DBI:mysql:database=hipathCDR;host=localhost','***','***',{ RaiseError => 1, AutoCommit => 1 }, );
my $sql = 'INSERT INTO calls (Datum,Zeit,Leitung,Teilnehmer,Rufzeit,Gespraechszeit,RufNr,Gebuehr,Typ) VALUES (?,?,?,?,?,?,?,?,?)';
my $sth = $dbh->prepare($sql);
# auto-flush on socket
$| = 1;
# creating a listening socket
my $socket = new IO::Socket::INET (
LocalHost => '0.0.0.0',
LocalPort => '4444',
Proto => 'tcp',
Listen => 5,
Reuse => 1
);
die "cannot create socket $!\n" unless $socket;
while(1)
{
# waiting for a new client connection
my $client_socket = $socket->accept();
# get information about a newly connected client
my $client_address = $client_socket->peerhost();
my $client_port = $client_socket->peerport();
# read up to 1024 characters from the connected client
my $data = "";
$client_socket->recv($data, 1024);
chomp ($data);
my #values = split(';', $data);
print "Array: #values\n";
$sth->execute(#values);
# write response data to the connected client
$data = "ok";
$client_socket->send($data);
# notify client that response has been sent
shutdown($client_socket, 1);
}
$socket->close();
You have more than one question, I am going to answer the second: Here is howto get all data.
my $data = "";
while ($client_socket->recv($data, 1024)){
$data .= $_;
}
It would more safer to parse the incoming CSV like data by Text::CSV.
Answer to question three:
Most probably your mysql was unavailable that time.
Answer to question one:
nohup ./script.pl &
Okay, I am just kidding: How can I run a Perl script as a system daemon in linux?
use Proc::Daemon;
Proc::Daemon::Init;
my $continue = 1;
$SIG{TERM} = sub { $continue = 0 };
...
while($continue)
{
...
}

Perl param() receiving from its own print HTML

I have a Perl script that reads in data from a database and prints out the result in HTML forms/tables. The form of each book also contains a submit button.
I want Perl to create a text file (or read into one already created) and print the title of the book that was inside the form submitted. But I can't seem to get param() to catch the submit action!
#!/usr/bin/perl -w
use warnings; # Allow for warnings to be sent if error's occur
use CGI; # Include CGI.pm module
use DBI;
use DBD::mysql; # Database data will come from mysql
my $dbh = DBI->connect('DBI:mysql:name?book_store', 'name', 'password')
or die("Could not make connection to database: $DBI::errstr"); # connect to the database with address and pass or return error
my $q = new CGI; # CGI object for basic stuff
my $ip = $q->remote_host(); # Get the user's ip
my $term = $q->param('searchterm'); # Set the search char to $term
$term =~ tr/A-Z/a-z/; # set all characters to lowercase for convenience of search
my $sql = '
SELECT *
FROM Books
WHERE Title LIKE ?
OR Description LIKE ?
OR Author LIKE ?
'; # Set the query string to search the database
my $sth = $dbh->prepare($sql); # Prepare to connect to the database
$sth->execute("%$term%", "%$term%", "%$term%")
or die "SQL Error: $DBI::errstr\n"; # Connect to the database or return an error
print $q->header;
print "<html>";
print "<body>";
print " <form name='book' action='bookcart.php' method=post> "; # Open a form for submitting the result of book selection
print "<table width=\"100%\" border=\"0\"> ";
my $title = $data[0];
my $desc = $data[1];
my $author = $data[2];
my $pub = $data[3];
my $isbn = $data[4];
my $photo = $data[5];
print "<tr> <td width=50%>Title: $title</td> <td width=50% rowspan=5><img src=$photo height=300px></td></tr><tr><td>Discreption Tags: $desc</td></tr><tr><td>Publication Date: $pub</td></tr><tr><td>Author: $author</td></tr><tr><td>ISBN: $isbn</td> </tr></table> <br>";
print "Add this to shopping cart:<input type='submit' name='submit' value='Add'>";
if ($q->param('submit')) {
open(FILE, ">>'$ip'.txt");
print FILE "$title\n";
close(FILE);
}
print "</form>"; # Close the form for submitting to shopping cart
You haven't used use strict, to force you to declare all your variables. This is a bad idea
You have used remote_host, which is the name of the client host system. Your server may not be able to resolve this value, in which case it will remain unset. If you want the IP address, use remote_addr
You have prepared and executed your SQL statement but have fetched no data from the query. You appear to expect the results to be in the array #data, but you haven't declared this array. You would have been told about this had you had use strict in effect
You have used the string '$ip'.txt for your file names so, if you were correctly using the IP address in stead of the host name, your files would look like '92.17.182.165'.txt. Do you really want the single quotes in there?
You don't check the status of your open call, so you have no idea whether the open succeeded, or the reason why it may have failed
I doubt if you have really spent the last 48 hours coding this. I think it is much more likely that you are throwing something together in a rush at the last minute, and using Stack Overflow to help you out of the hole you have dug for yourself.
Before asking for the aid of others you should at least use minimal good-practice coding methods such as applying use strict. You should also try your best to debug your code: it would have taken very little to find that $ip has the wrong value and #data is empty.
Use strict and warnings. You want to use strict for many reasons. A decent article on this is over at perlmonks, you can begin with this. Using strict and warnings
You don't necessarily need the following line, you are using DBI and can access mysql strictly with DBI.
use DBD::mysql;
Many of options are available with CGI, I would recommend reading the perldoc on this also based on user preferences and desired wants and needs.
I would not use the following:
my $q = new CGI;
# I would use as so..
my $q = CGI->new;
Use remote_addr instead of remote_host to retrieve your ip address.
The following line you are converting all uppercase to lowercase, unless it's a need to specifically read from your database with all lowercase, I find this useless.
$term =~ tr/A-Z/a-z/;
Next your $sql line, again user preference, but I would look into sprintf or using it directly inside your calls. Also you are trying to read an array of data that does not exist, where is the call to get back your data? I recommend reading the documentation for DBI also, many methods of returning your data. So you want your data back using an array for example...
Here is an untested example and hint to help get you started.
use strict;
use warnings;
use CGI qw( :standard );
use CGI::Carp qw( fatalsToBrowser ); # Track your syntax errors
use DBI;
# Get IP Address
my $ip = $ENV{'REMOTE_ADDR'};
# Get your query from param,
# I would also parse your data here
my $term = param('searchterm') || undef;
my $dbh = DBI->connect('DBI:mysql:db:host', 'user', 'pass',
{RaiseError => 1}) or die $DBI::errstr;
my $sql = sprintf ('SELECT * FROM Books WHERE Title LIKE %s
OR Description LIKE %s', $term, $term);
my $sth = $dbh->selectall_arrayref( $sql );
# Retrieve your result data from array ref and turn into
# a hash that has title for the key and a array ref to the data.
my %rows = ();
for my $i ( 0..$#{$sth} ) {
my ($title, $desc, $author, $pub, $isbn, $pic) = #{$sth->[$i]};
$rows{$title} = [ $desc, $author, $pub, $isbn, $pic ];
}
# Storing your table/column names
# in an array for mapping later.
my #cols;
$cols[0] = Tr(th('Title'), th('Desc'), th('Author'),
th('Published'), th('ISBN'), th('Photo'));
foreach (keys %rows) {
push #cols, Tr( td($_),
td($rows{$_}->[0]),
td($rows{$_}->[1]),
td($rows{$_}->[2]),
td($rows{$_}->[3]),
td(img({-src => $rows{$_}->[4]}));
}
print header,
start_html(-title => 'Example'),
start_form(-method => 'POST', -action => 'bookcart.php'), "\n",
table( {-border => undef, -width => '100%'}, #cols ),
submit(-name => 'Submit', -value => 'Add Entry'),
end_form,
end_html;
# Do something with if submit is clicked..
if ( param('Submit') ) {
......
}
This assumes that you're using the OO approach to CGI.pm, and that $q is the relevant object. This should work, assuming that you have $q = new CGI somewhere in your script.
Can you post the rest of the script?
I've created a mockup to test this, and it works as expected:
#!/usr/bin/perl
use CGI;
my $q = new CGI;
print $q->header;
print "<form><input type=submit name=submit value='add'></form>\n";
if ($q->param('submit')) {
print "submit is \"" . $q->param('submit') . "\"\n";
}
After the submit button is clicked, the page displays that submit is "add" which means the evaluation is going as planned.
I guess what you need to do is make sure that $q is your CGI object, and move forward from there.

Perl Import large .csv to MySQL, don't repeat data

I am trying to import several .csv files into a mysql database, the script below works except that it only imports the first row of my csv data into the database. Both my tables are populated with exactly one data entry.
Any help would be appreciated.
Thank you
#!/usr/bin/perl
use DBI;
use DBD::mysql;
use strict;
use warnings;
# MySQL CONFIG VARIABLES
my $host = "localhost";
my $user = "someuser";
my $pw = "somepassword";
my $database = "test";
my $dsn = "DBI:mysql:database=" . $database . ";host=" . $host;
my $dbh = DBI->connect($dsn, $user, $pw)
or die "Can't connect to the DB: $DBI::errstr\n";
print "Connected to DB!\n";
# enter the file name that you want import
my $filename = "/home/jonathan/dep/csv/linux_datetime_test_4.26.13_.csv";
open FILE, "<", $filename or die $!;
$_ = <FILE>;
$_ = <FILE>;
while (<FILE>) {
my #f = split(/,/,$_);
if (length($f[4]) < 10) {
print "No Weight\n";
}
else {
#insert the data into the db
print "insert into datetime_stamp\n";
}
my $sql = "INSERT INTO datetime_stamp (subject, date, time, weight)
VALUES('$f[1]', '$f[2]', '$f[3]', '$f[4]')";
print "$sql\n";
my $query = $dbh->do($sql);
my $sql = "INSERT INTO subj_weight (subject, weight) VALUES('$f[1]', '$f[2]')";
my $query = $dbh->do($sql);
close(FILE);
}
As has been commented, you close the input file after reading the first data entry, and so only populate your database with a single record.
However there are a few problems with your code you may want to consider:
You should set autoflush on the STDOUT file handle if you are printing diagnostics as the program runs. Otherwise perl won't print the output until either it has a buffer full of text to print or the file handle is closed when the program exits. That means you may not see the messages you have coded until long after the event
You should use Text::CSV to parse CSV data instead of relying on split
You can interpolate variables into a double-quoted string. That avoids the use of several concatenation operators and makes the intention clearer
Your open is near-perfect - an unusual thing - because you correctly use the three-parameter form of open as well as testing whether it succeeded and putting $! in the die string. However you should also always use a lexical file handle as well instead of the old-fashioned global ones
You don't chomp the lines you read from the input, so the last field will have a trailing newline. Using Text::CSV avoids the need for this
You use indices 1 through 4 of the data split from the input record. Perl indices start at zero, so that means you are droppping the first field. Is that correct?
Similarly you are inserting fields 1 and 2, which appear to be subject and date, into fields called subject and weight. It seems unlikely that this can be right
You should prepare your SQL statements, use placeholders, and provide the actual data in an execute call
You seem to diagnose the data read from the file ("No Weight") but insert the data into the database anyway. This may be correct but it seems unlikely
Here is a version of your program that includes these amendments. I hope it is of use to you.
#!/usr/bin/perl
use strict;
use warnings;
use DBI;
use Text::CSV;
use IO::Handle;
STDOUT->autoflush;
# MySQL config variables
my $host = "localhost";
my $user = "someuser";
my $pw = "somepassword";
my $database = "test";
my $dsn = "DBI:mysql:database=$database;host=$host";
my $dbh = DBI->connect($dsn, $user, $pw)
or die "Can't connect to the DB: $DBI::errstr\n";
print "Connected to DB!\n";
my $filename = "/home/jonathan/dep/csv/linux_datetime_test_4.26.13_.csv";
open my $fh, '<', $filename
or die qq{Unable to open "$filename" for input: $!};
my $csv = Text::CSV->new;
$csv->getline($fh) for 1, 2; # Drop header lines
my $insert_datetime_stamp = $dbh->prepare( 'INSERT INTO datetime_stamp (subject, date, time, weight) VALUES(?, ?, ?, ?)' );
my $insert_subj_weight = $dbh->prepare( 'INSERT INTO subj_weight (subject, weight) VALUES(?, ?)' );
while (my $row = $csv->getline($fh)) {
if (length($row->[4]) < 10) {
print qq{Invalid weight: "$row->[4]"\n};
}
else {
#insert the data into the db
print "insert into datetime_stamp\n";
$insert_datetime_stamp->execute(#$row[1..4]);
$insert_subj_weight->execute(#$row[1,4]);
}
}