how to store text file information into my sql database - mysql

this is my text file
StudentId Name Dept address city
1 Chellappa CSE 22 xx-colony 2nd street coimbatore
2 Vijay IT 23 yy colony coimbatore
In this file (22 xx-colony 2nd street) and (23 yy colony) is address to store in database address column
use DBI;
use strict;
my $driver = "mysql";
my $database = "TESTDB";
my $dsn = "DBI:$driver:database=$database";
my $userid = "root";
my $password = "1234";
my $dbh = DBI->connect($dsn, $userid, $password ) or die $DBI::errstr;
my $query = 'INSERT INTO student (StudentId,Name,Dept,address,city) VALUES (?,?,?,?,?)';
my $sth = $dbh->prepare($query) or die "Prepare failed: " . $dbh->errstr();
open my $fh, "<", "text.txt" or die $!;
<$fh>; #skip header
while (<$fh>)
{
chomp;
my #vals = split;
$sth->execute(#vals);
}
close $fh;
this code is doesn't work properly for that text file anyone can help me

DBD::mysql::st execute failed: called with 8 bind variables when 4 are
needed
That error is not generated by the code that you have shown us. Your code contains this:
my $query = 'INSERT INTO student (StudentId,Name,Dept,address,city) VALUES (?,?,?,?,?)';
That SQL query contains five bind points, not four. It's always important to be accurate when reporting these things.
Anyway, the important thing is that the number of bind points is different to the number of values you pass to execute(). And it's pretty simple to see where that problem comes from.
Your first data line is this:
1 Chellappa CSE 22 xx-colony 2nd street coimbatore
And you split that data into the #vals array with this code:
my #vals = split;
With no arguments, split() splits $_ on whitespace (you can read the documentation online).
So, after running that code on that data, you'll end up with eight values in #vals. Those values are:
1, Chellappa, CSE, 22, xx-colony, 2nd, street, coimbatore
So it's clear that your simple split() isn't processing your data in the way that you expect. You'll need to come up with a more sophisticated way to extract your five expected data items from your input record.
Unfortunately, you don't give us any detail on how your input file is structured, so we can't give you any more help on how to fix your problem.
Update: Guessing at what you might want here, the update statement could become something like:
$sth->update(#vals[0 .. 2], join ' ', #vals[3..6], $vals[7]);
But I have no idea how well that will work with other lines of data that might have the whitespace in other places.

Yes it can be done using join(). Since you have all the values separated by space you are getting this problem. If your file is created by you then you can just add comma separated values like (1, Chellappa, CSE, 22 xx-colony 2nd street,coimbatore). This will help you to fetch all the data using split as follows
split(',',).

Related

[Perl]read text file with delimiter and insert into mysql table in perl 5.10

I have text file(employeedata.txt) with values like this :
ABC#VVV#JHY#ABC#VVV#JHY#ABC#VVV#JHY#ABC#VVV
BBN#NJU#NULL#ABC#VVV#JHY#ABC#VVV#JHY#ABC#OLJ
ABC#BYR#MPL#ABC#VVV#JHY#ABC#TGB#JHY#ABC#NULL
NMP#JDC#NULL#ABC#VVV#JHY#ABC#XCD#JHY#ABC#NULL
UJK#SAB#NULL#ABC#VVV#JHY#ABC#NBG#JHY#ABC#MPL
my text file contains 5,000 lines and I have a Table called Employee with values like this:
id|EmployeLastName|EmployeFirstName|EmployeeAddress
In my file text, in each line, i have EmployeLastName in the first position, EmployeFirstName in the fourth position, EmployeeAddress in the last position
Example :
EmployeLastName#VVV#JHY#EmployeFirstName#VVV#JHY#ABC#VVV#JHY#ABC#EmployeeAddress
Now I want to read text file line by line and insert into table Employee by using perl 5.10.
I am a novice in perl. How can do it?
Well you need to do some reading on DBI driver to get a grip on the code provided bellow -- it is best invested time to work with DB.
NOTE: in this piece of code I read data from internal block __DATA__
use strict;
use warnings;
use Data::Dumper;
use DBI;
my $debug = 0;
my #fields = qw(id last first address); # Field names in Database
my(%record,$rv);
my $hostname = 'db_server_1'; # Database server name
my $database = 'db_employees'; # Database name
my $table = 'db_table'; # Table name
my $port = '3306'; # Database port [default]
# Define DSN
my $dsn = "DBI:mysql:database=$database;host=$hostname;port=$port";
# Connect to Database
my $dbh = DBI->connect($dsn, $user, $password, {RaiseError => 1});
# Define query
my $stq = qq(INSERT INTO $table (id,last,first,address) VALUES(?,?,?,?););
# Prepare query
my $sth = $dbh->prepare($stq);
$dbh->begin_work(); # Ok, we will do insert in one transaction
my $skip = <DATA>; # We skip header in data block
while( <DATA> ) {
#record{#fields} = split /#/; # Fill the hash with record data
print Dumper(\%row) if $debug; # Look at hash in debug mode
$rv = $sth->execute(#record{#fields}); # Execute query with data
print $DBI::errstr if $rv < 0; # If error lets see ERROR message
}
$dbh->commit(); # Commit the transaction
$dbh->disconnect(); # Disconnect from DataBase
__DATA__
id#EmployeLastName#EmployeFirstName#EmployeeAddress
1#Alexander#Makedonsky#267 Mozarella st., Pizza, Italy
2#Vladimir#Lenin#12 Glinka st., Moscow, Italy
3#Donald#Trump#765 Tower ave., Florida, USA
4#Angela#Merkel#789 Schulstrafe st., Berlin, Germany
You can read data from a file with following code
use strict;
use warnings;
my $debug = 1;
my #fields = qw(id last first address); # Field names in Database
my(%record);
my $filename = shift
or die "Provide filename on command line";
open DATA, "< $filename"
or die "Could not open $filename";
while( <DATA> ) {
#record{#fields} = split /#/;
print Dumper(\%record) if $debug;
}
close DATA;
As you very fresh in Perl programming then you probably should start from Learning Perl, then move on Programming Perl, when you get in trouble visit Perl Cookbook and if you decided to dive into database programming Programming the Perl DBI
Well it is nice to know Perl and DBI driver programming. But in your particular case you could load data from a file directly by utilizing MySQL commands.
Loading Data into a Table
Sometimes it is much much easier than it looks at first sight.

Inserting Data From Perl To MysQL

I have already connected MySQL to the database but the problem is that
I have some problems with inserting Data from Perl to Mysql.
The error that pops out is:
Use of uninitialized value in concatenation (.) or string at ./etl_server_info.pl line 204, (Which is the connection of the database )
DBD::mysql::st execute failed: called with 1 bind variables when 0 are needed at ($stmt->execute($sql);)
sub insert_record(){
my($data,$db_config)=#_;
my $result = -1; # -1 fail; 0 - succ
# connect to db
# connect to MySQL database
my $dsn = "DBI:mysql:database=".$db_config->{'Top_Data'}.";host=".$db_config->{'127.0.0.1'}.";port=".$db_config->{'3306'};
my $username = $db_config->{'username'};
my $password = $db_config->{'password'};
my %attr = (PrintError=>0,RaiseError=>1 );
my $dbh = DBI->connect($dsn,$username,$password,\%attr) or die $DBI::errstr;
print "We Have Successfully Connected To The Database \n";
# prepare sql statement
# execute insert
my $sql = 'insert into Top(Load_Average, CPU_us, CPU_id, CPU-wa, CPU_hi, CPU_si, CPU_st, Swap_Total, Swap_Free, Swap_Used, Memory_Total, Memeory_Free, Memory_Used, Memory_Buff, Date) values(float,float,float,float,float,float,float,float,varchar,varchar,varchar,varchar,varchar,varchar,date)';
my $stmt =$dbh->prepare($sql) or die "Its A Fail" . $dbh->errstr();
$stmt->execute($sql);
$stmt->finish();
$dbh->disconnect();
$result = 0;
return($result);
Your use of the $db_config variable looks suspicious to me. Either your config hash is strange, or you're using values instead of keys.
You haven't shown us where $db_config is set up, but I'd guess it looks something like this:
$db_config = {
name => 'Top_Data',
host => '127.0.0.1',
port => 3306,
username => 'someone',
password => 'a secret',
};
And then you would use it like this:
my $dsn = "DBI:mysql:database=".$db_config->{name}.";host=".$db_config->{host}.";port=".$db_config->{port};
Notice that I've used the key names (name, host and port) instead of the values (Top_Data, 127.0.0.1 and 3306).
I'll also point out that you can simplify this slightly by using Perl's ability to expand variables inside a double-quoted string.
my $dsn = "DBI:mysql:database=$db_config->{name};host=$db_config->{host};port=$db_config->{port}";
There's another problem later on, with your SQL statement.
my $sql = 'insert into Top(Load_Average, CPU_us, CPU_id, CPU-wa, CPU_hi,
CPU_si, CPU_st, Swap_Total, Swap_Free, Swap_Used, Memory_Total,
Memeory_Free, Memory_Used, Memory_Buff, Date)
values(float,float,float,float,float,float,float,float,
varchar,varchar,varchar,varchar,varchar,varchar,date)';
The values that you should be inserting are the actual data items. So where you have the strings "float", "varchar" or "date", you should actually have data items (a floating-point number, a string or a date).
Finally, having prepared your statement, you don't need to pass it to the execute() method. You should, however, look at using bind points in your SQL and passing your actual data items to the execute() call

perl script to connect two databases at same time

I am trying to connect to 2 databases on the same instance of MySQL from 1 Perl script.
I am using this in a migration script where I am grabbing data from the original database and inserting it into the new one.
Connecting to 1 database and then trying to initiate a second connection with the same user just changes the current database to the new one.
#!/usr/bin/perl
use DBI;
use strict;
my $driver = "mysql";
my $database1 = "db1";
my $dsn1 = "DBI:$driver:database=$database1";
my $userid = "userhead";
my $password = "pwdhead";
my $database2 = "db2";
my $dsn2 = "DBI:$driver:database=$database2";
my $dbh1 = DBI->connect($dsn1, $userid, $password ) or die $DBI::errstr;
my $dbh2 = DBI->connect($dsn2, $userid, $password ) or die $DBI::errstr;
my $sth = $dbh2->prepare("INSERT INTO Persons") $dbh1->prepare("SELECT *FROM Persons");
$sth->execute() or die $DBI::errstr;
print "Number of rows found :" + $sth->rows;
In the above example i am trying to copy from one database table to another datbase table. but i am getting error while running the script. Please help me out
At a guess, you're trying to use the same database handle to connect to both databases. If you need to operate two separate connections then you need two separate handles
This program uses the data_sources class method to discover all of the available MySQL databases and creates a connection to each of them, putting the handles in the array #dbh. You can use each element of that array as normal, for instance
my $stmt = $dbh[0]->prepare('SELECT * FROM table)
It may be that you prefer to set up the #databases array manually, or the username and password may be different for the two data sources, so some variation on this may be necessary
use strict;
use warnings 'all';
use DBI;
my $user = 'username';
my $pass = 'password';
my #databases = DBI->data_sources('mysql');
my #dbh = map { DBI->connect($_, $user, $pass) } #databases;
Update
You need to select data from the source table, fetch it one row at a time, and insert each row into the destination table
Here's an idea how that might work, but you need to adjust the number of question marks in the VALUES of the INSERT statement to match the number of columns
Note that, if you're just intending to copy the whole dataset, there aree better ways to go about this. In particular, if you have any foreign key constraints then you won't be able to add data until the table it it is dependent on is populated
#!/usr/bin/perl
use strict;
use warnings 'all';
use DBI;
my $userid = "userhead";
my $password = "pwdhead";
my ($dbase1, $dbase2) = qw/ db1 db2 /;
my $dsn1 = "DBI:mysql:database=$dbase1";
my $dsn2 = "DBI:mysql:database=$dbase2";
my $dbh1 = DBI->connect($dsn1, $userid, $password ) or die $DBI::errstr;
my $dbh2 = DBI->connect($dsn2, $userid, $password ) or die $DBI::errstr;
my $select = $dbh1->prepare("SELECT * FROM Persons");
my $insert = $dbh2->prepare("INSERT INTO Persons VALUES (?, ?, ?, ?, ?)");
$select->execute;
while ( my #row = $select->fetchrow_array ) {
$insert->execute(#row);
}
If you need to handle the columns from the source data separately then you can use named scalars instead of the array #row. Like this
while ( my ($id, $name) = $select->fetchrow_array ) {
my $lastname = '';
$insert->execute($id, $name, $lastname);
}
Plan A (especially if one-time task):
Run mysqldump on the source machine; feed the output to mysql on the target machine. This will be much faster and simpler. If you are on a Unix machine, do it with an exec() from Perl (if you like).
Plan B (especially if repeated task):
If the table is not "too big", do one SELECT to fetch all the rows into an array in Perl. Then INSERT the rows into the target machine. This can be sped up (with some effort) if you build a multi-row INSERT or create a CSV file and use LOAD DATA instead of INSERT.

save each row of a database as a seperate text file in a directory using perl?

I have a database table which is holding various column and rows of data.
I want to search one of the columns for keywords. the Column is called ''products".
I want to run the query on the table for each keyword and return the results for all rows that contains the keyword.
The results for each row should be printed to a seperate file, each file hould take the name as data value contained one of the column of that row (in this case item_id)
I want this done in a loop until all "products" column is searched for all rows of the table.
I have the following code which prints to screen, but all matching rows and columns is returned on the screen at once i,e Not separately to file as i wished (pls read (3) above).
also please note the output formating which is how i want it, i,e seperate lines e.t.c
can anyone help ?
#!/usr/bin/perl
use DBI;
#product = qw( orange apple); ###an array created for products
$count = 1;
# Connect to the database
details not provided
my $dbh = DBI->connect("DBI:mysql:$db;host=$db_host",$mysql_user, $mysql_passwd)
|| die "ERROR: Connecting: $DBI::errstr\n";
print "db connected\n";
foreach $product(#product)
{
# Prepare the SQL query for execution
my $sth = $dbh->prepare(<<End_SQL)
or die "Couldn't prepare statement: $DBI::errstr; stopped";
SELECT item_id, published, modified, availability, summary,
product, reference FROM $my_table WHERE product LIKE '%$product%'
End_SQL
# Execute the query
$sth->execute() or die "Couldn't execute statement: $DBI::errstr; stopped";
# Fetch each row and print it
while ( my ($item_id, $published, $modified, $availability, $summary, $product,
$reference) = $sth->fetchrow_array() )
{
print STDOUT ("ITEM ID: $item_id\n Product: $product\n Published:
$published Modified:$modified\n Availability: $availability\n Summary: $summary\n
Reference: $reference\n");};
$count++;
}
# Disconnect from the database
$dbh->disconnect();
Instead of print STDOUT, you must create a file and write to that
open(my $f, ">$item_id") or die "Cannot create file $item_id: $!";
print $f "ITEM ID: $item_id\n Product: $product\n Published: $published Modified:$modified\n Availability: $availability\n Summary: $summary\n Reference: $reference\n";
close($f);

Perl Import large .csv to MySQL, don't repeat data

I am trying to import several .csv files into a mysql database, the script below works except that it only imports the first row of my csv data into the database. Both my tables are populated with exactly one data entry.
Any help would be appreciated.
Thank you
#!/usr/bin/perl
use DBI;
use DBD::mysql;
use strict;
use warnings;
# MySQL CONFIG VARIABLES
my $host = "localhost";
my $user = "someuser";
my $pw = "somepassword";
my $database = "test";
my $dsn = "DBI:mysql:database=" . $database . ";host=" . $host;
my $dbh = DBI->connect($dsn, $user, $pw)
or die "Can't connect to the DB: $DBI::errstr\n";
print "Connected to DB!\n";
# enter the file name that you want import
my $filename = "/home/jonathan/dep/csv/linux_datetime_test_4.26.13_.csv";
open FILE, "<", $filename or die $!;
$_ = <FILE>;
$_ = <FILE>;
while (<FILE>) {
my #f = split(/,/,$_);
if (length($f[4]) < 10) {
print "No Weight\n";
}
else {
#insert the data into the db
print "insert into datetime_stamp\n";
}
my $sql = "INSERT INTO datetime_stamp (subject, date, time, weight)
VALUES('$f[1]', '$f[2]', '$f[3]', '$f[4]')";
print "$sql\n";
my $query = $dbh->do($sql);
my $sql = "INSERT INTO subj_weight (subject, weight) VALUES('$f[1]', '$f[2]')";
my $query = $dbh->do($sql);
close(FILE);
}
As has been commented, you close the input file after reading the first data entry, and so only populate your database with a single record.
However there are a few problems with your code you may want to consider:
You should set autoflush on the STDOUT file handle if you are printing diagnostics as the program runs. Otherwise perl won't print the output until either it has a buffer full of text to print or the file handle is closed when the program exits. That means you may not see the messages you have coded until long after the event
You should use Text::CSV to parse CSV data instead of relying on split
You can interpolate variables into a double-quoted string. That avoids the use of several concatenation operators and makes the intention clearer
Your open is near-perfect - an unusual thing - because you correctly use the three-parameter form of open as well as testing whether it succeeded and putting $! in the die string. However you should also always use a lexical file handle as well instead of the old-fashioned global ones
You don't chomp the lines you read from the input, so the last field will have a trailing newline. Using Text::CSV avoids the need for this
You use indices 1 through 4 of the data split from the input record. Perl indices start at zero, so that means you are droppping the first field. Is that correct?
Similarly you are inserting fields 1 and 2, which appear to be subject and date, into fields called subject and weight. It seems unlikely that this can be right
You should prepare your SQL statements, use placeholders, and provide the actual data in an execute call
You seem to diagnose the data read from the file ("No Weight") but insert the data into the database anyway. This may be correct but it seems unlikely
Here is a version of your program that includes these amendments. I hope it is of use to you.
#!/usr/bin/perl
use strict;
use warnings;
use DBI;
use Text::CSV;
use IO::Handle;
STDOUT->autoflush;
# MySQL config variables
my $host = "localhost";
my $user = "someuser";
my $pw = "somepassword";
my $database = "test";
my $dsn = "DBI:mysql:database=$database;host=$host";
my $dbh = DBI->connect($dsn, $user, $pw)
or die "Can't connect to the DB: $DBI::errstr\n";
print "Connected to DB!\n";
my $filename = "/home/jonathan/dep/csv/linux_datetime_test_4.26.13_.csv";
open my $fh, '<', $filename
or die qq{Unable to open "$filename" for input: $!};
my $csv = Text::CSV->new;
$csv->getline($fh) for 1, 2; # Drop header lines
my $insert_datetime_stamp = $dbh->prepare( 'INSERT INTO datetime_stamp (subject, date, time, weight) VALUES(?, ?, ?, ?)' );
my $insert_subj_weight = $dbh->prepare( 'INSERT INTO subj_weight (subject, weight) VALUES(?, ?)' );
while (my $row = $csv->getline($fh)) {
if (length($row->[4]) < 10) {
print qq{Invalid weight: "$row->[4]"\n};
}
else {
#insert the data into the db
print "insert into datetime_stamp\n";
$insert_datetime_stamp->execute(#$row[1..4]);
$insert_subj_weight->execute(#$row[1,4]);
}
}