perl tcp socket server to mysql - mysql

i have to create a little perl script which is creating an tcp socket and put the input from this socket into an mysql table.
Background: I get Call Data Records from a Siemens Hipath phone system via this TCP Socket.
I have to write the data, which is CSV in an DB for future use.
The format i get is following: 13.05.14;15:01:14;3;10;00:26;00:00:45;0123456789;;1;
Im new to perl, an have a few "noob" questions :)
1: How is it possible to run that script in background (as deamon) ?
2: The Script is only handling the last line of an deliverd csv line. If there are two lines, it ignores the first line. How can i fix that ?
3: Today i got this output from my script: DBD::mysql::st execute failed: MySQL server has gone away at ./hipath-CDR-Server.pl line 41. What happend there?
Can anyone help me with that maybe easy questions.
This is what i got till now:
#!/usr/bin/perl
use strict;
use warnings;
use DBI;
use IO::Socket::INET;
my $dbh = DBI->connect('DBI:mysql:database=hipathCDR;host=localhost','***','***',{ RaiseError => 1, AutoCommit => 1 }, );
my $sql = 'INSERT INTO calls (Datum,Zeit,Leitung,Teilnehmer,Rufzeit,Gespraechszeit,RufNr,Gebuehr,Typ) VALUES (?,?,?,?,?,?,?,?,?)';
my $sth = $dbh->prepare($sql);
# auto-flush on socket
$| = 1;
# creating a listening socket
my $socket = new IO::Socket::INET (
LocalHost => '0.0.0.0',
LocalPort => '4444',
Proto => 'tcp',
Listen => 5,
Reuse => 1
);
die "cannot create socket $!\n" unless $socket;
while(1)
{
# waiting for a new client connection
my $client_socket = $socket->accept();
# get information about a newly connected client
my $client_address = $client_socket->peerhost();
my $client_port = $client_socket->peerport();
# read up to 1024 characters from the connected client
my $data = "";
$client_socket->recv($data, 1024);
chomp ($data);
my #values = split(';', $data);
print "Array: #values\n";
$sth->execute(#values);
# write response data to the connected client
$data = "ok";
$client_socket->send($data);
# notify client that response has been sent
shutdown($client_socket, 1);
}
$socket->close();

You have more than one question, I am going to answer the second: Here is howto get all data.
my $data = "";
while ($client_socket->recv($data, 1024)){
$data .= $_;
}
It would more safer to parse the incoming CSV like data by Text::CSV.
Answer to question three:
Most probably your mysql was unavailable that time.
Answer to question one:
nohup ./script.pl &
Okay, I am just kidding: How can I run a Perl script as a system daemon in linux?
use Proc::Daemon;
Proc::Daemon::Init;
my $continue = 1;
$SIG{TERM} = sub { $continue = 0 };
...
while($continue)
{
...
}

Related

[Perl]read text file with delimiter and insert into mysql table in perl 5.10

I have text file(employeedata.txt) with values like this :
ABC#VVV#JHY#ABC#VVV#JHY#ABC#VVV#JHY#ABC#VVV
BBN#NJU#NULL#ABC#VVV#JHY#ABC#VVV#JHY#ABC#OLJ
ABC#BYR#MPL#ABC#VVV#JHY#ABC#TGB#JHY#ABC#NULL
NMP#JDC#NULL#ABC#VVV#JHY#ABC#XCD#JHY#ABC#NULL
UJK#SAB#NULL#ABC#VVV#JHY#ABC#NBG#JHY#ABC#MPL
my text file contains 5,000 lines and I have a Table called Employee with values like this:
id|EmployeLastName|EmployeFirstName|EmployeeAddress
In my file text, in each line, i have EmployeLastName in the first position, EmployeFirstName in the fourth position, EmployeeAddress in the last position
Example :
EmployeLastName#VVV#JHY#EmployeFirstName#VVV#JHY#ABC#VVV#JHY#ABC#EmployeeAddress
Now I want to read text file line by line and insert into table Employee by using perl 5.10.
I am a novice in perl. How can do it?
Well you need to do some reading on DBI driver to get a grip on the code provided bellow -- it is best invested time to work with DB.
NOTE: in this piece of code I read data from internal block __DATA__
use strict;
use warnings;
use Data::Dumper;
use DBI;
my $debug = 0;
my #fields = qw(id last first address); # Field names in Database
my(%record,$rv);
my $hostname = 'db_server_1'; # Database server name
my $database = 'db_employees'; # Database name
my $table = 'db_table'; # Table name
my $port = '3306'; # Database port [default]
# Define DSN
my $dsn = "DBI:mysql:database=$database;host=$hostname;port=$port";
# Connect to Database
my $dbh = DBI->connect($dsn, $user, $password, {RaiseError => 1});
# Define query
my $stq = qq(INSERT INTO $table (id,last,first,address) VALUES(?,?,?,?););
# Prepare query
my $sth = $dbh->prepare($stq);
$dbh->begin_work(); # Ok, we will do insert in one transaction
my $skip = <DATA>; # We skip header in data block
while( <DATA> ) {
#record{#fields} = split /#/; # Fill the hash with record data
print Dumper(\%row) if $debug; # Look at hash in debug mode
$rv = $sth->execute(#record{#fields}); # Execute query with data
print $DBI::errstr if $rv < 0; # If error lets see ERROR message
}
$dbh->commit(); # Commit the transaction
$dbh->disconnect(); # Disconnect from DataBase
__DATA__
id#EmployeLastName#EmployeFirstName#EmployeeAddress
1#Alexander#Makedonsky#267 Mozarella st., Pizza, Italy
2#Vladimir#Lenin#12 Glinka st., Moscow, Italy
3#Donald#Trump#765 Tower ave., Florida, USA
4#Angela#Merkel#789 Schulstrafe st., Berlin, Germany
You can read data from a file with following code
use strict;
use warnings;
my $debug = 1;
my #fields = qw(id last first address); # Field names in Database
my(%record);
my $filename = shift
or die "Provide filename on command line";
open DATA, "< $filename"
or die "Could not open $filename";
while( <DATA> ) {
#record{#fields} = split /#/;
print Dumper(\%record) if $debug;
}
close DATA;
As you very fresh in Perl programming then you probably should start from Learning Perl, then move on Programming Perl, when you get in trouble visit Perl Cookbook and if you decided to dive into database programming Programming the Perl DBI
Well it is nice to know Perl and DBI driver programming. But in your particular case you could load data from a file directly by utilizing MySQL commands.
Loading Data into a Table
Sometimes it is much much easier than it looks at first sight.

How to optimize this script performing INSERTS into a database?

So i already complete a script that will insert data into mysql table and move those file into a directory until all files are none. There around 51 files and it took around 9 sec to complete the execution. So my question is . is there a better way to speed up the execution process?
the codes are
our $DIR="/home/aimanhalim/LOG";
our $FILENAME_REGEX = "server_performance_";
# mariaDB config hash
our %db_config = ( "username"=>"root", "password"=> "", "db"=>"Top_Data", "ip" => "127.0.0.1", "port" => "3306");
main();
exit;
sub main()
{
my $start = time();
print "Searching file $FILENAME_REGEX in $DIR...\n";
opendir (my $dr , $DIR) or die "<ERROR> Cannot open dir: $DIR \n";
while( my $file = readdir $dr )
{
print "file in $DIR: [$file]\n";
next if (($file eq ".") || ($file eq "..") || ($file eq "DONE"));
#Opening The File in the directory
open(my $file_hndlr, "<$DIR/$file");
#Making Variables.
my $line_count = 0;
my %data = ();
my $dataRef = \%data;
my $move = "$DIR/$file";
print "$file\n";
while (<$file_hndlr>)
{
my $line = $_;
chomp($line);
print "line[$line_count] - [$line]\n";
if($line_count == 0)
{
# get load average from line 0
($dataRef) = get_load_average($line,$dataRef);
print Dumper($dataRef);
}
elsif ($line_count == 2)
{
($dataRef) = get_Cpu($line,$dataRef);
print Dumper($dataRef);
}
$line_count++;
}
#insert db
my ($result) = insert_record($dataRef,\%db_config,$file);
my $Done_File="/home/aimanhalim/LOG/DONE";
sub insert_record(){
my($data,$db_config,$file)=#_;
my $result = -1; # -1 fail; 0 - succ
# connect to db
# connect to MySQL database
my $dsn = "DBI:mysql:database=".$db_config->{'db'}.";host=".$db_config->{'ip'}.";port=".$db_config->{'port'};
my $username = $db_config->{'username'};
my $password = $db_config->{'password'};
my %attr = (PrintError=>0,RaiseError=>1 );
my $dbh = DBI->connect($dsn,$username,$password,\%attr) or die $DBI::errstr;
print "We Have Successfully Connected To The Database \n";
$stmt->execute(#param_bind);
****this line is insert data statement***
$stmt->finish();
print "The Data Has Been Inserted Successfully\n";
$result = 0;
return($result);
# commit
$dbh->commit();
# return succ / if fail rollback and return fail
$dbh->disconnect();
}
exit;
editted
so pretty much this is my code with some sniping here and there.
i tried to put the 'insert_record' below the comment #insert db but i dont think that do anything :U
You are connecting to the database for every file that you want to insert (if I read your code correctly, there seems to be a closing curly brace missing, it won't actually compile). Opening new database connections is (comparably) slow.
Open the connection once, before inserting the first file and re-use it for subsequent inserts into the database. Close the connection after your last file was inserted into the database. This should give you a noticable speed up.
(Depending on the amount of data, 9 seconds might actually not be too bad; but since there is no information on that, it's hard to say).

Inserting Data From Perl To MysQL

I have already connected MySQL to the database but the problem is that
I have some problems with inserting Data from Perl to Mysql.
The error that pops out is:
Use of uninitialized value in concatenation (.) or string at ./etl_server_info.pl line 204, (Which is the connection of the database )
DBD::mysql::st execute failed: called with 1 bind variables when 0 are needed at ($stmt->execute($sql);)
sub insert_record(){
my($data,$db_config)=#_;
my $result = -1; # -1 fail; 0 - succ
# connect to db
# connect to MySQL database
my $dsn = "DBI:mysql:database=".$db_config->{'Top_Data'}.";host=".$db_config->{'127.0.0.1'}.";port=".$db_config->{'3306'};
my $username = $db_config->{'username'};
my $password = $db_config->{'password'};
my %attr = (PrintError=>0,RaiseError=>1 );
my $dbh = DBI->connect($dsn,$username,$password,\%attr) or die $DBI::errstr;
print "We Have Successfully Connected To The Database \n";
# prepare sql statement
# execute insert
my $sql = 'insert into Top(Load_Average, CPU_us, CPU_id, CPU-wa, CPU_hi, CPU_si, CPU_st, Swap_Total, Swap_Free, Swap_Used, Memory_Total, Memeory_Free, Memory_Used, Memory_Buff, Date) values(float,float,float,float,float,float,float,float,varchar,varchar,varchar,varchar,varchar,varchar,date)';
my $stmt =$dbh->prepare($sql) or die "Its A Fail" . $dbh->errstr();
$stmt->execute($sql);
$stmt->finish();
$dbh->disconnect();
$result = 0;
return($result);
Your use of the $db_config variable looks suspicious to me. Either your config hash is strange, or you're using values instead of keys.
You haven't shown us where $db_config is set up, but I'd guess it looks something like this:
$db_config = {
name => 'Top_Data',
host => '127.0.0.1',
port => 3306,
username => 'someone',
password => 'a secret',
};
And then you would use it like this:
my $dsn = "DBI:mysql:database=".$db_config->{name}.";host=".$db_config->{host}.";port=".$db_config->{port};
Notice that I've used the key names (name, host and port) instead of the values (Top_Data, 127.0.0.1 and 3306).
I'll also point out that you can simplify this slightly by using Perl's ability to expand variables inside a double-quoted string.
my $dsn = "DBI:mysql:database=$db_config->{name};host=$db_config->{host};port=$db_config->{port}";
There's another problem later on, with your SQL statement.
my $sql = 'insert into Top(Load_Average, CPU_us, CPU_id, CPU-wa, CPU_hi,
CPU_si, CPU_st, Swap_Total, Swap_Free, Swap_Used, Memory_Total,
Memeory_Free, Memory_Used, Memory_Buff, Date)
values(float,float,float,float,float,float,float,float,
varchar,varchar,varchar,varchar,varchar,varchar,date)';
The values that you should be inserting are the actual data items. So where you have the strings "float", "varchar" or "date", you should actually have data items (a floating-point number, a string or a date).
Finally, having prepared your statement, you don't need to pass it to the execute() method. You should, however, look at using bind points in your SQL and passing your actual data items to the execute() call

RLY-8 POE relay using json

just got the DFrobot relay and trying to communicate with it using json commands
pdf is here
any help please how to send an on/off command from terminal?
That is pretty easy, just open a telnet to the device, an example:
telnet 192.168.1.10 2000
and copy paste one of the JSON commands from their pdf document, like:
{"relay1":"on","relay2":"on","relay3":"off","relay4":"off","relay5":"off", "relay6":"off","relay7":"off","relay8":"off"}
and hit enter. In case if you want more than telnet commands you can check this Github project.
please check : https://github.com/Dzduino/DFRobot-RLY-8-Web-Control/blob/master/index.php
You can use PHP Socket like bellow in an html or PHP page, please note that you have to use some local php server (Ex: XAMPP) to test it:
<?php
$addr = "192.168.1.10"; // RLY-8 Default Adress
$port = 2000; // RLY-8 Default port
$timeout = 30; // Connection Time out in Sec
if (isset($_POST["cmd"])){ // check if a submit was done, otherwise the communicatino will start after page loading
$cmd = $_POST["cmd"] ; // Capture the input Command
$fp = fsockopen ($addr, $port, $errno, $errstr, $timeout ); // initiate a socket connection
if (!$fp) {
echo "($errno) $errstr\n"; // return the error if no connection was established
} else {
fwrite ($fp, $cmd); // Send the command to the connected device
echo fread($fp, 128); // Echo the return string from the device
fclose ($fp); // close the connection
}
}
?>

Perl module to log in database and do a simple query?

I can't seem to figure out the issue with my .pm file and script. I am fairly new to Perl.
I have a database with name "project" and there is table with name "mailing".
mailing table has 7 entries, which I want to display using module.
So, I have this custom module to log in to database and do a query. This module is names as DB.pm
DB.pm is stored on my FEDORA 20 at /root/mysql/GUI/DB.pm.
DB.pm is defined as follows:
package GUI::DB;
use strict;
use DBI;
use vars qw(#ISA #EXPORT);
use Exporter;
#ISA = qw(Exporter);
#EXPORT = qw(dbConnect query);
#
# dbConnect - connect to the database, get the database handle
#
sub dbConnect {
# Read database settings from config file:
print "Works";
my $dsn = "DBI:mysql:project";
my $dbh = DBI->connect( $dsn,
'root',
'mydatabasepassword',
{ RaiseError => 1 }
);
return $dbh;
}
#
# query - execute a query with parameters
# query($dbh, $sql, #bindValues)
#
sub query {
my $dbh = shift;
my $sql ="SELECT * FROM mailing";
my #bindValues = #_; # 0 or serveral parameters
my #returnData = ();
# issue query
my $sth = $dbh->prepare($sql); //**line number 39 that is giving** error
if ( #bindValues ) {
$sth->execute(#bindValues);
} else {
$sth->execute();
}
if ( $sql =~ m/^select/i ) {
while ( my $row = $sth->fetchrow_hashref ) {
push #returnData, $row;
}
}
# finish the sql statement
$sth->finish();
return #returnData;
}
1;
Now I want to use this module inside my per script. This is what I tried:
#!/usr/bin/perl
use warnings;
use strict;
use lib '/root/mysql/';
use GUI::DB qw(dbConnect query);
dbConnect();
query();
This is the error I'm getting -->
Can't call method "prepare" on an undefined value at /root/mysql/GUI/DB.pm line 39.
Please help me with this. I am not sure how to proceed. I am guessing it has something to do with argument passing. Nothing is wrong with database. It works fine from CLI.
Thanks :)
_x_X_X_X_X_X_X_X_X_X_X_X_X_X_X_X__X_X_X_X_X__X
TILL HERE IT IS RESOLVED
_X_X_X_X_X_X_X_X_X_X_X_X_X_X_X_X_X__X_X_X
FURTHER ISSUE is SQL command is not working.
In the mailing table of my database I have email id with different domains.
For example, some id's are xyz#gmail.com, 12343#gmail.com , bae#yahoo.com as so on and I am assuming new email ids will be added to mailing tables every day with different domains.
I am trying to write a scripts that updates another table which holds a daily count of email addresses by their domain name. This is what I tried:
#!/usr/bin/perl
use warnings;
use strict;
use lib '/root/mysql/';
use 5.016;
use Data::Dumper;
use GUI::DB qw(dbConnect query);
my $data = dbConnect();
my #domain = query($data, "SELECT substr(addr,locate('\#',addr)+1) as maildomain, count (*) as mailcount FROM mailing GROUP BY maildomain ORDER BY mailcount DESC");
for my $key (#domain){
say Dumper ($key);
}
But I am getting an error,
You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '*) as mailcount FROM mailing GROUP BY maildomain ORDER BY mailcount DESC' at line 1 at /root/mysql/GUI/DB.pm line 44.
Same SQL statement works from CLI with no issues.
Any help would be appreciated. :)
1) Your error is saying that $dbh is undefined here:
sub query {
my $dbh = shift;
...
# issue query
my $sth = $dbh->prepare($sql); #<***LOOK HERE***
...which means $dbh must be undefined here:
sub query {
my $dbh = shift; #<***LOOK HERE***
...
# issue query
my $sth = $dbh->prepare($sql);
2) Let's see why. Your dbConnect() method returns $dbh:
sub dbConnect {
# Read database settings from config file:
print "Works";
my $dsn = "DBI:mysql:project";
my $dbh = DBI->connect(
$dsn,
'root',
'mydatabasepassword',
{ RaiseError => 1 }
);
return $dbh; #<***LOOK HERE*****
}
3) But, you call dbConnect() like this:
dbConnect();
Because you never save the return value anywhere, $dbh is discarded.
4) Furthermore, you call query() like this:
query();
Yet, you defined query() like this:
sub query {
my $dbh = shift;
The query() sub believes that the first argument will be the database handle--but you didn't call query() with any arguments.
You need to do this:
my $data_base_handle = dbConnect();
my #results = query($data_base_handle);
#do something with #results
Response to comment:
I printed #results, this is what I see HASH(0x1d05be8)
HASH(0x1d05ba0) HASH(0x1d05b58) HASH(0x1d05b10) HASH(0x1d05ac8)
HASH(0x1d05a80) HASH(0x1d05a38)
You wrote:
my $row = $sth->fetchrow_hashref;
...which asks DBI to return each row as a reference to a hash. Then you wrote:
push #returnData, $row;
...which pushed each hash reference into an array. So query() returns an array of hash references. The notation HASH(0x1d05be8) is what perl outputs when you print a hash reference.
If you want to see what's in those hashes, then do this:
use 5.016; #enable say()
use Data::Dumper;
...
...
for my $href (#results) {
say Dumper($href);
}
To access the data in a hash reference, you can do this:
use strict;
use warnings;
use 5.016;
use Data::Dumper;
my $href = {
c => 3,
a => 1,
b => 2,
};
my %hash = %{$href}; #dereference, {}, the reference into a hash, %
for my $key ( keys %hash ) {
say "$key $hash{$key}";
}
--output:--
c 3
a 1
b 2
Response to next comment:
(Answer posted in comments under op.)
By the way, perl is pretty good at text processing, so if you couldn't figure out the problem with your query, you could process the email addresses with perl:
use strict;
use warnings;
use 5.012;
use Data::Dumper;
use DBI;
use DBD::mysql;
# CONFIG VARIABLES
my $db_type = "mysql";
my $database = "my_db";
my $host = "localhost";
my $port = "3306";
my $user = "root";
my $pword = "";
# DATA SOURCE NAME
my $dsn = "dbi:$db_type:$database:$host:$port";
# PERL DBI CONNECT
my $dbh = DBI->connect($dsn, $user, $pword);
# PREPARE THE QUERY
my $tablename = "mailing";
my $select =<<"END_OF_SELECT";
select addr from $tablename
END_OF_SELECT
my $addr_aref = $dbh->selectcol_arrayref($select); #Returns a reference to a flat array containing all the email addresses
$dbh->disconnect;
my %count_for;
for my $addr (#{$addr_aref}) {
$addr =~ s/.*#//;
$count_for{$addr}++;
}
say Dumper(\%count_for);
--output:--
$VAR1 = {
'google.com' => 2,
'gorilla.com' => 1,
'yahoo.com' => 3
};