Autokill select MySQL queries perl script - mysql

I use the below perl script to kill all select connections done by users but i want to make an exception for the root user since root queries are either important or backup purposes.
Below is my code
#!/usr/bin/perl -w
use DBI;
use strict;
$| = 1;
#------------------------------------------------------------------------------
# !!! Configure check time and timeout. These are both in seconds.
my $check = 15; # check processes every $check seconds
my $slow_time = 60; # stop processes that run for >= $slow_time seconds
# !!! Configure log file - All slow queries also get logged to this file
my $logfile = "./check_mysql_query.log"; # log slow queries to this file
# !!! Configure the database connection parameters
my $db_string = "dbi:mysql:mysql"; # DBI resource to connect to
my $db_user = "root"; # DBI username to connect as
my $db_pass = "password"; # DBI password to connect with
# !!! Configure path to sendmail program
my $sendmail_bin = "/usr/sbin/sendmail";
#
#------------------------------------------------------------------------------
my ($dbh,$sth,$sth2,$thread,$state,$time,$query,$explain);
print "connecting\n";
my $opt = {
'RaiseError'=>0,
'PrintError'=>0
};
$dbh = DBI->connect($db_string,$db_user,$db_pass,$opt);
unless ($dbh) {
print "Error: Unable to connect to database: $DBI::errstr\n";
exit 1;
}
$SIG{'TERM'} = sub {
print "caught sig TERM!\nexiting!\n";
$dbh->disconnect;
exit 1;
};
print "preparing\n";
unless ($sth = $dbh->prepare("show full processlist")) {
print "error preparing query: $DBI::errstr\nexiting!\n";
$dbh->disconnect;
exit 1;
}
print "initialized.. starting loop\n";
while(1) {
unless ($sth->execute) {
print "statement execute failed: ".$sth->errstr."\nexiting!\n";
last;
}
while(my #tmp = $sth->fetchrow) {
$thread = $tmp[0];
$state = $tmp[4];
$time = $tmp[5];
$query = $tmp[7];
if ($state eq "Query" && $query =~ /^SELECT/ && $time >= $slow_time) {
print "killing slow query thread=$thread state=$state time=$time\n";
$dbh->do("kill $thread");
unless (log_query($logfile,$query)) {
print "log_query failed! exiting!\n";
last;
}
}
}
sleep($check);
}
$sth->finish;
$dbh->disconnect;
exit 1;
sub log_query {
my ($file,$query) = #_;
unless (open(O,">>".$file)) {
print "error opening log file '$file': $!\n";
return undef;
}
print O $query."\n-----\n";
close(O);
return 1;
}
script credit: http://code.google.com/p/mysql-killquery/
i removed most of the script code for simplicity.

I think I get from my reading that your second column will be the user. So assuming root logs into the database as user 'root', then you can probably do this:
while(my #tmp = $sth->fetchrow) {
next if $tmp[1] eq 'root';
...
}

Related

How to optimize this script performing INSERTS into a database?

So i already complete a script that will insert data into mysql table and move those file into a directory until all files are none. There around 51 files and it took around 9 sec to complete the execution. So my question is . is there a better way to speed up the execution process?
the codes are
our $DIR="/home/aimanhalim/LOG";
our $FILENAME_REGEX = "server_performance_";
# mariaDB config hash
our %db_config = ( "username"=>"root", "password"=> "", "db"=>"Top_Data", "ip" => "127.0.0.1", "port" => "3306");
main();
exit;
sub main()
{
my $start = time();
print "Searching file $FILENAME_REGEX in $DIR...\n";
opendir (my $dr , $DIR) or die "<ERROR> Cannot open dir: $DIR \n";
while( my $file = readdir $dr )
{
print "file in $DIR: [$file]\n";
next if (($file eq ".") || ($file eq "..") || ($file eq "DONE"));
#Opening The File in the directory
open(my $file_hndlr, "<$DIR/$file");
#Making Variables.
my $line_count = 0;
my %data = ();
my $dataRef = \%data;
my $move = "$DIR/$file";
print "$file\n";
while (<$file_hndlr>)
{
my $line = $_;
chomp($line);
print "line[$line_count] - [$line]\n";
if($line_count == 0)
{
# get load average from line 0
($dataRef) = get_load_average($line,$dataRef);
print Dumper($dataRef);
}
elsif ($line_count == 2)
{
($dataRef) = get_Cpu($line,$dataRef);
print Dumper($dataRef);
}
$line_count++;
}
#insert db
my ($result) = insert_record($dataRef,\%db_config,$file);
my $Done_File="/home/aimanhalim/LOG/DONE";
sub insert_record(){
my($data,$db_config,$file)=#_;
my $result = -1; # -1 fail; 0 - succ
# connect to db
# connect to MySQL database
my $dsn = "DBI:mysql:database=".$db_config->{'db'}.";host=".$db_config->{'ip'}.";port=".$db_config->{'port'};
my $username = $db_config->{'username'};
my $password = $db_config->{'password'};
my %attr = (PrintError=>0,RaiseError=>1 );
my $dbh = DBI->connect($dsn,$username,$password,\%attr) or die $DBI::errstr;
print "We Have Successfully Connected To The Database \n";
$stmt->execute(#param_bind);
****this line is insert data statement***
$stmt->finish();
print "The Data Has Been Inserted Successfully\n";
$result = 0;
return($result);
# commit
$dbh->commit();
# return succ / if fail rollback and return fail
$dbh->disconnect();
}
exit;
editted
so pretty much this is my code with some sniping here and there.
i tried to put the 'insert_record' below the comment #insert db but i dont think that do anything :U
You are connecting to the database for every file that you want to insert (if I read your code correctly, there seems to be a closing curly brace missing, it won't actually compile). Opening new database connections is (comparably) slow.
Open the connection once, before inserting the first file and re-use it for subsequent inserts into the database. Close the connection after your last file was inserted into the database. This should give you a noticable speed up.
(Depending on the amount of data, 9 seconds might actually not be too bad; but since there is no information on that, it's hard to say).

Display content and its related contents in Perl/Mysql

I am trying to achieve this sample output from a small Perl project like this
content1
relatedcontent1
relatedcontent2
relatedcontent2
content2
relatedcontent1
relatedcontent2
here is my code
#!C:/Perl64/bin/perl.exe
use strict;
use warnings;
use v5.10; # for say() function
use DBI;
use HTML::Table;
# MySQL database configurations
my $dsn = "DBI:mysql:naxum";
my $username = "root";
my $password = '';
print "Content-Type:text/html\r\n\r\n";
# connect to MySQL database
my %attr = ( PrintError=>0, # turn off error reporting via warn()
RaiseError=>1 # report error via die()
);
my $dbh = DBI->connect($dsn,$username,$password,\%attr);
# query data from the sponsor table
query_sponsor($dbh);
query_person_by_target($dbh);
sub query_sponsor{
# query from the table
my ($dbh) = #_;
my $sql = "SELECT name,id FROM sponsor";
my $sth = $dbh->prepare($sql);
# execute the query
$sth->execute();
print "<table>\n";
print "<thead>\n";
print "<tr>\n";
print "<th>Id</th>\n";
print "<th>Name</th>\n";
print "</tr>\n";
print "</thead>\n";
print "<tbody>\n";
while(my #row = $sth->fetchrow_array()){
print "<tr>\n";
print "<td>\n";
print $row['1'];
sub query_person_by_target{
my ($dbhPerson) = #_;
my $sqlPerson = "SELECT username, firstname FROM person WHERE sponsor_id = ?";
my $sthPerson = $dbhPerson->prepare($sqlPerson);
$sthPerson->execute($row['1']) or die "execution failed: $dbhPerson->errstr()";
while ( my #rowPerson = $sthPerson->fetchrow_array()){
print "<p>$rowPerson['0']</p>\n";
}
$sth->finish();
}
print "</td>\n";
print "<td>$row['0']</td>\n";
print "</tr>\n";
}
$sth->finish();
print "</tbody>\n";
print "</table>\n";
}
$dbh->disconnect();
However, I can't get the output what I am trying to achieve. Here is the result
content1
content2
content3
.....
relatedcontent1
It will just print one related content outside contents. Every content and its own at least 3 relatedcontent each.
You define your subroutines in the middle of other code in your program. This makes me think you expect them to be executed as they are defined, but that's not the case - they are executed as you call them. And you call them like this:
query_sponsor($dbh);
query_person_by_target($dbh);
So it should come as no surprise that you get all of the output from query_sponsor() followed by all of the output from query_person_by_name().
A far better approach would be to call query_person_by_target() from within query_sponsor(). I'd also split this into two phases - phase one to extract the data into a data structure and phase two to display that data.
As I don't have your database, this code is untested.
#!/usr/bin/perl
use strict;
use warnings;
use feature 'state'; # for state variables
use feature 'say';
use CGI 'header';
sub get_dbh {
# MySQL database configurations
my $dsn = "DBI:mysql:naxum";
my $username = "root";
my $password = '';
# connect to MySQL database
my %attr = (
PrintError=>0, # turn off error reporting via warn()
RaiseError=>1, # report error via die()
);
return DBI->connect($dsn, $username, $password, \%attr);
}
sub query_sponsor {
state $dbh = get_dbh();
my $sql = 'select name, id from sponsor';
my $sth = $dbh->prepare($sql);
$sth->execute;
my #sponsors;
while (my #row = $dbh->fetchrow_array) {
push #sponsors, {
name => $row[0],
id => $row[1],
people => get_people_for_sponsor($row[1]),
};
}
return #sponsors;
}
sub get_people_for_sponsor {
my ($sponsor_id) = #_;
state $dbh = get_dbh;
my $sql = 'select username, firstname
from person
where sponsor_id = ?';
my $sth = $dbh->prepare;
$sth->execute($sponsor_id);
my #people;
while (my #row = $sth->fetchrow_array) {
push #people, {
username => $row[0],
firstname => $row[1],
};
}
return \#people;
}
my #sponsors = query_sponsor();
# Now you have all of your data in #sponsors. You simply need
# to walk that array and use the data to build the output you
# want. My example is plain text - it shouldn't be too hard to
# convert it to HTML.
print header('text/plain');
for my $s (#sponsors) {
say "$s->{id}: $s->{name}";
for my $p (#{$s->{people}}) {
say "* $p->{firstname} / $p->{username}";
}
}
I'd also recommend that you look at using something like the Template Toolkit to produce your output. Putting raw HTML into your Perl program is a terrible idea - it is guaranteed to turn into an unmaintainable mess :-)
You have declared the two subroutines and you are calling one after one so it is executing like that for example consider the following one
sub1();
sub2();
sub sub1()
{
for(0..5)
{
print "hello\n";
sub sub2()
{
print "hi\n";
}
}
}
#output
hello
hello
hello
hello
hello
hello
hi
So you should remove the child query_person_by_target subroutine or call the child subroutine inside the parent subroutine query_sponsor and declare the child subroutine outside of the sub and loop, like below (not tested)
query_sponsor($dbh);
sub query_sponsor
{
# query from the table
my ($dbh) = #_;
my $sql = "SELECT name,id FROM sponsor";
my $sth = $dbh->prepare($sql);
# execute the query
$sth->execute();
print "<table>\n";
print "<thead>\n";
print "<tr>\n";
print "<th>Id</th>\n";
print "<th>Name</th>\n";
print "</tr>\n";
print "</thead>\n";
print "<tbody>\n";
while(my #row = $sth->fetchrow_array())
{
print "<tr>\n";
print "<td>\n";
print $row['1'];
query_person_by_target($dbh);
print "</td>\n";
print "<td>$row['0']</td>\n";
print "</tr>\n";
}
$sth->finish();
print "</tbody>\n";
print "</table>\n";
}
sub query_person_by_target{
my ($dbhPerson) = #_;
my $sqlPerson = "SELECT username, firstname FROM person WHERE sponsor_id = ?";
my $sthPerson = $dbhPerson->prepare($sqlPerson);
$sthPerson->execute($row['1']) or die "execution failed: $dbhPerson->errstr()";
while ( my #rowPerson = $sthPerson->fetchrow_array()){
print "<p>$rowPerson['0']</p>\n";
}
$sth->finish();
}
$dbh->disconnect();

Perl module to log in database and do a simple query?

I can't seem to figure out the issue with my .pm file and script. I am fairly new to Perl.
I have a database with name "project" and there is table with name "mailing".
mailing table has 7 entries, which I want to display using module.
So, I have this custom module to log in to database and do a query. This module is names as DB.pm
DB.pm is stored on my FEDORA 20 at /root/mysql/GUI/DB.pm.
DB.pm is defined as follows:
package GUI::DB;
use strict;
use DBI;
use vars qw(#ISA #EXPORT);
use Exporter;
#ISA = qw(Exporter);
#EXPORT = qw(dbConnect query);
#
# dbConnect - connect to the database, get the database handle
#
sub dbConnect {
# Read database settings from config file:
print "Works";
my $dsn = "DBI:mysql:project";
my $dbh = DBI->connect( $dsn,
'root',
'mydatabasepassword',
{ RaiseError => 1 }
);
return $dbh;
}
#
# query - execute a query with parameters
# query($dbh, $sql, #bindValues)
#
sub query {
my $dbh = shift;
my $sql ="SELECT * FROM mailing";
my #bindValues = #_; # 0 or serveral parameters
my #returnData = ();
# issue query
my $sth = $dbh->prepare($sql); //**line number 39 that is giving** error
if ( #bindValues ) {
$sth->execute(#bindValues);
} else {
$sth->execute();
}
if ( $sql =~ m/^select/i ) {
while ( my $row = $sth->fetchrow_hashref ) {
push #returnData, $row;
}
}
# finish the sql statement
$sth->finish();
return #returnData;
}
1;
Now I want to use this module inside my per script. This is what I tried:
#!/usr/bin/perl
use warnings;
use strict;
use lib '/root/mysql/';
use GUI::DB qw(dbConnect query);
dbConnect();
query();
This is the error I'm getting -->
Can't call method "prepare" on an undefined value at /root/mysql/GUI/DB.pm line 39.
Please help me with this. I am not sure how to proceed. I am guessing it has something to do with argument passing. Nothing is wrong with database. It works fine from CLI.
Thanks :)
_x_X_X_X_X_X_X_X_X_X_X_X_X_X_X_X__X_X_X_X_X__X
TILL HERE IT IS RESOLVED
_X_X_X_X_X_X_X_X_X_X_X_X_X_X_X_X_X__X_X_X
FURTHER ISSUE is SQL command is not working.
In the mailing table of my database I have email id with different domains.
For example, some id's are xyz#gmail.com, 12343#gmail.com , bae#yahoo.com as so on and I am assuming new email ids will be added to mailing tables every day with different domains.
I am trying to write a scripts that updates another table which holds a daily count of email addresses by their domain name. This is what I tried:
#!/usr/bin/perl
use warnings;
use strict;
use lib '/root/mysql/';
use 5.016;
use Data::Dumper;
use GUI::DB qw(dbConnect query);
my $data = dbConnect();
my #domain = query($data, "SELECT substr(addr,locate('\#',addr)+1) as maildomain, count (*) as mailcount FROM mailing GROUP BY maildomain ORDER BY mailcount DESC");
for my $key (#domain){
say Dumper ($key);
}
But I am getting an error,
You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '*) as mailcount FROM mailing GROUP BY maildomain ORDER BY mailcount DESC' at line 1 at /root/mysql/GUI/DB.pm line 44.
Same SQL statement works from CLI with no issues.
Any help would be appreciated. :)
1) Your error is saying that $dbh is undefined here:
sub query {
my $dbh = shift;
...
# issue query
my $sth = $dbh->prepare($sql); #<***LOOK HERE***
...which means $dbh must be undefined here:
sub query {
my $dbh = shift; #<***LOOK HERE***
...
# issue query
my $sth = $dbh->prepare($sql);
2) Let's see why. Your dbConnect() method returns $dbh:
sub dbConnect {
# Read database settings from config file:
print "Works";
my $dsn = "DBI:mysql:project";
my $dbh = DBI->connect(
$dsn,
'root',
'mydatabasepassword',
{ RaiseError => 1 }
);
return $dbh; #<***LOOK HERE*****
}
3) But, you call dbConnect() like this:
dbConnect();
Because you never save the return value anywhere, $dbh is discarded.
4) Furthermore, you call query() like this:
query();
Yet, you defined query() like this:
sub query {
my $dbh = shift;
The query() sub believes that the first argument will be the database handle--but you didn't call query() with any arguments.
You need to do this:
my $data_base_handle = dbConnect();
my #results = query($data_base_handle);
#do something with #results
Response to comment:
I printed #results, this is what I see HASH(0x1d05be8)
HASH(0x1d05ba0) HASH(0x1d05b58) HASH(0x1d05b10) HASH(0x1d05ac8)
HASH(0x1d05a80) HASH(0x1d05a38)
You wrote:
my $row = $sth->fetchrow_hashref;
...which asks DBI to return each row as a reference to a hash. Then you wrote:
push #returnData, $row;
...which pushed each hash reference into an array. So query() returns an array of hash references. The notation HASH(0x1d05be8) is what perl outputs when you print a hash reference.
If you want to see what's in those hashes, then do this:
use 5.016; #enable say()
use Data::Dumper;
...
...
for my $href (#results) {
say Dumper($href);
}
To access the data in a hash reference, you can do this:
use strict;
use warnings;
use 5.016;
use Data::Dumper;
my $href = {
c => 3,
a => 1,
b => 2,
};
my %hash = %{$href}; #dereference, {}, the reference into a hash, %
for my $key ( keys %hash ) {
say "$key $hash{$key}";
}
--output:--
c 3
a 1
b 2
Response to next comment:
(Answer posted in comments under op.)
By the way, perl is pretty good at text processing, so if you couldn't figure out the problem with your query, you could process the email addresses with perl:
use strict;
use warnings;
use 5.012;
use Data::Dumper;
use DBI;
use DBD::mysql;
# CONFIG VARIABLES
my $db_type = "mysql";
my $database = "my_db";
my $host = "localhost";
my $port = "3306";
my $user = "root";
my $pword = "";
# DATA SOURCE NAME
my $dsn = "dbi:$db_type:$database:$host:$port";
# PERL DBI CONNECT
my $dbh = DBI->connect($dsn, $user, $pword);
# PREPARE THE QUERY
my $tablename = "mailing";
my $select =<<"END_OF_SELECT";
select addr from $tablename
END_OF_SELECT
my $addr_aref = $dbh->selectcol_arrayref($select); #Returns a reference to a flat array containing all the email addresses
$dbh->disconnect;
my %count_for;
for my $addr (#{$addr_aref}) {
$addr =~ s/.*#//;
$count_for{$addr}++;
}
say Dumper(\%count_for);
--output:--
$VAR1 = {
'google.com' => 2,
'gorilla.com' => 1,
'yahoo.com' => 3
};

Perl DB interaction

I just started using Perl. I am able to connect to my MySQL database, create tables and get query results using my Perl Script. I came across a task that involves "You MUST use the provided DB.pm for all database interaction, and you must use it as it is (DB.pm cannot be modified except for the connection settings)."
What does that mean? Any one can guide me in the right direction ?
DB.pm file contains the following code
package GUI::DB;
use strict;
use DBI;
use vars qw(#ISA #EXPORT);
use Exporter;
#ISA = qw(Exporter);
#EXPORT = qw(dbConnect query);
#
# dbConnect - connect to the database, get the database handle
#
sub dbConnect {
# Read database settings from config file:
my $dsn = "DBI:mysql:database=test";
my $dbh = DBI->connect( $dsn,
'',
'',
{ RaiseError => 1 }
);
return $dbh;
}
#
# query - execute a query with parameters
# query($dbh, $sql, #bindValues)
#
sub query {
my $dbh = shift;
my $sql = shift;
my #bindValues = #_; # 0 or several parameters
my #returnData = ();
# issue query
my $sth = $dbh->prepare($sql);
if ( #bindValues ) {
$sth->execute(#bindValues);
} else {
$sth->execute();
}
if ( $sql =~ m/^select/i ) {
while ( my $row = $sth->fetchrow_hashref ) {
push #returnData, $row;
}
}
# finish the sql statement
$sth->finish();
return #returnData;
}
__END__
Probably, it means, that in your code you must use something like this:
use GUI::DB;
my $dbh = dbConnect();
my $sql = qq{SELECT * FROM my_table};
my #data = query($sql, $dbh);
You interact with the database through the provided module.

Can I check to see if an execution works without it dying in Perl mysql?

I am connecting to a database and I am taking an input from STDIN that I use as part of an execution.
So I have:
my $i = 0;
while($i != 1) {
print "Input: ";
my $input = <STDIN>;
chomp $input;
my $test = $dbh->prepare("show tables like $input");
and then I want to check that the input is a valid entry in the database and loop round again if it isn't:
if ($test->execute()) {
print "Input exists in database\n";
$i = 1;
}
else {
print "Input does not exist.\n";
}
} # end of while
I know that this does not work, but I would like something similar that isn't execute or die as I do not want to exit my program. Is this possible?
You have two options:
1) Disable the RaiseError attribute for the database handle. This can be done when creating the connection:
$dbh = DBI->connect($dsn, $user, $password, { RaiseError => 0 });
This would of course require you to handle errors yourself by testing $DBI::err on the appropriate places.
2) Catch the error. Either by using one of the Try/Catch frameworks (TryCatch or Try::Tiny are the ones I would recommend) or by using eval by hand. For example:
if (defined( eval { $test->execute() // 0 } ) {
print "Success";
} else {
pring "Bugger, I died...: $DBI:Err";
}
I actually found the solution I wanted a different way, but thanks to pmakholm for answering.
my $i = 0;
while($i != 1) {
print "Input: ";
my $input = <STDIN>;
chomp $input;
my $test = $dbh->prepare("show tables like $input");
my $var = $test->execute();
if ($var != 0) {
print "Input exists in database\n";
$i = 1;
}
else {
print "Input does not exist.\n";
}
} # end of while
What I didn't think of is that even if I enter nonsense into the database, I'm looking to see if that table exists - and if it doesn't it just returns an empty set, so I can check if is zero (checking if NULL or similar may be better but this works). I thought that it would return an error, but it doesn't. pmakholm - I shall use your method if I need to check that query works, so thanks.