Symfony2 add query logging in Symfony and in csv file - mysql

I need to add query logging in Symfony and in csv file. Sometime database can be busy or not available but I want to log all queries.
This log should be in csv format with columns:
-url
-datasource name
-SQL content
-parameters
-username
-start time of query execution (accuracy in ms)
-end time of query execution (accuracy in ms)
Is any help how can I do this? or what can be done for this?
Maybe Custom Function In which I can create the csv file with the current login user info with the url and time to execute the query to access that particular url?

In Symfony If you have a common SQL function which runs when each
query executes then this one helpful for you.
This is the thing I have implemented in my case. Hope This may help you.
public function execute($parameters = null)
{
if ($this->executed) {
return $this;
}
$executionStartTime = microtime(true);
$stmt = $this->execute($parameters);
$this->result = $stmt->fetchAll();
// Close cursor to allow query caching.
$stmt->closeCursor();
$this->executed = true;;
$executionEndTime = microtime(true);
//below code is added to logging for SQL queries to web server in csv format
$execution_time = $executionEndTime - $executionStartTime;
$getpageParameter = $parameters;
$getusername = $this->getUser()->getUsername();
$url = $_SERVER['HTTP_HOST'].$_SERVER['REQUEST_URI'];
$parms = json_encode($getpageParameter);
$data = array(
date ("Y-m-d H:i:s")."|".$url."|".$getusername."|".$parms."|".$executionStartTime."|".$executionEndTime,
);
if(!file_exists('/../../app/logs/querylog.csv')){
$column = array(
"DATE & TIME|URL|USERNAME|PARAMETERS|START TIME|END TIME"
);
$fp = fopen('/../../app/logs/querylog.csv', 'a+');
foreach ( $column as $line ) {
$val = explode("|", $line);
fputcsv($fp, $val);
}
fclose($fp);
}
$fp = fopen('/../../app/logs/querylog.csv', 'a+');
foreach ( $data as $line ) {
$val = explode("|", $line);
fputcsv($fp, $val);
}
fclose($fp);
return $this;
}

Related

How to optimize this script performing INSERTS into a database?

So i already complete a script that will insert data into mysql table and move those file into a directory until all files are none. There around 51 files and it took around 9 sec to complete the execution. So my question is . is there a better way to speed up the execution process?
the codes are
our $DIR="/home/aimanhalim/LOG";
our $FILENAME_REGEX = "server_performance_";
# mariaDB config hash
our %db_config = ( "username"=>"root", "password"=> "", "db"=>"Top_Data", "ip" => "127.0.0.1", "port" => "3306");
main();
exit;
sub main()
{
my $start = time();
print "Searching file $FILENAME_REGEX in $DIR...\n";
opendir (my $dr , $DIR) or die "<ERROR> Cannot open dir: $DIR \n";
while( my $file = readdir $dr )
{
print "file in $DIR: [$file]\n";
next if (($file eq ".") || ($file eq "..") || ($file eq "DONE"));
#Opening The File in the directory
open(my $file_hndlr, "<$DIR/$file");
#Making Variables.
my $line_count = 0;
my %data = ();
my $dataRef = \%data;
my $move = "$DIR/$file";
print "$file\n";
while (<$file_hndlr>)
{
my $line = $_;
chomp($line);
print "line[$line_count] - [$line]\n";
if($line_count == 0)
{
# get load average from line 0
($dataRef) = get_load_average($line,$dataRef);
print Dumper($dataRef);
}
elsif ($line_count == 2)
{
($dataRef) = get_Cpu($line,$dataRef);
print Dumper($dataRef);
}
$line_count++;
}
#insert db
my ($result) = insert_record($dataRef,\%db_config,$file);
my $Done_File="/home/aimanhalim/LOG/DONE";
sub insert_record(){
my($data,$db_config,$file)=#_;
my $result = -1; # -1 fail; 0 - succ
# connect to db
# connect to MySQL database
my $dsn = "DBI:mysql:database=".$db_config->{'db'}.";host=".$db_config->{'ip'}.";port=".$db_config->{'port'};
my $username = $db_config->{'username'};
my $password = $db_config->{'password'};
my %attr = (PrintError=>0,RaiseError=>1 );
my $dbh = DBI->connect($dsn,$username,$password,\%attr) or die $DBI::errstr;
print "We Have Successfully Connected To The Database \n";
$stmt->execute(#param_bind);
****this line is insert data statement***
$stmt->finish();
print "The Data Has Been Inserted Successfully\n";
$result = 0;
return($result);
# commit
$dbh->commit();
# return succ / if fail rollback and return fail
$dbh->disconnect();
}
exit;
editted
so pretty much this is my code with some sniping here and there.
i tried to put the 'insert_record' below the comment #insert db but i dont think that do anything :U
You are connecting to the database for every file that you want to insert (if I read your code correctly, there seems to be a closing curly brace missing, it won't actually compile). Opening new database connections is (comparably) slow.
Open the connection once, before inserting the first file and re-use it for subsequent inserts into the database. Close the connection after your last file was inserted into the database. This should give you a noticable speed up.
(Depending on the amount of data, 9 seconds might actually not be too bad; but since there is no information on that, it's hard to say).

"Undefined index" when trying to insert JSON file into SQL in PHP

im trying to insert multiple JSON files into a database(about 20-30 in total, im using 2 for now to test). All the files will have the same format. I inserted the files into an HTML table previously so i used the same loop so my script inserts any JSON file found in the directory into the database. I am however getting some errors 1) "Undefined index : Comments" and 2) Table 'serverd.serverd' doesn't exist. Any guidance would be appreciated. I have moved my brackets around but no luck.
'<?php
$connect =mysqli_connect("reservation","-----","-----","serverD") or
die("could not connect");
$dir = "/Users/-----/Desktop/reserve/sql";
if (is_dir($dir)) {
if ($dh = opendir($dir)) {
foreach(glob("*.json") as $filename) {
$jsondata = file_get_contents($filename);
$data = json_decode($jsondata, true);
$Manufacturer = $data['Comments']['Manufacturer'];
$Model = $data['Comments']['Model'];
$BIOSFamily = $data['Comments']['BIOSFamily'];
$BIOSDATE = $data['Comments']['BIOSDate'];
$SerialNumber = $data['Comments']['SerialNumber'];
$sql= " INSERT INTO serverD(Manufacturer, Model, BIOSFamily, BIOSDate, SerialNumber)
VALUES('$Manufacturer' , '$Model' , '$BIOSFamily' , '$BIOSDate' , '$SerialNumber')";
$query=mysqli_query($connect, $sql) or die (mysqli_error($connect));
}
}
}
?>'

Perl DB interaction

I just started using Perl. I am able to connect to my MySQL database, create tables and get query results using my Perl Script. I came across a task that involves "You MUST use the provided DB.pm for all database interaction, and you must use it as it is (DB.pm cannot be modified except for the connection settings)."
What does that mean? Any one can guide me in the right direction ?
DB.pm file contains the following code
package GUI::DB;
use strict;
use DBI;
use vars qw(#ISA #EXPORT);
use Exporter;
#ISA = qw(Exporter);
#EXPORT = qw(dbConnect query);
#
# dbConnect - connect to the database, get the database handle
#
sub dbConnect {
# Read database settings from config file:
my $dsn = "DBI:mysql:database=test";
my $dbh = DBI->connect( $dsn,
'',
'',
{ RaiseError => 1 }
);
return $dbh;
}
#
# query - execute a query with parameters
# query($dbh, $sql, #bindValues)
#
sub query {
my $dbh = shift;
my $sql = shift;
my #bindValues = #_; # 0 or several parameters
my #returnData = ();
# issue query
my $sth = $dbh->prepare($sql);
if ( #bindValues ) {
$sth->execute(#bindValues);
} else {
$sth->execute();
}
if ( $sql =~ m/^select/i ) {
while ( my $row = $sth->fetchrow_hashref ) {
push #returnData, $row;
}
}
# finish the sql statement
$sth->finish();
return #returnData;
}
__END__
Probably, it means, that in your code you must use something like this:
use GUI::DB;
my $dbh = dbConnect();
my $sql = qq{SELECT * FROM my_table};
my #data = query($sql, $dbh);
You interact with the database through the provided module.

Joomla 3.1 Database update query not working

Recently I was making an upload component for joomla 3.1 back-end.
based on How to Save Uploaded File's Name on Database
I was successful in moving the file to the hard-drive,
however I just cant get the update query to work based on the posted post above.
I don't get any SQL errors and saving works, but somehow ignores the database part.
I really hope I missed something obvious. (btw I don't know the joomla way of queries very well)
In phpmyadmin the following query works:
UPDATE hmdq7_mysites_projects
SET project_file = 'test'
WHERE id IN (3);
I have tried the following queries:
$id = JRequest::getVar('id');
$db =& JFactory::getDBO();
$sql = "UPDATE hmdq7_mysites_projects
SET project_file =' " . $filename. "'
WHERE id IN (".$id.");";
$db->setQuery($sql);
$db->query();
$colum = "project_file";
$id = JRequest::getVar('id');
$data = JRequest::getVar( 'jform', null, 'post', 'array' );
$data['project_file'] = strtolower( $file['name']['project_file'] );
$db =& JFactory::getDBO();
$query = $db->getQuery(true);
$query->update('#__mysites_projects');
$query->set($column.' = '.$db->quote($data));
$query->where('id'.'='.$db->quote($id));
$db->setQuery($query);
$db->query();
Here is the current code:
class MysitesControllerProject extends JControllerForm
{
function __construct() {
$this->view_list = 'projects';
parent::__construct();
}
function save(){
// ---------------------------- Uploading the file ---------------------
// Neccesary libraries and variables
jimport( 'joomla.filesystem.folder' );
jimport('joomla.filesystem.file');
$path= JPATH_SITE . DS . "images";
// Create the gonewsleter folder if not exists in images folder
if ( !JFolder::exists(JPATH_SITE . "/images" ) ) {
JFactory::getApplication()->enqueueMessage( $path , 'blue');
}
// Get the file data array from the request.
$file = JRequest::getVar( 'jform', null, 'files', 'array' );
// Make the file name safe.
$filename = JFile::makeSafe($file['name']['project_file']);
// Move the uploaded file into a permanent location.
if ( $filename != '' ) {
// Make sure that the full file path is safe.
$filepath = JPath::clean( JPATH_SITE . "/images/" . $filename );
// Move the uploaded file.
JFile::upload( $file['tmp_name']['project_file'], $filepath );
$colum = "project_file";
$id = JRequest::getVar('id');
$data = JRequest::getVar( 'jform', null, 'post', 'array' );
$data['project_file'] = strtolower( $file['name']['project_file'] );
$db =& JFactory::getDBO();
$query = $db->getQuery(true);
$query->update('#__mysites_projects');
$query->set($column.' = '.$db->quote($data));
$query->where('id'.'='.$db->quote($id));
$db->setQuery($query);
$db->query();
}
// ---------------------------- File Upload Ends ------------------------
JRequest::setVar('jform', $data );
return parent::save();
}
(Answered by the OP in comments. Converted to a community wiki answer. See Question with no answers, but issue solved in the comments (or extended in chat) )
The OP wrote:
Solved: after reviewing the post update record in database using jdatabase I made up some fixed test values. It turns out the query is correct but $data variable in the query had no data. $data['project_file'] = strtolower( $file['name']['project_file'] ); removed the array from first part and variable worked.

Cannot fetch sorted data from mysql database by perl

I am trying to create a cgi based on perl to display my album and now I am working on the sorting function on photos. I stored the information of each photo in mysql. To display all photos, I have to fetch the information first.
Here is the problem: I am expecting the fetched data from mysql is sorted by the file size of each photos, however the result from the fetchrow_array() is the data sorting according to the time being inserted into mysql.
In mysql shell, I tested
SELECT * FROM album ORDER BY filesize;
which gives the expected result sorted by the file size. Here is part of my source code:
#!/usr/bin/perl -w
use strict;
use CGI;
my $sort = 'filesize';
# Connect the database
my $dbh = do 'db.pl';
# Prepare to print out the pictures
my $query;
$query = $dbh->prepare("SELECT * FROM album ORDER BY ?") or die $DBI::errstr;
$query->execute($sort) or die $DBI::errstr;
# Print out all pictures
while( my #data = $query->fetchrow_array() ){
# Process fetched data
(my $id, my $user, my $filepath, my $filename, my $filesize, my $uploadtime, my $description, my $tfilepath, my $sessioninfo) = #data;
print '<fieldset>';
# Display thumbnail
print '<img src="', $tfilepath, '" title="', $description, '">';
# Display filename
print '</br>';
print $filename;
print '</fieldset>';
}
# Finish printing out all fetched pictures
$query->finish;
Am I using the wrong command? Or I am using a wrong approach to do the sorting function?
Thanks for helping!
ORDER BY takes a field name, not an expression.
my $query = "SELECT * FROM album ORDER BY ".$dbh->quote_identifier($sort);
my $sth = $dbh->prepare($query);
$sth->execute();
By the way, you have have bugs on the output side too. What if $description contains """, "&" or "<"? You need some escaping.
sub text_to_html {
my ($s) = #_;
$s =~ s/&/&/g;
$s =~ s/</</g;
$s =~ s/>/>/g;
$s =~ s/"/"/g;
$s =~ s/'/&apos;/g;
return $s;
}
By the way,
(my $id, my $user, my $filepath, my $filename,
my $filesize, my $uploadtime, my $description,
my $tfilepath, my $sessioninfo) = #data;
can be written as
my ($id, $user, $filepath, $filename,
$filesize, $uploadtime, $description,
$tfilepath, $sessioninfo) = #data;