How do I enable the MySQL function that logs each SQL query statement received from clients and the time that query statement has submitted?
Can I do that in phpmyadmin or NaviCat?
How do I analyse the log?
First, Remember that this logfile can grow very large on a busy server.
For mysql < 5.1.29:
To enable the query log, put this in /etc/my.cnf in the [mysqld] section
log = /path/to/query.log #works for mysql < 5.1.29
Also, to enable it from MySQL console
SET general_log = 1;
See http://dev.mysql.com/doc/refman/5.1/en/query-log.html
For mysql 5.1.29+
With mysql 5.1.29+ , the log option is deprecated. To specify the logfile and enable logging, use this in my.cnf in the [mysqld] section:
general_log_file = /path/to/query.log
general_log = 1
Alternately, to turn on logging from MySQL console (must also specify log file location somehow, or find the default location):
SET global general_log = 1;
Also note that there are additional options to log only slow queries, or those which do not use indexes.
Take a look on this answer to another related question. It shows how to enable, disable and to see the logs on live servers without restarting.
Log all queries in mysql
Here is a summary:
If you don't want or cannot restart the MySQL server you can proceed like this on your running server:
Create your log tables (see answer)
Enable Query logging on the database
(Note that the string 'table' should be put literally and not substituted by any table name. Thanks Nicholas Pickering)
SET global general_log = 1;
SET global log_output = 'table';
View the log
select * from mysql.general_log;
Disable Query logging on the database
SET global general_log = 0;
Clear query logs without disabling
TRUNCATE mysql.general_log
This was already in a comment, but deserves its own answer:
Without editing the config files: in mysql, as root, do
SET global general_log_file='/tmp/mysql.log';
SET global log_output = 'file';
SET global general_log = on;
Don't forget to turn it off afterwards:
SET global general_log = off;
I use this method for logging when I want to quickly optimize different page loads.
It's a little tip...
Logging to a TABLE
SET global general_log = 1;
SET global log_output = 'table';
You can then select from my mysql.general_log table to retrieve recent queries.
I can then do something similar to tail -f on the mysql.log, but with more refinements...
select * from mysql.general_log
where event_time > (now() - INTERVAL 8 SECOND) and thread_id not in(9 , 628)
and argument <> "SELECT 1" and argument <> ""
and argument <> "SET NAMES 'UTF8'" and argument <> "SHOW STATUS"
and command_type = "Query" and argument <> "SET PROFILING=1"
This makes it easy to see my queries that I can try and cut back. I use 8 seconds interval to only fetch queries executed within the last 8 seconds.
You can disable or enable the general query log (which logs all queries) with
SET GLOBAL general_log = 1 # (or 0 to disable)
// To see global variable is enabled or not and location of query log
SHOW VARIABLES like 'general%';
// Set query log on
SET GLOBAL general_log = ON;
I also wanted to enable the MySQL log file to see the queries and I have resolved this with the below instructions
Go to /etc/mysql/mysql.conf.d
open the mysqld.cnf
and enable the below lines
general_log_file = /var/log/mysql/mysql.log
general_log = 1
restart the MySQL with this command /etc/init.d/mysql restart
go to /var/log/mysql/ and check the logs
On Windows you can simply go to
C:\wamp\bin\mysql\mysql5.1.53\my.ini
Insert this line in my.ini
general_log_file = c:/wamp/logs/mysql_query_log.log
The my.ini file finally looks like this
...
...
...
socket = /tmp/mysql.sock
skip-locking
key_buffer = 16M
max_allowed_packet = 1M
table_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
basedir=c:/wamp/bin/mysql/mysql5.1.53
log = c:/wamp/logs/mysql_query_log.log #dump query logs in this file
log-error=c:/wamp/logs/mysql.log
datadir=c:/wamp/bin/mysql/mysql5.1.53/data
...
...
...
...
There is bug in MySQL 5.6 version.
Even mysqld show as :
Default options are read from the following files in the given order:
C:\Windows\my.ini C:\Windows\my.cnf C:\my.ini C:\my.cnf c:\Program Files (x86)\MySQL\MySQL Server 5.6\my.ini c:\Program Files (x86)\MySQL\MySQL Server 5.6\my.cnf
Realy settings are reading in following order :
Default options are read from the following files in the given order:
C:\ProgramData\MySQL\MySQL Server 5.6\my.ini C:\Windows\my.ini C:\Windows\my.cnf C:\my.ini C:\my.cnf c:\Program Files (x86)\MySQL\MySQL Server 5.6\my.ini c:\Program Files (x86)\MySQL\MySQL Server 5.6\my.cnf
Check file: "C:\ProgramData\MySQL\MySQL Server 5.6\my.ini"
Hope it help somebody.
for mysql>=5.5 only for slow queries (1 second and more)
my.cfg
[mysqld]
slow-query-log = 1
slow-query-log-file = /var/log/mysql/mysql-slow.log
long_query_time = 1
log-queries-not-using-indexes
To enable the query log in MAC Machine:
Open the following file:
vi /private/etc/my.cnf
Set the query log url under 'mysqld' section as follows:
[mysqld]
general_log_file=/Users/kumanan/Documents/mysql_query.log
Few machine’s are not logging query properly, So that case you can enable it from MySQL console
mysql> SET global general_log = 1;
Not exactly an answer to the question because the question already has great answers. This is a side info. Enabling general_log really put a dent on MySQL performance. I left general_log =1 accidentally on a production server and spent hours finding out why performance was not comparable to a similar setup on other servers. Then I found this which explains the impact of enabling general log. http://www.fromdual.com/general_query_log_vs_mysql_performance.
Gist of the story, don't put general_log=1 in the .cnf file. Instead use set global general_log =1 for a brief duration just to log enough to find out what you are trying to find out and then turn it off.
In phpMyAdmin 4.0, you go to Status > Monitor. In there you can enable the slow query log and general log, see a live monitor, select a portion of the graph, see the related queries and analyse them.
I had to drop and recreate the general log at one point. During the recreation, character sets got messed up and I ended up having this error in the logs:
[ERROR] Incorrect definition of table mysql.general_log: expected the type of column 'user_host' at position 1 to have character set 'utf8' but found character set 'latin1'
So if the standard answer of "check to make sure logging is on" doesn't work for you, check to make sure your fields have the right character set.
My OS Win10, MySQL server version - 5.7
The path to my.ini
C:\ProgramData\MySQL\MySQL Server 5.7\my.ini
Just add into my.ini file
general_log_file = C:/ProgramData/MySQL/MySQL Server 5.7/mysql.log
general_log = 1
You may come across a set of Hexadecimal values, like this (argument column):
mysql> select * from mysql.general_log LIMIT 1\G
*************************** 1. row ***************************
event_time: 2023-01-27 13:37:20.950778
user_host: root[root] # localhost []
thread_id: 1434
server_id: 1
command_type: Query
argument: 0x73656C656374202A2066726F6D207573657273
1 row in set (0.00 sec)
so to make it readable, just use:
select a.*, convert(a.argument using utf8) from mysql.general_log a;
And the return is something like this:
mysql> select a.*, convert(a.argument using utf8) from mysql.general_log a LIMIT 1\G
*************************** 1. row ***************************
event_time: 2023-01-27 13:37:20.950778
user_host: root[root] # localhost []
thread_id: 1434
server_id: 1
command_type: Query
argument: 0x73656C656374202A2066726F6D207573657273
convert(a.argument using utf8): select * from users
1 row in set, 1 warning (0.00 sec)
Ps: I used LIMIT 1 on examples, because my log table is too big.
Related
I have looked at many similar questions to this but I can't seem to find the answer. I would like to set up the slow query log for my MySQL database. I have seen many answers saying I should access the MySQL command line tool. I am not sure exactly how to find this tool but I tried accessing it by going to:
c:/xampp/mysql/bin/mysql -u root -p -h localhost
But here I get MariaDB, which seems to be different from any other answers/tutorials I have seen before. Typing in:
set log_slow_queries = ON;
gives me the error
ERROR 1193 (HY000): Unknown system variable 'log_slow_queries'
SET GLOBAL slow_query_log=1;
The Slow Query Log consists of log events for queries taking up to long_query_time seconds to finish. For instance, up to 10 seconds to complete. To see the time threshold currently set, issue the following:
SELECT ##long_query_time;
+-------------------+
| ##long_query_time |
+-------------------+
| 10.000000 |
+-------------------+
It can be set as a GLOBAL variable, in my.cnf or my.ini file. Or it can be set by the connection, though this is unusual. The value can be set between 0 to 10 (seconds). What value to use?
10 is so high as to be almost useless;
2 is a compromise;
0.5 and other fractions are possible;
0 captures everything; this could fill up disk dangerously fast, but can be very useful.
The capturing of slow queries is either turned on or off. And the file logged to is also specified. The below captures these concepts:
SELECT ##slow_query_log; -- Is capture currently active? (1=On, 0=Off)
SELECT ##slow_query_log_file; -- filename for capture. Resides in datadir
SELECT ##datadir; -- to see current value of the location for capture file
SET GLOBAL slow_query_log=0; -- Turn Off
-- make a backup of the Slow Query Log capture file. Then delete it.
SET GLOBAL slow_query_log=1; -- Turn it back On (new empty file is created)
For more information, please see the MySQL Manual Page The Slow Query Log
Note: The above information on turning on/off the slowlog was changed in 5.6(?); older version had another mechanism.
The "best" way to see what is slowing down your system:
long_query_time=...
turn on the slowlog
run for a few hours
turn off the slowlog (or raise the cutoff)
run pt-query-digest to find the 'worst' couple of queries. Or mysqldumpslow -s t
Go to xampp control panel click on config button for mysql and select my.ini then add these lines in my.ini file
slow_query_log = 1
slow-query-log-file=/path/of/the/log/file.log
I put above two lines under the log_error = "mysql_error.log". the modified part of the my.ini file should look like this
# The MySQL server
[mysqld]
port= 3306
socket = "C:/xampp/mysql/mysql.sock"
basedir = "C:/xampp/mysql"
tmpdir = "C:/xampp/tmp"
datadir = "C:/xampp/mysql/data"
pid_file = "mysql.pid"
# enable-named-pipe
key_buffer = 16M
max_allowed_packet = 1M
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
log_error = "mysql_error.log"
slow_query_log = 1
slow-query-log-file=/var/log/mysql-slow.log
Then Restart the MySQL server in xampp control panel. and now slow_query_log should be enabled, you can confirm it by running following command in the MySQL shell
show variables like '%slow%';
It might be obvious but it took me time before I realized my mistake: in the my.ini file you should put the slow_query_log settings in the [mysqld] group, not simply at the end of the my.ini file....
I need to log SQL to a file so i can check later which SQL run.
so i opened opt/lampp/etc/my.cnf and add these lines
log_slow_queries
log_queries_not_using_indexes =1
long_query_time = 1
slow_query_log = 1
slow_query_log_file = "/opt/lampp/logs/query.log"
but it did not logged the queries it even did not created the query.log file, so i created an empty file with the name, but still it's not working.
Edit
[mysqld]
log_slow_queries
log_queries_not_using_indexes =1
long_query_time = 1
slow_query_log = 1
general_log = 1
slow_query_log_file = /opt/lampp/logs/query.log
general_log_file = "/opt/lampp/logs/query.log"
This will only log slow queries. You need the general log if you want to see all queries.
general_log = 1
general_log_file = "/opt/lampp/logs/query.log"
Note that you'll need to restart the server for this to take effect. Also, you should only use this type of logging during testing as it does cause slowdown.
As other users mentioned, this could be a permissions issue. First, check what user MySQL is running as via ps -u -p $(pgrep mysql). The username will be displayed on the first column under USER. In your case, it seems the user is nobody. You can view the default group of a user via groups nobody. This should print something like nobody : nogroup.
To fix the permissions on the file, just run chown nobody:nogroup /opt/lampp/logs/query.log.
Be sure to give the correct permission :
chown mysql:mysql filename
also when i last did it , i had to restart the mysql service :
service mysqld restart
log_slow_queries
is deprecated
It now has to look like that:
slow_query_log
log_queries_not_using_indexes =1
long_query_time = 1
slow_query_log = 1
general_log = 1
slow_query_log_file = /var/log/mysql/mysql-slow.log
general_log_file = /var/log/mysql/mysql-slow.log
The process probably doesn't have permission to write to that directory. Make sure MySQL has permission to write there, or try logging somewhere less restricted.
This will only log slow queries. You need the general log if you want to see all queries.
general_log = 1
general_log_file = "/opt/lampp/logs/query.log"
Note that you'll need to restart the server for this to take effect. Also, you should only use this type of logging during testing as it does cause slowdown.
Also Note that mysql needs permissions over folder too, in my case, I changed:
general_log_file = "/opt/lampp/logs/query.log"
for
general_log_file = "/var/log/mysql/query.log"
But I have mysql installed from software center, without lampp, when I execute ls -l over /var/log/, it shows
drwx------ 8 mysql mysql 4096 sep 25 23:22 mysql
PD:I change the my.cn file and restart mysql, without create the query.log file in the specified path, mysql automatically create it
I'm using MySQL Server version: 5.5.8-log MySQL Community Server (GPL)
I want to log queries which are not using INDEX and is slow too !
I'm copying here my my.ini settings.
[mysqld]
port=3306
log = "E:/wamp/logs/genquery.log"
log_slow_queries
long_query_time = 1
slow_query_log = 1
slow_query_log_file = "E:/wamp/logs/slowquery.log"
what change i need to do ?
log_queries_not_using_indexes =1 //(or Yes) (From mysql)
Maybe useful for Linux user. (Testet: Ubuntu 16.04)
Get root in terminal and edit mysql configuration
su
vim /etc/mysql/conf.d/mysql.cnf
[mysqld]
slow_query_log=1
slow_query_log_file=/var/log/mysql/slow-query.log
long_query_time=1
log_queries_not_using_indexes=1
Add log file and restart mysql server
touch /var/log/mysql/slow-query.log
chown mysql:adm /var/log/mysql/slow-query.log
chmod 640 slow-query.log
service mysql restart
Test slow logging with SQL queries
/* Activate query log - Maybe useful to show errors (not necessary) */
SET GLOBAL SLOW_QUERY_LOG=ON;
/* Check if slow query log is working */
SELECT SLEEP(2);
log_queries_not_using_indexes
Command-Line Format --log-queries-not-using-indexes
Option-File Format log-queries-not-using-indexes
Option Sets Variable Yes, log_queries_not_using_indexes
Variable Name log_queries_not_using_indexes
Variable Scope Global
Dynamic Variable Yes
Permitted Values
Type boolean
Whether queries that do not use indexes are logged to the slow query log. See Section 5.2.4,
In addition to a1ex07's answer you can use the shell command mk-query-digest to output a report of your running queries without using the log.
See the full methodology: http://www.xaprb.com/blog/2009/08/18/how-to-find-un-indexed-queries-in-mysql-without-using-the-log/
As mentioned in the article it is also possible to group the queries by tables doing --group-by tables --report-format profile
Useful to fast detect unindexed queries.
Is there any query/way to show the last queries executed on ALL servers?
For those blessed with MySQL >= 5.1.12, you can control this option globally at runtime:
Execute SET GLOBAL log_output = 'TABLE';
Execute SET GLOBAL general_log = 'ON';
Take a look at the table mysql.general_log
If you prefer to output to a file instead of a table:
SET GLOBAL log_output = "FILE"; the default.
SET GLOBAL general_log_file = "/path/to/your/logfile.log";
SET GLOBAL general_log = 'ON';
I prefer this method to editing .cnf files because:
you're not editing the my.cnf file and potentially permanently turning on logging
you're not fishing around the filesystem looking for the query log - or even worse, distracted by the need for the perfect destination. /var/log /var/data/log /opt /home/mysql_savior/var
You don't have to restart the server and interrupt any current connections to it.
restarting the server leaves you where you started (log is by default still off)
For more information, see
MySQL 5.1 Reference Manual - Server System Variables - general_log
You can enable a general query log for that sort of diagnostic. Generally you don't log all SELECT queries on a production server though, it's a performance killer.
Edit your MySQL config, e.g. /etc/mysql/my.cnf - look for, or add, a line like this
[mysqld]
log = /var/log/mysql/mysql.log
Restart mysql to pick up that change, now you can
tail -f /var/log/mysql/mysql.log
Hey presto, you can watch the queries as they come in.
SELECT * FROM mysql.general_log WHERE command_type ='Query' LIMIT total;
You can do the flowing thing for monitoring mysql query logs.
Open mysql configuration file my.cnf
sudo nano /etc/mysql/my.cnf
Search following lines under a [mysqld] heading and uncomment these lines to enable log
general_log_file = /var/log/mysql/mysql.log
general_log = 1
Restart your mysql server for reflect changes
sudo service mysql start
Monitor mysql server log with following command in terminal
tail -f /var/log/mysql/mysql.log
1) If general mysql logging is enabled then we can check the queries in the log file or table based what we have mentioned in the config.
Check what is enabled with the following command
mysql> show variables like 'general_log%';
mysql> show variables like 'log_output%';
If we need query history in table then
Execute SET GLOBAL log_output = 'TABLE';
Execute SET GLOBAL general_log = 'ON';
Take a look at the table mysql.general_log
If you prefer to output to a file:
SET GLOBAL log_output = "FILE"; which is set by default.
SET GLOBAL general_log_file = "/path/to/your/logfile.log";
SET GLOBAL general_log = 'ON';
2) We can also check the queries in the .mysql_history file
cat ~/.mysql_history
Maybe you could find that out by looking at the query log.
If you don't feel like changing your MySQL configuration you could use an SQL profiler like "Neor Profile SQL" http://www.profilesql.com .
If mysql binlog is enabled you can check the commands ran by user by executing
following command in linux console by browsing to mysql binlog directory
mysqlbinlog binlog.000001 > /tmp/statements.sql
enabling
[mysqld]
log = /var/log/mysql/mysql.log
or general log will have an effect on performance of mysql
After reading Paul's answer, I went on digging for more information on https://dev.mysql.com/doc/refman/5.7/en/query-log.html
I found a really useful code by a person. Here's the summary of the context.
(Note: The following code is not mine)
This script is an example to keep the table clean which will help you to reduce your table size. As after a day, there will be about 180k queries of log. ( in a file, it would be 30MB per day)
You need to add an additional column (event_unix) and then you can use this script to keep the log clean... it will update the timestamp into a Unix-timestamp, delete the logs older than 1 day and then update the event_time into Timestamp from event_unix... sounds a bit confusing, but it's working great.
Commands for the new column:
SET GLOBAL general_log = 'OFF';
RENAME TABLE general_log TO general_log_temp;
ALTER TABLE `general_log_temp`
ADD COLUMN `event_unix` int(10) NOT NULL AFTER `event_time`;
RENAME TABLE general_log_temp TO general_log;
SET GLOBAL general_log = 'ON';
Cleanup script:
SET GLOBAL general_log = 'OFF';
RENAME TABLE general_log TO general_log_temp;
UPDATE general_log_temp SET event_unix = UNIX_TIMESTAMP(event_time);
DELETE FROM `general_log_temp` WHERE `event_unix` < UNIX_TIMESTAMP(NOW()) - 86400;
UPDATE general_log_temp SET event_time = FROM_UNIXTIME(event_unix);
RENAME TABLE general_log_temp TO general_log;
SET GLOBAL general_log = 'ON';
Credit goes to Sebastian Kaiser (Original writer of the code).
Hope someone will find it useful as I did.
You can look at the following in linux
cd /root
ls -al
vi .mysql_history It may help
I'm importing a MySQL dump and getting the following error.
$ mysql foo < foo.sql
ERROR 1153 (08S01) at line 96: Got a packet bigger than 'max_allowed_packet' bytes
Apparently there are attachments in the database, which makes for very large inserts.
This is on my local machine, a Mac with MySQL 5 installed from the MySQL package.
Where do I change max_allowed_packet to be able to import the dump?
Is there anything else I should set?
Just running mysql --max_allowed_packet=32M … resulted in the same error.
You probably have to change it for both the client (you are running to do the import) AND the daemon mysqld that is running and accepting the import.
For the client, you can specify it on the command line:
mysql --max_allowed_packet=100M -u root -p database < dump.sql
Also, change the my.cnf or my.ini file (usually found in /etc/mysql/) under the mysqld section and set:
max_allowed_packet=100M
or you could run these commands in a MySQL console connected to that same server:
set global net_buffer_length=1000000;
set global max_allowed_packet=1000000000;
(Use a very large value for the packet size.)
As michaelpryor said, you have to change it for both the client and the daemon mysqld server.
His solution for the client command-line is good, but the ini files don't always do the trick, depending on configuration.
So, open a terminal, type mysql to get a mysql prompt, and issue these commands:
set global net_buffer_length=1000000;
set global max_allowed_packet=1000000000;
Keep the mysql prompt open, and run your command-line SQL execution on a second terminal..
This can be changed in your my.ini file (on Windows, located in \Program Files\MySQL\MySQL Server) under the server section, for example:
[mysqld]
max_allowed_packet = 10M
Re my.cnf on Mac OS X when using MySQL from the mysql.com dmg package distribution
By default, my.cnf is nowhere to be found.
You need to copy one of /usr/local/mysql/support-files/my*.cnf to /etc/my.cnf and restart mysqld. (Which you can do in the MySQL preference pane if you installed it.)
The fix is to increase the MySQL daemon’s max_allowed_packet. You can do this to a running daemon by logging in as Super and running the following commands.
# mysql -u admin -p
mysql> set global net_buffer_length=1000000;
Query OK, 0 rows affected (0.00 sec)
mysql> set global max_allowed_packet=1000000000;
Query OK, 0 rows affected (0.00 sec)
Then to import your dump:
gunzip < dump.sql.gz | mysql -u admin -p database
In etc/my.cnf try changing the max_allowed _packet and net_buffer_length to
max_allowed_packet=100000000
net_buffer_length=1000000
if this is not working then try changing to
max_allowed_packet=100M
net_buffer_length=100K
On CENTOS 6 /etc/my.cnf , under [mysqld] section the correct syntax is:
[mysqld]
# added to avoid err "Got a packet bigger than 'max_allowed_packet' bytes"
#
net_buffer_length=1000000
max_allowed_packet=1000000000
#
I have resolved my issue by this query
SET GLOBAL max_allowed_packet=1073741824;
and check max_allowed_packet with this query
SHOW VARIABLES LIKE 'max_allowed_packet';
Use a max_allowed_packet variable issuing a command like
mysql --max_allowed_packet=32M
-u root -p database < dump.sql
Slightly unrelated to your problem, so here's one for Google.
If you didn't mysqldump the SQL, it might be that your SQL is broken.
I just got this error by accidentally having an unclosed string literal in my code. Sloppy fingers happen.
That's a fantastic error message to get for a runaway string, thanks for that MySQL!
Error:
ERROR 1153 (08S01) at line 6772: Got a packet bigger than
'max_allowed_packet' bytes Operation failed with exitcode 1
QUERY:
SET GLOBAL max_allowed_packet=1073741824;
SHOW VARIABLES LIKE 'max_allowed_packet';
Max value:
Default Value (MySQL >= 8.0.3) 67108864
Default Value (MySQL <= 8.0.2) 4194304
Minimum Value 1024
Maximum Value 1073741824
Sometimes type setting:
max_allowed_packet = 16M
in my.ini is not working.
Try to determine the my.ini as follows:
set-variable = max_allowed_packet = 32M
or
set-variable = max_allowed_packet = 1000000000
Then restart the server:
/etc/init.d/mysql restart
It is a security risk to have max_allowed_packet at higher value, as an attacker can push bigger sized packets and crash the system.
So, Optimum Value of max_allowed_packet to be tuned and tested.
It is to better to change when required (using set global max_allowed_packet = xxx)
than to have it as part of my.ini or my.conf.
I am working in a shared hosting environment and I have hosted a website based on Drupal. I cannot edit the my.ini file or my.conf file too.
So, I deleted all the tables which were related to Cache and hence I could resolve this issue. Still I am looking for a perfect solution / way to handle this problem.
Edit - Deleting the tables created problems for me, coz Drupal was expecting that these tables should be existing. So I emptied the contents of these tables which solved the problem.
Set max_allowed_packet to the same (or more) than what it was when you dumped it with mysqldump. If you can't do that, make the dump again with a smaller value.
That is, assuming you dumped it with mysqldump. If you used some other tool, you're on your own.