I have looked at many similar questions to this but I can't seem to find the answer. I would like to set up the slow query log for my MySQL database. I have seen many answers saying I should access the MySQL command line tool. I am not sure exactly how to find this tool but I tried accessing it by going to:
c:/xampp/mysql/bin/mysql -u root -p -h localhost
But here I get MariaDB, which seems to be different from any other answers/tutorials I have seen before. Typing in:
set log_slow_queries = ON;
gives me the error
ERROR 1193 (HY000): Unknown system variable 'log_slow_queries'
SET GLOBAL slow_query_log=1;
The Slow Query Log consists of log events for queries taking up to long_query_time seconds to finish. For instance, up to 10 seconds to complete. To see the time threshold currently set, issue the following:
SELECT ##long_query_time;
+-------------------+
| ##long_query_time |
+-------------------+
| 10.000000 |
+-------------------+
It can be set as a GLOBAL variable, in my.cnf or my.ini file. Or it can be set by the connection, though this is unusual. The value can be set between 0 to 10 (seconds). What value to use?
10 is so high as to be almost useless;
2 is a compromise;
0.5 and other fractions are possible;
0 captures everything; this could fill up disk dangerously fast, but can be very useful.
The capturing of slow queries is either turned on or off. And the file logged to is also specified. The below captures these concepts:
SELECT ##slow_query_log; -- Is capture currently active? (1=On, 0=Off)
SELECT ##slow_query_log_file; -- filename for capture. Resides in datadir
SELECT ##datadir; -- to see current value of the location for capture file
SET GLOBAL slow_query_log=0; -- Turn Off
-- make a backup of the Slow Query Log capture file. Then delete it.
SET GLOBAL slow_query_log=1; -- Turn it back On (new empty file is created)
For more information, please see the MySQL Manual Page The Slow Query Log
Note: The above information on turning on/off the slowlog was changed in 5.6(?); older version had another mechanism.
The "best" way to see what is slowing down your system:
long_query_time=...
turn on the slowlog
run for a few hours
turn off the slowlog (or raise the cutoff)
run pt-query-digest to find the 'worst' couple of queries. Or mysqldumpslow -s t
Go to xampp control panel click on config button for mysql and select my.ini then add these lines in my.ini file
slow_query_log = 1
slow-query-log-file=/path/of/the/log/file.log
I put above two lines under the log_error = "mysql_error.log". the modified part of the my.ini file should look like this
# The MySQL server
[mysqld]
port= 3306
socket = "C:/xampp/mysql/mysql.sock"
basedir = "C:/xampp/mysql"
tmpdir = "C:/xampp/tmp"
datadir = "C:/xampp/mysql/data"
pid_file = "mysql.pid"
# enable-named-pipe
key_buffer = 16M
max_allowed_packet = 1M
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
log_error = "mysql_error.log"
slow_query_log = 1
slow-query-log-file=/var/log/mysql-slow.log
Then Restart the MySQL server in xampp control panel. and now slow_query_log should be enabled, you can confirm it by running following command in the MySQL shell
show variables like '%slow%';
It might be obvious but it took me time before I realized my mistake: in the my.ini file you should put the slow_query_log settings in the [mysqld] group, not simply at the end of the my.ini file....
Related
I can't seem to get MySQL to change the global value of max_allowed_packet no matter where i change it.
In /usr/local/mysql/support-files/.my.cnf, I have:
...
# The MySQL server
[mysqld]
port = 3306
socket = /tmp/mysql.sock
skip-external-locking
key_buffer_size = 384M
max_allowed_packet = 32M
...
In ~/.my.cnf, I have:
[client]
socket = /var/mysql/mysql.sock
user=rc
password=shop
# database=application_data
[mysql]
prompt='MYSQL \u#\h:\d > '
# wait_timeout = 576000
[mysqld]
socket = /var/mysql/mysql.sock
max_allowed_packet=64M
[mysqldump]
I have different values for max_allowed_packet in the two cnf files to see if one supersedes the other. However, neither config is being used. When I check in MySQL to see the value in use, I get:
MYSQL vicinio#localhost:(none) > SHOW VARIABLES LIKE 'max_allowed_packet';
+--------------------+---------+
| Variable_name | Value |
+--------------------+---------+
| max_allowed_packet | 1048576 |
+--------------------+---------+
1 row in set (0.00 sec)
In my System Preferences, I can restart MySQL Server using a preference pane; I am pretty sure that pref pane is using the /Library/StartupItems/MySQLCOM/MySQLCOM script, and I can see in that script that the location it is using for mysql server is the same place I am changing configs:
# The path to the mysql.server init script. The official MySQL
# Mac OS X packages are being installed into /usr/local/mysql.
SCRIPT="/usr/local/mysql/support-files/mysql.server"
Any thoughts?
ETA: One additional note, I can change the global value in MySQL as a user, and that fixes the immediate problem, but every time the Mysql server is restarted the problem will recur. Not to mention I'd really like to understand why my configs aren't having the desired effect.
I have read various other q's on SO as well as googling but haven't found anything on point.
This might be helpful: https://dba.stackexchange.com/questions/45087/max-allowed-packet-in-mysql
Be sure to read that first comment closely. It has a couple of nuanced points for both the files and the settings in the database itself.
I am hoping that someone on here can help with an issue I am having on a webserver. Currently the server hosts a web application which we have purchased for helpdesking. Every 8 hours when someone navigates to the page they will be presented with this error:
[Warning]: PDO::__construct(): MySQL server has gone away (Database/class.SWIFT_Database.php:334)
I have tried setting the wait_timeout to the max value in the /etc/my.cnf file ie;
wait_timeout=31536000
Also from the mysql I have set the global wait_timeout to this value and set the session wait_time out to the same.
I have noticed some strange behaviour, when you restart the mysql service, the wait timeout resets to the 28800 default, despite it being set in the my.cnf. I can't find any other config files for mysql though happy to look if someone can point me in the direction.
Also this morning when I logged into the server and ran the following command, the session wait_timeout value had reverted!
mysql> select ##global.wait_timeout, ##session.wait_timeout;
+-----------------------+------------------------+
| ##global.wait_timeout | ##session.wait_timeout |
+-----------------------+------------------------+
| 31536000 | 28800 |
+-----------------------+------------------------+
1 row in set (0.00 sec)
I see that this is an issue for a lot of people online with various webapps running on mysql, but no one seems to have a fix. A lot of advice online points to the wait_timeout but it doesn't seem to be changing the bug that I am seeing. I have tried out the fixes on the mysql manual about this issue, but still no luck (link: http://dev.mysql.com/doc/refman/5.0/en/gone-away.html)
Any advice would be greatly appreciated. Server details and product versions below:
Server: OpenSuse 11.4
MySql Version: 5.1.53
Many Thanks In Advance!
While not very elegant, the following snippet has helped me get rid of the timeouts and keep persistent connections. It will throw the exception if the connection fails $limit times in a row, though if the problem is timeouts you will only need 1 retry at most.
$db = null;
$limit = 10;
$counter = 0;
while (true) {
try {
$db = new PDO('mysql:host=' . db_host . ';dbname=' . db_name, db_user, db_pass);
$db->exec( "SET CHARACTER SET utf8" );
$db->setAttribute( PDO::ATTR_DEFAULT_FETCH_MODE, PDO::FETCH_ASSOC );
$db->setAttribute( PDO::ATTR_PERSISTENT, true );
break;
}
catch (Exception $e) {
$db = null;
$counter++;
if ($counter == $limit)
throw $e;
}
}
This article on Drupal.org may be helpful in your situation:
MySQL comes with a default configuration of the resources it is going to use, specified in "my.cnf" (Linux) or "my.ini" (Windows) during the installation of MySQL.
In Linux this file is located at /etc/my.cnf to set global options, or /usr/local/var/mysql-data-dir/my.cnf to set server-specific options.
In Windows this file is located by default at C:\Program Files\MySQL\MySQL Server X.Y\my.ini.
Resources allowed by the default configuration are normally insufficient to run a resource-intensive application. You must modify the following resource specifications if they are available in your original configuration file, or add them to the configuration file if they are not already specified (because some are not present by default) :
Important: Remember to keep backup files before you do anything! You will also have to reload the MySQL service after making changes to these configuration files.
MyISAM specifications:
[mysqld]
port = 3306
socket = /tmp/mysql.sock
skip-external-locking
key_buffer = 384M
max_allowed_packet = 64M
table_cache = 4096
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 64M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size = 32M
InnoDB specifications:
innodb_buffer_pool_size = 384M
innodb_additional_mem_pool_size = 20M
innodb_log_file_size = 10M
innodb_log_buffer_size = 64M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 180
Note: It is assumed here that you are using the InnoDB database tables, as Drupal is a resource intensive application. If you are not using the InnoDB database tables try to change this, in view of the fact that you are getting the Warning: MySQL server has gone away - apparently meaning that your setup is resource intensive.
There are various reasons that a connection gets closed.
Reference:
https://dev.mysql.com/doc/refman/5.0/en/gone-away.html
I too faced the similar problem on using PDO where the hosting administrator kills the connection if it sleeps more than a minute. Hence I came up with my own class which will wrap the PDO class. This will detect whether a connection is closed and will try to reconnect on query execution.
Answer Below
PDO: MySQL server has gone away
I need to log SQL to a file so i can check later which SQL run.
so i opened opt/lampp/etc/my.cnf and add these lines
log_slow_queries
log_queries_not_using_indexes =1
long_query_time = 1
slow_query_log = 1
slow_query_log_file = "/opt/lampp/logs/query.log"
but it did not logged the queries it even did not created the query.log file, so i created an empty file with the name, but still it's not working.
Edit
[mysqld]
log_slow_queries
log_queries_not_using_indexes =1
long_query_time = 1
slow_query_log = 1
general_log = 1
slow_query_log_file = /opt/lampp/logs/query.log
general_log_file = "/opt/lampp/logs/query.log"
This will only log slow queries. You need the general log if you want to see all queries.
general_log = 1
general_log_file = "/opt/lampp/logs/query.log"
Note that you'll need to restart the server for this to take effect. Also, you should only use this type of logging during testing as it does cause slowdown.
As other users mentioned, this could be a permissions issue. First, check what user MySQL is running as via ps -u -p $(pgrep mysql). The username will be displayed on the first column under USER. In your case, it seems the user is nobody. You can view the default group of a user via groups nobody. This should print something like nobody : nogroup.
To fix the permissions on the file, just run chown nobody:nogroup /opt/lampp/logs/query.log.
Be sure to give the correct permission :
chown mysql:mysql filename
also when i last did it , i had to restart the mysql service :
service mysqld restart
log_slow_queries
is deprecated
It now has to look like that:
slow_query_log
log_queries_not_using_indexes =1
long_query_time = 1
slow_query_log = 1
general_log = 1
slow_query_log_file = /var/log/mysql/mysql-slow.log
general_log_file = /var/log/mysql/mysql-slow.log
The process probably doesn't have permission to write to that directory. Make sure MySQL has permission to write there, or try logging somewhere less restricted.
This will only log slow queries. You need the general log if you want to see all queries.
general_log = 1
general_log_file = "/opt/lampp/logs/query.log"
Note that you'll need to restart the server for this to take effect. Also, you should only use this type of logging during testing as it does cause slowdown.
Also Note that mysql needs permissions over folder too, in my case, I changed:
general_log_file = "/opt/lampp/logs/query.log"
for
general_log_file = "/var/log/mysql/query.log"
But I have mysql installed from software center, without lampp, when I execute ls -l over /var/log/, it shows
drwx------ 8 mysql mysql 4096 sep 25 23:22 mysql
PD:I change the my.cn file and restart mysql, without create the query.log file in the specified path, mysql automatically create it
How do I enable the MySQL function that logs each SQL query statement received from clients and the time that query statement has submitted?
Can I do that in phpmyadmin or NaviCat?
How do I analyse the log?
First, Remember that this logfile can grow very large on a busy server.
For mysql < 5.1.29:
To enable the query log, put this in /etc/my.cnf in the [mysqld] section
log = /path/to/query.log #works for mysql < 5.1.29
Also, to enable it from MySQL console
SET general_log = 1;
See http://dev.mysql.com/doc/refman/5.1/en/query-log.html
For mysql 5.1.29+
With mysql 5.1.29+ , the log option is deprecated. To specify the logfile and enable logging, use this in my.cnf in the [mysqld] section:
general_log_file = /path/to/query.log
general_log = 1
Alternately, to turn on logging from MySQL console (must also specify log file location somehow, or find the default location):
SET global general_log = 1;
Also note that there are additional options to log only slow queries, or those which do not use indexes.
Take a look on this answer to another related question. It shows how to enable, disable and to see the logs on live servers without restarting.
Log all queries in mysql
Here is a summary:
If you don't want or cannot restart the MySQL server you can proceed like this on your running server:
Create your log tables (see answer)
Enable Query logging on the database
(Note that the string 'table' should be put literally and not substituted by any table name. Thanks Nicholas Pickering)
SET global general_log = 1;
SET global log_output = 'table';
View the log
select * from mysql.general_log;
Disable Query logging on the database
SET global general_log = 0;
Clear query logs without disabling
TRUNCATE mysql.general_log
This was already in a comment, but deserves its own answer:
Without editing the config files: in mysql, as root, do
SET global general_log_file='/tmp/mysql.log';
SET global log_output = 'file';
SET global general_log = on;
Don't forget to turn it off afterwards:
SET global general_log = off;
I use this method for logging when I want to quickly optimize different page loads.
It's a little tip...
Logging to a TABLE
SET global general_log = 1;
SET global log_output = 'table';
You can then select from my mysql.general_log table to retrieve recent queries.
I can then do something similar to tail -f on the mysql.log, but with more refinements...
select * from mysql.general_log
where event_time > (now() - INTERVAL 8 SECOND) and thread_id not in(9 , 628)
and argument <> "SELECT 1" and argument <> ""
and argument <> "SET NAMES 'UTF8'" and argument <> "SHOW STATUS"
and command_type = "Query" and argument <> "SET PROFILING=1"
This makes it easy to see my queries that I can try and cut back. I use 8 seconds interval to only fetch queries executed within the last 8 seconds.
You can disable or enable the general query log (which logs all queries) with
SET GLOBAL general_log = 1 # (or 0 to disable)
// To see global variable is enabled or not and location of query log
SHOW VARIABLES like 'general%';
// Set query log on
SET GLOBAL general_log = ON;
I also wanted to enable the MySQL log file to see the queries and I have resolved this with the below instructions
Go to /etc/mysql/mysql.conf.d
open the mysqld.cnf
and enable the below lines
general_log_file = /var/log/mysql/mysql.log
general_log = 1
restart the MySQL with this command /etc/init.d/mysql restart
go to /var/log/mysql/ and check the logs
On Windows you can simply go to
C:\wamp\bin\mysql\mysql5.1.53\my.ini
Insert this line in my.ini
general_log_file = c:/wamp/logs/mysql_query_log.log
The my.ini file finally looks like this
...
...
...
socket = /tmp/mysql.sock
skip-locking
key_buffer = 16M
max_allowed_packet = 1M
table_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
basedir=c:/wamp/bin/mysql/mysql5.1.53
log = c:/wamp/logs/mysql_query_log.log #dump query logs in this file
log-error=c:/wamp/logs/mysql.log
datadir=c:/wamp/bin/mysql/mysql5.1.53/data
...
...
...
...
There is bug in MySQL 5.6 version.
Even mysqld show as :
Default options are read from the following files in the given order:
C:\Windows\my.ini C:\Windows\my.cnf C:\my.ini C:\my.cnf c:\Program Files (x86)\MySQL\MySQL Server 5.6\my.ini c:\Program Files (x86)\MySQL\MySQL Server 5.6\my.cnf
Realy settings are reading in following order :
Default options are read from the following files in the given order:
C:\ProgramData\MySQL\MySQL Server 5.6\my.ini C:\Windows\my.ini C:\Windows\my.cnf C:\my.ini C:\my.cnf c:\Program Files (x86)\MySQL\MySQL Server 5.6\my.ini c:\Program Files (x86)\MySQL\MySQL Server 5.6\my.cnf
Check file: "C:\ProgramData\MySQL\MySQL Server 5.6\my.ini"
Hope it help somebody.
for mysql>=5.5 only for slow queries (1 second and more)
my.cfg
[mysqld]
slow-query-log = 1
slow-query-log-file = /var/log/mysql/mysql-slow.log
long_query_time = 1
log-queries-not-using-indexes
To enable the query log in MAC Machine:
Open the following file:
vi /private/etc/my.cnf
Set the query log url under 'mysqld' section as follows:
[mysqld]
general_log_file=/Users/kumanan/Documents/mysql_query.log
Few machine’s are not logging query properly, So that case you can enable it from MySQL console
mysql> SET global general_log = 1;
Not exactly an answer to the question because the question already has great answers. This is a side info. Enabling general_log really put a dent on MySQL performance. I left general_log =1 accidentally on a production server and spent hours finding out why performance was not comparable to a similar setup on other servers. Then I found this which explains the impact of enabling general log. http://www.fromdual.com/general_query_log_vs_mysql_performance.
Gist of the story, don't put general_log=1 in the .cnf file. Instead use set global general_log =1 for a brief duration just to log enough to find out what you are trying to find out and then turn it off.
In phpMyAdmin 4.0, you go to Status > Monitor. In there you can enable the slow query log and general log, see a live monitor, select a portion of the graph, see the related queries and analyse them.
I had to drop and recreate the general log at one point. During the recreation, character sets got messed up and I ended up having this error in the logs:
[ERROR] Incorrect definition of table mysql.general_log: expected the type of column 'user_host' at position 1 to have character set 'utf8' but found character set 'latin1'
So if the standard answer of "check to make sure logging is on" doesn't work for you, check to make sure your fields have the right character set.
My OS Win10, MySQL server version - 5.7
The path to my.ini
C:\ProgramData\MySQL\MySQL Server 5.7\my.ini
Just add into my.ini file
general_log_file = C:/ProgramData/MySQL/MySQL Server 5.7/mysql.log
general_log = 1
You may come across a set of Hexadecimal values, like this (argument column):
mysql> select * from mysql.general_log LIMIT 1\G
*************************** 1. row ***************************
event_time: 2023-01-27 13:37:20.950778
user_host: root[root] # localhost []
thread_id: 1434
server_id: 1
command_type: Query
argument: 0x73656C656374202A2066726F6D207573657273
1 row in set (0.00 sec)
so to make it readable, just use:
select a.*, convert(a.argument using utf8) from mysql.general_log a;
And the return is something like this:
mysql> select a.*, convert(a.argument using utf8) from mysql.general_log a LIMIT 1\G
*************************** 1. row ***************************
event_time: 2023-01-27 13:37:20.950778
user_host: root[root] # localhost []
thread_id: 1434
server_id: 1
command_type: Query
argument: 0x73656C656374202A2066726F6D207573657273
convert(a.argument using utf8): select * from users
1 row in set, 1 warning (0.00 sec)
Ps: I used LIMIT 1 on examples, because my log table is too big.
I am running a drupal site. I got an error in my site user warning: Got a packet bigger than 'max_allowed_packet' bytes query. I have set the value as high as 128M . Even after that same error is reported.
What is the issue here?? Why is it not working ??
Is there a maximum limit for the value max_allowed_packet ?
This is the bleeding edge:
set global max_allowed_packet=1073741824;
Although, it is probably not a good idea to set it that high in your case.
As a side note, I experienced this error with mysqldump, and setting this maximum didn't help. This did the trick: $ mysqldump --max_allowed_packet=999M -u root -p table_name > table_name.sql
Often this can be caused by the variables not actually taking effect- you make the configuration change, but in the wrong my.cnf, or you forget to bounce the app, etc.
An easy way to check a running mysql instance is to do something like this in a shell:
mysqladmin variables -u root -p
and enter in your root password. This will dump all of the current variables (including max_allowed_packet), and will let you verify what it's set to. If it's set to 128M and you're still choking on it, then you'll need to increase it- but it's pretty unlikely.
first is you need to set your max_allowed_packet to 128M in your my.cnf file.
to find it, use "locate my.cnf" command in your command line.
the file should look like this:
#
!includedir /etc/my.cnf.d
#max_allowed_packet = 1024M
[mysqld]
port = 3306
key_buffer_size = 256M
# max_allowed_packet = 100M
table_open_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M
thread_concurrency = 8
bind-address = 202.90.158.47
# skip-networking
log = /var/log/mysql.access.log
log-error = /var/log/mysql.error.log
wait_timeout = 1
[mysqldump]
#max_allowed_packet = 101M
be sure to uncomment(remove the # sign before the max_allowed_packet = 128M line)
and finally, restart your sql using command "/etc/init.d/mysqld restart"
that should do the trick. :D
Edit your /etc/my.cnf, adding the max_allowed_packet variable.
It should look like this:
[mysqld]
max_allowed_packet=1000000000
Then restart your server.
Try setting max_allowed_packet = 128M as the very last option under the [mysqld] category of my.cnf.
When I had it as the first option, it would not work, but when I had it as the last option, it worked! I think this is because some other variables were over-riding max_allowed_packet.
After changing my.cnf, restart MySQL using sudo service mysql restart, and check the setting using SHOW VARIABLES LIKE 'max_allowed_packet';
You need to set the setting in all sections that apply to the action you are doing, and always in the [MySQLd] section. The setting applies to the buffer of the elements you are using. So under [MySQLd] is for your MySQL server proces deamon on linux / service on windows. And if you want to make a dump with MySQLDump add it as parameter on the command line or make a section [MySQLDump] in your my.ini as well for this tool with the same parameter to make it permanent. If you want to import the dump again with MySQL again use the parameter on the command line or make a section [MySQL] with again the same parameter in your my.ini to make the choice permanent for this tools also.
I kept on talking about my.ini because i am on windows but on linux that is my.cnf of course.
I decided to explain it here because it took me ages to figure this out because it is not explained anywhere. In examples however i noticed some ppl having the setting under multiple sections so i started to google more and found correlation between the sections and the actions they where doing. Now i never have this problem anymore and settings as high as the mentioned 128M here are not needed in most cases. However because it's the maximum the server will use for this buffer if you have the memory just set it high enough to never get into trouble with your actions. The size you actually need is a little bit larger than the largest record in your database.
I experienced this error with mysqldump with LONGBLOB fields, and setting this maximum didn't help. This did the trick:
$ mysqldump --max_allowed_packet=999M -u root -p table_name > table_name.sql