To visually analyze mysql-slow.log, I ran goaccess -f mysql-slow.log in my terminal and got this error:
Nothing valid to process. Verify your date/time/log format.
My mysql-slow.log looks like this:
# User#Host: towfiqpiash[towfiqpiash] # ip-xx-xx-xxx-xxx.ec2.internal [xx.xx.xxx.xxx] Id: 396
# Query_time: 0.000180 Lock_time: 0.000056 Rows_sent: 38 Rows_examined: 39
SET timestamp=1482221404;
SELECT AQ.answer_id FROM `answer_quality_log` as AQ WHERE AQ.active=1;
I need appropriate date-format, time-format and log-format for goaccess configuration. Any help appreciated.
Related
How to list the DDL statements (CREATE, ALTER, DROP) processed in a period of time when a general log was recorded?
I was able to use pt-query-digest (a free command-line tool, part of the Percona Toolkit) this way:
pt-query-digest --type genlog --no-report --output slowlog \
--filter '$event->{fingerprint} =~ /^create|alter|drop/' \
/usr/local/var/mysql/bkarwin.log
Output:
# Time: 2021-11-18T16:16:53.452721Z
# Thread_id: 24
# Query_time: 0.000000 Lock_time: 0.000000 Rows_sent: 0 Rows_examined: 0
create table foo (i int);
# Time: 2021-11-18T16:16:54.805065Z
# Thread_id: 24
# Query_time: 0.000000 Lock_time: 0.000000 Rows_sent: 0 Rows_examined: 0
drop table foo;
Those two statements are just test DDL statements I ran after enabling the general query log on my localhost MySQL instance.
The pt-query-digest tool also has options for restricting a date range, --since and --until. But I got an error when I tried it. It seemed to have a problem with the date format.
Pipeline process 5 (since) caused an error: Argument "2021-11-18T16:16:53.452721Z" isn't numeric in numeric ge (>=) at /usr/local/bin/pt-query-digest line 13660, <$fh> line 2.
Perhaps you can just skip the date range options and eyeball the output yourself.
I have seen a lot of slow query logs, but never one like this:
/usr/sbin/mysqld, Version: 5.1.46-log (SUSE MySQL RPM). started with:
Tcp port: 3306 Unix socket: /var/run/mysql/mysql.sock
Time Id Command Argument
# Time: 160627 9:10:05
# User#Host: sysop[sysop] # [127.0.0.1]
# Query_time: 3.768728 Lock_time: 0.034402 Rows_sent: 734 Rows_examined: 734
use asterisk;
SET timestamp=1467033005;
select PostID, lead_id, list_id from vdlist_temp where Posted is null;
# Time: 160627 10:35:11
# User#Host: sysop[sysop] # [192.168.0.248]
# Query_time: 35.563521 Lock_time: 0.000054 Rows_sent: 1222017 Rows_examined: 2444034
SET timestamp=1467038111;
SELECT `vicidial_list`.`lead_id`, `vicidial_list`.`source_id` FROM `vicidial_list` ORDER BY `vicidial_list`.`source_id`;
# User#Host: sysop[sysop] # [127.0.0.1]
# Query_time: 0.000095 Lock_time: 18446744073699.406250 Rows_sent: 2 Rows_examined: 1
SET timestamp=1467038111;
call spUpdate_VDList_from_temp(0, 0, 1324903);
# Time: 160627 10:35:12
# User#Host: sysop[sysop] # [127.0.0.1]
# Query_time: 0.000055 Lock_time: 18446744073699.359375 Rows_sent: 0 Rows_examined: 0
SET timestamp=1467038112;
call spMoveXDrop(10376163);
# Time: 160627 11:26:14
# User#Host: sysop[sysop] # [127.0.0.1]
# Query_time: 0.000057 Lock_time: 18446744073697.218750 Rows_sent: 3 Rows_examined: 0
SET timestamp=1467041174;
call spUpdate_VDList_from_temp(10795520, 616062301, 1955758);
This seems to show I have queries waiting over 500,000 years for a lock. (I wrote the stored procedure and I'm not quite that old!) Somehow I don't think that's right. These are all MyISAM tables. (Not my choice.) I did a mysqldump and restore of the database, rebooted the server and I'm still getting seeing lock times like this.
Can anyone give me a clue where to look for the problem? (Server times are all good.)
EDIT:This MySql Version: 5.1.46-log that comes with an OpenSource project Vicidial. It seems clear that the Lock_time is a bug. Problem is that I'm looking at the slow query log to track down user complaints of slow web server response. I was hoping someone would know what triggers this bug to help me locate the actual problem. As you can see from the log, most slow queries have sane Lock_times. Both stored procedures and PHP generated queries generate the insane Lock_time. The only thing I see in common is they all select or update from the table vicidial_list. I dumped, dropped and recreated that table to no avail.
Sometimes the clock seems to run backwards. This has been a problem for more than a decade. It seems to be harmless. Ignore it as a bogus value.
Note that if a -1 is stored in a BIGINT UNSIGNED, you get a value very similar to the 18446744073699.406250 that you are seeing.
Those queries are not part of stock VICIdial, so I might suggest some query optimization for those custom queries you're trying to run. Also, in more recent versions of our VICIbox ISO installer we are using MariaDB, and a much newer version as well, which has a lot of bug fixes and optimizations compared to the older version of MySQL you are using.
I am trying to understand what is causing my lamp server to timeout a few times a day since four days, after running fine for 417 days.
Looking at running processes top may point to mysql. So I looked in the slow queries log, and only found this.
Can you make sense of it for me? The line select sleep(60)particularly worries me. Especially, since it is not in my website codebase.
# Time: 150125 17:16:43
# User#Host: brujobs_live[brujobs_live] # localhost []
# Query_time: 12.912479 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 0
use brujobs_live;
SET timestamp=1422202603;
select sleep(60);
# Time: 150125 17:20:17
# User#Host: root[root] # localhost []
# Query_time: 60.000274 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 0
SET timestamp=1422202817;
select sleep(60);
It seems that your event_scheduler is ON.
That would at least explain "sleep(60)".
Read this, it may helps you:
http://dev.mysql.com/doc/refman/5.1/en/events-configuration.html
http://mysqlopt.blogspot.co.at/2012/03/mysql-event-scheduler.html
Why can't I use slow_query_log on MySQL 5.6 on CentOS 6.4?
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
...
## Logging
## *** THESE LOGS WORK JUST FINE ***
log_error = /var/log/mysql/error.log
general_log_file = /var/log/mysql/mysql.log
general_log = 1
## *** THESE LOGS BREAK MYSQL ***
#log_slow_queries = /var/log/mysql/slow.log
#long_query_time = 5
#log-queries-not-using-indexes
Here's the /var/log/mysql directory:
$ ls -lh
total 100K
-rw-r----- 1 mysql root 47K Nov 22 06:02 error.log
-rw-rw---- 1 mysql root 42K Nov 22 06:05 mysql.log
-rw-rw---- 1 mysql mysql 0 Nov 22 06:01 slow.log
If I uncomment the log_slow_query lines in the /etc/my.cnf I receive the following error:
$ /etc/init.d/mysql restart
Shutting down MySQL.. SUCCESS!
Starting MySQL..... ERROR! The server quit without updating PID file (/var/lib/mysql/server.domain.com.pid).
What am I missing?
Looks like MySQL changed the format. Now it's slow_query_log not log_slow_queries.
This works:
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 5
How do I enable slow query log on my server? I have enabled it on my local host by adding log-slow-queries =[path] in my.ini file, but don't know how to add this on my server. My server is Linux-based and has PHP version 5.2.16.
Enabling slow query log has nothing to do with PHP version. You have to enable it in the MySQL server. You can enable in two ways
In runtime
During the server start
If your server is above 5.1.6 you can set the slow query log in the runtime itself. For which you have to execute this queries.
set global log_slow_queries = 1;
set global slow_query_log_file = <some file name>;
Or alternatively you can set the this options in the my.cnf/my.ini option files
log_slow_queries = 1;
slow_query_log_file = <some file name>;
Where the option file is changed, the MySQL server need to be restarted.
Location of the mysql option file can be found here http://dev.mysql.com/doc/refman/4.1/en/mysql-config-wizard-file-location.html
FYI : log_slow_queries was removed in MySQL 5.6.1 and slow_query_log is used instead.
http://dev.mysql.com/doc/refman/5.6/en/server-options.html#option_mysqld_log-slow-queries
But for performance you can set the log output (option log_output) to TABLE.
Also you can have a look other slow query log options like long_query_time, log-queries-not-using-indexes
You can set it temporarily, by running the following commands:
set global slow_query_log = 1;
set global slow_query_log_file = '/var/log/mysql-slow.log';
but your changes will be undone when mysql is restarted.
You can set it permanently, by adding the following to your my.cnf file:
slow-query-log=1
slow-query-log-file=/var/log/mysql-slow.log
The location of my.cnf varies by OS, but is often found in /etc/my.cnf, or /etc/mysql/my.cnf:
After saving your changes, you will need to restart MySql. This can vary by OS, but here are some common examples:
sudo /etc/init.d/mysqld restart
and
sudo service mysqld restart
First (as root:)
mkdir /var/log/mysql
touch /var/log/mysql/log-slow-queries.log
chown mysql:mysql -R /var/log/mysql
Then under [mysqld] in /etc/my.conf (or wherever your configuration file is:)
slow_query_log = 1
log-queries-not-using-indexes
long_query_time=1
log-slow-queries=/var/log/mysql/log-slow-queries.log
Next:
service mysql restart
Finally:
mysql
mysql> SELECT SLEEP(10);
mysql> quit
Verification:
cat /var/log/mysql/log-slow-queries.log
/usr/sbin/mysqld, Version: 5.5.42-cll (MySQL Community Server (GPL)). started with:
Tcp port: 0 Unix socket: (null)
Time Id Command Argument
# Time: 150727 0:05:17
# User#Host: root[root] # localhost []
# Query_time: 10.000215 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 0
SET timestamp=1437973517;
select sleep(10);