I have seen a lot of slow query logs, but never one like this:
/usr/sbin/mysqld, Version: 5.1.46-log (SUSE MySQL RPM). started with:
Tcp port: 3306 Unix socket: /var/run/mysql/mysql.sock
Time Id Command Argument
# Time: 160627 9:10:05
# User#Host: sysop[sysop] # [127.0.0.1]
# Query_time: 3.768728 Lock_time: 0.034402 Rows_sent: 734 Rows_examined: 734
use asterisk;
SET timestamp=1467033005;
select PostID, lead_id, list_id from vdlist_temp where Posted is null;
# Time: 160627 10:35:11
# User#Host: sysop[sysop] # [192.168.0.248]
# Query_time: 35.563521 Lock_time: 0.000054 Rows_sent: 1222017 Rows_examined: 2444034
SET timestamp=1467038111;
SELECT `vicidial_list`.`lead_id`, `vicidial_list`.`source_id` FROM `vicidial_list` ORDER BY `vicidial_list`.`source_id`;
# User#Host: sysop[sysop] # [127.0.0.1]
# Query_time: 0.000095 Lock_time: 18446744073699.406250 Rows_sent: 2 Rows_examined: 1
SET timestamp=1467038111;
call spUpdate_VDList_from_temp(0, 0, 1324903);
# Time: 160627 10:35:12
# User#Host: sysop[sysop] # [127.0.0.1]
# Query_time: 0.000055 Lock_time: 18446744073699.359375 Rows_sent: 0 Rows_examined: 0
SET timestamp=1467038112;
call spMoveXDrop(10376163);
# Time: 160627 11:26:14
# User#Host: sysop[sysop] # [127.0.0.1]
# Query_time: 0.000057 Lock_time: 18446744073697.218750 Rows_sent: 3 Rows_examined: 0
SET timestamp=1467041174;
call spUpdate_VDList_from_temp(10795520, 616062301, 1955758);
This seems to show I have queries waiting over 500,000 years for a lock. (I wrote the stored procedure and I'm not quite that old!) Somehow I don't think that's right. These are all MyISAM tables. (Not my choice.) I did a mysqldump and restore of the database, rebooted the server and I'm still getting seeing lock times like this.
Can anyone give me a clue where to look for the problem? (Server times are all good.)
EDIT:This MySql Version: 5.1.46-log that comes with an OpenSource project Vicidial. It seems clear that the Lock_time is a bug. Problem is that I'm looking at the slow query log to track down user complaints of slow web server response. I was hoping someone would know what triggers this bug to help me locate the actual problem. As you can see from the log, most slow queries have sane Lock_times. Both stored procedures and PHP generated queries generate the insane Lock_time. The only thing I see in common is they all select or update from the table vicidial_list. I dumped, dropped and recreated that table to no avail.
Sometimes the clock seems to run backwards. This has been a problem for more than a decade. It seems to be harmless. Ignore it as a bogus value.
Note that if a -1 is stored in a BIGINT UNSIGNED, you get a value very similar to the 18446744073699.406250 that you are seeing.
Those queries are not part of stock VICIdial, so I might suggest some query optimization for those custom queries you're trying to run. Also, in more recent versions of our VICIbox ISO installer we are using MariaDB, and a much newer version as well, which has a lot of bug fixes and optimizations compared to the older version of MySQL you are using.
Related
How to list the DDL statements (CREATE, ALTER, DROP) processed in a period of time when a general log was recorded?
I was able to use pt-query-digest (a free command-line tool, part of the Percona Toolkit) this way:
pt-query-digest --type genlog --no-report --output slowlog \
--filter '$event->{fingerprint} =~ /^create|alter|drop/' \
/usr/local/var/mysql/bkarwin.log
Output:
# Time: 2021-11-18T16:16:53.452721Z
# Thread_id: 24
# Query_time: 0.000000 Lock_time: 0.000000 Rows_sent: 0 Rows_examined: 0
create table foo (i int);
# Time: 2021-11-18T16:16:54.805065Z
# Thread_id: 24
# Query_time: 0.000000 Lock_time: 0.000000 Rows_sent: 0 Rows_examined: 0
drop table foo;
Those two statements are just test DDL statements I ran after enabling the general query log on my localhost MySQL instance.
The pt-query-digest tool also has options for restricting a date range, --since and --until. But I got an error when I tried it. It seemed to have a problem with the date format.
Pipeline process 5 (since) caused an error: Argument "2021-11-18T16:16:53.452721Z" isn't numeric in numeric ge (>=) at /usr/local/bin/pt-query-digest line 13660, <$fh> line 2.
Perhaps you can just skip the date range options and eyeball the output yourself.
To visually analyze mysql-slow.log, I ran goaccess -f mysql-slow.log in my terminal and got this error:
Nothing valid to process. Verify your date/time/log format.
My mysql-slow.log looks like this:
# User#Host: towfiqpiash[towfiqpiash] # ip-xx-xx-xxx-xxx.ec2.internal [xx.xx.xxx.xxx] Id: 396
# Query_time: 0.000180 Lock_time: 0.000056 Rows_sent: 38 Rows_examined: 39
SET timestamp=1482221404;
SELECT AQ.answer_id FROM `answer_quality_log` as AQ WHERE AQ.active=1;
I need appropriate date-format, time-format and log-format for goaccess configuration. Any help appreciated.
I recently got new hardware but I'm having problem with mysql. It's slower than it was on the old server. When I ran sysbench tool on new server I got this:
sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password= --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 8
Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 7 times)
Done.
OLTP test statistics:
queries performed:
read: 365694
write: 0
other: 52242
total: 417936
transactions: 26121 (435.14 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 365694 (6091.93 per sec.)
other operations: 52242 (870.28 per sec.)
Test execution summary:
total time: 60.0293s
total number of events: 26121
total time taken by event execution: 479.9747
per-request statistics:
min: 0.52ms
avg: 18.38ms
max: 117.91ms
approx. 95 percentile: 53.21ms
Threads fairness:
events (avg/stddev): 3265.1250/19.37
execution time (avg/stddev): 59.9968/0.01
Old server produced this:
sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password= --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 8
Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 7 times)
Done.
OLTP test statistics:
queries performed:
read: 952294
write: 0
other: 136042
total: 1088336
transactions: 68021 (1133.58 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 952294 (15870.18 per sec.)
other operations: 136042 (2267.17 per sec.)
Test execution summary:
total time: 60.0052s
total number of events: 68021
total time taken by event execution: 479.5141
per-request statistics:
min: 3.27ms
avg: 7.05ms
max: 31.63ms
approx. 95 percentile: 8.60ms
Threads fairness:
events (avg/stddev): 8502.6250/11.10
execution time (avg/stddev): 59.9393/0.00
Old server has E3-1230 cpu and 16GB ram. New server has E5-1660 v3 and 32GB ram. OS is debian wheezy on both, and mysql version is 5.5.43. My.cnf below
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 256M
key_buffer_size = 256M
sort_buffer_size = 2M
read_rnd_buffer_size = 4M
innodb_buffer_pool_size = 1G
max_allowed_packet = 32M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
max_connections = 300
table_cache = 256M
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 200M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#slow_query_log_file = /var/log/mysql/mysql-slow.log
#slow_query_log = 1
#long_query_time = 2
#log_queries_not_using_indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
Any ideas?
query_cache_size = 200M
Turn off the Query cache. Even if you choose to keep it on, don't set the size bigger than 50M because of the cost of pruning.
table_cache = 256M
OUCH! You do not need a quarter billion tables!! Change to a few hundred (no suffix).
Do this on both machines, then diff the outputs:
SHOW VARIABLES LIKE 'innodb%';
Some defaults have changed over time; this should point them out.
Agree with Rick that your config is probably not optimal, but if you really are running the same mysql config on both machines, then that does not explain the discrepancy. All the potential reasons for this would take far too long to list. Assuming that you gathered the stats in the same way on both systems (cold caches) then the most likely cause is IO related, you could verify this using a disk benchmarking tool - I would recommend afio if you don't already have something you know well which is appropriate for the task. Fixing the problem is an even more complex question.
I am trying to understand what is causing my lamp server to timeout a few times a day since four days, after running fine for 417 days.
Looking at running processes top may point to mysql. So I looked in the slow queries log, and only found this.
Can you make sense of it for me? The line select sleep(60)particularly worries me. Especially, since it is not in my website codebase.
# Time: 150125 17:16:43
# User#Host: brujobs_live[brujobs_live] # localhost []
# Query_time: 12.912479 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 0
use brujobs_live;
SET timestamp=1422202603;
select sleep(60);
# Time: 150125 17:20:17
# User#Host: root[root] # localhost []
# Query_time: 60.000274 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 0
SET timestamp=1422202817;
select sleep(60);
It seems that your event_scheduler is ON.
That would at least explain "sleep(60)".
Read this, it may helps you:
http://dev.mysql.com/doc/refman/5.1/en/events-configuration.html
http://mysqlopt.blogspot.co.at/2012/03/mysql-event-scheduler.html
How do I enable slow query log on my server? I have enabled it on my local host by adding log-slow-queries =[path] in my.ini file, but don't know how to add this on my server. My server is Linux-based and has PHP version 5.2.16.
Enabling slow query log has nothing to do with PHP version. You have to enable it in the MySQL server. You can enable in two ways
In runtime
During the server start
If your server is above 5.1.6 you can set the slow query log in the runtime itself. For which you have to execute this queries.
set global log_slow_queries = 1;
set global slow_query_log_file = <some file name>;
Or alternatively you can set the this options in the my.cnf/my.ini option files
log_slow_queries = 1;
slow_query_log_file = <some file name>;
Where the option file is changed, the MySQL server need to be restarted.
Location of the mysql option file can be found here http://dev.mysql.com/doc/refman/4.1/en/mysql-config-wizard-file-location.html
FYI : log_slow_queries was removed in MySQL 5.6.1 and slow_query_log is used instead.
http://dev.mysql.com/doc/refman/5.6/en/server-options.html#option_mysqld_log-slow-queries
But for performance you can set the log output (option log_output) to TABLE.
Also you can have a look other slow query log options like long_query_time, log-queries-not-using-indexes
You can set it temporarily, by running the following commands:
set global slow_query_log = 1;
set global slow_query_log_file = '/var/log/mysql-slow.log';
but your changes will be undone when mysql is restarted.
You can set it permanently, by adding the following to your my.cnf file:
slow-query-log=1
slow-query-log-file=/var/log/mysql-slow.log
The location of my.cnf varies by OS, but is often found in /etc/my.cnf, or /etc/mysql/my.cnf:
After saving your changes, you will need to restart MySql. This can vary by OS, but here are some common examples:
sudo /etc/init.d/mysqld restart
and
sudo service mysqld restart
First (as root:)
mkdir /var/log/mysql
touch /var/log/mysql/log-slow-queries.log
chown mysql:mysql -R /var/log/mysql
Then under [mysqld] in /etc/my.conf (or wherever your configuration file is:)
slow_query_log = 1
log-queries-not-using-indexes
long_query_time=1
log-slow-queries=/var/log/mysql/log-slow-queries.log
Next:
service mysql restart
Finally:
mysql
mysql> SELECT SLEEP(10);
mysql> quit
Verification:
cat /var/log/mysql/log-slow-queries.log
/usr/sbin/mysqld, Version: 5.5.42-cll (MySQL Community Server (GPL)). started with:
Tcp port: 0 Unix socket: (null)
Time Id Command Argument
# Time: 150727 0:05:17
# User#Host: root[root] # localhost []
# Query_time: 10.000215 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 0
SET timestamp=1437973517;
select sleep(10);