Is it possible to timeout a query in MySQL?
That is, if any query exceeds the time I specify, it will be killed by MySQL and it will return an error instead of waiting for eternity.
There is a nice Perl script on CPAN to do just this:
http://search.cpan.org/~rsoliv/mysql-genocide-0.03/mysql-genocide
One only needs to schedule it to run with the proper parameters. Create a CRONtab file /etc/cron.d/mysql_query_timeout to schedule it to run every minute:
* * * * * root /path/to/mysql-genocide -t 7200 -s -K
Where 7200 is the maxiumum allowed execution time in seconds. The -s switch filters out all except SELECT queries. The -K switch instructs the script to kill the matching processes.
The root user should be able to run local mysql tools without authentication otherwise you will need to provide credentials on the command line.
I just set up the following bash script as a cron job to accomplish this with MySQL 5.0 (kills any query that has been executing for more than 30 seconds). Sharing it here in case it proves useful to anyone (apologies if my bash scripting style is inefficient or atrocious, it is not my primary development language):
#!/bin/bash
linecount=0
processes=$(echo "show processlist" | mysql -uroot -ppassword)
oldIfs=$IFS
IFS='
'
echo "Checking for slow MySQL queries..."
for line in $processes
do
if [ "$linecount" -gt 0 ]
then
pid=$(echo "$line" | cut -f1)
length=$(echo "$line" | cut -f6)
query=$(echo "$line" | cut -f8)
#Id User Host db Command Time State Info
if [ "$length" -gt 30 ]
then
#echo "$pid = $length"
echo "WARNING: Killing query with pid=$pid with total execution time of $length seconds! (query=$query)"
killoutput=$(echo "kill query $pid" | mysql -uroot -ppassword)
echo "Result of killing $pid: $killoutput"
fi
fi
linecount=`expr $linecount + 1`
done
IFS=$oldIfs
I thought it has been around a little longer, but according to this,
MySQL 5.7.4 introduces the ability to set server side execution time limits, specified in milliseconds, for top level read-only SELECT statements.
SELECT
MAX_STATEMENT_TIME = 1000 --in milliseconds
*
FROM table;
Note that this only works for read-only SELECT statements.
Starting with MySQL 5.1 you can create a stored procedure to query the information_schmea.PROCESSLIST table for all queries that match your criteria for "long running" then iterate over a cursor to kill them. Then setup that procedure to execute on a recurring basis in the event scheduler.
See: http://forge.mysql.com/tools/tool.php?id=106
The MySQL forum has some threads about this.
This post details how to set up timeouts on the server using innodb_lock_wait_timeout.
Here's a way to do it programmatically, assuming you're using JDBC.
I think this old question needs an updated answer.
You can set a GLOBAL timeout for all your read-only SELECT queries like this:
SET GLOBAL MAX_EXECUTION_TIME=1000;
The time specified is in milliseconds.
If you want the timeout only for a specific query, you can set it inline like this:
SELECT /*+ MAX_EXECUTION_TIME(1000) */ my_column FROM my_table WHERE ...
MySQL returns an error instead of waiting for eternity.
Note that this method only works for read-only SELECTs. If a SELECT statement is determined not to be read-only, then any timer set for it is cancelled and the following NOTE message is reported to the user:
Note 1908 Select is not a read only statement, disabling timer
For statements with subqueries, it limits the top SELECT only. It does not apply to SELECT statements within stored programs. Using the MAX_EXECUTION_TIME hint in SELECT statements within a stored program will be ignored.
I don't think the egrep above would find "2000".
Why not try just selecting the id as well, and avoiding all of that posh shell stuff:
mysql -e 'select id from information_schema.processlist where info is not null and time > 30;'
Since MySQL 5.7.8 there is max_execution_time option that defines the execution timeout for SELECT statements.
Here is my script :
mysql -e 'show processlist\G' |\
egrep -b5 'Time: [6-9]{3,10}' |\
grep 'Id:' |\
cut -d':' -f2 |\
grep -v '155' |\ ## Binary Log PID
sed 's/^ //' |\
while read id
do
mysql -e "kill $id;"
done
Related
I have a script which will execute Insert Query n times - for that i have used FOR loop , but the problem is the command which connects to remote mysql also executes n times. Here is the script for the better idea for my problem.
#!/bin/bash -X
#fields: id|alias|booking_time|contact_no|deleted|grace|number_in_queue|pax|seated_time|status|walk_in_time|queue_id|user_id
echo "Bash version ${BASH_VERSION}..."
for i in {1..5..1}
do
_alias="Name$i"
_contact_no=$(cat /dev/urandom | tr -dc '1-9' | fold -w 10 | head -n 1)
_deleted="FALSE"
_number_in_queue=$i
_pax=$(( $RANDOM % 10 + 20 ))
_status="waiting"
_queue_id=424
_user_id=550
mysql -u root -p restbucks << EOF #Want this to execute only One time
INSERT INTO queue_item VALUES ('','$_alias',now(),'$_contact_no','$_deleted',NULL,'$_number_in_queue','$_pax',now(),'$_status',now(),'$_queue_id','$_user_id');
EOF
done
Everytime i try to run the script , it will ask me for the password. What i want is that only once the connection made.
You can move the mysql connect before the for loop.
mysql -u root -p restbucks << EOF #this execute only One time
for i in {1..5..1}
do
.....
done
EOF
Also it is recommended that you can write your queries into a file and then finally execute the file using single connection.
You can refer bulk-mysql-query
That is because it is placed inside the for loop. Every time the control passes to the loop it gets executed. You don't need to connect to your server once you are connected. Try placing the statement before the FOR statement.
My DB seems to have almost 100 "sleep" connections and I believe it would be better to get rid of them. I have found the following script which helps me analyze all tables in a DB from the command line. Does anybody know of a similar one liner which can kill all "sleep" connections?
mysql -p -D<database> -B -e "SHOW TABLES" \
| awk '{print "CHECK TABLE "$1";"}' \
| mysql -p -D<database>
You can set the wait_timeout variable to a shorter time than the default and it should kill off the connections once they exceed this time value.
From what I've read there is some bugginess and connections will still randomly hang around, especially under load, but the wait_timeout variable would be the place to start.
Some thoughts:
You may use SHOW PROCESSLIST instead of SHOW TABLES. I'm not much familiar with awk linux utility but I think you can do all sorts of pattern matching there in.
Alternatively, you can write a script to query MySQL for "SHOW PROCESSLIST", parse the resultset for sleeping connections and accordingly use KILL <process id>
for i in `mysql -e "show processlist" | awk '/Sleep/ {print $1}'` ; do mysql -e "KILL $i;"; done
https://www.digitalocean.com/community/questions/how-can-i-kill-all-mysql-sleeping-queries
I' ve searched and searched, but I wasn't able to find an easy way to get this:
Query OK, 50000 rows affected (0.35 sec)
in milliseconds or microseconds.
How can I achieve it?
I came with the same problem, I did my queries from a linux console using time
$ time mysql --user="user" -D "DataBase" -e "SELECT SQL_NO_CACHE COUNT(1) FROM table"
------------
count(1)
------------
750
------------
real 0m0.269s
user 0m0.014s
sys 0m0.015s
or
$ time -f"%e" mysql --user="user" -D "DataBase" -e "SELECT SQL_NO_CACHE COUNT(1) FROM table"
------------
count(1)
------------
750
------------
0.24
It gives different values from "mysql" but at least is something you can work with, for example this script:
#!/bin
temp = 1
while [ $temp -le 1000]
do
/usr/bin/time -f"%e" -o"/home/admin/benchmark.txt" -a mysql --user="user" -D "DataBase" -e "SELECT SQL_NO_CACHE COUNT(1) FROM table" > /dev/null 2> /dev/null
let temp=$temp+1
done
Execute the query 1000 times, -f shows only the real time, -o the output file, -a appends to the output, > /dev/null 2> /dev/null ignores the query output so it doesn't print in console each time.
That time's calculated by the mysql monitor application and isn't done by the mysql server. It's not something you can retrieve programatically by doing (say) select last_query_execution_time() (which would be nice).
You can simulate it in a coarse way by doing the timing in your application, by taking system time before and after calling the query function. Hopefully the client-side overhead would be minimal compared to the mysql portion.
You could time it yourself in the code that runs the query:
Pseudo code:
double StartTime = <now>
Execute SQL Query
double QueryTime = <now> - StartTime
i have a busy web server with LAMP installed, and i was wondering, is there any way to count how many queries per second (mysql) are executed in the server ?
Thank you.
SELECT s1.variable_value / s2.variable_value
FROM information_schema.global_status s1, information_schema.global_status s2
WHERE s1.variable_name='queries'
AND s2.variable_name ='uptime';
Try Jeremy Zawodny's excellent utility mytop.
If you have the Perl module Time::HiRes installed, mytop will automatically use it to generate high-resoution query per second information.
There's useful information to be mined from the SHOW GLOBAL STATUS; command, including the number of queries executed (if your MySQL is 5.0.76 or later).
See http://dev.mysql.com/doc/refman/5.0/en/server-status-variables.html
You can use:
mysqladmin -u root -p status
which will return output like:
Uptime: 17134 Threads: 2 Questions: 1245 Slow queries: 0 Opens: 49 Flush tables: 1 Open tables: 42 Queries per second avg: 0.072
Here queries per second is: 0.072, which is questions/uptime.
When you use the "STATUS" command (not SHOW STATUS), MySQL will calculate the queries per second since server start for you.
Tested with MySQL 5.1.63.
We can have a small script for this. It will be some thing like the below.
declare -i a
declare -i b
declare -i c
a=`mysql -uroot -pxxxxx -e "show status like 'Queries'" |
tail -1 | awk '{print $2}'`
echo "$a"
sleep 1
b=`mysql -uroot -pxxxxx -e "show status like 'Queries'" |
tail -1 | awk '{print $2}'`
echo "$b"
c=$b-$a
echo "Number of Queries per second is: $c"
I'm doing a bash script that interacts with a MySQL datatabase using the mysql command line programme. I want to use table locks in my SQL. Can I do this?
mysql -e "LOCK TABLES mytable"
# do some bash stuff
mysql -u "UNLOCK TABLES"
The reason I ask, is because table locks are only kept for the session, so wouldn't the lock be released as soon as that mysql programme finishes?
[EDIT]
nos had the basic idea -- only run "mysql" once, and the solution nos provided should work, but it left the FIFO on disk.
nos was also correct that I screwed up: a simple "echo X >FIFO" will close the FIFO; I remembered wrongly. And my (removed) comments w.r.t. timing don't apply, sorry.
That said, you don't need a FIFO, you could use an inter-process pipe. And looking through my old MySQL scripts, some worked akin to this, but you cannot let any commands write to stdout (without some "exec" tricks).
#!/bin/bash
(
echo "LOCK TABLES mytable READ ;"
echo "Doing something..." >&2
echo "describe mytable;"
sleep 5
echo "UNLOCK tables;"
) | mysql ${ARGUMENTS}
Another option might be to assign a file descriptor to the FIFO, then have it run in the background. This is very similar to what nos did, but the "exec" option wouldn't require a subshell to run the bash commands; hence would allow you to set "RC" in the "other stuff":
#!/bin/bash
# Use the PID ($$) in the FIFO and remove it on exit:
FIFO="/tmp/mysql-pipe.$$"
mkfifo ${FIFO} || exit $?
RC=0
# Tie FD3 to the FIFO (only for writing), then start MySQL in the u
# background with its input from the FIFO:
exec 3<>${FIFO}
mysql ${ARGUMENTS} <${FIFO} &
MYSQL=$!
trap "rm -f ${FIFO};kill -1 ${MYSQL} 2>&-" 0
# Now lock the table...
echo "LOCK TABLES mytable WRITE;" >&3
# ... do your other stuff here, set RC ...
echo "DESCRIBE mytable;" >&3
sleep 5
RC=3
# ...
echo "UNLOCK TABLES;" >&3
exec 3>&-
# You probably wish to sleep for a bit, or wait on ${MYSQL} before you exit
exit ${RC}
Note that there are a few control issues:
This code has NO ERROR CHECKING for failure to lock (or any SQL commands
within the "other stuff"). And that's definitely non-trivial.
Since in the first example, the "other stuff" is within a subshell, you cannot easily
set the return code of the script from that context.
Here's one way, I'm sure there's an easier way though..
mkfifo /tmp/mysql-pipe
mysql mydb </tmp/mysql-pipe &
(
echo "LOCK TABLES mytable READ ;" 1>&6
echo "Doing something "
echo "UNLOCK tables;" 1>&6
) 6> /tmp/mysql-pipe
A very interesting approach I found out while looking into this issue for my own, is by using MySQL's SYSTEM command. I'm not still sure what exactly are the drawbacks, if any, but it will certainly work for a lot of cases:
Example:
mysql <<END_HEREDOC
LOCK TABLES mytable;
SYSTEM /path/to/script.sh
UNLOCK TABLES;
END_HEREDOC
It's worth noting that this only works on *nix, obviously, as does the SYSTEM command.
Credit goes to Daniel Kadosh: http://dev.mysql.com/doc/refman/5.5/en/lock-tables.html#c10447
Another approach without the mkfifo commands:
cat <(echo "LOCK TABLES mytable;") <(sleep 3600) | mysql &
LOCK_PID=$!
# BASH STUFF
kill $LOCK_PID
I think Amr's answer is the simplest. However I wanted to share this because someone else may also need a slightly different answer.
The sleep 3600 pauses the input for 1 hour. You can find other commands to make it pause here: https://unix.stackexchange.com/questions/42901/how-to-do-nothing-forever-in-an-elegant-way
The lock tables SQL runs immediately, then it will wait for the sleep timer.
Problem and limitation in existing answers
Answers by NVRAM, nos and xer0x
If commands between LOCK TABLES and UNLOCK TABLES are all SQL queries, you should be fine.
In this case, however, why don't we just simply construct a single SQL file and pipe it to the mysql command?
If there are commands other than issuing SQL queries in the critical section, you could be running into trouble.
The echo command that sends the lock statement to the file descriptor doesn't block and wait for mysql to respond.
Subsequent commands are therefore possible to be executed before the lock is actually acquired. Synchronization aren't guaranteed.
Answer by Amr Mostafa
The SYSTEM command is executed on the MySQL server. So the script or command to be executed must be present on the same MySQL server.
You will need terminal access to the machine/VM/container that host the server (or at least a mean to transfer your script to the server host).
SYSTEM command also works on Windows as of MySQL 8.0.19, but running it on a Windows server of course means you will be running a Windows command (e.g. batch file or PowerShell script).
A modified solution
Below is a example solution based on the answers by NVRAM and nos, but waits for lock:
#!/bin/bash
# creates named pipes for attaching to stdin and stdout of mysql
mkfifo /tmp/mysql.stdin.pipe /tmp/mysql.stdout.pipe
# unbuffered option to ensure mysql doesn't buffer the output, so we can read immediately
# batch and skip-column-names options are for ease of parsing the output
mysql --unbuffered --batch --skip-column-names $OTHER_MYSQL_OPTIONS < /tmp/mysql.stdin.pipe > /tmp/mysql.stdout.pipe &
PID_MYSQL=$!
# make sure to stop mysql and remove the pipes before leaving
cleanup_proc_pipe() {
kill $PID_MYSQL
rm -rf /tmp/mysql.stdin.pipe /tmp/mysql.stdout.pipe
}
trap cleanup_proc_pipe EXIT
# open file descriptors for writing and reading
exec 10>/tmp/mysql.stdin.pipe
exec 11</tmp/mysql.stdout.pipe
# update the cleanup procedure to close the file descriptors
cleanup_fd() {
exec 10>&-
exec 11>&-
cleanup_proc_pipe
}
trap cleanup_fd EXIT
# try to obtain lock with 5 seconds of timeout
echo 'SELECT GET_LOCK("my_lock", 5);' >&10
# read stdout of mysql with 6 seconds of timeout
if ! read -t 6 line <&11; then
echo "Timeout reading from mysql"
elif [[ $line == 1 ]]; then
echo "Lock acquired successfully"
echo "Doing some critical stuff..."
echo 'DO RELEASE_LOCK("my_lock");' >&10
else
echo "Timeout waiting for lock"
fi
The above example uses SELECT GET_LOCK() to enter the critical section. It produces output for us to parse the result and decide what to do next.
If you need to execute statements that doesn't produce output (e.g. LOCK TABLES and START TRANSACTION), you may perform a dummy SELECT 1; after such statement and read from the stdout with a reasonable timeout. E.g.:
# ...
echo 'LOCK TABLES my_table WRITE;' >&10
echo 'SELECT 1;' >&10
if ! read -t 10 line <&11; then
echo "Timeout reading from mysql"
elif [[ $line == 1 ]]; then
echo "Table lock acquired"
# ...
else
echo "Unexpected output?!"
fi
You may also want to attach a third named pipe to stderr of mysql to handle different cases of error.