I've noticed that if I execute a long running mysql query with php using mysql_query() (I know I'm not supposed to use that) and then the php process gets killed then the query continues to run on the mysql server. This is not a persistent connection. The connection is made with:
$db = mysql_connect($host, $login, $pass, false);
$sql = 'SELECT COUNT(*) FROM `huge_table`';
$result = mysql_query($sql, $db);
For example, let's say I have a 1 billion row table and a php process does this for some reason:
SELECT COUNT(*) FROM `huge_table`
And then it times out (say because I'm running php-fpm with request_terminate_timeout=5), so it kills the process after 5 seconds to make sure it doesn't hog things.
Eventhough the process is killed, the query still runs on mysql even far after wait_timeout.
Is there anyway to make sure that if the php process exits for whatever reason it also kills any running queries that it made?
I'm using tokudb 5.5.38-tokudb-7.1.7-e which is mysql 5.5.38
crickeys, when a PHP script starts to execute and it gets to the part where it executes a MySQL query, that query is handed over to MySQL. The control of the query is no longer in PHP's hands....PHP at the point is only waiting for a response from MySQL then it can proceed. Killing the PHP script doesn't affect the MySQL query because well, the query is MySQL's business.
Put another way, PHP comes to the door, knocks, hands over the goods and waits for you to bring back a response so he can be on his way. Shooting him won't affect what's going on behind the door.
You could run something like this to retrieve the longest running processes and kill them:
<?php
$con=mysqli_connect("example.com","peter","pass","my_db");
// Check connection
if (mysqli_connect_errno()) {
echo "Failed to connect to MySQL: " . mysqli_connect_error();
}
$result = mysqli_query($con,"SHOW FULL PROCESSLIST");
while($row = mysqli_fetch_array($result)) {
if ($row["Time"] > $max_excution_time ) {
$sql="KILL ".$row["Id"];
mysql_query($sql);
}
}
mysqli_close($con); ?>
Well you can use a destructor to call
mysql_close(); function.
I hope I understood your question...
You can use KILL.
KILL CONNECTION is the same as KILL with no modifier: It terminates the connection associated with the given thread_id.
KILL QUERY terminates the statement that the connection is currently executing, but leaves the connection itself intact.
You should KILL QUERY in the shutdown event, and then do a mysqli_close().
You might get some valuable information from this question about timeouts: Client times out, while MySQL query remains running?
Related
I would like to set a maximum execution time for sql queries like set_time_limit() in php. How can I do ?
I thought it has been around a little longer, but according to this,
MySQL 5.7.4 introduces the ability to set server side execution time limits, specified in milliseconds, for top level read-only SELECT statements.
SELECT
/*+ MAX_EXECUTION_TIME(1000) */ --in milliseconds
*
FROM table;
Note that this only works for read-only SELECT statements.
Update: This variable was added in MySQL 5.7.4 and renamed to max_execution_time in MySQL 5.7.8. (source)
If you're using the mysql native driver (common since php 5.3), and the mysqli extension, you can accomplish this with an asynchronous query:
<?php
// Heres an example query that will take a long time to execute.
$sql = "
select *
from information_schema.tables t1
join information_schema.tables t2
join information_schema.tables t3
join information_schema.tables t4
join information_schema.tables t5
join information_schema.tables t6
join information_schema.tables t7
join information_schema.tables t8
";
$mysqli = mysqli_connect('localhost', 'root', '');
$mysqli->query($sql, MYSQLI_ASYNC | MYSQLI_USE_RESULT);
$links = $errors = $reject = [];
$links[] = $mysqli;
// wait up to 1.5 seconds
$seconds = 1;
$microseconds = 500000;
$timeStart = microtime(true);
if (mysqli_poll($links, $errors, $reject, $seconds, $microseconds) > 0) {
echo "query finished executing. now we start fetching the data rows over the network...\n";
$result = $mysqli->reap_async_query();
if ($result) {
while ($row = $result->fetch_row()) {
// print_r($row);
if (microtime(true) - $timeStart > 1.5) {
// we exceeded our time limit in the middle of fetching our result set.
echo "timed out while fetching results\n";
var_dump($mysqli->close());
break;
}
}
}
} else {
echo "timed out while waiting for query to execute\n";
// kill the thread to stop the query from continuing to execute on
// the server, because we are abandoning it.
var_dump($mysqli->kill($mysqli->thread_id));
var_dump($mysqli->close());
}
The flags I'm giving to mysqli_query accomplish important things. It tells the client driver to enable asynchronous mode, while forces us to use more verbose code, but lets us use a timeout(and also issue concurrent queries if you want!). The other flag tells the client not to buffer the entire result set into memory.
By default, php configures its mysql client libraries to fetch the entire result set of your query into memory before it lets your php code start accessing rows in the result. This can take a long time to transfer a large result. We disable it, otherwise we risk that we might time out while waiting for the buffering to complete.
Note that there's two places where we need to check for exceeding a time limit:
The actual query execution
while fetching the results(data)
You can accomplish similar in the PDO and regular mysql extension. They don't support asynchronous queries, so you can't set a timeout on the query execution time. However, they do support unbuffered result sets, and so you can at least implement a timeout on the fetching of the data.
For many queries, mysql is able to start streaming the results to you almost immediately, and so unbuffered queries alone will allow you to somewhat effectively implement timeouts on certain queries. For example, a
select * from tbl_with_1billion_rows
can start streaming rows right away, but,
select sum(foo) from tbl_with_1billion_rows
needs to process the entire table before it can start returning the first row to you. This latter case is where the timeout on an asynchronous query will save you. It will also save you from plain old deadlocks and other stuff.
ps - I didn't include any timeout logic on the connection itself.
Please rewrite your query like
select /*+ MAX_EXECUTION_TIME(1000) */ * from table
this statement will kill your query after the specified time
You can find the answer on this other S.O. question:
MySQL - can I limit the maximum time allowed for a query to run?
a cron job that runs every second on your database server, connecting and doing something like this:
SHOW PROCESSLIST
Find all connections with a query time larger than your maximum desired time
Run KILL [process id] for each of those processes
pt_kill has an option for such. But it is on-demand, not continually monitoring. It does what #Rafa suggested. However see --sentinel for a hint of how to come close with cron.
echo "\nexec first time:";
$currentTime = microtime(true);
$users->paginate($request->length, ['*'], 'page', $request->start/$request->length + 1);
echo microtime(true) - $currentTime;
echo "\nexec second time:";
$currentTime = microtime(true);
$users->paginate($request->length, ['*'], 'page', $request->start/$request->length + 1);
echo microtime(true) - $currentTime;
The above is the code in my controller for testing the latency when query the mysql. As you can see I execute the same command twice. The execute latency is different.
exec first time: 2.7011959552765
exec second time: 0.78873896598816
The above is the output of the performance. During the Laravel document, the server provider contains the DI pattern to share the DB connection. If we are not recreate the connection then what happened is this result?
If the result is belongs to recreation, then how can I share the connection pool?
On a "cold" server, everything is on disk. A query must pull things into RAM to be performed. This typically means that the first time you run a query it takes X seconds; the second time it takes more like X/10 seconds.
Another issue is with "pagination", especially if it is done via LIMIT and OFFSET. All of the "offset" rows much be stepped over. So, as the user 'pages' through the data, the pages will come up slower and slower -- because of stepping over more and more rows. http://mysql.rjweb.org/doc.php/pagination
I use certain my.cnf settings like this. Does RDS instance allow such options?
slave-skip-errors = 1062,1054
replicate-ignore-db=verv_raw
replicate-ignore-table=verv.ox_session
replicate-wild-ignore-table=verv_raw.ox%
replicate-wild-ignore-table=verv_raw.ox%
I am aware of the procedure that skips one error at a time.
CALL mysql.rds_skip_repl_error;
But what I am looking for is an option to skip all errors on slave. Is it possible in RDS environment?
I solved it by creating a mysql event scheduler like this :
CREATE EVENT repl_error_skipper
ON SCHEDULE
EVERY 15 MINUTE
COMMENT 'Calling rds_skip_repl_error to skip replication error'
Do
CALL mysql.rds_skip_repl_error;
/*also you can add other logic */
To set other global variables you can find and set those (if available for changing) in rds parameter group (you will have to create new parameter group and set the variable values).
As mentioned, this command only skips one replication error. I wrote a PHP script to loop through this and ran it once a minute via cron job (my replica was log jammed with a series of thousands of bad queries than went through on the main DB)
for($i = 1; $i <= 30; $i++) {
$db = new mysqli('someserver.rds.amazonaws.com', 'root', 'password');
$res = $db->query('CALL mysql.rds_skip_repl_error;');
if(!$res) {
//echo 'Query failed: ' . $db->error . "\n";
return;
}
//var_dump($res->fetch_assoc());
$db->close();
sleep(1);
}
You'll need to tinker with this some (not every server would tolerate only one second between calls and 30 calls per minute), but it does work (albeit in a brute force manner). You must create a new DB connection every time to run this. The loop opens, runs and then closes the connection.
so I am just getting started using Memcache. I am ready to write some code, however I have an optimization question:
I wonder whether I should delay establishing a MySQL Connect as far as possible (and maybe not establish one at all, when everything can be read from the Memcache) OR establish it anyway to spare me coding time, based on the thought that not the connection but the actual querys make my server's CPU go crazy.
So, I have to choose between these two code examples:
1 - Connect to MySQL anyway
$memcache = new Memcache;
$memcache->connect('localhost', 11211) or die ("MEMCACHE: Could not connect!");
$db = mysql_connect('localhost', 'user', 'password') or die ("MySQL: Could not connect!");
mysql_select_db('database');
$sql = "SELECT id FROM table LIMIT 1";
$key = md5('query'.$sql);
//lookup value in memcache
$result = $memcache->get($key);
//check if we got something back
if($result == null) {
//fetch from database
$qry = mysql_query($sql) or die(mysql_error()." : $sql");
if(mysql_num_rows($qry)> 0) {
$result = mysql_fetch_object($qry);
//store in memcache for 60 seconds
$memcache->set($key,$result,0,60);
}
}
2 - Connect to MySQL as soon as it is needed
$memcache = new Memcache;
$memcache->connect('localhost', 11211) or die ("MEMCACHE: Could not connect!");
$sql = "SELECT id FROM table LIMIT 1";
$key = md5('query'.$sql);
//lookup value in memcache
$result = $memcache->get($key);
//check if we got something back
if($result == null) {
if(!$db){
$db = mysql_connect('localhost', 'user', 'password') or die ("MySQL: Could not connect!");
mysql_select_db('database');
}
//fetch from database
$qry = mysql_query($sql) or die(mysql_error()." : $sql");
if(mysql_num_rows($qry)> 0) {
$result = mysql_fetch_object($qry);
//store in memcache for 60 seconds
$memcache->set($key,$result,0,60);
}
}
The way to go is to connect to mySQL(and other stuff) only when you need it. that way you reduce the resources your app needs in this case network connections. and you don't put a load to the DB server.
General rule of thumb : use a resource only when you need it.
If speed is important, get the connection up front. It's not a big resource to have open, but it can take a while to establish.
Also, by connecting early you know on app startup if there's a problem (eg database server down) and you get confirmation that everything is good to go, rather than exuding some time later when you could have known earlier and fixed the problem before it was becme problem.
You might want to go further and run a heartbeat query to assert that the database is still there so similar reasons.
Note that this approach makes the database required to be up for your app to be up. You can do something in between: get the connection at startup, but if it`s not available fall back to a just in time approach, which gives you more flexibility. This is what I'd do.
I think it depends on concurrency. But cache the connections in a thread safe pool is better.
In many web applications, database connections are established and put in a thread safe pool, because establishing a connection is expensive.
It seems like getting data from memcached not directly from database, because it more fast and can hold so many threads.
Looking at the code you provided (usual last-century-style spaghetti) I'd vote for the first one.
Whatever logic added to the program flow will make your code more complex by a factor of ten. So, better leave it as plain as possible.
Or, even, I would advise not to use memcache at all, until you learn how to separate and encapsulate different matters.
Especially because there is no point in caching data you can get from db by the primary key.
I would like to set a maximum execution time for sql queries like set_time_limit() in php. How can I do ?
I thought it has been around a little longer, but according to this,
MySQL 5.7.4 introduces the ability to set server side execution time limits, specified in milliseconds, for top level read-only SELECT statements.
SELECT
/*+ MAX_EXECUTION_TIME(1000) */ --in milliseconds
*
FROM table;
Note that this only works for read-only SELECT statements.
Update: This variable was added in MySQL 5.7.4 and renamed to max_execution_time in MySQL 5.7.8. (source)
If you're using the mysql native driver (common since php 5.3), and the mysqli extension, you can accomplish this with an asynchronous query:
<?php
// Heres an example query that will take a long time to execute.
$sql = "
select *
from information_schema.tables t1
join information_schema.tables t2
join information_schema.tables t3
join information_schema.tables t4
join information_schema.tables t5
join information_schema.tables t6
join information_schema.tables t7
join information_schema.tables t8
";
$mysqli = mysqli_connect('localhost', 'root', '');
$mysqli->query($sql, MYSQLI_ASYNC | MYSQLI_USE_RESULT);
$links = $errors = $reject = [];
$links[] = $mysqli;
// wait up to 1.5 seconds
$seconds = 1;
$microseconds = 500000;
$timeStart = microtime(true);
if (mysqli_poll($links, $errors, $reject, $seconds, $microseconds) > 0) {
echo "query finished executing. now we start fetching the data rows over the network...\n";
$result = $mysqli->reap_async_query();
if ($result) {
while ($row = $result->fetch_row()) {
// print_r($row);
if (microtime(true) - $timeStart > 1.5) {
// we exceeded our time limit in the middle of fetching our result set.
echo "timed out while fetching results\n";
var_dump($mysqli->close());
break;
}
}
}
} else {
echo "timed out while waiting for query to execute\n";
// kill the thread to stop the query from continuing to execute on
// the server, because we are abandoning it.
var_dump($mysqli->kill($mysqli->thread_id));
var_dump($mysqli->close());
}
The flags I'm giving to mysqli_query accomplish important things. It tells the client driver to enable asynchronous mode, while forces us to use more verbose code, but lets us use a timeout(and also issue concurrent queries if you want!). The other flag tells the client not to buffer the entire result set into memory.
By default, php configures its mysql client libraries to fetch the entire result set of your query into memory before it lets your php code start accessing rows in the result. This can take a long time to transfer a large result. We disable it, otherwise we risk that we might time out while waiting for the buffering to complete.
Note that there's two places where we need to check for exceeding a time limit:
The actual query execution
while fetching the results(data)
You can accomplish similar in the PDO and regular mysql extension. They don't support asynchronous queries, so you can't set a timeout on the query execution time. However, they do support unbuffered result sets, and so you can at least implement a timeout on the fetching of the data.
For many queries, mysql is able to start streaming the results to you almost immediately, and so unbuffered queries alone will allow you to somewhat effectively implement timeouts on certain queries. For example, a
select * from tbl_with_1billion_rows
can start streaming rows right away, but,
select sum(foo) from tbl_with_1billion_rows
needs to process the entire table before it can start returning the first row to you. This latter case is where the timeout on an asynchronous query will save you. It will also save you from plain old deadlocks and other stuff.
ps - I didn't include any timeout logic on the connection itself.
Please rewrite your query like
select /*+ MAX_EXECUTION_TIME(1000) */ * from table
this statement will kill your query after the specified time
You can find the answer on this other S.O. question:
MySQL - can I limit the maximum time allowed for a query to run?
a cron job that runs every second on your database server, connecting and doing something like this:
SHOW PROCESSLIST
Find all connections with a query time larger than your maximum desired time
Run KILL [process id] for each of those processes
pt_kill has an option for such. But it is on-demand, not continually monitoring. It does what #Rafa suggested. However see --sentinel for a hint of how to come close with cron.