I've wrote a script to batch process domains and retrieve data on each one. For each domain retrieved, it connects to a remote page via curl and retrieves the data required for 30 domains at a time.
This page typical takes between 2 - 3 mins to load and return the curl result, at this point, the details are parsed and placed into an array (page rank tools function).
Upon running this script via CRON, I keep getting the error 'MySQL server has gone away'.
Can anyone tell me if I'm missing something obvious that could be causing this?
// script dies after 4 mins in time for next cron to start
set_time_limit(240);
include('../include_prehead.php');
$sql = "SELECT id, url FROM domains WHERE (provider_id = 9 OR provider_id = 10) AND google_page_rank IS NULL LIMIT 30";
$result = mysql_query($sql);
$row = mysql_fetch_assoc($result);
do {
$url_list[$row['id']] = $row['url'];
} while ($row = mysql_fetch_assoc($result));
// curl domain information page - typically takes about 3 minutes
$pr = page_rank_tools($url_list);
foreach ($pr AS $p) {
// each domain
if (isset($p['google_page_rank']) && isset($p['alexa_rank']) && isset($p['links_in_yahoo']) && isset($p['links_in_google'])) {
$sql = "UPDATE domains SET google_page_rank = '".$p['google_page_rank']."' , alexa_rank = '".$p['alexa_rank']."' , links_in_yahoo = '".$p['links_in_yahoo']."' , links_in_google = '".$p['links_in_google']."' WHERE id = '".$p['id']."'";
mysql_query($sql) or die(mysql_error());
}
}
Thanks
CJ
This happens because MySQL connection has its own timeout and while you are parsing your pages, well, it ends. You can try to increase this timeout with
ini_set('mysql.connect_timeout', 300);
ini_set('default_socket_timeout', 300);
(as mentioned in MySQL server has gone away - in exactly 60 seconds)
Or just call mysql_connect() again.
Because the curl take too long time, you can consider to connect again your database before entering the LOOP for update
There are many reasons why this error occurs. See a list here, it may be something you can fix quite easily
MySQL Server has gone away
Related
So im doing a project for which i have made an android application which communicates with a mysql database on phpMyAdmin. The android app sends requests to a php file which in turn communicates with the database. these php files are on the same server (000webhost) I have a bit of a problem though. Sometimes the connection doesnt work (eg. the data doesnt come through or i get a time-out error or something)
I have looked at the data traffic on phpMyAdmin and I think it is abnormally high (im not sure). The app is used by about 10 persons and the total queries should be about a 1000 a day. The queries are simple select and update actions. Though when I look at the data traffic (status) on phpMyadmin it says there has been 2.2 TiB data traffic total. Which I think is ridiculous. Also phpMyAdmin gives a lot of errors like "the rate of reading the first index entry is high" "and the rate of opening tables is high". And when I look at the monitor on phpMyAdmin it has random peaks of sudden 150 MiB data sent. Like, everything is incredibly high.
this is an example of a query I send to the database via php:
<?php
$con = mysqli_connect("*************");
$ID = $_POST["ID"];
$naam = $_POST["NAME"];
$inbier = $_POST["INBIER"];
$outbier = $_POST["OUTBIER"];
$pof = $_POST["POF"];
$adjust = $_POST["ADJUST"];
$statement = mysqli_prepare($con, "SELECT * FROM Bierlijst WHERE NAME = ?");
mysqli_stmt_bind_param($statement, "s", $naam);
mysqli_stmt_execute($statement);
mysqli_stmt_store_result($statement);
mysqli_stmt_bind_result($statement, $ID, $naam, $inbier, $outbier, $pof, $adjust);
$response = array();
$response["success"] = false;
while(mysqli_stmt_fetch($statement)){
$response["success"] = true;
$response["NAME"] = $naam;
$response["INBIER"] = $inbier;
$response["OUTBIER"] = $outbier;
$response["POF"] = $pof;
$response["ADJUST"] = $adjust;
}
echo json_encode($response);
?>
Is it possible that I'm running bad queries which are failing and looping and stuff?
It annoys me that the following query when fired up by an AJAX request takes 1 second to process where as when called during page refresh(synchronous) takes merely 2 ms. I have spent hours tracking down what goes wrong but I am helpless. I have tried Model->read, Model->find, Model->query() yet it takes the same amount of time. I think 1 second for a simple query like this is not natural. May be the CakePHP models wasting too much resources and time. But my instincts say it's related to query cache.
protected function _user_info($id= NULL){
//benchmarking
$time = -microtime(true);
if(!$id){
if($this->Auth->loggedIn())
$id = $this->Auth->user('id');
else
return NULL;
}
$this->loadModel('User');
/*$findOptions = array('conditions'=>array('User.id'=>$id),
'fields'=>'User.id, User.name, User.email, User.role, dp',
'limit'=>1,
'recursive'=>-1);
$r = $this->User->find('first', $findOptions);
*/
$r = $this->User->query("SELECT * FROM users WHERE id = '".$id."' LIMIT 1");
$time += microtime(true);
echo '<h1>'.$time.'</h1>'; //out- time taken for the query
return $r['User'];
}
Any kind of help would be awesome!
First, try normal Cake search style:
// You should have containable
$this->User->contain();
$r = $this->User->find('first',array('conditions'=>array('id'=>$id)));
Test it.
Cheers.
If you're on debug 2 you're not counting execution times but debugging overhead also.
With debug enabled cache won't be used for long which will mean the DB will be asked to DESCRIBE the table, an sql log will be created, expensive object reflection might be requested multiple times especially if you hit warnings, exceptions or non fatal errors and all this will take considerably longer.
I have designed a coding platform just like Spoj and Codeforces for competitions to be organised in my college on LAN.
I have used long polling there so that any announcements from the Admin can be broadcasted to all users with a JavaScript alert message. When anything is posted on the forum then the admin also gets a notification.
But for just 16 users (including the 1 Admin) accessing the site, the server went down showing too many sql connections. I restarted my laptop (server) and it continued for a while, then again went down; giving the same error message as before.
When I removed both long-poll processes everything continued smoothly.
Server-side code for long-poll:
include 'dbconnect.php';
$old_ann_id = $_GET['old_ann_id'];
$resultann = mysqli_query($con,"SELECT cmntid FROM announcements ORDER BY cmntid DESC LIMIT 1");
while($rowann = mysqli_fetch_array($resultann)){
$last_ann_id = $rowann['cmntid'];
}
while($last_ann_id <= $old_ann_id){
usleep(10000000);
clearstatcache();
$resultann = mysqli_query($con,"SELECT cmntid FROM announcements ORDER BY cmntid DESC LIMIT 1");
while($rowann = mysqli_fetch_array($resultann)){
$last_ann_id = $rowann['cmntid'];
}
}
$response = array();
$response['msg'] = 'new';
$response['old_ann_id'] = $last_ann_id;
$resultann = mysqli_query($con, "Select announcements from announcements where cmntid = $last_ann_id");
while($rowann = mysqli_fetch_array($resultann)){
$response['announcement'] = $rowann['announcements'];
}
echo json_encode($response);
Max connections is defined. Think the default is 100 or 151 connections depending on the version of MySQL. You can see the value in "Server variables and settings" in phpmyadmin (or directly by executing *show variables like "max_connections";* ).
If that is set to something very low (say 10) and you have (say) 15 users you will hit the limit rapidly. You are giving each long polling script its own connection, and that connection is probably sitting open until that long polling script ends. You could likely reduce this by having the script disconnect after each time it checks the database, then reconnect the next time it checks (ie, if your long polling script checks the db every 5 seconds you probably have well over 4.5 seconds of that 5 seconds currently where there is a connection to the db but where the connection is not being used)
However you could have a larger number of connections, but if you trigger the ajax polling multiple times per user, each could have several simultaneous connections. This is probably quite easy to do with a minor bug in your javascript.
Possibly worse if you are using a persistent connections you might leave connections open after the user has left the page that calls the long polling script.
EDIT - update based on your script.
Note I am not sure exactly what your dbconnect.php include is doing. I might be possible to easily call a connect / disconnect function in that include, but I have just put it in this example code as using the mysqlu_close and mysqli_connect functions.
<?php
include 'dbconnect.php';
$old_ann_id = $_GET['old_ann_id'];
$resultann = mysqli_query($con,"SELECT MAX(cmntid) FROM announcements");
if($rowann = mysqli_fetch_array($resultann))
{
$last_ann_id = $rowann['cmntid'];
}
$timeout = 0;
while($last_ann_id <=$old_ann_id and $timeout < 6)
{
$timeout++;
mysqli_close($con);
usleep(10000000);
clearstatcache();
$con = mysqli_connect("myhost","myuser","mypassw","mybd");
$resultann = mysqli_query($con,"SELECT MAX(cmntid) FROM announcements");
if($rowann = mysqli_fetch_array($resultann))
{
$last_ann_id = $rowann['cmntid'];
}
}
if ($last_ann_id >$old_ann_id)
{
$response = array();
$response['msg'] = 'new';
$response['old_ann_id'] = $last_ann_id;
$resultann=mysqli_query($con,"SELECT cmntid, announcements FROM announcements WHERE cmntid>$old_ann_id ORDER BY cmntid");
while($rowann = mysqli_fetch_array($resultann))
{
$response['announcement'][]=$rowann['announcements'];
$response['old_ann_id'] = $rowann['cmntid'];
}
mysqli_close($con);
echo json_encode($response);
}
else
{
echo "No annoucements - resubmit";
}
?>
I have added a count to the main loop. But it will drop out of the loop whether anything is found once it has executed 6 times. This way even if someone leaves the page the script will only be running for a short time afterwards (max a minute). You will have to amend you javascript to catch this and resubmit the ajax call.
Also I have changed the announcement in the response to be an array. This way if there are several announcements while the script is running all will be brought back.
I need to try to understand how MySQL processes/connections work. I have googled and dont see anything in laymans terms so I'm asking here. Here is the situation.
Our host is giving us grief over "too many MySQL processes". We are on a shared server. We are allowed .2 of the server mySQL processes - which they claim is 50 connections - and they say we are using .56.
From the technical support representative:
"Number of MySQL procs (average) - 0.59 meant that you were using
0.59% of the total MySQL connections available on the shared server. The acceptable value is 0.20 which is 50 connections. "
Here is what we are running:
Zen Cart: 1.5.1 35K products. Auto updating of 1-20
products every 10 hours via cron.
PHP version 5.3.16
MySQL version 5.1.62-cll
Architecture i686
Operating system linux
We generally have about 5000 hits per day on the site and Google bot loves to visit even though I have the crawl rate set to minimum in Google webmaster tools.
I'm hoping someone can explain MySQL processes to me in terms of what this host is talking about. Every time I ask them I get an obfuscated answer that is vague and unclear. Is a new MySQL process created every time a visitor visits the site? That does not seem right.
According to the tech we were using 150 connections at that particular time.
EDIT:
here is the connection function in zencart
function connect($zf_host, $zf_user, $zf_password, $zf_database, $zf_pconnect = 'false', $zp_real = false) {
$this->database = $zf_database;
$this->user = $zf_user;
$this->host = $zf_host;
$this->password = $zf_password;
$this->pConnect = $zf_pconnect;
$this->real = $zp_real;
if (!function_exists('mysql_connect')) die ('Call to undefined function: mysql_connect(). Please install the MySQL Connector for PHP');
$connectionRetry = 10;
while (!isset($this->link) || ($this->link == FALSE && $connectionRetry !=0) )
{
$this->link = #mysql_connect($zf_host, $zf_user, $zf_password, true);
$connectionRetry--;
}
if ($this->link) {
if (#mysql_select_db($zf_database, $this->link)) {
if (defined('DB_CHARSET') && version_compare(#mysql_get_server_info(), '4.1.0', '>=')) {
#mysql_query("SET NAMES '" . DB_CHARSET . "'", $this->link);
if (function_exists('mysql_set_charset')) {
#mysql_set_charset(DB_CHARSET, $this->link);
} else {
#mysql_query("SET CHARACTER SET '" . DB_CHARSET . "'", $this->link);
}
}
$this->db_connected = true;
if (getenv('TZ') && !defined('DISABLE_MYSQL_TZ_SET')) #mysql_query("SET time_zone = '" . substr_replace(date("O"),":",-2,0) . "'", $this->link);
return true;
} else {
$this->set_error(mysql_errno(),mysql_error(), $zp_real);
return false;
}
} else {
$this->set_error(mysql_errno(),mysql_error(), $zp_real);
return false;
}
I wonder if it is a problem with connection pooling. Try changing this line:
$this->link = #mysql_connect($zf_host, $zf_user, $zf_password, true);
to this:
$this->link = #mysql_connect($zf_host, $zf_user, $zf_password);
The manual is useful here - the forth parameter is false by default, but your code is forcing it to be true, which creates a new connection even if an existing one is already open (this is called connection pooling and saves creating new connections unnecessarily i.e. saves both time and memory).
I would offer a caveat though: modifying core code in a third-party system always needs to be done carefully. There may be a reason for the behaviour they've chosen, though there's not much in the way of comments to be able to tell. It may be worth asking a question via their support channels to see why it works this way, and whether they might consider changing it.
We have an Cron-Script, which detects - if some users got kicked out of our application.
We can detect this, if a specific value is 1 - but in the the stream, no new entries get set.
Scripts run every hour. Mostly non are detected. But since 2012-10-31 23:59:03 every user got detected. If i run the script on my local maschine or even on the same machine as the cron runs. Everything got handled as it should.
First things first, our script:
require_once ('cron_init.php');
ini_set('date.timezone', 'Europe/Berlin');
ini_set('max_execution_time', 30);
ini_set('memory_limit', -1);
error_reporting(E_ALL);
ini_set("display_errors", 1);
Zend_Date::setOptions(array('fix_dst' => true));
$userinfos = new Repricing_Dbservices_Userinfos();
$users = $userinfos->getUsersForRepricing();
$repricingstream = new Repricing_Dbservices_Repricingstream();
$error = new Repricing_Dbservices_Error();
if($users!==false AND count($users)>0){
$counter = 0;
$errCounter = 0;
$jetzt = new Zend_Date();
$jetzt->setTimezone('Europe/Berlin');
$jetzt = $jetzt->get(Zend_Date::TIMESTAMP);
foreach($users as $user){
$stream = $repricingstream->getStreamLimit($user);
$last = new Zend_Date($stream);
$last->setTimezone('Europe/Berlin');
$last = $last->get(Zend_Date::TIMESTAMP);
$diff = (($jetzt-$last)/60);
$error->setError(1, 'DIED', $diff, $user);
if($diff > 50 ){
$errCounter++;
$userinfos->setUserFree($user);
$error->setError(1, 'DIED', 'ANSTOSSEN', $user);
}
$counter++;
}
$error->setError(1, $errCounter, 'ANSTOSSEN_ALL', 'ALL');
}
Usually $diff >= 0 AND $diff <= 4 but, we detected, that $diff is always round about 381595. If we run it out of cron $diff is, as it should.
We also detected, that $jetzt is now ( as it should ) only $last is much more later. 381595 later. But that shouldnt be. The last stream-date is fully normal. We cant understand this behaviour of. Zend_Date with cron. Bevor 2012-10-21 23:59:03 the script run 2 weeks as it should. We cant explain, how come. Can you?
Consider this:
$right = new Zend_Date('2012-11-01 12:12:12', Zend_Date::ISO_8601);
var_dump( $right->getIso() ); // 2012-11-01T12:12:12+00:00
var_dump( $right->getTimestamp() ); // 1351771932
$wrong = new Zend_Date('2012-11-01 12:12:12', null, 'en_US');
var_dump( $wrong->getIso() ); // 2012-01-11T12:12:12+00:00
var_dump( $wrong->getTimestamp() ); // 1326283932
Now the real freaky part: on my PC it's the second behavior that is default - i.e., when no additional params are given to Zend_Date constructor.
The point is, Zend_Date is a bit... too helpful when trying to parse datetime strings. For example, it's taking the locale into account - but the locale both of server and client! And if the string cannot be parsed within this locale's rules, it silently gives up - and tries to use another rule.
That's why 2012-10-29 was parsed as October, 29 (despite of what locale suggested, as there's no 29th month) - but 2012-11-01 became January, 11 - and messed up your script big time. )