I have a database server running on a AIX box. When pinging the server I see increased ping times at certain times. I am directly connected to the box and usually see 0ms or 1ms ping times. Sometimes I see ping times of 10ms to 12ms. I am wondering a bit how ping is actually implemented in the TCP stack. If the number of other open connections could affect its responsiveness. Anyone have ideas?
We have found that processor preemption is causing problems with ping latency. (Where PHYP steals the whole processor over to itself or to another LPAR.) If you can, try switching to dedicated processors. There are more subtle ways to get good results but I'm the network guy, not the CPU perf guy so I'm sorta out of my league.
In 6.1 TL06 SP05 until about 6.1 TL07 SP05, processor folding was also broken. The work around was to set vpm_cpu_folding (I think thats the name of it) to 4. The same is true for 7.1 releases that came out at the same time.
Related
I have a medium sized database (7 GB) with around 200 concurrent users. i am getting some database lag issues, suddenly my node-mysql client freezes during selects and inserts.
as a process of troubleshooting i checked SHOW STATUS on the DB, everything seemed to be okay but just the Connections attribute has 262050.
want to understand if this number is okay or if the figure is exorbitant?
Exorbitant. Definitely.
Find a machine running your node app, and watch its logs and/or error outputs while you stop and restart the MySQL server.
Hopefully the node app will chatter away telling you that lots of connections were closed as you stop the MySQL server.
This looks like a connection leak. You probably have some sort of problem with your node connection pooling, or maybe with releasing connections after your node app uses then.
I have problem, when MySQL thread sometime stuck at status "Writing to net".
I have 4 Apache server (2.4) (requests are load-balanced on them) a 1 MySQL (MariaDB 10). Apache is executing php56. All Apache servers have same configuration. All servers runs on CentOS 7. SElinux is disabled on Apache servers for debug reasons. No problems in audit logs on DB server. All servers are virtual and located on same cluster (VMware).
Problem appear only on specific pages and specific queries to DB.
Usually there is around 100-200 separate queries on page and most of them takes 0.0001-0.0010 s. But then I have one query that takes around 1-2sec. The query itself take much lesser time (around 0.0045s).
Problematic query returns around 8984 rows and when executed from CLI from debug script, it is executed fast as expected.
Strange is that in time some Apache servers execute that page quickly, and some slowly. It changes (during day). Also I tried remove one Apache server from cluster and then send same request. If server is not under any load, it usually responds fast.
All server have enough resources (CPU and RAM) so it is definitely not load issue. They usually have around 4-10 active Apache workers (prefork) and have capacity for 100 active workers.
I tried debugging with tcpdump and when requesting page, I can see packet flow for fast queries and then it stops for a while and resumes. Not sure if the problem is on MySQL server or on Apache server.
My guess is that I am hitting some kind of limit, but I have no idea which one.
The solution is quite odd.
First few more details:
All Apache severs have same application data (PHP files, images, etc.) Mounted from NFS. The NFS share was working fine (low latency, no data corruption).
Solution:
When I was desperate I went through every possible log. Then I noticed that iptables are dropping some packets from NFS server. Well I said to myself that I should probably fix that, even when its not related.
But after I allowed all traffic from NFS to my Apache servers, MySQL status "writing to net" disappeared and all websites started to respond quickly.
When the system runs out of memory, ubuntu 12.04 kills the mysql process:
Out of memory: Kill process 17074 (mysqld) score 146 or sacrifice child
So the process ends up killed.
This happens at peaks of server load and mainly because of apache getting wild and eating the remaining available memory. Possible approaches could be:
Change somewhere somehow the priority of mysql, so it's not killed (probably a bad fix as something else will be killed)
Monitor the status of mysql and restart automatically whenever it's killed (the one I'm thinking about, but don't know how to do it).
How do you see it?
Abrupt termination of a database server is a very serious crash. You need to avoid this in a production system, because it may not restart cleanly.
The database server is a shared resource, and should almost never terminate in an unplanned fashion in production. The only thing that should cause unplanned termination is a catastrophic hardware or power failure. Most properly configured production data base servers have an unplanned termination once every ten years or less frequently. Seriously.
What to do?
Fix your apache configuration. Limit the number of worker threads and processes it can use, so it can't run wild. Learn how to do this. It's vital. See here: http://httpd.apache.org/docs/current/mod/mpm_common.html#maxrequestworkers
Fix the defects in your web app that are causing your apache to run wild.
If you can, move your mysqld server to a different server machine from apache, so the two don't contend for the same hardware resources.
Configure your mysqld to limit the number of connections it will accept from apache worker threads or other clients. Your web app probably handles the situation where a worker thread needs to wait for a connection. See here. http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_max_connections
Are you on an EC2 micro instance? You need to do some serious tuning. See here: http://ubuntuforums.org/showthread.php?t=1979049
You can check mysql status every minute (with cron) and restart if it is crashed:
* * * * * service mysql status | grep running || service mysql restart
I'm experiencing "mysql server has gone away" in my import script written in php
This http://dev.mysql.com/doc/refman/5.0/en/gone-away.html page has a list of possible reasons. But, how do I debug this, to know which of the reasons?
If some time out fired, I want to know which that his happened, which timeout etc. relevant details. If the query was broken, I want to know that this is the reason.
So, do I have a way to receive the details additionally to just the very global piece of information that the server "has gone away"?
Is there a way to log this stuff: timeouts for example?
I wouldn't think that there's a MySQL magic bullet to find out. It sounds like you need to use operating system and/or network diagnostic tools. Maybe within PHP you can do some things like try pinging the server, checking if the MySQL daemon is running, etc. You might also want to check into using a tool such as Wireshark or tcpdump to see if you're seeing anything funky on the network, such as reset packets or other dropped connection indicators.
If the MySQL server is remote, try doing a constant ping to it and see if you're seeing dropped or delayed packets. Make sure you check with large packets; I've seen systems on which a ping works fine, but routers screw up larger packet sizes due to MTU mismatches and such. On Windows, it is:
ping -t -l 1500 mysqlhost
On most Linux and Unix systems, it is:
ping -s 1500 mysqlhost
I'm working on debugging a slowness issue I've got with running ExpressionEngine (a PHP application) on IIS 7.
I don't think this is actually an issue with ExpressionEngine, but rather an issue with my PHP/MySQL setup.
The problem shows itself thusly:
Go to webite address
IE "spins" for 10-15 seconds, waiting to load. During this time:
processor usage is minimal on the server, and PHP's process is inactive
I see a connection for the site user in MySQL, but the thread is in "sleeping" more.
There is plenty of free memory on the server
pretty much, the server is doing nothing
After 10-15 seconds, I see the connection MySQL run some really quick queries (very fast) and the site loads in under a second.
This is a fairly complex site, but it doesn't make any sense that the whole system is just sitting there waiting for 10 seconds - not processing anything. I'm using FastCGI on IIS7, which seems to be working fine, and to me this seems like some sort of a timeout issue where FastCGI, PHP, or maybe even MySQL is waiting for something, not getting it, and after the timeout occurs, continuing to process.
Anyone had similar experiences?
Thanks!
P.S. - I should also add that the database (MySQL) and PHP are running on the same server.
mysql might be trying to do a reverse DNS on the connection from the web server. if you don't need to filter mysql connections by DNS name, then add skip-name-resolve to your mysql configuration file
Twitter plugins cause lots of issues in CMS systems. usually there is a function that checks if twitter is there/alive. When twitter gets busy, this slows down the system (thats why it can appear as intermittent) Find the twitter plugin, and the routine that checks if twitter is there. Comment out this code and return true (ie. don't ask twitter if it is there, just assume it is)
I have no idea why, but the solution to this was to install PHP 5.3. I had PHP 5.2.10 running, and I guess 5.3 added some extensive optimizations for Windows. Or fixed some other weird problem - who knows.
Actually, after some further digging, it appears the problem was with the Twitter plugin. It waits 25 seconds to come back from Twitter sometime with an error. Maybe this is related to the DNS?