I have a web site in closed beta, developed in Django, runs with Mysql on Debian.
In the last few days, the main page has been showing a slowdown. For every ten clicks, one or two receives extremely slow response (10 secs or more), others are as fast as they used to be.
When I was searching for the problem, I ran into this issue that I couldn't grasp:
top command shows that when I request the main page, mysql shoots up to 90% - 100% cpu usage. I get the page just as the cpu use gets back to normal. So, I thought, it is db.
Then I called ab with parameters -n 1000 -c 5, I got decent performance, about 100 pages per second, just as it was before the slowdown. I would imagine a worse performance as 10-20% of requests take 10 secs to load.
Is this conflict between ab and "real" clicks normal, or am I using ab in a wrong configuration?
ab doesn't execute many parts of the page (javascript i.e.) so you'll notice probably a sensible difference in pressure to the webserver.
Related
I have a WordPress website that has a decent amount of traffic, both guests and logged-in users. (Logged in users not served cache pages). I use WP Super Cache.
The site is very dynamic (main purpose is file downloading, rating, sharing, Buddypress section, etc.) It's quite large and active at all times.
I've taken an initiative on optimizing the basics of the site as best I can (much of it is run by a custom-written plugin done by myself) but I use a few third-party plugins as well (the aforementioned Buddypress, for example)
Anyway.
At any given time the site runs fine. But at "high traffic times" (weekends, etc) the site is constantly at >100% CPU usage and is very, very slow.
To add to the confusion, at similar "high traffic times" the site load is not necessarily high at all. It seems to come and go, but is typically associated with high volume of traffic.
Point is, it's quite sporadic. And can last for a few minutes to hours to maybe the entire day is just "high" CPU usage.
Before I throw in the towel and just ask my host to up my CPU power, I'd like to try to narrow it down. How can I do this? I'm not too adept with server maintenance, but hypothetically for example, maybe I can tell phpmyadmin to "log all queries for the next 30 minutes and rank them by number of times they've run and just how much CPU impact they've caused" etc. Whether or not that's a "thing" idk but that's what I'm asking here - what can I try to attempt to figure out the source of these slowdowns?
(I have tried disabling plugins here and there, but it's hard to do with how dynamic and mashed together the plugins are with each other. Disabling one thing kinda interferes with something else or worse.)
(I've also used plugins like Query Monitor and gone around the site optimizing and fixing any "slow queries" I've found, well, as much as I can. It's not the same, though. This issue happens on a LIVE site, with large traffic. Me on a test installation or as a single user is not the same environment. Besides, I may just not happen across the queries that are causing the issue.)
Running of "htop" just in case useful info: https://i.imgur.com/8eqDw4P.png
Rate Per Second = RPS
Suggestions to consider for your my.cnf [mysqld] section
innodb_io_capacity=600 # from 200 to enable higher IOPS
read_rnd_buffer_size=128K # from 256K to reduce handler_read_rnd_next RPS of 226691
innodb_buffer_pool_instances=4 # from 1 to reduce mutex contention
innodb_buffer_pool_size=4G # from 128M to reduce innodb_buffer_pool_reads RPS of 297
innodb_log_file_size=64M # from 5M for ~ 1 hr before log rotation
These configuration changes should expedite query completion. There are several more opportunities to improve performance.
Observation,
Slow_queries average 165 RPhr and can usually be corrected with appropriate indexes.
Your log is already active and would be used to summarize with mysqldumpslow -s t to identify most used to be corrected first.
For additional assistance, get in touch, please.
We are finding that the Regular Expression Cache in our JRuby application is out of control - it just keeps growing and growing until the app is grinding to a halt.
It eventually does garbage collect, but transaction time is becomes far too high (90 secs instead of 1-2 secs) long before that.
Is there a way to either stop this Regexp Cache from growing so much or limit the size of the cache?
first of all, since you already mentioned looking at the source at Very large retained heap size for org.jruby.RubyRegexp$RegexpCache in JRuby Rails App you probably realised there's no such support implemented.
would say you have 2-3 options to decide :
implement support for limiting or completely disabling the cache within JRuby's RubyRegexp
introduce a "hack" that will check available memory and clear out some of the cache RubyRegexp caches e.g. from another thread (at least until a PR is accepted into JRuby)
look into tuning or using a different GC (including some JVM options) so that the app performs more predictably ... this is application dependent and can no be answered (in general) without knowing the specifics
one hint related to how the JVM keeps soft references -XX:SoftRefLRUPolicyMSPerMB=250 it's 1000 (1 seconds) by default thus decreasing it means they will live shorter ... but it might just all relate to when they're collected (depends on GC and Java version I guess) so in the end you might find out to be fixing the symptom and not the real cause (as noted things such these can not be generalized esp. knowing very little about the app and/or JVM OPTS used)
I have a webpage in my LAN in order to input barcodes in real time to a db through a field (framework django + postgresql + nginx). It works fine, but lately we have a customer that uses 72 char barcodes (Code Matrix) that slows down inputs because before next scan, user must wait the redraw of the last one in the field (it takes about 1-2 seconds, redrawing one char after the other).
Is there a way to reduce latency of drawing scanned text in the html field?
Best thing would be to show directly all the scanned barcode, not one char after the other. The scanner is set to add an "Enter" after the scanned text.
At the end, as Brad stated, the problem is more related to scanner's settings (USB in HID mode), although PC speed is also an issue. After several tests, on a dual core linux machine I estimate delay due 85% to the scanner and 15% to PC/browser combo.
To solve the problem I first searched and downloaded the complete manual of our 2D barcode scanner (306 pages), then I focused on USB Keystroke Delay as a cause, but default setting was already set to 'No Delay'.
The setting that affected reading speed was USB Polling Interval, an option that applies only to the USB HID Keyboard Emulation Device.
The polling interval determines the rate at which data can be sent between the scanner and the host computer. A lower number indicates a faster data rate: default was 8ms, wich I lowered to 3ms without problems. Lower rates weren't any faster, probably because it was reached the treshold where PC becomes the bottleneck.
CAUTION: Ensure your host machine can handle the selected data rate, selecting a data rate that is too fast for your host machine may result in lost data: in my case when I lowered polling interval to 1ms there were no data loss within the working PC, but when testing inside a virtual machine there was data loss as soon as I reached 6ms.
Another interesting thing is that browsers tend to respond significantly slower after a long period of use with many tabs open (a couple of hours in my case), probably due to caching.
Tests done with Firefox and Chromium browsers on old dual core PC with OS Lubuntu (linux).
This probably has nothing to do with your page, but with the speed of the scanner interface. Most of those scanners intentionally rate-limit their input so as not to fill the computer's buffer, avoiding characters getting dropped. Think about this... when you copy/paste text, it doesn't take a long time to redraw characters. Everything appears instantly.
Most of those scanners are configurable. Check to see if there is an option on your scanner to increase its character rate.
On Honeywell and many other brand scanners the USB Keystroke Interval is marked as an INTERCHARACHTER DELAY.
Also if there is a BAUD rate that would be something to increase.
i recently have migrated between 2 servers (the newest has lower specs), and it freezes all the time even though there is no load on the server, below are my specs:
HP DL120G5 / Intel Quad-Core Xeon X3210 / 8GB RAM
free -m output:
total used free shared buffers cached
Mem: 7863 7603 260 0 176 5736
-/+ buffers/cache: 1690 6173
Swap: 4094 412 3681
as you can see there is 412 mb ysed in swap while there is almost 80% of the physical ram available
I don't know if this should cause any trouble, but almost no swap was used in my old server so I'm thinking this does not seem right.
i have cPanel license so i contacted their support and they noted that i have high iowait, and yes when i ran sar i noticed sometimes it exceeds 60%, most often it's 20% but sometimes it reaches to 60% or even 70%
i don't really know how to diagnose that, i was suspecting my drive is slow and this might cause the latency so i ran a test using dd and the speed was 250 mb/s so i think the transfer speed is ok plus the hardware is supposed to be brand new.
the high load usually happens when i use gzip or tar to extract files (backup or restore a cpanel account).
one important thing to mention is that top is reporting that mysql is using 100% to 125% of the CPU and sometimes it reaches much more, if i trace the mysql process i keep getting this error continually:
setsockopt(376, SOL_IP, IP_TOS, [8], 4) = -1 EOPNOTSUPP (Operation not supported)
i don't know what that means nor did i get useful information googling it.
i forgot to mention that it's a web hosting server for what it's worth, so it has the standard setup for web hosting (apache,php,mysql .. etc)
so how do i properly diagnose this issue and find the solution, or what might be the possible causes?
As you may have realized by now, the free -m output shows 7603MiB (~7.6GiB) USED, not free.
You're out of memory and it has started swapping which will drastically slow things down. Since most applications are unaware that the virtual memory is now coming from much slower disk, the system may very well appear to "hang" with no feedback describing the problem.
From your description, the first process I'd kill in order to regain control would be the Mysql. If you have ssh/rsh/telnet connectivity to this box from another machine, you may have to login from that in order to get a usable commandline to kill from.
My first thought (hypothesis?) for what's happening is...
MySQL is trying to do something that is not supported as this machine is currently configured. It could be missing a library or an environment variable is not set or any number things.
That operation allocates some memory but is failing and not cleaning up the allocation when it does. If this were a shell script, it could be fixed by putting an event trap command at the beginning that runs a function that releases memory and cleans up.
The code is written to keep retrying on failure so it rapidly uses up all your memory. Refering back to the shell script illustration, the trap function might also prompt to see if you really want to keep retrying.
Not a complete answer but hopefully will help.
How to measure current load of MySQL server? I know I can measure different things like CPU usage, RAM usage, disk IO etc but is there a generic load measure for example the server is at 40% load etc?
mysql> SHOW GLOBAL STATUS;
Found here.
The notion of "40% load" is not really well-defined. Your particular application may react differently to constraints on different resources. Applications will typically be bound by one of three factors: available (physical) memory, available CPU time, and disk IO.
On Linux (or possibly other *NIX) systems, you can get a snapshot of these with vmstat, or iostat (which provides more detail on disk IO).
However, to connect these to "40% load", you need to understand your database's performance characteristics under typical load. The best way to do this is to test with typical queries under varying amounts of load, until you observe response times increasing dramatically (this will mean you've hit a bottleneck in memory, CPU, or disk). This load should be considered your critical level, which you do not want to exceed.
is there a generic load measure for example the server is at 40% load ?
Yes! there is:
SELECT LOAD_FILE("/proc/loadavg")
Works on a linux machine. It displays the system load averages for the past 1, 5, and 15 minutes.
System load averages is the average number of processes that are either in a runnable or uninterruptable state. A process in a runnable state is either using the CPU or waiting to use the CPU.
A process in uninterruptable state is waiting for some I/O access, eg waiting for disk. The averages are taken over the three time intervals. Load averages are not normalized for the number of
CPUs in a system, so a load average of 1 means a single CPU system is loaded all the time while on a 4 CPU system it means it was idle 75% of the time.
So if you want to normalize you need to count de number of cpu's also.
you can do that too with
SELECT LOAD_FILE("/proc/cpuinfo")
see also 'man proc'
with top or htop you can follow the usage in Linux realtime
On linux based systems the standard check is usually uptime, a load index is returned according to metrics described here.
aside from all the good answers on this page (SHOW GLOBAL STATUS, VMSTAT, TOP...) there is also a very simple to use tool written by Jeremy Zawodny, it is perfect for non-admin users. It is called "mytop". more info # http://jeremy.zawodny.com/mysql/mytop/
Hi friend as per my research we have some command like
MYTOP: open source program written using PERL language
MTOP: also an open source program written on PERL, It works same as MYTOP but it monitors the queries which are taking longer time and kills them after specific time.
Link for details of above command