I created an application that works perfect in my computer but when I uploaded it to start server tests it becomes very slow, specially after a couple of uses (the first minutes work fine)...It even becomes unresponsive, as I move through a treetable a form should be updated from the database but stops working after a while...
I'm using an Amazon EC2 Linux server and a MySQL database...I checked if the connections to the database is what failed, but I'm using no more than 7 out of 150 max connections to the database.
Is this a common problem?
Any ideas on how to solve this?
Thanks!!!
Note: This is a copy of an internal vaadin forum thread: https://vaadin.com/forum#!/thread/4816326 ...Hope is not against the forum rules to do this...
It sounds like you may have a memory leak in your application somewhere that your computer is able to sustain, but your server is not. I would suggest trying some load testing on another machine and see what actions are causing it to spin out.
You can have a look at this SO answer to see how to do that:
https://stackoverflow.com/a/46227692/460802
Related
I'm working on an assignment for school, my files for the website are stored on a distant server which I access via VPN and remote server connection on macOS.
When I modify my html files, the changes aren't reflected immediately, sometimes after a day or two (in fact quite randomly, can be an afternoon, an hour).
It's a bit problematic when you try to have long code sessions. Sometimes, one page actualises but not the others.
I'm not having any problems with my php files, they actualise immediately.
I've tried several things without any changes:
Emptying the cache
Trying on different web browsers
Disconnecting from the server and VPN
Waiting :)
System infos :
macOS 10.12.2
Safari 10.0.2
Thanks for the help, I personally think it's a problem with the server, but I won't be able to change that, hopefully, it's something I can fix.
This is not the typical question, but I'm out of ideas and don't know where else to go. If there are better places to ask this, just point me there in the comments. Thanks.
Situation
We have this web application that uses Zend Framework, so runs in PHP on an Apache web server. We use MySQL for data storage and memcached for object caching.
The application has a very unique usage and load pattern. It is a mobile web application where every full hour a cronjob looks through the database for users that have some information waiting or action to do and sends this information to a (external) notification server, that pushes these notifications to them. After the users get these notifications, the go to the app and use it, mostly for a very short time. An hour later, same thing happens.
Problem
In the last few weeks usage of the application really started to grow. In the last few days we encountered very high load and doubling of application response times during and after the sending of these notifications (so basically every hour). The server doesn't crash or stop responding to requests, it just gets slower and slower and often takes 20 minutes to recover - until the same thing starts again at the full hour.
We have extensive monitoring in place (New Relic, collectd) but I can't figure out what's wrong; I can't find the bottlekneck. That's where you come in:
Can you help me figure out what's wrong and maybe how to fix it?
Additional information
The server is a 16 core Intel Xeon (8 cores with hyperthreading, I think) and 12GB RAM running Ubuntu 10.04 (Linux 3.2.4-20120307 x86_64). Apache is 2.2.x and PHP is Version 5.3.2-1ubuntu4.11.
If any configuration information would help analyze the problem, just comment and I will add it.
Graphs
info
phpinfo()
apc status
memcache status
collectd
Processes
CPU
Apache
Load
MySQL
Vmem
Disk
New Relic
Application performance
Server overview
Processes
Network
Disks
(Sorry the graphs are gifs and not the same time period, but I think the most important info is in there)
The problem is almost certainly MySQL based. If you look at the final graph mysql/mysql_threads you can see the number of threads hits 200 (which I assume is your setting for max_connections) at 20:00. Once the max_connections has been hit things do tend to take a while to recover.
Using mtop to monitor MySQL just before the hour will really help you figure out what is going on but if you cannot install this you could just using SHOW PROCESSLIST;. You will need to establish your connection to mysql before the problem hits. You will probably see lots of processes queued with only 1 process currently executing. This will be the most likely culprit.
Having identified the query causing the problems you can attack your code. Without understanding how your application is actually working my best guess would be that using an explicit transaction around the problem query(ies) will probably solve the problem.
Good luck!
I'm having trouble with a web service deployed on Tomcat. During peak traffic times the server is becoming non response and forces me to restart the entire server in order to get it working again.
First of all, I'm pretty new to all this. I built the server myself using various guides and blogs. Everything has been working great, but due to the larger load of traffic, I'm now getting out of my league a little. So, I need clear instructions on what to do or to be pointed towards exactly what I need to read up on.
I'm currently monitoring the service using JavaMelody, so I can see the spikes occurring, but I am unaware how to get more detailed information than this as to possible causes/solutions.
The server itself is quad core with 16gb ram, so the issue doesn't lie there, more likely in the fact I need to properly configure Tomcat to be able to use this (or setup a cluster...?)
JavaMelody shows the service crashing when the cpu usage only gets to about 20%, and about 300 hits a minute. Is there any max connection limits of memory settings that I should be configuring?
I also only have a single instance of the service deployed. I understand I can simply rename the war file and Tomcat deploys a second instance. Will doing this help?
Each request also opens (and immediately closes) a connection to mySQL to retrieve data, I probably need to be sure it's not getting throttled there too.
Sorry this is so long winded and has multiple questions. I can give more information as needed, I am just not certain what needs to be given at this time!
The server has 16Gs of ram but how much memory do you have dedicated to tomcat, -Xms and -Xmx?
There is an action in the admin section of a client's site, say Admin::Analytics (that I did not build but have to maintain) that compiles site usage analytics by performing a couple dozen, rather intensive database queries. This functionality has always been a bottleneck to application performance whenever the analytics report is being compiled. But, the bottleneck has become so bad lately that, when accessed, the site comes to a screeching halt and hangs indefinitely. Until yesterday I never had a reason to run the "top" command on the server, but doing so I realized that Admin::Analytics#index causes mysqld to spin at upwards of 350+% CPU power on the quad-core, production VPS.
I have downloaded fresh copies of production data and the production log. However, when I access Admin::Analytics#index locally on my development box, while using the production data, it loads in about 10 - 12 seconds (and utilizes ~ 150+% of my dual-core CPU), which sadly is normal. I suppose there could be a discrepancy in mysql settings that has suddenly come into play. Also, a mysqldump of the database is now 531 MB, when it was only 336 MB 28 days ago. Anyway, I do not have root access on the VPS, so tweaking mysqld performance would be cumbersome, and I would really like to get to the exact cause of this problem. However, the production logs don't contain info. on the queries; they merely report the length that these requests took, which average out to a few minutes apiece (although they seemed to have caused mysqld to stall for much longer than this and prompting me to request our host to reboot mysqld just to get our site back up in one instance).
I suppose I can try upping the log level in production to solicit info. on the database queries being performed by Admin::Analytics#index, but at the same time I'm afraid to replicate this behavior in production because I don't feel like calling our host up to restart mysqld again! This action contains a single database request in its controller, and a couple dozen prepared statements embedded in its view!
How would you proceed to benchmark/diagnose and optimize/fix this action?!
(Aside: Obviously I would like to completely replace this functionality with Google Analytics or a similar solution, but I need fix this problem before proceeding.)
I'd recommend taking a look at this article:
http://axonflux.com/building-and-scaling-a-startup
Particularly, query_reviewer and newrelic have been a life-saver for me.
I appreciate all the help with this, but what turned out to be the fix for this was to implement a couple of indexes on the Analytics table to cater to the queries in this action. A simple Rails migration to add the indexes and the action now loads in less than a second both on my dev box and on prod!
I have a web that currently runs off one Mediatemple VPS. I'm now at the stage were the site is getting bogged down with scaling issues and I need to move to a better setup.
Is this a sensible setup:
php on one server
mysql database on another
Cloud Files CDN used to server images, javascript and css
My main thinking is to put the MySQL database on its own server away from the rest of the files as it seems to be causing most of the problems.
Try grabbing the lowest hanging fruit by tackling the scaling problem that are easiest to resolve and don't involve hardware solutions. Identify the bottleneck that has the greatest effect on performance. If its your SQL server that is running slow and being funky, read the sql logs, and do some googling :).
Just make sure that your hardware is the problem and not your application.