I am trying to hit 350 users but Jmeter failing script by saying Connection timed out.
I have added following:
http.connection.stalecheck$Boolean=true in hc.parameter file
httpclient4.retrycount=1
hc.parameter.file=hc.parameter
Is there anything that I am missing to add on?
This normally indicates a problem on application under test side so I would recommend checking the logs of your application for anything suspicious.
If everything seems to be fine there - check logs of your web and database servers, for instance Apache HTTP Server allows 150 connections by default, MySQL - 100, etc. so you may need to identify whether you are suffering from this form of limits and what needs to be done to raise them
And finally it may be simply lack of CPU or free RAM on application under test side so next time you run your test keep an eye on baseline OS health metrics as application may respond slowly or even hang if it doesn't have any spare headroom. You can use JMeter PerfMon plugin to integrate this form of monitoring with your load test.
Related
I have a running instance of VerneMQ (cluster of 2 nodes) on Google kubernets and using MySQL (CloudSQL) for Auth. Server accepts connections over TLS
It works fine, but after a few days i start seeing this message on the log:
can't authenticate client {[],<<"Client-id">>} from X.X.X.X:16609 due to plugin_chain_exhausted
The client app (paho) complains that the server refused the connection for being "not authorized (code=5 in paho error)"
after a few retry it finally connects. but every time it get's harder and harder until it just won't connect anymore
If i restart VerneMQ everything get's back to normal
I have only 3 clients currently connected at most, at the same time.
clients already connected have no issues in pub/sub.
In my configuration i have (among other things):
log.console.level=debug
plugins.vmq_diversity=on
vmq_diversity.mysql.* = all of them set
allow_anonymous=off
vmq_diversity.auth_mysql.enabled=on
it's like the server degrades over time. the status webpage reports no problem
My verne server was build from the git repository about a month ago and runs on a docker container
what could be the cause?
what else could i check to find posibles causes? maybe a diversity missconfiguration?
Tks
To quickly explain the plugin_chain_exhausted log: with Verne you can run multiple authentication/authorization plugins, and they will be checked in a chain. If one plugin allows the client, it will be in. If no plugin allows the client, you'll see the log above.
This does not explain the behaviour you describe, though. I don't think I have seen that.
In any case, the first thing to check is whether you actually run multiple plugins. For instance: have you disabled the vmq.passwd and the vmq.acl plugins?
Hope you are all doing good. I have a problem with SSRS as it is stuck in the loading state (tried with several different browsers) even though executionlog3 says execution has been completed. I agree with executionlog3 as I don't see any running query in the database.
When I check SSRS logs, I see below error corresponds to the execution I do.
I have checked out a couple of websites but did not get any benefit. Any of you have an idea what could be the issue?
ERROR: Failed in BaseWorkerRequest::SendHttpResponse(bool), exception=System.ArgumentException: Value does not fall within the expected range.
This is most often because some intermediary has concluded that the TCP socket is not in use and has been closed. Because SSRS and the browser are both completely ignorant to the fact that the TCP socket is now closed, the server will continue to work on the report and the browser will continue to wait for it. The server will error out when it finally attempts to send the completed file over the socket only to find the session no longer exists, while the browser will continue to load until it times out.
Check to see if your requests are passing through a load balancer or firewall. Attempt connecting directly to the SSRS server to see if you can replicate the issue if they're not in the way. If it ends up being something like a load balancer, talk to the admins and ask them to extend the session timeout.
Alternatively, execute long running reports as subscriptions.
Error displaying the error page: Application Instantiation Error: Could not connect to MySQL.
This Error is shown sometimes (one every 20 pageviews) and I tried a lot of solutions without any change.
This error found with Joomla 3.5+
This will help you do some testings...
On MySql server run this query...
show processlist;
If this shows a lot of connections then you may be hitting the max_connections problem. Try to increase that to a reasonable number. But, do not make it too much because if it goes beyond the amount of available resources on your server then it will make MySql server crash and it will become a bigger problem where your website will go offline.
Try to check if you are closing sessions properly in your scripts using session_write_close(); You can find more about this function here: http://php.net/manual/en/function.session-write-close.php
This function will write user's current session on disk to be reloaded next time and most important part it will do for you with this problem is that it will close PHP's any connection made to the MySql server, so it will free up the MySql server for next visitor to come and open another page.
Lets say if you have 150 max_connections and at the same time you get 200 visitors, then PHP will still wait for some time for another 50 connections to be free before giving a timeout error where you get "Could not connect to server". But, if sessions are not closing properly, then its so bad that 50 connections will never get processed because 150 will never go away very soon, there is a separate MySql setting for that named wait_timeout. So, any MySql connection you open will keep waiting until you close yourself in script just like you open OR until wait_timeout happens. But, if you close sessions properly then next connection will become available as soon as your PHP script has finished processing and page is generated for the user for viewing and it will make the connection seat free for next connection to come in.
Server RAM was very low!
After upgrade from 0.5GB to 2GB all the kind of error starts with Error displaying the error page: gone.
I found the problem solution by using htop while connecting to the server via SSH, after that, I start opening pages rapidly which show the error plus the htop shows that the RAM was close to full.
show processlist; were helpful to find that the problem not about the connection number.which it may sometimes cost the same error in the same way.
I am currently trying to move my web server (php zendframework based) from Ubuntu to FreeBSD. Both the servers having the same hardware configuration. After migration, I did JMeter test (Http request (Json), Concurrent = 200) of the server, "Throughput" in FreeBSD server was double that of the Ubuntu server which is amazing.
However, when I increase the concurrent to 500, I see almost 50% request Failure due to "java.net.SocketException: Connection reset". But it works as normal in the Ubuntu server.
After many times testing, I found Ubuntu can handle 1500 concurrent httprequest without error, FreeBSD server can handle 200 concurrent request with double speed without error, but cannot handle more. In order to verify the result, I tried AB command. **ab -c 200 -n 5000 127.0.0.1/responseController. It fails and terminate if the ¬-c parameter is over 200, but works fine in Ubuntu.
For debugging I did following:
1. adjust httpd.conf, /boot/loader.conf, /etc/sysctl.conf somehow, but looks like nothing changed.
2. I try switch to mpn_worker_module in Apache configuration and its relevant configuration in php. Nothing changed but failure part log was different, which showed "request failure to respond" rather than "java.net.SocketException: Connection reset"
I did a lot of search but couldn't find the cause of this failure. I though the Json request would be waiting until response or timeout?
I am not sure which configuration file or parameter will make it work.
Please help.
Thanks for Michael Zhilin, yes "ipfw" did something to caused this, and yes "kern.ipc.soacceptqueue" is the bottleneck in this case.
I'm using Node.js to run a web-server for my web application. I'm also using the node-mysql module to interface with a MySQL server for all my persistent database needs.
Whenever there is a critical error within my Node.js application that crashes my app's process I get an email sent to me. So, I keep getting this email with an error saying "Too many connections". Here's an example of the error:
Error: Too many connections
at Function.Client._packetToUserObject (/apps/x/node_modules/mysql/lib/client.js:394:11)
at Client._handlePacket (/apps/x/node_modules/mysql/lib/client.js:307:43)
at Parser.EventEmitter.emit (events.js:96:17)
at Parser.write.emitPacket (/apps/x/node_modules/mysql/lib/parser.js:71:14)
at Parser.write (/apps/x/node_modules/mysql/lib/parser.js:576:7)
at Socket.EventEmitter.emit (events.js:96:17)
at TCP.onread (net.js:396:14)
As you can see all it tells me is that the error is coming from the mysql module, but it doesn't tell me where in my application code the issue is occurring.
My application opens a db connection anytime I need to run one or more queries. I immediately close the connection after all my queries and data has been collected. So, I don't understand how I could be exceeding the 151 max_connections limit.
Unless there is a place in my code where I forgot to call db.end() to close the connection, I don't see how my app would leak like this. Even if there was such a mistake, I wouldn't get these emails sent by the dozens. Yesterday, I received almost 100 emails with roughly the same error. How could this be happening? If my application had leaked and allocated connections over time, as soon as the first error occurred the app process would crash and all connections would be lost, preventing the app to crash again. Since I received ~100 emails, this means the app crashed ~100 times, and all within a short period of time. This could only mean that somewhere in my application a lot of connections where established in a short period of time, right?
How could I avoid this problem? This is very discouraging. All help is highly appreciated. Thanks
MySQL has a default MAX_CONNECTIONS = '100' not 151 unless you changed it. Also, in truth you have MAX_CONNECTIONS + 1. The plus 1 allows a root user to logon even after you have maxed out the conenctions in order to figure out what is actually being used. When your connections are maxed out try logging on as root and running the following command from MySQL.
mysql> SHOW FULL PROCESSLIST
Post the output of this command above. Once you actually know what is consuming your resources you can go about fixing it.It could easily be your code that is leaving open connections.
You should take a look at the follwoing documentation: Show Processlist
+1 for question. Investigations showed us that node-mysql opens the connections and doesn't close them. Because of that at one moment be reach the limit of max connections. The question is why node-mysql doesn't close the connections?