I'm using Apache HTTPClient (4.2.2) / Java7 to open many reusable connections to a tomcat 7 server (to simulate many users repeatedly hitting the service). Both client and server on Ubuntu 12 (but different machines). I made sure that systctl.conf and limits.conf allow this scenario.
This works well up to about 1500 simulated users / connections. The connections get reused as expected. Somewhere between 1500 and 1600 simulated users however, connections are no longer reused and closed/ re-opend all the time. Why might this be the case?
I don't think the problem is on the server side as when I start multiple simulation clients on different machines against the same server, the server has no problems reusing the connections as long as each client doesn't go beyond 1500 connections.
There can be various reasons as to why connections are not longer being re-used depending on the configuration of the connection manager OR server side configuration. The easiest way to find out the reason is to run HttpClient with context logging on as described in the 'context logging for connection management / request execution' example in the Logging Guide
You might need to increase the number of available workers,at least check if there are workers free when you run out of connections by going to server-status
Related
I have a Saas application on AWS ECS and databases on AWS RDS. We are planning to implement AWS RDS Proxy for pooling implementation. From the RDS proxy documentation, I saw that we don't need to make any changes to the application code. Currently, we are using application side connection pooling. When we implement an RDS proxy for pooling, does the current pooling have any impact?
Do we need to remove the application side pooling to work with RDS effectively?
My main concern is, if I choose 100% pooling in RDS proxy and from application pooling configuration if we limit that to say 100 max connection. Will that be a bottleneck?
TLDR: keep the connection pool in your application, and size it to the number of connections required by that one instance of your application (e.g. the ECS task or EKS pod).
With a database proxy in the middle, there are two separate legs to a "connection":
First, there is a connection from the application to the proxy. What you called the "application side pooling" is this type of connection. Since there's still overhead associated with creating a new instance of this type of connection, continuing to use a connection pool in your application probably is a good idea.
Second, there is a connection from the proxy to the database. These connections are managed by the proxy. The number of connections of this type is controlled by a proxy configuration. If you set this configuration to 100%, then you're allowing the proxy to use up to the database's max_connections value, and other clients may be starved for connections.
So, when your application wants to use a connection, it needs to get a connection from its local pool. Then, the proxy needs to pair that with a connection to the database. The proxy will reuse connections to the database where possible (this technique also is called multiplexing). Or, quoting the official docs: "You can open many simultaneous connections to the proxy, and the proxy keeps a smaller number of connections open to the DB instance or cluster. Doing so further minimizes the memory overhead for connections on the database server. This technique also reduces the chance of "too many connections" errors."
As your container orchestrator (e.g. ECS or EKS) scales your application horizontally, your application will open/close connections to the proxy, but the proxy will prevent your database from becoming overwhelmed by these changes.
I am using MySQL workbench on windows, which I want to connect to a Node-Red running on the IBM cloud. Since I don't run them on the same server the host 127.0.0.1 and port 3306 does not seem to work. What permissions should I give?
I'm going to make a LOT of assumptions here, because there really isn't enough information in your question.
First assumption, by "running on IBM" you mean that Node-RED is running on the IBM Cloud hosting service.
The short answer is you can not do what you want.
The longer version is that you probably could actually make this work but doing it is a REALLY bad idea.
Second assumption, you are doing this from home (even if you are doing it from a office location the same problems are likely to apply). This means you are connected to a local LAN using RFC 1918 address range (e.g. 192.168.0.x), this means you are behind a router that is performing NAT (Network Address Translation). This means you are going to need to set up portforwarding on the router so that when traffic arrives at the router it will send it on to your Windows machine. How you do this will depend on your router.
Next problem, your broadband probably doesn't have a static IP address which means it will change every time your connection drops. There are work arounds for this using things like Dynamic DNS. But that's too complicated to get into here.
Assuming you get all of that sorted out you still have the problem that you have now exposed your mysql database to the internet, so you need to make sure you have enabled all the right security measures to prevent people logging in and at best seeing all your data.
There are 2 much better solutions to this
Run Node-RED on the same machine or at least on the same local network as the database.
Use one of IBM Clouds hosted database solutions, these are a lot easier to connect to a IBM Cloud instance of Node-RED.
If you do not want to open ports to your network I recommend using a free MYSQL remote server
A simple website is https://remotemysql.com
Just take the screenshot of the credentials of your database after registration.
Keep in mind if your database is empty it will get deleted after some time.
I've created a Java Spring Boot application that launches 36 downloader droplets on digital ocean, which ssh tunnel to a database CPU Optimized droplet and downloads from an API into the database.
I've configured hikari as follows towards less pooling connections assuming the database may have trouble with too many and thinking they might not be required.
spring.datasource.hikari.maximumPoolSize=5
spring.datasource.hikari.connectionTimeout=200000
spring.datasource.hikari.maxLifetime=1800000
spring.datasource.hikari.validationTimeout=100000
I'm wondering if those settings may or may not be recommended and why. I've reduced the maximumPoolSize to 5 however I haven't found much information on whether that is considered too small for Java Spring Boot Application to run effectively.
Given each downloader is storing data in the database sequentially do I need to have more than a few pooling connections on each downloader?
I've configured the maximum connections in mysql to 250 and the maximum ssh connections on the database server to 200. I note that 114 sshD processes are created on the server. Can a server handle that many ssh tunneling connections?
Do you forsee any problems with this kind of distributed setup with Spring boot? One thing I have had to do before adjusting to these settings is place retry connection code around each database connection to prevent disconnection errors.
Thanks
Conteh
I am working in a project where we are using Go as a web server and MySQL.
We have been told to implement fault tolerance to handle a hardware crash. We were given 2 servers which have MySQL and the Go-server on them.
We have succesfully set up replication in MySQL, but we are struggling with the failover part. Our thought was to get an extra server with HAProxy to have a primary server and then being able to failover to the backup server.
We also considered using MySQL failover, but did not see how we could redirect the traffic using it.
Is this a reasonable plan? Or what would you recommend that we do instead?
If you want two identical servers connecting to their local MySQL instances, you need a way of deciding which one is the production server. There are a number of solutions for that, including
Setting up a reverse proxy, as you mention, but then, your proxy
itself becomes a SPOF,
Using a floating IP, also known as a failover
IP, but this only works if your host supports it. Cloud providers
typically support them, as well as some bare metal server providers.
There is nothing specific to Go as far as I know.
For our customer the application which is running is using MySQL database. However, this server is without monitoring. I want to install OpenNMS (which uses PostgreSQL) application to monitor the solution and send the traps to main NMS system.
Is there any problem having both on the same server?
No, there is no technical problem. Both default to different ports they listen on.
The only problem that could arise is that each individual DB might be slower compared to an installation on separate phyiscal machines because they are both share (and fight for) for the same resources (I/O, memory, CPU, network, ...)