I'm running my application with the Micro CloudFoundry, but I'm having trouble connecting to MySQL 'User 'usGh0jJk8EoZn' has exceeded the 'max_user_connections' resource'. How can I change this value?
I'm not quite sure you can change that value.
Before going down that road though, you may want to make sure that you are not leaking connections. Is your application running correctly when deployed locally (i.e. not using regular CloudFoundry nor Micro CF)? How are you connecting to the database? It may seem strange that you hit a connection limit if you're actually the sole user of your app, which I assume you are if using micro.
as ebottard said, it's well worth making sure your code isn't leaking connections. But, if you want to change the mysql setup for the instance running on Micro CloudFoundry, you can SSH in to the VM using the 'vcap' user.
Once connected, you will find the mysql configuration file at /var/vcap/jobs/mysql_node/config/my.cnf
For maximum connections you will also have to change the max_user_conns value in /var/vcap/jobs/mysql_node/config/mysql_node.yml
Please also take a look at;
http://docs.cloudfoundry.com/infrastructure/micro/using-mcf.html#logging-in-to-micro-cloud-foundry
Related
Right now I am connecting to a cluster endpoint that I have set up for an Aurora DB-MySQL compatible cluster, and after I do a "failover" from the AWS console, my web application is unable to properly connect to the DB that should be writable.
My setup is like this:
Java Web App (tomcat8) with HikariCP as the connection pool, with ConnecterJ as the driver for MySQL. I am evaluating Aurora-MySQL to see if it will satisfy some of the needs the application has. The web app sits in an EC2 instance that is in the same VPC and SG as the Aurora-MySQL cluster. I am connecting through the cluster endpoint to get to the database.
After a failover, I would expect HikariCP to break connections (it does), and then attempt to reconnect (it does), however, the application must be connecting to the wrong server, because anytime a write is hit to the database, a SQL Exception is thrown that says:
The MySQL server is running with the --read-only option so it cannot execute this statement
What is the solution here? Should I rework my code to flush DNS after all connections go down, or after I start receiving this error, and then try to re-initiate connections after that? That doesn't seem right...
I don't know why I keep asking questions if I just answer them (I should really be more patient), but here's an answer in case anyone stumbles upon this in a Google search:
RDS uses DNS changes when working with the cluster endpoint to make it looks "seamless". Since the IP behind the hostname can change, if there is any sort of caching going on, then you can see pretty quickly how a change won't be reflected. Here's a page from AWS' docs that go into it a bit more: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html
To resolve my issue, I went into the jvm's security file and then changed it to be 0 just to verify if what was happening was correct. Seems correct. Now I just need to figure out how to do it properly...
Is it possible to cache database connections when using PHP like you would in a J2EE container? If so, how?
There is no connection pooling in php.
mysql_pconnect and connection pooling are two different things.
There are many problems connected with mysql_pconnect and first you should read the manual and carefully use it, but this is not connection pooling.
Connection pooling is a technique where the application server manages the connections. When the application needs a connection it asks the application server for it and the application server returns one of the pooled connections if there is one free.
We can do connection scaling in php for that please go through following link: http://www.oracle.com/technetwork/articles/dsl/white-php-part1-355135.html
So no connection pooling in php.
As Julio said apache releases all resources when the request ends for the current reques. You can use mysql_pconnect but you are limited with that function and you must be very careful. Other choice is to use singleton pattern, but none of this is pooling.
This is a good article: https://blogs.oracle.com/opal/highly-scalable-connection-pooling-in-php
Also read this one http://www.apache2.es/2.2.2/mod/mod_dbd.html
Persistent connections are nothing like connection pooling. A persistent connection in php will only be reused if you make multiple db connects within the same request/script execution context. In most typical web dev scenarios you'll max out your connections way faster if you use mysql_pconnect because your script will have no way to get a reference to any open connections on your next request. The best way to use db connections in php is to make a singleton instance of a db object so that the connection is reused within the context of your script execution. This still incurs at least 1 db connect per request, but it's better than making multiple db connects per reqeust.
There is no real db connection pooling in php due to the nature of php. Php is not an application server that can sit there in between requests and manage references to a pool of open connections, at least not without some kind of major hack. I think in theory you could write an app server in php and run it as a commandline script that would just sit there in the background and keep a bunch of db connections open and pass references to them to your other scripts, but I don't know if that would be possible in practice, how you'd pass the references from your commandline script to other scripts, and I sort of doubt it would perform well even if you could pull it off. Anyway that's mostly speculation. I did just notice the link someone else posted to an apache module to allow connection pooling for prefork servers such as php. Looks interesting:
https://github.com/junamai2000/mod_namy_pool#readme
I suppose you're using mod_php, right?
When a PHP file finishes executing all it's state is killed so there's no way (in PHP code) to do connection pooling. Instead you have to rely on extensions.
You can mysql_pconnect so that your connections won't get closed after the page finishes, that way they get reused in the next request.
This might be all that you need but this isn't the same as connection pooling as there's no way to specify the number of connections to maintain opened.
You can use MySQLi.
For more info, scroll down to Connection pooling section # http://www.php.net/manual/en/mysqli.quickstart.connections.php#example-1622
Note that Connection pooling is also dependent on your server (i.e. Apache httpd) and its configuration.
If an unused persistent connection for a given combination of "host, username, password, socket, port and default database can not be found" in the open connection pool, then only mysqli opens a new connection otherwise it would reuse already open available persistent connections, which is in a way similar to the concept of connection pooling. The use of persistent connections can be enabled and disabled using the PHP directive mysqli.allow_persistent. The total number of connections opened by a script can be limited with mysqli.max_links (this may be interesting to you to address max_user_connections issue hitting hosting server's limit). The maximum number of persistent connections per PHP process can be restricted with mysqli.max_persistent.
In wider programming context, it's a task of web/app server however in this context, it's being handled by mysqli directive of PHP itself in a way supporting connection re-usability. You may also implement a singleton class to get a static instance of connection to reuse just like in Java. Just want to remind that java also doesn't support connection pooling as part of its standard JDBC, they're being different module/layers on top of JDBC drivers.
Coming to PHP, the good thing is that for the common databases in the PHP echosystem it does support Persistent Database Connections which persists the connection for 500 requests (config of max_requests in php.ini) and this avoids creating a new connection in each request. So check it out in docs in detail, it solves most of your challenges. Please note that PHP is not so much sophisticated in terms of extensive multi-threading mechanism and concurrent processing together with powerful asynchronous event handling, when compared to strictly object oriented Java. So in a way it is very less effective for PHP to have such in-built mechanism like pooling.
You cannot instantiate connection pools manually.
But you can use the "built in" connection pooling with the mysql_pconnect function.
I would like to suggest PDO::ATTR_PERSISTENT
Persistent connections are links that do not close when the execution of your script ends. When a persistent connection is requested, PHP checks if there's already an identical persistent connection (that remained open from earlier) - and if it exists, it uses it. If it does not exist, it creates the link.
Connection pooling works at MySQL server side like this.
If persistence connection is enabled into MySQL server config then MySQL keep a connection open and in sleep state after requested client (php script) finises its work and die.
When a 2nd request comes with same credential data (Same User Name, Same Password, Same Connection Parameter, Same Database name, Maybe from same IP, I am not sure about the IP) Then MySQL pool the previous connection from sleep state to active state and let the client use the connection. This helps MySQL to save time for initial resource for connection and reduce the total number of connection.
So the connection pooling option is actually available at MySQL server side. At PHP code end there is no option. mysql_pconnect() is just a wrapper that inform PHP to not send connection close request signal at the end of script run.
For features such as connection pooling - you need to install swoole extension first: https://openswoole.com/
It adds async features to php.
After that its trivial to add mysql and redis connection pooling:
https://github.com/open-smf/connection-pool
Some PHP frameworks come with pooling built-in: https://hyperf.wiki/2.2/#/en/pool
I saw the following statement on StackOverflow and was wondering about its meaning:
If you connect via 'localhost', the connection will automatically be established via the MySQL socket, which is really cheap anyways.
The discussion thread was pretty old, so I didn't want to comment on it.
Basically what I understand is, that using 'localhost' when connecting to your mysql database has certain advantages - such as "automatically established connections via MySQL socket". What does that mean exactly?
Currently I'm using
mysql_connect("73.21.24.201", [...]);
(changed to a random IP Address)
Does it make any difference? Can I change it to "localhost" without having to worry about it? (The mysql server is obviously on the same server/ip address as my website/application)
When you connect to 'localhost' you'll connect using a Unix socket, which is just a communications channel for the local processes to use. The big advantage of this is that you can disable networking completely in MySQL, and negate any processing overhead and security risks that go along with that.
When MySQL starts, it creates a socket file (typically at a place like /var/lib/mysql/mysql.sock) that your client program needs to be able to find. On a typical PHP (you didn't say, but I'm assuming) setup, it should know where to find this socket. If not, check /etc/my.cnf and /etc/php.ini to make sure the values match.
And finally, if that is PHP, stop using mysql_*() functions in PHP right now! They have been deprecated for years and are inefficient and insecure.
I'm running EC2 with MySQL RDS to serve dynamic websites' content.
The server was down due to 'too many connection' error on RDS database.
As it was urgent, I restart database server straight away and the problem gone.
However, I'm unable to see what queries produce those connection (as I didn't run SHOW PROCESS LISTS before reboot RDS).
CloudWatch show 250+ connections during period of issue which is obviously huge distinction from normal operating on other days.
I try to address the issue by see log in RDS, but there is quite a minimal message there.
The error message
2014-05-03 06:10:08 3628 [Warning] IP address '173.244.206.19' has been resolved to the host name '173.244.206.19.static.midphase.com', which resembles IPv4-address itself.
From above, 173.244.206.19 is not in our IP list both public and private. (but connection open to 0.0.0.0 secured with password which I'm going to limit IP remote in security group soon)
Questions
Is 173.244.206.19.static.midphase.com is something to do with RDS by default. I think this is obviously an attack sign but just would like to confirm.
What does 'resembles IPv4-address itself' mean? As this is RDS database server only, why server does need to resolve DNS?
Are there any way to digging into this for further detail (e.g. to see specific query).
I'm going to prevent this by only limit the IP access along with CloudWatch alarm setting for 10+ connections. Anything else I should do.
Thank you for reading through this guys. I'm the only developer in company start-up which take care for all front-end/backend/application/network. Therefore, apology if there are dummy questions out here.
However, your help would be really appreciated and will save a bit of my life writing the report.
You are maybe the target of a DDOS attack or a brute-force password discovery attempt.
I would report this to AWS support - as they can help to mitigate the effect of the attack.
As a best practice, we do not recommend to use 0.0.0.0/0 as source IP address for incoming connection rule in Security Group.
Try to restrict which IP addresses are authorised to connect to your database.
If you are accessing from on prem network, specify only your on-prem address range.
If you're accessing your database from an app server installed on EC2, use the ID of the App Server Security Group (sg-xxxx) as source authorised to connect to your database.
I have the following setup:
An EC2 instance hosting both an application server and a database (mysql), belonging to a security group: let's call it "AppServerSG", and assigned an elastic Public IP (AWS also assigns it a private IP).
Various EC2 worker instances which need to connect to the application server's database when booting up. These worker instances belong to another security group: let's call it "WorkerSG".
The inbound rules for the Security Groups look as follows.
For AppServerSG:
80 (HTTP) 0.0.0.0/0
3306 (MYSQL) WorkerSG
For WorkerSG
80 (HTTP) AppServerSG
So essentially only the application server should be reachable from outside, and the workers and application should be able to communicate with each other.
However connecting to the database from a worker instance only succeeds when the database host is set as the application server's private IP, not the public elastic IP.
The only way to connect to the database from a worker instance using the application server's public IP seems to require changing the MYSQL rule to allow all connections (0.0.0.0/0) on the AppServerSG, which is something I'm very reluctant to do out of security concerns.
Hard-coding the private IP into the worker instances is also not such a good idea, since every time the app server instance is stopped/restarted, it is assigned a new private IP, which would then require manually changing the database address that each worker instance needs to connect to.
I'm basically wondering if someone has run into similar trouble because this doesn't seem like the way things should work, so either I'm doing something wrong in my setup, or there's a workaround somehow.
Would very much appreciate the help !
Edit:
The motivation behind this setup is that in the event that I want to take the whole thing offline, I can safely bring it back online without having to change the configurations of the application server and the workers.
Had I used RDS, when taking the application offline/online again I would have to take a snapshot of the DB and stop it, then create a new DB based on the snapshot, which would have a different address, which would then bring me back to the problem of changing the configuration.
Honestly if I'm going to have to edit the configuration every time I restart the application anyway, I'd rather have the database on the application server and save myself the costs associated with RDS.
The main issue here is that I don't understand why the security groups don't seem to apply when I'm using the public elastic IP for the database address, is it by design on the AWS side, or a mistake in the configuration somewhere on my part ?
Really the recommended configuration would have you using an RDS DB instance, setting your DB security group to accept connections from the appropriate EC2 security groups only. In this configuration, you CAN set up your DB user like user#% and still enforce access to the DB only to the specified EC2 security groups.
In this way, you shift the burden of DB access control to the AWS security model, rather than MySQL user configuration. Of course, you would still need to configure DB users to have access only to those appropriate resources within the DB.