We have a cluster of Tomcat servers in AWS BeansTalk connected to AWS RDS (MySQL) with Multi-AZ availability.
Some days ago, the RDS instance had a patch applied to the OS which triggered a failover to another RDS instance based on the Multi-AZ availability.
The result was a Production system down during hours (it was at night) until we restarted the Tomcats in each instance. We had thousands of Connection refused errors to database.
According to AWS support, when a failover instance is launched, the endpoint is the same but its IP is changed, and my Tomcats had the old IP cached. So after restarting Tomcat the cache was cleared, the new IP was used and the connectivity issue was resolved. They refer me to this SO question.
That makes a lot of sense however I couldn't reproduce the issue in a controlled test with the same application in Production.
I changed the IP of a domain in /etc/hosts and my current BeansTalk Production Tomcat detected the IP change 30 seconds later, so it should have detected the RDS endpoint IP change too.
The Java ttl property in my BeansTalk environment is set as:
#networkaddress.cache.ttl=-1
So, by default it takes 30 secs as cache, that matches with my experiment.
[EDIT] As suggested in the comments, I've tried to simulate a failover through DNS. In this case, I've changed a CNAME record from a domain to another domain. I did the same test and Tomcat detected the change again 30 seconds later.
Do you have any idea why in this case the RDS endpoint IP change was not detected by Tomcat/JVM?
I have my Sails application on an AWS instance with all dependancies installed with no apparent issues. However, each time I try to launch the app I am getting the following error.
error: AdapterError: Connection is already registered
I have not managed to successfully lift sails yet on the instance and sails-mysql was freshly installed so no connections should be registered.
I have taken the following steps to deploy my app..
Set up a MySql RDS instance (EU-West)
Created and set up an Ubuntu AMD-64 t2.micro EC2 instance (EU-West)
Installed all prerequisites (Git, NVM, NodeJs, Sails, etc.)
Cloned my Sails project
Installed dependencies for Sails
Correctly configured my connection settings for Sails to use my RDS instance.
I know that my connection settings are correct as I have been able to run Sails on my local machine with a connection to my RDS instance and it would consistently lift without any issues.
I am also able to connect to my RDS instance using SequelPro with no problems.
I have had issues with dependencies in the past but have managed to fix those issues and have not had any of them on my local machine or with my EC2 instance.
After searching for a while I have come across a few users who have had similar issues but have managed to fix them with Waterline's teardown methods, however, I am unsure how to achieve this.
I have done my best to provide as much information as possible and any help would be massively appreciated.
Sails Version: 0.12.11
Thank you in advance.
I managed to fix the issue by carrying out the following:
Switched my environment to production in config/bootstrap.js
In connections.js add connectTimeout: 20000 to make sure the request does not time out before the connection is made.
eg. process.env.NODE_ENV = 'development'
Ensure that the security group inbounds rules for the RDS allows connections from the security group associated with my EC2 instance.
Type: MySQL/Aurora
Protocol: TCP
Port Range: 3306
Source: < Your security group ID >
Following the above points also meant I overcame the issue with handshake timeouts when communicating with the RDS.
I have a GCE instance running. Is there a way to schedule periodic restart?
I'm asking because I have a memory leak in JVM that I can't solve with out restarting the instance. Right now I restart it manually but it would be nice to automate it.
I'm new to GCE and want to migrate my web site there. I created a VM instance group hoping. I installed all the packages and set it up a couple days ago. But today I noticed my VM instance group has a different name (postfix, to be exact), and the disk is flushed empty. Is it possible to restore its status, or at least make sure it won't get wiped out again? I'm so surprised that GCE wiped out everything and I wonder if I'm missing something during setup.
A few details in case they are related:
I'm using a trusty image for the VM.
The cloud storage is chosen to be a regular persistent disk.
It was working with emphemeral IP, and yesterday I started to use Cloud DNS to host my domain. I should have used a static IP, but that mistake shouldn't cause the VM instance group to be flushed...
I'm using cloud sql as the database service.
Maybe I should just use VM instance, given I don't have much traffic now?
Any help will be greatly appreciated~
What are the advantages we get by using Elastic Beanstalk over maually creating EC2 instance and setting up tomcat server and deploy etc for a typical java web applicaion. Are load balancing, Monitoring and autoscaling the only advantages?
Suppose for my web application which uses database I installed the database in the EC2 instance itself. When Autoscalling takes place will the database gets created in the newly created instance or it will be accessing the database I created in the master instance... If it creates just a replica when autoscaling happens how will be data sync happens between the instances?
All the things you mentioned like load balancing, monitoring and auto-scaling are definitely advantages.
However, you have to kind of think about it this way: In a true Platform as a Service (PAAS), the goal is to separate the application from the platform. As a developer, you only worry about your application. The platform is "rented" to you. The platform "instances" are automatically updated, administered, scaled, balanced, etc. for you. You just upload your WAR file and it just works (at least theoretically).
EC2 by itself is not PAAS. It is more like IAAS (Infrastructure as a Service). You still have to take care of the server instances, install software on them, keep them updated, etc.
Elastic Beanstalk is a PAAS system. So are App Engine and Azure among many others.
In a true PAAS system, the DBMS is a separate component from the web application server(s). The reason is obvious: The DBMS cannot be possibly installed on the instances that are being used for the application server because, as instances are created and destroyed based on your traffic, the DBMS would be lost! Having the DBMS and application server on the same machine/instance is not generally a good idea anyway.
In a PAAS system, the DBMS is a separate service. For Amazon, it would be Amazon RDS. Just like with Elastic Beanstalk, where you don't have to worry about the application server and you just upload your WAR file, with RDS, you don't have to worry about the DBMS and you just deploy your database(s).
Elastic Beanstalk and RDS work very well together, especially when deployed in the same availability zone, where the latency would be very low.
Finally, using Elastic Beanstalk doesn't cost anything more than the deployed resources (EC2 instances and the load balancer). However, RDS is not cheap and would definitely be more expensive than using a single EC2 instance for both the application server and the DBMS.
Elastic Beanstalk does more than just load balancing, monitoring, and autoscaling.
1) Manages application versions by storing and managing different versions of your application, allowing you to easily switch back and forth between different versions of your applications.
2) Has the concept of "environments" for each application, allowing you to deploy different versions of your application in each environment. This is handy for example if you want to set up separate QA and DEV environments, and you want to easily deploy a build first in DEV then deploy the same version of the application in QA when your QA team is ready for the next build.
3) Externalizes the important container configuration properties (Tomcat memory settings, for example) to the Elastic Beanstalk console and API. Because of this you can easily save the settings and copy them between environments.
4) View application log files through the console and automatically roll and archive log files to S3. (Admittedly this feature is currently a little weak.)
I had an app deployed both in EC2 dedicated(Nginx & Gunicorn) and Beanstalk Environment(CentOS & Apache2).
My observations:
BeanStalk is Paas. Manually creating an EC2 instance(IAAS), is like doing everything from scratch, but you have solid control.
BeanStalk comes with by default CentOS and Apache(Httpd). You could choose OS in dedicated instance.
These things that mattered to me,
There were lots of 504 errors showing up in Beanstalk environment.
It was difficult to debug when BeanStalk server crashed, as logs would also not show up and could not ssh into machine. This is very important.
Installing/configuring tools like Celery, Redis (need to run another port) etc.,. in dedicated instance is lot more easier.
In my case, I had to scale up (Beanstalk)server in order to run installation of some packages(like pandoc). These things are more simpler in Ubuntu.
Scaling is a lot more easier in BeanStalk. Cloning servers is straightforward in BeanStalk.
I had taken micro in both the cases (dedicated & Beanstalk). I felt dedicated micro instance was better.
Automated deployment in Beanstalk. I had to write scripts to automate the same, which is fine, since it is only once.