I've a cluster, this cluster has four nodes.
If I stop one node, and edit the configuration file (add a new replicated cache),
When I'll start the node,
Will the cluster have the new replicated cache?
In the others three nodes, is it necessary change the configuration file?
Regards.
a) Yes, the new replicated cache will be created on the node. However, if you have the same cache (name) with different configurations, you're asking for trouble.
b) No, the configuration on other nodes will not change. You have to change it manually, either stopping the nodes, or running rolling upgrade.
You may also look into JMX operations for starting/stopping cache, but this does not allow to change the configuration (I am not 100% sure if starting a cache with unknown name wouldn't start a new cache with default configuration).
If you have programmatic access to CacheManager, you can start cache with configuration provided programmatically.
Related
I am working on logging of my application on Persistent Volume.
I am using OpenShift, I created storage(Persistent volume under nas-thin class) and allocated 64Gib to it. I added mount path to this PV for one of my pods where my application is running and generating logs in one of the folder named "logs".
My mount path is "/logs". Hence anything inside this folder will be root for my PVC.
I am appending my logs inside logs folder in a single file.
I tried to read about expanding PV but couldn't understand much.
What would happen if my log file size exceeds allocated PV size(which is 64Gib)?
That will depend on the persistent storage actually being used by the cluster. Some persistent volume providers will let you write more data than you actually defined. So you'll have to test how your storage and your application actually behave on your particular cluster.
That being said, it is generally a bad idea to have container workload log to a persistent volume. I would strongly recommend to log to STDOUT and then use an external logging system to manage your logs instead of writing to a file.
How will you deal with multiple replicas of your application running? Do you really want to go into each container to get the log files? How will you correlate logs between different replicas?
Applications running on OpenShift / Kubernetes should not manage their logs in files but write to STDOUT.
From the docs # http://docs.ejabberd.im/admin/guide/clustering/#clustering-setup
Adding a node into the cluster is done by starting a new ejabberd node within the same network, and running a command from a cluster node. On second node for example, as ejabberd is already started, run the following command as the ejabberd daemon user, using the ejabberdctl script: ejabberdctl join_cluster 'ejabberd#first'
How does this translate into deployment in the cloud- where instances can (hopefully) be shutdown/restarted based on a consistent image and behind a load balancer?
Can all of them, including the initial instance, use "example.com" as "first" in the example above (assuming "example.com" is setup in DNS to point to the cloud load balancer)?
Or must the first instance not attempt to join a cluster, and subsequent ones all use the ip address of that initial instance instead of "first" (and if this is the case- does it get wacky if that initial instance goes down)?
Can all of them, including the initial instance, use "example.com" as "first" in the example above (assuming "example.com" is setup in DNS to point to the cloud load balancer)?
No, the node name parameter is the node name of an Erlang ejabberd node. It should even more be on the internal Amazon network, not the public one, so it should not rely on a central DNS. It must be a name of an Erlang node as the newly started node with connect to the existing node to share the same "cluster schema" and do an initial sync of the data.
So, the deployment is done as follow:
first instance does not need to join a cluster indeed as there is no cluster schema to share.
New instance can use the node name of any other node of the cluster. It means they will add themselves to the ejabberd cluster schema. It means ejabberd knows that users can be on any node of this cluster. You can point to any running node in the cluster to add a new one, as they are all equivalent (there is no master).
You still need to configure the load balancer to balance traffic to public XMPP port on all nodes.
You only need to perform the cluster config for each once for each extra cluster node. The configuration with all the node is kept locally, so when you stop and restart a node, it will then automatically rejoined the cluster after it has been properly set up.
I'm setting up a high-availability environment for a customer. There are a pair of load-balanced hosts serving http requests to our Rails application, which is deployed in parallel on both hosts.
Additionally, there are two MySQL hosts available. I want to run MySQL Cluster (NDB) on both hosts (i.e., multi-master) to have a fully redundant configuration. I'm specifically trying to avoid a master-slave configuration based on database replication; I feel like that makes the writable node a single point of failure.
I'm looking for some guidance on how best to interface our Rails app to a multi-master MySQL cluster like this. Almost all of the database adapters I can find are for master-slave setups. Failover_adapter sounds very promising, but it's woefully outdated. I haven't managed to turn up anything similar developed in the last five years.
Is anyone aware of any gems to use or approaches to take to serve as an adapter between a Rails application and a multi-master MySQL cluster like I've described?
I've just recently set this up. There shouldn't be much work to be done.
In the mysqladapter gem it specifies the engine as InnoDB, which is obviously not suitable for clustering. You need to add this to an initialization file:
ActiveRecord::ConnectionAdapters::Mysql2Adapter
class ActiveRecord::ConnectionAdapters::Mysql2Adapter
def create_table(table_name, options = {})
super(table_name, options.reverse_merge(:options => "ENGINE=NDB"))
end
end
This is the original in the adapter.
def create_table(table_name, options = {})
super(table_name, options.reverse_merge(:options => "ENGINE=InnoDB"))
end
Another key aspect is you don't want migrations taking place at the same time so in the deploy.rb file:
task :migrate, :max_hosts => 1 do
# sleep 0.5
run "cd #{release_path} && bundle exec rake db:migrate RAILS_ENV=#{rails_env}"
end
Max hosts prevents cap from running the migrations in parallel. This is important as you don't want the cluster running create table type of things at the same time. It may even be worth putting a delay in I have commented above just for a little extra safety.
One more key aspect. Don't forget to set:
DataMemory =
IndexMemory =
The default values are supremely low. Typically index size is DataMemory/5-10
One more pitfall I've seen so far is in your mysqld nodes make sure you set:
ndb_autoincrement_prefetch_sz
to at least a 100. Otherwise bulk inserts will take forever. The default is 1.
EDIT:
ndb_autoincrement_prefetch_sz
Leave this variable completely alone. Don't set it. What it can cause is auto increment indexes to become out of sync on the cluster. Which is a nightmare to debug.
Additionally make sure you're NDB nodes don't run on the same server as the NDB MGM nodes.
Happy coding.
Ad
I ultimately was not able to find an adapter solution that did what I wanted.
I settled on using the mysql2 adapter and pointing it at a reverse-proxy (I used haproxy) in front of my database cluster that could handle load-balancing and failover between the master nodes.
I launched an Amazon EC2 with Amazon Linux and Amazon-EBS as root volume. I also started tomcat7 and mysql 5.5 on this EBS volume.
Later I decided to change from Amazon Linux to Ubuntu. To do that I need to launch another Amazon EC2 instance with a new EBS root volume. Now I want to copy tomcat7 and mysql from older EBS volume to new one. I have tables and data in mysql which I don't want to loose and an application running on tomcat. How to go about it?
A couple of thoughts and suggestions.
First, if you are going to be having any kind of significant load on your database, running it on EBS-backed volume is probably not a great idea as EBS-backed storage is incredibly slow relative to the machine's local/ephemeral storage (/mnt). Now obviously you don't want DB data on ephemeral storage, so there is really nothing you can do about it if you want to run MySQL on EC2. So my suggestion would be to utilize an RDS instance for your DB if your infrastructure requirements allow for it.
Second, if this is a production application, you are undoubtedly going to have some down time as you make this transition. The question is whether you need to absolutely minimize the amount of downtime. If so, then you need to have an idea as to the size of your database. Is it going to take a long time to dump/load? If not, you could probably just get your new instance up and running, and tested on an older copy of your database and then just dump and load the current database at the time of cutover.
If it is a large database then perhaps you can turn on MySQL binary logging. Then make a dump of the database at a known binary log position. Then install this dump on your new instance. Then when ready to cutover, you can replay the binary logs on the new instance to bring it current. Similarly, you could just set up the DB on the new instance as a replica until the cutover, at which point you make it the master.
You may even consider just using rsync to sync the physical database files if you don't want to mess with binary logging, though this can be a problematic approach if you are not that familiar with dealing with the actual physical database files.
As far as your application goes, that should be much simpler to migrate assuming it is just a collection of files. I would not copy the Tomcat7 installation itself, but rather just install Tomcat on Ubuntu and then adjust the configuration to match current.
As far as the cutover itself goes, this should be pretty straightforward and would vary in approach depending on whether you are using an elastic IP for your server or whether it is behind a load balancer,
We are using log4j for the logging functionality. The application is running in a Clustered environment. How can I configure log4j properties such that all the instances log to the same log file?
One solution is to have a directory dedicated for logging. That directory can be on a network share (NFS, etc.) that is mounted to a location that is the same for both processes. This could be as simple as mounting to the identical spot in the file structure or it could be done using environment variables ($LOGDIR) so each host could point to a different location in their local file structure.
The important thing is that the folder be shared so that multiple processes are writing to the same file. The normal shared-resource restrictions apply though; make sure the file isn't locked by one host while preventing the other from writing, etc. Also, use an output pattern that includes hostname/process name/thread id.
Another approach I've used is a database appender that writes to a log table. No share needed but you still need to design the table considering the issues with multi-process logging.