The machines in our cluster will restart due to software updates from time to time, so we wrote scripts to start Spark Workers and Master automatically.
However, if workers cannot see Master for a certain amount of time, they will give up and shut down eventually. So if the Master machine get restarted, the workers will get time out and shut down before they successfully connect to the restarted master.
So, the problem is, how to tell the workers do not give up connecting to the master?
Thank you!
Related
Currently I've set up replica cluster in AWS.
and I have to reboot the replica to update my cluster parameter group(It's like mysql config, but this is for AWS).
I'm not sure what's exactly going on in the replicating process under the hood. I think, If I reboot it, the replication could be stopped and I have to do the replication setting again from the beginning.
I can't find any guide on this online.
Anyone can give some know-how about this?
We have a web application running on local machine. And we would like to replicate the MySQL database to remote (web). I have looked at the tutorials online for that and I have one concern about the MASTER HOST ADDRESS. Since we usually shut down the laptops at the end of the day, the IP address will not be the same everyday. Is there any other way or method that replication can be achieved without specifying the IP from local to remote?
Thank you!
MySQL replication is pretty robust and will handle correctly situations where the master server is offline for prolonged periods of time.
your laptop could establish a VPN tunnel to the replica server or some relay server. private addressing that you'd use on both endpoints of the tunnel would be static. this approach will work fine regardless of type of internet connection used - one day it could be at home, another day at the office, another day using WiFi hotspot at a cafeteria.
Wireguard, OpenVPN would likely be easier to set up and troubleshoot, IPSec is another alternative.
things worth keeping in mind:
have long enough binlog retention period on the database master side - so the past transactions have chance to replicate to the slave. it's set by expire_log_days.
set infinite master-retry-count
I have gone over the pricing and documentation so many times but still do not understand pricing...
I picked a bare minimum server setup (CPU, RAM, etc). I am using this server as a development server (eventually) so it will be actively used about 6-8 hours a day, 5 days per week...when I entered these values in their "cost calculator" the result was a few bucks a month...perfect!
However I have been up and running for less than a week and already the price is $0.65 with a usage of 2,880.00 Minutes?!?!?!
So I am not being billed only for activity but for server uptime, entirely??? So even if the server sits idle, I am getting charged? Is there a way I can disable the instance during non-work hours? Re-enabling it when I arrive in the morning?
EDIT | how to stop compute engine instance without terminating the instance?
This may have answered my questions...
As the other question answered, you are billed by the minute while your server is running, whether or not it is actively using the CPU.
At the moment, there's no way to leave a server shut down and restart it later; you can use a persistent boot disk to store your development state and create/delete an instance (with the same name) each day that you want to use your server.
To use a persistent boot disk like this, you'll want to make sure that the "Delete boot disk when instance is deleted" checkbox is UNCHECKED -- you want your boot disk to stick around after the instance is deleted. The next time you create your instance, select "Use existing disk", and select the same disk.
You'll still pay $0.04/GB/month for the disk storage all the time, but you'll only pay for the instance running when you need it.
You can add a cron job that checks every 10 minutes to see if the load on the machine is less than 0.05 and no one is logged in and then runs "shutdown -p now" to shut down the machine if you're concerned about forgetting to shut down the machine.
When the system runs out of memory, ubuntu 12.04 kills the mysql process:
Out of memory: Kill process 17074 (mysqld) score 146 or sacrifice child
So the process ends up killed.
This happens at peaks of server load and mainly because of apache getting wild and eating the remaining available memory. Possible approaches could be:
Change somewhere somehow the priority of mysql, so it's not killed (probably a bad fix as something else will be killed)
Monitor the status of mysql and restart automatically whenever it's killed (the one I'm thinking about, but don't know how to do it).
How do you see it?
Abrupt termination of a database server is a very serious crash. You need to avoid this in a production system, because it may not restart cleanly.
The database server is a shared resource, and should almost never terminate in an unplanned fashion in production. The only thing that should cause unplanned termination is a catastrophic hardware or power failure. Most properly configured production data base servers have an unplanned termination once every ten years or less frequently. Seriously.
What to do?
Fix your apache configuration. Limit the number of worker threads and processes it can use, so it can't run wild. Learn how to do this. It's vital. See here: http://httpd.apache.org/docs/current/mod/mpm_common.html#maxrequestworkers
Fix the defects in your web app that are causing your apache to run wild.
If you can, move your mysqld server to a different server machine from apache, so the two don't contend for the same hardware resources.
Configure your mysqld to limit the number of connections it will accept from apache worker threads or other clients. Your web app probably handles the situation where a worker thread needs to wait for a connection. See here. http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_max_connections
Are you on an EC2 micro instance? You need to do some serious tuning. See here: http://ubuntuforums.org/showthread.php?t=1979049
You can check mysql status every minute (with cron) and restart if it is crashed:
* * * * * service mysql status | grep running || service mysql restart
I have a live MySQL DB which is too large to regularly copy the live data to the staging server.
Is there a way of getting just the changes from the last week with the intention of running that script weekly? Would every table have to have an updated timestamp field added to it to achieve this?
I don't know how large "too large to regularly copy" is, but I use SQLyog to synchronize databases. It intelligently does insert/update/deletes for only the records that have changed. I recommend it highly.
One way of going about this would be to make the staging server a replication slave of the production server. However, if you don't want the staging machine to be constantly up to date with the production master, you can keep the slave mode turned off.
Then weekly, run a script that starts the slave for a few hours, allowing it to bring itself up to date with the master, and stop the slave again.
START SLAVE;
-- Wait a while
-- Trial and error to determine how long it takes to come into sync
STOP SLAVE;
This will save it in a state consistent with the master for the current week. On the other hand, if you don't really need it as a weekly snapshot you can just leave the slave running all the time so it just stays in sync.