what metric to use for failed slave AWS server - mysql

Question

I have a master server running on a server in an independent data center, and a slave in AWS.
The replication failed with this error: "The incident LOST_EVENTS occured on the master. Message: error writing to the binary log".
Last time it went offline, it jumped from 4k bytes/second write throughput to 40k, and steadily increased to 252 k over a couple of weeks.
1) I'm wondering why write throughput would increase steadily after the failure?
2) I'm wondering what metric can be used within cloudwatch to create an SNS email to me when it does fail? Right now, I'm thinking the best thing to do is to run a simple bash script on the master, that compares Master_Log_File to Relay_Master_Log_File on 'show slave status;' and to forgo CloudWatch altogether.
edit update script:
Here's my script that I run every 10 minutes to check on the slave state (until I find an alernative metric in CloudWatch):
#!/bin/bash
a=$(mysql --host=*amazonaws.com --port=3306 -u whatever -ppass -N -B -e "show slave status;")
e=$(echo "$a" | awk -F\\t '{print $12}') #Slave_SQL_Running
d=$(echo "$a" | awk -F\\t '{print $26}') #Seconds_Behind_Master
if [ "$e" != 'Yes' ]; then
echo -e "slave mysql server down \n slave SQL running: $e \n seconds behind master: $d" | mail -s 'slave mysql server down' admin#email.com
fi

Answer 1

I didn't find a good metric from CloudWatch, so I made this script, which checks the slave status every 10 minutes through cron - it sends an email if it finds Slave_SQL_Running or Slave_IO_Running != 'Yes':
#!/bin/bash
a=$(mysql --host=host --port=3306 -u master -ppword -N -B -e "show slave status;")
b=$(echo "$a" | awk -F\\t '{print $6}') #Master_Log_File
c=$(echo "$a" | awk -F\\t '{print $10}') #Relay_Master_Log_File
e=$(echo "$a" | awk -F\\t '{print $12}') #Slave_SQL_Running
d=$(echo "$a" | awk -F\\t '{print $26}') #Seconds_Behind_Master
f=$(echo "$a" | awk -F\\t '{print $12}') #Slave_IO_Running
if [ "$e" != 'Yes' ] || [ "$f" != 'Yes' ]; then
echo -e "server id - slave mysql server down \n master log file: $b \n relay master log file: $c \n seconds behind master: $d \n Slave IO Running: $f \n Slave SQL Running: $e " | mail -s 'slave mysql server down' email#email.com
fi