Execute Remote Command just Once a Day for zabbix - zabbix

I want to execute action once in a day. Like whenever day's 1st alert is present in the Zabbix that time Action will restart the service/process. After that suppose same alert is come in Zabbix that time no Action will performed that time.
I am using Zabbix version 6.2.4 and on client side it is zabbix-agent2
I have created the Action on the trigger.
item --> get information process is running or not.
system.run["ps -aux | grep -i 'proces' | grep -v 'grep' | wc -l"]
trigger --> When process is down that time trigger id generated.
last(/Monitor process/system.run["ps -aux | grep -i 'proces' | grep -v 'grep' | wc -l"])<1
0 indicate --> Through the alert in Zabbix
1 indicate --> do not do anything.
Action --> When trigger is present in the Zabbix Action execute the service start script.
Up to this point all function is working fine.
But I want this action only once in a day. Like whenever day's 1st alert is present in the Zabbix that time Action will restart the service/process.

Problem history is not available as an Action condition, so you don't know in Zabbix if the Event already triggered, and when. You can however manage it inside the restart script.
Following script restarts the service when it's been more than 24 hours since last restart.
#!/bin/bash
SEMAPHORE=/tmp/zabbix.service1.lastrestart
first_time=0
day=$(expr 24 \* 60 \* 60)
delta=$(expr $(date +%s) - $(stat -c %Y $SEMAPHORE 2>/dev/null) 2>/dev/null)
if [ -z "$delta" ]; then
# file doesn't exists
first_time=1
elif [ $delta -gt $day ] ; then
# more than 1d since last restart
first_time=1
fi
if [ $first_time -eq 1 ]; then
systemctl restart service1
touch $SEMAPHORE
fi
Please note that system.run[] is deprecated, you should use it for tests only. In this case the appropriate key may be proc.num[], or even systemd.unit.info[] since you are using Agent2 newer than 4.4 (see https://www.zabbix.com/documentation/6.2/en/manual/discovery/low_level_discovery/examples/systemd?hl=systemd).

Related

Run a bash script one and stop

I have this simple script that checks if mysql on remote servers (db-test-1 and db-test-2) is in SST mode and sends a message to a slack channel. The script is running on a third server dedicated for running cron jobs. Here is the code below:
#!/bin/bash
time=$(date);
array=( db-test-1 db-test-2 )
for i in "${array[#]}"
do
S=$(ssh $i ps -ef |grep mysql | grep wsrep_sst_xtrabackup-v2);
if [[ "$S" != "" ]]; then
curl -X POST --data-urlencode "payload={\"channel\": \"#db-share-test\", \"username\": \"wsrep_local_state_comment\", \"text\": \"*$i*: ${time}\n>State transfer in progress, setting sleep higher mysqld\", \"icon_emoji\": \":scorpion:\"}" https://hooks.slack.com/services/G824ZJS9N/B6QS5JEKP/ZjV1hmM1k4dZGsf9HDC1o1jd
exit 0
else
curl -X POST --data-urlencode "payload={\"channel\": \"#db-share-test\", \"username\": \"wsrep_local_state_comment\", \"text\": \"*$i*: ${time}\n>State transfer is complete. Server is Synced now.\", \"icon_emoji\": \":scorpion:\"}" https://hooks.slack.com/services/G824ZJS9N/B6QS5JEKP/ZjV1hmM1k4dZGsf9HDC1o1jd
exit 2
fi
done
The two servers, db-test1 and db-test-2 are part of a PXC cluster. So when i start db-test-1 in SST to join the cluster, i get the following in my slack channel as expected:
*db-test-1*: Sun Aug 27 15:12:44 CST 2017
>State transfer in progress, setting sleep higher mysqld
[3:12]
*db-test-1*: Sun Aug 27 15:12:49 CST 2017
State transfer in progress, setting sleep higher mysqld
[3:12]
*db-test-1*: Sun Aug 27 15:12:51 CST 2017
State transfer in progress, setting sleep higher mysqld
[3:12]
*db-test-1*: Sun Aug 27 15:12:54 CST 2017
State transfer in progress, setting sleep higher mysqld
So the results are being displayed approximately every 3 seconds. However, the cron job executing this script is scheduled to run every minute, hence not sure why it is sending results every 3 seconds or so as shown above.
How can i ensure that the results are displayed every 1 minute to avoid my channel being thrown the same message every 3 seconds? Also, how can i make sure that when the SST is finished, a single message to slack to indicate that the state transfer is finished instead of sending this message none-stop every time the two db servers are not in SST mode?
Besides checking that the cron is properly set, probably something like:
#every_minute /path/to/script
or
*/1 * * * * /path/to/script
It could be also good to ensure that only one occurrence of the program is running, try adding this to your script used within the cron job
*/1 * * * * pgrep script > /dev/null || /path/to/script
or by using something like:
#!/bin/sh
if ps -ef | grep -v grep | grep your_script_name ; then
exit 0
fi
# your code goes below
# ...

userparameters and ZBX_NOTSUPPORTED

I want to ping an external ip from all of my servers that run zabbix agent.
I searched and find some articles about zabbix user parameters.
In /etc/zabbix/zabbix_agentd.conf.d/ I created a file named userparameter_ping.conf with following content:
UserParameter=checkip[*],ping -c4 8.8.8.8 && echo 0 || echo 1
I created an item named checkip in zabbix server with a graph but got no data. After some another digging I found zabbix_get and tested my userparameter but I got the error : ZBX_NOTSUPPORTED
# zabbix_get -s 172.20.4.43 -p 10050 -k checkip
my zabbix version :
Zabbix Agent (daemon) v2.4.5 (revision 53282) (21 April 2015)
Does anybody know what I can do to address this?
After some change and talks with folks in mailing list finally it worked but how :
first i created a file in :
/etc/zabbix/zabbix_agentd.conf.d/
and add this line :
UserParameter=checkip[*],ping -W1 -c2 $1 >/dev/null 2>&1 && echo 0 || echo 1
and run this command :
./sbin/zabbix_agentd -t checkip["8.8.8.8"]
checkip[8.8.8.8] [t|0]
so everything done but Timeout option is very important for us :
add time out in /etc/zabbix/zabbix_agentd.conf
Timeout=30
Timeout default is 3s so if we run
time ping -W1 -c2 8.8.8.8
see maybe it takes more than 3s so you got error :
ZBX_NOTSUPPORTED
It can be anything. For example timeout - default timeout is 3 sec and ping -c4 requires at least 3 seconds, permission/path to ping, not restarted agent, ...
Increase debug level, restart agent and check zabbix logs. Also you can test zabbix_agentd directly:
zabbix_agentd -t checkip[]
[m|ZBX_NOTSUPPORTED] [Timeout while executing a shell script.] => Timeout problem. Edit zabbix_agentd.conf and increase Timeout settings. Default 3 seconds are not the best for your ping, which needs 3+ seconds.
If you need more than 30s for the execution, you can use the nohup (command..) & combo to curb the timeout restriction.
That way, if you generate some file with the results, in the next pass, you can read the file and get back the results without any need to wait at all.
For those who may be experiencing other issues with the same error message.
It is important to run zabbix_agentd with the -c parameter:
./sbin/zabbix_agentd -c zabbix_agentd.conf --test checkip["8.8.8.8"]
Otherwise zabbix might not pick up on the command and will thus yield ZBX_NOTSUPPORTED.
It also helps to isolate the command into a script file, as Zabbix will butcher in-line commands in UserParameter= much more than you'd expect.
I defined two user parameters like this for sync checking between to samba DCs.
/etc/zabbix/zabbix_agentd.d/userparameter_samba.conf:
UserParameter=syncma, sudo samba-tool drs replicate smb1 smb2 cn=schema,cn=configuration,dc=domain,dc=com
UserParameter=syncam, sudo samba-tool drs replicate smb2 smb1 cn=schema,cn=configuration,dc=domain,dc=com
and also provided sudoer access for Zabbix user to execute the command. /etc/sudoers.d/zabbix:
Defaults:zabbix !syslog
Defaults:zabbix !requiretty
zabbix ALL=(ALL) NOPASSWD: /usr/bin/samba-tool
zabbix ALL=(ALL) NOPASSWD: /usr/bin/systemctl
And "EnableRemoteCommands" is enabled on my zabbix_aganetd.conf, sometimes when I run
zabbix_get -s CLIENT_IP -p10050 -k syncma or
zabbix_get -s CLIENT_IP -p10050 -k syncam
I get the error ZBX_NOTSUPPORTED: Timeout while executing a shell script.
but after executing /sbin/zabbix_agentd -t syncam on the client, Zabbix server just responses normally.
Replicate from smb2 to smb1 was successful.
and when it has a problem I get below error on my zabbix.log
failed to kill [ sudo samba-tool drs replicate smb1 smb2 cn=schema,cn=configuration,dc=domain,dc=com]: [1] Operation not permitted
It seems like it is a permission error! but It just resolved after executing /sbin/zabbix_agentd -t syncam but I am not sure the error is gone permanently or will happen at the next Zabbix item check interval.

Shell script to check if mysql is up or down

I want a bash shell script that i can run using a cron job to check if mysql on a remote server is running. If it is, then do nothing, other start the server.
The cronjob will be checking the remote server for a live (or not) mysql every minute. I can write the cron job myself, but i need help with the shell script that checks if a remote mysql is up or down. The response after a check if up or down is not important. But the check is important.
You can use below script
#!/bin/bash
USER=root
PASS=root123
mysqladmin -h remote_server_ip -u$USER -p$PASS processlist ###user should have mysql permission on remote server. Ideally you should use different user than root.
if [ $? -eq 0 ]
then
echo "do nothing"
else
ssh remote_server_ip ###remote server linux root server password should be shared with this server.
service mysqld start
fi
The script in the selected answer works great, but requires that you have the MySQL client installed on the local host. I needed something similar for a Docker container and didn't want to install the MySQL client. This is what I came up with:
# check for a connection to the database server
#
check=$(wget -O - -T 2 "http://$MYSQL_HOST:$MYSQL_PORT" 2>&1 | grep -o mariadb)
while [ -z "$check" ]; do
# wait a moment
#
sleep 5s
# check again
#
check=$(wget -O - -T 2 "http://$MYSQL_HOST:$MYSQL_PORT" 2>&1 | grep -o mariadb)
done
This is a little different, in that it will loop until a database connection can be made. I am also using MariaDB instead of the stock MySQL database. You can change this by changing the grep -o mariadb to something else - I'm not sure what MySQL returns on a successful connection, so you'll have to play with it a bit.

Issues with MySQL restart on running through a crontab scheduler

I have written a shell script which starts MySQL when its killed/terminated. I am running this shell script using a crontab.
My cron looks for the script file named mysql.sh under /root/mysql.sh
sh /root/mysql.sh
mysql.sh:
cd /root/validate-mysql-status
sh /root/validate-mysql-status/validate-mysql-status.sh
validate-mysql-status.sh:
# mysql root/admin username
MUSER="xxxx"
# mysql admin/root password
MPASS="xxxxxx"
# mysql server hostname
MHOST="localhost"
MSTART="/etc/init.d/mysql start"
# path mysqladmin
MADMIN="$(which mysqladmin)"
# see if MySQL server is alive or not
# 2&1 could be better but i would like to keep it simple
$MADMIN -h $MHOST -u $MUSER -p${MPASS} ping 2>/dev/null 1>/dev/null
if [ $? -ne 0 ]; then
# MySQL's status log file
MYSQL_STATUS_LOG=/root/validate-mysql-status/mysql-status.log
# If log file not exist, create a new file
if [ ! -f $MYSQL_STATUS_LOG ]; then
cat "Creating MySQL status log file.." > $MYSQL_STATUS_LOG
now="$(date)"
echo [$now] error : MySQL not running >> $MYSQL_STATUS_LOG
else
now="$(date)"
echo [$now] error : MySQL not running >> $MYSQL_STATUS_LOG
fi
# Restarting MySQL
/etc/init.d/mysql start
now1="$(date)"
echo [$now1] info : MySQL started >> $MYSQL_STATUS_LOG
cat $MYSQL_STATUS_LOG
fi
When I run the above mysql shell script manually using webmin's crontab, MySQL started successfully (when its killed).
However, when I schedule it using a cron job, MySQL doesn't starts. The logs are printed properly (it means my cron runs the scheduled script successfully, however MySQL is not restarting).
crontab -l displays:
* * * * * sh /root/mysql.sh
I found from URL's that we should give absolute path to restart MySQL through schedulers like cron. However, it haven't worked for me.
Can anyone please help me!
Thank You.
First, crontab normaly looks like this:
* * * * * /root/mysql.sh
So remove the surplus sh and put it at the beginning of the script - #!/bin/bash I suppose (why are you referring to sh instead of bash?) and don't forget to have an execute permission on the file (chmod +x /root/mysql.sh)
Second, running scripts within crontab is tricky, because the environment is different! You have to set it manually. We start with PATH: go to console and do echo $PATH, and then copy-paste the result into export PATH=<your path> to your cron script:
mysql.sh:
#!/bin/bash
export PATH=.:/bin:/usr/local/bin:/usr/bin:/opt/bin:/usr/games:./:/sbin:/usr/sbin:/usr/local/sbin
{
cd /root/validate-mysql-status
/root/validate-mysql-status/validate-mysql-status.sh
} >> OUT 2>> ERR
Note that I also redirected all the output to files so that you don't receive emails from cron.
Problem is how to know which other variables (besides PATH) matter. Try to go through set | less and try to figure out which variables might be important to set in the cron script too. If there are any MYSQL related variables, you must set them! You may also examine the cron script environment by putting set > cron.env to the cron script and then diff-ing it against console environment to look for significant differences.

Root access required for CUDA?

I am using GeForce 8400M GS on Ubuntu 10.04 and I am learning CUDA programming. I am writing and running few basic programs. I was using cudaMalloc, and it kept giving me an error until I ran the code as root. However, I had to run the code as root only once. After that, even if I run the code as normal user, I do not get an error on malloc. What's going on?
This is probably due to your GPU not being properly initialized at boot. I've come across this problem when using Ubuntu Server and other installations where an X server isn't being started automatically. Try the following to fix it:
Create a directory for a script to initialize your GPUs. I usually use /root/bin. In this directory, create a file called cudainit.sh with the following code in it (this script came from the Nvidia forums).
#!/bin/bash
/sbin/modprobe nvidia
if [ "$?" -eq 0 ]; then
# Count the number of NVIDIA controllers found.
N3D=`/usr/bin/lspci | grep -i NVIDIA | grep "3D controller" | wc -l`
NVGA=`/usr/bin/lspci | grep -i NVIDIA | grep "VGA compatible controller" | wc -l`
N=`expr $N3D + $NVGA - 1`
for i in `seq 0 $N`; do
mknod -m 666 /dev/nvidia$i c 195 $i;
done
mknod -m 666 /dev/nvidiactl c 195 255
else
exit 1
fi
Now we need to make this script run automatically at boot. Edit /etc/rc.local to look like the following.
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
#
# Init CUDA for all users
#
/root/bin/cudainit.sh
exit 0
Reboot your computer and try to run your CUDA program as a regular user. If I'm right about what the problem is, then it should be fixed.
To work with Ubuntu 14.04 I followed https://devtalk.nvidia.com/default/topic/699610/linux/334-21-driver-returns-999-on-cuinit-cuda-/ to add nvidia-uvm to etc/modules, and to add a line to a custom udev rule. Create /etc/udev/rules.d/70-nvidia-uvm.rules with this line:
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/bin/mknod -m 666 /dev/nvidia-uvm c $(grep nvidia-uvm /proc/devices | cut -d \ -f 1) 0;'"
I don't understand why sudo modprobe nvidia-uvm works to create a proper /dev/nvidia-uvm (as does sudo cuda_program) but the /etc/modules listing requires the udev rule.