zabbix_sender: send value error: ZBX_TCP_READ() timed out - zabbix

I have some problems sending my test data to Zabbix Server.
My configurations are set up, even have Zabbix agent installed and correctly working (sends data for monitoring CPU, memory,...).
This is the situation: I have a Zabbix installed on a Debian VM, and configured a host with correct IP, port, item(Zabbix trapper).
I want to send a value just for testing from my Windows 10 PC using "zabbix_sender"; later I want to find a way to get data from a .txt file for monitoring.
Used command from my cmd:
zabbix_sender -vv -z XXX.XXX.X.X -p XXXX -s "IT-CONS-PC4" -k trap -o "test"
Error:
zabbix_sender [8688]: DEBUG: send value error: ZBX_TCP_READ() timed out
Did someone else had this issue?

This errors out on the network level.
check that the local firewall on the Zabbix server allows incoming connections on the server port (10051 by default)
check that the VM network connectivity is correct
As a simple test, you can telnet from the box with zabbix_sender to the Zabbix server on port 10051. If that fails, you have a basic network issue.

after many an hour, this is what fixed it.
Active Agent Checks were failing with following error or similar:
active check configuration update from [zabbix.verticalcomputers.com:10051] started to fail (ZBX_TCP_READ() timed out)
For whatever reason, active agents wont be able to connect properly (active checks won't work, only passive), if server is behind a firewall NAT'd, and you don't have the following in your ifcfg-eth0 (or whatever NIC) file. It will work if you bypass the firewall and put a public IP right on the zabbix server.
NM_CONTROLLED=no
BOOTPROTO=static
If you use the CentOS 7 wizard, or nmtui to config your NIC, instead of manually, those lines don't get added.
I noticed this because when running "ip add", I'd get the following:
2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:15:5d:07:82:07 brd ff:ff:ff:ff:ff:ff
inet 10.32.2.25/24 brd 10.32.2.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
Notice the "noprefixroute". That was unsettling, so I dug for a long time online, with no leads. After adding the two lines to the NIC config mentioned above, and restarting the network, now looks like this:
2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:15:5d:07:82:07 brd ff:ff:ff:ff:ff:ff
inet 10.32.2.25/24 brd 10.32.2.255 scope global eth0
valid_lft forever preferred_lft forever

Late but i hope this could help you.
Check the last sections of scripts docs
Be sure that your zabbix server has connection with your zabbix agent (normally cause by firewalls)
1.1 by port 10050 for passive checks
1.2 by port 10051 for active checks
# you can do it with telnet from your zabbix server
> telnet <agent ip> <10050 or 10051>
Trying <agent ip>...
Connected to <agent ip>.
Escape character is '^]'.
You can modify your server/agent config file to increase Timeout directive. By default is 3 and you can set it up to 30 seconds. If you do this, be sure to modify in both server and agent.
2.1 Don't forget restarting the services service zabbix-agent restart and service zabbix-server restart

Related

Docker: Accessing to a mysql container from another container in same host (No route to host error)

I have a virtual machine with this IP: 10.23.23.23
On this VM, Docker is running and 2 containers are created:
Container1 (Apache running) : This container exposes the port 13080 and bin the port 80 of apache inside the container
Container2 (mysql) : This container exposes the port 5555 and bind the port 3306 of mysql inside the container.
In Container1, i am trying to access to container2, but i get the following error: SQLSTATE[HY000] [2002] No route to host
Notes:
The following command on VM host:
ip addr show docker0
returns:
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 02:42:cf:7e:ea:b7 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::41:cfff:fe7e:eab7/64 scope link
valid_lft forever preferred_lft forever
What can i do to be able to join the second container (mysql) from the first one ?
This seems to be IP:PORT exposing issue. If you want to refer containers using host IP make sure mysql is listening on all the IPs i.e 0.0.0.0 in the container and then give it a try using host ports.
Normally these issues occur when services in the container are running at localhost/127.0.0.1.

Can't connect to port 80 on Google Cloud Compute instance despite firewall rule

In summary, although I've set a firewall rule that allows tcp:80, my GCE instance, which is on the "default" network, is not accepting connections to port 80. It appears only port 22 is open on my instance. I can ping it, but can't traceroute to it in under 64 hops.
What follows is my investigation that led me to those conclusions.
gcloud beta compute firewall-rules list
NAME NETWORK DIRECTION PRIORITY ALLOW DENY
default-allow-http default INGRESS 1000 tcp:80
default-allow-https default INGRESS 1000 tcp:443
default-allow-icmp default INGRESS 65534 icmp
default-allow-internal default INGRESS 65534 tcp:0-65535,udp:0-65535,icmp
default-allow-rdp default INGRESS 65534 tcp:3389
default-allow-ssh default INGRESS 65534 tcp:22
temp default INGRESS 1000 tcp:8888
gcloud compute instances list
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
ssrf3 us-west1-c f1-micro true 10.138.0.4 35.197.33.182 RUNNING
gcloud compute instances describe ssrf3
...
name: ssrf3
networkInterfaces:
- accessConfigs:
- kind: compute#accessConfig
name: external-nat
natIP: 35.197.33.182
type: ONE_TO_ONE_NAT
kind: compute#networkInterface
name: nic0
network: https://www.googleapis.com/compute/v1/projects/hack-170416/global/networks/default
networkIP: 10.138.0.4
subnetwork: https://www.googleapis.com/compute/v1/projects/hack-170416/regions/us-west1/subnetworks/default
...
tags:
fingerprint: 6smc4R4d39I=
items:
- http-server
- https-server
I ssh into 35.197.33.182 (which is the ssrf3 instance) and run:
sudo nc -l -vv -p 80
On my local machine, I run:
nc 35.197.33.182 80 -vv
hey
but nothing happens.
So I try to ping the host. That looks healthy:
ping 35.197.33.182
PING 35.197.33.182 (35.197.33.182): 56 data bytes
64 bytes from 35.197.33.182: icmp_seq=0 ttl=57 time=69.172 ms
64 bytes from 35.197.33.182: icmp_seq=1 ttl=57 time=21.509 ms
Traceroute quits after 64 hops, without reaching the 35.197.33.182 destination.
So I check which ports are open with nmap:
nmap 35.197.33.182
Starting Nmap 7.12 ( https://nmap.org ) at 2017-06-18 16:39 PDT
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.06 seconds
nmap 35.197.33.182 -Pn
Starting Nmap 7.12 ( https://nmap.org ) at 2017-06-18 16:39 PDT
Nmap scan report for 182.33.197.35.bc.googleusercontent.com (35.197.33.182)
Host is up (0.022s latency).
Not shown: 999 filtered ports
PORT STATE SERVICE
22/tcp open ssh
Nmap done: 1 IP address (1 host up) scanned in 6.84 seconds
… even when I’m running nc -l -p 80 on 35.197.33.182.
Ensure that VM level firewall is not intervening. For example, Container-Optimized OS is a bit special in comparison to all other default images:
By default, the Container-Optimized OS host firewall allows only outgoing connections, and accepts incoming connections only through the SSH service. To accept incoming connections on a Container-Optimized OS instance, you must open the ports your services are listening on.
https://cloud.google.com/container-optimized-os/docs/how-to/firewall
Checking the two check boxes "Allow HTTP traffic" and "Allow HTTPS traffic" did the trick. This created two Firewall rules, that opened the ports 80 and 443.
Manually adding rules for those port didn't work for some reason, but it worked with checking the boxes.
On a quick glance, your setup seems to be correct.
You have allowed INGRESS tcp:80 for all instances in the default network.
Your VM is on the default network.
Traceroute will not give a good indication when you have VMs running on Cloud providers, because of the use of SDNs, virtual networks and whole bunch of intermediate networking infrastructure unfortunately.
One thing I notice is that your instance has 2 tags http-server and https-server. These could be used by some other firewall rules possibly which is somehow blocking traffic to your VM's tcp:80 port.
There are other variables in your setup and I'm happy to debug if needed further.
Tag based firewall rules
You can try tag based firewall rules which will apply the firewall rule only to instances which have the specified target tag.
Network tags are used by networks to identify which instances are
subject to certain firewall rules and network routes. For example, if
you have several VM instances that are serving a large website, tag
these instances with a shared word or term and then use that tag to
apply a firewall rule that allows HTTP access to those instances. Tags
are also reflected in the metadata server, so you can use them for
applications running on your instances. When you create a firewall
rule, you can provide either sourceRanges or sourceTags but not both.
# Add a new tag based firewall rule to allow ingress tcp:80
gcloud compute firewall-rules create rule-allow-tcp-80 --source-ranges 0.0.0.0/0 --target-tags allow-tcp-80 --allow tcp:80
# Add the allow-tcp-80 target tag to the VM ssrf3
gcloud compute instances add-tags ssrf3 --tags allow-tcp-80
It might take a few seconds to couple of minutes for the changes to take effect.
NOTE: Since you're opening up ports of VM's external IPs to the internet, take care to restrict access accordingly as per the needs of your application running on these ports.
After lots of trail and error, the following worked for me on ubuntu-1404-trusty-v20190514
, with a nodejs app listening on port 8080. Accept port 80 and 8080, and then redirect 80 to 8080.
sudo iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -i eth0 -p tcp --dport 8080 -j ACCEPT
sudo iptables -t nat -A OUTPUT -o lo -p tcp --dport 80 -j REDIRECT --to-port 8080
sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
Incase you are a windows server instance , You could try to turn off the Windows Defender and check if it's blocking the incoming connection.

tcpdump doesn't captures properly on specific port

I'm in a network and i wanna capture ftp packets from another server in the network but i have a problem with tcpdump about this.
I've used this command :
tcpdump -i eth0 dst X.X.X.X -A and port 21
But it doesn't shows anything! ( i tested and sure that ftp port is 21 )
But if i use this on my server it works properly.
tcpdump -i eth0 -A and port 21
I've this problem when i enter " port " in the command. but if i enter a command without specific port it works and captures properly.
What is the problem?
Thanks.
I don't have enough reputation to ask a question, so this is part question and part insight.
Is the IP you're filtering on the client or the server for the FTP connection?
For the first command, try using src x.x.x.x or just host x.x.x.x and port 21.
For the second command, the "and" is not necessary with the -A flag. This should look more like this:
tcpdump -A -i eth0 port 21
tcpdump -Ai eth0 port 21
Another thing I've seen is if there are vlan tags, normal filtering won't work without adding "vlan and " to your filter. For example:
tcpdump -A -i eth0 "vlan and host x.x.x.x and port 21"
Also keep in mind that FTP uses a control and data connection. The control is over port 21, but the data can vary depending on whether you're using active or passive FTP.

Zabbix proxy - Received empty response from Zabbix Agent

I am trying to setup zabbix proxy. My network is as below
Zabbix server IP: 192.168.101.11 (internal network)
Zabbix proxy server: 192.168.102.109 (internal network)
Zabbix agent: 172.1.16.2 (outside network but pingable from 102.109)
I can ping the zabbix agent IP from my proxy machine.
[root#102_109 ~]# ping 172.1.16.2
PING 172.1.16.2 (172.1.16.2) 56(84) bytes of data.
64 bytes from 172.1.16.2: icmp_seq=1 ttl=64 time=215 ms
64 bytes from 172.1.16.2: icmp_seq=2 ttl=64 time=214 ms
64 bytes from 172.1.16.2: icmp_seq=3 ttl=64 time=214 ms
64 bytes from 172.1.16.2: icmp_seq=4 ttl=64 time=214 ms
I can connect to the zabbix proxy from my zabbix server -
zabbix_get -k agent.ping -s 192.168.102.109
1
My zabbix_proxy.conf file (on 102.109) is as below
ProxyMode=0
Server=192.168.101.11
Hostname=CME_Proxy
LogFile=/tmp/zabbix_proxy.log
DBName=zabbix
DBUser=root
DBPassword=password
And on the zabbix agent machine (172.1.16.2) the configuration is as below.
EnableRemoteCommands=1
LogFile=/tmp/zabbix_agentd.log
Server=192.168.101.11,192.168.102.109
ServerActive=192.168.101.11,192.168.102.109
Hostname=172.1.16.2
AllowRoot=1
On my zabbix front end, I have configured the host as monitored by proxy (CME_Proxy) and there is only 1 item (agent.ping).
I am not able to get any data from the zabbix agent. From my proxy machine, when I run the following command, it returns a blank value.
zabbix_get -k agent.ping -s 172.1.16.2
<this is blank response>
Due to this, in the host configuration, zabbix shows error -
"Received empty response from Zabbix Agent at [172.1.16.2]. Assuming
that agent dropped connection because of access permissions."
Can someone please guide me if the way I have configured is correct? If not how to do this correctly. If you need additional data please let me know.
Thank you
Mukul
Figured it out:
In the agent config file, the following parameters
Server=192.168.101.11,192.168.102.109
ServerActive=192.168.101.11,192.168.102.109
should have been
Server=192.168.101.11,172.1.16.1
ServerActive=192.168.101.11,172.1.16.1
> server 172.1.16.2, some changes in zabbix_agentd.conf
you need specified who will have permission to request the data to agent (passive checks).
Server=192.168.102.109 # it will allow connections from proxy ip
# ServerActive=192.168.102.109 # comment ServerActive if you won't use active checks
at the web interface (set monitored by: CME_Proxy, or the same you defined in Hostname at zabbix_proxy.conf on 192.168.102.109)
> check communication: as you did before!
at the proxy terminal (192.168.102.109):
enter code here
zabbix_get -k agent.ping -s 172.1.16.2 # It should return 1.
PS: check Hostname in the zabbix_proxy.conf, it should be CME_Proxy, or the same you defined at web interface.
# FOR step-by-step guide of running latest zabbix version 5.0 follow these links
# https://blog.zabbix.com/zabbix-docker-containers/7150/
# https://techexpert.tips/zabbix/monitoring-docker-using-zabbix/
#it's simple just add all zabbix server IPs in zabbix host agent conf #file like below
Server=192.168.101.11,172.1.16.1
ServerActive=192.168.101.11,172.1.16.1
if you ur using zabbix server-agent model using docker containers then while deploying containers specify zabbix server Host/Container IPs which wants to connect to zabbix agent container
Assuming if you wanna deploy zabbix server and agent in the same server running docker containers just run below docker deploy commands
#Zabbix Server Container
sudo docker run --name zabbix-appliance -p 8080:80 -p 10051:10051 -d -h zabbix-server zabbix/zabbix-appliance
#Zabbix Agent container
sudo docker run --name=dockbix-agent-xxl --privileged -v /:/rootfs -v /var/run:/var/run -p 10050:10050 -e "ZA_Server=192.168.0.3,172.17.0.1" -e "ZA_ServerActive=192.168.0.3,172.17.0.1" -d monitoringartist/dockbix-agent-xxl-limited:latest
#Default username and password of zabbix server
#username: Admin password: zabbix
# For monitoring docker containers resources import a template from this cloned repository https://github.com/monitoringartist/zabbix-docker-monitoring

Zabbix JMX Tomcat monitoring

I have been trying to setup Zabbix to monitor my 2 tomcat servers on 2 different Amazon EC2 machines, but in vain.
The Z on the host is green, however te JMX is red with these errors
- ZBX_TCP_READ() failed: [4] Interrupted system call
- Someother error [111] connection refused
and many such errors, one after another, in the sense I resolve an error to see one more new error popping up.
These are some assumptions
All the machines run Ubuntu 12.10 and later
Server's IP address: 66.55.12.120 (Runs Zabbix server v2.2.4 (revision 46772) (23 June 2014) )
Agent's IP address: 87.52.45.198 ( Runs Zabbix agent v2.2.2 (revision 42525) (12 February 2014) )
My local machine's IP address: 76.89.54.111
Here is what I've done so far.
On Server Side:
1) Installed Zabbix_server using sudo apt-get install zabbix-server-mysql.
2) The GUI, mysql database all have been installed and configured.
3) The following are the only 3 changes that I've made in the file /etc/zabbix/zabbix_server.conf
...
JavaGateway=localhost
JavaGatewayPort=10052
StartJavaPollers=5
...
4) The Zabbix Java gateway was installed using sudo apt-get install zabbix-java-gateway.
5) The following are the only 3 changes that I've made in the file
/etc/zabbix/zabbix_java_gateway.conf
...
LISTEN_IP="127.0.0.1"
LISTEN_PORT=10052
START_POLLERS=5
...
On Client Side:
1) Installed Zabbix Client using
sudo apt-get install zabbix-agent
2) The following are the only 3 changes that I've made in the file
/etc/zabbix/zabbix_agentd.conf
...
Server=66.55.12.120
StartAgents=5
ServerActive=66.55.12.120:10051
Hostname=Security-test-JMX-EC2
... <br />
3) The Hostname is the same as the one that is mentioned while creating the Host on the GUI.
I believe that there are some issues with the IP and ports. So, here are the outbound rules for both the machines as obtained from Amazon EC2 Security Groups for the machines
OUTBOUND RULES for SERVER SECURITY GROUP:
Type Protocol Port Source Reasoning
Custom- TCP 8080 0.0.0.0/0
TCP Rule
All ICMP All N/A 0.0.0.0/0
Custom- TCP 10052 27.52.52.128/32 For access from Agent
TCP Rule
Custom- TCP 8081 76.84.120.130/32 To access Zabbix GUI from-
TCP Rule -my local machine's web browser
Custom- TCP 10051 27.52.52.128/32 As the agent responds to-
TCP Rule -the server on Port 10051TCP Rule-
-Must allow inbound communications-
- from the agent.
Custom- TCP 11000 27.52.52.128/32 The agent's JMX reporting-
TCP Rule -happens on port 11000(not on 12345).
OUTBOUND RULES for CLIENT SECURITY GROUP:
Type Protocol Port Source
HTTPS TCP 443 0.0.0.0/0
Custom- TCP 10050 66.55.12.120/32
TCP Rule
Custom- TCP 10052 66.55.12.120/32
TCP Rule
Custom- TCP 11000 66.55.12.120/32
TCP Rule
HTTP TCP 80 76.89.54.111/32
Custom- TCP 8080 76.89.54.111/32
TCP Rule
Custom- TCP 8443 76.89.54.111/32
TCP Rule
What am I missing? Please guide me.
Any help is appreciated.
Thanks
Goutham
If you can, then run VisualVm (probably using a tunneled X session) on the zabbix host, and see if you can connect to the target JVM with that. If you can't connect from that, you won't be able to connect from Zabbix.
Try with the following CATALINA_OPTS, replacing with the IP on the target that you want JMX to listen on:
export CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=falseom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=<LOCAL_IP>"
This will disable all JMX security so be aware!
Once you hopefully get it to connect, the "Tomcat JMX" items in Zabbix are also all incorrect! e.g.
Incorrect Zabbix default:
jmx["Catalina:type=GlobalRequestProcessor,name=http-8080",bytesReceived]
Correct entry:
jmx["Catalina:type=ThreadPool,name=\"http-bio-8080\"", bytesReceived]
Note the escaped quotes and incorrect thread name. Add the Mbeans plugin to VisualVM, and use that to browse the MBeans on the target VM, and check the Zabbix names.
It does work eventually, but is a real pain to setup. Zabbix is however one of the few open source monitoring tools that supports JMX at all!
By default, JMX does not work very well with firewalls. You might find related bug reports on Zabbix tracker useful: ZBX-5326 and ZBX-6815. The first one contains a workaround for Tomcat which might work for you.
#gvatreya wrote:
Server: (Runs Zabbix server)
Agent: (Runs Zabbix agent)
It looks like you have to start Zabbix Java gateway as well on host where it is installed (it is a daemon/service).
I configured as follows:
Server: (Runs Zabbix server, Zabbix Java gateway)
Agent: (Runs Zabbix agent)
I think it is possible to install it on a dedicated host.
Have you tried adding -Djava.net.preferIPv4Stack=true to the VM options?
to make it work add next java_opts to your tomcat startup script
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=2345
-Dcom.sun.management.jmxremote.rmi.port=12345
-Djava.rmi.server.hostname=<tomcat_hostname>