When I am trying to ping with packet size > 25152 I am getting 100% packet loss. can anyone please help me out on why is it happening ???
ping -c 1 -s 25153 time.nist.gov
--- ntp.glb.nist.gov ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
ping -c 1 -s 25152 time.nist.gov
PING ntp.glb.nist.gov (64.236.96.53) 25152(25180) bytes of data.
25160 bytes from dtc-nist01.ntp.aol.com (64.236.96.53): icmp_req=1 ttl=45 time=76.8 ms
--- ntp.glb.nist.gov ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 76.887/76.887/76.887/0.000 ms
As very quick google shows you are exceeding amount of fragments (data) that fits into one packet. In order to support that you would have to change configuration on your machine (and possibly the remote) to utilize it.
No, it’s not because of hardware, it’s because of IP restriction or it could be because the data choking happening in the router that you were pinging. It will consider large packets as data packets and drops it. If you try small packets, it never shows you any drop because the device will prioratized the input as voice or video.
Related
com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet successfully received from the server was 976,464 milliseconds ago. The last packet sent successfully to the server was 974,674 milliseconds ago.
This error occurs in JMeter when I ran the following test plan that would send 15 MB files to AWS RDS.
LoadTestPlan
JDBC Connection Configuration:
Max Wait ms: 0 (indefinite wait)
Max connections: 0 (no limit)
ThreadGroup
No. of threads: 200
Ramp up seconds: 100
Loop Count: Indefinite
Scheduled to run for 3 hours
JDBC Request
LOAD DATA LOCAL INFILE statement
RDS Configuration
Engine 5.7.33
Max connections: 200
Innodb lock wait timeout: 6000
Max allowed packet: 64 MB
There were many solutions for this Communication Links Failure but for me, some requests are successful and for some I get this error. Thus I am starting to think it is the network problem but I am using high speed Ethernet of 74 Mbps speed. Even if it is the network problem there must be some parameter that when adjusted should allow connections from even poor network to be successful.
JMter version: 5.4
With regards to your statement:
Max connections: 0 (no limit)
I don't think it's true, as per BasicDataSource Configuration Parameters I would say it's rather 8 (see maxTotal parameter)
So it looks like you're using 200 concurrent threads and having 8 connections in pool, try increasing these max connections to be equal to the number of JMeter threads. Or if you're not testing the database directly you should rather mimic your application JDBC configuration.
I know that JMeter tries to set maximum pool size equal to initial pool size as it evidenced by this line, however the source code of BasicDataSource suggests setting a negative number for "no limit"
More information: The Real Secret to Building a Database Test Plan With JMeter
I am using tcpdump to collect packet data through wlan0. But I find many packets with length of 0 like following picture shows. Well, length 0 packets... Is tcpdump reliable or I have missed something?
OK, those are TCP packets, and that's the TCP payload length. That can be zero, if a TCP packet is just acknowledging data sent by the other side of the connection, and not sending any data.
For example, the next to the last packet is from 121.186.151.61.dial.xw.sh.dynamic.163data.com.cn, port 80 (that's the ".http" at the end - it's not part of the domain name, it's an indication that it's from port 80) to android-1b46862a4910306b, port 53876, with 16 bytes of data. The last packet is android-1b46862a4910306b sending an ACK to 121.186.151.61.dial.xw.sh.dynamic.163data.com, probably acknowledging the previous 16 bytes.
The other zero-length packets in your sample output are TCP FIN packets, which don't have to have data in them (the host sending it is just saying "I don't have any more data to send you, and never will have any more data to send you on this connection - I'm done"), and TCP RST packets, which are just forcibly shutting down the connection and also don't have to have data in them.
I have a webapp (Tomcat/Hibernate/DBCP 1.4) that runs queries against MySQL, and this works fine for a certain load, say 50 queries a second. When I route the same moderate load through HAProxy (still just using a single database), I get a failure, maybe one for every 500 queries. My app reports:
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 196,898 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago.
at sun.reflect.GeneratedConstructorAccessor210.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3567)
...
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3017)
...
Meanwhile the HAProxy log is showing a lot of entries like:
27] mysql mysql/db03 0/0/34605 2364382 cD 3/3/3/3/0 0/0
Oct 15 15:43:12 localhost haproxy[3141]: 127.0.0.1:35500 [15/Oct/2012:15:42:50.0
The "cD" apparently indicates a state of client timeout. So whereas my webapp is saying that HAProxy is refusing to accepting new connections, HAProxy is saying that my webapp is not accepting data back.
I am not including my HAProxy configuration, because I've tried many different parameter values, with essentially the same result. In particular, I've set maxconn to both high and low values, in both global and server sections, and what always happens in the stats is that the max sessions rises to no more than about 7. My JDBC pool size is also high.
Is it generally ok to use a JDBC pool and a HAProxy pool together? Have people run into this kind of problem before?
I have an idea on how to solve this, which is to send a "validation query" before every query. But there's a certain overhead there, and I'd still like to know why my webapp succeeds when it goes straight to MySQL, but gets dropped connections on going through HAProxy.
How can I debug further and get more information than just "cD"? I tried running HAProxy in debug mode, but it doesn't seem to reveal anything more.
Try this:
tune.bufsize 20480
tune.maxrewrite 2048
See the ha-docs for their meaning. You have to do this with all eyes on it as you're entering the grey zone of potential lethal params. But it's worth a try to see if this works. I just solved a problem that made no sense vs. the documentation with this.
The defaults are 16k vs 1k.
When i was checking for mysql load time on site. i got result showing connections as TIME_WAIT. Even though i close connection on every page. Sometimes the site doesnt load saying too many connections. What could be solution to this problem?
Thanks in Advance for any replies or suggestions
If a client connects to a MySQL-Server, it usually opens a local port, example:
localhost:12345 -> mysqlserver:3306
If the client closes the connection, the client gets a TIME_WAIT. Due to TCP routing, a packet might arrive late on the temporary port. A connection in TIME_WAIT just discards these packets. Without a TIME_WAIT, the local port might be reused for another connection and might receive packets from a former connection.
On an high frequent application on the web which opens a mysql-connection per request, a high amount of TIME_WAIT connections is expectable. There is nothing wrong with it.
Problems can occur, if your local port range is too low, so you cannot open outgoing connections any more. The usual timeout is set to 60 seconds. So a problem can already occur on more than 400 requests per second on low ranges.
Check:
To check the amount of TIME_WAIT, you can use the following command:
$ cat /proc/net/sockstat
sockets: used 341
TCP: inuse 12 orphan 0 tw 33365 alloc 23 mem 16
UDP: inuse 9 mem 2
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
The value after "tw", in this case 33365, shows the amount of TIME_WAIT.
Solutions:
a. TIME_WAIT tuning (Linux based OS examples):
Reduce the timeout for TIME_WAIT:
# small values are ok, if your mysql server is in the same local network
echo 15 > /proc/sys/net/ipv4/tcp_fin_timeout
Increase the port range for local ports:
# check, what you highest listening ports are, before setting this
echo 15000 65000 > /proc/sys/net/ipv4/ip_local_port_range
The settings /proc/sys/net/ipv4/tcp_tw_recycle and /proc/sys/net/ipv4/tcp_tw_reuse might be interesting, too. (But we experienced strange side effects with these settings, so better avoid them. More informations in this answer)
b. Persistent Connections
Some programming languages and libraries support persistent connections. Another solution might be using a locally installed proxy like "ProxySQL". This reduces the amount of new and closed connections.
If you are getting alot of TIME_WAIT connections on the Mysql Server then that means that Mysql server is closing the connection. The most likely case in this instance would be that a host or several hosts got on a block list. You can clear this by running
mysqladmin flush-hosts
to get a list of the number of connections you have per ip run,
netstat -nat | awk {'print $5'} | cut -d ":" -f1 | sort | uniq -c | sort -n
you can also confirm this is happening by going to one of your clients that is having trouble connecting and telnet to port 3306. It will thow a message with something like,
telnet mysqlserver 3306
Trying 192.168.1.102...
Connected to mysqlserver.
Escape character is '^]'.
sHost 'clienthost.local' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'Connection closed by foreign host.
As #Zimbabao suggested in the comment, debug your code for any potential errors that may halt the execution of closing the Mysql connection.
If nothing works, check your my.cnf for a system variable called wait_timeout. If its not present add it to the section [mysqld] and restart your Mysql server.
[mysqld]
wait_timeout = 3600
Its the number of seconds the server waits for activity on a noninteractive connection before closing it. Further information can be found http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_wait_timeout
Tune the figure 3600 (1 hour) to your requirements.
HTH
Do the client's SYN and the servers SYN+ACK get delayed by Nagle? Will the client's ACK of the server's SYN get delayed?
Will connect return after rtt+spt or will it take rtt + spt + 2x Nagle Delay?
Or more generally, how do the Nagle Algorith and Delayed ACK's affect TCP Connection Setup?
No, NAGLE does not effect SYN or SYN+ACK. Neither does delayed ACK (logically enough, you want the SYN+ACK to come back as quick as possible, and there is no congestion state to cause a holdoff yet anyway).