We currently have a Zabbix app running in CentOS Linux that we use to log our network rtt and packet loss. We came in a internal discussion of what kind of ping we should use
Ping Flood:
ping -f -c 500 -s 1500 -I {$HOSTVLAN2} {$DEST1IPSEC} | grep packet | awk -F" " '{print $6}' | sed -e s'/%//' -e '/^\s*$/d'
'Default' ping:
ping -c 100 -i 0.2 -s 1500 -I {$HOSTVLAN2} {$DEST1IPSEC} | grep packet | awk -F" " '{print $6}' | sed -e s'/%//' -e '/^\s*$/d'
Thats a screen we made to compare results of packet loss:
So we would like an external view of this case. What you guys think about?
We came in several topics regarding network load, DDoS, realiable values of pkt loss etc..
thanks in advance
Related
I am trying to run the following command from Windows machine in the openshift docker container running Linux
oc exec openjdk-app-1-l9nrx -i -t --server https://xxx.cloud.ibm.com:30450 \
--token <token> -n dev-hg jcmd \
$(ps -ef | grep java | grep -v grep | awk '{print $2}') GC.heap_dump \
/tmp/heap1.hprof
It is trying to evaluate jcmd $(ps -ef | grep java | grep -v grep | awk '{print $2}') GC.heap_dump /tmp/heap1.hprof on local windows machine and I do not have linux commands. Also, I need the process ID of the application running in container and not my local.
Any quick help is appreciated.
Try this:
oc exec -it openjdk-app-1-l9nrx --server https://xxx.cloud.ibm.com:30450 \
--token <dont-share-your-token> -n dev-hg -- /bin/sh -c \
"jcmd $(ps -ef | grep java | grep -v grep | awk '{print \$2}')"
Or even:
oc exec -it openjdk-app-1-l9nrx --server https://xxx.cloud.ibm.com:30450 \
--token <dont-share-your-token> -n dev-hg -- /bin/sh -c \
"jcmd $(ps -ef | awk '/java/{print \$2}')"
The problem is that the $( ) piece is being interpreted locally. Surrounding it in double quotes won't help, as that kind of syntax is interpreted inside double quotes.
You have to replace your double quotes by single quotes (so $( ) is not interpreted), and then compensate for the awk single quotes:
oc exec openjdk-app-1-l9nrx -i -t --server https://xxx.cloud.ibm.com:30450 --token TOKEN -n dev-hg 'jcmd $(ps -ef | grep java | grep -v grep | awk '\''{print $2}'\'') GC.heap_dump /tmp/heap1.hprof'
Please add the tags unix and shell to your question, as this is more of a UNIX question than an Openshift one.
I have a simple pcap with some web traffic and am using tshark to obtain some header information from it:
I use the following command:
tshark -r ./capture-1-5 -Y "http2" -o tls.keylog_file:ssl-key.log \
-T fields -e frame.number -e _ws.col.Time -e ip.src -e tcp.srcport \
-e ip.dst -e tcp.dstport -e _ws.col.Protocol -e frame.len \
-e _ws.col.Info -E header=y -E separator="," -E quote=d \
-E occurrence=f > desegmented.csv
I realized that in this case all fragments are reassembled resulting in huge packets. However, I do not want reassembled packets. So, I add an extra option to tshark:
tshark -r ./capture-1-5 -Y "http2" -o tls.keylog_file:ssl-key.log \
-T fields -e frame.number -e _ws.col.Time -e ip.src -e tcp.srcport \
-e ip.dst -e tcp.dstport -e _ws.col.Protocol -e frame.len \
-e _ws.col.Info -E header=y -E separator="," -E quote=d \
-E occurrence=f -o tcp.desegment_tcp_streams:FALSE > segmented.csv
My intuition is that the resultant disassembled.csv file should be greater in size and should contain more rows given that the "packets above the MTU" will be shown as more than one packet.
However, I observe the opposite. The resultant file without assembly is smaller and has almost halved the number of rows.
-rw-r--r-- 1 root root 210K May 18 18:21 desegmented.csv
-rw-r--r-- 1 root root 97K May 18 18:21 segmented.csv
# cat desegmented.csv |wc -l
2635
# cat segmented.csv |wc -l
1233
Is this a normal behavior? I don't see (manually) where the packets start to disappear (and why) or see any pattern because of the two-way communication (missing packets here and there).
I assume that maybe, in the disassebmled.csv case, every packet or even the whole packet stream that resulted in at least one packet above the MTU is completely dropped.
I tried to also apply ip.defragment:FALSE but still the same results.
Thanks
For reproducing, the files can be downloaded from here
Thanks, #JimD., I have already come to a similar conclusion!
Packet capture itself has to be segmented to do this precisely.
So, tried to go one layer below, and make the packet capture itself to be segmented via
ethtool -K eth0 gso off tso off gro off sg off tx off rx off
(just to make sure).
The problem is that packet capturing is done in a docker container, so at multiple places, I have to issue this command to be fully working.
These places include the docker0 bridge, eth0 inside the container and the corresponding vethXXXXXX on the host, from which the second requires privileged containers that pose further issues :)
I have a master server running on a server in an independent data center, and a slave in AWS.
The replication failed with this error: "The incident LOST_EVENTS occured on the master. Message: error writing to the binary log".
Last time it went offline, it jumped from 4k bytes/second write throughput to 40k, and steadily increased to 252 k over a couple of weeks.
1) I'm wondering why write throughput would increase steadily after the failure?
2) I'm wondering what metric can be used within cloudwatch to create an SNS email to me when it does fail? Right now, I'm thinking the best thing to do is to run a simple bash script on the master, that compares Master_Log_File to Relay_Master_Log_File on 'show slave status;' and to forgo CloudWatch altogether.
edit update script:
Here's my script that I run every 10 minutes to check on the slave state (until I find an alernative metric in CloudWatch):
#!/bin/bash
a=$(mysql --host=*amazonaws.com --port=3306 -u whatever -ppass -N -B -e "show slave status;")
e=$(echo "$a" | awk -F\\t '{print $12}') #Slave_SQL_Running
d=$(echo "$a" | awk -F\\t '{print $26}') #Seconds_Behind_Master
if [ "$e" != 'Yes' ]; then
echo -e "slave mysql server down \n slave SQL running: $e \n seconds behind master: $d" | mail -s 'slave mysql server down' admin#email.com
fi
I didn't find a good metric from CloudWatch, so I made this script, which checks the slave status every 10 minutes through cron - it sends an email if it finds Slave_SQL_Running or Slave_IO_Running != 'Yes':
#!/bin/bash
a=$(mysql --host=host --port=3306 -u master -ppword -N -B -e "show slave status;")
b=$(echo "$a" | awk -F\\t '{print $6}') #Master_Log_File
c=$(echo "$a" | awk -F\\t '{print $10}') #Relay_Master_Log_File
e=$(echo "$a" | awk -F\\t '{print $12}') #Slave_SQL_Running
d=$(echo "$a" | awk -F\\t '{print $26}') #Seconds_Behind_Master
f=$(echo "$a" | awk -F\\t '{print $12}') #Slave_IO_Running
if [ "$e" != 'Yes' ] || [ "$f" != 'Yes' ]; then
echo -e "server id - slave mysql server down \n master log file: $b \n relay master log file: $c \n seconds behind master: $d \n Slave IO Running: $f \n Slave SQL Running: $e " | mail -s 'slave mysql server down' email#email.com
fi
I have two mysql instance running with --open-files-limit=65536. But it got ~193644 open file descriptor with lsof command?
$ lsof -n | grep mysql | wc -l
196410
$ lsof -n | grep mysql | grep ".MYI" | wc -l
83240
$ lsof -n | grep mysql | grep ".MYD" | wc -l
74053
$ sysctl fs.file-max
fs.file-max = 790612
$ lsof -n | wc -l
224647
Why there are so many open file descriptor? what could be the root cause of it? How to debug more?
Problem is with lsof version. I had lsof-4.87 on centos7 which is showing thread information and so it is duplicating open connections per thread. I changed lsof-4.82 & number got reduced
Because we develop using remote Mysql server , so cannot check query sql easily, if use local server you can tail - f general_log_file to see which sql are executed when call some http interface. So I installed a wireshark to capture these query sql send from local. At first I use local mysql to verify it.
The capture filter is
then I executed two query sql in mysql terminal
select version();
select now();
but very disappointing I cannot find these two sql packets in wireshark
I only found these four packets.
But from a post I knew
To filter out the mysql packets you just use the filter ‘mysql‘ or ‘mysql.query != “”‘ when you only want packets that request a query. After that you can add a custom column with the field name ‘mysql.query’ to have a list of queries that where executed.
and the effect is like this
It's convenient to capture only query sql and very clearly displayed these query sql. So how could I use wireshark to implement this?
hi #Jeff S.
I tried your command, please see below
#terminal 1
tshark -i lo0 -Y "mysql.command==3"
Capturing on 'Loopback'
# terminal 2
mysql -h127.0.0.1 -u root -p
select version();
#result: nothing output in terminal 1
and tshark -i lo0 -Y "mysql.command==3" -T fields -e mysql.query is same with tshark -i lo -Y "mysql.command==3" also nothing output. But if I only use tshark -i lo0, it has output
Capturing on 'Loopback'
1 0.000000 127.0.0.1 -> 127.0.0.1 TCP 68 57881 → 3306 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=32 TSval=1064967501 TSecr=0 SACK_PERM=1
2 0.000062 127.0.0.1 -> 127.0.0.1 TCP 68 3306 → 57881 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=16344 WS=32 TSval=1064967501 TSecr=1064967501 SACK_PERM=1
3 0.000072 127.0.0.1 -> 127.0.0.1 TCP 56 57881 → 3306 [ACK] Seq=1 Ack=1 Win=408288 Len=0 TSval=1064967501 TSecr=1064967501
4 0.000080 127.0.0.1 -> 127.0.0.1 TCP 56 [TCP Window Update] 3306 → 57881 [ACK] Seq=1 Ack=1 Win=408288 Len=0 TSval=1064967501 TSecr=1064967501
...
You can use tshark and save to a pcap or just export the fields you're interested in.
To save to a pcap (if you want to use wireshark to view later):
tshark -i lo -Y "mysql.command==3" -w outputfile.pcap
tshark -i lo -R "mysql.command==3" -w outputfile.pcap
-R is deprecated for single pass filters, but it will depend on your version
-i is interface so replace that with whatever interface you are using (e.g -i eth0)
To save to a text file:
tshark -i lo -Y "mysql.command==3" -T fields -e mysql.query > output.txt
You can also use BPF filters with tcpdump (and wireshark pre cap filters). They are more complex, but less taxing on your system if you're capturing a lot of traffic.
sudo tcpdump -i lo "dst port 3306 and tcp[(((tcp[12:1]&0xf0)>>2)+4):1]=0x03" -w outputfile.pcap
NOTE:
*This looks for 03 (similar mysql.command==3) within the TCP payload.
**Since this is a pretty loose filter, I also added 3306 to restrict to only traffic destined for that port.
***The filter is based on your screenshot. I cannot validate it right now so let me know if it doesn't work.
Example Output:
Useful answers here:
https://serverfault.com/questions/358978/how-to-capture-the-queries-run-on-mysql-server
In particular: SoMoSparky's answer of:
tshark -T fields -R mysql.query -e mysql.query
and user1038090's answer of:
tcpdump -i any -s 0 -l -vvv -w - dst port 3306 | strings | perl -e '
while(<>) { chomp; next if /^[^ ]+[ ]*$/;
if(/^(SELECT|UPDATE|DELETE|INSERT|SET|COMMIT|ROLLBACK|CREATE|DROP|ALTER)/i) {
if (defined $q) { print "$q\n"; }
$q=$_;
} else {
$_ =~ s/^[ \t]+//; $q.=" $_";
}
}'
I had similar "problem"
Try to check your mysql ssl
Probably the ssl was turned on hence the traffic was encrypted
You can refer to this post to check the ssl: https://dba.stackexchange.com/questions/36776/how-can-i-verify-im-using-ssl-to-connect-to-mysql
I tried another tshark command from this post, and it could capture query sql from local to remote mysql server.
tshark -i en0 -d tcp.port==3306,mysql -T fields -e mysql.query 'port 3306'
Capturing on 'Wi-Fi'
select version()
select now()
select rand()
but it also output some blank lines between these sql. I tried below command want to remove blank line but failed
tshark -i en0 -d tcp.port==6006,mysql -Y "frame.len>10" -T fields -e mysql.query 'port 6006'
And unfortunately this command cannot support capturing query sql to local mysql(5.7.12).
tshark -i lo -d tcp.port==3306,mysql -T fields -e mysql.query 'port 3306'
Capturing on 'Loopback'
Nothing output except blank lines.
Wireshark tool supports MySQL protocol:
https://www.wireshark.org/docs/dfref/m/mysql.html
Then config wireshark
a.menu Analyze --> Decode as --> add "field=tcp_port value=3306 current=MySQL"
b.filter ‘mysql‘ or ‘mysql.query != “”‘