I have a simple pcap with some web traffic and am using tshark to obtain some header information from it:
I use the following command:
tshark -r ./capture-1-5 -Y "http2" -o tls.keylog_file:ssl-key.log \
-T fields -e frame.number -e _ws.col.Time -e ip.src -e tcp.srcport \
-e ip.dst -e tcp.dstport -e _ws.col.Protocol -e frame.len \
-e _ws.col.Info -E header=y -E separator="," -E quote=d \
-E occurrence=f > desegmented.csv
I realized that in this case all fragments are reassembled resulting in huge packets. However, I do not want reassembled packets. So, I add an extra option to tshark:
tshark -r ./capture-1-5 -Y "http2" -o tls.keylog_file:ssl-key.log \
-T fields -e frame.number -e _ws.col.Time -e ip.src -e tcp.srcport \
-e ip.dst -e tcp.dstport -e _ws.col.Protocol -e frame.len \
-e _ws.col.Info -E header=y -E separator="," -E quote=d \
-E occurrence=f -o tcp.desegment_tcp_streams:FALSE > segmented.csv
My intuition is that the resultant disassembled.csv file should be greater in size and should contain more rows given that the "packets above the MTU" will be shown as more than one packet.
However, I observe the opposite. The resultant file without assembly is smaller and has almost halved the number of rows.
-rw-r--r-- 1 root root 210K May 18 18:21 desegmented.csv
-rw-r--r-- 1 root root 97K May 18 18:21 segmented.csv
# cat desegmented.csv |wc -l
2635
# cat segmented.csv |wc -l
1233
Is this a normal behavior? I don't see (manually) where the packets start to disappear (and why) or see any pattern because of the two-way communication (missing packets here and there).
I assume that maybe, in the disassebmled.csv case, every packet or even the whole packet stream that resulted in at least one packet above the MTU is completely dropped.
I tried to also apply ip.defragment:FALSE but still the same results.
Thanks
For reproducing, the files can be downloaded from here
Thanks, #JimD., I have already come to a similar conclusion!
Packet capture itself has to be segmented to do this precisely.
So, tried to go one layer below, and make the packet capture itself to be segmented via
ethtool -K eth0 gso off tso off gro off sg off tx off rx off
(just to make sure).
The problem is that packet capturing is done in a docker container, so at multiple places, I have to issue this command to be fully working.
These places include the docker0 bridge, eth0 inside the container and the corresponding vethXXXXXX on the host, from which the second requires privileged containers that pose further issues :)
Related
So I've got the following project OpenFHE-development and when I run the build process, there are lots of warnings. However, most of these warnings are fine to ignore (we vet them before pushing to the main branch)
Specifically, is there a way to take
pth/python -m sphinx -T -E -b readthedocssinglehtmllocalmedia -d _build/doctrees -D language=en . _build/localmedia
and convert it to
pth/python -m sphinx -T -E -b readthedocssinglehtmllocalmedia -d _build/doctrees -D language=en . _build/localmedia 2> errors.txt
(pipe the stderr to a file instead of having it display on stdout)?
Does not seem to be possible at the moment. See git discussion
I'm trying to configure the beaglebone black [wireless version - 4.9.82-ti-r102 #1 SMP PREEMPT] -- running debian 9.3 stretch.
This command works fine in a bash terminal:
sudo /usr/sbin/rtcwake -m mem -u -t $(date +%s -d "+2 minutes")
I've setup a cron job via
sudo crontab -e
In it, I have the following line:
10,40 * * * * /usr/sbin/rtcwake -m mem -u -t $(date +%s -d "20 minutes")
However, the device is not suspending.
What am I doing wrong? How do I debug this?
Thanks for your help
M
The % char is the problem in the crontab approach. That char has a special meaning in crontab (newline...start of stdin, see manual).
Instead of -t $(date +%s -d "1 minutes") you can simply use -s 60 for sleeping 60 seconds and not having to use % at all. Or you might escape it with \% or use '+%s' or "+%s".
OK,
In case there is anyone else trying to solve this. I don't know why this works, however it does...
1. make a script with the rtcwake comand in it....:
#!/bin/bash
# suspend then wake from a bash script
echo attempting to suspend!
/usr/sbin/rtcwake -m mem -u -t $(date +%s -d "1 minutes")
I called this wakesleep, and placed it in /usr/local/bin (on the PATH), make it executable...
sudo chmod +x /usr/local/bin/sleepwake
Then added a line in crontab, the su version:
sudo crontab -e
The added line in the root crontab is of this sort of format:
10 * * * * /usr/local/bin/sleepwake
... and it works. This is a mystery to me, why the previous approach did not work, but it works. Very keen to hear from someone what the difference between the two approaches is.
Because we develop using remote Mysql server , so cannot check query sql easily, if use local server you can tail - f general_log_file to see which sql are executed when call some http interface. So I installed a wireshark to capture these query sql send from local. At first I use local mysql to verify it.
The capture filter is
then I executed two query sql in mysql terminal
select version();
select now();
but very disappointing I cannot find these two sql packets in wireshark
I only found these four packets.
But from a post I knew
To filter out the mysql packets you just use the filter ‘mysql‘ or ‘mysql.query != “”‘ when you only want packets that request a query. After that you can add a custom column with the field name ‘mysql.query’ to have a list of queries that where executed.
and the effect is like this
It's convenient to capture only query sql and very clearly displayed these query sql. So how could I use wireshark to implement this?
hi #Jeff S.
I tried your command, please see below
#terminal 1
tshark -i lo0 -Y "mysql.command==3"
Capturing on 'Loopback'
# terminal 2
mysql -h127.0.0.1 -u root -p
select version();
#result: nothing output in terminal 1
and tshark -i lo0 -Y "mysql.command==3" -T fields -e mysql.query is same with tshark -i lo -Y "mysql.command==3" also nothing output. But if I only use tshark -i lo0, it has output
Capturing on 'Loopback'
1 0.000000 127.0.0.1 -> 127.0.0.1 TCP 68 57881 → 3306 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=32 TSval=1064967501 TSecr=0 SACK_PERM=1
2 0.000062 127.0.0.1 -> 127.0.0.1 TCP 68 3306 → 57881 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=16344 WS=32 TSval=1064967501 TSecr=1064967501 SACK_PERM=1
3 0.000072 127.0.0.1 -> 127.0.0.1 TCP 56 57881 → 3306 [ACK] Seq=1 Ack=1 Win=408288 Len=0 TSval=1064967501 TSecr=1064967501
4 0.000080 127.0.0.1 -> 127.0.0.1 TCP 56 [TCP Window Update] 3306 → 57881 [ACK] Seq=1 Ack=1 Win=408288 Len=0 TSval=1064967501 TSecr=1064967501
...
You can use tshark and save to a pcap or just export the fields you're interested in.
To save to a pcap (if you want to use wireshark to view later):
tshark -i lo -Y "mysql.command==3" -w outputfile.pcap
tshark -i lo -R "mysql.command==3" -w outputfile.pcap
-R is deprecated for single pass filters, but it will depend on your version
-i is interface so replace that with whatever interface you are using (e.g -i eth0)
To save to a text file:
tshark -i lo -Y "mysql.command==3" -T fields -e mysql.query > output.txt
You can also use BPF filters with tcpdump (and wireshark pre cap filters). They are more complex, but less taxing on your system if you're capturing a lot of traffic.
sudo tcpdump -i lo "dst port 3306 and tcp[(((tcp[12:1]&0xf0)>>2)+4):1]=0x03" -w outputfile.pcap
NOTE:
*This looks for 03 (similar mysql.command==3) within the TCP payload.
**Since this is a pretty loose filter, I also added 3306 to restrict to only traffic destined for that port.
***The filter is based on your screenshot. I cannot validate it right now so let me know if it doesn't work.
Example Output:
Useful answers here:
https://serverfault.com/questions/358978/how-to-capture-the-queries-run-on-mysql-server
In particular: SoMoSparky's answer of:
tshark -T fields -R mysql.query -e mysql.query
and user1038090's answer of:
tcpdump -i any -s 0 -l -vvv -w - dst port 3306 | strings | perl -e '
while(<>) { chomp; next if /^[^ ]+[ ]*$/;
if(/^(SELECT|UPDATE|DELETE|INSERT|SET|COMMIT|ROLLBACK|CREATE|DROP|ALTER)/i) {
if (defined $q) { print "$q\n"; }
$q=$_;
} else {
$_ =~ s/^[ \t]+//; $q.=" $_";
}
}'
I had similar "problem"
Try to check your mysql ssl
Probably the ssl was turned on hence the traffic was encrypted
You can refer to this post to check the ssl: https://dba.stackexchange.com/questions/36776/how-can-i-verify-im-using-ssl-to-connect-to-mysql
I tried another tshark command from this post, and it could capture query sql from local to remote mysql server.
tshark -i en0 -d tcp.port==3306,mysql -T fields -e mysql.query 'port 3306'
Capturing on 'Wi-Fi'
select version()
select now()
select rand()
but it also output some blank lines between these sql. I tried below command want to remove blank line but failed
tshark -i en0 -d tcp.port==6006,mysql -Y "frame.len>10" -T fields -e mysql.query 'port 6006'
And unfortunately this command cannot support capturing query sql to local mysql(5.7.12).
tshark -i lo -d tcp.port==3306,mysql -T fields -e mysql.query 'port 3306'
Capturing on 'Loopback'
Nothing output except blank lines.
Wireshark tool supports MySQL protocol:
https://www.wireshark.org/docs/dfref/m/mysql.html
Then config wireshark
a.menu Analyze --> Decode as --> add "field=tcp_port value=3306 current=MySQL"
b.filter ‘mysql‘ or ‘mysql.query != “”‘
I have just dumped a riak db (back-up). But the backup file is a binary file.
Is there a lib that it deserialize it into a human readable file (JSON w/e) ?
I haven't found anything on google, neither on Stack Overflow.
Found a solution for my current problem:
Connect to the env and then run following command:
wget https://s3-us-west-2.amazonaws.com/ps-tools/riak-data-migrator-0.2.9-bin.tar.gz
tar -xvzf riak-data-migrator-0.2.9-bin.tar.gz
cd riak-data-migrator-0.2.9
java -jar riak-data-migrator-0.2.9.jar -d -r /var/riak_export -a -h 127.0.0.1 -p 8087 -H 8098
(source: https://github.com/basho-labs/riak-data-migrator)
EDIT
Another way to export riak db https://www.npmjs.com/package/riak-bucket-exporter
#!/bin/bash
for bucket in $(curl http://localhost:8098/riak?buckets=true | sed -e 's/[{}:"]//gi' -e 's/buckets\[//' -e 's/\]//' -e 's/,/ /g')
do
echo "Exporting bucket $bucket"
rm -f $bucket.json
riak-bucket-exporter -H localhost -p 8098 $bucket
done
echo "Export done"
As all the suggestions listed so far appear to be broken in one way or another (at least for me and riak-kv#2.x), I ultimately resorted to homegrow a bash shell script that leverages riak-kv's HTTP API with no other prerequisites than curl and jq to accomplish an export of sorts.
It can be found in this gist here: https://gist.github.com/cueedee/0b26ec746c4ef578cd98e93c93d2b6e8 hoping that someone will find it useful.
I am trying to programmatically capture a stream of packets by using Tshark. The simplified terminal command I am using is:
tshark -i 2 -w output.pcap
This is pretty straightforward, but I then need to get a .csv file in order to easily analyze the information captured.
By opening the .pcap file in Wireshark and exporting it in .csv what I get is a file structured as follows:
"No.","Time","Source","Destination","Protocol","Length","Info"
but,again, I need to do this in an automatic way. So I tried using the command:
tshark -r output.pcap -T fields -e frame.number -e ip.src -e ip.dst -e frame.len -e frame.time -e frame.time_relative -E header=y -E separator=, > output.csv
but I can not find anywhere the name of the "Info" field I get when manually exporting the .csv.
Any ideas? Thanks!
Yes, you can if you use the latest Development Release.
See Wireshark Bug 2892.
Download the Development Release Version 1.9.0.
Use the following command:
$ tshark -i 2 -T fields -e frame.time -e col.Info
Output
Feb 28, 2013 20:58:24.604635000 Who has 10.10.128.203? Tell 10.10.128.1
Feb 28, 2013 20:58:24.678963000 Who has 10.10.128.163? Tell 10.10.128.1
Note
-e col.Info,
Use capital I
How about directly exporting the packets to a csv file.
sudo tshark > fileName.csv