The MySQL client/server protocol docs say:
Note
sending a MySQL Packet of the size 224-5 to 224-1 via compression leads to at least one extra compressed packet. If the uncompressed MySQL Packet is like
fe ff ff 03 ... -- length = 2^24-2, sequence id = 3
compressing it would result in the length of payload before compression in the Compressed Packet Header being:
length of mysql packet payload: 2^24-2
length of mysql packet header: 4
length of payload before compression: 2^24+2
which can not be represented in one compressed packet. Instead two or more packets have to be sent.
What I don't understand is how the packet is split.
Is the entire MySQL compressed, and then separated? If so, how is it split, and what should the length of payload before compression field be set to?
Or is the MySQL packet split into chunks of max size 224-1, and then sent in individual compressed packets with length of payload before compression set to ff ff ff in all but the last? This seems to be the more likely of the two, but I can't find any documentation for this detail.
Finally, if it is the latter, should the max packet size pre-compression be set to 224-5, to prevent a scenario like the following:
Payload size: 0x100fffe (0xffffff + 0xffff)
MySQL Packets:
#1: length of payload = ff ff ff (size = 0x1000003)
#2: length of payload = ff ff 00 (size = 0x10003)
Compressed packets:
MySQL Packet #1:
#1: length of payload before compression = ff ff ff
#2: length of payload before compression = 04 00 00
MySQL Packet #2:
#1: length of payload before compression = ff ff 00
Here 3 packets are compressed and sent, instead of just 2 if packets were split with max size 224-5 at the MySQL Packet layer.
Related
I'm trying to capture all DHCP Discover packets that don't end with ff that should be the last byte of a correct BOOTP request.
I can filter all DHCP Discover by using the correct offset ether[284:1] because it is at the beginning of the packet but what changes is obviously the length of the entire request.
Is there a way to dynamically calculate the length of the packet and use it as a proper offset?
Thanks
In the http/2 protocol we see the following statement for dynamic table size update:
SETTINGS_HEADER_TABLE_SIZE (0x1): Allows the sender to inform the
remote endpoint of the maximum size of the header compression
table used to decode header blocks, in octets. The encoder can
select any size equal to or less than this value by using
signaling specific to the header compression format inside a
header block (see [COMPRESSION]). The initial value is 4,096
octets.
The initial size for both encoder and decoder is 4096 bytes according to RFC.
In SETTINGS frame in wireshark, i can see the new table size passed to the ENDPOINT ( google.com in this case )
0000 00 00 12 04 00 00 00 00 00 **00 01 00 01 00** 00 00
0010 04 00 02 00 00 00 05 00 00 40 00
00 01 00 01 00 is a pattern for SETTINGS_HEADER_TABLE_SIZE = 65536
What i can't understand does it actually tells the endpoint that the dynamic table used to decode the headers from this ENDPOINT inside browser is 65536 bytes long, or does it tell the ENDPOINT that ENDPOINT dynamic table size should be 65536 ?
And reversed, i assume that the ENDPOINT must sent SETTINGS_HEADER_TABLE_SIZE to tell the browser its dynamic table used for decoding the headers from ENDPOINT but i don't see that option sent back by the ENDPOINT. Can someone explain this?
Also there is a signal for dynamic table size update, mentioned in RFC, which is sent inside the HEADERS frame.
A dynamic table size update starts with the '001' 3-bit pattern,
followed by the new maximum size, represented as an integer with a
5-bit prefix (see Section 5.1).
The new maximum size MUST be lower than or equal to the limit
determined by the protocol using HPACK. A value that exceeds this
limit MUST be treated as a decoding error. In HTTP/2, this limit is
the last value of the SETTINGS_HEADER_TABLE_SIZE parameter (see
Section 6.5.2 of [HTTP2]) received from the decoder and acknowledged
by the encoder (see Section 6.5.3 of [HTTP2]).
There is this line received from the decoder and acknowledged by the encoder, so does this signal is sent to limit the encoding dynamic table size ? I comletely lost, and it is not obvious from wireshark captures how this is handled correctly
UPDATE
Ok, i looked more on the logs of wireshark from firefox on the site of walmart.com ( since there is a lot of headers involved). Sometimes firefox sends the dynamic table size update signal in the headers frame, with the size smaller then the initial SETTINGS_HEADER_TABLE_SIZE sent by firefox on the beginning of connection. I wrote a firefox dynamic table on a paper and shrink it as if i expected the dynamic table size update would do. Turns out that shrinking it to smaller size produce incorrect headers.. So apparently the dynamic table size update affect only remote endpoint.. ( well i guess it is ). I also looked up on nigthttp and a c# implementation, and there they actually shrink the encoder table size, while sending dynamic table size update signal. I get a feeling that everyone have a complete different implementation for this protocol.. it's a complete nightmare to understand.
As you figured out there are multiple things which indicate the table size:
The maximum table size setting (as indicated in a HTTP/2 SETTINGS frame)
The actual used table size - which is encoded in a HEADERS frame in HPACK format
If we only look at the headers which are flowing from the client (browser) to a server we will see the following things going on:
As long as nobody has an information from the remote side the default values are used, which means the client expects that the server supports a maximum table size of 4kB (SETTINGS_HEADER_TABLE_SIZE) and it also uses this size as the initial table size.
The server can optionally inform the client through the HTTP/2 SETTINGS frame that it only supports smaller header tables. This information is contained in the SETTINGS_HEADER_TABLE_SIZE field, a SETTINGS frame which is sent from the server to the client.
The client can adjust the actually used [dynamic] header table size through the Dynamic Table Size Update in a HEADERS frame. This will always indicate the table size that is actually used on the encoder side - and which therefore also must be set on decoder side to be able to retrieve the same data. The sending side is free to set the actual used table size to anything between 0 and the maximum size that is supported by the remote side (in SETTINGS_HEADER_TABLE_SIZE). A typical strategy for implementations is to to always shrink the used table size when it's currently larger than what the remote supports. And to increase the table size when the remote supports bigger tables and the implementation also still can go bigger. There might be some race conditions where one end already set and used a larger table size than what the remote side actually supports, e.g. because the SETTINGS frame which indicates the lower limit was not received before a client encoded the first pair of headers. In that case the remote side might detect the use of a too big table size and reset the connection. To avoid this situations both sides of the connection should in reality at least support the default table size of 4kB, and ideally only increase the limit dynamically and never shrink it.
Now I mentioned that one pair of max. table size settings and actual table size settings is used for transmitting HEADERS from one end of the connection (client) to the other (server). But in total there is also a second pair of both, for the headers which are sent from the server to the client. For this case the client/browser also indicates in a SETTINGS frame how big the max. header table is that it supports and the server sends the size of the actual header table that is used.
I use both Tcpdump and libpcap(a program which uses libpcap) to capture TCP packets. And I notice there are some packets which are padded/added with additional 0x00 bytes at the end. For example, while the IP length indicated in the IP header says that the length is 40 bytes, tcpdump captures 46 bytes. and I notice there are 6 0x00 bytes at the end of the TCP packets.
They don't add those bytes.
The machine sending the packets does, because that's required on Ethernet.
A 40-byte IP packet, when sent on Ethernet, would be 54 bytes long, because there's a 14-byte Ethernet header before the IP header and payload.
However, the minimum packet length on Ethernet is 60 bytes (not including the 4-byte FCS at the end). That means that the packet has to be padded to 60 bytes, which means adding 6 bytes of padding at the end.
(That's one reason why the IP header has a length field - so that the receiver of the packet knows how much is IP and how much is padding.)
I'm trying to connect to minecraft server with as3.
The server returns a ByteArray that I am unable to understand.
Here's an example:
«¢00
*H÷
0[ÞJí"
nöí_Jí2Q»÷/½KW9ó`ä¦ËJ!ôàNÄÇgkÉÚY`*u\êRáåLøjTp9ÔÅڕQ̺ÐWÊýƶ[Ð5æsövxåIIæ>Z
u¾Cӷ.C¹iPWûóM×
I tried the following to interpret the data:
bytes.uncompress();
but I got this error:
Error: Error #2058: There was an error decompressing the data.
According to http://wiki.vg/Protocol#Packet_format, the Packet format for Minecraft is as shown below. You need to interpret the bits of the data that you receive as shown below and then send the compressed data to zlib, if the packet is compressed.
Packet format
Without compression
Field Name Field Type Notes
------------ ------------ -----------------------------------------------------------------------
Length VarInt Length of packet data + length of the packet ID
Packet ID VarInt
Data Byte Array Depends on the connection state and packet ID, see the sections below
With compression
Once a Set Compression packet is sent,
zlib compression
is enabled for all following packets. The format of a packet changes
slighty to include the size of the uncompressed packet.
Field Name Field Type Notes
------------ ------------ -----------------------------------------------------------------------
Packet Length VarInt Length of Data + length of Data Length
Data Length VarInt Length of uncompressed Data or 0
Data Byte Array zlib compressed packet, including packet ID (see the sections below)
How do you know that your packet is compressed? According to this same documentation, compression isn't enabled until a Set Compression is sent.
I have tcpdump like this
sudo tcpdmp tcp -n -i eth0 -w test.dmp
I want to calculate the number of tcp bytes going through eth0. I capture all the package using tcpdump as above. Is the file size equal the number of bytes or tcpdump add additional information into the dump file?
Yes, tcpdump adds additional information to the file.
It (currently) writes only in pcap format, which means there's a 24-byte header at the beginning of the file, giving information such as the link-layer header type for packets in the file, so the first thing you'd need to do would be to subtract 24 from the size of the file.
In addition, each packet has a 16-byte header giving an arrival time stamp for the packet, the length of the packet, and the number of bytes of packet data that was captured. This means that you would need to subtract 16*{number of packets} from the length - but the only way to get the number of packets is to read the file, so you can't get the number of bytes just by looking at the file size!
Note also that some versions of tcpdump did not default to a "snapshot length" of 0, so the number of bytes of packet data that is captured may be less than the number of packet bytes on the network.
Therefore, what you should do is write a program (use libpcap, as it already knows pcap format and you don't have to write your own code to understand it) that reads all the packets and adds up the "length of the packet" field (it's the len field in the struct pcap_pkthdr structure; do not use caplen, as that's the number of bytes of packet data that was captured) values for all the packets.
You say eth0, so the link-layer header type is probably Ethernet, and there is, for example, no radio meta-data, as might be the case if you were capturing in monitor mode on a Wi-Fi adapter. In the cases where there's extra meta-data in the link-layer header, you'd need to subtract that.