I'm trying to capture all DHCP Discover packets that don't end with ff that should be the last byte of a correct BOOTP request.
I can filter all DHCP Discover by using the correct offset ether[284:1] because it is at the beginning of the packet but what changes is obviously the length of the entire request.
Is there a way to dynamically calculate the length of the packet and use it as a proper offset?
Thanks
Related
Will tab/space like the below example increases the size of data sent?
return {
id:self.id,
username:self.username,
score:self.score,
level:self.level
};
vs
return {id:self.id,username:self.username
score:self.score,level:self.level};
Is there any size difference between 0/1 and true/false for Json?
Is there a size difference between "11" (string) and 11 (double)?
The Json will be sent 10 times every second with socket.emit of Socket.io.
There should be no difference in the size based on the server side object format. Any size would be type dependent and how your serializer actually converts the object properties.
Is there any size difference between 0/1 and true/false for Json?
There might be a dependence on the server type but number vs Boolean results would be within a string (serialized) as "mybool:true,mynumber:1"
So, if you were to optomize for size "a:true,b:1"` note the NAME is smaller so the serialzed content would be.
Is there a size difference between "11" (string) and 11 (double)? Similar to the second example "mysuperlongnameisgreat:"11",mysuperlongnumbername:11" vs {"a":"11","b":11}" thus the number is smaller by excluding those two quotes.
All that being said considering the total processing (IF speed is an issue) this has to be deserialized into a JavaScript object on the client side so IF you are using the number as a number it will need to parse that at some point to the proper type and thus may be more of an impact than serialized content size.
Note that with short names, you WILL have a negative impact on maintenance as it is much less intuitive to maintain short non-descriptive names than longer ones.
Using your example, it would be "smaller" to do (assuming a strongly typed server side)
return {
d:self.id,
n:self.username,
s:self.score,
v:self.level
};
In the http/2 protocol we see the following statement for dynamic table size update:
SETTINGS_HEADER_TABLE_SIZE (0x1): Allows the sender to inform the
remote endpoint of the maximum size of the header compression
table used to decode header blocks, in octets. The encoder can
select any size equal to or less than this value by using
signaling specific to the header compression format inside a
header block (see [COMPRESSION]). The initial value is 4,096
octets.
The initial size for both encoder and decoder is 4096 bytes according to RFC.
In SETTINGS frame in wireshark, i can see the new table size passed to the ENDPOINT ( google.com in this case )
0000 00 00 12 04 00 00 00 00 00 **00 01 00 01 00** 00 00
0010 04 00 02 00 00 00 05 00 00 40 00
00 01 00 01 00 is a pattern for SETTINGS_HEADER_TABLE_SIZE = 65536
What i can't understand does it actually tells the endpoint that the dynamic table used to decode the headers from this ENDPOINT inside browser is 65536 bytes long, or does it tell the ENDPOINT that ENDPOINT dynamic table size should be 65536 ?
And reversed, i assume that the ENDPOINT must sent SETTINGS_HEADER_TABLE_SIZE to tell the browser its dynamic table used for decoding the headers from ENDPOINT but i don't see that option sent back by the ENDPOINT. Can someone explain this?
Also there is a signal for dynamic table size update, mentioned in RFC, which is sent inside the HEADERS frame.
A dynamic table size update starts with the '001' 3-bit pattern,
followed by the new maximum size, represented as an integer with a
5-bit prefix (see Section 5.1).
The new maximum size MUST be lower than or equal to the limit
determined by the protocol using HPACK. A value that exceeds this
limit MUST be treated as a decoding error. In HTTP/2, this limit is
the last value of the SETTINGS_HEADER_TABLE_SIZE parameter (see
Section 6.5.2 of [HTTP2]) received from the decoder and acknowledged
by the encoder (see Section 6.5.3 of [HTTP2]).
There is this line received from the decoder and acknowledged by the encoder, so does this signal is sent to limit the encoding dynamic table size ? I comletely lost, and it is not obvious from wireshark captures how this is handled correctly
UPDATE
Ok, i looked more on the logs of wireshark from firefox on the site of walmart.com ( since there is a lot of headers involved). Sometimes firefox sends the dynamic table size update signal in the headers frame, with the size smaller then the initial SETTINGS_HEADER_TABLE_SIZE sent by firefox on the beginning of connection. I wrote a firefox dynamic table on a paper and shrink it as if i expected the dynamic table size update would do. Turns out that shrinking it to smaller size produce incorrect headers.. So apparently the dynamic table size update affect only remote endpoint.. ( well i guess it is ). I also looked up on nigthttp and a c# implementation, and there they actually shrink the encoder table size, while sending dynamic table size update signal. I get a feeling that everyone have a complete different implementation for this protocol.. it's a complete nightmare to understand.
As you figured out there are multiple things which indicate the table size:
The maximum table size setting (as indicated in a HTTP/2 SETTINGS frame)
The actual used table size - which is encoded in a HEADERS frame in HPACK format
If we only look at the headers which are flowing from the client (browser) to a server we will see the following things going on:
As long as nobody has an information from the remote side the default values are used, which means the client expects that the server supports a maximum table size of 4kB (SETTINGS_HEADER_TABLE_SIZE) and it also uses this size as the initial table size.
The server can optionally inform the client through the HTTP/2 SETTINGS frame that it only supports smaller header tables. This information is contained in the SETTINGS_HEADER_TABLE_SIZE field, a SETTINGS frame which is sent from the server to the client.
The client can adjust the actually used [dynamic] header table size through the Dynamic Table Size Update in a HEADERS frame. This will always indicate the table size that is actually used on the encoder side - and which therefore also must be set on decoder side to be able to retrieve the same data. The sending side is free to set the actual used table size to anything between 0 and the maximum size that is supported by the remote side (in SETTINGS_HEADER_TABLE_SIZE). A typical strategy for implementations is to to always shrink the used table size when it's currently larger than what the remote supports. And to increase the table size when the remote supports bigger tables and the implementation also still can go bigger. There might be some race conditions where one end already set and used a larger table size than what the remote side actually supports, e.g. because the SETTINGS frame which indicates the lower limit was not received before a client encoded the first pair of headers. In that case the remote side might detect the use of a too big table size and reset the connection. To avoid this situations both sides of the connection should in reality at least support the default table size of 4kB, and ideally only increase the limit dynamically and never shrink it.
Now I mentioned that one pair of max. table size settings and actual table size settings is used for transmitting HEADERS from one end of the connection (client) to the other (server). But in total there is also a second pair of both, for the headers which are sent from the server to the client. For this case the client/browser also indicates in a SETTINGS frame how big the max. header table is that it supports and the server sends the size of the actual header table that is used.
I am fairly new to both Kafka and Spark and trying to write a job (either Streaming or batch). I would like to read from Kafka a predefined number of messages (say x), process the collection through workers and then only start working on the next set of x messages. Basically each message in Kafka is 10 KB and I want to put 2 GB worth of messages in a single S3 file.
So is there any way of specifying the number of messages that the receiver fetches?
I have read that I can specify 'from offset' while creating DStream, but this use case is somewhat different. I need to be able to specify both 'from offset' and 'to offset'.
There's no way to set ending offset as the initial parameter (as you can for starting offset), but
you can use createDirectStream (the fourth overloaded version in the listing) which gives you the ability to get the offsets of the current micro batch using HasOffsetRanges (which gives you back OffsetRange).
That means that you'll have to compare values that you get from OffsetRange with your ending offset in every micro batch in order to see where you are and when to stop consuming from Kafka.
I guess you also need to think about the fact that each partition has its sequential offset. I assume it would be easiest if you could go a bit over 2GB, as much as it takes to finish the current micro-batch (could be couple of kB, depending on density of your messages), in order to avoid splitting the last batch on consumed and unconsumed part, which may require you to fiddle with offsets that Spark keeps in order to track what's consumed and what isn't.
Hope this helps.
I am trying to parse MAC timestamp fields of radiotap headers of 802.11 packets captured on monitor mode.
TSFT field of radiotap header is 64bit value in microseconds. Raw hex value is highlighted below.
The MAC timestamp value is represented in decimal by Wireshark
This decimal value is decimal value of 2b1c20cb00000000.
What I'm trying to do is get the time difference between two frames using hex value in radiotap header MAC timestamp field.
For example:
frame #2 has decimal value of 3106049021945315329 (2b1ae72100000001) and
frame #3 has 3106066889009266689 (2b1af76100000001).
subtracting this values gives 1AC47FFFFF5C1. And assuming this is in microseconds the value is equal to 470900214.330817 seconds.
What is the process following this steps to get time difference of 0.000071 seconds by using the values in MAC timestamp field of radiotap header
Thank you
The "MAC timestamp" field in the radiotap header is the value in microseconds of the MAC's 64-bit 802.11 Time Synchronization Function timer when the first bit of the MPDU arrived at the MAC.
This is taken directly from the MAC via the device driver for the particular WiFi card you have, and may or may not be accurate or correct, depending on the driver implementation.
The "Time" column displays the elapsed time since the first frame was received. This is calculated by libpcap using the system clock on the host and is the time the frame was first seen by libpcap.
Both of these time values are computed using different clocks, so cannot be directly compared. If the MAC timestamp field is correct and accurate (which yours appears not to be - maybe a driver issue) then it should be used as the reference time, and the libpcap time should only be used as a rough guide.
I have tcpdump like this
sudo tcpdmp tcp -n -i eth0 -w test.dmp
I want to calculate the number of tcp bytes going through eth0. I capture all the package using tcpdump as above. Is the file size equal the number of bytes or tcpdump add additional information into the dump file?
Yes, tcpdump adds additional information to the file.
It (currently) writes only in pcap format, which means there's a 24-byte header at the beginning of the file, giving information such as the link-layer header type for packets in the file, so the first thing you'd need to do would be to subtract 24 from the size of the file.
In addition, each packet has a 16-byte header giving an arrival time stamp for the packet, the length of the packet, and the number of bytes of packet data that was captured. This means that you would need to subtract 16*{number of packets} from the length - but the only way to get the number of packets is to read the file, so you can't get the number of bytes just by looking at the file size!
Note also that some versions of tcpdump did not default to a "snapshot length" of 0, so the number of bytes of packet data that is captured may be less than the number of packet bytes on the network.
Therefore, what you should do is write a program (use libpcap, as it already knows pcap format and you don't have to write your own code to understand it) that reads all the packets and adds up the "length of the packet" field (it's the len field in the struct pcap_pkthdr structure; do not use caplen, as that's the number of bytes of packet data that was captured) values for all the packets.
You say eth0, so the link-layer header type is probably Ethernet, and there is, for example, no radio meta-data, as might be the case if you were capturing in monitor mode on a Wi-Fi adapter. In the cases where there's extra meta-data in the link-layer header, you'd need to subtract that.