I am working on Click fast prototyping router api created by Eddie Kohler for my research.
I am having hard time in generating tcpdump files with RAW IP ENCAP. I know that my link type doesn't allow to capture in RAW IP link-type as the only link type listing on typing tcpdump -i (interface) -L is EN10MB and nothing else.
It is just not striking my mind. Are there ways to capture Raw IP packets (dump packets should start from IP headers and skip link level headers) in tcpdump or tshark?
Neither tcpdump nor tshark can arbitrarily choose a link-layer header type for a device, as the devices, and thus libpcap/WinPcap, don't allow arbitrary link-layer header types to be chosen (as tcpdump -i {interface} -L informed you).
Most network interfaces don't support "raw IP" as a link-layer header type; I'm not sure which, if any, do. If you want a capture file full of packets with that encapsulation, the easiest way to do it is probably to write a program that captures from an interface, discards all non-IP packets, and strips the link-layer header off of the IP packets and writes them out. For example, if you're capturing on a device with a DLT_ value of DLT_EN10MB (Ethernet), just throw away all packets that don't have 0x0800 or 0x86DD as the Ethernet type value, and strip off the first 14 bytes of packets that do have those Ethernet type values and write the resulting packets out in a pcap file with a DLT_ value of DLT_RAW.
Related
ALL,
The documentation here talks about 2 different functions.
Is pcap_next() call for pcap while pcap_next_ex() is for pcapng format? Or both those functions can read both formats?
The page doesn't indicate it.
I have a code that parses pcap file and uses the former call, and I'm just wondering if it will be enough to just check for pcapng file and use the latter call instead.
TIA!!
Is pcap_next() call for pcap while pcap_next_ex() is for pcapng format?
No. pcap_next() is for applications that don't bother checking for errors, and pcap_next_ex() is for applications that do. To quote the RETURN VALUE section of the man page:
pcap_next_ex() returns 1 if the packet was read without problems, 0 if packets are being read from a live capture and the packet buffer timeout expired, PCAP_ERROR if an error occurred while reading the packet, and PCAP_ERROR_BREAK if packets are being read from a ``savefile'' and there are no more packets to read from the savefile. If PCAP_ERROR is returned, pcap_geterr(3PCAP) or pcap_perror(3PCAP) may be called with p as an argument to fetch or display the error text.
pcap_next() returns a pointer to the packet data on success, and returns NULL if an error occurred, or if no packets were read from a live capture (if, for example, they were discarded because they didn't pass the packet filter, or if, on platforms that support a packet buffer timeout that starts before any packets arrive, the timeout expires before any packets arrive, or if the file descriptor for the capture device is in non-blocking mode and no packets were available to be read), or if no more packets are available in a "savefile." Unfortunately, there is no way to determine whether an error occurred or not.
You should use pcap_next_ex().
Or both those functions can read both formats?
Yes. (pcap_next_ex() was added before libpcap could even read pcapng files.)
I have a code that parses pcap file and uses the former call, and I'm just wondering if it will be enough to just check for pcapng file and use the latter call instead.
It would be enough just to use pcap_next_ex() regardless of whether the file is a pcap or pcapng file.
I'm used to sending a request to an endpoint with a method and whatever other options I need attached, e.g. axios({ method: `get`, url: `https://foo.bar/baz&q=123`}).
However, I came across this API https://vndb.org/d11 with docs specifying neither the method nor the url (aside from host, which I supposed isn't enough?) where I'm supposed to send my request, so how am I supposed to get any data out of it? Is it possible to access it via Postman or some such?
Host api.vndb.org
Port (plain tcp) 19534 ('VN')
Port (TLS) 19535
Sample request message looks like:
login {
"protocol" : 1,
"username" : "ayo"
}
0x04
0x04 - but this doesn't even look like JSON, so how do I send it across?
Looking at the VNDB API documentation it says:
all communication between the client and the server is done using one
TCP connection. This connection stays alive until it is explicitely
closed by either the client or the server.
so it's not like a normal REST API where you GET, POST etc. You open a connection and leave it open. It's session based, hence the initial login command and the 0x04 byte at the end of the JSON is the
End Of Transmission
character. So it works like a two way radio. You say "login, over" where the "over" is the 0x04 byte. That tells the server a command is waiting for it to process. I presume without the 0x04 character the command won't be interpreted. A bit like typing a command and not hitting the return key.
As for how to use the API, you could study this NodeJS VNDB client.
When an image is uploaded from the client's machine to the client (browser), it requires FileReader() API in html, thereafter a base64 encoded url (say) of the image is sent in chunks to the server, where it needs to be re-assembled. All of this is taken care by the developer.
However, when an image is sent from the server to the client, only the directory path of the image inside the server machine suffices, no chunking and encoding is required.
My questions are:
1. Does the server send the image in chunks to the html file. If it does not, how does sending full images not bottle server's network? What would happen in case of large video files?
2. In what form of binary data is the image sent to the client - base64url / ArrayBuffer / binarystring / text / etc.
3. If the server does send the image in chunks, who is doing the chunking and the re-assembly on the client thereafter?
Thanks.
HTML isn't really important here. What you care about are the transport protocols used - HTTP and TCP, most likely.
HTTP isn't chunked by default, though there are advanced headers that do allow that - those are mostly used to support seeking in large files (e.g. PDF, video). Technically, this isn't really chunking - it's just the infrastructure for allowing for partial downloads (i.e. "Give me data from byte 1024 to byte 2048.").
TCP is a stream-based protocol. From programmer point of view, that's all there is to it - no chunking. Technically, though, it will process your input data and send it as distinct packets that are reassembled in-order on the other side. This is a matter of practicality - it allows you to manage data latency, streaming, packet retransmission etc. TCP handles the specifics during connection negotiation - flow control, window sizing, congestion control...
That's not the end of it, though. All the other layers add their own bits - their own ways to package the payload and split it as necessary, their own ways to handle routing and filtering, their own ways to handle congestion...
Finally, just like HTTP natively supports downloads, it supports uploading data as well. Just send an HTTP request (usually POST or PUT) and write data in a format the server understands - anything from text through base-64 to raw binary data. The limiting factor in your case isn't the server, the browser or HTTP - it's JavaScript. The basic mechanism is still the same - a request followed by a response.
Now, to address your questions:
Server doesn't send images to the HTML file. HTML only contains an URL of the image[1], and when the browser sees an URL in the img tag, it will initiate a new, separate request just for the image data. It isn't fundamentally different from downloading a file from a link. As for the transfer itself, it follows pretty much exactly the same way as the original HTML document - HTTP headers, then the payload.
Usually, raw binary data. HTTP is a text-based protocol, but it's payload can be arbitrary. There's little reason to use Base-64 to transfer image data (though in the past, there have been HTTP and FTP servers that didn't support binary at all, so you had to use something like Base-64).
The HTTP server doesn't care (with the exception of "partial downloads" mentioned above). The underlying network protocols handle this.
[1] Nowadays, there's methods to embed images directly in the HTML text, but it's of varying practicality depending on the image size, caching requirements etc.
When an "onmessage" event fires in the web socket protocol are you guaranteed the full message or is it more like a straight TCP connection where you buffer the data first and then try to extract packets.
There is protocol level support for fragmented messages and streaming. But this behavior is not represented in the current Javascript API, (reference). So yes, if you receive a message, it is indeed an entire message even if it was sent as many fragments.
I used sharppcap to capture TCP packets. Now i wanna reconstruct HTTP packet from TCP packets but i don't know how. I read somewhere i can find start of HTTP packet in TCP data... i tried to convert byte[] TCP data to string using this code:
string s = System.Text.Encoding.UTF8.GetString(tcp_pack.Data);
but the string isn't readable. like a binary file that is opened with notepad.
is it because the data is encrypted or code is incorrect?
how can i reconstruct HTTP packet from TCP packets?
try "Encoding.BigEndianUnicode.GetString(tcp_pack.Data)"
There is no easy way do to this because you'll need to reconstruct the TCP session in memory.
Essentially, if a message being sent is larger than one packet, it's broken up into multiple packets. Therefore, you'll need to capture those packets, arrange them in the right order, and reassemble the data manually.
If the message is short/simple and doesn't get broken up into multiple parts then you'll need to find out what format the payload is in. Try decoding in ASCII first.