I used sharppcap to capture TCP packets. Now i wanna reconstruct HTTP packet from TCP packets but i don't know how. I read somewhere i can find start of HTTP packet in TCP data... i tried to convert byte[] TCP data to string using this code:
string s = System.Text.Encoding.UTF8.GetString(tcp_pack.Data);
but the string isn't readable. like a binary file that is opened with notepad.
is it because the data is encrypted or code is incorrect?
how can i reconstruct HTTP packet from TCP packets?
try "Encoding.BigEndianUnicode.GetString(tcp_pack.Data)"
There is no easy way do to this because you'll need to reconstruct the TCP session in memory.
Essentially, if a message being sent is larger than one packet, it's broken up into multiple packets. Therefore, you'll need to capture those packets, arrange them in the right order, and reassemble the data manually.
If the message is short/simple and doesn't get broken up into multiple parts then you'll need to find out what format the payload is in. Try decoding in ASCII first.
Related
I'm working on an HTTP parser and was wondering how can i parse traffic that is sent with no content length and no EOF?
Read the data until the connection is shutdown (from the other side).
This is how the original HTTP worked, but it's not very nice; if the server crashes, or you lose network connectivity in the middle of the transmission, you might not notice that the response is truncated.
If you request html webpage that has some text and an image, I believe http uses 2 responses for that of type text/html and image/html. Are both the http responses put in the same TCP response or not?
I would think that would be the most efficient way to do it unless the server expects another request for the image. Which brings me to my next question, does the server expect multiple requests for each http response or can it just send all the http responses that are needed for the html page?
ie.
TCP (GET /)
TCP (HTTP response 1, response 2, response 3)
Also can the tcp response have a fraction of one http response? For example if it is larger than the max packet size.
Typically, images have their own URLs. In HTTP/1.x, only the client can initiate a request. Even though the server might know that the client would need a resource, it has no way to send it to the client and must wait to receive a request for the resource from the client. Thus, images are fetched through a separate HTTP request, and thus get a separate HTTP response.
TCP Connections per HTTP request:
With regard to a TCP connection, a single connection can be used for multiple HTTP requests, for example in case several images are being fetched from the same server. However, this still requires a FIFO queue of requests and responses. Thus, browsers prefer separate TCP connections per HTTP request.
TCP Packets per HTTP request:
A single HTTP request/response can span across multiple TCP packets, if they don't fit in a single packet. Think of a 1 GB file upload or download request - its a single request/response, that would span over multiple packets.
When the server and client support "persistent connection" (Connection: keep-alive), then multiple HTTP requests can be pipelined in a single TCP packet, if they can fit in.
Try this:
(echo "HEAD /index.html HTTP/1.1\nHost: www.google.co.in\nConnection: keep-alive\n\nHEAD /index.html HTTP/1.1\nHost: www.google.co.in\n\n"; sleep 1) | telnet www.google.co.in 80
The two pipelined HEAD requests evidently fit in a single packet.
In general, I wouldn't worry if HTTP request(s) are payloaded in single/multiple TCP packet(s), and leave that to the TCP protocol. Same holds for HTTP response.
While you are on this subject, I would recommend some reading on HTTP/2 as well :)
I am working on a TCP-based proxy that must first do a REQ/REPLY handshake in json on a given connection. Because JSON is a self-delimiting protocol I reach for Go's json.Decoder to pull off this work which does the job nicely.
Here are the steps I take:
Dial a connection to a remote server
Write a single json request to a remote server (REQ)
Read a single json reply from the same remote server (completing the proxy handshake REPLY)
Upon a valid json handshake, pass the client connection onto another part of the code which will (going forward) switch to a text based protocol from this point on.
The problem is, when json.Decoder reads data into its internal buffer it can potentially read more data than it needs in which case the json.Decoder has a Buffered() method which gives back an io.Reader with the remainder of the data.
This data (available in the Buffered() method) is now the text-based protocol data which needs to get read from the connection after the json hand-shake did its work. But if I pass the connection forward as is without considering the left over buffer, the connection gets into a locked state because it is waiting to read this data which never comes. The code that deals with the text-based protocol expects a net.Conn going forward and once I pass the connection forward (after the json handshake has been made) the code utilizing the connection understands how to speak the text-based protocol at this point on. So there should be a clear boundary of work.
My question is what is the ideal way to solve this issue so I can still take advantage of the json.Decoder, but ensure that when I pass the connection to a different part of the code in my proxy I know the start of the data for the text-based protocol will still be readable. I somehow need to take the remaining data in the json.Decoder's Buffered() method and put that back in front of the connection so it can be properly read going forward.
Any insight is much appreciated.
You can try
type ConnWithBuffIncluded struct{ //Implement net.Conn so can be passed through pipeline
net.Conn
json.Decoder
}
func (x ConnWithBuffIncluded) Read(p []byte) (n int, err error){ //Will Read both sources
return io.MultiReader(x.Decoder.Buffered(), x.Conn).Read(p)
}
How do you know if a socket server or web server is done transmitting a HTTP GET request when using ProgressEvent.SOCKET_DATA ?
I doing my socket request with socket.writeUTFBytes('GET /index.php HTTP/1.1\r\n');
But the 'answer' is so big that i get multiple ProgressEvent.SOCKET_DATA. How do i know how much data it is supposed to transmit to me ? Or when it's done transmitting ?? Or even how many progressEvents i will get out of this request ? So far I'm using a timer that checks if the server is still transmitting but this isn't a very clean way of doing things..
How do i know how much data it is supposed to transmit to me? Or when it's done transmitting ??
By reading the Content-length header if that is sent by the server, or by waiting until the server closes the connection, or by reading until you've encountered a last-chunk (0<CRLF><CRLF>) if chunked transfer encoding is enabled, or any of the other indications that a full response has been received.
For simplicity, use a HTTPService or if that doesn't fit your needs, use a library that implements an HTTP client.
Or even how many progressEvents i will get out of this request ?
There is no way to tell.
I am working on Click fast prototyping router api created by Eddie Kohler for my research.
I am having hard time in generating tcpdump files with RAW IP ENCAP. I know that my link type doesn't allow to capture in RAW IP link-type as the only link type listing on typing tcpdump -i (interface) -L is EN10MB and nothing else.
It is just not striking my mind. Are there ways to capture Raw IP packets (dump packets should start from IP headers and skip link level headers) in tcpdump or tshark?
Neither tcpdump nor tshark can arbitrarily choose a link-layer header type for a device, as the devices, and thus libpcap/WinPcap, don't allow arbitrary link-layer header types to be chosen (as tcpdump -i {interface} -L informed you).
Most network interfaces don't support "raw IP" as a link-layer header type; I'm not sure which, if any, do. If you want a capture file full of packets with that encapsulation, the easiest way to do it is probably to write a program that captures from an interface, discards all non-IP packets, and strips the link-layer header off of the IP packets and writes them out. For example, if you're capturing on a device with a DLT_ value of DLT_EN10MB (Ethernet), just throw away all packets that don't have 0x0800 or 0x86DD as the Ethernet type value, and strip off the first 14 bytes of packets that do have those Ethernet type values and write the resulting packets out in a pcap file with a DLT_ value of DLT_RAW.