I am trying to do high speed packet sending on Linux using libpcap (~100mbps but >100k packets/second). The limitation appears to be (although I am not sure) the number of calls to pcap_send.
Winpcap has pcap_sendqueue_transmit, which allows multiple packets to be sent at once. I do not believe that libpcap has this function.
That being said, AFAIK, libpcap uses PF_PACKET https://stackoverflow.com/a/22996556/4381435. And packet mmap can peform this single system call send http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/networking/packet_mmap.txt?id=HEAD
So, is there some libpcap method of doing this that I can't find?
So, is there some libpcap method of doing this that I can't find?
No. You'd have to directly do the ring-buffer send yourself.
Related
I have a library which uses libpcap to capture packets. I'm using pcap_loop() in a dedicated thread for the capture and pcap_breakloop() to stop the capture.
The packet buffer timeout is set to 500ms.
In some rare cases I am missing the last packets that my application sends before calling pcap_breakloop().
Reading the libpcap documentation I ended up wondering if the packet loss is related to the packet buffer timeout. The documentation says:
packets are not delivered as soon as they arrive, but are delivered after a short delay (called a "packet buffer timeout")
What happens if pcap_breakloop() is called during this delay ? Are the packets in the buffer passed to the callback or are they dropped before pcap_loop() returns ?
I was unable to find the answer in the documentation.
Are the packets in the buffer passed to the callback
No.
or are they dropped before pcap_loop() returns ?
Yes. In capture mechanisms that buffer packets in kernel code and deliver them only when the buffer fills up or the timeout expires pcap_breakloop() doesn't force the packets to be delivered.
For some of those capture mechanisms there might be a way to force the timeout to, in effect, expire, but I don't know of any documented way to do that with Linux PF_PACKET sockets, BPF, or WinPcap/Npcap NPF.
Update, giving more details:
On Linux and Windows, pcap_breakloop() attempt to wake up anything that's blocked waiting for packets on the same pcap_t.
On Linux, this is implemented by having the poll() call in libpcap block on both the PF_PACKET socket being used for capturing and on an "event" descriptor; pcap_breakloop() causes the "event" descriptor to supply an event, so that the poll() wakes up even if there are no packets to pick up from the socket yet. That does not force the current chunk in the buffer (memory shared between the kernel and userland code) to be assigned to userland, so they're not provided to the caller of libpcap.
On Windows, with Npcap, an "event object" is used by the driver and Packet32 library (the libpcap part of Npcap calls routines in the Packet32 library) to allow the library to block waiting for packets and the driver to wake the library up when packets are available. pcap_breakloop() does a SetEvent() call on the handle for that object, which forces userland code waiting for packets to wake up; it then tries to read from the device. I'd have to spend more time looking at the driver code to see whether, if there are be buffered-but-not-delivered packets at that point, they will be delivered.
On all other platforms, pcap_breakloop() does not deliver a wakeup, as the capture mechanism either does no buffering or provides no mechanism to force a wakeup, so:
if no buffering is done, there's no packet buffer to flush;
if there's a timeout, code blocked on a read will be woken up when the timeout expires, and that buffer will be delivered to userland;
if there's no timeout, code blocked on a read could be blocked for an indefinite period of time.
The ideal situation would be if the capture mechanism provided, on all platforms that do buffering, a way for userland code to force the current buffer to be delivered, and thus to cause a wakeup. That would require changes to the NPF driver and Packet32 library in Npcap, and would require kernel changes in Linux, *BSD, macOS, Solaris, and AIX.
Update 2:
Note also that "break loop" means break out of the loop immediately, so even if all of the above were done, when the loop is exited, there might be packets remaining in libpcap's userland buffer. If you want those packets - even though, by calling pcap_breakloop(), you told libpcap "stop giving me packets" - you'll have put the pcap_t in non-blocking mode and call pcap_dispatch() to drain the userland buffer. (That won't drain the kernel buffer.)
I'd like to have pcap_dispatch() timeout if no packets are received within a set period of time. Similar to this SO question.
In the pcap(3) manpage, it says that not all platforms support that:
Not all platforms support a packet buffer timeout; on platforms that
don't, the packet buffer timeout is ignored. A zero value for the
timeout, on platforms that support a packet buffer timeout, will cause
a read to wait forever to allow enough packets to arrive, with no
timeout. A negative value is invalid; the result of setting the
timeout to a negative value is unpredictable.
And in this post, user862787 said that "Some OSes time out even if
no packets have arrived, others don't"
It's considered platform-specific because it is, but it's not
considered buggy (trust me, I'm the person who wrote that text in the
man page) - the timeout is to keep pcap_dispatch() from waiting
forever for a packet buffer to fill, not to keep it from waiting
forever for any packets to arrive at all. Some OSes time out even if
no packets have arrived, others don't. – user862787 Oct 19 '12 at
20:53
So how do I know which platforms support and which don't? I've searched and gone through the libpcap source but didn't find anything.
Specifically, what about Centos 8.1, kernel 4.18.0-147.el8.x86_64, libpcap 1.10 ?
On systems using the BPF capture mechanism - *BSD, macOS, Solaris 11, AIX - the timeout will occur even if no packets have arrived.
On most versions of most Linux distributions, it won't.
Don't depend on it doing so or not doing so; write your code not to depend on that.
I've searched and gone through the libpcap source but didn't find anything.
You need to look at the source for the capture mechanism libpcap uses on particular platforms, not at the libpcap source.
As all information I found about Qemu is related to Linux kernel, uboot or elf binaries I can't quite figure out how to load a binary blob from an embedded device into a specific address and execute part of it. The code I want to run does only arithmetics, so there are no hardware dependencies involved.
I would start qemu with something like
qemu-arm -singlestep -g8000
attach gdb, set initial register state and jump to my starting address to single step through it.
But how do I initially load binary data to a specific address and eventually set up an additional ram range?
how to load a binary blob from an embedded device into a specific address and execute part of it.
You can load binary blob into softmmu QEMU by the generic loader (-device loader).
I would start qemu with something like
qemu-arm -singlestep -g8000
This command line is for the linux-user QEMU invocation. It emulates userspace linux process of the guest architecture, it is unprivileged and does not provide support for any devices, including generic loader. Try using qemu-system-arm instead.
It's in fact easy with the Unicorn framework which works on top of Qemu. Based on the example in the websites doc section I wrote a Python script which loads the data, sets the registers, adds a hook which prints important per step information and start execution at the desired address until a target address.
I am writing monitoring program for a very high traffic network (HD videos are streamed through the network). Most packets are very large and I only want to watch the headers (IP and UDP/TCP only). Of course I want to avoid overhead of copying the entire data. Does libpcap necessarily give me a copy the whole packet? If yes, is there any library that matches my needs?
There appear to be two questions here:
the one in the title, which sounds as if it's asking whether libpcap copies the packet;
the one in the body, asking whether it always copies the entire packet.
For the first question:
There's probably at least one copy done by any code using the mechanisms atop which libpcap runs in various OSes - a copy from the mbufs/skbuff/STREAMS buffers/whatever to the mechanism's buffer. For Linux, when the tpacket mechanism is not being used, the skbuff might just be queued on the receive queue for the PF_PACKET socket libpcap is using.
There may be another copy - a copy from that buffer to userland; if libpcap is using a "zero-copy" mechanism, such as the Linux tpacket mechanism (which libpcap 1.0 and later use by default), the second copy doesn't happen. It will happen if a zero-copy mechanism isn't being used.
However, if you're using pcap_next() or pcap_next_ex() on a Linux system and the tpacket mechanism is being used, a separate copy, from the memory-mapped buffer to a private buffer; that doesn't happen if you use pcap_dispatch() or pcap_loop().
For the second question:
That's what the "snaplen" argument to pcap_open_live() and pcap_set_snaplen() is for - it lets you specify that no more than "snaplen" bytes of packet data should be captured, and that means that no more than that many bytes are copied.
Note that this length includes the link-layer headers, and that those can include "metadata" headers such as radiotap headers that you might get on 802.11 adapters. This header might be variable-length (for example, on 802.11, the 802.11 header is variable-length, and, if you're getting radiotap headers, those are variable-length as well).
In addition, both IPv4 and TCP headers can have options, and IPv6 packets can have extension headers, so the length of IP and TCP headers can also be variable.
This means that you might have to determine a "worst case" snapshot length to use; there's no way to explicitly say "don't give me anything past the TCP/UDP header", you can only say "give me no more than N bytes".
I heard this during a discussion that anti-viruses have this feature of stopping a program from executing by passing exceptions to it.
So how do you pass exceptions programmatically?
You can't "pass an exception" to another program. (I have at least never heard of any such terminology.)
Most operating systems allow one process to kill another by for instance sending it a signal, but that's different.