My passive network monitoring application needs packets to be captured from network interface (at higher packet rates). The packet capture module should be able to call a monitoring function upon capture of each packet (and also write the packet in to pcap file).
I thought of using DPDK as the packet capture module in my monitoring application (as we use pcap_loop and pfring_loop in libpcap and pfring respectively), but I am not sure whether this is one of the use cases of DPDK, or, is DPDK meant to be used like this?.
So my questions are..
Can I use DPDK to fulfill my requirements?, If yes how to start?.
OS: Linux.
Karnal version: 4.
DPDK version: Latest stable.
Capture on physical device.
The capturing application has root privileges and will be used by the network administrator (as part of passive asset discovery).
I want to use DPDK because it supports capture at line rate upto 10 Gbps
Thank you.
Based on the updates and clarification in comment the request is Can one replace an existing application which PF_RING API calls with DPDK API which is written in C?. Simple answer to it is yes it can be done.
Here is how one should start
identify the Platform (preferably Linux/BSD, windows 21.02 is still work in progress)
identify the processor list of supported CPU
Identify a NIC to use from LIST of DPDK NIC
Set up the Linux environment with Linux Enviroment
Explore basic example/skeleton for basicfwd usage
get the start of ethernet header for packet using DPDK API rte_pktmbuf_mtod. There are many samples in DPDK/examples folder which does the same.
Invoke the packet processing function logic between rx_burst and tx_burst of example/skeleton.
Newer versions of libpcap can themselves use DPDK, at least on Linux. The libpcap on your system might, or might not, be configured to use it. (There are also versions of libpcap modified to use PF_RING.)
Related
I made a simple Instant Message Chat Client and Server on TCP, that both run off Adobe AIR. It works great and it was a interesting way to learn basic networking programming.
My Question: Is it possible to change the data in the packet sent from the Chat Server before it arrives at the Client without using the Server or Client to do so? Like perhaps a program?
I am new to Network programming so I apologize if this is a dumb question.
Your question is very broad. So the answer is broad as well. Yes. It's possible.
For that you need to get the packets between the client and server to pass through a third program. There are quite a lot of ways to achieve that. Here's non-exhaustive list:
First, on your own machines (client/server) you could get access to the packet from the operating system using various low-level APIs. For instance iptables+nfqueue in Linux or the Windows Filtering Platform on Windows.
Second, you could get access to the packets by intentionally having them communicate through some proxy program which may or may not reside on the same server as the client or the server.
Third, you could get access to the packets by picking them up from the network itself. For instance, you could set up some Linux machine as a router and have it sit between the client and the server (as long as they're not on the same machine). That Linux machine will now have access to all of the packets that pass through it, and it can pass them to various user-space programs using hooks such as the previously mentioned nfqueue.
We are experiencing the issue when trying to connect to the cluster after updating the version of Java SDK.
The setup of the system is as follows:
We have a web application that is using Java SDK and a Couchbase cluster. In between we have a VIP (Virtual IP Address). We realise that isn’t ideal but we’re not able to change that immediately since VIP was mandated by Tech Ops. VIP is basically only there to reroute the initial request on application startup. That way we can make modifications on the cluster and ensure that when application starts it can find the cluster regardless of the actual nodes in the cluster and their IPs.
Prior to the issue we used JAVA SDK version 1.4.4. Our application would start and Java SDK would initiate a request on port 8091 to VIP. Please note that port 8091 is the only port open on VIP. VIP would reroute the request to one of the node cluster currently in use the cluster would respond to Java SDK. At that point Java SDK would discover all the nodes in the cluster and application would run fine. During up time if we would add, remove a node from the cluster Java SDK would update automatically and everything would run without the issue.
In the last sprint we updated the Java SDK to version 2.1.3. Our application would start and Java SDK would initiate a request on port 11210 to VIP. Since this port is not open the request would fail and Java SDK would throw an exception:
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException
at com.couchbase.client.java.util.Blocking.blockForSingle(Blocking.java:93)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:108)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:99)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:89)
No further request would be made on any port.
It appears the order in which port are being used has been changed between versions. Could somebody please confirm, or dispute, that the order in which ports are being used for cluster discovery has been changed between versions. Also could somebody please provide some advice on how we could resolve the issue. We are trying to understand the clients behavior, if we could open all those ports on the VIP would the client still then function correctly and at full performance?
The issue is happening on our production environment which we cannot use for testing out potential solutions since it will interfere with our products.
In v2.x of the Java SDK, it defaults to 11210 to get the cluster map to bootstrap the application. This is a huge improvement actually as now the map comes from the managed cache and not the cluster manager (8091). The SDK should use 8091 as a fall back if it cannot get the map on 11210 though. Regardless, you really want to get that map from 11210, trust me. It cleans up a lot of problems.
To resolve this long term and follow Couchbase best practices, upgrade to the Java 2.2.x SDK, get rid of the VIP entirely and go with a DNS SRV record instead. That gives you one DNS entry for the SDK connection object and you just manage the node list in DNS. It works great. I say SDK 2.2 as the DNS SRV record solution is fully supported there, in 2.1 it is experimental. VIPs are specifically recommended against by Couchbase these days. In older versions of the SDKs it was fine to do this and it helped with limiting the number of connections from the app to the DB nodes, but that is no longer necessary and can actually be a bad thing.
in addition to Kirk's long term answer (which I also advise you to follow), a shorter term solution may be to deactivate the 11210 bootstraping (carrier bootstrap) through the CouchbaseEnvironment by calling bootstrapCarrierEnabled(false) on the builder.
I don't guarantee that it'll work with a vIP even after that, but that may be worth a try if you're in a hurry.
I have done a fair amount of research these last few weeks trying to create a connection diagnostic tool, I don't so much want to just check to see if the connection is available but to diagnose if there is jitter, packet loss, etc..
So far it seems that Java doesn't support a true ICMP request and that there are a few workarounds out there but none of which achieve what I'm trying to do.
Does anyone know if this sort of tool can be built or should I start looking into other options?
It seems that InetAdress is using ICMP when its possible:
https://docs.oracle.com/javase/7/docs/api/java/net/InetAddress.html
take a look at public boolean isReachable(int timeout)
Test whether that address is reachable. Best effort is made by the
implementation to try to reach the host, but firewalls and server
configuration may block requests resulting in a unreachable status
while some specific ports may be accessible. A typical implementation
will use ICMP ECHO REQUESTs if the privilege can be obtained,
otherwise it will try to establish a TCP connection on port 7 (Echo)
of the destination host.
You can find a simple use example here:
How to test if a remote system is reachable
or here How to ping an IP address
I believe ICMP4J does exactly what you need: Internet Control Message Protocol for Java
You can use 'Exec' to run ping at the command line (assuming your OS supports this), or JNI to interface to a native application to do the pinging.
Creating your own implementation of the ICMP protocol would not be trivial.
If you do use Exec be aware of it's limitations which are not always obvious in initial testing:
http://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html
I was getting a bit confused on using of raw sockets and libcap. Can, anyone in simple point out advantages of using both. I read few links, but, its making me confused.
libpcap uses different mechanisms on different operating systems. On Linux, it uses PF_PACKET raw or cooked sockets, depending on whether it knows about the Linux link-layer type (ARPHRD_ value) for the interface and whether interfaces of that link-layer type produce a useful link-layer header (PPP interfaces don't, so you can't see the network-layer protocol identifier). On Irix, it uses PF_RAW sockets with the RAWPROTO_SNOOP protocol. On other systems, it doesn't use raw sockets at all.
Raw sockets can be used for a number of purposes, such as sending and receiving raw IPv4 packets without having to worry about the link-layer (i.e., they plug into the IP layer rather than into the network device driver). If you need access to the raw link layer, raw sockets on most OSes don't support that (Linux and Irix being obvious exceptions, as per the previous paragraph), but libpcap does.
"libpcap" and "raw sockets" are different topics.
Libpcap is a packet capture library for linux which is used to capture the traffic/packets that pass through a network interface like eth0.
On windows the Winpcap library does the same thing.
Raw sockets are a feature of the socket api provided by the OS that can be used to send packets with headers defined by the application and not the OS. So using raw sockets we can specify the IP , TCP headers and send the packets.
Raw sockets are available on Linux since all time. On Windows raw sockets were available only in Windows XP and Windows XP(SP1).
On windows the winpcap library has a feature to send packets with arbitrary content , which means raw socket functionality can be achieved with winpcap on Windows.
I have different sites running with 4 to 5 server at each location. All the locations have one monitoring server with Nagios. Now I want to create a central location and want to combine all the nagios services running at each location. Can anyone please point me to some documentation for these type of jobs.
There are two approaches that you can take.
Install a new Nagios core as you did at each location and perform active checks on each of the remote hosts. You'll likely end up installing NRPE on each of the remote hosts at each location and can read this document for the details: http://nagios.sourceforge.net/docs/nrpe/NRPE.pdf. If your remote servers are Windows servers, you can use NSClient to much of the same things that NRPE does for Linux hosts. This effectively centralizes your monitoring server. I also wrote some how-to style entries for using NRPE to run privileged commands http://blog.gnucom.cc/?p=479 or to run event handlers http://blog.gnucom.cc/?p=458. If you get tired of installing NRPE, you can use my script here http://blog.gnucom.cc/?p=185. I also have instructions to install NSClient here http://blog.gnucom.cc/?p=201.
Install a new Nagios core as you did at each location and perform passive checks by instructing the remote Nagios cores to feed their results to the new central Nagios core's passive command file. I haven't done this myself, so I'm going to point you to the communities documentation here http://nagios.sourceforge.net/docs/2_0/passivechecks.html. You could probably look at my event handler post to set up event handlers that send checks to the main server.
From my personal experience, the first option I mentioned is easier to implement, and is far easy to administer. However, as your server fleet grows you'll start seeing major CPU bottlenecks with the main Nagios core. This is where passive checks would become beneficial, as the main Nagios core simply waits for critical checks to be sent to it rather than having to check them itself.
Hope this helps. :)
A centralized view tool may be what you are looking for. There are a number of different options available.
Nagiosfusion
MK Livestatus
Nagcen
Thruk