RDMA and libpcap

By Vincent Jardin, 6WIND CTO

A few days ago, we received the following question on the Forum “Is RDMA (Remote Direct Memory Access) anyhow related to libpcap, since packet processing is ignored in RDMA which is very much similar to libpcap?”

For readers who are not familiar with libpcap and RDMA, here’s a brief summary of these two technologies:

• RDMA (http://www.rdmaconsortium.org/) is a direct memory access of networking packets from the memory of a user process without the involvement of the operating system’s networking stacks. This permits high-throughput, low-latency networking, which is especially useful in massively parallel computer clusters. RDMA was designed for High Performance Computing to avoid wasting CPU cycles and minimizing latency for highly distributed computing; it provides a set of communication protocols that is described by some RFCs at IETF.
• libpcap (http://www.tcpdump.org/) is a portable C/C++ library for network traffic capture. libpcap provides a RAW interface API for most operating systems. Libpcap has a well-known bottleneck which is its kernel stack packet capture features that deliver the packets through a system call to this userland library.

So, there is not any direct relation between RDMA and libpcap. However, I think it could be possible to use RDMA to optimize libpcap because  RDMA provides an approach to bypass the kernel.

If you benchmark the I/O mechanism for almost all multicore processor environments, all I/Os are designed to be performed using a bypass method:

• Cavium Simple Exec on OCTEON
• Freescale LWE using DPAA and uDPAA on QorIQ
• Intel DPDK
• NetLogic NetOS on XLS/XLR/XLP
• Tilera IPP/EPP including ZoL™ on Tilepro CPU

Even without using the RDMA-IP related protocols, you can use NICs that support RDMA in order to optimize your implementation of libpcap and then to bypass the OS stack. With this approach, packets would be available faster to libpcap in userspace.