XDP packet capture

The Linux network stack is generic. It handles interfaces, routing, sockets, firewalling, packet delivery, and many other things. This is useful, but it has a cost. For the purpose of a high-traffic probe, this is a waste of CPU time.

XDP (eXpress Data Path) makes packets hop on the express train from the NIC to the userspace, avoiding the kernel network stack altogether.

The kernel stack cost

In the classical capture path, every packet first takes the normal receive path. It reaches the driver, becomes a kernel networking object, crosses the generic stack, then reaches a capture API such as AF_PACKET/libpcap.

Kernel packet processing path

This path is portable and easy to use, but it is costly when the packet rate is high:

  • sk_buff allocation and kernel stack work happen before filtering;
  • packets that will be ignored still consume CPU;
  • userspace sees borrowed packet buffers one packet at a time;
  • the parser starts working only after the generic stack has already done its work.

XDP express train

With XDP and AF_XDP, selected packets can take a much shorter path.

XDP packet processing path

In zero-copy mode, packet bytes are written by DMA directly into a UMEM area. The parser reads packet descriptors from AF_XDP rings and reads packet bytes from UMEM frames.

This has three important effects:

  • redirected packets skip the generic kernel network stack;
  • packet bytes can avoid an intermediate copy;
  • packet memory is reused from a fixed UMEM pool instead of allocated per packet.

eBPF filtering

The eBPF program lives at the XDP hook of the network driver. It runs as soon as the driver receives the packet, before the generic kernel stack starts.

Its job is to make the first cheap decision:

  • return XDP_DROP for unrelated traffic;
  • return XDP_REDIRECT for packets that should go to userspace through AF_XDP;
  • return XDP_PASS for packets that should keep using normal Linux networking (usually not needed in a high-traffic probe).

A typical XDP filter for a telecom probe can inspect just enough headers to make that decision:

  • inspect Ethernet, VLAN, IP, and transport headers;
  • identify protocols worth decoding, such as SCTP or GTP;
  • redirect accepted packets to an AF_XDP and drop the rest;
  • expose counters for dropped and redirected packets.

Why it is fast

XDP is fast because it removes work from the hot path:

  • useless packets are dropped before sk_buff allocation;
  • selected packets skip the generic kernel stack;
  • AF_XDP avoids an intermediate copy in zero-copy mode;
  • userspace uses a fixed ring of reusable packet buffers;
  • the probe parser receives fewer packets and spends more time on useful traffic.

The trade-off is setup and portability. XDP needs kernel and driver support, especially for native XDP and AF_XDP zero-copy. For high-rate passive probes, this trade-off is worth it.

Author Avatar

Jeremy Caradec