The Snowden Effect! (by Casey Mullis)
LMTV Sharkfest | TRANSUM How-To Part 2 - Troubleshooting Network and Application Performance Problems (by Paul Offord)

Dealing With the Data Deluge: Accelerating Packet Capture (by Dan Joe Barry)

Dealing With the Data Deluge: Accelerating Packet Capture

 The digital universe is doubling in size every two years, and digital data volume is projected to reach 44 trillion gigabytes by 2020, according to IDC. Not only is there exponentially more data than ever before, but it’s coming in faster than ever – at network speeds up to 100Gbps. This poses clear challenges for network professionals who need new types of tools to protect the network.

Data drowning

Packet capture (PCAP) is a fundamental tool to help engineers and administrators manage and secure large and small-scale networks alike. A mechanism for intercepting data packets that are traversing a computer network, PCAP is a common capability deployed within an organization to monitor security events and network performance, identify data leaks, troubleshoot issues and even perform forensic analysis to determine the impact of network breaches.

However, current PCAP systems using commodity network interface cards (NICs)  are not equipped to deal with the demands of performing precision capture and replay at 10/40/100 Gbps speeds.

A ray of light shines, though, to give hope to network administrators. There are solutions today that have been built to facilitate packet capture at speeds topping 100 Gbps. The use of network acceleration technology, coupled with open source network monitoring and capture solutions, can enable organizations to keep up with the demands of precision packet capture and replay on high-speed networks.

Analyzing with PCAP - 

These new solutions are able to give engineers and administrators an accurate, real-time view of what is happening within a network infrastructure. Likewise, precision PCAP systems also provide organizations with the ability to re-create network events with high fidelity for verification and validation of architectural changes, troubleshooting and analysis.

It is imperative, as you explore analysis and security solutions for high-speed networks, to consider the coupling of open source tools with the speed and accuracy of programmable logic. Here are three key factors when comparing your options:

  • Precision time stamping: Consider tools that offer hardware-based, high-precision time stamping with nanosecond resolution for every frame captured and transmitted. Hardware-based time stamping avoids the unpredictable latency inherent in software-based solutions and enables a communication flow to be recorded precisely as it occurs. Precision time protocol (PTP) can also be supported for accurate synchronization across distributed network probes.
  • High-speed Capture and Replay: Field Programmable Gate Array (FPGA)-based network acceleration cards (NACs) are ideal for performing high-speed packet capture and replay at a variety of speeds, including 1/10/40/100 Gbps. Moreover, NACs allow for precise inter-frame gap (IFG) control, which is critical when replaying captured traffic for troubleshooting or simulation of traffic flows.
  • Intelligent Data Flow: Look for solutions that have the capability to identify and direct traffic flows immediately upon ingress to maintain capture and analysis performance at high speeds. In doing so, the load on user-space applications can be minimized and administrators are provided with the ability to dynamically identify and direct data flows into specific CPU cores based on the type of traffic being analyzed.

How PCAP Has Been Done

Software tools have traditionally been responsible for performing packet capture and analysis on an organization’s network infrastructure. In this case, software is installed on a designated monitoring host and configured to poll packets from a commodity network adapter placed in promiscuous mode and connected to the network via a Switched Port Analyzer (SPAN) interface. A typical architecture for low-speed PCAP using a commodity network interface card (NIC) and libpcap is illustrated in figure 1 below:

Figure 1: Conventional PCAP Architecture -  

Figure 1_NEW

As you can see, each time the network adapter receives an Ethernet frame, it generates an interrupt request and copies the data from the memory buffer on the adapter into kernel space. Normally the kernel space driver would determine if the packet is intended for this host and either drop the packet or pass it up the protocol stack until it reaches the user-space application it is destined for. However, when configured for promiscuous mode, all packets are captured in a kernel buffer regardless of destination host. Once the kernel buffer is full, a context switch is performed to transfer the data to a user-space buffer managed by libpcap, a system-independent interface for user-level packet capture, so that the data can be accessed by user-level applications. 

User-level applications can’t see this intermediate buffer, which is necessary to prevent applications from accessing kernel-managed memory. Given this architecture, it is clear that some amount of time will lapse between when a frame is received by the adapter and actually delivered to the user-space application for processing.

This lag time is not significant to PCAP accuracy at low data rates, but at higher rates this latency is compounded and CPUs become saturated trying to keep pace with incoming data leading to capture loss and timing issues. Consider, for example, that a 1 Gbps network link can push around 1.5 million packets per second, or one packet every 670 nanoseconds. Conversely, at 10 and 100 Gbps speeds systems are processing one packet every 67 or 6.7 nanoseconds respectively.

It is already difficult, at these speeds, to simply capture traffic in a conventional architecture without the added complexity of precise timing, categorization, flow identification and filtering. Performing lossless, high-fidelity packet capture, replay and real-time analysis of data flows at these rates requires a different approach to PCAP, one that moves the bulk of the data processing out of the user-space and into the hardware while also eliminating the inefficiency of user-to-kernel space interactions.

Speeding Up PCAP

Using a hardware-accelerated approach, it is possible to achieve the goals of PCAP on high-speed networks. The targeted use of programmable logic coupled with open source tools allows data to be accurately captured and processed within a network acceleration card (NAC) before it is passed into user-space applications. Figure 2 illustrates what an accelerated PCAP architecture might look like.

Figure 2: Accelerated PCAP Architecture - 

  Figure 2_NEW

To perform in-line event processing and line-rate packet analysis in hardware at 1/10/40/100 Gbps speeds, high-performance NACs use Field Programmable Gate Arrays (FPGAs). Due to their programmable nature, FPGAs play an important role in, and are an ideal fit for, many different markets. These semiconductor devices are based around a matrix of configurable logic blocks (CLBs) connected via programmable interconnects. FPGAs can be reprogrammed to desired application or functionality requirements after manufacturing. Through the use of FPGA-based NACs, network administrators can immediately improve an organization’s ability to monitor and react to events that occur within its network infrastructure.

Line-rate packet analysis is leveraged here to push most of the frame processing into the hardware of the capture device, which can be deployed within a commodity server or workstation, preserving CPU cycles for higher-level analysis. This approach ensures that by the time data is passed to the user-space buffer for access by applications it has already been time stamped, categorized, and filtered appropriately. 

Robust and cost-effective solutions can be built for a variety of purposes when these hardware devices are combined with open source applications. In general, high-performance NACs enable easy in-house development of scalable, high-performance network applications over PCAP. Even complex payload analysis and network-wide correlation algorithms can be easily scaled by the effective flow-based load-balancing mechanism built-in to the NAC. The more complex analysis that the application performs, the more critical it is that the PCAP stream from the capture device has no packet drops and that the frames are in the correct order. Tasks like protocol reconstruction, reassembly, event detection and QoS calculations are severely impacted by insufficient PCAP performance.

Support for IEEE 1588, or Precision Time Protocol (PTP), is an important factor to consider. In doing so, precise time synchronization is maintained in a distributed deployment where multiple accelerated PCAP probes are deployed throughout a network infrastructure. This allows frames to be merged from multiple ports on multiple NACs into a single, time-ordered analysis stream. Maintaining this level of temporal fidelity within the capture ensures that organizations can perform retrospective analysis of network events by replaying data in exactly the same way as it was captured, complete with precise timing and inter-frame gap control.

The ability to perform a retrospective review of activity and to gain a real-time view of what is happening within a network is critical to understanding and measuring performance, identifying bottlenecks, troubleshooting issues and securing the environment. As such, packet capture and analysis continues to play a critical role in managing and securing large and small-scale networks.

No Signs of Slowing

Modern high-speed network fabrics are faster than the traditional ways of performing PCAP can keep up with. This leads to large quantities of dropped packet data and imprecise collections.

Such performance is unacceptable, so it is clear that PCAP must be accelerated. To meet today’s speeds up to 100 Gbps, captured packet processing must be moved to the point of ingress. This can be achieved by using hardware acceleration to maintain precise, lossless capture. Combining open source software deployed on commodity servers and programmable logic, engineers and network architects can future-proof their networks and ensure high-performance PCAP. 

Daniel Joseph Barry 630x788The authorDaniel Joseph Barry is VP Positioning and Chief Evangelist at Napatech and has over 20 years experience in the IT and Telecom industry. Prior to joining Napatech in 2009, Dan Joe was Marketing Director at TPACK, a leading supplier of transport chip solutions to the Telecom sector.  From 2001 to 2005, he was Director of Sales and Business Development at optical component vendor NKT Integration (now Ignis Photonyx) following various positions in product development, business development and product management at Ericsson. Dan Joe joined Ericsson in 1995 from a position in the R&D department of Jutland Telecom (now TDC). He has an MBA and a BSc degree in Electronic Engineering from Trinity College Dublin.