Npcap internals

Abstract

Describes the internal structure and interfaces of Npcap: the NPF driver and Packet.dll

This portion of the manual describes the internal structure and interfaces of Npcap, starting from the lowest-level module. It is targeted at people who must extend or modify this software, or to the ones interested in how it works. Therefore, developers who just want to use Npcap in their software don't need to read it.

Npcap structure

Npcap is an architecture for packet capture and network analysis for the Win32 platforms. It includes a kernel-level packet filter, a low-level dynamic link library (packet.dll), and a high-level and system-independent library (wpcap.dll).

Why do we use the term architecture rather than library? Because packet capture is a low level mechanism that requires a strict interaction with the network adapter and with the operating system, in particular with its networking implementation, so a simple library is not sufficient.

Main components of Npcap.

First, a capture system needs to bypass the operating systems's protocol stack in order to access the raw data transiting on the network. This requires a portion running inside the kernel of OS, interacting directly with the network interface drivers. This portion is very system dependent, and in our solution it is realized as a device driver, called Netgroup Packet Filter (NPF); This driver offers basic features like packet capture and injection, as well as more advanced ones like a programmable filtering system and a monitoring engine. The filtering system can be used to restrict a capture session to a subset of the network traffic (e.g. it is possible to capture only the ftp traffic generated by a particular host); the monitoring engine provides a powerful but simple to use mechanism to obtain statistics on the traffic (e.g. it is possible to obtain the network load or the amount of data exchanged between two hosts).

Second, the capture system must export an interface that user-level applications will use to take advantage of the features provided by the kernel driver. Npcap provides two different libraries: packet.dll and wpcap.dll.

Packet.dll offers a low-level API that can be used to directly access the functions of the driver, with a programming interface independent from the Microsoft OS.

Wpcap.dll exports a more powerful set of high level capture primitives that are compatible with libpcap, the well known Unix capture library. These functions enable packet capture in a manner that is independent of the underlying network hardware and operating system.

Npcap driver internals

This section documents the internals of the Netgroup Packet Filter (NPF), the kernel portion of Npcap. Normal users are probably interested in how to use Npcap and not in its internal structure. Therefore the information present in this module is destined mainly to Npcap developers and maintainers, or to the people interested in how the driver works. In particular, a good knowledge of OSes, networking and Windows kernel programming and device drivers development is required to profitably read this section.

NPF is the Npcap component that does the hard work, processing the packets that transit on the network and exporting capture, injection and analysis capabilities to user-level.

The following paragraphs will describe the interaction of NPF with the OS and its basic structure.

NPF and NDIS

NDIS (Network Driver Interface Specification) is a standard that defines the communication between a network adapter (or, better, the driver that manages it) and the protocol drivers (that implement for example TCP/IP). Main NDIS purpose is to act as a wrapper that allows protocol drivers to send and receive packets onto a network (LAN or WAN) without caring either the particular adapter or the particular Win32 operating system.

NDIS supports four types of network drivers:

Miniport drivers. Miniport drivers directly manage network interface cards, referred to as NICs. The miniport drivers interface directly to the hardware at their lower edge and at their upper edge present an interface to allow upper layers to send packets on the network, to handle interrupts, to reset the NIC, to halt the NIC and to query and set the operational characteristics of the driver.
Miniport drivers implement only the hardware-specific operations necessary to manage a NIC, including sending and receiving data on the NIC. Operations common to all lowest level NIC drivers, such as synchronization, is provided by NDIS. Miniports do not call operating system routines directly; their interface to the operating system is NDIS.
A miniport does not keep track of bindings. It merely passes packets up to NDIS and NDIS makes sure that these packets are passed to the correct protocols.
Intermediate drivers. Intermediate drivers interface between an upper-level driver such as a protocol driver and a miniport. To the upper-level driver, an intermediate driver looks like a miniport. To a miniport, the intermediate driver looks like a protocol driver. An intermediate protocol driver can layer on top of another intermediate driver although such layering could have a negative effect on system performance. A typical reason for developing an intermediate driver is to perform media translation between an existing legacy protocol driver and a miniport that manages a NIC for a new media type unknown to the protocol driver. For instance, an intermediate driver could translate from LAN protocol to ATM protocol. An intermediate driver cannot communicate with user-mode applications, but only with other NDIS drivers.
Filter drivers. Filter drivers can monitor and modify traffic between protocol drivers and miniport drivers like an intermediate driver, but are much simpler. They have less processing overhead than intermediate drivers.
Transport drivers or protocol drivers. A protocol driver implements a network protocol stack such as IPX/SPX or TCP/IP, offering its services over one or more network interface cards. A protocol driver services application-layer clients at its upper edge and connects to one or more NIC driver(s) or intermediate NDIS driver(s) at its lower edge.

NPF is implemented as a filter driver. In order to provide complete access to the raw traffic and allow injection of packets, it is registered as a modifying filter driver in the compression FilterClass.

Notice that the various Windows operating systems have different versions of NDIS: NPF is NDIS 6.0 compliant, and so requires a Windows OS that supports NDIS 6.0: Windows Vista or later.

NPF structure basics

NPF is able to perform a number of different operations: capture, monitoring, packet injection. The following paragraphs will describe shortly each of these operations.

Packet Capture

The most important operation of NPF is packet capture. During a capture, the driver sniffs the packets using a network interface and delivers them intact to the user-level applications.

The capture process relies on two main components:

A packet filter that decides if an incoming packet has to be accepted and copied to the listening application. Most applications using NPF reject far more packets than those accepted, therefore a versatile and efficient packet filter is critical for good over-all performance. A packet filter is a function with boolean output that is applied to a packet. If the value of the function is true the capture driver copies the packet to the application; if it is false the packet is discarded. NPF packet filter is a bit more complex, because it determines not only if the packet should be kept, but also the amount of bytes to keep. The filtering system adopted by NPF derives from the BSD Packet Filter (BPF), a virtual processor able to execute filtering programs expressed in a pseudo-assembler and created at user level. The application takes a user-defined filter (e.g. “pick up all UDP packets”) and, using wpcap.dll, compiles them into a BPF program (e.g. “if the packet is IP and the protocol type field is equal to 17, then return true”). Then, the application uses the BIOCSETF IOCTL to inject the filter in the kernel. At this point, the program is executed for every incoming packet, and only the conformant packets are accepted. Unlike traditional solutions, NPF does not interpret the filters, but it executes them. For performance reasons, before using the filter NPF feeds it to a JIT compiler that translates it into a native 80x86 function. When a packet is captured, NPF calls this native function instead of invoking the filter interpreter, and this makes the process very fast. The concept behind this optimization is very similar to the one of Java jitters.
A circular buffer to store the packets and avoid loss. A packet is stored in the buffer with a header that maintains information like the timestamp and the size of the packet. Moreover, an alignment padding is inserted between the packets in order to speed-up the access to their data by the applications. Groups of packets can be copied with a single operation from the NPF buffer to the applications. This improves performances because it minimizes the number of reads. If the buffer is full when a new packet arrives, the packet is discarded and hence it's lost. Both kernel and user buffer can be changed at runtime for maximum versatility: packet.dll and wpcap.dll provide functions for this purpose.

The size of the user buffer is very important because it determines the maximum amount of data that can be copied from kernel space to user space within a single system call. On the other hand, it can be noticed that also the minimum amount of data that can be copied in a single call is extremely important. In presence of a large value for this variable, the kernel waits for the arrival of several packets before copying the data to the user. This guarantees a low number of system calls, i.e. low processor usage, which is a good setting for applications like sniffers. On the other side, a small value means that the kernel will copy the packets as soon as the application is ready to receive them. This is excellent for real time applications (like, for example, ARP redirectors or bridges) that need the better responsiveness from the kernel. From this point of view, NPF has a configurable behavior, that allows users to choose between best efficiency or best responsiveness (or any intermediate situation).

The wpcap library includes a couple of system calls that can be used both to set the timeout after which a read expires and the minimum amount of data that can be transferred to the application. By default, the read timeout is 1 second, and the minimum amount of data copied between the kernel and the application is 16K.

Packet injection

NPF allows to write raw packets to the network. To send data, a user-level application performs a WriteFile() system call on the NPF device file. The data is sent to the network as is, without encapsulating it in any protocol, therefore the application will have to build the various headers for each packet. The application usually does not need to generate the FCS because it is calculated by the network adapter hardware and it is attached automatically at the end of a packet before sending it to the network.

In normal situations, the sending rate of the packets to the network is not very high because of the need of a system call for each packet. For this reason, the possibility to send a single packet more than once with a single write system call has been added. The user-level application can set, with an IOCTL call (BIOCSWRITEREP), the number of times a single packet will be repeated: for example, if this value is set to 1000, every raw packet written by the application on the driver's device file will be sent 1000 times. This feature can be used to generate high speed traffic for testing purposes: the overload of context switches is no longer present, so performance is remarkably better.

Network monitoring

Npcap offers a kernel-level programmable monitoring module, able to calculate simple statistics on the network traffic. Statistics can be gathered without the need to copy the packets to the application, that simply receives and displays the results obtained from the monitoring engine. This allows to avoid great part of the capture overhead in terms of memory and CPU clocks.

The monitoring engine is made of a classifier followed by a counter. The packets are classified using the filtering engine of NPF, that provides a configurable way to select a subset of the traffic. The data that pass the filter go to the counter, that keeps some variables like the number of packets and the amount of bytes accepted by the filter and updates them with the data of the incoming packets. These variables are passed to the user-level application at regular intervals whose period can be configured by the user. No buffers are allocated at kernel and user level.