Skip to main content

Deep Packet Inspection (DPI)

WhiteOwl Networks integrates nDPI for real-time application-layer traffic classification. When enabled on a probe, DPI identifies over 280 application protocols and 30+ traffic categories directly from packet payloads, providing visibility beyond traditional port-based classification.

Overview​

Traditional NetFlow/IPFIX relies on port numbers to identify applications — port 443 is "HTTPS," port 53 is "DNS." This breaks down with modern traffic where most applications tunnel over HTTPS. DPI solves this by inspecting packet payloads to identify the actual application: Google, YouTube, Spotify, Slack, Zoom, and hundreds more.

How It Works​

┌─────────┐     IPFIX + DPI IDs      ┌─────────┐     Kafka      ┌──────────┐     Insert      ┌────────────┐
│  Probe  │ ──────────────────────── │ goflow2 │ ────────────── │ Enricher │ ──────────────── │ ClickHouse │
│  (nDPI) │   PEN 99999 fields       │         │   protobuf     │          │   UInt16 IDs     │            │
└─────────┘                          └─────────┘                └──────────┘                  └────────────┘
                                                                                                    │
                                                                                              dictGet()
                                                                                                    │
                                                                                              ┌────────────┐
                                                                                              │ Dictionary │
                                                                                              │  (CSV)     │
                                                                                              └────────────┘

The probe classifies each flow using nDPI and exports numeric protocol and category IDs as IPFIX enterprise fields. These IDs flow through the pipeline as lightweight integers. At query time, ClickHouse dictionaries resolve the IDs to human-readable names like "YouTube" or "Spotify."

Key Design Decisions​

Numeric IDs over strings: The probe sends app_protocol_id (UInt16) and app_category_id (UInt16) rather than protocol name strings. This avoids goflow2's limitation with enterprise string fields in protobuf and reduces bandwidth and storage overhead.

ClickHouse dictionaries for name resolution: Protocol and category names are resolved at query time using dictGet(). This means zero enricher overhead, easy updates when nDPI versions change, and clean separation between data storage and presentation.

Early classification with packet budget: DPI classification happens within the first 10 packets of each flow. Once nDPI reaches a conclusion (or exhausts its packet budget), the result is cached for the flow's lifetime.

Probe Configuration​

DPI is enabled per-probe using the dpi: true flag in the config.yaml, which can be enable in the UI or using a command line:

config.yaml
features:
  packet_capture: true
  synthetics: true
  dpi: true

command line
./chompy-probe -interface eth0 -dpi -collector 192.168.1.100 -port 2055

Requirements​

The probe host must have the nDPI library installed:

# Ubuntu/Debian
sudo apt install libndpi4.2t64 libndpi-dev

# RHEL/CentOS
sudo yum install nDPI-devel

The probe uses CGo bindings to call nDPI's C library directly, so the shared library must be available at runtime.

IPFIX Enterprise Fields​

DPI data is exported as enterprise IPFIX fields under Private Enterprise Number (PEN) 99999:

Field IDPENNameTypeDescription
90099999app_protocol_idUInt16nDPI protocol identifier (0-281)
90199999app_category_idUInt16nDPI category identifier (0-32)

These fields are appended to the standard IPFIX flow record alongside existing enterprise fields for TCP performance metrics (RTT, retransmits, TCP window sizes, microburst detection).

Dashboard Widgets​

DPI fields are available in the widget builder as "Application Protocol" and "Application Category." These work with all widget types: tables, bar charts, pie charts, and time series.

Widget Filter Examples​

Filter by specific application:

  • Field: Application Protocol, Operator: equals, Value: YouTube

Filter by category:

  • Field: Application Category, Operator: equals, Value: Streaming

Supported Protocols​

nDPI 4.2 supports 281 application protocols across 33 categories. Some commonly seen protocols:

Web: Google, Amazon, Cloudflare, HTTP, QUIC, TLS
Media: YouTube, Netflix, Hulu, Spotify, Twitch
Social: Facebook, Twitter, Instagram, Reddit, LinkedIn
Cloud: AmazonAWS, Microsoft Azure, Google Services
Communication: Zoom, MSTeams, Slack, Discord, WhatsApp, Telegram
Network: DNS, SNMP, ICMP, NTP, DHCP, NetFlow, MDNS
Security: TLS, SSH, OpenVPN, WireGuard, IPsec, Tor

For the complete list, see the ndpi_protocols.csv file in clickhouse/dictionaries/.

Verifying DPI on the Probe​

Run the probe in debug mode to see DPI classifications:

sudo ./chompy-probe -interface eth0 -dpi -debug

Look for log lines like:

DPI: 192.168.1.10:54321 -> 142.250.80.46:443 = Google (Web) [id:126 cat:5]

Common Issues​

All flows show "Unknown": The probe may not have the -dpi flag enabled, or the nDPI library is not installed on the probe host.

Dictionary not found: Ensure the CSV files are mounted into ClickHouse and the dictionary creation SQL has been run.