Remote Probe Architecture

Overview

The WhiteOwl remote probe system enables distributed network monitoring by deploying lightweight probe agents to remote locations. Probes connect back to the central WhiteOwl server to receive test assignments and report results.

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────┐
│                           Central Server                                 │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐    │
│  │   Backend   │  │  Synthetics │  │  PostgreSQL │  │  ClickHouse │    │
│  │  (Node.js)  │  │    (Go)     │  │   (Config)  │  │  (Metrics)  │    │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘    │
│         │                │                │                │            │
│         └────────────────┴────────────────┴────────────────┘            │
│                                   │                                      │
│                              Port 3001                                   │
└───────────────────────────────────┼──────────────────────────────────────┘
                                    │
                    ┌───────────────┼───────────────┐
                    │               │               │
            ┌───────▼───────┐ ┌─────▼─────┐ ┌──────▼──────┐
            │  Probe Site A │ │ Probe AWS │ │ Probe DC-2  │
            │ 192.168.1.x   │ │ 3.x.x.x   │ │ 10.0.0.x    │
            └───────────────┘ └───────────┘ └─────────────┘

---

Authentication

Probes authenticate with the central server using API keys.

How It Works

1. Key Generation — When a probe registers for the first time, the central server generates a unique 256-bit API key
2. Key Storage — The key is encrypted at rest in the central database using AES-256-GCM encryption
3. Probe Storage — The probe stores its key locally in config.yaml
4. Request Authentication — Every API request from the probe includes the key in the Authorization header

Security Features

• Unique per probe — Each probe has its own API key
• Encrypted at rest — Keys are encrypted in the database, not stored in plaintext
• Automatic generation — Keys are generated during registration, no manual setup required
• Re-registration support — If a probe reconnects with the same name and IP, it receives its existing key

Rotating API Keys

To rotate a probe's API key:

1. Delete the probe from Settings → Devices
2. Re-deploy the probe from Probe Deployment
3. A new API key will be generated automatically

Troubleshooting Authentication

If a probe shows 401 errors:

# Check if the probe has an API key configured
grep api_key /opt/chompy-probe/config.yaml

If the key is missing or empty, re-register the probe by restarting the service:

sudo systemctl restart chompy-probe

---

Remote Probe Components

Component	Technology	Purpose
Binary	Go	Single compiled binary for easy deployment
Packet Capture	gopacket/libpcap	NetFlow-style flow capture with SNI extraction
Synthetics Runner	Go	Execute assigned ping/http/dns/traceroute tests
Flow Exporter	Go	Send flow data to central ClickHouse

---

Probe Registration Flow

┌──────────────┐                           ┌──────────────┐
│    Probe     │                           │   Central    │
└──────┬───────┘                           └──────┬───────┘
       │                                          │
       │  1. POST /api/probe/register             │
       │  {probe_name, sampler_address, site_id}  │
       │─────────────────────────────────────────>│
       │                                          │
       │                              2. Check if probe exists
       │                              3. Generate API key (if new)
       │                              4. Store in PostgreSQL
       │                                          │
       │  5. Response: {device_id, api_key}       │
       │<─────────────────────────────────────────│
       │                                          │
       │  6. Store credentials locally            │
       │                                          │
       │  7. Start packet capture                 │
       │  8. Start synthetics runner              │
       │                                          │

Registration Scenarios

New Probe:

1. Probe sends registration request
2. Central creates new device entry with generated API key
3. Returns device_id and api_key
4. Probe stores credentials in config

Reconnecting Probe (same name + IP):

1. Probe sends registration request
2. Central finds existing device by name
3. Validates sampler_address matches
4. Returns existing device_id and api_key (generates if missing)
5. Preserves site assignment

Duplicate Name Rejection:

1. Probe sends registration with existing name but different IP
2. Central rejects with 409 Conflict
3. User must use unique name or remove existing device

---

Request/Response Examples

Register Probe

Request:

POST /api/probe/register
Content-Type: application/json

{
  "probe_name": "probe-dc1-rack5",
  "sampler_address": "10.0.5.100",
  "site_id": 7,
  "mgmt_ip": "10.0.5.100",
  "probe_version": "1.0.1"
}

Response:

{
  "success": true,
  "device_id": 57,
  "api_key": "a1b2c3d4e5f6...",
  "site_id": 7,
  "is_reregistration": false
}

---

Synthetic Test Execution Flow

Test Assignment (UI)

┌─────────────────────────────────────────────────────────────┐
│                    Synthetics UI                             │
│                                                              │
│  Test: "Google DNS Check"                                    │
│  Target: 8.8.8.8                                            │
│  Type: ICMP                                                  │
│                                                              │
│  Assigned Probes:                                           │
│  ☑ Central                                                  │
│  ☑ probe-192-168-100-178                                    │
│  ☐ probe-aws-us-east                                        │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Execution Flow

1. Test created in ClickHouse (synthetic_tests)

2. Probe assignments stored in synthetic_test_probes:

   • test_id: "e96ec6a9-..."
   • probe_device_id: 0 (Central)
   • probe_device_id: 57 (remote probe)

3. Central Synthetics Service:

   • Checks HasAnyProbeAssignment() for each test
   • If assigned to probes, checks IsTestAssignedToProbe(testID, 0)
   • Only runs if Central (device_id=0) is assigned
   • Skips tests assigned only to remote probes

4. Remote Probe:

   • Polls GET /api/probes/57/synthetic-tests every 30s
   • Receives only tests where probe_device_id=57 is enabled
   • Executes tests (ping, traceroute, http, dns)
   • Posts results to /api/probes/synthetic-results/{type}

---

Probe Configuration

Config File (config.yaml)

# Probe identification
probe_name: probe-192-168-100-178
site_name: Dallas
site_id: 7

# Central server connection
api_endpoint: http://192.168.100.132:3001

# Packet capture settings
capture:
  interface: ens33          # or "any" for all interfaces
  snaplen: 128
  promisc: true
  buffer_mb: 64
  sni_extraction: true

# Flow export settings
flow_export:
  destination: 192.168.100.132:9995
  protocol: netflow_v9
  active_timeout: 60
  idle_timeout: 15

# Synthetics settings
synthetics:
  enabled: true
  poll_interval: 30s

# Logging
log_level: info

---

Deployment

The Probe UI offers selections for SNMP, traceroute, and WireGuard. During installation of the probe, if these services do not exist the system will deploy them.

The system also creates a systemd service on the remote host.

Systemd Service

# /etc/systemd/system/chompy-probe.service
[Unit]
Description=Chompy Network Probe - {{ probe_name }}
Documentation=https://whiteowlnetworks.com/docs/probe
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
ExecStart=/opt/chompy-probe/chompy-probe -config /opt/chompy-probe/config.yaml
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=chompy-probe
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Required Dependencies

Package	Purpose
libpcap	Packet capture
traceroute	ICMP traceroute tests

---

Troubleshooting

401 Authentication Errors

If the API key is empty, the probe was created via UI without a key.

Solution: Re-register probe or manually set key.

Probe Not Receiving Tests

journalctl -u chompy-probe | grep "Fetched"

Traceroute Missing/Truncated

# Check traceroute is installed
which traceroute

Timestamps Incorrect

Verify timestamp format (should be: 2026-01-10 02:15:30.123). ISO format with T and Z needs conversion for ClickHouse DateTime64.