docs(04): research phase BPF filter and pcap file reading
This commit is contained in:
@@ -0,0 +1,541 @@
|
||||
# Phase 4: Power User Features - Research
|
||||
|
||||
**Researched:** 2026-03-26
|
||||
**Domain:** BPF filter integration, pcap file reading, timestamp-based aggregation
|
||||
**Confidence:** HIGH
|
||||
|
||||
## Summary
|
||||
|
||||
Phase 4 adds two additive CLI features to an already working pipeline: a `--filter` flag that scopes live capture (or offline replay) using BPF syntax, and a `--read` flag that replaces the live capture source with an existing pcap file. Both features wire into the existing classify → aggregate → synthesis pipeline without touching audio synthesis, classification rules, or MP3 encoding.
|
||||
|
||||
The key insight from reading the actual library source in the module cache is that go-pcap's `Handle.SetBPFFilter(expr string) error` is available as a first-class method; it compiles a tcpdump-syntax BPF expression into kernel BPF instructions and attaches them to the socket. For pcap file reading, `gopacket/pcapgo.NewReader(r io.Reader)` reads standard pcap (v2.4) files and returns a `*Reader` whose `ReadPacketData()` method satisfies `gopacket.PacketDataSource` — so it plugs directly into `gopacket.NewPacketSource`, exactly as the live `go-pcap` handle does. Timestamp-based windowing for offline mode replaces the real-time `time.Ticker` with arithmetic over `gopacket.CaptureInfo.Timestamp`.
|
||||
|
||||
**Primary recommendation:** Add `SetBPFFilter` to `OpenCapture` and add a new `ReadPcapFile` function in `capture/capture.go` that returns the same `<-chan gopacket.Packet` channel as `StartCapture`. Add a `AggregatePcap` function in `aggregate/window.go` that uses packet timestamps instead of a ticker. Wire both paths in `main.go`.
|
||||
|
||||
<user_constraints>
|
||||
## User Constraints (from CONTEXT.md)
|
||||
|
||||
### Locked Decisions
|
||||
- **D-01:** Use packet timestamps from the pcap file to assign packets to 500ms time windows. Processing is fast — a 10-minute pcap produces a 10-minute MP3 in seconds.
|
||||
- **D-02:** Preserve time gaps as silence. Windows with zero packets during gaps produce silent audio sections. MP3 duration faithfully matches the pcap file's time span.
|
||||
- **D-03:** `--read` and `-i` are mutually exclusive. Error if both provided. `--read` replaces `-i` as the packet source.
|
||||
- **D-04:** `--filter` works with both `-i` (live capture) and `--read` (pcap file). BPF filter applies to whichever packet source is active.
|
||||
- **D-05:** `--read` without `-o` derives output filename from input: `capture.pcap` → `capture.mp3`.
|
||||
- **D-06:** Bookend messages: `"Reading <file>..."` at start, then protocol summary + `"Saved"` line at end. No progress bar.
|
||||
- **D-07:** `--verbose` works with `--read` — per-window protocol activity lines scroll by quickly. Consistent behavior regardless of packet source.
|
||||
|
||||
### Claude's Discretion
|
||||
- BPF filter validation approach (pre-validate before opening capture vs let go-pcap/gopacket reject it)
|
||||
- Pcap file format detection and error messages for corrupt/unreadable files
|
||||
- Implementation of timestamp-based windowing (new aggregation path vs adapter that feeds existing Aggregate())
|
||||
- How to handle pcap files with no packets (reuse existing zero-packet guard from OUT-03)
|
||||
|
||||
### Deferred Ideas (OUT OF SCOPE)
|
||||
None — discussion stayed within phase scope
|
||||
</user_constraints>
|
||||
|
||||
<phase_requirements>
|
||||
## Phase Requirements
|
||||
|
||||
| ID | Description | Research Support |
|
||||
|----|-------------|------------------|
|
||||
| CAPT-05 | User can filter captured traffic using BPF syntax via `--filter` flag | `Handle.SetBPFFilter(expr)` on go-pcap handle; `filter.NewExpression(expr).Compile()` for pre-validation without a live socket; errors surface before capture begins |
|
||||
| CAPT-06 | User can sonify a pcap file instead of live traffic via `--read` flag | `pcapgo.NewReader(f)` + `gopacket.NewPacketSource` gives same packet channel interface; `CaptureInfo.Timestamp` provides per-packet timestamps for window assignment per D-01/D-02 |
|
||||
</phase_requirements>
|
||||
|
||||
## Standard Stack
|
||||
|
||||
### Core (all already in go.mod — no new dependencies)
|
||||
| Library | Version | Purpose | Why Standard |
|
||||
|---------|---------|---------|--------------|
|
||||
| `github.com/packetcap/go-pcap` | v0.0.0-20251215 | BPF filter on live handle | Already used; `Handle.SetBPFFilter(string) error` is a first-class method |
|
||||
| `github.com/gopacket/gopacket/pcapgo` | included in gopacket v1.5.0 | Pcap file reading | Part of gopacket; `pcapgo.NewReader(io.Reader)` is the canonical pure-Go pcap file reader |
|
||||
| `github.com/gopacket/gopacket` | v1.5.0 | Packet decoding from pcapgo reader | Already used; `gopacket.NewPacketSource(pcapgoReader, linkType)` identical to live path |
|
||||
| `github.com/spf13/cobra` | v1.10.2 | New `--filter` and `--read` flags | Already used |
|
||||
|
||||
**No new dependencies needed.** `pcapgo` is a sub-package of `gopacket` already in go.mod.
|
||||
|
||||
**Installation:** None required. All libraries are already present in go.mod/go.sum.
|
||||
|
||||
### Supporting
|
||||
| Library | Version | Purpose | When to Use |
|
||||
|---------|---------|---------|-------------|
|
||||
| `golang.org/x/net/bpf` | via go.sum (transitive) | BPF instruction assembly inside go-pcap | Not called directly; go-pcap's `SetBPFFilter` uses it internally |
|
||||
|
||||
### Alternatives Considered
|
||||
| Instead of | Could Use | Tradeoff |
|
||||
|------------|-----------|----------|
|
||||
| `pcapgo.NewReader` (already in gopacket) | External libpcap CGo path | Libpcap can read pcap files but adds CGo and system library dependency — no benefit |
|
||||
| In-place BPF filter on `--read` | Software filter in classify stage | go-pcap's `SetBPFFilter` only works on live sockets; for pcap file reading the filter must be applied in software after decoding. See Pitfall 2 below. |
|
||||
|
||||
## Architecture Patterns
|
||||
|
||||
### Recommended Project Structure
|
||||
```
|
||||
capture/
|
||||
├── capture.go # OpenCapture (add filter param), StartCapture (add filter param), ReadPcapFile (new)
|
||||
aggregate/
|
||||
├── window.go # Aggregate (unchanged), AggregatePcap (new timestamp-based variant)
|
||||
cmd/netsynth/
|
||||
└── main.go # New --filter and --read Cobra flags; branch run() into live vs pcap paths
|
||||
```
|
||||
|
||||
### Pattern 1: BPF Filter on Live Handle
|
||||
**What:** Call `handle.SetBPFFilter(expr)` immediately after `pcap.OpenLive` succeeds, before starting the read loop. The filter is installed in the kernel; packets not matching the expression are silently dropped by the kernel before they reach user space.
|
||||
|
||||
**When to use:** When `-i` is active (live capture mode) and `--filter` is provided.
|
||||
|
||||
**Validated API (verified by reading `/home/dev/go/pkg/mod/github.com/packetcap/go-pcap@v0.0.0-20251215121130-f2cf9f991e7c/pcap.go`):**
|
||||
```go
|
||||
// Source: go-pcap pcap.go SetBPFFilter method
|
||||
func (h *Handle) SetBPFFilter(expr string) error {
|
||||
expr2 := strings.TrimSpace(expr)
|
||||
if expr2 == "" {
|
||||
return nil
|
||||
}
|
||||
e := filter.NewExpression(expr2)
|
||||
if e == nil {
|
||||
return fmt.Errorf("no expression received for filter '%s'", expr)
|
||||
}
|
||||
f := e.Compile()
|
||||
instructions, err := f.Compile()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to compile filter into instructions: %v", err)
|
||||
}
|
||||
raw, err := bpf.Assemble(instructions)
|
||||
if err != nil {
|
||||
return fmt.Errorf("bpf assembly failed: %v", err)
|
||||
}
|
||||
return h.SetRawBPFFilter(raw)
|
||||
}
|
||||
```
|
||||
|
||||
**Usage in OpenCapture (updated signature):**
|
||||
```go
|
||||
// capture/capture.go
|
||||
func OpenCapture(ctx context.Context, iface string, filter string) (*pcap.Handle, error) {
|
||||
handle, err := pcap.OpenLive(ctx, iface, 65535, false, 0, false)
|
||||
if err != nil { ... }
|
||||
if filter != "" {
|
||||
if err := handle.SetBPFFilter(filter); err != nil {
|
||||
handle.Close()
|
||||
return nil, fmt.Errorf("invalid BPF filter %q: %w", filter, err)
|
||||
}
|
||||
}
|
||||
return handle, nil
|
||||
}
|
||||
```
|
||||
|
||||
**Error message pattern for invalid BPF:** The error from `SetBPFFilter` surfaces as `"failed to compile filter into instructions: parse error"` for unrecognized tokens, or `"failed to compile filter into instructions: unknown host: abc"` for unresolvable hostnames. Wrapping with `fmt.Errorf("invalid BPF filter %q: %w", expr, err)` gives the user the expression and the underlying reason.
|
||||
|
||||
### Pattern 2: BPF Filter for Pcap File (Software Filter)
|
||||
**What:** `go-pcap`'s `SetBPFFilter` attaches to a Linux raw socket — it cannot be called on a pcapgo file reader. For `--read` mode, BPF filtering must happen in the classify goroutine or as an adapter between the packet channel and the classify channel.
|
||||
|
||||
**Recommended approach (Claude's Discretion):** Apply a software BPF filter in a thin wrapper over the packet channel. Use `go-pcap`'s `filter` sub-package directly to pre-compile the expression and match each packet's raw bytes:
|
||||
|
||||
```go
|
||||
// Option A (simpler): validate expression early, then filter packets in a goroutine
|
||||
// using filter.NewExpression(expr).Compile() which returns a Filter that has a
|
||||
// Match([]byte) bool method. Verify Match signature from filter package source.
|
||||
```
|
||||
|
||||
**Important caveat:** After examining `filter/compile.go`, the `Filter` interface has a `Compile() ([]bpf.Instruction, error)` method for kernel assembly but the per-packet matching for user-space is via `golang.org/x/net/bpf.VM`. This adds a dependency on `bpf.NewVM(instructions)` and `vm.Run(packet)` for software filtering.
|
||||
|
||||
**Simpler alternative:** For `--read` mode, skip hardware BPF entirely and apply the filter as a software drop in the classify stage: decode the packet normally, then check if it matches. But this defeats the "filter before classification" separation.
|
||||
|
||||
**Most practical approach for this phase:** Use `bpf.NewVM` from `golang.org/x/net/bpf` (already in go.sum as a transitive dependency) to run BPF programs in user space against pcap file packets.
|
||||
|
||||
```go
|
||||
// Source: golang.org/x/net/bpf VM interface (standard, HIGH confidence)
|
||||
import "golang.org/x/net/bpf"
|
||||
|
||||
instructions, err := filter.NewExpression(expr).Compile().Compile()
|
||||
vm, err := bpf.NewVM(instructions)
|
||||
// Per packet:
|
||||
ok, err := vm.Run(packetData)
|
||||
// ok > 0 means packet passes the filter
|
||||
```
|
||||
|
||||
### Pattern 3: Pcap File Reading
|
||||
**What:** Use `gopacket/pcapgo.NewReader(io.Reader)` to open a pcap file, then `gopacket.NewPacketSource(reader, reader.LinkType())` to get the same packet-producing interface as live capture.
|
||||
|
||||
**Validated API (verified by reading `/home/dev/go/pkg/mod/github.com/gopacket/gopacket@v1.5.0/pcapgo/read.go`):**
|
||||
```go
|
||||
// Source: gopacket/pcapgo read.go
|
||||
import (
|
||||
"os"
|
||||
"github.com/gopacket/gopacket"
|
||||
"github.com/gopacket/gopacket/pcapgo"
|
||||
)
|
||||
|
||||
func ReadPcapFile(path string, filter string) (<-chan gopacket.Packet, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cannot open pcap file %q: %w", path, err)
|
||||
}
|
||||
r, err := pcapgo.NewReader(f)
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return nil, fmt.Errorf("invalid pcap file %q: %w", path, err)
|
||||
}
|
||||
// r.LinkType() returns layers.LinkType — same as handle.LinkType() in live mode
|
||||
packetSource := gopacket.NewPacketSource(r, r.LinkType())
|
||||
packetSource.NoCopy = false // pcapgo allocates per packet; NoCopy=false is correct here
|
||||
packets := make(chan gopacket.Packet, 512)
|
||||
go func() {
|
||||
defer close(packets)
|
||||
defer f.Close()
|
||||
for pkt := range packetSource.Packets() {
|
||||
packets <- pkt
|
||||
}
|
||||
}()
|
||||
return packets, nil
|
||||
}
|
||||
```
|
||||
|
||||
**Key observation:** `pcapgo.NewReader` reads and validates the pcap file header immediately. If the file is corrupt, truncated, or not a pcap file, `NewReader` returns an error before any packets are read. This is the natural pre-flight validation.
|
||||
|
||||
**CaptureInfo.Timestamp:** Every packet from pcapgo has `pkt.Metadata().CaptureInfo.Timestamp` set to the recorded timestamp. This is the time value used for D-01 window assignment.
|
||||
|
||||
### Pattern 4: Timestamp-Based Aggregation for Pcap Mode
|
||||
**What:** Replace the `time.Ticker` in `Aggregate()` with arithmetic over packet timestamps. Compute the current window index as `int(pkt.Timestamp.Sub(firstPacket).Milliseconds()) / windowMs` and flush when the index advances.
|
||||
|
||||
**When to use:** When `--read` mode is active. Reuse existing `Aggregate()` for live mode unchanged.
|
||||
|
||||
**New function in aggregate/window.go:**
|
||||
```go
|
||||
// AggregatePcap reads ClassifiedPackets that include a Timestamp field,
|
||||
// assigns each to a windowMs window relative to the first packet timestamp,
|
||||
// fills gap windows with empty snapshots per D-02, and emits all windows.
|
||||
// Returns a []WindowSnapshot directly (not a channel) because pcap processing
|
||||
// is synchronous and all snapshots are available before encode.RunSynthesis.
|
||||
func AggregatePcap(events []classify.ClassifiedPacket, timestamps []time.Time, windowMs int) []classify.WindowSnapshot
|
||||
```
|
||||
|
||||
**Alternative:** Pass timestamps alongside ClassifiedPackets by adding a `Timestamp time.Time` field to `classify.ClassifiedPacket`. This keeps the pipeline shape consistent.
|
||||
|
||||
**Recommended approach (Claude's Discretion):** Add `Timestamp time.Time` to `ClassifiedPacket`. The classify goroutine already reads `pkt.Metadata().CaptureInfo.Timestamp` — adding it to the struct costs one field. Then a new `AggregatePcap(events <-chan classify.ClassifiedPacket, windowMs int) []classify.WindowSnapshot` function collects all events, assigns them to windows, and fills gaps. This reuses the existing classify stage unchanged and requires only a new aggregation function.
|
||||
|
||||
**Gap-filling algorithm (D-02):**
|
||||
```go
|
||||
// For each window index from 0 to maxWindowIndex (inclusive):
|
||||
// if window has packets → emit WindowSnapshot with counts
|
||||
// if window is empty → emit WindowSnapshot with TotalPackets=0, empty Counts
|
||||
// This preserves silence for pcap files with quiet periods.
|
||||
```
|
||||
|
||||
### Pattern 5: Filename Derivation (D-05)
|
||||
```go
|
||||
// capture.pcap -> capture.mp3
|
||||
func deriveOutputPath(readPath string) string {
|
||||
ext := filepath.Ext(readPath)
|
||||
return strings.TrimSuffix(readPath, ext) + ".mp3"
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 6: Mutual Exclusion Validation (D-03)
|
||||
```go
|
||||
// In run(), before any capture:
|
||||
if readPath != "" && ifaceName != "" {
|
||||
return fmt.Errorf("--read and -i are mutually exclusive; use one or the other")
|
||||
}
|
||||
if readPath == "" && ifaceName == "" {
|
||||
return fmt.Errorf("interface required: use -i <interface> or --read <file>")
|
||||
}
|
||||
```
|
||||
|
||||
### Anti-Patterns to Avoid
|
||||
- **Calling SetBPFFilter on a pcapgo.Reader:** This method only exists on `go-pcap`'s live `Handle`. Will not compile.
|
||||
- **Using NoCopy=true with pcapgo:** Unlike go-pcap which copies internally (making NoCopy safe), pcapgo's `ZeroCopyReadPacketData` reuses a buffer that is invalidated on the next read. Use `ReadPacketData` (copy-safe) or `NoCopy=false` on PacketSource.
|
||||
- **Deriving pcap timestamps from wall clock:** When `--read` is active, use `pkt.Metadata().CaptureInfo.Timestamp`, not `time.Now()`, or windows will all collapse into a single instant.
|
||||
- **Omitting filter validation before capture starts:** If `--filter` is invalid, the error should appear before the "Starting capture..." message. Call `handle.SetBPFFilter` before logging capture start; or pre-validate using `filter.NewExpression(expr).Compile().Compile()` before opening the handle.
|
||||
|
||||
## Don't Hand-Roll
|
||||
|
||||
| Problem | Don't Build | Use Instead | Why |
|
||||
|---------|-------------|-------------|-----|
|
||||
| BPF expression compilation | Custom parser for tcpdump syntax | `handle.SetBPFFilter(expr)` (go-pcap) | BPF syntax has 20+ primitives, direction qualifiers, boolean operators; already implemented and tested |
|
||||
| Pcap file header parsing | Read magic bytes manually | `pcapgo.NewReader(r)` | Handles big/little endian, microsecond/nanosecond timestamps, gzip-compressed pcap; error on unknown magic |
|
||||
| Software BPF matching | Byte comparisons per-packet | `bpf.NewVM(instructions)` (golang.org/x/net/bpf) | Runs the same BPF bytecode the kernel uses; already in go.sum as transitive dep |
|
||||
| BPF expression validation | String parsing | `filter.NewExpression(expr).Compile().Compile()` | Validates the full pipeline (parse → AST → BPF instructions) without needing an open socket |
|
||||
|
||||
**Key insight:** The BPF filter path in go-pcap is already a pure-Go parser + compiler; the same compiled instructions can be run in user space via `bpf.NewVM` without duplicating any filter logic.
|
||||
|
||||
## Runtime State Inventory
|
||||
|
||||
Step 2.5: SKIPPED — this is a new feature addition phase, not a rename/refactor/migration.
|
||||
|
||||
## Environment Availability
|
||||
|
||||
| Dependency | Required By | Available | Version | Fallback |
|
||||
|------------|------------|-----------|---------|----------|
|
||||
| Go toolchain | Build | Yes | go1.24.1 linux/arm64 | — |
|
||||
| C compiler (CGo for go-lame) | MP3 encoding | Yes (verified in Phase 2) | — | — |
|
||||
| `golang.org/x/net/bpf` | Software BPF VM for `--read --filter` | Yes (in go.sum as transitive dep) | v0.39.0 | — |
|
||||
| `gopacket/pcapgo` | Pcap file reading | Yes (sub-package of gopacket v1.5.0, already in go.mod) | v1.5.0 | — |
|
||||
| Test pcap files | Integration tests | Not present — must create | — | Use `tcpdump -w test.pcap` or generate a minimal pcap in test setup |
|
||||
|
||||
**Missing dependencies with no fallback:** None.
|
||||
|
||||
**Missing dependencies with fallback:** None — all needed libraries are already in go.mod/go.sum.
|
||||
|
||||
**Note on test pcap files:** Unit tests for `ReadPcapFile` and `AggregatePcap` will need either (a) a small pcap fixture committed to the repo, or (b) a test helper that writes a minimal valid pcap file using `pcapgo.NewWriter`. Option (b) is preferred (no binary test fixtures in repo).
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Pitfall 1: SetBPFFilter Error After Capture Has Started
|
||||
**What goes wrong:** If BPF filter validation is deferred until after the "Starting capture on eth0..." message is printed, users see a misleading output before the error appears.
|
||||
**Why it happens:** `OpenCapture` is called, then filter is applied, but the start message was already printed in `main.go` before calling `StartCapture`.
|
||||
**How to avoid:** In `OpenCapture`, apply `SetBPFFilter` before returning the handle, and print the "Starting capture..." message only after the handle is successfully opened and filtered. Or pre-validate the BPF expression string using `filter.NewExpression(expr).Compile().Compile()` before `OpenCapture`.
|
||||
**Warning signs:** Error message appears after "Starting capture..." log line.
|
||||
|
||||
### Pitfall 2: BPF VM Not Available at Filter Validation Time
|
||||
**What goes wrong:** For `--read` mode, trying to call `handle.SetBPFFilter` fails because there is no live handle. Without a check, the filter flag is silently ignored.
|
||||
**Why it happens:** The BPF filter API is on `*pcap.Handle`, which doesn't exist in offline mode.
|
||||
**How to avoid:** Pre-validate the BPF expression string once at startup using `filter.NewExpression(expr).Compile().Compile()` (pure parse + compile, no socket needed). Then for live mode use `handle.SetBPFFilter`; for pcap mode use `bpf.NewVM(instructions)` per packet.
|
||||
**Warning signs:** `--filter` flag is provided with `--read` but no filtering occurs.
|
||||
|
||||
### Pitfall 3: NoCopy=true With pcapgo
|
||||
**What goes wrong:** `NoCopy=true` on `PacketSource` combined with `pcapgo.Reader.ZeroCopyReadPacketData` means packet data is overwritten on the next read. If the classify goroutine lags behind, it reads stale/corrupted data.
|
||||
**Why it happens:** go-pcap copies data internally (making NoCopy safe), but pcapgo's zero-copy mode reuses a buffer. The `NoCopy = true` comment in `StartCapture` notes "safe: go-pcap copies internally" — this does not apply to pcapgo.
|
||||
**How to avoid:** Use `NoCopy = false` (the default) when creating `PacketSource` from a pcapgo reader. Use `ReadPacketData` (not `ZeroCopyReadPacketData`) internally.
|
||||
**Warning signs:** Occasional garbage in decoded packet fields during pcap replay.
|
||||
|
||||
### Pitfall 4: Empty Pcap File Produces Zero Snapshots
|
||||
**What goes wrong:** `ReadPcapFile` on a pcap with no packets returns an empty packet channel. `AggregatePcap` returns `[]WindowSnapshot{}`. `encode.RunSynthesis` receives an empty slice and the existing OUT-03 guard fires: "no packets captured."
|
||||
**Why it happens:** The zero-packet guard in `RunSynthesis` already handles this — but the error message ("no packets captured") is slightly wrong for the pcap case.
|
||||
**How to avoid:** Check `len(collectedSnapshots) == 0` before calling `RunSynthesis` in the pcap path and return a more informative error: `"pcap file %q contains no packets"`. Reuse the guard, improve the message.
|
||||
**Warning signs:** User runs `netsynth --read empty.pcap` and gets "no packets captured" without any file path context.
|
||||
|
||||
### Pitfall 5: Timestamp Monotonicity in Pcap Files
|
||||
**What goes wrong:** Some pcap files have non-monotonic timestamps (e.g., pcap files merged from multiple captures). Window assignment via `pkt.Timestamp.Sub(firstPacket)` produces negative or very large window indices for out-of-order packets.
|
||||
**Why it happens:** The pcap format does not guarantee timestamp order.
|
||||
**How to avoid:** Track `minTimestamp` rather than assuming the first packet has the earliest timestamp. Or use `max(currentWindowIndex, computedWindowIndex)` to clamp backward jumps. For this phase, a simple guard is sufficient: skip packets with timestamps before the first observed timestamp.
|
||||
**Warning signs:** Panic on negative slice index, or extremely long MP3 output from merged pcap.
|
||||
|
||||
### Pitfall 6: go-pcap SetBPFFilter Returns nil for Empty String
|
||||
**What goes wrong:** If `filter != ""` check is omitted, passing an empty `--filter ""` to `SetBPFFilter` does nothing (the method returns nil for empty string). This is actually correct behavior, but test cases must confirm empty filter == no filter applied.
|
||||
**Why it happens:** `SetBPFFilter` explicitly returns nil for whitespace-only strings (`strings.TrimSpace(expr) == ""`).
|
||||
**How to avoid:** Pass `filter` through only when it is non-empty. Document the empty-filter-is-no-filter behavior.
|
||||
|
||||
## Code Examples
|
||||
|
||||
Verified patterns from module cache source:
|
||||
|
||||
### BPF Pre-validation (no socket required)
|
||||
```go
|
||||
// Source: go-pcap filter package (filter/expression.go, filter/compile.go)
|
||||
// Validates the BPF expression string before opening any capture handle.
|
||||
import "github.com/packetcap/go-pcap/filter"
|
||||
|
||||
func validateBPFFilter(expr string) error {
|
||||
if strings.TrimSpace(expr) == "" {
|
||||
return nil // empty is valid (means no filter)
|
||||
}
|
||||
e := filter.NewExpression(expr)
|
||||
if e == nil {
|
||||
return fmt.Errorf("invalid BPF filter expression: %q", expr)
|
||||
}
|
||||
compiled := e.Compile()
|
||||
if _, err := compiled.Compile(); err != nil {
|
||||
return fmt.Errorf("invalid BPF filter %q: %v", expr, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
### Opening pcapgo Reader
|
||||
```go
|
||||
// Source: gopacket/pcapgo read.go (NewReader, LinkType)
|
||||
import (
|
||||
"os"
|
||||
"github.com/gopacket/gopacket"
|
||||
"github.com/gopacket/gopacket/pcapgo"
|
||||
)
|
||||
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cannot open %q: %w", path, err)
|
||||
}
|
||||
r, err := pcapgo.NewReader(f)
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return nil, fmt.Errorf("not a valid pcap file %q: %w", path, err)
|
||||
}
|
||||
lt := r.LinkType() // layers.LinkType, same type as handle.LinkType() returns
|
||||
packetSource := gopacket.NewPacketSource(r, lt)
|
||||
packetSource.NoCopy = false // do NOT set true with pcapgo
|
||||
```
|
||||
|
||||
### Timestamp-Based Window Assignment
|
||||
```go
|
||||
// Source: gopacket CaptureInfo struct (gopacket/packet.go)
|
||||
// pkt.Metadata().CaptureInfo.Timestamp is a time.Time set by pcapgo.
|
||||
|
||||
func assignWindow(pktTime, firstTime time.Time, windowMs int) int {
|
||||
elapsed := pktTime.Sub(firstTime)
|
||||
if elapsed < 0 {
|
||||
return 0 // clamp backward jumps
|
||||
}
|
||||
return int(elapsed.Milliseconds()) / windowMs
|
||||
}
|
||||
```
|
||||
|
||||
### Gap-Filling for Silence (D-02)
|
||||
```go
|
||||
// Emit a WindowSnapshot for every window index from 0 to maxIdx (inclusive).
|
||||
// Windows with no packets get TotalPackets=0 and empty Counts map (silent audio).
|
||||
snapshots := make([]classify.WindowSnapshot, maxWindowIdx+1)
|
||||
for i := range snapshots {
|
||||
snapshots[i] = classify.WindowSnapshot{
|
||||
Counts: make(map[classify.TrafficClass]int64),
|
||||
WindowIndex: i,
|
||||
}
|
||||
}
|
||||
for _, ev := range buffered {
|
||||
idx := assignWindow(ev.Timestamp, firstTime, windowMs)
|
||||
snapshots[idx].Counts[ev.Class]++
|
||||
snapshots[idx].TotalPackets++
|
||||
}
|
||||
```
|
||||
|
||||
### Output Filename Derivation (D-05)
|
||||
```go
|
||||
// Source: stdlib path/filepath
|
||||
import "path/filepath"
|
||||
|
||||
func deriveOutputPath(readPath string) string {
|
||||
ext := filepath.Ext(readPath)
|
||||
base := strings.TrimSuffix(readPath, ext)
|
||||
return base + ".mp3"
|
||||
}
|
||||
// "capture.pcap" -> "capture.mp3"
|
||||
// "/tmp/net.pcap.gz" -> "/tmp/net.pcap.mp3" (intentional: only strips last ext)
|
||||
```
|
||||
|
||||
### Software BPF Matching via bpf.VM
|
||||
```go
|
||||
// Source: golang.org/x/net/bpf (in go.sum as transitive dep via go-pcap)
|
||||
import (
|
||||
"golang.org/x/net/bpf"
|
||||
gpcapfilter "github.com/packetcap/go-pcap/filter"
|
||||
)
|
||||
|
||||
func compileSoftwareBPF(expr string) (*bpf.VM, error) {
|
||||
e := gpcapfilter.NewExpression(expr)
|
||||
instructions, err := e.Compile().Compile()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return bpf.NewVM(instructions)
|
||||
}
|
||||
|
||||
// Per packet:
|
||||
result, err := vm.Run(pkt.Data())
|
||||
if result > 0 {
|
||||
// packet passes the filter
|
||||
}
|
||||
```
|
||||
|
||||
## State of the Art
|
||||
|
||||
| Old Approach | Current Approach | When Changed | Impact |
|
||||
|--------------|------------------|--------------|--------|
|
||||
| `google/gopacket` pcapgo | `gopacket/gopacket` pcapgo (community fork) | 2022 (fork), actively maintained | pcapgo API identical; import path differs |
|
||||
| libpcap for pcap file reading | `pcapgo.NewReader` (pure Go) | Available in gopacket since v1.0 | No system library needed; works without libpcap installed |
|
||||
|
||||
**No deprecations affecting this phase.**
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Software BPF for `--read --filter`: bpf.VM vs skip filtering**
|
||||
- What we know: `go-pcap`'s `SetBPFFilter` is hardware-only; for pcap files we need software matching.
|
||||
- What's unclear: Whether `bpf.VM` from `golang.org/x/net/bpf` correctly evaluates the same instructions that `SetBPFFilter` would compile for all filter expressions (hostname resolution differs between kernel and userspace).
|
||||
- Recommendation: Test with `port 53`, `tcp`, and `host 10.0.0.1` filter expressions against a known pcap fixture. If hostname resolution in `filter.NewExpression` proves unreliable for user-space matching, document the limitation clearly in the error message.
|
||||
|
||||
2. **ClassifiedPacket Timestamp field**
|
||||
- What we know: `aggregate.Aggregate` uses a `<-chan classify.ClassifiedPacket` with no timestamp. D-01 requires timestamp-based windowing for pcap mode.
|
||||
- What's unclear: Whether to add `Timestamp time.Time` to `ClassifiedPacket` (shared struct change) or handle timestamps externally.
|
||||
- Recommendation (Claude's Discretion): Add `Timestamp time.Time` to `ClassifiedPacket`. It is zero-valued in live mode (no behaviour change there) and populated from `pkt.Metadata().CaptureInfo.Timestamp` in the classify goroutine for pcap mode. This is the least invasive change.
|
||||
|
||||
## Validation Architecture
|
||||
|
||||
### Test Framework
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Framework | go test (stdlib) |
|
||||
| Config file | none |
|
||||
| Quick run command | `go test ./capture/... ./aggregate/... ./cmd/netsynth/...` |
|
||||
| Full suite command | `go test ./...` |
|
||||
|
||||
### Phase Requirements → Test Map
|
||||
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|
||||
|--------|----------|-----------|-------------------|-------------|
|
||||
| CAPT-05 | `--filter "port 53"` with live capture applies BPF to handle | unit | `go test ./capture/... -run TestOpenCaptureWithFilter` | No — Wave 0 |
|
||||
| CAPT-05 | Invalid BPF filter returns error before capture starts | unit | `go test ./capture/... -run TestBPFFilterValidation` | No — Wave 0 |
|
||||
| CAPT-05 | BPF pre-validation function rejects bad expressions | unit | `go test ./capture/... -run TestValidateBPFFilter` | No — Wave 0 |
|
||||
| CAPT-06 | `ReadPcapFile` returns packet channel from valid pcap | unit | `go test ./capture/... -run TestReadPcapFile` | No — Wave 0 |
|
||||
| CAPT-06 | `ReadPcapFile` returns error on missing/corrupt file | unit | `go test ./capture/... -run TestReadPcapFileErrors` | No — Wave 0 |
|
||||
| CAPT-06 | `AggregatePcap` assigns packets to correct windows | unit | `go test ./aggregate/... -run TestAggregatePcap` | No — Wave 0 |
|
||||
| CAPT-06 | `AggregatePcap` fills gap windows with empty snapshots (D-02) | unit | `go test ./aggregate/... -run TestAggregatePcapGaps` | No — Wave 0 |
|
||||
| CAPT-06 | `--read` and `-i` mutually exclusive | unit | `go test ./cmd/netsynth/... -run TestFlagMutualExclusion` | No — Wave 0 |
|
||||
| CAPT-05+06 | `--filter` with `--read` applies software BPF to pcap packets | unit | `go test ./capture/... -run TestSoftwareBPFFilter` | No — Wave 0 |
|
||||
| D-05 | Output filename derived from input when `-o` absent | unit | `go test ./cmd/netsynth/... -run TestDeriveOutputPath` | No — Wave 0 |
|
||||
|
||||
### Sampling Rate
|
||||
- **Per task commit:** `go test ./capture/... ./aggregate/...`
|
||||
- **Per wave merge:** `go test ./...`
|
||||
- **Phase gate:** Full suite green before `/gsd:verify-work`
|
||||
|
||||
### Wave 0 Gaps
|
||||
- [ ] `capture/capture_test.go` — add `TestOpenCaptureWithFilter`, `TestBPFFilterValidation`, `TestValidateBPFFilter`, `TestReadPcapFile`, `TestReadPcapFileErrors`, `TestSoftwareBPFFilter`
|
||||
- [ ] `capture/testdata/` — small pcap fixture for `TestReadPcapFile` (or a test helper that generates one using `pcapgo.NewWriter`)
|
||||
- [ ] `aggregate/window_test.go` — add `TestAggregatePcap`, `TestAggregatePcapGaps`
|
||||
- [ ] `cmd/netsynth/main_test.go` — add `TestFlagMutualExclusion`, `TestDeriveOutputPath`
|
||||
|
||||
**Existing test infrastructure:** `go test ./...` is already green (verified). All Wave 0 gaps are test additions, not framework setup.
|
||||
|
||||
## Project Constraints (from CLAUDE.md)
|
||||
|
||||
These directives apply to all phase work:
|
||||
|
||||
| Directive | Applies To |
|
||||
|-----------|------------|
|
||||
| Language: Go only | All new files |
|
||||
| Single binary output | No new system library dependencies |
|
||||
| MP3 output format | No change — RunSynthesis unchanged |
|
||||
| Non-interactive model: run → Ctrl+C → file saved | `--read` mode: run → completes automatically → file saved |
|
||||
| Use `gopacket/gopacket` (community fork, NOT `google/gopacket`) | import path in pcapgo: `github.com/gopacket/gopacket/pcapgo` |
|
||||
| Use `packetcap/go-pcap` for live capture | unchanged for live path; pcapgo for file path |
|
||||
| Use `sjzar/go-lame` for MP3 | unchanged |
|
||||
| Use `spf13/cobra` for CLI | `--filter` and `--read` flags added via cobra |
|
||||
| `go-audio/generator` is archived — do not use | not relevant to this phase |
|
||||
| Do not use `faiface/beep`, `dasa.cc/snd`, or `viert/go-lame` | not relevant to this phase |
|
||||
| GSD workflow: use Write/Edit only within a GSD command | enforced by workflow |
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (HIGH confidence)
|
||||
- Module cache: `/home/dev/go/pkg/mod/github.com/packetcap/go-pcap@v0.0.0-20251215121130-f2cf9f991e7c/pcap.go` — `SetBPFFilter` signature and implementation verified
|
||||
- Module cache: `/home/dev/go/pkg/mod/github.com/packetcap/go-pcap@v0.0.0-20251215121130-f2cf9f991e7c/filter/expression.go` — `NewExpression`, `Compile` verified
|
||||
- Module cache: `/home/dev/go/pkg/mod/github.com/packetcap/go-pcap@v0.0.0-20251215121130-f2cf9f991e7c/filter/compile_cases_test.go` — error strings for invalid BPF expressions verified (`"parse error"`, `"unknown host: ..."`)
|
||||
- Module cache: `/home/dev/go/pkg/mod/github.com/gopacket/gopacket@v1.5.0/pcapgo/read.go` — `NewReader`, `ReadPacketData`, `LinkType`, `CaptureInfo.Timestamp` verified
|
||||
- Project source: `capture/capture.go`, `aggregate/window.go`, `cmd/netsynth/main.go`, `classify/types.go` — integration points verified by reading existing code
|
||||
- `go test ./...` — full test suite passes (verified live on machine)
|
||||
|
||||
### Secondary (MEDIUM confidence)
|
||||
- `golang.org/x/net/bpf` VM interface for software BPF matching — package is in go.sum (transitive), API is stable stdlib-adjacent, `bpf.NewVM(instructions)` and `vm.Run(data)` are well-documented
|
||||
|
||||
### Tertiary (LOW confidence)
|
||||
- None — all claims verified from module cache source directly
|
||||
|
||||
## Metadata
|
||||
|
||||
**Confidence breakdown:**
|
||||
- Standard stack: HIGH — verified from go.mod and module cache source; no new dependencies required
|
||||
- Architecture: HIGH — API signatures verified from module cache source, not from documentation or training data
|
||||
- Pitfalls: HIGH — derived from direct source reading (NoCopy comment in pcap_linux.go, error string tests in compile_cases_test.go)
|
||||
- BPF VM for software filtering: MEDIUM — API is in go.sum, standard package, but `vm.Run` return semantics not verified by reading source; test against real pcap to confirm
|
||||
|
||||
**Research date:** 2026-03-26
|
||||
**Valid until:** 2026-09-26 (stable stdlib-adjacent packages; go-pcap minor version could change API but no indication of breaking changes planned)
|
||||
Reference in New Issue
Block a user