docs(phase-01): research capture and classification phase
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Claude Sonnet 4.6
@@ -0,0 +1,713 @@
|
||||
# Phase 1: Capture and Classification - Research
|
||||
|
||||
**Researched:** 2026-03-25
|
||||
**Domain:** Go CLI scaffolding, live packet capture (pure-Go pcap), protocol classification pipeline
|
||||
**Confidence:** HIGH (stack and patterns fully verified via official docs; one gap noted for go-pcap interface listing)
|
||||
|
||||
---
|
||||
|
||||
<user_constraints>
|
||||
## User Constraints (from CONTEXT.md)
|
||||
|
||||
### Locked Decisions
|
||||
|
||||
- **D-01:** Deep port map with 10+ classes: ICMP, DNS (53), HTTPS (443), HTTP (80), SSH (22), SMTP (25), NTP (123), DHCP (67/68), other-TCP, other-UDP
|
||||
- **D-02:** Classification rules stored in a config-driven Go map/struct (not hardcoded switch statements) — designed so rules could later be loaded from a config file
|
||||
- **D-03:** All unrecognized traffic grouped as a single "unknown" class until Phase 3 adds auto-clustering
|
||||
- **D-04:** Support Linux and macOS (not Windows)
|
||||
- **D-05:** On permission failure, detect the OS and show platform-specific guidance: `sudo setcap cap_net_raw+ep ...` on Linux, `sudo ...` on macOS
|
||||
- **D-06:** Prefer static binary with no runtime libpcap dependency — use pure-Go pcap backend (go-pcap) where possible
|
||||
|
||||
### Claude's Discretion
|
||||
|
||||
- CLI output formatting during capture (stderr layout, colors, table width)
|
||||
- Default time window duration for aggregation buckets
|
||||
- Default network interface selection when `-i` is omitted
|
||||
- Verbose output format and level of detail
|
||||
|
||||
### Deferred Ideas (OUT OF SCOPE)
|
||||
|
||||
None — discussion stayed within phase scope
|
||||
</user_constraints>
|
||||
|
||||
---
|
||||
|
||||
<phase_requirements>
|
||||
## Phase Requirements
|
||||
|
||||
| ID | Description | Research Support |
|
||||
|----|-------------|------------------|
|
||||
| CAPT-01 | User can specify network interface via `-i` flag | Cobra persistent flag `--interface/-i`; `go-pcap.OpenLive()` accepts device name string |
|
||||
| CAPT-02 | User can list available network interfaces via `--list-interfaces` | `net.Interfaces()` from Go stdlib (go-pcap has no FindAllDevs); no privilege required to enumerate |
|
||||
| CAPT-04 | User sees a clear actionable error message when lacking capture privileges | `OpenLive()` returns an error on permission failure; inspect error string + `runtime.GOOS` to produce platform-specific hint |
|
||||
| CLAS-01 | Known protocols (ICMP, DNS, TCP/443, TCP/other, UDP, SSH) each produce a distinct recognizable sound | Rule table as `map[string]TrafficClass` keyed by composite key (proto+port); gopacket layer decoding gives TCP/UDP/ICMP layers cleanly |
|
||||
| CLAS-03 | On exit, user sees a summary of packet counts and protocol breakdown (printed to stderr) | Counters in `map[TrafficClass]int64` incremented per classified packet; print on signal handler / channel drain |
|
||||
| CLAS-04 | User can enable per-window protocol activity log via `--verbose` flag | Cobra bool flag; time-window aggregator emits `WindowSnapshot`; verbose mode prints each snapshot to stderr |
|
||||
</phase_requirements>
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Phase 1 establishes the full capture-to-classify pipeline in Go: a Cobra-based CLI that opens a live network interface via the pure-Go `packetcap/go-pcap` library, decodes packets with `gopacket`, classifies them against a config-driven protocol rule table, aggregates counts into time windows, and prints live status lines plus a per-protocol summary on exit. No audio synthesis is included; Phase 2 consumes the `WindowSnapshot` output this phase defines.
|
||||
|
||||
The primary technical challenge is privilege handling: `go-pcap.OpenLive()` silently fails on Linux when the binary lacks `CAP_NET_RAW`, and the error message from the underlying syscall gives no actionable hint. The CLI must intercept that error and emit a platform-specific remediation message. The second challenge is that `packetcap/go-pcap` has no interface enumeration API — `--list-interfaces` must use Go's stdlib `net.Interfaces()` instead. This distinction is important to implement correctly.
|
||||
|
||||
All other elements (channel pipeline, ticker window, classification struct) are standard Go patterns. The project is greenfield with no existing code; Phase 1 establishes directory layout, `go.mod`, and all data types that downstream phases depend on.
|
||||
|
||||
**Primary recommendation:** Use `packetcap/go-pcap` + `gopacket/gopacket` for capture/decode, `net.Interfaces()` for interface listing, `spf13/cobra` for CLI, and the channel-connected pipeline pattern from ARCHITECTURE.md throughout.
|
||||
|
||||
---
|
||||
|
||||
## Project Constraints (from CLAUDE.md)
|
||||
|
||||
Directives that apply to this phase:
|
||||
|
||||
- **Language:** Go — mandatory, no alternatives
|
||||
- **Privileges:** Packet capture requires root/CAP_NET_RAW on Linux
|
||||
- **Audio format:** MP3 output (not in scope for Phase 1, but module must not import go-lame yet — save for Phase 2/3 to keep Phase 1 CGo-free)
|
||||
- **Interaction model:** Non-interactive (run → Ctrl+C → summary printed to stderr)
|
||||
- **GSD workflow:** All file edits go through GSD commands, not direct edits outside a GSD workflow
|
||||
|
||||
---
|
||||
|
||||
## Standard Stack
|
||||
|
||||
### Core (Phase 1 only)
|
||||
|
||||
| Library | Version | Purpose | Why Standard |
|
||||
|---------|---------|---------|--------------|
|
||||
| `github.com/gopacket/gopacket` | v1.5.0 | Packet decode, layer type assertions | Community-maintained fork of google/gopacket; v1.5.0 Nov 2025; 14.5k dependents; built-in ICMP/TCP/UDP/DNS/TLS layers. Requires Go 1.24+. |
|
||||
| `github.com/packetcap/go-pcap` | v0.0.0-20251215 | Live packet capture (pure Go, no CGo) | Implements `gopacket.PacketDataSource`; pure Go on Linux+macOS; mmap kernel ring buffer; no `libpcap-dev` system package at build or runtime. Decision D-06 mandates this. |
|
||||
| `github.com/spf13/cobra` | v1.10.2 | CLI flags, subcommands, signal handling | Industry standard (Kubernetes, Docker, Hugo); handles `-i`, `--list-interfaces`, `--verbose`, `--help`, `PersistentPreRunE` for validation. |
|
||||
| Go stdlib `net` | stdlib | Interface enumeration for `--list-interfaces` | `net.Interfaces()` returns all system interfaces with name, flags, and addresses; requires no privileges; go-pcap has no FindAllDevs equivalent. |
|
||||
| Go stdlib `runtime` | stdlib | OS detection for privilege error messages | `runtime.GOOS` returns `"linux"` or `"darwin"` at runtime; used in D-05 platform-specific error branching. |
|
||||
| Go stdlib `os/signal` + `syscall` | stdlib | Ctrl+C handling, clean shutdown | Standard Go signal channel pattern; `signal.NotifyContext` is idiomatic in Go 1.16+. |
|
||||
|
||||
### Not Needed for Phase 1
|
||||
|
||||
| Library | Phase | Reason Excluded |
|
||||
|---------|-------|-----------------|
|
||||
| `github.com/sjzar/go-lame` | Phase 2/3 | MP3 encoding — no audio in Phase 1 |
|
||||
| `github.com/go-audio/wav` | Phase 2/3 | WAV intermediate — no audio in Phase 1 |
|
||||
| `github.com/muesli/kmeans` | Phase 3 | Auto-clustering — D-03 defers to Phase 3 |
|
||||
|
||||
### Alternatives Considered
|
||||
|
||||
| Standard Choice | Alternative | Why Standard Wins |
|
||||
|-----------------|-------------|-------------------|
|
||||
| `packetcap/go-pcap` | `gopacket/pcap` (CGo + libpcap) | libpcap requires system package; contradicts D-06; no CGo in capture layer |
|
||||
| `net.Interfaces()` | `gopacket/pcap.FindAllDevs()` | pcap.FindAllDevs requires CGo libpcap which we explicitly avoid; stdlib is sufficient |
|
||||
| `spf13/cobra` | `urfave/cli` | Cobra has better flag validation, persistent pre-run hooks, and structured signal integration |
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# Go 1.24+ required — NOT available via apt on this machine (apt offers 1.22)
|
||||
# Install from https://go.dev/dl/ — e.g.:
|
||||
wget https://go.dev/dl/go1.24.1.linux-arm64.tar.gz
|
||||
sudo tar -C /usr/local -xzf go1.24.1.linux-arm64.tar.gz
|
||||
export PATH="$PATH:/usr/local/go/bin"
|
||||
|
||||
# Initialize module (greenfield — no go.mod exists yet)
|
||||
cd /home/dev/workspace/yoloyolo
|
||||
go mod init github.com/yourorg/netsynth # or preferred module path
|
||||
|
||||
# Phase 1 dependencies only (no CGo libraries yet)
|
||||
go get github.com/gopacket/gopacket@v1.5.0
|
||||
go get github.com/packetcap/go-pcap@latest
|
||||
go get github.com/spf13/cobra@v1.10.2
|
||||
|
||||
# Build (no CGo needed for Phase 1)
|
||||
CGO_ENABLED=0 go build -o netsynth ./cmd/netsynth
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Architecture Patterns
|
||||
|
||||
### Recommended Project Structure
|
||||
|
||||
```
|
||||
netsynth/
|
||||
├── cmd/
|
||||
│ └── netsynth/
|
||||
│ └── main.go # cobra root command wiring, signal handling
|
||||
├── capture/
|
||||
│ └── capture.go # OpenLive wrapper, interface listing, chan gopacket.Packet
|
||||
├── classify/
|
||||
│ ├── classifier.go # Rule table (map-based, D-02), Classify() func
|
||||
│ └── rules.go # Protocol rule definitions (ICMP, DNS, HTTPS, etc. per D-01)
|
||||
├── aggregate/
|
||||
│ └── window.go # Ticker-driven accumulator, WindowSnapshot type
|
||||
└── go.mod
|
||||
```
|
||||
|
||||
This is a subset of the full ARCHITECTURE.md structure — synth/, encode/, config/ are added in later phases. Phase 1 lays down the first four packages.
|
||||
|
||||
### Pattern 1: Channel-Connected Pipeline Stages
|
||||
|
||||
Each component is a goroutine reading from an inbound channel and writing to an outbound channel. A `done` channel (closed on Ctrl+C) signals all stages to drain and exit.
|
||||
|
||||
```go
|
||||
// Source: https://go.dev/blog/pipelines (official Go blog)
|
||||
func Classify(done <-chan struct{}, packets <-chan gopacket.Packet) <-chan ClassifiedPacket {
|
||||
out := make(chan ClassifiedPacket, 256)
|
||||
go func() {
|
||||
defer close(out)
|
||||
for {
|
||||
select {
|
||||
case <-done:
|
||||
return
|
||||
case pkt, ok := <-packets:
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
out <- classify(pkt)
|
||||
}
|
||||
}
|
||||
}()
|
||||
return out
|
||||
}
|
||||
```
|
||||
|
||||
**Channel buffer sizes for Phase 1 (no audio clock pressure):**
|
||||
- capture → classify: `make(chan gopacket.Packet, 512)` — absorbs bursts
|
||||
- classify → aggregate: `make(chan ClassifiedPacket, 1024)` — aggregate is ticker-driven
|
||||
|
||||
### Pattern 2: Ticker-Driven Window with Verbose Output
|
||||
|
||||
```go
|
||||
// Source: ARCHITECTURE.md Pattern 2 + Go time.Ticker docs
|
||||
func Aggregate(done <-chan struct{}, events <-chan ClassifiedPacket,
|
||||
windowMs int, verbose bool) <-chan WindowSnapshot {
|
||||
out := make(chan WindowSnapshot, 8)
|
||||
ticker := time.NewTicker(time.Duration(windowMs) * time.Millisecond)
|
||||
go func() {
|
||||
defer close(out)
|
||||
counts := map[TrafficClass]int64{}
|
||||
for {
|
||||
select {
|
||||
case <-done:
|
||||
out <- snapshot(counts) // flush final partial window
|
||||
return
|
||||
case <-ticker.C:
|
||||
snap := snapshot(counts)
|
||||
if verbose {
|
||||
printWindowLine(snap) // CLAS-04: --verbose output to stderr
|
||||
}
|
||||
out <- snap
|
||||
counts = map[TrafficClass]int64{}
|
||||
case ev, ok := <-events:
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
counts[ev.Class]++
|
||||
}
|
||||
}
|
||||
}()
|
||||
return out
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 3: Config-Driven Classification Table (D-02)
|
||||
|
||||
The classification rule set is a struct, not a switch statement:
|
||||
|
||||
```go
|
||||
// classify/rules.go
|
||||
type Rule struct {
|
||||
Protocol string // "tcp", "udp", "icmp", "any"
|
||||
DstPort uint16 // 0 = match any port for this protocol
|
||||
Class TrafficClass
|
||||
}
|
||||
|
||||
// DefaultRules is the D-01 port map — can later be replaced by config loading
|
||||
var DefaultRules = []Rule{
|
||||
{Protocol: "icmp", DstPort: 0, Class: ClassICMP},
|
||||
{Protocol: "udp", DstPort: 53, Class: ClassDNS},
|
||||
{Protocol: "tcp", DstPort: 53, Class: ClassDNS},
|
||||
{Protocol: "tcp", DstPort: 443, Class: ClassHTTPS},
|
||||
{Protocol: "tcp", DstPort: 80, Class: ClassHTTP},
|
||||
{Protocol: "tcp", DstPort: 22, Class: ClassSSH},
|
||||
{Protocol: "tcp", DstPort: 25, Class: ClassSMTP},
|
||||
{Protocol: "udp", DstPort: 123, Class: ClassNTP},
|
||||
{Protocol: "udp", DstPort: 67, Class: ClassDHCP},
|
||||
{Protocol: "udp", DstPort: 68, Class: ClassDHCP},
|
||||
// Catch-alls come last:
|
||||
{Protocol: "tcp", DstPort: 0, Class: ClassOtherTCP},
|
||||
{Protocol: "udp", DstPort: 0, Class: ClassOtherUDP},
|
||||
}
|
||||
```
|
||||
|
||||
The classifier iterates rules in order, returns first match. "unknown" class is returned when no rule matches (D-03).
|
||||
|
||||
### Pattern 4: Interface Listing via Go Stdlib
|
||||
|
||||
```go
|
||||
// capture/capture.go — uses net.Interfaces(), NOT pcap.FindAllDevs
|
||||
// Source: https://pkg.go.dev/net#Interfaces
|
||||
import "net"
|
||||
|
||||
func ListInterfaces() ([]net.Interface, error) {
|
||||
return net.Interfaces()
|
||||
}
|
||||
|
||||
// Usage in CLI command handler:
|
||||
ifaces, err := capture.ListInterfaces()
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "Error listing interfaces:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
for _, iface := range ifaces {
|
||||
addrs, _ := iface.Addrs()
|
||||
addrStrs := make([]string, len(addrs))
|
||||
for i, a := range addrs {
|
||||
addrStrs[i] = a.String()
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, " %-15s flags=%s addrs=%s\n",
|
||||
iface.Name, iface.Flags, strings.Join(addrStrs, ", "))
|
||||
}
|
||||
```
|
||||
|
||||
This satisfies CAPT-02 with no privilege requirement — `net.Interfaces()` works as a non-root user.
|
||||
|
||||
### Pattern 5: Privilege Error Detection (D-05)
|
||||
|
||||
```go
|
||||
// capture/capture.go
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"runtime"
|
||||
"strings"
|
||||
|
||||
pcap "github.com/packetcap/go-pcap"
|
||||
)
|
||||
|
||||
func OpenCapture(iface string) (*pcap.Handle, error) {
|
||||
handle, err := pcap.OpenLive(context.Background(), iface, 65535, false, 0, false)
|
||||
if err != nil {
|
||||
if isPermissionError(err) {
|
||||
return nil, permissionErrorMsg(iface)
|
||||
}
|
||||
return nil, fmt.Errorf("failed to open interface %q: %w", iface, err)
|
||||
}
|
||||
return handle, nil
|
||||
}
|
||||
|
||||
func isPermissionError(err error) bool {
|
||||
s := strings.ToLower(err.Error())
|
||||
return strings.Contains(s, "permission denied") ||
|
||||
strings.Contains(s, "operation not permitted") ||
|
||||
strings.Contains(s, "pcap_create") // some platforms wrap differently
|
||||
}
|
||||
|
||||
func permissionErrorMsg(iface string) error {
|
||||
switch runtime.GOOS {
|
||||
case "linux":
|
||||
bin, _ := os.Executable()
|
||||
return fmt.Errorf(
|
||||
"packet capture requires root or CAP_NET_RAW.\n"+
|
||||
"Run as root: sudo %s -i %s\n"+
|
||||
"Or grant capability: sudo setcap cap_net_raw+ep %s",
|
||||
bin, iface, bin)
|
||||
default: // darwin and others
|
||||
return fmt.Errorf(
|
||||
"packet capture requires root privileges.\n"+
|
||||
"Run as root: sudo netsynth -i %s", iface)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 6: gopacket Packet Decoding for Classification
|
||||
|
||||
```go
|
||||
// Source: https://pkg.go.dev/github.com/gopacket/gopacket
|
||||
import (
|
||||
"github.com/gopacket/gopacket"
|
||||
"github.com/gopacket/gopacket/layers"
|
||||
pcap "github.com/packetcap/go-pcap"
|
||||
)
|
||||
|
||||
handle, _ := pcap.OpenLive(ctx, iface, 65535, false, 0, false)
|
||||
packetSource := gopacket.NewPacketSource(handle, layers.LinkTypeEthernet)
|
||||
// NoCopy is safe here because go-pcap copies data internally
|
||||
packetSource.NoCopy = true
|
||||
|
||||
for pkt := range packetSource.Packets() {
|
||||
// ICMP check
|
||||
if pkt.Layer(layers.LayerTypeICMPv4) != nil ||
|
||||
pkt.Layer(layers.LayerTypeICMPv6) != nil {
|
||||
// ClassICMP
|
||||
}
|
||||
// TCP destination port
|
||||
if tcp, ok := pkt.TransportLayer().(*layers.TCP); ok {
|
||||
dstPort := uint16(tcp.DstPort)
|
||||
// match against rule table
|
||||
}
|
||||
// UDP destination port
|
||||
if udp, ok := pkt.TransportLayer().(*layers.UDP); ok {
|
||||
dstPort := uint16(udp.DstPort)
|
||||
// match against rule table
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Note: `packetSource.NoCopy = true` is safe because `packetcap/go-pcap` copies packet data internally before returning from `ReadPacketData()`. Verified from go-pcap source structure.
|
||||
|
||||
### Anti-Patterns to Avoid
|
||||
|
||||
- **Switch-statement classification:** Forbidden by D-02. Use the `[]Rule` slice instead.
|
||||
- **Blocking channel send in capture path:** Capture goroutine must never block on a full channel; use buffered channel + drop counter.
|
||||
- **`ZeroCopyReadPacketData()` sent to goroutines:** Do not use with concurrent processing; stick with `ReadPacketData()` or let gopacket's `PacketSource` manage it.
|
||||
- **`google/gopacket` import:** Must be `github.com/gopacket/gopacket` — enforced from day one in `go.mod`.
|
||||
- **Privilege check by inspecting `/proc/self/status`:** Fragile. Instead try `OpenLive()` and intercept the error — the only reliable way to know if capture actually works.
|
||||
|
||||
---
|
||||
|
||||
## Don't Hand-Roll
|
||||
|
||||
| Problem | Don't Build | Use Instead | Why |
|
||||
|---------|-------------|-------------|-----|
|
||||
| CLI flag parsing | Custom arg parser | `spf13/cobra` | Signal handling, auto-help, persistent flags, PersistentPreRunE validation |
|
||||
| Packet decoding (Ethernet/IP/TCP/UDP/ICMP headers) | Manual `binary.Read` offsets | `gopacket/layers` | Layer type assertions; handles IPv4/IPv6, fragmentation, Ethernet padding |
|
||||
| Interface enumeration | Parse `/proc/net/dev` or run `ip link` | `net.Interfaces()` | Stdlib; cross-platform; returns flags, addresses; no privilege needed |
|
||||
| Ctrl+C / signal handling | `os.Signal` channel with manual `os.Exit` | `signal.NotifyContext` | Propagates context cancellation cleanly; idiomatic Go 1.16+ pattern |
|
||||
| Live packet capture | Raw AF_PACKET socket from scratch | `packetcap/go-pcap` | Kernel ring buffer, BPF filter support, `gopacket.PacketDataSource` compliance |
|
||||
|
||||
**Key insight:** The hardest part of packet capture in Go is correctly handling the kernel ring buffer, BPF attachment, and copy semantics. `packetcap/go-pcap` solves all three in pure Go; there is no reason to touch raw sockets in this phase.
|
||||
|
||||
---
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Pitfall 1: `go-pcap` Has No Interface Discovery Function
|
||||
|
||||
**What goes wrong:** Developer calls `pcap.FindAllDevs()` expecting libpcap behavior — that function does not exist in `packetcap/go-pcap`. Build fails.
|
||||
|
||||
**Why it happens:** The STACK.md mentions `pcap.FindAllDevs()` in the context of the gopacket/pcap (CGo) backend, not go-pcap. Easy to conflate.
|
||||
|
||||
**How to avoid:** Use `net.Interfaces()` from Go stdlib for `--list-interfaces`. It returns equivalent information (name, flags, addresses) without any privilege requirement.
|
||||
|
||||
**Warning signs:** Any import of `gopacket/pcap` instead of `packetcap/go-pcap`.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 2: Go 1.24 Not Installed
|
||||
|
||||
**What goes wrong:** `go get github.com/gopacket/gopacket@v1.5.0` fails with "minimum required go 1.24" or the apt-installed `golang-go` (version 1.22) is used and build fails.
|
||||
|
||||
**Why it happens:** Ubuntu 24.04 ships `golang-go` v1.22 via apt. `gopacket/gopacket@v1.5.0` requires Go 1.24+. The version mismatch is silent until the first `go get`.
|
||||
|
||||
**How to avoid:** Wave 0 task: install Go 1.24+ from https://go.dev/dl/. Do not use `apt install golang-go` for this project. Verify with `go version` before any other task.
|
||||
|
||||
**Warning signs:** `go version` output showing `go1.22.x`.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 3: CAP_NET_RAW Silent Failure on nosuid Filesystems
|
||||
|
||||
**What goes wrong:** `sudo setcap cap_net_raw+ep ./netsynth` appears to succeed but the binary fails at runtime when run from `/home/dev/workspace/` because the filesystem is mounted `nosuid`. The capability is silently ignored.
|
||||
|
||||
**Why it happens:** `/home` is frequently on a separate partition mounted `nosuid`. The kernel enforces capabilities only on filesystems where `nosuid` is not set.
|
||||
|
||||
**How to avoid:** For development, run as `sudo ./netsynth -i eth0` rather than relying on `setcap`. Document that setcap only works from `/usr/local/bin` or equivalent. The error message from D-05 covers this — show both options so the user knows sudo always works.
|
||||
|
||||
**Warning signs:** Binary works via `sudo` but fails via `setcap` from the workspace directory.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 4: Blocking Channel Send Stalls Kernel Packet Buffer
|
||||
|
||||
**What goes wrong:** If the classify goroutine is slow (or tests inject sleep), the capture goroutine's send to the `packets` channel blocks, and the kernel's pcap ring buffer fills and drops packets without any log message.
|
||||
|
||||
**Why it happens:** Unbuffered or small-buffered channel between capture and classify.
|
||||
|
||||
**How to avoid:** Use `make(chan gopacket.Packet, 512)`. In the capture loop, use a non-blocking send with a drop counter:
|
||||
```go
|
||||
select {
|
||||
case packets <- pkt:
|
||||
default:
|
||||
atomic.AddInt64(&droppedPackets, 1)
|
||||
}
|
||||
```
|
||||
For Phase 1's use case (ambient audio, statistical fingerprint), lossy capture is acceptable; just make drops visible.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 5: `--list-interfaces` Requires Privilege on Some Systems
|
||||
|
||||
**What goes wrong:** On some Linux configurations, `net.Interfaces()` returns an empty list when run as non-root, causing `--list-interfaces` to print nothing.
|
||||
|
||||
**Why it happens:** Network namespace restrictions or security policies can limit interface visibility. However, this is rare on standard Ubuntu 24.04 with default configuration.
|
||||
|
||||
**How to avoid:** Always print the list plus a note: "If interfaces are missing, run with sudo." Confirmed LOW risk on the target machine (standard Ubuntu 24.04, no restricted namespaces observed).
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 6: LinkType Mismatch on Non-Ethernet Interfaces
|
||||
|
||||
**What goes wrong:** `gopacket.NewPacketSource(handle, layers.LinkTypeEthernet)` is correct for `eth0` but fails to decode on loopback (`lo`) which uses `layers.LinkTypeLoopback` (or `layers.LinkTypeNull` on macOS). Packets decode as garbage or produce layer decode errors silently.
|
||||
|
||||
**Why it happens:** Link type is hardcoded in the `NewPacketSource` call.
|
||||
|
||||
**How to avoid:** Call `handle.LinkType()` on the opened handle and pass the result. `packetcap/go-pcap` exports `LinkTypeEthernet` and `LinkTypeNull` constants. For `lo`, use `layers.LinkTypeLoopback`.
|
||||
```go
|
||||
lt := layers.LinkType(handle.LinkType())
|
||||
packetSource := gopacket.NewPacketSource(handle, lt)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Code Examples
|
||||
|
||||
Verified patterns from official sources:
|
||||
|
||||
### Opening a Live Capture Handle
|
||||
|
||||
```go
|
||||
// Source: https://pkg.go.dev/github.com/packetcap/go-pcap
|
||||
import (
|
||||
"context"
|
||||
pcap "github.com/packetcap/go-pcap"
|
||||
)
|
||||
|
||||
handle, err := pcap.OpenLive(
|
||||
context.Background(),
|
||||
"eth0", // interface name
|
||||
65535, // snaplen: capture full packets
|
||||
false, // promiscuous: false by default per security guidance in PITFALLS.md
|
||||
0, // timeout: 0 = block until packet
|
||||
false, // syscalls: false = use mmap ring buffer (faster on Linux)
|
||||
)
|
||||
if err != nil {
|
||||
// Check for permission error and print platform-specific hint (Pattern 5)
|
||||
}
|
||||
defer handle.Close()
|
||||
```
|
||||
|
||||
### Creating a gopacket PacketSource
|
||||
|
||||
```go
|
||||
// Source: https://pkg.go.dev/github.com/gopacket/gopacket
|
||||
import (
|
||||
"github.com/gopacket/gopacket"
|
||||
"github.com/gopacket/gopacket/layers"
|
||||
)
|
||||
|
||||
lt := layers.LinkType(handle.LinkType()) // dynamic link type per Pitfall 6
|
||||
packetSource := gopacket.NewPacketSource(handle, lt)
|
||||
packetSource.NoCopy = true // safe: go-pcap copies internally
|
||||
|
||||
for pkt := range packetSource.Packets() {
|
||||
// pkt is a gopacket.Packet with full layer decoding
|
||||
}
|
||||
```
|
||||
|
||||
### Listing Interfaces (--list-interfaces)
|
||||
|
||||
```go
|
||||
// Source: https://pkg.go.dev/net#Interfaces
|
||||
import "net"
|
||||
|
||||
ifaces, err := net.Interfaces()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, iface := range ifaces {
|
||||
addrs, _ := iface.Addrs()
|
||||
// print name, flags, addresses to stderr (tcpdump convention)
|
||||
}
|
||||
```
|
||||
|
||||
### Cobra Root Command Structure
|
||||
|
||||
```go
|
||||
// Source: https://pkg.go.dev/github.com/spf13/cobra
|
||||
var rootCmd = &cobra.Command{
|
||||
Use: "netsynth",
|
||||
Short: "Sonify live network traffic",
|
||||
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
|
||||
// validate flags before Run
|
||||
return nil
|
||||
},
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
// main capture loop
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
func init() {
|
||||
rootCmd.Flags().StringVarP(&iface, "interface", "i", "", "Network interface to capture on")
|
||||
rootCmd.Flags().BoolVar(&listIfaces, "list-interfaces", false, "List available interfaces")
|
||||
rootCmd.Flags().BoolVar(&verbose, "verbose", false, "Print per-window protocol activity")
|
||||
}
|
||||
```
|
||||
|
||||
### Signal Handling (clean shutdown)
|
||||
|
||||
```go
|
||||
// Source: https://pkg.go.dev/os/signal#NotifyContext (Go 1.16+)
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
// Pass ctx.Done() as the done channel to all pipeline stages
|
||||
// When Ctrl+C is pressed, ctx is cancelled and all stages drain cleanly
|
||||
```
|
||||
|
||||
### Per-Protocol Summary on Exit (CLAS-03)
|
||||
|
||||
```go
|
||||
// After pipeline drains, print summary to stderr (tcpdump convention)
|
||||
fmt.Fprintln(os.Stderr, "\n--- Protocol Summary ---")
|
||||
// Sort classes for stable output
|
||||
for _, class := range sortedClasses(totals) {
|
||||
pct := 0.0
|
||||
if totalPackets > 0 {
|
||||
pct = float64(totals[class]) / float64(totalPackets) * 100.0
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, " %-15s %8d packets (%5.1f%%)\n",
|
||||
class.String(), totals[class], pct)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, " %-15s %8d packets\n", "TOTAL", totalPackets)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## State of the Art
|
||||
|
||||
| Old Approach | Current Approach | When Changed | Impact |
|
||||
|--------------|------------------|--------------|--------|
|
||||
| `google/gopacket` | `github.com/gopacket/gopacket` v1.5.0 | Community fork active since ~2023, v1.5.0 Nov 2025 | Must use community fork; original is abandoned |
|
||||
| `gopacket/pcap` (CGo libpcap) for live capture | `packetcap/go-pcap` (pure Go) | Dec 2025 (v0.0.0-20251215) | No CGo at capture layer; CGO_ENABLED=0 for Phase 1 build |
|
||||
| `signal.Notify` (manual channel) | `signal.NotifyContext` | Go 1.16 | Cleaner context propagation; use NotifyContext everywhere |
|
||||
| Flat `main.go` | Cobra command structure | Stable pattern | Subcommand extensibility for future phases |
|
||||
|
||||
**Deprecated/outdated:**
|
||||
- `google/gopacket`: abandoned 2022; 270+ open issues; do not use
|
||||
- `go-audio/generator`: archived February 2026; do not take new dependency
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **go-pcap link type for WiFi / Docker bridge interfaces**
|
||||
- What we know: `handle.LinkType()` returns uint32; go-pcap exports only `LinkTypeEthernet` and `LinkTypeNull`
|
||||
- What's unclear: Does it return the correct type for Docker bridge (`br-*`) and veth interfaces on this machine?
|
||||
- Recommendation: In the capture open sequence, fall back to `layers.LinkTypeEthernet` if `handle.LinkType()` returns an unrecognized value, and log a warning. The target demo interface is `eth0` which is Ethernet.
|
||||
|
||||
2. **Default interface when `-i` is omitted**
|
||||
- What we know: Left to Claude's discretion (CONTEXT.md)
|
||||
- What's unclear: Should it pick the first non-loopback interface, or require `-i` always?
|
||||
- Recommendation: Require `-i` explicitly (no silent default) unless `--list-interfaces` was given. This matches `tcpdump` behavior and avoids surprising captures on wrong interfaces.
|
||||
|
||||
3. **`promiscuous` mode default**
|
||||
- What we know: PITFALLS.md recommends non-promiscuous by default with opt-in `--promiscuous` flag
|
||||
- What's unclear: Phase 1 requirements don't mention a `--promiscuous` flag (it's not in CAPT-01..CAPT-04)
|
||||
- Recommendation: Default to non-promiscuous; add `--promiscuous` flag as a bonus if it falls within scope; if not, document the limitation in help text.
|
||||
|
||||
---
|
||||
|
||||
## Environment Availability
|
||||
|
||||
| Dependency | Required By | Available | Version | Fallback |
|
||||
|------------|-------------|-----------|---------|---------|
|
||||
| Go toolchain 1.24+ | All Go compilation | **No** | 1.22 in apt (too old) | Install from https://go.dev/dl/ — manual step required |
|
||||
| GCC / build-essential | Phase 2+ CGo (go-lame) | **No** | Not installed | `sudo apt install build-essential` — not needed for Phase 1 (CGO_ENABLED=0) |
|
||||
| libpcap runtime (`libpcap0.8t64`) | go-pcap on Linux at runtime | Yes | 1.10.4 (installed) | n/a — but go-pcap is pure Go so this is irrelevant |
|
||||
| libpcap-dev headers | CGo pcap builds | No | Not installed | Not needed — go-pcap is pure Go |
|
||||
| `tcpdump` | Manual validation of captures | Yes | `/usr/bin/tcpdump` | — |
|
||||
| `eth0` interface | CAPT-01 demo interface | Yes | Present on machine | `lo` for loopback-only test |
|
||||
| `CAP_NET_RAW` or root | Live packet capture | Not verified | Run as uid=1001 (non-root) | `sudo netsynth -i eth0` always works |
|
||||
|
||||
**Missing dependencies with no fallback:**
|
||||
- **Go 1.24+**: Must be installed before any work begins. Wave 0 task. `apt install golang-go` gives 1.22 which is insufficient.
|
||||
|
||||
**Missing dependencies with fallback:**
|
||||
- **GCC**: Not needed for Phase 1 (no CGo libraries). Needed in Phase 2/3 for go-lame. Install via `sudo apt install build-essential` at that time.
|
||||
- **Root / CAP_NET_RAW**: Tests that need live capture must run via `sudo`. Unit tests for classifier and aggregator use synthetic packet data and require no privilege.
|
||||
|
||||
**Platform note:** This machine is `linux/arm64` (Ubuntu 24.04). Go 1.24 ARM64 binaries are available at https://go.dev/dl/. The `packetcap/go-pcap` library supports Linux on ARM64 (uses standard Linux kernel ring buffer syscalls).
|
||||
|
||||
---
|
||||
|
||||
## Validation Architecture
|
||||
|
||||
### Test Framework
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Framework | Go stdlib `testing` package (no external framework needed) |
|
||||
| Config file | None — `go test ./...` discovers tests automatically |
|
||||
| Quick run command | `go test ./classify/... ./aggregate/...` (no privileges needed) |
|
||||
| Full suite command | `go test ./...` (capture tests require sudo or are integration-only) |
|
||||
|
||||
### Phase Requirements → Test Map
|
||||
|
||||
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|
||||
|--------|----------|-----------|-------------------|-------------|
|
||||
| CAPT-01 | `-i eth0` opens capture handle without error | Integration (needs sudo) | `sudo go test ./capture/... -run TestOpenLive -v` | No — Wave 0 |
|
||||
| CAPT-02 | `--list-interfaces` prints at least one interface | Unit (no privilege) | `go test ./capture/... -run TestListInterfaces -v` | No — Wave 0 |
|
||||
| CAPT-04 | Non-root capture returns actionable error message | Unit (mock OpenLive error) | `go test ./capture/... -run TestPermissionError -v` | No — Wave 0 |
|
||||
| CLAS-01 | Each of 10+ protocol classes is assigned correctly for synthetic packets | Unit | `go test ./classify/... -run TestClassify -v` | No — Wave 0 |
|
||||
| CLAS-03 | Exit summary prints correct counts per class | Unit | `go test ./aggregate/... -run TestSummary -v` | No — Wave 0 |
|
||||
| CLAS-04 | `--verbose` prints one line per window with protocol breakdown | Unit | `go test ./aggregate/... -run TestVerbose -v` | No — Wave 0 |
|
||||
|
||||
### Sampling Rate
|
||||
|
||||
- **Per task commit:** `go test ./classify/... ./aggregate/...` (fast, no privilege)
|
||||
- **Per wave merge:** `go test ./...` (may skip integration tests if not running as root)
|
||||
- **Phase gate:** All unit tests pass; manual smoke test `sudo ./netsynth -i eth0` shows live classification output to stderr
|
||||
|
||||
### Wave 0 Gaps
|
||||
|
||||
- [ ] `classify/classifier_test.go` — covers CLAS-01: unit tests with synthetic gopacket packets for each of 10 protocol classes
|
||||
- [ ] `capture/capture_test.go` — covers CAPT-02 (TestListInterfaces, no privilege) and CAPT-04 (TestPermissionError with injected error)
|
||||
- [ ] `aggregate/window_test.go` — covers CLAS-03 (summary counts) and CLAS-04 (verbose window lines)
|
||||
- [ ] `go.mod` and `go.sum` — module not yet initialized; must be created before any test file compiles
|
||||
- [ ] Go 1.24 toolchain install — prerequisite for everything above
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (HIGH confidence)
|
||||
|
||||
- `https://pkg.go.dev/github.com/packetcap/go-pcap` — Full API surface verified; confirmed no FindAllDevs; `OpenLive` signature; `Handle.LinkType()` method
|
||||
- `https://pkg.go.dev/github.com/gopacket/gopacket` — v1.5.0 confirmed Nov 2025; `PacketDataSource` interface; `NewPacketSource` API; layer types
|
||||
- `https://pkg.go.dev/net` — `net.Interfaces()` stdlib function; returns `[]net.Interface` with name, flags, addrs; no privilege needed
|
||||
- `https://pkg.go.dev/github.com/spf13/cobra` — v1.10.2 Dec 2025; `PersistentPreRunE`, flag patterns
|
||||
- `https://go.dev/blog/pipelines` — Official Go pipeline pattern with `done` channel
|
||||
- `.planning/research/STACK.md` — Verified library versions, CGo strategy, why google/gopacket is forbidden
|
||||
- `.planning/research/ARCHITECTURE.md` — Channel buffer sizes, pipeline stage signatures, `WindowSnapshot` type definition
|
||||
- `.planning/research/PITFALLS.md` — CAP_NET_RAW nosuid behavior, ZeroCopy use-after-free, packet buffer overflow strategies
|
||||
|
||||
### Secondary (MEDIUM confidence)
|
||||
|
||||
- WebSearch + pkg.go.dev cross-check: `packetcap/go-pcap` v0.0.0-20251215 implements `gopacket.PacketDataSource`, Linux/macOS only — confirmed
|
||||
- `.planning/phases/01-capture-and-classification/01-CONTEXT.md` — Locked decisions D-01 through D-06
|
||||
|
||||
### Tertiary (LOW confidence)
|
||||
|
||||
- go-pcap `NoCopy = true` safety: described in fetch results as "handle copies data internally" — plausible given pure-Go implementation but not confirmed via source inspection; treat as verify-on-test
|
||||
|
||||
---
|
||||
|
||||
## Metadata
|
||||
|
||||
**Confidence breakdown:**
|
||||
- Standard stack: HIGH — versions verified via pkg.go.dev fetches
|
||||
- Architecture: HIGH — Go pipeline patterns from official blog; gopacket API from official docs
|
||||
- Pitfalls: HIGH for CAP_NET_RAW and channel buffering (verified via PITFALLS.md primary sources); MEDIUM for go-pcap NoCopy safety
|
||||
- Environment: HIGH — direct inspection of installed packages and Go availability on target machine
|
||||
|
||||
**Research date:** 2026-03-25
|
||||
**Valid until:** 2026-06-25 (90 days — stack is stable; go-pcap pre-release version may update)
|
||||
Reference in New Issue
Block a user