docs: complete project research
This commit is contained in:
+304
-272
@@ -1,344 +1,376 @@
|
||||
# Architecture Research
|
||||
# Architecture Patterns
|
||||
|
||||
**Domain:** Network traffic sonification CLI (Go)
|
||||
**Researched:** 2026-03-24
|
||||
**Confidence:** MEDIUM — Go audio synthesis patterns verified via official docs and real libraries; sonification architecture inferred from academic literature (SoNSTAR) and Go concurrency canon.
|
||||
**Domain:** Network traffic sonification CLI (Go) — v1.1 Custom Sound Mappings
|
||||
**Researched:** 2026-03-26
|
||||
**Confidence:** HIGH — based on direct code inspection of the existing v1.0 codebase
|
||||
|
||||
## Standard Architecture
|
||||
---
|
||||
|
||||
### System Overview
|
||||
## v1.1 Integration Overview
|
||||
|
||||
This document supersedes the pre-implementation v1.0 architecture research. It is grounded in the actual codebase (3,254 lines, 6 packages) and answers: what changes, what's new, and in what order.
|
||||
|
||||
---
|
||||
|
||||
## Existing Package Map (v1.0 Baseline)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ CLI Entry Point │
|
||||
│ (flags: interface, output path, duration) │
|
||||
└───────────────────────────┬─────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Capture Layer │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ PacketSource (gopacket/pcap) │ │
|
||||
│ │ Produces: chan Packet │ │
|
||||
│ └──────────────────────┬──────────────────────────────┘ │
|
||||
└─────────────────────────┼───────────────────────────────────┘
|
||||
│ raw packet stream
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Classification Layer │
|
||||
│ ┌─────────────────────────┐ ┌─────────────────────────┐ │
|
||||
│ │ Protocol Classifier │ │ Unknown Traffic │ │
|
||||
│ │ (ICMP, DNS, HTTPS, │ │ Clusterer │ │
|
||||
│ │ SSH, TCP-other, UDP) │ │ (feature-based bucketer)│ │
|
||||
│ └───────────┬─────────────┘ └────────────┬────────────┘ │
|
||||
│ └──────────────┬──────────────┘ │
|
||||
└─────────────────────────────┼───────────────────────────────┘
|
||||
│ classified packet events
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Aggregation Layer │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ Time-Window Accumulator │ │
|
||||
│ │ - fixed window (e.g. 500ms) │ │
|
||||
│ │ - counts + byte-volume per traffic class │ │
|
||||
│ │ Produces: chan WindowSnapshot │ │
|
||||
│ └──────────────────────┬──────────────────────────────┘ │
|
||||
└─────────────────────────┼───────────────────────────────────┘
|
||||
│ window snapshots
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Synthesis Layer │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Layer 0 │ │ Layer 1 │ │ Layer N │ │ Layer X │ │
|
||||
│ │ (ICMP) │ │ (DNS) │ │ (HTTPS) │ │ (auto) │ │
|
||||
│ │ Osc+Amp │ │ Osc+Amp │ │ Osc+Amp │ │ Osc+Amp │ │
|
||||
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
|
||||
│ └──────────────┴────────────┴──────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────▼──────┐ │
|
||||
│ │ Mixer │ │
|
||||
│ │ (sum+clip) │ │
|
||||
│ └─────┬──────┘ │
|
||||
└──────────────────────────┼──────────────────────────────────┘
|
||||
│ PCM sample stream (float32[])
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Encoding Layer │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ PCM Buffer Accumulator → LAME MP3 Encoder │ │
|
||||
│ │ (go-lame / CGo libmp3lame) │ │
|
||||
│ └──────────────────────┬──────────────────────────────┘ │
|
||||
└─────────────────────────┼───────────────────────────────────┘
|
||||
│ .mp3 file
|
||||
▼
|
||||
Output File
|
||||
cmd/netsynth/main.go CLI, pipeline wiring, Cobra flags
|
||||
capture/ go-pcap live capture + pcap file reader + BPF
|
||||
classify/
|
||||
types.go TrafficClass, ClassifiedPacket, WindowSnapshot
|
||||
classifier.go NewClassifier(rules []Rule) — first-match-wins
|
||||
rules.go DefaultRules []Rule (12 hardcoded rules)
|
||||
aggregate/
|
||||
window.go 500ms time-windowed snapshot accumulation
|
||||
synth/
|
||||
config.go ClassFreqConfigs — fixed map[TrafficClass]FreqConfig
|
||||
oscillator.go Phase-accumulator oscillator — sine only
|
||||
layer.go EMA amplitude smoothing per layer
|
||||
bank.go NewBank(tau) — one Layer per AllClasses()
|
||||
mixer.go PanGains, StereoFramesToInt16Bytes
|
||||
encode/
|
||||
mp3.go RunSynthesis(snapshots, path) — NewBank + EncodeMP3
|
||||
```
|
||||
|
||||
### Component Responsibilities
|
||||
---
|
||||
|
||||
| Component | Responsibility | Typical Implementation |
|
||||
|-----------|----------------|------------------------|
|
||||
| CLI Entry | Parse flags, wire all components, handle Ctrl+C via `os.Signal` | `main.go`, `cobra` or `flag` package |
|
||||
| PacketSource | Open interface via pcap/AF_PACKET, emit packets into channel | `gopacket.PacketSource.Packets()` → `<-chan gopacket.Packet` |
|
||||
| Protocol Classifier | Inspect decoded layers (IP, TCP, UDP, ICMP, DNS); assign class label | Pure Go switch on `packet.Layer()` type assertions |
|
||||
| Unknown Traffic Clusterer | Hash or bucket unclassified flows by port range / packet size signature; assign stable label ID | Simple feature-hash bucketer; no heavy ML needed for v1 |
|
||||
| Time-Window Accumulator | Batch packets into N-ms windows; emit packet-count and byte-volume per class | `ticker`-driven goroutine, map accumulation |
|
||||
| Sound Layer (per class) | Maintain a sine oscillator at a fixed root frequency; update amplitude from window snapshot | Oscillator struct with phase accumulator; amplitude lerp |
|
||||
| Mixer | Sum all layer outputs sample-by-sample; clamp/normalize to [-1, 1] | Simple additive sum with soft clip |
|
||||
| MP3 Encoder | Accept PCM float32 frames; encode to MP3 on flush/stop | go-lame (CGo) or pure-Go fallback |
|
||||
| Output File | Write encoded bytes to disk path from CLI flag | `os.File` + buffered writer |
|
||||
## What v1.1 Adds
|
||||
|
||||
## Recommended Project Structure
|
||||
Three independent but related features:
|
||||
|
||||
1. **TOML config file** — override frequencies and waveforms per built-in class
|
||||
2. **Additional waveforms** — square, sawtooth, triangle alongside existing sine
|
||||
3. **User-defined classification rules** — TOML-defined rules prepended before DefaultRules
|
||||
|
||||
---
|
||||
|
||||
## Integration Point Analysis
|
||||
|
||||
### Feature 1: TOML Config File
|
||||
|
||||
**Where config is consumed today:** `synth/config.go` holds a package-level `var ClassFreqConfigs`. `synth/bank.go:NewBank()` reads it directly with `ClassFreqConfigs[class]`. No config is passed through `encode.RunSynthesis` or `main.go`.
|
||||
|
||||
**Required change:** `NewBank` must accept a config parameter instead of reading the global. `encode.RunSynthesis` must accept and forward a config. `main.go` must load config from disk and pass it in.
|
||||
|
||||
**New package: `config/`**
|
||||
|
||||
This package does not exist yet in the codebase (the pre-implementation research anticipated it but it was deferred). It should own:
|
||||
- TOML struct definitions
|
||||
- File discovery logic (auto-detect `./netsynth.toml`, then `~/.config/netsynth/config.toml`)
|
||||
- Merging: loaded config overlays defaults, does not replace them entirely
|
||||
|
||||
```
|
||||
netsynth/
|
||||
├── main.go # CLI wiring, signal handling, top-level orchestration
|
||||
├── capture/
|
||||
│ └── capture.go # PacketSource wrapper, interface open/close, chan Packet
|
||||
├── classify/
|
||||
│ ├── classifier.go # Protocol dispatch, class label assignment
|
||||
│ └── cluster.go # Unknown traffic bucketer (feature hash)
|
||||
├── aggregate/
|
||||
│ └── window.go # Time-window accumulator, WindowSnapshot type
|
||||
├── synth/
|
||||
│ ├── oscillator.go # Phase-accumulator sine oscillator
|
||||
│ ├── layer.go # Per-traffic-class sound layer (osc + amp target)
|
||||
│ └── mixer.go # Sum layers → float32 PCM frames
|
||||
├── encode/
|
||||
│ └── mp3.go # PCM → MP3 via go-lame; file flush on close
|
||||
└── config/
|
||||
└── mapping.go # Protocol → frequency/harmonic assignment table
|
||||
config/
|
||||
config.go Config struct, Load(path string) (*Config, error)
|
||||
defaults.go DefaultConfig() — wraps existing ClassFreqConfigs values
|
||||
```
|
||||
|
||||
### Structure Rationale
|
||||
**TOML struct shape:**
|
||||
|
||||
- **capture/:** Isolates pcap/root-privilege boundary. Everything above it operates on typed Go channels with no pcap dependency.
|
||||
- **classify/:** Cleanly separates rule-based (known protocol) from heuristic (unknown cluster) logic. Each can be tested with synthetic packet fixtures independently.
|
||||
- **aggregate/:** The only stateful time-domain component. Isolating it makes window size configurable without touching synthesis.
|
||||
- **synth/:** Pure PCM math — no I/O, no pcap. Fully unit-testable with deterministic inputs. The mixer owns the sample rate constant.
|
||||
- **encode/:** CGo boundary lives here and nowhere else. If LAME is replaced (e.g., pure Go encoder), only this package changes.
|
||||
- **config/:** Static frequency-to-protocol table. Separating it avoids magic numbers scattered across synth/.
|
||||
```toml
|
||||
[[class]]
|
||||
name = "HTTPS"
|
||||
frequency_hz = 200.0
|
||||
waveform = "sawtooth"
|
||||
|
||||
## Architectural Patterns
|
||||
[[class]]
|
||||
name = "myservice" # user-defined class (Feature 3)
|
||||
frequency_hz = 350.0
|
||||
waveform = "triangle"
|
||||
```
|
||||
|
||||
### Pattern 1: Channel-Connected Pipeline Stages
|
||||
The `Config` struct passed into `NewBank` should merge with `ClassFreqConfigs`:
|
||||
|
||||
**What:** Each component is a goroutine that reads from an inbound channel and writes to an outbound channel. The `done` channel (closed on Ctrl+C) signals all stages to drain and exit cleanly.
|
||||
|
||||
**When to use:** Always — this is the idiomatic Go pipeline pattern described in the Go Blog.
|
||||
|
||||
**Trade-offs:** Slightly more setup than direct function calls; pays off immediately with clean shutdown and testability of individual stages.
|
||||
|
||||
**Example:**
|
||||
```go
|
||||
// Each stage signature follows this pattern
|
||||
func Classify(done <-chan struct{}, packets <-chan gopacket.Packet) <-chan ClassifiedPacket {
|
||||
out := make(chan ClassifiedPacket, 256)
|
||||
go func() {
|
||||
defer close(out)
|
||||
for {
|
||||
select {
|
||||
case <-done:
|
||||
return
|
||||
case pkt, ok := <-packets:
|
||||
if !ok { return }
|
||||
out <- classify(pkt)
|
||||
}
|
||||
}
|
||||
}()
|
||||
return out
|
||||
// config/config.go
|
||||
type ClassConfig struct {
|
||||
Name string `toml:"name"`
|
||||
FrequencyHz float64 `toml:"frequency_hz"`
|
||||
Waveform string `toml:"waveform"` // "sine" | "square" | "sawtooth" | "triangle"
|
||||
}
|
||||
|
||||
type Config struct {
|
||||
Classes []ClassConfig `toml:"class"`
|
||||
Rules []RuleConfig `toml:"rule"` // Feature 3
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 2: Ticker-Driven Window Flush
|
||||
**TOML library:** Use `github.com/BurntSushi/toml`. It is the de-facto standard for TOML in Go (used by Hugo, dep, buf, etc.). Already a transitive dependency in many Go module graphs. Provides struct-tag-based decode, good error messages.
|
||||
|
||||
**What:** The aggregation goroutine owns a `time.Ticker`. On each tick it snapshots accumulated counters and sends a `WindowSnapshot` downstream, then resets counters.
|
||||
---
|
||||
|
||||
**When to use:** Anywhere time-based batching converts a high-frequency stream into low-frequency control signals.
|
||||
### Feature 2: Additional Waveforms
|
||||
|
||||
**Trade-offs:** Fixed window size (e.g. 500ms) is simple but loses sub-window dynamics. Sliding windows add complexity with marginal benefit for ambient synthesis.
|
||||
**Where waveform logic lives today:** `synth/oscillator.go:Advance()` — pure sine via `math.Sin`. The `HarmonicDef.Ratio` and `HarmonicDef.Amplitude` fields are stored in `FreqConfig.Harmonics` but the waveform function is hardcoded.
|
||||
|
||||
**Required change:** `Oscillator.Advance` must dispatch on a waveform type. Two clean approaches:
|
||||
|
||||
**Option A (recommended): Waveform enum on Oscillator**
|
||||
|
||||
Add a `waveform` field to `Oscillator`. `Advance` switches on it. `NewOscillator` gains a waveform parameter.
|
||||
|
||||
**Example:**
|
||||
```go
|
||||
func Aggregate(done <-chan struct{}, events <-chan ClassifiedPacket, windowMs int) <-chan WindowSnapshot {
|
||||
out := make(chan WindowSnapshot, 8)
|
||||
ticker := time.NewTicker(time.Duration(windowMs) * time.Millisecond)
|
||||
go func() {
|
||||
defer close(out)
|
||||
counts := map[TrafficClass]int{}
|
||||
for {
|
||||
select {
|
||||
case <-done:
|
||||
return
|
||||
case <-ticker.C:
|
||||
out <- snapshot(counts)
|
||||
counts = map[TrafficClass]int{}
|
||||
case ev, ok := <-events:
|
||||
if !ok { return }
|
||||
counts[ev.Class]++
|
||||
}
|
||||
}
|
||||
}()
|
||||
return out
|
||||
type Waveform int
|
||||
|
||||
const (
|
||||
WaveformSine Waveform = iota
|
||||
WaveformSquare
|
||||
WaveformSawtooth
|
||||
WaveformTriangle
|
||||
)
|
||||
|
||||
type Oscillator struct {
|
||||
phase float64
|
||||
freq float64
|
||||
sr float64
|
||||
waveform Waveform
|
||||
}
|
||||
|
||||
func (o *Oscillator) sampleAt(phase, ratio float64) float64 {
|
||||
p := phase * float64(ratio)
|
||||
p -= math.Floor(p) // wrap to [0, 1)
|
||||
switch o.waveform {
|
||||
case WaveformSquare:
|
||||
if p < 0.5 { return 1.0 }
|
||||
return -1.0
|
||||
case WaveformSawtooth:
|
||||
return 2.0*p - 1.0
|
||||
case WaveformTriangle:
|
||||
if p < 0.5 { return 4.0*p - 1.0 }
|
||||
return 3.0 - 4.0*p
|
||||
default: // WaveformSine
|
||||
return math.Sin(2 * math.Pi * p)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 3: Per-Layer Amplitude Lerp
|
||||
**Option B: Function field on Oscillator**
|
||||
|
||||
**What:** Each sound layer holds a current amplitude and a target amplitude. On each audio frame the current value moves toward the target by a smoothing coefficient. The layer's oscillator always runs; silence is achieved by targeting amplitude = 0.
|
||||
Store `waveFn func(phase float64) float64`. More flexible but harder to serialize/configure.
|
||||
|
||||
**When to use:** Whenever window snapshots drive synthesis — avoids clicks/pops from abrupt amplitude changes.
|
||||
Option A is preferred because waveform type maps cleanly to the TOML `waveform` string field without reflection tricks.
|
||||
|
||||
**Trade-offs:** Adds minimal CPU overhead (one multiply per frame per layer); necessary for perceptually smooth audio.
|
||||
**`FreqConfig` change:** Add `Waveform` field:
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Primary Flow: Packets to PCM
|
||||
|
||||
```
|
||||
Network Interface
|
||||
│
|
||||
▼ (gopacket pcap handle)
|
||||
PacketSource.Packets() chan
|
||||
│
|
||||
▼ (classify goroutine)
|
||||
ClassifiedPacket chan
|
||||
│
|
||||
▼ (aggregate goroutine, ticker)
|
||||
WindowSnapshot chan ─────────────────────────────────┐
|
||||
│
|
||||
(synth goroutine,
|
||||
per window snap:
|
||||
update amplitude targets)
|
||||
│
|
||||
PCM frame generator loop
|
||||
(renders N frames per window,
|
||||
one frame = sum of all layers)
|
||||
│
|
||||
▼
|
||||
PCM []float32 blocks
|
||||
│
|
||||
▼
|
||||
LAME encoder (streaming)
|
||||
│
|
||||
▼
|
||||
MP3 bytes → output file
|
||||
```go
|
||||
type FreqConfig struct {
|
||||
BaseHz float64
|
||||
Harmonics []HarmonicDef
|
||||
Pan float64
|
||||
Waveform Waveform // NEW: defaults to WaveformSine
|
||||
}
|
||||
```
|
||||
|
||||
### Shutdown Flow
|
||||
`NewLayer` passes `cfg.Waveform` to `NewOscillator`. `NewOscillator` signature changes to accept the waveform.
|
||||
|
||||
```
|
||||
Ctrl+C → os.Signal → close(done) channel
|
||||
│
|
||||
├── capture goroutine: drain + close packet chan
|
||||
├── classify goroutine: drain + close event chan
|
||||
├── aggregate goroutine: drain + close snapshot chan
|
||||
└── synth goroutine: flush remaining PCM → encoder.Flush() → file.Close()
|
||||
**What does NOT change:** `HarmonicDef`, `EMAAlpha`, `Layer.UpdateTarget`, `Layer.AdvanceSample`, `OscillatorBank.RenderWindow`, `mixer.go`, `encode/mp3.go`. The waveform change is contained to `oscillator.go` and the `FreqConfig` struct.
|
||||
|
||||
---
|
||||
|
||||
### Feature 3: User-Defined Classification Rules
|
||||
|
||||
**Where rules are wired today:** `main.go` lines 111, 175 — both `runLiveMode` and `runPcapMode` call `classify.NewClassifier(classify.DefaultRules)` directly. No config is passed.
|
||||
|
||||
**Required change:** User rules from TOML prepend before `DefaultRules`. `Classifier` already supports arbitrary `[]Rule` — `NewClassifier(rules []Rule)` is the constructor. No change to `classifier.go` itself.
|
||||
|
||||
**`RuleConfig` TOML struct:**
|
||||
|
||||
```toml
|
||||
[[rule]]
|
||||
protocol = "tcp"
|
||||
dst_port = 8443
|
||||
class = "myservice"
|
||||
```
|
||||
|
||||
### Key Data Types
|
||||
|
||||
1. **`gopacket.Packet`** → raw decoded packet from pcap; carries layer stack.
|
||||
2. **`ClassifiedPacket{Packet, Class TrafficClass, Bytes int}`** → labeled event.
|
||||
3. **`WindowSnapshot{ClassCounts map[TrafficClass]int, ClassBytes map[TrafficClass]int}`** → per-window aggregate; drives amplitude targets.
|
||||
4. **`[]float32` PCM block** → mixer output at 44100 Hz, mono; flows into LAME.
|
||||
|
||||
## Build Order (Phase Implications)
|
||||
|
||||
Build in dependency order — each layer is independently testable before the next is added:
|
||||
|
||||
```
|
||||
1. capture/ → can test: "does it open an interface and emit packets?"
|
||||
2. classify/ → can test: "does ICMP get labeled ICMP?" (synthetic packets)
|
||||
3. aggregate/ → can test: "does a 500ms window count correctly?"
|
||||
4. synth/ → can test: "does mixer output expected amplitude?" (no pcap needed)
|
||||
5. encode/ → can test: "does PCM produce valid MP3 bytes?"
|
||||
6. main.go wiring → integration: full end-to-end pipeline
|
||||
```go
|
||||
// config/config.go
|
||||
type RuleConfig struct {
|
||||
Protocol string `toml:"protocol"`
|
||||
DstPort uint16 `toml:"dst_port"`
|
||||
Class string `toml:"class"` // must match a name in [[class]] or a builtin class name
|
||||
}
|
||||
```
|
||||
|
||||
This ordering means:
|
||||
- **Phase 1** can deliver a working capture + classify pipeline writing JSON/text summaries — validating the hardest privilege/pcap risk early.
|
||||
- **Phase 2** delivers the synthesis engine in isolation — testable with synthetic `WindowSnapshot` inputs before any real traffic.
|
||||
- **Phase 3** wires them together with the MP3 encoder.
|
||||
**Merging in main.go:**
|
||||
|
||||
## Anti-Patterns
|
||||
```go
|
||||
userRules := config.ToClassifyRules(cfg.Rules) // []classify.Rule
|
||||
allRules := append(userRules, classify.DefaultRules...)
|
||||
classifier := classify.NewClassifier(allRules)
|
||||
```
|
||||
|
||||
### Anti-Pattern 1: Synchronous Per-Packet Audio Rendering
|
||||
**New `TrafficClass` values:** User-defined classes in TOML produce new `TrafficClass` string values (e.g., `"myservice"`). `AllClasses()` in `classify/types.go` is currently a hardcoded slice. For user-defined classes, `AllClasses()` cannot be the source of truth for bank layer construction. `NewBank` must instead iterate over whatever classes have a `FreqConfig` entry.
|
||||
|
||||
**What people do:** Generate one audio sample or tone event per packet — a 10 Gbps link produces 14M packets/sec, making synchronous render impossible.
|
||||
This is a critical integration point: `bank.go:NewBank` currently ranges over `classify.AllClasses()`. If user classes can appear, `NewBank` must accept the full config map and range over that instead.
|
||||
|
||||
**Why it's wrong:** Breaks at any real traffic volume; produces click-heavy output, not smooth drone.
|
||||
---
|
||||
|
||||
**Do this instead:** Batch packets into time windows (500ms–1s) and drive amplitude targets from the batch, not individual packets.
|
||||
## New vs Modified Components
|
||||
|
||||
### Anti-Pattern 2: Blocking Channel Sends in the Capture Path
|
||||
### New
|
||||
|
||||
**What people do:** Use unbuffered channels between PacketSource and classifier; slow classifier stalls the pcap ring buffer and causes kernel drops.
|
||||
| Component | Location | Purpose |
|
||||
|-----------|----------|---------|
|
||||
| `config` package | `config/config.go` | TOML struct, `Load()`, file discovery, merge with defaults |
|
||||
| `config/defaults.go` | optional split | `DefaultConfig()` wrapping existing `ClassFreqConfigs` values |
|
||||
|
||||
**Why it's wrong:** libpcap's kernel buffer is fixed-size; if userspace can't drain it fast enough, packets are silently dropped. For audio purposes this introduces silent gaps.
|
||||
### Modified
|
||||
|
||||
**Do this instead:** Use buffered channels (capacity 256–1024) between capture and classify. Drop packets on full buffer with a counter — acceptable for sonification, fatal to log completeness tools.
|
||||
| Component | Change | Impact |
|
||||
|-----------|--------|--------|
|
||||
| `synth/oscillator.go` | Add `Waveform` type + `waveform` field; dispatch in `Advance` | Self-contained; no caller signature breaks except `NewOscillator` |
|
||||
| `synth/config.go` | Add `Waveform Waveform` field to `FreqConfig`; default to `WaveformSine` | Requires `NewLayer` to pass waveform to `NewOscillator` |
|
||||
| `synth/layer.go` | Pass `cfg.Waveform` to `NewOscillator` | One-line change |
|
||||
| `synth/bank.go` | Accept `map[classify.TrafficClass]FreqConfig` param instead of reading global; range over param keys not `AllClasses()` | Decouples bank from global; enables user classes |
|
||||
| `encode/mp3.go` | Accept `*config.Config` or merged `FreqConfig` map; pass to `NewBank` | Thin forwarding change |
|
||||
| `cmd/netsynth/main.go` | Add `--config` flag; load config; prepend user rules; pass config to `RunSynthesis` | Touches both `runLiveMode` and `runPcapMode` |
|
||||
| `classify/rules.go` | No change — `DefaultRules` stays as the fallback | Unchanged |
|
||||
| `classify/classifier.go` | No change — already accepts `[]Rule` | Unchanged |
|
||||
| `classify/types.go` | `AllClasses()` may need a note that it returns only builtins; bank no longer relies on it | Low risk; document only |
|
||||
|
||||
### Anti-Pattern 3: CGo MP3 Encoding in the Hot Audio Loop
|
||||
---
|
||||
|
||||
**What people do:** Call `lame.Encode()` synchronously inside the frame-render loop, stalling synthesis.
|
||||
## Data Flow Changes
|
||||
|
||||
**Why it's wrong:** CGo calls carry overhead; libmp3lame may block on I/O; this disrupts the synthesis clock.
|
||||
### v1.0 Flow (config hardcoded)
|
||||
|
||||
**Do this instead:** The synth goroutine pushes PCM blocks onto a buffered channel; a separate encoder goroutine drains and encodes. On shutdown, close the PCM channel and drain completely before `lame.Close()`.
|
||||
```
|
||||
main.go
|
||||
└─ classify.NewClassifier(classify.DefaultRules)
|
||||
└─ encode.RunSynthesis(snapshots, path)
|
||||
└─ synth.NewBank(1.0)
|
||||
└─ ClassFreqConfigs[class] ← global, hardcoded
|
||||
```
|
||||
|
||||
### Anti-Pattern 4: Global Mutable State for Class Frequency Mapping
|
||||
### v1.1 Flow (config injected)
|
||||
|
||||
**What people do:** Use a global `map[TrafficClass]float64` for frequency assignments modified at runtime.
|
||||
```
|
||||
main.go
|
||||
└─ config.Load(configPath) ← NEW: resolve path, parse TOML, merge defaults
|
||||
└─ cfg *config.Config
|
||||
└─ classify.NewClassifier(
|
||||
append(config.ToClassifyRules(cfg.Rules), classify.DefaultRules...)
|
||||
) ← user rules prepend built-ins
|
||||
└─ encode.RunSynthesis(snapshots, path, cfg.FreqConfigs())
|
||||
└─ synth.NewBank(1.0, freqConfigs) ← map passed in, not read from global
|
||||
└─ freqConfigs[class] ← merged: user overrides + defaults
|
||||
```
|
||||
|
||||
**Why it's wrong:** Race conditions; hard to test; makes the mapping invisible to callers.
|
||||
---
|
||||
|
||||
**Do this instead:** Pass the mapping table as an immutable struct at construction time. Auto-clustered classes append to a local slice protected by a mutex inside the clusterer — not a global.
|
||||
## Suggested Build Order
|
||||
|
||||
## Integration Points
|
||||
The following order minimizes integration risk. Each step is independently testable before the next begins.
|
||||
|
||||
### External Services
|
||||
### Step 1: Waveform types in `synth/oscillator.go`
|
||||
|
||||
| Dependency | Integration Pattern | Notes |
|
||||
|------------|---------------------|-------|
|
||||
| libpcap / pcap.h | CGo via gopacket/pcap — requires libpcap-dev at build time | Can substitute AF_PACKET (linux only) to avoid CGo in capture; still needs root |
|
||||
| libmp3lame | CGo via go-lame — requires libmp3lame-dev at build time | Binary distribution requires static linking or Docker; pure-Go MP3 (e.g. oto + gmp3) is an option but quality/speed tradeoff |
|
||||
No external dependencies. Pure math. Testable with golden-sample unit tests (square wave sample at phase 0.25 should be 1.0, etc.). Does not affect `Layer`, `Bank`, or `encode` yet.
|
||||
|
||||
### Internal Boundaries
|
||||
**Files changed:** `synth/oscillator.go` only.
|
||||
|
||||
| Boundary | Communication | Notes |
|
||||
|----------|---------------|-------|
|
||||
| capture ↔ classify | `chan gopacket.Packet` (buffered 512) | classify must never block capture |
|
||||
| classify ↔ aggregate | `chan ClassifiedPacket` (buffered 1024) | aggregate is slower (ticker-driven); buffer absorbs bursts |
|
||||
| aggregate ↔ synth | `chan WindowSnapshot` (buffered 4) | synth consumes synchronously per window; small buffer is fine |
|
||||
| synth ↔ encode | `chan []float32` (buffered 8 blocks) | encoder runs in separate goroutine to decouple CGo latency |
|
||||
| all stages ↔ main | `chan struct{}` done channel | closed on Ctrl+C; all stages select on it |
|
||||
### Step 2: Wire `Waveform` through `FreqConfig` and `Layer`
|
||||
|
||||
## Scaling Considerations
|
||||
Add `Waveform` to `FreqConfig`. Update `NewLayer` to pass it to `NewOscillator`. `ClassFreqConfigs` entries default to `WaveformSine` (zero value — valid if `WaveformSine = 0`).
|
||||
|
||||
This is a single-binary CLI tool, not a distributed service. Scaling concerns are throughput-based:
|
||||
Existing tests continue to pass without modification since all existing configs use the zero-value waveform.
|
||||
|
||||
| Traffic Rate | Architecture Adjustments |
|
||||
|--------------|--------------------------|
|
||||
| Home/office (< 10K pps) | Default design handles easily with no tuning |
|
||||
| Datacenter (100K–1M pps) | Increase capture buffer size; consider AF_PACKET with TPACKET_V3 ring buffer instead of pcap; classify goroutine may need fan-out to 2–4 workers |
|
||||
| Line-rate 10G (> 5M pps) | Out of scope for v1 ambient audio tool — synthesis granularity at 500ms windows means exact packet-level accuracy is not required |
|
||||
**Files changed:** `synth/config.go`, `synth/layer.go`.
|
||||
|
||||
### Scaling Priorities
|
||||
### Step 3: Decouple `NewBank` from the global
|
||||
|
||||
1. **First bottleneck:** Kernel pcap buffer drops — mitigated by buffered channels and accepting lossy capture (fine for sonification).
|
||||
2. **Second bottleneck:** CGo encoding latency coupling synthesis clock — mitigated by decoupled encoder goroutine.
|
||||
Change `NewBank(tau float64)` to `NewBank(tau float64, cfgs map[classify.TrafficClass]FreqConfig)`. Update `encode/mp3.go:RunSynthesis` to pass `synth.ClassFreqConfigs` as default.
|
||||
|
||||
At this point the system is functionally identical to v1.0 but `NewBank` no longer reads a global.
|
||||
|
||||
**Files changed:** `synth/bank.go`, `encode/mp3.go`.
|
||||
|
||||
### Step 4: `config` package — TOML structs and file discovery
|
||||
|
||||
Implement `config.Load()`, file discovery, and the `DefaultConfig()` function that wraps `synth.ClassFreqConfigs`. No TOML parsing yet — start with the struct definitions and the merge logic.
|
||||
|
||||
Add `github.com/BurntSushi/toml` dependency (`go get`).
|
||||
|
||||
**Files added:** `config/config.go`.
|
||||
|
||||
### Step 5: `--config` flag and user rule merging in `main.go`
|
||||
|
||||
Wire `config.Load()` into `run()`. Pass user rules to both `runLiveMode` and `runPcapMode`. Pass merged `FreqConfig` map to `RunSynthesis`.
|
||||
|
||||
At this point a minimal TOML config (empty file, or `[[rule]]` only) can be validated end-to-end.
|
||||
|
||||
**Files changed:** `cmd/netsynth/main.go`.
|
||||
|
||||
### Step 6: Custom frequency and waveform overrides in config
|
||||
|
||||
Implement the `[[class]]` TOML section parsing. Add `FreqConfigs()` method to `Config` that returns the merged map (user overrides applied over defaults). Write table-driven tests: "TOML sets HTTPS to 200 Hz sawtooth, bank layer for HTTPS uses 200 Hz sawtooth."
|
||||
|
||||
**Files changed:** `config/config.go`.
|
||||
|
||||
### Step 7: User-defined classes end-to-end
|
||||
|
||||
Support `[[class]]` entries with names not in `classify.AllClasses()`. These become new `TrafficClass` values. User `[[rule]]` entries pointing to these classes are prepended to `DefaultRules`. The bank creates layers for all classes in the merged `FreqConfig` map.
|
||||
|
||||
This step requires the most cross-package coordination but by this point each piece is already in place.
|
||||
|
||||
**Files changed:** `config/config.go`, `cmd/netsynth/main.go` (verification that unknown class names don't panic).
|
||||
|
||||
---
|
||||
|
||||
## Component Boundaries After v1.1
|
||||
|
||||
| Component | Responsibility | Communicates With |
|
||||
|-----------|---------------|-------------------|
|
||||
| `config` | TOML parsing, file discovery, merge logic, `DefaultConfig()` | `synth` (FreqConfig type), `classify` (Rule type) |
|
||||
| `synth/oscillator` | Phase-accumulator for sine/square/sawtooth/triangle | Used by `Layer` |
|
||||
| `synth/bank` | Accepts freq config map, constructs one `Layer` per entry | `encode` passes config map in |
|
||||
| `encode` | Receives config map from `main`, passes to `NewBank` | Thin pass-through |
|
||||
| `cmd/netsynth/main` | Loads config, merges rules, wires all stages | All packages |
|
||||
| `classify` | Rules engine (unchanged); `DefaultRules` stays as package-level var | `main` constructs with merged rules |
|
||||
|
||||
---
|
||||
|
||||
## Critical Integration Constraints
|
||||
|
||||
### `AllClasses()` Is Not the Source of Truth for Bank Construction
|
||||
|
||||
`bank.go` currently iterates `classify.AllClasses()` to construct layers. After v1.1, the bank must iterate the keys of the `FreqConfig` map passed to it. User-defined classes will not appear in `AllClasses()`. If this is not changed, user-defined class packets will be aggregated in `WindowSnapshot.Counts` but have no corresponding layer — they will produce silence and no error.
|
||||
|
||||
**Fix:** `NewBank` iterates `maps.Keys(cfgs)` (or equivalent range over the map), not `classify.AllClasses()`.
|
||||
|
||||
### Class Name Validation Must Happen at Config Load Time
|
||||
|
||||
If a `[[rule]]` references a class name that has no corresponding `[[class]]` entry and is not a builtin, the system will silently mis-classify packets into a layer that doesn't exist. Validate at `config.Load()` time: every class name in `[[rule]]` must resolve to either a builtin `TrafficClass` or a `[[class]]` entry in the same config.
|
||||
|
||||
### `encode.RunSynthesis` Signature Change Is a Breaking API Change
|
||||
|
||||
`encode.RunSynthesis(snapshots []classify.WindowSnapshot, outputPath string)` will need to accept the config. If any external code (tests, future callers) uses this signature, they will break. Keep the change to a single place and update all call sites in the same commit.
|
||||
|
||||
---
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
### Anti-Pattern: Reading Global `ClassFreqConfigs` from Multiple Places
|
||||
|
||||
If `NewBank`, `encode.RunSynthesis`, and config loading all reference the package-level `synth.ClassFreqConfigs`, the merge point becomes ambiguous. The fix (Step 3 above) centralizes config reading to one place: `config.DefaultConfig()` reads from `ClassFreqConfigs` once when building defaults; everything downstream receives the already-merged map.
|
||||
|
||||
### Anti-Pattern: Storing `Waveform` as a String Everywhere
|
||||
|
||||
Keeping `waveform` as a `string` from TOML all the way into `Oscillator` means every advance call parses or switches on a string. Parse the string to a `Waveform` int type at config-load time. The `Oscillator` field should be a typed `Waveform`, not `string`.
|
||||
|
||||
### Anti-Pattern: User Rules Appended After DefaultRules
|
||||
|
||||
User rules must **prepend** `DefaultRules`, not append. `DefaultRules` ends with catch-all rules (`DstPort: 0`) that match any TCP or UDP packet. Appending user rules after these catch-alls means they will never be reached.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- Go Pipeline patterns: [Go Concurrency Patterns: Pipelines and cancellation](https://go.dev/blog/pipelines) — HIGH confidence, official Go blog
|
||||
- SoNSTAR network sonification architecture: [Sonification of Network Traffic Flow for Monitoring and Situational Awareness, arXiv 1712.07029](https://arxiv.org/abs/1712.07029) — MEDIUM confidence (abstract only accessed)
|
||||
- gopacket channel API: [gopacket pkg.go.dev](https://pkg.go.dev/github.com/google/gopacket) — HIGH confidence, official package docs
|
||||
- bleep synthesizer architecture (Go): [GitHub bspaans/bleep](https://github.com/bspaans/bleep) — MEDIUM confidence (README inspection)
|
||||
- Waveform synthesis PCM patterns in Go: [Audio From Scratch With Go — Dylan Meeus](https://dylanmeeus.github.io/posts/audio-from-scratch-pt8/) — MEDIUM confidence
|
||||
- go-lame MP3 encoding: [go-lame pkg.go.dev](https://pkg.go.dev/github.com/sunicy/go-lame) — MEDIUM confidence
|
||||
- Drone amplitude/frequency modulation patterns: [Drone auralization model, Acta Acustica 2024](https://acta-acustica.edpsciences.org/articles/aacus/full_html/2024/01/aacus240076/aacus240076.html) — MEDIUM confidence
|
||||
- Direct code inspection: `synth/config.go`, `synth/oscillator.go`, `synth/bank.go`, `synth/layer.go`, `classify/classifier.go`, `classify/rules.go`, `classify/types.go`, `encode/mp3.go`, `cmd/netsynth/main.go` — HIGH confidence
|
||||
- BurntSushi/toml usage in production Go projects (Hugo, dep): MEDIUM confidence (well-known in Go ecosystem)
|
||||
- Phase-accumulator waveform synthesis formulas (square, sawtooth, triangle): HIGH confidence (standard DSP, textbook formulas)
|
||||
|
||||
---
|
||||
*Architecture research for: NetSynth — network-traffic-to-audio synthesis CLI (Go)*
|
||||
*Researched: 2026-03-24*
|
||||
|
||||
*Architecture research for: NetSynth v1.1 — custom sound mappings integration*
|
||||
*Researched: 2026-03-26*
|
||||
|
||||
+232
-137
@@ -1,184 +1,279 @@
|
||||
# Feature Research
|
||||
|
||||
**Domain:** Network traffic sonification CLI tool (packet capture → ambient MP3)
|
||||
**Researched:** 2026-03-24
|
||||
**Confidence:** MEDIUM — this is a niche domain; most comparable tools are research prototypes or GUI applications, not CLI tools. Table stakes are derived from tcpdump/packet-capture CLI conventions and sonification research literature.
|
||||
**Domain:** Network traffic sonification CLI tool (packet capture -> ambient MP3)
|
||||
**Researched:** 2026-03-24 (v1.0), updated 2026-03-26 (v1.1 custom sound mappings)
|
||||
**Confidence:** MEDIUM — niche domain; comparable tools are research prototypes or GUI applications, not CLI tools. Table stakes are derived from tcpdump/packet-capture CLI conventions and sonification research literature.
|
||||
|
||||
---
|
||||
|
||||
## Feature Landscape
|
||||
## v1.1 Feature Research: Custom Sound Mappings via TOML Config
|
||||
|
||||
### Table Stakes (Users Expect These)
|
||||
This section addresses the milestone question: "How do custom sound mapping config files typically work in audio/network tools? What are expected behaviors for config file loading, merging with defaults, validation, and error reporting?"
|
||||
|
||||
Features users assume exist. Missing these = product feels incomplete.
|
||||
### Config File Loading: Standard Behaviors Expected by CLI Users
|
||||
|
||||
| Feature | Why Expected | Complexity | Notes |
|
||||
|---------|--------------|------------|-------|
|
||||
| Network interface selection (`-i eth0`) | tcpdump/tshark convention; every capture tool has this | LOW | `gopacket` exposes interface list; needs `--list-interfaces` companion flag |
|
||||
| Output file path flag (`-o output.mp3`) | Any file-producing CLI must let you name the output | LOW | Sensible default (e.g. `netsynth-<timestamp>.mp3`) reduces friction |
|
||||
| Graceful Ctrl+C capture stop with file save | Users expect the tool to cleanly finalize the MP3 on interrupt | MEDIUM | Need signal handler; partial synthesis must be flushed to encoder before exit |
|
||||
| Per-protocol sound distinction | Core value prop: ping sounds different from HTTPS noise | MEDIUM | Minimum recognizable set: ICMP, DNS, TCP (port 443), TCP (other), UDP |
|
||||
| Packet count / traffic summary on exit | Every capture tool prints capture statistics; users want to know what was heard | LOW | Print to stderr so it doesn't interfere with stdout pipeline use |
|
||||
| Privilege error message | `pcap` silently fails or panics without root/CAP_NET_RAW; users need a clear message | LOW | Detect EACCES / EPERM on open; print actionable message (`sudo` or capability hint) |
|
||||
| List available interfaces (`--list-interfaces`) | Users don't know interface names on unfamiliar machines | LOW | Wrap `pcap.FindAllDevs()`; print name + description |
|
||||
| Minimum viable duration guard | Zero-packet capture should not produce a corrupt/empty MP3 | LOW | Check sample count before encoding; exit with clear error if nothing was captured |
|
||||
Based on patterns from established CLI tools (git, golangci-lint, mise, hugo), users expect:
|
||||
|
||||
### Differentiators (Competitive Advantage)
|
||||
1. **Auto-discovery with a defined search order.** The tool looks in a conventional set of locations without requiring an explicit flag. Failing silently (no config found = run with defaults) is correct behavior.
|
||||
|
||||
Features that set the product apart. Not required, but valuable.
|
||||
2. **Explicit override via a flag.** `--config` (or `-c`) lets users point at a non-standard path. If `--config` is supplied and the file does not exist, that is an error — not silent fallback.
|
||||
|
||||
3. **Discovery search order (standard precedence):**
|
||||
- `--config path/to/file.toml` (explicit flag, highest priority)
|
||||
- `./netsynth.toml` (working directory, project-local)
|
||||
- `$XDG_CONFIG_HOME/netsynth/config.toml` (defaults to `~/.config/netsynth/config.toml`)
|
||||
- No config found → run with all defaults (not an error)
|
||||
|
||||
This is the pattern used by git (`.git/config` -> `~/.gitconfig` -> `/etc/gitconfig`), golangci-lint (`.golangci.yml` in working dir), and mise (`mise.toml` -> `~/.config/mise/config.toml`). **HIGH confidence** — XDG Base Directory Specification is the Linux/macOS standard.
|
||||
|
||||
4. **Partial overrides only — not a replacement config.** The config file expresses only what the user wants to change. Absent keys retain default values. This is universally expected: users do not want to replicate the full default table in order to change one frequency.
|
||||
|
||||
### Config File Merging: How Defaults and User Config Combine
|
||||
|
||||
The dominant pattern across well-designed CLI tools:
|
||||
|
||||
**Merge strategy: user values override defaults, defaults fill gaps.**
|
||||
|
||||
```
|
||||
builtin defaults <-- loaded first (in-code, always present)
|
||||
+
|
||||
user config file <-- loaded second (overrides per-key)
|
||||
=
|
||||
effective config <-- what the program runs with
|
||||
```
|
||||
|
||||
For NetSynth's classification rules specifically, there are two distinct semantics that must be clearly chosen:
|
||||
|
||||
- **Override by name:** User supplies a `[rule.DNS]` block that replaces the built-in DNS sound parameters. The predefined DNS rule's classification logic is kept; only its sound output changes.
|
||||
- **Prepend user rules:** User-defined rules are inserted before the built-in rule list, allowing them to match first (first-match-wins). This enables the user to add entirely new protocol-to-sound mappings.
|
||||
|
||||
Both are needed. They serve different use cases:
|
||||
- Sound overrides (change frequency/waveform for a known protocol) use the override-by-name pattern.
|
||||
- Custom traffic rules (classify "tcp port 8443 as MyApp") use prepend semantics.
|
||||
|
||||
### Validation: What Users Expect When Config Has Errors
|
||||
|
||||
Based on patterns in go-toml v2's strict mode and golangci-lint error reporting:
|
||||
|
||||
**Expected validation behaviors (roughly in order of importance):**
|
||||
|
||||
| Behavior | Why Expected | Go Implementation Note |
|
||||
|----------|--------------|----------------------|
|
||||
| Unknown keys caught and reported | Prevents silent typos (user writes `frequncy`, expects it to work) | `go-toml/v2` `DisallowUnknownFields()` or BurntSushi's `Undecoded()` check |
|
||||
| Line number in error message | Users need to know where the problem is | Both go-toml/v2 DecodeError and BurntSushi include position info |
|
||||
| Human-readable field path | "invalid value for `rules[0].waveform`" not "decode error" | go-toml v2's `DecodeError` produces contextualized messages |
|
||||
| Invalid enum values rejected | `waveform = "sqaure"` (typo) should list valid options | Post-decode validation loop with explicit error message listing valid values |
|
||||
| Out-of-range numbers rejected | `frequency = -50` or `frequency = 25000` should fail with reason | Post-decode bounds check with message |
|
||||
| Missing required fields in new rules | A user rule block missing `protocol` is ambiguous | Post-decode presence check |
|
||||
| Config error prevents startup | Do not silently ignore errors and run with partial config | Error should exit with non-zero and print the problem before capturing any packets |
|
||||
|
||||
**Critical:** Validation errors must surface before capture begins. A user who runs the tool, captures for 30 minutes, then gets a corrupt MP3 because a config value was silently ignored would rightly be frustrated.
|
||||
|
||||
### Error Reporting: Standard UX Patterns
|
||||
|
||||
From studying tools in the same class (golangci-lint, hugo, suricata):
|
||||
|
||||
- Print config errors to **stderr** (not stdout).
|
||||
- Prefix with the config file path: `netsynth.toml:12: unknown field "frequncy"`.
|
||||
- List ALL errors found in one pass rather than stopping at the first error. Users prefer fixing 5 things in one edit over 5 sequential runs.
|
||||
- Warn (not error) for non-fatal issues such as "config file found but empty" or "unknown field in a comment-like position" — but for NetSynth's scope, unknown keys should be hard errors to prevent silent misconfigurations.
|
||||
- On `--config path` flag with missing file: hard error immediately.
|
||||
- On auto-discovered config with missing file: silent success (no config = defaults).
|
||||
|
||||
---
|
||||
|
||||
## Table Stakes for v1.1
|
||||
|
||||
Features users expect in any CLI tool that introduces a config file. Missing these makes v1.1 feel incomplete.
|
||||
|
||||
| Feature | Why Expected | Complexity | Depends On |
|
||||
|---------|--------------|------------|------------|
|
||||
| TOML config file auto-discovery (`./netsynth.toml`, `~/.config/netsynth/config.toml`) | Standard CLI convention; users expect zero-flag discovery | LOW | New: config loader module |
|
||||
| `--config` flag for explicit path | Required when multiple configs exist or working dir is wrong | LOW | New: config loader + cobra flag |
|
||||
| Partial override semantics (absent keys retain defaults) | Users must not copy the entire default table to change one field | LOW | New: merge logic |
|
||||
| Custom frequency per known traffic class | Core v1.1 ask; directly maps to `synth.FreqConfig.BaseHz` | LOW | Existing `synth.ClassFreqConfigs` |
|
||||
| Custom waveform per known traffic class | Core v1.1 ask; maps to `synth.Oscillator.Advance()` harmonic shape | MEDIUM | Existing oscillator (needs waveform type support) |
|
||||
| User-defined classification rules with custom sounds | Core v1.1 ask; prepend to `classify.DefaultRules` | MEDIUM | Existing `classify.Rule` struct (needs `Class` name generation) |
|
||||
| Config validation with line-number errors | Users cannot fix config errors without location info | LOW | go-toml v2 DecodeError (built-in) |
|
||||
| Unknown field detection | Prevents silent typos | LOW | go-toml v2 `DisallowUnknownFields()` |
|
||||
| Startup-time validation (fail before capture) | No wasted captures with bad config | LOW | Load config in `cmd` root before starting capture |
|
||||
| Clear error message listing valid enum values | `waveform` has exactly 4 valid values; list them on error | LOW | Post-decode validation |
|
||||
|
||||
## Differentiators for v1.1
|
||||
|
||||
Features that make the config experience polished beyond the minimum.
|
||||
|
||||
| Feature | Value Proposition | Complexity | Notes |
|
||||
|---------|-------------------|------------|-------|
|
||||
| Auto-clustering of unrecognized traffic | Unknown traffic still gets a unique voice instead of being silently dropped — honest audio fingerprint | HIGH | Requires unsupervised clustering (e.g. flow-feature vector → k-means or simple hash bucketing); each cluster gets a deterministic frequency mapping |
|
||||
| Ambient/drone style output (layered sine harmonics) | Distinct from event-ping tools (Peep, SoNSTAR); slow tonal evolution makes long captures listenable | HIGH | Synthesize per-protocol drone layers; amplitude driven by time-windowed packet rate; mix layers before encoding |
|
||||
| Time-windowed amplitude evolution | Traffic volume changes over time are reflected in the audio; the mix evolves rather than being static | MEDIUM | Segment capture into N-second windows; compute per-layer gain per window; apply smooth gain ramps between windows |
|
||||
| Configurable time window duration (`--window 10`) | Lets users tune responsiveness vs. smoothness; research tools (SoNSTAR) expose this parameter | LOW | Default 10s; range 1–60s is sensible |
|
||||
| BPF capture filter support (`--filter "tcp port 443"`) | Power users want to scope what gets sonified; tcpdump BPF syntax is universally known | MEDIUM | Pass expression directly to `gopacket`/`pcap`; validate at startup before capture begins |
|
||||
| Offline pcap file input (`--read capture.pcap`) | Lets users sonify historical captures, not just live traffic; useful for analysis and demos | MEDIUM | Replace live capture source with `pcap.OpenOffline()`; time-compress or time-expand to fixed output duration |
|
||||
| Verbose protocol activity log to stderr (`--verbose`) | Developers and curious users want to see what was classified | LOW | Print per-window protocol breakdown table to stderr during capture |
|
||||
| Configurable output duration when reading pcap file (`--duration 30`) | Offline pcap may span hours; need ability to compress to a target audio length | LOW | Only meaningful with `--read`; scale time windows proportionally |
|
||||
| Single static binary (`go build`) | Eliminates dependency hell on target machines | LOW (build-time) | Go native; no CGo for MP3 encoding avoids runtime `.so` requirements — choose a pure-Go MP3 encoder |
|
||||
| `netsynth --print-config` command to dump effective config as TOML | Users want to see what defaults they're overriding; essential for creating a starting-point config file | LOW | Marshal `ClassFreqConfigs` + active rules to TOML; makes discoverability easy |
|
||||
| Config documentation via inline comments in generated TOML | When `--print-config` outputs commented TOML, users get self-documenting starting point | LOW | Write comment strings alongside marshaled output |
|
||||
| Named custom rules (user assigns a label) | User writes `name = "MyApp"` in a rule block; that name appears in the exit summary and `--verbose` output | LOW | Extend `classify.Rule` to carry optional display name |
|
||||
| Waveform preview hint in config error message | "valid waveforms: sine, square, sawtooth, triangle" inline with the error | LOW | Hard-code the valid set in the validator |
|
||||
| Harmonic override per class (not just base frequency) | Advanced users can tune the timbre, not just the pitch | MEDIUM | Requires exposing `HarmonicDef` slice in TOML schema; nesting adds parsing complexity |
|
||||
|
||||
### Anti-Features (Commonly Requested, Often Problematic)
|
||||
## Anti-Features for v1.1
|
||||
|
||||
Features that seem good but create problems.
|
||||
Features that seem natural but should be avoided.
|
||||
|
||||
| Feature | Why Requested | Why Problematic | Alternative |
|
||||
|---------|---------------|-----------------|-------------|
|
||||
| Real-time audio playback (speakers while capturing) | Feels more immediate; Peep and dmeldrum6/Network-Sonification do this | Requires platform audio APIs (ALSA/CoreAudio/WASAPI), cross-platform complexity triples; conflicts with file-output simplicity; latency/buffering bugs; CGo or external library dependency | File output only; user can pipe MP3 to `mpv`/`afplay` themselves after capture |
|
||||
| GUI or web dashboard | Visually richer; existing tools like Network-Sonification are GUI-first | Negates single-binary CLI value; doubles scope; Go GUI toolkits are immature or require CGo | Emit stderr text summary; let external tools consume the MP3 |
|
||||
| Custom sound mapping configuration file | Power-user request; SoNSTAR supports per-user sound uploads | Configuration surface area is large; predefined + auto-cluster covers the use case adequately for v1; config files introduce parsing/validation work | Well-chosen defaults + auto-cluster for unknowns; defer custom mapping to v2 |
|
||||
| Rhythmic/percussive output mode | Some sonification tools use discrete note triggers per packet | Ambient/drone style is the deliberate differentiator; per-packet triggers at high traffic volumes produce noise, not information | Stick to amplitude-modulated harmonic drones; volume changes carry the rhythm implicitly |
|
||||
| Deep-packet inspection / payload parsing | Users might want to hear HTTP body content, TLS handshake details | Requires reassembly, encryption handling, legal concerns about payload interception; massive complexity | Classify by header fields only (port, protocol, flags, packet size); that is sufficient for the audio fingerprint goal |
|
||||
| Streaming MP3 output (write while capturing) | Real-time preview of what's being synthesized | MP3 frame boundaries and VBR headers require the full file to be finalized; streaming output would produce a non-standard file | Write to temp buffer during capture, finalize and flush on Ctrl+C |
|
||||
| Anomaly detection / alerting | Natural extension once you have classified traffic | Adds a monitoring-tool responsibility on top of the audio-fingerprint responsibility; these are different user jobs | Stick to "produce an audio fingerprint"; anomaly detection is a separate tool |
|
||||
| Anti-Feature | Why Avoid | What to Do Instead |
|
||||
|--------------|-----------|-------------------|
|
||||
| Config file hot-reload during capture | Appears useful but mid-capture parameter change would corrupt synthesis state and produce jarring audio discontinuities | Require restart to apply config changes; document this explicitly |
|
||||
| Environment variable config overrides | Adds a third precedence layer (flags > env > file > defaults) that increases combinatorial test surface with low user demand for this tool | Stick to flags + file + defaults; NetSynth is not a server needing 12-factor config |
|
||||
| Multiple config file includes / inheritance (`extends = "base.toml"`) | Sounds powerful, creates debugging nightmares when users do not understand the merge order | Single user config file merged with in-code defaults is sufficient; if a user needs multiple environments they can use `--config` |
|
||||
| YAML or JSON config format as alternatives | "Why not YAML?" is a common request; supporting multiple formats multiplies parser dependency surface and doubles validation code paths | TOML only; document the choice (TOML is unambiguous, has clean table syntax, is the standard for Go tooling) |
|
||||
| Silent partial load on validation error | Some tools load what they can and warn about the rest | Hard error on any invalid field; the user's intent for that field is unknown, so continuing is worse than stopping |
|
||||
| Config wizard / interactive setup | Out of scope for a CLI tool with a non-interactive model | Provide `--print-config` with comments as a self-service starting point |
|
||||
| Stereo pan position in config | Requested but explicitly deferred in PROJECT.md for this milestone | Out of scope for v1.1; document as v1.2 candidate |
|
||||
|
||||
---
|
||||
|
||||
## Feature Dependencies
|
||||
## Feature Dependencies for v1.1
|
||||
|
||||
```
|
||||
[Interface selection / list-interfaces]
|
||||
└──required-by──> [Live capture]
|
||||
[Config file loader (TOML parse + merge)]
|
||||
|
|
||||
+--provides--> [Custom frequency overrides] (maps to synth.FreqConfig.BaseHz)
|
||||
|
|
||||
+--provides--> [Custom waveform per class] (requires oscillator waveform dispatch)
|
||||
| |
|
||||
| +--requires--> [Waveform type in oscillator] (new: sine/square/sawtooth/triangle)
|
||||
|
|
||||
+--provides--> [User-defined classification rules]
|
||||
|
|
||||
+--requires--> [Dynamic TrafficClass generation] (new: user rule class names)
|
||||
+--prepended-to--> [classify.DefaultRules]
|
||||
|
||||
[Live capture] ──OR── [Offline pcap input]
|
||||
└──required-by──> [Protocol classification]
|
||||
└──required-by──> [Auto-clustering of unknowns]
|
||||
└──required-by──> [Per-protocol drone layer synthesis]
|
||||
└──required-by──> [Time-windowed amplitude evolution]
|
||||
└──required-by──> [Layer mixing]
|
||||
└──required-by──> [MP3 encoding & file output]
|
||||
|
||||
[BPF capture filter] ──enhances──> [Live capture]
|
||||
[Configurable time window] ──tunes──> [Time-windowed amplitude evolution]
|
||||
[Offline pcap + --duration] ──requires──> [Offline pcap input]
|
||||
[Verbose flag] ──enhances──> [Protocol classification] (reporting only, no data dependency)
|
||||
[Graceful Ctrl+C] ──requires──> [MP3 encoding & file output] (must flush before exit)
|
||||
[--config flag] --overrides--> [Config file loader search path]
|
||||
[--print-config] --reads--> [Effective config after merge] (new subcommand)
|
||||
```
|
||||
|
||||
### Dependency Notes
|
||||
### Dependency Notes for v1.1
|
||||
|
||||
- **Protocol classification requires Live capture OR Offline pcap:** These are the two data sources; everything downstream is source-agnostic.
|
||||
- **Time-windowed amplitude evolution requires Protocol classification:** You need classified packet counts per window before you can derive per-layer gain values.
|
||||
- **MP3 encoding requires Layer mixing:** You cannot encode until all layers for a time window are mixed to a PCM buffer.
|
||||
- **Graceful Ctrl+C requires MP3 encoding:** The signal handler must trigger the encode-and-flush path, not just `os.Exit`.
|
||||
- **Auto-clustering enhances Protocol classification:** It extends classification to traffic that doesn't match predefined rules; the audio pipeline treats cluster-assigned tones identically to predefined protocol tones.
|
||||
- **BPF filter conflicts with Offline pcap input (partial):** `pcap` supports BPF on offline files, so this works technically, but user expectation for offline mode is usually "sonify all traffic in the file" — document the interaction clearly.
|
||||
- **Waveform type is a new concept in the oscillator.** The v1.0 `Oscillator.Advance()` only does additive sine. To support square/sawtooth/triangle, the oscillator needs a `WaveformType` field and dispatch logic. This is an internal change, but it's required before waveform config can be wired up.
|
||||
- **User-defined rules require dynamic `TrafficClass` values.** v1.0 `TrafficClass` is a string type with predefined constants. User rules name their own classes (e.g., `"MyApp"`). The classifier already uses `TrafficClass` as a string; the `synth` layer needs to handle classes not in `ClassFreqConfigs` by looking up user-supplied sound parameters.
|
||||
- **Config loading must happen in `cmd` before the capture pipeline starts.** The cobra root command's `RunE` (or `PersistentPreRunE`) function loads and validates config, then passes effective config into the pipeline constructors. This is a structural change to `cmd/root.go`.
|
||||
- **`--print-config` is independent** of capture and can be implemented as a separate cobra subcommand reading only the config loader output.
|
||||
|
||||
---
|
||||
|
||||
## MVP Definition
|
||||
## Implementation Complexity Summary
|
||||
|
||||
### Launch With (v1)
|
||||
| Feature | Complexity | Reason |
|
||||
|---------|------------|--------|
|
||||
| Config file loader (TOML parse + merge + validation) | LOW-MEDIUM | go-toml v2 handles parsing; merge logic is a loop; validation is a post-decode pass |
|
||||
| Custom frequency per class | LOW | Direct map lookup override; one line per class |
|
||||
| Custom waveform per class | MEDIUM | Oscillator needs waveform dispatch (new `WaveformType`); synthesis loop changes |
|
||||
| User-defined classification rules | MEDIUM | Dynamic class names; synth layer must handle unknown class names via config lookup |
|
||||
| `--config` flag + auto-discovery | LOW | Cobra flag + os.Stat checks on 2-3 paths |
|
||||
| `--print-config` subcommand | LOW | Marshal effective config to TOML; add comments |
|
||||
| Named custom rules in exit summary | LOW | `classify.Rule` struct gains optional `Name string` field |
|
||||
|
||||
Minimum viable product — what's needed to validate the concept.
|
||||
|
||||
- [ ] Network interface selection (`-i`) and `--list-interfaces` — required for capture
|
||||
- [ ] Live packet capture with Ctrl+C stop — core interaction model
|
||||
- [ ] Protocol classification: ICMP, DNS, TCP/443, TCP/other, UDP — minimum set for a recognizable fingerprint
|
||||
- [ ] Auto-clustering of unrecognized traffic (simple hash-bucketing by port/proto is acceptable for v1) — honest representation of full traffic
|
||||
- [ ] Per-protocol ambient drone layer synthesis (sine harmonics, amplitude modulated by packet rate) — core differentiator
|
||||
- [ ] Time-windowed amplitude evolution (10s default) — makes the output dynamic
|
||||
- [ ] MP3 encoding and file output with sensible default filename — deliverable artifact
|
||||
- [ ] Capture statistics summary on exit (stderr) — basic UX courtesy
|
||||
- [ ] Privilege error detection and clear message — prevents silent failure
|
||||
|
||||
### Add After Validation (v1.x)
|
||||
|
||||
Features to add once core is working.
|
||||
|
||||
- [ ] BPF capture filter (`--filter`) — add when users report wanting to scope captures
|
||||
- [ ] Offline pcap file input (`--read`) — add when users want to sonify historical captures
|
||||
- [ ] Verbose protocol activity log (`--verbose`) — add when debugging/demo use cases emerge
|
||||
- [ ] Configurable time window duration (`--window`) — add if users report default feels too slow or too fast
|
||||
- [ ] Configurable output duration for pcap input (`--duration`) — depends on offline input being implemented
|
||||
|
||||
### Future Consideration (v2+)
|
||||
|
||||
Features to defer until product-market fit is established.
|
||||
|
||||
- [ ] Custom sound mapping configuration — defer; needs user research on what customization actually matters
|
||||
- [ ] Improved clustering algorithm (k-means on flow features vs. simple hash) — defer until users report clusters feel meaningless
|
||||
- [ ] Multi-interface capture — defer; adds complexity to packet deduplication
|
||||
**No new external dependencies required.** go-toml v2 is the only addition to `go.mod`.
|
||||
|
||||
---
|
||||
|
||||
## Feature Prioritization Matrix
|
||||
## TOML Schema Sketch (Informational)
|
||||
|
||||
| Feature | User Value | Implementation Cost | Priority |
|
||||
|---------|------------|---------------------|----------|
|
||||
| Interface selection + list-interfaces | HIGH | LOW | P1 |
|
||||
| Live capture with Ctrl+C stop | HIGH | LOW | P1 |
|
||||
| Protocol classification (ICMP, DNS, TCP, UDP) | HIGH | MEDIUM | P1 |
|
||||
| Ambient drone synthesis (per-protocol layers) | HIGH | HIGH | P1 |
|
||||
| Time-windowed amplitude evolution | HIGH | MEDIUM | P1 |
|
||||
| MP3 encoding + file output | HIGH | MEDIUM | P1 |
|
||||
| Capture stats on exit | MEDIUM | LOW | P1 |
|
||||
| Privilege error message | MEDIUM | LOW | P1 |
|
||||
| Auto-clustering of unknown traffic | MEDIUM | MEDIUM | P1 |
|
||||
| BPF capture filter | MEDIUM | MEDIUM | P2 |
|
||||
| Offline pcap file input | MEDIUM | MEDIUM | P2 |
|
||||
| Verbose flag | LOW | LOW | P2 |
|
||||
| Configurable time window | LOW | LOW | P2 |
|
||||
| Custom sound mapping | LOW | HIGH | P3 |
|
||||
| Real-time playback | LOW | HIGH | P3 (anti-feature, avoid) |
|
||||
This is not a binding decision — it informs the roadmap's implementation phase. The schema should feel natural to a user who has seen other Go tool configs (golangci-lint, goreleaser).
|
||||
|
||||
**Priority key:**
|
||||
- P1: Must have for launch
|
||||
- P2: Should have, add when possible
|
||||
- P3: Nice to have, future consideration
|
||||
```toml
|
||||
# Override built-in protocol sounds
|
||||
[classes.HTTPS]
|
||||
frequency = 220.0
|
||||
waveform = "square" # sine | square | sawtooth | triangle
|
||||
|
||||
[classes.DNS]
|
||||
frequency = 90.0
|
||||
|
||||
# Add custom classification rules (prepended before built-in rules, first-match-wins)
|
||||
[[rules]]
|
||||
name = "Internal API"
|
||||
protocol = "tcp"
|
||||
port = 8443
|
||||
frequency = 300.0
|
||||
waveform = "sawtooth"
|
||||
|
||||
[[rules]]
|
||||
name = "Game Traffic"
|
||||
protocol = "udp"
|
||||
port = 27015
|
||||
frequency = 450.0
|
||||
waveform = "triangle"
|
||||
```
|
||||
|
||||
Key schema design choices:
|
||||
- `[classes.X]` uses the same class name strings already used in `--verbose` output and exit summary (`HTTPS`, `DNS`, etc.) — no new naming system to learn.
|
||||
- `[[rules]]` is a TOML array of tables, consistent with how goreleaser and other tools express lists of items.
|
||||
- `protocol` and `port` map directly to the existing `classify.Rule` fields, minimizing translation.
|
||||
- Waveform is an enum string, not an integer — readable and self-documenting in the config file.
|
||||
|
||||
---
|
||||
|
||||
## v1.0 Feature Landscape (Retained from Original Research)
|
||||
|
||||
### Table Stakes (v1.0)
|
||||
|
||||
| Feature | Why Expected | Complexity | Notes |
|
||||
|---------|--------------|------------|-------|
|
||||
| Network interface selection (`-i eth0`) | tcpdump/tshark convention | LOW | Implemented: v1.0 |
|
||||
| Output file path flag (`-o output.mp3`) | Any file-producing CLI | LOW | Implemented: v1.0 |
|
||||
| Graceful Ctrl+C with file save | Users expect clean finalize | MEDIUM | Implemented: v1.0 |
|
||||
| Per-protocol sound distinction | Core value prop | MEDIUM | Implemented: v1.0, 12 rules |
|
||||
| Packet count / traffic summary on exit | Every capture tool does this | LOW | Implemented: v1.0 |
|
||||
| Privilege error message | Silent pcap failure is confusing | LOW | Implemented: v1.0 |
|
||||
| List available interfaces (`--list-interfaces`) | Users don't know interface names | LOW | Implemented: v1.0 |
|
||||
| Minimum viable duration guard | Zero-packet = no corrupt MP3 | LOW | Implemented: v1.0 |
|
||||
|
||||
### Differentiators (v1.0)
|
||||
|
||||
| Feature | Value Proposition | Complexity | Status |
|
||||
|---------|-------------------|------------|--------|
|
||||
| Auto-clustering of unrecognized traffic | Honest audio fingerprint | HIGH | Implemented: hash-bucket, 4 classes |
|
||||
| Ambient/drone style (layered sine harmonics) | Distinct from event-ping tools | HIGH | Implemented: v1.0 |
|
||||
| Time-windowed amplitude evolution | Mix evolves dynamically | MEDIUM | Implemented: 500ms windows + EMA |
|
||||
| BPF capture filter (`--filter`) | Power users scope what's sonified | MEDIUM | Implemented: v1.0 |
|
||||
| Offline pcap file input (`--read`) | Sonify historical captures | MEDIUM | Implemented: v1.0 |
|
||||
| Verbose protocol activity log (`--verbose`) | Developers see classifications | LOW | Implemented: v1.0 |
|
||||
|
||||
### Anti-Features (v1.0)
|
||||
|
||||
| Feature | Why Avoided |
|
||||
|---------|-------------|
|
||||
| Real-time audio playback | Platform audio API complexity; file output is correct |
|
||||
| GUI or web dashboard | Negates single-binary CLI value |
|
||||
| Custom sound mapping (v1.0) | Deferred to v1.1 — now the current milestone |
|
||||
| Rhythmic/percussive output | Ambient/drone is the deliberate differentiator |
|
||||
| Deep-packet inspection | Massive complexity; header classification sufficient |
|
||||
| Streaming MP3 output | MP3 finalization requires full buffer |
|
||||
| Anomaly detection / alerting | Different user job |
|
||||
|
||||
---
|
||||
|
||||
## Competitor Feature Analysis
|
||||
|
||||
| Feature | SoNSTAR (Python, research) | Network-Sonification (C#, Windows GUI) | Peep (C, Unix, 2000) | NetSynth (our approach) |
|
||||
|---------|----------------------------|-----------------------------------------|----------------------|------------------------|
|
||||
| Interface selection | Interactive prompt | GUI dropdown | Config file | CLI flag `-i` |
|
||||
| Protocol coverage | TCP flag states | TCP, UDP, HTTP, HTTPS, DNS, ICMP | Any syslog-able event | ICMP, DNS, TCP, UDP + auto-cluster |
|
||||
| Sound style | Recorded natural sounds (forest ambience) | Waveform shapes per protocol (sine/square/triangle) | Discrete event sounds | Synthesized harmonic drones |
|
||||
| Output | Real-time audio (Max/MSP) | Real-time audio (WPF) | Real-time audio (Unix audio) | MP3 file |
|
||||
| Time aggregation | Configurable window (default 20s) | Per-packet event | Per-event | Configurable window (default 10s) |
|
||||
| CLI/scriptable | Partial (Python prompts) | No (GUI only) | Yes (daemon) | Yes (single binary, flags) |
|
||||
| Offline pcap input | No | No | No | v1.x |
|
||||
| Auto-clustering | No | No | No | Yes (v1 hash-bucket) |
|
||||
| Single binary | No | No | No | Yes (Go) |
|
||||
| Open source | Yes | Yes | Yes | Intended |
|
||||
| Feature | SoNSTAR (Python) | Network-Sonification (C# GUI) | Peep (C, Unix) | NetSynth v1.0 | NetSynth v1.1 |
|
||||
|---------|-----------------|-------------------------------|----------------|----------------|----------------|
|
||||
| Custom sound config | No | No | Config file (fixed format) | No | Yes (TOML) |
|
||||
| Config file discovery | n/a | n/a | Hardcoded path | n/a | XDG + working dir |
|
||||
| Partial override semantics | n/a | n/a | Full replacement | n/a | Partial override |
|
||||
| Waveform selection | Recorded samples | sine/square/triangle | Fixed | sine only | sine/square/sawtooth/triangle |
|
||||
| Custom classification rules | No | No | No | No | Yes (user-defined port/proto rules) |
|
||||
| Named custom classes | n/a | n/a | n/a | n/a | Yes (appears in summary output) |
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [SoNSTAR: Sonification of Networks for SiTuational AwaReness — Paul Vickers](https://paulvickers.github.io/SoNSTAR/)
|
||||
- [Sonification of network traffic flow for monitoring and situational awareness — PLOS One](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0195948)
|
||||
- [SoNSTAR GitHub repository — nuson/SoNSTAR](https://github.com/nuson/SoNSTAR)
|
||||
- [Network-Sonification GitHub — dmeldrum6/Network-Sonification](https://github.com/dmeldrum6/Network-Sonification)
|
||||
- [Peep (The Network Auralizer): Monitoring Your Network With Sound — USENIX 2000](https://www.usenix.org/legacyurl/peep-network-auralizer-monitoring-your-network-sound)
|
||||
- [Sonification of DDoS Attacks — Imperva](https://www.imperva.com/blog/archive/sonification-of-ddos-attacks/)
|
||||
- [Data Sonification Toolkit — Sound and data parameters](https://www.sonificationkit.com/data-sonification/concepts/sound-and-data-parameters)
|
||||
- [tcpdump man page — tcpdump.org](https://www.tcpdump.org/manpages/tcpdump.1.html)
|
||||
- [The Sound of Data: A gentle introduction to sonification — Programming Historian](https://programminghistorian.org/en/lessons/sonification)
|
||||
- [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir/latest/) — standard for `~/.config` discovery path
|
||||
- [adrg/xdg — Go XDG implementation](https://github.com/adrg/xdg) — if explicit XDG library is needed (probably not for NetSynth's 2-path lookup)
|
||||
- [pelletier/go-toml v2 — strict mode and DecodeError](https://pkg.go.dev/github.com/pelletier/go-toml/v2) — recommended TOML library; DisallowUnknownFields() and human-readable errors
|
||||
- [BurntSushi/toml — Undecoded() for unknown key detection](https://github.com/BurntSushi/toml) — alternative; simpler API but less actively maintained
|
||||
- [Building CLI Applications with Go: Cobra and Viper Guide (2026)](https://dasroot.net/posts/2026/03/building-cli-applications-go-cobra-viper/) — config loading patterns in Cobra CLI tools
|
||||
- [A Guide to TOML in Golang — kelche.co](https://www.kelche.co/blog/go/toml/) — go-toml v2 vs BurntSushi comparison and practical examples
|
||||
- [Configuration | mise-en-place](https://mise.jdx.dev/configuration.html) — example of working-dir + XDG config discovery
|
||||
- [golangci-lint configuration](https://golangci-lint.run/docs/configuration/cli/) — real-world example of partial override config in a Go CLI tool
|
||||
- [Online Tone Generator — waveform types](https://onlinetonegenerator.com/) — confirms sine/square/sawtooth/triangle as the standard 4 waveform set
|
||||
|
||||
---
|
||||
*Feature research for: network traffic sonification CLI (NetSynth)*
|
||||
*Researched: 2026-03-24*
|
||||
*v1.0 research: 2026-03-24*
|
||||
*v1.1 custom sound mappings research: 2026-03-26*
|
||||
|
||||
+391
-213
@@ -1,333 +1,511 @@
|
||||
# Pitfalls Research
|
||||
|
||||
**Domain:** Network-traffic-to-audio synthesis CLI tool (Go)
|
||||
**Researched:** 2026-03-24
|
||||
**Confidence:** HIGH (packet capture / CGo pitfalls verified against official issues and docs; audio synthesis pitfalls cross-referenced against encoder project post-mortems and DSP literature)
|
||||
**Researched:** 2026-03-26 (v1.1 update — TOML config, waveform types, user-defined rules)
|
||||
**Confidence:** HIGH (TOML decoder behaviors verified against pkg.go.dev official docs and issue trackers; audio synthesis aliasing verified against DSP literature; config merging verified against BurntSushi/toml issue #47 and go-toml issue #252)
|
||||
|
||||
---
|
||||
|
||||
## Critical Pitfalls
|
||||
## v1.1 Milestone Pitfalls (New)
|
||||
|
||||
### Pitfall 1: Using `google/gopacket` Instead of the Active Community Fork
|
||||
These pitfalls are specific to adding TOML config, waveform types, and user-defined classification rules to the existing NetSynth codebase.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall A1: TOML Unmarshal Silently Overwrites Pre-filled Defaults with Zero Values
|
||||
|
||||
**What goes wrong:**
|
||||
The original `github.com/google/gopacket` repository is unmaintained. Bugs go unpatched, open PRs accumulate, and compatibility with newer Go versions degrades. Projects that import it are pinned to a stale library.
|
||||
You initialize a `Config` struct with built-in defaults, then call `toml.Unmarshal` to layer in user overrides. Any field the user *omits* from their TOML file is set to its Go zero value (`0`, `""`, `false`, `nil`) by the decoder — overwriting your defaults. A user who writes only `[sounds.DNS]` in their config file to change the DNS tone ends up wiping every other class back to zero Hz.
|
||||
|
||||
**Why it happens:**
|
||||
`google/gopacket` has enormous search mindshare and most tutorials still reference it. Developers reach for the first result without checking maintenance status.
|
||||
Both `BurntSushi/toml` and `pelletier/go-toml` v1 do not distinguish between "key was absent" and "key was explicitly set to zero". The decoder reflects over the struct and writes zero for every absent key. This was explicitly reported as a bug in BurntSushi/toml issue #47 and go-toml issue #252. go-toml v2 partially addresses it but still zeros primitive-type fields that are absent.
|
||||
|
||||
**How to avoid:**
|
||||
Import `github.com/gopacket/gopacket` (the community fork, v1.5.0 released November 2025, requires Go 1.24+). Major projects including Cilium have already migrated. Treat `google/gopacket` as deprecated.
|
||||
**Consequences:**
|
||||
- All non-overridden traffic classes play silence (0 Hz oscillator)
|
||||
- Classification rules get zeroed if user only partially fills `[[rules]]`
|
||||
- EMA tau, whisper floor, gain, and other synth parameters reset to 0
|
||||
|
||||
**Warning signs:**
|
||||
- `go.mod` referencing `github.com/google/gopacket`
|
||||
- Build errors on Go 1.21+ not fixed upstream
|
||||
**Prevention:**
|
||||
Use pointer fields (`*float64`, `*string`) in the decoded struct to distinguish "not provided" (nil pointer) from "explicitly set to zero" (non-nil pointer to 0). Apply a merge step: iterate over the decoded struct, and for each pointer field that is nil, keep the built-in default. For slice fields (like `[]RuleConfig`), nil slice means "user did not provide rules" — preserve defaults; non-nil empty slice (`[]RuleConfig{}`) means "user explicitly cleared rules" — respect that.
|
||||
|
||||
```go
|
||||
// In config struct, use pointers for optional overrides:
|
||||
type SoundConfig struct {
|
||||
FreqHz *float64 `toml:"freq_hz"`
|
||||
Waveform *string `toml:"waveform"`
|
||||
}
|
||||
|
||||
// Merge: for each class, override only non-nil fields
|
||||
func mergeSound(base synth.FreqConfig, override SoundConfig) synth.FreqConfig {
|
||||
if override.FreqHz != nil {
|
||||
base.BaseHz = *override.FreqHz
|
||||
}
|
||||
if override.Waveform != nil {
|
||||
base.WaveformType = *override.Waveform
|
||||
}
|
||||
return base
|
||||
}
|
||||
```
|
||||
|
||||
**Detection:**
|
||||
- User reports that classes they did not configure now produce no sound
|
||||
- Unit test: load a config that overrides only one class; verify all other classes retain built-in Hz values
|
||||
|
||||
**Phase to address:**
|
||||
Phase 1 (packet capture scaffolding) — set the correct import path from day one; migrating later is a find-and-replace across the whole codebase.
|
||||
Config loading phase (first phase of v1.1). Get the pointer-and-merge pattern established before wiring config into the bank. Retrofitting after the bank construction is wired is a significant churn.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 2: CGo Destroys the "Single Binary" Promise
|
||||
### Pitfall A2: BurntSushi/toml Silently Ignores Typos in Field Names
|
||||
|
||||
**What goes wrong:**
|
||||
`gopacket/pcap` requires `libpcap` via CGo. By default Go produces a dynamically linked binary. On a target machine without `libpcap.so` installed, the binary silently or loudly fails with `error while loading shared libraries: libpcap.so.0.8`. The "just copy the binary" distribution story breaks completely.
|
||||
A user writes `freq_hz = 440` but the struct tag is `toml:"freq_hz"` — this works. However if the user writes `freqhz = 440` or `FreqHz = 440` or a misspelled `frek_hz = 440`, the library silently ignores the key. The user's override is never applied. No error is returned. The user thinks their config is active; it is not.
|
||||
|
||||
**Why it happens:**
|
||||
CGo is enabled by default and Go gives no compile-time warning that the resulting binary has a runtime C dependency. The binary runs perfectly on the build machine (which has libpcap-dev installed) and fails on clean machines.
|
||||
`BurntSushi/toml` by default silently discards keys that do not map to any struct field. This is the documented default behavior ("will ignore options in the TOML file that you don't use"). It is the opposite of "strict mode."
|
||||
|
||||
**How to avoid:**
|
||||
Choose one of these strategies before writing a line of capture code:
|
||||
1. **Fully static build**: `CGO_ENABLED=1 go build -ldflags "-linkmode 'external' -extldflags '-static'"` with `libpcap.a` present. Requires `musl-gcc` or equivalent on Alpine/musl.
|
||||
2. **pcapgo (pure Go)**: `gopacket/pcapgo` provides an `EthernetHandle` that avoids CGo entirely — lower performance but zero C dependency. Sufficient for ambient audio capture at non-Gbps rates.
|
||||
3. **Document the dependency explicitly**: If CGo/dynamic linking is accepted, `README` must state "requires `libpcap` (`apt install libpcap-dev` / `brew install libpcap`)".
|
||||
**Consequences:**
|
||||
- Silent misconfiguration: user's customization is invisible
|
||||
- Debugging is very hard — no error to trace back to the TOML file
|
||||
|
||||
**Warning signs:**
|
||||
- `CGO_ENABLED` not explicitly set in your build script
|
||||
- `ldd ./netsynth` shows `libpcap.so` as a dependency
|
||||
- No CI test on a minimal (Alpine, scratch Docker) container
|
||||
**Prevention:**
|
||||
Use `toml.Decode` (not `toml.Unmarshal`) to obtain `MetaData`, then call `md.Undecoded()` and return an error listing any keys that were not decoded. This is BurntSushi's documented strict-mode pattern.
|
||||
|
||||
```go
|
||||
md, err := toml.Decode(string(data), &cfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if keys := md.Undecoded(); len(keys) > 0 {
|
||||
return fmt.Errorf("unknown config keys (check for typos): %v", keys)
|
||||
}
|
||||
```
|
||||
|
||||
**Detection:**
|
||||
- Config change that should audibly alter the sound has no effect
|
||||
- Undecoded keys present but no warning/error logged
|
||||
|
||||
**Phase to address:**
|
||||
Phase 1 — this is a foundational architecture decision. Changing from dynamic to static after the fact is painful and causes build pipeline rewrites.
|
||||
Config loading phase. Implement strict decoding from the first config load function. Do not add this as an afterthought — it is the primary mechanism protecting users from silent misconfiguration.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 3: `CAP_NET_RAW` + Binary Location = Silent Failure on Linux
|
||||
### Pitfall A3: Naive Square/Sawtooth/Triangle Generation Produces Audible Aliasing Distortion
|
||||
|
||||
**What goes wrong:**
|
||||
On Ubuntu and many Linux distributions, `setcap cap_net_raw+eip ./netsynth` appears to succeed but the binary fails at runtime if it lives in `/home/user/bin`, `/tmp`, or any filesystem mounted `nosuid`. The kernel silently ignores the capability. AppArmor compounds this by enforcing path-based restrictions.
|
||||
Implementing waveforms by direct time-domain math — `sign(sin(phase))` for square, `2*frac(phase)-1` for sawtooth, `1-2*abs(frac(phase)-0.5)` for triangle — produces a waveform with infinite harmonics. At 44100 Hz, harmonics above 22050 Hz fold back into the audible range as aliasing. At the frequencies used in NetSynth (65–1047 Hz), aliasing from a naive square wave produces a buzzing distortion that is especially audible at higher drone frequencies and sounds like corruption rather than timbre.
|
||||
|
||||
**Why it happens:**
|
||||
Developers test from their build directory (`~/projects/netsynth/`) — a path frequently on a `nosuid` filesystem. The tool appears broken with no clear error message beyond "permission denied" or "you must be root."
|
||||
The mathematical waveforms are not bandlimited — they have infinite harmonic content. Direct sampling them at 44100 Hz aliases all energy above Nyquist back into the audible band. Developers who test at low frequencies (60–120 Hz) may not notice because the aliased harmonics land at very high frequencies with low perceptual impact; the problem worsens significantly above 400 Hz where aliases fold into the 1–5 kHz perceptually prominent range.
|
||||
|
||||
**How to avoid:**
|
||||
- Install to `/usr/local/bin` or `/usr/bin` for capability-based operation
|
||||
- Document two run modes: `sudo ./netsynth` (always works) vs. `setcap` (requires standard path)
|
||||
- In the CLI, detect permission failure and emit a clear message: "Packet capture requires root or CAP_NET_RAW. Run as root or: sudo setcap cap_net_raw+eip $(which netsynth)"
|
||||
- Test capability-mode explicitly from a non-home path in CI
|
||||
**Consequences:**
|
||||
- Square/sawtooth at SSH (330 Hz) and higher frequencies sounds harsh and buzzy
|
||||
- The effect worsens at higher frequencies, making SMTP (440 Hz) and DHCP (600 Hz) drones sound distorted
|
||||
- Aliasing cannot be filtered out post-synthesis (it is interleaved with desired signal)
|
||||
|
||||
**Warning signs:**
|
||||
- Testing only via `sudo go run .`
|
||||
- No test of the installed-binary path in README instructions
|
||||
- macOS-only development (macOS uses a different privilege model; Linux `nosuid` behavior won't surface)
|
||||
**Prevention:**
|
||||
Use additive synthesis — the approach already in use for sine waves in `oscillator.go`. The existing `Oscillator.Advance(harmonics []HarmonicDef)` computes `sin(2π * phase * ratio)` for each partial. Square, sawtooth, and triangle waveforms are all expressible as harmonic series:
|
||||
|
||||
- **Square:** odd harmonics only, amplitude `1/k` for harmonic `k`: ratios 1, 3, 5, 7, ... with amplitudes 1.0, 0.33, 0.20, 0.14, ... Truncate at Nyquist.
|
||||
- **Sawtooth:** all harmonics, amplitude `1/k`: ratios 1, 2, 3, 4, ... with amplitudes 1.0, 0.5, 0.33, 0.25, ... Truncate at Nyquist.
|
||||
- **Triangle:** odd harmonics, amplitude `1/k²`, alternating sign: ratios 1, 3, 5, ... with amplitudes 1.0, 0.11, 0.04, ... Truncate at Nyquist.
|
||||
|
||||
The truncation (only sum harmonics where `freq * ratio < sampleRate / 2`) is the critical step that makes the synthesis bandlimited. The existing `[]HarmonicDef` structure in `synth/config.go` already supports this — waveform type selection just requires generating the right harmonic series for each `FreqConfig`.
|
||||
|
||||
Waveform presets should be pre-computed `[]HarmonicDef` slices, not runtime computation of naive waveform math:
|
||||
|
||||
```go
|
||||
// BandlimitedHarmonics returns a bandlimited harmonic series for the given waveform type.
|
||||
// It truncates harmonics at Nyquist (sampleRate/2) to prevent aliasing.
|
||||
func BandlimitedHarmonics(waveform string, baseHz float64, sampleRate int) []HarmonicDef {
|
||||
nyquist := float64(sampleRate) / 2.0
|
||||
var defs []HarmonicDef
|
||||
switch waveform {
|
||||
case "square":
|
||||
for k := 1; float64(k)*baseHz < nyquist; k += 2 { // odd only
|
||||
defs = append(defs, HarmonicDef{Ratio: k, Amplitude: 1.0 / float64(k)})
|
||||
}
|
||||
case "sawtooth":
|
||||
for k := 1; float64(k)*baseHz < nyquist; k++ {
|
||||
defs = append(defs, HarmonicDef{Ratio: k, Amplitude: 1.0 / float64(k)})
|
||||
}
|
||||
case "triangle":
|
||||
sign := 1.0
|
||||
for k := 1; float64(k)*baseHz < nyquist; k += 2 {
|
||||
defs = append(defs, HarmonicDef{Ratio: k, Amplitude: sign / float64(k*k)})
|
||||
sign = -sign
|
||||
}
|
||||
default: // "sine"
|
||||
defs = []HarmonicDef{{Ratio: 1, Amplitude: 1.0}}
|
||||
}
|
||||
return defs
|
||||
}
|
||||
```
|
||||
|
||||
**Detection:**
|
||||
- Audible buzzing or grainy texture on drone layers above 300 Hz with non-sine waveforms
|
||||
- Square/sawtooth waveforms sound harsher than expected at high frequencies
|
||||
|
||||
**Phase to address:**
|
||||
Phase 1 (capture scaffolding) and the CLI UX phase — the error message is user-facing and needs to be explicit.
|
||||
Waveform type implementation phase. The design decision (additive synthesis, not direct waveform math) must be made before coding waveform support. Switching from direct math to additive after the fact requires rewriting the oscillator API.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 4: Packet Buffer Overflow Under Moderate Traffic Load
|
||||
### Pitfall A4: Waveform String Validation Fails Silently, Falls Back to Silence
|
||||
|
||||
**What goes wrong:**
|
||||
At high packet rates (busy LAN, server NIC), gopacket's kernel ring buffer fills faster than the processing goroutine consumes it. The OS drops packets silently. The tool appears to work, but 50-98% of packets never reach the classifier. The audio output misrepresents actual traffic.
|
||||
A user writes `waveform = "Sawtooth"` (capital S) or `waveform = "saw"` (abbreviation). The config loading code does a simple equality check (`if waveform == "sawtooth"`), finds no match, and either panics, silently emits silence, or applies a default without telling the user. In all cases the user's intent is invisible.
|
||||
|
||||
**Why it happens:**
|
||||
The default pcap buffer is 1-2 MB. Each packet triggers a cgo call (with `pcap` backend), creating per-packet overhead that compounds at speed. Developers test on quiet home networks and never observe drops.
|
||||
String-based enumerations in config files have no compile-time type checking. Case sensitivity and abbreviations are user expectations that must be explicitly handled.
|
||||
|
||||
**How to avoid:**
|
||||
- Set a large capture buffer explicitly: `handle.SetBufferSize(32 * 1024 * 1024)` (32 MB)
|
||||
- Use a non-blocking channel between capture and classification goroutines with a buffer of at least 1000 packets; drop metrics count drops so they are visible
|
||||
- For high-throughput scenarios, prefer `afpacket` backend over `pcap` — `afpacket` eliminates per-packet CGo calls and dramatically improves throughput (benchmark: 1.27 MB/s → 21.17 MB/s)
|
||||
- NetSynth's ambient audio goal tolerates lossy capture — document this explicitly so users understand the tool provides a statistical fingerprint, not a perfect census
|
||||
**Consequences:**
|
||||
- Silent misconfiguration: wrong waveform with no feedback
|
||||
- Hard to debug: config appears valid, sound is just wrong
|
||||
|
||||
**Warning signs:**
|
||||
- Capture and classification in a single goroutine
|
||||
- No `SetBufferSize` call
|
||||
- Testing only on loopback (`lo`) which has near-zero real packet rates
|
||||
**Prevention:**
|
||||
Normalize waveform strings at parse time (`strings.ToLower`, `strings.TrimSpace`), validate against the accepted set, and return an explicit error with the accepted values if the string is unrecognized:
|
||||
|
||||
```go
|
||||
var validWaveforms = map[string]struct{}{
|
||||
"sine": {}, "square": {}, "sawtooth": {}, "triangle": {},
|
||||
}
|
||||
func validateWaveform(s string) (string, error) {
|
||||
normalized := strings.ToLower(strings.TrimSpace(s))
|
||||
if _, ok := validWaveforms[normalized]; !ok {
|
||||
return "", fmt.Errorf("unknown waveform %q: must be one of sine, square, sawtooth, triangle", s)
|
||||
}
|
||||
return normalized, nil
|
||||
}
|
||||
```
|
||||
|
||||
**Phase to address:**
|
||||
Phase 1/2 (capture pipeline) — the goroutine architecture must be designed for async processing from the start. Retrofitting is a significant rewrite.
|
||||
Config validation step (same phase as config loading). Implement all string field validation in a single `validate(cfg Config) error` function called immediately after decoding.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 5: ZeroCopy Packet Data Use-After-Free
|
||||
### Pitfall A5: User Rules Appended After Catch-All Rules Are Unreachable
|
||||
|
||||
**What goes wrong:**
|
||||
`ZeroCopyReadPacketData()` returns a slice pointing into a buffer owned by the pcap handle. The next call to `ZeroCopyReadPacketData()` invalidates the previous slice's backing memory. If any goroutine holds a reference to old packet bytes and reads them after the next call, it reads corrupted or incorrect data. This produces silent data corruption — wrong protocol classifications, no crash.
|
||||
The existing `DefaultRules` slice ends with two catch-alls:
|
||||
|
||||
```go
|
||||
{Protocol: "tcp", DstPort: 0, Class: ClassOtherTCP},
|
||||
{Protocol: "udp", DstPort: 0, Class: ClassOtherUDP},
|
||||
```
|
||||
|
||||
If user-defined rules are simply appended to this slice (`append(DefaultRules, userRules...)`), the catch-alls match first (DstPort=0 matches any port for that protocol), and the user's rules are unreachable. Every custom rule maps to `ClassOtherTCP` or `ClassOtherUDP` instead. The user gets no sound from their custom class.
|
||||
|
||||
**Why it happens:**
|
||||
The zero-copy API looks identical to the copying API. Developers reach for it for performance without reading the "each call invalidates previous data" contract.
|
||||
The first-match-wins semantics of `Classifier.Classify()` mean ordering is semantically critical. `DefaultRules` is a named var that exists precisely as an ordered slice — the comment `// Catch-alls (must be last)` documents this constraint. But "must be last in the defaults" does not automatically mean "must be last in the final merged slice." Developers who concatenate slices without thinking about this invariant break the system.
|
||||
|
||||
**How to avoid:**
|
||||
Use `ReadPacketData()` (copies data) unless you have profiling evidence that allocation is a bottleneck. If `ZeroCopyReadPacketData()` is used, never pass the slice to another goroutine without first copying it: `data := append([]byte(nil), raw...)`.
|
||||
**Consequences:**
|
||||
- All user-defined rules are silently swallowed by catch-alls
|
||||
- User's custom class never activates
|
||||
- No error — the pipeline works, just wrong
|
||||
|
||||
**Warning signs:**
|
||||
- `ZeroCopyReadPacketData` in a goroutine-per-packet pattern
|
||||
- Intermittent wrong protocol classifications that are not reproducible
|
||||
- Using `gopacket.Lazy` decode with concurrent goroutines (the gopacket docs explicitly warn against this combination)
|
||||
**Prevention:**
|
||||
Always insert user rules *before* catch-all rules. The merge strategy must be: `specificDefaultRules + userRules + catchAllRules`. Implement this with an explicit split in the default rule set:
|
||||
|
||||
```go
|
||||
// In classify/rules.go, split into two exported slices:
|
||||
var SpecificRules = []Rule{ /* ICMP through DHCP */ }
|
||||
var CatchAllRules = []Rule{
|
||||
{Protocol: "tcp", DstPort: 0, Class: ClassOtherTCP},
|
||||
{Protocol: "udp", DstPort: 0, Class: ClassOtherUDP},
|
||||
}
|
||||
|
||||
// Merge function used by config loading:
|
||||
func MergeRules(userRules []Rule) []Rule {
|
||||
result := make([]Rule, 0, len(SpecificRules)+len(userRules)+len(CatchAllRules))
|
||||
result = append(result, SpecificRules...)
|
||||
result = append(result, userRules...)
|
||||
result = append(result, CatchAllRules...)
|
||||
return result
|
||||
}
|
||||
```
|
||||
|
||||
Alternatively, annotate each default rule with a `CatchAll bool` field and sort before use. The split-slice approach is simpler and more explicit.
|
||||
|
||||
**Detection:**
|
||||
- User-defined rule that should match traffic does not produce its custom sound
|
||||
- `--verbose` output shows traffic being classified as `OtherTCP`/`OtherUDP` instead of the custom class
|
||||
- Test: write a rule for port 8080, send HTTP traffic to port 8080, verify it hits the custom class and not `ClassOtherTCP`
|
||||
|
||||
**Phase to address:**
|
||||
Phase 1 (capture/decode) — establish the correct API choice at the read loop level.
|
||||
User-defined rules phase. The `classify/rules.go` split must be the first code change before any config loading logic references the rule slice.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 6: MP3 Output Is Corrupt or Unplayable Due to LAME Initialization Errors
|
||||
### Pitfall A6: User-Defined Classes Have No synth.FreqConfig Entry — Bank Panics or Plays Silence
|
||||
|
||||
**What goes wrong:**
|
||||
MP3 encoding via CGo LAME bindings requires calling `InitParams()` after setting all encoder parameters. Skipping or reordering this call produces a file with a valid `.mp3` extension that most players refuse to open or that plays as noise. The encoder returns no error from the encode calls themselves.
|
||||
`OscillatorBank.NewBank()` iterates over `classify.AllClasses()` and looks up each class in `ClassFreqConfigs`. A user-defined rule creates a new `TrafficClass` (e.g., `"my-api"`). This class is not in `AllClasses()`, so the bank has no layer for it. The aggregator increments a count for `"my-api"`, `RenderWindow` looks up `b.layers["my-api"]`, gets nil, and either panics (nil pointer dereference on `layer.AdvanceSample()`) or silently contributes nothing to the mix.
|
||||
|
||||
**Why it happens:**
|
||||
The LAME C API is stateful and order-dependent. Go wrappers vary in how much they enforce initialization order. Many tutorial examples show minimal code that happens to work for 44100 Hz stereo but silently breaks for other configurations.
|
||||
`classify.AllClasses()` is a hardcoded list of the 14 built-in classes. The synth bank is constructed once at startup from this static list. User-defined classes are a runtime extension that the bank knows nothing about.
|
||||
|
||||
**How to avoid:**
|
||||
- Always call `InitParams()` before writing any frames
|
||||
- Restrict to known-safe parameters: sample rate 44100 or 48000, stereo or mono (LAME does not support dual-channel mode)
|
||||
- Write a single integration test that encodes 1 second of silence and confirms the output file is valid (use `mp3val` or `ffprobe` in CI)
|
||||
- Consider `shine-mp3` (pure Go port) as an alternative that eliminates CGo entirely; output files are larger but the library has no C initialization state
|
||||
**Consequences:**
|
||||
- Nil pointer panic in `RenderWindow` if the layer map lookup is not nil-guarded
|
||||
- Or silent: user-defined class traffic is captured and aggregated but never rendered to audio
|
||||
- In either case the user's primary feature request (custom sounds for custom classes) silently fails
|
||||
|
||||
**Warning signs:**
|
||||
- No test that validates the output MP3 with an external tool
|
||||
- Sample rate set to anything other than 44100 or 48000
|
||||
- Encoder parameters set after `InitParams()` has been called
|
||||
**Prevention:**
|
||||
The bank must be constructed from the *full* set of active classes, including user-defined ones. The construction path should be:
|
||||
|
||||
1. Load config (parse TOML, validate)
|
||||
2. Compute effective rule set (built-in + user rules)
|
||||
3. Extract the complete set of `TrafficClass` values referenced by all rules
|
||||
4. Pass this full class set to `NewBank` (or equivalent) so a layer is created for every reachable class
|
||||
5. Wire user-defined class frequencies from config into the bank
|
||||
|
||||
`AllClasses()` in `classify/types.go` should either remain the static built-in list (used for display/iteration of built-ins) or be replaced by a dynamic function that takes the active rule set as input. Do not rely on the hardcoded list in the bank-construction path when user-defined classes are possible.
|
||||
|
||||
**Detection:**
|
||||
- Panic: `runtime error: invalid memory address or nil pointer dereference` in `synth/bank.go:RenderWindow`
|
||||
- Or: user-defined class produces no sound, no error
|
||||
- Test: create a config with one user rule using a custom class; verify the bank is built with a layer for that class and that layer produces sound
|
||||
|
||||
**Phase to address:**
|
||||
Audio synthesis / encoding phase — establish the encode pipeline with an end-to-end smoke test (silence → valid MP3) before wiring up synthesis.
|
||||
User-defined rules phase, specifically the bank initialization step. This is the deepest integration point — it touches the pipeline at capture → classify → aggregate → synthesize.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 7: PCM Sample Overflow Produces Wrap-Around Distortion
|
||||
### Pitfall A7: Config Auto-Discovery Follows Wrong Order or Ignores XDG Variables
|
||||
|
||||
**What goes wrong:**
|
||||
Synthesizing audio as `int16` samples and summing multiple sine layers without clamping causes integer overflow. The value wraps around (e.g., 32767 + 100 = -32667 in int16), producing a sharp click or a buzzing distortion that corrupts the ambient soundscape. This is not clipping — it is a distinctly worse artifact.
|
||||
The spec calls for auto-discovery from `./netsynth.toml` then `~/.config/netsynth/config.toml`. A naive implementation uses `os.UserHomeDir()` to build the fallback path. On systems where `$XDG_CONFIG_HOME` is set to a non-default location (common on NixOS, custom dotfile managers, CI environments), the tool ignores the user's configured config directory and looks in `~/.config` anyway. The user has a config at `$XDG_CONFIG_HOME/netsynth/config.toml` that is never found.
|
||||
|
||||
Additionally, `os.UserHomeDir()` returns an error if `$HOME` is unset (e.g., inside some Docker containers or cron jobs). If this error is not handled, the path construction silently produces `"/.config/netsynth/config.toml"` (an absolute path starting with `/.config`) rather than failing with a useful message.
|
||||
|
||||
**Why it happens:**
|
||||
Developers model audio math in their head as real-valued floats, implement it in int16 for "efficiency," and forget that Go integer overflow is undefined-behavior-free but still wraps. With 6-8 drone layers simultaneously active, summing them easily exceeds ±32767.
|
||||
Go's `os.UserConfigDir()` already implements the XDG lookup (`$XDG_CONFIG_HOME` → `~/.config` on Linux, `~/Library/Application Support` on macOS). Most developers reach for `os.UserHomeDir()` + hardcoded `".config"` string because it is the first function they find in the stdlib.
|
||||
|
||||
**How to avoid:**
|
||||
Synthesize internally in `float64` in the range `[-1.0, 1.0]`. Apply a normalisation/soft-limiter pass before converting to `int16` for encoding. Clamp before cast: `sample := int16(math.Max(-1.0, math.Min(1.0, floatSample)) * 32767)`. Never do mixed-type audio math that passes through int16 as an intermediate.
|
||||
**Consequences:**
|
||||
- User's config is silently ignored when `$XDG_CONFIG_HOME` is non-default
|
||||
- Confusing behavior difference between development machines and CI
|
||||
|
||||
**Warning signs:**
|
||||
- Audio synthesis structs storing amplitude as `int16` or `int32`
|
||||
- Adding layer outputs with `+=` without a final normalisation step
|
||||
- Distorted output that correlates with traffic spikes (more active layers = more overflow)
|
||||
**Prevention:**
|
||||
Use `os.UserConfigDir()` (stdlib, Go 1.13+) for the platform-appropriate config directory. This correctly respects `$XDG_CONFIG_HOME` on Linux and `APPDATA` on Windows (if ever relevant). The discovery order should be:
|
||||
|
||||
```go
|
||||
func configSearchPaths() []string {
|
||||
var paths []string
|
||||
// 1. Current directory (highest precedence)
|
||||
paths = append(paths, "netsynth.toml")
|
||||
// 2. XDG/platform config dir
|
||||
if cfgDir, err := os.UserConfigDir(); err == nil {
|
||||
paths = append(paths, filepath.Join(cfgDir, "netsynth", "config.toml"))
|
||||
}
|
||||
return paths
|
||||
}
|
||||
```
|
||||
|
||||
If `--config` flag is set, use that path exclusively and return a clear error if the file is absent (do not fall through to auto-discovery when explicit path is provided).
|
||||
|
||||
**Detection:**
|
||||
- Config not loaded on systems where `$XDG_CONFIG_HOME=/custom/path`
|
||||
- Silent "no config found" behavior when a config clearly exists at the XDG path
|
||||
|
||||
**Phase to address:**
|
||||
Audio synthesis phase — establish the internal sample representation as `float64` from the start.
|
||||
Config loading phase. Implement the path discovery with `os.UserConfigDir()` from the start. Fix before the feature ships.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 8: Tone-per-Protocol Mapping Produces Perceptual Chaos
|
||||
### Pitfall A8: Explicit --config Flag Does Not Error on Missing File
|
||||
|
||||
**What goes wrong:**
|
||||
Assigning arbitrary frequencies to protocols (e.g., DNS=440 Hz, HTTPS=880 Hz, ICMP=1320 Hz, SSH=1760 Hz, 6 auto-clusters=random) creates a soundscape where all tones are in the same frequency range, fighting each other. At moderate traffic the result is an undifferentiated buzz rather than distinct recognizable layers.
|
||||
When `--config path/to/file.toml` is specified, the user expects an error if the file does not exist. If the config loader falls through to auto-discovery when the explicit path is missing, or silently uses defaults, the user has no way to detect a typo in their `--config` argument. They run a session, get "unexpected" default sounds, and have no indication their config was never loaded.
|
||||
|
||||
**Why it happens:**
|
||||
Developers choose frequencies programmatically (e.g., multiples of a base frequency) without considering auditory scene analysis — the human perceptual process by which listeners separate simultaneous sounds into distinct streams. Sounds too close in frequency mask each other.
|
||||
Auto-discovery logic is convenient to write as "try these paths, use first found." Developers reuse this logic even for the `--config` code path.
|
||||
|
||||
**How to avoid:**
|
||||
Space protocol tones across register bands: low drones (80-200 Hz) for high-volume background traffic (HTTPS bulk), mid tones (300-600 Hz) for control traffic (DNS, NTP), high tones (800-1600 Hz) for interactive protocols (SSH, ICMP). Use harmonic or musical intervals (octaves, fifths) rather than arithmetic spacing. Keep auto-cluster frequencies in the 200-500 Hz mid-range so they don't obscure the "signature" tones. Limit simultaneous active layers to avoid masking.
|
||||
**Prevention:**
|
||||
Separate the two code paths:
|
||||
- `--config` specified → `os.Open(flagValue)`, return error immediately if `errors.Is(err, os.ErrNotExist)`
|
||||
- No flag → `configSearchPaths()` loop, silently skip missing files, proceed with defaults if none found
|
||||
|
||||
**Warning signs:**
|
||||
- Frequency assignments as an arithmetic sequence: `baseFreq + n*200`
|
||||
- No perceptual test — only waveform-level correctness checks
|
||||
- Auto-cluster frequencies chosen randomly from the full audible range
|
||||
**Detection:**
|
||||
- `--config missing.toml` runs without error, uses defaults
|
||||
- User misses that their config file path has a typo
|
||||
|
||||
**Phase to address:**
|
||||
Audio mapping / synthesis phase — the frequency mapping table should be designed up front with the perceptual goals in mind, not patched after "it sounds like noise" feedback.
|
||||
Config loading phase. A one-line `if flagValue != "" { /* require it */ }` branch is sufficient.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 9: Time Window Too Short — Unstable, Jittery Audio
|
||||
### Pitfall A9: User Rules That Target the Same Port as Built-in Rules Are Silently Shadowed
|
||||
|
||||
**What goes wrong:**
|
||||
Aggregating traffic into windows shorter than ~500ms causes rapid amplitude oscillation in the synthesized drones. A single ICMP ping becomes a brief tone burst; a DNS query causes a momentary volume spike. The output sounds jittery and event-driven rather than ambient.
|
||||
A user writes a rule for `{Protocol: "tcp", DstPort: 443, Class: "my-api"}` intending to reclassify their internal HTTPS traffic. If built-in `ClassHTTPS` still appears before the user rule in the merged slice, the built-in rule wins every time. The user's intent ("I want my port-443 traffic to sound different") is silently defeated.
|
||||
|
||||
**Why it happens:**
|
||||
Developers choose a "natural" update interval (100ms or 200ms matches CPU scheduling intuition) without considering audio envelope times. Human perception of tonal stability requires note durations of at least 200-500ms; drones need even longer.
|
||||
First-match-wins with `SpecificRules + userRules + CatchAllRules` means built-in specific rules still precede user rules. A user trying to *override* a built-in mapping must replace it, not add after it.
|
||||
|
||||
**How to avoid:**
|
||||
- Use a minimum window of 500ms for amplitude updates; 1-2s for tonal shifts
|
||||
- Apply amplitude smoothing (exponential moving average with a decay of ~2-5s) so a single-packet burst doesn't cause an immediate amplitude jump
|
||||
- Separate the "data collection" window (can be shorter) from the "audio parameter update" window (should be longer)
|
||||
**Consequences:**
|
||||
- User's specific rule is unreachable if a built-in rule covers the same port/protocol
|
||||
- No error, no warning
|
||||
- Functionally the same as Pitfall A5 but for specific (non-catch-all) built-in rules
|
||||
|
||||
**Warning signs:**
|
||||
- `time.Tick(100 * time.Millisecond)` driving audio parameter updates
|
||||
- No smoothing/interpolation between amplitude values
|
||||
- Testing with ping floods (bursty) rather than continuous traffic
|
||||
**Prevention:**
|
||||
Two viable strategies:
|
||||
1. **User rules first:** `userRules + specificDefaultRules + catchAllRules`. User rules always take precedence. Built-ins serve as fallback. This is the simplest design and most aligned with user expectations ("I configure what I care about; defaults handle everything else").
|
||||
2. **Conflict detection:** After merging, scan for duplicate `(protocol, dstPort)` pairs and emit a warning: `"User rule for tcp:443 shadows built-in HTTPS rule. Did you mean to replace it?"`.
|
||||
|
||||
Option 1 is recommended for simplicity. Document it clearly: "User-defined rules are evaluated before built-in rules."
|
||||
|
||||
**Detection:**
|
||||
- User-defined rule for a built-in port (80, 443, 22, etc.) never activates
|
||||
- Verbose output shows built-in class instead of user class for the expected traffic
|
||||
|
||||
**Phase to address:**
|
||||
Traffic aggregation / audio mapping phase — establish the window and smoothing strategy before wiring traffic data to audio parameters.
|
||||
User-defined rules phase, merge strategy design. Address at the same time as Pitfall A5.
|
||||
|
||||
---
|
||||
|
||||
## Technical Debt Patterns
|
||||
### Pitfall A10: New TrafficClass Strings From Config Are Not Validated — Empty String or Whitespace Is a Valid Key
|
||||
|
||||
| Shortcut | Immediate Benefit | Long-term Cost | When Acceptable |
|
||||
|----------|-------------------|----------------|-----------------|
|
||||
| `google/gopacket` instead of `gopacket/gopacket` | Familiar, more tutorials | Unmaintained; Go compat breaks | Never |
|
||||
| `sudo ./netsynth` only, no `setcap` docs | Simpler setup instructions | Users won't run as root in practice; tool appears broken | MVP only — document the limitation |
|
||||
| Dynamic libpcap linking (no static build) | Faster to compile | Binary doesn't work on target machines without libpcap installed | Only acceptable if distributing via package manager that declares the dep |
|
||||
| `ReadPacketData` (copying) instead of `ZeroCopy` | Safe, simple | ~20% memory overhead at high packet rates | Always acceptable; optimize only if profiling proves allocation bottleneck |
|
||||
| Sine-wave-only synthesis (no ADSR, no envelope) | Much simpler code | Tonal changes are abrupt, not perceptually smooth | Acceptable for v1 ambient/drone if EMA smoothing is applied to amplitude |
|
||||
| Hard-coded frequency table (no config) | No CLI complexity | Can't tune without recompiling | Acceptable for v1 per PROJECT.md out-of-scope decision |
|
||||
**What goes wrong:**
|
||||
A user writes:
|
||||
|
||||
```toml
|
||||
[[rules]]
|
||||
protocol = "tcp"
|
||||
dst_port = 9200
|
||||
class = ""
|
||||
```
|
||||
|
||||
The string `""` decodes without error. It is a valid Go map key. It gets inserted into the `WindowSnapshot.Counts` map and the aggregator increments `Counts[""]`. The bank has no layer for `""`. The behavior is undefined — silent or panic depending on nil-guard presence.
|
||||
|
||||
Similarly, `class = " elasticsearch "` (padded spaces) decodes to a string with leading/trailing whitespace that does not match any configured sound entry (because the config sound entry key is `"elasticsearch"` without spaces).
|
||||
|
||||
**Prevention:**
|
||||
Validate all `Class` string values from user rules in the `validate()` step:
|
||||
```go
|
||||
if strings.TrimSpace(rule.Class) == "" {
|
||||
return fmt.Errorf("rule %d: class name must not be empty", i)
|
||||
}
|
||||
rule.Class = strings.TrimSpace(rule.Class)
|
||||
```
|
||||
Also validate that class names do not collide with reserved built-in class names (`"ICMP"`, `"DNS"`, etc.) unless the user is explicitly overriding a built-in sound (which is a distinct feature — it should be opt-in, not accidental).
|
||||
|
||||
**Phase to address:**
|
||||
Config validation step.
|
||||
|
||||
---
|
||||
|
||||
## Integration Gotchas
|
||||
## v1.0 Pitfalls (Retained for Reference)
|
||||
|
||||
The following pitfalls from the initial MVP research remain valid. They are retained in condensed form for reference.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B1: Using `google/gopacket` Instead of the Active Community Fork
|
||||
|
||||
**What goes wrong:** Import of the unmaintained original — 270 open issues, Go compat degrades.
|
||||
**Prevention:** Import `github.com/gopacket/gopacket` (v1.5.0, requires Go 1.24+).
|
||||
**Phase:** Phase 1 — set correct import path from day one.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B2: CGo Destroys the "Single Binary" Promise
|
||||
|
||||
**What goes wrong:** `gopacket/pcap` (CGo + libpcap) produces a dynamically-linked binary that fails on machines without `libpcap.so`.
|
||||
**Prevention:** Use `packetcap/go-pcap` (pure Go capture, already the chosen stack). Verify with `ldd ./netsynth`.
|
||||
**Phase:** Phase 1 — foundational architecture decision.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B3: `CAP_NET_RAW` + Binary Location = Silent Failure on Linux
|
||||
|
||||
**What goes wrong:** `setcap` is silently ignored on `nosuid` filesystems. Binary appears broken from home directories.
|
||||
**Prevention:** Install to `/usr/local/bin`; document two run modes; emit clear privilege error.
|
||||
**Phase:** Phase 1 + CLI UX.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B4: Packet Buffer Overflow Under Moderate Traffic Load
|
||||
|
||||
**What goes wrong:** Default capture buffer fills faster than the classifier consumes it; silent packet drops misrepresent traffic.
|
||||
**Prevention:** Large capture buffer (32 MB); buffered channel between capture and classify goroutines.
|
||||
**Phase:** Phase 1/2 (capture pipeline architecture).
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B5: ZeroCopy Packet Data Use-After-Free
|
||||
|
||||
**What goes wrong:** `ZeroCopyReadPacketData()` invalidates previous slice on each call; silent data corruption in concurrent code.
|
||||
**Prevention:** Use `ReadPacketData()` (copying API) unless profiling proves allocation bottleneck.
|
||||
**Phase:** Phase 1 (capture/decode).
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B6: MP3 Output Is Corrupt Due to LAME Initialization Order
|
||||
|
||||
**What goes wrong:** Skipping `InitParams()` or setting parameters out of order produces unplayable MP3.
|
||||
**Prevention:** Always call `InitParams()` before writing frames; smoke test with `ffprobe`.
|
||||
**Phase:** Audio synthesis / encoding phase.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B7: PCM Sample Overflow Produces Wrap-Around Distortion
|
||||
|
||||
**What goes wrong:** Summing `int16` layers overflows and wraps (32767 + 100 = -32667), producing buzzing distortion.
|
||||
**Prevention:** Synthesize in `float64 [-1.0, 1.0]`; clamp before int16 cast. Already implemented in `synth/mixer.go`.
|
||||
**Phase:** Audio synthesis (already addressed in v1.0).
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B8: Tone-per-Protocol Mapping Produces Perceptual Chaos
|
||||
|
||||
**What goes wrong:** Frequencies too close together mask each other; output is undifferentiated buzz.
|
||||
**Prevention:** Space protocols across register bands; use harmonic/musical intervals. Already addressed in v1.0.
|
||||
**Phase:** Audio mapping (already addressed in v1.0).
|
||||
|
||||
---
|
||||
|
||||
## v1.1 Phase-Specific Warnings
|
||||
|
||||
| Phase Topic | Likely Pitfall | Mitigation |
|
||||
|-------------|---------------|------------|
|
||||
| TOML struct design | A1: zero-value overwrites defaults | Use pointer fields for all optional overrides |
|
||||
| Config strict decode | A2: typos silently ignored | Use `md.Undecoded()` as strict mode check |
|
||||
| Waveform implementation | A3: naive waveform aliases | Use additive synthesis (bandlimited harmonic series) — compatible with existing `[]HarmonicDef` API |
|
||||
| Waveform string input | A4: case/abbreviation mismatches | Normalize + validate with clear error listing accepted values |
|
||||
| Rule merge ordering (catch-alls) | A5: user rules after catch-alls are unreachable | Split `DefaultRules` into `SpecificRules` + `CatchAllRules`; user rules go in between |
|
||||
| Bank construction | A6: custom class has no synth layer | Derive full class set from merged rule slice; pass to bank constructor |
|
||||
| Config discovery | A7: XDG ignored, `~/.config` hardcoded | Use `os.UserConfigDir()` not `os.UserHomeDir() + "/.config"` |
|
||||
| --config flag path | A8: missing explicit path silently ignored | Two distinct code paths: flag path (require) vs auto-discovery (skip-missing) |
|
||||
| Rule merge ordering (specific built-ins) | A9: user rule shadowed by built-in for same port | User rules first in merged slice (`userRules + specificDefaults + catchAlls`) |
|
||||
| Class name validation | A10: empty/whitespace class name is valid Go string | Validate and trim all class strings in `validate()` |
|
||||
|
||||
---
|
||||
|
||||
## Integration Gotchas (v1.1 Additions)
|
||||
|
||||
| Integration | Common Mistake | Correct Approach |
|
||||
|-------------|----------------|------------------|
|
||||
| `gopacket/pcap` handle | Not calling `handle.Close()` on signal — leaks capture resources | Use `defer handle.Close()` and ensure the goroutine exits before process termination |
|
||||
| LAME CGo encoder | Not flushing the encoder before closing — truncated final MP3 frame | Call `encoder.Flush()` / `lame.EncodeFlush()` after the sample loop ends |
|
||||
| OS signal handling (`SIGINT`) | Goroutine receives SIGINT but the capture loop is blocked on `ReadPacketData` | Use `handle.SetReadDeadline(time.Now())` or close the handle to unblock |
|
||||
| MP3 encoder sample format | Passing `float64` samples directly to LAME (expects `int16` or `float32` depending on binding) | Explicitly convert and clamp to the binding's expected type; check each binding's API |
|
||||
| `pcapgo.EthernetHandle` | Only captures Ethernet frames — fails on WiFi (802.11), loopback, or tunnel interfaces | For non-Ethernet interfaces, use the `pcap` backend or check link type at startup |
|
||||
|
||||
---
|
||||
|
||||
## Performance Traps
|
||||
|
||||
| Trap | Symptoms | Prevention | When It Breaks |
|
||||
|------|----------|------------|----------------|
|
||||
| Single goroutine: capture + classify + synthesize | CPU-bound synthesis blocks packet reads; drops spike under any real traffic | Three-stage pipeline: capture goroutine → classify channel → synthesis goroutine | Breaks on any network with > ~1000 pps |
|
||||
| One goroutine per packet | Goroutine creation overhead exceeds packet processing time; OOM on busy networks | Channel-based batching: one reader, N classifiers from a worker pool | Breaks above ~10k pps |
|
||||
| Recomputing sine wave sample-by-sample in inner loop using `math.Sin` | CPU pegged at 100% during synthesis; output can't keep pace | Precompute wavetable per frequency; iterate with phase accumulator | Breaks with > 4-5 simultaneous drone layers at 44100 Hz |
|
||||
| Blocking channel between capture and synthesis with no buffer | Any synthesis stall causes packet drops | Buffered channel of 1000+ packets; separate goroutines | Breaks immediately on any CPU scheduling hiccup |
|
||||
|
||||
---
|
||||
|
||||
## Security Mistakes
|
||||
|
||||
| Mistake | Risk | Prevention |
|
||||
|---------|------|------------|
|
||||
| Requesting full `root` and keeping it throughout capture | Privilege escalation if a parsing bug in gopacket can be exploited via crafted packets | Drop privileges after opening the capture handle: `syscall.Setuid(originalUID)` |
|
||||
| Promiscuous mode on by default without user opt-in | Captures all LAN traffic, not just traffic to/from the host — legal and privacy risk on shared networks | Default to non-promiscuous; add `--promiscuous` flag with a warning message |
|
||||
| No limit on capture duration or file size | Unbounded run produces an arbitrarily large MP3 or consumes all memory in the aggregator maps | Add `--max-duration` flag (default: warn at 10min, hard limit at 1hr); prune old flow state periodically |
|
||||
| Logging decoded packet payloads in debug mode | Inadvertently logs credentials or private data | Never log packet payload bytes; log only headers and metadata |
|
||||
|
||||
---
|
||||
|
||||
## UX Pitfalls
|
||||
|
||||
| Pitfall | User Impact | Better Approach |
|
||||
|---------|-------------|-----------------|
|
||||
| Silent failure when interface doesn't exist | User specifies `-i eth1` on a machine with only `ens3`; tool exits with cryptic libpcap error | List available interfaces at startup with `pcap.FindAllDevs()` and suggest correct name |
|
||||
| No progress feedback during capture | User has no idea if the tool is working; assumes it hung | Print periodic status line: "Capturing... 1,234 packets classified (HTTPS:45% DNS:30% ICMP:8% other:17%)" |
|
||||
| Output MP3 path collision without warning | Re-running overwrites previous output | Warn if output file exists; suggest timestamped default filename |
|
||||
| Ctrl+C produces empty or invalid MP3 | User interrupts too quickly before any traffic is captured | Detect zero-packet case and emit an error instead of an empty file |
|
||||
| No indication of which interface is being captured | Confusing when multiple interfaces exist | Print "Capturing on: eth0 (192.168.1.5)" at startup |
|
||||
|
||||
---
|
||||
|
||||
## "Looks Done But Isn't" Checklist
|
||||
|
||||
- [ ] **Packet capture:** Binary runs as non-root user with `setcap` — verify from a non-`/home` path, not just from the build directory
|
||||
- [ ] **MP3 output:** File validates with `ffprobe` or `mp3val` — not just "has .mp3 extension and non-zero size"
|
||||
- [ ] **Static binary:** `ldd ./netsynth` shows "not a dynamic executable" (or explicitly "requires libpcap" if dynamic is accepted)
|
||||
- [ ] **Signal handling:** Ctrl+C during capture produces a valid (playable) MP3, not a truncated file
|
||||
- [ ] **High-traffic:** Drop counter is zero (or documented/acceptable) when tested against a network with > 1000 pps
|
||||
- [ ] **Audio layers:** Output with 6+ simultaneous traffic types does not distort — no wrap-around clipping audible
|
||||
- [ ] **Empty capture:** Graceful error message when zero packets were captured, not a silent empty file
|
||||
- [ ] **Interface not found:** Helpful error with available interface list, not a libpcap raw error string
|
||||
|
||||
---
|
||||
|
||||
## Recovery Strategies
|
||||
|
||||
| Pitfall | Recovery Cost | Recovery Steps |
|
||||
|---------|---------------|----------------|
|
||||
| Wrong gopacket fork | LOW | `go mod edit -replace github.com/google/gopacket=github.com/gopacket/gopacket@v1.5.0`; update import paths |
|
||||
| Dynamic binary on clean machine | MEDIUM | Add static build Makefile target; update CI; update README |
|
||||
| PCM overflow / distortion | LOW | Refactor synthesis to float64 internal representation; add clamp before int16 cast |
|
||||
| Corrupt MP3 (missing flush) | LOW | Add `Flush()` call in the shutdown path |
|
||||
| Perceptual chaos (tone mapping) | MEDIUM | Redesign frequency table (no code change to synthesis engine); requires subjective listening tests |
|
||||
| Time window jitter | LOW | Add EMA smoothing and increase window; no architectural change needed |
|
||||
| `ZeroCopy` data corruption | MEDIUM | Replace `ZeroCopyReadPacketData` with `ReadPacketData`; audit all goroutine handoffs |
|
||||
|
||||
---
|
||||
|
||||
## Pitfall-to-Phase Mapping
|
||||
|
||||
| Pitfall | Prevention Phase | Verification |
|
||||
|---------|------------------|--------------|
|
||||
| Wrong gopacket fork | Phase 1: Packet Capture | `go.mod` references `gopacket/gopacket`; `go list -m github.com/gopacket/gopacket` |
|
||||
| CGo / single binary contract | Phase 1: Packet Capture | `ldd` output on CI; test on clean Alpine container |
|
||||
| CAP_NET_RAW binary location | Phase 1 + CLI UX phase | Test `setcap` from `/usr/local/bin`; verify helpful error message from non-root |
|
||||
| Packet buffer overflow | Phase 1/2: Capture Pipeline | `SetBufferSize` call present; goroutine architecture is async (channel-separated) |
|
||||
| ZeroCopy use-after-free | Phase 1: Capture/Decode | Code review: no `ZeroCopy` passed to goroutines without copy; or use `ReadPacketData` |
|
||||
| LAME init errors / corrupt MP3 | Audio synthesis phase | CI smoke test: 1s silence → `ffprobe` validates output file |
|
||||
| PCM overflow wrap-around | Audio synthesis phase | Unit test: 8 simultaneous max-amplitude layers produce no distortion |
|
||||
| Perceptual tone chaos | Audio mapping phase | Subjective listen test with mixed traffic capture; frequency table reviewed against auditory masking |
|
||||
| Time window jitter | Traffic aggregation / mapping phase | Capture test with bursty traffic; verify EMA smoothing produces stable amplitude |
|
||||
| Config → Bank wire-up | Pass `classify.AllClasses()` to bank; custom classes missing | Derive layer set from `Classifier.ActiveClasses()` — all classes reachable via the effective rule set |
|
||||
| Waveform → FreqConfig | Add `WaveformType string` to `FreqConfig`; forget to generate harmonics at bank init | Generate `[]HarmonicDef` from waveform+freq at bank/layer construction time, not at sample render time |
|
||||
| User rules → Classifier | Replace `DefaultRules` var directly; breaks tests relying on it | Keep `DefaultRules` immutable; construct `mergedRules` for runtime use |
|
||||
| Config file absent | Return error if no config found | Return nil (no config = all defaults). Only error on explicit `--config` path that is missing |
|
||||
| Sound overrides for built-in class | User sets freq for "HTTPS" — must hit `ClassHTTPS` layer | Match config sound keys case-insensitively against `TrafficClass` string values; map `"HTTPS"` → `classify.ClassHTTPS` |
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [gopacket/gopacket (community fork, v1.5.0)](https://github.com/gopacket/gopacket) — active fork status
|
||||
- [google/gopacket issue #329: 98% packet loss under high traffic](https://github.com/google/gopacket/issues/329) — buffer overflow and afpacket solution
|
||||
- [google/gopacket issue #1016: current project status](https://github.com/google/gopacket/issues/1016) — unmaintained status of original repo
|
||||
- [google/gopacket issue #1167: static linking libpcap.a](https://github.com/google/gopacket/issues/1167) — static build complications
|
||||
- [ZeroCopyReadPacketData docs (gopacket/pcap)](https://pkg.go.dev/github.com/google/gopacket/pcap) — memory ownership contract
|
||||
- [linuxvox.com: CAP_NET_RAW outside /usr/bin](https://linuxvox.com/blog/raw-capture-capabilities-cap-net-raw-cap-net-admin-not-working-outside-usr-bin-and-friends-for-packet-capture-program-using-libpcap/) — nosuid and AppArmor restrictions
|
||||
- [braheezy.github.io: What I Learned About MP3 Encoding](https://braheezy.github.io/posts/what-i-learned-about-mp3-encoding/) — Go MP3 encoding pitfalls
|
||||
- [github.com/braheezy/shine-mp3](https://github.com/braheezy/shine-mp3) — pure Go MP3 encoder (no CGo)
|
||||
- [Eli Bendersky: Building Static Binaries with Go on Linux](https://eli.thegreenplace.net/2024/building-static-binaries-with-go-on-linux/) — CGo static linking strategy
|
||||
- [SoNSTAR: Sonification of Networks for Situational Awareness](https://github.com/nuson/SoNSTAR) — reference architecture for network sonification
|
||||
- [PLOS One: Sonification of Network Traffic Flow](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0195948) — time window and design lessons
|
||||
- [KVR Audio: PCM float-to-int clipping and wrap-around](https://www.kvraudio.com/forum/viewtopic.php?t=414666) — PCM overflow consequences
|
||||
- [bjornroche.com: ABCs of PCM Digital Audio](http://blog.bjornroche.com/2013/05/the-abcs-of-pcm-uncompressed-digital.html) — sample format fundamentals
|
||||
- [BurntSushi/toml pkg.go.dev](https://pkg.go.dev/github.com/BurntSushi/toml) — `Undecoded()` strict mode, pointer field behavior, `MetaData` API
|
||||
- [BurntSushi/toml issue #47: Unmarshal with default values](https://github.com/BurntSushi/toml/issues/47) — confirms default-overwrite behavior
|
||||
- [pelletier/go-toml issue #252: Unmarshal overrides origin values if key is omitted](https://github.com/pelletier/go-toml/issues/252) — confirms same behavior in v1; v2 partially resolves
|
||||
- [pelletier/go-toml v2 pkg.go.dev](https://pkg.go.dev/github.com/pelletier/go-toml/v2) — strict decoder mode documentation
|
||||
- [golang/go issue #29960: os: add UserConfigDir](https://github.com/golang/go/issues/29960) — rationale for `os.UserConfigDir()` (XDG-aware)
|
||||
- [WolfSound: Basic Waveforms in Synthesis](https://thewolfsound.com/sine-saw-square-triangle-pulse-basic-waveforms-in-synthesis/) — aliasing and harmonic series for square/saw/triangle
|
||||
- [CCRMA: Alias-Free Digital Synthesis of Classic Analog Waveforms](https://ccrma.stanford.edu/~stilti/papers/blit.pdf) — bandlimited synthesis theory
|
||||
- [McGill Bandlimited Synthesis of Classic Waveforms](https://www.music.mcgill.ca/~gary/307/week5/bandlimited.html) — truncated harmonic series approach
|
||||
- [Teensy Forum: triangle & sawtooth oscillators aliasing](https://forum.pjrc.com/threads/61269-triangle-amp-sawtooth-oscillators-how-to-deal-with-aliasing) — practical aliasing impact at different frequencies
|
||||
- [adrg/xdg package](https://github.com/adrg/xdg) — XDG Base Directory Specification Go implementation (reference; stdlib `os.UserConfigDir()` is sufficient for NetSynth's needs)
|
||||
|
||||
---
|
||||
*Pitfalls research for: network-traffic-to-audio synthesis CLI (Go) — NetSynth*
|
||||
*Researched: 2026-03-24*
|
||||
*Pitfalls research for: NetSynth v1.1 — TOML config, waveform types, user-defined rules*
|
||||
*Updated: 2026-03-26*
|
||||
|
||||
+180
-131
@@ -1,166 +1,215 @@
|
||||
# Stack Research
|
||||
# Technology Stack
|
||||
|
||||
**Domain:** Go CLI tool — network packet capture, traffic classification, audio synthesis, MP3 encoding
|
||||
**Researched:** 2026-03-24
|
||||
**Confidence:** MEDIUM-HIGH (packet capture and CLI: HIGH; audio synthesis in Go: MEDIUM; MP3 encoding: MEDIUM)
|
||||
**Project:** NetSynth v1.1 — Custom Sound Mappings
|
||||
**Researched:** 2026-03-26
|
||||
**Scope:** Additions/changes only. Existing stack (gopacket, go-pcap, go-lame, cobra) is validated and unchanged.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Stack
|
||||
## Existing Stack (Do Not Re-research)
|
||||
|
||||
### Core Technologies
|
||||
|
||||
| Technology | Version | Purpose | Why Recommended |
|
||||
|------------|---------|---------|-----------------|
|
||||
| `github.com/gopacket/gopacket` | v1.5.0 | Packet capture, protocol decoding | The canonical Go packet library. Community fork (`gopacket/gopacket`) supersedes the original Google repo (`google/gopacket`) as of 2024; released v1.5.0 in November 2025, minimum Go 1.24. 14.5k dependents; has ICMP, TCP, UDP, DNS, TLS layer decoders built in. |
|
||||
| `github.com/packetcap/go-pcap` | v0.0.0-20251215 | Pure-Go live packet capture backend | Replaces CGo libpcap dependency for live capture. 100% native Go, Linux + macOS, mmap-based kernel ring buffer for performance. Implements the `gopacket.PacketDataSource` interface so gopacket decodes packets on top of it. Enables CGO_ENABLED=0 builds and cross-compilation. |
|
||||
| `github.com/sjzar/go-lame` | v0.0.9 | MP3 encoding | Embeds libmp3lame C source directly via CGo — no external `libmp3lame` system package required. Published April 2025. Exposes sample rate, channels, quality control. Produces LAME-quality MP3, unlike the pure-Go shine-mp3 port which produces larger, lower-quality output. Tradeoff: requires CGo, so `CGO_ENABLED=1` and a C compiler at build time. |
|
||||
| `github.com/spf13/cobra` | v1.10.2 | CLI flag parsing and command structure | The industry standard for Go CLIs (Kubernetes, Docker, Hugo, etc.). v1.10.2 released December 2025. Handles `--interface`, `--output` flags, Ctrl+C signal plumbing, and `--help` generation automatically. No alternatives worth considering for this scope. |
|
||||
|
||||
### Supporting Libraries
|
||||
|
||||
| Library | Version | Purpose | When to Use |
|
||||
|---------|---------|---------|-------------|
|
||||
| `github.com/muesli/kmeans` | v0.3.1 | K-means clustering for unrecognized traffic patterns | Use to auto-cluster packets that don't match known protocol rules. Feed feature vectors: [port, protocol_num, packet_size_bin, direction]. Last release July 2022 but mathematically stable; the algorithm doesn't change. Alternatively, implement a simple incremental classifier directly (see Architecture notes below). |
|
||||
| `github.com/go-audio/wav` | latest | WAV file I/O as intermediate format | Use to write synthesized PCM as WAV before MP3 encoding pass. "Battle tested" per maintainer. Simplifies the PCM → encoder pipeline: synthesize float64 samples → write WAV → re-read as PCM → LAME encode. |
|
||||
| `golang.org/x/sys/unix` | stdlib | Raw socket / CAP_NET_RAW privilege checks | Use for detecting if the process has required privileges and for signaling (SIGINT for clean shutdown). Part of Go extended stdlib — no external version pinning needed. |
|
||||
|
||||
### Development Tools
|
||||
|
||||
| Tool | Purpose | Notes |
|
||||
|------|---------|-------|
|
||||
| `go build -ldflags="-s -w"` | Stripped binary production builds | Reduces binary size significantly; combine with `upx` if size is critical |
|
||||
| `goreleaser` | Cross-platform release builds | Handles CGo cross-compilation complexity with Docker-based build matrix; useful for distributing Linux x86_64 + ARM64 binaries |
|
||||
| `golangci-lint` | Static analysis | Catches nil pointer dereferences common in packet-handling code |
|
||||
| Wireshark / `tcpdump` | Manual verification of packet capture | Essential for confirming gopacket is decoding the right protocols before plugging into audio synthesis |
|
||||
| Technology | Version | Status |
|
||||
|------------|---------|--------|
|
||||
| `github.com/gopacket/gopacket` | v1.5.0 | Validated in v1.0, unchanged |
|
||||
| `github.com/packetcap/go-pcap` | v0.0.0-20251215 | Validated in v1.0, unchanged |
|
||||
| `github.com/sjzar/go-lame` | v0.0.9 | Validated in v1.0, unchanged |
|
||||
| `github.com/spf13/cobra` | v1.10.2 | Validated in v1.0, unchanged |
|
||||
| Hand-rolled sine oscillator + EMA | — | Validated in v1.0, extend in place |
|
||||
| Ordered `[]Rule` classifier | — | Validated in v1.0, extend in place |
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
## New Dependencies for v1.1
|
||||
|
||||
```bash
|
||||
# Initialize module
|
||||
go mod init netsynth
|
||||
### TOML Config Parsing
|
||||
|
||||
# Core dependencies
|
||||
go get github.com/gopacket/gopacket@v1.5.0
|
||||
go get github.com/packetcap/go-pcap@latest
|
||||
go get github.com/sjzar/go-lame@v0.0.9
|
||||
go get github.com/spf13/cobra@v1.10.2
|
||||
| Technology | Version | Purpose | Why |
|
||||
|------------|---------|---------|-----|
|
||||
| `github.com/BurntSushi/toml` | v1.6.0 | Parse `netsynth.toml` config files | Single-function `toml.Decode()` into a struct. The `MetaData.Undecoded()` method catches unknown keys in user configs — surfacing typos like `frequncy` rather than silently ignoring them. This is the right behavior for a config file tool. v1.6.0 released December 2025, Go 1.18+ required. Zero indirect dependencies. |
|
||||
|
||||
# Supporting
|
||||
go get github.com/muesli/kmeans@v0.3.1
|
||||
go get github.com/go-audio/wav@latest
|
||||
**Version confirmed:** v1.6.0, December 18, 2025, via pkg.go.dev and GitHub releases page.
|
||||
|
||||
# Build (CGo required for go-lame)
|
||||
CGO_ENABLED=1 go build -ldflags="-s -w" -o netsynth ./cmd/netsynth
|
||||
```
|
||||
**Why not `pelletier/go-toml v2`:** go-toml v2.3.0 (March 2026) is faster but the performance difference is irrelevant — config is read once at startup. go-toml v2's `Strict` mode can detect unknown keys but requires more setup than BurntSushi's `MetaData.Undecoded()`. BurntSushi's API is simpler for this use case and has clearer error message patterns for user-facing config mistakes.
|
||||
|
||||
**Root/privilege requirement at runtime (not build time):**
|
||||
```bash
|
||||
sudo ./netsynth --interface eth0 --output traffic.mp3
|
||||
# OR: grant capability instead of running as root
|
||||
sudo setcap cap_net_raw+ep ./netsynth
|
||||
```
|
||||
### Config Auto-Discovery
|
||||
|
||||
---
|
||||
No new dependency. Use Go stdlib only:
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
| Recommended | Alternative | When to Use Alternative |
|
||||
|-------------|-------------|-------------------------|
|
||||
| `gopacket/gopacket` (community fork) | `google/gopacket` (original) | Never for new projects — original repo has 270 open issues, community fork actively merges fixes |
|
||||
| `packetcap/go-pcap` (pure Go) | `gopacket/pcap` (CGo + libpcap) | Use libpcap path only if you need advanced BPF filter syntax or BSD/Windows support — it requires `libpcap-dev` system package |
|
||||
| `sjzar/go-lame` (embedded C source) | `braheezy/shine-mp3` (pure Go) | Use shine-mp3 if CGo is truly impossible (e.g., WASM target) — but accept that output quality and file size are worse |
|
||||
| `sjzar/go-lame` (embedded C source) | `viert/go-lame` (dynamic link) | Never — viert/go-lame requires libmp3lame installed on the target system, defeating single-binary distribution |
|
||||
| Hand-rolled additive synthesis | `dasa.cc/snd`, `bspaans/bleep` | Use a library only if you need MIDI scheduling or real-time playback; for file output, the synthesis math is simple enough to own directly (see Architecture notes) |
|
||||
| `muesli/kmeans` | `mpraski/clusters` | Use mpraski if you need online (incremental) clustering — it supports add-one-point updates vs muesli's batch-only approach |
|
||||
|
||||
---
|
||||
|
||||
## What NOT to Use
|
||||
|
||||
| Avoid | Why | Use Instead |
|
||||
|-------|-----|-------------|
|
||||
| `google/gopacket` (original) | Effectively unmaintained since 2022; 270 open issues, PRs not merged | `github.com/gopacket/gopacket` (community fork, v1.5.0) |
|
||||
| `viert/go-lame` or `sunicy/go-lame` | Dynamic-links against system `libmp3lame` — breaks single-binary distribution, fails on machines without the library | `github.com/sjzar/go-lame` (embeds C source statically) |
|
||||
| `braheezy/shine-mp3` (pure Go MP3) | Last commit 2023, explicitly not production-ready per its own README, produces larger lower-quality files, no bitrate control | `sjzar/go-lame` for quality, or WAV output if you must avoid CGo |
|
||||
| `go-audio/generator` | **Archived February 2026, read-only** — do not take a new dependency on it | Write your own oscillator (20 lines of Go) or use `dasa.cc/snd` |
|
||||
| `faiface/beep` | Designed for real-time audio playback via PortAudio/oto; pulls in platform audio drivers that are irrelevant for file output | Roll a minimal additive synthesizer directly (see below) |
|
||||
| `dasa.cc/snd` | Plays audio through hardware; brings in real-time audio scheduling complexity unnecessary for batch file output | Roll a minimal additive synthesizer directly |
|
||||
| urfave/cli | Fine for simpler tools, but Cobra's flag validation, help generation, and signal handling are better for a tool with multiple flags and clean shutdown semantics | `github.com/spf13/cobra` |
|
||||
|
||||
---
|
||||
|
||||
## Stack Patterns by Variant
|
||||
|
||||
**If CGo is acceptable (recommended path):**
|
||||
- Use `sjzar/go-lame` for real MP3 quality
|
||||
- Use `packetcap/go-pcap` for the capture layer (pure Go on Linux/macOS)
|
||||
- Build with `CGO_ENABLED=1`; single binary is still self-contained because LAME C source is embedded
|
||||
|
||||
**If pure Go / no CGo is required (e.g., restricted build environment):**
|
||||
- Use `braheezy/shine-mp3` for MP3 — accept lower quality and larger files
|
||||
- Use `packetcap/go-pcap` for capture — already pure Go
|
||||
- Build with `CGO_ENABLED=0`; truly static binary
|
||||
|
||||
**If Linux-only deployment is acceptable:**
|
||||
- Consider `packetcap/go-pcap`'s mmap ring buffer mode for high-traffic interfaces (default on Linux)
|
||||
- Privilege: `CAP_NET_RAW` setcap is cleaner than running as root
|
||||
|
||||
**For the audio synthesis layer — roll your own, don't use a library:**
|
||||
The ambient/drone requirement is additive synthesis: N sine wave oscillators, each with a frequency and time-varying amplitude. This is 30-50 lines of Go:
|
||||
```go
|
||||
// Conceptual — not a library call
|
||||
for t := 0; t < numSamples; t++ {
|
||||
sample := 0.0
|
||||
for _, layer := range layers {
|
||||
sample += layer.Amplitude(t) * math.Sin(2*math.Pi*layer.Freq*float64(t)/sampleRate)
|
||||
// Probe order: --config flag > ./netsynth.toml > ~/.config/netsynth/config.toml
|
||||
func findConfigPath(flagValue string) (string, bool) {
|
||||
if flagValue != "" {
|
||||
return flagValue, true
|
||||
}
|
||||
pcm[t] = int16(sample * 32767)
|
||||
if _, err := os.Stat("./netsynth.toml"); err == nil {
|
||||
return "./netsynth.toml", true
|
||||
}
|
||||
if dir, err := os.UserConfigDir(); err == nil {
|
||||
p := filepath.Join(dir, "netsynth", "config.toml")
|
||||
if _, err := os.Stat(p); err == nil {
|
||||
return p, true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
```
|
||||
No library adds value here. Libraries designed for real-time playback add complexity (audio thread management, ring buffers, OS audio drivers) that hurts a batch file-output tool.
|
||||
|
||||
`os.UserConfigDir()` returns `$XDG_CONFIG_HOME` if set, else `$HOME/.config` on Linux/macOS — confirmed against Go stdlib docs. No third-party XDG library needed.
|
||||
|
||||
### Additional Waveform Types
|
||||
|
||||
No new dependency. Extend the existing `synth.Oscillator` in place.
|
||||
|
||||
Square, sawtooth, and triangle are pure math — each is ~3 lines. The existing oscillator uses a phase accumulator (0.0–1.0 range), which is the right representation for all four waveforms:
|
||||
|
||||
```go
|
||||
// Waveform enum addition to synth package
|
||||
type Waveform int
|
||||
|
||||
const (
|
||||
WaveformSine Waveform = iota
|
||||
WaveformSquare
|
||||
WaveformSawtooth
|
||||
WaveformTriangle
|
||||
)
|
||||
|
||||
// Per-sample generation (replaces math.Sin call in Advance())
|
||||
func sample(phase float64, w Waveform) float64 {
|
||||
switch w {
|
||||
case WaveformSquare:
|
||||
if phase < 0.5 { return 1.0 }
|
||||
return -1.0
|
||||
case WaveformSawtooth:
|
||||
return 2*phase - 1.0
|
||||
case WaveformTriangle:
|
||||
if phase < 0.5 { return 4*phase - 1.0 }
|
||||
return 3.0 - 4*phase
|
||||
default: // WaveformSine
|
||||
return math.Sin(2 * math.Pi * phase)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `Oscillator` struct gains a `Waveform` field; `Advance()` dispatches to `sample()`. Harmonics still work the same way — each harmonic's phase is `phase * ratio`, which maps correctly for all waveform types.
|
||||
|
||||
---
|
||||
|
||||
## Installation Delta
|
||||
|
||||
```bash
|
||||
# Add only this new dependency
|
||||
go get github.com/BurntSushi/toml@v1.6.0
|
||||
```
|
||||
|
||||
No changes to build flags. `CGO_ENABLED=1` still required for go-lame.
|
||||
|
||||
---
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Where Config Feeds Existing Code
|
||||
|
||||
The TOML config needs to override two existing data structures:
|
||||
|
||||
1. **`synth.ClassFreqConfigs`** (map in `synth/config.go`) — user can override `BaseHz` and add a `Waveform` field per class
|
||||
2. **`classify.DefaultRules`** (slice in `classify/rules.go`) — user can prepend custom rules before the defaults
|
||||
|
||||
The config loader should apply overrides at startup before any other initialization. The cleanest integration is:
|
||||
|
||||
```
|
||||
cmd/netsynth/main.go
|
||||
-> config.Load(path) // returns *AppConfig
|
||||
-> classify.MergeRules(cfg) // prepend user rules to DefaultRules
|
||||
-> synth.ApplyOverrides(cfg) // patch ClassFreqConfigs entries
|
||||
```
|
||||
|
||||
Both `classify.DefaultRules` and `synth.ClassFreqConfigs` are currently package-level vars — they can be replaced or cloned at startup without changing the downstream pipeline.
|
||||
|
||||
### TOML Struct Shape
|
||||
|
||||
The config schema maps naturally to the existing types:
|
||||
|
||||
```toml
|
||||
# netsynth.toml
|
||||
[[rules]]
|
||||
protocol = "tcp"
|
||||
dst_port = 8443
|
||||
class = "my-https-alt"
|
||||
|
||||
[sounds.my-https-alt]
|
||||
frequency = 195.0
|
||||
waveform = "square"
|
||||
|
||||
[sounds.ICMP]
|
||||
frequency = 80.0 # override built-in
|
||||
waveform = "triangle"
|
||||
```
|
||||
|
||||
```go
|
||||
type AppConfig struct {
|
||||
Rules []RuleConfig `toml:"rules"`
|
||||
Sounds map[string]SoundConfig `toml:"sounds"`
|
||||
}
|
||||
|
||||
type RuleConfig struct {
|
||||
Protocol string `toml:"protocol"`
|
||||
DstPort uint16 `toml:"dst_port"`
|
||||
Class string `toml:"class"`
|
||||
}
|
||||
|
||||
type SoundConfig struct {
|
||||
Frequency float64 `toml:"frequency"`
|
||||
Waveform string `toml:"waveform"` // "sine"|"square"|"sawtooth"|"triangle"
|
||||
}
|
||||
```
|
||||
|
||||
Use `toml.Decode()` and check `meta.Undecoded()` to warn on unknown keys.
|
||||
|
||||
---
|
||||
|
||||
## What NOT to Add
|
||||
|
||||
| Avoid | Why | What to Do Instead |
|
||||
|-------|-----|-------------------|
|
||||
| `adrg/xdg` or any XDG library | `os.UserConfigDir()` in stdlib already handles `$XDG_CONFIG_HOME` on Linux — confirmed | Use `os.UserConfigDir()` directly |
|
||||
| `pelletier/go-toml v2` | No advantage over BurntSushi for a single-file startup read; `MetaData.Undecoded()` in BurntSushi is more ergonomic for typo detection | `github.com/BurntSushi/toml` |
|
||||
| `spf13/viper` | Massive dependency (brings in 20+ transitive deps) for a use case that is one TOML file — Viper adds remote config, env var binding, hot reload, none of which are needed | `BurntSushi/toml` + manual flag override |
|
||||
| Any waveform/audio library | Square/sawtooth/triangle are 3 lines of math each; no library adds value | Extend `synth.Oscillator` in place |
|
||||
| `gopkg.in/yaml.v3` or JSON config | TOML is explicitly specified for this milestone and is the right format for user-editable config files (comments supported, less noisy than JSON) | TOML only |
|
||||
|
||||
---
|
||||
|
||||
## Version Compatibility
|
||||
|
||||
| Package | Compatible With | Notes |
|
||||
|---------|-----------------|-------|
|
||||
| `gopacket/gopacket@v1.5.0` | Go 1.24+ | v1.5.0 bumped minimum Go to 1.24; use Go 1.24.x toolchain |
|
||||
| `packetcap/go-pcap` | Linux, macOS (Darwin) | No Windows support; this is acceptable per project constraints |
|
||||
| `sjzar/go-lame@v0.0.9` | Any Go + C compiler; CGO_ENABLED=1 | Embeds LAME C source; no system library dependency |
|
||||
| `spf13/cobra@v1.10.2` | Go 1.20+ | No issues with Go 1.24 |
|
||||
| `muesli/kmeans@v0.3.1` | Go 1.12+ | Stable; no compatibility concerns |
|
||||
| Package | Version | Compatible With | Notes |
|
||||
|---------|---------|-----------------|-------|
|
||||
| `BurntSushi/toml` | v1.6.0 | Go 1.18+ | No issues with Go 1.24 |
|
||||
| `os.UserConfigDir()` | stdlib | Go 1.13+ | Returns `$XDG_CONFIG_HOME` or `$HOME/.config` on Linux |
|
||||
|
||||
---
|
||||
|
||||
## Audio Synthesis Architecture Note
|
||||
## Confidence Assessment
|
||||
|
||||
Do not reach for an audio library. The synthesis requirement is:
|
||||
1. Map each traffic class (ICMP, DNS, HTTPS, SSH, unknown-cluster-N) to a base frequency
|
||||
2. Accumulate packet counts per class per time window (e.g., 500ms buckets)
|
||||
3. Drive oscillator amplitude from smoothed packet rate (exponential moving average)
|
||||
4. Sum N oscillators into PCM samples at 44100 Hz, 16-bit, mono
|
||||
5. Write PCM to WAV via `go-audio/wav`, then encode WAV to MP3 via `sjzar/go-lame`
|
||||
|
||||
The WAV intermediate step decouples synthesis from encoding and gives you a debug artifact. Total synthesis code: ~100 lines. No external library needed.
|
||||
| Area | Confidence | Source |
|
||||
|------|------------|--------|
|
||||
| BurntSushi/toml v1.6.0 version | HIGH | pkg.go.dev confirmed, GitHub releases confirmed |
|
||||
| `os.UserConfigDir()` XDG behavior | HIGH | Official Go stdlib docs at pkg.go.dev/os |
|
||||
| Waveform math (no library needed) | HIGH | Trivial math, Dylan Meeus Go audio blog confirms the same approach |
|
||||
| go-toml v2.3.0 version | HIGH | pkg.go.dev confirmed |
|
||||
| Recommendation of BurntSushi over go-toml v2 | MEDIUM | Based on API ergonomics for the specific `Undecoded()` use case; both would work |
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- `github.com/gopacket/gopacket` releases page — v1.5.0 confirmed, November 2025
|
||||
- `pkg.go.dev/github.com/packetcap/go-pcap` — v0.0.0-20251215, pure Go, Linux/macOS confirmed
|
||||
- `pkg.go.dev/github.com/sjzar/go-lame` — v0.0.9, April 2025, embedded C source confirmed
|
||||
- `pkg.go.dev/github.com/spf13/cobra` — v1.10.2, December 2025
|
||||
- `github.com/go-audio/generator` — archived February 2026 (read-only), do not use
|
||||
- `braheezy.github.io/posts/what-i-learned-about-mp3-encoding/` — author's first-hand account of Go MP3 encoding options, concluded shine-mp3 is not production-grade
|
||||
- `github.com/google/gopacket/issues/1016` — maintenance status discussion confirming community fork is preferred
|
||||
- WebSearch: muesli/kmeans v0.3.1 last release July 2022 — LOW confidence on ongoing maintenance, but algorithm is stable
|
||||
- WebSearch: cobra v1.9.1/v1.10.2 — MEDIUM confidence, confirmed via pkg.go.dev
|
||||
- `pkg.go.dev/github.com/BurntSushi/toml` — v1.6.0 confirmed, December 18, 2025
|
||||
- `github.com/BurntSushi/toml/releases` — v1.6.0 release notes, TOML 1.1 enabled by default
|
||||
- `pkg.go.dev/github.com/pelletier/go-toml/v2` — v2.3.0 confirmed, March 24, 2026
|
||||
- `pkg.go.dev/os#UserConfigDir` — XDG_CONFIG_HOME behavior on Linux confirmed via official Go docs
|
||||
- `dylanmeeus.github.io/posts/audio-from-scratch-pt8/` — Go waveform synthesis from scratch, confirms no library needed
|
||||
- `github.com/golang/go/issues/76320` — UserConfigDir XDG_CONFIG_HOME discussion (Nov 2025), confirms existing stdlib support on Linux
|
||||
|
||||
---
|
||||
*Stack research for: NetSynth — Go CLI network-traffic-to-audio tool*
|
||||
*Researched: 2026-03-24*
|
||||
|
||||
*Stack research for: NetSynth v1.1 — Custom Sound Mappings milestone*
|
||||
*Researched: 2026-03-26*
|
||||
|
||||
+111
-112
@@ -1,188 +1,187 @@
|
||||
# Project Research Summary
|
||||
|
||||
**Project:** NetSynth
|
||||
**Domain:** Network traffic sonification CLI — Go, packet capture, audio synthesis, MP3 encoding
|
||||
**Researched:** 2026-03-24
|
||||
**Confidence:** MEDIUM-HIGH
|
||||
**Project:** NetSynth v1.1 — Custom Sound Mappings
|
||||
**Domain:** Network traffic sonification CLI tool (Go)
|
||||
**Researched:** 2026-03-26
|
||||
**Confidence:** HIGH
|
||||
|
||||
## Executive Summary
|
||||
|
||||
NetSynth is a Go CLI tool that captures live network traffic, classifies packets by protocol, and synthesizes an ambient drone MP3 where each protocol layer produces a distinct tonal frequency whose amplitude evolves with traffic volume. There is no direct precedent for this exact form factor: comparable tools (SoNSTAR, Peep, Network-Sonification) all produce real-time audio through OS audio APIs rather than file output, are not single binaries, and do not auto-cluster unknown traffic. The recommended approach builds on well-understood Go concurrency primitives (channel-connected pipeline stages, ticker-driven time windows) rather than audio or ML libraries — the synthesis math is ~100 lines of Go and no external audio framework adds value for batch file output.
|
||||
NetSynth v1.1 extends the working v1.0 CLI by adding user-customizable sound mappings through a TOML config file. The v1.0 codebase is a clean 6-package Go project (3,254 lines) with a well-separated pipeline: capture → classify → aggregate → synthesize → encode. The v1.1 milestone threads a new `config` package through this pipeline, enabling users to override frequencies and waveforms per traffic class, add new classification rules, and reference those custom classes in the synth layer. The recommended implementation path is incremental — introduce waveform types at the oscillator level first, then decouple bank construction from the global config, then add the TOML loader and wire it all together. Each step is independently testable before the next begins.
|
||||
|
||||
The recommended stack is `gopacket/gopacket` v1.5.0 (community fork, not the abandoned Google repo) for packet decode, `packetcap/go-pcap` for pure-Go live capture on Linux/macOS, `sjzar/go-lame` v0.0.9 for embedded-CGo MP3 encoding, and `spf13/cobra` v1.10.2 for CLI structure. The audio synthesis layer should be hand-rolled — additive sine oscillators with exponential-moving-average amplitude smoothing. This stack requires CGo at build time but produces a self-contained binary with no runtime library dependencies beyond `CAP_NET_RAW` or root for packet capture.
|
||||
The stack requires one new dependency: `github.com/BurntSushi/toml` v1.6.0 (zero indirect deps, `MetaData.Undecoded()` provides strict-mode typo detection). All other additions — waveform math, config file discovery, and class name validation — are stdlib-only. The single notable technical choice is waveform synthesis strategy: naive direct-math square/sawtooth/triangle waveforms produce audible aliasing at the frequencies NetSynth uses (65–1047 Hz). The existing additive synthesis infrastructure (`[]HarmonicDef`) is the correct approach, generating bandlimited harmonic series for each waveform type rather than direct time-domain computation.
|
||||
|
||||
The two hardest risks are at opposite ends of the pipeline. On the capture side: privilege requirements, CGo binary distribution contracts, and kernel buffer drops all bite in production but not in local development. On the audio side: PCM integer overflow, perceptual tone masking between protocol frequencies, and LAME initialization order all produce silent or subtle corruption that integration tests must specifically cover. Both risk clusters must be resolved in Phase 1 and Phase 2 respectively — they cannot be retrofitted.
|
||||
The highest-risk integration points are bank construction (which must be extended to handle user-defined classes not in the static `AllClasses()` list) and TOML merge semantics (TOML decoders zero-out absent fields, silently overwriting defaults unless pointer fields are used). Both are well-understood problems with clear prevention patterns that must be established before wiring config into the pipeline. The merge ordering for classification rules also requires deliberate design: user rules must precede specific built-in rules, which must precede catch-alls — three-layer ordering, not two.
|
||||
|
||||
## Key Findings
|
||||
|
||||
### Recommended Stack
|
||||
|
||||
The stack is straightforward for Go developers with one critical trap: `github.com/google/gopacket` is unmaintained (270 open issues, no active merges since 2022) and must never be used — the import path is `github.com/gopacket/gopacket` (community fork, v1.5.0, Go 1.24+ required). For the capture backend, `packetcap/go-pcap` is pure Go and eliminates libpcap CGo entirely; the MP3 encoder `sjzar/go-lame` embeds LAME C source and requires CGo but no system library on the target machine. Audio synthesis should be written directly — no audio library is appropriate for batch file output. See `.planning/research/STACK.md` for full alternatives matrix.
|
||||
The existing stack (gopacket v1.5.0, packetcap/go-pcap, sjzar/go-lame v0.0.9, cobra v1.10.2, hand-rolled sine oscillator + EMA) is unchanged. The only new external dependency is `BurntSushi/toml` v1.6.0, selected over `pelletier/go-toml v2` because `MetaData.Undecoded()` is more ergonomic for typo detection on a single startup config read, and over `spf13/viper` because Viper pulls in 20+ transitive deps for features (remote config, env var binding, hot reload) that are irrelevant here. See `.planning/research/STACK.md` for full alternatives analysis.
|
||||
|
||||
**Core technologies:**
|
||||
- `github.com/gopacket/gopacket` v1.5.0: packet decode (ICMP, DNS, TCP, UDP, TLS layers) — only maintained Go packet library
|
||||
- `github.com/packetcap/go-pcap`: live capture backend — pure Go, mmap ring buffer, Linux/macOS, no CGo
|
||||
- `github.com/sjzar/go-lame` v0.0.9: MP3 encoding — embeds LAME C source, no runtime `.so` dependency, requires CGo at build time
|
||||
- `github.com/spf13/cobra` v1.10.2: CLI flag parsing — industry standard, handles signal plumbing and help generation
|
||||
- `github.com/go-audio/wav`: WAV intermediate format — decouples synthesis from encoding, provides debug artifact
|
||||
- Hand-rolled additive synthesizer: 30-50 lines of Go sine oscillators, no library needed
|
||||
- `github.com/gopacket/gopacket` v1.5.0 — packet decode — only maintained Go packet library, Go 1.24+
|
||||
- `github.com/packetcap/go-pcap` — pure-Go live capture backend — no CGo, mmap ring buffer, Linux/macOS
|
||||
- `github.com/sjzar/go-lame` v0.0.9 — MP3 encoding — embeds LAME C source, CGO_ENABLED=1, no system library required
|
||||
- `github.com/spf13/cobra` v1.10.2 — CLI structure and flag handling — industry standard
|
||||
- `github.com/BurntSushi/toml` v1.6.0 — TOML config parsing — zero transitive deps, strict-mode via `Undecoded()`
|
||||
- Hand-rolled additive oscillator (sine today, square/sawtooth/triangle in v1.1) — no audio library needed
|
||||
|
||||
### Expected Features
|
||||
|
||||
NetSynth has a well-defined feature set. All precedent tools provide real-time audio output, not file output — this is both a differentiator and a source of user confusion to address in UX copy. Auto-clustering of unknown traffic is unique to NetSynth among comparable tools. See `.planning/research/FEATURES.md` for full prioritization matrix and competitor analysis.
|
||||
**Must have (table stakes) for v1.1:**
|
||||
- TOML config auto-discovery (`./netsynth.toml`, `~/.config/netsynth/config.toml`) — XDG Base Directory Specification standard
|
||||
- `--config` flag for explicit path, with hard error if file is absent
|
||||
- Partial override semantics — absent keys retain defaults; users must not replicate the full table to change one field
|
||||
- Custom frequency per traffic class — direct override of `synth.ClassFreqConfigs`
|
||||
- Custom waveform per traffic class — sine/square/sawtooth/triangle selection
|
||||
- User-defined classification rules with custom class names, prepended before built-in rules
|
||||
- Startup-time config validation with line-number errors (fail before capture begins, not after)
|
||||
- Unknown field detection — prevents silent typos (`frequncy` must be caught, not silently ignored)
|
||||
|
||||
**Must have (table stakes):**
|
||||
- Interface selection (`-i eth0`) and `--list-interfaces` — packet capture CLI convention
|
||||
- Live capture with graceful Ctrl+C stop producing a valid MP3 — core interaction model
|
||||
- Per-protocol sound distinction: ICMP, DNS, TCP/443, TCP/other, UDP — minimum fingerprint set
|
||||
- Ambient drone synthesis with time-windowed amplitude evolution — core differentiator
|
||||
- MP3 encoding and file output with sensible default filename — deliverable artifact
|
||||
- Capture statistics summary on exit (stderr) — expected by every capture tool user
|
||||
- Privilege error detection with actionable message — prevents silent failure
|
||||
- Minimum viable duration guard — zero-packet capture must not produce a corrupt file
|
||||
|
||||
**Should have (competitive):**
|
||||
- Auto-clustering of unrecognized traffic into stable drone layers — honest representation, unique feature
|
||||
- BPF capture filter (`--filter`) — power user scope control
|
||||
- Offline pcap file input (`--read`) — historical analysis and demos
|
||||
- Verbose protocol activity log (`--verbose`) — debugging and demo use cases
|
||||
- Configurable time window duration (`--window`) — tuning for responsiveness vs. smoothness
|
||||
**Should have (differentiators):**
|
||||
- `netsynth --print-config` subcommand dumping effective config as commented TOML — critical for discoverability
|
||||
- Named custom rules (display name appears in exit summary and `--verbose` output)
|
||||
- Clear error message listing valid waveform values on invalid input
|
||||
|
||||
**Defer (v2+):**
|
||||
- Custom sound mapping configuration file — needs user research first; well-chosen defaults cover v1
|
||||
- Improved clustering (full k-means on flow features) — hash-bucketing is sufficient for v1
|
||||
- Multi-interface capture — adds deduplication complexity
|
||||
- Real-time audio playback — anti-feature; triples cross-platform complexity, conflicts with file-output simplicity
|
||||
- Harmonic override per class (expose `HarmonicDef` slice in TOML) — niche, adds TOML nesting complexity
|
||||
- Stereo pan position in config — explicitly deferred per project constraints
|
||||
- Config hot-reload during capture — mid-capture state change corrupts synthesis; not worth the complexity
|
||||
- Multiple config file includes/inheritance — single file merged with in-code defaults is sufficient
|
||||
|
||||
### Architecture Approach
|
||||
|
||||
The architecture is a channel-connected pipeline of five stages, each a goroutine communicating via buffered channels, with a `done` channel closed on Ctrl+C driving clean shutdown across all stages. The stages are: Capture (pcap handle → `chan gopacket.Packet`), Classification (rule-based protocol dispatch + unknown traffic bucketer → `chan ClassifiedPacket`), Aggregation (ticker-driven time-window accumulator → `chan WindowSnapshot`), Synthesis (per-class sine oscillators with EMA amplitude smoothing → PCM blocks), and Encoding (LAME encoder goroutine consuming PCM blocks, separate from synthesis to decouple CGo latency). This is the idiomatic Go pipeline pattern and directly follows the build order the architecture research prescribes. See `.planning/research/ARCHITECTURE.md` for full data flow diagrams and channel buffer size recommendations.
|
||||
v1.1 adds a `config` package and threads it through the existing pipeline via dependency injection. The key architectural shift: `synth.NewBank` currently reads the package-level global `ClassFreqConfigs`; after v1.1 it accepts a `map[TrafficClass]FreqConfig` parameter, enabling user-defined classes and eliminating hidden global state. The `config` package owns TOML parsing, file discovery, default config wrapping, and merge logic. `classify.DefaultRules` splits into `SpecificRules` + `CatchAllRules` so user rules can be inserted between them. The oscillator gains a `Waveform` enum field with additive-synthesis dispatch. All changes are contained to well-bounded components; `classify/classifier.go` and `encode/mp3.go` change only at their call sites. See `.planning/research/ARCHITECTURE.md` for full data flow diagrams and step-by-step build order.
|
||||
|
||||
**Major components:**
|
||||
1. `capture/` — pcap handle wrapper; privilege/interface boundary; all downstream code is pcap-free
|
||||
2. `classify/` — protocol rule dispatch + feature-hash bucketer for unknown traffic; testable with synthetic packets
|
||||
3. `aggregate/` — ticker-driven time-window accumulator; the only stateful time-domain component
|
||||
4. `synth/` — sine oscillators + EMA amplitude smoothing + mixer; pure PCM math, no I/O
|
||||
5. `encode/` — CGo LAME boundary; decoupled encoder goroutine; if encoder changes, only this package changes
|
||||
6. `config/` — static frequency-to-protocol mapping table; prevents magic numbers in synth/
|
||||
1. `config/` (new) — TOML struct definitions, `Load()`, auto-discovery via `os.UserConfigDir()`, pointer-field merge, `DefaultConfig()`
|
||||
2. `synth/oscillator.go` (modified) — `Waveform` enum, `sampleAt()` dispatch, bandlimited harmonic series generation at config load time
|
||||
3. `synth/bank.go` (modified) — accepts freq config map param; iterates map keys, not hardcoded `AllClasses()`
|
||||
4. `classify/rules.go` (modified) — split into `SpecificRules` + `CatchAllRules`; `MergeRules(userRules)` export
|
||||
5. `cmd/netsynth/main.go` (modified) — `--config` flag, `config.Load()`, user rule merge, config forwarded to `RunSynthesis`
|
||||
|
||||
### Critical Pitfalls
|
||||
|
||||
Nine pitfalls identified, ranging from critical (causes data corruption or broken binaries) to moderate (causes poor audio quality). The top five require architectural decisions in Phase 1 or Phase 2 — they cannot be patched later without significant rewrite. See `.planning/research/PITFALLS.md` for recovery strategies and the full "Looks Done But Isn't" checklist.
|
||||
1. **TOML decoder zeros absent fields, silently overwriting defaults (Pitfall A1)** — Use pointer fields (`*float64`, `*string`) for all optional overrides in the decoded struct. Apply an explicit merge function that only writes non-nil values over the built-in defaults. Establish this pattern before any config is wired into the bank.
|
||||
|
||||
1. **Wrong gopacket fork (`google/` vs `gopacket/`)** — use `github.com/gopacket/gopacket` from day one; import path migration across the whole codebase is the recovery cost
|
||||
2. **CGo breaking the single-binary promise** — decide the static vs. dynamic linking strategy before writing capture code; verify with `ldd` on a clean Alpine container in CI
|
||||
3. **`CAP_NET_RAW` + nosuid filesystem = silent failure** — install to `/usr/local/bin` for capability mode; always provide a `sudo ./netsynth` fallback with clear error messaging
|
||||
4. **PCM integer overflow producing wrap-around distortion** — synthesize internally in `float64 [-1.0, 1.0]`, clamp before casting to `int16`; never do audio math in integer types
|
||||
5. **Perceptual tone masking (all protocols in the same frequency band)** — design the frequency table with register separation: low drones for bulk traffic, mid for control, high for interactive; use harmonic intervals not arithmetic spacing
|
||||
2. **User rules appended after catch-alls are unreachable (Pitfall A5/A9)** — `DefaultRules` ends with catch-all rules (`DstPort: 0`) that match any TCP/UDP packet. Appending user rules after them makes user rules unreachable. Split into `SpecificRules` + `CatchAllRules`; merge order must be `userRules + SpecificRules + CatchAllRules`.
|
||||
|
||||
3. **User-defined classes have no bank layer — panic or silence (Pitfall A6)** — `NewBank` currently iterates `classify.AllClasses()` (a hardcoded list of 14 built-in classes). User classes will not be in that list. `NewBank` must iterate the keys of the merged `FreqConfig` map instead. Validate at config load that every rule's class name resolves to a configured sound entry.
|
||||
|
||||
4. **Naive square/sawtooth/triangle waveforms produce audible aliasing (Pitfall A3)** — Direct time-domain math generates infinite harmonics that alias above Nyquist. Use additive synthesis: generate a bandlimited `[]HarmonicDef` series (odd harmonics for square/triangle, all harmonics for sawtooth, truncated at Nyquist) at config load time. The existing `HarmonicDef` infrastructure already supports this approach.
|
||||
|
||||
5. **BurntSushi/toml silently ignores unknown keys by default (Pitfall A2)** — Use `toml.Decode()` (not `Unmarshal`) to obtain `MetaData`, then call `md.Undecoded()` and return an error listing any unrecognized keys. Implement strict decoding from the first config load function.
|
||||
|
||||
## Implications for Roadmap
|
||||
|
||||
Based on research, the architecture's build-order prescription maps directly to a three-phase roadmap. Each phase is independently testable before the next is wired in.
|
||||
The v1.1 work has clear dependency ordering that directly dictates phase structure. The build order in ARCHITECTURE.md (7 steps, each independently testable) maps naturally to implementation phases.
|
||||
|
||||
### Phase 1: Capture and Classification Pipeline
|
||||
### Phase 1: Waveform Types in the Oscillator
|
||||
|
||||
**Rationale:** The packet capture layer carries the highest technical risk (privilege, CGo, binary distribution, buffer overflow). Validating it first — before any audio code exists — means the hardest pitfalls are resolved while the codebase is small. The architecture research explicitly names this as the correct first step.
|
||||
**Rationale:** Zero external dependencies; pure math testable in isolation with golden-sample unit tests. The critical design decision — additive synthesis vs. direct math — must be made and locked in here. Switching after integration is a full oscillator rewrite.
|
||||
**Delivers:** `Waveform` enum, `sampleAt()` dispatch, `BandlimitedHarmonics()` generator; all four waveform types produce correct, alias-free output at all NetSynth frequencies.
|
||||
**Addresses:** Custom waveform per class (table stakes)
|
||||
**Avoids:** Pitfall A3 (aliasing from naive waveforms)
|
||||
**Files changed:** `synth/oscillator.go` only
|
||||
|
||||
**Delivers:** A working CLI that opens a network interface, classifies packets by protocol, and prints a live traffic summary to stderr. No audio output yet — just proof the pipeline works. The privilege error message and `--list-interfaces` flag ship here.
|
||||
### Phase 2: Decouple Bank from Global Config
|
||||
|
||||
**Addresses:** Interface selection, live capture, protocol classification, privilege detection, `--list-interfaces`, capture statistics
|
||||
**Rationale:** Prerequisite for config injection. `NewBank` must accept an injected config map before the `config` package exists. Wiring `Waveform` through `FreqConfig` → `Layer` → `Oscillator` is included here; zero-value default (`WaveformSine = 0`) means existing tests pass unchanged.
|
||||
**Delivers:** `NewBank(tau, cfgs map[TrafficClass]FreqConfig)`, `FreqConfig.Waveform` field, `synth/layer.go` updated. System is functionally identical to v1.0 but injectable.
|
||||
**Avoids:** Global-read anti-pattern (multiple places reading `ClassFreqConfigs`, ambiguous merge point)
|
||||
**Files changed:** `synth/config.go`, `synth/layer.go`, `synth/bank.go`, `encode/mp3.go`
|
||||
|
||||
**Avoids:** Wrong gopacket fork (Pitfall 1), CGo distribution contract (Pitfall 2), CAP_NET_RAW binary location (Pitfall 3), packet buffer overflow (Pitfall 4), ZeroCopy use-after-free (Pitfall 5)
|
||||
### Phase 3: Config Package — TOML Loading and Merge
|
||||
|
||||
### Phase 2: Audio Synthesis Engine
|
||||
**Rationale:** Core new infrastructure. Builds on the injectable bank signature from Phase 2. All config correctness patterns (pointer fields, strict decode, validation, string normalization) must be established here in isolation before any features are wired to the bank. Retrofitting these patterns after pipeline integration is significantly more expensive.
|
||||
**Delivers:** `config/` package with `Load()`, `os.UserConfigDir()` discovery, pointer-field merge, `validate()` with string normalization and enum checking, `DefaultConfig()` wrapping existing values.
|
||||
**Addresses:** Config auto-discovery, `--config` flag, partial override semantics, unknown field detection, startup validation, clear error messages
|
||||
**Avoids:** Pitfalls A1 (zero-value overwrite), A2 (silent typos), A4 (waveform string case), A7 (XDG ignored), A8 (missing explicit path), A10 (empty class name)
|
||||
|
||||
**Rationale:** The synthesis engine is pure PCM math with no pcap dependency. It can be built and tested in isolation with synthetic `WindowSnapshot` inputs before any real traffic flows through it. This is the architecture research's explicit recommendation. Separating synthesis from capture also means audio bugs are diagnosed without needing a live network.
|
||||
### Phase 4: Classification Rule Merging
|
||||
|
||||
**Delivers:** A synthesizer that accepts `WindowSnapshot` inputs and produces a valid MP3 file. End-to-end smoke test: silence input → `ffprobe`-validated MP3 output. The frequency mapping table, EMA amplitude smoothing, and the LAME encoder goroutine all ship here.
|
||||
**Rationale:** `classify/rules.go` must be split before any config-loading logic references the rule slice. The three-layer merge order is a design decision that, if wrong, produces silent failures with no error messages — it must be validated with unit tests before pipeline integration.
|
||||
**Delivers:** `classify.SpecificRules`, `classify.CatchAllRules`, `classify.MergeRules(userRules []Rule) []Rule`; user rules prepend correctly with both catch-all and specific-rule ordering.
|
||||
**Avoids:** Pitfall A5 (unreachable rules after catch-alls), Pitfall A9 (user rule shadowed by built-in specific rule for same port)
|
||||
|
||||
**Uses:** `sjzar/go-lame`, `go-audio/wav`, hand-rolled oscillator, `config/mapping.go` frequency table
|
||||
### Phase 5: Wire Config Through Pipeline — Frequency and Waveform Overrides
|
||||
|
||||
**Implements:** `synth/` (oscillator, layer, mixer), `encode/` (LAME goroutine), `aggregate/` (time-window accumulator), `config/` (frequency mapping)
|
||||
**Rationale:** Connects the config package to the synth layer for built-in classes only. Validates the full pipeline end-to-end before adding the complexity of user-defined classes. `encode.RunSynthesis` signature change is a breaking API change — all call sites must be updated in a single commit.
|
||||
**Delivers:** `--config` cobra flag, `config.Load()` in `main.go`, merged freq config map passed to `RunSynthesis` and `NewBank`; end-to-end test: TOML sets HTTPS to 200 Hz sawtooth, bank produces 200 Hz sawtooth layer.
|
||||
**Implements:** Config → synth integration
|
||||
|
||||
**Avoids:** LAME initialization errors (Pitfall 6), PCM overflow wrap-around (Pitfall 7), perceptual tone masking (Pitfall 8), time window jitter (Pitfall 9)
|
||||
### Phase 6: User-Defined Classes End-to-End
|
||||
|
||||
### Phase 3: Pipeline Integration and CLI Polish
|
||||
**Rationale:** The most complex integration; requires all prior phases. User-defined classes create new `TrafficClass` strings that must exist in both the merged rule set and the bank's layer map. The AllClasses() decoupling from Phase 2 makes this tractable.
|
||||
**Delivers:** `[[rules]]` TOML section, dynamic `TrafficClass` values from config, bank layers constructed from merged FreqConfig map keys, class name cross-validation at config load.
|
||||
**Avoids:** Pitfall A6 (user-defined class has no bank layer — nil panic or silence)
|
||||
|
||||
**Rationale:** Wire the Phase 1 capture/classify pipeline to the Phase 2 synthesis engine via the aggregation layer. Add Ctrl+C shutdown producing a valid MP3 (requires coordinated drain across all goroutines). Add auto-clustering of unknown traffic. Add UX features (progress output, file collision warning, graceful empty-capture error).
|
||||
### Phase 7: Print-Config and UX Polish
|
||||
|
||||
**Delivers:** The complete v1 MVP: live capture → protocol classification + auto-clustering → time-windowed synthesis → MP3 file output. Graceful Ctrl+C with valid MP3. Full UX surface (startup interface announcement, per-window progress line, exit statistics).
|
||||
|
||||
**Addresses:** Auto-clustering, graceful Ctrl+C stop, capture statistics, progress feedback, output file collision warning, zero-packet guard, `main.go` pipeline wiring
|
||||
|
||||
**Uses:** All Phase 1 and Phase 2 components; `muesli/kmeans` or hash-bucketing for unknown traffic clustering
|
||||
|
||||
### Phase 4: Power User Features (v1.x)
|
||||
|
||||
**Rationale:** These features add value for specific user segments but have no blocking dependencies on each other — add in any order based on user feedback after the core is validated.
|
||||
|
||||
**Delivers:** BPF capture filter (`--filter`), offline pcap file input (`--read`), verbose protocol log (`--verbose`), configurable time window (`--window`), configurable output duration for pcap input (`--duration`)
|
||||
|
||||
**Addresses:** All P2 features from the prioritization matrix in FEATURES.md
|
||||
**Rationale:** `--print-config` is independent of capture and must wait until all config structure is stable (Phase 6). Additive, zero regression risk.
|
||||
**Delivers:** `netsynth --print-config` subcommand with commented TOML output of effective config; optional `name` field on user rules displayed in exit summary.
|
||||
|
||||
### Phase Ordering Rationale
|
||||
|
||||
- **Capture before synthesis:** The privilege and CGo pitfalls are foundational — an audio-first approach would hide them until integration and make them expensive to fix.
|
||||
- **Synthesis in isolation:** Pure PCM math is independently testable. Building it against synthetic inputs before real traffic makes audio bugs fast to diagnose.
|
||||
- **Integration as its own phase:** The shutdown coordination (Ctrl+C → drain → flush encoder → close file) across five goroutines is non-trivial; it deserves focused attention rather than being an afterthought of feature development.
|
||||
- **Power features deferred:** BPF filter and offline pcap do not validate the core concept; they add complexity to the capture layer that should wait until the pipeline is stable.
|
||||
- Phases 1–2 are internal refactors with no user-visible change — the right starting point for establishing patterns safely
|
||||
- Phase 3 owns all config safety in one isolated package before anything is wired — retrofitting pointer-field merge after bank integration means touching multiple packages simultaneously
|
||||
- Phase 4 (rule splitting) must precede Phase 6 (user rules) or catch-all ordering bugs surface silently at integration with no clear failure signal
|
||||
- Phase 5 validates the full pipeline with familiar built-in classes before Phase 6 introduces the harder user-defined class problem
|
||||
- Phase 7 is pure additive polish with zero risk of breaking earlier phases
|
||||
|
||||
### Research Flags
|
||||
|
||||
Phases likely needing deeper research during planning:
|
||||
Phases with well-documented patterns — skip additional research:
|
||||
- **Phase 1:** DSP textbook math; harmonic series are fully specified
|
||||
- **Phase 2:** Standard dependency injection refactor; no unknowns
|
||||
- **Phase 3:** BurntSushi/toml API is well-documented; pointer-field merge is a known TOML pattern
|
||||
- **Phase 4:** Simple slice manipulation; no external dependencies
|
||||
- **Phase 7:** Cobra subcommand and TOML marshal are standard patterns
|
||||
|
||||
- **Phase 2 (Audio Synthesis):** The perceptual frequency mapping table requires listening tests, not just code correctness. Research the auditory masking literature before finalizing `config/mapping.go`. Consider consulting the SoNSTAR PLOS One paper on time window choices.
|
||||
- **Phase 3 (Auto-clustering):** The decision between simple hash-bucketing and k-means clustering (muesli/kmeans) depends on what "meaningfully distinct drone layers" means in practice. This needs a working synthesis engine to evaluate — defer the decision to Phase 3 planning.
|
||||
- **Phase 4 (Offline pcap):** Time-compression of multi-hour pcap files to a fixed audio duration needs a clear algorithm decision (proportional window scaling vs. fixed window with truncation). Research this when Phase 4 is planned.
|
||||
|
||||
Phases with standard patterns (skip research-phase):
|
||||
|
||||
- **Phase 1 (Capture pipeline):** Go channel pipelines and gopacket usage are thoroughly documented. The pitfalls are known and avoidable with the guidance in PITFALLS.md.
|
||||
- **Phase 3 (Pipeline integration):** Go done-channel shutdown patterns are canonical (Go Blog: Pipelines). No novel research needed.
|
||||
Phases that may benefit from a targeted research pass or design review:
|
||||
- **Phase 5:** `encode.RunSynthesis` signature change is a breaking API change — verify all test call sites and plan a single-commit update
|
||||
- **Phase 6:** User-defined class name collision with built-in `TrafficClass` string values (e.g., user names a class `"HTTPS"`) requires a design decision: treat as override of built-in sound vs. reject as ambiguous. Not resolved in research; decide before coding Phase 6.
|
||||
|
||||
## Confidence Assessment
|
||||
|
||||
| Area | Confidence | Notes |
|
||||
|------|------------|-------|
|
||||
| Stack | HIGH | Core libraries confirmed via pkg.go.dev; version numbers verified; `go-audio/generator` archived status confirmed February 2026 |
|
||||
| Features | MEDIUM | Niche domain with few direct CLI comparators; feature set derived from tcpdump conventions and sonification research, not user surveys |
|
||||
| Architecture | MEDIUM-HIGH | Go pipeline patterns are HIGH confidence (official Go Blog); audio synthesis architecture inferred from SoNSTAR paper (MEDIUM, abstract-level access only) |
|
||||
| Pitfalls | HIGH | Packet capture pitfalls verified against official gopacket issues and libpcap docs; audio pitfalls cross-referenced against DSP literature and encoder post-mortems |
|
||||
| Stack | HIGH | Existing stack validated in v1.0; BurntSushi/toml v1.6.0 confirmed via pkg.go.dev and GitHub releases; `os.UserConfigDir()` XDG behavior confirmed against official Go stdlib docs |
|
||||
| Features | HIGH | Config file conventions verified against XDG spec, git, golangci-lint, and mise patterns; TOML schema grounded in existing v1.0 codebase types |
|
||||
| Architecture | HIGH | Based on direct code inspection of the 3,254-line v1.0 codebase; all integration points identified with specific file/line references and build order prescribed |
|
||||
| Pitfalls | HIGH | TOML default-overwrite behavior verified against upstream issue trackers (BurntSushi/toml #47, go-toml #252); aliasing prevention verified against DSP literature (CCRMA paper, McGill bandlimited synthesis notes) |
|
||||
|
||||
**Overall confidence:** MEDIUM-HIGH
|
||||
**Overall confidence:** HIGH
|
||||
|
||||
### Gaps to Address
|
||||
|
||||
- **Frequency mapping validation:** The correct frequency assignments for the protocol drone layers require subjective listening tests with real traffic. The research prescribes the *approach* (register separation, harmonic intervals) but not specific Hz values. Validate during Phase 2 with a listening session before Phase 3 integration.
|
||||
- **Auto-clustering granularity:** How many unknown-traffic clusters are perceptually useful? The research suggests hash-bucketing is acceptable for v1, but does not validate how many distinct cluster tones are distinguishable simultaneously. Validate during Phase 3 with real mixed traffic.
|
||||
- **muesli/kmeans maintenance status:** Last release July 2022 (LOW confidence on ongoing maintenance). If Go 1.24 compatibility issues emerge, the alternative is `mpraski/clusters` (online clustering) or a hand-rolled hash bucketer. Plan for substitution.
|
||||
- **macOS privilege model:** CAP_NET_RAW pitfall was verified for Linux. macOS uses a different privilege model (BPF device permissions). If macOS is a target, verify the privilege flow and error messages during Phase 1.
|
||||
- **User class name collision with built-in class strings:** If a user writes `class = "HTTPS"` in a `[[rules]]` block, the intent could be "override built-in sound" or "create a parallel custom class." The merge logic needs an explicit decision before Phase 6: treat matching names as overrides (simplest) or require a separate TOML section. This is a UX design question — resolve before coding Phase 6.
|
||||
- **BurntSushi/toml vs. go-toml v2:** Both work for this use case. BurntSushi is recommended for ergonomics, but if the team prefers go-toml v2's `DisallowUnknownFields()` pattern, it is equally valid with minor API differences. Either choice is fine; just make one and be consistent.
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (HIGH confidence)
|
||||
|
||||
- `github.com/gopacket/gopacket` releases — v1.5.0 November 2025, Go 1.24+ confirmed
|
||||
- `pkg.go.dev/github.com/packetcap/go-pcap` — pure Go, Linux/macOS confirmed
|
||||
- `pkg.go.dev/github.com/sjzar/go-lame` — v0.0.9 April 2025, embedded C source confirmed
|
||||
- `pkg.go.dev/github.com/spf13/cobra` — v1.10.2 December 2025
|
||||
- Go Blog: Pipelines and cancellation — canonical Go pipeline pattern
|
||||
- `google/gopacket` issue #1016 — unmaintained status confirmed
|
||||
- `google/gopacket` issue #329 — 98% packet drop under high traffic, afpacket solution
|
||||
- linuxvox.com: CAP_NET_RAW + nosuid filesystem behavior
|
||||
- Direct code inspection: `synth/config.go`, `synth/oscillator.go`, `synth/bank.go`, `synth/layer.go`, `classify/classifier.go`, `classify/rules.go`, `classify/types.go`, `encode/mp3.go`, `cmd/netsynth/main.go`
|
||||
- `pkg.go.dev/github.com/BurntSushi/toml` — v1.6.0 API, `Undecoded()` strict mode, pointer field behavior
|
||||
- `github.com/BurntSushi/toml/issues/47` — default-overwrite behavior when using `Unmarshal` confirmed
|
||||
- `pkg.go.dev/os#UserConfigDir` — XDG_CONFIG_HOME behavior on Linux confirmed via official Go stdlib docs
|
||||
- `ccrma.stanford.edu/~stilti/papers/blit.pdf` — bandlimited synthesis theory (alias-free waveforms)
|
||||
- `music.mcgill.ca/~gary/307/week5/bandlimited.html` — truncated harmonic series approach confirmed
|
||||
|
||||
### Secondary (MEDIUM confidence)
|
||||
|
||||
- SoNSTAR PLOS One paper (arXiv 1712.07029) — time window design and sonification architecture
|
||||
- braheezy.github.io: Go MP3 encoding options — shine-mp3 not production-grade
|
||||
- Dylan Meeus: Audio From Scratch With Go — PCM synthesis patterns
|
||||
- `github.com/go-audio/generator` — archived February 2026, do not use (confirmed read-only)
|
||||
- XDG Base Directory Specification — config discovery precedence order
|
||||
- `mise.jdx.dev/configuration.html` — working-dir + XDG config discovery pattern
|
||||
- `golangci-lint.run/docs/configuration/cli/` — partial override config in Go CLI tools
|
||||
- `dylanmeeus.github.io/posts/audio-from-scratch-pt8/` — Go waveform synthesis from scratch, confirms no library needed
|
||||
- `pkg.go.dev/github.com/pelletier/go-toml/v2` — v2.3.0 strict decoder comparison; `pelletier/go-toml/issues/252` partial v2 resolution of default-overwrite
|
||||
|
||||
### Tertiary (LOW confidence)
|
||||
|
||||
- `muesli/kmeans` v0.3.1 (July 2022) — last release date only; ongoing Go 1.24 compatibility unverified
|
||||
- Drone auralization model, Acta Acustica 2024 — amplitude/frequency modulation patterns (MEDIUM, used for perceptual guidance)
|
||||
- Competitor feature table (SoNSTAR, Network-Sonification, Peep) — niche domain, limited documentation; used for context only, not binding decisions
|
||||
|
||||
---
|
||||
*Research completed: 2026-03-24*
|
||||
*Research completed: 2026-03-26*
|
||||
*Ready for roadmap: yes*
|
||||
|
||||
Reference in New Issue
Block a user