docs: complete project research
This commit is contained in:
+363
-249
@@ -1,208 +1,332 @@
|
||||
# Architecture Patterns
|
||||
|
||||
**Domain:** Network traffic sonification CLI (Go) — v1.1 Custom Sound Mappings
|
||||
**Researched:** 2026-03-26
|
||||
**Confidence:** HIGH — based on direct code inspection of the existing v1.0 codebase
|
||||
**Domain:** Network traffic sonification CLI (Go) — v1.2 Extended Protocol Coverage with Grouped Families
|
||||
**Researched:** 2026-03-26 (v1.0/v1.1), updated 2026-03-27 (v1.2 grouped protocol families)
|
||||
**Confidence:** HIGH — based on direct code inspection of the shipped v1.1 codebase
|
||||
|
||||
---
|
||||
|
||||
## v1.1 Integration Overview
|
||||
## v1.2 Integration Overview
|
||||
|
||||
This document supersedes the pre-implementation v1.0 architecture research. It is grounded in the actual codebase (3,254 lines, 6 packages) and answers: what changes, what's new, and in what order.
|
||||
This document supersedes the pre-implementation v1.0/v1.1 architecture research. It is grounded in the actual shipped v1.1 codebase (~4,675 lines, 7 packages) and answers specifically: how does the *group concept* thread through classify, synth, config, and CLI? What changes, what stays, and in what order?
|
||||
|
||||
---
|
||||
|
||||
## Existing Package Map (v1.0 Baseline)
|
||||
## Current Package Map (v1.1 Baseline — the starting point)
|
||||
|
||||
```
|
||||
cmd/netsynth/main.go CLI, pipeline wiring, Cobra flags
|
||||
capture/ go-pcap live capture + pcap file reader + BPF
|
||||
classify/
|
||||
types.go TrafficClass, ClassifiedPacket, WindowSnapshot
|
||||
types.go TrafficClass (string type), ClassifiedPacket, WindowSnapshot, AllClasses()
|
||||
classifier.go NewClassifier(rules []Rule) — first-match-wins
|
||||
rules.go DefaultRules []Rule (12 hardcoded rules)
|
||||
rules.go DefaultRules []Rule (12 built-in rules)
|
||||
aggregate/
|
||||
window.go 500ms time-windowed snapshot accumulation
|
||||
synth/
|
||||
config.go ClassFreqConfigs — fixed map[TrafficClass]FreqConfig
|
||||
oscillator.go Phase-accumulator oscillator — sine only
|
||||
layer.go EMA amplitude smoothing per layer
|
||||
bank.go NewBank(tau) — one Layer per AllClasses()
|
||||
config.go FreqConfig{BaseHz, Harmonics, Pan, WaveformType}, ClassFreqConfigs map
|
||||
oscillator.go Phase-accumulator oscillator — sine+additive harmonics
|
||||
layer.go EMA amplitude smoothing per Layer
|
||||
bank.go NewBank(tau, cfgs map) — one Layer per config entry
|
||||
mixer.go PanGains, StereoFramesToInt16Bytes
|
||||
encode/
|
||||
mp3.go RunSynthesis(snapshots, path) — NewBank + EncodeMP3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What v1.1 Adds
|
||||
|
||||
Three independent but related features:
|
||||
|
||||
1. **TOML config file** — override frequencies and waveforms per built-in class
|
||||
2. **Additional waveforms** — square, sawtooth, triangle alongside existing sine
|
||||
3. **User-defined classification rules** — TOML-defined rules prepended before DefaultRules
|
||||
|
||||
---
|
||||
|
||||
## Integration Point Analysis
|
||||
|
||||
### Feature 1: TOML Config File
|
||||
|
||||
**Where config is consumed today:** `synth/config.go` holds a package-level `var ClassFreqConfigs`. `synth/bank.go:NewBank()` reads it directly with `ClassFreqConfigs[class]`. No config is passed through `encode.RunSynthesis` or `main.go`.
|
||||
|
||||
**Required change:** `NewBank` must accept a config parameter instead of reading the global. `encode.RunSynthesis` must accept and forward a config. `main.go` must load config from disk and pass it in.
|
||||
|
||||
**New package: `config/`**
|
||||
|
||||
This package does not exist yet in the codebase (the pre-implementation research anticipated it but it was deferred). It should own:
|
||||
- TOML struct definitions
|
||||
- File discovery logic (auto-detect `./netsynth.toml`, then `~/.config/netsynth/config.toml`)
|
||||
- Merging: loaded config overlays defaults, does not replace them entirely
|
||||
|
||||
```
|
||||
config/
|
||||
config.go Config struct, Load(path string) (*Config, error)
|
||||
defaults.go DefaultConfig() — wraps existing ClassFreqConfigs values
|
||||
config.go TOML parse, merge, validate, auto-freq assignment, PrintConfig
|
||||
encode/
|
||||
mp3.go RunSynthesis(snapshots, path, freqCfgs) — NewBank + EncodeMP3
|
||||
```
|
||||
|
||||
**TOML struct shape:**
|
||||
|
||||
```toml
|
||||
[[class]]
|
||||
name = "HTTPS"
|
||||
frequency_hz = 200.0
|
||||
waveform = "sawtooth"
|
||||
|
||||
[[class]]
|
||||
name = "myservice" # user-defined class (Feature 3)
|
||||
frequency_hz = 350.0
|
||||
waveform = "triangle"
|
||||
```
|
||||
|
||||
The `Config` struct passed into `NewBank` should merge with `ClassFreqConfigs`:
|
||||
|
||||
```go
|
||||
// config/config.go
|
||||
type ClassConfig struct {
|
||||
Name string `toml:"name"`
|
||||
FrequencyHz float64 `toml:"frequency_hz"`
|
||||
Waveform string `toml:"waveform"` // "sine" | "square" | "sawtooth" | "triangle"
|
||||
}
|
||||
|
||||
type Config struct {
|
||||
Classes []ClassConfig `toml:"class"`
|
||||
Rules []RuleConfig `toml:"rule"` // Feature 3
|
||||
}
|
||||
```
|
||||
|
||||
**TOML library:** Use `github.com/BurntSushi/toml`. It is the de-facto standard for TOML in Go (used by Hugo, dep, buf, etc.). Already a transitive dependency in many Go module graphs. Provides struct-tag-based decode, good error messages.
|
||||
|
||||
---
|
||||
|
||||
### Feature 2: Additional Waveforms
|
||||
## What v1.2 Adds
|
||||
|
||||
**Where waveform logic lives today:** `synth/oscillator.go:Advance()` — pure sine via `math.Sin`. The `HarmonicDef.Ratio` and `HarmonicDef.Amplitude` fields are stored in `FreqConfig.Harmonics` but the waveform function is hardcoded.
|
||||
Two related but separable capabilities:
|
||||
|
||||
**Required change:** `Oscillator.Advance` must dispatch on a waveform type. Two clean approaches:
|
||||
1. **More built-in protocol rules** — extended DefaultRules covering Mail, Remote Access, Database, Discovery, File Transfer, VoIP, etc.
|
||||
2. **Group concept** — related protocols share a recognizable sound family (shared base frequency, detuned members, optional shared waveform character)
|
||||
|
||||
**Option A (recommended): Waveform enum on Oscillator**
|
||||
These must be designed together because the group concept directly affects frequency allocation, and frequency allocation directly affects DefaultRules ordering decisions.
|
||||
|
||||
Add a `waveform` field to `Oscillator`. `Advance` switches on it. `NewOscillator` gains a waveform parameter.
|
||||
---
|
||||
|
||||
## The Group Concept: What It Means Architecturally
|
||||
|
||||
A "group" means: protocols in the same family share a musical neighborhood. Listeners learn "low rumble = infrastructure traffic" and "mid-range shimmer = web traffic" without needing to identify individual tones.
|
||||
|
||||
The group concept requires:
|
||||
- A **group identifier** stored somewhere accessible by both classify (for `--print-config` ordering) and synth (for within-group detuning computation)
|
||||
- A **group base frequency** from which individual protocol frequencies are derived by small detuning offsets
|
||||
- **Within-group waveform consistency** — all members of a group use the same waveform type so the family sounds recognizable even as individual pitches differ
|
||||
|
||||
The key architectural question is: *where does the group concept live?*
|
||||
|
||||
---
|
||||
|
||||
## Decision: Group Lives in `synth/config.go`, Not in `classify/`
|
||||
|
||||
### Option A: Group field on `classify.Rule` or new `TrafficClassGroup` type in `classify`
|
||||
|
||||
Adding `Group string` to `classify.Rule` would put group information at the classification boundary, visible from the capture pipeline forward. But:
|
||||
- The classifier does not care about groups — it only cares about protocol matching
|
||||
- `AllClasses()` in `classify/types.go` already returns ordered class names without needing sound semantics
|
||||
- Pollutes the `classify` package with audio-domain concerns
|
||||
|
||||
### Option B: Group field in `synth.FreqConfig` (recommended)
|
||||
|
||||
`FreqConfig` already owns all sound parameters (frequency, harmonics, pan, waveform). Adding a `Group string` field keeps group logic inside the audio domain. The `classify` package stays clean — it only knows about `TrafficClass` names.
|
||||
|
||||
```go
|
||||
type Waveform int
|
||||
|
||||
const (
|
||||
WaveformSine Waveform = iota
|
||||
WaveformSquare
|
||||
WaveformSawtooth
|
||||
WaveformTriangle
|
||||
)
|
||||
|
||||
type Oscillator struct {
|
||||
phase float64
|
||||
freq float64
|
||||
sr float64
|
||||
waveform Waveform
|
||||
}
|
||||
|
||||
func (o *Oscillator) sampleAt(phase, ratio float64) float64 {
|
||||
p := phase * float64(ratio)
|
||||
p -= math.Floor(p) // wrap to [0, 1)
|
||||
switch o.waveform {
|
||||
case WaveformSquare:
|
||||
if p < 0.5 { return 1.0 }
|
||||
return -1.0
|
||||
case WaveformSawtooth:
|
||||
return 2.0*p - 1.0
|
||||
case WaveformTriangle:
|
||||
if p < 0.5 { return 4.0*p - 1.0 }
|
||||
return 3.0 - 4.0*p
|
||||
default: // WaveformSine
|
||||
return math.Sin(2 * math.Pi * p)
|
||||
}
|
||||
// synth/config.go
|
||||
type FreqConfig struct {
|
||||
BaseHz float64
|
||||
Harmonics []HarmonicDef
|
||||
Pan float64
|
||||
WaveformType WaveformType
|
||||
Group string // NEW: e.g., "mail", "web", "remote-access", "" for ungrouped
|
||||
}
|
||||
```
|
||||
|
||||
**Option B: Function field on Oscillator**
|
||||
**`classify.TrafficClass` does NOT get a Group field.** The group is a sound-design concept, not a classification concept. Classification says "this is SMTP"; synthesis says "SMTP belongs to the Mail group at 440 Hz + 0 cents detuning."
|
||||
|
||||
Store `waveFn func(phase float64) float64`. More flexible but harder to serialize/configure.
|
||||
### Why this is the right boundary
|
||||
|
||||
Option A is preferred because waveform type maps cleanly to the TOML `waveform` string field without reflection tricks.
|
||||
| Package | Knows About | Does Not Know About |
|
||||
|---------|-------------|---------------------|
|
||||
| `classify` | Protocol, port, traffic class name | Group, frequency, waveform |
|
||||
| `synth` | Frequency, harmonics, pan, waveform, group | Protocol, port |
|
||||
| `config` | Merges both sides; reads TOML; owns PrintConfig | Packet capture |
|
||||
|
||||
**`FreqConfig` change:** Add `Waveform` field:
|
||||
---
|
||||
|
||||
## Frequency Allocation Strategy for Groups
|
||||
|
||||
### Current allocation (v1.1)
|
||||
|
||||
The 14 existing classes span 65 Hz to 1047 Hz in a roughly logarithmic spread:
|
||||
|
||||
```
|
||||
ICMP: 65 Hz (sub-bass)
|
||||
DNS: 110 Hz
|
||||
HTTPS: 175 Hz
|
||||
HTTP: 220 Hz
|
||||
SSH: 330 Hz
|
||||
SMTP: 440 Hz
|
||||
NTP: 520 Hz
|
||||
DHCP: 600 Hz
|
||||
other-TCP: 700 Hz
|
||||
other-UDP: 780 Hz
|
||||
unknown-1: 862 Hz
|
||||
unknown-2: 920 Hz
|
||||
unknown-3: 981 Hz
|
||||
unknown-4: 1047 Hz
|
||||
auto-range: 1200-2350 Hz (FNV-32a hash for user-defined classes)
|
||||
```
|
||||
|
||||
### v1.2 group-based allocation
|
||||
|
||||
New protocols need frequency homes. With ~25-35 total built-in classes after expansion, maintaining individual hand-tuned frequencies becomes fragile and collision-prone. The group model solves this by assigning each group a frequency band, then placing members within that band at small detuning offsets.
|
||||
|
||||
**Recommended group bands (within the 65-1100 Hz range already owned by built-ins):**
|
||||
|
||||
| Group | Band Center | Role | Protocols |
|
||||
|-------|-------------|------|-----------|
|
||||
| Infrastructure | 65-130 Hz | Core network services (barely heard, always present) | ICMP, NTP, DHCP, mDNS/SSDP |
|
||||
| Web | 175-280 Hz | HTTP-family traffic | HTTP, HTTPS, HTTP/3, WebSocket |
|
||||
| Mail | 380-480 Hz | Email protocols | SMTP, IMAP, POP3, SMTP-submission |
|
||||
| Remote Access | 300-360 Hz | Interactive sessions | SSH, RDP, Telnet, VNC |
|
||||
| Database | 500-580 Hz | Backend data stores | MySQL, PostgreSQL, Redis, MongoDB |
|
||||
| File Transfer | 620-700 Hz | File movement protocols | FTP, SFTP, FTPS, SMB, NFS |
|
||||
| VoIP / Streaming | 720-820 Hz | Real-time media | SIP, RTP, RTSP |
|
||||
| DNS | 110 Hz | Singleton — already well-placed | DNS (UDP+TCP) |
|
||||
| Unknown | 850-1100 Hz | Auto-bucketed unrecognized traffic | unknown-1 through unknown-4 |
|
||||
|
||||
**Within-group detuning model:** Each group has a `GroupBaseHz`. Members are offset by small cent-based detuning or fixed Hz offsets:
|
||||
|
||||
```
|
||||
Mail group base: 420 Hz
|
||||
SMTP: 420 Hz (group base, no detune)
|
||||
IMAP: 430 Hz (+10 Hz)
|
||||
POP3: 440 Hz (+20 Hz)
|
||||
SMTP-submission: 450 Hz (+30 Hz)
|
||||
```
|
||||
|
||||
Offsets of 10-30 Hz are audible as distinct pitches but harmonically related enough to sound like a family. Do NOT use harmonic ratios (2x, 3x) for detuning — that would place group members an octave or fifth apart, destroying the "family sound" effect.
|
||||
|
||||
**This is a design constraint, not a library feature.** The detuning is expressed as explicit `BaseHz` values in `ClassFreqConfigs`. There is no runtime detuning computation needed — it is baked into the default config map.
|
||||
|
||||
### Frequency rebalancing required
|
||||
|
||||
Adding ~15-20 new protocols means the existing 65-1100 Hz range must be replanned. The existing classes should be reassigned to group-coherent frequencies even if this moves them from v1.1 values. This is a conscious breaking change to the default sound, acceptable in a minor version bump where the goal is improved audio design.
|
||||
|
||||
The `unknown-1..4` range at 850-1047 Hz is preserved as-is — those are intentionally dissonant and serve a distinct purpose.
|
||||
|
||||
---
|
||||
|
||||
## Component-by-Component Changes
|
||||
|
||||
### `classify/rules.go` — Extend DefaultRules
|
||||
|
||||
**What changes:** Add ~15-20 new `Rule` entries for the expanded protocol set.
|
||||
|
||||
**What does NOT change:** `Rule` struct, `NewClassifier`, the first-match-wins algorithm.
|
||||
|
||||
**Required care:** The catch-all rules (`DstPort: 0` for TCP and UDP) must remain last. New protocol-specific rules must be inserted before these catch-alls. `DefaultRules` is an ordered slice — insertion order matters.
|
||||
|
||||
```go
|
||||
// New rules inserted before catch-alls, e.g.:
|
||||
{Protocol: "tcp", DstPort: 143, Class: ClassIMAP},
|
||||
{Protocol: "tcp", DstPort: 110, Class: ClassPOP3},
|
||||
{Protocol: "tcp", DstPort: 587, Class: ClassSMTPSubmit},
|
||||
{Protocol: "tcp", DstPort: 3306, Class: ClassMySQL},
|
||||
{Protocol: "tcp", DstPort: 5432, Class: ClassPostgres},
|
||||
{Protocol: "tcp", DstPort: 6379, Class: ClassRedis},
|
||||
{Protocol: "tcp", DstPort: 3389, Class: ClassRDP},
|
||||
{Protocol: "tcp", DstPort: 5900, Class: ClassVNC},
|
||||
{Protocol: "udp", DstPort: 5060, Class: ClassSIP},
|
||||
{Protocol: "tcp", DstPort: 5060, Class: ClassSIP},
|
||||
// ... etc
|
||||
```
|
||||
|
||||
### `classify/types.go` — New Constants and Updated AllClasses()
|
||||
|
||||
**What changes:** Add new `TrafficClass` constants for each new protocol. Update `AllClasses()` to include them in display-sensible order (grouped by family, matching PrintConfig output order).
|
||||
|
||||
**What does NOT change:** `ClassifiedPacket`, `WindowSnapshot`, the `TrafficClass` string type itself.
|
||||
|
||||
```go
|
||||
// New constants (representative subset):
|
||||
const (
|
||||
ClassIMAP TrafficClass = "IMAP"
|
||||
ClassPOP3 TrafficClass = "POP3"
|
||||
ClassSMTPSubmit TrafficClass = "SMTP-submit"
|
||||
ClassMySQL TrafficClass = "MySQL"
|
||||
ClassPostgres TrafficClass = "PostgreSQL"
|
||||
ClassRedis TrafficClass = "Redis"
|
||||
ClassMongoDB TrafficClass = "MongoDB"
|
||||
ClassRDP TrafficClass = "RDP"
|
||||
ClassVNC TrafficClass = "VNC"
|
||||
ClassTelnet TrafficClass = "Telnet"
|
||||
ClassFTP TrafficClass = "FTP"
|
||||
ClassFTPS TrafficClass = "FTPS"
|
||||
ClassSMB TrafficClass = "SMB"
|
||||
ClassNFS TrafficClass = "NFS"
|
||||
ClassSIP TrafficClass = "SIP"
|
||||
ClassHTTP3 TrafficClass = "HTTP3" // UDP 443 (QUIC)
|
||||
ClassmDNS TrafficClass = "mDNS" // UDP 5353
|
||||
ClassSSDPDisc TrafficClass = "SSDP" // UDP 1900
|
||||
// ... additional as determined by feature research
|
||||
)
|
||||
```
|
||||
|
||||
**`AllClasses()` ordering:** Grouped by family in the same order as the group bands above. This order is consumed by `config.PrintConfig` for output ordering — so the output will naturally group protocols by family with no additional logic needed in `config`.
|
||||
|
||||
### `synth/config.go` — Add Group Field and Rebalanced ClassFreqConfigs
|
||||
|
||||
**What changes:**
|
||||
|
||||
1. Add `Group string` to `FreqConfig`:
|
||||
|
||||
```go
|
||||
type FreqConfig struct {
|
||||
BaseHz float64
|
||||
Harmonics []HarmonicDef
|
||||
Pan float64
|
||||
Waveform Waveform // NEW: defaults to WaveformSine
|
||||
WaveformType WaveformType
|
||||
Group string // NEW: sound family identifier; "" = ungrouped
|
||||
}
|
||||
```
|
||||
|
||||
`NewLayer` passes `cfg.Waveform` to `NewOscillator`. `NewOscillator` signature changes to accept the waveform.
|
||||
2. Add `ClassFreqConfigs` entries for all new protocols with group-coherent frequencies and within-group detuning.
|
||||
|
||||
**What does NOT change:** `HarmonicDef`, `EMAAlpha`, `Layer.UpdateTarget`, `Layer.AdvanceSample`, `OscillatorBank.RenderWindow`, `mixer.go`, `encode/mp3.go`. The waveform change is contained to `oscillator.go` and the `FreqConfig` struct.
|
||||
3. Update existing class frequencies to align with the group bands (frequency rebalancing). This is the highest-risk change in v1.2 because it alters the default sound of existing classes.
|
||||
|
||||
4. Update `NumLayers` constant from 14 to the new total (e.g., 30-35 depending on final protocol list):
|
||||
|
||||
```go
|
||||
const (
|
||||
NumLayers = 32 // updated count; GainPerLayer recomputed automatically
|
||||
GainPerLayer = 1.0 / float64(NumLayers)
|
||||
)
|
||||
```
|
||||
|
||||
**What does NOT change:** `FreqConfig` field types (only addition), `WaveformType`, `HarmonicDef`, `WaveformPresetHarmonics`, `SampleRate`, `WindowMs`, `SamplesPerWindow`, `WhisperFloor`. The `Group` field is metadata — it has no effect on synthesis math.
|
||||
|
||||
**Pan assignment:** With 30+ layers, the current hand-tuned pan positions become crowded. Recommendation: assign pan by group position (infrastructure = center, web = slight left, mail = slight right, etc.) rather than per-protocol. This creates a spatial "map" of the network that reinforces the group concept aurally.
|
||||
|
||||
### `config/config.go` — PrintConfig Group Ordering
|
||||
|
||||
**What changes:** `PrintConfig` currently emits classes in `AllClasses()` order. With `AllClasses()` updated to group-coherent order, `PrintConfig` output will naturally be grouped. No structural change to `PrintConfig` is required.
|
||||
|
||||
**Optional enhancement:** Emit a group comment header between group sections:
|
||||
|
||||
```toml
|
||||
# --- Infrastructure ---
|
||||
# ICMP -- 65.0 Hz (default)
|
||||
[sounds.ICMP]
|
||||
frequency = 65.0
|
||||
waveform = "sine"
|
||||
|
||||
# NTP -- 110.0 Hz (default)
|
||||
[sounds.NTP]
|
||||
...
|
||||
|
||||
# --- Web ---
|
||||
# HTTP -- 175.0 Hz (default)
|
||||
[sounds.HTTP]
|
||||
...
|
||||
```
|
||||
|
||||
This requires `PrintConfig` to detect group transitions in the class ordering and emit a comment. The `Group` field on `FreqConfig` provides this information. Implementation: iterate `AllClasses()`, track previous group, emit a comment when group changes.
|
||||
|
||||
**What does NOT change:** TOML schema shape (no `group` key in `[sounds.*]` blocks — groups are not user-configurable in v1.2), `SoundOverride` struct, `RawRule`, `LoadResult`, `Load`, `merge`, `addAutoFreqEntries`.
|
||||
|
||||
**Group field in TOML config:** Users should NOT be able to override the `group` of a class. Group is a built-in design decision, not a user parameter. The `SoundOverride` struct stays with only `Frequency` and `Waveform` pointer fields.
|
||||
|
||||
### `synth/bank.go` — No Changes Required
|
||||
|
||||
`NewBank` already ranges over the passed-in `cfgs` map keys. Adding 20 new entries to `ClassFreqConfigs` and passing them through the existing chain requires no changes to `bank.go`. The `gainPerLayer` computation (`1.0 / float64(len(cfgs))`) automatically scales down with more layers.
|
||||
|
||||
### `encode/mp3.go` — No Changes Required
|
||||
|
||||
`RunSynthesis` already accepts the full `freqCfgs` map and passes it to `NewBank`. No signature change needed.
|
||||
|
||||
### `cmd/netsynth/main.go` — No Changes Required
|
||||
|
||||
The pipeline wiring (`config.Load` → `NewClassifier` → `encode.RunSynthesis`) is already correct. Adding more built-in classes and a group field to `FreqConfig` is transparent to `main.go`.
|
||||
|
||||
---
|
||||
|
||||
### Feature 3: User-Defined Classification Rules
|
||||
## Data Flow: Unchanged for v1.2
|
||||
|
||||
**Where rules are wired today:** `main.go` lines 111, 175 — both `runLiveMode` and `runPcapMode` call `classify.NewClassifier(classify.DefaultRules)` directly. No config is passed.
|
||||
The v1.1 data flow is correct and does not need to change:
|
||||
|
||||
**Required change:** User rules from TOML prepend before `DefaultRules`. `Classifier` already supports arbitrary `[]Rule` — `NewClassifier(rules []Rule)` is the constructor. No change to `classifier.go` itself.
|
||||
|
||||
**`RuleConfig` TOML struct:**
|
||||
|
||||
```toml
|
||||
[[rule]]
|
||||
protocol = "tcp"
|
||||
dst_port = 8443
|
||||
class = "myservice"
|
||||
```
|
||||
main.go
|
||||
└─ config.Load(configPath)
|
||||
└─ LoadResult{FreqCfgs, UserRules, ConfigPath, AutoClasses}
|
||||
└─ classify.NewClassifier(append(userRules, DefaultRules...))
|
||||
└─ encode.RunSynthesis(snapshots, path, result.FreqCfgs)
|
||||
└─ synth.NewBank(tau, freqCfgs)
|
||||
└─ one Layer per freqCfgs entry
|
||||
```
|
||||
|
||||
```go
|
||||
// config/config.go
|
||||
type RuleConfig struct {
|
||||
Protocol string `toml:"protocol"`
|
||||
DstPort uint16 `toml:"dst_port"`
|
||||
Class string `toml:"class"` // must match a name in [[class]] or a builtin class name
|
||||
}
|
||||
```
|
||||
The only change to the data flowing through this pipeline is: `freqCfgs` now contains ~30-35 entries (up from 14), each `FreqConfig` has a new `Group` field (ignored by synthesis, used only by `PrintConfig`).
|
||||
|
||||
**Merging in main.go:**
|
||||
---
|
||||
|
||||
```go
|
||||
userRules := config.ToClassifyRules(cfg.Rules) // []classify.Rule
|
||||
allRules := append(userRules, classify.DefaultRules...)
|
||||
classifier := classify.NewClassifier(allRules)
|
||||
```
|
||||
## Integration Points Summary
|
||||
|
||||
**New `TrafficClass` values:** User-defined classes in TOML produce new `TrafficClass` string values (e.g., `"myservice"`). `AllClasses()` in `classify/types.go` is currently a hardcoded slice. For user-defined classes, `AllClasses()` cannot be the source of truth for bank layer construction. `NewBank` must instead iterate over whatever classes have a `FreqConfig` entry.
|
||||
|
||||
This is a critical integration point: `bank.go:NewBank` currently ranges over `classify.AllClasses()`. If user classes can appear, `NewBank` must accept the full config map and range over that instead.
|
||||
| Integration Point | Change Type | Risk |
|
||||
|-------------------|-------------|------|
|
||||
| `classify/types.go` — new constants + `AllClasses()` | Additive | LOW |
|
||||
| `classify/rules.go` — extended `DefaultRules` | Additive (new entries before catch-alls) | MEDIUM (order matters) |
|
||||
| `synth/config.go` — `Group` field on `FreqConfig` | Additive (new field) | LOW (no synthesis impact) |
|
||||
| `synth/config.go` — `ClassFreqConfigs` entries for new protocols | Additive | LOW |
|
||||
| `synth/config.go` — `ClassFreqConfigs` frequency rebalancing for existing protocols | Modifying defaults | HIGH (changes default audio output) |
|
||||
| `synth/config.go` — `NumLayers` constant update | Single constant | MEDIUM (affects `GainPerLayer`) |
|
||||
| `config/config.go` — group-header comment in `PrintConfig` | Optional enhancement | LOW (isolated to string output) |
|
||||
| `synth/bank.go` | No change | — |
|
||||
| `encode/mp3.go` | No change | — |
|
||||
| `cmd/netsynth/main.go` | No change | — |
|
||||
|
||||
---
|
||||
|
||||
@@ -210,167 +334,157 @@ This is a critical integration point: `bank.go:NewBank` currently ranges over `c
|
||||
|
||||
### New
|
||||
|
||||
| Component | Location | Purpose |
|
||||
|-----------|----------|---------|
|
||||
| `config` package | `config/config.go` | TOML struct, `Load()`, file discovery, merge with defaults |
|
||||
| `config/defaults.go` | optional split | `DefaultConfig()` wrapping existing `ClassFreqConfigs` values |
|
||||
None — no new packages or files are required.
|
||||
|
||||
### Modified
|
||||
|
||||
| Component | Change | Impact |
|
||||
|-----------|--------|--------|
|
||||
| `synth/oscillator.go` | Add `Waveform` type + `waveform` field; dispatch in `Advance` | Self-contained; no caller signature breaks except `NewOscillator` |
|
||||
| `synth/config.go` | Add `Waveform Waveform` field to `FreqConfig`; default to `WaveformSine` | Requires `NewLayer` to pass waveform to `NewOscillator` |
|
||||
| `synth/layer.go` | Pass `cfg.Waveform` to `NewOscillator` | One-line change |
|
||||
| `synth/bank.go` | Accept `map[classify.TrafficClass]FreqConfig` param instead of reading global; range over param keys not `AllClasses()` | Decouples bank from global; enables user classes |
|
||||
| `encode/mp3.go` | Accept `*config.Config` or merged `FreqConfig` map; pass to `NewBank` | Thin forwarding change |
|
||||
| `cmd/netsynth/main.go` | Add `--config` flag; load config; prepend user rules; pass config to `RunSynthesis` | Touches both `runLiveMode` and `runPcapMode` |
|
||||
| `classify/rules.go` | No change — `DefaultRules` stays as the fallback | Unchanged |
|
||||
| `classify/classifier.go` | No change — already accepts `[]Rule` | Unchanged |
|
||||
| `classify/types.go` | `AllClasses()` may need a note that it returns only builtins; bank no longer relies on it | Low risk; document only |
|
||||
| Component | What Changes | What Stays Same |
|
||||
|-----------|--------------|-----------------|
|
||||
| `classify/types.go` | New `TrafficClass` constants; `AllClasses()` extended + reordered by group | `ClassifiedPacket`, `WindowSnapshot`, string type |
|
||||
| `classify/rules.go` | New `Rule` entries for all new protocols | `Rule` struct, match algorithm, catch-all placement |
|
||||
| `synth/config.go` | `Group` field on `FreqConfig`; new entries in `ClassFreqConfigs`; existing frequency rebalancing; `NumLayers` updated | All synthesis constants, waveform types, `HarmonicDef` |
|
||||
| `config/config.go` | Optional: group-header comments in `PrintConfig` | All loading, parsing, merge, validation logic |
|
||||
|
||||
---
|
||||
|
||||
## Data Flow Changes
|
||||
## Build Order
|
||||
|
||||
### v1.0 Flow (config hardcoded)
|
||||
The following order ensures each step is independently testable.
|
||||
|
||||
```
|
||||
main.go
|
||||
└─ classify.NewClassifier(classify.DefaultRules)
|
||||
└─ encode.RunSynthesis(snapshots, path)
|
||||
└─ synth.NewBank(1.0)
|
||||
└─ ClassFreqConfigs[class] ← global, hardcoded
|
||||
```
|
||||
### Step 1: Define new TrafficClass constants and extend AllClasses()
|
||||
|
||||
### v1.1 Flow (config injected)
|
||||
**Files:** `classify/types.go` only.
|
||||
|
||||
```
|
||||
main.go
|
||||
└─ config.Load(configPath) ← NEW: resolve path, parse TOML, merge defaults
|
||||
└─ cfg *config.Config
|
||||
└─ classify.NewClassifier(
|
||||
append(config.ToClassifyRules(cfg.Rules), classify.DefaultRules...)
|
||||
) ← user rules prepend built-ins
|
||||
└─ encode.RunSynthesis(snapshots, path, cfg.FreqConfigs())
|
||||
└─ synth.NewBank(1.0, freqConfigs) ← map passed in, not read from global
|
||||
└─ freqConfigs[class] ← merged: user overrides + defaults
|
||||
```
|
||||
Add all new protocol constants. Update `AllClasses()` to include them in group-coherent order. Tests: verify `AllClasses()` returns the expected slice with all new members.
|
||||
|
||||
---
|
||||
No impact on synthesis, config, or main yet — new classes simply have no rules or FreqConfig entries yet, which is harmless.
|
||||
|
||||
## Suggested Build Order
|
||||
### Step 2: Extend DefaultRules for new protocols
|
||||
|
||||
The following order minimizes integration risk. Each step is independently testable before the next begins.
|
||||
**Files:** `classify/rules.go` only.
|
||||
|
||||
### Step 1: Waveform types in `synth/oscillator.go`
|
||||
Add new `Rule` entries before the catch-alls. Tests: `NewClassifier(DefaultRules).Classify(pkt)` for each new protocol's port returns the expected new class. Confirm catch-alls still fire for unrecognized traffic.
|
||||
|
||||
No external dependencies. Pure math. Testable with golden-sample unit tests (square wave sample at phase 0.25 should be 1.0, etc.). Does not affect `Layer`, `Bank`, or `encode` yet.
|
||||
Dependency: Step 1 must complete first (new constants must exist).
|
||||
|
||||
**Files changed:** `synth/oscillator.go` only.
|
||||
### Step 3: Add Group field to FreqConfig
|
||||
|
||||
### Step 2: Wire `Waveform` through `FreqConfig` and `Layer`
|
||||
**Files:** `synth/config.go` only.
|
||||
|
||||
Add `Waveform` to `FreqConfig`. Update `NewLayer` to pass it to `NewOscillator`. `ClassFreqConfigs` entries default to `WaveformSine` (zero value — valid if `WaveformSine = 0`).
|
||||
Add `Group string` to `FreqConfig`. Set `Group` on all existing 14 `ClassFreqConfigs` entries. Existing tests continue to pass (new field has zero value by default; synthesis ignores it). No callers need updating.
|
||||
|
||||
Existing tests continue to pass without modification since all existing configs use the zero-value waveform.
|
||||
This step is independent of Steps 1 and 2.
|
||||
|
||||
**Files changed:** `synth/config.go`, `synth/layer.go`.
|
||||
### Step 4: Add ClassFreqConfigs entries for new protocols (without frequency rebalancing)
|
||||
|
||||
### Step 3: Decouple `NewBank` from the global
|
||||
**Files:** `synth/config.go` only.
|
||||
|
||||
Change `NewBank(tau float64)` to `NewBank(tau float64, cfgs map[classify.TrafficClass]FreqConfig)`. Update `encode/mp3.go:RunSynthesis` to pass `synth.ClassFreqConfigs` as default.
|
||||
Add `FreqConfig` entries for each new protocol using placeholder frequencies that avoid the 65-1100 Hz range (use 2400-4000 Hz temporarily). This makes the end-to-end pipeline work for new classes without disrupting existing audio.
|
||||
|
||||
At this point the system is functionally identical to v1.0 but `NewBank` no longer reads a global.
|
||||
Tests: verify `ClassFreqConfigs` contains entries for all constants from `classify.AllClasses()`.
|
||||
|
||||
**Files changed:** `synth/bank.go`, `encode/mp3.go`.
|
||||
Dependency: Step 3 must complete first (Group field must exist). Step 1 must complete first (constants must exist).
|
||||
|
||||
### Step 4: `config` package — TOML structs and file discovery
|
||||
### Step 5: Frequency rebalancing — reassign all ClassFreqConfigs to group-coherent values
|
||||
|
||||
Implement `config.Load()`, file discovery, and the `DefaultConfig()` function that wraps `synth.ClassFreqConfigs`. No TOML parsing yet — start with the struct definitions and the merge logic.
|
||||
**Files:** `synth/config.go` only.
|
||||
|
||||
Add `github.com/BurntSushi/toml` dependency (`go get`).
|
||||
This is the riskiest step. Assign final group-coherent frequencies to all protocols (existing and new) following the band design above. Update `NumLayers` to the final count.
|
||||
|
||||
**Files added:** `config/config.go`.
|
||||
Test strategy: write a test that verifies all entries in `ClassFreqConfigs` have `BaseHz` in a specified range per group; verify no two entries have identical `BaseHz`; verify `NumLayers` matches `len(ClassFreqConfigs)`.
|
||||
|
||||
### Step 5: `--config` flag and user rule merging in `main.go`
|
||||
Manual validation: run `netsynth --read testfile.pcap` with varied traffic and listen. This step requires subjective audio evaluation that tests cannot replace.
|
||||
|
||||
Wire `config.Load()` into `run()`. Pass user rules to both `runLiveMode` and `runPcapMode`. Pass merged `FreqConfig` map to `RunSynthesis`.
|
||||
Dependency: Steps 3 and 4 must complete first.
|
||||
|
||||
At this point a minimal TOML config (empty file, or `[[rule]]` only) can be validated end-to-end.
|
||||
### Step 6: PrintConfig group-header comments (optional polish)
|
||||
|
||||
**Files changed:** `cmd/netsynth/main.go`.
|
||||
**Files:** `config/config.go` only.
|
||||
|
||||
### Step 6: Custom frequency and waveform overrides in config
|
||||
Add group transition detection in `PrintConfig`. When the `Group` field changes between consecutive `AllClasses()` entries, emit a `# --- GroupName ---` comment.
|
||||
|
||||
Implement the `[[class]]` TOML section parsing. Add `FreqConfigs()` method to `Config` that returns the merged map (user overrides applied over defaults). Write table-driven tests: "TOML sets HTTPS to 200 Hz sawtooth, bank layer for HTTPS uses 200 Hz sawtooth."
|
||||
Tests: verify `PrintConfig` output contains group header comments; verify ungrouped classes have no group header; verify user-defined classes (no group field) are not preceded by a group header.
|
||||
|
||||
**Files changed:** `config/config.go`.
|
||||
|
||||
### Step 7: User-defined classes end-to-end
|
||||
|
||||
Support `[[class]]` entries with names not in `classify.AllClasses()`. These become new `TrafficClass` values. User `[[rule]]` entries pointing to these classes are prepended to `DefaultRules`. The bank creates layers for all classes in the merged `FreqConfig` map.
|
||||
|
||||
This step requires the most cross-package coordination but by this point each piece is already in place.
|
||||
|
||||
**Files changed:** `config/config.go`, `cmd/netsynth/main.go` (verification that unknown class names don't panic).
|
||||
|
||||
---
|
||||
|
||||
## Component Boundaries After v1.1
|
||||
|
||||
| Component | Responsibility | Communicates With |
|
||||
|-----------|---------------|-------------------|
|
||||
| `config` | TOML parsing, file discovery, merge logic, `DefaultConfig()` | `synth` (FreqConfig type), `classify` (Rule type) |
|
||||
| `synth/oscillator` | Phase-accumulator for sine/square/sawtooth/triangle | Used by `Layer` |
|
||||
| `synth/bank` | Accepts freq config map, constructs one `Layer` per entry | `encode` passes config map in |
|
||||
| `encode` | Receives config map from `main`, passes to `NewBank` | Thin pass-through |
|
||||
| `cmd/netsynth/main` | Loads config, merges rules, wires all stages | All packages |
|
||||
| `classify` | Rules engine (unchanged); `DefaultRules` stays as package-level var | `main` constructs with merged rules |
|
||||
Dependency: Steps 3 and 5 must complete first (Group field must be populated in ClassFreqConfigs).
|
||||
|
||||
---
|
||||
|
||||
## Critical Integration Constraints
|
||||
|
||||
### `AllClasses()` Is Not the Source of Truth for Bank Construction
|
||||
### NumLayers Must Match len(ClassFreqConfigs) for Correct Gain Scaling
|
||||
|
||||
`bank.go` currently iterates `classify.AllClasses()` to construct layers. After v1.1, the bank must iterate the keys of the `FreqConfig` map passed to it. User-defined classes will not appear in `AllClasses()`. If this is not changed, user-defined class packets will be aggregated in `WindowSnapshot.Counts` but have no corresponding layer — they will produce silence and no error.
|
||||
`synth/config.go` defines `NumLayers = 14` as a constant. `GainPerLayer = 1.0 / float64(NumLayers)`. However, `bank.go:NewBank` computes `gainPerLayer` dynamically from `len(cfgs)`:
|
||||
|
||||
**Fix:** `NewBank` iterates `maps.Keys(cfgs)` (or equivalent range over the map), not `classify.AllClasses()`.
|
||||
```go
|
||||
gainPerLayer: 1.0 / float64(len(cfgs)),
|
||||
```
|
||||
|
||||
### Class Name Validation Must Happen at Config Load Time
|
||||
This means `NumLayers` and `GainPerLayer` in `config.go` are currently informational constants, not what the bank actually uses at runtime. The bank's dynamic computation is correct. Update `NumLayers` for documentation accuracy, but the audio math is not at risk even if this is temporarily inconsistent.
|
||||
|
||||
If a `[[rule]]` references a class name that has no corresponding `[[class]]` entry and is not a builtin, the system will silently mis-classify packets into a layer that doesn't exist. Validate at `config.Load()` time: every class name in `[[rule]]` must resolve to either a builtin `TrafficClass` or a `[[class]]` entry in the same config.
|
||||
### DefaultRules Catch-Alls Must Remain Last
|
||||
|
||||
### `encode.RunSynthesis` Signature Change Is a Breaking API Change
|
||||
`classify/rules.go` ends with `{Protocol: "tcp", DstPort: 0, ...}` and `{Protocol: "udp", DstPort: 0, ...}`. All new protocol rules must be inserted before these. A rule added after the catch-alls will never fire. This constraint must be enforced by convention (comment in the file) since Go has no static ordering guarantees for slice initialization beyond the literal order.
|
||||
|
||||
`encode.RunSynthesis(snapshots []classify.WindowSnapshot, outputPath string)` will need to accept the config. If any external code (tests, future callers) uses this signature, they will break. Keep the change to a single place and update all call sites in the same commit.
|
||||
### AllClasses() Ordering Drives PrintConfig Ordering
|
||||
|
||||
`config.PrintConfig` iterates `classify.AllClasses()` to emit the `[sounds.*]` section. If `AllClasses()` is updated to group-coherent order (Step 1), `PrintConfig` output naturally reflects groups. If Step 6's group-header comments are added, they depend on `AllClasses()` presenting entries in group-sorted order. The two changes must be coordinated.
|
||||
|
||||
### Frequency Rebalancing Is a Default-Breaking Change
|
||||
|
||||
Users who rely on the default audio profile (no config file) will hear different tones for previously-familiar classes. This is an intentional design improvement but should be documented in the release notes. Users who have overridden frequencies via TOML config are unaffected (their overrides take precedence).
|
||||
|
||||
### Group Field Is Not a TOML Configurable
|
||||
|
||||
The `SoundOverride` struct in `config/config.go` must not gain a `group` field. Groups are built-in design decisions. Allowing users to reassign protocols to different groups via TOML would create inconsistencies in `PrintConfig` output ordering and the `AllClasses()` group semantics. If this is requested in a future milestone, it warrants a separate design decision.
|
||||
|
||||
---
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
### Anti-Pattern: Reading Global `ClassFreqConfigs` from Multiple Places
|
||||
### Anti-Pattern: Computing Detuning at Runtime from Group Metadata
|
||||
|
||||
If `NewBank`, `encode.RunSynthesis`, and config loading all reference the package-level `synth.ClassFreqConfigs`, the merge point becomes ambiguous. The fix (Step 3 above) centralizes config reading to one place: `config.DefaultConfig()` reads from `ClassFreqConfigs` once when building defaults; everything downstream receives the already-merged map.
|
||||
It is tempting to define group base frequencies and detune offsets as data structures, then derive individual `BaseHz` values at synthesis startup. This adds complexity with no benefit — the final `BaseHz` values are static per-class constants that belong in `ClassFreqConfigs` directly. Bake the detuning arithmetic into the initial constant assignment, document it with a comment (`// Mail group: +10 Hz from group base 420 Hz`), and move on.
|
||||
|
||||
### Anti-Pattern: Storing `Waveform` as a String Everywhere
|
||||
### Anti-Pattern: Group as a `classify.TrafficClass` Field
|
||||
|
||||
Keeping `waveform` as a `string` from TOML all the way into `Oscillator` means every advance call parses or switches on a string. Parse the string to a `Waveform` int type at config-load time. The `Oscillator` field should be a typed `Waveform`, not `string`.
|
||||
Groups are audio concepts, not classification concepts. `classify.Rule` should not know about groups. `TrafficClass` is already a string type used as a map key throughout the codebase — adding a structured type would be a larger refactor than the feature warrants. Keep groups in `synth.FreqConfig`.
|
||||
|
||||
### Anti-Pattern: User Rules Appended After DefaultRules
|
||||
### Anti-Pattern: Adding Group-Based Routing to bank.go
|
||||
|
||||
User rules must **prepend** `DefaultRules`, not append. `DefaultRules` ends with catch-all rules (`DstPort: 0`) that match any TCP or UDP packet. Appending user rules after these catch-alls means they will never be reached.
|
||||
Do not make `OscillatorBank` aware of groups for mixing purposes (e.g., summing all Mail group layers before applying gain). The current architecture mixes all layers uniformly. Group-coherent mixing is a more complex feature than v1.2 requires and changes the synthesis model. Defer if requested.
|
||||
|
||||
### Anti-Pattern: Overcrowding 65-1100 Hz with Too Many Protocols
|
||||
|
||||
At 30+ built-in classes, some protocols will be heard less often than others. Resist the urge to map rarely-seen protocols (Telnet, FTP) to prominent mid-range frequencies. Relegate low-signal protocols to the upper end of each group band so they contribute texture without dominating when not present.
|
||||
|
||||
### Anti-Pattern: Hand-Tuning Frequencies Without Listening
|
||||
|
||||
Frequency assignments must be validated by ear, not just by looking at Hz values on a spreadsheet. Schedule a listening session with a realistic pcap file after Step 5 before Step 6. The architecture makes this easy — any frequency change is a one-line edit in `ClassFreqConfigs`.
|
||||
|
||||
---
|
||||
|
||||
## Scalability Considerations
|
||||
|
||||
| Concern | With 14 Classes (v1.1) | With 32 Classes (v1.2) |
|
||||
|---------|----------------------|----------------------|
|
||||
| Gain per layer | ~7.1% of max | ~3.1% of max |
|
||||
| Frequency collisions | None | Possible if not planned |
|
||||
| AllClasses() iteration | 14 items, trivial | 32 items, still trivial |
|
||||
| PrintConfig output lines | ~60 lines | ~130 lines |
|
||||
| EMA smoothing convergence | No change | No change (per-layer, independent) |
|
||||
| NewBank construction time | Negligible | Negligible |
|
||||
|
||||
The main practical change is perceptual loudness: with 32 layers each at 3.1% gain, the ambient mix becomes quieter when many protocols are active simultaneously. This is acceptable and matches the ambient/drone aesthetic. The `WhisperFloor` mechanism ensures inactive layers contribute minimally, so sessions with only 5-6 active protocols still sound full.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- Direct code inspection: `synth/config.go`, `synth/oscillator.go`, `synth/bank.go`, `synth/layer.go`, `classify/classifier.go`, `classify/rules.go`, `classify/types.go`, `encode/mp3.go`, `cmd/netsynth/main.go` — HIGH confidence
|
||||
- BurntSushi/toml usage in production Go projects (Hugo, dep): MEDIUM confidence (well-known in Go ecosystem)
|
||||
- Phase-accumulator waveform synthesis formulas (square, sawtooth, triangle): HIGH confidence (standard DSP, textbook formulas)
|
||||
- Direct code inspection: `synth/config.go`, `synth/bank.go`, `synth/layer.go`, `synth/oscillator.go`, `classify/types.go`, `classify/rules.go`, `classify/classifier.go`, `config/config.go`, `encode/mp3.go`, `cmd/netsynth/main.go` — HIGH confidence
|
||||
- Musical interval theory (detuning, harmonic relationships): HIGH confidence — standard acoustic physics
|
||||
- v1.2 protocol list: determined from feature research (see FEATURES.md for rationale on which protocols to include)
|
||||
|
||||
---
|
||||
|
||||
*Architecture research for: NetSynth v1.1 — custom sound mappings integration*
|
||||
*Researched: 2026-03-26*
|
||||
*Architecture research for: NetSynth v1.2 — extended protocol coverage with grouped families*
|
||||
*Researched: 2026-03-27*
|
||||
|
||||
+292
-205
@@ -1,279 +1,366 @@
|
||||
# Feature Research
|
||||
|
||||
**Domain:** Network traffic sonification CLI tool (packet capture -> ambient MP3)
|
||||
**Researched:** 2026-03-24 (v1.0), updated 2026-03-26 (v1.1 custom sound mappings)
|
||||
**Researched:** 2026-03-24 (v1.0), updated 2026-03-26 (v1.1 custom sound mappings), updated 2026-03-27 (v1.2 extended protocol coverage)
|
||||
**Confidence:** MEDIUM — niche domain; comparable tools are research prototypes or GUI applications, not CLI tools. Table stakes are derived from tcpdump/packet-capture CLI conventions and sonification research literature.
|
||||
|
||||
---
|
||||
|
||||
## v1.1 Feature Research: Custom Sound Mappings via TOML Config
|
||||
## v1.2 Feature Research: Extended Protocol Coverage with Grouped Protocol Families
|
||||
|
||||
This section addresses the milestone question: "How do custom sound mapping config files typically work in audio/network tools? What are expected behaviors for config file loading, merging with defaults, validation, and error reporting?"
|
||||
This section addresses the milestone question: "How should protocol grouping work for network sonification? What are the most common protocols seen on typical networks? Which protocols are worth adding? What's the right granularity — individual protocol vs group?"
|
||||
|
||||
### Config File Loading: Standard Behaviors Expected by CLI Users
|
||||
### What Real Networks Actually See
|
||||
|
||||
Based on patterns from established CLI tools (git, golangci-lint, mise, hugo), users expect:
|
||||
Based on IANA well-known ports, nDPI's traffic classification taxonomy (450+ protocols, 17 application categories), Wireshark's protocol dissector set, and widely-cited network security references, traffic on real networks breaks down into recognizable families.
|
||||
|
||||
1. **Auto-discovery with a defined search order.** The tool looks in a conventional set of locations without requiring an explicit flag. Failing silently (no config found = run with defaults) is correct behavior.
|
||||
**Home network (residential broadband):**
|
||||
The dominant traffic types are HTTPS (streaming, browsing, cloud sync), DNS (constant background noise, every connection starts here), NTP (infrequent but present on all devices), DHCP (device join/renew events), ICMP (ping, router probe), and mDNS/SSDP (device discovery on the local segment). IoT device traffic adds MQTT. Video calls add RTP/SRTP.
|
||||
|
||||
2. **Explicit override via a flag.** `--config` (or `-c`) lets users point at a non-standard path. If `--config` is supplied and the file does not exist, that is an error — not silent fallback.
|
||||
**Office network (enterprise LAN):**
|
||||
Same HTTPS/DNS/NTP/DHCP base, plus: LDAP/Kerberos (domain auth), SMB (file sharing), RDP (remote desktop), SMTP/IMAP/POP3 (mail), SNMP (monitoring), syslog (log aggregation), SSH (server access), FTP (legacy file transfer still common in many environments).
|
||||
|
||||
3. **Discovery search order (standard precedence):**
|
||||
- `--config path/to/file.toml` (explicit flag, highest priority)
|
||||
- `./netsynth.toml` (working directory, project-local)
|
||||
- `$XDG_CONFIG_HOME/netsynth/config.toml` (defaults to `~/.config/netsynth/config.toml`)
|
||||
- No config found → run with all defaults (not an error)
|
||||
**Server host (Linux box exposed to internet):**
|
||||
SSH (constant scan attempts), HTTPS (serving), DNS (resolver queries), ICMP (reachability probes), NTP (drift correction), syslog (local log collection), PostgreSQL/MySQL/Redis (local app traffic), SMTP (outbound mail relay).
|
||||
|
||||
This is the pattern used by git (`.git/config` -> `~/.gitconfig` -> `/etc/gitconfig`), golangci-lint (`.golangci.yml` in working dir), and mise (`mise.toml` -> `~/.config/mise/config.toml`). **HIGH confidence** — XDG Base Directory Specification is the Linux/macOS standard.
|
||||
**Cloud workload:**
|
||||
HTTPS dominates. DNS for service discovery. Redis/PostgreSQL for app data. Kafka/AMQP for message queues. gRPC (still port 443 via HTTP/2). ICMP suppressed. NTP locked down.
|
||||
|
||||
4. **Partial overrides only — not a replacement config.** The config file expresses only what the user wants to change. Absent keys retain default values. This is universally expected: users do not want to replicate the full default table in order to change one frequency.
|
||||
### Why Protocol Grouping Matters for Sonification
|
||||
|
||||
### Config File Merging: How Defaults and User Config Combine
|
||||
Without grouping, adding 20+ individual protocols creates:
|
||||
1. **Spectrum crowding:** 20+ tones across the audible range become indistinguishable mud.
|
||||
2. **No perceptual structure:** Listeners cannot form a mental model of what they are hearing.
|
||||
3. **Frequency allocation complexity:** Designing 20+ non-conflicting frequency slots is difficult.
|
||||
|
||||
The dominant pattern across well-designed CLI tools:
|
||||
The nDPI project (the leading open-source DPI library, used by ntopng) faced the same problem and solved it with 17 application categories that group hundreds of protocols. For sonification, the goal is different — not to identify every protocol, but to produce a soundscape where related activity sounds related.
|
||||
|
||||
**Merge strategy: user values override defaults, defaults fill gaps.**
|
||||
**Recommended approach: Protocol families, not individual protocols.**
|
||||
|
||||
```
|
||||
builtin defaults <-- loaded first (in-code, always present)
|
||||
+
|
||||
user config file <-- loaded second (overrides per-key)
|
||||
=
|
||||
effective config <-- what the program runs with
|
||||
```
|
||||
A "Mail" family chord — one tonal cluster covering SMTP + IMAP + POP3 — is more musically coherent and more perceptually useful than three separate isolated tones. A user can hear "mail activity is elevated" rather than "something is happening on port 143."
|
||||
|
||||
For NetSynth's classification rules specifically, there are two distinct semantics that must be clearly chosen:
|
||||
### Protocol Grouping Taxonomy for NetSynth v1.2
|
||||
|
||||
- **Override by name:** User supplies a `[rule.DNS]` block that replaces the built-in DNS sound parameters. The predefined DNS rule's classification logic is kept; only its sound output changes.
|
||||
- **Prepend user rules:** User-defined rules are inserted before the built-in rule list, allowing them to match first (first-match-wins). This enables the user to add entirely new protocol-to-sound mappings.
|
||||
After surveying IANA port assignments, nDPI's category taxonomy, and the Wireshark protocol support matrix, the following family groupings are recommended. Each family occupies a frequency region (not individual tones), and protocols within a family use slight detuning or waveform variation to stay distinct.
|
||||
|
||||
Both are needed. They serve different use cases:
|
||||
- Sound overrides (change frequency/waveform for a known protocol) use the override-by-name pattern.
|
||||
- Custom traffic rules (classify "tcp port 8443 as MyApp") use prepend semantics.
|
||||
**Family: Web (already covered)**
|
||||
- HTTPS / TLS (port 443) — already built in
|
||||
- HTTP (port 80) — already built in
|
||||
- HTTP/3 / QUIC (port 443 UDP) — port-based detection is possible (UDP 443)
|
||||
- Alt-HTTPS ports (8443, 8080) — users can add via custom rules
|
||||
|
||||
### Validation: What Users Expect When Config Has Errors
|
||||
**Family: Mail**
|
||||
- SMTP (port 25) — already built in (single class)
|
||||
- SMTP submission (port 587, 465) — distinct submission path, worth adding
|
||||
- IMAP (port 143, 993) — pull email; very common on office networks
|
||||
- POP3 (port 110, 995) — legacy pull email; still common with older mail clients
|
||||
|
||||
Based on patterns in go-toml v2's strict mode and golangci-lint error reporting:
|
||||
**Family: Remote Access**
|
||||
- SSH (port 22) — already built in
|
||||
- Telnet (port 23) — unencrypted, legacy; worth representing (security signal)
|
||||
- RDP (port 3389) — Windows remote desktop; ubiquitous in enterprise
|
||||
- VNC (port 5900) — remote frame buffer; common on servers/developer machines
|
||||
|
||||
**Expected validation behaviors (roughly in order of importance):**
|
||||
**Family: Infrastructure (already partially covered)**
|
||||
- DNS (port 53) — already built in
|
||||
- DHCP (ports 67/68) — already built in
|
||||
- NTP (port 123) — already built in
|
||||
- mDNS (port 5353 UDP) — Bonjour/Avahi discovery; prominent on home/office LANs
|
||||
- SSDP (port 1900 UDP) — UPnP device discovery; common with IoT devices
|
||||
- LLMNR (port 5355) — Windows local name resolution; common on Windows networks
|
||||
- SNMP (port 161/162 UDP) — network monitoring; common on office/server networks
|
||||
- Syslog (port 514 UDP) — log forwarding; common on server and enterprise networks
|
||||
|
||||
| Behavior | Why Expected | Go Implementation Note |
|
||||
|----------|--------------|----------------------|
|
||||
| Unknown keys caught and reported | Prevents silent typos (user writes `frequncy`, expects it to work) | `go-toml/v2` `DisallowUnknownFields()` or BurntSushi's `Undecoded()` check |
|
||||
| Line number in error message | Users need to know where the problem is | Both go-toml/v2 DecodeError and BurntSushi include position info |
|
||||
| Human-readable field path | "invalid value for `rules[0].waveform`" not "decode error" | go-toml v2's `DecodeError` produces contextualized messages |
|
||||
| Invalid enum values rejected | `waveform = "sqaure"` (typo) should list valid options | Post-decode validation loop with explicit error message listing valid values |
|
||||
| Out-of-range numbers rejected | `frequency = -50` or `frequency = 25000` should fail with reason | Post-decode bounds check with message |
|
||||
| Missing required fields in new rules | A user rule block missing `protocol` is ambiguous | Post-decode presence check |
|
||||
| Config error prevents startup | Do not silently ignore errors and run with partial config | Error should exit with non-zero and print the problem before capturing any packets |
|
||||
**Family: File Transfer**
|
||||
- FTP (port 21) — still widely used in legacy environments, NAS devices
|
||||
- TFTP (port 69 UDP) — boot/config transfer; common in network infrastructure (switches, PXE boot)
|
||||
- SMB (port 445) — Windows file sharing; ubiquitous on any Windows or Samba network
|
||||
|
||||
**Critical:** Validation errors must surface before capture begins. A user who runs the tool, captures for 30 minutes, then gets a corrupt MP3 because a config value was silently ignored would rightly be frustrated.
|
||||
**Family: Database**
|
||||
- MySQL/MariaDB (port 3306) — most common SQL database port
|
||||
- PostgreSQL (port 5432) — second most common SQL database
|
||||
- Redis (port 6379) — in-memory cache; present on almost every modern app server
|
||||
- MongoDB (port 27017) — document store; very common in web apps
|
||||
|
||||
### Error Reporting: Standard UX Patterns
|
||||
**Family: VoIP / Real-Time**
|
||||
- SIP (port 5060/5061) — VoIP signaling; present in any office with IP phones
|
||||
- RTP (ports 16384-32767 UDP, dynamic) — voice/video payload; hard to detect by port alone
|
||||
|
||||
From studying tools in the same class (golangci-lint, hugo, suricata):
|
||||
**Family: Directory / Authentication**
|
||||
- LDAP (port 389, 636) — Active Directory / OpenLDAP; present on any enterprise network
|
||||
- Kerberos (port 88) — Active Directory authentication; present on any Windows domain network
|
||||
|
||||
- Print config errors to **stderr** (not stdout).
|
||||
- Prefix with the config file path: `netsynth.toml:12: unknown field "frequncy"`.
|
||||
- List ALL errors found in one pass rather than stopping at the first error. Users prefer fixing 5 things in one edit over 5 sequential runs.
|
||||
- Warn (not error) for non-fatal issues such as "config file found but empty" or "unknown field in a comment-like position" — but for NetSynth's scope, unknown keys should be hard errors to prevent silent misconfigurations.
|
||||
- On `--config path` flag with missing file: hard error immediately.
|
||||
- On auto-discovered config with missing file: silent success (no config = defaults).
|
||||
Note: gopacket/layers does NOT natively decode SMTP, IMAP, POP3, FTP, TFTP, SMB, MySQL, PostgreSQL, Redis, SIP, RTP, LDAP, or Kerberos at the application layer. Classification must happen at the transport layer (port number matching), which is exactly how the existing Rule classifier works. No new dependencies required.
|
||||
|
||||
### Which Protocols to Include: Priority Tiers
|
||||
|
||||
**Tier 1 — Add immediately (high real-world frequency, low complexity)**
|
||||
|
||||
| Protocol | Ports | Network Type | Detection Method | Family |
|
||||
|----------|-------|-------------|-----------------|--------|
|
||||
| IMAP / IMAPS | TCP 143, 993 | Home, Office, Server | Port match | Mail |
|
||||
| POP3 / POP3S | TCP 110, 995 | Home, Office | Port match | Mail |
|
||||
| SMTP Submission | TCP 587, 465 | Home, Office, Server | Port match | Mail |
|
||||
| FTP | TCP 20, 21 | Office, Server, NAS | Port match | File Transfer |
|
||||
| SMB | TCP 445 | Office, Windows networks | Port match | File Transfer |
|
||||
| RDP | TCP 3389 | Office, Enterprise | Port match | Remote Access |
|
||||
| mDNS | UDP 5353 | Home, Office | Port match | Infrastructure |
|
||||
| SSDP | UDP 1900 | Home, IoT | Port match | Infrastructure |
|
||||
| SNMP | UDP 161, 162 | Office, Server | Port match | Infrastructure |
|
||||
| MySQL | TCP 3306 | Server, Cloud | Port match | Database |
|
||||
| PostgreSQL | TCP 5432 | Server, Cloud | Port match | Database |
|
||||
| Redis | TCP 6379 | Server, Cloud | Port match | Database |
|
||||
|
||||
**Tier 2 — Include if groups are being formed (moderate frequency)**
|
||||
|
||||
| Protocol | Ports | Network Type | Detection Method | Family |
|
||||
|----------|-------|-------------|-----------------|--------|
|
||||
| Telnet | TCP 23 | Legacy, embedded | Port match | Remote Access |
|
||||
| VNC | TCP 5900 | Office, Developer | Port match | Remote Access |
|
||||
| TFTP | UDP 69 | Infrastructure, PXE | Port match | File Transfer |
|
||||
| SIP | TCP/UDP 5060, 5061 | Office, VoIP | Port match | VoIP |
|
||||
| LDAP / LDAPS | TCP 389, 636 | Enterprise | Port match | Directory/Auth |
|
||||
| Kerberos | UDP/TCP 88 | Enterprise, Windows | Port match | Directory/Auth |
|
||||
| Syslog | UDP 514 | Server, Enterprise | Port match | Infrastructure |
|
||||
| MongoDB | TCP 27017 | Server, Cloud | Port match | Database |
|
||||
| QUIC/HTTP3 | UDP 443 | Home, Office | Port match (UDP 443) | Web |
|
||||
|
||||
**Tier 3 — Defer or leave to user custom rules**
|
||||
|
||||
| Protocol | Reason to Defer |
|
||||
|----------|----------------|
|
||||
| MQTT (TCP 1883/8883) | IoT-specific; not present on most general networks |
|
||||
| AMQP (TCP 5672) | Message queue; only relevant on specific server workloads |
|
||||
| Kafka (TCP 9092) | Rarely seen outside distributed systems environments |
|
||||
| BGP (TCP 179) | Routing protocol; not visible on end-host captures |
|
||||
| OSPF | Link-state routing; IP protocol 89, not TCP/UDP; gopacket has native decoder but it is infrastructure traffic only |
|
||||
| RADIUS (UDP 1812/1813) | Authentication forwarding; only on network infrastructure |
|
||||
| gRPC | Uses HTTP/2 on port 443; indistinguishable from HTTPS at transport layer |
|
||||
| XMPP (TCP 5222) | Near-obsolete for general messaging |
|
||||
|
||||
### What Granularity Is Right: Individual Protocol vs Group?
|
||||
|
||||
**Recommendation: Implement individual protocol classes, but group them for frequency allocation.**
|
||||
|
||||
This gives the user the most information value while maintaining musical coherence:
|
||||
- Each protocol gets a distinct `TrafficClass` constant and rule (e.g., `ClassIMAP`, `ClassSMB`).
|
||||
- Related protocols are assigned frequencies within a shared family frequency band (e.g., all Mail protocols live in 400-550 Hz).
|
||||
- Within a family band, slight detuning (5-15 Hz apart) produces audible distinction without crowding.
|
||||
- Family identity is perceivable because the tones are harmonically close.
|
||||
|
||||
This matches how nDPI handles the tension: individual protocol identity for precise classification, category grouping for user-facing display and policy. NetSynth's "display" is sonic — the grouping manifests as tonal proximity.
|
||||
|
||||
The alternative — collapsing IMAP+SMTP+POP3 into a single "Mail" class — loses information. A user cannot tell whether mail noise is inbound (IMAP) or outbound (SMTP). Individual classes preserve that distinction.
|
||||
|
||||
### Frequency Allocation for New Families
|
||||
|
||||
The current frequency map uses 65 Hz – 1047 Hz across 14 classes. Adding ~20 new protocols requires rethinking the allocation.
|
||||
|
||||
**Current allocation issues:**
|
||||
- The range from 65-110 Hz (ICMP, DNS) is very low; adding families in this region creates muddiness.
|
||||
- The unknown buckets at 862-1047 Hz occupy space that could be used for real protocols.
|
||||
- The spread from 600-780 Hz (DHCP, OtherTCP, OtherUDP) is dense.
|
||||
|
||||
**Recommended approach for v1.2:**
|
||||
- Assign families to distinct octave/register bands rather than a linear frequency sweep.
|
||||
- Use musically meaningful intervals within each family (minor thirds, perfect fourths — intervals that sound related without clashing).
|
||||
- Keep the existing 10 class frequencies backward-compatible; assign new protocols to new frequency slots.
|
||||
- Push unknown buckets above 1200 Hz (the current auto-assign range already does this via FNV hash).
|
||||
|
||||
**Proposed family frequency bands:**
|
||||
|
||||
| Family | Band | Rationale |
|
||||
|--------|------|-----------|
|
||||
| Infrastructure (DNS, DHCP, NTP, mDNS, SNMP, Syslog, SSDP) | 80-200 Hz | Low, grounding tones; infrastructure is the "bass" of the network |
|
||||
| Web (HTTP, HTTPS, QUIC) | 220-320 Hz | Mid-low; dominant traffic, warm register |
|
||||
| Mail (SMTP, IMAP, POP3) | 350-480 Hz | Mid; distinct from web, harmonically separate |
|
||||
| Remote Access (SSH, Telnet, RDP, VNC) | 500-620 Hz | Mid-high; noticeable — admin activity is important |
|
||||
| File Transfer (FTP, TFTP, SMB) | 650-750 Hz | Upper-mid; distinct texture |
|
||||
| Database (MySQL, PostgreSQL, Redis, MongoDB) | 800-950 Hz | Upper register; database chatter is server-side signal |
|
||||
| Directory / Auth (LDAP, Kerberos) | 960-1050 Hz | High; auth traffic is sparse but significant |
|
||||
| VoIP (SIP, RTP) | 1100-1200 Hz | High; real-time traffic stands out |
|
||||
| Unknown buckets | 1300+ Hz | Highest; unclassified traffic is "noise above the signal" |
|
||||
| ICMP | 65 Hz | Remains as fundamental ping pulse below all families |
|
||||
|
||||
---
|
||||
|
||||
## Table Stakes for v1.1
|
||||
## Table Stakes for v1.2
|
||||
|
||||
Features users expect in any CLI tool that introduces a config file. Missing these makes v1.1 feel incomplete.
|
||||
Features required to call v1.2 complete. Missing these means the milestone goal ("expanded protocol classification with grouped protocol families") is not delivered.
|
||||
|
||||
| Feature | Why Expected | Complexity | Depends On |
|
||||
|---------|--------------|------------|------------|
|
||||
| TOML config file auto-discovery (`./netsynth.toml`, `~/.config/netsynth/config.toml`) | Standard CLI convention; users expect zero-flag discovery | LOW | New: config loader module |
|
||||
| `--config` flag for explicit path | Required when multiple configs exist or working dir is wrong | LOW | New: config loader + cobra flag |
|
||||
| Partial override semantics (absent keys retain defaults) | Users must not copy the entire default table to change one field | LOW | New: merge logic |
|
||||
| Custom frequency per known traffic class | Core v1.1 ask; directly maps to `synth.FreqConfig.BaseHz` | LOW | Existing `synth.ClassFreqConfigs` |
|
||||
| Custom waveform per known traffic class | Core v1.1 ask; maps to `synth.Oscillator.Advance()` harmonic shape | MEDIUM | Existing oscillator (needs waveform type support) |
|
||||
| User-defined classification rules with custom sounds | Core v1.1 ask; prepend to `classify.DefaultRules` | MEDIUM | Existing `classify.Rule` struct (needs `Class` name generation) |
|
||||
| Config validation with line-number errors | Users cannot fix config errors without location info | LOW | go-toml v2 DecodeError (built-in) |
|
||||
| Unknown field detection | Prevents silent typos | LOW | go-toml v2 `DisallowUnknownFields()` |
|
||||
| Startup-time validation (fail before capture) | No wasted captures with bad config | LOW | Load config in `cmd` root before starting capture |
|
||||
| Clear error message listing valid enum values | `waveform` has exactly 4 valid values; list them on error | LOW | Post-decode validation |
|
||||
| Mail family (IMAP, IMAPS, POP3, POP3S, SMTP submission 587/465) | Requested directly in todo; mail traffic is high-frequency on any office network | LOW | Existing Rule/TrafficClass pattern; add constants + rules |
|
||||
| Remote Access expansion (RDP, VNC, Telnet) | SSH is already present; the family is incomplete without RDP on enterprise captures | LOW | Same as above |
|
||||
| File Transfer family (FTP, SMB, TFTP) | FTP/SMB appear on almost every office or NAS-connected home network | LOW | Same as above |
|
||||
| Infrastructure expansion (mDNS, SSDP, SNMP, Syslog) | These are constant background noise on every LAN; without them they land in other-UDP | LOW | Same as above |
|
||||
| Database family (MySQL, PostgreSQL, Redis, MongoDB) | Any developer machine has these; they currently all land in other-TCP | LOW | Same as above |
|
||||
| Frequency rebalancing to accommodate new classes | Without rebalancing, the new classes crowd the existing spectrum | MEDIUM | requires touching synth.ClassFreqConfigs; backward-compatible if existing class constants keep their names |
|
||||
| TrafficClass constants and AllClasses() updated | Config, synth, and print-config must know about new classes | LOW | classify/types.go extension |
|
||||
| DefaultRules updated with new port rules | New classes only work if packets reach them via rules | LOW | classify/rules.go extension |
|
||||
| --print-config reflects new classes | Users need to see and override the new classes | LOW | Falls out automatically once ClassFreqConfigs and AllClasses() are updated |
|
||||
| Group concept exposed in --print-config | Grouped comments (# Mail family, # Database family) make the config readable | LOW | PrintConfig formatting only; no struct changes needed |
|
||||
|
||||
## Differentiators for v1.1
|
||||
## Differentiators for v1.2
|
||||
|
||||
Features that make the config experience polished beyond the minimum.
|
||||
Features that make the extended protocol coverage polished beyond the minimum.
|
||||
|
||||
| Feature | Value Proposition | Complexity | Notes |
|
||||
|---------|-------------------|------------|-------|
|
||||
| `netsynth --print-config` command to dump effective config as TOML | Users want to see what defaults they're overriding; essential for creating a starting-point config file | LOW | Marshal `ClassFreqConfigs` + active rules to TOML; makes discoverability easy |
|
||||
| Config documentation via inline comments in generated TOML | When `--print-config` outputs commented TOML, users get self-documenting starting point | LOW | Write comment strings alongside marshaled output |
|
||||
| Named custom rules (user assigns a label) | User writes `name = "MyApp"` in a rule block; that name appears in the exit summary and `--verbose` output | LOW | Extend `classify.Rule` to carry optional display name |
|
||||
| Waveform preview hint in config error message | "valid waveforms: sine, square, sawtooth, triangle" inline with the error | LOW | Hard-code the valid set in the validator |
|
||||
| Harmonic override per class (not just base frequency) | Advanced users can tune the timbre, not just the pitch | MEDIUM | Requires exposing `HarmonicDef` slice in TOML schema; nesting adds parsing complexity |
|
||||
| Within-family tonal design: shared waveform within a family | Mail protocols all use the same waveform (e.g., sawtooth); listeners perceive the family by timbre as well as frequency | LOW | Assign waveform by family at design time; no new code needed |
|
||||
| Within-family detuning: minor third intervals between protocols in a family | Protocols within a family are harmonically close; the family "chord" is identifiable | LOW | Frequency assignment arithmetic at design time |
|
||||
| Directory/Auth family (LDAP, Kerberos) | Present on every enterprise network; their absence means enterprise traffic sounds like "other-TCP" noise | LOW | 2 more class constants + rules |
|
||||
| VoIP family (SIP, SIP-TLS) | IP phone traffic is prominent in offices; SIP port 5060 is easily matched | LOW | 2 more class constants + rules |
|
||||
| QUIC / HTTP3 class (UDP 443) | HTTP/3 now represents a significant fraction of web traffic; treating it identically to HTTPS when it arrives via UDP is perceptually meaningful | LOW | 1 rule: UDP port 443 maps to ClassQUIC or ClassHTTPS3 |
|
||||
|
||||
## Anti-Features for v1.1
|
||||
## Anti-Features for v1.2
|
||||
|
||||
Features that seem natural but should be avoided.
|
||||
Features that seem natural for this milestone but should be avoided.
|
||||
|
||||
| Anti-Feature | Why Avoid | What to Do Instead |
|
||||
|--------------|-----------|-------------------|
|
||||
| Config file hot-reload during capture | Appears useful but mid-capture parameter change would corrupt synthesis state and produce jarring audio discontinuities | Require restart to apply config changes; document this explicitly |
|
||||
| Environment variable config overrides | Adds a third precedence layer (flags > env > file > defaults) that increases combinatorial test surface with low user demand for this tool | Stick to flags + file + defaults; NetSynth is not a server needing 12-factor config |
|
||||
| Multiple config file includes / inheritance (`extends = "base.toml"`) | Sounds powerful, creates debugging nightmares when users do not understand the merge order | Single user config file merged with in-code defaults is sufficient; if a user needs multiple environments they can use `--config` |
|
||||
| YAML or JSON config format as alternatives | "Why not YAML?" is a common request; supporting multiple formats multiplies parser dependency surface and doubles validation code paths | TOML only; document the choice (TOML is unambiguous, has clean table syntax, is the standard for Go tooling) |
|
||||
| Silent partial load on validation error | Some tools load what they can and warn about the rest | Hard error on any invalid field; the user's intent for that field is unknown, so continuing is worse than stopping |
|
||||
| Config wizard / interactive setup | Out of scope for a CLI tool with a non-interactive model | Provide `--print-config` with comments as a self-service starting point |
|
||||
| Stereo pan position in config | Requested but explicitly deferred in PROJECT.md for this milestone | Out of scope for v1.1; document as v1.2 candidate |
|
||||
| Application-layer (DPI) detection | gopacket does not decode SMTP, SMB, Redis, etc. at the application layer. Implementing DPI requires a full protocol parser per protocol — massive scope. | Port-number matching is sufficient for classification purposes; DPI adds complexity without enough sonification value |
|
||||
| Single "Mail" or "Database" class (collapsing all protocols in family) | Loses per-protocol information. "Mail" does not tell you if it is inbound or outbound. "Database" does not distinguish Redis latency spikes from a slow Postgres query. | Keep individual TrafficClass per protocol; use frequency proximity for family grouping |
|
||||
| Dynamic port detection (e.g., FTP data port 20 bidirectional, ephemeral RTP ports) | FTP uses negotiated dynamic ports for data transfer; RTP uses ports negotiated over SIP. Tracking these requires stateful flow tracking across packets — significant architectural change. | Classify on well-known control/server ports only; dynamic data flows land in other-TCP/UDP. Document this limitation. |
|
||||
| Runtime group concept (Group struct with members) | Adding a Group abstraction to TrafficClass, Rule, or FreqConfig requires touching multiple packages and complicates the TOML schema. | Groups are a classification/display concept only, not a data structure. Implement them as naming conventions and config comment sections. |
|
||||
| Backward-incompatible frequency changes to existing 10 classes | Users who have existing TOML configs relying on the current frequencies would have their carefully tuned soundscapes broken | Keep ICMP=65Hz, DNS=110Hz, HTTPS=175Hz, HTTP=220Hz, SSH=330Hz, SMTP=440Hz, NTP=520Hz, DHCP=600Hz, OtherTCP=700Hz, OtherUDP=780Hz; assign new protocols to unoccupied slots |
|
||||
| Replacing other-TCP / other-UDP with something smarter | The catch-all classes serve an important role: unrecognized traffic is still represented. Removing them creates silent gaps. | Keep other-TCP and other-UDP as catch-alls; new specific classes reduce how much traffic lands there |
|
||||
| SRC port matching rules | Some protocols run on ephemeral source ports; adding src-port rules would double rule count and create false matches. Current architecture matches dst-port only. | Stick to dst-port matching. This is how nmap, iptables, and most classifiers work by default. |
|
||||
|
||||
## Feature Dependencies for v1.2
|
||||
|
||||
```
|
||||
[classify/types.go: add ~20 new TrafficClass constants]
|
||||
|
|
||||
+--enables--> [classify/rules.go: add new Rule entries per protocol]
|
||||
| |
|
||||
| +--feeds--> [classify.Classifier: matches packets to new classes]
|
||||
|
|
||||
+--enables--> [synth/config.go: add FreqConfig entries for new classes]
|
||||
| |
|
||||
| +--requires--> [Frequency rebalancing: shift new classes into family bands]
|
||||
| | (backward-compatible: existing 10 classes unchanged)
|
||||
| |
|
||||
| +--feeds--> [synth.Bank: synthesizes new layers]
|
||||
| (NumLayers constant must increase from 14 to cover new classes)
|
||||
|
|
||||
+--enables--> [classify.AllClasses(): include new classes in display order]
|
||||
|
|
||||
+--feeds--> [config.PrintConfig(): groups appear in --print-config output]
|
||||
+--feeds--> [aggregate.Summary: new classes appear in exit summary]
|
||||
```
|
||||
|
||||
### Dependency Notes for v1.2
|
||||
|
||||
- **NumLayers constant must increase.** `synth/config.go` has `NumLayers = 14` and derives `GainPerLayer = 1.0 / float64(NumLayers)` from it. Adding 20 protocols brings total classes to ~34. `NumLayers` must be updated, or the gain calculation must become dynamic. This is a straightforward arithmetic change but affects all layers' amplitude. Test the mix with more layers to confirm it still sounds balanced.
|
||||
|
||||
- **AllClasses() ordering determines --print-config output order.** Currently returns a flat slice. For v1.2, ordering by family group (all Mail classes together, all Database classes together) makes --print-config more readable. This is a display concern only — the order has no effect on classification.
|
||||
|
||||
- **No new external dependencies required.** All new protocols are detected via port number using the existing Rule struct. gopacket is not being asked to decode new application-layer protocols.
|
||||
|
||||
- **QUIC/HTTP3 requires a UDP 443 rule.** The current rules only match TCP 443 for HTTPS. Adding a separate rule for UDP 443 is one line. The question is naming: `ClassHTTPS3` or `ClassQUIC`. QUIC is the transport; HTTP/3 is the application. For sonification purposes, `ClassQUIC` is clearer because it describes the observable port behavior.
|
||||
|
||||
- **SIP detection covers control plane only.** SIP signals calls on port 5060/5061, but the actual voice/video payload travels via RTP on dynamically negotiated ports (typically in 16384-32767 range). The current classifier cannot detect RTP payloads without stateful flow tracking. Classify SIP only; document that RTP payload traffic lands in other-UDP.
|
||||
|
||||
---
|
||||
|
||||
## Feature Dependencies for v1.1
|
||||
## Protocol List: Final Recommended Set
|
||||
|
||||
```
|
||||
[Config file loader (TOML parse + merge)]
|
||||
|
|
||||
+--provides--> [Custom frequency overrides] (maps to synth.FreqConfig.BaseHz)
|
||||
|
|
||||
+--provides--> [Custom waveform per class] (requires oscillator waveform dispatch)
|
||||
| |
|
||||
| +--requires--> [Waveform type in oscillator] (new: sine/square/sawtooth/triangle)
|
||||
|
|
||||
+--provides--> [User-defined classification rules]
|
||||
|
|
||||
+--requires--> [Dynamic TrafficClass generation] (new: user rule class names)
|
||||
+--prepended-to--> [classify.DefaultRules]
|
||||
This is the complete recommended protocol class list for v1.2, including existing + new.
|
||||
|
||||
[--config flag] --overrides--> [Config file loader search path]
|
||||
[--print-config] --reads--> [Effective config after merge] (new subcommand)
|
||||
```
|
||||
### Existing (unchanged, backward-compatible)
|
||||
| Class | Protocol | Port | Transport |
|
||||
|-------|----------|------|-----------|
|
||||
| ICMP | ICMP/ICMPv6 | — | ICMP |
|
||||
| DNS | Domain Name System | 53 | TCP+UDP |
|
||||
| HTTPS | HTTP Secure / TLS | 443 | TCP |
|
||||
| HTTP | HTTP | 80 | TCP |
|
||||
| SSH | Secure Shell | 22 | TCP |
|
||||
| SMTP | Mail Transfer (server-to-server) | 25 | TCP |
|
||||
| NTP | Network Time Protocol | 123 | UDP |
|
||||
| DHCP | Dynamic Host Config | 67, 68 | UDP |
|
||||
| other-TCP | Unclassified TCP | — | TCP |
|
||||
| other-UDP | Unclassified UDP | — | UDP |
|
||||
| unknown-1..4 | Hash-bucketed unknowns | — | any |
|
||||
|
||||
### Dependency Notes for v1.1
|
||||
### New: Tier 1 (high frequency, recommended for v1.2)
|
||||
| Class | Protocol | Port | Transport | Family |
|
||||
|-------|----------|------|-----------|--------|
|
||||
| IMAP | IMAP / IMAPS | 143, 993 | TCP | Mail |
|
||||
| POP3 | POP3 / POP3S | 110, 995 | TCP | Mail |
|
||||
| SMTP-Submission | SMTP client submission | 587, 465 | TCP | Mail |
|
||||
| FTP | File Transfer Protocol | 20, 21 | TCP | File Transfer |
|
||||
| SMB | Server Message Block | 445 | TCP | File Transfer |
|
||||
| RDP | Remote Desktop Protocol | 3389 | TCP | Remote Access |
|
||||
| mDNS | Multicast DNS (Bonjour) | 5353 | UDP | Infrastructure |
|
||||
| SSDP | Simple Service Discovery | 1900 | UDP | Infrastructure |
|
||||
| SNMP | Simple Network Mgmt | 161, 162 | UDP | Infrastructure |
|
||||
| MySQL | MySQL database | 3306 | TCP | Database |
|
||||
| PostgreSQL | PostgreSQL database | 5432 | TCP | Database |
|
||||
| Redis | Redis in-memory store | 6379 | TCP | Database |
|
||||
|
||||
- **Waveform type is a new concept in the oscillator.** The v1.0 `Oscillator.Advance()` only does additive sine. To support square/sawtooth/triangle, the oscillator needs a `WaveformType` field and dispatch logic. This is an internal change, but it's required before waveform config can be wired up.
|
||||
- **User-defined rules require dynamic `TrafficClass` values.** v1.0 `TrafficClass` is a string type with predefined constants. User rules name their own classes (e.g., `"MyApp"`). The classifier already uses `TrafficClass` as a string; the `synth` layer needs to handle classes not in `ClassFreqConfigs` by looking up user-supplied sound parameters.
|
||||
- **Config loading must happen in `cmd` before the capture pipeline starts.** The cobra root command's `RunE` (or `PersistentPreRunE`) function loads and validates config, then passes effective config into the pipeline constructors. This is a structural change to `cmd/root.go`.
|
||||
- **`--print-config` is independent** of capture and can be implemented as a separate cobra subcommand reading only the config loader output.
|
||||
### New: Tier 2 (moderate frequency, recommended for v1.2 completeness)
|
||||
| Class | Protocol | Port | Transport | Family |
|
||||
|-------|----------|------|-----------|--------|
|
||||
| Telnet | Telnet (unencrypted shell) | 23 | TCP | Remote Access |
|
||||
| VNC | VNC / Remote Frame Buffer | 5900 | TCP | Remote Access |
|
||||
| TFTP | Trivial File Transfer | 69 | UDP | File Transfer |
|
||||
| SIP | SIP VoIP signaling | 5060, 5061 | TCP+UDP | VoIP |
|
||||
| LDAP | Directory Access Protocol | 389, 636 | TCP | Directory/Auth |
|
||||
| Kerberos | Kerberos authentication | 88 | TCP+UDP | Directory/Auth |
|
||||
| Syslog | System log forwarding | 514 | UDP | Infrastructure |
|
||||
| MongoDB | MongoDB document store | 27017 | TCP | Database |
|
||||
| QUIC | QUIC / HTTP/3 transport | 443 | UDP | Web |
|
||||
|
||||
**Total: 11 existing known + 21 new = 32 known protocol classes + 4 unknown buckets = 36 total.**
|
||||
|
||||
---
|
||||
|
||||
## Implementation Complexity Summary
|
||||
|
||||
| Feature | Complexity | Reason |
|
||||
|---------|------------|--------|
|
||||
| Config file loader (TOML parse + merge + validation) | LOW-MEDIUM | go-toml v2 handles parsing; merge logic is a loop; validation is a post-decode pass |
|
||||
| Custom frequency per class | LOW | Direct map lookup override; one line per class |
|
||||
| Custom waveform per class | MEDIUM | Oscillator needs waveform dispatch (new `WaveformType`); synthesis loop changes |
|
||||
| User-defined classification rules | MEDIUM | Dynamic class names; synth layer must handle unknown class names via config lookup |
|
||||
| `--config` flag + auto-discovery | LOW | Cobra flag + os.Stat checks on 2-3 paths |
|
||||
| `--print-config` subcommand | LOW | Marshal effective config to TOML; add comments |
|
||||
| Named custom rules in exit summary | LOW | `classify.Rule` struct gains optional `Name string` field |
|
||||
| Area | Complexity | Reason |
|
||||
|------|------------|--------|
|
||||
| New TrafficClass constants (~21) | LOW | Add string constants; no logic change |
|
||||
| New Rule entries in DefaultRules (~25 rules for 21 classes, some need 2 ports) | LOW | Add Rule structs; existing matcher handles them |
|
||||
| New FreqConfig entries (~21) | LOW | Add map entries with chosen Hz values and waveform |
|
||||
| Frequency rebalancing design | MEDIUM | Must assign ~21 new Hz values that (a) stay within audible range, (b) are musically coherent within families, (c) do not collide with existing 10 classes |
|
||||
| NumLayers update | LOW | One constant change; test mix amplitude |
|
||||
| AllClasses() family-ordered output | LOW | Reorder the returned slice by family |
|
||||
| PrintConfig family section headers | LOW | Add comment lines between family groups in PrintConfig |
|
||||
| Test updates | LOW | Add new classes to classifier tests; confirm no regressions |
|
||||
|
||||
**No new external dependencies required.** go-toml v2 is the only addition to `go.mod`.
|
||||
**No new external dependencies required for v1.2.**
|
||||
|
||||
---
|
||||
|
||||
## TOML Schema Sketch (Informational)
|
||||
## Competitor Feature Analysis (Updated for v1.2)
|
||||
|
||||
This is not a binding decision — it informs the roadmap's implementation phase. The schema should feel natural to a user who has seen other Go tool configs (golangci-lint, goreleaser).
|
||||
|
||||
```toml
|
||||
# Override built-in protocol sounds
|
||||
[classes.HTTPS]
|
||||
frequency = 220.0
|
||||
waveform = "square" # sine | square | sawtooth | triangle
|
||||
|
||||
[classes.DNS]
|
||||
frequency = 90.0
|
||||
|
||||
# Add custom classification rules (prepended before built-in rules, first-match-wins)
|
||||
[[rules]]
|
||||
name = "Internal API"
|
||||
protocol = "tcp"
|
||||
port = 8443
|
||||
frequency = 300.0
|
||||
waveform = "sawtooth"
|
||||
|
||||
[[rules]]
|
||||
name = "Game Traffic"
|
||||
protocol = "udp"
|
||||
port = 27015
|
||||
frequency = 450.0
|
||||
waveform = "triangle"
|
||||
```
|
||||
|
||||
Key schema design choices:
|
||||
- `[classes.X]` uses the same class name strings already used in `--verbose` output and exit summary (`HTTPS`, `DNS`, etc.) — no new naming system to learn.
|
||||
- `[[rules]]` is a TOML array of tables, consistent with how goreleaser and other tools express lists of items.
|
||||
- `protocol` and `port` map directly to the existing `classify.Rule` fields, minimizing translation.
|
||||
- Waveform is an enum string, not an integer — readable and self-documenting in the config file.
|
||||
|
||||
---
|
||||
|
||||
## v1.0 Feature Landscape (Retained from Original Research)
|
||||
|
||||
### Table Stakes (v1.0)
|
||||
|
||||
| Feature | Why Expected | Complexity | Notes |
|
||||
|---------|--------------|------------|-------|
|
||||
| Network interface selection (`-i eth0`) | tcpdump/tshark convention | LOW | Implemented: v1.0 |
|
||||
| Output file path flag (`-o output.mp3`) | Any file-producing CLI | LOW | Implemented: v1.0 |
|
||||
| Graceful Ctrl+C with file save | Users expect clean finalize | MEDIUM | Implemented: v1.0 |
|
||||
| Per-protocol sound distinction | Core value prop | MEDIUM | Implemented: v1.0, 12 rules |
|
||||
| Packet count / traffic summary on exit | Every capture tool does this | LOW | Implemented: v1.0 |
|
||||
| Privilege error message | Silent pcap failure is confusing | LOW | Implemented: v1.0 |
|
||||
| List available interfaces (`--list-interfaces`) | Users don't know interface names | LOW | Implemented: v1.0 |
|
||||
| Minimum viable duration guard | Zero-packet = no corrupt MP3 | LOW | Implemented: v1.0 |
|
||||
|
||||
### Differentiators (v1.0)
|
||||
|
||||
| Feature | Value Proposition | Complexity | Status |
|
||||
|---------|-------------------|------------|--------|
|
||||
| Auto-clustering of unrecognized traffic | Honest audio fingerprint | HIGH | Implemented: hash-bucket, 4 classes |
|
||||
| Ambient/drone style (layered sine harmonics) | Distinct from event-ping tools | HIGH | Implemented: v1.0 |
|
||||
| Time-windowed amplitude evolution | Mix evolves dynamically | MEDIUM | Implemented: 500ms windows + EMA |
|
||||
| BPF capture filter (`--filter`) | Power users scope what's sonified | MEDIUM | Implemented: v1.0 |
|
||||
| Offline pcap file input (`--read`) | Sonify historical captures | MEDIUM | Implemented: v1.0 |
|
||||
| Verbose protocol activity log (`--verbose`) | Developers see classifications | LOW | Implemented: v1.0 |
|
||||
|
||||
### Anti-Features (v1.0)
|
||||
|
||||
| Feature | Why Avoided |
|
||||
|---------|-------------|
|
||||
| Real-time audio playback | Platform audio API complexity; file output is correct |
|
||||
| GUI or web dashboard | Negates single-binary CLI value |
|
||||
| Custom sound mapping (v1.0) | Deferred to v1.1 — now the current milestone |
|
||||
| Rhythmic/percussive output | Ambient/drone is the deliberate differentiator |
|
||||
| Deep-packet inspection | Massive complexity; header classification sufficient |
|
||||
| Streaming MP3 output | MP3 finalization requires full buffer |
|
||||
| Anomaly detection / alerting | Different user job |
|
||||
|
||||
---
|
||||
|
||||
## Competitor Feature Analysis
|
||||
|
||||
| Feature | SoNSTAR (Python) | Network-Sonification (C# GUI) | Peep (C, Unix) | NetSynth v1.0 | NetSynth v1.1 |
|
||||
| Feature | SoNSTAR (Python) | Network-Sonification (C# GUI) | Peep (C, Unix) | NetSynth v1.1 | NetSynth v1.2 |
|
||||
|---------|-----------------|-------------------------------|----------------|----------------|----------------|
|
||||
| Custom sound config | No | No | Config file (fixed format) | No | Yes (TOML) |
|
||||
| Config file discovery | n/a | n/a | Hardcoded path | n/a | XDG + working dir |
|
||||
| Partial override semantics | n/a | n/a | Full replacement | n/a | Partial override |
|
||||
| Waveform selection | Recorded samples | sine/square/triangle | Fixed | sine only | sine/square/sawtooth/triangle |
|
||||
| Custom classification rules | No | No | No | No | Yes (user-defined port/proto rules) |
|
||||
| Named custom classes | n/a | n/a | n/a | n/a | Yes (appears in summary output) |
|
||||
| Protocol count | ~8 (TCP flow types) | ~10 | ~6 | 10 known + 4 unknown | ~32 known + 4 unknown |
|
||||
| Family grouping | No | No | No | No | Yes (7 families) |
|
||||
| Tonal family identity | No | No | No | No | Yes (freq proximity + shared waveform) |
|
||||
| Database protocols | No | No | No | No | Yes (MySQL, PostgreSQL, Redis, MongoDB) |
|
||||
| Mail family (IMAP/POP3) | No | No | No | SMTP only | Yes (SMTP + IMAP + POP3) |
|
||||
| Enterprise protocols (RDP, LDAP, Kerberos, SMB) | No | No | No | No | Yes |
|
||||
| Infrastructure expansion (mDNS, SNMP, Syslog) | No | No | No | No | Yes |
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir/latest/) — standard for `~/.config` discovery path
|
||||
- [adrg/xdg — Go XDG implementation](https://github.com/adrg/xdg) — if explicit XDG library is needed (probably not for NetSynth's 2-path lookup)
|
||||
- [pelletier/go-toml v2 — strict mode and DecodeError](https://pkg.go.dev/github.com/pelletier/go-toml/v2) — recommended TOML library; DisallowUnknownFields() and human-readable errors
|
||||
- [BurntSushi/toml — Undecoded() for unknown key detection](https://github.com/BurntSushi/toml) — alternative; simpler API but less actively maintained
|
||||
- [Building CLI Applications with Go: Cobra and Viper Guide (2026)](https://dasroot.net/posts/2026/03/building-cli-applications-go-cobra-viper/) — config loading patterns in Cobra CLI tools
|
||||
- [A Guide to TOML in Golang — kelche.co](https://www.kelche.co/blog/go/toml/) — go-toml v2 vs BurntSushi comparison and practical examples
|
||||
- [Configuration | mise-en-place](https://mise.jdx.dev/configuration.html) — example of working-dir + XDG config discovery
|
||||
- [golangci-lint configuration](https://golangci-lint.run/docs/configuration/cli/) — real-world example of partial override config in a Go CLI tool
|
||||
- [Online Tone Generator — waveform types](https://onlinetonegenerator.com/) — confirms sine/square/sawtooth/triangle as the standard 4 waveform set
|
||||
- [IANA Service Name and Transport Protocol Port Number Registry](https://www.iana.org/assignments/service-names-port-numbers) — authoritative port assignments
|
||||
- [List of TCP and UDP port numbers — Wikipedia](https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers) — comprehensive reference for well-known ports
|
||||
- [Common Ports Cheat Sheet — StationX](https://www.stationx.net/common-ports-cheat-sheet/) — grouped protocol reference used for family taxonomy
|
||||
- [nDPI Protocols List — ntop](https://www.ntop.org/guides/nDPI/protocols.html) — nDPI's 450+ protocol list and 17-category taxonomy; source for family grouping inspiration
|
||||
- [nDPI 5.0: Enhanced Traffic Fingerprinting — ntop blog](https://www.ntop.org/ndpi-5-0-enhanced-traffic-fingerprinting-and-fpc-many-new-protocols/) — confirms category-based grouping as the production approach for managing large protocol sets
|
||||
- [SoNSTAR: Sonification of Network Traffic — Paul Vickers](https://paulvickers.github.io/SoNSTAR/) — academic network sonification tool; uses TCP/IP flow features rather than protocol families
|
||||
- [Sonification of network traffic flow for monitoring and situational awareness — PLoS One 2018](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0195948) — research literature on what protocol-level groupings are perceptually useful in sonification
|
||||
- [pkg.go.dev/github.com/gopacket/gopacket/layers](https://pkg.go.dev/github.com/gopacket/gopacket/layers) — confirmed that gopacket natively decodes DNS, DHCP, NTP, ICMP, OSPF, BGP, but NOT SMTP, IMAP, FTP, SMB, MySQL, Redis, SIP, RTP, LDAP at the application layer; port-based classification is the correct approach for v1.2
|
||||
- [Realtime High-Speed Network Traffic Monitoring Using ntopng — LISA 2014](https://luca.ntop.org/Lisa2014.pdf) — confirms category-based protocol grouping as standard in production monitoring tools
|
||||
- Internal codebase review: `/home/dev/workspace/yoloyolo/classify/rules.go`, `types.go`, `synth/config.go` — confirmed existing 10 known classes, Rule struct pattern, FreqConfig pattern, NumLayers=14 constant, and auto-assign frequency range (1200-2350 Hz)
|
||||
|
||||
---
|
||||
*v1.0 research: 2026-03-24*
|
||||
*v1.1 custom sound mappings research: 2026-03-26*
|
||||
*v1.2 extended protocol coverage research: 2026-03-27*
|
||||
|
||||
+419
-377
@@ -1,511 +1,553 @@
|
||||
# Pitfalls Research
|
||||
# Domain Pitfalls
|
||||
|
||||
**Domain:** Network-traffic-to-audio synthesis CLI tool (Go)
|
||||
**Researched:** 2026-03-26 (v1.1 update — TOML config, waveform types, user-defined rules)
|
||||
**Confidence:** HIGH (TOML decoder behaviors verified against pkg.go.dev official docs and issue trackers; audio synthesis aliasing verified against DSP literature; config merging verified against BurntSushi/toml issue #47 and go-toml issue #252)
|
||||
**Domain:** NetSynth — Network traffic sonification CLI tool (Go)
|
||||
**Researched:** 2026-03-26 (v1.1 original); 2026-03-27 (v1.2 update — extended protocol coverage, grouped sound families)
|
||||
**Confidence:** HIGH (all pitfalls grounded in direct codebase inspection; audio masking values from Glasberg & Moore 1990 ERB model; TOML behaviors from BurntSushi docs and issue history; Go performance from first-principles analysis of existing code)
|
||||
|
||||
---
|
||||
|
||||
## v1.1 Milestone Pitfalls (New)
|
||||
## v1.2 Milestone Pitfalls (New)
|
||||
|
||||
These pitfalls are specific to adding TOML config, waveform types, and user-defined classification rules to the existing NetSynth codebase.
|
||||
These pitfalls are specific to adding expanded protocol classification with grouped sound families to the existing NetSynth v1.1 codebase. They are ordered by severity: critical pitfalls cause incorrect output or broken configs without obvious errors; moderate pitfalls degrade audio quality or developer experience; minor pitfalls are friction points with clear workarounds.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall A1: TOML Unmarshal Silently Overwrites Pre-filled Defaults with Zero Values
|
||||
## Critical Pitfalls
|
||||
|
||||
### Pitfall C1: Frequency Rebalancing Silently Invalidates User v1.1 Configs
|
||||
|
||||
**What goes wrong:**
|
||||
You initialize a `Config` struct with built-in defaults, then call `toml.Unmarshal` to layer in user overrides. Any field the user *omits* from their TOML file is set to its Go zero value (`0`, `""`, `false`, `nil`) by the decoder — overwriting your defaults. A user who writes only `[sounds.DNS]` in their config file to change the DNS tone ends up wiping every other class back to zero Hz.
|
||||
Users who created a `netsynth.toml` under v1.1 specified absolute Hz values for built-in classes — for example `[sounds.HTTPS] frequency = 175.0`. If v1.2 rebalances that class to a new default (say 340 Hz to make room for new protocols), the user's config now overrides the new default back to the old v1.1 value (175 Hz). The user gets a v1.1 sound for HTTPS even after upgrading, with no warning that their override value has become stale relative to the rebalanced layout.
|
||||
|
||||
The reverse is also possible: a user who did NOT set a frequency override had HTTPS at 175 Hz; after rebalancing it moves silently to a different Hz value. Their soundscape has changed without explanation.
|
||||
|
||||
**Why it happens:**
|
||||
Both `BurntSushi/toml` and `pelletier/go-toml` v1 do not distinguish between "key was absent" and "key was explicitly set to zero". The decoder reflects over the struct and writes zero for every absent key. This was explicitly reported as a bug in BurntSushi/toml issue #47 and go-toml issue #252. go-toml v2 partially addresses it but still zeros primitive-type fields that are absent.
|
||||
The `merge()` function in `config/config.go` applies any `[sounds.X] frequency = Y` from the user's TOML unconditionally. There is no concept of "this override is relative to a previous default" — it is applied as a fixed Hz value. Changing the default in `ClassFreqConfigs` does not trigger any validation that existing user overrides remain intentional.
|
||||
|
||||
**Consequences:**
|
||||
- All non-overridden traffic classes play silence (0 Hz oscillator)
|
||||
- Classification rules get zeroed if user only partially fills `[[rules]]`
|
||||
- EMA tau, whisper floor, gain, and other synth parameters reset to 0
|
||||
- v1.1 user configs produce different-than-expected audio on v1.2 without any error or warning
|
||||
- Users with explicit overrides are stuck at v1.1 frequency values — the rebalancing has zero effect for them
|
||||
- Users without overrides hear an unexplained soundscape change after upgrade
|
||||
|
||||
**Prevention:**
|
||||
Use pointer fields (`*float64`, `*string`) in the decoded struct to distinguish "not provided" (nil pointer) from "explicitly set to zero" (non-nil pointer to 0). Apply a merge step: iterate over the decoded struct, and for each pointer field that is nil, keep the built-in default. For slice fields (like `[]RuleConfig`), nil slice means "user did not provide rules" — preserve defaults; non-nil empty slice (`[]RuleConfig{}`) means "user explicitly cleared rules" — respect that.
|
||||
Two complementary strategies:
|
||||
|
||||
1. **Minimize rebalancing scope.** Assign new protocol classes to frequency ranges not yet occupied by v1.1 built-ins. The current v1.1 built-in range is 65–780 Hz (known protocols) and 862–1047 Hz (unknown buckets), with a gap at 781–861 Hz. New families can be allocated into ranges above 1100 Hz (e.g., 1100–4000 Hz), leaving all existing Hz assignments untouched. This eliminates the backward-compatibility problem entirely for users who have not overridden values in that range.
|
||||
|
||||
2. **Changelog + --print-config.** If rebalancing IS required, document every changed Hz value in the release notes and update `--print-config` so users can diff their effective config against what they saved. Add a comment to `--print-config` output when a user override matches a value that was the v1.1 default (warn that it may be stale).
|
||||
|
||||
**Detection:**
|
||||
- User reports HTTPS sounds wrong after upgrade
|
||||
- `--print-config` shows `(override)` for a class the user never intentionally customized — they set it once to the default value and now that exact value is stale
|
||||
|
||||
**Phase to address:**
|
||||
Frequency allocation design phase (first phase of v1.2). The spectrum layout must be finalized before writing any `ClassFreqConfigs` entries. Treat the 65–1047 Hz range as frozen for backward compatibility.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall C2: `autoAssignFreq` Range Collision With New Built-in Frequencies
|
||||
|
||||
**What goes wrong:**
|
||||
`autoAssignFreq()` in `config/config.go` assigns custom user classes to frequencies in `[1200, 2350]` Hz using 24 steps of 50 Hz each. If new v1.2 built-in protocol classes are assigned frequencies in this same range (e.g., placing SIP at 1500 Hz or IMAP at 1200 Hz), a user's custom class may hash to the same frequency as a new built-in. The user's custom class and the new built-in will produce the same tone — the soundscape loses discriminability, and the user has no way to know their custom class has collided.
|
||||
|
||||
**Why it happens:**
|
||||
The `[1200, 2350]` range was deliberately chosen in v1.1 as "unused 1200-2350 Hz range" (per the STACK.md annotation). If v1.2 extends built-ins into that range without also updating `autoAssignFreq`, the guarantee is broken.
|
||||
|
||||
**Consequences:**
|
||||
- Silent frequency collision: two classes (one built-in, one user-defined) play the same tone
|
||||
- User's custom classification is perceptually indistinguishable from the colliding built-in
|
||||
- `--print-config` will show different Hz values in the TOML text, but the audio output is identical
|
||||
|
||||
**Prevention:**
|
||||
Update `autoAssignFreq` in `config/config.go` whenever new built-in frequency ranges are allocated. Specifically:
|
||||
- After finalizing all v1.2 `ClassFreqConfigs` frequencies, compute the highest built-in Hz value
|
||||
- Set `autoAssignFreq` base above that value, e.g., `baseHz = 4500.0` with a range that is guaranteed to be clear of all built-ins
|
||||
- Add a compile-time assertion (test) that verifies no `ClassFreqConfigs` entry falls inside the auto-assign range
|
||||
|
||||
```go
|
||||
// In config struct, use pointers for optional overrides:
|
||||
type SoundConfig struct {
|
||||
FreqHz *float64 `toml:"freq_hz"`
|
||||
Waveform *string `toml:"waveform"`
|
||||
}
|
||||
|
||||
// Merge: for each class, override only non-nil fields
|
||||
func mergeSound(base synth.FreqConfig, override SoundConfig) synth.FreqConfig {
|
||||
if override.FreqHz != nil {
|
||||
base.BaseHz = *override.FreqHz
|
||||
// In synth/config_test.go:
|
||||
func TestAutoAssignRangeIsEmpty(t *testing.T) {
|
||||
const autoBase = 4500.0
|
||||
const autoTop = 6000.0
|
||||
for class, cfg := range ClassFreqConfigs {
|
||||
if cfg.BaseHz >= autoBase && cfg.BaseHz <= autoTop {
|
||||
t.Errorf("built-in class %q frequency %.1f Hz falls in auto-assign range [%.0f, %.0f]",
|
||||
class, cfg.BaseHz, autoBase, autoTop)
|
||||
}
|
||||
if override.Waveform != nil {
|
||||
base.WaveformType = *override.Waveform
|
||||
}
|
||||
return base
|
||||
}
|
||||
```
|
||||
|
||||
**Detection:**
|
||||
- User reports that classes they did not configure now produce no sound
|
||||
- Unit test: load a config that overrides only one class; verify all other classes retain built-in Hz values
|
||||
- Two classes with different names produce identical tones
|
||||
- `--print-config` shows correct but coincidentally matching Hz values for a user class and a built-in
|
||||
|
||||
**Phase to address:**
|
||||
Config loading phase (first phase of v1.1). Get the pointer-and-merge pattern established before wiring config into the bank. Retrofitting after the bank construction is wired is a significant churn.
|
||||
Frequency allocation design phase. After all new built-in Hz values are set, update `autoAssignFreq` constants and add the compile-time range check before merging.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall A2: BurntSushi/toml Silently Ignores Typos in Field Names
|
||||
### Pitfall C3: Hardcoded `NumLayers = 14` Constant Becomes a Lie — But `GainPerLayer` Stays Wrong
|
||||
|
||||
**What goes wrong:**
|
||||
A user writes `freq_hz = 440` but the struct tag is `toml:"freq_hz"` — this works. However if the user writes `freqhz = 440` or `FreqHz = 440` or a misspelled `frek_hz = 440`, the library silently ignores the key. The user's override is never applied. No error is returned. The user thinks their config is active; it is not.
|
||||
|
||||
**Why it happens:**
|
||||
`BurntSushi/toml` by default silently discards keys that do not map to any struct field. This is the documented default behavior ("will ignore options in the TOML file that you don't use"). It is the opposite of "strict mode."
|
||||
|
||||
**Consequences:**
|
||||
- Silent misconfiguration: user's customization is invisible
|
||||
- Debugging is very hard — no error to trace back to the TOML file
|
||||
|
||||
**Prevention:**
|
||||
Use `toml.Decode` (not `toml.Unmarshal`) to obtain `MetaData`, then call `md.Undecoded()` and return an error listing any keys that were not decoded. This is BurntSushi's documented strict-mode pattern.
|
||||
|
||||
`synth/config.go` defines:
|
||||
```go
|
||||
md, err := toml.Decode(string(data), &cfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if keys := md.Undecoded(); len(keys) > 0 {
|
||||
return fmt.Errorf("unknown config keys (check for typos): %v", keys)
|
||||
}
|
||||
NumLayers = 14
|
||||
GainPerLayer = 1.0 / float64(NumLayers) // 0.0714
|
||||
```
|
||||
|
||||
The current `bank.go` correctly uses `1.0 / float64(len(cfgs))` at bank construction time, so the bank itself handles more layers correctly. However, `GainPerLayer` is still exported as a package-level constant. If any code outside `bank.go` references `synth.GainPerLayer` for gain calculations (including tests, the encode pipeline, or future code added in v1.2), it will use the stale `0.0714` value even when 25+ layers are active. The bank sounds louder than expected at low class counts, or softer at high class counts, depending on which reference is used.
|
||||
|
||||
Additionally, `TestNumLayersMatchesAllClasses` in `synth/config_test.go` checks:
|
||||
```go
|
||||
if len(synth.ClassFreqConfigs) != len(classify.AllClasses())
|
||||
```
|
||||
This test enforces that `ClassFreqConfigs` and `AllClasses()` stay in sync. Adding new classes to one without the other causes this test to fail — which is the right behavior, but the fix is non-obvious: you must update both `ClassFreqConfigs` (with a new `FreqConfig` entry) AND `AllClasses()` (by appending to the return value). Miss either and the test blocks the build.
|
||||
|
||||
**Why it happens:**
|
||||
`NumLayers` was introduced in v1.0 when the layer count was static. It was not removed when `NewBank` was refactored to use `len(cfgs)` dynamically. The constant is now a documentation artifact that can mislead future code.
|
||||
|
||||
**Consequences:**
|
||||
- Any code added in v1.2 that references `synth.GainPerLayer` uses an incorrect value
|
||||
- Possible audio clipping (if gain is too high) or inaudibly quiet output (if computed with wrong layer count)
|
||||
- `TestNumLayersMatchesAllClasses` fails if `AllClasses()` and `ClassFreqConfigs` are updated independently
|
||||
|
||||
**Prevention:**
|
||||
At the start of v1.2 protocol rule addition:
|
||||
1. Remove `NumLayers` and `GainPerLayer` constants from `synth/config.go` (or mark them deprecated with a clear comment)
|
||||
2. Rename `TestNumLayersMatchesAllClasses` to `TestClassFreqConfigsMatchesAllClasses` and update its comment to explain the invariant
|
||||
3. When adding each new protocol class: update `AllClasses()` and `ClassFreqConfigs` atomically in the same commit — the test will catch any missed entry
|
||||
|
||||
**Detection:**
|
||||
- Config change that should audibly alter the sound has no effect
|
||||
- Undecoded keys present but no warning/error logged
|
||||
- `TestNumLayersMatchesAllClasses` fails after adding new classes to only one of the two locations
|
||||
- Audio output is unexpectedly loud or quiet compared to previous version
|
||||
|
||||
**Phase to address:**
|
||||
Config loading phase. Implement strict decoding from the first config load function. Do not add this as an afterthought — it is the primary mechanism protecting users from silent misconfiguration.
|
||||
First code phase of v1.2, before adding any new protocol classes. Removing the stale constant and renaming the test is a two-minute cleanup that prevents confusion throughout the milestone.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall A3: Naive Square/Sawtooth/Triangle Generation Produces Audible Aliasing Distortion
|
||||
### Pitfall C4: `TestFrequenciesInRange` Hardcodes `[60, 1100]` — Will Fail for New High-Frequency Classes
|
||||
|
||||
**What goes wrong:**
|
||||
Implementing waveforms by direct time-domain math — `sign(sin(phase))` for square, `2*frac(phase)-1` for sawtooth, `1-2*abs(frac(phase)-0.5)` for triangle — produces a waveform with infinite harmonics. At 44100 Hz, harmonics above 22050 Hz fold back into the audible range as aliasing. At the frequencies used in NetSynth (65–1047 Hz), aliasing from a naive square wave produces a buzzing distortion that is especially audible at higher drone frequencies and sounds like corruption rather than timbre.
|
||||
|
||||
**Why it happens:**
|
||||
The mathematical waveforms are not bandlimited — they have infinite harmonic content. Direct sampling them at 44100 Hz aliases all energy above Nyquist back into the audible band. Developers who test at low frequencies (60–120 Hz) may not notice because the aliased harmonics land at very high frequencies with low perceptual impact; the problem worsens significantly above 400 Hz where aliases fold into the 1–5 kHz perceptually prominent range.
|
||||
|
||||
**Consequences:**
|
||||
- Square/sawtooth at SSH (330 Hz) and higher frequencies sounds harsh and buzzy
|
||||
- The effect worsens at higher frequencies, making SMTP (440 Hz) and DHCP (600 Hz) drones sound distorted
|
||||
- Aliasing cannot be filtered out post-synthesis (it is interleaved with desired signal)
|
||||
|
||||
**Prevention:**
|
||||
Use additive synthesis — the approach already in use for sine waves in `oscillator.go`. The existing `Oscillator.Advance(harmonics []HarmonicDef)` computes `sin(2π * phase * ratio)` for each partial. Square, sawtooth, and triangle waveforms are all expressible as harmonic series:
|
||||
|
||||
- **Square:** odd harmonics only, amplitude `1/k` for harmonic `k`: ratios 1, 3, 5, 7, ... with amplitudes 1.0, 0.33, 0.20, 0.14, ... Truncate at Nyquist.
|
||||
- **Sawtooth:** all harmonics, amplitude `1/k`: ratios 1, 2, 3, 4, ... with amplitudes 1.0, 0.5, 0.33, 0.25, ... Truncate at Nyquist.
|
||||
- **Triangle:** odd harmonics, amplitude `1/k²`, alternating sign: ratios 1, 3, 5, ... with amplitudes 1.0, 0.11, 0.04, ... Truncate at Nyquist.
|
||||
|
||||
The truncation (only sum harmonics where `freq * ratio < sampleRate / 2`) is the critical step that makes the synthesis bandlimited. The existing `[]HarmonicDef` structure in `synth/config.go` already supports this — waveform type selection just requires generating the right harmonic series for each `FreqConfig`.
|
||||
|
||||
Waveform presets should be pre-computed `[]HarmonicDef` slices, not runtime computation of naive waveform math:
|
||||
|
||||
`synth/config_test.go` contains:
|
||||
```go
|
||||
// BandlimitedHarmonics returns a bandlimited harmonic series for the given waveform type.
|
||||
// It truncates harmonics at Nyquist (sampleRate/2) to prevent aliasing.
|
||||
func BandlimitedHarmonics(waveform string, baseHz float64, sampleRate int) []HarmonicDef {
|
||||
nyquist := float64(sampleRate) / 2.0
|
||||
var defs []HarmonicDef
|
||||
switch waveform {
|
||||
case "square":
|
||||
for k := 1; float64(k)*baseHz < nyquist; k += 2 { // odd only
|
||||
defs = append(defs, HarmonicDef{Ratio: k, Amplitude: 1.0 / float64(k)})
|
||||
func TestFrequenciesInRange(t *testing.T) {
|
||||
for class, cfg := range synth.ClassFreqConfigs {
|
||||
if cfg.BaseHz < 60 || cfg.BaseHz > 1100 {
|
||||
t.Errorf("class %q BaseHz=%.1f is out of range [60, 1100]", class, cfg.BaseHz)
|
||||
}
|
||||
case "sawtooth":
|
||||
for k := 1; float64(k)*baseHz < nyquist; k++ {
|
||||
defs = append(defs, HarmonicDef{Ratio: k, Amplitude: 1.0 / float64(k)})
|
||||
}
|
||||
case "triangle":
|
||||
sign := 1.0
|
||||
for k := 1; float64(k)*baseHz < nyquist; k += 2 {
|
||||
defs = append(defs, HarmonicDef{Ratio: k, Amplitude: sign / float64(k*k)})
|
||||
sign = -sign
|
||||
}
|
||||
default: // "sine"
|
||||
defs = []HarmonicDef{{Ratio: 1, Amplitude: 1.0}}
|
||||
}
|
||||
return defs
|
||||
}
|
||||
```
|
||||
|
||||
**Detection:**
|
||||
- Audible buzzing or grainy texture on drone layers above 300 Hz with non-sine waveforms
|
||||
- Square/sawtooth waveforms sound harsher than expected at high frequencies
|
||||
|
||||
**Phase to address:**
|
||||
Waveform type implementation phase. The design decision (additive synthesis, not direct waveform math) must be made before coding waveform support. Switching from direct math to additive after the fact requires rewriting the oscillator API.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall A4: Waveform String Validation Fails Silently, Falls Back to Silence
|
||||
|
||||
**What goes wrong:**
|
||||
A user writes `waveform = "Sawtooth"` (capital S) or `waveform = "saw"` (abbreviation). The config loading code does a simple equality check (`if waveform == "sawtooth"`), finds no match, and either panics, silently emits silence, or applies a default without telling the user. In all cases the user's intent is invisible.
|
||||
If any new v1.2 protocol class is assigned a frequency above 1100 Hz (which is necessary if new families extend into the 1100–4000 Hz range), this test fails immediately. The test was written for v1.0's 14-class spectrum. It will now become a false blocker, making every correct new assignment fail CI.
|
||||
|
||||
**Why it happens:**
|
||||
String-based enumerations in config files have no compile-time type checking. Case sensitivity and abbreviations are user expectations that must be explicitly handled.
|
||||
Range-assertion tests like this encode a snapshot of the system state at the time they were written. They do not automatically update when the valid range evolves.
|
||||
|
||||
**Consequences:**
|
||||
- Silent misconfiguration: wrong waveform with no feedback
|
||||
- Hard to debug: config appears valid, sound is just wrong
|
||||
- CI red on every correct new class addition until the test is updated
|
||||
- Developer wastes time diagnosing a failing test that is wrong, not the code
|
||||
- Risk: developer deletes the test entirely rather than updating it, losing the coverage
|
||||
|
||||
**Prevention:**
|
||||
Normalize waveform strings at parse time (`strings.ToLower`, `strings.TrimSpace`), validate against the accepted set, and return an explicit error with the accepted values if the string is unrecognized:
|
||||
Update the test when the frequency allocation design is finalized. The new range should accommodate whatever spectrum is decided, e.g.:
|
||||
|
||||
```go
|
||||
var validWaveforms = map[string]struct{}{
|
||||
"sine": {}, "square": {}, "sawtooth": {}, "triangle": {},
|
||||
}
|
||||
func validateWaveform(s string) (string, error) {
|
||||
normalized := strings.ToLower(strings.TrimSpace(s))
|
||||
if _, ok := validWaveforms[normalized]; !ok {
|
||||
return "", fmt.Errorf("unknown waveform %q: must be one of sine, square, sawtooth, triangle", s)
|
||||
func TestFrequenciesInRange(t *testing.T) {
|
||||
for class, cfg := range synth.ClassFreqConfigs {
|
||||
if cfg.BaseHz < 60 || cfg.BaseHz > 4000 {
|
||||
t.Errorf("class %q BaseHz=%.1f is out of range [60, 4000]", class, cfg.BaseHz)
|
||||
}
|
||||
return normalized, nil
|
||||
}
|
||||
```
|
||||
|
||||
**Phase to address:**
|
||||
Config validation step (same phase as config loading). Implement all string field validation in a single `validate(cfg Config) error` function called immediately after decoding.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall A5: User Rules Appended After Catch-All Rules Are Unreachable
|
||||
|
||||
**What goes wrong:**
|
||||
The existing `DefaultRules` slice ends with two catch-alls:
|
||||
|
||||
```go
|
||||
{Protocol: "tcp", DstPort: 0, Class: ClassOtherTCP},
|
||||
{Protocol: "udp", DstPort: 0, Class: ClassOtherUDP},
|
||||
```
|
||||
|
||||
If user-defined rules are simply appended to this slice (`append(DefaultRules, userRules...)`), the catch-alls match first (DstPort=0 matches any port for that protocol), and the user's rules are unreachable. Every custom rule maps to `ClassOtherTCP` or `ClassOtherUDP` instead. The user gets no sound from their custom class.
|
||||
|
||||
**Why it happens:**
|
||||
The first-match-wins semantics of `Classifier.Classify()` mean ordering is semantically critical. `DefaultRules` is a named var that exists precisely as an ordered slice — the comment `// Catch-alls (must be last)` documents this constraint. But "must be last in the defaults" does not automatically mean "must be last in the final merged slice." Developers who concatenate slices without thinking about this invariant break the system.
|
||||
|
||||
**Consequences:**
|
||||
- All user-defined rules are silently swallowed by catch-alls
|
||||
- User's custom class never activates
|
||||
- No error — the pipeline works, just wrong
|
||||
|
||||
**Prevention:**
|
||||
Always insert user rules *before* catch-all rules. The merge strategy must be: `specificDefaultRules + userRules + catchAllRules`. Implement this with an explicit split in the default rule set:
|
||||
|
||||
```go
|
||||
// In classify/rules.go, split into two exported slices:
|
||||
var SpecificRules = []Rule{ /* ICMP through DHCP */ }
|
||||
var CatchAllRules = []Rule{
|
||||
{Protocol: "tcp", DstPort: 0, Class: ClassOtherTCP},
|
||||
{Protocol: "udp", DstPort: 0, Class: ClassOtherUDP},
|
||||
}
|
||||
|
||||
// Merge function used by config loading:
|
||||
func MergeRules(userRules []Rule) []Rule {
|
||||
result := make([]Rule, 0, len(SpecificRules)+len(userRules)+len(CatchAllRules))
|
||||
result = append(result, SpecificRules...)
|
||||
result = append(result, userRules...)
|
||||
result = append(result, CatchAllRules...)
|
||||
return result
|
||||
}
|
||||
```
|
||||
|
||||
Alternatively, annotate each default rule with a `CatchAll bool` field and sort before use. The split-slice approach is simpler and more explicit.
|
||||
|
||||
**Detection:**
|
||||
- User-defined rule that should match traffic does not produce its custom sound
|
||||
- `--verbose` output shows traffic being classified as `OtherTCP`/`OtherUDP` instead of the custom class
|
||||
- Test: write a rule for port 8080, send HTTP traffic to port 8080, verify it hits the custom class and not `ClassOtherTCP`
|
||||
|
||||
**Phase to address:**
|
||||
User-defined rules phase. The `classify/rules.go` split must be the first code change before any config loading logic references the rule slice.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall A6: User-Defined Classes Have No synth.FreqConfig Entry — Bank Panics or Plays Silence
|
||||
|
||||
**What goes wrong:**
|
||||
`OscillatorBank.NewBank()` iterates over `classify.AllClasses()` and looks up each class in `ClassFreqConfigs`. A user-defined rule creates a new `TrafficClass` (e.g., `"my-api"`). This class is not in `AllClasses()`, so the bank has no layer for it. The aggregator increments a count for `"my-api"`, `RenderWindow` looks up `b.layers["my-api"]`, gets nil, and either panics (nil pointer dereference on `layer.AdvanceSample()`) or silently contributes nothing to the mix.
|
||||
|
||||
**Why it happens:**
|
||||
`classify.AllClasses()` is a hardcoded list of the 14 built-in classes. The synth bank is constructed once at startup from this static list. User-defined classes are a runtime extension that the bank knows nothing about.
|
||||
|
||||
**Consequences:**
|
||||
- Nil pointer panic in `RenderWindow` if the layer map lookup is not nil-guarded
|
||||
- Or silent: user-defined class traffic is captured and aggregated but never rendered to audio
|
||||
- In either case the user's primary feature request (custom sounds for custom classes) silently fails
|
||||
|
||||
**Prevention:**
|
||||
The bank must be constructed from the *full* set of active classes, including user-defined ones. The construction path should be:
|
||||
|
||||
1. Load config (parse TOML, validate)
|
||||
2. Compute effective rule set (built-in + user rules)
|
||||
3. Extract the complete set of `TrafficClass` values referenced by all rules
|
||||
4. Pass this full class set to `NewBank` (or equivalent) so a layer is created for every reachable class
|
||||
5. Wire user-defined class frequencies from config into the bank
|
||||
|
||||
`AllClasses()` in `classify/types.go` should either remain the static built-in list (used for display/iteration of built-ins) or be replaced by a dynamic function that takes the active rule set as input. Do not rely on the hardcoded list in the bank-construction path when user-defined classes are possible.
|
||||
|
||||
**Detection:**
|
||||
- Panic: `runtime error: invalid memory address or nil pointer dereference` in `synth/bank.go:RenderWindow`
|
||||
- Or: user-defined class produces no sound, no error
|
||||
- Test: create a config with one user rule using a custom class; verify the bank is built with a layer for that class and that layer produces sound
|
||||
|
||||
**Phase to address:**
|
||||
User-defined rules phase, specifically the bank initialization step. This is the deepest integration point — it touches the pipeline at capture → classify → aggregate → synthesize.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall A7: Config Auto-Discovery Follows Wrong Order or Ignores XDG Variables
|
||||
|
||||
**What goes wrong:**
|
||||
The spec calls for auto-discovery from `./netsynth.toml` then `~/.config/netsynth/config.toml`. A naive implementation uses `os.UserHomeDir()` to build the fallback path. On systems where `$XDG_CONFIG_HOME` is set to a non-default location (common on NixOS, custom dotfile managers, CI environments), the tool ignores the user's configured config directory and looks in `~/.config` anyway. The user has a config at `$XDG_CONFIG_HOME/netsynth/config.toml` that is never found.
|
||||
|
||||
Additionally, `os.UserHomeDir()` returns an error if `$HOME` is unset (e.g., inside some Docker containers or cron jobs). If this error is not handled, the path construction silently produces `"/.config/netsynth/config.toml"` (an absolute path starting with `/.config`) rather than failing with a useful message.
|
||||
|
||||
**Why it happens:**
|
||||
Go's `os.UserConfigDir()` already implements the XDG lookup (`$XDG_CONFIG_HOME` → `~/.config` on Linux, `~/Library/Application Support` on macOS). Most developers reach for `os.UserHomeDir()` + hardcoded `".config"` string because it is the first function they find in the stdlib.
|
||||
|
||||
**Consequences:**
|
||||
- User's config is silently ignored when `$XDG_CONFIG_HOME` is non-default
|
||||
- Confusing behavior difference between development machines and CI
|
||||
|
||||
**Prevention:**
|
||||
Use `os.UserConfigDir()` (stdlib, Go 1.13+) for the platform-appropriate config directory. This correctly respects `$XDG_CONFIG_HOME` on Linux and `APPDATA` on Windows (if ever relevant). The discovery order should be:
|
||||
|
||||
```go
|
||||
func configSearchPaths() []string {
|
||||
var paths []string
|
||||
// 1. Current directory (highest precedence)
|
||||
paths = append(paths, "netsynth.toml")
|
||||
// 2. XDG/platform config dir
|
||||
if cfgDir, err := os.UserConfigDir(); err == nil {
|
||||
paths = append(paths, filepath.Join(cfgDir, "netsynth", "config.toml"))
|
||||
}
|
||||
return paths
|
||||
}
|
||||
```
|
||||
|
||||
If `--config` flag is set, use that path exclusively and return a clear error if the file is absent (do not fall through to auto-discovery when explicit path is provided).
|
||||
Alternatively, replace the range test with a `TestFrequenciesUnique` variant that only checks for collisions (already present), which remains valid regardless of range expansion. The range test can instead verify family-level groupings: all Mail family classes are in [X, Y] Hz range, all VoIP classes in [A, B] Hz range.
|
||||
|
||||
**Detection:**
|
||||
- Config not loaded on systems where `$XDG_CONFIG_HOME=/custom/path`
|
||||
- Silent "no config found" behavior when a config clearly exists at the XDG path
|
||||
- CI fails on `TestFrequenciesInRange` after adding first new class above 1100 Hz
|
||||
- The test name suggests a range violation but the code is correct
|
||||
|
||||
**Phase to address:**
|
||||
Config loading phase. Implement the path discovery with `os.UserConfigDir()` from the start. Fix before the feature ships.
|
||||
Immediately when frequency allocation is decided — before adding any `ClassFreqConfigs` entries outside [60, 1100].
|
||||
|
||||
---
|
||||
|
||||
### Pitfall A8: Explicit --config Flag Does Not Error on Missing File
|
||||
### Pitfall C5: `[families]` TOML Block Rejected by Strict Unknown-Key Validation
|
||||
|
||||
**What goes wrong:**
|
||||
When `--config path/to/file.toml` is specified, the user expects an error if the file does not exist. If the config loader falls through to auto-discovery when the explicit path is missing, or silently uses defaults, the user has no way to detect a typo in their `--config` argument. They run a session, get "unexpected" default sounds, and have no indication their config was never loaded.
|
||||
If v1.2 adds a `[families]` section (or any new top-level TOML key) to the config schema to support per-family sound configuration, a user who adds this to their TOML file will get an error when running an unpatched v1.1 binary:
|
||||
|
||||
```
|
||||
config: unknown key "families" — check spelling
|
||||
```
|
||||
|
||||
The `parseFile()` function uses `md.Undecoded()` as strict mode, which rejects any key not in `rawConfig`. The `rawConfig` struct only knows about `sounds` and `rules`. Adding `families` to the user's TOML will break the tool for anyone running the v1.1 binary — even if they do not care about family features yet.
|
||||
|
||||
**Why it happens:**
|
||||
Auto-discovery logic is convenient to write as "try these paths, use first found." Developers reuse this logic even for the `--config` code path.
|
||||
|
||||
**Prevention:**
|
||||
Separate the two code paths:
|
||||
- `--config` specified → `os.Open(flagValue)`, return error immediately if `errors.Is(err, os.ErrNotExist)`
|
||||
- No flag → `configSearchPaths()` loop, silently skip missing files, proceed with defaults if none found
|
||||
|
||||
**Detection:**
|
||||
- `--config missing.toml` runs without error, uses defaults
|
||||
- User misses that their config file path has a typo
|
||||
|
||||
**Phase to address:**
|
||||
Config loading phase. A one-line `if flagValue != "" { /* require it */ }` branch is sufficient.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall A9: User Rules That Target the Same Port as Built-in Rules Are Silently Shadowed
|
||||
|
||||
**What goes wrong:**
|
||||
A user writes a rule for `{Protocol: "tcp", DstPort: 443, Class: "my-api"}` intending to reclassify their internal HTTPS traffic. If built-in `ClassHTTPS` still appears before the user rule in the merged slice, the built-in rule wins every time. The user's intent ("I want my port-443 traffic to sound different") is silently defeated.
|
||||
|
||||
**Why it happens:**
|
||||
First-match-wins with `SpecificRules + userRules + CatchAllRules` means built-in specific rules still precede user rules. A user trying to *override* a built-in mapping must replace it, not add after it.
|
||||
This is the correct and intended behavior of `md.Undecoded()` (introduced specifically to prevent silent misconfiguration). But it means the config schema is strictly versioned: any new field must be added to `rawConfig` before any user can write it to their TOML.
|
||||
|
||||
**Consequences:**
|
||||
- User's specific rule is unreachable if a built-in rule covers the same port/protocol
|
||||
- No error, no warning
|
||||
- Functionally the same as Pitfall A5 but for specific (non-catch-all) built-in rules
|
||||
- User adds `[families]` to their config, upgrades their TOML, then tries to run v1.1 binary (e.g., from a build that hasn't shipped yet) — immediate error
|
||||
- More critically: if v1.2 adds `[families]` to `rawConfig` but the user's v1.1 config does not have `[families]` at all — this direction is fine, since absent keys are not reported by `Undecoded()`
|
||||
|
||||
**Prevention:**
|
||||
Two viable strategies:
|
||||
1. **User rules first:** `userRules + specificDefaultRules + catchAllRules`. User rules always take precedence. Built-ins serve as fallback. This is the simplest design and most aligned with user expectations ("I configure what I care about; defaults handle everything else").
|
||||
2. **Conflict detection:** After merging, scan for duplicate `(protocol, dstPort)` pairs and emit a warning: `"User rule for tcp:443 shadows built-in HTTPS rule. Did you mean to replace it?"`.
|
||||
The direction of concern is: user writes v1.2 TOML, runs v1.1 binary. Since this project does not provide v1.1 binary distribution to end users (it's a CLI built from source), this pitfall is primarily about development workflow and test fixtures:
|
||||
|
||||
Option 1 is recommended for simplicity. Document it clearly: "User-defined rules are evaluated before built-in rules."
|
||||
1. Update `rawConfig` struct to include `families` (or whatever the new group config key is named) before any tests or documentation reference the new config format
|
||||
2. Any test fixture `.toml` files should use the schema matching the current binary's `rawConfig` struct
|
||||
3. For documentation examples: do not publish TOML samples containing `[families]` until the code that handles it is shipped
|
||||
|
||||
The v1.1 → v1.2 migration path for the config struct should be:
|
||||
- `rawConfig` adds `Groups map[string]GroupOverride` (or similar) — unknown key validation now accepts it
|
||||
- If absent in user's TOML (the common case): `raw.Groups` is nil — safe, no behavior change
|
||||
- If present: processed in the new merge step
|
||||
|
||||
**Detection:**
|
||||
- User-defined rule for a built-in port (80, 443, 22, etc.) never activates
|
||||
- Verbose output shows built-in class instead of user class for the expected traffic
|
||||
- Test fixture containing new config key causes `config: unknown key` error in a test that still uses old `rawConfig`
|
||||
- `--print-config` output contains new group annotations but user running v1.1 binary sees error
|
||||
|
||||
**Phase to address:**
|
||||
User-defined rules phase, merge strategy design. Address at the same time as Pitfall A5.
|
||||
Config schema extension phase. Update `rawConfig` and `PrintConfig` before any code that generates or consumes the new TOML format.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall A10: New TrafficClass Strings From Config Are Not Validated — Empty String or Whitespace Is a Valid Key
|
||||
## Moderate Pitfalls
|
||||
|
||||
### Pitfall C6: Within-Family Detuning Causes Critical Band Masking at High Frequencies
|
||||
|
||||
**What goes wrong:**
|
||||
A user writes:
|
||||
The v1.2 spec calls for "within-group sound design: shared base frequency, different waveforms or slight detuning." If two classes in the same family are assigned frequencies closer than one critical bandwidth, the human auditory system treats them as a single tone rather than two distinct sounds. The "slightly detuned" design goal backfires: instead of sounding like two related-but-distinct protocols, the two tones merge perceptually into one broader tone with beating artifacts.
|
||||
|
||||
```toml
|
||||
[[rules]]
|
||||
protocol = "tcp"
|
||||
dst_port = 9200
|
||||
class = ""
|
||||
```
|
||||
Critical bandwidth (ERB) formula: `ERB(f) = 24.7 * (4.37 * f/1000 + 1)` Hz.
|
||||
|
||||
The string `""` decodes without error. It is a valid Go map key. It gets inserted into the `WindowSnapshot.Counts` map and the aggregator increments `Counts[""]`. The bank has no layer for `""`. The behavior is undefined — silent or panic depending on nil-guard presence.
|
||||
Concrete values for the NetSynth frequency range:
|
||||
- At 100 Hz: ~35 Hz critical bandwidth (tones must be >35 Hz apart)
|
||||
- At 500 Hz: ~48 Hz critical bandwidth (tones must be >48 Hz apart)
|
||||
- At 1000 Hz: ~72 Hz critical bandwidth (tones must be >72 Hz apart)
|
||||
- At 2000 Hz: ~117 Hz critical bandwidth (tones must be >117 Hz apart)
|
||||
|
||||
Similarly, `class = " elasticsearch "` (padded spaces) decodes to a string with leading/trailing whitespace that does not match any configured sound entry (because the config sound entry key is `"elasticsearch"` without spaces).
|
||||
The v1.1 `autoAssignFreq` used 50 Hz steps in the 1200–2350 Hz range. At 1200 Hz, critical bandwidth is ~88 Hz. A 50 Hz step at that frequency is inside the critical band — the two tones will mask each other.
|
||||
|
||||
**Why it happens:**
|
||||
Frequency step sizes that feel visually reasonable (e.g., 50 Hz) do not scale with the logarithmic nature of human pitch perception. The critical band narrows in Hz as frequency decreases but the absolute Hz separation needed for perceptual distinctness increases with frequency.
|
||||
|
||||
**Consequences:**
|
||||
- Same-family protocols sound identical rather than "related but distinct"
|
||||
- Within-family distinguishability — a core design goal — is not achieved even though the Hz values differ
|
||||
- The bug is hard to detect: `--print-config` shows different Hz values, but the audio is perceptually undifferentiated
|
||||
|
||||
**Prevention:**
|
||||
Validate all `Class` string values from user rules in the `validate()` step:
|
||||
```go
|
||||
if strings.TrimSpace(rule.Class) == "" {
|
||||
return fmt.Errorf("rule %d: class name must not be empty", i)
|
||||
}
|
||||
rule.Class = strings.TrimSpace(rule.Class)
|
||||
```
|
||||
Also validate that class names do not collide with reserved built-in class names (`"ICMP"`, `"DNS"`, etc.) unless the user is explicitly overriding a built-in sound (which is a distinct feature — it should be opt-in, not accidental).
|
||||
Use a logarithmic (musical interval) separation for within-family detuning rather than fixed-Hz steps. A minor third (ratio 1.2) or major second (ratio 1.125) provides psychoacoustically safe separation across the full frequency range used by NetSynth:
|
||||
|
||||
- Family base at 800 Hz, member 2 at 800 * 1.125 = 900 Hz (100 Hz gap, safe)
|
||||
- Family base at 2000 Hz, member 2 at 2000 * 1.125 = 2250 Hz (250 Hz gap, safe vs 117 Hz critical band)
|
||||
|
||||
Rule of thumb for NetSynth protocol family design: within a family, space members at least 1.25x the critical bandwidth of the lower tone. Using a minor second (semitone, ratio 1.059) as the minimum separation gives ~75 Hz at 1300 Hz — marginal. Use at least a major second (ratio 1.122) for reliable perceptual separation.
|
||||
|
||||
**Detection:**
|
||||
- Two protocols in the same family sound identical in listening test
|
||||
- The beating artifact (amplitude modulation at the difference frequency) is audible when two closely-spaced tones are both active
|
||||
|
||||
**Phase to address:**
|
||||
Config validation step.
|
||||
Frequency allocation and family sound design phase. Compute critical bandwidths for all proposed family member frequencies before finalizing the allocation. A short spreadsheet checking `ERB(f) < |f2 - f1|` for each pair catches this before any code is written.
|
||||
|
||||
---
|
||||
|
||||
## v1.0 Pitfalls (Retained for Reference)
|
||||
### Pitfall C7: Adding ~20 Rules to the Rule Slice Does Not Degrade Classification Performance, But Dual-Port Rules Do
|
||||
|
||||
The following pitfalls from the initial MVP research remain valid. They are retained in condensed form for reference.
|
||||
**What goes wrong:**
|
||||
The existing `DefaultRules` slice has 12 entries. Adding 20–30 more for protocols like IMAP (143), IMAPS (993), POP3 (110), LDAP (389), RDP (3389), FTP (21), SIP (5060), SNMP (161), etc., increases the linear scan from ~12 comparisons to ~40 comparisons per packet.
|
||||
|
||||
At 44100 Hz / 22050 SamplesPerWindow = 2 windows/sec, and typical home/office network rates of 1000–5000 packets/sec, the classifier is called ~2500 times/sec. Each call does a linear scan over 40 rules. At ~4 ns per comparison (cache-warm slice iteration), 40-rule scan ≈ 160 ns per packet. For 5000 packets/sec, that is 0.8 ms/sec total classifier CPU — negligible.
|
||||
|
||||
The actual risk is not linear scan overhead but **dual-port rule confusion**: many protocols have both a plain and a TLS/secure variant on different ports (HTTP:80 and HTTPS:443, SMTP:25 and SMTPS:587, IMAP:143 and IMAPS:993, POP3:110 and POP3S:995, LDAP:389 and LDAPS:636). If these are naively placed into separate classes with different frequencies, a mail server using SMTPS at 587 will be classified differently from one using SMTP at 25 — even though they are the same protocol family. The frequency space gets overcrowded with variants that sound like separate protocols but represent the same thing.
|
||||
|
||||
**Why it happens:**
|
||||
The classification decision "same class vs separate classes" for secure and insecure protocol variants is not obvious. The default instinct is to add more rules = more specificity = better, but perceptually the user wants "I can hear that I have mail traffic" not "I can tell the exact TLS variant."
|
||||
|
||||
**Consequences:**
|
||||
- Frequency spectrum crowded with 2x the expected number of mail-related tones
|
||||
- Secure and insecure variants of the same protocol family cancel each other's coherence
|
||||
- The "family" identity becomes invisible — SMTP and SMTPS sound like different protocols
|
||||
|
||||
**Prevention:**
|
||||
Group insecure and secure variants of the same protocol into the same `TrafficClass`:
|
||||
- `ClassSMTPFamily` covers ports 25, 465, 587
|
||||
- `ClassIMAPFamily` covers ports 143, 993
|
||||
- `ClassPOP3Family` covers ports 110, 995
|
||||
|
||||
Add multiple `Rule` entries for the same class (one per port). The classifier already supports this — multiple rules with different ports mapping to the same class are correct. The frequency allocation should then be: one slot per protocol family, not one slot per port.
|
||||
|
||||
```go
|
||||
// In classify/rules.go:
|
||||
{Protocol: "tcp", DstPort: 25, Class: ClassMailSMTP},
|
||||
{Protocol: "tcp", DstPort: 587, Class: ClassMailSMTP},
|
||||
{Protocol: "tcp", DstPort: 465, Class: ClassMailSMTP},
|
||||
```
|
||||
|
||||
**Detection:**
|
||||
- Frequency assignment table has 30+ entries for ~15 conceptual protocol families
|
||||
- Listening test: mail traffic sounds like 3 separate overlapping tones instead of one identifiable mail layer
|
||||
|
||||
**Phase to address:**
|
||||
Protocol list design phase (before any rule code). Define the class list (family → single class name → list of ports) before implementing rules. The family grouping decision should drive the `TrafficClass` constant list.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B1: Using `google/gopacket` Instead of the Active Community Fork
|
||||
### Pitfall C8: `AllClasses()` and `ClassFreqConfigs` Must Both Be Updated Atomically — Two Callsites, Not One
|
||||
|
||||
**What goes wrong:** Import of the unmaintained original — 270 open issues, Go compat degrades.
|
||||
**Prevention:** Import `github.com/gopacket/gopacket` (v1.5.0, requires Go 1.24+).
|
||||
**Phase:** Phase 1 — set correct import path from day one.
|
||||
**What goes wrong:**
|
||||
Adding a new protocol class to NetSynth requires touching three locations:
|
||||
1. A new `ClassXxx TrafficClass = "xxx"` constant in `classify/types.go`
|
||||
2. A new entry in `classify.AllClasses()` return slice in `classify/types.go`
|
||||
3. A new entry in `synth.ClassFreqConfigs` map in `synth/config.go`
|
||||
|
||||
If any one of these is missing:
|
||||
- Missing from `AllClasses()`: `--print-config` does not emit it; `config/config.go`'s `PrintConfig` classifies it as "user-defined" rather than "built-in"; `TestNumLayersMatchesAllClasses` fails
|
||||
- Missing from `ClassFreqConfigs`: `TestAllClassesHaveConfig` fails; `copyDefaults()` does not include it; user can't override it in TOML
|
||||
- Missing constant (using a string literal instead): compiles, but typos create a second unintended class
|
||||
|
||||
When adding 20+ new classes, the three-location update is repeated 20+ times. The likelihood of a missed entry in one location is high.
|
||||
|
||||
**Why it happens:**
|
||||
Go does not have enum types that automatically enforce that a new member must be registered in every relevant collection. The `classify.TrafficClass` type is a `string` type alias — adding a constant does not force updates to `AllClasses()` or `ClassFreqConfigs`.
|
||||
|
||||
**Consequences:**
|
||||
- Test failure that is diagnostic but potentially confusing ("I added the class, why does the test fail?")
|
||||
- Less dangerous but still: `--print-config` shows wrong annotation (user-defined vs built-in) for new classes
|
||||
|
||||
**Prevention:**
|
||||
Write a single source-of-truth Go data structure that drives all three, rather than maintaining them independently:
|
||||
|
||||
```go
|
||||
// In classify/types.go: define the authoritative ordered list
|
||||
var builtinClassDefs = []struct {
|
||||
Class TrafficClass
|
||||
Display string
|
||||
}{
|
||||
{ClassICMP, "ICMP"},
|
||||
// ... all classes ...
|
||||
{ClassMailSMTP, "mail-smtp"},
|
||||
}
|
||||
|
||||
// AllClasses() derives from this:
|
||||
func AllClasses() []TrafficClass {
|
||||
classes := make([]TrafficClass, len(builtinClassDefs))
|
||||
for i, def := range builtinClassDefs {
|
||||
classes[i] = def.Class
|
||||
}
|
||||
return classes
|
||||
}
|
||||
```
|
||||
|
||||
Then `ClassFreqConfigs` can be validated against `AllClasses()` at test time rather than being separately maintained. Adding a new class means updating only `builtinClassDefs` — the rest is derived.
|
||||
|
||||
Alternatively: add a comment above both `AllClasses()` and `ClassFreqConfigs` stating "KEEP IN SYNC — adding a class requires updating both" and rely on the existing `TestNumLayersMatchesAllClasses` test to catch mismatches.
|
||||
|
||||
**Detection:**
|
||||
- `TestNumLayersMatchesAllClasses` fails
|
||||
- `TestAllClassesHaveConfig` fails
|
||||
|
||||
**Phase to address:**
|
||||
First code phase of v1.2 protocol additions, before adding any new classes. Decide on the sync strategy and document it clearly so all 20+ additions follow the same pattern.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B2: CGo Destroys the "Single Binary" Promise
|
||||
### Pitfall C9: Port-Range and Multi-Port Rules Require Protocol Rule Schema Extension
|
||||
|
||||
**What goes wrong:** `gopacket/pcap` (CGo + libpcap) produces a dynamically-linked binary that fails on machines without `libpcap.so`.
|
||||
**Prevention:** Use `packetcap/go-pcap` (pure Go capture, already the chosen stack). Verify with `ldd ./netsynth`.
|
||||
**Phase:** Phase 1 — foundational architecture decision.
|
||||
**What goes wrong:**
|
||||
Some protocols use dynamic or high-number ports that cannot be expressed as a single `DstPort uint16` rule. Examples:
|
||||
- RTP (VoIP media) uses ephemeral UDP ports in a range (typically 16384–32767 or 49152–65535)
|
||||
- mDNS (multicast DNS/discovery) uses UDP port 5353 but also matches on IP multicast addresses
|
||||
- NetBIOS uses ports 137, 138, 139 — three separate ports for the same protocol family
|
||||
|
||||
The current `Rule` struct only supports `{Protocol, DstPort, Class}`. Adding RTP and other range-based protocols cannot be represented without extending the rule schema.
|
||||
|
||||
**Why it happens:**
|
||||
The v1.0 rule design was sufficient for well-known single-port protocols. Port ranges are a natural extension that was not anticipated. Extending the schema now risks breaking the existing TOML rule syntax (`[[rules]]` blocks) that v1.1 users have written.
|
||||
|
||||
**Consequences:**
|
||||
- RTP, mDNS, and other range-based protocols cannot be classified with the current rule model
|
||||
- Attempting to add them as single-port rules misses the vast majority of their traffic
|
||||
- If the rule struct is extended (e.g., `DstPortMin uint16, DstPortMax uint16`), all existing rule-reading code must be updated, and the TOML format changes
|
||||
|
||||
**Prevention:**
|
||||
Decide explicitly which protocols to include in v1.2 scope. If a protocol requires port-range matching, either:
|
||||
1. Exclude it from v1.2 and note it as requiring a rule schema extension in a future milestone
|
||||
2. Implement the range extension in the rule struct first — but verify it does not break existing TOML `[[rules]]` parsing (it should not, since adding optional fields to `RawRule` with pointer types is backward-compatible with existing configs that omit those fields)
|
||||
|
||||
For the v1.2 protocol list, favor protocols with well-known single static ports (IMAP:143, POP3:110, LDAP:389, RDP:3389, etc.) and defer RTP, dynamic SIP media, and NetBIOS-style multi-port protocols to a future "advanced rule types" milestone.
|
||||
|
||||
**Detection:**
|
||||
- RTP traffic appears as `ClassOtherUDP` even after adding a rule
|
||||
- Attempting to write a TOML rule for RTP using a single port produces incorrect results
|
||||
|
||||
**Phase to address:**
|
||||
Protocol list design phase. Before implementation, filter the candidate protocol list to only those expressible with current `{Protocol, DstPort, Class}` semantics, or decide up front to extend the schema and account for the additional complexity.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B3: `CAP_NET_RAW` + Binary Location = Silent Failure on Linux
|
||||
## Minor Pitfalls
|
||||
|
||||
**What goes wrong:** `setcap` is silently ignored on `nosuid` filesystems. Binary appears broken from home directories.
|
||||
**Prevention:** Install to `/usr/local/bin`; document two run modes; emit clear privilege error.
|
||||
**Phase:** Phase 1 + CLI UX.
|
||||
### Pitfall C10: `--print-config` Group Annotations Must Not Break Existing TOML Output Parsing
|
||||
|
||||
**What goes wrong:**
|
||||
`PrintConfig()` in `config/config.go` emits commented TOML that users may use as a template. If v1.2 changes the output format — for example, adding group header comments like `# === Web Family ===` above related classes — and a user pipes `--print-config` output back to a config file, the comments are harmless. However, if `PrintConfig` emits actual TOML key-value pairs for a `[families]` section that the current `rawConfig` struct cannot parse, loading that output as a config file will fail with `unknown key "families"`.
|
||||
|
||||
**Prevention:**
|
||||
All new group-related output in `--print-config` must either be:
|
||||
1. Comments only (lines starting with `#`) — safe, TOML ignores them
|
||||
2. Actual config keys that `rawConfig` can parse — requires adding them to `rawConfig` first
|
||||
|
||||
Never emit a `[families]` or `[groups]` TOML block in `--print-config` output before the corresponding struct field exists in `rawConfig`.
|
||||
|
||||
**Phase to address:** Config output phase.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B4: Packet Buffer Overflow Under Moderate Traffic Load
|
||||
### Pitfall C11: New Class Constants Named Inconsistently With Existing Pattern
|
||||
|
||||
**What goes wrong:** Default capture buffer fills faster than the classifier consumes it; silent packet drops misrepresent traffic.
|
||||
**Prevention:** Large capture buffer (32 MB); buffered channel between capture and classify goroutines.
|
||||
**Phase:** Phase 1/2 (capture pipeline architecture).
|
||||
**What goes wrong:**
|
||||
The existing constants follow `ClassHTTPS`, `ClassSSH`, `ClassNTP` — protocol name in CamelCase. For grouped protocols, if constants are named `ClassMailSMTP`, `ClassMailIMAP`, `ClassMailPOP3`, the "Mail" prefix creates a new naming convention that does not match the flat naming of existing classes. The `TrafficClass` string values (e.g., `"mail-smtp"`, `"mail-imap"`) become the identifiers users reference in TOML — if these are kebab-case with family prefix (e.g., `[sounds.mail-smtp]`), that is a new pattern that does not match existing class names like `[sounds.HTTPS]` (uppercase) or `[sounds.other-TCP]` (mixed case with hyphen).
|
||||
|
||||
**Prevention:**
|
||||
Decide the naming convention for grouped class string values before adding any constants:
|
||||
- Option A: `"smtp"`, `"imap"`, `"pop3"` — flat names, consistent with `"SSH"`, `"DNS"` (but drops family grouping in the config key)
|
||||
- Option B: `"mail-smtp"`, `"mail-imap"` — family-prefixed, makes grouping visible in TOML but is a new pattern
|
||||
|
||||
Existing classes use all-caps for protocols (`"HTTPS"`, `"ICMP"`) and lowercase-hyphenated for non-standard ones (`"other-TCP"`, `"unknown-1"`). New classes should follow the lowercase-hyphenated pattern for multi-word names. Document the convention at the top of `classify/types.go`.
|
||||
|
||||
**Phase to address:** Protocol list design phase, before writing constants.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B5: ZeroCopy Packet Data Use-After-Free
|
||||
### Pitfall C12: Too Many Active Layers Degrades Ambient Distinctness (Perceptual Density Threshold)
|
||||
|
||||
**What goes wrong:** `ZeroCopyReadPacketData()` invalidates previous slice on each call; silent data corruption in concurrent code.
|
||||
**Prevention:** Use `ReadPacketData()` (copying API) unless profiling proves allocation bottleneck.
|
||||
**Phase:** Phase 1 (capture/decode).
|
||||
**What goes wrong:**
|
||||
v1.1 has 14 layers. v1.2 will add 20–30 more, reaching a total of ~35–44 layers. When all layers are simultaneously active at whisper-floor amplitude, the combined output is 35 × WhisperFloor × gainPerLayer = 35 × 0.03 × (1/35) = 3% of max amplitude — still quiet. The whisper floor plus gain-per-layer math continues to work correctly.
|
||||
|
||||
The perceptual problem is different: with 35 simultaneous drone layers, the ambient sound loses definition. Below 5–8 simultaneous distinct tones, listeners can track individual threads. Above 10–12, the output becomes a dense textural wash. This is not a technical bug but a UX risk: the "network fingerprint" value proposition weakens because the output sounds less like "I can identify HTTPS vs SSH" and more like "everything is one undifferentiated cloud."
|
||||
|
||||
**Prevention:**
|
||||
Group protocols into families specifically to mitigate this: a family's members should share enough spectral character (same or nearby frequency, similar waveform) that they fuse into a single perceptible "family layer" rather than adding N separate threads. The "distinct family tone" becomes the perceptual unit, not each individual protocol.
|
||||
|
||||
Additionally, consider whether `WhisperFloor` should be reduced for high layer counts. At 35 layers, 35 × 0.03 × (1/35) = 0.03 (3% amplitude from whisper alone when all active) — which is fine. The math is self-correcting. The concern is purely perceptual richness, not clipping or silence.
|
||||
|
||||
**Phase to address:** Sound design review after all frequencies are assigned. Listening test with a mix of protocols active simultaneously is the definitive check.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B6: MP3 Output Is Corrupt Due to LAME Initialization Order
|
||||
|
||||
**What goes wrong:** Skipping `InitParams()` or setting parameters out of order produces unplayable MP3.
|
||||
**Prevention:** Always call `InitParams()` before writing frames; smoke test with `ffprobe`.
|
||||
**Phase:** Audio synthesis / encoding phase.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B7: PCM Sample Overflow Produces Wrap-Around Distortion
|
||||
|
||||
**What goes wrong:** Summing `int16` layers overflows and wraps (32767 + 100 = -32667), producing buzzing distortion.
|
||||
**Prevention:** Synthesize in `float64 [-1.0, 1.0]`; clamp before int16 cast. Already implemented in `synth/mixer.go`.
|
||||
**Phase:** Audio synthesis (already addressed in v1.0).
|
||||
|
||||
---
|
||||
|
||||
### Pitfall B8: Tone-per-Protocol Mapping Produces Perceptual Chaos
|
||||
|
||||
**What goes wrong:** Frequencies too close together mask each other; output is undifferentiated buzz.
|
||||
**Prevention:** Space protocols across register bands; use harmonic/musical intervals. Already addressed in v1.0.
|
||||
**Phase:** Audio mapping (already addressed in v1.0).
|
||||
|
||||
---
|
||||
|
||||
## v1.1 Phase-Specific Warnings
|
||||
## Phase-Specific Warnings (v1.2)
|
||||
|
||||
| Phase Topic | Likely Pitfall | Mitigation |
|
||||
|-------------|---------------|------------|
|
||||
| TOML struct design | A1: zero-value overwrites defaults | Use pointer fields for all optional overrides |
|
||||
| Config strict decode | A2: typos silently ignored | Use `md.Undecoded()` as strict mode check |
|
||||
| Waveform implementation | A3: naive waveform aliases | Use additive synthesis (bandlimited harmonic series) — compatible with existing `[]HarmonicDef` API |
|
||||
| Waveform string input | A4: case/abbreviation mismatches | Normalize + validate with clear error listing accepted values |
|
||||
| Rule merge ordering (catch-alls) | A5: user rules after catch-alls are unreachable | Split `DefaultRules` into `SpecificRules` + `CatchAllRules`; user rules go in between |
|
||||
| Bank construction | A6: custom class has no synth layer | Derive full class set from merged rule slice; pass to bank constructor |
|
||||
| Config discovery | A7: XDG ignored, `~/.config` hardcoded | Use `os.UserConfigDir()` not `os.UserHomeDir() + "/.config"` |
|
||||
| --config flag path | A8: missing explicit path silently ignored | Two distinct code paths: flag path (require) vs auto-discovery (skip-missing) |
|
||||
| Rule merge ordering (specific built-ins) | A9: user rule shadowed by built-in for same port | User rules first in merged slice (`userRules + specificDefaults + catchAlls`) |
|
||||
| Class name validation | A10: empty/whitespace class name is valid Go string | Validate and trim all class strings in `validate()` |
|
||||
| Frequency spectrum design | C1: existing Hz overrides become stale | Allocate new classes above 1100 Hz; leave v1.1 range frozen |
|
||||
| Frequency spectrum design | C6: within-family tones too close | Enforce >1 critical bandwidth separation; use musical interval ratios |
|
||||
| Auto-assign range update | C2: new built-ins collide with user custom class auto-assign range | Move auto-assign base above highest new built-in Hz; add range-check test |
|
||||
| Test suite update | C4: TestFrequenciesInRange fails on new Hz values | Update range bound in test before adding any class above 1100 Hz |
|
||||
| Protocol list curation | C7: secure + insecure variants fill 2x slots | Decide: one class per family (covering all ports) or one class per variant |
|
||||
| Protocol list curation | C9: RTP and range-based protocols not expressible | Exclude from v1.2 or extend rule schema; decide before writing rules |
|
||||
| Adding class constants | C3: NumLayers stale constant misleads | Remove or document-only; update test name; keep AllClasses+ClassFreqConfigs atomic |
|
||||
| Adding class constants | C8: three-location update forgetting one | Establish single source of truth or add checklist in classify/types.go comment |
|
||||
| Group config schema | C5: new TOML key rejected by old binary | Add to rawConfig struct before documenting or emitting the key anywhere |
|
||||
| --print-config update | C10: output contains unparseable TOML | All group output must be comments only, or rawConfig must accept the new keys |
|
||||
| Class constant naming | C11: naming convention drift | Decide convention once in design phase; document in types.go |
|
||||
| Perceptual density | C12: 35+ layers is a wash | Design family groupings to fuse into ~10 perceptual units, not 35 threads |
|
||||
|
||||
---
|
||||
|
||||
## Integration Gotchas (v1.1 Additions)
|
||||
## Backward Compatibility Summary (v1.1 → v1.2)
|
||||
|
||||
| Integration | Common Mistake | Correct Approach |
|
||||
|-------------|----------------|------------------|
|
||||
| Config → Bank wire-up | Pass `classify.AllClasses()` to bank; custom classes missing | Derive layer set from `Classifier.ActiveClasses()` — all classes reachable via the effective rule set |
|
||||
| Waveform → FreqConfig | Add `WaveformType string` to `FreqConfig`; forget to generate harmonics at bank init | Generate `[]HarmonicDef` from waveform+freq at bank/layer construction time, not at sample render time |
|
||||
| User rules → Classifier | Replace `DefaultRules` var directly; breaks tests relying on it | Keep `DefaultRules` immutable; construct `mergedRules` for runtime use |
|
||||
| Config file absent | Return error if no config found | Return nil (no config = all defaults). Only error on explicit `--config` path that is missing |
|
||||
| Sound overrides for built-in class | User sets freq for "HTTPS" — must hit `ClassHTTPS` layer | Match config sound keys case-insensitively against `TrafficClass` string values; map `"HTTPS"` → `classify.ClassHTTPS` |
|
||||
| Change Type | Impact on v1.1 User Configs | Mitigation |
|
||||
|-------------|----------------------------|------------|
|
||||
| New built-in classes added | None — absent TOML keys silently default; existing overrides unaffected | Safe |
|
||||
| Existing built-in Hz values changed | User overrides silently re-apply old v1.1 Hz values, masking the change | Freeze v1.1 Hz range; do not reassign existing classes |
|
||||
| New top-level TOML key added (e.g., `[families]`) | v1.1 binary rejects config with new key via `Undecoded()` | Acceptable since user controls which binary they run |
|
||||
| Auto-assign range shifted | User custom classes get different Hz values than before | Announce in changelog; update `autoAssignFreq` constants and document |
|
||||
| `[[rules]]` schema extended (port range fields) | Existing rules without new fields: no change (pointer types are nil = absent) | Backward-compatible if new fields are optional pointers |
|
||||
| Class string values renamed | TOML `[sounds.old-name]` silently produces "unknown class" warning (not error) | Do not rename existing class strings |
|
||||
|
||||
---
|
||||
|
||||
## v1.1 Pitfalls (Retained)
|
||||
|
||||
The following pitfalls from v1.1 research remain valid and fully resolved in the codebase. They are retained in condensed form for reference. See the original v1.1 entries for full detail.
|
||||
|
||||
### Pitfall A1: TOML Unmarshal Silently Overwrites Defaults With Zero Values
|
||||
Use pointer fields (`*float64`, `*string`) — implemented in `config/config.go` via `SoundOverride`.
|
||||
|
||||
### Pitfall A2: BurntSushi/toml Silently Ignores Typos
|
||||
Use `md.Undecoded()` — implemented in `parseFile()`.
|
||||
|
||||
### Pitfall A3: Naive Square/Sawtooth/Triangle Produces Aliasing
|
||||
Use bandlimited additive synthesis — implemented in `WaveformPresetHarmonics()`.
|
||||
|
||||
### Pitfall A4: Waveform String Validation Fails Silently
|
||||
Normalize + validate — implemented in `parseWaveform()`.
|
||||
|
||||
### Pitfall A5: User Rules After Catch-All Rules Are Unreachable
|
||||
User rules prepend before built-ins — implemented: user rules inserted first in merged slice.
|
||||
|
||||
### Pitfall A6: User-Defined Classes Have No synth.FreqConfig Entry
|
||||
`addAutoFreqEntries()` handles this — implemented in `config/config.go`.
|
||||
|
||||
### Pitfall A7: Config Auto-Discovery Ignores XDG Variables
|
||||
Use `os.UserConfigDir()` — implemented in `discoverPath()`.
|
||||
|
||||
### Pitfall A8: Explicit --config Flag Does Not Error on Missing File
|
||||
Separate code paths for explicit vs auto-discovery — implemented in `resolvePath()`.
|
||||
|
||||
### Pitfall A9: User Rule Shadowed by Built-in for Same Port
|
||||
User rules evaluate first — implemented via prepend ordering.
|
||||
|
||||
### Pitfall A10: Empty/Whitespace Class Name Is a Valid Go String
|
||||
Validated in `validateRules()`.
|
||||
|
||||
---
|
||||
|
||||
## v1.0 Pitfalls (Retained, Condensed)
|
||||
|
||||
### Pitfall B1: `google/gopacket` (Unmaintained)
|
||||
Use `gopacket/gopacket` v1.5.0.
|
||||
|
||||
### Pitfall B2: CGo Destroys Single Binary
|
||||
Use `packetcap/go-pcap` (pure Go capture).
|
||||
|
||||
### Pitfall B3: CAP_NET_RAW + nosuid Filesystem
|
||||
Install to `/usr/local/bin`; emit clear privilege error.
|
||||
|
||||
### Pitfall B4: Packet Buffer Overflow
|
||||
Large capture buffer (32 MB); buffered channel between goroutines.
|
||||
|
||||
### Pitfall B5: ZeroCopy Packet Use-After-Free
|
||||
Use `ReadPacketData()` (copying API).
|
||||
|
||||
### Pitfall B6: LAME Initialization Order
|
||||
Call `InitParams()` before writing frames.
|
||||
|
||||
### Pitfall B7: PCM Sample Overflow
|
||||
Synthesize in float64 [-1, 1]; clamp before int16 cast.
|
||||
|
||||
### Pitfall B8: Tone-per-Protocol Frequency Masking
|
||||
Space protocols across register bands using musical intervals.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [BurntSushi/toml pkg.go.dev](https://pkg.go.dev/github.com/BurntSushi/toml) — `Undecoded()` strict mode, pointer field behavior, `MetaData` API
|
||||
- [BurntSushi/toml issue #47: Unmarshal with default values](https://github.com/BurntSushi/toml/issues/47) — confirms default-overwrite behavior
|
||||
- [pelletier/go-toml issue #252: Unmarshal overrides origin values if key is omitted](https://github.com/pelletier/go-toml/issues/252) — confirms same behavior in v1; v2 partially resolves
|
||||
- [pelletier/go-toml v2 pkg.go.dev](https://pkg.go.dev/github.com/pelletier/go-toml/v2) — strict decoder mode documentation
|
||||
- [golang/go issue #29960: os: add UserConfigDir](https://github.com/golang/go/issues/29960) — rationale for `os.UserConfigDir()` (XDG-aware)
|
||||
- [WolfSound: Basic Waveforms in Synthesis](https://thewolfsound.com/sine-saw-square-triangle-pulse-basic-waveforms-in-synthesis/) — aliasing and harmonic series for square/saw/triangle
|
||||
- [CCRMA: Alias-Free Digital Synthesis of Classic Analog Waveforms](https://ccrma.stanford.edu/~stilti/papers/blit.pdf) — bandlimited synthesis theory
|
||||
- [McGill Bandlimited Synthesis of Classic Waveforms](https://www.music.mcgill.ca/~gary/307/week5/bandlimited.html) — truncated harmonic series approach
|
||||
- [Teensy Forum: triangle & sawtooth oscillators aliasing](https://forum.pjrc.com/threads/61269-triangle-amp-sawtooth-oscillators-how-to-deal-with-aliasing) — practical aliasing impact at different frequencies
|
||||
- [adrg/xdg package](https://github.com/adrg/xdg) — XDG Base Directory Specification Go implementation (reference; stdlib `os.UserConfigDir()` is sufficient for NetSynth's needs)
|
||||
- Direct inspection of `/home/dev/workspace/yoloyolo/synth/config.go`, `config.go`, `classify/types.go`, `synth/bank.go`, `synth/config_test.go` — codebase analysis is HIGH confidence
|
||||
- [Glasberg & Moore (1990) ERB formula via Wikipedia Critical Band article](https://en.wikipedia.org/wiki/Critical_band) — critical bandwidth values at specific Hz, HIGH confidence
|
||||
- [BurntSushi/toml pkg.go.dev — Undecoded() strict mode](https://pkg.go.dev/github.com/BurntSushi/toml) — unknown key validation behavior, HIGH confidence
|
||||
- [BurntSushi/toml issue #47: default value behavior](https://github.com/BurntSushi/toml/issues/47) — TOML zero-value overwrite behavior, HIGH confidence
|
||||
- [Evanjones.ca: The Unreasonable Effectiveness of Linear Search](https://www.evanjones.ca/linear-search.html) — linear search competitive with map for N < ~100, MEDIUM confidence
|
||||
- [vitalvas.com: Slice vs Map Performance in Golang (2025)](https://blog.vitalvas.com/post/2025/10/03/slice-vs-map-performance-in-golang/) — map outperforms slice at N > 10, MEDIUM confidence; 40-rule estimate of 160 ns is first-principles, not benchmarked
|
||||
- [QSC Blog: Auditory Masking and its Effect on Perception](https://blogs.qsc.com/live-sound/auditory-masking-and-its-effect-on-our-perception-of-sound/) — masking principles, MEDIUM confidence
|
||||
- [Wikipedia: List of TCP and UDP port numbers](https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers) — port numbers for IMAP, POP3, LDAP, RDP, SIP, SNMP, Syslog, HIGH confidence
|
||||
- [RF Wireless World: Well-Known Port Numbers](https://www.rfwireless-world.com/terminology/well-known-port-numbers) — port reference, MEDIUM confidence
|
||||
|
||||
---
|
||||
*Pitfalls research for: NetSynth v1.1 — TOML config, waveform types, user-defined rules*
|
||||
*Updated: 2026-03-26*
|
||||
*Pitfalls research for: NetSynth v1.2 — Extended protocol coverage, grouped sound families*
|
||||
*Updated: 2026-03-27*
|
||||
|
||||
+143
-147
@@ -1,8 +1,8 @@
|
||||
# Technology Stack
|
||||
|
||||
**Project:** NetSynth v1.1 — Custom Sound Mappings
|
||||
**Project:** NetSynth v1.2 — Extended Protocol Coverage with Grouped Sound Families
|
||||
**Researched:** 2026-03-26
|
||||
**Scope:** Additions/changes only. Existing stack (gopacket, go-pcap, go-lame, cobra) is validated and unchanged.
|
||||
**Scope:** Additions/changes only. Existing stack (gopacket, go-pcap, go-lame, cobra, BurntSushi/toml) is validated and unchanged.
|
||||
|
||||
---
|
||||
|
||||
@@ -10,160 +10,125 @@
|
||||
|
||||
| Technology | Version | Status |
|
||||
|------------|---------|--------|
|
||||
| `github.com/gopacket/gopacket` | v1.5.0 | Validated in v1.0, unchanged |
|
||||
| `github.com/packetcap/go-pcap` | v0.0.0-20251215 | Validated in v1.0, unchanged |
|
||||
| `github.com/sjzar/go-lame` | v0.0.9 | Validated in v1.0, unchanged |
|
||||
| `github.com/spf13/cobra` | v1.10.2 | Validated in v1.0, unchanged |
|
||||
| Hand-rolled sine oscillator + EMA | — | Validated in v1.0, extend in place |
|
||||
| Ordered `[]Rule` classifier | — | Validated in v1.0, extend in place |
|
||||
| `github.com/gopacket/gopacket` | v1.5.0 | Validated in v1.0/v1.1, unchanged |
|
||||
| `github.com/packetcap/go-pcap` | v0.0.0-20251215 | Validated in v1.0/v1.1, unchanged |
|
||||
| `github.com/sjzar/go-lame` | v0.0.9 | Validated in v1.0/v1.1, unchanged |
|
||||
| `github.com/spf13/cobra` | v1.10.2 | Validated in v1.0/v1.1, unchanged |
|
||||
| `github.com/BurntSushi/toml` | v1.6.0 | Validated in v1.1, unchanged |
|
||||
| Hand-rolled additive synth + EMA | — | Validated, extend in place |
|
||||
| Ordered `[]Rule` classifier | — | Validated, extend in place |
|
||||
|
||||
---
|
||||
|
||||
## New Dependencies for v1.1
|
||||
## New Dependencies for v1.2
|
||||
|
||||
### TOML Config Parsing
|
||||
**None required.**
|
||||
|
||||
| Technology | Version | Purpose | Why |
|
||||
|------------|---------|---------|-----|
|
||||
| `github.com/BurntSushi/toml` | v1.6.0 | Parse `netsynth.toml` config files | Single-function `toml.Decode()` into a struct. The `MetaData.Undecoded()` method catches unknown keys in user configs — surfacing typos like `frequncy` rather than silently ignoring them. This is the right behavior for a config file tool. v1.6.0 released December 2025, Go 1.18+ required. Zero indirect dependencies. |
|
||||
|
||||
**Version confirmed:** v1.6.0, December 18, 2025, via pkg.go.dev and GitHub releases page.
|
||||
|
||||
**Why not `pelletier/go-toml v2`:** go-toml v2.3.0 (March 2026) is faster but the performance difference is irrelevant — config is read once at startup. go-toml v2's `Strict` mode can detect unknown keys but requires more setup than BurntSushi's `MetaData.Undecoded()`. BurntSushi's API is simpler for this use case and has clearer error message patterns for user-facing config mistakes.
|
||||
|
||||
### Config Auto-Discovery
|
||||
|
||||
No new dependency. Use Go stdlib only:
|
||||
|
||||
```go
|
||||
// Probe order: --config flag > ./netsynth.toml > ~/.config/netsynth/config.toml
|
||||
func findConfigPath(flagValue string) (string, bool) {
|
||||
if flagValue != "" {
|
||||
return flagValue, true
|
||||
}
|
||||
if _, err := os.Stat("./netsynth.toml"); err == nil {
|
||||
return "./netsynth.toml", true
|
||||
}
|
||||
if dir, err := os.UserConfigDir(); err == nil {
|
||||
p := filepath.Join(dir, "netsynth", "config.toml")
|
||||
if _, err := os.Stat(p); err == nil {
|
||||
return p, true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
```
|
||||
|
||||
`os.UserConfigDir()` returns `$XDG_CONFIG_HOME` if set, else `$HOME/.config` on Linux/macOS — confirmed against Go stdlib docs. No third-party XDG library needed.
|
||||
|
||||
### Additional Waveform Types
|
||||
|
||||
No new dependency. Extend the existing `synth.Oscillator` in place.
|
||||
|
||||
Square, sawtooth, and triangle are pure math — each is ~3 lines. The existing oscillator uses a phase accumulator (0.0–1.0 range), which is the right representation for all four waveforms:
|
||||
|
||||
```go
|
||||
// Waveform enum addition to synth package
|
||||
type Waveform int
|
||||
|
||||
const (
|
||||
WaveformSine Waveform = iota
|
||||
WaveformSquare
|
||||
WaveformSawtooth
|
||||
WaveformTriangle
|
||||
)
|
||||
|
||||
// Per-sample generation (replaces math.Sin call in Advance())
|
||||
func sample(phase float64, w Waveform) float64 {
|
||||
switch w {
|
||||
case WaveformSquare:
|
||||
if phase < 0.5 { return 1.0 }
|
||||
return -1.0
|
||||
case WaveformSawtooth:
|
||||
return 2*phase - 1.0
|
||||
case WaveformTriangle:
|
||||
if phase < 0.5 { return 4*phase - 1.0 }
|
||||
return 3.0 - 4*phase
|
||||
default: // WaveformSine
|
||||
return math.Sin(2 * math.Pi * phase)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `Oscillator` struct gains a `Waveform` field; `Advance()` dispatches to `sample()`. Harmonics still work the same way — each harmonic's phase is `phase * ratio`, which maps correctly for all waveform types.
|
||||
All features for extended protocol coverage and grouped sound families can be implemented by extending existing packages in place. No new external dependencies are needed.
|
||||
|
||||
---
|
||||
|
||||
## Installation Delta
|
||||
## gopacket Protocol Decoder Coverage
|
||||
|
||||
```bash
|
||||
# Add only this new dependency
|
||||
go get github.com/BurntSushi/toml@v1.6.0
|
||||
```
|
||||
This is the critical research question for v1.2. The `layers` package in `gopacket/gopacket v1.5.0` is the authoritative source.
|
||||
|
||||
No changes to build flags. `CGO_ENABLED=1` still required for go-lame.
|
||||
### Protocols with Native gopacket Layer Decoders
|
||||
|
||||
These protocols have a dedicated `LayerType` constant and `DecodeFromBytes` implementation in `github.com/gopacket/gopacket/layers`. They auto-register via UDP/TCP port dispatch — `pkt.Layer(layers.LayerTypeSIP)` just works after gopacket decodes the packet.
|
||||
|
||||
| Protocol | LayerType Constant | Port Auto-Registered | Notes |
|
||||
|----------|-------------------|---------------------|-------|
|
||||
| ICMP v4 | `LayerTypeICMPv4` | IP protocol 1 | Already used in v1.0 |
|
||||
| ICMP v6 | `LayerTypeICMPv6` | IP protocol 58 | Already used in v1.0 |
|
||||
| DNS | `LayerTypeDNS` | UDP/TCP 53 | Already used in v1.0 |
|
||||
| DHCP v4 | `LayerTypeDHCPv4` | UDP 67, 68 | Already used in v1.0 |
|
||||
| DHCP v6 | `LayerTypeDHCPv6` | UDP 546, 547 | NEW: can add DHCPv6 classification rule |
|
||||
| NTP | `LayerTypeNTP` | UDP 123 | Already used in v1.0 |
|
||||
| TLS | `LayerTypeTLS` | TCP 443, 636, 989-995, 5061, etc. | Can use to improve HTTPS/SMTPS/LDAPS detection |
|
||||
| SIP | `LayerTypeSIP` | UDP/TCP/SCTP 5060, 5082, 5083 | NEW: native layer decoder available |
|
||||
| RADIUS | `LayerTypeRADIUS` | UDP 1812 | Possible addition for network infra traffic |
|
||||
| SCTP | `LayerTypeSCTP` | IP protocol 132 | Available if needed |
|
||||
| GRE | `LayerTypeGRE` | IP protocol 47 | Tunnel protocol, probably skip |
|
||||
| Modbus TCP | `LayerTypeModbusTCP` | TCP/UDP 502 | Industrial — niche |
|
||||
|
||||
Source: `github.com/gopacket/gopacket/blob/master/layers/layertypes.go` and `layers/ports.go` — confirmed via direct inspection.
|
||||
|
||||
### Protocols WITHOUT gopacket Layer Decoders (Port-Based Classification Only)
|
||||
|
||||
These protocols do NOT have a `LayerType` in gopacket. Classification must use the existing `Rule{Protocol, DstPort, Class}` mechanism — matching by transport protocol + destination port number. This is already how most of the v1.0 rules work (SSH, HTTP, HTTPS, SMTP are all port-based).
|
||||
|
||||
| Protocol | Standard Port(s) | Transport | Classification Approach |
|
||||
|----------|-----------------|-----------|------------------------|
|
||||
| FTP | 21 (control), 20 (data) | TCP | Port-based rule: `{tcp, 21, ClassFTP}` |
|
||||
| IMAP | 143, 993 (TLS) | TCP | Port-based rules: `{tcp, 143}`, `{tcp, 993}` |
|
||||
| POP3 | 110, 995 (TLS) | TCP | Port-based rules: `{tcp, 110}`, `{tcp, 995}` |
|
||||
| SNMP | 161 (queries), 162 (traps) | UDP | Port-based rules: `{udp, 161}`, `{udp, 162}` |
|
||||
| LDAP | 389, 636 (TLS) | TCP | Port-based rules: `{tcp, 389}`, `{tcp, 636}` (note: 636 already hits LayerTypeTLS) |
|
||||
| RDP | 3389 | TCP | Port-based rule: `{tcp, 3389}` |
|
||||
| SMB | 445 (direct), 139 (NetBIOS) | TCP | Port-based rules: `{tcp, 445}`, `{tcp, 139}` |
|
||||
| mDNS | 5353 | UDP | Port-based rule: `{udp, 5353}` — gopacket uses LayerTypeDNS registered on 53, not 5353 |
|
||||
| QUIC / HTTP3 | 443 | UDP | Port-based rule: `{udp, 443}` distinguishes from HTTPS/TLS on TCP 443 |
|
||||
| Telnet | 23 | TCP | Port-based rule: `{tcp, 23}` |
|
||||
| HTTP alt | 8080, 8443 | TCP | Can add as additional Web family rules |
|
||||
|
||||
**mDNS detail:** gopacket's DNS layer registers only on UDP port 53. mDNS on UDP 5353 will decode as raw UDP payload — the existing `hashBucket` fallback handles it. A `{udp, 5353, ClassMDNS}` rule is correct and sufficient for classification without needing any layer decoder.
|
||||
|
||||
**QUIC detail:** QUIC uses UDP port 443 (same port HTTPS uses on TCP). The existing `{tcp, 443, ClassHTTPS}` rule only fires on TCP. A `{udp, 443, ClassQUIC}` rule is unambiguous — UDP 443 is QUIC/HTTP3 traffic on modern networks. No deep packet inspection needed for classification purposes.
|
||||
|
||||
**SIP detail:** gopacket v1.5.0 has a native SIP decoder (`LayerTypeSIP`) registered on UDP/TCP 5060. This means `pkt.Layer(layers.LayerTypeSIP)` works after gopacket decodes the packet. However, since the existing classifier already dispatches by transport + port via the `Rule` struct, a simple `{udp, 5060, ClassSIP}` / `{tcp, 5060, ClassSIP}` rule pair is simpler and more consistent than adding a special Layer-based code path. Use port-based rules. The native SIP layer decoder is available if future features need SIP message parsing (call rates, request types), but v1.2 only needs classification.
|
||||
|
||||
---
|
||||
|
||||
## Integration Points
|
||||
## In-Place Extensions Required
|
||||
|
||||
### Where Config Feeds Existing Code
|
||||
### 1. classify package — New TrafficClass constants and DefaultRules
|
||||
|
||||
The TOML config needs to override two existing data structures:
|
||||
Add new `TrafficClass` constants to `classify/types.go` for each new protocol. Extend `classify/rules.go` `DefaultRules` with new ordered entries.
|
||||
|
||||
1. **`synth.ClassFreqConfigs`** (map in `synth/config.go`) — user can override `BaseHz` and add a `Waveform` field per class
|
||||
2. **`classify.DefaultRules`** (slice in `classify/rules.go`) — user can prepend custom rules before the defaults
|
||||
|
||||
The config loader should apply overrides at startup before any other initialization. The cleanest integration is:
|
||||
**Proposed new classes by family:**
|
||||
|
||||
```
|
||||
cmd/netsynth/main.go
|
||||
-> config.Load(path) // returns *AppConfig
|
||||
-> classify.MergeRules(cfg) // prepend user rules to DefaultRules
|
||||
-> synth.ApplyOverrides(cfg) // patch ClassFreqConfigs entries
|
||||
Mail family: ClassIMAP, ClassPOP3, ClassSMTPS (SMTP over TLS = 465/587)
|
||||
Web family: ClassHTTP (existing), ClassHTTPS (existing), ClassHTTP8080, ClassQUIC
|
||||
Remote family: ClassSSH (existing), ClassRDP, ClassTelnet
|
||||
Discovery: ClassMDNS, ClassDHCP (existing), ClassDHCPv6
|
||||
File Transfer: ClassFTP
|
||||
Directory: ClassLDAP
|
||||
Monitoring: ClassSNMP
|
||||
Messaging: ClassSIP
|
||||
Infra: ClassSMB
|
||||
```
|
||||
|
||||
Both `classify.DefaultRules` and `synth.ClassFreqConfigs` are currently package-level vars — they can be replaced or cloned at startup without changing the downstream pipeline.
|
||||
The exact set is a product decision (FEATURES.md), but every entry requires only a new `TrafficClass` string constant and a `Rule{Protocol, DstPort, Class}` entry in `DefaultRules`. No code path changes needed.
|
||||
|
||||
### TOML Struct Shape
|
||||
**Insertion point in DefaultRules:** New rules must come before the existing catch-alls (`{tcp, 0, ClassOtherTCP}` and `{udp, 0, ClassOtherUDP}`). Ordering within the new rules does not matter since they are distinct ports.
|
||||
|
||||
The config schema maps naturally to the existing types:
|
||||
### 2. synth package — Frequency map and group detuning
|
||||
|
||||
```toml
|
||||
# netsynth.toml
|
||||
[[rules]]
|
||||
protocol = "tcp"
|
||||
dst_port = 8443
|
||||
class = "my-https-alt"
|
||||
Extend `synth/config.go` `ClassFreqConfigs` with an entry for each new `TrafficClass`. No API change — it's a map addition.
|
||||
|
||||
[sounds.my-https-alt]
|
||||
frequency = 195.0
|
||||
waveform = "square"
|
||||
**Group-based frequency allocation approach (no new code needed):**
|
||||
|
||||
[sounds.ICMP]
|
||||
frequency = 80.0 # override built-in
|
||||
waveform = "triangle"
|
||||
```
|
||||
Group related protocols into a frequency band, using slight detuning within the band for distinction. The existing `FreqConfig.BaseHz` + `FreqConfig.Harmonics` already supports this — give family members adjacent base frequencies (e.g., 5-15 Hz apart at low frequencies, 15-30 Hz at mid frequencies) with the same harmonic shape but different waveform types.
|
||||
|
||||
Example for Mail family:
|
||||
```go
|
||||
type AppConfig struct {
|
||||
Rules []RuleConfig `toml:"rules"`
|
||||
Sounds map[string]SoundConfig `toml:"sounds"`
|
||||
}
|
||||
|
||||
type RuleConfig struct {
|
||||
Protocol string `toml:"protocol"`
|
||||
DstPort uint16 `toml:"dst_port"`
|
||||
Class string `toml:"class"`
|
||||
}
|
||||
|
||||
type SoundConfig struct {
|
||||
Frequency float64 `toml:"frequency"`
|
||||
Waveform string `toml:"waveform"` // "sine"|"square"|"sawtooth"|"triangle"
|
||||
}
|
||||
ClassSMTP: {BaseHz: 440.0, Harmonics: ...sawtooth..., Pan: -0.55} // existing
|
||||
ClassIMAP: {BaseHz: 450.0, Harmonics: ...sawtooth..., Pan: 0.55} // same family, detuned +10 Hz
|
||||
ClassPOP3: {BaseHz: 435.0, Harmonics: ...sawtooth..., Pan: -0.3} // same family, detuned -5 Hz
|
||||
```
|
||||
|
||||
Use `toml.Decode()` and check `meta.Undecoded()` to warn on unknown keys.
|
||||
The `WaveformType` field already encodes "same character within group." The existing bandlimited synthesis code handles all this correctly.
|
||||
|
||||
**NumLayers constant:** Currently hardcoded to 14 in `synth/config.go`. Must be updated to reflect the new total class count. Alternatively, compute it dynamically from `len(ClassFreqConfigs)`. The dynamic approach is more maintainable and requires touching only `synth/config.go`.
|
||||
|
||||
**GainPerLayer:** Computed as `1.0 / float64(NumLayers)`. With more layers active simultaneously, individual gain drops. This is the correct behavior — prevents clipping. Verify mix levels after adding classes.
|
||||
|
||||
### 3. config package — --print-config output
|
||||
|
||||
`--print-config` currently emits commented TOML grouped by class. With protocol families, adding a `Group` field to `FreqConfig` or a separate group-to-classes mapping in `synth/config.go` allows `--print-config` to emit sections with comment headers like `# Mail family`. This is cosmetic; no behavioral change needed.
|
||||
|
||||
No new dependency needed. Add a `GroupName string` field to `FreqConfig` (zero value = ungrouped) or a `var ClassGroups = map[string][]TrafficClass{...}` in `synth/config.go`.
|
||||
|
||||
---
|
||||
|
||||
@@ -171,20 +136,48 @@ Use `toml.Decode()` and check `meta.Undecoded()` to warn on unknown keys.
|
||||
|
||||
| Avoid | Why | What to Do Instead |
|
||||
|-------|-----|-------------------|
|
||||
| `adrg/xdg` or any XDG library | `os.UserConfigDir()` in stdlib already handles `$XDG_CONFIG_HOME` on Linux — confirmed | Use `os.UserConfigDir()` directly |
|
||||
| `pelletier/go-toml v2` | No advantage over BurntSushi for a single-file startup read; `MetaData.Undecoded()` in BurntSushi is more ergonomic for typo detection | `github.com/BurntSushi/toml` |
|
||||
| `spf13/viper` | Massive dependency (brings in 20+ transitive deps) for a use case that is one TOML file — Viper adds remote config, env var binding, hot reload, none of which are needed | `BurntSushi/toml` + manual flag override |
|
||||
| Any waveform/audio library | Square/sawtooth/triangle are 3 lines of math each; no library adds value | Extend `synth.Oscillator` in place |
|
||||
| `gopkg.in/yaml.v3` or JSON config | TOML is explicitly specified for this milestone and is the right format for user-editable config files (comments supported, less noisy than JSON) | TOML only |
|
||||
| Any deep packet inspection library (gopacket TLS layer for HTTPS detection) | v1.2 goal is protocol family classification by port, not payload analysis. TLS handshake parsing adds complexity for no classification benefit since port is unambiguous. | Port-based `Rule{tcp, 443, ClassHTTPS}` — already working |
|
||||
| `github.com/google/gopacket` (original) | Superseded by community fork; 270 open issues, not maintained | `gopacket/gopacket v1.5.0` (already in use) |
|
||||
| Any SNMP library (e.g., `gosnmp`) | v1.2 only needs to detect SNMP traffic, not decode OIDs or walk MIBs | `{udp, 161, ClassSNMP}` port rule |
|
||||
| Any SIP parsing library | v1.2 only needs to detect SIP presence for sonification, not parse SIP messages, headers, or call state | `{udp, 5060, ClassSIP}` + `{tcp, 5060, ClassSIP}` port rules |
|
||||
| Separate "group" abstraction layer in classify | A `Group` field on `FreqConfig` (in synth) is sufficient for --print-config display. The classifier itself doesn't need to know about groups — families emerge from frequency proximity in the audio output. | `GroupName string` in `synth.FreqConfig` |
|
||||
| Dynamic port range rules (e.g., "all TCP 1024-65535 → ClassOtherTCP") | Existing catch-alls (`DstPort: 0`) already cover this. Current Rule struct is optimized for exact-match dispatch. | Keep existing catch-all rules |
|
||||
|
||||
---
|
||||
|
||||
## Version Compatibility
|
||||
## Frequency Rebalancing Scope
|
||||
|
||||
Current v1.1 spectrum allocation (for reference):
|
||||
|
||||
```
|
||||
65 Hz — ICMP
|
||||
110 Hz — DNS
|
||||
175 Hz — HTTPS
|
||||
220 Hz — HTTP
|
||||
330 Hz — SSH
|
||||
440 Hz — SMTP
|
||||
520 Hz — NTP
|
||||
600 Hz — DHCP
|
||||
700 Hz — OtherTCP
|
||||
780 Hz — OtherUDP
|
||||
862 Hz — Unknown-1 (dissonant band)
|
||||
920 Hz — Unknown-2
|
||||
981 Hz — Unknown-3
|
||||
1047 Hz — Unknown-4
|
||||
```
|
||||
|
||||
Adding ~8-12 new protocol classes requires rebalancing. The 65-780 Hz "known protocol" band currently has 8 classes spread over ~715 Hz (average spacing ~90 Hz). Adding 8+ new entries will compress that to ~40-50 Hz average spacing — still audibly distinct with different waveforms.
|
||||
|
||||
The unknown-1-4 dissonant band (862-1047 Hz) should stay — it provides the "something unknown" sound character. The rebalancing task is purely a `synth/config.go` constant edit, not a code change.
|
||||
|
||||
---
|
||||
|
||||
## Version Compatibility (Unchanged)
|
||||
|
||||
| Package | Version | Compatible With | Notes |
|
||||
|---------|---------|-----------------|-------|
|
||||
| `BurntSushi/toml` | v1.6.0 | Go 1.18+ | No issues with Go 1.24 |
|
||||
| `os.UserConfigDir()` | stdlib | Go 1.13+ | Returns `$XDG_CONFIG_HOME` or `$HOME/.config` on Linux |
|
||||
| `gopacket/gopacket` | v1.5.0 | Go 1.24+ | New protocol rules use existing API — no compat concerns |
|
||||
| All other existing packages | (unchanged) | (unchanged) | No updates needed |
|
||||
|
||||
---
|
||||
|
||||
@@ -192,24 +185,27 @@ Use `toml.Decode()` and check `meta.Undecoded()` to warn on unknown keys.
|
||||
|
||||
| Area | Confidence | Source |
|
||||
|------|------------|--------|
|
||||
| BurntSushi/toml v1.6.0 version | HIGH | pkg.go.dev confirmed, GitHub releases confirmed |
|
||||
| `os.UserConfigDir()` XDG behavior | HIGH | Official Go stdlib docs at pkg.go.dev/os |
|
||||
| Waveform math (no library needed) | HIGH | Trivial math, Dylan Meeus Go audio blog confirms the same approach |
|
||||
| go-toml v2.3.0 version | HIGH | pkg.go.dev confirmed |
|
||||
| Recommendation of BurntSushi over go-toml v2 | MEDIUM | Based on API ergonomics for the specific `Undecoded()` use case; both would work |
|
||||
| gopacket LayerType SIP exists at v1.5.0 | HIGH | Direct inspection of `layers/layertypes.go` and `layers/sip.go` via GitHub |
|
||||
| gopacket LayerType TLS exists at v1.5.0 | HIGH | Direct inspection of `layers/layertypes.go` and `layers/ports.go` via GitHub |
|
||||
| gopacket port registrations (ports.go) | HIGH | Direct inspection of `layers/ports.go` via GitHub; explicit list of pre-registered UDP/TCP ports |
|
||||
| mDNS NOT registered in gopacket layers | HIGH | Port 5353 absent from `layers/ports.go` pre-registration list; confirmed via GitHub |
|
||||
| QUIC NOT registered in gopacket layers | HIGH | No `quic.go` in layers directory; no port 443 UDP registration in `layers/ports.go` |
|
||||
| SNMP, LDAP, RDP, SMB, FTP, IMAP, POP3 NOT in gopacket layers | HIGH | No corresponding .go files found in layers directory |
|
||||
| Port-based Rule classification sufficiency for all new protocols | HIGH | All protocols have well-known IANA port assignments; existing Rule struct handles them identically to SSH/HTTP/SMTP |
|
||||
| No new external dependencies needed | HIGH | All new functionality is data additions (constants, map entries) to existing packages |
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- `pkg.go.dev/github.com/BurntSushi/toml` — v1.6.0 confirmed, December 18, 2025
|
||||
- `github.com/BurntSushi/toml/releases` — v1.6.0 release notes, TOML 1.1 enabled by default
|
||||
- `pkg.go.dev/github.com/pelletier/go-toml/v2` — v2.3.0 confirmed, March 24, 2026
|
||||
- `pkg.go.dev/os#UserConfigDir` — XDG_CONFIG_HOME behavior on Linux confirmed via official Go docs
|
||||
- `dylanmeeus.github.io/posts/audio-from-scratch-pt8/` — Go waveform synthesis from scratch, confirms no library needed
|
||||
- `github.com/golang/go/issues/76320` — UserConfigDir XDG_CONFIG_HOME discussion (Nov 2025), confirms existing stdlib support on Linux
|
||||
- `github.com/gopacket/gopacket/blob/master/layers/layertypes.go` — LayerTypeSIP (id 133), LayerTypeTLS (id 140) confirmed
|
||||
- `github.com/gopacket/gopacket/blob/master/layers/sip.go` — SIP decoder implementation confirmed
|
||||
- `github.com/gopacket/gopacket/blob/master/layers/ports.go` — UDP/TCP port pre-registration list; mDNS (5353), SNMP (161/162), QUIC (UDP 443) absent; SIP (5060, 5082, 5083) present
|
||||
- `github.com/gopacket/gopacket/tree/master/layers` — directory listing; no mdns.go, quic.go, snmp.go, ldap.go, smb.go, rdp.go, ftp.go, imap.go, or pop3.go files
|
||||
- `pkg.go.dev/github.com/gopacket/gopacket/layers` — package index confirming layer types
|
||||
- IANA port assignments — standard reference for FTP/21, IMAP/143, POP3/110, SNMP/161, LDAP/389, RDP/3389, SMB/445, mDNS/5353, SIP/5060, QUIC/UDP-443
|
||||
|
||||
---
|
||||
|
||||
*Stack research for: NetSynth v1.1 — Custom Sound Mappings milestone*
|
||||
*Stack research for: NetSynth v1.2 — Extended Protocol Coverage with Grouped Sound Families*
|
||||
*Researched: 2026-03-26*
|
||||
|
||||
+113
-118
@@ -1,187 +1,182 @@
|
||||
# Project Research Summary
|
||||
|
||||
**Project:** NetSynth v1.1 — Custom Sound Mappings
|
||||
**Domain:** Network traffic sonification CLI tool (Go)
|
||||
**Researched:** 2026-03-26
|
||||
**Project:** NetSynth v1.2 — Extended Protocol Coverage with Grouped Sound Families
|
||||
**Domain:** Network traffic sonification CLI (Go) — packet capture to ambient MP3
|
||||
**Researched:** 2026-03-27
|
||||
**Confidence:** HIGH
|
||||
|
||||
## Executive Summary
|
||||
|
||||
NetSynth v1.1 extends the working v1.0 CLI by adding user-customizable sound mappings through a TOML config file. The v1.0 codebase is a clean 6-package Go project (3,254 lines) with a well-separated pipeline: capture → classify → aggregate → synthesize → encode. The v1.1 milestone threads a new `config` package through this pipeline, enabling users to override frequencies and waveforms per traffic class, add new classification rules, and reference those custom classes in the synth layer. The recommended implementation path is incremental — introduce waveform types at the oscillator level first, then decouple bank construction from the global config, then add the TOML loader and wire it all together. Each step is independently testable before the next begins.
|
||||
NetSynth v1.2 extends an already-shipped Go CLI tool that captures live network traffic and synthesizes it into an ambient MP3 soundscape. The existing v1.1 codebase (~4,675 lines, 7 packages) provides a validated pipeline: go-pcap capture → port-based classification → EMA-smoothed additive synthesis → LAME MP3 encoding. The v1.2 milestone adds ~21 new protocol classes across 7 sound families (Mail, Remote Access, File Transfer, Infrastructure expansion, Database, Directory/Auth, VoIP), expanding from 14 total classes to ~35. No new external dependencies are required — every new protocol is detectable via the existing port-based Rule classifier, and group identity is expressed purely through frequency proximity and waveform consistency in `ClassFreqConfigs`.
|
||||
|
||||
The stack requires one new dependency: `github.com/BurntSushi/toml` v1.6.0 (zero indirect deps, `MetaData.Undecoded()` provides strict-mode typo detection). All other additions — waveform math, config file discovery, and class name validation — are stdlib-only. The single notable technical choice is waveform synthesis strategy: naive direct-math square/sawtooth/triangle waveforms produce audible aliasing at the frequencies NetSynth uses (65–1047 Hz). The existing additive synthesis infrastructure (`[]HarmonicDef`) is the correct approach, generating bandlimited harmonic series for each waveform type rather than direct time-domain computation.
|
||||
The recommended approach is strictly additive: extend three locations in the codebase (`classify/types.go` for constants, `classify/rules.go` for port rules, `synth/config.go` for frequency configs) in a fixed order to keep tests green throughout. The highest-value protocols for v1.2 are the Tier 1 set (IMAP, POP3, FTP, SMB, RDP, mDNS, SSDP, SNMP, MySQL, PostgreSQL, Redis) because they appear on nearly every network type. The group-as-frequency-proximity design means family identity emerges from the audio itself — no new data structures are needed for the group concept beyond a `Group string` metadata field on `FreqConfig`.
|
||||
|
||||
The highest-risk integration points are bank construction (which must be extended to handle user-defined classes not in the static `AllClasses()` list) and TOML merge semantics (TOML decoders zero-out absent fields, silently overwriting defaults unless pointer fields are used). Both are well-understood problems with clear prevention patterns that must be established before wiring config into the pipeline. The merge ordering for classification rules also requires deliberate design: user rules must precede specific built-in rules, which must precede catch-alls — three-layer ordering, not two.
|
||||
The key risks are all backward-compatibility and perceptual-design concerns rather than engineering complexity. The critical risks are: (1) frequency rebalancing silently invalidating existing user TOML configs, (2) the `autoAssignFreq` range in `config.go` colliding with new built-in frequencies if not updated, and (3) within-family detuning that is psychoacoustically too narrow to be perceptually distinct. These are all preventable with explicit design choices made before any code is written. The milestone complexity is LOW — all changes are data additions to existing packages, and the build order is clear from the architecture research.
|
||||
|
||||
## Key Findings
|
||||
|
||||
### Recommended Stack
|
||||
|
||||
The existing stack (gopacket v1.5.0, packetcap/go-pcap, sjzar/go-lame v0.0.9, cobra v1.10.2, hand-rolled sine oscillator + EMA) is unchanged. The only new external dependency is `BurntSushi/toml` v1.6.0, selected over `pelletier/go-toml v2` because `MetaData.Undecoded()` is more ergonomic for typo detection on a single startup config read, and over `spf13/viper` because Viper pulls in 20+ transitive deps for features (remote config, env var binding, hot reload) that are irrelevant here. See `.planning/research/STACK.md` for full alternatives analysis.
|
||||
The v1.2 stack is identical to v1.1 — no new dependencies are required. All existing packages (gopacket/gopacket v1.5.0, packetcap/go-pcap, sjzar/go-lame v0.0.9, spf13/cobra v1.10.2, BurntSushi/toml v1.6.0) are validated and unchanged. The gopacket `layers` package has native decoders for SIP (LayerTypeSIP, id 133) and TLS (LayerTypeTLS, id 140) at v1.5.0, but port-based `Rule` matching is the correct and simpler approach for all v1.2 protocols — using native layer decoders would add code paths without classification benefit, since port numbers are unambiguous for every protocol in scope.
|
||||
|
||||
**Core technologies:**
|
||||
- `github.com/gopacket/gopacket` v1.5.0 — packet decode — only maintained Go packet library, Go 1.24+
|
||||
- `github.com/packetcap/go-pcap` — pure-Go live capture backend — no CGo, mmap ring buffer, Linux/macOS
|
||||
- `github.com/sjzar/go-lame` v0.0.9 — MP3 encoding — embeds LAME C source, CGO_ENABLED=1, no system library required
|
||||
- `github.com/spf13/cobra` v1.10.2 — CLI structure and flag handling — industry standard
|
||||
- `github.com/BurntSushi/toml` v1.6.0 — TOML config parsing — zero transitive deps, strict-mode via `Undecoded()`
|
||||
- Hand-rolled additive oscillator (sine today, square/sawtooth/triangle in v1.1) — no audio library needed
|
||||
**Core technologies (unchanged from v1.1):**
|
||||
- `gopacket/gopacket v1.5.0`: Packet capture and protocol layer decoding — community fork, actively maintained, Go 1.24+
|
||||
- `packetcap/go-pcap`: Pure-Go live capture via mmap ring buffer — no CGo, Linux/macOS
|
||||
- `sjzar/go-lame v0.0.9`: MP3 encoding with embedded LAME C source — no system library dependency
|
||||
- `spf13/cobra v1.10.2`: CLI flags, signal handling, --help generation
|
||||
- `BurntSushi/toml v1.6.0`: TOML config loading with strict unknown-key validation
|
||||
- Hand-rolled additive synth + EMA: oscillators with bandlimited harmonics, exponential moving average amplitude smoothing per layer
|
||||
|
||||
### Expected Features
|
||||
|
||||
**Must have (table stakes) for v1.1:**
|
||||
- TOML config auto-discovery (`./netsynth.toml`, `~/.config/netsynth/config.toml`) — XDG Base Directory Specification standard
|
||||
- `--config` flag for explicit path, with hard error if file is absent
|
||||
- Partial override semantics — absent keys retain defaults; users must not replicate the full table to change one field
|
||||
- Custom frequency per traffic class — direct override of `synth.ClassFreqConfigs`
|
||||
- Custom waveform per traffic class — sine/square/sawtooth/triangle selection
|
||||
- User-defined classification rules with custom class names, prepended before built-in rules
|
||||
- Startup-time config validation with line-number errors (fail before capture begins, not after)
|
||||
- Unknown field detection — prevents silent typos (`frequncy` must be caught, not silently ignored)
|
||||
**Must have (table stakes for v1.2):**
|
||||
- Mail family: IMAP/IMAPS (TCP 143, 993), POP3/POP3S (TCP 110, 995), SMTP submission (TCP 587, 465) — present on every office and home network
|
||||
- Remote Access expansion: RDP (TCP 3389), VNC (TCP 5900), Telnet (TCP 23) — completes the SSH family
|
||||
- File Transfer family: FTP (TCP 20, 21), SMB (TCP 445), TFTP (UDP 69) — ubiquitous on NAS and Windows networks
|
||||
- Infrastructure expansion: mDNS (UDP 5353), SSDP (UDP 1900), SNMP (UDP 161/162), Syslog (UDP 514) — constant background on all LAN segments
|
||||
- Database family: MySQL (TCP 3306), PostgreSQL (TCP 5432), Redis (TCP 6379), MongoDB (TCP 27017) — all currently land in other-TCP
|
||||
- Frequency allocation into family bands (group concept expressed via Hz proximity, not a new data structure)
|
||||
- AllClasses() and ClassFreqConfigs updated atomically — existing test `TestNumLayersMatchesAllClasses` enforces this invariant
|
||||
- --print-config reflects all new classes, organized with group-header comments
|
||||
|
||||
**Should have (differentiators):**
|
||||
- `netsynth --print-config` subcommand dumping effective config as commented TOML — critical for discoverability
|
||||
- Named custom rules (display name appears in exit summary and `--verbose` output)
|
||||
- Clear error message listing valid waveform values on invalid input
|
||||
**Should have (differentiators for v1.2):**
|
||||
- Directory/Auth family: LDAP/LDAPS (TCP 389, 636), Kerberos (UDP/TCP 88) — every enterprise/Windows network
|
||||
- VoIP family: SIP/SIP-TLS (UDP/TCP 5060, 5061) — IP phone traffic on office networks
|
||||
- QUIC/HTTP3 class (UDP 443) — distinct from HTTPS on TCP 443; significant fraction of modern web traffic
|
||||
- Within-family waveform consistency: protocols in a family share the same waveform type for timbral family identity
|
||||
- Group-header section comments in --print-config output (cosmetic, high value for user discoverability)
|
||||
|
||||
**Defer (v2+):**
|
||||
- Harmonic override per class (expose `HarmonicDef` slice in TOML) — niche, adds TOML nesting complexity
|
||||
- Stereo pan position in config — explicitly deferred per project constraints
|
||||
- Config hot-reload during capture — mid-capture state change corrupts synthesis; not worth the complexity
|
||||
- Multiple config file includes/inheritance — single file merged with in-code defaults is sufficient
|
||||
- Dynamic port protocols: RTP (negotiated ephemeral ports), FTP data channel — require stateful flow tracking across packets
|
||||
- Deep packet inspection for application-layer classification — massive scope, no sonification benefit over port matching
|
||||
- Collapsed "Mail" or "Database" class — loses per-protocol identity (can't distinguish IMAP inbound from SMTP outbound)
|
||||
- MQTT, AMQP, Kafka, BGP, OSPF — IoT/routing protocols absent on general networks; niche
|
||||
- Port-range rule support in the Rule struct — needed for RTP; valid v1.3 enhancement
|
||||
- Runtime group concept as a data structure — groups emerge from frequency proximity; no struct needed in v1.2
|
||||
|
||||
### Architecture Approach
|
||||
|
||||
v1.1 adds a `config` package and threads it through the existing pipeline via dependency injection. The key architectural shift: `synth.NewBank` currently reads the package-level global `ClassFreqConfigs`; after v1.1 it accepts a `map[TrafficClass]FreqConfig` parameter, enabling user-defined classes and eliminating hidden global state. The `config` package owns TOML parsing, file discovery, default config wrapping, and merge logic. `classify.DefaultRules` splits into `SpecificRules` + `CatchAllRules` so user rules can be inserted between them. The oscillator gains a `Waveform` enum field with additive-synthesis dispatch. All changes are contained to well-bounded components; `classify/classifier.go` and `encode/mp3.go` change only at their call sites. See `.planning/research/ARCHITECTURE.md` for full data flow diagrams and step-by-step build order.
|
||||
The v1.2 architecture is strictly additive to the existing pipeline: `config.Load` → `classify.NewClassifier` → `encode.RunSynthesis` → `synth.NewBank` → MP3. The only new concept is `Group string` on `FreqConfig` in the synth package — this metadata field is used only by `PrintConfig` for section headers and has zero effect on synthesis math. `bank.go`, `encode/mp3.go`, and `main.go` require no changes. Group identity is expressed architecturally through frequency proximity alone: protocols in the same family are assigned `BaseHz` values within a shared band, and within-band detuning of at least a major second interval (ratio 1.122) ensures psychoacoustic distinctness while maintaining timbral family coherence.
|
||||
|
||||
**Major components:**
|
||||
1. `config/` (new) — TOML struct definitions, `Load()`, auto-discovery via `os.UserConfigDir()`, pointer-field merge, `DefaultConfig()`
|
||||
2. `synth/oscillator.go` (modified) — `Waveform` enum, `sampleAt()` dispatch, bandlimited harmonic series generation at config load time
|
||||
3. `synth/bank.go` (modified) — accepts freq config map param; iterates map keys, not hardcoded `AllClasses()`
|
||||
4. `classify/rules.go` (modified) — split into `SpecificRules` + `CatchAllRules`; `MergeRules(userRules)` export
|
||||
5. `cmd/netsynth/main.go` (modified) — `--config` flag, `config.Load()`, user rule merge, config forwarded to `RunSynthesis`
|
||||
**Major components and their v1.2 changes:**
|
||||
1. `classify/types.go` — new TrafficClass constants + AllClasses() extended in group-coherent order (additive)
|
||||
2. `classify/rules.go` — new Rule entries before catch-alls for all new protocols; multi-port-to-single-class mapping for secure/insecure variants (additive; ordering is critical)
|
||||
3. `synth/config.go` — Group field on FreqConfig; new ClassFreqConfigs entries; frequency rebalancing for family bands; NumLayers constant cleanup (highest-risk change due to backward compatibility)
|
||||
4. `config/config.go` — optional group-header comments in PrintConfig (isolated to string output, no behavioral change)
|
||||
5. `bank.go`, `encode/mp3.go`, `main.go` — no changes required
|
||||
|
||||
**Recommended build order (each step independently testable):**
|
||||
1. Test and constant cleanup — remove stale NumLayers/GainPerLayer, update TestFrequenciesInRange bounds
|
||||
2. Protocol list and frequency design (no code) — finalize all Hz values in family bands, check ERB constraints
|
||||
3. New TrafficClass constants + AllClasses() in classify/types.go
|
||||
4. New DefaultRules in classify/rules.go (depends on step 3)
|
||||
5. Add Group field to FreqConfig — independent of steps 3/4
|
||||
6. Add ClassFreqConfigs entries with finalized frequencies; update autoAssignFreq base
|
||||
7. PrintConfig group-header comments (optional polish)
|
||||
|
||||
### Critical Pitfalls
|
||||
|
||||
1. **TOML decoder zeros absent fields, silently overwriting defaults (Pitfall A1)** — Use pointer fields (`*float64`, `*string`) for all optional overrides in the decoded struct. Apply an explicit merge function that only writes non-nil values over the built-in defaults. Establish this pattern before any config is wired into the bank.
|
||||
1. **Frequency rebalancing silently invalidates v1.1 user TOML configs** — Users with explicit Hz overrides will retain stale v1.1 values after rebalancing; users without overrides hear unexplained soundscape changes. Prevention: assign all new protocols to frequency ranges above 1047 Hz (unoccupied by v1.1 built-ins), leaving the 65–1047 Hz existing layout frozen. If existing class frequencies must move, document every Hz change in release notes.
|
||||
|
||||
2. **User rules appended after catch-alls are unreachable (Pitfall A5/A9)** — `DefaultRules` ends with catch-all rules (`DstPort: 0`) that match any TCP/UDP packet. Appending user rules after them makes user rules unreachable. Split into `SpecificRules` + `CatchAllRules`; merge order must be `userRules + SpecificRules + CatchAllRules`.
|
||||
2. **autoAssignFreq range [1200, 2350] Hz collision with new built-ins** — If new built-in classes use frequencies in the 1200–2350 Hz auto-assign range, user-defined custom classes can hash to the same frequency silently. Prevention: push autoAssignFreq base above all built-in frequencies (e.g., 4500 Hz) after finalizing v1.2 ClassFreqConfigs. Add a compile-time test asserting no built-in falls in the auto-assign range.
|
||||
|
||||
3. **User-defined classes have no bank layer — panic or silence (Pitfall A6)** — `NewBank` currently iterates `classify.AllClasses()` (a hardcoded list of 14 built-in classes). User classes will not be in that list. `NewBank` must iterate the keys of the merged `FreqConfig` map instead. Validate at config load that every rule's class name resolves to a configured sound entry.
|
||||
3. **NumLayers/GainPerLayer exported constant is stale** — bank.go uses `1.0 / float64(len(cfgs))` dynamically; the exported `synth.GainPerLayer` constant is frozen at 14-layer value. Any new v1.2 code referencing `synth.GainPerLayer` will compute wrong gain. Prevention: remove or deprecate the constant before adding any new classes; bank.go's dynamic computation is the sole authoritative source.
|
||||
|
||||
4. **Naive square/sawtooth/triangle waveforms produce audible aliasing (Pitfall A3)** — Direct time-domain math generates infinite harmonics that alias above Nyquist. Use additive synthesis: generate a bandlimited `[]HarmonicDef` series (odd harmonics for square/triangle, all harmonics for sawtooth, truncated at Nyquist) at config load time. The existing `HarmonicDef` infrastructure already supports this approach.
|
||||
4. **TestFrequenciesInRange hardcodes [60, 1100] Hz** — CI fails immediately when adding classes above 1100 Hz. Prevention: update the test bounds to the new valid range (e.g., [60, 4000]) before adding any ClassFreqConfigs entries outside the current range. This is a false blocker if not addressed first.
|
||||
|
||||
5. **BurntSushi/toml silently ignores unknown keys by default (Pitfall A2)** — Use `toml.Decode()` (not `Unmarshal`) to obtain `MetaData`, then call `md.Undecoded()` and return an error listing any unrecognized keys. Implement strict decoding from the first config load function.
|
||||
5. **Within-family detuning too narrow for psychoacoustic distinctness** — Fixed small Hz steps (e.g., 10 Hz) fall inside the critical band (ERB) at higher frequencies — at 1000 Hz ERB is ~72 Hz; at 2000 Hz it is ~117 Hz. Tones within the critical band merge perceptually. Prevention: use logarithmic interval separations — at minimum a major second (ratio 1.122). Verify all within-family pairs satisfy `|f2 - f1| > ERB(min(f1,f2))` before committing Hz values.
|
||||
|
||||
6. **Three-location atomic update required for each new class** — Adding a protocol requires updating AllClasses(), ClassFreqConfigs, and the TrafficClass constant. Missing any one causes test failures that are diagnostic but confusing. Prevention: update all three in the same commit, or introduce a single `builtinClassDefs` slice that drives both AllClasses() and validates ClassFreqConfigs.
|
||||
|
||||
7. **Secure/insecure protocol variants create frequency overcrowding if treated as separate classes** — SMTP:25, SMTP-submit:587, SMTPS:465 as three classes produces three frequencies where users want one "mail" sound. Prevention: map all ports of a protocol family to a single TrafficClass using multiple Rule entries. Port-to-class is many-to-one within a family.
|
||||
|
||||
## Implications for Roadmap
|
||||
|
||||
The v1.1 work has clear dependency ordering that directly dictates phase structure. The build order in ARCHITECTURE.md (7 steps, each independently testable) maps naturally to implementation phases.
|
||||
Based on the research, v1.2 should follow a 4-phase sequence driven by dependency order and the need to resolve design decisions before code decisions.
|
||||
|
||||
### Phase 1: Waveform Types in the Oscillator
|
||||
### Phase 1: Test and Constant Cleanup
|
||||
|
||||
**Rationale:** Zero external dependencies; pure math testable in isolation with golden-sample unit tests. The critical design decision — additive synthesis vs. direct math — must be made and locked in here. Switching after integration is a full oscillator rewrite.
|
||||
**Delivers:** `Waveform` enum, `sampleAt()` dispatch, `BandlimitedHarmonics()` generator; all four waveform types produce correct, alias-free output at all NetSynth frequencies.
|
||||
**Addresses:** Custom waveform per class (table stakes)
|
||||
**Avoids:** Pitfall A3 (aliasing from naive waveforms)
|
||||
**Files changed:** `synth/oscillator.go` only
|
||||
**Rationale:** Three existing tests and one exported constant actively block v1.2 work if not addressed first. Cleaning these up prevents confusing CI failures throughout the milestone.
|
||||
**Delivers:** Stale NumLayers/GainPerLayer constant removed or deprecated; TestFrequenciesInRange updated to accept new range; TestNumLayersMatchesAllClasses renamed and its invariant documented clearly.
|
||||
**Addresses:** Pitfalls C3 (stale GainPerLayer), C4 (hardcoded range test blocking correct additions)
|
||||
**Avoids:** Wasted debugging time on pre-existing issues presenting as new failures
|
||||
|
||||
### Phase 2: Decouple Bank from Global Config
|
||||
### Phase 2: Protocol List and Frequency Design (No Code)
|
||||
|
||||
**Rationale:** Prerequisite for config injection. `NewBank` must accept an injected config map before the `config` package exists. Wiring `Waveform` through `FreqConfig` → `Layer` → `Oscillator` is included here; zero-value default (`WaveformSine = 0`) means existing tests pass unchanged.
|
||||
**Delivers:** `NewBank(tau, cfgs map[TrafficClass]FreqConfig)`, `FreqConfig.Waveform` field, `synth/layer.go` updated. System is functionally identical to v1.0 but injectable.
|
||||
**Avoids:** Global-read anti-pattern (multiple places reading `ClassFreqConfigs`, ambiguous merge point)
|
||||
**Files changed:** `synth/config.go`, `synth/layer.go`, `synth/bank.go`, `encode/mp3.go`
|
||||
**Rationale:** The frequency allocation must be designed before any ClassFreqConfigs entries are written. Getting this wrong after the fact requires touching every entry. The autoAssignFreq collision and psychoacoustic ERB constraints must be resolved at design time.
|
||||
**Delivers:** Final protocol class list (Tier 1 + Tier 2 from FEATURES.md, multi-port collapsed to single class per family), complete Hz allocation for all ~35 classes in family bands, autoAssignFreq base set above all built-ins, ERB check confirming all within-family pairs are perceptually distinct.
|
||||
**Addresses:** Pitfalls C1 (backward compat), C2 (auto-assign collision), C6 (critical band masking), C7 (secure/insecure variant crowding)
|
||||
**Avoids:** Frequency rebalancing causing retroactive rework; tone merging within families discovered only during listening tests
|
||||
|
||||
### Phase 3: Config Package — TOML Loading and Merge
|
||||
### Phase 3: Classification Layer
|
||||
|
||||
**Rationale:** Core new infrastructure. Builds on the injectable bank signature from Phase 2. All config correctness patterns (pointer fields, strict decode, validation, string normalization) must be established here in isolation before any features are wired to the bank. Retrofitting these patterns after pipeline integration is significantly more expensive.
|
||||
**Delivers:** `config/` package with `Load()`, `os.UserConfigDir()` discovery, pointer-field merge, `validate()` with string normalization and enum checking, `DefaultConfig()` wrapping existing values.
|
||||
**Addresses:** Config auto-discovery, `--config` flag, partial override semantics, unknown field detection, startup validation, clear error messages
|
||||
**Avoids:** Pitfalls A1 (zero-value overwrite), A2 (silent typos), A4 (waveform string case), A7 (XDG ignored), A8 (missing explicit path), A10 (empty class name)
|
||||
**Rationale:** New constants and rules must exist before synth entries can reference them. This phase is purely additive to the classify package with no audio impact — safe to land and test in isolation.
|
||||
**Delivers:** All ~21 new TrafficClass constants, AllClasses() updated in family-grouped order, DefaultRules extended with new port rules (multi-port-to-single-class pattern applied per family).
|
||||
**Addresses:** Protocol coverage table stakes (Mail, Remote Access, File Transfer, Infrastructure, Database, Directory/Auth, VoIP families from FEATURES.md)
|
||||
**Avoids:** Pitfall C8 (three-location atomicity — enforced by existing test), Pitfall C9 (range-based protocols like RTP deferred)
|
||||
**Uses:** Port-based Rule classification — same pattern as existing SSH/HTTP/DNS rules, no new code paths
|
||||
|
||||
### Phase 4: Classification Rule Merging
|
||||
### Phase 4: Synthesis and Config Layer
|
||||
|
||||
**Rationale:** `classify/rules.go` must be split before any config-loading logic references the rule slice. The three-layer merge order is a design decision that, if wrong, produces silent failures with no error messages — it must be validated with unit tests before pipeline integration.
|
||||
**Delivers:** `classify.SpecificRules`, `classify.CatchAllRules`, `classify.MergeRules(userRules []Rule) []Rule`; user rules prepend correctly with both catch-all and specific-rule ordering.
|
||||
**Avoids:** Pitfall A5 (unreachable rules after catch-alls), Pitfall A9 (user rule shadowed by built-in specific rule for same port)
|
||||
|
||||
### Phase 5: Wire Config Through Pipeline — Frequency and Waveform Overrides
|
||||
|
||||
**Rationale:** Connects the config package to the synth layer for built-in classes only. Validates the full pipeline end-to-end before adding the complexity of user-defined classes. `encode.RunSynthesis` signature change is a breaking API change — all call sites must be updated in a single commit.
|
||||
**Delivers:** `--config` cobra flag, `config.Load()` in `main.go`, merged freq config map passed to `RunSynthesis` and `NewBank`; end-to-end test: TOML sets HTTPS to 200 Hz sawtooth, bank produces 200 Hz sawtooth layer.
|
||||
**Implements:** Config → synth integration
|
||||
|
||||
### Phase 6: User-Defined Classes End-to-End
|
||||
|
||||
**Rationale:** The most complex integration; requires all prior phases. User-defined classes create new `TrafficClass` strings that must exist in both the merged rule set and the bank's layer map. The AllClasses() decoupling from Phase 2 makes this tractable.
|
||||
**Delivers:** `[[rules]]` TOML section, dynamic `TrafficClass` values from config, bank layers constructed from merged FreqConfig map keys, class name cross-validation at config load.
|
||||
**Avoids:** Pitfall A6 (user-defined class has no bank layer — nil panic or silence)
|
||||
|
||||
### Phase 7: Print-Config and UX Polish
|
||||
|
||||
**Rationale:** `--print-config` is independent of capture and must wait until all config structure is stable (Phase 6). Additive, zero regression risk.
|
||||
**Delivers:** `netsynth --print-config` subcommand with commented TOML output of effective config; optional `name` field on user rules displayed in exit summary.
|
||||
**Rationale:** ClassFreqConfigs entries depend on both the protocol list from Phase 2 design and the TrafficClass constants from Phase 3. The Group field on FreqConfig and PrintConfig group headers are the final polish on this phase.
|
||||
**Delivers:** All new FreqConfig entries with finalized frequencies from Phase 2; Group field added to FreqConfig; autoAssignFreq base updated and compile-time range test added; optional PrintConfig group-header section comments.
|
||||
**Addresses:** Table stakes (--print-config reflects new classes); differentiators (within-family tonal design, group headers)
|
||||
**Avoids:** Pitfall C2 (confirmed by new compile-time test), Pitfall C5 (no new TOML top-level keys since groups are not user-configurable)
|
||||
**Manual validation required:** A listening test with a real or synthetic pcap file is needed after this phase — automated tests cannot substitute for ear confirmation that family identity is perceptually clear.
|
||||
|
||||
### Phase Ordering Rationale
|
||||
|
||||
- Phases 1–2 are internal refactors with no user-visible change — the right starting point for establishing patterns safely
|
||||
- Phase 3 owns all config safety in one isolated package before anything is wired — retrofitting pointer-field merge after bank integration means touching multiple packages simultaneously
|
||||
- Phase 4 (rule splitting) must precede Phase 6 (user rules) or catch-all ordering bugs surface silently at integration with no clear failure signal
|
||||
- Phase 5 validates the full pipeline with familiar built-in classes before Phase 6 introduces the harder user-defined class problem
|
||||
- Phase 7 is pure additive polish with zero risk of breaking earlier phases
|
||||
- Phase 1 before everything because pre-existing test blockers cause false CI failures throughout the milestone
|
||||
- Phase 2 (design) before code because Hz allocation is the hardest-to-change decision with the broadest blast radius; fixing retroactively touches every ClassFreqConfigs entry
|
||||
- Phase 3 before Phase 4 because TrafficClass constants must exist before ClassFreqConfigs can reference them
|
||||
- Phase 4 is last because it depends on both the design (Phase 2) and the constants (Phase 3)
|
||||
|
||||
### Research Flags
|
||||
|
||||
Phases with well-documented patterns — skip additional research:
|
||||
- **Phase 1:** DSP textbook math; harmonic series are fully specified
|
||||
- **Phase 2:** Standard dependency injection refactor; no unknowns
|
||||
- **Phase 3:** BurntSushi/toml API is well-documented; pointer-field merge is a known TOML pattern
|
||||
- **Phase 4:** Simple slice manipulation; no external dependencies
|
||||
- **Phase 7:** Cobra subcommand and TOML marshal are standard patterns
|
||||
Phases with well-documented patterns (no additional research needed):
|
||||
- **Phase 1:** Straightforward constant and test cleanup; all relevant code is in the existing codebase
|
||||
- **Phase 3:** Port rules follow the exact same pattern as existing SSH/HTTP/DNS rules; no new patterns or unknowns
|
||||
- **Phase 4:** FreqConfig additions follow the exact same pattern as existing entries; the Group field is a non-functional metadata addition
|
||||
|
||||
Phases that may benefit from a targeted research pass or design review:
|
||||
- **Phase 5:** `encode.RunSynthesis` signature change is a breaking API change — verify all test call sites and plan a single-commit update
|
||||
- **Phase 6:** User-defined class name collision with built-in `TrafficClass` string values (e.g., user names a class `"HTTPS"`) requires a design decision: treat as override of built-in sound vs. reject as ambiguous. Not resolved in research; decide before coding Phase 6.
|
||||
Phases that require design validation:
|
||||
- **Phase 2:** The frequency allocation should be validated with a listening test on a real pcap file before committing to final Hz values. The ERB computations are straightforward math; the perceptual result requires ear confirmation. This is inherent to audio design work.
|
||||
|
||||
## Confidence Assessment
|
||||
|
||||
| Area | Confidence | Notes |
|
||||
|------|------------|-------|
|
||||
| Stack | HIGH | Existing stack validated in v1.0; BurntSushi/toml v1.6.0 confirmed via pkg.go.dev and GitHub releases; `os.UserConfigDir()` XDG behavior confirmed against official Go stdlib docs |
|
||||
| Features | HIGH | Config file conventions verified against XDG spec, git, golangci-lint, and mise patterns; TOML schema grounded in existing v1.0 codebase types |
|
||||
| Architecture | HIGH | Based on direct code inspection of the 3,254-line v1.0 codebase; all integration points identified with specific file/line references and build order prescribed |
|
||||
| Pitfalls | HIGH | TOML default-overwrite behavior verified against upstream issue trackers (BurntSushi/toml #47, go-toml #252); aliasing prevention verified against DSP literature (CCRMA paper, McGill bandlimited synthesis notes) |
|
||||
| Stack | HIGH | Validated against shipped v1.1 codebase; no new dependencies; gopacket layer types confirmed via direct GitHub source inspection |
|
||||
| Features | MEDIUM-HIGH | Protocol selection based on IANA port registry, nDPI taxonomy, Wireshark dissectors; real-world traffic frequency is inference, not measurement |
|
||||
| Architecture | HIGH | Derived from direct inspection of shipped v1.1 code (~4,675 lines); all integration points identified with specific file references and build order |
|
||||
| Pitfalls | HIGH | All critical pitfalls grounded in specific code locations (config.go merge(), bank.go gainPerLayer, synth/config_test.go assertions); audio masking values from Glasberg & Moore 1990 ERB model |
|
||||
|
||||
**Overall confidence:** HIGH
|
||||
**Overall confidence: HIGH**
|
||||
|
||||
### Gaps to Address
|
||||
|
||||
- **User class name collision with built-in class strings:** If a user writes `class = "HTTPS"` in a `[[rules]]` block, the intent could be "override built-in sound" or "create a parallel custom class." The merge logic needs an explicit decision before Phase 6: treat matching names as overrides (simplest) or require a separate TOML section. This is a UX design question — resolve before coding Phase 6.
|
||||
- **BurntSushi/toml vs. go-toml v2:** Both work for this use case. BurntSushi is recommended for ergonomics, but if the team prefers go-toml v2's `DisallowUnknownFields()` pattern, it is equally valid with minor API differences. Either choice is fine; just make one and be consistent.
|
||||
- **Frequency allocation requires listening validation:** No automated test replaces ear-testing. Plan a listening session with a diverse pcap file after Phase 4 before declaring the milestone complete.
|
||||
- **Protocol frequency on real networks:** Tier 1/2 ranking is based on typical network types, not measurement on the target user's actual network. mDNS and SSDP are prominent on home/office LANs but absent on cloud workloads. The catch-all classes handle unrecognized traffic regardless, so this is a coverage quality concern, not a correctness concern.
|
||||
- **Rule schema for range-based protocols:** RTP and NetBIOS (multi-port, non-contiguous) are explicitly deferred. If desired in v1.3+, the Rule struct needs `DstPortMin/Max` fields — a known future gap, not a v1.2 concern.
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (HIGH confidence)
|
||||
|
||||
- Direct code inspection: `synth/config.go`, `synth/oscillator.go`, `synth/bank.go`, `synth/layer.go`, `classify/classifier.go`, `classify/rules.go`, `classify/types.go`, `encode/mp3.go`, `cmd/netsynth/main.go`
|
||||
- `pkg.go.dev/github.com/BurntSushi/toml` — v1.6.0 API, `Undecoded()` strict mode, pointer field behavior
|
||||
- `github.com/BurntSushi/toml/issues/47` — default-overwrite behavior when using `Unmarshal` confirmed
|
||||
- `pkg.go.dev/os#UserConfigDir` — XDG_CONFIG_HOME behavior on Linux confirmed via official Go stdlib docs
|
||||
- `ccrma.stanford.edu/~stilti/papers/blit.pdf` — bandlimited synthesis theory (alias-free waveforms)
|
||||
- `music.mcgill.ca/~gary/307/week5/bandlimited.html` — truncated harmonic series approach confirmed
|
||||
- `github.com/gopacket/gopacket/blob/master/layers/layertypes.go` — LayerTypeSIP (id 133), LayerTypeTLS (id 140) confirmed at v1.5.0
|
||||
- `github.com/gopacket/gopacket/blob/master/layers/ports.go` — UDP/TCP port pre-registration confirmed; mDNS/SNMP/QUIC absent
|
||||
- `github.com/gopacket/gopacket/tree/master/layers` — directory listing; no mdns.go, quic.go, snmp.go, ldap.go, smb.go, rdp.go
|
||||
- Direct codebase inspection: `synth/config.go`, `synth/bank.go`, `classify/types.go`, `classify/rules.go`, `classify/classifier.go`, `config/config.go`, `encode/mp3.go`, `cmd/netsynth/main.go`
|
||||
- Glasberg & Moore 1990 ERB model: `ERB(f) = 24.7 * (4.37 * f/1000 + 1)` — critical bandwidth values for all relevant frequencies
|
||||
|
||||
### Secondary (MEDIUM confidence)
|
||||
|
||||
- XDG Base Directory Specification — config discovery precedence order
|
||||
- `mise.jdx.dev/configuration.html` — working-dir + XDG config discovery pattern
|
||||
- `golangci-lint.run/docs/configuration/cli/` — partial override config in Go CLI tools
|
||||
- `dylanmeeus.github.io/posts/audio-from-scratch-pt8/` — Go waveform synthesis from scratch, confirms no library needed
|
||||
- `pkg.go.dev/github.com/pelletier/go-toml/v2` — v2.3.0 strict decoder comparison; `pelletier/go-toml/issues/252` partial v2 resolution of default-overwrite
|
||||
- IANA Service Name and Transport Protocol Port Number Registry — authoritative port assignments for all new protocols
|
||||
- nDPI Protocols List (ntop) — 450+ protocol taxonomy; 17-category grouping model as design precedent
|
||||
- nDPI 5.0 Enhanced Traffic Fingerprinting blog post — category-based grouping confirmed as production approach
|
||||
- SoNSTAR: Sonification of Network Traffic (Paul Vickers) — academic network sonification reference
|
||||
- Sonification of network traffic flow (PLoS One 2018) — research on perceptually useful protocol groupings
|
||||
|
||||
### Tertiary (LOW confidence)
|
||||
|
||||
- Competitor feature table (SoNSTAR, Network-Sonification, Peep) — niche domain, limited documentation; used for context only, not binding decisions
|
||||
- Protocol prevalence on "typical" networks (home/office/server/cloud) — inferred from nDPI taxonomy and Wireshark dissector popularity; not empirically measured on target networks
|
||||
|
||||
---
|
||||
*Research completed: 2026-03-26*
|
||||
*Research completed: 2026-03-27*
|
||||
*Ready for roadmap: yes*
|
||||
|
||||
Reference in New Issue
Block a user