Files
yoloyolo/.planning/research/ARCHITECTURE.md
T
2026-03-27 08:25:02 +01:00

491 lines
25 KiB
Markdown

# Architecture Patterns
**Domain:** Network traffic sonification CLI (Go) — v1.2 Extended Protocol Coverage with Grouped Families
**Researched:** 2026-03-26 (v1.0/v1.1), updated 2026-03-27 (v1.2 grouped protocol families)
**Confidence:** HIGH — based on direct code inspection of the shipped v1.1 codebase
---
## v1.2 Integration Overview
This document supersedes the pre-implementation v1.0/v1.1 architecture research. It is grounded in the actual shipped v1.1 codebase (~4,675 lines, 7 packages) and answers specifically: how does the *group concept* thread through classify, synth, config, and CLI? What changes, what stays, and in what order?
---
## Current Package Map (v1.1 Baseline — the starting point)
```
cmd/netsynth/main.go CLI, pipeline wiring, Cobra flags
capture/ go-pcap live capture + pcap file reader + BPF
classify/
types.go TrafficClass (string type), ClassifiedPacket, WindowSnapshot, AllClasses()
classifier.go NewClassifier(rules []Rule) — first-match-wins
rules.go DefaultRules []Rule (12 built-in rules)
aggregate/
window.go 500ms time-windowed snapshot accumulation
synth/
config.go FreqConfig{BaseHz, Harmonics, Pan, WaveformType}, ClassFreqConfigs map
oscillator.go Phase-accumulator oscillator — sine+additive harmonics
layer.go EMA amplitude smoothing per Layer
bank.go NewBank(tau, cfgs map) — one Layer per config entry
mixer.go PanGains, StereoFramesToInt16Bytes
config/
config.go TOML parse, merge, validate, auto-freq assignment, PrintConfig
encode/
mp3.go RunSynthesis(snapshots, path, freqCfgs) — NewBank + EncodeMP3
```
---
## What v1.2 Adds
Two related but separable capabilities:
1. **More built-in protocol rules** — extended DefaultRules covering Mail, Remote Access, Database, Discovery, File Transfer, VoIP, etc.
2. **Group concept** — related protocols share a recognizable sound family (shared base frequency, detuned members, optional shared waveform character)
These must be designed together because the group concept directly affects frequency allocation, and frequency allocation directly affects DefaultRules ordering decisions.
---
## The Group Concept: What It Means Architecturally
A "group" means: protocols in the same family share a musical neighborhood. Listeners learn "low rumble = infrastructure traffic" and "mid-range shimmer = web traffic" without needing to identify individual tones.
The group concept requires:
- A **group identifier** stored somewhere accessible by both classify (for `--print-config` ordering) and synth (for within-group detuning computation)
- A **group base frequency** from which individual protocol frequencies are derived by small detuning offsets
- **Within-group waveform consistency** — all members of a group use the same waveform type so the family sounds recognizable even as individual pitches differ
The key architectural question is: *where does the group concept live?*
---
## Decision: Group Lives in `synth/config.go`, Not in `classify/`
### Option A: Group field on `classify.Rule` or new `TrafficClassGroup` type in `classify`
Adding `Group string` to `classify.Rule` would put group information at the classification boundary, visible from the capture pipeline forward. But:
- The classifier does not care about groups — it only cares about protocol matching
- `AllClasses()` in `classify/types.go` already returns ordered class names without needing sound semantics
- Pollutes the `classify` package with audio-domain concerns
### Option B: Group field in `synth.FreqConfig` (recommended)
`FreqConfig` already owns all sound parameters (frequency, harmonics, pan, waveform). Adding a `Group string` field keeps group logic inside the audio domain. The `classify` package stays clean — it only knows about `TrafficClass` names.
```go
// synth/config.go
type FreqConfig struct {
BaseHz float64
Harmonics []HarmonicDef
Pan float64
WaveformType WaveformType
Group string // NEW: e.g., "mail", "web", "remote-access", "" for ungrouped
}
```
**`classify.TrafficClass` does NOT get a Group field.** The group is a sound-design concept, not a classification concept. Classification says "this is SMTP"; synthesis says "SMTP belongs to the Mail group at 440 Hz + 0 cents detuning."
### Why this is the right boundary
| Package | Knows About | Does Not Know About |
|---------|-------------|---------------------|
| `classify` | Protocol, port, traffic class name | Group, frequency, waveform |
| `synth` | Frequency, harmonics, pan, waveform, group | Protocol, port |
| `config` | Merges both sides; reads TOML; owns PrintConfig | Packet capture |
---
## Frequency Allocation Strategy for Groups
### Current allocation (v1.1)
The 14 existing classes span 65 Hz to 1047 Hz in a roughly logarithmic spread:
```
ICMP: 65 Hz (sub-bass)
DNS: 110 Hz
HTTPS: 175 Hz
HTTP: 220 Hz
SSH: 330 Hz
SMTP: 440 Hz
NTP: 520 Hz
DHCP: 600 Hz
other-TCP: 700 Hz
other-UDP: 780 Hz
unknown-1: 862 Hz
unknown-2: 920 Hz
unknown-3: 981 Hz
unknown-4: 1047 Hz
auto-range: 1200-2350 Hz (FNV-32a hash for user-defined classes)
```
### v1.2 group-based allocation
New protocols need frequency homes. With ~25-35 total built-in classes after expansion, maintaining individual hand-tuned frequencies becomes fragile and collision-prone. The group model solves this by assigning each group a frequency band, then placing members within that band at small detuning offsets.
**Recommended group bands (within the 65-1100 Hz range already owned by built-ins):**
| Group | Band Center | Role | Protocols |
|-------|-------------|------|-----------|
| Infrastructure | 65-130 Hz | Core network services (barely heard, always present) | ICMP, NTP, DHCP, mDNS/SSDP |
| Web | 175-280 Hz | HTTP-family traffic | HTTP, HTTPS, HTTP/3, WebSocket |
| Mail | 380-480 Hz | Email protocols | SMTP, IMAP, POP3, SMTP-submission |
| Remote Access | 300-360 Hz | Interactive sessions | SSH, RDP, Telnet, VNC |
| Database | 500-580 Hz | Backend data stores | MySQL, PostgreSQL, Redis, MongoDB |
| File Transfer | 620-700 Hz | File movement protocols | FTP, SFTP, FTPS, SMB, NFS |
| VoIP / Streaming | 720-820 Hz | Real-time media | SIP, RTP, RTSP |
| DNS | 110 Hz | Singleton — already well-placed | DNS (UDP+TCP) |
| Unknown | 850-1100 Hz | Auto-bucketed unrecognized traffic | unknown-1 through unknown-4 |
**Within-group detuning model:** Each group has a `GroupBaseHz`. Members are offset by small cent-based detuning or fixed Hz offsets:
```
Mail group base: 420 Hz
SMTP: 420 Hz (group base, no detune)
IMAP: 430 Hz (+10 Hz)
POP3: 440 Hz (+20 Hz)
SMTP-submission: 450 Hz (+30 Hz)
```
Offsets of 10-30 Hz are audible as distinct pitches but harmonically related enough to sound like a family. Do NOT use harmonic ratios (2x, 3x) for detuning — that would place group members an octave or fifth apart, destroying the "family sound" effect.
**This is a design constraint, not a library feature.** The detuning is expressed as explicit `BaseHz` values in `ClassFreqConfigs`. There is no runtime detuning computation needed — it is baked into the default config map.
### Frequency rebalancing required
Adding ~15-20 new protocols means the existing 65-1100 Hz range must be replanned. The existing classes should be reassigned to group-coherent frequencies even if this moves them from v1.1 values. This is a conscious breaking change to the default sound, acceptable in a minor version bump where the goal is improved audio design.
The `unknown-1..4` range at 850-1047 Hz is preserved as-is — those are intentionally dissonant and serve a distinct purpose.
---
## Component-by-Component Changes
### `classify/rules.go` — Extend DefaultRules
**What changes:** Add ~15-20 new `Rule` entries for the expanded protocol set.
**What does NOT change:** `Rule` struct, `NewClassifier`, the first-match-wins algorithm.
**Required care:** The catch-all rules (`DstPort: 0` for TCP and UDP) must remain last. New protocol-specific rules must be inserted before these catch-alls. `DefaultRules` is an ordered slice — insertion order matters.
```go
// New rules inserted before catch-alls, e.g.:
{Protocol: "tcp", DstPort: 143, Class: ClassIMAP},
{Protocol: "tcp", DstPort: 110, Class: ClassPOP3},
{Protocol: "tcp", DstPort: 587, Class: ClassSMTPSubmit},
{Protocol: "tcp", DstPort: 3306, Class: ClassMySQL},
{Protocol: "tcp", DstPort: 5432, Class: ClassPostgres},
{Protocol: "tcp", DstPort: 6379, Class: ClassRedis},
{Protocol: "tcp", DstPort: 3389, Class: ClassRDP},
{Protocol: "tcp", DstPort: 5900, Class: ClassVNC},
{Protocol: "udp", DstPort: 5060, Class: ClassSIP},
{Protocol: "tcp", DstPort: 5060, Class: ClassSIP},
// ... etc
```
### `classify/types.go` — New Constants and Updated AllClasses()
**What changes:** Add new `TrafficClass` constants for each new protocol. Update `AllClasses()` to include them in display-sensible order (grouped by family, matching PrintConfig output order).
**What does NOT change:** `ClassifiedPacket`, `WindowSnapshot`, the `TrafficClass` string type itself.
```go
// New constants (representative subset):
const (
ClassIMAP TrafficClass = "IMAP"
ClassPOP3 TrafficClass = "POP3"
ClassSMTPSubmit TrafficClass = "SMTP-submit"
ClassMySQL TrafficClass = "MySQL"
ClassPostgres TrafficClass = "PostgreSQL"
ClassRedis TrafficClass = "Redis"
ClassMongoDB TrafficClass = "MongoDB"
ClassRDP TrafficClass = "RDP"
ClassVNC TrafficClass = "VNC"
ClassTelnet TrafficClass = "Telnet"
ClassFTP TrafficClass = "FTP"
ClassFTPS TrafficClass = "FTPS"
ClassSMB TrafficClass = "SMB"
ClassNFS TrafficClass = "NFS"
ClassSIP TrafficClass = "SIP"
ClassHTTP3 TrafficClass = "HTTP3" // UDP 443 (QUIC)
ClassmDNS TrafficClass = "mDNS" // UDP 5353
ClassSSDPDisc TrafficClass = "SSDP" // UDP 1900
// ... additional as determined by feature research
)
```
**`AllClasses()` ordering:** Grouped by family in the same order as the group bands above. This order is consumed by `config.PrintConfig` for output ordering — so the output will naturally group protocols by family with no additional logic needed in `config`.
### `synth/config.go` — Add Group Field and Rebalanced ClassFreqConfigs
**What changes:**
1. Add `Group string` to `FreqConfig`:
```go
type FreqConfig struct {
BaseHz float64
Harmonics []HarmonicDef
Pan float64
WaveformType WaveformType
Group string // NEW: sound family identifier; "" = ungrouped
}
```
2. Add `ClassFreqConfigs` entries for all new protocols with group-coherent frequencies and within-group detuning.
3. Update existing class frequencies to align with the group bands (frequency rebalancing). This is the highest-risk change in v1.2 because it alters the default sound of existing classes.
4. Update `NumLayers` constant from 14 to the new total (e.g., 30-35 depending on final protocol list):
```go
const (
NumLayers = 32 // updated count; GainPerLayer recomputed automatically
GainPerLayer = 1.0 / float64(NumLayers)
)
```
**What does NOT change:** `FreqConfig` field types (only addition), `WaveformType`, `HarmonicDef`, `WaveformPresetHarmonics`, `SampleRate`, `WindowMs`, `SamplesPerWindow`, `WhisperFloor`. The `Group` field is metadata — it has no effect on synthesis math.
**Pan assignment:** With 30+ layers, the current hand-tuned pan positions become crowded. Recommendation: assign pan by group position (infrastructure = center, web = slight left, mail = slight right, etc.) rather than per-protocol. This creates a spatial "map" of the network that reinforces the group concept aurally.
### `config/config.go` — PrintConfig Group Ordering
**What changes:** `PrintConfig` currently emits classes in `AllClasses()` order. With `AllClasses()` updated to group-coherent order, `PrintConfig` output will naturally be grouped. No structural change to `PrintConfig` is required.
**Optional enhancement:** Emit a group comment header between group sections:
```toml
# --- Infrastructure ---
# ICMP -- 65.0 Hz (default)
[sounds.ICMP]
frequency = 65.0
waveform = "sine"
# NTP -- 110.0 Hz (default)
[sounds.NTP]
...
# --- Web ---
# HTTP -- 175.0 Hz (default)
[sounds.HTTP]
...
```
This requires `PrintConfig` to detect group transitions in the class ordering and emit a comment. The `Group` field on `FreqConfig` provides this information. Implementation: iterate `AllClasses()`, track previous group, emit a comment when group changes.
**What does NOT change:** TOML schema shape (no `group` key in `[sounds.*]` blocks — groups are not user-configurable in v1.2), `SoundOverride` struct, `RawRule`, `LoadResult`, `Load`, `merge`, `addAutoFreqEntries`.
**Group field in TOML config:** Users should NOT be able to override the `group` of a class. Group is a built-in design decision, not a user parameter. The `SoundOverride` struct stays with only `Frequency` and `Waveform` pointer fields.
### `synth/bank.go` — No Changes Required
`NewBank` already ranges over the passed-in `cfgs` map keys. Adding 20 new entries to `ClassFreqConfigs` and passing them through the existing chain requires no changes to `bank.go`. The `gainPerLayer` computation (`1.0 / float64(len(cfgs))`) automatically scales down with more layers.
### `encode/mp3.go` — No Changes Required
`RunSynthesis` already accepts the full `freqCfgs` map and passes it to `NewBank`. No signature change needed.
### `cmd/netsynth/main.go` — No Changes Required
The pipeline wiring (`config.Load``NewClassifier``encode.RunSynthesis`) is already correct. Adding more built-in classes and a group field to `FreqConfig` is transparent to `main.go`.
---
## Data Flow: Unchanged for v1.2
The v1.1 data flow is correct and does not need to change:
```
main.go
└─ config.Load(configPath)
└─ LoadResult{FreqCfgs, UserRules, ConfigPath, AutoClasses}
└─ classify.NewClassifier(append(userRules, DefaultRules...))
└─ encode.RunSynthesis(snapshots, path, result.FreqCfgs)
└─ synth.NewBank(tau, freqCfgs)
└─ one Layer per freqCfgs entry
```
The only change to the data flowing through this pipeline is: `freqCfgs` now contains ~30-35 entries (up from 14), each `FreqConfig` has a new `Group` field (ignored by synthesis, used only by `PrintConfig`).
---
## Integration Points Summary
| Integration Point | Change Type | Risk |
|-------------------|-------------|------|
| `classify/types.go` — new constants + `AllClasses()` | Additive | LOW |
| `classify/rules.go` — extended `DefaultRules` | Additive (new entries before catch-alls) | MEDIUM (order matters) |
| `synth/config.go``Group` field on `FreqConfig` | Additive (new field) | LOW (no synthesis impact) |
| `synth/config.go``ClassFreqConfigs` entries for new protocols | Additive | LOW |
| `synth/config.go``ClassFreqConfigs` frequency rebalancing for existing protocols | Modifying defaults | HIGH (changes default audio output) |
| `synth/config.go``NumLayers` constant update | Single constant | MEDIUM (affects `GainPerLayer`) |
| `config/config.go` — group-header comment in `PrintConfig` | Optional enhancement | LOW (isolated to string output) |
| `synth/bank.go` | No change | — |
| `encode/mp3.go` | No change | — |
| `cmd/netsynth/main.go` | No change | — |
---
## New vs Modified Components
### New
None — no new packages or files are required.
### Modified
| Component | What Changes | What Stays Same |
|-----------|--------------|-----------------|
| `classify/types.go` | New `TrafficClass` constants; `AllClasses()` extended + reordered by group | `ClassifiedPacket`, `WindowSnapshot`, string type |
| `classify/rules.go` | New `Rule` entries for all new protocols | `Rule` struct, match algorithm, catch-all placement |
| `synth/config.go` | `Group` field on `FreqConfig`; new entries in `ClassFreqConfigs`; existing frequency rebalancing; `NumLayers` updated | All synthesis constants, waveform types, `HarmonicDef` |
| `config/config.go` | Optional: group-header comments in `PrintConfig` | All loading, parsing, merge, validation logic |
---
## Build Order
The following order ensures each step is independently testable.
### Step 1: Define new TrafficClass constants and extend AllClasses()
**Files:** `classify/types.go` only.
Add all new protocol constants. Update `AllClasses()` to include them in group-coherent order. Tests: verify `AllClasses()` returns the expected slice with all new members.
No impact on synthesis, config, or main yet — new classes simply have no rules or FreqConfig entries yet, which is harmless.
### Step 2: Extend DefaultRules for new protocols
**Files:** `classify/rules.go` only.
Add new `Rule` entries before the catch-alls. Tests: `NewClassifier(DefaultRules).Classify(pkt)` for each new protocol's port returns the expected new class. Confirm catch-alls still fire for unrecognized traffic.
Dependency: Step 1 must complete first (new constants must exist).
### Step 3: Add Group field to FreqConfig
**Files:** `synth/config.go` only.
Add `Group string` to `FreqConfig`. Set `Group` on all existing 14 `ClassFreqConfigs` entries. Existing tests continue to pass (new field has zero value by default; synthesis ignores it). No callers need updating.
This step is independent of Steps 1 and 2.
### Step 4: Add ClassFreqConfigs entries for new protocols (without frequency rebalancing)
**Files:** `synth/config.go` only.
Add `FreqConfig` entries for each new protocol using placeholder frequencies that avoid the 65-1100 Hz range (use 2400-4000 Hz temporarily). This makes the end-to-end pipeline work for new classes without disrupting existing audio.
Tests: verify `ClassFreqConfigs` contains entries for all constants from `classify.AllClasses()`.
Dependency: Step 3 must complete first (Group field must exist). Step 1 must complete first (constants must exist).
### Step 5: Frequency rebalancing — reassign all ClassFreqConfigs to group-coherent values
**Files:** `synth/config.go` only.
This is the riskiest step. Assign final group-coherent frequencies to all protocols (existing and new) following the band design above. Update `NumLayers` to the final count.
Test strategy: write a test that verifies all entries in `ClassFreqConfigs` have `BaseHz` in a specified range per group; verify no two entries have identical `BaseHz`; verify `NumLayers` matches `len(ClassFreqConfigs)`.
Manual validation: run `netsynth --read testfile.pcap` with varied traffic and listen. This step requires subjective audio evaluation that tests cannot replace.
Dependency: Steps 3 and 4 must complete first.
### Step 6: PrintConfig group-header comments (optional polish)
**Files:** `config/config.go` only.
Add group transition detection in `PrintConfig`. When the `Group` field changes between consecutive `AllClasses()` entries, emit a `# --- GroupName ---` comment.
Tests: verify `PrintConfig` output contains group header comments; verify ungrouped classes have no group header; verify user-defined classes (no group field) are not preceded by a group header.
Dependency: Steps 3 and 5 must complete first (Group field must be populated in ClassFreqConfigs).
---
## Critical Integration Constraints
### NumLayers Must Match len(ClassFreqConfigs) for Correct Gain Scaling
`synth/config.go` defines `NumLayers = 14` as a constant. `GainPerLayer = 1.0 / float64(NumLayers)`. However, `bank.go:NewBank` computes `gainPerLayer` dynamically from `len(cfgs)`:
```go
gainPerLayer: 1.0 / float64(len(cfgs)),
```
This means `NumLayers` and `GainPerLayer` in `config.go` are currently informational constants, not what the bank actually uses at runtime. The bank's dynamic computation is correct. Update `NumLayers` for documentation accuracy, but the audio math is not at risk even if this is temporarily inconsistent.
### DefaultRules Catch-Alls Must Remain Last
`classify/rules.go` ends with `{Protocol: "tcp", DstPort: 0, ...}` and `{Protocol: "udp", DstPort: 0, ...}`. All new protocol rules must be inserted before these. A rule added after the catch-alls will never fire. This constraint must be enforced by convention (comment in the file) since Go has no static ordering guarantees for slice initialization beyond the literal order.
### AllClasses() Ordering Drives PrintConfig Ordering
`config.PrintConfig` iterates `classify.AllClasses()` to emit the `[sounds.*]` section. If `AllClasses()` is updated to group-coherent order (Step 1), `PrintConfig` output naturally reflects groups. If Step 6's group-header comments are added, they depend on `AllClasses()` presenting entries in group-sorted order. The two changes must be coordinated.
### Frequency Rebalancing Is a Default-Breaking Change
Users who rely on the default audio profile (no config file) will hear different tones for previously-familiar classes. This is an intentional design improvement but should be documented in the release notes. Users who have overridden frequencies via TOML config are unaffected (their overrides take precedence).
### Group Field Is Not a TOML Configurable
The `SoundOverride` struct in `config/config.go` must not gain a `group` field. Groups are built-in design decisions. Allowing users to reassign protocols to different groups via TOML would create inconsistencies in `PrintConfig` output ordering and the `AllClasses()` group semantics. If this is requested in a future milestone, it warrants a separate design decision.
---
## Anti-Patterns to Avoid
### Anti-Pattern: Computing Detuning at Runtime from Group Metadata
It is tempting to define group base frequencies and detune offsets as data structures, then derive individual `BaseHz` values at synthesis startup. This adds complexity with no benefit — the final `BaseHz` values are static per-class constants that belong in `ClassFreqConfigs` directly. Bake the detuning arithmetic into the initial constant assignment, document it with a comment (`// Mail group: +10 Hz from group base 420 Hz`), and move on.
### Anti-Pattern: Group as a `classify.TrafficClass` Field
Groups are audio concepts, not classification concepts. `classify.Rule` should not know about groups. `TrafficClass` is already a string type used as a map key throughout the codebase — adding a structured type would be a larger refactor than the feature warrants. Keep groups in `synth.FreqConfig`.
### Anti-Pattern: Adding Group-Based Routing to bank.go
Do not make `OscillatorBank` aware of groups for mixing purposes (e.g., summing all Mail group layers before applying gain). The current architecture mixes all layers uniformly. Group-coherent mixing is a more complex feature than v1.2 requires and changes the synthesis model. Defer if requested.
### Anti-Pattern: Overcrowding 65-1100 Hz with Too Many Protocols
At 30+ built-in classes, some protocols will be heard less often than others. Resist the urge to map rarely-seen protocols (Telnet, FTP) to prominent mid-range frequencies. Relegate low-signal protocols to the upper end of each group band so they contribute texture without dominating when not present.
### Anti-Pattern: Hand-Tuning Frequencies Without Listening
Frequency assignments must be validated by ear, not just by looking at Hz values on a spreadsheet. Schedule a listening session with a realistic pcap file after Step 5 before Step 6. The architecture makes this easy — any frequency change is a one-line edit in `ClassFreqConfigs`.
---
## Scalability Considerations
| Concern | With 14 Classes (v1.1) | With 32 Classes (v1.2) |
|---------|----------------------|----------------------|
| Gain per layer | ~7.1% of max | ~3.1% of max |
| Frequency collisions | None | Possible if not planned |
| AllClasses() iteration | 14 items, trivial | 32 items, still trivial |
| PrintConfig output lines | ~60 lines | ~130 lines |
| EMA smoothing convergence | No change | No change (per-layer, independent) |
| NewBank construction time | Negligible | Negligible |
The main practical change is perceptual loudness: with 32 layers each at 3.1% gain, the ambient mix becomes quieter when many protocols are active simultaneously. This is acceptable and matches the ambient/drone aesthetic. The `WhisperFloor` mechanism ensures inactive layers contribute minimally, so sessions with only 5-6 active protocols still sound full.
---
## Sources
- Direct code inspection: `synth/config.go`, `synth/bank.go`, `synth/layer.go`, `synth/oscillator.go`, `classify/types.go`, `classify/rules.go`, `classify/classifier.go`, `config/config.go`, `encode/mp3.go`, `cmd/netsynth/main.go` — HIGH confidence
- Musical interval theory (detuning, harmonic relationships): HIGH confidence — standard acoustic physics
- v1.2 protocol list: determined from feature research (see FEATURES.md for rationale on which protocols to include)
---
*Architecture research for: NetSynth v1.2 — extended protocol coverage with grouped families*
*Researched: 2026-03-27*