Files
yoloyolo/.planning/research/SUMMARY.md
T

188 lines
18 KiB
Markdown
Raw Normal View History

2026-03-24 22:36:28 +01:00
# Project Research Summary
2026-03-26 16:54:31 +01:00
**Project:** NetSynth v1.1 — Custom Sound Mappings
**Domain:** Network traffic sonification CLI tool (Go)
**Researched:** 2026-03-26
**Confidence:** HIGH
2026-03-24 22:36:28 +01:00
## Executive Summary
2026-03-26 16:54:31 +01:00
NetSynth v1.1 extends the working v1.0 CLI by adding user-customizable sound mappings through a TOML config file. The v1.0 codebase is a clean 6-package Go project (3,254 lines) with a well-separated pipeline: capture → classify → aggregate → synthesize → encode. The v1.1 milestone threads a new `config` package through this pipeline, enabling users to override frequencies and waveforms per traffic class, add new classification rules, and reference those custom classes in the synth layer. The recommended implementation path is incremental — introduce waveform types at the oscillator level first, then decouple bank construction from the global config, then add the TOML loader and wire it all together. Each step is independently testable before the next begins.
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
The stack requires one new dependency: `github.com/BurntSushi/toml` v1.6.0 (zero indirect deps, `MetaData.Undecoded()` provides strict-mode typo detection). All other additions — waveform math, config file discovery, and class name validation — are stdlib-only. The single notable technical choice is waveform synthesis strategy: naive direct-math square/sawtooth/triangle waveforms produce audible aliasing at the frequencies NetSynth uses (651047 Hz). The existing additive synthesis infrastructure (`[]HarmonicDef`) is the correct approach, generating bandlimited harmonic series for each waveform type rather than direct time-domain computation.
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
The highest-risk integration points are bank construction (which must be extended to handle user-defined classes not in the static `AllClasses()` list) and TOML merge semantics (TOML decoders zero-out absent fields, silently overwriting defaults unless pointer fields are used). Both are well-understood problems with clear prevention patterns that must be established before wiring config into the pipeline. The merge ordering for classification rules also requires deliberate design: user rules must precede specific built-in rules, which must precede catch-alls — three-layer ordering, not two.
2026-03-24 22:36:28 +01:00
## Key Findings
### Recommended Stack
2026-03-26 16:54:31 +01:00
The existing stack (gopacket v1.5.0, packetcap/go-pcap, sjzar/go-lame v0.0.9, cobra v1.10.2, hand-rolled sine oscillator + EMA) is unchanged. The only new external dependency is `BurntSushi/toml` v1.6.0, selected over `pelletier/go-toml v2` because `MetaData.Undecoded()` is more ergonomic for typo detection on a single startup config read, and over `spf13/viper` because Viper pulls in 20+ transitive deps for features (remote config, env var binding, hot reload) that are irrelevant here. See `.planning/research/STACK.md` for full alternatives analysis.
2026-03-24 22:36:28 +01:00
**Core technologies:**
2026-03-26 16:54:31 +01:00
- `github.com/gopacket/gopacket` v1.5.0 — packet decode — only maintained Go packet library, Go 1.24+
- `github.com/packetcap/go-pcap` — pure-Go live capture backend — no CGo, mmap ring buffer, Linux/macOS
- `github.com/sjzar/go-lame` v0.0.9 — MP3 encoding — embeds LAME C source, CGO_ENABLED=1, no system library required
- `github.com/spf13/cobra` v1.10.2 — CLI structure and flag handling — industry standard
- `github.com/BurntSushi/toml` v1.6.0 — TOML config parsing — zero transitive deps, strict-mode via `Undecoded()`
- Hand-rolled additive oscillator (sine today, square/sawtooth/triangle in v1.1) — no audio library needed
2026-03-24 22:36:28 +01:00
### Expected Features
2026-03-26 16:54:31 +01:00
**Must have (table stakes) for v1.1:**
- TOML config auto-discovery (`./netsynth.toml`, `~/.config/netsynth/config.toml`) — XDG Base Directory Specification standard
- `--config` flag for explicit path, with hard error if file is absent
- Partial override semantics — absent keys retain defaults; users must not replicate the full table to change one field
- Custom frequency per traffic class — direct override of `synth.ClassFreqConfigs`
- Custom waveform per traffic class — sine/square/sawtooth/triangle selection
- User-defined classification rules with custom class names, prepended before built-in rules
- Startup-time config validation with line-number errors (fail before capture begins, not after)
- Unknown field detection — prevents silent typos (`frequncy` must be caught, not silently ignored)
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
**Should have (differentiators):**
- `netsynth --print-config` subcommand dumping effective config as commented TOML — critical for discoverability
- Named custom rules (display name appears in exit summary and `--verbose` output)
- Clear error message listing valid waveform values on invalid input
2026-03-24 22:36:28 +01:00
**Defer (v2+):**
2026-03-26 16:54:31 +01:00
- Harmonic override per class (expose `HarmonicDef` slice in TOML) — niche, adds TOML nesting complexity
- Stereo pan position in config — explicitly deferred per project constraints
- Config hot-reload during capture — mid-capture state change corrupts synthesis; not worth the complexity
- Multiple config file includes/inheritance — single file merged with in-code defaults is sufficient
2026-03-24 22:36:28 +01:00
### Architecture Approach
2026-03-26 16:54:31 +01:00
v1.1 adds a `config` package and threads it through the existing pipeline via dependency injection. The key architectural shift: `synth.NewBank` currently reads the package-level global `ClassFreqConfigs`; after v1.1 it accepts a `map[TrafficClass]FreqConfig` parameter, enabling user-defined classes and eliminating hidden global state. The `config` package owns TOML parsing, file discovery, default config wrapping, and merge logic. `classify.DefaultRules` splits into `SpecificRules` + `CatchAllRules` so user rules can be inserted between them. The oscillator gains a `Waveform` enum field with additive-synthesis dispatch. All changes are contained to well-bounded components; `classify/classifier.go` and `encode/mp3.go` change only at their call sites. See `.planning/research/ARCHITECTURE.md` for full data flow diagrams and step-by-step build order.
2026-03-24 22:36:28 +01:00
**Major components:**
2026-03-26 16:54:31 +01:00
1. `config/` (new) — TOML struct definitions, `Load()`, auto-discovery via `os.UserConfigDir()`, pointer-field merge, `DefaultConfig()`
2. `synth/oscillator.go` (modified) — `Waveform` enum, `sampleAt()` dispatch, bandlimited harmonic series generation at config load time
3. `synth/bank.go` (modified) — accepts freq config map param; iterates map keys, not hardcoded `AllClasses()`
4. `classify/rules.go` (modified) — split into `SpecificRules` + `CatchAllRules`; `MergeRules(userRules)` export
5. `cmd/netsynth/main.go` (modified) — `--config` flag, `config.Load()`, user rule merge, config forwarded to `RunSynthesis`
2026-03-24 22:36:28 +01:00
### Critical Pitfalls
2026-03-26 16:54:31 +01:00
1. **TOML decoder zeros absent fields, silently overwriting defaults (Pitfall A1)** — Use pointer fields (`*float64`, `*string`) for all optional overrides in the decoded struct. Apply an explicit merge function that only writes non-nil values over the built-in defaults. Establish this pattern before any config is wired into the bank.
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
2. **User rules appended after catch-alls are unreachable (Pitfall A5/A9)**`DefaultRules` ends with catch-all rules (`DstPort: 0`) that match any TCP/UDP packet. Appending user rules after them makes user rules unreachable. Split into `SpecificRules` + `CatchAllRules`; merge order must be `userRules + SpecificRules + CatchAllRules`.
3. **User-defined classes have no bank layer — panic or silence (Pitfall A6)**`NewBank` currently iterates `classify.AllClasses()` (a hardcoded list of 14 built-in classes). User classes will not be in that list. `NewBank` must iterate the keys of the merged `FreqConfig` map instead. Validate at config load that every rule's class name resolves to a configured sound entry.
4. **Naive square/sawtooth/triangle waveforms produce audible aliasing (Pitfall A3)** — Direct time-domain math generates infinite harmonics that alias above Nyquist. Use additive synthesis: generate a bandlimited `[]HarmonicDef` series (odd harmonics for square/triangle, all harmonics for sawtooth, truncated at Nyquist) at config load time. The existing `HarmonicDef` infrastructure already supports this approach.
5. **BurntSushi/toml silently ignores unknown keys by default (Pitfall A2)** — Use `toml.Decode()` (not `Unmarshal`) to obtain `MetaData`, then call `md.Undecoded()` and return an error listing any unrecognized keys. Implement strict decoding from the first config load function.
2026-03-24 22:36:28 +01:00
## Implications for Roadmap
2026-03-26 16:54:31 +01:00
The v1.1 work has clear dependency ordering that directly dictates phase structure. The build order in ARCHITECTURE.md (7 steps, each independently testable) maps naturally to implementation phases.
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
### Phase 1: Waveform Types in the Oscillator
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
**Rationale:** Zero external dependencies; pure math testable in isolation with golden-sample unit tests. The critical design decision — additive synthesis vs. direct math — must be made and locked in here. Switching after integration is a full oscillator rewrite.
**Delivers:** `Waveform` enum, `sampleAt()` dispatch, `BandlimitedHarmonics()` generator; all four waveform types produce correct, alias-free output at all NetSynth frequencies.
**Addresses:** Custom waveform per class (table stakes)
**Avoids:** Pitfall A3 (aliasing from naive waveforms)
**Files changed:** `synth/oscillator.go` only
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
### Phase 2: Decouple Bank from Global Config
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
**Rationale:** Prerequisite for config injection. `NewBank` must accept an injected config map before the `config` package exists. Wiring `Waveform` through `FreqConfig``Layer``Oscillator` is included here; zero-value default (`WaveformSine = 0`) means existing tests pass unchanged.
**Delivers:** `NewBank(tau, cfgs map[TrafficClass]FreqConfig)`, `FreqConfig.Waveform` field, `synth/layer.go` updated. System is functionally identical to v1.0 but injectable.
**Avoids:** Global-read anti-pattern (multiple places reading `ClassFreqConfigs`, ambiguous merge point)
**Files changed:** `synth/config.go`, `synth/layer.go`, `synth/bank.go`, `encode/mp3.go`
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
### Phase 3: Config Package — TOML Loading and Merge
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
**Rationale:** Core new infrastructure. Builds on the injectable bank signature from Phase 2. All config correctness patterns (pointer fields, strict decode, validation, string normalization) must be established here in isolation before any features are wired to the bank. Retrofitting these patterns after pipeline integration is significantly more expensive.
**Delivers:** `config/` package with `Load()`, `os.UserConfigDir()` discovery, pointer-field merge, `validate()` with string normalization and enum checking, `DefaultConfig()` wrapping existing values.
**Addresses:** Config auto-discovery, `--config` flag, partial override semantics, unknown field detection, startup validation, clear error messages
**Avoids:** Pitfalls A1 (zero-value overwrite), A2 (silent typos), A4 (waveform string case), A7 (XDG ignored), A8 (missing explicit path), A10 (empty class name)
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
### Phase 4: Classification Rule Merging
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
**Rationale:** `classify/rules.go` must be split before any config-loading logic references the rule slice. The three-layer merge order is a design decision that, if wrong, produces silent failures with no error messages — it must be validated with unit tests before pipeline integration.
**Delivers:** `classify.SpecificRules`, `classify.CatchAllRules`, `classify.MergeRules(userRules []Rule) []Rule`; user rules prepend correctly with both catch-all and specific-rule ordering.
**Avoids:** Pitfall A5 (unreachable rules after catch-alls), Pitfall A9 (user rule shadowed by built-in specific rule for same port)
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
### Phase 5: Wire Config Through Pipeline — Frequency and Waveform Overrides
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
**Rationale:** Connects the config package to the synth layer for built-in classes only. Validates the full pipeline end-to-end before adding the complexity of user-defined classes. `encode.RunSynthesis` signature change is a breaking API change — all call sites must be updated in a single commit.
**Delivers:** `--config` cobra flag, `config.Load()` in `main.go`, merged freq config map passed to `RunSynthesis` and `NewBank`; end-to-end test: TOML sets HTTPS to 200 Hz sawtooth, bank produces 200 Hz sawtooth layer.
**Implements:** Config → synth integration
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
### Phase 6: User-Defined Classes End-to-End
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
**Rationale:** The most complex integration; requires all prior phases. User-defined classes create new `TrafficClass` strings that must exist in both the merged rule set and the bank's layer map. The AllClasses() decoupling from Phase 2 makes this tractable.
**Delivers:** `[[rules]]` TOML section, dynamic `TrafficClass` values from config, bank layers constructed from merged FreqConfig map keys, class name cross-validation at config load.
**Avoids:** Pitfall A6 (user-defined class has no bank layer — nil panic or silence)
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
### Phase 7: Print-Config and UX Polish
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
**Rationale:** `--print-config` is independent of capture and must wait until all config structure is stable (Phase 6). Additive, zero regression risk.
**Delivers:** `netsynth --print-config` subcommand with commented TOML output of effective config; optional `name` field on user rules displayed in exit summary.
2026-03-24 22:36:28 +01:00
### Phase Ordering Rationale
2026-03-26 16:54:31 +01:00
- Phases 12 are internal refactors with no user-visible change — the right starting point for establishing patterns safely
- Phase 3 owns all config safety in one isolated package before anything is wired — retrofitting pointer-field merge after bank integration means touching multiple packages simultaneously
- Phase 4 (rule splitting) must precede Phase 6 (user rules) or catch-all ordering bugs surface silently at integration with no clear failure signal
- Phase 5 validates the full pipeline with familiar built-in classes before Phase 6 introduces the harder user-defined class problem
- Phase 7 is pure additive polish with zero risk of breaking earlier phases
2026-03-24 22:36:28 +01:00
### Research Flags
2026-03-26 16:54:31 +01:00
Phases with well-documented patterns — skip additional research:
- **Phase 1:** DSP textbook math; harmonic series are fully specified
- **Phase 2:** Standard dependency injection refactor; no unknowns
- **Phase 3:** BurntSushi/toml API is well-documented; pointer-field merge is a known TOML pattern
- **Phase 4:** Simple slice manipulation; no external dependencies
- **Phase 7:** Cobra subcommand and TOML marshal are standard patterns
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
Phases that may benefit from a targeted research pass or design review:
- **Phase 5:** `encode.RunSynthesis` signature change is a breaking API change — verify all test call sites and plan a single-commit update
- **Phase 6:** User-defined class name collision with built-in `TrafficClass` string values (e.g., user names a class `"HTTPS"`) requires a design decision: treat as override of built-in sound vs. reject as ambiguous. Not resolved in research; decide before coding Phase 6.
2026-03-24 22:36:28 +01:00
## Confidence Assessment
| Area | Confidence | Notes |
|------|------------|-------|
2026-03-26 16:54:31 +01:00
| Stack | HIGH | Existing stack validated in v1.0; BurntSushi/toml v1.6.0 confirmed via pkg.go.dev and GitHub releases; `os.UserConfigDir()` XDG behavior confirmed against official Go stdlib docs |
| Features | HIGH | Config file conventions verified against XDG spec, git, golangci-lint, and mise patterns; TOML schema grounded in existing v1.0 codebase types |
| Architecture | HIGH | Based on direct code inspection of the 3,254-line v1.0 codebase; all integration points identified with specific file/line references and build order prescribed |
| Pitfalls | HIGH | TOML default-overwrite behavior verified against upstream issue trackers (BurntSushi/toml #47, go-toml #252); aliasing prevention verified against DSP literature (CCRMA paper, McGill bandlimited synthesis notes) |
2026-03-24 22:36:28 +01:00
2026-03-26 16:54:31 +01:00
**Overall confidence:** HIGH
2026-03-24 22:36:28 +01:00
### Gaps to Address
2026-03-26 16:54:31 +01:00
- **User class name collision with built-in class strings:** If a user writes `class = "HTTPS"` in a `[[rules]]` block, the intent could be "override built-in sound" or "create a parallel custom class." The merge logic needs an explicit decision before Phase 6: treat matching names as overrides (simplest) or require a separate TOML section. This is a UX design question — resolve before coding Phase 6.
- **BurntSushi/toml vs. go-toml v2:** Both work for this use case. BurntSushi is recommended for ergonomics, but if the team prefers go-toml v2's `DisallowUnknownFields()` pattern, it is equally valid with minor API differences. Either choice is fine; just make one and be consistent.
2026-03-24 22:36:28 +01:00
## Sources
### Primary (HIGH confidence)
2026-03-26 16:54:31 +01:00
- Direct code inspection: `synth/config.go`, `synth/oscillator.go`, `synth/bank.go`, `synth/layer.go`, `classify/classifier.go`, `classify/rules.go`, `classify/types.go`, `encode/mp3.go`, `cmd/netsynth/main.go`
- `pkg.go.dev/github.com/BurntSushi/toml` — v1.6.0 API, `Undecoded()` strict mode, pointer field behavior
- `github.com/BurntSushi/toml/issues/47` — default-overwrite behavior when using `Unmarshal` confirmed
- `pkg.go.dev/os#UserConfigDir` — XDG_CONFIG_HOME behavior on Linux confirmed via official Go stdlib docs
- `ccrma.stanford.edu/~stilti/papers/blit.pdf` — bandlimited synthesis theory (alias-free waveforms)
- `music.mcgill.ca/~gary/307/week5/bandlimited.html` — truncated harmonic series approach confirmed
2026-03-24 22:36:28 +01:00
### Secondary (MEDIUM confidence)
2026-03-26 16:54:31 +01:00
- XDG Base Directory Specification — config discovery precedence order
- `mise.jdx.dev/configuration.html` — working-dir + XDG config discovery pattern
- `golangci-lint.run/docs/configuration/cli/` — partial override config in Go CLI tools
- `dylanmeeus.github.io/posts/audio-from-scratch-pt8/` — Go waveform synthesis from scratch, confirms no library needed
- `pkg.go.dev/github.com/pelletier/go-toml/v2` — v2.3.0 strict decoder comparison; `pelletier/go-toml/issues/252` partial v2 resolution of default-overwrite
2026-03-24 22:36:28 +01:00
### Tertiary (LOW confidence)
2026-03-26 16:54:31 +01:00
- Competitor feature table (SoNSTAR, Network-Sonification, Peep) — niche domain, limited documentation; used for context only, not binding decisions
2026-03-24 22:36:28 +01:00
---
2026-03-26 16:54:31 +01:00
*Research completed: 2026-03-26*
2026-03-24 22:36:28 +01:00
*Ready for roadmap: yes*