gurix/yoloyolo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6015b0e4b9c1144e40f79ebe71fce16c1a2a305e
yoloyolo/.planning/phases/03-pipeline-integration-and-mvp/03-RESEARCH.md
T

25 KiB
Raw Blame History

Phase 3: Pipeline Integration and MVP - Research

Researched: 2026-03-26 Domain: Go pipeline wiring, hash-bucketing classifier extension, audio configuration update, encoding feedback UX Confidence: HIGH

Summary

Phase 3 is almost entirely an integration and light-extension phase. Phases 1 and 2 produced a fully functional capture-classify-aggregate pipeline AND a fully functional audio synthesis/MP3 encoder. The only reason they are not yet connected is a single TODO(phase-3) stub in main.go at line 99. The mechanical wiring is one line: replace _ = collectedSnapshots with encode.RunSynthesis(collectedSnapshots, outputPath). The rest of the phase extends the TrafficClass type system from 11 classes to 14 (replacing ClassUnknown with ClassUnknown1-ClassUnknown4), updates synth/config.go to add tone configs for the 4 new classes and recalculate GainPerLayer, and adds encoding-progress feedback to the UX flow in main.go.

The project's existing patterns — io.Writer injection, AllClasses() driving loops in bank.go, config-driven ClassFreqConfigs map — mean that all synthesis and summary code automatically picks up the new classes with zero changes beyond updating the type definitions and config map. Hash-bucketing is a deterministic (dstPort*31 + protocolNum) % 4 style operation inserted at the end of classifier.Classify() as a tail clause replacing the bare ClassUnknown return.

Primary recommendation: Execute in four focused, independently testable tasks: (1) extend classify/types.go; (2) add hash-bucketing to classifier.Classify(); (3) update synth/config.go for 14 layers; (4) wire main.go and add encoding feedback. Tests for tasks 1-3 follow existing package test patterns. Task 4 confirms end-to-end correctness by building and running with a loopback interface.

<user_constraints>

User Constraints (from CONTEXT.md)

Locked Decisions

  • D-01: Hash-bucketing approach — deterministic hash of (dst_port, protocol) into 4 fixed buckets. No k-means, no warmup period, no external dependency.
  • D-02: 4 buckets for unknown traffic. Total layer count becomes 14 (10 known + 4 unknown buckets).
  • D-03: Hash-bucketing happens inside the existing classifier — classifier returns unknown-1 through unknown-4 instead of plain unknown. New TrafficClass constants added.
  • D-04: Replace ClassUnknown entirely with ClassUnknown1-ClassUnknown4. AllClasses() returns 14 classes. Clean break, no fallback.
  • D-05: All 4 unknown buckets live in a dedicated 850-1100 Hz dissonant range, above the known protocol range (60-800 Hz). Slightly detuned intervals between them.
  • D-06: Same dissonant harmonic character for all 4 buckets (matching Phase 2's D-04/D-06 for the original "unknown" tone). Differentiated by pitch only, preserving the "something foreign" sonic identity.
  • D-07: Status line during encoding: print Encoding N windows to <path>... then Saved <path> (Xs, N KB, encoded in Xs) on completion. Minimal but confirms encoding is happening.
  • D-08: Protocol summary prints BEFORE encoding. Ctrl+C → protocol summary → "Encoding..." → "Saved". User sees capture stats immediately.
  • D-09: Saved line includes path, audio duration, file size, and encoding time. e.g., Saved out.mp3 (12.5s, 198 KB, encoded in 0.3s).
  • D-10: Unknown buckets appear in the protocol summary naturally as unknown-1: 42, unknown-2: 17, etc. Falls out of existing summary logic since they're full TrafficClass values.

Claude's Discretion

  • Exact hash function for port/protocol → bucket mapping (within the 4-bucket constraint)
  • Exact Hz values for the 4 unknown bucket tones (within 850-1100 Hz range, detuned intervals)
  • GainPerLayer recalculation for 14 layers (was 1/11 for 11 layers)
  • Stereo panning positions for the 4 unknown buckets
  • Encoding time measurement implementation

Deferred Ideas (OUT OF SCOPE)

None — discussion stayed within phase scope </user_constraints>

<phase_requirements>

Phase Requirements

ID Description Research Support
CAPT-03 Capture runs until user presses Ctrl+C, then gracefully flushes and saves MP3 The existing signal.NotifyContext shutdown path already drains all channels before the accumulator loop exits; RunSynthesis simply needs to be called with the collected snapshots and the output path. D-07/D-08/D-09 add the feedback messages around the call.
CLAS-02 Unrecognized traffic is auto-clustered and assigned unique tones automatically Hash-bucketing in classifier.Classify() tail clause maps (dstPort, protocolNum) deterministically to ClassUnknown1-ClassUnknown4. New constants in classify/types.go, new FreqConfig entries in synth/config.go covering 850-1100 Hz.
</phase_requirements>

Standard Stack

Core (no changes — inherited from Phases 1 and 2)

Library Version Purpose
github.com/gopacket/gopacket v1.5.0 Packet capture/decoding — already in go.mod
github.com/packetcap/go-pcap v0.0.0-20251215 Live capture backend — already in go.mod
github.com/sjzar/go-lame v0.0.9 MP3 encoding — already in go.mod
github.com/spf13/cobra v1.10.2 CLI — already in go.mod

No new dependencies required for Phase 3.

Installation: None — all dependencies are already in go.mod and go.sum.

Architecture Patterns

Existing Project Structure (Phase 3 touches these files)

classify/
├── types.go         # ADD ClassUnknown1-4, REMOVE ClassUnknown, UPDATE AllClasses()
├── classifier.go    # ADD hash-bucketing tail clause in Classify()
├── rules.go         # No change
└── classifier_test.go  # UPDATE: test unknown bucket routing
synth/
├── config.go        # ADD 4 FreqConfig entries, UPDATE NumLayers=14, GainPerLayer=1/14
├── bank.go          # No change (loops over AllClasses() — picks up new classes automatically)
└── bank_test.go     # UPDATE: test 14-layer count assertion
cmd/netsynth/
└── main.go          # WIRE RunSynthesis, ADD encoding feedback messages (D-07/D-08/D-09)

Pattern 1: TrafficClass Extension

What: Add constants to classify/types.go, update AllClasses(), add config entries to synth/config.go. All synthesis and summary loops that iterate AllClasses() automatically pick up the 4 new classes.

How it works end-to-end:

  • NewBank(tau) in bank.go calls classify.AllClasses() to populate the layers map
  • RenderWindow() loops classify.AllClasses() for both amplitude updates and frame mixing
  • PrintSummary() in aggregate/summary.go sorts and prints whatever keys are in totals — it is map-driven, not AllClasses()-driven, so no change needed there

Example — types.go change:

const (
    ClassICMP     TrafficClass = "ICMP"
    ClassDNS      TrafficClass = "DNS"
    ClassHTTPS    TrafficClass = "HTTPS"
    ClassHTTP     TrafficClass = "HTTP"
    ClassSSH      TrafficClass = "SSH"
    ClassSMTP     TrafficClass = "SMTP"
    ClassNTP      TrafficClass = "NTP"
    ClassDHCP     TrafficClass = "DHCP"
    ClassOtherTCP TrafficClass = "other-TCP"
    ClassOtherUDP TrafficClass = "other-UDP"
    // ClassUnknown REMOVED — replaced by 4 hash buckets (D-04)
    ClassUnknown1 TrafficClass = "unknown-1"
    ClassUnknown2 TrafficClass = "unknown-2"
    ClassUnknown3 TrafficClass = "unknown-3"
    ClassUnknown4 TrafficClass = "unknown-4"
)

func AllClasses() []TrafficClass {
    return []TrafficClass{
        ClassICMP, ClassDNS, ClassHTTPS, ClassHTTP, ClassSSH,
        ClassSMTP, ClassNTP, ClassDHCP, ClassOtherTCP, ClassOtherUDP,
        ClassUnknown1, ClassUnknown2, ClassUnknown3, ClassUnknown4,
    }
}

Pattern 2: Hash-Bucketing Tail Clause

What: The existing Classify() method returns ClassUnknown as a bare fallback in three places: after the ICMP rule loop, at the end of the TCP block (no matching port rule), at the end of the UDP block (no matching port rule), and in the final fall-through (no recognized transport layer). Each of these becomes a hashBucket(result.DstPort, result.Protocol) call.

Hash function design (Claude's Discretion):

A deterministic hash of (dstPort, protocolNum) into 4 buckets:

// hashBucket maps an unrecognized packet to one of 4 unknown classes.
// Uses a simple polynomial hash to spread common port ranges across buckets.
func hashBucket(dstPort uint16, protocol string) TrafficClass {
    var protoNum uint16
    switch protocol {
    case "tcp":
        protoNum = 6
    case "udp":
        protoNum = 17
    case "icmp":
        protoNum = 1
    default:
        protoNum = 0
    }
    // Polynomial mix — prime multipliers prevent trivially adjacent ports
    // landing in the same bucket.
    h := uint32(dstPort)*31 + uint32(protoNum)*7
    switch h % 4 {
    case 0:
        return ClassUnknown1
    case 1:
        return ClassUnknown2
    case 2:
        return ClassUnknown3
    default:
        return ClassUnknown4
    }
}

Why this is correct: ARP packets (no transport layer) have dstPort=0, protocol="" → bucket 0 → ClassUnknown1. This is fine; the dissonant tone still distinguishes them from known traffic. Determinism means the same port always lands in the same bucket across runs, which is desirable: if a user always sees a custom service on port 8443, it always gets the same tone.

Where to apply: Replace the 4 bare ClassUnknown return points in classifier.go with hashBucket(result.DstPort, result.Protocol).

Pattern 3: synth/config.go Update

What: Remove the ClassUnknown entry, add 4 new entries in the 850-1100 Hz dissonant range. Update NumLayers from 11 to 14 and GainPerLayer from 1.0/11 to 1.0/14.

Recommended Hz values (Claude's Discretion — within D-05 850-1100 Hz constraint):

The original ClassUnknown was at 437 Hz (deliberately flat from SMTP's 440 Hz). The 4 new buckets should be spread across 850-1100 Hz with slightly detuned intervals to create subtle beating when multiple unknown buckets are active simultaneously. Suggested values:

Class Hz Rationale
ClassUnknown1 862 ~11 cents flat from 900 Hz
ClassUnknown2 920 ~3 Hz above a "clean" interval from Unknown1
ClassUnknown3 981 ~7 cents flat from 990 Hz
ClassUnknown4 1047 ~9 cents sharp from 1040 Hz

Harmonics follow Phase 2's D-06 dissonant profile: {1, 1.0}, {2, 0.8}, {3, 0.4} (rich upper harmonics = "foreign" character).

Recommended pan positions (Claude's Discretion):

  • ClassUnknown1: 0.6 (right-of-center)
  • ClassUnknown2: -0.6 (left-of-center)
  • ClassUnknown3: 0.9 (wide right)
  • ClassUnknown4: -0.9 (wide left)

Spreading unknowns wide in the stereo field sonically separates them from the known protocols, which occupy center and moderate positions.

GainPerLayer recalculation: 1.0 / 14 ≈ 0.0714. The existing TestMixerNoClip test exercises all-layers-at-max — it asserts [-1, 1] bounds. With 14 layers at GainPerLayer=1/14 and PanGains() applying constant-power panning, worst-case is 14 * (1/14) * maxPanGain = 1.0 * maxPanGain. The PanGains function returns values where gainL^2 + gainR^2 = 1, so gainL, gainR <= 1.0. Clipping cannot occur. The test will pass with the updated constant.

Pattern 4: main.go Wiring and Feedback

What: Replace the TODO stub at line 99 with:

  1. Move PrintSummary call BEFORE RunSynthesis (D-08: summary first)
  2. Print encoding status line before calling RunSynthesis (D-07)
  3. Measure encoding time with time.Now() (Claude's Discretion)
  4. Call encode.RunSynthesis(collectedSnapshots, outputPath)
  5. Compute audio duration from snapshot count: float64(len(collectedSnapshots)) * float64(aggregate.DefaultWindowMs) / 1000.0
  6. Stat the output file for size
  7. Print the "Saved" line (D-09)

Import addition: "github.com/netsynth/netsynth/encode" — the only new import in main.go.

Example wiring sequence:

// Print exit summary FIRST (D-08)
dropped := atomic.LoadInt64(droppedPtr)
if dropped > 0 {
    fmt.Fprintf(os.Stderr, "\nWarning: %d packets dropped (channel buffer full)\n", dropped)
}
aggregate.PrintSummary(os.Stderr, totals)

// Encode to MP3 with feedback (D-07/D-09/CAPT-03)
fmt.Fprintf(os.Stderr, "Encoding %d windows to %s...\n", len(collectedSnapshots), outputPath)
encodeStart := time.Now()
if err := encode.RunSynthesis(collectedSnapshots, outputPath); err != nil {
    return fmt.Errorf("synthesis failed: %w", err)
}
encodeElapsed := time.Since(encodeStart)
audioDuration := float64(len(collectedSnapshots)) * float64(aggregate.DefaultWindowMs) / 1000.0
info, err := os.Stat(outputPath)
if err != nil {
    return fmt.Errorf("stat output: %w", err)
}
fmt.Fprintf(os.Stderr, "Saved %s (%.1fs, %d KB, encoded in %.1fs)\n",
    outputPath,
    audioDuration,
    info.Size()/1024,
    encodeElapsed.Seconds(),
)
return nil

Note: The existing return nil at line 109 is replaced by this block.

Anti-Patterns to Avoid

  • Keeping ClassUnknown as a fallback: D-04 mandates clean removal. Any remaining ClassUnknown reference after Phase 3 is a bug — the compiler will catch it in synth/config.go if the map entry is not updated, but it won't catch it if a default return in classifier.go still uses it.
  • Forgetting to update bank_test.go: TestNewBankHas11Layers hardcodes 11 — it must be updated to 14 or the test suite will fail after the AllClasses() change.
  • Calling PrintSummary after RunSynthesis: D-08 requires summary first. The user may wait several seconds for encoding; seeing the protocol counts immediately is the UX goal.
  • Creating the output file before the zero-packet check: RunSynthesis already guards this (Phase 2 OUT-03). The main.go wiring doesn't need a redundant check — just handle the returned error.
  • Hardcoding audio duration from time elapsed: Audio duration is deterministic from snapshot count × window duration, not from wall clock. Use len(collectedSnapshots) * DefaultWindowMs / 1000 seconds.

Don't Hand-Roll

Problem Don't Build Use Instead
MP3 encoding Custom MP3 writer encode.RunSynthesis() — already exists and tested
Synthesis Custom oscillator synth.NewBank()/bank.RenderWindow() — already exists
Signal handling Custom signal loop signal.NotifyContext — already wired in main.go
Protocol classification New classifier Extend existing Classify() with hashBucket() tail
Elapsed time Custom timer time.Now() / time.Since() — stdlib

Key insight: Phase 3 is almost all integration. Custom solutions for any of these would contradict already-shipped, tested implementations.

Common Pitfalls

Pitfall 1: Missing ClassUnknown removal causes silent map miss in bank.go

What goes wrong: If ClassUnknown is left in synth/config.go but removed from AllClasses(), NewBank() will never create a layer for it and the config entry is dead. Conversely, if AllClasses() still returns ClassUnknown but ClassFreqConfigs has no entry for it, NewBank() will call NewLayer(cfg, ...) with a zero FreqConfig — a silent oscillator at 0 Hz that wastes memory and produces no sound. The compiler does not catch this.

How to avoid: Update types.go, config.go, and classifier.go in the same plan wave so they're always consistent. Write a test that calls AllClasses() and asserts every returned class has a corresponding entry in synth.ClassFreqConfigs.

Warning signs: TestNewBankHas11Layers still passes but producing 11 ≠ 14 audio layers.

Pitfall 2: bank_test.go hardcoded layer count

What goes wrong: TestNewBankHas11Layers asserts len(b.layers) != 11. After updating AllClasses() to return 14 entries, this test fails. The test suite will be broken before any synthesis runs.

How to avoid: Update TestNewBankHas11Layers to assert 14 when updating synth/config.go and classify/types.go.

Pitfall 3: classifier_test.go TestClassifyUnknown expects ClassUnknown

What goes wrong: TestClassifyUnknown at line 222 asserts got.Class != classify.ClassUnknown. After D-04 removes ClassUnknown, this will fail to compile.

How to avoid: Update TestClassifyUnknown to assert that the returned class is one of ClassUnknown1-ClassUnknown4 (i.e., the string has "unknown-" prefix), not the removed constant.

Pitfall 4: Audio duration shows 0 seconds for short captures

What goes wrong: len(collectedSnapshots) * DefaultWindowMs / 1000 uses integer arithmetic. For captures shorter than 1 second (< 2 windows at 500ms), integer division truncates to 0.

How to avoid: Cast to float64 before dividing: float64(len(collectedSnapshots)) * float64(aggregate.DefaultWindowMs) / 1000.0.

Pitfall 5: os.Stat on output path fails when RunSynthesis returns an error

What goes wrong: If RunSynthesis returns an error (e.g., zero packets), calling os.Stat(outputPath) immediately after will fail or stat an empty path because the file was never created. The error handling sequence matters.

How to avoid: Only call os.Stat in the happy path — after confirming RunSynthesis returned nil.

Code Examples

Hash-bucketing in classifier.go

// At the end of Classify(), replace all ClassUnknown return sites:

// No recognized transport layer → deterministic bucket assignment (D-01/D-03)
result.Class = hashBucket(result.DstPort, result.Protocol)
return result

// hashBucket maps an unrecognized packet to ClassUnknown1-4.
func hashBucket(dstPort uint16, protocol string) TrafficClass {
    var protoNum uint16
    switch protocol {
    case "tcp":
        protoNum = 6
    case "udp":
        protoNum = 17
    case "icmp":
        protoNum = 1
    }
    h := uint32(dstPort)*31 + uint32(protoNum)*7
    switch h % 4 {
    case 0:
        return ClassUnknown1
    case 1:
        return ClassUnknown2
    case 2:
        return ClassUnknown3
    default:
        return ClassUnknown4
    }
}

synth/config.go updated constants

const (
    SampleRate       = 44100
    WindowMs         = 500
    SamplesPerWindow = SampleRate * WindowMs / 1000  // 22050
    NumLayers        = 14
    GainPerLayer     = 1.0 / float64(NumLayers)      // ~0.0714
    WhisperFloor     = 0.03
)

main.go integration (key flow)

// After snapshot accumulator loop completes:
dropped := atomic.LoadInt64(droppedPtr)
if dropped > 0 {
    fmt.Fprintf(os.Stderr, "\nWarning: %d packets dropped (channel buffer full)\n", dropped)
}
// D-08: summary before encoding
aggregate.PrintSummary(os.Stderr, totals)

// D-07: encoding status
fmt.Fprintf(os.Stderr, "Encoding %d windows to %s...\n", len(collectedSnapshots), outputPath)
encodeStart := time.Now()
if err := encode.RunSynthesis(collectedSnapshots, outputPath); err != nil {
    return fmt.Errorf("synthesis failed: %w", err)
}
encodeElapsed := time.Since(encodeStart)

// D-09: saved confirmation with stats
audioDuration := float64(len(collectedSnapshots)) * float64(aggregate.DefaultWindowMs) / 1000.0
info, statErr := os.Stat(outputPath)
if statErr != nil {
    return fmt.Errorf("stat output file: %w", statErr)
}
fmt.Fprintf(os.Stderr, "Saved %s (%.1fs, %d KB, encoded in %.1fs)\n",
    outputPath, audioDuration, info.Size()/1024, encodeElapsed.Seconds())
return nil

Environment Availability

Dependency Required By Available Version Fallback
Go toolchain Build Verified (go.mod exists, tests pass) go 1.24.1 —
C compiler (gcc) sjzar/go-lame CGo build Assumed (Phase 2 already built) — —
ffprobe encode/mp3_test.go validation Unknown — Tests skip via t.Skip if absent

All tests pass as of Phase 3 start (go test ./... returns ok for all 6 packages). No new external dependencies required.

Validation Architecture

Test Framework

Property Value
Framework Go standard testing package
Config file none — go test ./... convention
Quick run command go test ./classify/... ./synth/... ./cmd/... -count=1
Full suite command go test ./... -count=1

Phase Requirements → Test Map

Req ID Behavior Test Type Automated Command File Exists?
CLAS-02 hashBucket() distributes packets to ClassUnknown1-4 (not ClassUnknown) unit go test ./classify/... -run TestClassifyUnknown -count=1 Update existing ✅
CLAS-02 All 4 buckets are reachable (hash covers all 4 outputs) unit go test ./classify/... -run TestHashBucketDistribution -count=1 New ❌ Wave 0
CLAS-02 AllClasses() returns exactly 14 classes, none is ClassUnknown unit go test ./classify/... -run TestAllClassesCount -count=1 New ❌ Wave 0
CLAS-02 NewBank() creates exactly 14 layers unit go test ./synth/... -run TestNewBankHas11Layers -count=1 Update existing ✅
CLAS-02 ClassFreqConfigs has entry for every class in AllClasses() unit go test ./synth/... -run TestClassFreqConfigsComplete -count=1 New ❌ Wave 0
CAPT-03 RunSynthesis is called in main; non-empty capture produces MP3 integration go test ./encode/... -run TestMP3Valid -count=1 Exists ✅ (passes already)
CAPT-03 Encoding messages printed (Encoding/Saved) — manual verify manual Build and run with sudo ./netsynth -i lo -o /tmp/test.mp3 N/A

Sampling Rate

  • Per task commit: go test ./classify/... ./synth/... -count=1
  • Per wave merge: go test ./... -count=1
  • Phase gate: Full suite green before /gsd:verify-work

Wave 0 Gaps

  • classify/types_test.go (or add to classifier_test.go) — TestAllClassesCount and TestAllClassesNoUnknown
  • classify/classifier_test.go — TestHashBucketDistribution covering all 4 bucket outputs
  • synth/config_test.go — TestClassFreqConfigsComplete asserting every class in AllClasses() has a FreqConfig entry
  • Update synth/bank_test.go:TestNewBankHas11Layers → assert 14 (not 11)
  • Update classify/classifier_test.go:TestClassifyUnknown → assert unknown- prefix, not removed ClassUnknown constant

Open Questions

  1. TestMixerNoClip with 14 layers — will it still pass?

    • What we know: GainPerLayer = 1/14 ≈ 0.0714. With PanGains constant-power law, stereo channels cannot exceed 14 * (1/14) * 1.0 = 1.0. Mathematically safe.
    • What's unclear: Whether the fast-EMA convergence test (tau=0.01, 10 render passes) produces audible clipping at a transient peak during the first few windows.
    • Recommendation: Run TestMixerNoClip immediately after NumLayers and GainPerLayer changes. If it fails, the test scenario (all 14 layers at 1000 pkts simultaneously) is an extreme worst-case that won't occur in real captures; the EMA will naturally limit overshoot.

  2. Unknown packets with no DstPort (ARP, raw Ethernet, etc.) always land in same bucket?

    • What we know: dstPort=0, protocol="" → protoNum=0, h = 0*31 + 0*7 = 0, 0 % 4 = 0 → always ClassUnknown1.
    • What's unclear: Whether this is a concern. ARP produces low packet rates and the behavior is deterministic — acceptable per D-01.
    • Recommendation: Document in a comment in hashBucket(). No code change needed.

Sources

Primary (HIGH confidence)

  • Direct code reading: cmd/netsynth/main.go, classify/types.go, classify/classifier.go, synth/config.go, synth/bank.go, encode/mp3.go, aggregate/summary.go — all integration points verified by inspection.
  • go test ./... run result: all 6 packages pass — baseline confirmed.
  • go.mod — all required dependencies already present at pinned versions.

Secondary (MEDIUM confidence)

  • Phase 2 STATE.md decisions ([Phase 02]: GainPerLayer applied in bank.go during mixing) — confirm loop structure, no surprises.
  • Phase 3 CONTEXT.md — all implementation decisions locked (D-01 through D-10).

Tertiary (LOW confidence)

  • Tone Hz recommendations (862, 920, 981, 1047) — chosen within D-05 constraints by applying chromatic detuning logic; no user validation yet. Subjective listening validation remains the final arbiter.

Metadata

Confidence breakdown:

  • Integration wiring: HIGH — one-line change, existing APIs match
  • Hash-bucketing logic: HIGH — deterministic, no external deps, existing test patterns apply
  • Tone Hz values: MEDIUM — within D-05 constraint, detuning math is sound, but subjective validation needed
  • Test gap identification: HIGH — all test files inspected, hardcoded constants located

Research date: 2026-03-26 Valid until: Phase 3 complete (low churn domain; these are all in-repo findings)

Reference in New Issue View Git Blame Copy Permalink