345 lines
21 KiB
Markdown
345 lines
21 KiB
Markdown
# Architecture Research
|
||
|
||
**Domain:** Network traffic sonification CLI (Go)
|
||
**Researched:** 2026-03-24
|
||
**Confidence:** MEDIUM — Go audio synthesis patterns verified via official docs and real libraries; sonification architecture inferred from academic literature (SoNSTAR) and Go concurrency canon.
|
||
|
||
## Standard Architecture
|
||
|
||
### System Overview
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ CLI Entry Point │
|
||
│ (flags: interface, output path, duration) │
|
||
└───────────────────────────┬─────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Capture Layer │
|
||
│ ┌─────────────────────────────────────────────────────┐ │
|
||
│ │ PacketSource (gopacket/pcap) │ │
|
||
│ │ Produces: chan Packet │ │
|
||
│ └──────────────────────┬──────────────────────────────┘ │
|
||
└─────────────────────────┼───────────────────────────────────┘
|
||
│ raw packet stream
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Classification Layer │
|
||
│ ┌─────────────────────────┐ ┌─────────────────────────┐ │
|
||
│ │ Protocol Classifier │ │ Unknown Traffic │ │
|
||
│ │ (ICMP, DNS, HTTPS, │ │ Clusterer │ │
|
||
│ │ SSH, TCP-other, UDP) │ │ (feature-based bucketer)│ │
|
||
│ └───────────┬─────────────┘ └────────────┬────────────┘ │
|
||
│ └──────────────┬──────────────┘ │
|
||
└─────────────────────────────┼───────────────────────────────┘
|
||
│ classified packet events
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Aggregation Layer │
|
||
│ ┌─────────────────────────────────────────────────────┐ │
|
||
│ │ Time-Window Accumulator │ │
|
||
│ │ - fixed window (e.g. 500ms) │ │
|
||
│ │ - counts + byte-volume per traffic class │ │
|
||
│ │ Produces: chan WindowSnapshot │ │
|
||
│ └──────────────────────┬──────────────────────────────┘ │
|
||
└─────────────────────────┼───────────────────────────────────┘
|
||
│ window snapshots
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Synthesis Layer │
|
||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||
│ │ Layer 0 │ │ Layer 1 │ │ Layer N │ │ Layer X │ │
|
||
│ │ (ICMP) │ │ (DNS) │ │ (HTTPS) │ │ (auto) │ │
|
||
│ │ Osc+Amp │ │ Osc+Amp │ │ Osc+Amp │ │ Osc+Amp │ │
|
||
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
|
||
│ └──────────────┴────────────┴──────────────┘ │
|
||
│ │ │
|
||
│ ┌─────▼──────┐ │
|
||
│ │ Mixer │ │
|
||
│ │ (sum+clip) │ │
|
||
│ └─────┬──────┘ │
|
||
└──────────────────────────┼──────────────────────────────────┘
|
||
│ PCM sample stream (float32[])
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Encoding Layer │
|
||
│ ┌─────────────────────────────────────────────────────┐ │
|
||
│ │ PCM Buffer Accumulator → LAME MP3 Encoder │ │
|
||
│ │ (go-lame / CGo libmp3lame) │ │
|
||
│ └──────────────────────┬──────────────────────────────┘ │
|
||
└─────────────────────────┼───────────────────────────────────┘
|
||
│ .mp3 file
|
||
▼
|
||
Output File
|
||
```
|
||
|
||
### Component Responsibilities
|
||
|
||
| Component | Responsibility | Typical Implementation |
|
||
|-----------|----------------|------------------------|
|
||
| CLI Entry | Parse flags, wire all components, handle Ctrl+C via `os.Signal` | `main.go`, `cobra` or `flag` package |
|
||
| PacketSource | Open interface via pcap/AF_PACKET, emit packets into channel | `gopacket.PacketSource.Packets()` → `<-chan gopacket.Packet` |
|
||
| Protocol Classifier | Inspect decoded layers (IP, TCP, UDP, ICMP, DNS); assign class label | Pure Go switch on `packet.Layer()` type assertions |
|
||
| Unknown Traffic Clusterer | Hash or bucket unclassified flows by port range / packet size signature; assign stable label ID | Simple feature-hash bucketer; no heavy ML needed for v1 |
|
||
| Time-Window Accumulator | Batch packets into N-ms windows; emit packet-count and byte-volume per class | `ticker`-driven goroutine, map accumulation |
|
||
| Sound Layer (per class) | Maintain a sine oscillator at a fixed root frequency; update amplitude from window snapshot | Oscillator struct with phase accumulator; amplitude lerp |
|
||
| Mixer | Sum all layer outputs sample-by-sample; clamp/normalize to [-1, 1] | Simple additive sum with soft clip |
|
||
| MP3 Encoder | Accept PCM float32 frames; encode to MP3 on flush/stop | go-lame (CGo) or pure-Go fallback |
|
||
| Output File | Write encoded bytes to disk path from CLI flag | `os.File` + buffered writer |
|
||
|
||
## Recommended Project Structure
|
||
|
||
```
|
||
netsynth/
|
||
├── main.go # CLI wiring, signal handling, top-level orchestration
|
||
├── capture/
|
||
│ └── capture.go # PacketSource wrapper, interface open/close, chan Packet
|
||
├── classify/
|
||
│ ├── classifier.go # Protocol dispatch, class label assignment
|
||
│ └── cluster.go # Unknown traffic bucketer (feature hash)
|
||
├── aggregate/
|
||
│ └── window.go # Time-window accumulator, WindowSnapshot type
|
||
├── synth/
|
||
│ ├── oscillator.go # Phase-accumulator sine oscillator
|
||
│ ├── layer.go # Per-traffic-class sound layer (osc + amp target)
|
||
│ └── mixer.go # Sum layers → float32 PCM frames
|
||
├── encode/
|
||
│ └── mp3.go # PCM → MP3 via go-lame; file flush on close
|
||
└── config/
|
||
└── mapping.go # Protocol → frequency/harmonic assignment table
|
||
```
|
||
|
||
### Structure Rationale
|
||
|
||
- **capture/:** Isolates pcap/root-privilege boundary. Everything above it operates on typed Go channels with no pcap dependency.
|
||
- **classify/:** Cleanly separates rule-based (known protocol) from heuristic (unknown cluster) logic. Each can be tested with synthetic packet fixtures independently.
|
||
- **aggregate/:** The only stateful time-domain component. Isolating it makes window size configurable without touching synthesis.
|
||
- **synth/:** Pure PCM math — no I/O, no pcap. Fully unit-testable with deterministic inputs. The mixer owns the sample rate constant.
|
||
- **encode/:** CGo boundary lives here and nowhere else. If LAME is replaced (e.g., pure Go encoder), only this package changes.
|
||
- **config/:** Static frequency-to-protocol table. Separating it avoids magic numbers scattered across synth/.
|
||
|
||
## Architectural Patterns
|
||
|
||
### Pattern 1: Channel-Connected Pipeline Stages
|
||
|
||
**What:** Each component is a goroutine that reads from an inbound channel and writes to an outbound channel. The `done` channel (closed on Ctrl+C) signals all stages to drain and exit cleanly.
|
||
|
||
**When to use:** Always — this is the idiomatic Go pipeline pattern described in the Go Blog.
|
||
|
||
**Trade-offs:** Slightly more setup than direct function calls; pays off immediately with clean shutdown and testability of individual stages.
|
||
|
||
**Example:**
|
||
```go
|
||
// Each stage signature follows this pattern
|
||
func Classify(done <-chan struct{}, packets <-chan gopacket.Packet) <-chan ClassifiedPacket {
|
||
out := make(chan ClassifiedPacket, 256)
|
||
go func() {
|
||
defer close(out)
|
||
for {
|
||
select {
|
||
case <-done:
|
||
return
|
||
case pkt, ok := <-packets:
|
||
if !ok { return }
|
||
out <- classify(pkt)
|
||
}
|
||
}
|
||
}()
|
||
return out
|
||
}
|
||
```
|
||
|
||
### Pattern 2: Ticker-Driven Window Flush
|
||
|
||
**What:** The aggregation goroutine owns a `time.Ticker`. On each tick it snapshots accumulated counters and sends a `WindowSnapshot` downstream, then resets counters.
|
||
|
||
**When to use:** Anywhere time-based batching converts a high-frequency stream into low-frequency control signals.
|
||
|
||
**Trade-offs:** Fixed window size (e.g. 500ms) is simple but loses sub-window dynamics. Sliding windows add complexity with marginal benefit for ambient synthesis.
|
||
|
||
**Example:**
|
||
```go
|
||
func Aggregate(done <-chan struct{}, events <-chan ClassifiedPacket, windowMs int) <-chan WindowSnapshot {
|
||
out := make(chan WindowSnapshot, 8)
|
||
ticker := time.NewTicker(time.Duration(windowMs) * time.Millisecond)
|
||
go func() {
|
||
defer close(out)
|
||
counts := map[TrafficClass]int{}
|
||
for {
|
||
select {
|
||
case <-done:
|
||
return
|
||
case <-ticker.C:
|
||
out <- snapshot(counts)
|
||
counts = map[TrafficClass]int{}
|
||
case ev, ok := <-events:
|
||
if !ok { return }
|
||
counts[ev.Class]++
|
||
}
|
||
}
|
||
}()
|
||
return out
|
||
}
|
||
```
|
||
|
||
### Pattern 3: Per-Layer Amplitude Lerp
|
||
|
||
**What:** Each sound layer holds a current amplitude and a target amplitude. On each audio frame the current value moves toward the target by a smoothing coefficient. The layer's oscillator always runs; silence is achieved by targeting amplitude = 0.
|
||
|
||
**When to use:** Whenever window snapshots drive synthesis — avoids clicks/pops from abrupt amplitude changes.
|
||
|
||
**Trade-offs:** Adds minimal CPU overhead (one multiply per frame per layer); necessary for perceptually smooth audio.
|
||
|
||
## Data Flow
|
||
|
||
### Primary Flow: Packets to PCM
|
||
|
||
```
|
||
Network Interface
|
||
│
|
||
▼ (gopacket pcap handle)
|
||
PacketSource.Packets() chan
|
||
│
|
||
▼ (classify goroutine)
|
||
ClassifiedPacket chan
|
||
│
|
||
▼ (aggregate goroutine, ticker)
|
||
WindowSnapshot chan ─────────────────────────────────┐
|
||
│
|
||
(synth goroutine,
|
||
per window snap:
|
||
update amplitude targets)
|
||
│
|
||
PCM frame generator loop
|
||
(renders N frames per window,
|
||
one frame = sum of all layers)
|
||
│
|
||
▼
|
||
PCM []float32 blocks
|
||
│
|
||
▼
|
||
LAME encoder (streaming)
|
||
│
|
||
▼
|
||
MP3 bytes → output file
|
||
```
|
||
|
||
### Shutdown Flow
|
||
|
||
```
|
||
Ctrl+C → os.Signal → close(done) channel
|
||
│
|
||
├── capture goroutine: drain + close packet chan
|
||
├── classify goroutine: drain + close event chan
|
||
├── aggregate goroutine: drain + close snapshot chan
|
||
└── synth goroutine: flush remaining PCM → encoder.Flush() → file.Close()
|
||
```
|
||
|
||
### Key Data Types
|
||
|
||
1. **`gopacket.Packet`** → raw decoded packet from pcap; carries layer stack.
|
||
2. **`ClassifiedPacket{Packet, Class TrafficClass, Bytes int}`** → labeled event.
|
||
3. **`WindowSnapshot{ClassCounts map[TrafficClass]int, ClassBytes map[TrafficClass]int}`** → per-window aggregate; drives amplitude targets.
|
||
4. **`[]float32` PCM block** → mixer output at 44100 Hz, mono; flows into LAME.
|
||
|
||
## Build Order (Phase Implications)
|
||
|
||
Build in dependency order — each layer is independently testable before the next is added:
|
||
|
||
```
|
||
1. capture/ → can test: "does it open an interface and emit packets?"
|
||
2. classify/ → can test: "does ICMP get labeled ICMP?" (synthetic packets)
|
||
3. aggregate/ → can test: "does a 500ms window count correctly?"
|
||
4. synth/ → can test: "does mixer output expected amplitude?" (no pcap needed)
|
||
5. encode/ → can test: "does PCM produce valid MP3 bytes?"
|
||
6. main.go wiring → integration: full end-to-end pipeline
|
||
```
|
||
|
||
This ordering means:
|
||
- **Phase 1** can deliver a working capture + classify pipeline writing JSON/text summaries — validating the hardest privilege/pcap risk early.
|
||
- **Phase 2** delivers the synthesis engine in isolation — testable with synthetic `WindowSnapshot` inputs before any real traffic.
|
||
- **Phase 3** wires them together with the MP3 encoder.
|
||
|
||
## Anti-Patterns
|
||
|
||
### Anti-Pattern 1: Synchronous Per-Packet Audio Rendering
|
||
|
||
**What people do:** Generate one audio sample or tone event per packet — a 10 Gbps link produces 14M packets/sec, making synchronous render impossible.
|
||
|
||
**Why it's wrong:** Breaks at any real traffic volume; produces click-heavy output, not smooth drone.
|
||
|
||
**Do this instead:** Batch packets into time windows (500ms–1s) and drive amplitude targets from the batch, not individual packets.
|
||
|
||
### Anti-Pattern 2: Blocking Channel Sends in the Capture Path
|
||
|
||
**What people do:** Use unbuffered channels between PacketSource and classifier; slow classifier stalls the pcap ring buffer and causes kernel drops.
|
||
|
||
**Why it's wrong:** libpcap's kernel buffer is fixed-size; if userspace can't drain it fast enough, packets are silently dropped. For audio purposes this introduces silent gaps.
|
||
|
||
**Do this instead:** Use buffered channels (capacity 256–1024) between capture and classify. Drop packets on full buffer with a counter — acceptable for sonification, fatal to log completeness tools.
|
||
|
||
### Anti-Pattern 3: CGo MP3 Encoding in the Hot Audio Loop
|
||
|
||
**What people do:** Call `lame.Encode()` synchronously inside the frame-render loop, stalling synthesis.
|
||
|
||
**Why it's wrong:** CGo calls carry overhead; libmp3lame may block on I/O; this disrupts the synthesis clock.
|
||
|
||
**Do this instead:** The synth goroutine pushes PCM blocks onto a buffered channel; a separate encoder goroutine drains and encodes. On shutdown, close the PCM channel and drain completely before `lame.Close()`.
|
||
|
||
### Anti-Pattern 4: Global Mutable State for Class Frequency Mapping
|
||
|
||
**What people do:** Use a global `map[TrafficClass]float64` for frequency assignments modified at runtime.
|
||
|
||
**Why it's wrong:** Race conditions; hard to test; makes the mapping invisible to callers.
|
||
|
||
**Do this instead:** Pass the mapping table as an immutable struct at construction time. Auto-clustered classes append to a local slice protected by a mutex inside the clusterer — not a global.
|
||
|
||
## Integration Points
|
||
|
||
### External Services
|
||
|
||
| Dependency | Integration Pattern | Notes |
|
||
|------------|---------------------|-------|
|
||
| libpcap / pcap.h | CGo via gopacket/pcap — requires libpcap-dev at build time | Can substitute AF_PACKET (linux only) to avoid CGo in capture; still needs root |
|
||
| libmp3lame | CGo via go-lame — requires libmp3lame-dev at build time | Binary distribution requires static linking or Docker; pure-Go MP3 (e.g. oto + gmp3) is an option but quality/speed tradeoff |
|
||
|
||
### Internal Boundaries
|
||
|
||
| Boundary | Communication | Notes |
|
||
|----------|---------------|-------|
|
||
| capture ↔ classify | `chan gopacket.Packet` (buffered 512) | classify must never block capture |
|
||
| classify ↔ aggregate | `chan ClassifiedPacket` (buffered 1024) | aggregate is slower (ticker-driven); buffer absorbs bursts |
|
||
| aggregate ↔ synth | `chan WindowSnapshot` (buffered 4) | synth consumes synchronously per window; small buffer is fine |
|
||
| synth ↔ encode | `chan []float32` (buffered 8 blocks) | encoder runs in separate goroutine to decouple CGo latency |
|
||
| all stages ↔ main | `chan struct{}` done channel | closed on Ctrl+C; all stages select on it |
|
||
|
||
## Scaling Considerations
|
||
|
||
This is a single-binary CLI tool, not a distributed service. Scaling concerns are throughput-based:
|
||
|
||
| Traffic Rate | Architecture Adjustments |
|
||
|--------------|--------------------------|
|
||
| Home/office (< 10K pps) | Default design handles easily with no tuning |
|
||
| Datacenter (100K–1M pps) | Increase capture buffer size; consider AF_PACKET with TPACKET_V3 ring buffer instead of pcap; classify goroutine may need fan-out to 2–4 workers |
|
||
| Line-rate 10G (> 5M pps) | Out of scope for v1 ambient audio tool — synthesis granularity at 500ms windows means exact packet-level accuracy is not required |
|
||
|
||
### Scaling Priorities
|
||
|
||
1. **First bottleneck:** Kernel pcap buffer drops — mitigated by buffered channels and accepting lossy capture (fine for sonification).
|
||
2. **Second bottleneck:** CGo encoding latency coupling synthesis clock — mitigated by decoupled encoder goroutine.
|
||
|
||
## Sources
|
||
|
||
- Go Pipeline patterns: [Go Concurrency Patterns: Pipelines and cancellation](https://go.dev/blog/pipelines) — HIGH confidence, official Go blog
|
||
- SoNSTAR network sonification architecture: [Sonification of Network Traffic Flow for Monitoring and Situational Awareness, arXiv 1712.07029](https://arxiv.org/abs/1712.07029) — MEDIUM confidence (abstract only accessed)
|
||
- gopacket channel API: [gopacket pkg.go.dev](https://pkg.go.dev/github.com/google/gopacket) — HIGH confidence, official package docs
|
||
- bleep synthesizer architecture (Go): [GitHub bspaans/bleep](https://github.com/bspaans/bleep) — MEDIUM confidence (README inspection)
|
||
- Waveform synthesis PCM patterns in Go: [Audio From Scratch With Go — Dylan Meeus](https://dylanmeeus.github.io/posts/audio-from-scratch-pt8/) — MEDIUM confidence
|
||
- go-lame MP3 encoding: [go-lame pkg.go.dev](https://pkg.go.dev/github.com/sunicy/go-lame) — MEDIUM confidence
|
||
- Drone amplitude/frequency modulation patterns: [Drone auralization model, Acta Acustica 2024](https://acta-acustica.edpsciences.org/articles/aacus/full_html/2024/01/aacus240076/aacus240076.html) — MEDIUM confidence
|
||
|
||
---
|
||
*Architecture research for: NetSynth — network-traffic-to-audio synthesis CLI (Go)*
|
||
*Researched: 2026-03-24*
|