docs: complete project research
This commit is contained in:
@@ -0,0 +1,344 @@
|
||||
# Architecture Research
|
||||
|
||||
**Domain:** Network traffic sonification CLI (Go)
|
||||
**Researched:** 2026-03-24
|
||||
**Confidence:** MEDIUM — Go audio synthesis patterns verified via official docs and real libraries; sonification architecture inferred from academic literature (SoNSTAR) and Go concurrency canon.
|
||||
|
||||
## Standard Architecture
|
||||
|
||||
### System Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ CLI Entry Point │
|
||||
│ (flags: interface, output path, duration) │
|
||||
└───────────────────────────┬─────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Capture Layer │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ PacketSource (gopacket/pcap) │ │
|
||||
│ │ Produces: chan Packet │ │
|
||||
│ └──────────────────────┬──────────────────────────────┘ │
|
||||
└─────────────────────────┼───────────────────────────────────┘
|
||||
│ raw packet stream
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Classification Layer │
|
||||
│ ┌─────────────────────────┐ ┌─────────────────────────┐ │
|
||||
│ │ Protocol Classifier │ │ Unknown Traffic │ │
|
||||
│ │ (ICMP, DNS, HTTPS, │ │ Clusterer │ │
|
||||
│ │ SSH, TCP-other, UDP) │ │ (feature-based bucketer)│ │
|
||||
│ └───────────┬─────────────┘ └────────────┬────────────┘ │
|
||||
│ └──────────────┬──────────────┘ │
|
||||
└─────────────────────────────┼───────────────────────────────┘
|
||||
│ classified packet events
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Aggregation Layer │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ Time-Window Accumulator │ │
|
||||
│ │ - fixed window (e.g. 500ms) │ │
|
||||
│ │ - counts + byte-volume per traffic class │ │
|
||||
│ │ Produces: chan WindowSnapshot │ │
|
||||
│ └──────────────────────┬──────────────────────────────┘ │
|
||||
└─────────────────────────┼───────────────────────────────────┘
|
||||
│ window snapshots
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Synthesis Layer │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Layer 0 │ │ Layer 1 │ │ Layer N │ │ Layer X │ │
|
||||
│ │ (ICMP) │ │ (DNS) │ │ (HTTPS) │ │ (auto) │ │
|
||||
│ │ Osc+Amp │ │ Osc+Amp │ │ Osc+Amp │ │ Osc+Amp │ │
|
||||
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
|
||||
│ └──────────────┴────────────┴──────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────▼──────┐ │
|
||||
│ │ Mixer │ │
|
||||
│ │ (sum+clip) │ │
|
||||
│ └─────┬──────┘ │
|
||||
└──────────────────────────┼──────────────────────────────────┘
|
||||
│ PCM sample stream (float32[])
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Encoding Layer │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ PCM Buffer Accumulator → LAME MP3 Encoder │ │
|
||||
│ │ (go-lame / CGo libmp3lame) │ │
|
||||
│ └──────────────────────┬──────────────────────────────┘ │
|
||||
└─────────────────────────┼───────────────────────────────────┘
|
||||
│ .mp3 file
|
||||
▼
|
||||
Output File
|
||||
```
|
||||
|
||||
### Component Responsibilities
|
||||
|
||||
| Component | Responsibility | Typical Implementation |
|
||||
|-----------|----------------|------------------------|
|
||||
| CLI Entry | Parse flags, wire all components, handle Ctrl+C via `os.Signal` | `main.go`, `cobra` or `flag` package |
|
||||
| PacketSource | Open interface via pcap/AF_PACKET, emit packets into channel | `gopacket.PacketSource.Packets()` → `<-chan gopacket.Packet` |
|
||||
| Protocol Classifier | Inspect decoded layers (IP, TCP, UDP, ICMP, DNS); assign class label | Pure Go switch on `packet.Layer()` type assertions |
|
||||
| Unknown Traffic Clusterer | Hash or bucket unclassified flows by port range / packet size signature; assign stable label ID | Simple feature-hash bucketer; no heavy ML needed for v1 |
|
||||
| Time-Window Accumulator | Batch packets into N-ms windows; emit packet-count and byte-volume per class | `ticker`-driven goroutine, map accumulation |
|
||||
| Sound Layer (per class) | Maintain a sine oscillator at a fixed root frequency; update amplitude from window snapshot | Oscillator struct with phase accumulator; amplitude lerp |
|
||||
| Mixer | Sum all layer outputs sample-by-sample; clamp/normalize to [-1, 1] | Simple additive sum with soft clip |
|
||||
| MP3 Encoder | Accept PCM float32 frames; encode to MP3 on flush/stop | go-lame (CGo) or pure-Go fallback |
|
||||
| Output File | Write encoded bytes to disk path from CLI flag | `os.File` + buffered writer |
|
||||
|
||||
## Recommended Project Structure
|
||||
|
||||
```
|
||||
netsynth/
|
||||
├── main.go # CLI wiring, signal handling, top-level orchestration
|
||||
├── capture/
|
||||
│ └── capture.go # PacketSource wrapper, interface open/close, chan Packet
|
||||
├── classify/
|
||||
│ ├── classifier.go # Protocol dispatch, class label assignment
|
||||
│ └── cluster.go # Unknown traffic bucketer (feature hash)
|
||||
├── aggregate/
|
||||
│ └── window.go # Time-window accumulator, WindowSnapshot type
|
||||
├── synth/
|
||||
│ ├── oscillator.go # Phase-accumulator sine oscillator
|
||||
│ ├── layer.go # Per-traffic-class sound layer (osc + amp target)
|
||||
│ └── mixer.go # Sum layers → float32 PCM frames
|
||||
├── encode/
|
||||
│ └── mp3.go # PCM → MP3 via go-lame; file flush on close
|
||||
└── config/
|
||||
└── mapping.go # Protocol → frequency/harmonic assignment table
|
||||
```
|
||||
|
||||
### Structure Rationale
|
||||
|
||||
- **capture/:** Isolates pcap/root-privilege boundary. Everything above it operates on typed Go channels with no pcap dependency.
|
||||
- **classify/:** Cleanly separates rule-based (known protocol) from heuristic (unknown cluster) logic. Each can be tested with synthetic packet fixtures independently.
|
||||
- **aggregate/:** The only stateful time-domain component. Isolating it makes window size configurable without touching synthesis.
|
||||
- **synth/:** Pure PCM math — no I/O, no pcap. Fully unit-testable with deterministic inputs. The mixer owns the sample rate constant.
|
||||
- **encode/:** CGo boundary lives here and nowhere else. If LAME is replaced (e.g., pure Go encoder), only this package changes.
|
||||
- **config/:** Static frequency-to-protocol table. Separating it avoids magic numbers scattered across synth/.
|
||||
|
||||
## Architectural Patterns
|
||||
|
||||
### Pattern 1: Channel-Connected Pipeline Stages
|
||||
|
||||
**What:** Each component is a goroutine that reads from an inbound channel and writes to an outbound channel. The `done` channel (closed on Ctrl+C) signals all stages to drain and exit cleanly.
|
||||
|
||||
**When to use:** Always — this is the idiomatic Go pipeline pattern described in the Go Blog.
|
||||
|
||||
**Trade-offs:** Slightly more setup than direct function calls; pays off immediately with clean shutdown and testability of individual stages.
|
||||
|
||||
**Example:**
|
||||
```go
|
||||
// Each stage signature follows this pattern
|
||||
func Classify(done <-chan struct{}, packets <-chan gopacket.Packet) <-chan ClassifiedPacket {
|
||||
out := make(chan ClassifiedPacket, 256)
|
||||
go func() {
|
||||
defer close(out)
|
||||
for {
|
||||
select {
|
||||
case <-done:
|
||||
return
|
||||
case pkt, ok := <-packets:
|
||||
if !ok { return }
|
||||
out <- classify(pkt)
|
||||
}
|
||||
}
|
||||
}()
|
||||
return out
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 2: Ticker-Driven Window Flush
|
||||
|
||||
**What:** The aggregation goroutine owns a `time.Ticker`. On each tick it snapshots accumulated counters and sends a `WindowSnapshot` downstream, then resets counters.
|
||||
|
||||
**When to use:** Anywhere time-based batching converts a high-frequency stream into low-frequency control signals.
|
||||
|
||||
**Trade-offs:** Fixed window size (e.g. 500ms) is simple but loses sub-window dynamics. Sliding windows add complexity with marginal benefit for ambient synthesis.
|
||||
|
||||
**Example:**
|
||||
```go
|
||||
func Aggregate(done <-chan struct{}, events <-chan ClassifiedPacket, windowMs int) <-chan WindowSnapshot {
|
||||
out := make(chan WindowSnapshot, 8)
|
||||
ticker := time.NewTicker(time.Duration(windowMs) * time.Millisecond)
|
||||
go func() {
|
||||
defer close(out)
|
||||
counts := map[TrafficClass]int{}
|
||||
for {
|
||||
select {
|
||||
case <-done:
|
||||
return
|
||||
case <-ticker.C:
|
||||
out <- snapshot(counts)
|
||||
counts = map[TrafficClass]int{}
|
||||
case ev, ok := <-events:
|
||||
if !ok { return }
|
||||
counts[ev.Class]++
|
||||
}
|
||||
}
|
||||
}()
|
||||
return out
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 3: Per-Layer Amplitude Lerp
|
||||
|
||||
**What:** Each sound layer holds a current amplitude and a target amplitude. On each audio frame the current value moves toward the target by a smoothing coefficient. The layer's oscillator always runs; silence is achieved by targeting amplitude = 0.
|
||||
|
||||
**When to use:** Whenever window snapshots drive synthesis — avoids clicks/pops from abrupt amplitude changes.
|
||||
|
||||
**Trade-offs:** Adds minimal CPU overhead (one multiply per frame per layer); necessary for perceptually smooth audio.
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Primary Flow: Packets to PCM
|
||||
|
||||
```
|
||||
Network Interface
|
||||
│
|
||||
▼ (gopacket pcap handle)
|
||||
PacketSource.Packets() chan
|
||||
│
|
||||
▼ (classify goroutine)
|
||||
ClassifiedPacket chan
|
||||
│
|
||||
▼ (aggregate goroutine, ticker)
|
||||
WindowSnapshot chan ─────────────────────────────────┐
|
||||
│
|
||||
(synth goroutine,
|
||||
per window snap:
|
||||
update amplitude targets)
|
||||
│
|
||||
PCM frame generator loop
|
||||
(renders N frames per window,
|
||||
one frame = sum of all layers)
|
||||
│
|
||||
▼
|
||||
PCM []float32 blocks
|
||||
│
|
||||
▼
|
||||
LAME encoder (streaming)
|
||||
│
|
||||
▼
|
||||
MP3 bytes → output file
|
||||
```
|
||||
|
||||
### Shutdown Flow
|
||||
|
||||
```
|
||||
Ctrl+C → os.Signal → close(done) channel
|
||||
│
|
||||
├── capture goroutine: drain + close packet chan
|
||||
├── classify goroutine: drain + close event chan
|
||||
├── aggregate goroutine: drain + close snapshot chan
|
||||
└── synth goroutine: flush remaining PCM → encoder.Flush() → file.Close()
|
||||
```
|
||||
|
||||
### Key Data Types
|
||||
|
||||
1. **`gopacket.Packet`** → raw decoded packet from pcap; carries layer stack.
|
||||
2. **`ClassifiedPacket{Packet, Class TrafficClass, Bytes int}`** → labeled event.
|
||||
3. **`WindowSnapshot{ClassCounts map[TrafficClass]int, ClassBytes map[TrafficClass]int}`** → per-window aggregate; drives amplitude targets.
|
||||
4. **`[]float32` PCM block** → mixer output at 44100 Hz, mono; flows into LAME.
|
||||
|
||||
## Build Order (Phase Implications)
|
||||
|
||||
Build in dependency order — each layer is independently testable before the next is added:
|
||||
|
||||
```
|
||||
1. capture/ → can test: "does it open an interface and emit packets?"
|
||||
2. classify/ → can test: "does ICMP get labeled ICMP?" (synthetic packets)
|
||||
3. aggregate/ → can test: "does a 500ms window count correctly?"
|
||||
4. synth/ → can test: "does mixer output expected amplitude?" (no pcap needed)
|
||||
5. encode/ → can test: "does PCM produce valid MP3 bytes?"
|
||||
6. main.go wiring → integration: full end-to-end pipeline
|
||||
```
|
||||
|
||||
This ordering means:
|
||||
- **Phase 1** can deliver a working capture + classify pipeline writing JSON/text summaries — validating the hardest privilege/pcap risk early.
|
||||
- **Phase 2** delivers the synthesis engine in isolation — testable with synthetic `WindowSnapshot` inputs before any real traffic.
|
||||
- **Phase 3** wires them together with the MP3 encoder.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
### Anti-Pattern 1: Synchronous Per-Packet Audio Rendering
|
||||
|
||||
**What people do:** Generate one audio sample or tone event per packet — a 10 Gbps link produces 14M packets/sec, making synchronous render impossible.
|
||||
|
||||
**Why it's wrong:** Breaks at any real traffic volume; produces click-heavy output, not smooth drone.
|
||||
|
||||
**Do this instead:** Batch packets into time windows (500ms–1s) and drive amplitude targets from the batch, not individual packets.
|
||||
|
||||
### Anti-Pattern 2: Blocking Channel Sends in the Capture Path
|
||||
|
||||
**What people do:** Use unbuffered channels between PacketSource and classifier; slow classifier stalls the pcap ring buffer and causes kernel drops.
|
||||
|
||||
**Why it's wrong:** libpcap's kernel buffer is fixed-size; if userspace can't drain it fast enough, packets are silently dropped. For audio purposes this introduces silent gaps.
|
||||
|
||||
**Do this instead:** Use buffered channels (capacity 256–1024) between capture and classify. Drop packets on full buffer with a counter — acceptable for sonification, fatal to log completeness tools.
|
||||
|
||||
### Anti-Pattern 3: CGo MP3 Encoding in the Hot Audio Loop
|
||||
|
||||
**What people do:** Call `lame.Encode()` synchronously inside the frame-render loop, stalling synthesis.
|
||||
|
||||
**Why it's wrong:** CGo calls carry overhead; libmp3lame may block on I/O; this disrupts the synthesis clock.
|
||||
|
||||
**Do this instead:** The synth goroutine pushes PCM blocks onto a buffered channel; a separate encoder goroutine drains and encodes. On shutdown, close the PCM channel and drain completely before `lame.Close()`.
|
||||
|
||||
### Anti-Pattern 4: Global Mutable State for Class Frequency Mapping
|
||||
|
||||
**What people do:** Use a global `map[TrafficClass]float64` for frequency assignments modified at runtime.
|
||||
|
||||
**Why it's wrong:** Race conditions; hard to test; makes the mapping invisible to callers.
|
||||
|
||||
**Do this instead:** Pass the mapping table as an immutable struct at construction time. Auto-clustered classes append to a local slice protected by a mutex inside the clusterer — not a global.
|
||||
|
||||
## Integration Points
|
||||
|
||||
### External Services
|
||||
|
||||
| Dependency | Integration Pattern | Notes |
|
||||
|------------|---------------------|-------|
|
||||
| libpcap / pcap.h | CGo via gopacket/pcap — requires libpcap-dev at build time | Can substitute AF_PACKET (linux only) to avoid CGo in capture; still needs root |
|
||||
| libmp3lame | CGo via go-lame — requires libmp3lame-dev at build time | Binary distribution requires static linking or Docker; pure-Go MP3 (e.g. oto + gmp3) is an option but quality/speed tradeoff |
|
||||
|
||||
### Internal Boundaries
|
||||
|
||||
| Boundary | Communication | Notes |
|
||||
|----------|---------------|-------|
|
||||
| capture ↔ classify | `chan gopacket.Packet` (buffered 512) | classify must never block capture |
|
||||
| classify ↔ aggregate | `chan ClassifiedPacket` (buffered 1024) | aggregate is slower (ticker-driven); buffer absorbs bursts |
|
||||
| aggregate ↔ synth | `chan WindowSnapshot` (buffered 4) | synth consumes synchronously per window; small buffer is fine |
|
||||
| synth ↔ encode | `chan []float32` (buffered 8 blocks) | encoder runs in separate goroutine to decouple CGo latency |
|
||||
| all stages ↔ main | `chan struct{}` done channel | closed on Ctrl+C; all stages select on it |
|
||||
|
||||
## Scaling Considerations
|
||||
|
||||
This is a single-binary CLI tool, not a distributed service. Scaling concerns are throughput-based:
|
||||
|
||||
| Traffic Rate | Architecture Adjustments |
|
||||
|--------------|--------------------------|
|
||||
| Home/office (< 10K pps) | Default design handles easily with no tuning |
|
||||
| Datacenter (100K–1M pps) | Increase capture buffer size; consider AF_PACKET with TPACKET_V3 ring buffer instead of pcap; classify goroutine may need fan-out to 2–4 workers |
|
||||
| Line-rate 10G (> 5M pps) | Out of scope for v1 ambient audio tool — synthesis granularity at 500ms windows means exact packet-level accuracy is not required |
|
||||
|
||||
### Scaling Priorities
|
||||
|
||||
1. **First bottleneck:** Kernel pcap buffer drops — mitigated by buffered channels and accepting lossy capture (fine for sonification).
|
||||
2. **Second bottleneck:** CGo encoding latency coupling synthesis clock — mitigated by decoupled encoder goroutine.
|
||||
|
||||
## Sources
|
||||
|
||||
- Go Pipeline patterns: [Go Concurrency Patterns: Pipelines and cancellation](https://go.dev/blog/pipelines) — HIGH confidence, official Go blog
|
||||
- SoNSTAR network sonification architecture: [Sonification of Network Traffic Flow for Monitoring and Situational Awareness, arXiv 1712.07029](https://arxiv.org/abs/1712.07029) — MEDIUM confidence (abstract only accessed)
|
||||
- gopacket channel API: [gopacket pkg.go.dev](https://pkg.go.dev/github.com/google/gopacket) — HIGH confidence, official package docs
|
||||
- bleep synthesizer architecture (Go): [GitHub bspaans/bleep](https://github.com/bspaans/bleep) — MEDIUM confidence (README inspection)
|
||||
- Waveform synthesis PCM patterns in Go: [Audio From Scratch With Go — Dylan Meeus](https://dylanmeeus.github.io/posts/audio-from-scratch-pt8/) — MEDIUM confidence
|
||||
- go-lame MP3 encoding: [go-lame pkg.go.dev](https://pkg.go.dev/github.com/sunicy/go-lame) — MEDIUM confidence
|
||||
- Drone amplitude/frequency modulation patterns: [Drone auralization model, Acta Acustica 2024](https://acta-acustica.edpsciences.org/articles/aacus/full_html/2024/01/aacus240076/aacus240076.html) — MEDIUM confidence
|
||||
|
||||
---
|
||||
*Architecture research for: NetSynth — network-traffic-to-audio synthesis CLI (Go)*
|
||||
*Researched: 2026-03-24*
|
||||
@@ -0,0 +1,184 @@
|
||||
# Feature Research
|
||||
|
||||
**Domain:** Network traffic sonification CLI tool (packet capture → ambient MP3)
|
||||
**Researched:** 2026-03-24
|
||||
**Confidence:** MEDIUM — this is a niche domain; most comparable tools are research prototypes or GUI applications, not CLI tools. Table stakes are derived from tcpdump/packet-capture CLI conventions and sonification research literature.
|
||||
|
||||
---
|
||||
|
||||
## Feature Landscape
|
||||
|
||||
### Table Stakes (Users Expect These)
|
||||
|
||||
Features users assume exist. Missing these = product feels incomplete.
|
||||
|
||||
| Feature | Why Expected | Complexity | Notes |
|
||||
|---------|--------------|------------|-------|
|
||||
| Network interface selection (`-i eth0`) | tcpdump/tshark convention; every capture tool has this | LOW | `gopacket` exposes interface list; needs `--list-interfaces` companion flag |
|
||||
| Output file path flag (`-o output.mp3`) | Any file-producing CLI must let you name the output | LOW | Sensible default (e.g. `netsynth-<timestamp>.mp3`) reduces friction |
|
||||
| Graceful Ctrl+C capture stop with file save | Users expect the tool to cleanly finalize the MP3 on interrupt | MEDIUM | Need signal handler; partial synthesis must be flushed to encoder before exit |
|
||||
| Per-protocol sound distinction | Core value prop: ping sounds different from HTTPS noise | MEDIUM | Minimum recognizable set: ICMP, DNS, TCP (port 443), TCP (other), UDP |
|
||||
| Packet count / traffic summary on exit | Every capture tool prints capture statistics; users want to know what was heard | LOW | Print to stderr so it doesn't interfere with stdout pipeline use |
|
||||
| Privilege error message | `pcap` silently fails or panics without root/CAP_NET_RAW; users need a clear message | LOW | Detect EACCES / EPERM on open; print actionable message (`sudo` or capability hint) |
|
||||
| List available interfaces (`--list-interfaces`) | Users don't know interface names on unfamiliar machines | LOW | Wrap `pcap.FindAllDevs()`; print name + description |
|
||||
| Minimum viable duration guard | Zero-packet capture should not produce a corrupt/empty MP3 | LOW | Check sample count before encoding; exit with clear error if nothing was captured |
|
||||
|
||||
### Differentiators (Competitive Advantage)
|
||||
|
||||
Features that set the product apart. Not required, but valuable.
|
||||
|
||||
| Feature | Value Proposition | Complexity | Notes |
|
||||
|---------|-------------------|------------|-------|
|
||||
| Auto-clustering of unrecognized traffic | Unknown traffic still gets a unique voice instead of being silently dropped — honest audio fingerprint | HIGH | Requires unsupervised clustering (e.g. flow-feature vector → k-means or simple hash bucketing); each cluster gets a deterministic frequency mapping |
|
||||
| Ambient/drone style output (layered sine harmonics) | Distinct from event-ping tools (Peep, SoNSTAR); slow tonal evolution makes long captures listenable | HIGH | Synthesize per-protocol drone layers; amplitude driven by time-windowed packet rate; mix layers before encoding |
|
||||
| Time-windowed amplitude evolution | Traffic volume changes over time are reflected in the audio; the mix evolves rather than being static | MEDIUM | Segment capture into N-second windows; compute per-layer gain per window; apply smooth gain ramps between windows |
|
||||
| Configurable time window duration (`--window 10`) | Lets users tune responsiveness vs. smoothness; research tools (SoNSTAR) expose this parameter | LOW | Default 10s; range 1–60s is sensible |
|
||||
| BPF capture filter support (`--filter "tcp port 443"`) | Power users want to scope what gets sonified; tcpdump BPF syntax is universally known | MEDIUM | Pass expression directly to `gopacket`/`pcap`; validate at startup before capture begins |
|
||||
| Offline pcap file input (`--read capture.pcap`) | Lets users sonify historical captures, not just live traffic; useful for analysis and demos | MEDIUM | Replace live capture source with `pcap.OpenOffline()`; time-compress or time-expand to fixed output duration |
|
||||
| Verbose protocol activity log to stderr (`--verbose`) | Developers and curious users want to see what was classified | LOW | Print per-window protocol breakdown table to stderr during capture |
|
||||
| Configurable output duration when reading pcap file (`--duration 30`) | Offline pcap may span hours; need ability to compress to a target audio length | LOW | Only meaningful with `--read`; scale time windows proportionally |
|
||||
| Single static binary (`go build`) | Eliminates dependency hell on target machines | LOW (build-time) | Go native; no CGo for MP3 encoding avoids runtime `.so` requirements — choose a pure-Go MP3 encoder |
|
||||
|
||||
### Anti-Features (Commonly Requested, Often Problematic)
|
||||
|
||||
Features that seem good but create problems.
|
||||
|
||||
| Feature | Why Requested | Why Problematic | Alternative |
|
||||
|---------|---------------|-----------------|-------------|
|
||||
| Real-time audio playback (speakers while capturing) | Feels more immediate; Peep and dmeldrum6/Network-Sonification do this | Requires platform audio APIs (ALSA/CoreAudio/WASAPI), cross-platform complexity triples; conflicts with file-output simplicity; latency/buffering bugs; CGo or external library dependency | File output only; user can pipe MP3 to `mpv`/`afplay` themselves after capture |
|
||||
| GUI or web dashboard | Visually richer; existing tools like Network-Sonification are GUI-first | Negates single-binary CLI value; doubles scope; Go GUI toolkits are immature or require CGo | Emit stderr text summary; let external tools consume the MP3 |
|
||||
| Custom sound mapping configuration file | Power-user request; SoNSTAR supports per-user sound uploads | Configuration surface area is large; predefined + auto-cluster covers the use case adequately for v1; config files introduce parsing/validation work | Well-chosen defaults + auto-cluster for unknowns; defer custom mapping to v2 |
|
||||
| Rhythmic/percussive output mode | Some sonification tools use discrete note triggers per packet | Ambient/drone style is the deliberate differentiator; per-packet triggers at high traffic volumes produce noise, not information | Stick to amplitude-modulated harmonic drones; volume changes carry the rhythm implicitly |
|
||||
| Deep-packet inspection / payload parsing | Users might want to hear HTTP body content, TLS handshake details | Requires reassembly, encryption handling, legal concerns about payload interception; massive complexity | Classify by header fields only (port, protocol, flags, packet size); that is sufficient for the audio fingerprint goal |
|
||||
| Streaming MP3 output (write while capturing) | Real-time preview of what's being synthesized | MP3 frame boundaries and VBR headers require the full file to be finalized; streaming output would produce a non-standard file | Write to temp buffer during capture, finalize and flush on Ctrl+C |
|
||||
| Anomaly detection / alerting | Natural extension once you have classified traffic | Adds a monitoring-tool responsibility on top of the audio-fingerprint responsibility; these are different user jobs | Stick to "produce an audio fingerprint"; anomaly detection is a separate tool |
|
||||
|
||||
---
|
||||
|
||||
## Feature Dependencies
|
||||
|
||||
```
|
||||
[Interface selection / list-interfaces]
|
||||
└──required-by──> [Live capture]
|
||||
|
||||
[Live capture] ──OR── [Offline pcap input]
|
||||
└──required-by──> [Protocol classification]
|
||||
└──required-by──> [Auto-clustering of unknowns]
|
||||
└──required-by──> [Per-protocol drone layer synthesis]
|
||||
└──required-by──> [Time-windowed amplitude evolution]
|
||||
└──required-by──> [Layer mixing]
|
||||
└──required-by──> [MP3 encoding & file output]
|
||||
|
||||
[BPF capture filter] ──enhances──> [Live capture]
|
||||
[Configurable time window] ──tunes──> [Time-windowed amplitude evolution]
|
||||
[Offline pcap + --duration] ──requires──> [Offline pcap input]
|
||||
[Verbose flag] ──enhances──> [Protocol classification] (reporting only, no data dependency)
|
||||
[Graceful Ctrl+C] ──requires──> [MP3 encoding & file output] (must flush before exit)
|
||||
```
|
||||
|
||||
### Dependency Notes
|
||||
|
||||
- **Protocol classification requires Live capture OR Offline pcap:** These are the two data sources; everything downstream is source-agnostic.
|
||||
- **Time-windowed amplitude evolution requires Protocol classification:** You need classified packet counts per window before you can derive per-layer gain values.
|
||||
- **MP3 encoding requires Layer mixing:** You cannot encode until all layers for a time window are mixed to a PCM buffer.
|
||||
- **Graceful Ctrl+C requires MP3 encoding:** The signal handler must trigger the encode-and-flush path, not just `os.Exit`.
|
||||
- **Auto-clustering enhances Protocol classification:** It extends classification to traffic that doesn't match predefined rules; the audio pipeline treats cluster-assigned tones identically to predefined protocol tones.
|
||||
- **BPF filter conflicts with Offline pcap input (partial):** `pcap` supports BPF on offline files, so this works technically, but user expectation for offline mode is usually "sonify all traffic in the file" — document the interaction clearly.
|
||||
|
||||
---
|
||||
|
||||
## MVP Definition
|
||||
|
||||
### Launch With (v1)
|
||||
|
||||
Minimum viable product — what's needed to validate the concept.
|
||||
|
||||
- [ ] Network interface selection (`-i`) and `--list-interfaces` — required for capture
|
||||
- [ ] Live packet capture with Ctrl+C stop — core interaction model
|
||||
- [ ] Protocol classification: ICMP, DNS, TCP/443, TCP/other, UDP — minimum set for a recognizable fingerprint
|
||||
- [ ] Auto-clustering of unrecognized traffic (simple hash-bucketing by port/proto is acceptable for v1) — honest representation of full traffic
|
||||
- [ ] Per-protocol ambient drone layer synthesis (sine harmonics, amplitude modulated by packet rate) — core differentiator
|
||||
- [ ] Time-windowed amplitude evolution (10s default) — makes the output dynamic
|
||||
- [ ] MP3 encoding and file output with sensible default filename — deliverable artifact
|
||||
- [ ] Capture statistics summary on exit (stderr) — basic UX courtesy
|
||||
- [ ] Privilege error detection and clear message — prevents silent failure
|
||||
|
||||
### Add After Validation (v1.x)
|
||||
|
||||
Features to add once core is working.
|
||||
|
||||
- [ ] BPF capture filter (`--filter`) — add when users report wanting to scope captures
|
||||
- [ ] Offline pcap file input (`--read`) — add when users want to sonify historical captures
|
||||
- [ ] Verbose protocol activity log (`--verbose`) — add when debugging/demo use cases emerge
|
||||
- [ ] Configurable time window duration (`--window`) — add if users report default feels too slow or too fast
|
||||
- [ ] Configurable output duration for pcap input (`--duration`) — depends on offline input being implemented
|
||||
|
||||
### Future Consideration (v2+)
|
||||
|
||||
Features to defer until product-market fit is established.
|
||||
|
||||
- [ ] Custom sound mapping configuration — defer; needs user research on what customization actually matters
|
||||
- [ ] Improved clustering algorithm (k-means on flow features vs. simple hash) — defer until users report clusters feel meaningless
|
||||
- [ ] Multi-interface capture — defer; adds complexity to packet deduplication
|
||||
|
||||
---
|
||||
|
||||
## Feature Prioritization Matrix
|
||||
|
||||
| Feature | User Value | Implementation Cost | Priority |
|
||||
|---------|------------|---------------------|----------|
|
||||
| Interface selection + list-interfaces | HIGH | LOW | P1 |
|
||||
| Live capture with Ctrl+C stop | HIGH | LOW | P1 |
|
||||
| Protocol classification (ICMP, DNS, TCP, UDP) | HIGH | MEDIUM | P1 |
|
||||
| Ambient drone synthesis (per-protocol layers) | HIGH | HIGH | P1 |
|
||||
| Time-windowed amplitude evolution | HIGH | MEDIUM | P1 |
|
||||
| MP3 encoding + file output | HIGH | MEDIUM | P1 |
|
||||
| Capture stats on exit | MEDIUM | LOW | P1 |
|
||||
| Privilege error message | MEDIUM | LOW | P1 |
|
||||
| Auto-clustering of unknown traffic | MEDIUM | MEDIUM | P1 |
|
||||
| BPF capture filter | MEDIUM | MEDIUM | P2 |
|
||||
| Offline pcap file input | MEDIUM | MEDIUM | P2 |
|
||||
| Verbose flag | LOW | LOW | P2 |
|
||||
| Configurable time window | LOW | LOW | P2 |
|
||||
| Custom sound mapping | LOW | HIGH | P3 |
|
||||
| Real-time playback | LOW | HIGH | P3 (anti-feature, avoid) |
|
||||
|
||||
**Priority key:**
|
||||
- P1: Must have for launch
|
||||
- P2: Should have, add when possible
|
||||
- P3: Nice to have, future consideration
|
||||
|
||||
---
|
||||
|
||||
## Competitor Feature Analysis
|
||||
|
||||
| Feature | SoNSTAR (Python, research) | Network-Sonification (C#, Windows GUI) | Peep (C, Unix, 2000) | NetSynth (our approach) |
|
||||
|---------|----------------------------|-----------------------------------------|----------------------|------------------------|
|
||||
| Interface selection | Interactive prompt | GUI dropdown | Config file | CLI flag `-i` |
|
||||
| Protocol coverage | TCP flag states | TCP, UDP, HTTP, HTTPS, DNS, ICMP | Any syslog-able event | ICMP, DNS, TCP, UDP + auto-cluster |
|
||||
| Sound style | Recorded natural sounds (forest ambience) | Waveform shapes per protocol (sine/square/triangle) | Discrete event sounds | Synthesized harmonic drones |
|
||||
| Output | Real-time audio (Max/MSP) | Real-time audio (WPF) | Real-time audio (Unix audio) | MP3 file |
|
||||
| Time aggregation | Configurable window (default 20s) | Per-packet event | Per-event | Configurable window (default 10s) |
|
||||
| CLI/scriptable | Partial (Python prompts) | No (GUI only) | Yes (daemon) | Yes (single binary, flags) |
|
||||
| Offline pcap input | No | No | No | v1.x |
|
||||
| Auto-clustering | No | No | No | Yes (v1 hash-bucket) |
|
||||
| Single binary | No | No | No | Yes (Go) |
|
||||
| Open source | Yes | Yes | Yes | Intended |
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [SoNSTAR: Sonification of Networks for SiTuational AwaReness — Paul Vickers](https://paulvickers.github.io/SoNSTAR/)
|
||||
- [Sonification of network traffic flow for monitoring and situational awareness — PLOS One](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0195948)
|
||||
- [SoNSTAR GitHub repository — nuson/SoNSTAR](https://github.com/nuson/SoNSTAR)
|
||||
- [Network-Sonification GitHub — dmeldrum6/Network-Sonification](https://github.com/dmeldrum6/Network-Sonification)
|
||||
- [Peep (The Network Auralizer): Monitoring Your Network With Sound — USENIX 2000](https://www.usenix.org/legacyurl/peep-network-auralizer-monitoring-your-network-sound)
|
||||
- [Sonification of DDoS Attacks — Imperva](https://www.imperva.com/blog/archive/sonification-of-ddos-attacks/)
|
||||
- [Data Sonification Toolkit — Sound and data parameters](https://www.sonificationkit.com/data-sonification/concepts/sound-and-data-parameters)
|
||||
- [tcpdump man page — tcpdump.org](https://www.tcpdump.org/manpages/tcpdump.1.html)
|
||||
- [The Sound of Data: A gentle introduction to sonification — Programming Historian](https://programminghistorian.org/en/lessons/sonification)
|
||||
|
||||
---
|
||||
*Feature research for: network traffic sonification CLI (NetSynth)*
|
||||
*Researched: 2026-03-24*
|
||||
@@ -0,0 +1,333 @@
|
||||
# Pitfalls Research
|
||||
|
||||
**Domain:** Network-traffic-to-audio synthesis CLI tool (Go)
|
||||
**Researched:** 2026-03-24
|
||||
**Confidence:** HIGH (packet capture / CGo pitfalls verified against official issues and docs; audio synthesis pitfalls cross-referenced against encoder project post-mortems and DSP literature)
|
||||
|
||||
---
|
||||
|
||||
## Critical Pitfalls
|
||||
|
||||
### Pitfall 1: Using `google/gopacket` Instead of the Active Community Fork
|
||||
|
||||
**What goes wrong:**
|
||||
The original `github.com/google/gopacket` repository is unmaintained. Bugs go unpatched, open PRs accumulate, and compatibility with newer Go versions degrades. Projects that import it are pinned to a stale library.
|
||||
|
||||
**Why it happens:**
|
||||
`google/gopacket` has enormous search mindshare and most tutorials still reference it. Developers reach for the first result without checking maintenance status.
|
||||
|
||||
**How to avoid:**
|
||||
Import `github.com/gopacket/gopacket` (the community fork, v1.5.0 released November 2025, requires Go 1.24+). Major projects including Cilium have already migrated. Treat `google/gopacket` as deprecated.
|
||||
|
||||
**Warning signs:**
|
||||
- `go.mod` referencing `github.com/google/gopacket`
|
||||
- Build errors on Go 1.21+ not fixed upstream
|
||||
|
||||
**Phase to address:**
|
||||
Phase 1 (packet capture scaffolding) — set the correct import path from day one; migrating later is a find-and-replace across the whole codebase.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 2: CGo Destroys the "Single Binary" Promise
|
||||
|
||||
**What goes wrong:**
|
||||
`gopacket/pcap` requires `libpcap` via CGo. By default Go produces a dynamically linked binary. On a target machine without `libpcap.so` installed, the binary silently or loudly fails with `error while loading shared libraries: libpcap.so.0.8`. The "just copy the binary" distribution story breaks completely.
|
||||
|
||||
**Why it happens:**
|
||||
CGo is enabled by default and Go gives no compile-time warning that the resulting binary has a runtime C dependency. The binary runs perfectly on the build machine (which has libpcap-dev installed) and fails on clean machines.
|
||||
|
||||
**How to avoid:**
|
||||
Choose one of these strategies before writing a line of capture code:
|
||||
1. **Fully static build**: `CGO_ENABLED=1 go build -ldflags "-linkmode 'external' -extldflags '-static'"` with `libpcap.a` present. Requires `musl-gcc` or equivalent on Alpine/musl.
|
||||
2. **pcapgo (pure Go)**: `gopacket/pcapgo` provides an `EthernetHandle` that avoids CGo entirely — lower performance but zero C dependency. Sufficient for ambient audio capture at non-Gbps rates.
|
||||
3. **Document the dependency explicitly**: If CGo/dynamic linking is accepted, `README` must state "requires `libpcap` (`apt install libpcap-dev` / `brew install libpcap`)".
|
||||
|
||||
**Warning signs:**
|
||||
- `CGO_ENABLED` not explicitly set in your build script
|
||||
- `ldd ./netsynth` shows `libpcap.so` as a dependency
|
||||
- No CI test on a minimal (Alpine, scratch Docker) container
|
||||
|
||||
**Phase to address:**
|
||||
Phase 1 — this is a foundational architecture decision. Changing from dynamic to static after the fact is painful and causes build pipeline rewrites.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 3: `CAP_NET_RAW` + Binary Location = Silent Failure on Linux
|
||||
|
||||
**What goes wrong:**
|
||||
On Ubuntu and many Linux distributions, `setcap cap_net_raw+eip ./netsynth` appears to succeed but the binary fails at runtime if it lives in `/home/user/bin`, `/tmp`, or any filesystem mounted `nosuid`. The kernel silently ignores the capability. AppArmor compounds this by enforcing path-based restrictions.
|
||||
|
||||
**Why it happens:**
|
||||
Developers test from their build directory (`~/projects/netsynth/`) — a path frequently on a `nosuid` filesystem. The tool appears broken with no clear error message beyond "permission denied" or "you must be root."
|
||||
|
||||
**How to avoid:**
|
||||
- Install to `/usr/local/bin` or `/usr/bin` for capability-based operation
|
||||
- Document two run modes: `sudo ./netsynth` (always works) vs. `setcap` (requires standard path)
|
||||
- In the CLI, detect permission failure and emit a clear message: "Packet capture requires root or CAP_NET_RAW. Run as root or: sudo setcap cap_net_raw+eip $(which netsynth)"
|
||||
- Test capability-mode explicitly from a non-home path in CI
|
||||
|
||||
**Warning signs:**
|
||||
- Testing only via `sudo go run .`
|
||||
- No test of the installed-binary path in README instructions
|
||||
- macOS-only development (macOS uses a different privilege model; Linux `nosuid` behavior won't surface)
|
||||
|
||||
**Phase to address:**
|
||||
Phase 1 (capture scaffolding) and the CLI UX phase — the error message is user-facing and needs to be explicit.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 4: Packet Buffer Overflow Under Moderate Traffic Load
|
||||
|
||||
**What goes wrong:**
|
||||
At high packet rates (busy LAN, server NIC), gopacket's kernel ring buffer fills faster than the processing goroutine consumes it. The OS drops packets silently. The tool appears to work, but 50-98% of packets never reach the classifier. The audio output misrepresents actual traffic.
|
||||
|
||||
**Why it happens:**
|
||||
The default pcap buffer is 1-2 MB. Each packet triggers a cgo call (with `pcap` backend), creating per-packet overhead that compounds at speed. Developers test on quiet home networks and never observe drops.
|
||||
|
||||
**How to avoid:**
|
||||
- Set a large capture buffer explicitly: `handle.SetBufferSize(32 * 1024 * 1024)` (32 MB)
|
||||
- Use a non-blocking channel between capture and classification goroutines with a buffer of at least 1000 packets; drop metrics count drops so they are visible
|
||||
- For high-throughput scenarios, prefer `afpacket` backend over `pcap` — `afpacket` eliminates per-packet CGo calls and dramatically improves throughput (benchmark: 1.27 MB/s → 21.17 MB/s)
|
||||
- NetSynth's ambient audio goal tolerates lossy capture — document this explicitly so users understand the tool provides a statistical fingerprint, not a perfect census
|
||||
|
||||
**Warning signs:**
|
||||
- Capture and classification in a single goroutine
|
||||
- No `SetBufferSize` call
|
||||
- Testing only on loopback (`lo`) which has near-zero real packet rates
|
||||
|
||||
**Phase to address:**
|
||||
Phase 1/2 (capture pipeline) — the goroutine architecture must be designed for async processing from the start. Retrofitting is a significant rewrite.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 5: ZeroCopy Packet Data Use-After-Free
|
||||
|
||||
**What goes wrong:**
|
||||
`ZeroCopyReadPacketData()` returns a slice pointing into a buffer owned by the pcap handle. The next call to `ZeroCopyReadPacketData()` invalidates the previous slice's backing memory. If any goroutine holds a reference to old packet bytes and reads them after the next call, it reads corrupted or incorrect data. This produces silent data corruption — wrong protocol classifications, no crash.
|
||||
|
||||
**Why it happens:**
|
||||
The zero-copy API looks identical to the copying API. Developers reach for it for performance without reading the "each call invalidates previous data" contract.
|
||||
|
||||
**How to avoid:**
|
||||
Use `ReadPacketData()` (copies data) unless you have profiling evidence that allocation is a bottleneck. If `ZeroCopyReadPacketData()` is used, never pass the slice to another goroutine without first copying it: `data := append([]byte(nil), raw...)`.
|
||||
|
||||
**Warning signs:**
|
||||
- `ZeroCopyReadPacketData` in a goroutine-per-packet pattern
|
||||
- Intermittent wrong protocol classifications that are not reproducible
|
||||
- Using `gopacket.Lazy` decode with concurrent goroutines (the gopacket docs explicitly warn against this combination)
|
||||
|
||||
**Phase to address:**
|
||||
Phase 1 (capture/decode) — establish the correct API choice at the read loop level.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 6: MP3 Output Is Corrupt or Unplayable Due to LAME Initialization Errors
|
||||
|
||||
**What goes wrong:**
|
||||
MP3 encoding via CGo LAME bindings requires calling `InitParams()` after setting all encoder parameters. Skipping or reordering this call produces a file with a valid `.mp3` extension that most players refuse to open or that plays as noise. The encoder returns no error from the encode calls themselves.
|
||||
|
||||
**Why it happens:**
|
||||
The LAME C API is stateful and order-dependent. Go wrappers vary in how much they enforce initialization order. Many tutorial examples show minimal code that happens to work for 44100 Hz stereo but silently breaks for other configurations.
|
||||
|
||||
**How to avoid:**
|
||||
- Always call `InitParams()` before writing any frames
|
||||
- Restrict to known-safe parameters: sample rate 44100 or 48000, stereo or mono (LAME does not support dual-channel mode)
|
||||
- Write a single integration test that encodes 1 second of silence and confirms the output file is valid (use `mp3val` or `ffprobe` in CI)
|
||||
- Consider `shine-mp3` (pure Go port) as an alternative that eliminates CGo entirely; output files are larger but the library has no C initialization state
|
||||
|
||||
**Warning signs:**
|
||||
- No test that validates the output MP3 with an external tool
|
||||
- Sample rate set to anything other than 44100 or 48000
|
||||
- Encoder parameters set after `InitParams()` has been called
|
||||
|
||||
**Phase to address:**
|
||||
Audio synthesis / encoding phase — establish the encode pipeline with an end-to-end smoke test (silence → valid MP3) before wiring up synthesis.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 7: PCM Sample Overflow Produces Wrap-Around Distortion
|
||||
|
||||
**What goes wrong:**
|
||||
Synthesizing audio as `int16` samples and summing multiple sine layers without clamping causes integer overflow. The value wraps around (e.g., 32767 + 100 = -32667 in int16), producing a sharp click or a buzzing distortion that corrupts the ambient soundscape. This is not clipping — it is a distinctly worse artifact.
|
||||
|
||||
**Why it happens:**
|
||||
Developers model audio math in their head as real-valued floats, implement it in int16 for "efficiency," and forget that Go integer overflow is undefined-behavior-free but still wraps. With 6-8 drone layers simultaneously active, summing them easily exceeds ±32767.
|
||||
|
||||
**How to avoid:**
|
||||
Synthesize internally in `float64` in the range `[-1.0, 1.0]`. Apply a normalisation/soft-limiter pass before converting to `int16` for encoding. Clamp before cast: `sample := int16(math.Max(-1.0, math.Min(1.0, floatSample)) * 32767)`. Never do mixed-type audio math that passes through int16 as an intermediate.
|
||||
|
||||
**Warning signs:**
|
||||
- Audio synthesis structs storing amplitude as `int16` or `int32`
|
||||
- Adding layer outputs with `+=` without a final normalisation step
|
||||
- Distorted output that correlates with traffic spikes (more active layers = more overflow)
|
||||
|
||||
**Phase to address:**
|
||||
Audio synthesis phase — establish the internal sample representation as `float64` from the start.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 8: Tone-per-Protocol Mapping Produces Perceptual Chaos
|
||||
|
||||
**What goes wrong:**
|
||||
Assigning arbitrary frequencies to protocols (e.g., DNS=440 Hz, HTTPS=880 Hz, ICMP=1320 Hz, SSH=1760 Hz, 6 auto-clusters=random) creates a soundscape where all tones are in the same frequency range, fighting each other. At moderate traffic the result is an undifferentiated buzz rather than distinct recognizable layers.
|
||||
|
||||
**Why it happens:**
|
||||
Developers choose frequencies programmatically (e.g., multiples of a base frequency) without considering auditory scene analysis — the human perceptual process by which listeners separate simultaneous sounds into distinct streams. Sounds too close in frequency mask each other.
|
||||
|
||||
**How to avoid:**
|
||||
Space protocol tones across register bands: low drones (80-200 Hz) for high-volume background traffic (HTTPS bulk), mid tones (300-600 Hz) for control traffic (DNS, NTP), high tones (800-1600 Hz) for interactive protocols (SSH, ICMP). Use harmonic or musical intervals (octaves, fifths) rather than arithmetic spacing. Keep auto-cluster frequencies in the 200-500 Hz mid-range so they don't obscure the "signature" tones. Limit simultaneous active layers to avoid masking.
|
||||
|
||||
**Warning signs:**
|
||||
- Frequency assignments as an arithmetic sequence: `baseFreq + n*200`
|
||||
- No perceptual test — only waveform-level correctness checks
|
||||
- Auto-cluster frequencies chosen randomly from the full audible range
|
||||
|
||||
**Phase to address:**
|
||||
Audio mapping / synthesis phase — the frequency mapping table should be designed up front with the perceptual goals in mind, not patched after "it sounds like noise" feedback.
|
||||
|
||||
---
|
||||
|
||||
### Pitfall 9: Time Window Too Short — Unstable, Jittery Audio
|
||||
|
||||
**What goes wrong:**
|
||||
Aggregating traffic into windows shorter than ~500ms causes rapid amplitude oscillation in the synthesized drones. A single ICMP ping becomes a brief tone burst; a DNS query causes a momentary volume spike. The output sounds jittery and event-driven rather than ambient.
|
||||
|
||||
**Why it happens:**
|
||||
Developers choose a "natural" update interval (100ms or 200ms matches CPU scheduling intuition) without considering audio envelope times. Human perception of tonal stability requires note durations of at least 200-500ms; drones need even longer.
|
||||
|
||||
**How to avoid:**
|
||||
- Use a minimum window of 500ms for amplitude updates; 1-2s for tonal shifts
|
||||
- Apply amplitude smoothing (exponential moving average with a decay of ~2-5s) so a single-packet burst doesn't cause an immediate amplitude jump
|
||||
- Separate the "data collection" window (can be shorter) from the "audio parameter update" window (should be longer)
|
||||
|
||||
**Warning signs:**
|
||||
- `time.Tick(100 * time.Millisecond)` driving audio parameter updates
|
||||
- No smoothing/interpolation between amplitude values
|
||||
- Testing with ping floods (bursty) rather than continuous traffic
|
||||
|
||||
**Phase to address:**
|
||||
Traffic aggregation / audio mapping phase — establish the window and smoothing strategy before wiring traffic data to audio parameters.
|
||||
|
||||
---
|
||||
|
||||
## Technical Debt Patterns
|
||||
|
||||
| Shortcut | Immediate Benefit | Long-term Cost | When Acceptable |
|
||||
|----------|-------------------|----------------|-----------------|
|
||||
| `google/gopacket` instead of `gopacket/gopacket` | Familiar, more tutorials | Unmaintained; Go compat breaks | Never |
|
||||
| `sudo ./netsynth` only, no `setcap` docs | Simpler setup instructions | Users won't run as root in practice; tool appears broken | MVP only — document the limitation |
|
||||
| Dynamic libpcap linking (no static build) | Faster to compile | Binary doesn't work on target machines without libpcap installed | Only acceptable if distributing via package manager that declares the dep |
|
||||
| `ReadPacketData` (copying) instead of `ZeroCopy` | Safe, simple | ~20% memory overhead at high packet rates | Always acceptable; optimize only if profiling proves allocation bottleneck |
|
||||
| Sine-wave-only synthesis (no ADSR, no envelope) | Much simpler code | Tonal changes are abrupt, not perceptually smooth | Acceptable for v1 ambient/drone if EMA smoothing is applied to amplitude |
|
||||
| Hard-coded frequency table (no config) | No CLI complexity | Can't tune without recompiling | Acceptable for v1 per PROJECT.md out-of-scope decision |
|
||||
|
||||
---
|
||||
|
||||
## Integration Gotchas
|
||||
|
||||
| Integration | Common Mistake | Correct Approach |
|
||||
|-------------|----------------|------------------|
|
||||
| `gopacket/pcap` handle | Not calling `handle.Close()` on signal — leaks capture resources | Use `defer handle.Close()` and ensure the goroutine exits before process termination |
|
||||
| LAME CGo encoder | Not flushing the encoder before closing — truncated final MP3 frame | Call `encoder.Flush()` / `lame.EncodeFlush()` after the sample loop ends |
|
||||
| OS signal handling (`SIGINT`) | Goroutine receives SIGINT but the capture loop is blocked on `ReadPacketData` | Use `handle.SetReadDeadline(time.Now())` or close the handle to unblock |
|
||||
| MP3 encoder sample format | Passing `float64` samples directly to LAME (expects `int16` or `float32` depending on binding) | Explicitly convert and clamp to the binding's expected type; check each binding's API |
|
||||
| `pcapgo.EthernetHandle` | Only captures Ethernet frames — fails on WiFi (802.11), loopback, or tunnel interfaces | For non-Ethernet interfaces, use the `pcap` backend or check link type at startup |
|
||||
|
||||
---
|
||||
|
||||
## Performance Traps
|
||||
|
||||
| Trap | Symptoms | Prevention | When It Breaks |
|
||||
|------|----------|------------|----------------|
|
||||
| Single goroutine: capture + classify + synthesize | CPU-bound synthesis blocks packet reads; drops spike under any real traffic | Three-stage pipeline: capture goroutine → classify channel → synthesis goroutine | Breaks on any network with > ~1000 pps |
|
||||
| One goroutine per packet | Goroutine creation overhead exceeds packet processing time; OOM on busy networks | Channel-based batching: one reader, N classifiers from a worker pool | Breaks above ~10k pps |
|
||||
| Recomputing sine wave sample-by-sample in inner loop using `math.Sin` | CPU pegged at 100% during synthesis; output can't keep pace | Precompute wavetable per frequency; iterate with phase accumulator | Breaks with > 4-5 simultaneous drone layers at 44100 Hz |
|
||||
| Blocking channel between capture and synthesis with no buffer | Any synthesis stall causes packet drops | Buffered channel of 1000+ packets; separate goroutines | Breaks immediately on any CPU scheduling hiccup |
|
||||
|
||||
---
|
||||
|
||||
## Security Mistakes
|
||||
|
||||
| Mistake | Risk | Prevention |
|
||||
|---------|------|------------|
|
||||
| Requesting full `root` and keeping it throughout capture | Privilege escalation if a parsing bug in gopacket can be exploited via crafted packets | Drop privileges after opening the capture handle: `syscall.Setuid(originalUID)` |
|
||||
| Promiscuous mode on by default without user opt-in | Captures all LAN traffic, not just traffic to/from the host — legal and privacy risk on shared networks | Default to non-promiscuous; add `--promiscuous` flag with a warning message |
|
||||
| No limit on capture duration or file size | Unbounded run produces an arbitrarily large MP3 or consumes all memory in the aggregator maps | Add `--max-duration` flag (default: warn at 10min, hard limit at 1hr); prune old flow state periodically |
|
||||
| Logging decoded packet payloads in debug mode | Inadvertently logs credentials or private data | Never log packet payload bytes; log only headers and metadata |
|
||||
|
||||
---
|
||||
|
||||
## UX Pitfalls
|
||||
|
||||
| Pitfall | User Impact | Better Approach |
|
||||
|---------|-------------|-----------------|
|
||||
| Silent failure when interface doesn't exist | User specifies `-i eth1` on a machine with only `ens3`; tool exits with cryptic libpcap error | List available interfaces at startup with `pcap.FindAllDevs()` and suggest correct name |
|
||||
| No progress feedback during capture | User has no idea if the tool is working; assumes it hung | Print periodic status line: "Capturing... 1,234 packets classified (HTTPS:45% DNS:30% ICMP:8% other:17%)" |
|
||||
| Output MP3 path collision without warning | Re-running overwrites previous output | Warn if output file exists; suggest timestamped default filename |
|
||||
| Ctrl+C produces empty or invalid MP3 | User interrupts too quickly before any traffic is captured | Detect zero-packet case and emit an error instead of an empty file |
|
||||
| No indication of which interface is being captured | Confusing when multiple interfaces exist | Print "Capturing on: eth0 (192.168.1.5)" at startup |
|
||||
|
||||
---
|
||||
|
||||
## "Looks Done But Isn't" Checklist
|
||||
|
||||
- [ ] **Packet capture:** Binary runs as non-root user with `setcap` — verify from a non-`/home` path, not just from the build directory
|
||||
- [ ] **MP3 output:** File validates with `ffprobe` or `mp3val` — not just "has .mp3 extension and non-zero size"
|
||||
- [ ] **Static binary:** `ldd ./netsynth` shows "not a dynamic executable" (or explicitly "requires libpcap" if dynamic is accepted)
|
||||
- [ ] **Signal handling:** Ctrl+C during capture produces a valid (playable) MP3, not a truncated file
|
||||
- [ ] **High-traffic:** Drop counter is zero (or documented/acceptable) when tested against a network with > 1000 pps
|
||||
- [ ] **Audio layers:** Output with 6+ simultaneous traffic types does not distort — no wrap-around clipping audible
|
||||
- [ ] **Empty capture:** Graceful error message when zero packets were captured, not a silent empty file
|
||||
- [ ] **Interface not found:** Helpful error with available interface list, not a libpcap raw error string
|
||||
|
||||
---
|
||||
|
||||
## Recovery Strategies
|
||||
|
||||
| Pitfall | Recovery Cost | Recovery Steps |
|
||||
|---------|---------------|----------------|
|
||||
| Wrong gopacket fork | LOW | `go mod edit -replace github.com/google/gopacket=github.com/gopacket/gopacket@v1.5.0`; update import paths |
|
||||
| Dynamic binary on clean machine | MEDIUM | Add static build Makefile target; update CI; update README |
|
||||
| PCM overflow / distortion | LOW | Refactor synthesis to float64 internal representation; add clamp before int16 cast |
|
||||
| Corrupt MP3 (missing flush) | LOW | Add `Flush()` call in the shutdown path |
|
||||
| Perceptual chaos (tone mapping) | MEDIUM | Redesign frequency table (no code change to synthesis engine); requires subjective listening tests |
|
||||
| Time window jitter | LOW | Add EMA smoothing and increase window; no architectural change needed |
|
||||
| `ZeroCopy` data corruption | MEDIUM | Replace `ZeroCopyReadPacketData` with `ReadPacketData`; audit all goroutine handoffs |
|
||||
|
||||
---
|
||||
|
||||
## Pitfall-to-Phase Mapping
|
||||
|
||||
| Pitfall | Prevention Phase | Verification |
|
||||
|---------|------------------|--------------|
|
||||
| Wrong gopacket fork | Phase 1: Packet Capture | `go.mod` references `gopacket/gopacket`; `go list -m github.com/gopacket/gopacket` |
|
||||
| CGo / single binary contract | Phase 1: Packet Capture | `ldd` output on CI; test on clean Alpine container |
|
||||
| CAP_NET_RAW binary location | Phase 1 + CLI UX phase | Test `setcap` from `/usr/local/bin`; verify helpful error message from non-root |
|
||||
| Packet buffer overflow | Phase 1/2: Capture Pipeline | `SetBufferSize` call present; goroutine architecture is async (channel-separated) |
|
||||
| ZeroCopy use-after-free | Phase 1: Capture/Decode | Code review: no `ZeroCopy` passed to goroutines without copy; or use `ReadPacketData` |
|
||||
| LAME init errors / corrupt MP3 | Audio synthesis phase | CI smoke test: 1s silence → `ffprobe` validates output file |
|
||||
| PCM overflow wrap-around | Audio synthesis phase | Unit test: 8 simultaneous max-amplitude layers produce no distortion |
|
||||
| Perceptual tone chaos | Audio mapping phase | Subjective listen test with mixed traffic capture; frequency table reviewed against auditory masking |
|
||||
| Time window jitter | Traffic aggregation / mapping phase | Capture test with bursty traffic; verify EMA smoothing produces stable amplitude |
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [gopacket/gopacket (community fork, v1.5.0)](https://github.com/gopacket/gopacket) — active fork status
|
||||
- [google/gopacket issue #329: 98% packet loss under high traffic](https://github.com/google/gopacket/issues/329) — buffer overflow and afpacket solution
|
||||
- [google/gopacket issue #1016: current project status](https://github.com/google/gopacket/issues/1016) — unmaintained status of original repo
|
||||
- [google/gopacket issue #1167: static linking libpcap.a](https://github.com/google/gopacket/issues/1167) — static build complications
|
||||
- [ZeroCopyReadPacketData docs (gopacket/pcap)](https://pkg.go.dev/github.com/google/gopacket/pcap) — memory ownership contract
|
||||
- [linuxvox.com: CAP_NET_RAW outside /usr/bin](https://linuxvox.com/blog/raw-capture-capabilities-cap-net-raw-cap-net-admin-not-working-outside-usr-bin-and-friends-for-packet-capture-program-using-libpcap/) — nosuid and AppArmor restrictions
|
||||
- [braheezy.github.io: What I Learned About MP3 Encoding](https://braheezy.github.io/posts/what-i-learned-about-mp3-encoding/) — Go MP3 encoding pitfalls
|
||||
- [github.com/braheezy/shine-mp3](https://github.com/braheezy/shine-mp3) — pure Go MP3 encoder (no CGo)
|
||||
- [Eli Bendersky: Building Static Binaries with Go on Linux](https://eli.thegreenplace.net/2024/building-static-binaries-with-go-on-linux/) — CGo static linking strategy
|
||||
- [SoNSTAR: Sonification of Networks for Situational Awareness](https://github.com/nuson/SoNSTAR) — reference architecture for network sonification
|
||||
- [PLOS One: Sonification of Network Traffic Flow](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0195948) — time window and design lessons
|
||||
- [KVR Audio: PCM float-to-int clipping and wrap-around](https://www.kvraudio.com/forum/viewtopic.php?t=414666) — PCM overflow consequences
|
||||
- [bjornroche.com: ABCs of PCM Digital Audio](http://blog.bjornroche.com/2013/05/the-abcs-of-pcm-uncompressed-digital.html) — sample format fundamentals
|
||||
|
||||
---
|
||||
*Pitfalls research for: network-traffic-to-audio synthesis CLI (Go) — NetSynth*
|
||||
*Researched: 2026-03-24*
|
||||
@@ -0,0 +1,166 @@
|
||||
# Stack Research
|
||||
|
||||
**Domain:** Go CLI tool — network packet capture, traffic classification, audio synthesis, MP3 encoding
|
||||
**Researched:** 2026-03-24
|
||||
**Confidence:** MEDIUM-HIGH (packet capture and CLI: HIGH; audio synthesis in Go: MEDIUM; MP3 encoding: MEDIUM)
|
||||
|
||||
---
|
||||
|
||||
## Recommended Stack
|
||||
|
||||
### Core Technologies
|
||||
|
||||
| Technology | Version | Purpose | Why Recommended |
|
||||
|------------|---------|---------|-----------------|
|
||||
| `github.com/gopacket/gopacket` | v1.5.0 | Packet capture, protocol decoding | The canonical Go packet library. Community fork (`gopacket/gopacket`) supersedes the original Google repo (`google/gopacket`) as of 2024; released v1.5.0 in November 2025, minimum Go 1.24. 14.5k dependents; has ICMP, TCP, UDP, DNS, TLS layer decoders built in. |
|
||||
| `github.com/packetcap/go-pcap` | v0.0.0-20251215 | Pure-Go live packet capture backend | Replaces CGo libpcap dependency for live capture. 100% native Go, Linux + macOS, mmap-based kernel ring buffer for performance. Implements the `gopacket.PacketDataSource` interface so gopacket decodes packets on top of it. Enables CGO_ENABLED=0 builds and cross-compilation. |
|
||||
| `github.com/sjzar/go-lame` | v0.0.9 | MP3 encoding | Embeds libmp3lame C source directly via CGo — no external `libmp3lame` system package required. Published April 2025. Exposes sample rate, channels, quality control. Produces LAME-quality MP3, unlike the pure-Go shine-mp3 port which produces larger, lower-quality output. Tradeoff: requires CGo, so `CGO_ENABLED=1` and a C compiler at build time. |
|
||||
| `github.com/spf13/cobra` | v1.10.2 | CLI flag parsing and command structure | The industry standard for Go CLIs (Kubernetes, Docker, Hugo, etc.). v1.10.2 released December 2025. Handles `--interface`, `--output` flags, Ctrl+C signal plumbing, and `--help` generation automatically. No alternatives worth considering for this scope. |
|
||||
|
||||
### Supporting Libraries
|
||||
|
||||
| Library | Version | Purpose | When to Use |
|
||||
|---------|---------|---------|-------------|
|
||||
| `github.com/muesli/kmeans` | v0.3.1 | K-means clustering for unrecognized traffic patterns | Use to auto-cluster packets that don't match known protocol rules. Feed feature vectors: [port, protocol_num, packet_size_bin, direction]. Last release July 2022 but mathematically stable; the algorithm doesn't change. Alternatively, implement a simple incremental classifier directly (see Architecture notes below). |
|
||||
| `github.com/go-audio/wav` | latest | WAV file I/O as intermediate format | Use to write synthesized PCM as WAV before MP3 encoding pass. "Battle tested" per maintainer. Simplifies the PCM → encoder pipeline: synthesize float64 samples → write WAV → re-read as PCM → LAME encode. |
|
||||
| `golang.org/x/sys/unix` | stdlib | Raw socket / CAP_NET_RAW privilege checks | Use for detecting if the process has required privileges and for signaling (SIGINT for clean shutdown). Part of Go extended stdlib — no external version pinning needed. |
|
||||
|
||||
### Development Tools
|
||||
|
||||
| Tool | Purpose | Notes |
|
||||
|------|---------|-------|
|
||||
| `go build -ldflags="-s -w"` | Stripped binary production builds | Reduces binary size significantly; combine with `upx` if size is critical |
|
||||
| `goreleaser` | Cross-platform release builds | Handles CGo cross-compilation complexity with Docker-based build matrix; useful for distributing Linux x86_64 + ARM64 binaries |
|
||||
| `golangci-lint` | Static analysis | Catches nil pointer dereferences common in packet-handling code |
|
||||
| Wireshark / `tcpdump` | Manual verification of packet capture | Essential for confirming gopacket is decoding the right protocols before plugging into audio synthesis |
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Initialize module
|
||||
go mod init netsynth
|
||||
|
||||
# Core dependencies
|
||||
go get github.com/gopacket/gopacket@v1.5.0
|
||||
go get github.com/packetcap/go-pcap@latest
|
||||
go get github.com/sjzar/go-lame@v0.0.9
|
||||
go get github.com/spf13/cobra@v1.10.2
|
||||
|
||||
# Supporting
|
||||
go get github.com/muesli/kmeans@v0.3.1
|
||||
go get github.com/go-audio/wav@latest
|
||||
|
||||
# Build (CGo required for go-lame)
|
||||
CGO_ENABLED=1 go build -ldflags="-s -w" -o netsynth ./cmd/netsynth
|
||||
```
|
||||
|
||||
**Root/privilege requirement at runtime (not build time):**
|
||||
```bash
|
||||
sudo ./netsynth --interface eth0 --output traffic.mp3
|
||||
# OR: grant capability instead of running as root
|
||||
sudo setcap cap_net_raw+ep ./netsynth
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
| Recommended | Alternative | When to Use Alternative |
|
||||
|-------------|-------------|-------------------------|
|
||||
| `gopacket/gopacket` (community fork) | `google/gopacket` (original) | Never for new projects — original repo has 270 open issues, community fork actively merges fixes |
|
||||
| `packetcap/go-pcap` (pure Go) | `gopacket/pcap` (CGo + libpcap) | Use libpcap path only if you need advanced BPF filter syntax or BSD/Windows support — it requires `libpcap-dev` system package |
|
||||
| `sjzar/go-lame` (embedded C source) | `braheezy/shine-mp3` (pure Go) | Use shine-mp3 if CGo is truly impossible (e.g., WASM target) — but accept that output quality and file size are worse |
|
||||
| `sjzar/go-lame` (embedded C source) | `viert/go-lame` (dynamic link) | Never — viert/go-lame requires libmp3lame installed on the target system, defeating single-binary distribution |
|
||||
| Hand-rolled additive synthesis | `dasa.cc/snd`, `bspaans/bleep` | Use a library only if you need MIDI scheduling or real-time playback; for file output, the synthesis math is simple enough to own directly (see Architecture notes) |
|
||||
| `muesli/kmeans` | `mpraski/clusters` | Use mpraski if you need online (incremental) clustering — it supports add-one-point updates vs muesli's batch-only approach |
|
||||
|
||||
---
|
||||
|
||||
## What NOT to Use
|
||||
|
||||
| Avoid | Why | Use Instead |
|
||||
|-------|-----|-------------|
|
||||
| `google/gopacket` (original) | Effectively unmaintained since 2022; 270 open issues, PRs not merged | `github.com/gopacket/gopacket` (community fork, v1.5.0) |
|
||||
| `viert/go-lame` or `sunicy/go-lame` | Dynamic-links against system `libmp3lame` — breaks single-binary distribution, fails on machines without the library | `github.com/sjzar/go-lame` (embeds C source statically) |
|
||||
| `braheezy/shine-mp3` (pure Go MP3) | Last commit 2023, explicitly not production-ready per its own README, produces larger lower-quality files, no bitrate control | `sjzar/go-lame` for quality, or WAV output if you must avoid CGo |
|
||||
| `go-audio/generator` | **Archived February 2026, read-only** — do not take a new dependency on it | Write your own oscillator (20 lines of Go) or use `dasa.cc/snd` |
|
||||
| `faiface/beep` | Designed for real-time audio playback via PortAudio/oto; pulls in platform audio drivers that are irrelevant for file output | Roll a minimal additive synthesizer directly (see below) |
|
||||
| `dasa.cc/snd` | Plays audio through hardware; brings in real-time audio scheduling complexity unnecessary for batch file output | Roll a minimal additive synthesizer directly |
|
||||
| urfave/cli | Fine for simpler tools, but Cobra's flag validation, help generation, and signal handling are better for a tool with multiple flags and clean shutdown semantics | `github.com/spf13/cobra` |
|
||||
|
||||
---
|
||||
|
||||
## Stack Patterns by Variant
|
||||
|
||||
**If CGo is acceptable (recommended path):**
|
||||
- Use `sjzar/go-lame` for real MP3 quality
|
||||
- Use `packetcap/go-pcap` for the capture layer (pure Go on Linux/macOS)
|
||||
- Build with `CGO_ENABLED=1`; single binary is still self-contained because LAME C source is embedded
|
||||
|
||||
**If pure Go / no CGo is required (e.g., restricted build environment):**
|
||||
- Use `braheezy/shine-mp3` for MP3 — accept lower quality and larger files
|
||||
- Use `packetcap/go-pcap` for capture — already pure Go
|
||||
- Build with `CGO_ENABLED=0`; truly static binary
|
||||
|
||||
**If Linux-only deployment is acceptable:**
|
||||
- Consider `packetcap/go-pcap`'s mmap ring buffer mode for high-traffic interfaces (default on Linux)
|
||||
- Privilege: `CAP_NET_RAW` setcap is cleaner than running as root
|
||||
|
||||
**For the audio synthesis layer — roll your own, don't use a library:**
|
||||
The ambient/drone requirement is additive synthesis: N sine wave oscillators, each with a frequency and time-varying amplitude. This is 30-50 lines of Go:
|
||||
```go
|
||||
// Conceptual — not a library call
|
||||
for t := 0; t < numSamples; t++ {
|
||||
sample := 0.0
|
||||
for _, layer := range layers {
|
||||
sample += layer.Amplitude(t) * math.Sin(2*math.Pi*layer.Freq*float64(t)/sampleRate)
|
||||
}
|
||||
pcm[t] = int16(sample * 32767)
|
||||
}
|
||||
```
|
||||
No library adds value here. Libraries designed for real-time playback add complexity (audio thread management, ring buffers, OS audio drivers) that hurts a batch file-output tool.
|
||||
|
||||
---
|
||||
|
||||
## Version Compatibility
|
||||
|
||||
| Package | Compatible With | Notes |
|
||||
|---------|-----------------|-------|
|
||||
| `gopacket/gopacket@v1.5.0` | Go 1.24+ | v1.5.0 bumped minimum Go to 1.24; use Go 1.24.x toolchain |
|
||||
| `packetcap/go-pcap` | Linux, macOS (Darwin) | No Windows support; this is acceptable per project constraints |
|
||||
| `sjzar/go-lame@v0.0.9` | Any Go + C compiler; CGO_ENABLED=1 | Embeds LAME C source; no system library dependency |
|
||||
| `spf13/cobra@v1.10.2` | Go 1.20+ | No issues with Go 1.24 |
|
||||
| `muesli/kmeans@v0.3.1` | Go 1.12+ | Stable; no compatibility concerns |
|
||||
|
||||
---
|
||||
|
||||
## Audio Synthesis Architecture Note
|
||||
|
||||
Do not reach for an audio library. The synthesis requirement is:
|
||||
1. Map each traffic class (ICMP, DNS, HTTPS, SSH, unknown-cluster-N) to a base frequency
|
||||
2. Accumulate packet counts per class per time window (e.g., 500ms buckets)
|
||||
3. Drive oscillator amplitude from smoothed packet rate (exponential moving average)
|
||||
4. Sum N oscillators into PCM samples at 44100 Hz, 16-bit, mono
|
||||
5. Write PCM to WAV via `go-audio/wav`, then encode WAV to MP3 via `sjzar/go-lame`
|
||||
|
||||
The WAV intermediate step decouples synthesis from encoding and gives you a debug artifact. Total synthesis code: ~100 lines. No external library needed.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- `github.com/gopacket/gopacket` releases page — v1.5.0 confirmed, November 2025
|
||||
- `pkg.go.dev/github.com/packetcap/go-pcap` — v0.0.0-20251215, pure Go, Linux/macOS confirmed
|
||||
- `pkg.go.dev/github.com/sjzar/go-lame` — v0.0.9, April 2025, embedded C source confirmed
|
||||
- `pkg.go.dev/github.com/spf13/cobra` — v1.10.2, December 2025
|
||||
- `github.com/go-audio/generator` — archived February 2026 (read-only), do not use
|
||||
- `braheezy.github.io/posts/what-i-learned-about-mp3-encoding/` — author's first-hand account of Go MP3 encoding options, concluded shine-mp3 is not production-grade
|
||||
- `github.com/google/gopacket/issues/1016` — maintenance status discussion confirming community fork is preferred
|
||||
- WebSearch: muesli/kmeans v0.3.1 last release July 2022 — LOW confidence on ongoing maintenance, but algorithm is stable
|
||||
- WebSearch: cobra v1.9.1/v1.10.2 — MEDIUM confidence, confirmed via pkg.go.dev
|
||||
|
||||
---
|
||||
*Stack research for: NetSynth — Go CLI network-traffic-to-audio tool*
|
||||
*Researched: 2026-03-24*
|
||||
@@ -0,0 +1,188 @@
|
||||
# Project Research Summary
|
||||
|
||||
**Project:** NetSynth
|
||||
**Domain:** Network traffic sonification CLI — Go, packet capture, audio synthesis, MP3 encoding
|
||||
**Researched:** 2026-03-24
|
||||
**Confidence:** MEDIUM-HIGH
|
||||
|
||||
## Executive Summary
|
||||
|
||||
NetSynth is a Go CLI tool that captures live network traffic, classifies packets by protocol, and synthesizes an ambient drone MP3 where each protocol layer produces a distinct tonal frequency whose amplitude evolves with traffic volume. There is no direct precedent for this exact form factor: comparable tools (SoNSTAR, Peep, Network-Sonification) all produce real-time audio through OS audio APIs rather than file output, are not single binaries, and do not auto-cluster unknown traffic. The recommended approach builds on well-understood Go concurrency primitives (channel-connected pipeline stages, ticker-driven time windows) rather than audio or ML libraries — the synthesis math is ~100 lines of Go and no external audio framework adds value for batch file output.
|
||||
|
||||
The recommended stack is `gopacket/gopacket` v1.5.0 (community fork, not the abandoned Google repo) for packet decode, `packetcap/go-pcap` for pure-Go live capture on Linux/macOS, `sjzar/go-lame` v0.0.9 for embedded-CGo MP3 encoding, and `spf13/cobra` v1.10.2 for CLI structure. The audio synthesis layer should be hand-rolled — additive sine oscillators with exponential-moving-average amplitude smoothing. This stack requires CGo at build time but produces a self-contained binary with no runtime library dependencies beyond `CAP_NET_RAW` or root for packet capture.
|
||||
|
||||
The two hardest risks are at opposite ends of the pipeline. On the capture side: privilege requirements, CGo binary distribution contracts, and kernel buffer drops all bite in production but not in local development. On the audio side: PCM integer overflow, perceptual tone masking between protocol frequencies, and LAME initialization order all produce silent or subtle corruption that integration tests must specifically cover. Both risk clusters must be resolved in Phase 1 and Phase 2 respectively — they cannot be retrofitted.
|
||||
|
||||
## Key Findings
|
||||
|
||||
### Recommended Stack
|
||||
|
||||
The stack is straightforward for Go developers with one critical trap: `github.com/google/gopacket` is unmaintained (270 open issues, no active merges since 2022) and must never be used — the import path is `github.com/gopacket/gopacket` (community fork, v1.5.0, Go 1.24+ required). For the capture backend, `packetcap/go-pcap` is pure Go and eliminates libpcap CGo entirely; the MP3 encoder `sjzar/go-lame` embeds LAME C source and requires CGo but no system library on the target machine. Audio synthesis should be written directly — no audio library is appropriate for batch file output. See `.planning/research/STACK.md` for full alternatives matrix.
|
||||
|
||||
**Core technologies:**
|
||||
- `github.com/gopacket/gopacket` v1.5.0: packet decode (ICMP, DNS, TCP, UDP, TLS layers) — only maintained Go packet library
|
||||
- `github.com/packetcap/go-pcap`: live capture backend — pure Go, mmap ring buffer, Linux/macOS, no CGo
|
||||
- `github.com/sjzar/go-lame` v0.0.9: MP3 encoding — embeds LAME C source, no runtime `.so` dependency, requires CGo at build time
|
||||
- `github.com/spf13/cobra` v1.10.2: CLI flag parsing — industry standard, handles signal plumbing and help generation
|
||||
- `github.com/go-audio/wav`: WAV intermediate format — decouples synthesis from encoding, provides debug artifact
|
||||
- Hand-rolled additive synthesizer: 30-50 lines of Go sine oscillators, no library needed
|
||||
|
||||
### Expected Features
|
||||
|
||||
NetSynth has a well-defined feature set. All precedent tools provide real-time audio output, not file output — this is both a differentiator and a source of user confusion to address in UX copy. Auto-clustering of unknown traffic is unique to NetSynth among comparable tools. See `.planning/research/FEATURES.md` for full prioritization matrix and competitor analysis.
|
||||
|
||||
**Must have (table stakes):**
|
||||
- Interface selection (`-i eth0`) and `--list-interfaces` — packet capture CLI convention
|
||||
- Live capture with graceful Ctrl+C stop producing a valid MP3 — core interaction model
|
||||
- Per-protocol sound distinction: ICMP, DNS, TCP/443, TCP/other, UDP — minimum fingerprint set
|
||||
- Ambient drone synthesis with time-windowed amplitude evolution — core differentiator
|
||||
- MP3 encoding and file output with sensible default filename — deliverable artifact
|
||||
- Capture statistics summary on exit (stderr) — expected by every capture tool user
|
||||
- Privilege error detection with actionable message — prevents silent failure
|
||||
- Minimum viable duration guard — zero-packet capture must not produce a corrupt file
|
||||
|
||||
**Should have (competitive):**
|
||||
- Auto-clustering of unrecognized traffic into stable drone layers — honest representation, unique feature
|
||||
- BPF capture filter (`--filter`) — power user scope control
|
||||
- Offline pcap file input (`--read`) — historical analysis and demos
|
||||
- Verbose protocol activity log (`--verbose`) — debugging and demo use cases
|
||||
- Configurable time window duration (`--window`) — tuning for responsiveness vs. smoothness
|
||||
|
||||
**Defer (v2+):**
|
||||
- Custom sound mapping configuration file — needs user research first; well-chosen defaults cover v1
|
||||
- Improved clustering (full k-means on flow features) — hash-bucketing is sufficient for v1
|
||||
- Multi-interface capture — adds deduplication complexity
|
||||
- Real-time audio playback — anti-feature; triples cross-platform complexity, conflicts with file-output simplicity
|
||||
|
||||
### Architecture Approach
|
||||
|
||||
The architecture is a channel-connected pipeline of five stages, each a goroutine communicating via buffered channels, with a `done` channel closed on Ctrl+C driving clean shutdown across all stages. The stages are: Capture (pcap handle → `chan gopacket.Packet`), Classification (rule-based protocol dispatch + unknown traffic bucketer → `chan ClassifiedPacket`), Aggregation (ticker-driven time-window accumulator → `chan WindowSnapshot`), Synthesis (per-class sine oscillators with EMA amplitude smoothing → PCM blocks), and Encoding (LAME encoder goroutine consuming PCM blocks, separate from synthesis to decouple CGo latency). This is the idiomatic Go pipeline pattern and directly follows the build order the architecture research prescribes. See `.planning/research/ARCHITECTURE.md` for full data flow diagrams and channel buffer size recommendations.
|
||||
|
||||
**Major components:**
|
||||
1. `capture/` — pcap handle wrapper; privilege/interface boundary; all downstream code is pcap-free
|
||||
2. `classify/` — protocol rule dispatch + feature-hash bucketer for unknown traffic; testable with synthetic packets
|
||||
3. `aggregate/` — ticker-driven time-window accumulator; the only stateful time-domain component
|
||||
4. `synth/` — sine oscillators + EMA amplitude smoothing + mixer; pure PCM math, no I/O
|
||||
5. `encode/` — CGo LAME boundary; decoupled encoder goroutine; if encoder changes, only this package changes
|
||||
6. `config/` — static frequency-to-protocol mapping table; prevents magic numbers in synth/
|
||||
|
||||
### Critical Pitfalls
|
||||
|
||||
Nine pitfalls identified, ranging from critical (causes data corruption or broken binaries) to moderate (causes poor audio quality). The top five require architectural decisions in Phase 1 or Phase 2 — they cannot be patched later without significant rewrite. See `.planning/research/PITFALLS.md` for recovery strategies and the full "Looks Done But Isn't" checklist.
|
||||
|
||||
1. **Wrong gopacket fork (`google/` vs `gopacket/`)** — use `github.com/gopacket/gopacket` from day one; import path migration across the whole codebase is the recovery cost
|
||||
2. **CGo breaking the single-binary promise** — decide the static vs. dynamic linking strategy before writing capture code; verify with `ldd` on a clean Alpine container in CI
|
||||
3. **`CAP_NET_RAW` + nosuid filesystem = silent failure** — install to `/usr/local/bin` for capability mode; always provide a `sudo ./netsynth` fallback with clear error messaging
|
||||
4. **PCM integer overflow producing wrap-around distortion** — synthesize internally in `float64 [-1.0, 1.0]`, clamp before casting to `int16`; never do audio math in integer types
|
||||
5. **Perceptual tone masking (all protocols in the same frequency band)** — design the frequency table with register separation: low drones for bulk traffic, mid for control, high for interactive; use harmonic intervals not arithmetic spacing
|
||||
|
||||
## Implications for Roadmap
|
||||
|
||||
Based on research, the architecture's build-order prescription maps directly to a three-phase roadmap. Each phase is independently testable before the next is wired in.
|
||||
|
||||
### Phase 1: Capture and Classification Pipeline
|
||||
|
||||
**Rationale:** The packet capture layer carries the highest technical risk (privilege, CGo, binary distribution, buffer overflow). Validating it first — before any audio code exists — means the hardest pitfalls are resolved while the codebase is small. The architecture research explicitly names this as the correct first step.
|
||||
|
||||
**Delivers:** A working CLI that opens a network interface, classifies packets by protocol, and prints a live traffic summary to stderr. No audio output yet — just proof the pipeline works. The privilege error message and `--list-interfaces` flag ship here.
|
||||
|
||||
**Addresses:** Interface selection, live capture, protocol classification, privilege detection, `--list-interfaces`, capture statistics
|
||||
|
||||
**Avoids:** Wrong gopacket fork (Pitfall 1), CGo distribution contract (Pitfall 2), CAP_NET_RAW binary location (Pitfall 3), packet buffer overflow (Pitfall 4), ZeroCopy use-after-free (Pitfall 5)
|
||||
|
||||
### Phase 2: Audio Synthesis Engine
|
||||
|
||||
**Rationale:** The synthesis engine is pure PCM math with no pcap dependency. It can be built and tested in isolation with synthetic `WindowSnapshot` inputs before any real traffic flows through it. This is the architecture research's explicit recommendation. Separating synthesis from capture also means audio bugs are diagnosed without needing a live network.
|
||||
|
||||
**Delivers:** A synthesizer that accepts `WindowSnapshot` inputs and produces a valid MP3 file. End-to-end smoke test: silence input → `ffprobe`-validated MP3 output. The frequency mapping table, EMA amplitude smoothing, and the LAME encoder goroutine all ship here.
|
||||
|
||||
**Uses:** `sjzar/go-lame`, `go-audio/wav`, hand-rolled oscillator, `config/mapping.go` frequency table
|
||||
|
||||
**Implements:** `synth/` (oscillator, layer, mixer), `encode/` (LAME goroutine), `aggregate/` (time-window accumulator), `config/` (frequency mapping)
|
||||
|
||||
**Avoids:** LAME initialization errors (Pitfall 6), PCM overflow wrap-around (Pitfall 7), perceptual tone masking (Pitfall 8), time window jitter (Pitfall 9)
|
||||
|
||||
### Phase 3: Pipeline Integration and CLI Polish
|
||||
|
||||
**Rationale:** Wire the Phase 1 capture/classify pipeline to the Phase 2 synthesis engine via the aggregation layer. Add Ctrl+C shutdown producing a valid MP3 (requires coordinated drain across all goroutines). Add auto-clustering of unknown traffic. Add UX features (progress output, file collision warning, graceful empty-capture error).
|
||||
|
||||
**Delivers:** The complete v1 MVP: live capture → protocol classification + auto-clustering → time-windowed synthesis → MP3 file output. Graceful Ctrl+C with valid MP3. Full UX surface (startup interface announcement, per-window progress line, exit statistics).
|
||||
|
||||
**Addresses:** Auto-clustering, graceful Ctrl+C stop, capture statistics, progress feedback, output file collision warning, zero-packet guard, `main.go` pipeline wiring
|
||||
|
||||
**Uses:** All Phase 1 and Phase 2 components; `muesli/kmeans` or hash-bucketing for unknown traffic clustering
|
||||
|
||||
### Phase 4: Power User Features (v1.x)
|
||||
|
||||
**Rationale:** These features add value for specific user segments but have no blocking dependencies on each other — add in any order based on user feedback after the core is validated.
|
||||
|
||||
**Delivers:** BPF capture filter (`--filter`), offline pcap file input (`--read`), verbose protocol log (`--verbose`), configurable time window (`--window`), configurable output duration for pcap input (`--duration`)
|
||||
|
||||
**Addresses:** All P2 features from the prioritization matrix in FEATURES.md
|
||||
|
||||
### Phase Ordering Rationale
|
||||
|
||||
- **Capture before synthesis:** The privilege and CGo pitfalls are foundational — an audio-first approach would hide them until integration and make them expensive to fix.
|
||||
- **Synthesis in isolation:** Pure PCM math is independently testable. Building it against synthetic inputs before real traffic makes audio bugs fast to diagnose.
|
||||
- **Integration as its own phase:** The shutdown coordination (Ctrl+C → drain → flush encoder → close file) across five goroutines is non-trivial; it deserves focused attention rather than being an afterthought of feature development.
|
||||
- **Power features deferred:** BPF filter and offline pcap do not validate the core concept; they add complexity to the capture layer that should wait until the pipeline is stable.
|
||||
|
||||
### Research Flags
|
||||
|
||||
Phases likely needing deeper research during planning:
|
||||
|
||||
- **Phase 2 (Audio Synthesis):** The perceptual frequency mapping table requires listening tests, not just code correctness. Research the auditory masking literature before finalizing `config/mapping.go`. Consider consulting the SoNSTAR PLOS One paper on time window choices.
|
||||
- **Phase 3 (Auto-clustering):** The decision between simple hash-bucketing and k-means clustering (muesli/kmeans) depends on what "meaningfully distinct drone layers" means in practice. This needs a working synthesis engine to evaluate — defer the decision to Phase 3 planning.
|
||||
- **Phase 4 (Offline pcap):** Time-compression of multi-hour pcap files to a fixed audio duration needs a clear algorithm decision (proportional window scaling vs. fixed window with truncation). Research this when Phase 4 is planned.
|
||||
|
||||
Phases with standard patterns (skip research-phase):
|
||||
|
||||
- **Phase 1 (Capture pipeline):** Go channel pipelines and gopacket usage are thoroughly documented. The pitfalls are known and avoidable with the guidance in PITFALLS.md.
|
||||
- **Phase 3 (Pipeline integration):** Go done-channel shutdown patterns are canonical (Go Blog: Pipelines). No novel research needed.
|
||||
|
||||
## Confidence Assessment
|
||||
|
||||
| Area | Confidence | Notes |
|
||||
|------|------------|-------|
|
||||
| Stack | HIGH | Core libraries confirmed via pkg.go.dev; version numbers verified; `go-audio/generator` archived status confirmed February 2026 |
|
||||
| Features | MEDIUM | Niche domain with few direct CLI comparators; feature set derived from tcpdump conventions and sonification research, not user surveys |
|
||||
| Architecture | MEDIUM-HIGH | Go pipeline patterns are HIGH confidence (official Go Blog); audio synthesis architecture inferred from SoNSTAR paper (MEDIUM, abstract-level access only) |
|
||||
| Pitfalls | HIGH | Packet capture pitfalls verified against official gopacket issues and libpcap docs; audio pitfalls cross-referenced against DSP literature and encoder post-mortems |
|
||||
|
||||
**Overall confidence:** MEDIUM-HIGH
|
||||
|
||||
### Gaps to Address
|
||||
|
||||
- **Frequency mapping validation:** The correct frequency assignments for the protocol drone layers require subjective listening tests with real traffic. The research prescribes the *approach* (register separation, harmonic intervals) but not specific Hz values. Validate during Phase 2 with a listening session before Phase 3 integration.
|
||||
- **Auto-clustering granularity:** How many unknown-traffic clusters are perceptually useful? The research suggests hash-bucketing is acceptable for v1, but does not validate how many distinct cluster tones are distinguishable simultaneously. Validate during Phase 3 with real mixed traffic.
|
||||
- **muesli/kmeans maintenance status:** Last release July 2022 (LOW confidence on ongoing maintenance). If Go 1.24 compatibility issues emerge, the alternative is `mpraski/clusters` (online clustering) or a hand-rolled hash bucketer. Plan for substitution.
|
||||
- **macOS privilege model:** CAP_NET_RAW pitfall was verified for Linux. macOS uses a different privilege model (BPF device permissions). If macOS is a target, verify the privilege flow and error messages during Phase 1.
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (HIGH confidence)
|
||||
|
||||
- `github.com/gopacket/gopacket` releases — v1.5.0 November 2025, Go 1.24+ confirmed
|
||||
- `pkg.go.dev/github.com/packetcap/go-pcap` — pure Go, Linux/macOS confirmed
|
||||
- `pkg.go.dev/github.com/sjzar/go-lame` — v0.0.9 April 2025, embedded C source confirmed
|
||||
- `pkg.go.dev/github.com/spf13/cobra` — v1.10.2 December 2025
|
||||
- Go Blog: Pipelines and cancellation — canonical Go pipeline pattern
|
||||
- `google/gopacket` issue #1016 — unmaintained status confirmed
|
||||
- `google/gopacket` issue #329 — 98% packet drop under high traffic, afpacket solution
|
||||
- linuxvox.com: CAP_NET_RAW + nosuid filesystem behavior
|
||||
|
||||
### Secondary (MEDIUM confidence)
|
||||
|
||||
- SoNSTAR PLOS One paper (arXiv 1712.07029) — time window design and sonification architecture
|
||||
- braheezy.github.io: Go MP3 encoding options — shine-mp3 not production-grade
|
||||
- Dylan Meeus: Audio From Scratch With Go — PCM synthesis patterns
|
||||
- `github.com/go-audio/generator` — archived February 2026, do not use (confirmed read-only)
|
||||
|
||||
### Tertiary (LOW confidence)
|
||||
|
||||
- `muesli/kmeans` v0.3.1 (July 2022) — last release date only; ongoing Go 1.24 compatibility unverified
|
||||
- Drone auralization model, Acta Acustica 2024 — amplitude/frequency modulation patterns (MEDIUM, used for perceptual guidance)
|
||||
|
||||
---
|
||||
*Research completed: 2026-03-24*
|
||||
*Ready for roadmap: yes*
|
||||
Reference in New Issue
Block a user