--- phase: 04-power-user-features plan: 01 type: execute wave: 1 depends_on: [] files_modified: - classify/types.go - capture/capture.go - capture/bpf.go - capture/pcap_reader.go - aggregate/window.go - capture/capture_test.go - capture/bpf_test.go - capture/pcap_reader_test.go - aggregate/window_test.go autonomous: true requirements: [CAPT-05, CAPT-06] must_haves: truths: - "BPF filter expressions can be validated without a live socket" - "A valid pcap file can be read and packets emitted on a channel" - "Invalid/missing pcap files produce clear error messages" - "Pcap packets are aggregated into time windows using packet timestamps, not wall clock" - "Gap windows (no packets) produce empty snapshots preserving silence" - "BPF filter can be applied to live capture handles" - "Software BPF filtering works for pcap file packets" artifacts: - path: "capture/bpf.go" provides: "BPF validation and software filter compilation" exports: ["ValidateBPFFilter", "CompileSoftwareBPF"] - path: "capture/pcap_reader.go" provides: "Pcap file reading into packet channel" exports: ["ReadPcapFile"] - path: "aggregate/window.go" provides: "Timestamp-based aggregation for pcap mode" exports: ["AggregatePcap"] - path: "classify/types.go" provides: "Timestamp field on ClassifiedPacket" contains: "Timestamp time.Time" key_links: - from: "capture/pcap_reader.go" to: "gopacket/pcapgo" via: "pcapgo.NewReader" pattern: "pcapgo\\.NewReader" - from: "capture/bpf.go" to: "packetcap/go-pcap/filter" via: "filter.NewExpression" pattern: "filter\\.NewExpression" - from: "aggregate/window.go" to: "classify/types.go" via: "ClassifiedPacket.Timestamp for window assignment" pattern: "ev\\.Timestamp" --- Add core library functions for BPF filtering and pcap file reading. Purpose: Build the foundation that Plan 02 wires into the CLI. BPF validation, pcap reading, software BPF matching, and timestamp-based aggregation are all independently testable library functions. Output: New files capture/bpf.go, capture/pcap_reader.go; updated classify/types.go (Timestamp field), updated capture/capture.go (filter param on OpenCapture), updated aggregate/window.go (AggregatePcap); all with tests. @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md @.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/04-power-user-features/04-CONTEXT.md @.planning/phases/04-power-user-features/04-RESEARCH.md @capture/capture.go @capture/capture_test.go @classify/types.go @classify/classifier.go @aggregate/window.go @aggregate/window_test.go From classify/types.go: ```go type TrafficClass string type ClassifiedPacket struct { Class TrafficClass SrcPort uint16 DstPort uint16 Protocol string Length int } type WindowSnapshot struct { Counts map[TrafficClass]int64 TotalPackets int64 WindowIndex int } ``` From capture/capture.go: ```go func OpenCapture(ctx context.Context, iface string) (*pcap.Handle, error) func StartCapture(ctx context.Context, iface string) (<-chan gopacket.Packet, *int64, error) ``` From aggregate/window.go: ```go const DefaultWindowMs = 500 func Aggregate(done <-chan struct{}, events <-chan classify.ClassifiedPacket, windowMs int, onSnapshot func(classify.WindowSnapshot)) <-chan classify.WindowSnapshot ``` From go-pcap (verified in module cache): ```go // pcap.Handle method: func (h *Handle) SetBPFFilter(expr string) error // filter package: func NewExpression(expr string) *Expression func (e *Expression) Compile() *Filter func (f *Filter) Compile() ([]bpf.Instruction, error) ``` From gopacket/pcapgo (verified in module cache): ```go func NewReader(r io.Reader) (*Reader, error) func (r *Reader) LinkType() layers.LinkType // Reader implements gopacket.PacketDataSource ``` From golang.org/x/net/bpf: ```go func NewVM(filter []bpf.Instruction) (*VM, error) func (vm *VM) Run(in []byte) (int, error) // >0 means packet passes ``` Task 1: Add Timestamp to ClassifiedPacket, BPF validation, pcap reading, and software BPF filter classify/types.go, capture/bpf.go, capture/pcap_reader.go, capture/capture.go, capture/bpf_test.go, capture/pcap_reader_test.go, capture/capture_test.go classify/types.go, capture/capture.go, capture/capture_test.go, classify/classifier.go - TestValidateBPFFilter: ValidateBPFFilter("port 53") returns nil; ValidateBPFFilter("invalid garbage xyz") returns non-nil error containing "invalid BPF filter" - TestValidateBPFFilterEmpty: ValidateBPFFilter("") returns nil (empty = no filter) - TestCompileSoftwareBPF: CompileSoftwareBPF("tcp") returns a non-nil *bpf.VM and nil error - TestCompileSoftwareBPFInvalid: CompileSoftwareBPF("invalid garbage") returns nil VM and non-nil error - TestReadPcapFile: ReadPcapFile with a programmatically-generated pcap (using pcapgo.NewWriter to write 3 test packets) returns a channel that yields exactly 3 packets, then closes - TestReadPcapFileNotFound: ReadPcapFile("/nonexistent/file.pcap", "") returns error containing "cannot open" - TestReadPcapFileInvalid: ReadPcapFile on a non-pcap file (e.g., a temp file with garbage bytes) returns error containing "invalid pcap file" or "not a valid pcap" - TestReadPcapFileWithFilter: ReadPcapFile with filter="tcp" on a pcap containing both TCP and UDP packets returns only TCP packets - TestOpenCaptureWithFilter: OpenCapture now accepts a filter string parameter (signature change verified by compilation) - TestClassifiedPacketTimestamp: ClassifiedPacket struct has a Timestamp field of type time.Time **1. Update classify/types.go** — Add `Timestamp time.Time` field to `ClassifiedPacket`: ```go import "time" type ClassifiedPacket struct { Class TrafficClass SrcPort uint16 DstPort uint16 Protocol string Length int Timestamp time.Time // Set from pkt.Metadata().CaptureInfo.Timestamp; zero in live mode } ``` This is a backward-compatible addition. Live mode leaves it zero-valued (no behavior change). Per D-01, pcap mode uses it for window assignment. **2. Create capture/bpf.go** — Two exported functions: ```go package capture import ( "fmt" "strings" "golang.org/x/net/bpf" gpcapfilter "github.com/packetcap/go-pcap/filter" ) // ValidateBPFFilter checks if a BPF expression string is valid without needing a live socket. // Returns nil for empty strings (empty = no filter). Per CAPT-05. func ValidateBPFFilter(expr string) error { if strings.TrimSpace(expr) == "" { return nil } e := gpcapfilter.NewExpression(expr) if e == nil { return fmt.Errorf("invalid BPF filter expression: %q", expr) } compiled := e.Compile() if _, err := compiled.Compile(); err != nil { return fmt.Errorf("invalid BPF filter %q: %v", expr, err) } return nil } // CompileSoftwareBPF compiles a BPF expression into a VM for user-space packet matching. // Used for --read mode where kernel BPF is unavailable. Per D-04 (filter works with --read). func CompileSoftwareBPF(expr string) (*bpf.VM, error) { e := gpcapfilter.NewExpression(expr) if e == nil { return nil, fmt.Errorf("invalid BPF filter expression: %q", expr) } instructions, err := e.Compile().Compile() if err != nil { return nil, fmt.Errorf("BPF compile error for %q: %v", expr, err) } return bpf.NewVM(instructions) } ``` **3. Create capture/pcap_reader.go** — ReadPcapFile function: ```go package capture import ( "fmt" "os" "github.com/gopacket/gopacket" "github.com/gopacket/gopacket/layers" "github.com/gopacket/gopacket/pcapgo" "golang.org/x/net/bpf" ) // ReadPcapFile opens a pcap file and returns a channel of packets. // If filter is non-empty, applies software BPF filtering (kernel BPF unavailable for files). // Per D-03: replaces -i as packet source. Per D-04: --filter works with --read. func ReadPcapFile(path string, filter string) (<-chan gopacket.Packet, error) { f, err := os.Open(path) if err != nil { return nil, fmt.Errorf("cannot open pcap file %q: %w", path, err) } r, err := pcapgo.NewReader(f) if err != nil { f.Close() return nil, fmt.Errorf("not a valid pcap file %q: %w", path, err) } // Compile software BPF filter if provided var vm *bpf.VM if filter != "" { vm, err = CompileSoftwareBPF(filter) if err != nil { f.Close() return nil, err } } lt := layers.LinkType(r.LinkType()) packetSource := gopacket.NewPacketSource(r, lt) packetSource.NoCopy = false // CRITICAL: do NOT use NoCopy=true with pcapgo (Pitfall 3) packets := make(chan gopacket.Packet, 512) go func() { defer close(packets) defer f.Close() for pkt := range packetSource.Packets() { // Apply software BPF filter if active if vm != nil { result, err := vm.Run(pkt.Data()) if err != nil || result == 0 { continue // packet does not match filter } } packets <- pkt } }() return packets, nil } ``` **4. Update capture/capture.go** — Add `filter string` parameter to `OpenCapture` and `StartCapture`. In `OpenCapture`, after successfully opening the handle, call `handle.SetBPFFilter(filter)` if filter is non-empty. If SetBPFFilter fails, close the handle and return error wrapped with `fmt.Errorf("invalid BPF filter %q: %w", filter, err)`. Update `StartCapture` signature to `StartCapture(ctx context.Context, iface string, filter string)` and pass filter through to `OpenCapture`. **5. Create test files:** - `capture/bpf_test.go` — Tests for ValidateBPFFilter and CompileSoftwareBPF per behavior list above. - `capture/pcap_reader_test.go` — Uses `pcapgo.NewWriter` to create temporary pcap files programmatically (no binary fixtures). Write a helper `createTestPcap(t *testing.T, packets [][]byte) string` that creates a temp file with pcapgo.NewWriter, writes Ethernet+IP+TCP/UDP raw packets, and returns the path. Test ReadPcapFile with valid pcap, nonexistent file, invalid file (garbage bytes), and filter. - `capture/capture_test.go` — Add TestOpenCaptureSignature that verifies the new signature compiles (the existing tests don't call OpenCapture directly due to privilege requirements; just ensure the file compiles with the new signature). **6. Update cmd/netsynth/main.go call site** — The call `capture.StartCapture(ctx, ifaceName)` must become `capture.StartCapture(ctx, ifaceName, "")` to match the updated signature. This is a minimal change to keep existing code compiling. (The actual --filter flag wiring happens in Plan 02.) cd /home/dev/workspace/yoloyolo && export PATH="/home/dev/tools/go-install/go/bin:$PATH" && go test ./capture/... -run "TestValidateBPF|TestCompileSoftware|TestReadPcap|TestClassifiedPacket" -v -count=1 && go build ./... - classify/types.go contains `Timestamp time.Time` inside ClassifiedPacket struct - capture/bpf.go contains `func ValidateBPFFilter(expr string) error` - capture/bpf.go contains `func CompileSoftwareBPF(expr string) (*bpf.VM, error)` - capture/pcap_reader.go contains `func ReadPcapFile(path string, filter string) (<-chan gopacket.Packet, error)` - capture/pcap_reader.go contains `packetSource.NoCopy = false` - capture/capture.go contains `func OpenCapture(ctx context.Context, iface string, filter string)` - capture/capture.go contains `handle.SetBPFFilter(filter)` - capture/capture.go contains `func StartCapture(ctx context.Context, iface string, filter string)` - cmd/netsynth/main.go contains `capture.StartCapture(ctx, ifaceName, "")` - `go test ./capture/... -v` passes all new tests - `go build ./...` succeeds (all packages compile) BPF validation rejects invalid expressions and accepts valid ones. Pcap reader opens valid files and returns packet channels, rejects invalid/missing files. Software BPF filters pcap packets. OpenCapture/StartCapture accept filter parameter. ClassifiedPacket has Timestamp field. All tests pass, all packages compile. Task 2: Timestamp-based pcap aggregation with gap-filling aggregate/window.go, aggregate/window_test.go aggregate/window.go, aggregate/window_test.go, classify/types.go - TestAggregatePcapBasic: AggregatePcap with 3 packets at times T+0ms, T+100ms, T+200ms (all in window 0 for 500ms windows) returns 1 snapshot with TotalPackets=3 - TestAggregatePcapMultipleWindows: AggregatePcap with packets at T+0ms, T+600ms, T+1200ms returns 3 snapshots (window 0, 1, 2) each with TotalPackets=1 - TestAggregatePcapGaps: AggregatePcap with packets at T+0ms and T+1500ms (skip window 1 and 2) returns 4 snapshots: window 0 with 1 packet, windows 1-2 with TotalPackets=0, window 3 with 1 packet (per D-02: gaps are silent) - TestAggregatePcapEmpty: AggregatePcap with empty channel returns empty slice (no snapshots) - TestAggregatePcapWindowIndex: Each returned snapshot has correct sequential WindowIndex 0,1,2,... - TestAggregatePcapClassCounts: Packets of different classes in same window have correct per-class counts in snapshot.Counts - TestAggregatePcapOnSnapshot: If onSnapshot callback is provided, it is called for each emitted snapshot (for --verbose support per D-07) Add `AggregatePcap` function to `aggregate/window.go`. This function reads from a `<-chan classify.ClassifiedPacket` (same channel type as live mode), collects all events into a slice, then assigns each to a time window using `pkt.Timestamp.Sub(firstTimestamp).Milliseconds() / windowMs`. **Function signature:** ```go // AggregatePcap reads all ClassifiedPackets (with Timestamp set), assigns to time windows // using packet timestamps per D-01, fills gap windows with empty snapshots per D-02, // and returns all WindowSnapshots. Calls onSnapshot for each if non-nil (D-07: --verbose). // Returns synchronously after channel closes (pcap processing is finite). func AggregatePcap(events <-chan classify.ClassifiedPacket, windowMs int, onSnapshot func(classify.WindowSnapshot)) []classify.WindowSnapshot ``` **Implementation details:** 1. Drain the `events` channel into a `[]classify.ClassifiedPacket` slice. 2. If slice is empty, return `nil` (empty slice). 3. Find `minTimestamp` by scanning all events (handles non-monotonic pcaps per Pitfall 5). 4. Compute `maxWindowIdx` = `int(maxTimestamp.Sub(minTimestamp).Milliseconds()) / windowMs`. 5. Create `snapshots := make([]classify.WindowSnapshot, maxWindowIdx+1)`. Initialize each with `Counts: make(map[classify.TrafficClass]int64)` and `WindowIndex: i`. 6. For each event, compute `idx := int(ev.Timestamp.Sub(minTimestamp).Milliseconds()) / windowMs`. Clamp to `[0, maxWindowIdx]` for safety. Increment `snapshots[idx].Counts[ev.Class]++` and `snapshots[idx].TotalPackets++`. 7. If `onSnapshot` is non-nil, call it for each snapshot in order (supports --verbose per D-07). 8. Return `snapshots`. **Tests in aggregate/window_test.go** — Add test functions per behavior list. Use helper that creates `classify.ClassifiedPacket` values with specific `Timestamp` values relative to a base time `time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC)`. Send packets into a buffered channel, close it, then call `AggregatePcap`. cd /home/dev/workspace/yoloyolo && export PATH="/home/dev/tools/go-install/go/bin:$PATH" && go test ./aggregate/... -run "TestAggregatePcap" -v -count=1 - aggregate/window.go contains `func AggregatePcap(events <-chan classify.ClassifiedPacket, windowMs int, onSnapshot func(classify.WindowSnapshot)) []classify.WindowSnapshot` - aggregate/window_test.go contains `TestAggregatePcapBasic` - aggregate/window_test.go contains `TestAggregatePcapGaps` - aggregate/window_test.go contains `TestAggregatePcapEmpty` - `go test ./aggregate/... -run TestAggregatePcap -v` shows all tests PASS - Gap test verifies that windows with no packets have TotalPackets=0 AggregatePcap assigns packets to correct time windows using timestamps (D-01). Gap windows produce empty snapshots with TotalPackets=0 (D-02). onSnapshot callback fires for each window (D-07). All aggregation tests pass. Full package compilation and test suite: ```bash cd /home/dev/workspace/yoloyolo && export PATH="/home/dev/tools/go-install/go/bin:$PATH" && go test ./... -count=1 && go build ./... ``` All existing tests must continue to pass. All new tests must pass. `go build ./...` must succeed. 1. `ValidateBPFFilter("port 53")` returns nil; `ValidateBPFFilter("invalid xyz")` returns error 2. `ReadPcapFile` reads programmatically-generated pcap files and emits correct packet count 3. `ReadPcapFile` with filter only passes matching packets 4. `AggregatePcap` assigns packets to correct windows and fills gaps with empty snapshots 5. `OpenCapture` and `StartCapture` accept filter parameter 6. `ClassifiedPacket` has `Timestamp time.Time` field 7. `go test ./...` all green, `go build ./...` succeeds After completion, create `.planning/phases/04-power-user-features/04-01-SUMMARY.md`