0 Fork 0

Files

T

Tianyi Cui 2f6d3b8539 test: property-based tests for protocol-shaped code (RFC 001)

Adds fast-check + one tests/properties.spec.ts per protocol-shaped package
(llm/BlockAssembler, session, tools/schema DSL, agent-loop scheduling). The
tools suite includes the RFC 001<->005 composition property (generated args
satisfying a spec pass validateArgs), closing the validator/InferArgs drift
risk from ADR 0011. Loop properties are deterministic (settle on agent/status,
no sleeps).

The BlockAssembler suite found a real bug on first run: a duplicate block-end
at the same index overwrote an already-flushed block, so the streamed prefix
disagreed with final blocks(). Fixed (first close wins, matching the existing
straggler rule) + regression test. Graduates RFC 001 -> ADR 0013.

2026-06-14 00:06:25 +08:00

2.2 KiB

Raw Blame History

RFC 001: Property-based testing for protocol-shaped code

Status: implemented — see ADR 0013. (It found a real BlockAssembler duplicate-block-end bug on first run.)

Problem

Example-based tests pin the cases we thought of. The harness's core is protocol-shaped — chunk streams, event logs, schema conversion — where the input space is combinatorial and the interesting bugs live in interleavings nobody wrote an example for (the streamBlocks ordering bug survived 100% line coverage of the happy paths).

Proposal

Adopt fast-check (vitest integration) with generators for our vocabulary:

BlockAssembler: arbitrary chunk sequences (valid and malformed — duplicate indices, stragglers after block-end, missing block-start). Invariants: flushReady() + flushRemaining() ≡ blocks() in order; streamBlocks ≡ generate().message.content; memory bounded (partials map size ≤ distinct indices); idempotent re-assembly.
Session: arbitrary event logs (seeded generators over SessionEventMap). Invariants: deriveMessages deterministic; replay-from-seed produces identical derivation; seq strictly monotonic; derived history unaffected by non-message events.
Schema DSL: arbitrary SchemaSpecs. Invariants: generated JSON Schema's required array equals the required: true keys at every nesting level; conversion is total (never throws); generated args satisfying InferArgs validate against the generated schema (once RFC 005's validator exists — the two RFCs compose).
Inbox/loop: arbitrary send/steer/abort schedules against a scripted adapter. Invariants: no message lost (every send/steer appears in the log exactly once), turn numbers strictly increase, status transitions follow idle→running→idle/disposed.

Plan

One tests/properties.spec.ts per package; fast-check as devDependency; numRuns tuned so the suite stays under ~10s locally, with a nightly CI job running 100× the iterations. Failures persist their seed in the report so agents can reproduce deterministically.

Risks

Generator quality determines value — invest in generators that produce realistic-but-adversarial streams, not uniform noise. Property flake from timeouts must be treated as a finding, not retried away.

2.2 KiB Raw Blame History Unescape Escape