Tianyi Cui
4dafad4db6
Eight proposals grouped by category, each with problem statement, concrete plan, and risks: property-based testing over the protocol-shaped core (chunk streams, event logs, schema DSL); mutation testing as the counterweight to the 100%-coverage gate; deterministic tests + a universal replay-invariant fixture + nightly race stress; architectural conformance (dependency-cruiser rules and the LlmAdapter conformance kit); runtime arg validation at the model boundary with a structured error taxonomy and dev-mode invariants; doc-sync enforcement (typechecked doc snippets, API reports); supply-chain checks and nightly vendor-drift verification against the manifest; and deep-readonly public surfaces (logged-vs-in-flight mutability boundary). AGENTS.md points at docs/adr and docs/rfc.
1.7 KiB
1.7 KiB
RFC 003: Deterministic tests, the replay invariant fixture, and race stress
Status: proposed
Problem
Several loop tests synchronize with setTimeout(30) sleeps — flakiness debt
that wastes agent cycles on retries and can mask ordering bugs. Separately,
our core architectural promise (any session log replays to identical derived
history) is asserted in two tests but is cheap to assert everywhere. And
the inbox wakeup race was verified by hand exactly once; nothing re-verifies
it continuously.
Proposal
Three measures:
- No wall-clock sleeps in tests. Replace
setTimeout(N)waits with event-driven waits (the existingwaitForIdlepattern, extended towaitForStatus,waitForEvent(n)) or vitest fake timers where time itself is under test. Enforce with a lint rule banningsetTimeoutinpackages/*/testsoutside an allowlisted helper module. - Universal replay fixture. A shared test helper wraps the loop harness
so that after every test, the agent's session log is replayed into a fresh
Session and
deriveMessages()equality is asserted automatically. The invariant then gets checked hundreds of times per CI run across every scenario the suite produces, not twice. - Nightly race stress. A CI job running the agent-loop and inbox suites
with
vitest --repeat=200(and--shuffle) to flush scheduling-dependent failures; any flake found is a bug to fix, never a retry.
Plan
Land 1 and 2 together (they touch the same helpers); add the nightly job after the suite is sleep-free so repeats are fast.
Risks
Fake timers interact subtly with Promise scheduling in the loop — prefer event-driven waits; reserve fake timers for timer-service behavior itself.