Tianyi Cui
4dafad4db6
Eight proposals grouped by category, each with problem statement, concrete plan, and risks: property-based testing over the protocol-shaped core (chunk streams, event logs, schema DSL); mutation testing as the counterweight to the 100%-coverage gate; deterministic tests + a universal replay-invariant fixture + nightly race stress; architectural conformance (dependency-cruiser rules and the LlmAdapter conformance kit); runtime arg validation at the model boundary with a structured error taxonomy and dev-mode invariants; doc-sync enforcement (typechecked doc snippets, API reports); supply-chain checks and nightly vendor-drift verification against the manifest; and deep-readonly public surfaces (logged-vs-in-flight mutability boundary). AGENTS.md points at docs/adr and docs/rfc.
1.5 KiB
1.5 KiB
RFC 002: Mutation testing as the coverage counterweight
Status: proposed
Problem
The per-file 100% coverage gate (ADR 0007) proves every line executes under test — not that any assertion would notice if the line were wrong. Under agent-written tests, coverage pressure can produce execution-without-assertion. Mutation testing measures what coverage cannot: whether the suite kills deliberately injected bugs.
Proposal
Stryker (@stryker-mutator/vitest-runner) over packages/*/src:
- PR-scoped incremental runs (changed files only) as a CI job — fast enough to gate merges once tuned.
- Nightly full runs with a tracked mutation score; start by recording, then set the threshold at the observed baseline and ratchet upward (same policy as coverage: thresholds only ever tighten).
- Surviving mutants are work items: an agent picks a survivor, writes the killing test, repeats — a well-shaped autonomous loop.
- Equivalent mutants (provably behavior-preserving) get annotated exclusions
with reasons, mirroring the
/* v8 ignore */policy.
Plan
- Add Stryker config scoped to one package (llm — smallest, most algorithmic) and measure runtime.
- Expand to all packages; record baseline scores in the config.
- Wire the nightly job; add the incremental PR job once runtime is acceptable.
Risks
Runtime: mutation testing is expensive; per-file 100% coverage helps (every mutant is at least reached). If PR-scoped runs stay too slow, keep them nightly-only and rely on the score ratchet.