deepseek-harness/docs/rfc/implemented/testing/2026-07-04-hook-snapshot-matrix.md

RFC: Hook snapshot matrix — end-to-end goldens for both bridges

Status: implemented

Problem

The hook bridges — dsh-hooks-claude (7 Claude Code hook points) and dsh-hooks-codex (5 Codex points) — map external hook commands onto the harness interception seams. They carry deep unit and coverage-spec coverage (every decision arm, every payload dialect, driven against a mocked seam) plus one key-gated e2e (hooks.e2e.ts, a live PreToolUse block). But the full-transcript snapshot tier — the one net that boots the real acp-agent subprocess, replays a recorded session keyless, and diffs the normalized ACP stdout + re-persisted log against committed goldens — covered exactly ONE hook: a Claude UserPromptSubmit block (hook-cc-promptsubmit-block).

That is the tier a mocked unit test structurally cannot be: it exercises the REAL bridge translating a REAL hook process's outcome into the REAL seam decision, then the REAL loop's reaction, rendered exactly as an editor sees it. A bridge-translation or loop-structure regression that left every unit green would still escape it for every hook point but one — and for the Codex bridge, the ACP example did not even LOAD it, so no Codex hook could fire end-to-end at all.

Decision

Two coupled changes, in one PR:

1. The ACP example ships BOTH hook bridges

examples/acp-agent/cordis.yml and cordis.snapshot.yml now load dsh-hooks-codex alongside dsh-hooks-claude, each pointed at its own config file (./hooks.json for Claude, ./codex-hooks.json for Codex — the two dialects cannot share one file). This is a genuine product-surface change, not test-only wiring: the shipped ACP server (and the demo:acp front door) now carries both bridges.

It is safe because a bridge whose config file is absent is a silent no-op: apply() catches the read failure, logs through ctx.logger, and registers nothing — zero listeners, zero session events. The acp-agent app ships no stdout logger, so the warning cannot reach the ACP JSON-RPC channel. A scenario (or a real project) that wants only Claude hooks ships only hooks.json; the Codex bridge sees no codex-hooks.json and vanishes. This was verified empirically: with both bridges loaded, all pre-existing snapshots (none of which ship a codex-hooks.json) are byte-identical.

Loading both is the minimum that lets the snapshot tier exercise each dialect against the same real app the product ships. Recording (which boots cordis.yml) loads both by construction, and replay inherits them the same way: cordis.snapshot.yml is an include-overlay of cordis.yml that swaps only the llm entry (see single-source the acp-agent replay config), so a bridge added to the live tree is in the replay tree with no second edit.

2. A snapshot scenario per hook point × its headline outcome, both dialects

Thirteen scenarios under examples/acp-agent/tests/snapshots/, naming hook-<dialect>-<point>-<outcome>:

• Authored, no model turn (keyless, no sidecar — the derived replay script is empty; the rejected turn carrying hook/* events is compared): hook-cc-promptsubmit-block, hook-codex-promptsubmit-block.
• Recorded against the real API, hook active during recording (the model's reaction to the decision is part of the captured transcript, replayed keyless thereafter): hook-{cc,codex}-promptsubmit-context (allow + additionalContext fold), hook-cc-pretool-deny / hook-codex-pretool-block (deny → isError tool result), hook-cc-pretool-ask (ask → degrades to deny with the approval-required reason), hook-{cc,codex}-posttool-block (block with feedback), hook-{cc,codex}-posttool-context (accept + additionalContext), hook-{cc,codex}-stop-continue (a blocking Stop hook forces one extra step via steering).

Each hook command emits only FIXED LITERAL strings (no timestamps/pids/$RANDOM/cwd echoes); the snapshot normalizer scrubs the one volatile field a hook/result carries (durationMs). The Stop scenarios self-limit with a marker file (.stop_fired) so the force-continue does not loop — the stop_hook_active loop-guard is still a bridge TODO, so an unconditional Stop hook would force-continue every step.

Three hook points are deliberately NOT snapshotted

Discovered while building the matrix, and documented here because the omission is a decision, not an oversight:

• SessionStart and SubagentStart inject context through a detached, best-effort void runPoint(...).then(agent.inject()) with NO turn binding. The resulting context/message races the work it precedes (the first model request / the child's first turn) and lands at a nondeterministic log position. A recorded golden does not even reproduce on its own replay — a 10× replay stability check failed 10/10 for both. They stay on the bridges' unit coverage, which drives the seam directly without the timing race. (If the injection is ever made turn-bound and deterministic — the direction the TODO(session-start-gating) points — these become snapshottable.)
• SubagentStop is observe-only: its subagent/end handler passes no turn (so no hook/* log events) and does no injection. It writes NOTHING to the transcript, so a golden would be byte-identical to the no-hook run and could never be proven to fail — a guard that cannot bite. It stays on unit coverage (bridge.spec.ts already asserts the observe-only call).

The matrix therefore covers every hook point that has a DETERMINISTIC, OBSERVABLE transcript footprint, for both dialects.

Consequences

• Every bridge seam mapping with an observable transcript is now guarded at the full-transcript tier, in the real app, for both dialects — including the Codex bridge, which had no end-to-end coverage at all. Recorded goldens capture the model's real reaction to a denied/blocked/force-continued turn, which a hand-authored transcript could only guess at.
• The block scenarios are keyless (no model turn); the rest replay keyless from recorded fixtures. pnpm run test:snapshot:record regenerates the recorded fixtures from the live API and self-skips without a key like every recorded scenario.
• The prove-red discipline holds: tampering a hook config's output (e.g. changing a deny reason) turns its scenario red on replay — the hook process runs FOR REAL during replay (only the model is replayed), so the golden guards the actual hook→seam→loop path, not a mock of it.
• The acp-agent demo now loads a Codex bridge it will usually no-op (no codex-hooks.json in a typical project), which is the intended fail-soft behavior, not a cost.