Rename RFCs to Agent Notes
This commit is contained in:
452 files changed
+1443
-1414
No files matched your search
@@ -0,0 +1,27 @@
|
||||
# Agent Note: Property-based testing for protocol-shaped code
|
||||
|
||||
Status: implemented
|
||||
|
||||
> Merges the original proposal and the decision record for one topic. It found a real BlockAssembler duplicate-`block-end` bug on first run.
|
||||
|
||||
## Problem
|
||||
|
||||
Example-based tests pin the cases we thought of. The harness's core is protocol-shaped — chunk streams, event logs, schema conversion, inbox scheduling — where the input space is combinatorial and the interesting bugs live in interleavings nobody wrote an example for. The motivating evidence: a block-assembly ordering bug once survived 100% line coverage of the happy paths. Per-file 100% coverage proves every line ran, not that every interleaving is correct.
|
||||
|
||||
## Decision
|
||||
|
||||
Adopt `fast-check` (a root devDependency) with one `tests/properties.spec.ts` per protocol-shaped package, generators tuned for *realistic-but-adversarial* inputs (not uniform noise) and `numRuns` kept so the suite stays well under ~10s locally. Failures print a reproducible seed. (The original proposal also sketched a nightly CI job running 100× the iterations; that was not shipped — the property suite runs only in the normal `push`/`pull_request` CI, and a scheduled high-iteration job remains possible future work.)
|
||||
|
||||
- **dsh-llm / BlockAssembler:** arbitrary chunk streams (valid + malformed: duplicate indices, stragglers, missing block-start). Invariants: `blocks()` count ≤ distinct indices seen; re-assembly idempotent (`blocks()` is stable across repeated calls and `message().content` mirrors it); `blocks()` never throws and yields only valid content-block tags; `finish` reflects the last `finish` chunk, defaulting to `{kind:'stop'}` when none arrives.
|
||||
- **dsh-session:** arbitrary event logs. Invariants: `deriveMessages` deterministic; replay-from-seed identical; seq strictly monotonic; non-message events never affect derived history; derived content is decoupled from the log.
|
||||
- **dsh-tools:** arbitrary `SchemaSpec`. Invariants: JSON Schema `required` equals the `required:true` keys at every level; conversion total; **and the composition with [runtime arg validation](../architecture/2026-06-11-runtime-arg-validation.md)** — generated args satisfying a spec pass `validateArgs`, and targeted corruptions (dropped required key, non-object top level) are rejected. This closes the validator/`InferArgs` drift risk.
|
||||
- **dsh-agent-loop:** arbitrary send schedules against a never-exhausting adapter, driven through the `agent/status` settle signal (no wall-clock sleeps). Invariants: no message lost; turn numbers strictly increase; status transitions stay on the legal machine.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Generator quality is the value lever — the generators bias toward small index pools and short strings so collisions and interleavings are common.
|
||||
- **It already paid off:** the BlockAssembler stream found a real bug — a duplicate `block-end` at the same index rewrote a completed block. Fixed (first close wins, matching the existing straggler rule) with a dedicated regression test.
|
||||
- A property flake from a timeout is a finding, not something to retry away. The loop properties are deterministic by construction (settle on `agent/status`), so a hang is a real defect.
|
||||
- Property tests supplement, not replace, the example tests that pin specific branches for the 100%-coverage gate.
|
||||
|
||||
<!-- agent-note-format: alternatives-not-recorded (pre-format Agent Note) -->
|
||||
@@ -0,0 +1,80 @@
|
||||
# Agent Note: ACP snapshot tests — record-once / replay-deterministic
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
Unit tests do not exercise the complete ACP subprocess transcript, while real-API tests are nondeterministic and key-gated. Editor-facing `session/update` output can therefore regress despite green unit coverage, as the [default-export postmortem](../../../../docs/postmortem/0001-acp-default-export-drops-inject.md) demonstrated.
|
||||
|
||||
The blocker for a full-transcript test is the model: the agent's output is driven by a non-deterministic LLM, and a key-gated test that hits the real API on every run is neither deterministic nor CI-runnable. We want the fidelity of a real run with the determinism of a fixture.
|
||||
|
||||
This Agent Note records the decision to add a third test tier — **snapshot tests** — and the design choices that make it deterministic, keyless-in-CI, and cheap to maintain.
|
||||
|
||||
## Decision
|
||||
|
||||
A snapshot test boots the real ACP example, drives its stdio protocol from a deterministic script, and compares normalized output with committed expected outputs. A session log recorded once from the real API supplies all later model streams. The fixture is the product's ordinary persisted JSONL.
|
||||
|
||||
### The fixture is the persisted session JSONL
|
||||
|
||||
Each scenario's `session.jsonl` is harvested from a real run. `assistant/chunk` events reproduce the model streams; tool, message, and boundary events capture the harness behavior. One ordinary session artifact therefore serves as both replay source and behavioral expected output.
|
||||
|
||||
### Replay derives the model script from the log
|
||||
|
||||
`llm-replay` short-circuits the provider-agnostic `llm/stream` waterfall. `deriveReplayScript()` groups recorded chunks by `(turn, step)` and serves one group per model call. The loop makes one stream call per step, so the grouping is exact and includes error finish chunks without special handling.
|
||||
|
||||
### The in-memory replay entry honors the full LLM contract
|
||||
|
||||
`deriveReplayScript` produces a list of `ReplayEntry`, the in-memory unit the replay listener serves positionally:
|
||||
|
||||
```
|
||||
{ kind: 'chunks', chunks: StreamChunk[] }
|
||||
| { kind: 'throw', chunks: StreamChunk[], message: string, code: string }
|
||||
| { kind: 'hang' }
|
||||
```
|
||||
|
||||
Logs derive chunk entries. Pre-stream throws and hangs have no reconstructable chunk representation, so those scenarios provide `replay.override.json`. A throw entry may include prefix chunks for mid-stream failure. Explicit overrides avoid inferring adapter behavior from lossy turn-end reasons.
|
||||
|
||||
### Positional replay, one in-flight stream
|
||||
|
||||
Replay is positional and therefore permits only one in-flight model stream per scenario. Concurrent-session snapshots require request-keyed entries. Changed call order requires re-recording, and missing or exhausted fixtures fail loudly.
|
||||
|
||||
### Recording harvests the log; keyless replay needs a providerless config
|
||||
|
||||
Recording runs the scenario with the real `llm-deepseek` adapter and the JSONL persistence backend, then copies the produced `.jsonl` into the scenario dir. Per-event appends are durable, but the harness shuts the subprocess down gracefully (close stdin → `await ctx.dispose()`) before harvesting so the final events are flushed. `llm-replay` itself does no recording — it is replay-only.
|
||||
|
||||
Replay uses a `cordis.snapshot.yml` overlay that replaces the real adapter with `llm-replay` while retaining the live composition. Recording uses the ordinary config and a harness-supplied persistence root. Replay mode skips `.env` loading, so a stray API key cannot trigger a live call. See the [single-source config Agent Note](2026-07-04-single-source-acp-replay-config.md).
|
||||
|
||||
### Two surfaces: normalize, then compare
|
||||
|
||||
A snapshot run asserts **two** normalized surfaces, because the harness's external surfaces are distinct:
|
||||
|
||||
1. The **stdout transcript** — the framed `session/update` JSON-RPC the editor sees. Catches regressions in the ACP bridge's event→update translation (`streamSessionEventUpdate`). Compared against a committed `stdout.expected.jsonl`.
|
||||
2. The **re-persisted session JSONL**, normalized and compared with `session.jsonl`. The same fixture is both replay source and expected log. Prompt text is scrubbed; one scenario per header class pins readable prompt and tool content as described in the [header-pinning Agent Note](2026-07-06-pin-request-header-content-in-one-scenario.md). Override scenarios derive model behavior solely from their sidecar.
|
||||
|
||||
The surfaces are complementary: stdout covers bridge projection, while JSONL covers loop, tool, and boundary structure that the projection omits.
|
||||
|
||||
Normalization replaces session, cwd, protocol-id, timestamp, path, and process volatility while preserving deterministic sequence numbers. Scenarios constrain real bash use to stable commands. The stdout expected output remains wire-shaped JSONL and every raw line must parse as JSON. Vitest updates only the stdout expected output; normalized session equality never overwrites the replay fixture.
|
||||
|
||||
### Isolation: normalization now, sandbox later
|
||||
|
||||
Tool determinism comes from a temporary cwd, scrubbed environment, fresh non-login shell, constrained commands, and normalization. It does not claim OS confinement. A sandboxed executor can replace the local backend through the existing [capability seam](../architecture/2026-06-13-capability-seams.md) if a stronger tier is needed.
|
||||
|
||||
### The replay plugin is its own package
|
||||
|
||||
`@deepseek-ai/dsh-llm-replay` is a support package rather than example-local glue. It replaces the real adapter by short-circuiting `llm/stream` with streams reconstructed from JSONL, and its package placement keeps the replay logic under normal coverage gates.
|
||||
|
||||
### Two subcommands, replay in the default gate
|
||||
|
||||
`pnpm run test:snapshot` replays committed fixtures keylessly; `test:snapshot:record` uses the real API and rewrites the harvested session log and stdout expected output. Missing fixtures fail loud. Every scenario carries `input.json`, `stdout.expected.jsonl`, and `session.jsonl`; no-model cases use a header-only log. `replay.override.json` is required only for scenarios marked `overridden`, because its presence replaces derived replay. Fixture guards reject missing, mismatched, and orphaned files. Both commands accept scenario filters.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **A hand-authored `llm.json` of model chunks** — the earlier draft; reusing the real session log makes the fixture a genuine product of the system rather than a hand-built mock, and doubles it as a behavioral expected output.
|
||||
- **A byte-level HTTP-record library (Polly/nock/MSW)** — rejected: adapter-specific, awkward with streaming SSE, and lower-level than the thing under test.
|
||||
- **Synthesizing throw/cancel entries from `turn/end {kind:'error'|'aborted'}`** — rejected: it couples `llm-replay` to loop-internal turn-closing semantics, and the `turn/end` reason is lossy (it cannot distinguish a thrown 401 from a finish-error); the explicit `replay.override.json` sidecar is the cleaner seam.
|
||||
|
||||
## Consequences
|
||||
|
||||
The new tier adds reviewed per-scenario input, session, stdout, optional override, and optional workspace fixtures. Workspace seeds are copied into the temporary cwd for both record and replay. In return the tier provides deterministic keyless transcript coverage through the real Loader and tool composition. The subprocess, input, workspace, normalization, and replay harness can support examples beyond ACP.
|
||||
|
||||
This Agent Note relates to but does not supersede the [proposed determinism Agent Note](../../proposed/testing/2026-06-11-deterministic-and-stress-testing.md): that proposal's "universal replay fixture" re-derives session *message history* after every test (an internal-consistency invariant), whereas snapshot tests pin the *external protocol output*. They are complementary — one guards the event-sourcing invariant, the other guards the editor-facing contract.
|
||||
@@ -0,0 +1,98 @@
|
||||
# Agent Note: Real-API e2e in CI against the external DeepSeek API
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
The harness leans hard on real-API tests by policy: [docs/testing.md](../../../../docs/testing.md) argues that a no-key suite proves the plumbing but not the product, and the [ACP inject postmortem](../../../../docs/postmortem/0001-acp-default-export-drops-inject.md) is the standing proof — 178 keyless tests stayed green while a real editor session crashed instantly. The real-API e2e suite (`pnpm run test:e2e`, the `*.e2e.ts` files) exists precisely to close that gap: it drives the agent against the live DeepSeek API — real model calls, real bash tools, multi-turn, resume, ACP-over-stdio.
|
||||
|
||||
The default gate ([.github/workflows/ci.yml](../../../../.github/workflows/ci.yml)) is deliberately keyless: it carries no secret and runs for forks. `test:e2e` self-skips without a key (`describe.skipIf(!process.env.DEEPSEEK_API_KEY)`), so adding it there would report green without exercising the real suite. A separate secret-bearing workflow is required to make real-API coverage a merge signal.
|
||||
|
||||
This Agent Note records the decision to add a **second, secret-consuming workflow** that runs the real-API suite in CI, and — because introducing the first CI secret into a repo that may later go public is a security/isolation decision — the threat model it relies on and what changes when the repo becomes public.
|
||||
|
||||
## Decision
|
||||
|
||||
Add a dedicated workflow, [.github/workflows/e2e.yml](../../../../.github/workflows/e2e.yml), separate from ci.yml. It runs only `pnpm run test:e2e` against the external API using a repo secret, on trusted events, with a preflight that converts a missing secret into a loud failure instead of a false green. The keyless workflow remains separate so forkable quality gates and secret-consuming real-API gates keep different trigger and credential policies.
|
||||
|
||||
### A separate workflow, not a job in ci.yml
|
||||
|
||||
ci.yml's value is that it is keyless, forkable, and always-green: any contributor (including an outside fork) gets a complete keyless signal with no secret in the blast radius. Adding a secret-consuming job there would couple that always-green gate to credential availability and a different trigger policy. Keeping the secret-bearing work in its own file isolates the secret, trigger, and concurrency policy, and preserves ci.yml's property for forks. Different lifecycles → different files.
|
||||
|
||||
### Cost is not the constraint; reliability is
|
||||
|
||||
Internal inference cost is not the limiting constraint, so the workflow optimizes for coverage and signal. It runs every matching `*.e2e.ts` file on multiple triggers and every trusted PR, implementing the [docs/testing.md](../../../../docs/testing.md) with-key policy.
|
||||
|
||||
### Triggers: trusted events only
|
||||
|
||||
`workflow_dispatch` + `push` to `main`/`master` + nightly `schedule` (`17 0 * * *`, 08:17 Asia/Shanghai) + `pull_request`. Push gives a post-merge signal; schedule catches external-API drift; dispatch is the manual escape hatch; and trusted pull requests get a pre-merge gate. That pre-merge signal deliberately accepts the larger key-exposure surface described under § Security.
|
||||
|
||||
### The untrusted-PR gate
|
||||
|
||||
GitHub withholds repo secrets from two kinds of PR: those from **forks**, and **Dependabot** PRs (same-repo branch, so `head.repo.fork == false`, but secrets are still withheld). A job-level `if:` skips the whole job for both:
|
||||
|
||||
```
|
||||
github.event_name != 'pull_request'
|
||||
|| !(github.event.pull_request.head.repo.fork || github.event.pull_request.user.login == 'dependabot[bot]')
|
||||
```
|
||||
|
||||
The Dependabot clause keys on the PR **author** (`pull_request.user.login`), not `github.actor` (the run trigger): a maintainer who reopens or re-runs a Dependabot PR would make `github.actor` a human while the PR is still keyless, and an author-based test stays correct across that. A job skipped by a **job-level** `if:` reports as a *successful* check (unlike a workflow/trigger-level skip, which stays pending), so this workflow is safe to mark as a required status check if desired — a fork/Dependabot PR's skipped-but-green check does not block the merge.
|
||||
|
||||
The gate is a *clean-skip nicety*, not the secret's security boundary (see § Security — the boundary is GitHub's own fork-secret withholding under `pull_request`). Without the gate, forks still could not read the key; they would just hit a confusing preflight hard-fail and waste compute.
|
||||
|
||||
### Preflight: fail loud, never false-green
|
||||
|
||||
Because the job only runs on trusted events where the secret is expected, the preflight is an unconditional presence check: empty key → `exit 1` with a `::error::` annotation naming the secret to configure. This is the crux that makes a self-skipping suite safe to gate on. Without it, a deleted/renamed/misconfigured secret would make `test:e2e` skip every real suite and report all-green — a silent regression of the entire safety net. The guard turns "secret missing" from an invisible false pass into a visible failure. (Its correctness was verified live: the run before the secret existed failed at exactly this step.)
|
||||
|
||||
### Secret mapping and hygiene
|
||||
|
||||
The repo secret is named `DEEPSEEK_API_KEY_EXTERNAL`; it is mapped to the `DEEPSEEK_API_KEY` env var the adapters and tests read (`process.env.DEEPSEEK_API_KEY`). The distinct secret name documents intent (this is the *external* public-API key, not an internal-endpoint key) and lets an internal-endpoint key coexist later without collision. Hygiene choices, each defensive:
|
||||
|
||||
- **Step-scoped secret.** `DEEPSEEK_API_KEY` is set in the `env:` of only the preflight and e2e steps, never job-level — so checkout/setup-node/install never see it. A compromised install-time lifecycle script in a dependency cannot read a secret that isn't in its environment.
|
||||
- **`permissions: contents: read`.** The job only reads the repo to run tests; it needs no write scopes (no PR comments, no status writes), so the `GITHUB_TOKEN` is dropped to least privilege.
|
||||
- **`DEEPSEEK_BASE_URL` pinned** to `https://api.deepseek.com` on the e2e step. The adapter would default to this when unset ([packages/llm/llm-deepseek/src/index.ts](../../../../packages/llm/llm-deepseek/src/index.ts) `PUBLIC_BASE_URL`), but pinning is self-documenting and hermetic — a stray repo-root `.env` (which `vitest.e2e.config.ts` loads if present) cannot silently redirect the run to another endpoint.
|
||||
- **No secret echoed.** The preflight prints only `DEEPSEEK_API_KEY present.` — not the value or its length.
|
||||
|
||||
### Scope, runtime shape
|
||||
|
||||
The job runs only `test:e2e` on Node 24; keyless gates and version compatibility belong to the main CI workflow. Tests run unbuilt through the workspace paths map with a bounded configurable worker pool, per-test retries, and a job timeout. Superseded PR runs are cancelled, while push and scheduled runs complete for post-merge signal.
|
||||
|
||||
## Security
|
||||
|
||||
The repository's first CI secret requires a recorded threat model because access differs between same-repository, fork, and Dependabot pull requests and changes when the repository becomes public.
|
||||
|
||||
### Who can reach the secret today (private repo)
|
||||
|
||||
- **No write access (fork PRs): cannot.** Two independent facts block it. First, the workflow uses `pull_request`, **not** `pull_request_target` — GitHub does not pass repo secrets to fork-PR runs of `pull_request`, so `secrets.DEEPSEEK_API_KEY_EXTERNAL` resolves to empty on a fork runner. Second, the `if:` gate skips fork PRs entirely. The withholding is the real boundary; the gate is defense-in-depth and UX.
|
||||
- **Write (push) access: can.** A same-repo branch PR receives secrets, so a write-access author could modify test code (or an install lifecycle script, or the workflow YAML on their branch) to exfiltrate the key. This is **inherent to GitHub Actions, not introduced here**: anyone with push access to any repo can already exfiltrate any of its Actions secrets by authoring a workflow. Write access ⇒ secret access, always. The mitigation lives in who is granted write and in branch protection, not in this file.
|
||||
|
||||
So "everyone who could open a PR can steal it" is false: only the write-access set can, and that set could already steal any secret the repo holds.
|
||||
|
||||
### The residual exposure the `pull_request` trigger adds
|
||||
|
||||
Because PR runs are enabled, the key is handed to **the code on a write-access author's PR branch** before merge. This is a larger surface than `push` + `schedule` + `workflow_dispatch`, accepted for a pre-merge signal within the trusted write set. If that calculus changes, drop the `pull_request` trigger while retaining post-merge, nightly, and on-demand coverage.
|
||||
|
||||
### What changes when the repo goes public
|
||||
|
||||
The secret stays protected from the public **through this workflow**: `pull_request` behaves identically on a public repo — fork PRs (now openable by anyone) still receive no secret, and on public repos GitHub additionally gates fork-PR runs behind maintainer approval, where even an approved run gets no secret (approving the run is not the same as handing over the key). The write-access set is unchanged by visibility, so the insider reality is also unchanged.
|
||||
|
||||
What gets worse is the *surrounding* model, and these are the things to address before flipping visibility:
|
||||
|
||||
- **Logs become world-readable.** A careless secret echo that today leaks to org members would leak to the entire internet and be scraped within minutes. Secret-handling discipline (no value/length echoes — already done) matters far more.
|
||||
- **The `pull_request_target` footgun becomes catastrophic.** If anyone ever "fixes" PR runs by switching the trigger to `pull_request_target`, the workflow would run untrusted fork code in the base-repo context **with** secrets — a full key-leak vector. This is benign-ish on a private repo and disastrous on a public one. A `SECURITY —` comment on the trigger in e2e.yml forbids the change and points here.
|
||||
- **Rotate on flip.** The key lived in a private repo's CI; treat going-public as "assume exposed" and rotate `DEEPSEEK_API_KEY_EXTERNAL` at that moment.
|
||||
- **Settle the secret behind controls.** Confirm Settings → Actions → *"Send secrets to workflows from fork pull requests"* stays **off** (the one setting that would actually break the fork boundary), and consider moving the key into a GitHub **Environment** with required reviewers so even merged code uses it only under controlled conditions and rotation has a single home.
|
||||
|
||||
None of these require changing the workflow to go public; they are operational steps plus the already-added `pull_request_target` guard comment.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **A secret-consuming job inside ci.yml** — rejected: it would couple the keyless, forkable, always-green gate to credential availability and a different trigger/concurrency policy; different lifecycles, different files.
|
||||
- **Omitting the `pull_request` trigger** (the smaller key-exposure surface) — rejected for the pre-merge signal; the Security section carries the accepted exposure analysis.
|
||||
|
||||
## Consequences
|
||||
|
||||
A second CI workflow and the first repo secret to maintain. The real-API suite now gates merges (pre-merge on trusted PRs, post-merge on the main branch) and runs nightly, so a real break in the agent's interaction with the external API surfaces in CI rather than only in a developer's local run — at the cost of real (but internally free) API calls on every trusted PR and merge. The preflight makes secret misconfiguration self-announcing instead of silently disabling the net.
|
||||
|
||||
The design carries a documented constraint surface: the `pull_request` trigger's key-exposure tradeoff (drop it to harden), the `if:` gate's dependence on the author-based Dependabot test, and the hard prohibition on `pull_request_target`. The going-public checklist above is the operational companion — this Agent Note is the place a future maintainer should re-read before changing the trigger set or flipping repo visibility, rather than re-deriving the fork/secret model from scratch.
|
||||
|
||||
The scheduled trigger auto-disables after 60 days of repo inactivity (a GitHub behavior); push/PR/dispatch are backstops, and an active monorepo will not hit it. Runner egress to `https://api.deepseek.com` is assumed — GitHub-hosted `ubuntu-latest` has it; an egress-restricted self-hosted runner would need connectivity confirmed before relying on the nightly.
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
# Agent Note: Use `session.jsonl` as the only snapshot session-log artifact
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
Model-driving ACP snapshot scenarios ship both `session.jsonl` and `session.expected.jsonl`. For normal recorded scenarios, `session.jsonl` is the replay fixture harvested from a real run, and the replay test normalizes the newly persisted log and compares it to `session.expected.jsonl`. In the current fixtures, the two normalized logs are identical for ordinary recorded scenarios.
|
||||
|
||||
Authored override scenarios (`error-finish`, `cancel`) currently use `replay.override.json` to drive model behavior and keep `session.jsonl` as a minimal dummy fixture, while `session.expected.jsonl` holds the expected persisted log. The override file is a JSON array of `ReplayEntry` objects: `{ "kind": "chunks", "chunks": StreamChunk[] }`, `{ "kind": "throw", "chunks": StreamChunk[], "message": string, "code": string }`, or `{ "kind": "hang" }`. That split is also unnecessary: when an override sidecar exists, `llm-replay` replaces the derived script and does not need `session.jsonl` for model chunks, so `session.jsonl` can still be the expected session-log artifact for the scenario.
|
||||
|
||||
## Decision
|
||||
|
||||
The `session.expected.jsonl` concept is removed entirely. Every scenario has at most one committed session-log artifact, `session.jsonl`:
|
||||
|
||||
- For recorded scenarios, `session.jsonl` remains the raw harvested log. Replay still derives model chunks from it, and the snapshot test compares the replay run's normalized persisted log against normalized `session.jsonl`.
|
||||
- For authored override scenarios, `replay.override.json` drives model behavior and `session.jsonl` holds the expected produced session log. The replay adapter ignores the fixture for model chunks when the override exists, so the same file can be the expected log without affecting replay behavior.
|
||||
- For no-model scenarios, `session.jsonl` can stay as the minimal fixture needed to boot `llm-replay`; no session-log comparison is needed unless the scenario creates a persisted session.
|
||||
|
||||
Stdout expected outputs remain unchanged; they are the editor-facing projection and are not redundant with the session fixture.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Normalizing both sides against a shared (replay-run) context** — rejected: `normalizeSessionLog` scrubs cwd by exact string match, so the fixture's recorded cwd would survive unscrubbed and every compare would fail. Each side normalizes against its own header-derived context — the implementation note below carries the mechanics.
|
||||
|
||||
## Verification
|
||||
|
||||
`session.expected.jsonl` appears nowhere in the snapshot harness, fixtures, orphan guards, or docs; the snapshot test derives the expected session log from `session.jsonl` for every model scenario; authored sidecar scenarios commit their expected produced log as `session.jsonl` with `replay.override.json` as the model-behavior override; and the orphan-fixture guards know which files each scenario kind requires. The [ACP snapshot tests Agent Note](2026-06-19-acp-snapshot-tests.md) describes the reduced fixture set.
|
||||
|
||||
## Consequences
|
||||
|
||||
Reviewers lose one artifact name that made the expected persisted log visually separate from the replay fixture. The stdout expected output still protects the editor transcript, and comparing replay output to `session.jsonl` preserves the loop/persistence regression check without duplicating files.
|
||||
|
||||
## Implementation note
|
||||
|
||||
Each side is normalized against its own header values because recording and replay have different ids, paths, and timestamps. `fixtureContext()` derives the fixture context from its header, making already-normalized fixtures idempotent. Session logs use plain equality rather than file-snapshot updates, so comparison never rewrites fixtures.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Agent Note: Persist the seed boundary so fork-child replay routes correctly
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
The [per-session snapshot replay Agent Note](2026-06-22-subagent-snapshot-replay.md) made the snapshot tier express a nested-agent shape: a parent plus one recorded log per in-process subagent, each replayed as its own script keyed by calling session. It noted (§ Scope, final bullet) that a fork snapshot was "a trivial future addition, not a gap in the keying." That was wrong about a fork child specifically — not the keying, but the *script derivation*.
|
||||
|
||||
A subagent script is derived from a recorded session log by [`deriveReplayScript`](../../../../packages/support/llm-replay): it groups the log's `assistant/chunk` events by `(turn, step)` into one replay entry per `stream()` call. This is correct for a **spawn** child, whose log contains only its own model calls.
|
||||
|
||||
A **fork** child is different. The fork backend seeds the child session with a *balanced completed-turn prefix of the parent's log* ([`dsh-subagent-inprocess`](../../../../packages/subagent/subagent-inprocess)), and that seed becomes the child session's persisted `log` (`Session`'s constructor copies the seed into `this.log`). So a fork child's `.jsonl` begins with the **parent's** events — including the parent's `assistant/chunk` events — and only then carries the child's own turn.
|
||||
|
||||
Deriving the child script from the whole fork-child log therefore replays the **parent's** recorded responses as the **child's** model calls: the live fork child's first `stream()` would receive the parent's first recorded chunk sequence instead of its own. The recorded scenarios are all spawn today, so this never fired — but a fork snapshot would have mis-routed silently, exactly the class of bug the snapshot tier exists to catch.
|
||||
|
||||
## Decision
|
||||
|
||||
Record where a session's **inherited** prefix ends, persist it, and have the replay harness derive a child's script from its **own** events only.
|
||||
|
||||
### 1. `seedLength` on the session header
|
||||
|
||||
`SessionHeader` gains an optional `seedLength: number` — how many leading events were inherited via a seed rather than produced by this session. The fork backend stamps it (= the seeded-prefix length) when it creates the child; a fresh spawn leaves it absent (≡ 0). It is threaded through `CreateSessionOptions.meta` (and `CreateAgentOptions.meta`), set in `SessionStore.prepare`.
|
||||
|
||||
`seedLength` is **explicit**, never inferred from `seed.length`. A reconstruction (resume/load) seeds the session with its WHOLE stored log, so `seed.length` there is the full length, not the original boundary — the resume path passes the persisted `seedLength` back from the loaded header instead. (Same shape as `createdAt`, which is also explicitly preserved on reconstruction rather than re-defaulted to now.)
|
||||
|
||||
### 2. Both persistence backends round-trip it
|
||||
|
||||
- **JSONL**: a `seedLength` field on the header line (`toHeaderLine`/`fromHeaderLine`).
|
||||
- **SQLite**: a `seed_length` column on the `sessions` table.
|
||||
|
||||
The SQLite layout containing `seed_length`, `source_event_seqs`, and `surface_op` is schema version 4. Earlier version 3 layouts were ambiguous, so every non-current `user_version` is rejected without migration under the pre-release policy.
|
||||
|
||||
### 3. Replay derives a child script after the boundary
|
||||
|
||||
`dsh-llm-replay`'s `parseSessionHeader` now also reads `seedLength` (absent ⇒ 0), and `loadSessionScripts` derives a child's entries from `parseSessionLog(text).slice(seedLength)` — the events at or after the boundary, i.e. the child's own model calls. For a spawn child `seedLength` is 0 and this is a no-op, so spawn scenarios are byte-for-byte unchanged.
|
||||
|
||||
This closes the routing correctness gap, and two recorded fork scenarios exercise it end to end — see [Record fork and mixed spawn+fork snapshot scenarios](2026-06-22-fork-snapshot-scenarios.md).
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Derive the boundary heuristically in `llm-replay`** (the seeded prefix is contiguous parent events ending at the last `turn/end` before the child's first `user/message`). Rejected: a brittle heuristic in the test harness that re-derives a fact the producer already knows. Persisting the boundary at its source (the fork backend) is the "explicit > implicit at package seams" rule applied across the persistence boundary — the reader of a child fixture never has to reconstruct where the inheritance ended.
|
||||
- **Pin the format version instead of bumping** (the `SESSION_FORMAT_VERSION = 0` "unstable" stance the event log uses). Rejected for the SQLite *table* layout: `SCHEMA_VERSION` is the monotonic bump-and-reject knob (a small enumerable set of revisions worth telling apart), distinct from the event-vocabulary `version`. Adding a column is precisely the breaking table change it versions, so it bumps.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A new persisted header field across core + both backends; the core-data-structures catalog (`persistence.md`) is updated in the same change (its `SessionHeader` / `CreateSessionOptions` `type-equiv` blocks).
|
||||
- Existing SQLite databases at schema v2 are rejected on open (no user data pre-release).
|
||||
- Spawn replay is unchanged (`seedLength` 0). Fork replay now routes a child to its own script; covered by a regression in `llm-replay`'s tests (a child fixture whose seeded prefix carries a parent chunk — the derived child script must exclude it, proven red without the slice) and a persistence round-trip test (both backends, via the shared coordinator contract).
|
||||
@@ -0,0 +1,29 @@
|
||||
# Agent Note: Record fork and mixed spawn+fork snapshot scenarios
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
The [seed-boundary Agent Note](2026-06-22-fork-child-replay-seed-boundary.md) made fork-child replay route correctly: `dsh-llm-replay` derives a child's script from the events at or after its persisted `seedLength` boundary, so a fork child's inherited parent prefix is not replayed as the child's own model calls. But it shipped with **no recorded fork scenario** — the slice was exercised only by `llm-replay`'s unit tests (a synthetic child fixture) and a persistence round-trip test. The full-transcript snapshot tier, the one net that boots the real `acp-agent` and replays an end-to-end nested transcript, had only spawn children (`subagent-spawn`, `subagent-multi`). A fork-routing regression that left the unit tests green would still have escaped the tier built to catch transcript regressions.
|
||||
|
||||
The snapshot infrastructure to express a fork scenario was already in place — both in-process backends are wired into `cordis.yml` / `cordis.snapshot.yml` as two model-facing tools (`subagent` → spawn, `subagent_fork` → fork), the harness harvests every child log, and replay forwards per-child fixtures keyed by `seedLength`. What was missing was a *recorded scenario* that drives a fork child through it.
|
||||
|
||||
## Decision
|
||||
|
||||
Record two scenarios against the real API, both replayed keyless in the default gate:
|
||||
|
||||
- **`subagent-fork`** — the parent completes a turn that establishes a fact, then delegates one subtask via `subagent_fork`. The fork child inherits the conversation (its log carries a non-zero `seedLength`), so it can answer from the parent's context. This is the focused regression: the child fixture's `seedLength` is the boundary the replay slice depends on, recorded from a real fork rather than hand-synthesized.
|
||||
- **`subagent-mixed`** — the parent completes a turn, then delegates once via `subagent` (a fresh spawn child, `seedLength` 0) and once via `subagent_fork` (a fork child, non-zero `seedLength`) in one transcript. This is the mixed spawn+fork scenario the seed-boundary and per-session-replay Agent Notes both named as a future addition: one transcript exercises both transports and both branches of the slice (`seedLength` 0 = no-op, `seedLength > 0` = trim the inherited prefix), with the two children ordered spawn-then-fork by `createdAt`.
|
||||
|
||||
### Why a completed turn-1 is required
|
||||
|
||||
The fork backend seeds the child with the parent's **balanced completed-turn prefix**. A parent that forks on its very first turn has no completed turn to inherit, so the seed is empty (≡ a fresh spawn, `seedLength` 0) — which would NOT exercise the slice. Both scenarios therefore use a two-prompt input: the first prompt completes a turn (establishing a codeword the child is later asked to recall), the second delegates the fork. The recalled codeword in the child's transcript is incidental to the model's behavior; the load-bearing artifact is the child fixture's recorded `seedLength`, which the replay slice consumes.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The fork-routing slice is now guarded at the full-transcript tier, not just by unit tests. Removing the `slice(seedLength)` (replaying the whole child log) turns **both** new scenarios red — the fork child receives the parent's recorded chunks instead of its own — proving the guard bites (verified red→green when the scenarios landed).
|
||||
- `subagent-mixed` is the first snapshot scenario to drive two *different* subagent backends in one transcript, exercising the per-session replay keying across a spawn and a fork child simultaneously.
|
||||
- Out-of-process (ACP) subagent replay remains a different shape (each child is its own process with its own replay) and is still tracked as `TODO(acp-subagent-replay)` — these scenarios are in-process only.
|
||||
- Re-recording (`pnpm run test:snapshot:record`) regenerates all four fork/spawn fixtures from the live API; the two new scenarios self-skip without a key like every recorded scenario.
|
||||
|
||||
<!-- agent-note-format: alternatives-not-recorded (pre-format Agent Note) -->
|
||||
@@ -0,0 +1,56 @@
|
||||
# Agent Note: Per-session snapshot replay for nested agents
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
The snapshot tier (`pnpm run test:snapshot`) boots the real `acp-agent` subprocess, replays a recorded session through [`dsh-llm-replay`](../../../../packages/support/llm-replay), and diffs the normalized stdout transcript + re-persisted session log against committed expected outputs. It is the only tier that exercises the full editor-facing transcript end to end.
|
||||
|
||||
It was built for ONE session per process, and that assumption is wired into two places:
|
||||
|
||||
- **`dsh-llm-replay` keyed nothing.** It served the Nth `llm/stream` call the Nth recorded entry from a single global cursor. With a parent agent AND an in-process subagent both streaming on one context, the calls interleave and the single cursor hands the child the parent's script (and vice versa).
|
||||
- **The harness harvested one log.** `findSessionLog` walked the sessions root and returned the FIRST `.jsonl` it found. A subagent runs as a second `Session` with its own log in the same cwd bucket, so the child's transcript was silently dropped.
|
||||
|
||||
This was the `TODO(subagent-snapshots)` deferral recorded in the [subagent seam Agent Note](../feature/2026-06-21-subagent-capability-seam.md): the in-process backends (PR2) shipped with unit + e2e coverage, but the full-transcript snapshot tier could not express a nested-agent shape until this infrastructure landed. This Agent Note is that stacked follow-up.
|
||||
|
||||
## Decision
|
||||
|
||||
Replay is keyed **per calling session**, and the harness harvests **every** session log.
|
||||
|
||||
### 1. The calling session id rides on the model request
|
||||
|
||||
`GenerateOptions` gains an optional `sessionId`, stamped from `agent.session.id` during request assembly. Adapters ignore it; an `llm/stream` listener uses it to route by the issuing session. Its type is `Branded<'SessionId'>` (from `dsh-brand`) rather than `SessionId` from `dsh-session`, because that package imports `Message` from `dsh-llm` and importing back would create a cycle. The types are equivalent, so a session id assigns without a cast. Moving the brand to a dedicated ids package remains separate work because it would touch every id import.
|
||||
|
||||
### 2. Replay binds live sessions to recorded scripts by first-call order
|
||||
|
||||
A nested scenario records more than one log: the parent (`session.jsonl`) plus one per subagent child (`session.1.jsonl`, …). `dsh-llm-replay` loads them all, derives one script per recorded session, and orders the scripts by header `createdAt` (the parent is created before its children).
|
||||
|
||||
Live session ids are freshly random every run and never equal the recorded ones, so a live session cannot bind to a script by id equality. Instead it binds by **first-call order**: the first live session to make any model call claims the first ordered script (the parent — earliest `createdAt`, and necessarily the first to stream, because it must run a turn before it can delegate), the next new live session claims the next script, and so on. Each session then advances its own cursor independently.
|
||||
|
||||
This keys by WHO calls, not by global call order — so it stays correct even if subagents ever run concurrently or in the background (a global cursor would interleave them). A call carrying no `sessionId` (a direct unit-test `stream()`) is treated as one anonymous session bound to the primary script, so the single-session path is byte-for-byte the old behavior. More distinct live sessions than recorded scripts is a fail-loud error (an unrecorded subagent appeared), never a silent mis-route.
|
||||
|
||||
Child fixtures sort by `createdAt`, which matches call order while siblings run strictly sequentially. The id tiebreak only makes degenerate collisions deterministic. Concurrent or background children must introduce an explicit first-call ordinal instead of relying on timestamps.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
The alternative considered and rejected was a **call-ordered merge of the parent and child logs** into one global script (sound only because in-process subagent execution is strictly nested — the parent blocks on the child). It is simpler for today's synchronous cut but bakes in the parent-blocks-on-child invariant that a future backgrounded/concurrent subagent would break; per-session keying does not.
|
||||
|
||||
### 3. The harness harvests every log, primary-first
|
||||
|
||||
`harvestSessionLogs` collects every `.jsonl` across every cwd bucket under the sessions root (the JSONL backend puts a parent and its same-cwd child in the same bucket), parses each header, and orders them primary-first: the top-level session (no `parentSession`) leads, then each child by ascending `createdAt`. `RunResult.sessionLogs` is the plural result; the spec writes each back to its fixture on record (`session.jsonl` + `session.<n>.jsonl`) and diffs each harvested log against its fixture on replay. The normalizer already accepted plural session ids and collapses any stray UUID, so no normalizer change was needed.
|
||||
|
||||
### 4. Scenarios
|
||||
|
||||
Two nested scenarios were added and recorded against the real API:
|
||||
|
||||
- **`subagent-spawn`** — the parent delegates one subtask via the `subagent` tool to a fresh spawn child (2 sessions).
|
||||
- **`subagent-multi`** — the parent delegates two subtasks, each to its own spawn child (3 sessions), stressing the per-session keying with three concurrent scripts and the `createdAt` ordering of two children under one parent.
|
||||
|
||||
Both replay keyless in the default gate.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The `TODO(subagent-snapshots)` deferral is resolved: nested-agent transcripts are now a first-class snapshot shape.
|
||||
- `GenerateOptions.sessionId` is a small, honest core-seam addition useful beyond replay (telemetry, request routing).
|
||||
- The `subagent` tool is bound to a single provider, so both children in `subagent-multi` are spawn (fresh). The keying routes by session, not by backend, so it is already correct for fork. The script *derivation* was not: a fork child's log begins with the seeded parent prefix (the parent's `assistant/chunk` events), so deriving its script from the whole log would replay the parent's responses as the child's. That correctness gap is closed by persisting a seed boundary — see [Persist the seed boundary so fork-child replay routes correctly](2026-06-22-fork-child-replay-seed-boundary.md) — and recorded fork + mixed spawn+fork scenarios now exercise both transports through one transcript (see [Record fork and mixed spawn+fork snapshot scenarios](2026-06-22-fork-snapshot-scenarios.md)).
|
||||
- Out-of-process (ACP) subagents are a different replay shape entirely (each child is its own PROCESS with its own replay), tracked as `TODO(acp-subagent-replay)` in the PR3 plan.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Agent Note: Hook snapshot matrix — end-to-end expected outputs for both bridges
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
The hook bridges — [`dsh-hooks-claude`](../../../../packages/hooks/hooks-claude) (7 Claude Code hook points) and [`dsh-hooks-codex`](../../../../packages/hooks/hooks-codex) (5 Codex points) — map external hook commands onto the harness interception seams. They carry deep unit and coverage-spec coverage (every decision arm, every payload dialect, driven against a mocked seam) plus one key-gated e2e (`hooks.e2e.ts`, a live `PreToolUse` block). But the full-transcript snapshot tier — the one net that boots the real `acp-agent` subprocess, replays a recorded session keyless, and diffs the normalized ACP stdout + re-persisted log against committed expected outputs — covered exactly ONE hook: a Claude `UserPromptSubmit` block (`hook-cc-promptsubmit-block`).
|
||||
|
||||
That is the tier a mocked unit test structurally cannot be: it exercises the REAL bridge translating a REAL hook process's outcome into the REAL seam decision, then the REAL loop's reaction, rendered exactly as an editor sees it. A bridge-translation or loop-structure regression that left every unit green would still escape it for every hook point but one — and for the Codex bridge, the ACP example did not even LOAD it, so no Codex hook could fire end-to-end at all.
|
||||
|
||||
## Decision
|
||||
|
||||
The implementation has two coupled parts:
|
||||
|
||||
### 1. The ACP example ships BOTH hook bridges
|
||||
|
||||
`examples/acp-agent/cordis.yml` and `cordis.snapshot.yml` now load `dsh-hooks-codex` alongside `dsh-hooks-claude`, each pointed at its own config file (`./hooks.json` for Claude, `./codex-hooks.json` for Codex — the two dialects cannot share one file). This is a genuine product-surface change, not test-only wiring: the shipped ACP server (and the `demo:acp` front door) now carries both bridges.
|
||||
|
||||
It is safe because a bridge whose config file is absent is a **silent no-op**: `apply()` catches the read failure, logs through `ctx.logger`, and registers nothing — zero listeners, zero session events. The `acp-agent` app ships no stdout logger, so the warning cannot reach the ACP JSON-RPC channel. A scenario (or a real project) that wants only Claude hooks ships only `hooks.json`; the Codex bridge sees no `codex-hooks.json` and vanishes. This was verified empirically: with both bridges loaded, all pre-existing snapshots (none of which ship a `codex-hooks.json`) are byte-identical.
|
||||
|
||||
Loading both is the minimum that lets the snapshot tier exercise each dialect against the same real app the product ships. Recording (which boots `cordis.yml`) loads both by construction, and replay inherits them the same way: `cordis.snapshot.yml` is an include-overlay of `cordis.yml` that swaps only the llm entry (see [single-source the acp-agent replay config](2026-07-04-single-source-acp-replay-config.md)), so a bridge added to the live tree is in the replay tree with no second edit.
|
||||
|
||||
### 2. A snapshot scenario per hook point × its headline outcome, both dialects
|
||||
|
||||
Thirteen scenarios under `examples/acp-agent/tests/snapshots/`, naming `hook-<dialect>-<point>-<outcome>`:
|
||||
|
||||
- **Authored, no model turn** (keyless, no sidecar — the derived replay script is empty; the `rejected` turn carrying `hook/*` events is compared): `hook-cc-promptsubmit-block`, `hook-codex-promptsubmit-block`.
|
||||
- **Recorded against the real API, hook active during recording** (the model's reaction to the decision is part of the captured transcript, replayed keyless thereafter): `hook-{cc,codex}-promptsubmit-context` (allow + additionalContext fold), `hook-cc-pretool-deny` / `hook-codex-pretool-block` (deny → `isError` tool result), `hook-cc-pretool-ask` (ask → degrades to deny with the approval-required reason), `hook-{cc,codex}-posttool-block` (block with feedback), `hook-{cc,codex}-posttool-context` (accept + additionalContext), `hook-{cc,codex}-stop-continue` (a blocking Stop hook forces one extra step via steering).
|
||||
|
||||
Each hook command emits only FIXED LITERAL strings (no timestamps/pids/`$RANDOM`/cwd echoes); the snapshot normalizer scrubs the one volatile field a `hook/result` carries (`durationMs`). The `Stop` scenarios self-limit with a marker file (`.stop_fired`) so the force-continue does not loop — the `stop_hook_active` loop-guard is still a bridge `TODO`, so an unconditional Stop hook would force-continue every step.
|
||||
|
||||
The `PostToolUse` block scenarios self-limit at the mechanism they prove. The Claude hook persists a workspace marker after its first rejection, so one recovery call is allowed; the Codex prompt makes one call and reports the injected result. Each expected output pins one blocked call without repeated block/retry cycles.
|
||||
|
||||
### Three hook points are deliberately NOT snapshotted
|
||||
|
||||
Discovered while building the matrix, and documented here because the omission is a decision, not an oversight:
|
||||
|
||||
- **`SessionStart` and `SubagentStart`** inject context through a detached, best-effort `void runPoint(...).then(agent.inject())` with NO turn binding. The resulting `context/message` races the work it precedes (the first model request / the child's first turn) and lands at a nondeterministic log position. A recorded expected output does not even reproduce on its own replay — a 10× replay stability check failed 10/10 for both. They stay on the bridges' unit coverage, which drives the seam directly without the timing race. (If the injection is ever made turn-bound and deterministic — the direction the `TODO(session-start-gating)` points — these become snapshottable.)
|
||||
- **`SubagentStop`** is observe-only: its `subagent/end` handler passes no turn (so no `hook/*` log events) and does no injection. It writes NOTHING to the transcript, so an expected output would be byte-identical to the no-hook run and could never be proven to fail — a guard that cannot bite. It stays on unit coverage (`bridge.spec.ts` already asserts the observe-only call).
|
||||
|
||||
The matrix therefore covers every hook point that has a DETERMINISTIC, OBSERVABLE transcript footprint, for both dialects.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Every bridge seam mapping with an observable transcript is now guarded at the full-transcript tier, in the real app, for both dialects — including the Codex bridge, which had no end-to-end coverage at all. Recorded expected outputs capture the model's real reaction to a denied/blocked/force-continued turn, which a hand-authored transcript could only guess at.
|
||||
- The `UserPromptSubmit` block scenarios are authored keylessly (no model turn); the rest replay keylessly from recorded fixtures. `pnpm run test:snapshot:record` regenerates the recorded fixtures from the live API and self-skips without a key like every recorded scenario.
|
||||
- The prove-red discipline holds: tampering a hook config's output (e.g. changing a deny reason) turns its scenario red on replay — the hook process runs FOR REAL during replay (only the model is replayed), so the expected output guards the actual hook→seam→loop path, not a mock of it.
|
||||
- The `acp-agent` demo now loads a Codex bridge it will usually no-op (no `codex-hooks.json` in a typical project), which is the intended fail-soft behavior, not a cost.
|
||||
|
||||
<!-- agent-note-format: alternatives-not-recorded (pre-format Agent Note) -->
|
||||
@@ -0,0 +1,25 @@
|
||||
# Agent Note: Single-source the acp-agent replay config
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
`examples/acp-agent` shipped two hand-maintained configs: `cordis.yml` (the live tree) and a `cordis.snapshot.yml` that mirrored it entry-for-entry with only the llm backend swapped — stripped of comments, the entire difference was the eight-line `llm-deepseek` stanza versus the two-line `llm-replay` stanza. Every app-shape change had to be made twice, and nothing gated the symmetry: if the copies drifted, the snapshot tier would silently exercise a different app than the one that ships — the ["green units, broken product" class of gap](../../../../docs/postmortem/0001-acp-default-export-drops-inject.md) the snapshot tier exists to close, reintroduced one level up, with reviewer vigilance as the only defense.
|
||||
|
||||
## Decision
|
||||
|
||||
`cordis.snapshot.yml` includes the live config, disables the named DeepSeek adapter by id and name, and inserts the replay adapter. Every other entry therefore comes from the shipping tree. Replay selects the overlay; recording still boots `cordis.yml`, and the load guard permits the intentionally disabled entry.
|
||||
|
||||
One vendored-plugin fact the overlay depends on, deliberately: the include applies `patches` when it loads the file — its `refresh()`/`internal/update` paths re-read without re-patching — which is exactly enough for a one-shot replay boot (the replay app loads no `hmr` and nothing rewrites the config mid-run). The snapshot suite is the proof: all scenarios pass unchanged on the overlay, byte-identical expected outputs included.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
### Why not the alternatives?
|
||||
|
||||
Keeping the full twin with a symmetry verify-gate was the recorded fallback — it would have removed the silent-drift class but kept a 125-line near-copy whose only content was one entry's difference, growing with every plugin the app gains. A bin-side swap (parse the config, replace the entry, delete the file) would have put YAML surgery inside a published artifact and moved the replay delta out of sight; the overlay keeps the delta declarative, readable, and next to the base config — the teaching value the twin's defenders actually wanted.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A plugin added to `cordis.yml` is in the replay tree with no second edit; the drift class is structurally gone rather than gated.
|
||||
- The overlay depends on entries carrying stable `id:`s. The `name` assertion on the disable patch guards mis-targeting (a reused id skips the patch instead of disabling the wrong plugin). An id RENAME degrades the patch to a skip whose warning needs a logger the replay app deliberately lacks — the observable result is a futile keyless `llm-deepseek` entry alongside `llm-replay`, with replay output still correct (`llm-replay` owns the stream short-circuit); config rot for review to catch, not wrong snapshots. A top-level insert whose id collides with an existing entry resolves last-wins through the loader's id map — the current config has no collision, and a new patch line is where one would be introduced.
|
||||
- If a future replay tree needs a second divergence (another backend swapped), it is one more patch line, not a second fork of the file.
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
# Agent Note: Pin request-header content in one snapshot scenario
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
An ACP snapshot suite needs to prove the exact composed system prompt and tool-schema list sent in each `request/header`, but duplicating that content inside every `session.jsonl` makes a prompt or schema edit rewrite dozens of giant one-line JSON records. Keeping one raw header avoids the duplication but still makes prompt review poor: prose is JSON-escaped onto one line and mixed with thousands of characters of tool schemas.
|
||||
|
||||
## Decision
|
||||
|
||||
Exactly one scenario per header-composition class is flagged `pinsHeader`. Its directory splits the pin by review format: `system-prompt.expected.md` contains the normalized full prompt sequence as ordinary Markdown, `tool-schemas.expected.json` contains the corresponding complete schema sequence as structured JSON, and `session.jsonl` retains config, reason, and any model-visible prefix while storing `header.system` and `header.tools` as `"{{system}}"` / `"{{tools}}"`. Every other JSONL uses the same prompt and tool tokens and also tokenizes session-prefix content. The pin mechanics live in [`dsh-acp-snapshot`](../../../../packages/support/acp-snapshot/README.md), whose suite factory enforces one pin per class.
|
||||
|
||||
The pure `scrubSystemPrompts` and `scrubToolSchemas` normalizers independently tokenize every stored full header. `scrubRequestHeaders` also tokenizes session-prefix content for non-pinning scenarios while retaining header count, field presence, config, reason, and prefix message count. Record and refresh write-back apply the appropriate scrub before writing JSONL and regenerate both sidecars from the normalized live full-header sequence, so neither path can reintroduce prompt/schema bulk into JSONL or leave a review artifact stale.
|
||||
|
||||
Guards make the split self-enforcing. On disk, every `session*.jsonl` is a fixed point of both prompt and schema scrubbers, only non-pinning fixtures must be fixed points of the full header scrub, both sidecars exist exactly beside pinning fixtures in canonical newline-terminated formats, and each class has one pin. Live, every `request/header` produced by a parent, spawn child, fork child, initial request, resume, or in-instance change must match the reconstructed class sequence after volatile-value normalization. A header without a string prompt, without an array-valued tool list, or beyond the pin's declared changed-header count fails loud.
|
||||
|
||||
One pin covers the whole suite because every session — parent, spawn child, fork child — composes the identical tool list and the identical prompt modulo cwd, and the uniformity guard fails the suite the moment that stops holding. If header composition ever becomes session-dependent by design (a restricted subagent toolset, say), the divergent shape gets its own pinning scenario.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Re-record or hand-edit every fixture per change** — preserves exact headers but buries behavioral diffs under duplicated prompt and schema content.
|
||||
- **Scrub at compare time only, keeping fixtures raw** — lets compares pass while committed fixtures retain stale duplicate content and rewrite wholesale on the next recording. Stored tokens state honestly what each JSONL does not pin.
|
||||
- **Scrub everywhere, pin nowhere** — loses the only end-to-end record of the composed header as actually sent (prompt assembly, registered-tool order, full schemas). The generated tool catalog documents each tool in isolation; only a real fixture pins the composed set.
|
||||
- **Keep the one full pin entirely in JSONL** — removes suite-wide duplication but leaves prompt and schema changes as one escaped line. Markdown and structured JSON give each surface its natural review format without weakening the reconstructed-header assertion.
|
||||
- **Slim the session log itself (log a content digest, store the header elsewhere)** — violates the reconstructability contract: the product log must reproduce each request bit-for-bit ([reconstructable-requests Agent Note](../architecture/2026-07-05-reconstructable-requests.md)). Header bulk is a test-artifact concern, solved in test normalization; the live log is untouched.
|
||||
|
||||
## Verification
|
||||
|
||||
The suite replays every scenario against the split pins. Unit coverage exercises the independent and full scrubbers, both full-header sidecar formats, record/refresh regeneration, normalized prompt/schema extraction, fixed-point enforcement, required-file symmetry, reconstructed-header uniformity, and changed-header count rejection.
|
||||
|
||||
## Consequences
|
||||
|
||||
A system-prompt change produces a line-oriented Markdown diff in one file per affected composition class; a tool-description change produces a structured JSON diff in one file per class; ordinary behavioral fixtures remain untouched. Session fixtures display tokens for omitted content, and the live uniformity guard makes each split pin authoritative for every session in its class. Each pinning scenario carries two generated, newline-canonicalized sidecars.
|
||||
@@ -0,0 +1,38 @@
|
||||
# Agent Note: Extract the ACP snapshot suite into a support package
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
The ACP snapshot tier ([snapshot Agent Note](2026-06-19-acp-snapshot-tests.md)) was built from three modules living inside one example's test directory: `snapshot-harness.ts` (boot the real bin subprocess, drive it over ACP JSON-RPC, harvest the persisted logs), `snapshot-normalize.ts` (the pure expected-output normalizers), and the ~150-line scenario body plus fixture guards in `acp.snapshot.ts` (record/replay modes, the stdout expected-output and log comparisons, the pinned-header uniformity guard, the orphan/required-file/single-pin meta-tests).
|
||||
|
||||
A second ACP example wanting snapshot coverage — the sandbox/approval composition is the immediate consumer — could only copy those modules, forking exactly the logic that must not drift: record write-back, header scrubbing, child-session harvest ordering. The spawn/client glue was also triplicated across `acp.e2e.ts`, `hooks.e2e.ts`, and the harness. Location decided test rigor: the per-file 100% coverage gate measures `packages/*/*/src` only, so none of this machinery was measured — the same gap that had moved `dsh-llm-replay` out of `examples/` into [packages/support](../../../../packages/support/README.md). And the harness's ACP client hardcoded `requestPermission → cancelled`, so an approval round-trip — the headline behavior of the sandbox composition — could not be expressed at the snapshot tier at all.
|
||||
|
||||
## Decision
|
||||
|
||||
The machinery lives in [`packages/support/acp-snapshot`](../../../../packages/support/acp-snapshot/README.md) (`@deepseek-ai/dsh-acp-snapshot`); an example's `*.snapshot.ts` is its scenario table, its agent paths, and one factory call, over its own `snapshots/` fixtures and `cordis.snapshot.yml` overlay ([single-source replay config](2026-07-04-single-source-acp-replay-config.md)). Reading `DSH_SNAPSHOT` stays at that edge — the library takes a resolved `mode`.
|
||||
|
||||
**`src/launcher.ts`** — `launchAcpTestAgent` owns the common unbuilt-process boundary: absolute tsx loader resolution, `TSX_TSCONFIG_PATH`, isolated harness homes, stdio wiring, a raw-byte stdout tee, stderr and update capture, fail-closed permission fallback, update waiters, and graceful or signalled shutdown. Snapshot scenarios and ordinary e2e suites supply the same `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath`); a test that plays a user supplies only its permission handler. The ACP and hook e2e suites plus the sandbox/approval e2e suite use this launcher instead of rebuilding the SDK client boundary.
|
||||
|
||||
**`src/harness.ts`** — `runScenario` and the input-script/result types layer deterministic steps, temp workspaces, snapshot environment, and persisted-log harvest over the launcher. Its `session/request_permission` handler consumes an optional `InputScript.permissionAnswers` FIFO queue, each entry selecting by option **kind** (ids are agent-issued randoms a committed script cannot know; kinds are the ACP-stable vocabulary, mapped to the offered `optionId` at answer time); an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run — the agent itself is answered `cancelled`, so the scenario bug fails the harness rather than being absorbed as an agent-side denial. This is what lets an approval suite drive allow/reject round-trips deterministically from `input.json`.
|
||||
|
||||
**`src/normalize.ts`** — the pure normalizers, hook-free by policy: when a future event carries a new volatile field (an approval duration, say), the shared normalizer learns it in the same change, keeping one home for what "normalized" means rather than per-suite scrub extensions.
|
||||
|
||||
**`src/suite.ts`** — the `Scenario` type and `defineAcpSnapshotSuite(options)`, registering the per-scenario compares, record/refresh fixture write-back, the header pin with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL a `scrubSystemPrompts` fixed point, non-pinning fixtures also `scrubRequestHeaders` fixed points). A scenario directory's `session.jsonl` plus contiguous `session.<n>.jsonl` siblings are its ordered primary/child inventory, so the scenario table declares policy without duplicating a child count. The pinned-header contract ([pinned-header Agent Note](2026-07-06-pin-request-header-content-in-one-scenario.md)) is per-suite: each header class flags exactly one `pinsHeader` scenario, whose `system-prompt.expected.md` and JSONL tool list split the composed header into reviewable artifacts; the uniformity guard compares both against every live header in that class. A pinning scenario declares any legitimate changed-header count, and its Markdown artifact records every full changed prompt. The pure helpers (`sessionFixtureNames`, `fixtureContext`, `normalizedHeaders`, `normalizedSystemPrompts`, `formatSystemPromptSnapshot`, `headerChangeCount`) are exported from the module for direct unit coverage.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Copy the modules into each example** — the fork this Agent Note exists to prevent: the record/guard logic is exactly the code that must stay byte-identical across suites, and examples are outside the coverage gate, so each copy is also unmeasured.
|
||||
- **A shared module directory under `examples/`** — keeps the code outside the coverage gate and forces relative imports across example boundaries, against the package-name import convention; `examples/` leaves stay thin by design.
|
||||
- **A `/testing` subpath export of `dsh-acp-demo`** — couples test infrastructure into a product package's surface and dependency set; `packages/support/` exists precisely for real-but-lower-compatibility dev/test packages, with `dsh-llm-replay` as the precedent this package completes.
|
||||
- **Export raw test-body functions instead of a suite factory** — each example would re-own the `describe`/`it` skeleton (~80 lines of registration boilerplate per suite) for no flexibility gain; the factory keeps consumers to a scenario table plus one call, and the exported pure helpers preserve unit-testability inside the factory design.
|
||||
- **An injectable ACP `Client` factory instead of declarative `permissionAnswers`** — maximally flexible, but it leaks SDK client construction to every consumer and reopens per-example drift in exactly the layer being unified; a declarative queue keeps `input.json` the single scripting surface and compatible with expected-output normalization.
|
||||
- **Generalize beyond ACP (a transport-agnostic snapshot harness)** — no second transport exists; the harness is ACP-shaped end to end (SDK client, JSON-RPC frames, `session/update` waiters), and a speculative abstraction would be a seam split ahead of any consumer.
|
||||
|
||||
## Testing
|
||||
|
||||
Extraction parity was proven mechanically: after the move, `pnpm run test:snapshot` matched the base commit's result with zero byte changes under `examples/acp-agent/tests/snapshots/`. The package's `src/` holds per-file 100% statements/branches/functions/lines under the gating unit run, driven through the real launcher by a scripted fake ACP bin (`tests/fixtures/fake-acp-agent.ts`, behavior scripted per scenario via a `behavior.json` beside the fixture): `harness.spec.ts` directly covers launcher defaults, captures, update waiting, shutdown, and environment/config variants, then covers every scenario step op, both expect-error arms, the permission queue (selection, fallback, impossible-click), workspace seeding, and the harvest ordering/noise/fallback branches; `suite.spec.ts` runs the factory for real at collection time — a replay suite over committed synthetic fixtures and a record suite over a temp copy (write-back never touches the committed tree; `ACP_SNAPSHOT_SPEC_BOOTSTRAP=1` re-bootstraps it) — plus direct cases for the pure helpers. The fake bin substitutes the `session/new` cwd, not `process.cwd()`, into scripted logs, matching what the real bin's header carries (darwin realpaths `/var/folders/…` to `/private/var/folders/…`).
|
||||
|
||||
## Consequences
|
||||
|
||||
A new example gets the whole snapshot tier from a scenario table plus fixtures, while an ordinary ACP e2e gets the same tested process/client boundary from one launcher call. The costs: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run — a shape no other package has, stated in its README; and each suite pins its own ~8 KB header fixture (a genuinely distinct composition deserves its own pin; an identical one would be caught by that suite's uniformity guard).
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-18-tui-terminal-state-snapshots.md: 192e872ab63cf4ff8a121ea0a2ee9345379cfa26
|
||||
2026-07-18-tui-terminal-state-snapshots.zh.md: 9766a8087632daa1be0dcfb191696dbad354ff68
|
||||
@@ -0,0 +1,70 @@
|
||||
# Agent Note: Snapshot semantic terminal state for the TUI
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-07-18-tui-terminal-state-snapshots.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The TUI is a stateful renderer. Its user-visible result depends on ANSI parsing, differential frames, wrapping, scrollback, viewport position, terminal width, focus, cursor state, and each tool's presentation intent. Unit tests that collect `Terminal.write()` fragments can prove event handling, but they cannot prove the final screen a terminal displays. The same screen may also be emitted through different write fragments, so pinning those fragments creates false regressions.
|
||||
|
||||
Component-line snapshots stop before ANSI reaches a terminal and miss cursor movement, clearing, styling, overlay composition, and reflow. Raster screenshots include font and platform rendering noise that is unrelated to the TUI contract. A completed flow built by directly appending plausible session events has another blind spot: it proves the renderer accepts those shapes, not that the production agent loop and tool implementations produce them.
|
||||
|
||||
The TUI therefore needs a deterministic, reviewable representation of terminal state, recorded model journeys that execute the real downstream stack, and a smaller test at the real process and PTY boundary.
|
||||
|
||||
## Decision
|
||||
|
||||
TUI coverage has four complementary layers:
|
||||
|
||||
1. `packages/ui/tui/tests/tui.spec.ts` tests event mapping, input routing, disposal, and error behavior directly.
|
||||
2. `packages/ui/tui/tests/tui.snapshot.ts` mounts the production TUI against a headless terminal emulator for transient states that a completed session log cannot retain: in-flight streaming, pending tool calls, overlays, expansion, compaction reflow, errors, and shutdown.
|
||||
3. `examples/tui-agent/tests/tui.snapshot.ts` replays committed JSONL session logs through the production agent loop and real tools, then compares the resulting semantic terminal state.
|
||||
4. `examples/tui-agent/tests/tui-keyless-smoke.e2e.ts` boots the real Loader composition in a PTY, drives a scripted conversation through streaming and `ask_user_question`, and verifies startup, input, exit, failure reporting, and terminal restoration.
|
||||
|
||||
The runnable TUI has its own `examples/tui-agent` leaf beside the readline `repl-agent` and `acp-agent` leaves. It reuses the repl-agent backend and tool composition through an asserted include patch while fixing the shared terminal app to `ui.mode: tui`; TUI snapshots and PTY tests live with that leaf.
|
||||
|
||||
### Recorded-session replay
|
||||
|
||||
Each example-level scenario directory owns `session.jsonl`, optional child logs `session.<n>.jsonl`, and `terminal.expected.txt`. The primary log supplies user-authored `user/message` prompts and the recorded `assistant/chunk` sequence. `dsh-llm-replay` derives one model-call script per session, binds child logs to fresh child sessions, and is the only mocked boundary. The agent loop, bash and filesystem implementations, Code Mode worker, subagent provider, workflow worker, Cordis tools, presenters, and TUI are production implementations.
|
||||
|
||||
The suite rejects a journey when its tool-call sequence differs, an expected event count is missing, a tool result is an error, a turn ends in error, a workflow lifecycle is incomplete, or the live child-session count differs from the fixture set. These assertions prevent an attractive terminal expected output from hiding a failed or bypassed production path.
|
||||
|
||||
The live-model fixtures use `DSH_SNAPSHOT=record`; record mode rewrites their primary and child JSONL logs and terminal expected outputs. The deterministic Cordis toolchain keeps an authored complete JSONL script because reliably coercing a live model through five exact tool boundaries and two children is not a stable recording contract. `DSH_SNAPSHOT=refresh` replays every committed script keylessly and rewrites only derived terminal expected outputs. Plain replay compares without writing, and unknown mode values fail loud.
|
||||
|
||||
### Semantic terminal projection
|
||||
|
||||
The package-local `HeadlessTerminal` implements the same pi-tui `Terminal` interface as the process terminal and feeds every ANSI write into the pinned `@xterm/headless` parser. Snapshot code waits for synchronized frames to quiesce before reading state, so a checkpoint represents a completed screen rather than a timer-dependent write prefix.
|
||||
|
||||
Each expected output projects dimensions, active-buffer and viewport coordinates, lifecycle and cursor state, rows, wrap markers, and non-default style ranges into text. Scroll-heavy cards capture the used buffer; overlays capture the visible viewport. Text and style remain separate so a reviewer can distinguish content changes from presentation changes without decoding ANSI bytes.
|
||||
|
||||
Every checkpoint enforces theme independence across the complete terminal state: no RGB colors, no palette entries beyond ANSI 0–15, and no explicit background colors. Reverse video remains valid for selection because it uses terminal defaults. Both suites own closed inventories that reject missing scenarios, missing checkpoints, and orphaned expected output files.
|
||||
|
||||
### Required scenario matrix
|
||||
|
||||
| Layer | Scenario | Contract pinned |
|
||||
|---|---|---|
|
||||
| Recorded journey | Multi-turn conversation | Recorded reasoning/text chunks, two input turns, retained history, token totals, and idle editor state |
|
||||
| Recorded journey | Todo plan | Real `todo_write` execution, result card, and persistent plan rendering |
|
||||
| Recorded journey | Bash terminal card | Real local executor output, description, exit status, and completed terminal card |
|
||||
| Recorded journey | Parallel filesystem reads | Two calls from one assistant message, real file contents, ordering, and separate completed cards |
|
||||
| Recorded journey | Code Mode | Real `run_code` worker execution, two `tool/code-dispatch` events, captured program output, and completed card |
|
||||
| Recorded journey | Dynamic workflow | Real workflow worker, phase lifecycle, replayed child session, structured return value, and completed card |
|
||||
| Recorded journey | Cordis dynamic toolchain | Real mount, Code Mode inspect, direct subagent, workflow child, unmount, and all production presenters |
|
||||
| Transient state | Streaming and pending advanced calls | In-flight reasoning/text plus pending Code Mode, workflow, and Cordis cards that disappear from completed logs |
|
||||
| Transient state | Cards, interaction, layout, failure, and shutdown | Collapsed/expanded card families, question validation, compaction replacement, resize reflow, help/errors, cursor restoration, and terminal stop |
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Snapshot raw terminal writes** — rejected because differential rendering may change write boundaries without changing the screen, while cursor and clear sequences are unreadable in review.
|
||||
- **Snapshot component render lines before terminal output** — rejected because it does not test ANSI parsing, cursor movement, overlays, viewport behavior, or independent components in one frame.
|
||||
- **Build every completed flow by appending session events** — rejected because a hand-authored event sequence can drift from the agent loop, tool execution, child-session binding, or worker behavior while its presentation test stays green. Direct event construction remains limited to transient renderer states.
|
||||
- **Reuse ACP stdout expected outputs as the TUI oracle** — rejected because a recorded model journey is transport-neutral but its presentation is not. TUI scenarios own terminal expected outputs while using the same JSONL replay vocabulary.
|
||||
- **Commit raster screenshots** — rejected because fonts, glyph metrics, antialiasing, and host terminal themes make them platform-sensitive and make semantic style changes difficult to review.
|
||||
- **Use only PTY end-to-end tests** — rejected because raw PTY output is a stream of historical drawing operations, not queryable final state. PTY tests retain the real Loader/input/teardown boundary, while the emulator owns broad state coverage.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Completed advanced snapshots now fail when the real Code Mode, workflow, subagent, filesystem, bash, or Cordis path breaks, rather than accepting a fabricated result event.
|
||||
- TUI visual regressions produce readable cell-and-style diffs, while JSONL fixtures retain the exact model chunks that made the production path execute.
|
||||
- The emulator uses xterm's proposed buffer API. An xterm upgrade requires rerunning and reviewing the semantic projection; terminal-specific behavior still needs the PTY smoke.
|
||||
- Expected outputs deliberately encode wrapping and viewport behavior at fixed sizes. Intentional layout changes use keyless refresh, while model-journey changes use record mode and review both JSONL and terminal diffs.
|
||||
@@ -0,0 +1,70 @@
|
||||
# Agent Note: TUI 语义终端状态快照
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-07-18-tui-terminal-state-snapshots.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
TUI 是有状态的渲染器。用户最终看到的结果取决于 ANSI 解析、差分帧、换行、回滚缓冲、视口位置、终端宽度、焦点、光标状态,以及各工具的呈现意图。收集 `Terminal.write()` 片段的单元测试可以验证事件处理,却无法验证终端最终显示的画面。同一画面也可能由不同的写入片段产生,因此固定这些片段会制造误报。
|
||||
|
||||
组件行快照止于 ANSI 进入终端之前,无法覆盖光标移动、清屏、样式、浮层组合和重排。栅格截图会带入与 TUI 契约无关的字体和平台渲染噪声。直接追加看似合理的会话事件来构造完整流程还存在另一处盲区:这种测试只能证明渲染器接受这些数据形态,无法证明生产环境的 agent loop(智能体循环)和工具实现会生成这些事件。
|
||||
|
||||
因此,TUI 既需要确定、便于评审的终端状态表示,也需要通过已录制模型流程执行真实下游组件,并保留一项范围更小、覆盖真实进程与 PTY 边界的测试。
|
||||
|
||||
## 决策
|
||||
|
||||
TUI 覆盖分为四个互补层次:
|
||||
|
||||
1. `packages/ui/tui/tests/tui.spec.ts` 直接测试事件映射、输入路由、资源释放和错误行为。
|
||||
2. `packages/ui/tui/tests/tui.snapshot.ts` 将生产 TUI 挂载到无界面终端模拟器,覆盖完整会话日志无法保留的瞬态:进行中的流式输出、待完成工具调用、浮层、展开状态、压缩重排、错误和关闭过程。
|
||||
3. `examples/tui-agent/tests/tui.snapshot.ts` 通过生产 agent loop 和真实工具回放已提交的 JSONL 会话日志,再比较生成的语义终端状态。
|
||||
4. `examples/tui-agent/tests/tui-keyless-smoke.e2e.ts` 在 PTY 中启动真实 Loader 组合,驱动一段经过流式输出和 `ask_user_question` 的脚本化会话,并验证启动、输入、退出、失败报告和终端恢复。
|
||||
|
||||
可运行 TUI 在 `examples/tui-agent` 中拥有独立叶节点,与 readline `repl-agent` 和 `acp-agent` 叶节点并列。它通过带断言的 include patch 复用 repl-agent 的后端与工具组合,只把共享终端应用固定为 `ui.mode: tui`;TUI 快照和 PTY 测试也归属这个叶节点。
|
||||
|
||||
### 已录制会话回放
|
||||
|
||||
每个示例级场景目录都包含 `session.jsonl`、可选的子会话日志 `session.<n>.jsonl`,以及 `terminal.expected.txt`。主日志提供用户来源的 `user/message` 提示词和已录制的 `assistant/chunk` 序列。`dsh-llm-replay` 为每个会话派生一份模型调用脚本,并将子日志绑定到新建的子会话;这是测试中唯一的 mock 边界。agent loop、bash 与文件系统实现、Code Mode worker、subagent 提供方、工作流 worker、Cordis 工具、呈现器和 TUI 都使用生产实现。
|
||||
|
||||
如果工具调用顺序不符、预期事件数量不足、工具结果报错、轮次以错误结束、工作流生命周期不完整,或者实时子会话数量与 fixture(测试前置数据)集合不一致,测试都会失败。即使终端预期输出表面正确,这些断言也能阻止失败或被绕过的生产路径混入结果。
|
||||
|
||||
真实模型 fixture 通过 `DSH_SNAPSHOT=record` 更新;录制模式会重写其主会话与子会话 JSONL 日志以及终端预期输出。确定性的 Cordis 工具链保留一份人工编写的完整 JSONL 脚本,因为要求真实模型稳定经过五个指定工具边界和两个子会话并不是可靠的录制契约。`DSH_SNAPSHOT=refresh` 会无密钥回放所有已提交脚本,并且只重写派生的终端预期输出。普通回放只比较而不写入,未知模式值会快速失败。
|
||||
|
||||
### 语义终端投影
|
||||
|
||||
包内的 `HeadlessTerminal` 实现与进程终端相同的 pi-tui `Terminal` 接口,并把每次 ANSI 写入交给固定版本的 `@xterm/headless` 解析器。读取状态前,快照代码会等待同步帧稳定,因此每个检查点表示已经完成的画面,而不是依赖计时的写入前缀。
|
||||
|
||||
每份预期输出把终端尺寸、活动缓冲区和视口坐标、生命周期与光标状态、各行、换行标记以及非默认样式区间投影为文本。滚动内容较多的卡片捕获已使用缓冲区;浮层捕获可见视口。文本和样式相互分离,评审人无需解码 ANSI 字节即可区分内容变化与呈现变化。
|
||||
|
||||
每个检查点还会对完整终端状态强制执行主题无关性:禁止 RGB 颜色、禁止 ANSI 0–15 以外的调色板项,也禁止显式背景色。选择行使用终端默认色进行反显,因此仍然有效。两套测试都拥有封闭清单,会拒绝缺失的场景、缺失的检查点和遗留预期输出文件。
|
||||
|
||||
### 必需场景矩阵
|
||||
|
||||
| 层次 | 场景 | 固定的契约 |
|
||||
|---|---|---|
|
||||
| 已录制流程 | 多轮会话 | 已录制的推理与文本分片、两轮输入、保留历史、token 总量和空闲编辑器状态 |
|
||||
| 已录制流程 | Todo 计划 | 真实 `todo_write` 执行、结果卡片和持久计划渲染 |
|
||||
| 已录制流程 | Bash 终端卡片 | 真实本地执行器输出、说明、退出状态和已完成终端卡片 |
|
||||
| 已录制流程 | 并行文件读取 | 同一条 assistant 消息中的两次调用、真实文件内容、顺序和两个独立完成卡片 |
|
||||
| 已录制流程 | Code Mode | 真实 `run_code` worker 执行、两条 `tool/code-dispatch` 事件、捕获的程序输出和已完成卡片 |
|
||||
| 已录制流程 | 动态工作流 | 真实工作流 worker、阶段生命周期、回放的子会话、结构化返回值和已完成卡片 |
|
||||
| 已录制流程 | Cordis 动态工具链 | 真实挂载、Code Mode 检查、直接 subagent、工作流子会话、卸载和全部生产呈现器 |
|
||||
| 瞬态 | 流式输出与待完成高级调用 | 进行中的推理和文本,以及完整日志中不会保留的待完成 Code Mode、工作流和 Cordis 卡片 |
|
||||
| 瞬态 | 卡片、交互、布局、失败和关闭 | 折叠与展开的卡片族、问题校验、压缩替换、尺寸重排、帮助与错误、光标恢复和终端停止 |
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
- **快照原始终端写入**:不予采纳,因为差分渲染可能在画面不变时改变写入边界,而且光标与清屏序列难以评审。
|
||||
- **快照进入终端输出之前的组件渲染行**:不予采纳,因为它无法测试 ANSI 解析、光标移动、浮层、视口行为,也无法测试独立组件在同一帧中的相互作用。
|
||||
- **通过追加会话事件构造所有完整流程**:不予采纳,因为人工编写的事件序列可能与 agent loop、工具执行、子会话绑定或 worker 行为发生偏差,但呈现测试仍然保持绿色。直接构造事件只用于渲染器瞬态。
|
||||
- **复用 ACP stdout 预期输出作为 TUI 判定依据**:不予采纳,因为已录制模型流程与传输方式无关,其呈现方式却并非如此。TUI 场景使用同一套 JSONL 回放词汇,但拥有独立的终端预期输出。
|
||||
- **提交栅格截图**:不予采纳,因为字体、字形度量、抗锯齿和宿主终端主题会使结果依赖平台,也会增加语义样式变更的评审难度。
|
||||
- **只使用 PTY 端到端测试**:不予采纳,因为原始 PTY 输出是一系列历史绘制操作,而不是可查询的最终状态。PTY 测试保留真实 Loader、输入与清理边界,模拟器负责广泛的状态覆盖。
|
||||
|
||||
## 后果
|
||||
|
||||
- 当真实 Code Mode、工作流、subagent、文件系统、bash 或 Cordis 路径损坏时,已完成高级快照会失败,不会继续接受伪造的结果事件。
|
||||
- TUI 视觉回归会产生便于阅读的单元格和样式 diff,而 JSONL fixture 会保留触发生产路径的确切模型分片。
|
||||
- 模拟器使用 xterm 的拟议缓冲区 API。升级 xterm 时必须重新运行并评审语义投影;终端特有行为仍需由 PTY 冒烟测试覆盖。
|
||||
- 预期输出有意固定指定尺寸下的换行与视口行为。预期布局变更使用无密钥刷新;模型流程变更使用录制模式,并同时评审 JSONL 与终端 diff。
|
||||
Reference in New Issue
Block a user