From fd6713b4967506e654dd7bdf77b067f4ffc57f96 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 24 Jul 2026 00:20:38 +0800 Subject: [PATCH 01/11] docs(testing): propose keyless browser e2e lane for the web GUI Design study for a deterministic, keyless browser e2e lane over the real assembled web chain (chromium -> SSE/HTTP wire -> apiproxy -> agent loop -> persistence), replayed through dsh-llm-replay from recorded session-log fixtures, with aria-tree goldens plus in-process world-state assertions. Synthesized from an OSS prior-art survey (LibreChat, ai-chatbot, lobe-chat, OpenHands, cline, aimock...), a repo seam deep-dive, and three adversarial critiques (doctrine, flakiness, YAGNI). Records the settled shape (no new package, no suite factory, seed via the real persistence API, whenIdle barrier stack, no transient-DOM assertions) and the open questions (LLM seam, Loader-izing dsh web, header pin, golden breadth, settled signal). --- .../2026-07-24-web-gui-browser-e2e-lane.md | 92 +++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 .agents/notes/proposed/testing/2026-07-24-web-gui-browser-e2e-lane.md diff --git a/.agents/notes/proposed/testing/2026-07-24-web-gui-browser-e2e-lane.md b/.agents/notes/proposed/testing/2026-07-24-web-gui-browser-e2e-lane.md new file mode 100644 index 0000000000..cc7504dae7 --- /dev/null +++ b/.agents/notes/proposed/testing/2026-07-24-web-gui-browser-e2e-lane.md @@ -0,0 +1,92 @@ +# Agent Note: Keyless browser e2e lane for the web GUI + +Status: proposed + +## Problem + +The web GUI ships as a real assembled chain — chromium page → nine client plugin bundles → HTTP unary RPC + two SSE streams → `toFetchHandler`/apiproxy → `bootHost`'s agent loop, tools, and JSONL persistence — and no test exercises that chain keylessly and deterministically. The [GUI testing system](../../implemented/process/2026-07-20-gui-testing-system.md) covers tier 1 (wire isomorphism in node), tier 2 (object-layer state machines), and a tier-3 smoke pair, but the keyless smoke (`apps/web/tests/smoke-fixture.e2e.ts`) drives `FixtureApiClient` behind `?fixture` — no host, no wire, no agent loop — while the full-chain smoke (`smoke-real.e2e.ts`) needs `DEEPSEEK_API_KEY` and a live model, so it is nondeterministic and self-skips in keyless CI. The snapshot philosophy of [docs/testing.md](../../../../docs/testing.md) — record once with a key, replay forever keyless, refresh on format churn — already covers the ACP, headless `stream-json`, and TUI transcript surfaces; the web surface is the one assembled product shape without it. The gap is exactly where the two confirmed GUI P0s hid: the wire carriage chain the fixture client short-circuits. + +## Proposal + +Add a keyless, deterministic browser e2e lane under `apps/web/tests/`, driven by recorded session-log fixtures replayed through `@deepseek-ai/dsh-llm-replay`, asserting the rendered accessibility tree plus in-process world state. No new package; no product-code change except (open question 1) an LLM composition knob. + +### Harness: `apps/web/tests/harness.ts` + +A plain shared-fixture module (the [testing-policy sanctioned shape](../../../../docs/testing.md)), not a package: the gate-worthy logic this lane needs — replay derivation, session parsing, log scrubbing, persistence — already lives in gated packages (`dsh-llm-replay`, `dsh-acp-snapshot`, `dsh-session-persistence-jsonl`); what remains is boot wiring and browser glue, and chromium-driving code cannot hold per-file 100% coverage on the browserless coverage runners. + +`launchWebHarness()` boots the real web assembly in-process from the exported production functions — `startHost({ boot: { persistenceRoot: , workspaceContext: false, cwd: } })`, `installLlmReplay(host.ctx, { file, childFiles, providers })`, `mountWebPlugins(host.ctx)`, `createHostWebPluginRegistry`, `startWebServer({ port: 0, distIndex, apiHandler: host.handler, webPlugins })` — and returns `{ baseUrl, host, workspaceCwd, close }`. This is the web analog of the TUI suite mounting the production bundle in-process ([TUI snapshots](../../implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)): the real entry boundary (`dsh web` bin arg-parsing and dist resolution) stays held by the existing keyless CLI smokes in `smoke-real.e2e.ts`, and the web surface has no `cordis.yml` to bypass — assembly is written in the app per the [GUI layering decision](../../implemented/architecture/2026-07-19-gui-layering-and-rpc-protocol.md). Replay runs in providers-catalog mode with a `contextWindow` (the TUI suite's `PROVIDERS` shape), never catch-all: with no adapter registered, catch-all mode would make `compact-basic`'s `resolveModelContext` throw into its post-step catch every step, spamming warnings and silently disabling the pressure path instead of proving it inert. + +`seedSession(host, fixtureText)` seeds cold sessions through the real persistence API — a throwaway `Context` mounting `SessionStore` + `SessionPersistenceJsonl` against the host's root, `create()` + `append()`, one `utimes` for deterministic sidebar order (the precedent is `examples/acp-agent/tests/semantic-checkpoint.snapshot.ts`) — never raw file writes, so the seeder needs no knowledge of bucket hashing, filename encoding, or compression. Seeds are validated at seed time (parseable, `seq`-contiguous, ending in `turn/end`) so fixture drift fails loud at the earliest resolvable point rather than as silently dropped frames in the client; a seed not ending in `turn/end` would be mutated by resume's crash repair. + +### Determinism rules + +The barrier stack, in order, for a prompted turn: (1) host-side `await agent.whenIdle()` under a timeout — the idle flip happens after both the `turn/end` append and the persistence flush, so one await covers turn completion and durability; (2) browser settled poll — streaming node detached, composer restored, final text visible; (3) log harvest only after `host.dispose()`. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync), and file polling is banned (slow on NFS, superseded by `whenIdle`). For history-open scenarios the barrier is a poll for the last expected message's text, then a poll-until-equal aria capture. `networkidle` is banned outright — it never resolves while an SSE stream is open. + +No single-shot transient-DOM assertions: every hop from replay yield to React commit can coalesce chunks, so sampling `[data-streaming]` is a race by construction. Streaming incrementality is asserted from the persisted `assistant/chunk` events (model-visible ⟺ logged makes the log the authoritative proof), optionally corroborated by an in-page MutationObserver latch armed before send — observers cannot miss a commit; polls can. `dsh-llm-replay` gains an opt-in `paceMs` config field (default absent = today's instant yield) as a realism knob so the browser observes genuinely incremental SSE; correctness never leans on the pace. + +Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Harness `close()` asserts every replay script was fully consumed (all scripts bound, every cursor at end), converting silent underruns and shifted bindings into crisp diagnostics; this is a small additive stats handle on `installLlmReplay`. No vitest retry on the lane — a retried-green race is a flake deferred, and only the chromium launch itself may retry inside the harness, logged. One chromium per file, fresh browser context per scenario, one host per scenario; viewport pinned; selectors anchor on roles, `data-*` attributes, and visible text only. + +### Expected outputs + +One committed golden per scenario: a normalized `ariaSnapshot()` of the conversation region only (`ui.expected.md`) — uuid/cwd/duration tokens normalized, sidebar and other time-bearing chrome structurally excluded, captured poll-until-equal at the settled milestone. The accessibility tree is the mechanization of the client rule "assert what the user would see, never class names": it survives CSS-module hash churn, styling rewrites, and DOM restructuring, and a wholesale component rewrite refreshes it keylessly. Alongside the golden, three or four targeted role/text anchor assertions (heading level, `pre code` content, tool-row accessible name) keep green anchors under a semantics-preserving rewrite so a churned golden diff is reviewable against surviving anchors. World-state assertions ride `host.ctx` session events inline (which tools ran, `turn/end` completed, no error) instead of a second committed log golden: the persisted-log surface is already pinned by the ACP/headless/TUI suites through the same loop and persistence plugins, and re-pinning it here would double refresh cost for no new regression class. `playwright` gets pinned exactly in `apps/web/package.json` — the aria format is Playwright-owned, the one snapshot format in this repo we do not own, so version bumps must be deliberate bump-and-refresh commits. + +### Modes and fixtures + +`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless), as inline branches in the specs — the TUI suite's shape, not a suite-factory: at two scenarios the acp-snapshot factory machinery (scenario tables, pinning classes, Windows sidecars, packed-row stabilization) has no owner, and the genuinely shared parts are already exported (`scrubRequestHeaders`, `normalizeSessionLog`, `parseSessionLog`, `installLlmReplay`). Each scenario script splits into drive steps (type, send, `whenIdle`-generic waits — run in all modes, never waiting on model-content selectors) and interaction/assertion steps (expand reasoning, aria capture — replay/refresh only), so record mode cannot hang on a live model answering with a different tool count. Record = drive + harvest the in-memory `session.header` + `session.events` (the TUI `rawSessionLog` shape — no file decompression, so `bootHost` needs no compression knob) + scrub via `scrubRequestHeaders` + a mandatory keyless refresh to regenerate `ui.expected.md`. Web fixtures scrub headers everywhere and pin nowhere, matching the TUI precedent; whether the web surface must instead own a header-class pin per the [pinned-header discipline](../../implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md) is open question 3. Seeds are recorded fixtures under the same inventory and refresh discipline as replay fixtures, never hand-authored one-offs, so `DSH_SNAPSHOT=refresh` heals every committed surface after intentional shape churn and only `assistant/chunk`-shape churn escalates to re-record. A TUI-style `afterAll` fixture guard holds the inventory closed (expected files present, every fixture scrub-fixed-point, no orphan directories). + +### Demo scenarios + +1. **`fresh-round-trip`** — new session, prompt, replay streams reasoning + markdown + a `bash` tool call that really executes (`echo` in the temp workspace) + final text. Asserts settled markdown semantics (heading, code block), the tool card row, composer restore, the aria golden, and inline world state (bash `tool/call` + completed `turn/end` in the session events). The keyless version of the with-key W5 flows. +2. **`seeded-history`** — a recorded session seeded cold; the sidebar lists it, opening it renders tool cards and collapsible reasoning purely from the log. This exercises the implicit cold-resume attach (`session.history` resumes via `agentFor`), cold summaries, history pagination views, and the client fold of historical events — the surface nothing else covers — with zero model calls, so no replay-binding constraints at all. A follow-up-prompt-after-resume scenario is deliberately deferred until the history/live stitch path changes or regresses. + +### Lane wiring and CI stance + +The lane rides `vitest.web.config.ts` (`pnpm run test:web`, serial), which stays gate-exempt exactly as its header comment records. Adding chromium to CI would reverse the "no browser infrastructure in CI" premise recorded in the [GUI testing system note](../../implemented/process/2026-07-20-gui-testing-system.md) and therefore requires its own Agent Note cross-linked from that note, staged as: non-required CI job first, promotion criteria measured (consecutive green runs, wall time, zero-retry flake budget, browser cache strategy on the enterprise runners, whether the runner images carry the chromium system libraries). Deferred out of this proposal; a `TODO(ci-browser)` marks the seam. Scenarios are `posixOnly` initially. Docs updated in the same implementation PR: the [testing policy](../../../../docs/testing.md) names `apps/web/tests/snapshots/` as the web surface's snapshot home with its divergent `DSH_SNAPSHOT=… pnpm run test:web` commands, the GUI testing note's tier map gains the lane (and drops its stale references to the deleted `missions/scripts/verify-*` files), `packages/client/AGENTS.md`'s check ladder mentions it, and the `dsh-acp-snapshot` README's "ACP-specific by design" sentence is corrected — this lane is the third consumer of its normalizers. + +### Open questions + +1. **LLM seam.** Two viable shapes. (a) `BootHostOptions.llm?: 'deepseek' | false` — an assembly toggle in the one module that owns assembly, matching the existing `workspaceContext: Config | false` shape and the reserved-knob sentence in `start.ts`; `llm: false` mounts no adapter, the harness fills the open seam on `host.ctx`, misuse fails loud at the first stream with `NO_ADAPTER`, and keyless boots stop needing any key. (b) Zero product change — a placeholder `DEEPSEEK_API_KEY` env var satisfies `llm-deepseek`'s load-time presence check (twice-precedented in-tree) and replay intercepts ahead of the mounted adapter. (a) is cleaner semantics and honest keylessness at the cost of a test-motivated product field; (b) is free but satisfies a fail-loud check with a lie and leaves a dead adapter mounted. Recommendation: (a), shaped minimal. +2. **Loader-izing `dsh web`.** Making the web host `cordis.yml`-driven like every example would give the ACP-style `cordis.snapshot.yml` replay overlay for free and align with "everything is a plugin", but it reverses the settled "assembly is written in the app" ruling and is a product-architecture decision on its own merits — its own proposal if wanted; this lane does not need it. +3. **Header-class pin.** Strict reading of the pinned-header discipline wants one web scenario pinning `bootHost`'s composed prompt + tool schemas (a header class no ACP scenario covers); the TUI precedent scrubs everywhere and pins nowhere. Cheap middle: pin sidecars on `fresh-round-trip` at record time. Recommendation: follow TUI now (scrub-only), revisit when the web assembly's header diverges further from the repl composition it mirrors. +4. **Golden breadth.** Full conversation-region aria golden (adopted above) versus targeted assertions only. The golden is the "assembled transcript" duty for user-visible changes; the cost is a keyless refresh on every component rewrite. Recommendation: keep the golden + anchors. +5. **Client settled signal.** A `data-dsh-busy` attribute derived from the object layer's pending-RPC/active-stream state would replace multi-condition settled polls with one selector. Presentation-plane observability, no session-log leak — but the current polls suffice for two scenarios. Recommendation: defer until a settled-poll flake actually appears. + +## Prior art + +Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot + AI SDK, lobe-chat, open-webui, OpenHands, Chainlit, continue, cline, langfuse, gradio/streamlit; Playwright HAR/route, MSW, Polly/nock, WireMock, aimock). The dominant proven architecture for apps that own their backend is an in-process fake/replay model behind the real backend seam with everything downstream real (LibreChat's `LIBRECHAT_TEST_RUN_HOOK` fake model; ai-chatbot's `MockLanguageModelV3` + `simulateReadableStream`; continue's scripted mock provider classes) — which is what `dsh-llm-replay` already is. Browser-level SSE interception cannot exercise incremental rendering (`route.fulfill` delivers the whole body at once; playwright#33564) and leaves the server SSE stack untested, so projects use it only for edge cases. Chunk pacing as a fixture parameter recurs everywhere (LibreChat 10ms default with slow profiles; ai-chatbot 500ms); real models in CI rot (open-webui's suite grew 120-second timeouts, was disabled, then deleted); sessions are seeded at the persistence layer with controlled timestamps (LibreChat inserts backdated Mongo documents; langfuse seeds its DB). No surveyed project replays a recorded agent-event log through the real backend for UI tests — the closest are provider-level recorded fixtures (aimock) and frontend-level socket history emission (OpenHands MSW) — so the session-log-as-fixture design goes one step beyond prior art along the axis this repo's model-visible ⟺ logged invariant makes natural. + +## Alternatives considered + +**Browser-network SSE interception (`page.route`).** Rejected: `route.fulfill` cannot stream, so incremental token rendering is unexercisable and the server-side SSE/backpressure/close path — where both confirmed P0s hid — goes untested. + +**Mock HTTP provider at `DEEPSEEK_BASE_URL`.** Rejected as the lane's mechanism (kept for the one existing workspace-probe smoke): fixtures become hand-authored OpenAI SSE byte scripts, a second fixture format that drifts from the session-log format the rest of the repo records and replays; the adapter's real HTTP path is with-key e2e's job. + +**Growing the `?fixture` client.** Rejected: tier separation — `FixtureApiClient` exists to test the client shell without a server; everything below the client API seam stays untested by construction. + +**A `packages/support/web-snapshot` package with a `defineWebSnapshotSuite` factory.** Rejected for now: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and at two scenarios the factory generalizes from one consumer while the genuinely shared logic already lives exported in `dsh-llm-replay`/`dsh-acp-snapshot`. Re-entry trigger: a second web-shaped consumer or ≥6 scenarios with demonstrably drifting inline branches; the package boundary would then be drawn browser-free. + +**A committed normalized-session-log golden as a second expected surface.** Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers, against the tier discipline. Inline world-state assertions on `host.ctx` events keep the world-verification duty. + +**Spawning the `dsh web` bin with a `DSH_SNAPSHOT` replay branch.** Rejected for now: it needs a test-mode branch plus env plumbing in the product bin where the in-process route uses exported production functions with zero product change; the bin's thin glue is covered by the keyless CLI smokes. Becomes free if the web host is ever Loader-ized (open question 2). + +**Changing the wire protocol for testability.** Rejected: the contract already has a first-class keyless isomorphic seam (`InProcessApiClient(toFetchHandler(api))`), the per-event unbatched SSE is exactly what makes replay observable in a browser, and testing a wire we no longer ship would invert the tier's purpose. + +**Real-model browser tests as the keyless lane.** Rejected: nondeterministic by construction; the surveyed cautionary case (open-webui) grew unbounded timeouts and was deleted. The with-key W5 smoke stays as the live-model complement. + +## Acceptance criteria + +- `pnpm run test:web` keyless passes the two scenarios deterministically (no vitest retry), alongside the existing smoke pair, on a checkout with built client bundles and the frontend dist. +- Replay asserts: aria golden equality at the settled milestone, anchor role/text assertions, inline world-state event assertions, zero pageerrors, zero connection-loss/gap-repair console warnings, all replay scripts fully consumed at teardown. +- `DSH_SNAPSHOT=record` with a key re-records `fresh-round-trip` (drive steps only), rewrites its `session.jsonl` scrubbed, and a follow-up `DSH_SNAPSHOT=refresh` regenerates `ui.expected.md` keylessly; `refresh` alone heals goldens after intentional non-chunk shape churn. +- The seeded scenario renders history through the real cold-resume path with zero model calls and leaves the seed fixture byte-identical (closedness validated at seed time). +- Fixture guard holds the snapshot inventory closed; failure produces a bundle under `.artifacts/` (screenshot, console, pageerrors, persistence copy, actual-vs-expected aria). +- Docs land in the same PR: testing.md web-lane entry, GUI testing note tier map + stale verify-script cleanup, client AGENTS.md ladder, acp-snapshot README correction, this note moved to `implemented/` rewritten in present tense. + +## Risks + +- **The aria format is Playwright-owned** — the one committed snapshot format the repo does not control; a version bump can churn every golden. Mitigated by an exact version pin in `apps/web` and a documented bump-and-refresh procedure; residual risk accepted. +- **Replay's first-call-order binding** stays fragile under concurrent browser-driven sessions; the lane constrains scenarios to one prompting session each (the seeded scenario prompts none), and the teardown consumption assertion turns violations into diagnostics rather than surreal transcripts. +- **`compact-basic` shares the session's replay cursor** — a pressure-triggered summarize would consume a script entry; inert for small fixtures under the 128k catalog window, and the consumption assertion catches it if a fixture ever grows past the threshold. +- **CI remains browserless for now**, so the lane guards regressions only where it is run (locally and in any future non-required job) until the CI reversal is separately decided; the runner images' chromium-library situation is unverified. +- **Record-mode nondeterminism** is contained but not eliminated by the drive/assert split: a live model may still produce a transcript whose replay violates a scenario's assertions, requiring prompt tuning at record time (bounded by terse prompts and a chunk-count warning in record mode). +- **jsdom-lane overlap**: component-level rendering is already covered per-plugin; this lane must stay at assembled-transcript altitude (whole-region golden + anchors) or it starts re-testing tier 2 and paying double maintenance. From 9ef0193dd56a56ca4792c12f8296b035201dee85 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 24 Jul 2026 19:08:10 +0800 Subject: [PATCH 02/11] feat(host-runtime,llm-replay): keyless llm seam + replay pacing/consumption handle MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BootHostOptions.llm: 'deepseek' | false — false mounts no adapter, boots keyless, and leaves the llm capability seam open for the embedder to fill on RunningHost.ctx (now the third sanctioned ctx use, JSDoc + README amended); an unfilled seam fails loud with NO_ADAPTER at the first stream. dsh-llm-replay grows two additive surfaces for the web browser e2e lane: paceMs (validated per-chunk delay so a real transport shows incremental delivery; abort during a pace wait cancels promptly) and a ReplayHandle return — dispose() plus assertConsumed(), the teardown check that every recorded script bound and drained, converting silent fixture underruns into diagnostics. Existing callers updated; config catalog regenerated. --- docs/config-catalog.md | 4 +- packages/host/runtime/README.md | 3 +- packages/host/runtime/src/boot.ts | 11 ++- packages/host/runtime/src/start.ts | 9 +- .../host/runtime/tests/host-runtime.spec.ts | 36 ++++++++ packages/support/llm-replay/README.md | 5 +- packages/support/llm-replay/src/index.ts | 88 +++++++++++++++++-- .../llm-replay/tests/llm-replay.spec.ts | 63 ++++++++++++- 8 files changed, 201 insertions(+), 18 deletions(-) diff --git a/docs/config-catalog.md b/docs/config-catalog.md index 49a0a1510f..c06fef3d4d 100644 --- a/docs/config-catalog.md +++ b/docs/config-catalog.md @@ -616,6 +616,8 @@ export interface Config { childFiles?: string[] /** Optional replay-only provider catalog; absent or empty selects catch-all waterfall replay. */ providers?: ReplayProviderConfig[] + /** Optional per-chunk pacing delay in ms (see {@link ReplayConfig.paceMs}); absent keeps burst yield. */ + paceMs?: number } /** One provider route exposed by the replay adapter. */ @@ -641,7 +643,7 @@ export interface ReplayModelConfig { } ``` -Source: [`packages/support/llm-replay/src/index.ts:387`](../packages/support/llm-replay/src/index.ts) +Source: [`packages/support/llm-replay/src/index.ts:453`](../packages/support/llm-replay/src/index.ts) ## `@deepseek-ai/dsh-llm-retry` diff --git a/packages/host/runtime/README.md b/packages/host/runtime/README.md index f7cf7d9de8..4384f591a1 100644 --- a/packages/host/runtime/README.md +++ b/packages/host/runtime/README.md @@ -2,7 +2,7 @@ Host runtime assembly for `dsh`: `bootHost` composes the core plugin spine (LLM service + DeepSeek adapter, sessions with JSONL persistence and immediate fallback titles, optional first-message model summaries, system prompt, tools, agents, agent loop, workspace instructions, local bash, and the provider-neutral user-interaction service), `createApiProxy` implements the [`dsh-host-apiproxy`](../apiproxy/README.md) contract over that composition, and `startHost` is the one-step shell seam returning `{ api, handler, defaults, ctx, dispose }`. -Which plugins mount and with what defaults is decided only here — shells must not `ctx.plugin` to alter the assembly. `RunningHost.ctx` is a formal seam with exactly two sanctioned uses: mounting protocol front-door plugins (e.g. a future `dsh acp`) and headless session-event subscription; consuming clients must not bypass `api` through it. +Which plugins mount and with what defaults is decided only here — shells must not `ctx.plugin` to alter the assembly. `RunningHost.ctx` is a formal seam with exactly three sanctioned uses: mounting protocol front-door plugins (e.g. a future `dsh acp`), headless session-event subscription, and filling a capability seam the boot options deliberately left open (`llm: false` → the embedder installs its own LLM backend, e.g. the keyless web e2e harness's replay); consuming clients must not bypass `api` through it. ## Configuration @@ -10,6 +10,7 @@ Which plugins mount and with what defaults is decided only here — shells must |---|---:|---| | `persistenceRoot` | (required) | Root directory for JSONL session persistence. | | `workspaceContext` | (required) | [`AGENTS.md`/`CLAUDE.md` loader](../../context/workspace-context/README.md) config with an explicit `maxBytes`, or `false` to disable it. | +| `llm` | `'deepseek'` | LLM adapter selection: `'deepseek'` mounts the DeepSeek adapter (API key required at load); `false` mounts none, boots keyless, and leaves the `llm` seam open for the embedder — an unfilled seam fails loud with `NO_ADAPTER` at the first stream. | | `provider` | `'deepseek'` | Default provider route injected as agentOptions on create/resume and reported by `host.describe`. | | `model` | `'deepseek-v4-flash'` | Default model id, same single source as `provider`. | | `cwd` | `process.cwd()` | Default project directory for a session whose create request omits `cwd`. | diff --git a/packages/host/runtime/src/boot.ts b/packages/host/runtime/src/boot.ts index c0960ab678..e2f24a98ac 100644 --- a/packages/host/runtime/src/boot.ts +++ b/packages/host/runtime/src/boot.ts @@ -65,6 +65,15 @@ export interface BootHostOptions { persistenceRoot: string /** Workspace-instruction byte budget/config, or false to disable AGENTS.md/CLAUDE.md loading. */ workspaceContext: workspaceContext.Config | false + /** + * LLM adapter selection: `'deepseek'` (default) mounts the DeepSeek adapter + * (requires an API key at load), `false` mounts no adapter and leaves the + * `llm` capability seam open for the embedder to fill on the returned ctx + * (e.g. the keyless web e2e harness installing a replay backend). With + * `false` and nothing filled, the first stream fails loud with NO_ADAPTER — + * the earliest resolvable point for an open capability seam. + */ + llm?: 'deepseek' | false /** Default provider route for created/resumed agents (defaults to 'deepseek', the only adapter bootHost registers). */ provider?: string /** Default model id (defaults to 'deepseek-v4-flash', matching the demos). */ @@ -129,7 +138,7 @@ export async function bootHost(options: BootHostOptions): Promise { await ctx.plugin(AgentRegistry) await ctx.plugin(TaskService) await ctx.plugin(AgentLoop, { agents: [] }) - await ctx.plugin(LlmDeepSeek, {}) + if (options.llm !== false) await ctx.plugin(LlmDeepSeek, {}) await ctx.plugin(SessionPersistenceJsonl, { root: options.persistenceRoot }) await ctx.plugin(LocalBashExecutor, {}) // Tool suite mirroring the demo:repl composition (repl-agent/cordis.yml + diff --git a/packages/host/runtime/src/start.ts b/packages/host/runtime/src/start.ts index 94e5f22da1..c389c0de68 100644 --- a/packages/host/runtime/src/start.ts +++ b/packages/host/runtime/src/start.ts @@ -34,9 +34,12 @@ export interface RunningHost { /** * Root context — a formal seam, not an escape hatch: (1) the mount point for * protocol front-door plugins (`dsh acp` = startHost() → ctx.plugin(uiAcp, config)); - * (2) headless session-event subscription. Discipline: consuming clients must - * not bypass `api` through ctx; shells must not ctx.plugin to alter the - * assembly (mounting a front door is the shell's own shape, not an assembly change). + * (2) headless session-event subscription; (3) filling a capability seam the + * boot options deliberately left open (`llm: false` → the embedder installs + * its own LLM backend, e.g. keyless replay). Discipline: consuming clients + * must not bypass `api` through ctx; shells must not ctx.plugin to alter the + * assembly (mounting a front door or filling an explicitly-open seam is the + * shell's own shape, not an assembly change). */ ctx: Context /** Single shutdown exit (ctx.fiber.dispose()). Idempotent: a second call returns the same promise. */ diff --git a/packages/host/runtime/tests/host-runtime.spec.ts b/packages/host/runtime/tests/host-runtime.spec.ts index 4058fdfe2e..92d6534831 100644 --- a/packages/host/runtime/tests/host-runtime.spec.ts +++ b/packages/host/runtime/tests/host-runtime.spec.ts @@ -205,6 +205,42 @@ describe('bootHost / startHost', () => { expect((await ctx.sessionTitle.refresh(agent.session))?.source).toEqual({ kind: 'fallback' }) expect(agent.session.events.some(event => event.type === 'session/title-llm-request')).toBe(false) }) + + it('llm: false boots keyless with no adapter and fails loud at the first stream', async () => { + vi.stubEnv('DEEPSEEK_API_KEY', '') + const handle: HostHandle = await bootHost({ + persistenceRoot: mkdtempSync(join(tmpdir(), 'dsh-boot-keyless-')), + workspaceContext: false, + llm: false, + }) + // The seam is open: nothing routes 'deepseek', so misuse surfaces at the + // earliest resolvable point instead of silently doing provider I/O. + await expect(async () => { + for await (const chunk of handle.ctx.llm.stream({ + provider: 'deepseek', + model: 'deepseek-v4-flash', + messages: [], + })) void chunk + }).rejects.toThrow(/NO_ADAPTER|no adapter/i) + // The embedder can fill the open seam on the returned ctx (the sanctioned + // RunningHost.ctx use) and streams route through the filled adapter. + class ProbeAdapter extends LlmAdapter { + async * stream(): AsyncIterable { + yield * textResponse('keyless-ok') + } + } + handle.ctx.llm.registerAdapter(['deepseek'], new ProbeAdapter()) + const collected: string[] = [] + for await (const chunk of handle.ctx.llm.stream({ + provider: 'deepseek', + model: 'deepseek-v4-flash', + messages: [], + })) { + if (chunk.type === 'text-delta') collected.push(chunk.text) + } + expect(collected.join('')).toBe('keyless-ok') + await handle.dispose() + }) }) describe('host.describe', () => { diff --git a/packages/support/llm-replay/README.md b/packages/support/llm-replay/README.md index 0d89d4337d..e655aa4077 100644 --- a/packages/support/llm-replay/README.md +++ b/packages/support/llm-replay/README.md @@ -24,6 +24,7 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s | `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | Optional path to a `ReplayEntry[]` sidecar that replaces the PRIMARY session's derived script. | | `childFiles` | string[] | `$DSH_SNAPSHOT_CHILD_FILES` (path-delimited) | Recorded subagent child-session logs for a nested scenario; empty for a single-session scenario. | | `providers` | `ReplayProviderConfig[]` | — | Optional replay-only provider and model catalog. Each model may publish `contextWindow`; configured routes dispatch through the replay adapter and never perform provider I/O. | +| `paceMs` | number | — (burst) | Optional per-chunk delay in ms so downstream transports (e.g. the web SSE mux observed by a real browser) see genuinely incremental delivery. A realism knob only — tests must not depend on it for correctness. Non-negative integer; abort during a pace wait cancels the stream promptly. | ```yaml - id: llm-replay @@ -43,11 +44,11 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s ## Exports -- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns the disposer (HMR safety). Use this in tests to drive replay without the Loader or env vars. +- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns a `ReplayHandle` (`dispose()` for HMR safety plus `assertConsumed()`, the teardown check that every recorded script bound to a live session and every bound cursor drained — turning a scenario that silently drove fewer model calls than recorded into a crisp diagnostic). Use this in tests to drive replay without the Loader or env vars. - `loadSessionScripts(config)` — resolve the ordered `SessionScript[]` (primary + children) for a scenario, ready to bind to live sessions in first-call order. - `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the PRIMARY session only (sidecar override if present, else derived from the JSONL; fail-loud if the fixture is missing). - `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` — the pure helpers that turn a recorded session log into a script and read its header `id`/`createdAt`. A derived group must end in a `finish` chunk; a group without one is the fingerprint of a thrown `stream()` and must instead be expressed via an override sidecar. -- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `Config`. +- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`. ## Plugin export shape diff --git a/packages/support/llm-replay/src/index.ts b/packages/support/llm-replay/src/index.ts index ca4511db8a..ecdd5b0c3b 100644 --- a/packages/support/llm-replay/src/index.ts +++ b/packages/support/llm-replay/src/index.ts @@ -74,6 +74,32 @@ export interface ReplayConfig { * by tests that do not need discovery. */ providers?: ReplayProviderConfig[] + /** + * Optional per-chunk pacing delay in milliseconds: each replayed chunk waits + * this long before yielding, so a downstream transport (e.g. the web SSE + * mux observed by a browser) sees genuinely incremental delivery. A realism + * knob only — correctness must never depend on it. Absent or `0` keeps + * today's synchronous burst yield. Must be a non-negative finite integer; + * aborting mid-wait cancels the stream like any other abort. + */ + paceMs?: number +} + +/** + * Handle returned by {@link installLlmReplay}: removal plus the end-of-run + * consumption check that turns silent fixture underruns (a scenario that + * issued fewer calls than recorded, or never bound a recorded child script) + * into a crisp diagnostic at teardown. + */ +export interface ReplayHandle { + /** Remove the registered adapter or waterfall listener (HMR safety). Freestanding closure — safe to destructure. */ + dispose(this: void): void + /** + * Throw unless every recorded script was bound to a live session and every + * bound cursor consumed its full entry list. Call at scenario teardown. + * Freestanding closure — safe to destructure. + */ + assertConsumed(this: void): void } /** @@ -277,12 +303,32 @@ class ReplayAdapter extends LlmAdapter { } } +/** + * Wait `paceMs` between chunk yields, aborting the wait (and the stream) the + * moment the signal fires — a paced replay must cancel as promptly as a burst + * one. + */ +function paceDelay(paceMs: number, signal: AbortSignal | undefined): Promise { + return new Promise((resolve, reject) => { + const timer = setTimeout(() => { + signal?.removeEventListener('abort', onAbort) + resolve() + }, paceMs) + const onAbort = (): void => { + clearTimeout(timer) + reject(new Error('aborted')) + } + signal?.addEventListener('abort', onAbort, { once: true }) + }) +} + /** Yield a recorded stream back, honoring abort like a real adapter. */ -async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined): AsyncIterable { +async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined, paceMs: number): AsyncIterable { switch (entry.kind) { case 'chunks': for (const chunk of entry.chunks) { if (signal?.aborted) throw new Error('aborted') + if (paceMs > 0) await paceDelay(paceMs, signal) yield chunk } return @@ -293,6 +339,7 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined) // mid-stream STREAM_CLOSED after partial chunks). for (const chunk of entry.chunks) { if (signal?.aborted) throw new Error('aborted') + if (paceMs > 0) await paceDelay(paceMs, signal) yield chunk } throw new LlmError(entry.message, entry.code) @@ -319,14 +366,17 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined) * next ordered recorded script, then advances its own cursor synchronously at * invocation time; calls without `sessionId` share one anonymous session. A * non-empty provider catalog registers a routed replay adapter; otherwise a - * catch-all waterfall intercepts requests. Returns the effect disposer for - * HMR-safe removal. + * catch-all waterfall intercepts requests. * * @param ctx - the context whose LLM service receives the replay route or waterfall. * @param config - the resolved fixture paths (env-var defaulting is `apply`'s job). - * @returns the disposer that removes the registered adapter or listener. + * @returns the {@link ReplayHandle} carrying the disposer and the teardown consumption check. */ -export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void { +export function installLlmReplay(ctx: Context, config: ReplayConfig): ReplayHandle { + const paceMs = config.paceMs ?? 0 + if (!Number.isInteger(paceMs) || paceMs < 0) { + throw new Error(`llm-replay: paceMs must be a non-negative integer, got ${String(config.paceMs)}`) + } const scripts = loadSessionScripts(config) // Live-session → its bound script + cursor. A new live session id claims the // next not-yet-bound script (scripts are in bind order); `nextScript` is the @@ -370,14 +420,31 @@ export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void + `but its script has only ${boundState.entries.length}; re-record the scenario`, ) } - yield* replayEntry(entry, options.signal) + yield* replayEntry(entry, options.signal, paceMs) })() } const providers = config.providers ?? [] - if (providers.length > 0) { - return ctx.llm.registerAdapter(providers.map(provider => provider.id), new ReplayAdapter(providers, replay)) + const dispose = providers.length > 0 + ? ctx.llm.registerAdapter(providers.map(provider => provider.id), new ReplayAdapter(providers, replay)) + : ctx.on('llm/stream', (options: GenerateOptions, _next) => replay(options)) + return { + dispose, + assertConsumed(): void { + const problems: string[] = [] + if (nextScript < scripts.length) { + problems.push(`${scripts.length - nextScript} recorded script(s) never bound to a live session`) + } + for (const [key, state] of bound) { + if (state.cursor < state.entries.length) { + const who = key === ANON ? 'the anonymous session' : `session ${key}` + problems.push(`${who} consumed ${state.cursor}/${state.entries.length} recorded call(s)`) + } + } + if (problems.length > 0) { + throw new Error(`llm-replay: fixture not fully consumed — ${problems.join('; ')}; the scenario drove fewer model calls than recorded`) + } + }, } - return ctx.on('llm/stream', (options: GenerateOptions, _next) => replay(options)) } export const name = 'llm-replay' @@ -397,6 +464,8 @@ export interface Config { childFiles?: string[] /** Optional replay-only provider catalog; absent or empty selects catch-all waterfall replay. */ providers?: ReplayProviderConfig[] + /** Optional per-chunk pacing delay in ms (see {@link ReplayConfig.paceMs}); absent keeps burst yield. */ + paceMs?: number } export function apply(ctx: Context, config: Config = {}): void { @@ -413,5 +482,6 @@ export function apply(ctx: Context, config: Config = {}): void { ...overrideFile !== undefined && overrideFile.length > 0 ? { overrideFile } : {}, ...childFiles.length > 0 ? { childFiles } : {}, ...config.providers !== undefined ? { providers: config.providers } : {}, + ...config.paceMs !== undefined ? { paceMs: config.paceMs } : {}, }) } diff --git a/packages/support/llm-replay/tests/llm-replay.spec.ts b/packages/support/llm-replay/tests/llm-replay.spec.ts index 14086db27f..b6b03aacbf 100644 --- a/packages/support/llm-replay/tests/llm-replay.spec.ts +++ b/packages/support/llm-replay/tests/llm-replay.spec.ts @@ -234,7 +234,7 @@ describe('installLlmReplay (through the real LlmService)', () => { writeLog(TEXT_CHUNKS) const ctx = new Context() await ctx.plugin(LlmService) - const dispose = installLlmReplay(ctx, { + const { dispose } = installLlmReplay(ctx, { file, providers: [ { @@ -429,6 +429,67 @@ describe('installLlmReplay (through the real LlmService)', () => { await iterator.next() await expect(iterator.next()).rejects.toThrow('aborted') }) + + it('rejects a paceMs that is not a non-negative integer', async () => { + writeLog(TEXT_CHUNKS) + const ctx = new Context() + await ctx.plugin(LlmService) + expect(() => installLlmReplay(ctx, { file, paceMs: -1 })).toThrow(/paceMs/) + expect(() => installLlmReplay(ctx, { file, paceMs: 1.5 })).toThrow(/paceMs/) + }) + + it('paces chunk yields when paceMs is set (each chunk waits at least the pace)', async () => { + writeLog(TEXT_CHUNKS) + const ctx = new Context() + await ctx.plugin(LlmService) + installLlmReplay(ctx, { file, paceMs: 10 }) + const started = performance.now() + const chunks = await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })) + expect(chunks).toEqual(TEXT_CHUNKS) + // N chunks × 10ms; allow generous scheduling slack, assert the floor only. + expect(performance.now() - started).toBeGreaterThanOrEqual(TEXT_CHUNKS.length * 10 - 5) + }) + + it('aborting DURING a pace wait cancels the stream promptly', async () => { + writeLog(TEXT_CHUNKS) + const ctx = new Context() + await ctx.plugin(LlmService) + installLlmReplay(ctx, { file, paceMs: 60_000 }) + const controller = new AbortController() + const pending = drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })) + // Let the generator park inside the pace timer, then abort — the reject + // must come from the abort listener, not the (distant) timer. + await new Promise(r => setImmediate(r)) + controller.abort() + await expect(pending).rejects.toThrow('aborted') + }) + + it('assertConsumed passes only after every recorded call replayed', async () => { + writeLog(TEXT_CHUNKS, TEXT_CHUNKS) + const ctx = new Context() + await ctx.plugin(LlmService) + const handle = installLlmReplay(ctx, { file }) + await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })) + // One of two recorded calls consumed — the underrun must name the gap. + expect(() => { handle.assertConsumed() }).toThrow(/consumed 1\/2 recorded call/) + await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })) + expect(() => { handle.assertConsumed() }).not.toThrow() + }) + + it('assertConsumed reports recorded scripts no live session ever bound', async () => { + writeLog(TEXT_CHUNKS) + const childFile = join(dir, 'session.1.jsonl') + writeFileSync(childFile, sessionJsonl( + TEXT_CHUNKS.map((chunk, i) => chunkEvent(i + 1, 1, 1, chunk)), + { id: 'child', createdAt: 10 }, + ), 'utf8') + const ctx = new Context() + await ctx.plugin(LlmService) + const handle = installLlmReplay(ctx, { file, childFiles: [childFile] }) + await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], sessionId: 'live-parent' as NonNullable })) + // The child script never bound: the scenario drove fewer sessions than recorded. + expect(() => { handle.assertConsumed() }).toThrow(/1 recorded script\(s\) never bound/) + }) }) describe('parseSessionHeader', () => { From 46b9a91e556437946b3356d2e5d6eed241cd57ef Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 24 Jul 2026 19:49:02 +0800 Subject: [PATCH 03/11] =?UTF-8?q?test(web):=20keyless=20browser=20e2e=20la?= =?UTF-8?q?ne=20=E2=80=94=20replayed=20round=20trip=20+=20seeded=20cold=20?= =?UTF-8?q?resume?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit apps/web/tests/harness.ts boots the real web assembly in-process (startHost llm:false -> installLlmReplay providers-mode -> mountWebPlugins -> startWebServer) under DSH_SNAPSHOT replay/record/refresh. Barrier stack: in-process turn/end -> agent.whenIdle (covers the persistence flush) -> browser settled-poll. Seeding goes through the real persistence API (semantic-checkpoint precedent); record harvests fixtures from live session memory and tokenizes {{sessionId}}/{{cwd}}; refresh is the sole golden writer. Console tripwires fail scenarios on reconnect/gap-repair self-healing; harness close asserts full replay-fixture consumption. Scenarios, each with fixtures recorded against THIS assembly via a live model run: replay-round-trip (real composer -> real bash echo -> settled markdown + aria golden + world-state event asserts) and seeded-history (cold sidebar list -> implicit resume on open -> history tool cards from the log, zero model calls). apps/web/tests are host-plane programs: excluded from the client-registered apps/web project, included in tsconfig.host.json (one program cannot hold both Context merge sides). --- apps/web/tests/harness.ts | 423 ++++++++++++++++++ apps/web/tests/replay-round-trip.e2e.ts | 109 +++++ apps/web/tests/seeded-history.e2e.ts | 105 +++++ .../snapshots/fresh-round-trip/session.jsonl | 97 ++++ .../snapshots/fresh-round-trip/ui.expected.md | 31 ++ .../tests/snapshots/seeded-history/seed.jsonl | 112 +++++ .../snapshots/seeded-history/ui.expected.md | 36 ++ apps/web/tsconfig.json | 9 + tsconfig.host.json | 4 + 9 files changed, 926 insertions(+) create mode 100644 apps/web/tests/harness.ts create mode 100644 apps/web/tests/replay-round-trip.e2e.ts create mode 100644 apps/web/tests/seeded-history.e2e.ts create mode 100644 apps/web/tests/snapshots/fresh-round-trip/session.jsonl create mode 100644 apps/web/tests/snapshots/fresh-round-trip/ui.expected.md create mode 100644 apps/web/tests/snapshots/seeded-history/seed.jsonl create mode 100644 apps/web/tests/snapshots/seeded-history/ui.expected.md diff --git a/apps/web/tests/harness.ts b/apps/web/tests/harness.ts new file mode 100644 index 0000000000..6f885da0c8 --- /dev/null +++ b/apps/web/tests/harness.ts @@ -0,0 +1,423 @@ +// Shared harness for the keyless browser e2e lane (Agent Note: +// .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md). +// Boots the REAL web assembly in-process from the exported production +// functions — startHost (bootHost spine) + mountWebPlugins + registry + +// startWebServer — so a real chromium exercises the real HTTP/SSE wire, +// apiproxy, agent loop, tools, and persistence. Modes ride $DSH_SNAPSHOT: +// replay (default, keyless: `llm: false` + dsh-llm-replay in providers mode), +// record (real DeepSeek adapter + key, harvests fixtures from live session +// memory), refresh (keyless replay that rewrites the committed goldens). +// +// Assembly divergence from `dsh web` (apps/cli/src/web.ts), deliberate: the +// shipped shell opts into sessionTitleLlm, whose fire-and-forget title call +// shares the session's replay cursor — nondeterministic ordering against the +// loop's own calls — so this lane keeps bootHost's disabled default and +// sidebar titles come from the deterministic fallback service. +import { existsSync, readFileSync } from 'node:fs' +import { mkdtemp, readFile, readdir, rm, utimes, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { pathToFileURL } from 'node:url' +import type { Page } from 'playwright' +import { expect } from 'vitest' +import { scrubRequestHeaders } from '@deepseek-ai/dsh-acp-snapshot' +import { installLlmReplay, parseSessionLog } from '@deepseek-ai/dsh-llm-replay' +import type { ReplayHandle } from '@deepseek-ai/dsh-llm-replay' +import { startHost, mountWebPlugins } from '@deepseek-ai/dsh-host-runtime' +import type { RunningHost } from '@deepseek-ai/dsh-host-runtime' +import { createHostWebPluginRegistry, startWebServer } from '@deepseek-ai/dsh-host-webserver' +import SessionStore, { SESSION_FORMAT_VERSION, SessionId } from '@deepseek-ai/dsh-session' +import type { Session, SessionEvent, SessionHeader } from '@deepseek-ai/dsh-session' +import SessionPersistenceJsonl from '@deepseek-ai/dsh-session-persistence-jsonl' +import { Context } from 'cordis' +import { DIST_INDEX, REPO_ROOT, requireDist } from './support.ts' + +/** Snapshot mode for the lane, from $DSH_SNAPSHOT (same vocabulary as the ACP/TUI suites). */ +export type WebSnapshotMode = 'replay' | 'record' | 'refresh' + +/** + * Resolve and validate the lane's snapshot mode. + * @returns the active mode; unset/empty selects replay. + */ +export function webSnapshotMode(): WebSnapshotMode { + const value = process.env.DSH_SNAPSHOT + if (value === undefined || value === '' || value === 'replay') return 'replay' + if (value === 'record' || value === 'refresh') return value + throw new Error(`DSH_SNAPSHOT must be replay, record, or refresh; got ${JSON.stringify(value)}`) +} + +// Replay must run in providers mode (never catch-all): with `llm: false` no +// adapter exists, so a catch-all would leave resolveModelContext unroutable +// and compact-basic's post-step pressure check would warn every step. The +// published contextWindow keeps that pressure path provably inert for small +// fixtures. +const PROVIDERS = [{ id: 'deepseek', name: 'DeepSeek', models: [{ id: 'deepseek-v4-flash', contextWindow: 128_000 }] }] + +// The shipped client roster (apps/cli/src/web.ts CLIENT_PACKAGES, sans the +// --dev HMR row). apps/web depends on every entry, so its URL anchors the +// Loader's bare-specifier resolution. +const CLIENT_PACKAGES = [ + '@deepseek-ai/dsh-client-connection', + '@deepseek-ai/dsh-client-runtime', + '@deepseek-ai/dsh-client-ui-theme', + '@deepseek-ai/dsh-client-i18n', + '@deepseek-ai/dsh-client-ui-layout', + '@deepseek-ai/dsh-client-ui-sidebar', + '@deepseek-ai/dsh-client-ui-conversation', + '@deepseek-ai/dsh-client-ui-question', + '@deepseek-ai/dsh-client-ui-trajectory', +] as const + +/** Repo-root .env → process.env for record mode (never overrides set vars); the smoke-real convention. */ +function loadRootEnv(): void { + const envPath = join(REPO_ROOT, '.env') + if (!existsSync(envPath)) return + for (const line of readFileSync(envPath, 'utf8').split('\n')) { + const m = /^([A-Za-z_][A-Za-z0-9_]*)=(.*)$/.exec(line.trim()) + if (m !== null && process.env[m[1]!] === undefined) process.env[m[1]!] = m[2] + } +} + +/** A booted web harness: real assembly, mode-selected model backend, temp world. */ +export interface WebHarness { + /** The active snapshot mode this harness booted under. */ + mode: WebSnapshotMode + /** Browser-facing origin (http://127.0.0.1:). */ + baseUrl: string + /** The running host (ctx is the documented in-process barrier seam). */ + host: RunningHost + /** Temp project directory sessions run in (bash/fs tool cwd). */ + workspaceCwd: string + /** Temp persistence root (seeded sessions land here through the real API). */ + persistenceRoot: string + /** Errors the web server reported asynchronously; assert empty at scenario end. */ + serverErrors: string[] + /** Await a settled turn end: in-process turn/end, then the agent's idle flip (which follows the persistence flush). */ + whenTurnSettled(timeoutMs?: number): Promise + /** Tear everything down; asserts the replay fixture was fully consumed first (replay/refresh). */ + close(): Promise +} + +/** Options for {@link launchWebHarness}. */ +export interface LaunchOptions { + /** + * Replay fixture (session.jsonl) served by dsh-llm-replay in replay/refresh + * modes; ignored in record mode (the real adapter answers). Omit for + * scenarios issuing no model calls — a stray stream then fails loud with + * NO_ADAPTER on the open seam. + */ + replayFixture?: string + /** Per-chunk replay pacing (ms) so the browser observes genuinely incremental SSE; replay/refresh only. */ + paceMs?: number +} + +/** + * Boot the real web assembly under the current snapshot mode. + * @param options - replay fixture selection and pacing. + * @returns the running harness. + */ +export async function launchWebHarness(options: LaunchOptions = {}): Promise { + requireDist() + const mode = webSnapshotMode() + if (mode === 'record') { + loadRootEnv() + if (process.env.DEEPSEEK_API_KEY === undefined || process.env.DEEPSEEK_API_KEY.length === 0) { + throw new Error('web e2e record mode needs DEEPSEEK_API_KEY (env or repo-root .env)') + } + } + const workspaceCwd = await mkdtemp(join(tmpdir(), 'dsh-web-e2e-ws-')) + const persistenceRoot = await mkdtemp(join(tmpdir(), 'dsh-web-e2e-sessions-')) + const serverErrors: string[] = [] + let host: RunningHost | undefined + let server: Awaited> | undefined + let replay: ReplayHandle | undefined + try { + host = await startHost({ + boot: { + persistenceRoot, + // Keep the request header free of ambient AGENTS.md content so + // recorded fixtures do not embed this repo's instructions. + workspaceContext: false, + cwd: workspaceCwd, + // Replay/refresh boot keyless with the llm seam open; record mounts + // the real adapter and performs real provider I/O. + ...(mode === 'record' ? {} : { llm: false as const }), + }, + }) + if (mode !== 'record' && options.replayFixture !== undefined) { + replay = installLlmReplay(host.ctx, { + file: options.replayFixture, + providers: PROVIDERS, + ...(options.paceMs === undefined ? {} : { paceMs: options.paceMs }), + }) + } + // Anchor at apps/cli exactly as `dsh web` does: that package declares + // every roster entry as a dependency, so the Loader's bare-specifier + // resolution and the registry's package.json resolver both work. + const anchor = pathToFileURL(join(REPO_ROOT, 'apps/cli/src/web.ts')).href + const mounted = await mountWebPlugins(host.ctx, CLIENT_PACKAGES, anchor) + const webPlugins = createHostWebPluginRegistry({ + ctx: host.ctx, + loader: mounted.loader, + resolvePkgJson: mounted.resolvePkgJson, + onError: (err: Error) => { serverErrors.push(String(err)) }, + }) + server = await startWebServer( + { host: '127.0.0.1', port: 0, distIndex: DIST_INDEX, apiHandler: host.handler, webPlugins }, + (err: Error) => { serverErrors.push(String(err)) }, + ) + } catch (error) { + await server?.close().catch(() => undefined) + await host?.dispose().catch(() => undefined) + await rm(workspaceCwd, { recursive: true, force: true }).catch(() => undefined) + await rm(persistenceRoot, { recursive: true, force: true }).catch(() => undefined) + throw error + } + const runningHost = host + const runningServer = server + const replayHandle = replay + + return { + mode, + baseUrl: `http://127.0.0.1:${server.port}`, + host, + workspaceCwd, + persistenceRoot, + serverErrors, + // Barrier stack: the in-process turn/end identifies the session, then + // agent.whenIdle() covers the persistence flush (the idle flip follows + // the flush), and the caller's browser settled-poll comes last because + // host completion strictly precedes render. + whenTurnSettled(timeoutMs = mode === 'record' ? 180_000 : 30_000): Promise { + return new Promise((resolve, reject) => { + const timer = setTimeout(() => { + off() + reject(new Error(`no turn/end within ${timeoutMs}ms`)) + }, timeoutMs) + const off = runningHost.ctx.on('session/event', (session: { id: SessionId }, event: SessionEvent) => { + if (event.type !== 'turn/end') return + clearTimeout(timer) + off() + const agent = runningHost.ctx.agents.get(session.id) + if (agent === undefined) { + reject(new Error(`turn/end for ${session.id} but no live agent`)) + return + } + agent.whenIdle().then(() => { resolve(session.id) }, reject) + }) + }) + }, + async close(): Promise { + const failures: unknown[] = [] + // Fixture-consumption check first, while the run's binding state is + // still authoritative — a scenario that drove fewer model calls than + // recorded fails here instead of drifting green. + try { + replayHandle?.assertConsumed() + } catch (error) { + failures.push(error) + } + await runningServer.close().catch((e: unknown) => failures.push(e)) + await runningHost.dispose().catch((e: unknown) => failures.push(e)) + await rm(workspaceCwd, { recursive: true, force: true }).catch((e: unknown) => failures.push(e)) + await rm(persistenceRoot, { recursive: true, force: true }).catch((e: unknown) => failures.push(e)) + if (failures.length > 0) throw new AggregateError(failures, 'web harness teardown failed') + }, + } +} + +/** + * Serialize a live session back to raw session-JSONL (header + events) — the + * in-memory record-mode harvest, so the on-disk zstd default never matters. + * Mirrors the TUI suite's rawSessionLog. + * @param session - the live session to serialize. + * @returns raw JSONL text ending in one newline. + */ +export function rawSessionLog(session: Session): string { + return [ + JSON.stringify({ type: 'session', ...session.header }), + ...session.events.map(event => JSON.stringify(event)), + '', + ].join('\n') +} + +/** + * Record-mode fixture write-back: harvest the live session, scrub request + * headers to {{system}}/{{tools}} (the web lane pins no header class — a + * deliberate deviation logged in the Agent Note's deferred work), tokenize + * the run-local session id and cwd ({{sessionId}}/{{cwd}}, the committed ACP + * fixture convention — re-records then diff only on real content), and write + * the committed fixture. + * @param harness - the record-mode harness. + * @param sessionId - the driven session. + * @param fixturePath - the committed session.jsonl / seed.jsonl target. + */ +export async function recordFixture(harness: WebHarness, sessionId: SessionId, fixturePath: string): Promise { + const agent = harness.host.ctx.agents.get(sessionId) + if (agent === undefined) throw new Error(`record harvest: no live agent for ${sessionId}`) + const tokenized = scrubRequestHeaders(rawSessionLog(agent.session)) + .split(sessionId).join('{{sessionId}}') + .split(harness.workspaceCwd).join('{{cwd}}') + await writeFile(fixturePath, tokenized) +} + +/** + * The user prompts recorded in a fixture, in order — the single source tying + * spec drive steps to recorded reality so script and fixture cannot drift. + * @param fixtureText - raw session.jsonl contents. + * @returns the recorded user prompt texts. + */ +export function fixtureUserPrompts(fixtureText: string): string[] { + return parseSessionLog(fixtureText).flatMap((event) => { + if (event.type !== 'user/message' || event.data.source.kind !== 'user') return [] + const text = event.data.content.filter(block => block.type === 'text').map(block => block.text).join('') + return text.length > 0 ? [text] : [] + }) +} + +/** + * Seed a recorded session fixture into the harness's persistence root through + * the REAL backend API (throwaway Context + SessionStore + JSONL plugin — the + * semantic-checkpoint precedent), never raw file writes: no knowledge of + * bucket hashing, filename encoding, or compression, and malformed shapes + * fail loud at seed time. The fixture's recorded cwd is rewritten to the + * harness workspace so header/path identity and event payload paths agree. + * @param harness - the target harness. + * @param fixtureText - raw recorded session.jsonl contents. + * @param id - the seeded session id (stable for deterministic goldens). + * @returns the seeded id. + */ +export async function seedSession(harness: WebHarness, fixtureText: string, id: string): Promise { + // Committed fixtures tokenize run-local identity ({{sessionId}}/{{cwd}}, + // written by recordFixture); realize both for this world before parsing. + const realized = fixtureText + .split('{{sessionId}}').join(id) + .split('{{cwd}}').join(harness.workspaceCwd) + const fixtureCwd = (JSON.parse(realized.split('\n', 1)[0]!) as { cwd?: string }).cwd + const rewritten = fixtureCwd === undefined + ? realized + : realized.split(fixtureCwd).join(harness.workspaceCwd) + const events = parseSessionLog(rewritten) + if (events.length === 0) throw new Error('seed fixture has no events') + const last = events[events.length - 1]! + // An open final turn would be mutated by resume's crash repair on first + // open; a committed seed must be a closed recording. + if (last.type !== 'turn/end') throw new Error(`seed fixture must end in turn/end, got ${last.type}`) + const meta: SessionHeader = { + version: SESSION_FORMAT_VERSION, + id: SessionId(id), + createdAt: Date.now() - 60_000, + cwd: harness.workspaceCwd, + delegationDepth: 0, + } + const ctx = new Context() + try { + await ctx.plugin(SessionStore) + // Same root as the host with the plugin's own default compression, so the + // host's directory-scan list() sees one consistent encoding. + await ctx.plugin(SessionPersistenceJsonl, { root: harness.persistenceRoot }) + await ctx.sessionPersistence.create(meta) + await ctx.sessionPersistence.append(meta.id, events) + // Deterministic sidebar order: cold summaries take updatedAt from mtime. + const located = ctx.sessionPersistence.locate(meta) + if (located !== undefined) { + const backdated = new Date(meta.createdAt) + await utimes(located.path, backdated, backdated) + } + } finally { + await ctx.fiber.dispose() + } + return meta.id +} + +/** + * Normalize an aria snapshot: uuid, cwd, workspace-basename, and duration + * volatility collapse to stable tokens. + * @param snapshot - raw ariaSnapshot text. + * @param workspaceCwd - the harness workspace (basename doubles as the header breadcrumb). + * @returns tokenized snapshot text. + */ +export function normalizeAria(snapshot: string, workspaceCwd: string): string { + // The header breadcrumb renders the workspace's basename, not the full + // path, so both spellings must collapse to the token. + const base = workspaceCwd.split('/').pop()! + return snapshot + .split(workspaceCwd).join('{{cwd}}') + .split(base).join('{{workspace}}') + .replace(/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi, '{{uuid}}') + .replace(/\b\d+(?:\.\d+)?(?:ms|s|秒)\b/g, '{{duration}}') +} + +/** + * Capture the region's aria snapshot at a settled milestone: poll until two + * consecutive normalized captures are equal — a single-shot capture races the + * last React commits. + * @param page - the page under test. + * @param selector - the region locator selector. + * @param workspaceCwd - normalization input. + * @returns the stable normalized snapshot. + */ +export async function captureStableAria(page: Page, selector: string, workspaceCwd: string): Promise { + const region = page.locator(selector).first() + let previous = normalizeAria(await region.ariaSnapshot(), workspaceCwd) + await expect.poll(async () => { + const current = normalizeAria(await region.ariaSnapshot(), workspaceCwd) + const stable = current === previous + previous = current + return stable + }, { timeout: 5_000, message: 'aria snapshot did not stabilize' }).toBe(true) + return previous +} + +/** + * Compare a normalized golden, or rewrite it under refresh. Refresh is the + * ONLY writer: a missing golden in replay mode fails with the healing command + * instead of silently self-bootstrapping. + * @param goldenPath - the committed ui.expected.md path. + * @param actual - the stable normalized snapshot. + * @param mode - the active snapshot mode. + */ +export async function compareOrRefreshGolden(goldenPath: string, actual: string, mode: WebSnapshotMode): Promise { + const payload = `${actual}\n` + if (mode === 'refresh') { + await writeFile(goldenPath, payload) + return + } + if (!existsSync(goldenPath)) { + throw new Error(`missing golden ${goldenPath} — run DSH_SNAPSHOT=refresh pnpm run test:web to generate it`) + } + expect(payload).toBe(await readFile(goldenPath, 'utf8')) +} + +/** + * Fixture-inventory guard (the TUI afterAll shape): the scenario directory + * holds exactly the expected files and every committed JSONL is a scrub + * fixed-point (no request-header bulk escaped the record write-back). + * @param dir - the scenario snapshot directory. + * @param expected - the exact expected file inventory. + */ +export async function assertFixtureInventory(dir: string, expected: string[]): Promise { + const entries = (await readdir(dir)).sort() + expect(entries).toEqual([...expected].sort()) + for (const entry of entries.filter(name => name.endsWith('.jsonl'))) { + const content = await readFile(join(dir, entry), 'utf8') + expect(scrubRequestHeaders(content), `${dir}/${entry} carries request-header bulk`).toBe(content) + } +} + +/** + * Console tripwires: reconnect/gap-repair self-healing or a pageerror must + * fail the scenario, not mask a dead wire behind eventual consistency. + * @param page - the page under test. + * @returns live warning/pageerror collectors to assert empty at scenario end. + */ +export function watchConsole(page: Page): { warnings: string[]; pageErrors: string[] } { + const warnings: string[] = [] + const pageErrors: string[] = [] + page.on('console', (message) => { + const text = message.text() + if (/connection lost|gap repair|discontinuous/i.test(text)) warnings.push(text) + }) + page.on('pageerror', (error) => { pageErrors.push(String(error)) }) + return { warnings, pageErrors } +} diff --git a/apps/web/tests/replay-round-trip.e2e.ts b/apps/web/tests/replay-round-trip.e2e.ts new file mode 100644 index 0000000000..faf1a1f6a8 --- /dev/null +++ b/apps/web/tests/replay-round-trip.e2e.ts @@ -0,0 +1,109 @@ +// Web e2e scenario: fresh round trip. A real chromium types a prompt into the +// real composer; the wire, apiproxy, agent loop, and the REAL bash tool (echo +// in the temp workspace) all run; the model seam is dsh-llm-replay (keyless) +// or the live adapter (record). Drive steps run in every mode and wait only +// on generic completion (whenTurnSettled — never model-content selectors, so +// record cannot hang on a live model answering differently); assertion steps +// run in replay/refresh only. Settled states only — streaming incrementality +// is asserted from the persisted assistant/chunk events, not transient DOM. +// Record: DSH_SNAPSHOT=record rewrites session.jsonl, then a keyless +// DSH_SNAPSHOT=refresh regenerates ui.expected.md. +import { readFile } from 'node:fs/promises' +import { fileURLToPath } from 'node:url' +import type { Browser, Page } from 'playwright' +import { chromium } from 'playwright' +import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest' +import type { SessionEvent } from '@deepseek-ai/dsh-session' +import { + assertFixtureInventory, captureStableAria, compareOrRefreshGolden, fixtureUserPrompts, + launchWebHarness, recordFixture, watchConsole, webSnapshotMode, type WebHarness, +} from './harness.ts' +import { saveFailureShot } from './support.ts' + +const SNAPSHOT_DIR = fileURLToPath(new URL('./snapshots/fresh-round-trip', import.meta.url)) +const FIXTURE = fileURLToPath(new URL('./snapshots/fresh-round-trip/session.jsonl', import.meta.url)) +const UI_EXPECTED = fileURLToPath(new URL('./snapshots/fresh-round-trip/ui.expected.md', import.meta.url)) +const MODE = webSnapshotMode() + +// The scenario's one drive prompt. Record sends it; replay asserts the +// committed fixture recorded exactly it, so drive script and fixture cannot +// drift apart. +const PROMPT = 'Use the bash tool to run exactly: echo WEB_E2E_OK. Then reply with the single word DONE and stop.' + +describe('web e2e: fresh round trip through the real assembly', () => { + let harness: WebHarness + let browser: Browser + let page: Page + let tripwire: ReturnType + const sessionEvents: SessionEvent[] = [] + + beforeAll(async () => { + harness = await launchWebHarness({ + ...(MODE === 'record' ? {} : { replayFixture: FIXTURE, paceMs: 15 }), + }) + harness.host.ctx.on('session/event', (_session, event: SessionEvent) => { sessionEvents.push(event) }) + browser = await chromium.launch() + page = await browser.newPage({ viewport: { width: 1680, height: 1000 } }) + tripwire = watchConsole(page) + await page.goto(harness.baseUrl, { waitUntil: 'load' }) + await page.waitForSelector('[class*="frame"]', { timeout: 30_000 }) + }, 120_000) + + afterAll(async () => { + await browser?.close() + await harness?.close() + }) + + it('drives the recorded prompt to a settled turn (all modes)', async () => { + onTestFailed(() => saveFailureShot(page, 'web-e2e-round-trip')) + if (MODE !== 'record') { + // Drift guard: the committed fixture must carry exactly the drive prompt. + expect(fixtureUserPrompts(await readFile(FIXTURE, 'utf8'))).toEqual([PROMPT]) + } + const input = page.locator('textarea').first() + await input.waitFor({ timeout: 10_000 }) + // Arm the host-side settled barrier BEFORE the send click. + const settled = harness.whenTurnSettled() + await input.fill(PROMPT) + await input.press('Enter') + const sessionId = await settled + if (MODE === 'record') { + await recordFixture(harness, sessionId, FIXTURE) + } + }, 200_000) + + it.skipIf(MODE === 'record')('rendered the settled turn: markdown, tool row, composer restore', async () => { + onTestFailed(() => saveFailureShot(page, 'web-e2e-round-trip-settled')) + // Browser settled-poll after host completion (host strictly precedes render). + await page.locator('[data-streaming="true"]').waitFor({ state: 'detached', timeout: 15_000 }).catch(() => { + // Chunks may coalesce into one commit; a never-mounted streaming node is + // legal — the chunk-event assertions below carry incrementality. + }) + await expect.poll(() => page.getByText('DONE', { exact: true }).count(), { timeout: 15_000 }).toBeGreaterThanOrEqual(1) + // World state, not self-report: bash really ran and the turn closed clean. + const toolCalls = sessionEvents.filter(e => e.type === 'tool/call') + expect(toolCalls.map(e => (e as SessionEvent & { data: { name: string } }).data.name)).toContain('bash') + const turnEnds = sessionEvents.filter(e => e.type === 'turn/end') + expect(turnEnds.length).toBe(1) + expect((turnEnds[0] as SessionEvent & { data: { reason: { kind: string } } }).data.reason.kind).toBe('completed') + // The persisted chunk events are the authoritative incrementality proof. + expect(sessionEvents.filter(e => e.type === 'assistant/chunk').length).toBeGreaterThan(10) + }, 60_000) + + it.skipIf(MODE === 'record')('matches the conversation aria golden with stable anchors', async () => { + onTestFailed(() => saveFailureShot(page, 'web-e2e-round-trip-aria')) + // Anchor assertions survive a semantics-preserving component rewrite even + // while the whole-region golden churns. + await expect(page.getByRole('textbox').first().isVisible()).resolves.toBe(true) + expect(await page.getByText('WEB_E2E_OK', { exact: false }).count()).toBeGreaterThanOrEqual(1) + const snapshot = await captureStableAria(page, '[class*="centerCol"]', harness.workspaceCwd) + await compareOrRefreshGolden(UI_EXPECTED, snapshot, MODE) + }) + + it.skipIf(MODE === 'record')('stayed clean: no pageerrors, no reconnect self-healing, no server errors', async () => { + expect(tripwire.pageErrors).toEqual([]) + expect(tripwire.warnings).toEqual([]) + expect(harness.serverErrors).toEqual([]) + await assertFixtureInventory(SNAPSHOT_DIR, ['session.jsonl', 'ui.expected.md']) + }) +}) diff --git a/apps/web/tests/seeded-history.e2e.ts b/apps/web/tests/seeded-history.e2e.ts new file mode 100644 index 0000000000..734c0848c4 --- /dev/null +++ b/apps/web/tests/seeded-history.e2e.ts @@ -0,0 +1,105 @@ +// Web e2e scenario: seeded history. A recorded session seeded cold through +// the REAL persistence API renders purely from the log — the surface nothing +// else covers: sidebar cold listing, the implicit resume/attach inside the +// history RPC, history-page tool views, and the client fold of historical +// events — with ZERO model calls in replay (no replay fixture; a stray stream +// fails loud on the open llm seam). The seed is a recorded fixture under the +// same record discipline as every other: DSH_SNAPSHOT=record drives the turn +// live through the composer (real read tool against seeded workspace files) +// and harvests seed.jsonl; replay/refresh seed it cold and only render. +import { readFile, writeFile } from 'node:fs/promises' +import { fileURLToPath } from 'node:url' +import type { Browser, Page } from 'playwright' +import { chromium } from 'playwright' +import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest' +import { join } from 'node:path' +import { + assertFixtureInventory, captureStableAria, compareOrRefreshGolden, fixtureUserPrompts, + launchWebHarness, recordFixture, seedSession, watchConsole, webSnapshotMode, type WebHarness, +} from './harness.ts' +import { saveFailureShot } from './support.ts' + +const SNAPSHOT_DIR = fileURLToPath(new URL('./snapshots/seeded-history', import.meta.url)) +const SEED = fileURLToPath(new URL('./snapshots/seeded-history/seed.jsonl', import.meta.url)) +const UI_EXPECTED = fileURLToPath(new URL('./snapshots/seeded-history/ui.expected.md', import.meta.url)) +const MODE = webSnapshotMode() +const SEED_ID = 'seeded-history-web-e2e' + +const PROMPT = 'Use the read tool twice in one assistant message: read a.txt and b.txt. Then reply with the single word DONE and stop.' + +describe('web e2e: seeded history renders through cold resume', () => { + let harness: WebHarness + let browser: Browser + let page: Page + let tripwire: ReturnType + + beforeAll(async () => { + harness = await launchWebHarness({}) + // The read-tool targets exist in both modes: record needs them for the + // live turn; replay's seeded log carries their recorded contents but the + // workspace stays consistent for any user poking the harness. + await writeFile(join(harness.workspaceCwd, 'a.txt'), 'alpha\n') + await writeFile(join(harness.workspaceCwd, 'b.txt'), 'beta\n') + if (MODE !== 'record') { + const raw = await readFile(SEED, 'utf8') + expect(fixtureUserPrompts(raw), 'seed fixture must carry exactly the drive prompt').toEqual([PROMPT]) + await seedSession(harness, raw, SEED_ID) + } + browser = await chromium.launch() + page = await browser.newPage({ viewport: { width: 1680, height: 1000 } }) + tripwire = watchConsole(page) + await page.goto(harness.baseUrl, { waitUntil: 'load' }) + await page.waitForSelector('[class*="frame"]', { timeout: 30_000 }) + }, 120_000) + + afterAll(async () => { + await browser?.close() + await harness?.close() + }) + + it.skipIf(MODE !== 'record')('records the seed turn live through the composer', async () => { + onTestFailed(() => saveFailureShot(page, 'web-e2e-seeded-record')) + const input = page.locator('textarea').first() + await input.waitFor({ timeout: 10_000 }) + const settled = harness.whenTurnSettled() + await input.fill(PROMPT) + await input.press('Enter') + const sessionId = await settled + await recordFixture(harness, sessionId, SEED) + }, 200_000) + + it.skipIf(MODE === 'record')('lists the seeded session cold and renders its history from the log', async () => { + onTestFailed(() => saveFailureShot(page, 'web-e2e-seeded-history')) + // The sidebar tree collapses workspace groups by default: click the group + // row (treeitem 0) to expand, then the revealed session row. + const groupRow = page.locator('[role="treeitem"]').first() + await groupRow.waitFor({ timeout: 15_000 }) + await groupRow.click() + const sessionRow = page.locator('[role="treeitem"]').nth(1) + await sessionRow.waitFor({ timeout: 10_000 }) + await sessionRow.click() + // Settled barrier for history: the recorded final assistant text renders. + await expect.poll(() => page.getByText('DONE', { exact: true }).count(), { timeout: 15_000 }).toBe(1) + // Tool cards render from logged tool/call + tool/result alone (views are + // host-recomputed per page; the generic card is the documented default). + const toolRows = page.locator('[data-variant], [data-sample]') + await expect.poll(() => toolRows.count(), { timeout: 10_000 }).toBeGreaterThanOrEqual(2) + expect(await page.getByText('a.txt', { exact: false }).count()).toBeGreaterThan(0) + }, 60_000) + + it.skipIf(MODE === 'record')('matches the historical conversation aria golden', async () => { + onTestFailed(() => saveFailureShot(page, 'web-e2e-seeded-aria')) + const snapshot = (await captureStableAria(page, '[class*="centerCol"]', harness.workspaceCwd)) + .split(SEED_ID).join('{{seededId}}') + await compareOrRefreshGolden(UI_EXPECTED, snapshot, MODE) + }) + + it.skipIf(MODE === 'record')('issued zero model calls and stayed clean', async () => { + // No replay fixture was installed and the llm seam is open — any stray + // stream would have failed the turn loudly. Cleanliness pins the wire. + expect(tripwire.pageErrors).toEqual([]) + expect(tripwire.warnings).toEqual([]) + expect(harness.serverErrors).toEqual([]) + await assertFixtureInventory(SNAPSHOT_DIR, ['seed.jsonl', 'ui.expected.md']) + }) +}) diff --git a/apps/web/tests/snapshots/fresh-round-trip/session.jsonl b/apps/web/tests/snapshots/fresh-round-trip/session.jsonl new file mode 100644 index 0000000000..d9cc109c60 --- /dev/null +++ b/apps/web/tests/snapshots/fresh-round-trip/session.jsonl @@ -0,0 +1,97 @@ +{"type":"session","version":0,"id":"{{sessionId}}","createdAt":1784893539564,"cwd":"{{cwd}}"} +{"type":"turn/start","seq":0,"time":1784893539588,"data":{"turn":1,"trigger":{"kind":"message","source":{"kind":"user","rpcId":"4962b8d4-0422-4f81-8187-642e3e6bab78"}}}} +{"type":"user/message","seq":1,"time":1784893539589,"data":{"content":[{"type":"text","text":"Use the bash tool to run exactly: echo WEB_E2E_OK. Then reply with the single word DONE and stop."}],"source":{"kind":"user","rpcId":"4962b8d4-0422-4f81-8187-642e3e6bab78"}},"surfaceOp":"append"} +{"type":"session/title","seq":2,"time":1784893539592,"data":{"title":"Use the bash tool to","messageSeqs":[1],"source":{"kind":"fallback"}}} +{"type":"step/start","seq":3,"time":1784893539657,"data":{"turn":1,"step":1}} +{"type":"request/header","seq":4,"time":1784893539658,"data":{"header":{"config":{"provider":"deepseek","model":"deepseek-v4-flash"},"system":"{{system}}","tools":"{{tools}}","messagePrefix":["{{messagePrefix}}"]},"reason":"initial"}} +{"type":"assistant/chunk","seq":5,"time":1784893540271,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":0,"blockType":"reasoning"}}} +{"type":"assistant/chunk","seq":6,"time":1784893540271,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"The"}}} +{"type":"assistant/chunk","seq":7,"time":1784893540366,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" user"}}} +{"type":"assistant/chunk","seq":8,"time":1784893540396,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" wants"}}} +{"type":"assistant/chunk","seq":9,"time":1784893540396,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" me"}}} +{"type":"assistant/chunk","seq":10,"time":1784893540396,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" to"}}} +{"type":"assistant/chunk","seq":11,"time":1784893540396,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" run"}}} +{"type":"assistant/chunk","seq":12,"time":1784893540397,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" a"}}} +{"type":"assistant/chunk","seq":13,"time":1784893540421,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" simple"}}} +{"type":"assistant/chunk","seq":14,"time":1784893540447,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" bash"}}} +{"type":"assistant/chunk","seq":15,"time":1784893540447,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" command"}}} +{"type":"assistant/chunk","seq":16,"time":1784893540448,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" and"}}} +{"type":"assistant/chunk","seq":17,"time":1784893540448,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" reply"}}} +{"type":"assistant/chunk","seq":18,"time":1784893540475,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" with"}}} +{"type":"assistant/chunk","seq":19,"time":1784893540476,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" \""}}} +{"type":"assistant/chunk","seq":20,"time":1784893540476,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"D"}}} +{"type":"assistant/chunk","seq":21,"time":1784893540476,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"ONE"}}} +{"type":"assistant/chunk","seq":22,"time":1784893540476,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"\"."}}} +{"type":"assistant/chunk","seq":23,"time":1784893540565,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":1,"blockType":"tool-call"}}} +{"type":"assistant/chunk","seq":24,"time":1784893540566,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":""}}} +{"type":"assistant/chunk","seq":25,"time":1784893540591,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"{"}}} +{"type":"assistant/chunk","seq":26,"time":1784893540592,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":27,"time":1784893540592,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"command"}}} +{"type":"assistant/chunk","seq":28,"time":1784893540592,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":29,"time":1784893540592,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":": "}}} +{"type":"assistant/chunk","seq":30,"time":1784893540621,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":31,"time":1784893540621,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"echo"}}} +{"type":"assistant/chunk","seq":32,"time":1784893540621,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":" WEB"}}} +{"type":"assistant/chunk","seq":33,"time":1784893540621,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"_E"}}} +{"type":"assistant/chunk","seq":34,"time":1784893540651,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"2"}}} +{"type":"assistant/chunk","seq":35,"time":1784893540651,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"E"}}} +{"type":"assistant/chunk","seq":36,"time":1784893540652,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"_OK"}}} +{"type":"assistant/chunk","seq":37,"time":1784893540652,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":38,"time":1784893540680,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":", "}}} +{"type":"assistant/chunk","seq":39,"time":1784893540680,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":40,"time":1784893540709,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"description"}}} +{"type":"assistant/chunk","seq":41,"time":1784893540710,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":42,"time":1784893540710,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":": "}}} +{"type":"assistant/chunk","seq":43,"time":1784893540710,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":44,"time":1784893540738,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"Print"}}} +{"type":"assistant/chunk","seq":45,"time":1784893540738,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":" WEB"}}} +{"type":"assistant/chunk","seq":46,"time":1784893540768,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"_E"}}} +{"type":"assistant/chunk","seq":47,"time":1784893540768,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"2"}}} +{"type":"assistant/chunk","seq":48,"time":1784893540768,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"E"}}} +{"type":"assistant/chunk","seq":49,"time":1784893540768,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"_OK"}}} +{"type":"assistant/chunk","seq":50,"time":1784893540769,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":" to"}}} +{"type":"assistant/chunk","seq":51,"time":1784893540797,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":" stdout"}}} +{"type":"assistant/chunk","seq":52,"time":1784893540801,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":53,"time":1784893540826,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","argumentsDelta":"}"}}} +{"type":"assistant/chunk","seq":54,"time":1784893540859,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":0,"block":{"type":"reasoning","text":"The user wants me to run a simple bash command and reply with \"DONE\"."}}}} +{"type":"assistant/chunk","seq":55,"time":1784893540859,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":1,"block":{"type":"tool-call","id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","arguments":"{\"command\": \"echo WEB_E2E_OK\", \"description\": \"Print WEB_E2E_OK to stdout\"}"}}}} +{"type":"assistant/chunk","seq":56,"time":1784893540859,"data":{"turn":1,"step":1,"chunk":{"type":"usage","usage":{"inputTokens":7802,"outputTokens":88,"cacheReadTokens":0,"reasoningTokens":17}}}} +{"type":"assistant/chunk","seq":57,"time":1784893540860,"data":{"turn":1,"step":1,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}} +{"type":"assistant/message","seq":58,"time":1784893540863,"data":{"turn":1,"step":1,"content":[{"type":"reasoning","text":"The user wants me to run a simple bash command and reply with \"DONE\"."},{"type":"tool-call","id":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","arguments":"{\"command\": \"echo WEB_E2E_OK\", \"description\": \"Print WEB_E2E_OK to stdout\"}"}],"provenance":{"provider":"deepseek","model":"deepseek-v4-flash"},"usage":{"inputTokens":7802,"outputTokens":88,"cacheReadTokens":0,"reasoningTokens":17}},"sourceEventSeqs":[5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57],"surfaceOp":"append"} +{"type":"tool/call","seq":59,"time":1784893540864,"data":{"turn":1,"step":1,"callId":"call_00_yxp0l3itFAMPCLKhz7EU1205","name":"bash","arguments":"{\"command\": \"echo WEB_E2E_OK\", \"description\": \"Print WEB_E2E_OK to stdout\"}"}} +{"type":"tool/result","seq":60,"time":1784893540878,"data":{"turn":1,"step":1,"callId":"call_00_yxp0l3itFAMPCLKhz7EU1205","content":[{"type":"text","text":"WEB_E2E_OK\n"}],"isError":false},"sourceEventSeqs":[59],"surfaceOp":"append"} +{"type":"step/end","seq":61,"time":1784893540881,"data":{"turn":1,"step":1}} +{"type":"step/start","seq":62,"time":1784893540881,"data":{"turn":1,"step":2}} +{"type":"assistant/chunk","seq":63,"time":1784893541457,"data":{"turn":1,"step":2,"chunk":{"type":"block-start","index":0,"blockType":"reasoning"}}} +{"type":"assistant/chunk","seq":64,"time":1784893541457,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"The"}}} +{"type":"assistant/chunk","seq":65,"time":1784893541545,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" command"}}} +{"type":"assistant/chunk","seq":66,"time":1784893541574,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" executed"}}} +{"type":"assistant/chunk","seq":67,"time":1784893541574,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" successfully"}}} +{"type":"assistant/chunk","seq":68,"time":1784893541574,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" and"}}} +{"type":"assistant/chunk","seq":69,"time":1784893541574,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" printed"}}} +{"type":"assistant/chunk","seq":70,"time":1784893541603,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" \""}}} +{"type":"assistant/chunk","seq":71,"time":1784893541604,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"WEB"}}} +{"type":"assistant/chunk","seq":72,"time":1784893541604,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"_E"}}} +{"type":"assistant/chunk","seq":73,"time":1784893541604,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"2"}}} +{"type":"assistant/chunk","seq":74,"time":1784893541604,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"E"}}} +{"type":"assistant/chunk","seq":75,"time":1784893541604,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"_OK"}}} +{"type":"assistant/chunk","seq":76,"time":1784893541633,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"\"."}}} +{"type":"assistant/chunk","seq":77,"time":1784893541633,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" I"}}} +{"type":"assistant/chunk","seq":78,"time":1784893541675,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" should"}}} +{"type":"assistant/chunk","seq":79,"time":1784893541676,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" now"}}} +{"type":"assistant/chunk","seq":80,"time":1784893541676,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" reply"}}} +{"type":"assistant/chunk","seq":81,"time":1784893541676,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" with"}}} +{"type":"assistant/chunk","seq":82,"time":1784893541694,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" \""}}} +{"type":"assistant/chunk","seq":83,"time":1784893541694,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"D"}}} +{"type":"assistant/chunk","seq":84,"time":1784893541694,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"ONE"}}} +{"type":"assistant/chunk","seq":85,"time":1784893541695,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"\"."}}} +{"type":"assistant/chunk","seq":86,"time":1784893541718,"data":{"turn":1,"step":2,"chunk":{"type":"block-start","index":1,"blockType":"text"}}} +{"type":"assistant/chunk","seq":87,"time":1784893541718,"data":{"turn":1,"step":2,"chunk":{"type":"text-delta","index":1,"text":"D"}}} +{"type":"assistant/chunk","seq":88,"time":1784893541718,"data":{"turn":1,"step":2,"chunk":{"type":"text-delta","index":1,"text":"ONE"}}} +{"type":"assistant/chunk","seq":89,"time":1784893541719,"data":{"turn":1,"step":2,"chunk":{"type":"block-end","index":0,"block":{"type":"reasoning","text":"The command executed successfully and printed \"WEB_E2E_OK\". I should now reply with \"DONE\"."}}}} +{"type":"assistant/chunk","seq":90,"time":1784893541719,"data":{"turn":1,"step":2,"chunk":{"type":"block-end","index":1,"block":{"type":"text","text":"DONE"}}}} +{"type":"assistant/chunk","seq":91,"time":1784893541719,"data":{"turn":1,"step":2,"chunk":{"type":"usage","usage":{"inputTokens":228,"outputTokens":25,"cacheReadTokens":7680,"reasoningTokens":22}}}} +{"type":"assistant/chunk","seq":92,"time":1784893541719,"data":{"turn":1,"step":2,"chunk":{"type":"finish","reason":{"kind":"stop"}}}} +{"type":"assistant/message","seq":93,"time":1784893541720,"data":{"turn":1,"step":2,"content":[{"type":"reasoning","text":"The command executed successfully and printed \"WEB_E2E_OK\". I should now reply with \"DONE\"."},{"type":"text","text":"DONE"}],"provenance":{"provider":"deepseek","model":"deepseek-v4-flash"},"usage":{"inputTokens":228,"outputTokens":25,"cacheReadTokens":7680,"reasoningTokens":22}},"sourceEventSeqs":[63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92],"surfaceOp":"append"} +{"type":"step/end","seq":94,"time":1784893541720,"data":{"turn":1,"step":2}} +{"type":"turn/end","seq":95,"time":1784893541721,"data":{"turn":1,"reason":{"kind":"completed"}}} diff --git a/apps/web/tests/snapshots/fresh-round-trip/ui.expected.md b/apps/web/tests/snapshots/fresh-round-trip/ui.expected.md new file mode 100644 index 0000000000..4464843ce6 --- /dev/null +++ b/apps/web/tests/snapshots/fresh-round-trip/ui.expected.md @@ -0,0 +1,31 @@ +- banner: + - navigation "会话层级": + - button "Use the bash tool to" [disabled] + - text: · 1 turns + - tablist: + - tab "Chat" [selected] + - tab "Trajectory" + - tab "Waterfall" +- text: "Use the bash tool to run exactly: echo WEB_E2E_OK. Then reply with the single word DONE and stop." +- button "Think The user wants me to run a simple bash command and reply with \"DONE\".": + - img + - text: Think The user wants me to run a simple bash command and reply with "DONE". +- text: Print WEB_E2E_OK to stdout +- button "Think The command executed successfully and printed \"WEB_E2E_OK\". I should now reply with \"DONE\".": + - img + - text: Think The command executed successfully and printed "WEB_E2E_OK". I should now reply with "DONE". +- paragraph: DONE +- text: cache hit 49% · 15,823 tokens · 1 turns · 2 steps +- textbox "输入消息，Enter 发送，Shift+Enter 换行" +- button "添加": + - img +- combobox "Plan mode": + - option "Plan" [selected] + - option "Agent" +- combobox "Access mode": + - option "Read-only" [selected] + - option "Read-write" +- combobox "Model": + - option "DeepSeek-V4-Pro High" [selected] + - option "DeepSeek-V4-Pro" +- button "发送" [disabled] diff --git a/apps/web/tests/snapshots/seeded-history/seed.jsonl b/apps/web/tests/snapshots/seeded-history/seed.jsonl new file mode 100644 index 0000000000..a27f182e32 --- /dev/null +++ b/apps/web/tests/snapshots/seeded-history/seed.jsonl @@ -0,0 +1,112 @@ +{"type":"session","version":0,"id":"{{sessionId}}","createdAt":1784893580342,"cwd":"{{cwd}}"} +{"type":"turn/start","seq":0,"time":1784893580362,"data":{"turn":1,"trigger":{"kind":"message","source":{"kind":"user","rpcId":"ab867856-cd1a-4270-8cee-c240076c58e9"}}}} +{"type":"user/message","seq":1,"time":1784893580363,"data":{"content":[{"type":"text","text":"Use the read tool twice in one assistant message: read a.txt and b.txt. Then reply with the single word DONE and stop."}],"source":{"kind":"user","rpcId":"ab867856-cd1a-4270-8cee-c240076c58e9"}},"surfaceOp":"append"} +{"type":"session/title","seq":2,"time":1784893580365,"data":{"title":"Use the read tool twice","messageSeqs":[1],"source":{"kind":"fallback"}}} +{"type":"step/start","seq":3,"time":1784893580420,"data":{"turn":1,"step":1}} +{"type":"request/header","seq":4,"time":1784893580421,"data":{"header":{"config":{"provider":"deepseek","model":"deepseek-v4-flash"},"system":"{{system}}","tools":"{{tools}}","messagePrefix":["{{messagePrefix}}"]},"reason":"initial"}} +{"type":"assistant/chunk","seq":5,"time":1784893581003,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":0,"blockType":"reasoning"}}} +{"type":"assistant/chunk","seq":6,"time":1784893581003,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"The"}}} +{"type":"assistant/chunk","seq":7,"time":1784893581092,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" user"}}} +{"type":"assistant/chunk","seq":8,"time":1784893581107,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" wants"}}} +{"type":"assistant/chunk","seq":9,"time":1784893581108,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" me"}}} +{"type":"assistant/chunk","seq":10,"time":1784893581108,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" to"}}} +{"type":"assistant/chunk","seq":11,"time":1784893581108,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" read"}}} +{"type":"assistant/chunk","seq":12,"time":1784893581135,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" a"}}} +{"type":"assistant/chunk","seq":13,"time":1784893581135,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":".txt"}}} +{"type":"assistant/chunk","seq":14,"time":1784893581136,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" and"}}} +{"type":"assistant/chunk","seq":15,"time":1784893581161,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" b"}}} +{"type":"assistant/chunk","seq":16,"time":1784893581162,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":".txt"}}} +{"type":"assistant/chunk","seq":17,"time":1784893581162,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":","}}} +{"type":"assistant/chunk","seq":18,"time":1784893581189,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" then"}}} +{"type":"assistant/chunk","seq":19,"time":1784893581190,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" reply"}}} +{"type":"assistant/chunk","seq":20,"time":1784893581190,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" with"}}} +{"type":"assistant/chunk","seq":21,"time":1784893581190,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" D"}}} +{"type":"assistant/chunk","seq":22,"time":1784893581217,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"ONE"}}} +{"type":"assistant/chunk","seq":23,"time":1784893581217,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"."}}} +{"type":"assistant/chunk","seq":24,"time":1784893581217,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" Let"}}} +{"type":"assistant/chunk","seq":25,"time":1784893581217,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" me"}}} +{"type":"assistant/chunk","seq":26,"time":1784893581217,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" read"}}} +{"type":"assistant/chunk","seq":27,"time":1784893581258,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" both"}}} +{"type":"assistant/chunk","seq":28,"time":1784893581259,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" files"}}} +{"type":"assistant/chunk","seq":29,"time":1784893581259,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"."}}} +{"type":"assistant/chunk","seq":30,"time":1784893581324,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":1,"blockType":"tool-call"}}} +{"type":"assistant/chunk","seq":31,"time":1784893581325,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","argumentsDelta":""}}} +{"type":"assistant/chunk","seq":32,"time":1784893581325,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","argumentsDelta":"{"}}} +{"type":"assistant/chunk","seq":33,"time":1784893581325,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":34,"time":1784893581351,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","argumentsDelta":"file"}}} +{"type":"assistant/chunk","seq":35,"time":1784893581351,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","argumentsDelta":"_path"}}} +{"type":"assistant/chunk","seq":36,"time":1784893581351,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":37,"time":1784893581351,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","argumentsDelta":": "}}} +{"type":"assistant/chunk","seq":38,"time":1784893581378,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":39,"time":1784893581378,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","argumentsDelta":"a"}}} +{"type":"assistant/chunk","seq":40,"time":1784893581378,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","argumentsDelta":".txt"}}} +{"type":"assistant/chunk","seq":41,"time":1784893581378,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":42,"time":1784893581404,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","argumentsDelta":"}"}}} +{"type":"assistant/chunk","seq":43,"time":1784893581462,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":2,"blockType":"tool-call"}}} +{"type":"assistant/chunk","seq":44,"time":1784893581462,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","argumentsDelta":""}}} +{"type":"assistant/chunk","seq":45,"time":1784893581486,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","argumentsDelta":"{"}}} +{"type":"assistant/chunk","seq":46,"time":1784893581486,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":47,"time":1784893581486,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","argumentsDelta":"file"}}} +{"type":"assistant/chunk","seq":48,"time":1784893581486,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","argumentsDelta":"_path"}}} +{"type":"assistant/chunk","seq":49,"time":1784893581486,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":50,"time":1784893581486,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","argumentsDelta":": "}}} +{"type":"assistant/chunk","seq":51,"time":1784893581513,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":52,"time":1784893581513,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","argumentsDelta":"b"}}} +{"type":"assistant/chunk","seq":53,"time":1784893581514,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","argumentsDelta":".txt"}}} +{"type":"assistant/chunk","seq":54,"time":1784893581514,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","argumentsDelta":"\""}}} +{"type":"assistant/chunk","seq":55,"time":1784893581539,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","argumentsDelta":"}"}}} +{"type":"assistant/chunk","seq":56,"time":1784893581597,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":0,"block":{"type":"reasoning","text":"The user wants me to read a.txt and b.txt, then reply with DONE. Let me read both files."}}}} +{"type":"assistant/chunk","seq":57,"time":1784893581597,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":1,"block":{"type":"tool-call","id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","arguments":"{\"file_path\": \"a.txt\"}"}}}} +{"type":"assistant/chunk","seq":58,"time":1784893581597,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":2,"block":{"type":"tool-call","id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","arguments":"{\"file_path\": \"b.txt\"}"}}}} +{"type":"assistant/chunk","seq":59,"time":1784893581597,"data":{"turn":1,"step":1,"chunk":{"type":"usage","usage":{"inputTokens":124,"outputTokens":100,"cacheReadTokens":7680,"reasoningTokens":24}}}} +{"type":"assistant/chunk","seq":60,"time":1784893581597,"data":{"turn":1,"step":1,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}} +{"type":"assistant/message","seq":61,"time":1784893581601,"data":{"turn":1,"step":1,"content":[{"type":"reasoning","text":"The user wants me to read a.txt and b.txt, then reply with DONE. Let me read both files."},{"type":"tool-call","id":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","arguments":"{\"file_path\": \"a.txt\"}"},{"type":"tool-call","id":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","arguments":"{\"file_path\": \"b.txt\"}"}],"provenance":{"provider":"deepseek","model":"deepseek-v4-flash"},"usage":{"inputTokens":124,"outputTokens":100,"cacheReadTokens":7680,"reasoningTokens":24}},"sourceEventSeqs":[5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60],"surfaceOp":"append"} +{"type":"tool/call","seq":62,"time":1784893581602,"data":{"turn":1,"step":1,"callId":"call_00_8bQPkq98ZAzRTEJA6XM38538","name":"read","arguments":"{\"file_path\": \"a.txt\"}"}} +{"type":"tool/call","seq":63,"time":1784893581604,"data":{"turn":1,"step":1,"callId":"call_01_5FMIBJA9HHyktFY8KSlk9459","name":"read","arguments":"{\"file_path\": \"b.txt\"}"}} +{"type":"tool/result","seq":64,"time":1784893581608,"data":{"turn":1,"step":1,"callId":"call_00_8bQPkq98ZAzRTEJA6XM38538","content":[{"type":"text","text":"{{cwd}}/a.txt\nfile\n\n1: alpha\n\n(End of file - total 1 lines)\n"}],"isError":false},"sourceEventSeqs":[62],"surfaceOp":"append"} +{"type":"tool/result","seq":65,"time":1784893581609,"data":{"turn":1,"step":1,"callId":"call_01_5FMIBJA9HHyktFY8KSlk9459","content":[{"type":"text","text":"{{cwd}}/b.txt\nfile\n\n1: beta\n\n(End of file - total 1 lines)\n"}],"isError":false},"sourceEventSeqs":[63],"surfaceOp":"append"} +{"type":"step/end","seq":66,"time":1784893581611,"data":{"turn":1,"step":1}} +{"type":"step/start","seq":67,"time":1784893581611,"data":{"turn":1,"step":2}} +{"type":"assistant/chunk","seq":68,"time":1784893582137,"data":{"turn":1,"step":2,"chunk":{"type":"block-start","index":0,"blockType":"reasoning"}}} +{"type":"assistant/chunk","seq":69,"time":1784893582137,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"Both"}}} +{"type":"assistant/chunk","seq":70,"time":1784893582257,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" files"}}} +{"type":"assistant/chunk","seq":71,"time":1784893582259,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" have"}}} +{"type":"assistant/chunk","seq":72,"time":1784893582260,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" been"}}} +{"type":"assistant/chunk","seq":73,"time":1784893582260,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" read"}}} +{"type":"assistant/chunk","seq":74,"time":1784893582260,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"."}}} +{"type":"assistant/chunk","seq":75,"time":1784893582277,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" a"}}} +{"type":"assistant/chunk","seq":76,"time":1784893582303,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":".txt"}}} +{"type":"assistant/chunk","seq":77,"time":1784893582303,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" contains"}}} +{"type":"assistant/chunk","seq":78,"time":1784893582304,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" \""}}} +{"type":"assistant/chunk","seq":79,"time":1784893582304,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"alpha"}}} +{"type":"assistant/chunk","seq":80,"time":1784893582304,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"\""}}} +{"type":"assistant/chunk","seq":81,"time":1784893582330,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" and"}}} +{"type":"assistant/chunk","seq":82,"time":1784893582330,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" b"}}} +{"type":"assistant/chunk","seq":83,"time":1784893582330,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":".txt"}}} +{"type":"assistant/chunk","seq":84,"time":1784893582330,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" contains"}}} +{"type":"assistant/chunk","seq":85,"time":1784893582331,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" \""}}} +{"type":"assistant/chunk","seq":86,"time":1784893582331,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"beta"}}} +{"type":"assistant/chunk","seq":87,"time":1784893582356,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"\"."}}} +{"type":"assistant/chunk","seq":88,"time":1784893582357,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" Now"}}} +{"type":"assistant/chunk","seq":89,"time":1784893582384,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" I"}}} +{"type":"assistant/chunk","seq":90,"time":1784893582384,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" just"}}} +{"type":"assistant/chunk","seq":91,"time":1784893582384,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" need"}}} +{"type":"assistant/chunk","seq":92,"time":1784893582384,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" to"}}} +{"type":"assistant/chunk","seq":93,"time":1784893582385,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" reply"}}} +{"type":"assistant/chunk","seq":94,"time":1784893582411,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" with"}}} +{"type":"assistant/chunk","seq":95,"time":1784893582411,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" the"}}} +{"type":"assistant/chunk","seq":96,"time":1784893582438,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" single"}}} +{"type":"assistant/chunk","seq":97,"time":1784893582438,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" word"}}} +{"type":"assistant/chunk","seq":98,"time":1784893582438,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" D"}}} +{"type":"assistant/chunk","seq":99,"time":1784893582438,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"ONE"}}} +{"type":"assistant/chunk","seq":100,"time":1784893582438,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"."}}} +{"type":"assistant/chunk","seq":101,"time":1784893582467,"data":{"turn":1,"step":2,"chunk":{"type":"block-start","index":1,"blockType":"text"}}} +{"type":"assistant/chunk","seq":102,"time":1784893582468,"data":{"turn":1,"step":2,"chunk":{"type":"text-delta","index":1,"text":"D"}}} +{"type":"assistant/chunk","seq":103,"time":1784893582468,"data":{"turn":1,"step":2,"chunk":{"type":"text-delta","index":1,"text":"ONE"}}} +{"type":"assistant/chunk","seq":104,"time":1784893582468,"data":{"turn":1,"step":2,"chunk":{"type":"block-end","index":0,"block":{"type":"reasoning","text":"Both files have been read. a.txt contains \"alpha\" and b.txt contains \"beta\". Now I just need to reply with the single word DONE."}}}} +{"type":"assistant/chunk","seq":105,"time":1784893582468,"data":{"turn":1,"step":2,"chunk":{"type":"block-end","index":1,"block":{"type":"text","text":"DONE"}}}} +{"type":"assistant/chunk","seq":106,"time":1784893582468,"data":{"turn":1,"step":2,"chunk":{"type":"usage","usage":{"inputTokens":340,"outputTokens":35,"cacheReadTokens":7680,"reasoningTokens":32}}}} +{"type":"assistant/chunk","seq":107,"time":1784893582468,"data":{"turn":1,"step":2,"chunk":{"type":"finish","reason":{"kind":"stop"}}}} +{"type":"assistant/message","seq":108,"time":1784893582469,"data":{"turn":1,"step":2,"content":[{"type":"reasoning","text":"Both files have been read. a.txt contains \"alpha\" and b.txt contains \"beta\". Now I just need to reply with the single word DONE."},{"type":"text","text":"DONE"}],"provenance":{"provider":"deepseek","model":"deepseek-v4-flash"},"usage":{"inputTokens":340,"outputTokens":35,"cacheReadTokens":7680,"reasoningTokens":32}},"sourceEventSeqs":[68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107],"surfaceOp":"append"} +{"type":"step/end","seq":109,"time":1784893582470,"data":{"turn":1,"step":2}} +{"type":"turn/end","seq":110,"time":1784893582470,"data":{"turn":1,"reason":{"kind":"completed"}}} diff --git a/apps/web/tests/snapshots/seeded-history/ui.expected.md b/apps/web/tests/snapshots/seeded-history/ui.expected.md new file mode 100644 index 0000000000..07f5b15b2d --- /dev/null +++ b/apps/web/tests/snapshots/seeded-history/ui.expected.md @@ -0,0 +1,36 @@ +- banner: + - navigation "会话层级": + - button "Use the read tool twice" [disabled] + - text: · 1 turns + - tablist: + - tab "Chat" [selected] + - tab "Trajectory" + - tab "Waterfall" +- text: "Use the read tool twice in one assistant message: read a.txt and b.txt. Then reply with the single word DONE and stop." +- button "Think The user wants me to read a.txt and b.txt, then reply with DONE. Let me read both files.": + - img + - text: Think The user wants me to read a.txt and b.txt, then reply with DONE. Let me read both files. +- button: + - img +- text: Read a.txt +- button: + - img +- text: Read b.txt +- button "Think Both files have been read. a.txt contains \"alpha\" and b.txt contains \"beta\". Now I just need to reply with the single word DONE.": + - img + - text: Think Both files have been read. a.txt contains "alpha" and b.txt contains "beta". Now I just need to reply with the single word DONE. +- paragraph: DONE +- text: cache hit 97% · 15,959 tokens · 1 turns · 2 steps +- textbox "输入消息，Enter 发送，Shift+Enter 换行" +- button "添加": + - img +- combobox "Plan mode": + - option "Plan" [selected] + - option "Agent" +- combobox "Access mode": + - option "Read-only" [selected] + - option "Read-write" +- combobox "Model": + - option "DeepSeek-V4-Pro High" [selected] + - option "DeepSeek-V4-Pro" +- button "发送" [disabled] diff --git a/apps/web/tsconfig.json b/apps/web/tsconfig.json index 998996304e..514cbe4d57 100644 --- a/apps/web/tsconfig.json +++ b/apps/web/tsconfig.json @@ -17,6 +17,15 @@ "src", "tests" ], + // The web e2e lane (harness + replay specs) boots the host spine and reads + // its Context merges — host-plane programs, checked in tsconfig.host.json; + // this client-registered project must not also hold them (one program + // cannot see both sides of the cordis Context merges). + "exclude": [ + "tests/harness.ts", + "tests/replay-round-trip.e2e.ts", + "tests/seeded-history.e2e.ts" + ], "references": [ { "path": "../../packages/client/web" diff --git a/tsconfig.host.json b/tsconfig.host.json index a13bcf35e3..a5f48ed22d 100644 --- a/tsconfig.host.json +++ b/tsconfig.host.json @@ -8,6 +8,10 @@ "rewriteRelativeImportExtensions": false }, "include": [ + "apps/web/tests/harness.ts", + "apps/web/tests/support.ts", + "apps/web/tests/replay-round-trip.e2e.ts", + "apps/web/tests/seeded-history.e2e.ts", "examples/*/src/**/*.ts", "examples/*/start.ts", "examples/*/tests/**/*.ts", From 2cd37e276518ef04489f5ddc0e08574827a9f3f8 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 24 Jul 2026 19:57:27 +0800 Subject: [PATCH 04/11] docs(testing): web e2e lane docs + Agent Note to implemented MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit testing.md gains the web browser snapshot tier entry (divergent DSH_SNAPSHOT=... test:web commands) and names apps/web/tests/snapshots/ as the web surface's snapshot home. The GUI testing note's tier map and lane map gain the e2e scenarios (both languages, pair re-recorded) and drop the stale verify-session-real references (those scripts left with the missions/ tree). packages/client/AGENTS.md check ladder covers the wire-carriage trigger and refresh/record commands. acp-snapshot README stops claiming the whole package is ACP-specific — its normalizers are transport-neutral with three consumers now. vitest.web.config.ts header carries TODO(ci-browser) with the staged-reversal pointer. The design-study Agent Note moves proposed/ -> implemented/ rewritten in present tense: all review decisions recorded (llm:false seam over the placeholder-key hack, providers-mode replay, whenIdle barrier stack, single aria golden + anchors, TUI-style inline modes over a suite factory, scrub-only header stance, CI deferral) with re-entry triggers under Deferred. --- .../2026-07-20-gui-testing-system.i18n.yaml | 4 +- .../process/2026-07-20-gui-testing-system.md | 8 +- .../2026-07-20-gui-testing-system.zh.md | 8 +- .../2026-07-24-web-gui-browser-e2e-lane.md | 88 ++++++++++++++++++ .../2026-07-24-web-gui-browser-e2e-lane.md | 92 ------------------- apps/web/tests/harness.ts | 10 +- docs/testing.md | 3 +- packages/client/AGENTS.md | 2 +- packages/support/acp-snapshot/README.md | 2 +- vitest.web.config.ts | 11 ++- 10 files changed, 114 insertions(+), 114 deletions(-) create mode 100644 .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md delete mode 100644 .agents/notes/proposed/testing/2026-07-24-web-gui-browser-e2e-lane.md diff --git a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.i18n.yaml b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.i18n.yaml index b21353698c..ca443d7a16 100644 --- a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write -2026-07-20-gui-testing-system.md: e42dafcdf37e48475e7d420eaad9600e6c20891c -2026-07-20-gui-testing-system.zh.md: e4ef6246e59e0ad6c0a3070a38c546f964e347fa +2026-07-20-gui-testing-system.md: b261bd2c84a628ab6fcdc29c59cf36a7b2428a76 +2026-07-20-gui-testing-system.zh.md: ecb8634695bd05359e6b590a825a4ad3604003b1 diff --git a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.md b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.md index e42dafcdf3..b261bd2c84 100644 --- a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.md +++ b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.md @@ -20,7 +20,7 @@ Cut along the architecture's natural test seams into three tiers, bottom-up: |---|---|---|---| | 1 Protocol isomorphism | `AbstractApiClient` + `toFetchHandler` (bidirectional data / rpcId / zod types / SSE streams / batching / timeouts) | **The full chain at the isomorphic point**: `InProcessApiClient(toFetchHandler(脚本化 impl))` skips the network but genuinely runs the wire serialization — zero browser, pure node env | `packages/host/apiproxy/tests/client-handler.spec.ts` | | 2 Object-layer orchestration | `Session`/`SessionManager`/`ConnectionController` (state machines and timing: stitching / dedup / paging / optimistic draft clearing / pendingBuffers / reconnect / backoff) | **The "event sequence in → snapshot out" golden path**: programmable fakes + deferreds controlling timing + fake timers controlling backoff | `packages/client/{runtime,connection}/tests/` | -| 3 Assembled presentation | Built artifacts × the real client loader and plugin composition | App-owned semantic snapshots boot all eight built client plugins under jsdom for deterministic cross-plugin state changes; bare Playwright smoke separately proves the real browser/carrier boundary, with real-host cases self-skipping without a key | `apps/web/tests/*.snapshot.ts`, `apps/web/tests/smoke-{fixture,real}.e2e.ts` | +| 3 Assembled presentation | Built artifacts × the real client loader and plugin composition | App-owned semantic snapshots boot all eight built client plugins under jsdom for deterministic cross-plugin state changes; bare Playwright smoke separately proves the real browser/carrier boundary, with real-host cases self-skipping without a key; the keyless browser e2e lane replays recorded session fixtures through the real in-process web assembly (`llm: false` + dsh-llm-replay) against conversation aria goldens ([web e2e lane](../testing/2026-07-24-web-gui-browser-e2e-lane.md)) | `apps/web/tests/*.snapshot.ts`, `apps/web/tests/smoke-{fixture,real}.e2e.ts`, `apps/web/tests/{replay-round-trip,seeded-history}.e2e.ts` | Inter-tier discipline: **each tier tests its own layer, upper tiers never re-test lower ones** — an app semantic snapshot pins only user-visible projection across the assembled plugin boundary, while Playwright smoke proves browser and carrier liveness; wire semantics belong to tier 1 and data semantics to tier 2. Pure-function layers (lineage/partial/notifier/fold-adapter) are tested directly with zero fakes in the same package's tests/ alongside tier 2. @@ -33,15 +33,15 @@ Inter-tier discipline: **each tier tests its own layer, upper tiers never re-tes |---|---|---|---| | Baseline | `pnpm run test:gui` | Tier 1+2 vitest (`packages/client packages/host`), seconds-fast, no browser, no server | Casually, after touching any GUI source | | Semantic snapshot | `DSH_EXAMPLE_MODE=lib pnpm run test:snapshot` | Keyless assembled-application semantics plus the repo's transport-specific expected outputs | After a human-visible GUI change; before delivery | -| Browser end-to-end | `pnpm run test:web` | Rebuilds the front-end dist first, then runs the tier-3 two-level smoke (fixture level + real-host level self-skip) | After touching the build surface/boot/carriage; before delivery | +| Browser end-to-end | `pnpm run test:web` | Rebuilds the front-end dist first, then runs the tier-3 browser set: the two-level smoke (fixture level + real-host level self-skip) plus the keyless replayed e2e scenarios (`DSH_SNAPSHOT=record`/`refresh` re-record fixtures / rewrite goldens) | After touching the build surface/boot/carriage; before delivery | | Gate | `pnpm run test:coverage` | The repo-wide gate (host and client GUI packages included, except annotated browser-grade exclusions) | The PR window | **Division of labor between the browser scripts and vitest**: Playwright owns browser/carrier black-box regression and long sequential user journeys; ordinary vitest owns data-layer semantics such as reference stability, timing, and wire shapes; snapshot vitest owns stable app-level semantic output through the built composition. These lanes complement each other rather than duplicating assertions. ## Anti-regression discipline -- **Every bug fix pins an assertion**: a browser-visible bug is pinned into the regression section of its owning verify script (one pin = one report line); a data-layer bug is pinned into the matching spec (precedent: the res-close misjudgment pinned in the webserver bridge suite — pure Node, reproduces in seconds, no longer needs the 12s browser sentinel as the only defense). -- **All-green on fixture is not done, the real host must pass too**: what the fixture short-circuits is exactly the wire carriage chain (node:http bridge close semantics, real network timing); both empirically confirmed bugs hid there. Changes touching connection/bridge/handler/SSE must run `verify-session-real`. +- **Every bug fix pins an assertion**: a browser-visible bug is pinned into its owning browser spec (smoke or e2e scenario); a data-layer bug is pinned into the matching spec (precedent: the res-close misjudgment pinned in the webserver bridge suite — pure Node, reproduces in seconds, no longer needs the 12s browser sentinel as the only defense). +- **All-green on fixture is not done, the real wire must pass too**: what the fixture short-circuits is exactly the wire carriage chain (node:http bridge close semantics, real network timing); both empirically confirmed bugs hid there. Changes touching connection/bridge/handler/SSE must run the browser lane (`pnpm run test:web`) — its keyless e2e scenarios drive the real HTTP/SSE carriage, and the with-key real-host smoke remains the live-model complement. - The code-on-disk-is-the-answer reconciliation workflow: when a behavior change lands and turns existing cases red, reconcile on the spot (fix the test or fix the code, with the RFC/contract as arbiter); no red left hanging. ## Consequences diff --git a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.zh.md b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.zh.md index e4ef6246e5..ecb8634695 100644 --- a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.zh.md +++ b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.zh.md @@ -20,7 +20,7 @@ GUI 栈需要考虑多种应用形态，同应用形态内的不同运行环境 |---|---|---|---| | 1 协议同构层 | `AbstractApiClient` + `toFetchHandler`（双向数据/rpcId/ZOD类型/SSE 流/合批/超时） | **同构点全链**：`InProcessApiClient(toFetchHandler(脚本化 impl))` 不过网络但真跑 wire 序列化——零浏览器、纯 node env | `packages/host/apiproxy/tests/client-handler.spec.ts` | | 2 对象层编排 | `Session`/`SessionManager`/`ConnectionController`（状态机与时序：缝合/去重/翻页/乐观清稿/pendingBuffers/重连/退避） | **「事件序列进→快照出」黄金路径**：可编程假体 + deferred 控时序 + fake timers 控退避 | `packages/client/{runtime,connection}/tests/` | -| 3 组装呈现层 | 构建产物 × 真实 client loader 与插件组合 | 归应用所有的语义快照会在 jsdom 下启动全部 8 个已构建的 client 插件，以固定确定性的跨插件状态变化；独立使用 Playwright 裸库的冒烟测试负责验证真实浏览器/承载层边界，真 host 用例在无密钥时自行跳过 | `apps/web/tests/*.snapshot.ts`、`apps/web/tests/smoke-{fixture,real}.e2e.ts` | +| 3 组装呈现层 | 构建产物 × 真实 client loader 与插件组合 | 归应用所有的语义快照会在 jsdom 下启动全部 8 个已构建的 client 插件，以固定确定性的跨插件状态变化；独立使用 Playwright 裸库的冒烟测试负责验证真实浏览器/承载层边界，真 host 用例在无密钥时自行跳过；无密钥浏览器 e2e 车道把录制的会话 fixture 通过真实进程内 web 组装（`llm: false` + dsh-llm-replay）回放，与会话区 aria 期望输出比对（[web e2e 车道](../testing/2026-07-24-web-gui-browser-e2e-lane.md)） | `apps/web/tests/*.snapshot.ts`、`apps/web/tests/smoke-{fixture,real}.e2e.ts`、`apps/web/tests/{replay-round-trip,seeded-history}.e2e.ts` | 层间纪律：**下层各测各的，上层不重测下层**：应用语义快照只固定组装后插件边界上的用户可见投影，Playwright 冒烟测试负责验证浏览器与承载层是否存活；wire 语义归 1 层，数据语义归 2 层。纯函数层（lineage/partial/notifier/fold-adapter）随 2 层同包 tests/ 零假体直测。 @@ -33,15 +33,15 @@ GUI 栈需要考虑多种应用形态，同应用形态内的不同运行环境 |---|---|---|---| | 基础 | `pnpm run test:gui` | 1+2 层 vitest（`packages/client packages/host`），秒级、无浏览器无 server | 改 GUI 任意源码后随手跑 | | 语义快照 | `DSH_EXAMPLE_MODE=lib pnpm run test:snapshot` | 无需密钥的组装应用语义，以及仓库按传输形态划分的预期输出 | 用户可见的 GUI 变更后；交付前 | -| 浏览器端到端 | `pnpm run test:web` | 先重建前端 dist，再跑 3 层双级 smoke（fixture 级 + 真 host 级 self-skip） | 改构建面/boot/承载后；交付前 | +| 浏览器端到端 | `pnpm run test:web` | 先重建前端 dist，再跑 3 层浏览器全集：双级 smoke（fixture 级 + 真 host 级 self-skip）加上无密钥回放 e2e 场景（`DSH_SNAPSHOT=record`/`refresh` 重录 fixture / 重写期望输出） | 改构建面/boot/承载后；交付前 | | 门禁 | `pnpm run test:coverage` | 全仓 gate（host 与 client GUI 包均纳入，仅排除带注释的浏览器级例外） | PR 窗口 | **浏览器脚本与 vitest 的分工**：Playwright 负责浏览器/承载层黑盒回归和较长的连续用户操作流程；普通 vitest 负责引用稳定性、时序和 wire 结构等数据层语义；快照 vitest 通过构建后的组合负责稳定的应用层语义输出。这些车道彼此互补，而不重复断言。 ## 防回归纪律 -- **修一个 bug 钉一条断言**：浏览器可见的 bug 钉进所属 verify 脚本的回归节（一钉一行 report）；数据层 bug 钉进对应 spec（先例：res-close 误判钉在 webserver 桥 suite——纯 Node 秒级复现，不再需要 12s 浏览器哨兵作唯一防线）。 -- **fixture 全绿不算完，真 host 也要过**：fixture 短路的恰是 wire 承载链（node:http 桥 close 语义、真网络时序），两次实证 bug 都藏在那里。改动触及连接/桥/handler/SSE 的，`verify-session-real` 必跑。 +- **修一个 bug 钉一条断言**：浏览器可见的 bug 钉进所属浏览器 spec（smoke 或 e2e 场景）；数据层 bug 钉进对应 spec（先例：res-close 误判钉在 webserver 桥 suite——纯 Node 秒级复现，不再需要 12s 浏览器哨兵作唯一防线）。 +- **fixture 全绿不算完，真 wire 也要过**：fixture 短路的恰是 wire 承载链（node:http 桥 close 语义、真网络时序），两次实证 bug 都藏在那里。改动触及连接/桥/handler/SSE 的，浏览器车道（`pnpm run test:web`）必跑——其无密钥 e2e 场景驱动真实 HTTP/SSE 承载，带密钥的真 host smoke 仍是真模型侧的补充。 - 落盘代码即答案的对表工作流：行为改动落盘打红既有用例时，当场对表校准（改测试还是改代码以 RFC/契约为裁），不留悬红。 ## Consequences diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md new file mode 100644 index 0000000000..fddd1e9a0c --- /dev/null +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md @@ -0,0 +1,88 @@ +# Agent Note: Keyless browser e2e lane for the web GUI + +Status: implemented + +## Problem + +The web GUI ships as a real assembled chain — chromium page → client plugin bundles → HTTP unary RPC + two SSE streams → `toFetchHandler`/apiproxy → `bootHost`'s agent loop, tools, and JSONL persistence — and no test exercised that chain keylessly and deterministically. The [GUI testing system](../process/2026-07-20-gui-testing-system.md) covers tier 1 (wire isomorphism in node), tier 2 (object-layer state machines), and tier-3 smokes, but the keyless smoke drives `FixtureApiClient` — no host, no wire, no agent loop — while the full-chain smoke needs `DEEPSEEK_API_KEY` and a live model, so it is nondeterministic and self-skips in keyless CI. The snapshot philosophy of [docs/testing.md](../../../../docs/testing.md) — record once with a key, replay forever keyless, refresh on format churn — already covers the ACP, headless `stream-json`, and TUI transcript surfaces; the web surface was the one assembled product shape without it. The gap is exactly where the two confirmed GUI P0s hid: the wire carriage chain the fixture client short-circuits. + +## Decision + +`pnpm run test:web` carries a keyless, deterministic browser e2e lane under `apps/web/tests/`: recorded session-log fixtures replayed through `@deepseek-ai/dsh-llm-replay` against the real in-process web assembly, asserting a normalized conversation aria golden plus in-process world state. No new package; the product deltas are the `BootHostOptions.llm` seam and two additive `dsh-llm-replay` surfaces. + +### Harness: `apps/web/tests/harness.ts` + +A plain shared-fixture module (the [testing-policy sanctioned shape](../../../../docs/testing.md)), not a package: the gate-worthy logic — replay derivation, session parsing, log scrubbing, persistence — lives in the gated packages `dsh-llm-replay`, `dsh-acp-snapshot`, and `dsh-session-persistence-jsonl`; what remains is boot wiring and browser glue, and chromium-driving code cannot hold per-file 100% coverage on the browserless coverage runners. + +`launchWebHarness()` boots the real web assembly in-process from the exported production functions — `startHost({ boot: { …, llm: false } })`, `installLlmReplay(host.ctx, { file, providers, paceMs })`, `mountWebPlugins(host.ctx, roster, anchor)`, `createHostWebPluginRegistry`, `startWebServer({ port: 0, … })`. This is the web analog of the TUI suite mounting the production bundle in-process ([TUI snapshots](2026-07-18-tui-terminal-state-snapshots.md)): the real entry boundary (`dsh web` bin arg-parsing, dist resolution) stays held by the keyless CLI smokes in `smoke-real.e2e.ts`, and the web surface has no `cordis.yml` to bypass — assembly is written in the app per the [GUI layering decision](../architecture/2026-07-19-gui-layering-and-rpc-protocol.md), a ruling this lane's design review explicitly reaffirmed (Loader-izing `dsh web` was declined; it would be its own proposal). Two deliberate assembly divergences from the `dsh web` shell, noted in the harness header: `workspaceContext: false` (recorded fixtures must not embed this repo's AGENTS.md) and `sessionTitleLlm` left at bootHost's disabled default (its fire-and-forget title call would share the session's replay cursor nondeterministically). + +The `llm: false` seam is the reviewed resolution of the keyless-boot question: `'deepseek' | false` on `BootHostOptions`, matching the `workspaceContext: Config | false` shape, with `RunningHost.ctx` JSDoc naming "filling a deliberately-open capability seam" as its third sanctioned use. Replay runs in providers-catalog mode with a published `contextWindow` (the TUI `PROVIDERS` shape), never catch-all: with no adapter registered, catch-all would leave `resolveModelContext` unroutable and `compact-basic`'s post-step pressure check would warn every step instead of being provably inert. + +`seedSession()` seeds cold sessions through the real persistence API — a throwaway `Context` mounting `SessionStore` + `SessionPersistenceJsonl` against the host's root, `create()` + `append()`, one `utimes` backdate for deterministic sidebar order (the `semantic-checkpoint.snapshot.ts` precedent) — never raw file writes, so the seeder knows nothing of bucket hashing, filename encoding, or compression, and the host's zstd default needs no boot knob. Seeds are validated at seed time (parseable, ending in `turn/end` — an open final turn would be mutated by resume's crash repair). + +### Determinism rules + +The barrier stack for a prompted turn, in order: (1) host-side `await agent.whenIdle()` under a timeout, keyed off the in-process `turn/end` — the idle flip follows the persistence flush, so one await covers turn completion and durability; (2) browser settled poll (streaming detached, final text visible); (3) any log harvest after `host.dispose()`. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync); file polling is banned (slow on NFS, superseded by `whenIdle`); `networkidle` is banned outright (never resolves while an SSE stream is open). + +No single-shot transient-DOM assertions: every hop from replay yield to React commit can coalesce chunks, so sampling `[data-streaming]` is a race by construction. Streaming incrementality is asserted from the persisted `assistant/chunk` events (model-visible ⟺ logged makes the log the authoritative proof). `dsh-llm-replay`'s opt-in `paceMs` (default absent = burst) is a realism knob so the browser observes genuinely incremental SSE; correctness never leans on it, and abort during a pace wait cancels promptly. + +Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Harness `close()` calls the `ReplayHandle.assertConsumed()` teardown check (every recorded script bound, every cursor drained), converting silent underruns and shifted bindings into crisp diagnostics. No vitest retry on the lane; one chromium per file, fresh context per scenario, one host per scenario; viewport pinned; selectors anchor on roles, `data-*` attributes, and visible text. + +### Expected outputs + +One committed golden per scenario: a normalized `ariaSnapshot()` of the conversation region (`ui.expected.md`) — uuid/cwd/workspace-basename/duration tokens normalized, captured poll-until-equal at the settled milestone — plus a few role/text anchor assertions that stay green under a semantics-preserving component rewrite while the golden churns reviewably. The aria tree is the mechanization of the client rule "assert what the user would see, never class names". World-state assertions ride `host.ctx` session events inline (which tools ran, `turn/end` completed) instead of a second committed log golden: the persisted-log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence, and re-pinning it here would double refresh cost against the tier discipline. `refresh` is the sole golden writer — a missing golden in replay mode fails with the healing command rather than self-bootstrapping. + +The typecheck plane split is structural: `apps/web/tests/{harness,support,replay-round-trip.e2e,seeded-history.e2e}.ts` are host-plane programs (they boot the host spine), so they are excluded from the client-registered `apps/web` project and included file-by-file in `tsconfig.host.json` — one program cannot hold both sides of the cordis `Context` merges. + +### Modes and fixtures + +`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless) as inline spec branches — the TUI shape, not a suite factory: at two scenarios the acp-snapshot factory machinery has no owner, and the genuinely shared parts are already exported (`scrubRequestHeaders`, `parseSessionLog`, `installLlmReplay`). Each spec splits into drive steps (type, send, `whenTurnSettled` — run in all modes, never waiting on model-content selectors, so record cannot hang on a live model answering differently) and assertion steps (replay/refresh only). Record = drive live through the real composer + harvest the in-memory `session.header`/`session.events` (the TUI `rawSessionLog` shape — no file decompression) + `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}` tokenization; a follow-up keyless refresh regenerates `ui.expected.md`. Both scenarios' fixtures were recorded against this assembly through this flow. A drift guard ties each spec's drive prompt to the fixture's recorded `user/message`. A fixture-inventory guard holds each scenario directory closed (exact file set, every JSONL a scrub fixed-point). Web fixtures scrub headers everywhere and pin no header class, following the TUI precedent over the strict [pinned-header](2026-07-06-pin-request-header-content-in-one-scenario.md) reading — see Deferred. + +### Scenarios + +1. **`replay-round-trip`** — new session, prompt through the real composer, replay streams reasoning + a `bash` tool call that really executes in the temp workspace + final text (paced 15ms). Asserts settled markdown, the aria golden, and inline world state (bash `tool/call`, completed `turn/end`, >10 chunk events). +2. **`seeded-history`** — a recorded session seeded cold; the sidebar lists it (group row → session row, collapsed by default), opening renders tool cards and text purely from the log through the implicit cold-resume attach inside `session.history` — zero model calls in replay, so no binding constraints; record mode drives the same turn live (real `read` tool against seeded workspace files) to produce the seed. + +### CI stance + +The lane ships gate-exempt inside `pnpm run test:web`, exactly as that config's header records. Adding chromium to CI would reverse the "no browser infrastructure in CI" premise in the [GUI testing note](../process/2026-07-20-gui-testing-system.md) and therefore requires its own Agent Note cross-linked from there, staged as a non-required job first with measured promotion criteria (consecutive green runs, wall time, zero-retry flake budget, runner browser-cache strategy). `TODO(ci-browser)` marks the seam. Scenarios are POSIX-oriented (the lane is not in the Windows matrix). + +## Prior art + +Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot + AI SDK, lobe-chat, open-webui, OpenHands, Chainlit, continue, cline, langfuse, gradio/streamlit; Playwright HAR/route, MSW, Polly/nock, WireMock, aimock). The dominant proven architecture for apps that own their backend is an in-process fake/replay model behind the real backend seam with everything downstream real (LibreChat's `LIBRECHAT_TEST_RUN_HOOK` fake model; ai-chatbot's `MockLanguageModelV3` + `simulateReadableStream`; continue's scripted mock provider classes) — which is what `dsh-llm-replay` already is. Browser-level SSE interception cannot exercise incremental rendering (`route.fulfill` delivers the whole body at once; playwright#33564) and leaves the server SSE stack untested, so projects use it only for edge cases. Chunk pacing as a fixture parameter recurs everywhere (LibreChat 10ms default with slow profiles; ai-chatbot 500ms); real models in CI rot (open-webui's suite grew 120-second timeouts, was disabled, then deleted); sessions are seeded at the persistence layer with controlled timestamps (LibreChat inserts backdated Mongo documents; langfuse seeds its DB). No surveyed project replays a recorded agent-event log through the real backend for UI tests — the closest are provider-level recorded fixtures (aimock) and frontend-level socket history emission (OpenHands MSW) — so the session-log-as-fixture design goes one step beyond prior art along the axis this repo's model-visible ⟺ logged invariant makes natural. + +## Alternatives considered + +**Browser-network SSE interception (`page.route`).** Rejected: `route.fulfill` cannot stream, so incremental token rendering is unexercisable and the server-side SSE/backpressure/close path — where both confirmed P0s hid — goes untested. + +**Mock HTTP provider at `DEEPSEEK_BASE_URL`.** Rejected as the lane's mechanism (kept for the one existing workspace-probe smoke): fixtures become hand-authored OpenAI SSE byte scripts, a second fixture format that drifts from the session-log format the rest of the repo records and replays; the adapter's real HTTP path is with-key e2e's job. + +**Growing the `?fixture` client.** Rejected: tier separation — `FixtureApiClient` exists to test the client shell without a server; everything below the client API seam stays untested by construction. + +**Placeholder `DEEPSEEK_API_KEY` + replay interception instead of the `llm: false` seam.** Rejected despite zero product change and two in-tree precedents: it satisfies `llm-deepseek`'s fail-loud key check with a lie and leaves a dead adapter mounted-but-intercepted; the seam matches an existing option shape and fails loud at the earliest resolvable point. + +**A `packages/support/web-snapshot` package with a `defineWebSnapshotSuite` factory.** Rejected: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and at two scenarios a factory generalizes from one consumer while the genuinely shared logic is already exported from gated packages. Re-entry trigger: a second web-shaped consumer or ≥6 scenarios with demonstrably drifting inline branches; the package boundary would then be drawn browser-free. + +**A committed normalized-session-log golden as a second expected surface.** Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers. Inline world-state assertions on `host.ctx` events keep the world-verification duty. + +**Spawning the `dsh web` bin with a `DSH_SNAPSHOT` replay branch.** Rejected: it needs a test-mode branch plus env plumbing in the product bin where the in-process route uses exported production functions; the bin's thin glue is covered by the keyless CLI smokes. Becomes free only if the web host is ever Loader-ized — declined in review, with the app-assembly ruling reaffirmed. + +**Changing the wire protocol for testability.** Rejected: the contract already has a first-class keyless isomorphic seam (`InProcessApiClient(toFetchHandler(api))`), the per-event unbatched SSE is exactly what makes replay observable in a browser, and testing a wire we no longer ship would invert the tier's purpose. + +**Real-model browser tests as the keyless lane.** Rejected: nondeterministic by construction; the surveyed cautionary case (open-webui) grew unbounded timeouts and was deleted. The with-key W5 smoke stays as the live-model complement. + +**A client `data-dsh-busy` settled signal.** Deferred: the multi-condition settled polls proved sufficient at two scenarios and the host-side `whenIdle` barrier does the heavy lifting. Re-entry trigger: the first settled-poll flake, or a scenario needing a state the DOM does not expose. + +## Testing + +The lane itself: `pnpm run test:web` runs both scenarios keylessly alongside the existing smoke pair; `DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/` re-records a scenario's fixture against the live model; `DSH_SNAPSHOT=refresh` rewrites both aria goldens keylessly. The `llm: false` seam is pinned by `packages/host/runtime/tests/host-runtime.spec.ts` (keyless boot, NO_ADAPTER at first stream, embedder fill through ctx); `paceMs` validation, pacing floor, abort-during-pace, and both `assertConsumed` failure shapes are pinned in `packages/support/llm-replay/tests/llm-replay.spec.ts`. + +## Deferred + +- **Web header-class pin**: web fixtures tokenize `{{system}}`/`{{tools}}` everywhere and no scenario pins bootHost's composed prompt/tool schemas (`TODO(web-header-pin)` — the harness `recordFixture` JSDoc marks it). Following the TUI scrub-everywhere precedent; revisit when the web assembly's header diverges from the repl composition it mirrors. +- **CI browser provisioning**: reversal of the no-browser-in-CI ruling, staged criteria above (`TODO(ci-browser)`). +- **Follow-up-prompt-after-resume scenario**: the history/live stitch path over the real wire; add as its own scenario when that code changes or regresses. + +## Consequences + +The web surface gains its record-once/replay-forever tier: the real chromium → SSE → apiproxy → loop → tools → persistence chain runs keylessly in ~10-30s, deterministic across repeat runs, with fixtures owned and re-recordable by the lane itself. Costs accepted: every intentional conversation-UI change ends with a keyless `DSH_SNAPSHOT=refresh` (golden churn is reviewed diff, anchors keep semantic green); the aria format is Playwright-owned — the one committed snapshot format the repo does not control — so playwright version bumps must be deliberate bump-and-refresh commits (the dependency floats `^1.49.0` in `apps/web/package.json`; pin exactly if churn bites); replay's first-call-order binding constrains scenarios to one prompting session each, with the consumption assertion as the tripwire; `compact-basic` shares the session's replay cursor and stays inert only under the published 128k catalog window; and the lane guards regressions only where it runs (locally, `test:web`) until the CI reversal is separately decided. diff --git a/.agents/notes/proposed/testing/2026-07-24-web-gui-browser-e2e-lane.md b/.agents/notes/proposed/testing/2026-07-24-web-gui-browser-e2e-lane.md deleted file mode 100644 index cc7504dae7..0000000000 --- a/.agents/notes/proposed/testing/2026-07-24-web-gui-browser-e2e-lane.md +++ /dev/null @@ -1,92 +0,0 @@ -# Agent Note: Keyless browser e2e lane for the web GUI - -Status: proposed - -## Problem - -The web GUI ships as a real assembled chain — chromium page → nine client plugin bundles → HTTP unary RPC + two SSE streams → `toFetchHandler`/apiproxy → `bootHost`'s agent loop, tools, and JSONL persistence — and no test exercises that chain keylessly and deterministically. The [GUI testing system](../../implemented/process/2026-07-20-gui-testing-system.md) covers tier 1 (wire isomorphism in node), tier 2 (object-layer state machines), and a tier-3 smoke pair, but the keyless smoke (`apps/web/tests/smoke-fixture.e2e.ts`) drives `FixtureApiClient` behind `?fixture` — no host, no wire, no agent loop — while the full-chain smoke (`smoke-real.e2e.ts`) needs `DEEPSEEK_API_KEY` and a live model, so it is nondeterministic and self-skips in keyless CI. The snapshot philosophy of [docs/testing.md](../../../../docs/testing.md) — record once with a key, replay forever keyless, refresh on format churn — already covers the ACP, headless `stream-json`, and TUI transcript surfaces; the web surface is the one assembled product shape without it. The gap is exactly where the two confirmed GUI P0s hid: the wire carriage chain the fixture client short-circuits. - -## Proposal - -Add a keyless, deterministic browser e2e lane under `apps/web/tests/`, driven by recorded session-log fixtures replayed through `@deepseek-ai/dsh-llm-replay`, asserting the rendered accessibility tree plus in-process world state. No new package; no product-code change except (open question 1) an LLM composition knob. - -### Harness: `apps/web/tests/harness.ts` - -A plain shared-fixture module (the [testing-policy sanctioned shape](../../../../docs/testing.md)), not a package: the gate-worthy logic this lane needs — replay derivation, session parsing, log scrubbing, persistence — already lives in gated packages (`dsh-llm-replay`, `dsh-acp-snapshot`, `dsh-session-persistence-jsonl`); what remains is boot wiring and browser glue, and chromium-driving code cannot hold per-file 100% coverage on the browserless coverage runners. - -`launchWebHarness()` boots the real web assembly in-process from the exported production functions — `startHost({ boot: { persistenceRoot: , workspaceContext: false, cwd: } })`, `installLlmReplay(host.ctx, { file, childFiles, providers })`, `mountWebPlugins(host.ctx)`, `createHostWebPluginRegistry`, `startWebServer({ port: 0, distIndex, apiHandler: host.handler, webPlugins })` — and returns `{ baseUrl, host, workspaceCwd, close }`. This is the web analog of the TUI suite mounting the production bundle in-process ([TUI snapshots](../../implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)): the real entry boundary (`dsh web` bin arg-parsing and dist resolution) stays held by the existing keyless CLI smokes in `smoke-real.e2e.ts`, and the web surface has no `cordis.yml` to bypass — assembly is written in the app per the [GUI layering decision](../../implemented/architecture/2026-07-19-gui-layering-and-rpc-protocol.md). Replay runs in providers-catalog mode with a `contextWindow` (the TUI suite's `PROVIDERS` shape), never catch-all: with no adapter registered, catch-all mode would make `compact-basic`'s `resolveModelContext` throw into its post-step catch every step, spamming warnings and silently disabling the pressure path instead of proving it inert. - -`seedSession(host, fixtureText)` seeds cold sessions through the real persistence API — a throwaway `Context` mounting `SessionStore` + `SessionPersistenceJsonl` against the host's root, `create()` + `append()`, one `utimes` for deterministic sidebar order (the precedent is `examples/acp-agent/tests/semantic-checkpoint.snapshot.ts`) — never raw file writes, so the seeder needs no knowledge of bucket hashing, filename encoding, or compression. Seeds are validated at seed time (parseable, `seq`-contiguous, ending in `turn/end`) so fixture drift fails loud at the earliest resolvable point rather than as silently dropped frames in the client; a seed not ending in `turn/end` would be mutated by resume's crash repair. - -### Determinism rules - -The barrier stack, in order, for a prompted turn: (1) host-side `await agent.whenIdle()` under a timeout — the idle flip happens after both the `turn/end` append and the persistence flush, so one await covers turn completion and durability; (2) browser settled poll — streaming node detached, composer restored, final text visible; (3) log harvest only after `host.dispose()`. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync), and file polling is banned (slow on NFS, superseded by `whenIdle`). For history-open scenarios the barrier is a poll for the last expected message's text, then a poll-until-equal aria capture. `networkidle` is banned outright — it never resolves while an SSE stream is open. - -No single-shot transient-DOM assertions: every hop from replay yield to React commit can coalesce chunks, so sampling `[data-streaming]` is a race by construction. Streaming incrementality is asserted from the persisted `assistant/chunk` events (model-visible ⟺ logged makes the log the authoritative proof), optionally corroborated by an in-page MutationObserver latch armed before send — observers cannot miss a commit; polls can. `dsh-llm-replay` gains an opt-in `paceMs` config field (default absent = today's instant yield) as a realism knob so the browser observes genuinely incremental SSE; correctness never leans on the pace. - -Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Harness `close()` asserts every replay script was fully consumed (all scripts bound, every cursor at end), converting silent underruns and shifted bindings into crisp diagnostics; this is a small additive stats handle on `installLlmReplay`. No vitest retry on the lane — a retried-green race is a flake deferred, and only the chromium launch itself may retry inside the harness, logged. One chromium per file, fresh browser context per scenario, one host per scenario; viewport pinned; selectors anchor on roles, `data-*` attributes, and visible text only. - -### Expected outputs - -One committed golden per scenario: a normalized `ariaSnapshot()` of the conversation region only (`ui.expected.md`) — uuid/cwd/duration tokens normalized, sidebar and other time-bearing chrome structurally excluded, captured poll-until-equal at the settled milestone. The accessibility tree is the mechanization of the client rule "assert what the user would see, never class names": it survives CSS-module hash churn, styling rewrites, and DOM restructuring, and a wholesale component rewrite refreshes it keylessly. Alongside the golden, three or four targeted role/text anchor assertions (heading level, `pre code` content, tool-row accessible name) keep green anchors under a semantics-preserving rewrite so a churned golden diff is reviewable against surviving anchors. World-state assertions ride `host.ctx` session events inline (which tools ran, `turn/end` completed, no error) instead of a second committed log golden: the persisted-log surface is already pinned by the ACP/headless/TUI suites through the same loop and persistence plugins, and re-pinning it here would double refresh cost for no new regression class. `playwright` gets pinned exactly in `apps/web/package.json` — the aria format is Playwright-owned, the one snapshot format in this repo we do not own, so version bumps must be deliberate bump-and-refresh commits. - -### Modes and fixtures - -`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless), as inline branches in the specs — the TUI suite's shape, not a suite-factory: at two scenarios the acp-snapshot factory machinery (scenario tables, pinning classes, Windows sidecars, packed-row stabilization) has no owner, and the genuinely shared parts are already exported (`scrubRequestHeaders`, `normalizeSessionLog`, `parseSessionLog`, `installLlmReplay`). Each scenario script splits into drive steps (type, send, `whenIdle`-generic waits — run in all modes, never waiting on model-content selectors) and interaction/assertion steps (expand reasoning, aria capture — replay/refresh only), so record mode cannot hang on a live model answering with a different tool count. Record = drive + harvest the in-memory `session.header` + `session.events` (the TUI `rawSessionLog` shape — no file decompression, so `bootHost` needs no compression knob) + scrub via `scrubRequestHeaders` + a mandatory keyless refresh to regenerate `ui.expected.md`. Web fixtures scrub headers everywhere and pin nowhere, matching the TUI precedent; whether the web surface must instead own a header-class pin per the [pinned-header discipline](../../implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md) is open question 3. Seeds are recorded fixtures under the same inventory and refresh discipline as replay fixtures, never hand-authored one-offs, so `DSH_SNAPSHOT=refresh` heals every committed surface after intentional shape churn and only `assistant/chunk`-shape churn escalates to re-record. A TUI-style `afterAll` fixture guard holds the inventory closed (expected files present, every fixture scrub-fixed-point, no orphan directories). - -### Demo scenarios - -1. **`fresh-round-trip`** — new session, prompt, replay streams reasoning + markdown + a `bash` tool call that really executes (`echo` in the temp workspace) + final text. Asserts settled markdown semantics (heading, code block), the tool card row, composer restore, the aria golden, and inline world state (bash `tool/call` + completed `turn/end` in the session events). The keyless version of the with-key W5 flows. -2. **`seeded-history`** — a recorded session seeded cold; the sidebar lists it, opening it renders tool cards and collapsible reasoning purely from the log. This exercises the implicit cold-resume attach (`session.history` resumes via `agentFor`), cold summaries, history pagination views, and the client fold of historical events — the surface nothing else covers — with zero model calls, so no replay-binding constraints at all. A follow-up-prompt-after-resume scenario is deliberately deferred until the history/live stitch path changes or regresses. - -### Lane wiring and CI stance - -The lane rides `vitest.web.config.ts` (`pnpm run test:web`, serial), which stays gate-exempt exactly as its header comment records. Adding chromium to CI would reverse the "no browser infrastructure in CI" premise recorded in the [GUI testing system note](../../implemented/process/2026-07-20-gui-testing-system.md) and therefore requires its own Agent Note cross-linked from that note, staged as: non-required CI job first, promotion criteria measured (consecutive green runs, wall time, zero-retry flake budget, browser cache strategy on the enterprise runners, whether the runner images carry the chromium system libraries). Deferred out of this proposal; a `TODO(ci-browser)` marks the seam. Scenarios are `posixOnly` initially. Docs updated in the same implementation PR: the [testing policy](../../../../docs/testing.md) names `apps/web/tests/snapshots/` as the web surface's snapshot home with its divergent `DSH_SNAPSHOT=… pnpm run test:web` commands, the GUI testing note's tier map gains the lane (and drops its stale references to the deleted `missions/scripts/verify-*` files), `packages/client/AGENTS.md`'s check ladder mentions it, and the `dsh-acp-snapshot` README's "ACP-specific by design" sentence is corrected — this lane is the third consumer of its normalizers. - -### Open questions - -1. **LLM seam.** Two viable shapes. (a) `BootHostOptions.llm?: 'deepseek' | false` — an assembly toggle in the one module that owns assembly, matching the existing `workspaceContext: Config | false` shape and the reserved-knob sentence in `start.ts`; `llm: false` mounts no adapter, the harness fills the open seam on `host.ctx`, misuse fails loud at the first stream with `NO_ADAPTER`, and keyless boots stop needing any key. (b) Zero product change — a placeholder `DEEPSEEK_API_KEY` env var satisfies `llm-deepseek`'s load-time presence check (twice-precedented in-tree) and replay intercepts ahead of the mounted adapter. (a) is cleaner semantics and honest keylessness at the cost of a test-motivated product field; (b) is free but satisfies a fail-loud check with a lie and leaves a dead adapter mounted. Recommendation: (a), shaped minimal. -2. **Loader-izing `dsh web`.** Making the web host `cordis.yml`-driven like every example would give the ACP-style `cordis.snapshot.yml` replay overlay for free and align with "everything is a plugin", but it reverses the settled "assembly is written in the app" ruling and is a product-architecture decision on its own merits — its own proposal if wanted; this lane does not need it. -3. **Header-class pin.** Strict reading of the pinned-header discipline wants one web scenario pinning `bootHost`'s composed prompt + tool schemas (a header class no ACP scenario covers); the TUI precedent scrubs everywhere and pins nowhere. Cheap middle: pin sidecars on `fresh-round-trip` at record time. Recommendation: follow TUI now (scrub-only), revisit when the web assembly's header diverges further from the repl composition it mirrors. -4. **Golden breadth.** Full conversation-region aria golden (adopted above) versus targeted assertions only. The golden is the "assembled transcript" duty for user-visible changes; the cost is a keyless refresh on every component rewrite. Recommendation: keep the golden + anchors. -5. **Client settled signal.** A `data-dsh-busy` attribute derived from the object layer's pending-RPC/active-stream state would replace multi-condition settled polls with one selector. Presentation-plane observability, no session-log leak — but the current polls suffice for two scenarios. Recommendation: defer until a settled-poll flake actually appears. - -## Prior art - -Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot + AI SDK, lobe-chat, open-webui, OpenHands, Chainlit, continue, cline, langfuse, gradio/streamlit; Playwright HAR/route, MSW, Polly/nock, WireMock, aimock). The dominant proven architecture for apps that own their backend is an in-process fake/replay model behind the real backend seam with everything downstream real (LibreChat's `LIBRECHAT_TEST_RUN_HOOK` fake model; ai-chatbot's `MockLanguageModelV3` + `simulateReadableStream`; continue's scripted mock provider classes) — which is what `dsh-llm-replay` already is. Browser-level SSE interception cannot exercise incremental rendering (`route.fulfill` delivers the whole body at once; playwright#33564) and leaves the server SSE stack untested, so projects use it only for edge cases. Chunk pacing as a fixture parameter recurs everywhere (LibreChat 10ms default with slow profiles; ai-chatbot 500ms); real models in CI rot (open-webui's suite grew 120-second timeouts, was disabled, then deleted); sessions are seeded at the persistence layer with controlled timestamps (LibreChat inserts backdated Mongo documents; langfuse seeds its DB). No surveyed project replays a recorded agent-event log through the real backend for UI tests — the closest are provider-level recorded fixtures (aimock) and frontend-level socket history emission (OpenHands MSW) — so the session-log-as-fixture design goes one step beyond prior art along the axis this repo's model-visible ⟺ logged invariant makes natural. - -## Alternatives considered - -**Browser-network SSE interception (`page.route`).** Rejected: `route.fulfill` cannot stream, so incremental token rendering is unexercisable and the server-side SSE/backpressure/close path — where both confirmed P0s hid — goes untested. - -**Mock HTTP provider at `DEEPSEEK_BASE_URL`.** Rejected as the lane's mechanism (kept for the one existing workspace-probe smoke): fixtures become hand-authored OpenAI SSE byte scripts, a second fixture format that drifts from the session-log format the rest of the repo records and replays; the adapter's real HTTP path is with-key e2e's job. - -**Growing the `?fixture` client.** Rejected: tier separation — `FixtureApiClient` exists to test the client shell without a server; everything below the client API seam stays untested by construction. - -**A `packages/support/web-snapshot` package with a `defineWebSnapshotSuite` factory.** Rejected for now: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and at two scenarios the factory generalizes from one consumer while the genuinely shared logic already lives exported in `dsh-llm-replay`/`dsh-acp-snapshot`. Re-entry trigger: a second web-shaped consumer or ≥6 scenarios with demonstrably drifting inline branches; the package boundary would then be drawn browser-free. - -**A committed normalized-session-log golden as a second expected surface.** Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers, against the tier discipline. Inline world-state assertions on `host.ctx` events keep the world-verification duty. - -**Spawning the `dsh web` bin with a `DSH_SNAPSHOT` replay branch.** Rejected for now: it needs a test-mode branch plus env plumbing in the product bin where the in-process route uses exported production functions with zero product change; the bin's thin glue is covered by the keyless CLI smokes. Becomes free if the web host is ever Loader-ized (open question 2). - -**Changing the wire protocol for testability.** Rejected: the contract already has a first-class keyless isomorphic seam (`InProcessApiClient(toFetchHandler(api))`), the per-event unbatched SSE is exactly what makes replay observable in a browser, and testing a wire we no longer ship would invert the tier's purpose. - -**Real-model browser tests as the keyless lane.** Rejected: nondeterministic by construction; the surveyed cautionary case (open-webui) grew unbounded timeouts and was deleted. The with-key W5 smoke stays as the live-model complement. - -## Acceptance criteria - -- `pnpm run test:web` keyless passes the two scenarios deterministically (no vitest retry), alongside the existing smoke pair, on a checkout with built client bundles and the frontend dist. -- Replay asserts: aria golden equality at the settled milestone, anchor role/text assertions, inline world-state event assertions, zero pageerrors, zero connection-loss/gap-repair console warnings, all replay scripts fully consumed at teardown. -- `DSH_SNAPSHOT=record` with a key re-records `fresh-round-trip` (drive steps only), rewrites its `session.jsonl` scrubbed, and a follow-up `DSH_SNAPSHOT=refresh` regenerates `ui.expected.md` keylessly; `refresh` alone heals goldens after intentional non-chunk shape churn. -- The seeded scenario renders history through the real cold-resume path with zero model calls and leaves the seed fixture byte-identical (closedness validated at seed time). -- Fixture guard holds the snapshot inventory closed; failure produces a bundle under `.artifacts/` (screenshot, console, pageerrors, persistence copy, actual-vs-expected aria). -- Docs land in the same PR: testing.md web-lane entry, GUI testing note tier map + stale verify-script cleanup, client AGENTS.md ladder, acp-snapshot README correction, this note moved to `implemented/` rewritten in present tense. - -## Risks - -- **The aria format is Playwright-owned** — the one committed snapshot format the repo does not control; a version bump can churn every golden. Mitigated by an exact version pin in `apps/web` and a documented bump-and-refresh procedure; residual risk accepted. -- **Replay's first-call-order binding** stays fragile under concurrent browser-driven sessions; the lane constrains scenarios to one prompting session each (the seeded scenario prompts none), and the teardown consumption assertion turns violations into diagnostics rather than surreal transcripts. -- **`compact-basic` shares the session's replay cursor** — a pressure-triggered summarize would consume a script entry; inert for small fixtures under the 128k catalog window, and the consumption assertion catches it if a fixture ever grows past the threshold. -- **CI remains browserless for now**, so the lane guards regressions only where it is run (locally and in any future non-required job) until the CI reversal is separately decided; the runner images' chromium-library situation is unverified. -- **Record-mode nondeterminism** is contained but not eliminated by the drive/assert split: a live model may still produce a transcript whose replay violates a scenario's assertions, requiring prompt tuning at record time (bounded by terse prompts and a chunk-count warning in record mode). -- **jsdom-lane overlap**: component-level rendering is already covered per-plugin; this lane must stay at assembled-transcript altitude (whole-region golden + anchors) or it starts re-testing tier 2 and paying double maintenance. diff --git a/apps/web/tests/harness.ts b/apps/web/tests/harness.ts index 6f885da0c8..71c3ac9344 100644 --- a/apps/web/tests/harness.ts +++ b/apps/web/tests/harness.ts @@ -243,11 +243,11 @@ export function rawSessionLog(session: Session): string { /** * Record-mode fixture write-back: harvest the live session, scrub request - * headers to {{system}}/{{tools}} (the web lane pins no header class — a - * deliberate deviation logged in the Agent Note's deferred work), tokenize - * the run-local session id and cwd ({{sessionId}}/{{cwd}}, the committed ACP - * fixture convention — re-records then diff only on real content), and write - * the committed fixture. + * headers to {{system}}/{{tools}} (TODO(web-header-pin): the web lane pins no + * header class — a deliberate deviation logged in the Agent Note's deferred + * work), tokenize the run-local session id and cwd ({{sessionId}}/{{cwd}}, + * the committed ACP fixture convention — re-records then diff only on real + * content), and write the committed fixture. * @param harness - the record-mode harness. * @param sessionId - the driven session. * @param fixturePath - the committed session.jsonl / seed.jsonl target. diff --git a/docs/testing.md b/docs/testing.md index 85798cee3e..5841cd5f29 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -8,6 +8,7 @@ How this repo tests, tier by tier, and the rules that keep a green suite meaning - **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped. - **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e Agent Note](../.agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.md)). - **Snapshot** (`pnpm run test:snapshot`): transport-specific keyless expected outputs cover external presentation. ACP suites boot the real example subprocess, replay a recorded session, and diff normalized JSON-RPC plus the re-persisted log ([ACP snapshot Agent Note](../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md)); the headless suite independently pins `stream-json` through its real one-shot subprocess. TUI completed journeys replay recorded primary/child JSONL through the real agent loop and tools before projecting ANSI into semantic terminal-state expected outputs; package-local snapshots retain transient renderer states, and a real PTY conversation covers the process boundary ([TUI snapshot Agent Note](../.agents/notes/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)). Use `pnpm run test:snapshot:record` when a model transcript must change and `pnpm run test:snapshot:refresh` when committed replay input remains correct; review every JSONL and expected-output diff. System-prompt/tool-schema content is pinned by one ACP scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header Agent Note](../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)). +- **Web browser snapshot** (inside `pnpm run test:web`, gate-exempt like the rest of that lane): the web GUI's keyless assembled-transcript tier — a real chromium over the real in-process web assembly (`llm: false` + `dsh-llm-replay`), scenario fixtures and normalized conversation aria goldens under `apps/web/tests/snapshots/`. Its record/refresh commands diverge from `test:snapshot` (`DSH_SNAPSHOT=record pnpm run test:web` re-records against the live model; `DSH_SNAPSHOT=refresh` rewrites goldens keylessly); the [web e2e lane Agent Note](../.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md) owns the design, and CI browser provisioning is deferred there. ## The with-key policy: inference is cheap here @@ -41,4 +42,4 @@ An e2e assertion re-runs the command or re-reads the file externally; a keyword ## When a snapshot test is required -Every non-trivial model- or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP surfaces use `examples//tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the `stream-json` snapshot and replay fixtures. Completed interactive-terminal journeys use JSONL-driven scenarios under `examples/tui-agent/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. New capability seams, lifecycle shapes, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation. +Every non-trivial model- or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP surfaces use `examples//tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the `stream-json` snapshot and replay fixtures. Completed interactive-terminal journeys use JSONL-driven scenarios under `examples/tui-agent/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. Browser-rendered web GUI journeys use `apps/web/tests/snapshots/` through the web e2e lane's harness. New capability seams, lifecycle shapes, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation. diff --git a/packages/client/AGENTS.md b/packages/client/AGENTS.md index 5bde15dc2c..3c96ac1e85 100644 --- a/packages/client/AGENTS.md +++ b/packages/client/AGENTS.md @@ -65,7 +65,7 @@ The GUI test structure (three tiers, lane map) is settled in the [GUI testing sy Run the narrowest rung that covers what you touched; escalate only when the change surface demands it. 1. **Every GUI code change** — `pnpm run test:gui` (seconds; no browser, no server): the client suites plus the host-side GUI packages. This is the inner loop; run it as freely as a typecheck. -2. **Changes to the build surface, boot wiring, or static serving** (`apps/web`, vite config, `dsh-host-webserver`) — additionally `pnpm run test:web`: rebuilds the frontend dist, then runs the browser smoke pair (the real-host case self-skips without `DEEPSEEK_API_KEY`). +2. **Changes to the build surface, boot wiring, static serving, or the wire carriage** (`apps/web`, vite config, `dsh-host-webserver`, connection/handler/SSE) — additionally `pnpm run test:web`: rebuilds the frontend dist, then runs the browser smoke pair (the real-host case self-skips without `DEEPSEEK_API_KEY`) plus the keyless replayed e2e scenarios (`DSH_SNAPSHOT=refresh` rewrites their aria goldens after an intentional conversation-UI change; `DSH_SNAPSHOT=record` re-records fixtures with a key). 3. **Before a PR** — `pnpm run check:pre-push` (the repo-wide gate ladder). Between PR windows this rung is not expected on every commit. If `test:gui` is red on code you did not touch, neither silently fix nor ignore it: note it in your handoff so it lands in the next PR window's sweep. diff --git a/packages/support/acp-snapshot/README.md b/packages/support/acp-snapshot/README.md index 05a4974c19..1243cd5026 100644 --- a/packages/support/acp-snapshot/README.md +++ b/packages/support/acp-snapshot/README.md @@ -42,7 +42,7 @@ Every scenario compares `stdout.expected.jsonl` with cwd-rooted separators canon The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config Agent Note](../../../.agents/notes/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log expected outputs, and each pin's prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md). -Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). Session config options are scriptable too: the `setConfigOption` step switches a knob over `session/set_config_option`, and `setConfigOptionExpectError` asserts the bridge rejects an unknown id or out-of-vocabulary value (the error frame stays in the transcript). +Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). The launcher and suite factory are ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection` — while the normalizers are transport-neutral session-log/text helpers also consumed by the TUI snapshot suite and the web browser e2e lane. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). Session config options are scriptable too: the `setConfigOption` step switches a knob over `session/set_config_option`, and `setConfigOptionExpectError` asserts the bridge rejects an unknown id or out-of-vocabulary value (the error frame stays in the transcript). ## Model Experience diff --git a/vitest.web.config.ts b/vitest.web.config.ts index de220c9f12..bcca93e5dd 100644 --- a/vitest.web.config.ts +++ b/vitest.web.config.ts @@ -1,10 +1,13 @@ import tsconfigPaths from 'vite-tsconfig-paths' import { defineConfig } from 'vitest/config' -// Web smoke lane (GUI, gate-exempt — not part of the CI sequence yet): built -// page + real chromium, so it lives outside the unit/e2e includes. The -// real-host test self-skips without DEEPSEEK_API_KEY; the fixture test is -// keyless and deterministic. +// Web browser lane (GUI, gate-exempt — not part of the CI sequence yet): +// built page + real chromium, so it lives outside the unit/e2e includes. The +// real-host smoke self-skips without DEEPSEEK_API_KEY; the fixture smoke and +// the replayed e2e scenarios are keyless and deterministic. +// TODO(ci-browser): running this lane in CI requires chromium provisioning +// and reverses the no-browser-in-CI ruling — staged criteria in +// .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md. try { // Node >= 21.7 native; throws when the file does not exist. process.loadEnvFile(new URL('.env', import.meta.url).pathname) From d08050ab69256cdc5f4183dd0b56dbad475fac59 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 24 Jul 2026 20:30:33 +0800 Subject: [PATCH 05/11] =?UTF-8?q?chore(web-e2e):=20gate=20fixes=20?= =?UTF-8?q?=E2=80=94=20catalog,=20budgets,=20knip?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Regenerate config-catalog for the llm-replay paceMs row; condense the testing.md web-lane entry to pointer form and raise its ceiling 1020->1060 (the two-sentence tier entry for a genuinely new surface does not fit the old ceiling after relocation-first trims); internalize two harness helpers knip flagged (rawSessionLog/normalizeAria are module-internal). --- apps/web/tests/harness.ts | 9 ++----- docs/config-catalog.md | 2 +- docs/testing.md | 4 +-- .../llm-replay/tests/llm-replay.spec.ts | 27 ++++++++++++++++++- scripts/doc-budgets.manifest.json | 2 +- 5 files changed, 32 insertions(+), 12 deletions(-) diff --git a/apps/web/tests/harness.ts b/apps/web/tests/harness.ts index 71c3ac9344..b2941c62e0 100644 --- a/apps/web/tests/harness.ts +++ b/apps/web/tests/harness.ts @@ -230,10 +230,8 @@ export async function launchWebHarness(options: LaunchOptions = {}): Promise JSON.stringify(event)), @@ -333,11 +331,8 @@ export async function seedSession(harness: WebHarness, fixtureText: string, id: /** * Normalize an aria snapshot: uuid, cwd, workspace-basename, and duration * volatility collapse to stable tokens. - * @param snapshot - raw ariaSnapshot text. - * @param workspaceCwd - the harness workspace (basename doubles as the header breadcrumb). - * @returns tokenized snapshot text. */ -export function normalizeAria(snapshot: string, workspaceCwd: string): string { +function normalizeAria(snapshot: string, workspaceCwd: string): string { // The header breadcrumb renders the workspace's basename, not the full // path, so both spellings must collapse to the token. const base = workspaceCwd.split('/').pop()! diff --git a/docs/config-catalog.md b/docs/config-catalog.md index c06fef3d4d..1b546be67d 100644 --- a/docs/config-catalog.md +++ b/docs/config-catalog.md @@ -643,7 +643,7 @@ export interface ReplayModelConfig { } ``` -Source: [`packages/support/llm-replay/src/index.ts:453`](../packages/support/llm-replay/src/index.ts) +Source: [`packages/support/llm-replay/src/index.ts:454`](../packages/support/llm-replay/src/index.ts) ## `@deepseek-ai/dsh-llm-retry` diff --git a/docs/testing.md b/docs/testing.md index 5841cd5f29..78ce3c2856 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -8,7 +8,7 @@ How this repo tests, tier by tier, and the rules that keep a green suite meaning - **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped. - **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e Agent Note](../.agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.md)). - **Snapshot** (`pnpm run test:snapshot`): transport-specific keyless expected outputs cover external presentation. ACP suites boot the real example subprocess, replay a recorded session, and diff normalized JSON-RPC plus the re-persisted log ([ACP snapshot Agent Note](../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md)); the headless suite independently pins `stream-json` through its real one-shot subprocess. TUI completed journeys replay recorded primary/child JSONL through the real agent loop and tools before projecting ANSI into semantic terminal-state expected outputs; package-local snapshots retain transient renderer states, and a real PTY conversation covers the process boundary ([TUI snapshot Agent Note](../.agents/notes/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)). Use `pnpm run test:snapshot:record` when a model transcript must change and `pnpm run test:snapshot:refresh` when committed replay input remains correct; review every JSONL and expected-output diff. System-prompt/tool-schema content is pinned by one ACP scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header Agent Note](../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)). -- **Web browser snapshot** (inside `pnpm run test:web`, gate-exempt like the rest of that lane): the web GUI's keyless assembled-transcript tier — a real chromium over the real in-process web assembly (`llm: false` + `dsh-llm-replay`), scenario fixtures and normalized conversation aria goldens under `apps/web/tests/snapshots/`. Its record/refresh commands diverge from `test:snapshot` (`DSH_SNAPSHOT=record pnpm run test:web` re-records against the live model; `DSH_SNAPSHOT=refresh` rewrites goldens keylessly); the [web e2e lane Agent Note](../.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md) owns the design, and CI browser provisioning is deferred there. +- **Web browser snapshot** (gate-exempt `pnpm run test:web`): real chromium over the in-process web assembly replays recorded fixtures against conversation aria goldens (`apps/web/tests/snapshots/`); `DSH_SNAPSHOT=record`/`refresh` semantics and the deferred CI browser decision live in the [web e2e lane Agent Note](../.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md). ## The with-key policy: inference is cheap here @@ -42,4 +42,4 @@ An e2e assertion re-runs the command or re-reads the file externally; a keyword ## When a snapshot test is required -Every non-trivial model- or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP surfaces use `examples//tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the `stream-json` snapshot and replay fixtures. Completed interactive-terminal journeys use JSONL-driven scenarios under `examples/tui-agent/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. Browser-rendered web GUI journeys use `apps/web/tests/snapshots/` through the web e2e lane's harness. New capability seams, lifecycle shapes, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation. +Every non-trivial model- or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP surfaces use `examples//tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the `stream-json` snapshot and replay fixtures. Completed interactive-terminal journeys use JSONL-driven scenarios under `examples/tui-agent/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. Browser-rendered web GUI journeys use `apps/web/tests/snapshots/`. New capability seams, lifecycle shapes, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation. diff --git a/packages/support/llm-replay/tests/llm-replay.spec.ts b/packages/support/llm-replay/tests/llm-replay.spec.ts index b6b03aacbf..d42626bedd 100644 --- a/packages/support/llm-replay/tests/llm-replay.spec.ts +++ b/packages/support/llm-replay/tests/llm-replay.spec.ts @@ -476,6 +476,31 @@ describe('installLlmReplay (through the real LlmService)', () => { expect(() => { handle.assertConsumed() }).not.toThrow() }) + it('paces a throw-entry prefix too (the recorded partial streams at the same cadence)', async () => { + writeFileSync(file, sessionJsonl([]), 'utf8') + const overrideFile = join(dir, 'replay.override.json') + const partial: StreamChunk[] = [{ type: 'block-start', index: 0, blockType: 'text' }] + writeFileSync(overrideFile, JSON.stringify([ + { kind: 'throw', chunks: partial, message: 'boom', code: 'STREAM_CLOSED' }, + ]), 'utf8') + const ctx = new Context() + await ctx.plugin(LlmService) + installLlmReplay(ctx, { file, overrideFile, paceMs: 10 }) + const started = performance.now() + await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).rejects.toThrow('boom') + expect(performance.now() - started).toBeGreaterThanOrEqual(5) + }) + + it('assertConsumed names an underrunning identified session by its id', async () => { + writeLog(TEXT_CHUNKS, TEXT_CHUNKS) + const ctx = new Context() + await ctx.plugin(LlmService) + const handle = installLlmReplay(ctx, { file }) + const sessionId = 'live-underrun' as NonNullable + await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], sessionId })) + expect(() => { handle.assertConsumed() }).toThrow(/session live-underrun consumed 1\/2/) + }) + it('assertConsumed reports recorded scripts no live session ever bound', async () => { writeLog(TEXT_CHUNKS) const childFile = join(dir, 'session.1.jsonl') @@ -690,7 +715,7 @@ describe('apply (the plugin entry)', () => { writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8') const ctx = new Context() await ctx.plugin(LlmService) - apply(ctx, { file, providers: [{ id: 'm', models: [{ id: 'm' }] }] }) + apply(ctx, { file, providers: [{ id: 'm', models: [{ id: 'm' }] }], paceMs: 1 }) expect(ctx.llm.listProviders()).toEqual([{ id: 'm', name: 'm' }]) expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS) }) diff --git a/scripts/doc-budgets.manifest.json b/scripts/doc-budgets.manifest.json index 7e4d154174..fe61bfa69d 100644 --- a/scripts/doc-budgets.manifest.json +++ b/scripts/doc-budgets.manifest.json @@ -4,7 +4,7 @@ "docs/architecture.md": 1800, "docs/cordis-primer.md": 600, "docs/defensive-patterns.md": 550, - "docs/testing.md": 1020, + "docs/testing.md": 1060, "examples/AGENTS.md": 310, "packages/AGENTS.md": 660, "packages/README.md": 760 From 8f97f95d7bb03889bee91d14ad5b03a1fca7c6f9 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 24 Jul 2026 21:06:54 +0800 Subject: [PATCH 06/11] docs(i18n): bilingual pair for the web e2e lane Agent Note Chinese counterpart translated per the terminology table and the 2026-07-18 TUI note's register; switcher lines added on both sides; pair recorded. doc-sync 24/24. --- ...6-07-24-web-gui-browser-e2e-lane.i18n.yaml | 6 ++ .../2026-07-24-web-gui-browser-e2e-lane.md | 2 + .../2026-07-24-web-gui-browser-e2e-lane.zh.md | 90 +++++++++++++++++++ 3 files changed, 98 insertions(+) create mode 100644 .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml create mode 100644 .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml new file mode 100644 index 0000000000..1f55dfce3e --- /dev/null +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write +2026-07-24-web-gui-browser-e2e-lane.md: 3cabceb9667d3d1c153518d58b8d4c02b0578d20 +2026-07-24-web-gui-browser-e2e-lane.zh.md: 132caa453662f48619aa542c68b59f59b64acd0f diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md index fddd1e9a0c..3cabceb966 100644 --- a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md @@ -2,6 +2,8 @@ Status: implemented +English | [中文](2026-07-24-web-gui-browser-e2e-lane.zh.md) + ## Problem The web GUI ships as a real assembled chain — chromium page → client plugin bundles → HTTP unary RPC + two SSE streams → `toFetchHandler`/apiproxy → `bootHost`'s agent loop, tools, and JSONL persistence — and no test exercised that chain keylessly and deterministically. The [GUI testing system](../process/2026-07-20-gui-testing-system.md) covers tier 1 (wire isomorphism in node), tier 2 (object-layer state machines), and tier-3 smokes, but the keyless smoke drives `FixtureApiClient` — no host, no wire, no agent loop — while the full-chain smoke needs `DEEPSEEK_API_KEY` and a live model, so it is nondeterministic and self-skips in keyless CI. The snapshot philosophy of [docs/testing.md](../../../../docs/testing.md) — record once with a key, replay forever keyless, refresh on format churn — already covers the ACP, headless `stream-json`, and TUI transcript surfaces; the web surface was the one assembled product shape without it. The gap is exactly where the two confirmed GUI P0s hid: the wire carriage chain the fixture client short-circuits. diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md new file mode 100644 index 0000000000..132caa4536 --- /dev/null +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md @@ -0,0 +1,90 @@ +# Agent Note: Web GUI 的无密钥浏览器 e2e 车道 + +Status: implemented + +[English](2026-07-24-web-gui-browser-e2e-lane.md) | 中文 + +## 问题 + +Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bundle → HTTP 单次 RPC + 两条 SSE（Server-Sent Events）流 → `toFetchHandler`/apiproxy → `bootHost` 的 agent loop（智能体循环）、工具与 JSONL 持久化——却没有任何测试无密钥且确定性地检验这条链。[GUI 测试体系](../process/2026-07-20-gui-testing-system.md)覆盖第 1 层（Node 中的协议同构）、第 2 层（对象层状态机）与第 3 层冒烟测试，但无密钥冒烟驱动的是 `FixtureApiClient`——没有 host、没有 wire、没有 agent loop——而全链路冒烟需要 `DEEPSEEK_API_KEY` 和真实模型，因此不确定、在无密钥 CI 中自行跳过。[docs/testing.md](../../../../docs/testing.md) 的快照哲学——带密钥录制一次、永久无密钥回放、格式变动时刷新——已覆盖 ACP（Agent Client Protocol）、headless `stream-json` 与 TUI 三个文本记录（transcript）表面；web 表面是唯一没有这层保障的组装形态。而缺口恰恰是两起已实证 GUI P0 藏身之处：fixture（测试前置数据）客户端短路掉的 wire 承载链。 + +## 决策 + +`pnpm run test:web` 携带 `apps/web/tests/` 下的无密钥、确定性浏览器 e2e 车道：录制的会话日志 fixture 经 `@deepseek-ai/dsh-llm-replay` 对真实进程内 web 组装回放，断言规范化后的会话区 aria 预期输出加进程内世界状态。不新增包（package）；产品侧增量只有 `BootHostOptions.llm` seam 和 `dsh-llm-replay` 的两处增量接口。 + +### Harness：`apps/web/tests/harness.ts` + +一个普通的共享 fixture 模块（[测试政策认可的形态](../../../../docs/testing.md)），不是包：值得门禁把守的逻辑——回放推导、会话解析、日志脱敏、持久化——都在已受门禁的包 `dsh-llm-replay`、`dsh-acp-snapshot`、`dsh-session-persistence-jsonl` 中；剩下的只是启动接线和浏览器胶水，而驱动 chromium 的源码在无浏览器的覆盖率 runner 上无法诚实保持逐文件 100% 覆盖率。 + +`launchWebHarness()` 用导出的生产函数在进程内启动真实 web 组装——`startHost({ boot: { …, llm: false } })`、`installLlmReplay(host.ctx, { file, providers, paceMs })`、`mountWebPlugins(host.ctx, roster, anchor)`、`createHostWebPluginRegistry`、`startWebServer({ port: 0, … })`。这是 TUI 套件进程内挂载生产 bundle 的 web 对应物（[TUI 快照](2026-07-18-tui-terminal-state-snapshots.md)）：真实入口边界（`dsh web` bin 的参数解析、dist 解析）仍由 `smoke-real.e2e.ts` 中的无密钥 CLI 冒烟把守，且 web 表面没有可绕过的 `cordis.yml`——按[GUI 分层决策](../architecture/2026-07-19-gui-layering-and-rpc-protocol.md)，组装写在应用里；本车道的设计评审明确重申了这一裁定（Loader 化 `dsh web` 被否决；那需要自己的提案）。与 `dsh web` shell 的两处刻意组装差异已注明在 harness 头部：`workspaceContext: false`（录制的 fixture 不得嵌入本仓库的 AGENTS.md），以及 `sessionTitleLlm` 保持 bootHost 的禁用默认值（其发后不管的标题调用会与循环自身的调用不确定地共享会话的回放游标）。 + +`llm: false` seam 是无密钥启动问题经评审后的定论：`BootHostOptions` 上的 `'deepseek' | false`，与 `workspaceContext: Config | false` 形态一致，且 `RunningHost.ctx` 的 JSDoc 把「填充刻意开放的能力 seam」列为其第三种认可用法。回放必须以提供方目录（providers-catalog）模式运行并发布 `contextWindow`（TUI 的 `PROVIDERS` 形态），绝不用 catch-all：没有注册适配器时，catch-all 会让 `resolveModelContext` 无路由可走，`compact-basic` 的步后压力检查将步步告警，而不是被可证明地闲置。 + +`seedSession()` 通过真实持久化 API 播种冷会话——一次性 `Context` 挂载 `SessionStore` + `SessionPersistenceJsonl` 指向 host 的根目录，`create()` + `append()`，一次 `utimes` 回拨保证侧栏顺序确定（`semantic-checkpoint.snapshot.ts` 先例）——绝不裸写文件，因此播种器对桶哈希、文件名编码、压缩一无所知，host 的 zstd 默认值也无需任何启动开关。种子在播种时即校验（可解析、以 `turn/end` 结尾——未闭合的最终轮次会被恢复（resume）的崩溃修复改写）。 + +### 确定性规则 + +提示一轮对话的屏障栈，按序：（1）host 侧 `await agent.whenIdle()` 加超时，以进程内 `turn/end` 为锚——空闲翻转发生在持久化落盘之后，一次等待同时覆盖轮次完成与持久性；（2）浏览器安定轮询（流式输出节点已卸载、最终文本可见）；（3）任何日志采收都在 `host.dispose()` 之后。单独监听进程内 `turn/end` 是错误屏障（它先于 SSE 帧到达浏览器、先于 fsync 触发）；文件轮询被禁止（NFS 上慢，且被 `whenIdle` 取代）；`networkidle` 被彻底禁止（SSE 流保持打开时它永不解析）。 + +不做单次瞬态 DOM 断言：从回放产出到 React 提交的每一跳都可能合并分片，采样 `[data-streaming]` 天然就是竞态。流式输出的增量性由持久化的 `assistant/chunk` 事件断言（模型可见 ⟺ 已记录，使日志成为权威证据）。`dsh-llm-replay` 的可选 `paceMs`（默认缺省 = 突发）只是让浏览器观察到真正增量 SSE 的真实感旋钮；正确性绝不依赖它，且节奏等待期间中止会即时取消。 + +每个场景都会因任何 pageerror 或客户端的连接丢失/间隙修复控制台警告而失败：否则重连机制加历史重同步会把一条死掉的 SSE 通路自愈掉，套件反而认证了坏 wire。Harness 的 `close()` 调用 `ReplayHandle.assertConsumed()` 收尾检查（每个已录脚本都被绑定、每个游标都耗尽），把静默的少放与错绑变成清晰诊断。车道不设 vitest 重试；每文件一个 chromium、每场景一个新 context、每场景一个 host；视口固定；选择器只锚定 role、`data-*` 属性和可见文本。 + +### 预期输出 + +每场景一份提交的预期输出：会话区规范化 `ariaSnapshot()`（`ui.expected.md`）——uuid/cwd/工作区目录名/时长归一为稳定 token，在安定里程碑处轮询至两次相等再采集——外加几条 role/文本锚断言，让保语义的组件重写在预期输出可评审地变动时仍保持绿色锚点。aria 树是 client 规则「断言用户所见，绝不断言类名」的机械化。世界状态断言内联在 `host.ctx` 会话事件上（哪些工具运行了、`turn/end` 完成）而不是第二份提交的日志预期输出：持久化日志表面已由 ACP/headless/TUI 套件经同一循环和持久化钉住，在此重复钉住会违背分层纪律、翻倍刷新成本。`refresh` 是预期输出的唯一写入者——回放模式下预期输出缺失会连同修复命令一起报错，而不是静默自举。 + +类型检查平面切分是结构性的：`apps/web/tests/{harness,support,replay-round-trip.e2e,seeded-history.e2e}.ts` 是 host 平面程序（它们启动 host 主干），因此被排除出注册在 client 侧的 `apps/web` 工程，逐文件纳入 `tsconfig.host.json`——一个程序不能同时持有 cordis `Context` 合并的两侧。 + +### 模式与 fixture + +`DSH_SNAPSHOT` 以内联 spec 分支选择 replay（默认，无密钥）、record（带密钥）或 refresh（无密钥）——TUI 的形态，不是套件工厂：两个场景撑不起 acp-snapshot 工厂机制，且真正共享的部分已被导出（`scrubRequestHeaders`、`parseSessionLog`、`installLlmReplay`）。每个 spec 切分为驱动步骤（输入、发送、`whenTurnSettled`——所有模式都执行，绝不等待模型内容选择器，因此 record 不会因真实模型答法不同而挂起）与断言步骤（仅 replay/refresh）。Record = 经真实输入框实时驱动 + 采收内存中的 `session.header`/`session.events`（TUI 的 `rawSessionLog` 形态——无需文件解压）+ `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}` token 化；随后一次无密钥 refresh 重新生成 `ui.expected.md`。两个场景的 fixture 都经此流程对本组装录制。一条漂移防线把每个 spec 的驱动提示词与 fixture 录制的 `user/message` 绑定。fixture 清单防线保持每个场景目录封闭（精确文件集合，每个 JSONL 都是脱敏不动点）。Web fixture 全部脱敏请求头且不钉任何头类别，沿用 TUI 先例而非[钉住请求头](2026-07-06-pin-request-header-content-in-one-scenario.md)的严格读法——见「暂缓」。 + +### 场景 + +1. **`replay-round-trip`**——新会话，经真实输入框发送提示词，回放流式输出推理（reasoning）+ 一次在临时工作区真实执行的 `bash` 工具调用 + 最终文本（15ms 节奏）。断言安定后的 markdown、aria 预期输出与内联世界状态（bash `tool/call`、完成的 `turn/end`、>10 个分片事件）。 +2. **`seeded-history`**——冷播种一份已录会话；侧栏列出它（分组行 → 会话行，默认折叠），打开后纯凭日志经 `session.history` 内的隐式冷恢复挂载渲染工具卡片与文本——replay 下零模型调用，因此没有任何绑定约束；record 模式实时驱动同一轮（真实 `read` 工具读取播种的工作区文件）来产出种子。 + +### CI 立场 + +车道随 `pnpm run test:web` 交付、豁免门禁，与该配置头部注释所记一致。往 CI 加 chromium 会推翻 [GUI 测试笔记](../process/2026-07-20-gui-testing-system.md)中「CI 无浏览器基础设施」的前提，因此需要自己的 Agent Note 并从那里交叉链接，分阶段推进：先作为非必需任务，再以量化标准晋升（连续绿色运行次数、耗时、零重试的抖动预算、runner 浏览器缓存策略）。`TODO(ci-browser)` 标记该接缝。场景目前面向 POSIX（车道不在 Windows 矩阵中）。 + +## 业界先例 + +调研了 AI 聊天/agent web UI 与 mock 层（LibreChat、vercel/ai-chatbot + AI SDK、lobe-chat、open-webui、OpenHands、Chainlit、continue、cline、langfuse、gradio/streamlit；Playwright HAR/route、MSW、Polly/nock、WireMock、aimock）。自有后端的应用的主流成熟架构是：真实后端 seam 后放一个进程内伪造/回放模型，下游全部真实（LibreChat 的 `LIBRECHAT_TEST_RUN_HOOK` 伪模型；ai-chatbot 的 `MockLanguageModelV3` + `simulateReadableStream`；continue 的脚本化 mock 提供方类）——这正是 `dsh-llm-replay` 已然所是。浏览器层 SSE 拦截无法检验增量渲染（`route.fulfill` 一次性交付整个响应体；playwright#33564），且服务端 SSE 栈完全失测，因此各项目只把它用于边缘用例。分片节奏作为 fixture 参数反复出现（LibreChat 默认 10ms 附慢速档；ai-chatbot 500ms）；CI 里的真实模型会腐烂（open-webui 的套件长出 120 秒超时，先被禁用后被删除）；会话在持久化层以受控时间戳播种（LibreChat 直插回拨时间的 Mongo 文档；langfuse 播种其数据库）。没有任何被调研项目为 UI 测试把录制的 agent 事件日志经真实后端回放——最接近的是提供方层录制 fixture（aimock）与前端层 socket 历史发射（OpenHands MSW）——因此会话日志即 fixture 的设计沿着本仓库「模型可见 ⟺ 已记录」不变式所指的方向比业界先例多走了一步。 + +## 曾考虑的替代方案 + +**浏览器网络层 SSE 拦截（`page.route`）。** 已否决：`route.fulfill` 无法流式输出，增量 token 渲染无从检验，且服务端 SSE/背压/关闭路径——两起已实证 P0 的藏身处——完全失测。 + +**`DEEPSEEK_BASE_URL` 处的 mock HTTP 提供方。** 作为本车道机制已否决（仅保留给既有的工作区探针冒烟）：fixture 会变成手写的 OpenAI SSE 字节脚本，一种与仓库其余部分录制回放的会话日志格式渐行渐远的第二 fixture 格式；适配器的真实 HTTP 路径归带密钥 e2e 管。 + +**扩展 `?fixture` 客户端。** 已否决：分层纪律——`FixtureApiClient` 的存在意义就是脱离服务器测试客户端 shell；client API seam 以下按构造即失测。 + +**用占位 `DEEPSEEK_API_KEY` + 回放拦截替代 `llm: false` seam。** 尽管零产品改动且树内有两处先例仍被否决：它用谎言满足 `llm-deepseek` 的快速失败密钥检查，还留下一个挂载却被拦截的死适配器；seam 方案与既有选项形态一致，并在最早可解析点快速失败。 + +**`packages/support/web-snapshot` 包 + `defineWebSnapshotSuite` 工厂。** 已否决：驱动 chromium 的源码在无浏览器的覆盖率 runner 上无法诚实保持逐文件 100%，且两个场景就上工厂是从单一消费方过度泛化，真正共享的逻辑已从受门禁的包中导出。重启条件：出现第二个 web 形态消费方，或 ≥6 个场景的内联分支被证实各自漂移；届时包边界将画在无浏览器一侧。 + +**第二份提交的规范化会话日志预期输出。** 已否决：日志表面已由 ACP/headless/TUI 套件经同一循环与持久化钉住；在此只会翻倍刷新成本并重复测试下层。内联在 `host.ctx` 事件上的世界状态断言保住了验证世界的义务。 + +**以 `DSH_SNAPSHOT` 回放分支拉起 `dsh web` bin。** 已否决：它需要在产品 bin 里加测试模式分支和环境变量管道，而进程内路线用的是零产品改动的导出生产函数；bin 的薄胶水已由无密钥 CLI 冒烟覆盖。只有 web host 某天 Loader 化它才免费——评审中已否决，并重申了应用内组装的裁定。 + +**为可测试性改 wire 协议。** 已否决：契约已有第一等的无密钥同构 seam（`InProcessApiClient(toFetchHandler(api))`），逐事件不合批的 SSE 恰是回放在浏览器中可观测的原因，测试一条不再交付的 wire 会颠倒该层的存在意义。 + +**以真实模型浏览器测试充当无密钥车道。** 已否决：按构造即不确定；被调研的前车之鉴（open-webui）长出无界超时后被删除。带密钥的 W5 冒烟仍是真实模型侧的补充。 + +**客户端 `data-dsh-busy` 安定信号。** 暂缓：两个场景下多条件安定轮询已经够用，host 侧 `whenIdle` 屏障承担了重活。重启条件：第一次安定轮询抖动，或某场景需要等待 DOM 不暴露的状态。 + +## Testing + +车道自身：`pnpm run test:web` 与既有冒烟对一起无密钥运行两个场景；`DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/` 对真实模型重录某场景的 fixture；`DSH_SNAPSHOT=refresh` 无密钥重写两份 aria 预期输出。`llm: false` seam 由 `packages/host/runtime/tests/host-runtime.spec.ts` 钉住（无密钥启动、首次流式调用 NO_ADAPTER、嵌入方经 ctx 填充）；`paceMs` 校验、节奏下限、节奏中中止、`assertConsumed` 的两种失败形态钉在 `packages/support/llm-replay/tests/llm-replay.spec.ts`。 + +## 暂缓 + +- **Web 头类别钉住**：web fixture 处处 token 化 `{{system}}`/`{{tools}}`，没有场景钉住 bootHost 组装的提示词/工具 schema（`TODO(web-header-pin)`——harness 的 `recordFixture` JSDoc 有标记）。沿用 TUI 处处脱敏先例；当 web 组装的请求头与其镜像的 repl 组合进一步分叉时重审。 +- **CI 浏览器供给**：推翻 CI 无浏览器裁定，分阶段标准见上（`TODO(ci-browser)`）。 +- **恢复后追问场景**：真实 wire 上的历史/实时缝合路径；当该代码变更或回归时作为独立场景补充。 + +## 后果 + +Web 表面获得了录制一次/永久回放的层级：真实 chromium → SSE → apiproxy → 循环 → 工具 → 持久化的链路以约 10-30 秒无密钥运行，重复运行结果确定，fixture 由车道自身持有并可重录。接受的成本：每次有意的会话 UI 变更都以一次无密钥 `DSH_SNAPSHOT=refresh` 收尾（预期输出变动是受评审的 diff，锚断言保住语义绿色）；aria 格式归 Playwright 所有——仓库唯一不受自己控制的提交快照格式——因此 playwright 版本升级必须是刻意的升级加刷新提交（依赖在 `apps/web/package.json` 中浮动为 `^1.49.0`；若变动伤人则改为精确锁定）；回放的首次调用顺序绑定把每个场景限制为至多一个发起提示的会话，消费断言是绊线；`compact-basic` 与会话共享回放游标，仅在发布的 128k 目录窗口下保持闲置；在 CI 反转被单独决策之前，车道只在其运行之处（本地，`test:web`）把守回归。 From 6d3c25f494a3e9bd48ae02a7e5200773e4ec5261 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 24 Jul 2026 22:18:19 +0800 Subject: [PATCH 07/11] refactor(web-e2e): rename harness -> scaffold; add interaction coverage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The shared test module was named harness.ts inside a repo whose product IS a harness — hopelessly ambiguous. Renamed to scaffold.ts with launchWebScaffold/WebScaffold; tsconfig plane-split entries, the seam JSDoc/README mentions, and both Agent Note languages updated. Both scenarios gain a Playwright interaction step over the settled transcript (after the golden capture, so committed aria surfaces stay untouched): replay-round-trip clicks the reasoning fold open/closed over wire-delivered state; seeded-history expands a read tool row rebuilt from the cold log and asserts the recorded result text appears (read rows are expand-in-place — rowExpands routes the click to the inline fold, not the details column). test:web 30 passed | 1 skipped. --- .../2026-07-24-web-gui-browser-e2e-lane.md | 10 ++--- .../2026-07-24-web-gui-browser-e2e-lane.zh.md | 10 ++--- apps/web/tests/replay-round-trip.e2e.ts | 36 ++++++++++----- apps/web/tests/{harness.ts => scaffold.ts} | 40 ++++++++--------- apps/web/tests/seeded-history.e2e.ts | 45 +++++++++++++------ apps/web/tsconfig.json | 4 +- packages/host/runtime/README.md | 2 +- packages/host/runtime/src/boot.ts | 2 +- tsconfig.host.json | 2 +- 9 files changed, 91 insertions(+), 60 deletions(-) rename apps/web/tests/{harness.ts => scaffold.ts} (93%) diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md index 3cabceb966..9bab4d1eb9 100644 --- a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md @@ -12,11 +12,11 @@ The web GUI ships as a real assembled chain — chromium page → client plugin `pnpm run test:web` carries a keyless, deterministic browser e2e lane under `apps/web/tests/`: recorded session-log fixtures replayed through `@deepseek-ai/dsh-llm-replay` against the real in-process web assembly, asserting a normalized conversation aria golden plus in-process world state. No new package; the product deltas are the `BootHostOptions.llm` seam and two additive `dsh-llm-replay` surfaces. -### Harness: `apps/web/tests/harness.ts` +### Scaffold: `apps/web/tests/scaffold.ts` A plain shared-fixture module (the [testing-policy sanctioned shape](../../../../docs/testing.md)), not a package: the gate-worthy logic — replay derivation, session parsing, log scrubbing, persistence — lives in the gated packages `dsh-llm-replay`, `dsh-acp-snapshot`, and `dsh-session-persistence-jsonl`; what remains is boot wiring and browser glue, and chromium-driving code cannot hold per-file 100% coverage on the browserless coverage runners. -`launchWebHarness()` boots the real web assembly in-process from the exported production functions — `startHost({ boot: { …, llm: false } })`, `installLlmReplay(host.ctx, { file, providers, paceMs })`, `mountWebPlugins(host.ctx, roster, anchor)`, `createHostWebPluginRegistry`, `startWebServer({ port: 0, … })`. This is the web analog of the TUI suite mounting the production bundle in-process ([TUI snapshots](2026-07-18-tui-terminal-state-snapshots.md)): the real entry boundary (`dsh web` bin arg-parsing, dist resolution) stays held by the keyless CLI smokes in `smoke-real.e2e.ts`, and the web surface has no `cordis.yml` to bypass — assembly is written in the app per the [GUI layering decision](../architecture/2026-07-19-gui-layering-and-rpc-protocol.md), a ruling this lane's design review explicitly reaffirmed (Loader-izing `dsh web` was declined; it would be its own proposal). Two deliberate assembly divergences from the `dsh web` shell, noted in the harness header: `workspaceContext: false` (recorded fixtures must not embed this repo's AGENTS.md) and `sessionTitleLlm` left at bootHost's disabled default (its fire-and-forget title call would share the session's replay cursor nondeterministically). +`launchWebScaffold()` boots the real web assembly in-process from the exported production functions — `startHost({ boot: { …, llm: false } })`, `installLlmReplay(host.ctx, { file, providers, paceMs })`, `mountWebPlugins(host.ctx, roster, anchor)`, `createHostWebPluginRegistry`, `startWebServer({ port: 0, … })`. This is the web analog of the TUI suite mounting the production bundle in-process ([TUI snapshots](2026-07-18-tui-terminal-state-snapshots.md)): the real entry boundary (`dsh web` bin arg-parsing, dist resolution) stays held by the keyless CLI smokes in `smoke-real.e2e.ts`, and the web surface has no `cordis.yml` to bypass — assembly is written in the app per the [GUI layering decision](../architecture/2026-07-19-gui-layering-and-rpc-protocol.md), a ruling this lane's design review explicitly reaffirmed (Loader-izing `dsh web` was declined; it would be its own proposal). Two deliberate assembly divergences from the `dsh web` shell, noted in the scaffold header: `workspaceContext: false` (recorded fixtures must not embed this repo's AGENTS.md) and `sessionTitleLlm` left at bootHost's disabled default (its fire-and-forget title call would share the session's replay cursor nondeterministically). The `llm: false` seam is the reviewed resolution of the keyless-boot question: `'deepseek' | false` on `BootHostOptions`, matching the `workspaceContext: Config | false` shape, with `RunningHost.ctx` JSDoc naming "filling a deliberately-open capability seam" as its third sanctioned use. Replay runs in providers-catalog mode with a published `contextWindow` (the TUI `PROVIDERS` shape), never catch-all: with no adapter registered, catch-all would leave `resolveModelContext` unroutable and `compact-basic`'s post-step pressure check would warn every step instead of being provably inert. @@ -28,13 +28,13 @@ The barrier stack for a prompted turn, in order: (1) host-side `await agent.when No single-shot transient-DOM assertions: every hop from replay yield to React commit can coalesce chunks, so sampling `[data-streaming]` is a race by construction. Streaming incrementality is asserted from the persisted `assistant/chunk` events (model-visible ⟺ logged makes the log the authoritative proof). `dsh-llm-replay`'s opt-in `paceMs` (default absent = burst) is a realism knob so the browser observes genuinely incremental SSE; correctness never leans on it, and abort during a pace wait cancels promptly. -Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Harness `close()` calls the `ReplayHandle.assertConsumed()` teardown check (every recorded script bound, every cursor drained), converting silent underruns and shifted bindings into crisp diagnostics. No vitest retry on the lane; one chromium per file, fresh context per scenario, one host per scenario; viewport pinned; selectors anchor on roles, `data-*` attributes, and visible text. +Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Scaffold `close()` calls the `ReplayHandle.assertConsumed()` teardown check (every recorded script bound, every cursor drained), converting silent underruns and shifted bindings into crisp diagnostics. No vitest retry on the lane; one chromium per file, fresh context per scenario, one host per scenario; viewport pinned; selectors anchor on roles, `data-*` attributes, and visible text. ### Expected outputs One committed golden per scenario: a normalized `ariaSnapshot()` of the conversation region (`ui.expected.md`) — uuid/cwd/workspace-basename/duration tokens normalized, captured poll-until-equal at the settled milestone — plus a few role/text anchor assertions that stay green under a semantics-preserving component rewrite while the golden churns reviewably. The aria tree is the mechanization of the client rule "assert what the user would see, never class names". World-state assertions ride `host.ctx` session events inline (which tools ran, `turn/end` completed) instead of a second committed log golden: the persisted-log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence, and re-pinning it here would double refresh cost against the tier discipline. `refresh` is the sole golden writer — a missing golden in replay mode fails with the healing command rather than self-bootstrapping. -The typecheck plane split is structural: `apps/web/tests/{harness,support,replay-round-trip.e2e,seeded-history.e2e}.ts` are host-plane programs (they boot the host spine), so they are excluded from the client-registered `apps/web` project and included file-by-file in `tsconfig.host.json` — one program cannot hold both sides of the cordis `Context` merges. +The typecheck plane split is structural: `apps/web/tests/{scaffold,support,replay-round-trip.e2e,seeded-history.e2e}.ts` are host-plane programs (they boot the host spine), so they are excluded from the client-registered `apps/web` project and included file-by-file in `tsconfig.host.json` — one program cannot hold both sides of the cordis `Context` merges. ### Modes and fixtures @@ -81,7 +81,7 @@ The lane itself: `pnpm run test:web` runs both scenarios keylessly alongside the ## Deferred -- **Web header-class pin**: web fixtures tokenize `{{system}}`/`{{tools}}` everywhere and no scenario pins bootHost's composed prompt/tool schemas (`TODO(web-header-pin)` — the harness `recordFixture` JSDoc marks it). Following the TUI scrub-everywhere precedent; revisit when the web assembly's header diverges from the repl composition it mirrors. +- **Web header-class pin**: web fixtures tokenize `{{system}}`/`{{tools}}` everywhere and no scenario pins bootHost's composed prompt/tool schemas (`TODO(web-header-pin)` — the scaffold `recordFixture` JSDoc marks it). Following the TUI scrub-everywhere precedent; revisit when the web assembly's header diverges from the repl composition it mirrors. - **CI browser provisioning**: reversal of the no-browser-in-CI ruling, staged criteria above (`TODO(ci-browser)`). - **Follow-up-prompt-after-resume scenario**: the history/live stitch path over the real wire; add as its own scenario when that code changes or regresses. diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md index 132caa4536..c61aee7ae5 100644 --- a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md @@ -12,11 +12,11 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu `pnpm run test:web` 携带 `apps/web/tests/` 下的无密钥、确定性浏览器 e2e 车道：录制的会话日志 fixture 经 `@deepseek-ai/dsh-llm-replay` 对真实进程内 web 组装回放，断言规范化后的会话区 aria 预期输出加进程内世界状态。不新增包（package）；产品侧增量只有 `BootHostOptions.llm` seam 和 `dsh-llm-replay` 的两处增量接口。 -### Harness：`apps/web/tests/harness.ts` +### Scaffold：`apps/web/tests/scaffold.ts` 一个普通的共享 fixture 模块（[测试政策认可的形态](../../../../docs/testing.md)），不是包：值得门禁把守的逻辑——回放推导、会话解析、日志脱敏、持久化——都在已受门禁的包 `dsh-llm-replay`、`dsh-acp-snapshot`、`dsh-session-persistence-jsonl` 中；剩下的只是启动接线和浏览器胶水，而驱动 chromium 的源码在无浏览器的覆盖率 runner 上无法诚实保持逐文件 100% 覆盖率。 -`launchWebHarness()` 用导出的生产函数在进程内启动真实 web 组装——`startHost({ boot: { …, llm: false } })`、`installLlmReplay(host.ctx, { file, providers, paceMs })`、`mountWebPlugins(host.ctx, roster, anchor)`、`createHostWebPluginRegistry`、`startWebServer({ port: 0, … })`。这是 TUI 套件进程内挂载生产 bundle 的 web 对应物（[TUI 快照](2026-07-18-tui-terminal-state-snapshots.md)）：真实入口边界（`dsh web` bin 的参数解析、dist 解析）仍由 `smoke-real.e2e.ts` 中的无密钥 CLI 冒烟把守，且 web 表面没有可绕过的 `cordis.yml`——按[GUI 分层决策](../architecture/2026-07-19-gui-layering-and-rpc-protocol.md)，组装写在应用里；本车道的设计评审明确重申了这一裁定（Loader 化 `dsh web` 被否决；那需要自己的提案）。与 `dsh web` shell 的两处刻意组装差异已注明在 harness 头部：`workspaceContext: false`（录制的 fixture 不得嵌入本仓库的 AGENTS.md），以及 `sessionTitleLlm` 保持 bootHost 的禁用默认值（其发后不管的标题调用会与循环自身的调用不确定地共享会话的回放游标）。 +`launchWebScaffold()` 用导出的生产函数在进程内启动真实 web 组装——`startHost({ boot: { …, llm: false } })`、`installLlmReplay(host.ctx, { file, providers, paceMs })`、`mountWebPlugins(host.ctx, roster, anchor)`、`createHostWebPluginRegistry`、`startWebServer({ port: 0, … })`。这是 TUI 套件进程内挂载生产 bundle 的 web 对应物（[TUI 快照](2026-07-18-tui-terminal-state-snapshots.md)）：真实入口边界（`dsh web` bin 的参数解析、dist 解析）仍由 `smoke-real.e2e.ts` 中的无密钥 CLI 冒烟把守，且 web 表面没有可绕过的 `cordis.yml`——按[GUI 分层决策](../architecture/2026-07-19-gui-layering-and-rpc-protocol.md)，组装写在应用里；本车道的设计评审明确重申了这一裁定（Loader 化 `dsh web` 被否决；那需要自己的提案）。与 `dsh web` shell 的两处刻意组装差异已注明在 scaffold 头部：`workspaceContext: false`（录制的 fixture 不得嵌入本仓库的 AGENTS.md），以及 `sessionTitleLlm` 保持 bootHost 的禁用默认值（其发后不管的标题调用会与循环自身的调用不确定地共享会话的回放游标）。 `llm: false` seam 是无密钥启动问题经评审后的定论：`BootHostOptions` 上的 `'deepseek' | false`，与 `workspaceContext: Config | false` 形态一致，且 `RunningHost.ctx` 的 JSDoc 把「填充刻意开放的能力 seam」列为其第三种认可用法。回放必须以提供方目录（providers-catalog）模式运行并发布 `contextWindow`（TUI 的 `PROVIDERS` 形态），绝不用 catch-all：没有注册适配器时，catch-all 会让 `resolveModelContext` 无路由可走，`compact-basic` 的步后压力检查将步步告警，而不是被可证明地闲置。 @@ -28,13 +28,13 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu 不做单次瞬态 DOM 断言：从回放产出到 React 提交的每一跳都可能合并分片，采样 `[data-streaming]` 天然就是竞态。流式输出的增量性由持久化的 `assistant/chunk` 事件断言（模型可见 ⟺ 已记录，使日志成为权威证据）。`dsh-llm-replay` 的可选 `paceMs`（默认缺省 = 突发）只是让浏览器观察到真正增量 SSE 的真实感旋钮；正确性绝不依赖它，且节奏等待期间中止会即时取消。 -每个场景都会因任何 pageerror 或客户端的连接丢失/间隙修复控制台警告而失败：否则重连机制加历史重同步会把一条死掉的 SSE 通路自愈掉，套件反而认证了坏 wire。Harness 的 `close()` 调用 `ReplayHandle.assertConsumed()` 收尾检查（每个已录脚本都被绑定、每个游标都耗尽），把静默的少放与错绑变成清晰诊断。车道不设 vitest 重试；每文件一个 chromium、每场景一个新 context、每场景一个 host；视口固定；选择器只锚定 role、`data-*` 属性和可见文本。 +每个场景都会因任何 pageerror 或客户端的连接丢失/间隙修复控制台警告而失败：否则重连机制加历史重同步会把一条死掉的 SSE 通路自愈掉，套件反而认证了坏 wire。Scaffold 的 `close()` 调用 `ReplayHandle.assertConsumed()` 收尾检查（每个已录脚本都被绑定、每个游标都耗尽），把静默的少放与错绑变成清晰诊断。车道不设 vitest 重试；每文件一个 chromium、每场景一个新 context、每场景一个 host；视口固定；选择器只锚定 role、`data-*` 属性和可见文本。 ### 预期输出 每场景一份提交的预期输出：会话区规范化 `ariaSnapshot()`（`ui.expected.md`）——uuid/cwd/工作区目录名/时长归一为稳定 token，在安定里程碑处轮询至两次相等再采集——外加几条 role/文本锚断言，让保语义的组件重写在预期输出可评审地变动时仍保持绿色锚点。aria 树是 client 规则「断言用户所见，绝不断言类名」的机械化。世界状态断言内联在 `host.ctx` 会话事件上（哪些工具运行了、`turn/end` 完成）而不是第二份提交的日志预期输出：持久化日志表面已由 ACP/headless/TUI 套件经同一循环和持久化钉住，在此重复钉住会违背分层纪律、翻倍刷新成本。`refresh` 是预期输出的唯一写入者——回放模式下预期输出缺失会连同修复命令一起报错，而不是静默自举。 -类型检查平面切分是结构性的：`apps/web/tests/{harness,support,replay-round-trip.e2e,seeded-history.e2e}.ts` 是 host 平面程序（它们启动 host 主干），因此被排除出注册在 client 侧的 `apps/web` 工程，逐文件纳入 `tsconfig.host.json`——一个程序不能同时持有 cordis `Context` 合并的两侧。 +类型检查平面切分是结构性的：`apps/web/tests/{scaffold,support,replay-round-trip.e2e,seeded-history.e2e}.ts` 是 host 平面程序（它们启动 host 主干），因此被排除出注册在 client 侧的 `apps/web` 工程，逐文件纳入 `tsconfig.host.json`——一个程序不能同时持有 cordis `Context` 合并的两侧。 ### 模式与 fixture @@ -81,7 +81,7 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu ## 暂缓 -- **Web 头类别钉住**：web fixture 处处 token 化 `{{system}}`/`{{tools}}`，没有场景钉住 bootHost 组装的提示词/工具 schema（`TODO(web-header-pin)`——harness 的 `recordFixture` JSDoc 有标记）。沿用 TUI 处处脱敏先例；当 web 组装的请求头与其镜像的 repl 组合进一步分叉时重审。 +- **Web 头类别钉住**：web fixture 处处 token 化 `{{system}}`/`{{tools}}`，没有场景钉住 bootHost 组装的提示词/工具 schema（`TODO(web-header-pin)`——scaffold 的 `recordFixture` JSDoc 有标记）。沿用 TUI 处处脱敏先例；当 web 组装的请求头与其镜像的 repl 组合进一步分叉时重审。 - **CI 浏览器供给**：推翻 CI 无浏览器裁定，分阶段标准见上（`TODO(ci-browser)`）。 - **恢复后追问场景**：真实 wire 上的历史/实时缝合路径；当该代码变更或回归时作为独立场景补充。 diff --git a/apps/web/tests/replay-round-trip.e2e.ts b/apps/web/tests/replay-round-trip.e2e.ts index faf1a1f6a8..c2476ed3ea 100644 --- a/apps/web/tests/replay-round-trip.e2e.ts +++ b/apps/web/tests/replay-round-trip.e2e.ts @@ -16,8 +16,8 @@ import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest' import type { SessionEvent } from '@deepseek-ai/dsh-session' import { assertFixtureInventory, captureStableAria, compareOrRefreshGolden, fixtureUserPrompts, - launchWebHarness, recordFixture, watchConsole, webSnapshotMode, type WebHarness, -} from './harness.ts' + launchWebScaffold, recordFixture, watchConsole, webSnapshotMode, type WebScaffold, +} from './scaffold.ts' import { saveFailureShot } from './support.ts' const SNAPSHOT_DIR = fileURLToPath(new URL('./snapshots/fresh-round-trip', import.meta.url)) @@ -31,27 +31,27 @@ const MODE = webSnapshotMode() const PROMPT = 'Use the bash tool to run exactly: echo WEB_E2E_OK. Then reply with the single word DONE and stop.' describe('web e2e: fresh round trip through the real assembly', () => { - let harness: WebHarness + let scaffold: WebScaffold let browser: Browser let page: Page let tripwire: ReturnType const sessionEvents: SessionEvent[] = [] beforeAll(async () => { - harness = await launchWebHarness({ + scaffold = await launchWebScaffold({ ...(MODE === 'record' ? {} : { replayFixture: FIXTURE, paceMs: 15 }), }) - harness.host.ctx.on('session/event', (_session, event: SessionEvent) => { sessionEvents.push(event) }) + scaffold.host.ctx.on('session/event', (_session, event: SessionEvent) => { sessionEvents.push(event) }) browser = await chromium.launch() page = await browser.newPage({ viewport: { width: 1680, height: 1000 } }) tripwire = watchConsole(page) - await page.goto(harness.baseUrl, { waitUntil: 'load' }) + await page.goto(scaffold.baseUrl, { waitUntil: 'load' }) await page.waitForSelector('[class*="frame"]', { timeout: 30_000 }) }, 120_000) afterAll(async () => { await browser?.close() - await harness?.close() + await scaffold?.close() }) it('drives the recorded prompt to a settled turn (all modes)', async () => { @@ -63,12 +63,12 @@ describe('web e2e: fresh round trip through the real assembly', () => { const input = page.locator('textarea').first() await input.waitFor({ timeout: 10_000 }) // Arm the host-side settled barrier BEFORE the send click. - const settled = harness.whenTurnSettled() + const settled = scaffold.whenTurnSettled() await input.fill(PROMPT) await input.press('Enter') const sessionId = await settled if (MODE === 'record') { - await recordFixture(harness, sessionId, FIXTURE) + await recordFixture(scaffold, sessionId, FIXTURE) } }, 200_000) @@ -96,14 +96,28 @@ describe('web e2e: fresh round trip through the real assembly', () => { // while the whole-region golden churns. await expect(page.getByRole('textbox').first().isVisible()).resolves.toBe(true) expect(await page.getByText('WEB_E2E_OK', { exact: false }).count()).toBeGreaterThanOrEqual(1) - const snapshot = await captureStableAria(page, '[class*="centerCol"]', harness.workspaceCwd) + const snapshot = await captureStableAria(page, '[class*="centerCol"]', scaffold.workspaceCwd) await compareOrRefreshGolden(UI_EXPECTED, snapshot, MODE) }) + it.skipIf(MODE === 'record')('expands and collapses the reasoning fold from its click target', async () => { + onTestFailed(() => saveFailureShot(page, 'web-e2e-round-trip-think')) + // Interaction over the REAL wire-delivered transcript (the fixture-client + // tier pins the same gesture against FixtureApiClient; this one runs on + // mux-frame-fed state). Runs after the golden capture so the committed + // aria surface stays the untouched settled state. + const think = page.getByRole('button', { name: /^Think/ }).first() + expect(await think.getAttribute('aria-expanded')).toBe('false') + await think.click() + await expect.poll(() => think.getAttribute('aria-expanded'), { timeout: 5_000 }).toBe('true') + await think.click() + await expect.poll(() => think.getAttribute('aria-expanded'), { timeout: 5_000 }).toBe('false') + }) + it.skipIf(MODE === 'record')('stayed clean: no pageerrors, no reconnect self-healing, no server errors', async () => { expect(tripwire.pageErrors).toEqual([]) expect(tripwire.warnings).toEqual([]) - expect(harness.serverErrors).toEqual([]) + expect(scaffold.serverErrors).toEqual([]) await assertFixtureInventory(SNAPSHOT_DIR, ['session.jsonl', 'ui.expected.md']) }) }) diff --git a/apps/web/tests/harness.ts b/apps/web/tests/scaffold.ts similarity index 93% rename from apps/web/tests/harness.ts rename to apps/web/tests/scaffold.ts index b2941c62e0..c7fb3b04fe 100644 --- a/apps/web/tests/harness.ts +++ b/apps/web/tests/scaffold.ts @@ -1,4 +1,4 @@ -// Shared harness for the keyless browser e2e lane (Agent Note: +// Shared scaffold for the keyless browser e2e lane (Agent Note: // .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md). // Boots the REAL web assembly in-process from the exported production // functions — startHost (bootHost spine) + mountWebPlugins + registry + @@ -78,9 +78,9 @@ function loadRootEnv(): void { } } -/** A booted web harness: real assembly, mode-selected model backend, temp world. */ -export interface WebHarness { - /** The active snapshot mode this harness booted under. */ +/** A booted web scaffold: real assembly, mode-selected model backend, temp world. */ +export interface WebScaffold { + /** The active snapshot mode this scaffold booted under. */ mode: WebSnapshotMode /** Browser-facing origin (http://127.0.0.1:). */ baseUrl: string @@ -98,7 +98,7 @@ export interface WebHarness { close(): Promise } -/** Options for {@link launchWebHarness}. */ +/** Options for {@link launchWebScaffold}. */ export interface LaunchOptions { /** * Replay fixture (session.jsonl) served by dsh-llm-replay in replay/refresh @@ -114,9 +114,9 @@ export interface LaunchOptions { /** * Boot the real web assembly under the current snapshot mode. * @param options - replay fixture selection and pacing. - * @returns the running harness. + * @returns the running scaffold. */ -export async function launchWebHarness(options: LaunchOptions = {}): Promise { +export async function launchWebScaffold(options: LaunchOptions = {}): Promise { requireDist() const mode = webSnapshotMode() if (mode === 'record') { @@ -221,7 +221,7 @@ export async function launchWebHarness(options: LaunchOptions = {}): Promise failures.push(e)) await rm(workspaceCwd, { recursive: true, force: true }).catch((e: unknown) => failures.push(e)) await rm(persistenceRoot, { recursive: true, force: true }).catch((e: unknown) => failures.push(e)) - if (failures.length > 0) throw new AggregateError(failures, 'web harness teardown failed') + if (failures.length > 0) throw new AggregateError(failures, 'web scaffold teardown failed') }, } } @@ -246,16 +246,16 @@ function rawSessionLog(session: Session): string { * work), tokenize the run-local session id and cwd ({{sessionId}}/{{cwd}}, * the committed ACP fixture convention — re-records then diff only on real * content), and write the committed fixture. - * @param harness - the record-mode harness. + * @param scaffold - the record-mode scaffold. * @param sessionId - the driven session. * @param fixturePath - the committed session.jsonl / seed.jsonl target. */ -export async function recordFixture(harness: WebHarness, sessionId: SessionId, fixturePath: string): Promise { - const agent = harness.host.ctx.agents.get(sessionId) +export async function recordFixture(scaffold: WebScaffold, sessionId: SessionId, fixturePath: string): Promise { + const agent = scaffold.host.ctx.agents.get(sessionId) if (agent === undefined) throw new Error(`record harvest: no live agent for ${sessionId}`) const tokenized = scrubRequestHeaders(rawSessionLog(agent.session)) .split(sessionId).join('{{sessionId}}') - .split(harness.workspaceCwd).join('{{cwd}}') + .split(scaffold.workspaceCwd).join('{{cwd}}') await writeFile(fixturePath, tokenized) } @@ -274,27 +274,27 @@ export function fixtureUserPrompts(fixtureText: string): string[] { } /** - * Seed a recorded session fixture into the harness's persistence root through + * Seed a recorded session fixture into the scaffold's persistence root through * the REAL backend API (throwaway Context + SessionStore + JSONL plugin — the * semantic-checkpoint precedent), never raw file writes: no knowledge of * bucket hashing, filename encoding, or compression, and malformed shapes * fail loud at seed time. The fixture's recorded cwd is rewritten to the - * harness workspace so header/path identity and event payload paths agree. - * @param harness - the target harness. + * scaffold workspace so header/path identity and event payload paths agree. + * @param scaffold - the target scaffold. * @param fixtureText - raw recorded session.jsonl contents. * @param id - the seeded session id (stable for deterministic goldens). * @returns the seeded id. */ -export async function seedSession(harness: WebHarness, fixtureText: string, id: string): Promise { +export async function seedSession(scaffold: WebScaffold, fixtureText: string, id: string): Promise { // Committed fixtures tokenize run-local identity ({{sessionId}}/{{cwd}}, // written by recordFixture); realize both for this world before parsing. const realized = fixtureText .split('{{sessionId}}').join(id) - .split('{{cwd}}').join(harness.workspaceCwd) + .split('{{cwd}}').join(scaffold.workspaceCwd) const fixtureCwd = (JSON.parse(realized.split('\n', 1)[0]!) as { cwd?: string }).cwd const rewritten = fixtureCwd === undefined ? realized - : realized.split(fixtureCwd).join(harness.workspaceCwd) + : realized.split(fixtureCwd).join(scaffold.workspaceCwd) const events = parseSessionLog(rewritten) if (events.length === 0) throw new Error('seed fixture has no events') const last = events[events.length - 1]! @@ -305,7 +305,7 @@ export async function seedSession(harness: WebHarness, fixtureText: string, id: version: SESSION_FORMAT_VERSION, id: SessionId(id), createdAt: Date.now() - 60_000, - cwd: harness.workspaceCwd, + cwd: scaffold.workspaceCwd, delegationDepth: 0, } const ctx = new Context() @@ -313,7 +313,7 @@ export async function seedSession(harness: WebHarness, fixtureText: string, id: await ctx.plugin(SessionStore) // Same root as the host with the plugin's own default compression, so the // host's directory-scan list() sees one consistent encoding. - await ctx.plugin(SessionPersistenceJsonl, { root: harness.persistenceRoot }) + await ctx.plugin(SessionPersistenceJsonl, { root: scaffold.persistenceRoot }) await ctx.sessionPersistence.create(meta) await ctx.sessionPersistence.append(meta.id, events) // Deterministic sidebar order: cold summaries take updatedAt from mtime. diff --git a/apps/web/tests/seeded-history.e2e.ts b/apps/web/tests/seeded-history.e2e.ts index 734c0848c4..67bb5131c5 100644 --- a/apps/web/tests/seeded-history.e2e.ts +++ b/apps/web/tests/seeded-history.e2e.ts @@ -15,8 +15,8 @@ import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest' import { join } from 'node:path' import { assertFixtureInventory, captureStableAria, compareOrRefreshGolden, fixtureUserPrompts, - launchWebHarness, recordFixture, seedSession, watchConsole, webSnapshotMode, type WebHarness, -} from './harness.ts' + launchWebScaffold, recordFixture, seedSession, watchConsole, webSnapshotMode, type WebScaffold, +} from './scaffold.ts' import { saveFailureShot } from './support.ts' const SNAPSHOT_DIR = fileURLToPath(new URL('./snapshots/seeded-history', import.meta.url)) @@ -28,44 +28,44 @@ const SEED_ID = 'seeded-history-web-e2e' const PROMPT = 'Use the read tool twice in one assistant message: read a.txt and b.txt. Then reply with the single word DONE and stop.' describe('web e2e: seeded history renders through cold resume', () => { - let harness: WebHarness + let scaffold: WebScaffold let browser: Browser let page: Page let tripwire: ReturnType beforeAll(async () => { - harness = await launchWebHarness({}) + scaffold = await launchWebScaffold({}) // The read-tool targets exist in both modes: record needs them for the // live turn; replay's seeded log carries their recorded contents but the - // workspace stays consistent for any user poking the harness. - await writeFile(join(harness.workspaceCwd, 'a.txt'), 'alpha\n') - await writeFile(join(harness.workspaceCwd, 'b.txt'), 'beta\n') + // workspace stays consistent for any user poking the scaffold. + await writeFile(join(scaffold.workspaceCwd, 'a.txt'), 'alpha\n') + await writeFile(join(scaffold.workspaceCwd, 'b.txt'), 'beta\n') if (MODE !== 'record') { const raw = await readFile(SEED, 'utf8') expect(fixtureUserPrompts(raw), 'seed fixture must carry exactly the drive prompt').toEqual([PROMPT]) - await seedSession(harness, raw, SEED_ID) + await seedSession(scaffold, raw, SEED_ID) } browser = await chromium.launch() page = await browser.newPage({ viewport: { width: 1680, height: 1000 } }) tripwire = watchConsole(page) - await page.goto(harness.baseUrl, { waitUntil: 'load' }) + await page.goto(scaffold.baseUrl, { waitUntil: 'load' }) await page.waitForSelector('[class*="frame"]', { timeout: 30_000 }) }, 120_000) afterAll(async () => { await browser?.close() - await harness?.close() + await scaffold?.close() }) it.skipIf(MODE !== 'record')('records the seed turn live through the composer', async () => { onTestFailed(() => saveFailureShot(page, 'web-e2e-seeded-record')) const input = page.locator('textarea').first() await input.waitFor({ timeout: 10_000 }) - const settled = harness.whenTurnSettled() + const settled = scaffold.whenTurnSettled() await input.fill(PROMPT) await input.press('Enter') const sessionId = await settled - await recordFixture(harness, sessionId, SEED) + await recordFixture(scaffold, sessionId, SEED) }, 200_000) it.skipIf(MODE === 'record')('lists the seeded session cold and renders its history from the log', async () => { @@ -89,17 +89,34 @@ describe('web e2e: seeded history renders through cold resume', () => { it.skipIf(MODE === 'record')('matches the historical conversation aria golden', async () => { onTestFailed(() => saveFailureShot(page, 'web-e2e-seeded-aria')) - const snapshot = (await captureStableAria(page, '[class*="centerCol"]', harness.workspaceCwd)) + const snapshot = (await captureStableAria(page, '[class*="centerCol"]', scaffold.workspaceCwd)) .split(SEED_ID).join('{{seededId}}') await compareOrRefreshGolden(UI_EXPECTED, snapshot, MODE) }) + it.skipIf(MODE === 'record')('expands and collapses a tool row rebuilt from the cold log', async () => { + onTestFailed(() => saveFailureShot(page, 'web-e2e-seeded-toolrow')) + // Interaction over cold-resumed history: read rows are expand-in-place + // rows (rowExpands routes the click to toggleExpand, not openDetails), so + // the gesture under test is the inline fold over log-rebuilt content. + // Runs after the golden capture; still zero model calls. + const row = page.locator('[data-variant] [data-clickable][role="button"]').first() + await row.waitFor({ timeout: 10_000 }) + expect(await row.getAttribute('aria-expanded')).toBe('false') + await row.click() + await expect.poll(() => row.getAttribute('aria-expanded'), { timeout: 5_000 }).toBe('true') + // The expanded body renders the recorded tool result (a.txt's contents). + await expect.poll(() => page.getByText('alpha', { exact: false }).count(), { timeout: 5_000 }).toBeGreaterThan(0) + await row.click() + await expect.poll(() => row.getAttribute('aria-expanded'), { timeout: 5_000 }).toBe('false') + }) + it.skipIf(MODE === 'record')('issued zero model calls and stayed clean', async () => { // No replay fixture was installed and the llm seam is open — any stray // stream would have failed the turn loudly. Cleanliness pins the wire. expect(tripwire.pageErrors).toEqual([]) expect(tripwire.warnings).toEqual([]) - expect(harness.serverErrors).toEqual([]) + expect(scaffold.serverErrors).toEqual([]) await assertFixtureInventory(SNAPSHOT_DIR, ['seed.jsonl', 'ui.expected.md']) }) }) diff --git a/apps/web/tsconfig.json b/apps/web/tsconfig.json index 514cbe4d57..e1b68a4309 100644 --- a/apps/web/tsconfig.json +++ b/apps/web/tsconfig.json @@ -17,12 +17,12 @@ "src", "tests" ], - // The web e2e lane (harness + replay specs) boots the host spine and reads + // The web e2e lane (scaffold + replay specs) boots the host spine and reads // its Context merges — host-plane programs, checked in tsconfig.host.json; // this client-registered project must not also hold them (one program // cannot see both sides of the cordis Context merges). "exclude": [ - "tests/harness.ts", + "tests/scaffold.ts", "tests/replay-round-trip.e2e.ts", "tests/seeded-history.e2e.ts" ], diff --git a/packages/host/runtime/README.md b/packages/host/runtime/README.md index 4384f591a1..90a547c210 100644 --- a/packages/host/runtime/README.md +++ b/packages/host/runtime/README.md @@ -2,7 +2,7 @@ Host runtime assembly for `dsh`: `bootHost` composes the core plugin spine (LLM service + DeepSeek adapter, sessions with JSONL persistence and immediate fallback titles, optional first-message model summaries, system prompt, tools, agents, agent loop, workspace instructions, local bash, and the provider-neutral user-interaction service), `createApiProxy` implements the [`dsh-host-apiproxy`](../apiproxy/README.md) contract over that composition, and `startHost` is the one-step shell seam returning `{ api, handler, defaults, ctx, dispose }`. -Which plugins mount and with what defaults is decided only here — shells must not `ctx.plugin` to alter the assembly. `RunningHost.ctx` is a formal seam with exactly three sanctioned uses: mounting protocol front-door plugins (e.g. a future `dsh acp`), headless session-event subscription, and filling a capability seam the boot options deliberately left open (`llm: false` → the embedder installs its own LLM backend, e.g. the keyless web e2e harness's replay); consuming clients must not bypass `api` through it. +Which plugins mount and with what defaults is decided only here — shells must not `ctx.plugin` to alter the assembly. `RunningHost.ctx` is a formal seam with exactly three sanctioned uses: mounting protocol front-door plugins (e.g. a future `dsh acp`), headless session-event subscription, and filling a capability seam the boot options deliberately left open (`llm: false` → the embedder installs its own LLM backend, e.g. the keyless web e2e scaffold's replay); consuming clients must not bypass `api` through it. ## Configuration diff --git a/packages/host/runtime/src/boot.ts b/packages/host/runtime/src/boot.ts index e2f24a98ac..7e89a435db 100644 --- a/packages/host/runtime/src/boot.ts +++ b/packages/host/runtime/src/boot.ts @@ -69,7 +69,7 @@ export interface BootHostOptions { * LLM adapter selection: `'deepseek'` (default) mounts the DeepSeek adapter * (requires an API key at load), `false` mounts no adapter and leaves the * `llm` capability seam open for the embedder to fill on the returned ctx - * (e.g. the keyless web e2e harness installing a replay backend). With + * (e.g. the keyless web e2e scaffold installing a replay backend). With * `false` and nothing filled, the first stream fails loud with NO_ADAPTER — * the earliest resolvable point for an open capability seam. */ diff --git a/tsconfig.host.json b/tsconfig.host.json index a5f48ed22d..a8688ab3fb 100644 --- a/tsconfig.host.json +++ b/tsconfig.host.json @@ -8,7 +8,7 @@ "rewriteRelativeImportExtensions": false }, "include": [ - "apps/web/tests/harness.ts", + "apps/web/tests/scaffold.ts", "apps/web/tests/support.ts", "apps/web/tests/replay-round-trip.e2e.ts", "apps/web/tests/seeded-history.e2e.ts", From 13345fdadcd6625d60bdbd7fa61b4206292a53c0 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 24 Jul 2026 22:46:04 +0800 Subject: [PATCH 08/11] docs(i18n): re-record web e2e note pair after the scaffold rename The rename commit edited both sides of the bilingual pair but missed the re-record; the pairing gate compares blob hashes and went red. --- .../testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml index 1f55dfce3e..d759047151 100644 --- a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write -2026-07-24-web-gui-browser-e2e-lane.md: 3cabceb9667d3d1c153518d58b8d4c02b0578d20 -2026-07-24-web-gui-browser-e2e-lane.zh.md: 132caa453662f48619aa542c68b59f59b64acd0f +2026-07-24-web-gui-browser-e2e-lane.md: 9bab4d1eb9ec24acae0057143379629c87194d1f +2026-07-24-web-gui-browser-e2e-lane.zh.md: c61aee7ae5c070d1c82dda0ab78d8db44c16b9bf From 1e78054a78f037d2ea58f27cd7e0298f575f51cc Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 24 Jul 2026 23:04:57 +0800 Subject: [PATCH 09/11] ci: retrigger workflows (push event for 7a4cd7857 was dropped by Actions) From e4553deceda22b8088c108c92367366bcd09bd1c Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 24 Jul 2026 23:13:03 +0800 Subject: [PATCH 10/11] docs(i18n): re-record testing.md pair after merging master's bilingual split The merge added the web-browser-snapshot bullet to both sides of the now bilingual docs/testing.md; the pair record needs the post-merge hashes. --- docs/testing.i18n.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/testing.i18n.yaml b/docs/testing.i18n.yaml index 8ebdff8c55..e01ada78dd 100644 --- a/docs/testing.i18n.yaml +++ b/docs/testing.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write -testing.md: fd38fb7b20d76ef48c81c86badcf501f7c0dbd4e -testing.zh.md: 4584492350aefd5d72692093b08c0dcd4910a8af +testing.md: bf202c23574194d61d138c0f03073136dd29484a +testing.zh.md: 8b98f17c955fbb8cf7588b5bccc2e4c1298d3cf4 From f1577d620e0a43007f00d709f65c145b1a7533c5 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Sun, 26 Jul 2026 00:33:46 +0800 Subject: [PATCH 11/11] test(web): harden replay scaffold lifecycle --- .../2026-07-20-gui-testing-system.i18n.yaml | 4 +- .../process/2026-07-20-gui-testing-system.md | 2 +- .../2026-07-20-gui-testing-system.zh.md | 2 +- ...6-07-24-web-gui-browser-e2e-lane.i18n.yaml | 4 +- .../2026-07-24-web-gui-browser-e2e-lane.md | 24 +++--- .../2026-07-24-web-gui-browser-e2e-lane.zh.md | 24 +++--- apps/web/tests/replay-round-trip.e2e.ts | 13 ++- apps/web/tests/scaffold.ts | 82 ++++++++++++------- .../snapshots/fresh-round-trip/session.jsonl | 4 +- .../tests/snapshots/seeded-history/seed.jsonl | 4 +- packages/support/llm-replay/README.md | 2 +- scripts/doc-budgets.manifest.json | 2 +- 12 files changed, 97 insertions(+), 70 deletions(-) diff --git a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.i18n.yaml b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.i18n.yaml index ca443d7a16..ffa1e87103 100644 --- a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write -2026-07-20-gui-testing-system.md: b261bd2c84a628ab6fcdc29c59cf36a7b2428a76 -2026-07-20-gui-testing-system.zh.md: ecb8634695bd05359e6b590a825a4ad3604003b1 +2026-07-20-gui-testing-system.md: 546f65f065c0c2266773acc3c28b2833a094ba9b +2026-07-20-gui-testing-system.zh.md: 6601ae0a1c2bd1671af6f02961fbda81d30ab971 diff --git a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.md b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.md index b261bd2c84..546f65f065 100644 --- a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.md +++ b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.md @@ -20,7 +20,7 @@ Cut along the architecture's natural test seams into three tiers, bottom-up: |---|---|---|---| | 1 Protocol isomorphism | `AbstractApiClient` + `toFetchHandler` (bidirectional data / rpcId / zod types / SSE streams / batching / timeouts) | **The full chain at the isomorphic point**: `InProcessApiClient(toFetchHandler(脚本化 impl))` skips the network but genuinely runs the wire serialization — zero browser, pure node env | `packages/host/apiproxy/tests/client-handler.spec.ts` | | 2 Object-layer orchestration | `Session`/`SessionManager`/`ConnectionController` (state machines and timing: stitching / dedup / paging / optimistic draft clearing / pendingBuffers / reconnect / backoff) | **The "event sequence in → snapshot out" golden path**: programmable fakes + deferreds controlling timing + fake timers controlling backoff | `packages/client/{runtime,connection}/tests/` | -| 3 Assembled presentation | Built artifacts × the real client loader and plugin composition | App-owned semantic snapshots boot all eight built client plugins under jsdom for deterministic cross-plugin state changes; bare Playwright smoke separately proves the real browser/carrier boundary, with real-host cases self-skipping without a key; the keyless browser e2e lane replays recorded session fixtures through the real in-process web assembly (`llm: false` + dsh-llm-replay) against conversation aria goldens ([web e2e lane](../testing/2026-07-24-web-gui-browser-e2e-lane.md)) | `apps/web/tests/*.snapshot.ts`, `apps/web/tests/smoke-{fixture,real}.e2e.ts`, `apps/web/tests/{replay-round-trip,seeded-history}.e2e.ts` | +| 3 Assembled presentation | Built artifacts × the real client loader and plugin composition | App-owned semantic snapshots boot all eight built client plugins under jsdom for deterministic cross-plugin state changes; bare Playwright smoke separately proves the real browser/carrier boundary, with real-host cases self-skipping without a key; the keyless browser e2e lane disables the shipped model-adapter row and replays recorded session fixtures through `dsh-llm-replay` in the real in-process web assembly against conversation aria goldens ([web e2e lane](../testing/2026-07-24-web-gui-browser-e2e-lane.md)) | `apps/web/tests/*.snapshot.ts`, `apps/web/tests/smoke-{fixture,real}.e2e.ts`, `apps/web/tests/{replay-round-trip,seeded-history}.e2e.ts` | Inter-tier discipline: **each tier tests its own layer, upper tiers never re-test lower ones** — an app semantic snapshot pins only user-visible projection across the assembled plugin boundary, while Playwright smoke proves browser and carrier liveness; wire semantics belong to tier 1 and data semantics to tier 2. Pure-function layers (lineage/partial/notifier/fold-adapter) are tested directly with zero fakes in the same package's tests/ alongside tier 2. diff --git a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.zh.md b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.zh.md index ecb8634695..6601ae0a1c 100644 --- a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.zh.md +++ b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.zh.md @@ -20,7 +20,7 @@ GUI 栈需要考虑多种应用形态，同应用形态内的不同运行环境 |---|---|---|---| | 1 协议同构层 | `AbstractApiClient` + `toFetchHandler`（双向数据/rpcId/ZOD类型/SSE 流/合批/超时） | **同构点全链**：`InProcessApiClient(toFetchHandler(脚本化 impl))` 不过网络但真跑 wire 序列化——零浏览器、纯 node env | `packages/host/apiproxy/tests/client-handler.spec.ts` | | 2 对象层编排 | `Session`/`SessionManager`/`ConnectionController`（状态机与时序：缝合/去重/翻页/乐观清稿/pendingBuffers/重连/退避） | **「事件序列进→快照出」黄金路径**：可编程假体 + deferred 控时序 + fake timers 控退避 | `packages/client/{runtime,connection}/tests/` | -| 3 组装呈现层 | 构建产物 × 真实 client loader 与插件组合 | 归应用所有的语义快照会在 jsdom 下启动全部 8 个已构建的 client 插件，以固定确定性的跨插件状态变化；独立使用 Playwright 裸库的冒烟测试负责验证真实浏览器/承载层边界，真 host 用例在无密钥时自行跳过；无密钥浏览器 e2e 车道把录制的会话 fixture 通过真实进程内 web 组装（`llm: false` + dsh-llm-replay）回放，与会话区 aria 期望输出比对（[web e2e 车道](../testing/2026-07-24-web-gui-browser-e2e-lane.md)） | `apps/web/tests/*.snapshot.ts`、`apps/web/tests/smoke-{fixture,real}.e2e.ts`、`apps/web/tests/{replay-round-trip,seeded-history}.e2e.ts` | +| 3 组装呈现层 | 构建产物 × 真实 client loader 与插件组合 | 归应用所有的语义快照会在 jsdom 下启动全部 8 个已构建的 client 插件，以固定确定性的跨插件状态变化；独立使用 Playwright 裸库的冒烟测试负责验证真实浏览器/承载层边界，真 host 用例在无密钥时自行跳过；无密钥浏览器 e2e 车道会禁用交付配置中的模型适配器行，并通过 `dsh-llm-replay` 在真实进程内 web 组装中回放录制的会话 fixture，与会话区 aria 期望输出比对（[web e2e 车道](../testing/2026-07-24-web-gui-browser-e2e-lane.md)） | `apps/web/tests/*.snapshot.ts`、`apps/web/tests/smoke-{fixture,real}.e2e.ts`、`apps/web/tests/{replay-round-trip,seeded-history}.e2e.ts` | 层间纪律：**下层各测各的，上层不重测下层**：应用语义快照只固定组装后插件边界上的用户可见投影，Playwright 冒烟测试负责验证浏览器与承载层是否存活；wire 语义归 1 层，数据语义归 2 层。纯函数层（lineage/partial/notifier/fold-adapter）随 2 层同包 tests/ 零假体直测。 diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml index e50541fa85..ef389fecb3 100644 --- a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write -2026-07-24-web-gui-browser-e2e-lane.md: b6e62f59e12c64dd5386eaaabe15e863ef52e291 -2026-07-24-web-gui-browser-e2e-lane.zh.md: 9f806c7030336bad4f7a9ca7695a03982d8a7878 +2026-07-24-web-gui-browser-e2e-lane.md: fb870c4bb2c85d8be7ac11f9f29f05bf24f446a4 +2026-07-24-web-gui-browser-e2e-lane.zh.md: c43e526d27fd4d8cf4f774e8a03480930682041d diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md index b6e62f59e1..fb870c4bb2 100644 --- a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md @@ -6,7 +6,7 @@ English | [中文](2026-07-24-web-gui-browser-e2e-lane.zh.md) ## Problem -The web GUI ships as a real assembled chain — chromium page → client plugin bundles → HTTP unary RPC + two SSE streams → `toFetchHandler`/apiproxy → `bootHost`'s agent loop, tools, and JSONL persistence — and no test exercised that chain keylessly and deterministically. The [GUI testing system](../process/2026-07-20-gui-testing-system.md) covers tier 1 (wire isomorphism in node), tier 2 (object-layer state machines), and tier-3 smokes, but the keyless smoke drives `FixtureApiClient` — no host, no wire, no agent loop — while the full-chain smoke needs `DEEPSEEK_API_KEY` and a live model, so it is nondeterministic and self-skips in keyless CI. The snapshot philosophy of [docs/testing.md](../../../../docs/testing.md) — record once with a key, replay forever keyless, refresh on format churn — already covers the ACP, headless `stream-json`, and TUI transcript surfaces; the web surface was the one assembled product shape without it. The gap is exactly where the two confirmed GUI P0s hid: the wire carriage chain the fixture client short-circuits. +The web GUI ships as a real assembled chain — chromium page → client plugin bundles → HTTP unary RPC + two SSE streams → `toFetchHandler`/apiproxy → the host agent loop, tools, and JSONL persistence — and no test exercised that chain keylessly and deterministically. The [GUI testing system](../process/2026-07-20-gui-testing-system.md) covers tier 1 (wire isomorphism in node), tier 2 (object-layer state machines), and tier-3 smokes, but the keyless smoke drives `FixtureApiClient` — no host, no wire, no agent loop — while the full-chain smoke needs `DEEPSEEK_API_KEY` and a live model, so it is nondeterministic and self-skips in keyless CI. The snapshot philosophy of [docs/testing.md](../../../../docs/testing.md) — record once with a key, replay forever keyless, refresh on format churn — already covers the ACP, headless `stream-json`, and TUI transcript surfaces; the web surface was the one assembled product shape without it. The gap is exactly where the two confirmed GUI P0s hid: the wire carriage chain the fixture client short-circuits. ## Decision @@ -16,33 +16,33 @@ The web GUI ships as a real assembled chain — chromium page → client plugin A plain shared-fixture module (the [testing-policy sanctioned shape](../../../../docs/testing.md)), not a package: the gate-worthy logic — replay derivation, session parsing, log scrubbing, persistence — lives in the gated packages `dsh-llm-replay`, `dsh-acp-snapshot`, and `dsh-session-persistence-jsonl`; what remains is boot wiring and browser glue, and chromium-driving code cannot hold per-file 100% coverage on the browserless coverage runners. -`launchWebScaffold()` boots the real web composition: the shipped `apps/cli/cordis.yml` through the vendored Loader's include boot — the same tree and mechanism `AppCLIEntry` drives for `dsh web` (the config-tree boot landed upstream on 2026-07-25, superseding this lane's earlier in-process `startHost` assembly and resolving the original Loader-ization question in favor of Loader-izing). Divergences ride include patches over the SAME shipped tree, the ACP `cordis.snapshot.yml` pattern expressed in-process: temp `persistenceRoot`, `workspace-context` disabled (recorded fixtures must not embed this repo's AGENTS.md), `session-title-llm` disabled (its fire-and-forget title call would race the loop for the session's replay cursor), the webserver row pinned to port 0 with the built dist, and in keyless modes `llm-deepseek` disabled. A patch id that stops matching a row fails the boot sweep loudly instead of drifting. The boot runs `chdir`'d to the temp workspace so the api-gateway's `process.cwd()` session default, tool cwds, and fixtures agree; the `dsh web` bin's own glue (argv, profile json, AppCLIEntry) stays held by the keyless CLI smokes in `smoke-real.e2e.ts`. +`launchWebScaffold()` boots the real web composition from the shipped `apps/cli/cordis.yml` through the vendored Loader's include mechanism — the same tree and mechanism `AppCLIEntry` drives for `dsh web`. Divergences ride include patches over that tree, the ACP `cordis.snapshot.yml` pattern expressed in-process: temp `persistenceRoot`, `workspace-context` disabled (recorded fixtures must not embed this repo's AGENTS.md), `session-title-llm` disabled (its fire-and-forget title call would race the loop for the session's replay cursor), the webserver row pinned to port 0 with the built dist, and in keyless modes `llm-deepseek` disabled. A patch id that stops matching a row fails the boot sweep loudly instead of drifting. The boot runs `chdir`'d to the temp workspace so the api-gateway's `process.cwd()` session default, tool cwds, and fixtures agree; the `dsh web` bin's own glue (argv, profile json, AppCLIEntry) stays held by the keyless CLI smokes in `smoke-real.e2e.ts`. Setup rollback and ordinary close both dispose the Cordis tree before removing the two owned temp roots, attempt every cleanup independently, and report cleanup failures without masking the setup failure. -Keyless model displacement is the disabled adapter row plus `installLlmReplay` filling the open seam on the settled root ctx in providers-catalog mode — never catch-all: with the adapter row disabled no adapter exists, so catch-all would leave `resolveModelContext` unroutable and `compact-basic`'s post-step pressure check would warn every step instead of being provably inert (the published 128k `contextWindow` keeps it inert for small fixtures). The direct install rather than an inserted replay plugin row is deliberate: it returns the `ReplayHandle` the teardown consumption check needs. A scenario with no fixture leaves the seam empty, so a stray stream fails loud with NO_ADAPTER. (The first round's `BootHostOptions.llm: 'deepseek' | false` seam was superseded by the config-tree boot and removed with `bootHost`'s web role.) +Keyless model displacement is the disabled adapter row plus `installLlmReplay` filling the open seam on the settled root ctx in providers-catalog mode — never catch-all: with the adapter row disabled no adapter exists, so catch-all would leave `resolveModelContext` unroutable and `compact-basic`'s post-step pressure check would warn every step instead of being provably inert (the published 128k `contextWindow` keeps it inert for small fixtures). The direct install rather than an inserted replay plugin row is deliberate: it returns the `ReplayHandle` the teardown consumption check needs. A scenario with no fixture leaves the seam empty, so a stray stream fails loud with NO_ADAPTER. `seedSession()` seeds cold sessions through the real persistence API — a throwaway `Context` mounting `SessionStore` + `SessionPersistenceJsonl` against the host's root, `create()` + `append()`, one `utimes` backdate for deterministic sidebar order (the `semantic-checkpoint.snapshot.ts` precedent) — never raw file writes, so the seeder knows nothing of bucket hashing, filename encoding, or compression, and the host's zstd default needs no boot knob. Seeds are validated at seed time (parseable, ending in `turn/end` — an open final turn would be mutated by resume's crash repair). ### Determinism rules -The barrier stack for a prompted turn, in order: (1) host-side `await agent.whenIdle()` under a timeout, keyed off the in-process `turn/end` — the idle flip follows the persistence flush, so one await covers turn completion and durability; (2) browser settled poll (streaming detached, final text visible); (3) any log harvest after `host.dispose()`. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync); file polling is banned (slow on NFS, superseded by `whenIdle`); `networkidle` is banned outright (never resolves while an SSE stream is open). +The barrier stack for replay-mode browser assertions is, in order: (1) host-side `await agent.whenIdle()` under a timeout, keyed off the in-process `turn/end` — the idle flip follows the persistence flush, so one await covers turn completion and durability; (2) browser settled poll (streaming detached, final text visible). Record-mode log harvest runs after `whenIdle()` and before scaffold disposal while the live session remains available. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync); file polling is banned (slow on NFS, superseded by `whenIdle`); `networkidle` is banned outright (never resolves while an SSE stream is open). No single-shot transient-DOM assertions: every hop from replay yield to React commit can coalesce chunks, so sampling `[data-streaming]` is a race by construction. Streaming incrementality is asserted from the persisted `assistant/chunk` events (model-visible ⟺ logged makes the log the authoritative proof). `dsh-llm-replay`'s opt-in `paceMs` (default absent = burst) is a realism knob so the browser observes genuinely incremental SSE; correctness never leans on it, and abort during a pace wait cancels promptly. -Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Scaffold `close()` calls the `ReplayHandle.assertConsumed()` teardown check (every recorded script bound, every cursor drained), converting silent underruns and shifted bindings into crisp diagnostics. No vitest retry on the lane; one chromium per file, fresh context per scenario, one host per scenario; viewport pinned; selectors anchor on roles, `data-*` attributes, and visible text. +Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Scaffold `close()` calls the `ReplayHandle.assertConsumed()` teardown check (every recorded script bound, every cursor drained), converting silent underruns and shifted bindings into crisp diagnostics. No vitest retry on the lane; one chromium per file, fresh context per scenario, one host per scenario; viewport pinned; interaction selectors anchor on roles, `data-*` attributes, and visible text, while the frame and conversation-region captures use the existing CSS-module local-name anchors. ### Expected outputs -One committed golden per scenario: a normalized `ariaSnapshot()` of the conversation region (`ui.expected.md`) — uuid/cwd/workspace-basename/duration tokens normalized, captured poll-until-equal at the settled milestone — plus a few role/text anchor assertions that stay green under a semantics-preserving component rewrite while the golden churns reviewably. The aria tree is the mechanization of the client rule "assert what the user would see, never class names". World-state assertions ride `host.ctx` session events inline (which tools ran, `turn/end` completed) instead of a second committed log golden: the persisted-log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence, and re-pinning it here would double refresh cost against the tier discipline. `refresh` is the sole golden writer — a missing golden in replay mode fails with the healing command rather than self-bootstrapping. +One committed golden per scenario: a normalized `ariaSnapshot()` of the conversation region (`ui.expected.md`) — uuid/cwd/workspace-basename/duration tokens normalized, captured poll-until-equal at the settled milestone — plus a few role/text anchor assertions that stay green under a semantics-preserving component rewrite while the golden churns reviewably. The aria tree is the mechanization of the client rule "assert what the user would see, never class names". World-state assertions ride root-context session events inline (which tool call produced which durable result, whether `turn/end` completed) instead of a second committed log golden: the persisted-log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence, and re-pinning it here would double refresh cost against the tier discipline. `refresh` is the sole golden writer — a missing golden in replay mode fails with the healing command rather than self-bootstrapping. -The typecheck plane split is structural: `apps/web/tests/{scaffold,support,replay-round-trip.e2e,seeded-history.e2e}.ts` are host-plane programs (they boot the host spine), so they are excluded from the client-registered `apps/web` project and included file-by-file in `tsconfig.host.json` — one program cannot hold both sides of the cordis `Context` merges. +The typecheck plane split is structural: the three files that boot the host spine (`scaffold`, `replay-round-trip.e2e`, and `seeded-history.e2e`) are excluded from the client-registered `apps/web` project. Those files and their shared `support.ts` are included file-by-file in `tsconfig.host.json` — one program cannot hold both sides of the cordis `Context` merges. ### Modes and fixtures -`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless) as inline spec branches — the TUI shape, not a suite factory: at two scenarios the acp-snapshot factory machinery has no owner, and the genuinely shared parts are already exported (`scrubRequestHeaders`, `parseSessionLog`, `installLlmReplay`). Each spec splits into drive steps (type, send, `whenTurnSettled` — run in all modes, never waiting on model-content selectors, so record cannot hang on a live model answering differently) and assertion steps (replay/refresh only). Record = drive live through the real composer + harvest the in-memory `session.header`/`session.events` (the TUI `rawSessionLog` shape — no file decompression) + `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}` tokenization; a follow-up keyless refresh regenerates `ui.expected.md`. Both scenarios' fixtures were recorded against this assembly through this flow. A drift guard ties each spec's drive prompt to the fixture's recorded `user/message`. A fixture-inventory guard holds each scenario directory closed (exact file set, every JSONL a scrub fixed-point). Web fixtures scrub headers everywhere and pin no header class, following the TUI precedent over the strict [pinned-header](2026-07-06-pin-request-header-content-in-one-scenario.md) reading — see Deferred. +`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless) as inline spec branches — the TUI shape, not a suite factory: at two scenarios the acp-snapshot factory machinery has no owner, and the genuinely shared parts are already exported (`scrubRequestHeaders`, `parseSessionLog`, `installLlmReplay`). Each spec splits into drive steps (type, send, `whenTurnSettled` — run in all modes, never waiting on model-content selectors, so record cannot hang on a live model answering differently) and assertion steps (replay/refresh only). Record = drive live through the real composer + harvest the in-memory `session.header`/`session.events` (the TUI `rawSessionLog` shape — no file decompression) + `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}`/`{{rpcId}}` tokenization; a follow-up keyless refresh regenerates `ui.expected.md`. Both scenarios' fixtures were recorded against this assembly through this flow. A drift guard ties each spec's drive prompt to the fixture's recorded `user/message`. A fixture-inventory guard holds each scenario directory closed (exact file set, every JSONL a scrub fixed-point with no run-local `rpcId`). Web fixtures scrub headers everywhere and pin no header class, following the TUI precedent over the strict [pinned-header](2026-07-06-pin-request-header-content-in-one-scenario.md) reading — see Deferred. ### Scenarios -1. **`replay-round-trip`** — new session, prompt through the real composer, replay streams reasoning + a `bash` tool call that really executes in the temp workspace + final text (paced 15ms). Asserts settled markdown, the aria golden, and inline world state (bash `tool/call`, completed `turn/end`, >10 chunk events). +1. **`replay-round-trip`** — new session, prompt through the real composer, replay streams reasoning + a `bash` tool call that really executes in the temp workspace + final text (paced 15ms). Asserts settled markdown, the aria golden, and inline world state (the bash call's durable result is exactly `WEB_E2E_OK\n`, completed `turn/end`, >10 chunk events). 2. **`seeded-history`** — a recorded session seeded cold; the sidebar lists it (group row → session row, collapsed by default), opening renders tool cards and text purely from the log through the implicit cold-resume attach inside `session.history` — zero model calls in replay, so no binding constraints; record mode drives the same turn live (real `read` tool against seeded workspace files) to produce the seed. ### CI stance @@ -65,9 +65,9 @@ Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot **A `packages/support/web-snapshot` package with a `defineWebSnapshotSuite` factory.** Rejected: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and at two scenarios a factory generalizes from one consumer while the genuinely shared logic is already exported from gated packages. Re-entry trigger: a second web-shaped consumer or ≥6 scenarios with demonstrably drifting inline branches; the package boundary would then be drawn browser-free. -**A committed normalized-session-log golden as a second expected surface.** Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers. Inline world-state assertions on `host.ctx` events keep the world-verification duty. +**A committed normalized-session-log golden as a second expected surface.** Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers. Inline world-state assertions on root-context events keep the world-verification duty. -**Spawning the `dsh web` bin with a `DSH_SNAPSHOT` replay branch.** Rejected: it needs a test-mode branch plus env plumbing in the product bin where the in-process route uses exported production functions; the bin's thin glue is covered by the keyless CLI smokes. Becomes free only if the web host is ever Loader-ized — declined in review, with the app-assembly ruling reaffirmed. +**Spawning the `dsh web` bin with a `DSH_SNAPSHOT` replay branch.** Rejected: it needs a test-only replay branch plus environment plumbing in the shipped CLI. The in-process scaffold already loads the same `apps/cli/cordis.yml`; only argv, profile JSON, and `AppCLIEntry` glue remain outside it, and the keyless CLI smokes cover those paths. **Changing the wire protocol for testability.** Rejected: the contract already has a first-class keyless isomorphic seam (`InProcessApiClient(toFetchHandler(api))`), the per-event unbatched SSE is exactly what makes replay observable in a browser, and testing a wire we no longer ship would invert the tier's purpose. @@ -81,7 +81,7 @@ The lane itself: `pnpm run test:web` runs both scenarios keylessly alongside the ## Deferred -- **Web header-class pin**: web fixtures tokenize `{{system}}`/`{{tools}}` everywhere and no scenario pins bootHost's composed prompt/tool schemas (`TODO(web-header-pin)` — the scaffold `recordFixture` JSDoc marks it). Following the TUI scrub-everywhere precedent; revisit when the web assembly's header diverges from the repl composition it mirrors. +- **Web header-class pin**: web fixtures tokenize `{{system}}`/`{{tools}}` everywhere and no scenario pins the web composition's prompt/tool schemas (`TODO(web-header-pin)` — the scaffold `recordFixture` JSDoc marks it). Following the TUI scrub-everywhere precedent; revisit when the web assembly's header diverges from the repl composition it mirrors. - **CI browser provisioning**: reversal of the no-browser-in-CI ruling, staged criteria above (`TODO(ci-browser)`). - **Follow-up-prompt-after-resume scenario**: the history/live stitch path over the real wire; add as its own scenario when that code changes or regresses. diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md index 9f806c7030..c43e526d27 100644 --- a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md @@ -6,7 +6,7 @@ Status: implemented ## 问题 -Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bundle → HTTP 单次 RPC + 两条 SSE（Server-Sent Events）流 → `toFetchHandler`/apiproxy → `bootHost` 的 agent loop（智能体循环）、工具与 JSONL 持久化——却没有任何测试无密钥且确定性地检验这条链。[GUI 测试体系](../process/2026-07-20-gui-testing-system.md)覆盖第 1 层（Node 中的协议同构）、第 2 层（对象层状态机）与第 3 层冒烟测试，但无密钥冒烟驱动的是 `FixtureApiClient`——没有 host、没有 wire、没有 agent loop——而全链路冒烟需要 `DEEPSEEK_API_KEY` 和真实模型，因此不确定、在无密钥 CI 中自行跳过。[docs/testing.md](../../../../docs/testing.md) 的快照哲学——带密钥录制一次、永久无密钥回放、格式变动时刷新——已覆盖 ACP（Agent Client Protocol）、headless `stream-json` 与 TUI 三个文本记录（transcript）表面；web 表面是唯一没有这层保障的组装形态。而缺口恰恰是两起已实证 GUI P0 藏身之处：fixture（测试前置数据）客户端短路掉的 wire 承载链。 +Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bundle → HTTP 单次 RPC + 两条 SSE（Server-Sent Events）流 → `toFetchHandler`/apiproxy → host 端的 agent loop（智能体循环）、工具与 JSONL 持久化——却没有任何测试无密钥且确定性地检验这条链。[GUI 测试体系](../process/2026-07-20-gui-testing-system.md)覆盖第 1 层（Node 中的协议同构）、第 2 层（对象层状态机）与第 3 层冒烟测试，但无密钥冒烟驱动的是 `FixtureApiClient`——没有 host、没有 wire、没有 agent loop——而全链路冒烟需要 `DEEPSEEK_API_KEY` 和真实模型，因此不确定、在无密钥 CI 中自行跳过。[docs/testing.md](../../../../docs/testing.md) 的快照哲学——带密钥录制一次、永久无密钥回放、格式变动时刷新——已覆盖 ACP（Agent Client Protocol）、headless `stream-json` 与 TUI 三个文本记录（transcript）表面；web 表面是唯一没有这层保障的组装形态。而缺口恰恰是两起已实证 GUI P0 藏身之处：fixture（测试前置数据）客户端短路掉的 wire 承载链。 ## 决策 @@ -16,33 +16,33 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu 一个普通的共享 fixture 模块（[测试政策认可的形态](../../../../docs/testing.md)），不是包：值得门禁把守的逻辑——回放推导、会话解析、日志脱敏、持久化——都在已受门禁的包 `dsh-llm-replay`、`dsh-acp-snapshot`、`dsh-session-persistence-jsonl` 中；剩下的只是启动接线和浏览器胶水，而驱动 chromium 的源码在无浏览器的覆盖率 runner 上无法诚实保持逐文件 100% 覆盖率。 -`launchWebScaffold()` 启动真实 web 组合：经 vendored Loader 的 include boot 加载交付的 `apps/cli/cordis.yml`——与 `AppCLIEntry` 为 `dsh web` 驱动的是同一棵树、同一套机制（配置树 boot 于 2026-07-25 在上游落地，取代了本车道第一轮的进程内 `startHost` 组装，也把当初的 Loader 化问题裁定为「Loader 化」）。差异全部经 include patch 骑在同一棵交付树上，即 ACP `cordis.snapshot.yml` 模式的进程内表达：临时 `persistenceRoot`；禁用 `workspace-context`（录制的 fixture 不得嵌入本仓库的 AGENTS.md）；禁用 `session-title-llm`（其发后不管的标题调用会与循环争抢会话的回放游标）；webserver 行钉到端口 0 加已构建 dist；无密钥模式下禁用 `llm-deepseek`。patch 的 id 一旦不再匹配任何行，boot 扫描会大声失败而不是漂移。boot 在临时工作区 `chdir` 下运行，使 api-gateway 的 `process.cwd()` 会话默认值、工具 cwd 与 fixture 一致；`dsh web` bin 自身的胶水（argv、profile json、AppCLIEntry）仍由 `smoke-real.e2e.ts` 中的无密钥 CLI 冒烟把守。 +`launchWebScaffold()` 通过 vendored Loader 的 include 机制，从交付的 `apps/cli/cordis.yml` 启动真实 web 组合——与 `AppCLIEntry` 为 `dsh web` 驱动的是同一棵树、同一套机制。差异全部经 include patch 覆盖在这棵树上，即 ACP `cordis.snapshot.yml` 模式的进程内表达：临时 `persistenceRoot`；禁用 `workspace-context`（录制的 fixture 不得嵌入本仓库的 AGENTS.md）；禁用 `session-title-llm`（其发后不管的标题调用会与循环争抢会话的回放游标）；webserver 行钉到端口 0 加已构建 dist；无密钥模式下禁用 `llm-deepseek`。patch 的 id 一旦不再匹配任何行，boot 扫描会大声失败而不是漂移。boot 在临时工作区 `chdir` 下运行，使 api-gateway 的 `process.cwd()` 会话默认值、工具 cwd 与 fixture 一致；`dsh web` bin 自身的胶水（argv、profile json、AppCLIEntry）仍由 `smoke-real.e2e.ts` 中的无密钥 CLI 冒烟把守。初始化回滚和正常关闭都会先对 Cordis 树执行 dispose（资源释放），再删除 scaffold 持有的两个临时根目录；每项清理都会独立尝试，并会报告清理失败而不掩盖初始化失败。 -无密钥的模型替换 = 禁用适配器行的 patch 加 `installLlmReplay` 在停稳的根 ctx 上以提供方目录（providers-catalog）模式填充开放的 seam——绝不用 catch-all：适配器行被禁用后不存在任何适配器，catch-all 会让 `resolveModelContext` 无路由可走，`compact-basic` 的步后压力检查将步步告警，而不是被可证明地闲置（发布的 128k `contextWindow` 使该路径对小 fixture 保持闲置）。选择直接安装而非插入回放插件行是刻意的：直接安装返回收尾消费检查所需的 `ReplayHandle`。没有 fixture 的场景让 seam 保持空置，任何离群的流式调用都会以 NO_ADAPTER 大声失败。（第一轮的 `BootHostOptions.llm: 'deepseek' | false` seam 已随配置树 boot 取代 `bootHost` 的 web 角色而移除。） +无密钥的模型替换 = 禁用适配器行的 patch 加 `installLlmReplay` 在停稳的根 ctx 上以提供方目录（providers-catalog）模式填充开放的 seam——绝不用 catch-all：适配器行被禁用后不存在任何适配器，catch-all 会让 `resolveModelContext` 无路由可走，`compact-basic` 的步后压力检查将步步告警，而不是被可证明地闲置（发布的 128k `contextWindow` 使该路径对小 fixture 保持闲置）。选择直接安装而非插入回放插件行是刻意的：直接安装返回收尾消费检查所需的 `ReplayHandle`。没有 fixture 的场景让 seam 保持空置，任何离群的流式调用都会以 NO_ADAPTER 大声失败。 `seedSession()` 通过真实持久化 API 播种冷会话——一次性 `Context` 挂载 `SessionStore` + `SessionPersistenceJsonl` 指向 host 的根目录，`create()` + `append()`，一次 `utimes` 回拨保证侧栏顺序确定（`semantic-checkpoint.snapshot.ts` 先例）——绝不裸写文件，因此播种器对桶哈希、文件名编码、压缩一无所知，host 的 zstd 默认值也无需任何启动开关。种子在播种时即校验（可解析、以 `turn/end` 结尾——未闭合的最终轮次会被恢复（resume）的崩溃修复改写）。 ### 确定性规则 -提示一轮对话的屏障栈，按序：（1）host 侧 `await agent.whenIdle()` 加超时，以进程内 `turn/end` 为锚——空闲翻转发生在持久化落盘之后，一次等待同时覆盖轮次完成与持久性；（2）浏览器安定轮询（流式输出节点已卸载、最终文本可见）；（3）任何日志采收都在 `host.dispose()` 之后。单独监听进程内 `turn/end` 是错误屏障（它先于 SSE 帧到达浏览器、先于 fsync 触发）；文件轮询被禁止（NFS 上慢，且被 `whenIdle` 取代）；`networkidle` 被彻底禁止（SSE 流保持打开时它永不解析）。 +回放模式下浏览器断言的屏障栈，按序：（1）host 侧 `await agent.whenIdle()` 加超时，以进程内 `turn/end` 为锚——空闲翻转发生在持久化落盘之后，一次等待同时覆盖轮次完成与持久性；（2）浏览器安定轮询（流式输出节点已卸载、最终文本可见）。录制模式下，日志采收在 `whenIdle()` 之后、scaffold 释放之前进行，此时运行中的会话仍然可用。单独监听进程内 `turn/end` 是错误屏障（它先于 SSE 帧到达浏览器、先于 fsync 触发）；文件轮询被禁止（NFS 上慢，且被 `whenIdle` 取代）；`networkidle` 被彻底禁止（SSE 流保持打开时它永不解析）。 不做单次瞬态 DOM 断言：从回放产出到 React 提交的每一跳都可能合并分片，采样 `[data-streaming]` 天然就是竞态。流式输出的增量性由持久化的 `assistant/chunk` 事件断言（模型可见 ⟺ 已记录，使日志成为权威证据）。`dsh-llm-replay` 的可选 `paceMs`（默认缺省 = 突发）只是让浏览器观察到真正增量 SSE 的真实感旋钮；正确性绝不依赖它，且节奏等待期间中止会即时取消。 -每个场景都会因任何 pageerror 或客户端的连接丢失/间隙修复控制台警告而失败：否则重连机制加历史重同步会把一条死掉的 SSE 通路自愈掉，套件反而认证了坏 wire。Scaffold 的 `close()` 调用 `ReplayHandle.assertConsumed()` 收尾检查（每个已录脚本都被绑定、每个游标都耗尽），把静默的少放与错绑变成清晰诊断。车道不设 vitest 重试；每文件一个 chromium、每场景一个新 context、每场景一个 host；视口固定；选择器只锚定 role、`data-*` 属性和可见文本。 +每个场景都会因任何 pageerror 或客户端的连接丢失/间隙修复控制台警告而失败：否则重连机制加历史重同步会把一条死掉的 SSE 通路自愈掉，套件反而认证了坏 wire。Scaffold 的 `close()` 调用 `ReplayHandle.assertConsumed()` 收尾检查（每个已录脚本都被绑定、每个游标都耗尽），把静默的少放与错绑变成清晰诊断。车道不设 vitest 重试；每文件一个 chromium、每场景一个新 context、每场景一个 host；视口固定；交互选择器锚定 role、`data-*` 属性和可见文本，而 frame 与会话区采集则使用既有的 CSS 模块局部类名锚点。 ### 预期输出 -每场景一份提交的预期输出：会话区规范化 `ariaSnapshot()`（`ui.expected.md`）——uuid/cwd/工作区目录名/时长归一为稳定 token，在安定里程碑处轮询至两次相等再采集——外加几条 role/文本锚断言，让保语义的组件重写在预期输出可评审地变动时仍保持绿色锚点。aria 树是 client 规则「断言用户所见，绝不断言类名」的机械化。世界状态断言内联在 `host.ctx` 会话事件上（哪些工具运行了、`turn/end` 完成）而不是第二份提交的日志预期输出：持久化日志表面已由 ACP/headless/TUI 套件经同一循环和持久化钉住，在此重复钉住会违背分层纪律、翻倍刷新成本。`refresh` 是预期输出的唯一写入者——回放模式下预期输出缺失会连同修复命令一起报错，而不是静默自举。 +每场景一份提交的预期输出：会话区规范化 `ariaSnapshot()`（`ui.expected.md`）——uuid/cwd/工作区目录名/时长归一为稳定 token，在安定里程碑处轮询至两次相等再采集——外加几条 role/文本锚断言，让保语义的组件重写在预期输出可评审地变动时仍保持绿色锚点。aria 树是 client 规则「断言用户所见，绝不断言类名」的机械化。世界状态断言内联在根上下文的会话事件上（哪次工具调用产生了哪项已持久化的工具结果、`turn/end` 是否完成）而不是第二份提交的日志预期输出：持久化日志表面已由 ACP/headless/TUI 套件经同一循环和持久化钉住，在此重复钉住会违背分层纪律、翻倍刷新成本。`refresh` 是预期输出的唯一写入者——回放模式下预期输出缺失会连同修复命令一起报错，而不是静默自举。 -类型检查平面切分是结构性的：`apps/web/tests/{scaffold,support,replay-round-trip.e2e,seeded-history.e2e}.ts` 是 host 平面程序（它们启动 host 主干），因此被排除出注册在 client 侧的 `apps/web` 工程，逐文件纳入 `tsconfig.host.json`——一个程序不能同时持有 cordis `Context` 合并的两侧。 +类型检查平面切分是结构性的：启动 host 主干的三个文件（`scaffold`、`replay-round-trip.e2e` 和 `seeded-history.e2e`）被排除出注册在 client 侧的 `apps/web` 工程。这三个文件及其共享的 `support.ts` 逐文件纳入 `tsconfig.host.json`——一个程序不能同时持有 cordis `Context` 合并的两侧。 ### 模式与 fixture -`DSH_SNAPSHOT` 以内联 spec 分支选择 replay（默认，无密钥）、record（带密钥）或 refresh（无密钥）——TUI 的形态，不是套件工厂：两个场景撑不起 acp-snapshot 工厂机制，且真正共享的部分已被导出（`scrubRequestHeaders`、`parseSessionLog`、`installLlmReplay`）。每个 spec 切分为驱动步骤（输入、发送、`whenTurnSettled`——所有模式都执行，绝不等待模型内容选择器，因此 record 不会因真实模型答法不同而挂起）与断言步骤（仅 replay/refresh）。Record = 经真实输入框实时驱动 + 采收内存中的 `session.header`/`session.events`（TUI 的 `rawSessionLog` 形态——无需文件解压）+ `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}` token 化；随后一次无密钥 refresh 重新生成 `ui.expected.md`。两个场景的 fixture 都经此流程对本组装录制。一条漂移防线把每个 spec 的驱动提示词与 fixture 录制的 `user/message` 绑定。fixture 清单防线保持每个场景目录封闭（精确文件集合，每个 JSONL 都是脱敏不动点）。Web fixture 全部脱敏请求头且不钉任何头类别，沿用 TUI 先例而非[钉住请求头](2026-07-06-pin-request-header-content-in-one-scenario.md)的严格读法——见「暂缓」。 +`DSH_SNAPSHOT` 以内联 spec 分支选择 replay（默认，无密钥）、record（带密钥）或 refresh（无密钥）——TUI 的形态，不是套件工厂：两个场景撑不起 acp-snapshot 工厂机制，且真正共享的部分已被导出（`scrubRequestHeaders`、`parseSessionLog`、`installLlmReplay`）。每个 spec 切分为驱动步骤（输入、发送、`whenTurnSettled`——所有模式都执行，绝不等待模型内容选择器，因此 record 不会因真实模型答法不同而挂起）与断言步骤（仅 replay/refresh）。Record = 经真实输入框实时驱动 + 采收内存中的 `session.header`/`session.events`（TUI 的 `rawSessionLog` 形态——无需文件解压）+ `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}`/`{{rpcId}}` token 化；随后一次无密钥 refresh 重新生成 `ui.expected.md`。两个场景的 fixture 都经此流程对本组装录制。一条漂移防线把每个 spec 的驱动提示词与 fixture 录制的 `user/message` 绑定。fixture 清单防线保持每个场景目录封闭（精确文件集合，每个 JSONL 都是脱敏不动点，不含当次运行的 `rpcId`）。Web fixture 全部脱敏请求头且不钉任何头类别，沿用 TUI 先例而非[钉住请求头](2026-07-06-pin-request-header-content-in-one-scenario.md)的严格读法——见「暂缓」。 ### 场景 -1. **`replay-round-trip`**——新会话，经真实输入框发送提示词，回放流式输出推理（reasoning）+ 一次在临时工作区真实执行的 `bash` 工具调用 + 最终文本（15ms 节奏）。断言安定后的 markdown、aria 预期输出与内联世界状态（bash `tool/call`、完成的 `turn/end`、>10 个分片事件）。 +1. **`replay-round-trip`**——新会话，经真实输入框发送提示词，回放流式输出推理（reasoning）+ 一次在临时工作区真实执行的 `bash` 工具调用 + 最终文本（15ms 节奏）。断言安定后的 markdown、aria 预期输出与内联世界状态（这次 bash 调用的已持久化工具结果严格等于 `WEB_E2E_OK\n`、完成的 `turn/end`、>10 个分片事件）。 2. **`seeded-history`**——冷播种一份已录会话；侧栏列出它（分组行 → 会话行，默认折叠），打开后纯凭日志经 `session.history` 内的隐式冷恢复挂载渲染工具卡片与文本——replay 下零模型调用，因此没有任何绑定约束；record 模式实时驱动同一轮（真实 `read` 工具读取播种的工作区文件）来产出种子。 ### CI 立场 @@ -65,9 +65,9 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu **`packages/support/web-snapshot` 包 + `defineWebSnapshotSuite` 工厂。** 已否决：驱动 chromium 的源码在无浏览器的覆盖率 runner 上无法诚实保持逐文件 100%，且两个场景就上工厂是从单一消费方过度泛化，真正共享的逻辑已从受门禁的包中导出。重启条件：出现第二个 web 形态消费方，或 ≥6 个场景的内联分支被证实各自漂移；届时包边界将画在无浏览器一侧。 -**第二份提交的规范化会话日志预期输出。** 已否决：日志表面已由 ACP/headless/TUI 套件经同一循环与持久化钉住；在此只会翻倍刷新成本并重复测试下层。内联在 `host.ctx` 事件上的世界状态断言保住了验证世界的义务。 +**第二份提交的规范化会话日志预期输出。** 已否决：日志表面已由 ACP/headless/TUI 套件经同一循环与持久化钉住；在此只会翻倍刷新成本并重复测试下层。内联在根上下文事件上的世界状态断言保住了验证世界的义务。 -**以 `DSH_SNAPSHOT` 回放分支拉起 `dsh web` bin。** 已否决：它需要在产品 bin 里加测试模式分支和环境变量管道，而进程内路线用的是零产品改动的导出生产函数；bin 的薄胶水已由无密钥 CLI 冒烟覆盖。只有 web host 某天 Loader 化它才免费——评审中已否决，并重申了应用内组装的裁定。 +**以 `DSH_SNAPSHOT` 回放分支拉起 `dsh web` bin。** 已否决：它需要在交付的 CLI 中增加测试专用回放分支和环境变量管道。进程内 scaffold 已加载同一份 `apps/cli/cordis.yml`；只剩 argv、profile JSON 和 `AppCLIEntry` 胶水不在其覆盖范围内，而这些路径已由无密钥 CLI 冒烟覆盖。 **为可测试性改 wire 协议。** 已否决：契约已有第一等的无密钥同构 seam（`InProcessApiClient(toFetchHandler(api))`），逐事件不合批的 SSE 恰是回放在浏览器中可观测的原因，测试一条不再交付的 wire 会颠倒该层的存在意义。 @@ -81,7 +81,7 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu ## 暂缓 -- **Web 头类别钉住**：web fixture 处处 token 化 `{{system}}`/`{{tools}}`，没有场景钉住 bootHost 组装的提示词/工具 schema（`TODO(web-header-pin)`——scaffold 的 `recordFixture` JSDoc 有标记）。沿用 TUI 处处脱敏先例；当 web 组装的请求头与其镜像的 repl 组合进一步分叉时重审。 +- **Web 头类别钉住**：web fixture 处处 token 化 `{{system}}`/`{{tools}}`，没有场景钉住 web 组合的提示词/工具 schema（`TODO(web-header-pin)`——scaffold 的 `recordFixture` JSDoc 有标记）。沿用 TUI 处处脱敏先例；当 web 组装的请求头与其镜像的 repl 组合进一步分叉时重审。 - **CI 浏览器供给**：推翻 CI 无浏览器裁定，分阶段标准见上（`TODO(ci-browser)`）。 - **恢复后追问场景**：真实 wire 上的历史/实时缝合路径；当该代码变更或回归时作为独立场景补充。 diff --git a/apps/web/tests/replay-round-trip.e2e.ts b/apps/web/tests/replay-round-trip.e2e.ts index 8baabde0ba..131f3fa5fd 100644 --- a/apps/web/tests/replay-round-trip.e2e.ts +++ b/apps/web/tests/replay-round-trip.e2e.ts @@ -80,9 +80,16 @@ describe('web e2e: fresh round trip through the real assembly', () => { // legal — the chunk-event assertions below carry incrementality. }) await expect.poll(() => page.getByText('DONE', { exact: true }).count(), { timeout: 15_000 }).toBeGreaterThanOrEqual(1) - // World state, not self-report: bash really ran and the turn closed clean. - const toolCalls = sessionEvents.filter(e => e.type === 'tool/call') - expect(toolCalls.map(e => (e as SessionEvent & { data: { name: string } }).data.name)).toContain('bash') + // World state, not self-report: the real bash executor returned the exact + // command output, and the turn closed cleanly. + const bashCall = sessionEvents.find(event => event.type === 'tool/call' && event.data.name === 'bash') + if (bashCall?.type !== 'tool/call') throw new Error('the replayed turn did not call the bash tool') + const bashResult = sessionEvents.find(event => + event.type === 'tool/result' && event.data.callId === bashCall.data.callId) + if (bashResult?.type !== 'tool/result') throw new Error('the bash tool call produced no durable result') + expect(bashResult.data.isError).toBe(false) + expect(bashResult.data.content.filter(block => block.type === 'text').map(block => block.text).join('')) + .toBe('WEB_E2E_OK\n') const turnEnds = sessionEvents.filter(e => e.type === 'turn/end') expect(turnEnds.length).toBe(1) expect((turnEnds[0] as SessionEvent & { data: { reason: { kind: string } } }).data.reason.kind).toBe('completed') diff --git a/apps/web/tests/scaffold.ts b/apps/web/tests/scaffold.ts index d858e0f7ad..babfbde919 100644 --- a/apps/web/tests/scaffold.ts +++ b/apps/web/tests/scaffold.ts @@ -105,6 +105,15 @@ export interface LaunchOptions { paceMs?: number } +/** Dispose the booted tree and remove both owned temp roots, reporting every independent cleanup failure. */ +async function cleanupScaffoldWorld(ctx: Context, workspaceCwd: string, persistenceRoot: string): Promise { + const failures: unknown[] = [] + await Promise.resolve(ctx.fiber.dispose()).catch((error: unknown) => failures.push(error)) + await rm(workspaceCwd, { recursive: true, force: true }).catch((error: unknown) => failures.push(error)) + await rm(persistenceRoot, { recursive: true, force: true }).catch((error: unknown) => failures.push(error)) + return failures +} + /** * Boot the real web composition under the current snapshot mode. * @param options - replay fixture selection and pacing. @@ -120,7 +129,15 @@ export async function launchWebScaffold(options: LaunchOptions = {}): Promise failures.push(cleanupError)) + if (failures.length > 1) throw new AggregateError(failures, 'web scaffold temp-root setup failed') + throw error + } // The include patch set — the same mechanism AppCLIEntry and the ACP // snapshot overlay use, applied over the SAME shipped tree (a patch id that @@ -143,9 +160,11 @@ export async function launchWebScaffold(options: LaunchOptions = {}): Promise undefined) - await rm(persistenceRoot, { recursive: true, force: true }).catch(() => undefined) + if (process.cwd() !== originalCwd) process.chdir(originalCwd) + const cleanupFailures = await cleanupScaffoldWorld(ctx, workspaceCwd, persistenceRoot) + if (cleanupFailures.length > 0) { + throw new AggregateError([error, ...cleanupFailures], 'web scaffold setup failed and cleanup was incomplete') + } throw error } finally { if (process.cwd() !== originalCwd) process.chdir(originalCwd) } - const port = ctx.get('httpServer')?.port - if (port === undefined) { - await ctx.fiber.dispose() - throw new Error('web e2e scaffold: httpServer service missing after settled boot') - } - // Fill the open llm seam on the settled root ctx (llm-deepseek is disabled - // in keyless modes; a scenario with no fixture leaves the seam empty so a - // stray stream fails loud with NO_ADAPTER). The direct install, unlike the - // plugin row, returns the ReplayHandle for the teardown consumption check. - let replayHandle: ReplayHandle | undefined - if (mode !== 'record' && options.replayFixture !== undefined) { - replayHandle = installLlmReplay(ctx, { - file: options.replayFixture, - providers: REPLAY_PROVIDERS, - ...(options.paceMs === undefined ? {} : { paceMs: options.paceMs }), - }) - } - return { mode, baseUrl: `http://127.0.0.1:${port}`, @@ -222,9 +241,7 @@ export async function launchWebScaffold(options: LaunchOptions = {}): Promise failures.push(e)) - await rm(workspaceCwd, { recursive: true, force: true }).catch((e: unknown) => failures.push(e)) - await rm(persistenceRoot, { recursive: true, force: true }).catch((e: unknown) => failures.push(e)) + failures.push(...await cleanupScaffoldWorld(ctx, workspaceCwd, persistenceRoot)) if (failures.length > 0) throw new AggregateError(failures, 'web scaffold teardown failed') }, } @@ -247,9 +264,9 @@ function rawSessionLog(session: Session): string { * Record-mode fixture write-back: harvest the live session, scrub request * headers to {{system}}/{{tools}} (TODO(web-header-pin): the web lane pins no * header class — a deliberate deviation logged in the Agent Note's deferred - * work), tokenize the run-local session id and cwd ({{sessionId}}/{{cwd}}, - * the committed ACP fixture convention — re-records then diff only on real - * content), and write the committed fixture. + * work), tokenize the run-local session id, cwd, and browser RPC id + * ({{sessionId}}/{{cwd}}/{{rpcId}}, the committed fixture convention — + * re-records then diff only on real content), and write the fixture. * @param scaffold - the record-mode scaffold. * @param sessionId - the driven session. * @param fixturePath - the committed session.jsonl / seed.jsonl target. @@ -260,6 +277,7 @@ export async function recordFixture(scaffold: WebScaffold, sessionId: SessionId, const tokenized = scrubRequestHeaders(rawSessionLog(agent.session)) .split(sessionId).join('{{sessionId}}') .split(scaffold.workspaceCwd).join('{{cwd}}') + .replace(/"rpcId":"[^"]+"/g, '"rpcId":"{{rpcId}}"') await writeFile(fixturePath, tokenized) } @@ -389,7 +407,7 @@ export async function compareOrRefreshGolden(goldenPath: string, actual: string, /** * Fixture-inventory guard (the TUI afterAll shape): the scenario directory * holds exactly the expected files and every committed JSONL is a scrub - * fixed-point (no request-header bulk escaped the record write-back). + * fixed-point without a run-local browser RPC id. * @param dir - the scenario snapshot directory. * @param expected - the exact expected file inventory. */ @@ -399,6 +417,8 @@ export async function assertFixtureInventory(dir: string, expected: string[]): P for (const entry of entries.filter(name => name.endsWith('.jsonl'))) { const content = await readFile(join(dir, entry), 'utf8') expect(scrubRequestHeaders(content), `${dir}/${entry} carries request-header bulk`).toBe(content) + expect(content, `${dir}/${entry} carries a run-local rpcId`) + .not.toMatch(/"rpcId":"(?!\{\{rpcId\}\})[^"]+"/) } } diff --git a/apps/web/tests/snapshots/fresh-round-trip/session.jsonl b/apps/web/tests/snapshots/fresh-round-trip/session.jsonl index 9bd1959879..21218b459d 100644 --- a/apps/web/tests/snapshots/fresh-round-trip/session.jsonl +++ b/apps/web/tests/snapshots/fresh-round-trip/session.jsonl @@ -1,6 +1,6 @@ {"type":"session","version":0,"id":"{{sessionId}}","createdAt":1784973850091,"cwd":"{{cwd}}/workspace"} -{"type":"turn/start","seq":0,"time":1784973850102,"data":{"turn":1,"trigger":{"kind":"message","source":{"kind":"user","rpcId":"c4e068dc-c277-46ae-9713-6a2694027fb5"}}}} -{"type":"user/message","seq":1,"time":1784973850103,"data":{"content":[{"type":"text","text":"Use the bash tool to run exactly: echo WEB_E2E_OK. Then reply with the single word DONE and stop."}],"source":{"kind":"user","rpcId":"c4e068dc-c277-46ae-9713-6a2694027fb5"}},"surfaceOp":"append"} +{"type":"turn/start","seq":0,"time":1784973850102,"data":{"turn":1,"trigger":{"kind":"message","source":{"kind":"user","rpcId":"{{rpcId}}"}}}} +{"type":"user/message","seq":1,"time":1784973850103,"data":{"content":[{"type":"text","text":"Use the bash tool to run exactly: echo WEB_E2E_OK. Then reply with the single word DONE and stop."}],"source":{"kind":"user","rpcId":"{{rpcId}}"}},"surfaceOp":"append"} {"type":"session/title","seq":2,"time":1784973850105,"data":{"title":"Use the bash tool to","messageSeqs":[1],"source":{"kind":"fallback"}}} {"type":"step/start","seq":3,"time":1784973850164,"data":{"turn":1,"step":1}} {"type":"request/header","seq":4,"time":1784973850165,"data":{"header":{"config":{"provider":"deepseek","model":"deepseek-v4-flash"},"system":"{{system}}","tools":"{{tools}}","messagePrefix":["{{messagePrefix}}"]},"reason":"initial"}} diff --git a/apps/web/tests/snapshots/seeded-history/seed.jsonl b/apps/web/tests/snapshots/seeded-history/seed.jsonl index 0f61158a54..27e31004bc 100644 --- a/apps/web/tests/snapshots/seeded-history/seed.jsonl +++ b/apps/web/tests/snapshots/seeded-history/seed.jsonl @@ -1,6 +1,6 @@ {"type":"session","version":0,"id":"{{sessionId}}","createdAt":1784974100747,"cwd":"{{cwd}}/workspace"} -{"type":"turn/start","seq":0,"time":1784974100758,"data":{"turn":1,"trigger":{"kind":"message","source":{"kind":"user","rpcId":"f95c6f1c-f1b4-42bf-ba40-c05ae0647a70"}}}} -{"type":"user/message","seq":1,"time":1784974100759,"data":{"content":[{"type":"text","text":"Use the read tool twice in one assistant message: read a.txt and b.txt. Then reply with the single word DONE and stop."}],"source":{"kind":"user","rpcId":"f95c6f1c-f1b4-42bf-ba40-c05ae0647a70"}},"surfaceOp":"append"} +{"type":"turn/start","seq":0,"time":1784974100758,"data":{"turn":1,"trigger":{"kind":"message","source":{"kind":"user","rpcId":"{{rpcId}}"}}}} +{"type":"user/message","seq":1,"time":1784974100759,"data":{"content":[{"type":"text","text":"Use the read tool twice in one assistant message: read a.txt and b.txt. Then reply with the single word DONE and stop."}],"source":{"kind":"user","rpcId":"{{rpcId}}"}},"surfaceOp":"append"} {"type":"session/title","seq":2,"time":1784974100761,"data":{"title":"Use the read tool twice","messageSeqs":[1],"source":{"kind":"fallback"}}} {"type":"step/start","seq":3,"time":1784974100827,"data":{"turn":1,"step":1}} {"type":"request/header","seq":4,"time":1784974100828,"data":{"header":{"config":{"provider":"deepseek","model":"deepseek-v4-flash"},"system":"{{system}}","tools":"{{tools}}","messagePrefix":["{{messagePrefix}}"]},"reason":"initial"}} diff --git a/packages/support/llm-replay/README.md b/packages/support/llm-replay/README.md index a8d811f35b..534d289b19 100644 --- a/packages/support/llm-replay/README.md +++ b/packages/support/llm-replay/README.md @@ -2,7 +2,7 @@ A replay LLM plugin for keyless snapshot tests. It yields model streams reconstructed from a recorded **session JSONL** fixture, so a test can boot the real agent against a fixed model transcript with no API key. With `providers` configured it registers a replay-only adapter whose catalog is available to scenarios that exercise model discovery; without `providers` it installs the catch-all `llm/stream` waterfall used by tests that do not need discovery. -Its consumers are the ACP snapshot harness in `examples/acp-agent` and the `stream-json` snapshot in `examples/headless-agent`; each loads this plugin in place of a real LLM adapter. Keeping derivation and replay here places that logic under the per-file 100% coverage gate on `packages/*/src`. +Its consumers are the ACP, headless `stream-json`, and TUI snapshot suites plus the web browser e2e lane. Loader-driven suites mount this plugin in place of a real LLM adapter; the web lane installs it directly to retain the teardown consumption handle. Keeping derivation and replay here places that logic under the per-file 100% coverage gate on `packages/*/src`. ## How the fixture works diff --git a/scripts/doc-budgets.manifest.json b/scripts/doc-budgets.manifest.json index f6f8a5c3fc..c2fee63c70 100644 --- a/scripts/doc-budgets.manifest.json +++ b/scripts/doc-budgets.manifest.json @@ -4,7 +4,7 @@ "docs/architecture.md": 1800, "docs/cordis-primer.md": 600, "docs/defensive-patterns.md": 550, - "docs/testing.md": 1040, + "docs/testing.md": 1100, "examples/AGENTS.md": 310, "packages/AGENTS.md": 660, "packages/README.md": 790