From 2cd37e276518ef04489f5ddc0e08574827a9f3f8 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 24 Jul 2026 19:57:27 +0800 Subject: [PATCH] docs(testing): web e2e lane docs + Agent Note to implemented MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit testing.md gains the web browser snapshot tier entry (divergent DSH_SNAPSHOT=... test:web commands) and names apps/web/tests/snapshots/ as the web surface's snapshot home. The GUI testing note's tier map and lane map gain the e2e scenarios (both languages, pair re-recorded) and drop the stale verify-session-real references (those scripts left with the missions/ tree). packages/client/AGENTS.md check ladder covers the wire-carriage trigger and refresh/record commands. acp-snapshot README stops claiming the whole package is ACP-specific — its normalizers are transport-neutral with three consumers now. vitest.web.config.ts header carries TODO(ci-browser) with the staged-reversal pointer. The design-study Agent Note moves proposed/ -> implemented/ rewritten in present tense: all review decisions recorded (llm:false seam over the placeholder-key hack, providers-mode replay, whenIdle barrier stack, single aria golden + anchors, TUI-style inline modes over a suite factory, scrub-only header stance, CI deferral) with re-entry triggers under Deferred. --- .../2026-07-20-gui-testing-system.i18n.yaml | 4 +- .../process/2026-07-20-gui-testing-system.md | 8 +- .../2026-07-20-gui-testing-system.zh.md | 8 +- .../2026-07-24-web-gui-browser-e2e-lane.md | 88 ++++++++++++++++++ .../2026-07-24-web-gui-browser-e2e-lane.md | 92 ------------------- apps/web/tests/harness.ts | 10 +- docs/testing.md | 3 +- packages/client/AGENTS.md | 2 +- packages/support/acp-snapshot/README.md | 2 +- vitest.web.config.ts | 11 ++- 10 files changed, 114 insertions(+), 114 deletions(-) create mode 100644 .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md delete mode 100644 .agents/notes/proposed/testing/2026-07-24-web-gui-browser-e2e-lane.md diff --git a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.i18n.yaml b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.i18n.yaml index b21353698c..ca443d7a16 100644 --- a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write -2026-07-20-gui-testing-system.md: e42dafcdf37e48475e7d420eaad9600e6c20891c -2026-07-20-gui-testing-system.zh.md: e4ef6246e59e0ad6c0a3070a38c546f964e347fa +2026-07-20-gui-testing-system.md: b261bd2c84a628ab6fcdc29c59cf36a7b2428a76 +2026-07-20-gui-testing-system.zh.md: ecb8634695bd05359e6b590a825a4ad3604003b1 diff --git a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.md b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.md index e42dafcdf3..b261bd2c84 100644 --- a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.md +++ b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.md @@ -20,7 +20,7 @@ Cut along the architecture's natural test seams into three tiers, bottom-up: |---|---|---|---| | 1 Protocol isomorphism | `AbstractApiClient` + `toFetchHandler` (bidirectional data / rpcId / zod types / SSE streams / batching / timeouts) | **The full chain at the isomorphic point**: `InProcessApiClient(toFetchHandler(脚本化 impl))` skips the network but genuinely runs the wire serialization — zero browser, pure node env | `packages/host/apiproxy/tests/client-handler.spec.ts` | | 2 Object-layer orchestration | `Session`/`SessionManager`/`ConnectionController` (state machines and timing: stitching / dedup / paging / optimistic draft clearing / pendingBuffers / reconnect / backoff) | **The "event sequence in → snapshot out" golden path**: programmable fakes + deferreds controlling timing + fake timers controlling backoff | `packages/client/{runtime,connection}/tests/` | -| 3 Assembled presentation | Built artifacts × the real client loader and plugin composition | App-owned semantic snapshots boot all eight built client plugins under jsdom for deterministic cross-plugin state changes; bare Playwright smoke separately proves the real browser/carrier boundary, with real-host cases self-skipping without a key | `apps/web/tests/*.snapshot.ts`, `apps/web/tests/smoke-{fixture,real}.e2e.ts` | +| 3 Assembled presentation | Built artifacts × the real client loader and plugin composition | App-owned semantic snapshots boot all eight built client plugins under jsdom for deterministic cross-plugin state changes; bare Playwright smoke separately proves the real browser/carrier boundary, with real-host cases self-skipping without a key; the keyless browser e2e lane replays recorded session fixtures through the real in-process web assembly (`llm: false` + dsh-llm-replay) against conversation aria goldens ([web e2e lane](../testing/2026-07-24-web-gui-browser-e2e-lane.md)) | `apps/web/tests/*.snapshot.ts`, `apps/web/tests/smoke-{fixture,real}.e2e.ts`, `apps/web/tests/{replay-round-trip,seeded-history}.e2e.ts` | Inter-tier discipline: **each tier tests its own layer, upper tiers never re-test lower ones** — an app semantic snapshot pins only user-visible projection across the assembled plugin boundary, while Playwright smoke proves browser and carrier liveness; wire semantics belong to tier 1 and data semantics to tier 2. Pure-function layers (lineage/partial/notifier/fold-adapter) are tested directly with zero fakes in the same package's tests/ alongside tier 2. @@ -33,15 +33,15 @@ Inter-tier discipline: **each tier tests its own layer, upper tiers never re-tes |---|---|---|---| | Baseline | `pnpm run test:gui` | Tier 1+2 vitest (`packages/client packages/host`), seconds-fast, no browser, no server | Casually, after touching any GUI source | | Semantic snapshot | `DSH_EXAMPLE_MODE=lib pnpm run test:snapshot` | Keyless assembled-application semantics plus the repo's transport-specific expected outputs | After a human-visible GUI change; before delivery | -| Browser end-to-end | `pnpm run test:web` | Rebuilds the front-end dist first, then runs the tier-3 two-level smoke (fixture level + real-host level self-skip) | After touching the build surface/boot/carriage; before delivery | +| Browser end-to-end | `pnpm run test:web` | Rebuilds the front-end dist first, then runs the tier-3 browser set: the two-level smoke (fixture level + real-host level self-skip) plus the keyless replayed e2e scenarios (`DSH_SNAPSHOT=record`/`refresh` re-record fixtures / rewrite goldens) | After touching the build surface/boot/carriage; before delivery | | Gate | `pnpm run test:coverage` | The repo-wide gate (host and client GUI packages included, except annotated browser-grade exclusions) | The PR window | **Division of labor between the browser scripts and vitest**: Playwright owns browser/carrier black-box regression and long sequential user journeys; ordinary vitest owns data-layer semantics such as reference stability, timing, and wire shapes; snapshot vitest owns stable app-level semantic output through the built composition. These lanes complement each other rather than duplicating assertions. ## Anti-regression discipline -- **Every bug fix pins an assertion**: a browser-visible bug is pinned into the regression section of its owning verify script (one pin = one report line); a data-layer bug is pinned into the matching spec (precedent: the res-close misjudgment pinned in the webserver bridge suite — pure Node, reproduces in seconds, no longer needs the 12s browser sentinel as the only defense). -- **All-green on fixture is not done, the real host must pass too**: what the fixture short-circuits is exactly the wire carriage chain (node:http bridge close semantics, real network timing); both empirically confirmed bugs hid there. Changes touching connection/bridge/handler/SSE must run `verify-session-real`. +- **Every bug fix pins an assertion**: a browser-visible bug is pinned into its owning browser spec (smoke or e2e scenario); a data-layer bug is pinned into the matching spec (precedent: the res-close misjudgment pinned in the webserver bridge suite — pure Node, reproduces in seconds, no longer needs the 12s browser sentinel as the only defense). +- **All-green on fixture is not done, the real wire must pass too**: what the fixture short-circuits is exactly the wire carriage chain (node:http bridge close semantics, real network timing); both empirically confirmed bugs hid there. Changes touching connection/bridge/handler/SSE must run the browser lane (`pnpm run test:web`) — its keyless e2e scenarios drive the real HTTP/SSE carriage, and the with-key real-host smoke remains the live-model complement. - The code-on-disk-is-the-answer reconciliation workflow: when a behavior change lands and turns existing cases red, reconcile on the spot (fix the test or fix the code, with the RFC/contract as arbiter); no red left hanging. ## Consequences diff --git a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.zh.md b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.zh.md index e4ef6246e5..ecb8634695 100644 --- a/.agents/notes/implemented/process/2026-07-20-gui-testing-system.zh.md +++ b/.agents/notes/implemented/process/2026-07-20-gui-testing-system.zh.md @@ -20,7 +20,7 @@ GUI 栈需要考虑多种应用形态，同应用形态内的不同运行环境 |---|---|---|---| | 1 协议同构层 | `AbstractApiClient` + `toFetchHandler`（双向数据/rpcId/ZOD类型/SSE 流/合批/超时） | **同构点全链**：`InProcessApiClient(toFetchHandler(脚本化 impl))` 不过网络但真跑 wire 序列化——零浏览器、纯 node env | `packages/host/apiproxy/tests/client-handler.spec.ts` | | 2 对象层编排 | `Session`/`SessionManager`/`ConnectionController`（状态机与时序：缝合/去重/翻页/乐观清稿/pendingBuffers/重连/退避） | **「事件序列进→快照出」黄金路径**：可编程假体 + deferred 控时序 + fake timers 控退避 | `packages/client/{runtime,connection}/tests/` | -| 3 组装呈现层 | 构建产物 × 真实 client loader 与插件组合 | 归应用所有的语义快照会在 jsdom 下启动全部 8 个已构建的 client 插件，以固定确定性的跨插件状态变化；独立使用 Playwright 裸库的冒烟测试负责验证真实浏览器/承载层边界，真 host 用例在无密钥时自行跳过 | `apps/web/tests/*.snapshot.ts`、`apps/web/tests/smoke-{fixture,real}.e2e.ts` | +| 3 组装呈现层 | 构建产物 × 真实 client loader 与插件组合 | 归应用所有的语义快照会在 jsdom 下启动全部 8 个已构建的 client 插件，以固定确定性的跨插件状态变化；独立使用 Playwright 裸库的冒烟测试负责验证真实浏览器/承载层边界，真 host 用例在无密钥时自行跳过；无密钥浏览器 e2e 车道把录制的会话 fixture 通过真实进程内 web 组装（`llm: false` + dsh-llm-replay）回放，与会话区 aria 期望输出比对（[web e2e 车道](../testing/2026-07-24-web-gui-browser-e2e-lane.md)） | `apps/web/tests/*.snapshot.ts`、`apps/web/tests/smoke-{fixture,real}.e2e.ts`、`apps/web/tests/{replay-round-trip,seeded-history}.e2e.ts` | 层间纪律：**下层各测各的，上层不重测下层**：应用语义快照只固定组装后插件边界上的用户可见投影，Playwright 冒烟测试负责验证浏览器与承载层是否存活；wire 语义归 1 层，数据语义归 2 层。纯函数层（lineage/partial/notifier/fold-adapter）随 2 层同包 tests/ 零假体直测。 @@ -33,15 +33,15 @@ GUI 栈需要考虑多种应用形态，同应用形态内的不同运行环境 |---|---|---|---| | 基础 | `pnpm run test:gui` | 1+2 层 vitest（`packages/client packages/host`），秒级、无浏览器无 server | 改 GUI 任意源码后随手跑 | | 语义快照 | `DSH_EXAMPLE_MODE=lib pnpm run test:snapshot` | 无需密钥的组装应用语义，以及仓库按传输形态划分的预期输出 | 用户可见的 GUI 变更后；交付前 | -| 浏览器端到端 | `pnpm run test:web` | 先重建前端 dist，再跑 3 层双级 smoke（fixture 级 + 真 host 级 self-skip） | 改构建面/boot/承载后；交付前 | +| 浏览器端到端 | `pnpm run test:web` | 先重建前端 dist，再跑 3 层浏览器全集：双级 smoke（fixture 级 + 真 host 级 self-skip）加上无密钥回放 e2e 场景（`DSH_SNAPSHOT=record`/`refresh` 重录 fixture / 重写期望输出） | 改构建面/boot/承载后；交付前 | | 门禁 | `pnpm run test:coverage` | 全仓 gate（host 与 client GUI 包均纳入，仅排除带注释的浏览器级例外） | PR 窗口 | **浏览器脚本与 vitest 的分工**：Playwright 负责浏览器/承载层黑盒回归和较长的连续用户操作流程；普通 vitest 负责引用稳定性、时序和 wire 结构等数据层语义；快照 vitest 通过构建后的组合负责稳定的应用层语义输出。这些车道彼此互补，而不重复断言。 ## 防回归纪律 -- **修一个 bug 钉一条断言**：浏览器可见的 bug 钉进所属 verify 脚本的回归节（一钉一行 report）；数据层 bug 钉进对应 spec（先例：res-close 误判钉在 webserver 桥 suite——纯 Node 秒级复现，不再需要 12s 浏览器哨兵作唯一防线）。 -- **fixture 全绿不算完，真 host 也要过**：fixture 短路的恰是 wire 承载链（node:http 桥 close 语义、真网络时序），两次实证 bug 都藏在那里。改动触及连接/桥/handler/SSE 的，`verify-session-real` 必跑。 +- **修一个 bug 钉一条断言**：浏览器可见的 bug 钉进所属浏览器 spec（smoke 或 e2e 场景）；数据层 bug 钉进对应 spec（先例：res-close 误判钉在 webserver 桥 suite——纯 Node 秒级复现，不再需要 12s 浏览器哨兵作唯一防线）。 +- **fixture 全绿不算完，真 wire 也要过**：fixture 短路的恰是 wire 承载链（node:http 桥 close 语义、真网络时序），两次实证 bug 都藏在那里。改动触及连接/桥/handler/SSE 的，浏览器车道（`pnpm run test:web`）必跑——其无密钥 e2e 场景驱动真实 HTTP/SSE 承载，带密钥的真 host smoke 仍是真模型侧的补充。 - 落盘代码即答案的对表工作流：行为改动落盘打红既有用例时，当场对表校准（改测试还是改代码以 RFC/契约为裁），不留悬红。 ## Consequences diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md new file mode 100644 index 0000000000..fddd1e9a0c --- /dev/null +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md @@ -0,0 +1,88 @@ +# Agent Note: Keyless browser e2e lane for the web GUI + +Status: implemented + +## Problem + +The web GUI ships as a real assembled chain — chromium page → client plugin bundles → HTTP unary RPC + two SSE streams → `toFetchHandler`/apiproxy → `bootHost`'s agent loop, tools, and JSONL persistence — and no test exercised that chain keylessly and deterministically. The [GUI testing system](../process/2026-07-20-gui-testing-system.md) covers tier 1 (wire isomorphism in node), tier 2 (object-layer state machines), and tier-3 smokes, but the keyless smoke drives `FixtureApiClient` — no host, no wire, no agent loop — while the full-chain smoke needs `DEEPSEEK_API_KEY` and a live model, so it is nondeterministic and self-skips in keyless CI. The snapshot philosophy of [docs/testing.md](../../../../docs/testing.md) — record once with a key, replay forever keyless, refresh on format churn — already covers the ACP, headless `stream-json`, and TUI transcript surfaces; the web surface was the one assembled product shape without it. The gap is exactly where the two confirmed GUI P0s hid: the wire carriage chain the fixture client short-circuits. + +## Decision + +`pnpm run test:web` carries a keyless, deterministic browser e2e lane under `apps/web/tests/`: recorded session-log fixtures replayed through `@deepseek-ai/dsh-llm-replay` against the real in-process web assembly, asserting a normalized conversation aria golden plus in-process world state. No new package; the product deltas are the `BootHostOptions.llm` seam and two additive `dsh-llm-replay` surfaces. + +### Harness: `apps/web/tests/harness.ts` + +A plain shared-fixture module (the [testing-policy sanctioned shape](../../../../docs/testing.md)), not a package: the gate-worthy logic — replay derivation, session parsing, log scrubbing, persistence — lives in the gated packages `dsh-llm-replay`, `dsh-acp-snapshot`, and `dsh-session-persistence-jsonl`; what remains is boot wiring and browser glue, and chromium-driving code cannot hold per-file 100% coverage on the browserless coverage runners. + +`launchWebHarness()` boots the real web assembly in-process from the exported production functions — `startHost({ boot: { …, llm: false } })`, `installLlmReplay(host.ctx, { file, providers, paceMs })`, `mountWebPlugins(host.ctx, roster, anchor)`, `createHostWebPluginRegistry`, `startWebServer({ port: 0, … })`. This is the web analog of the TUI suite mounting the production bundle in-process ([TUI snapshots](2026-07-18-tui-terminal-state-snapshots.md)): the real entry boundary (`dsh web` bin arg-parsing, dist resolution) stays held by the keyless CLI smokes in `smoke-real.e2e.ts`, and the web surface has no `cordis.yml` to bypass — assembly is written in the app per the [GUI layering decision](../architecture/2026-07-19-gui-layering-and-rpc-protocol.md), a ruling this lane's design review explicitly reaffirmed (Loader-izing `dsh web` was declined; it would be its own proposal). Two deliberate assembly divergences from the `dsh web` shell, noted in the harness header: `workspaceContext: false` (recorded fixtures must not embed this repo's AGENTS.md) and `sessionTitleLlm` left at bootHost's disabled default (its fire-and-forget title call would share the session's replay cursor nondeterministically). + +The `llm: false` seam is the reviewed resolution of the keyless-boot question: `'deepseek' | false` on `BootHostOptions`, matching the `workspaceContext: Config | false` shape, with `RunningHost.ctx` JSDoc naming "filling a deliberately-open capability seam" as its third sanctioned use. Replay runs in providers-catalog mode with a published `contextWindow` (the TUI `PROVIDERS` shape), never catch-all: with no adapter registered, catch-all would leave `resolveModelContext` unroutable and `compact-basic`'s post-step pressure check would warn every step instead of being provably inert. + +`seedSession()` seeds cold sessions through the real persistence API — a throwaway `Context` mounting `SessionStore` + `SessionPersistenceJsonl` against the host's root, `create()` + `append()`, one `utimes` backdate for deterministic sidebar order (the `semantic-checkpoint.snapshot.ts` precedent) — never raw file writes, so the seeder knows nothing of bucket hashing, filename encoding, or compression, and the host's zstd default needs no boot knob. Seeds are validated at seed time (parseable, ending in `turn/end` — an open final turn would be mutated by resume's crash repair). + +### Determinism rules + +The barrier stack for a prompted turn, in order: (1) host-side `await agent.whenIdle()` under a timeout, keyed off the in-process `turn/end` — the idle flip follows the persistence flush, so one await covers turn completion and durability; (2) browser settled poll (streaming detached, final text visible); (3) any log harvest after `host.dispose()`. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync); file polling is banned (slow on NFS, superseded by `whenIdle`); `networkidle` is banned outright (never resolves while an SSE stream is open). + +No single-shot transient-DOM assertions: every hop from replay yield to React commit can coalesce chunks, so sampling `[data-streaming]` is a race by construction. Streaming incrementality is asserted from the persisted `assistant/chunk` events (model-visible ⟺ logged makes the log the authoritative proof). `dsh-llm-replay`'s opt-in `paceMs` (default absent = burst) is a realism knob so the browser observes genuinely incremental SSE; correctness never leans on it, and abort during a pace wait cancels promptly. + +Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Harness `close()` calls the `ReplayHandle.assertConsumed()` teardown check (every recorded script bound, every cursor drained), converting silent underruns and shifted bindings into crisp diagnostics. No vitest retry on the lane; one chromium per file, fresh context per scenario, one host per scenario; viewport pinned; selectors anchor on roles, `data-*` attributes, and visible text. + +### Expected outputs + +One committed golden per scenario: a normalized `ariaSnapshot()` of the conversation region (`ui.expected.md`) — uuid/cwd/workspace-basename/duration tokens normalized, captured poll-until-equal at the settled milestone — plus a few role/text anchor assertions that stay green under a semantics-preserving component rewrite while the golden churns reviewably. The aria tree is the mechanization of the client rule "assert what the user would see, never class names". World-state assertions ride `host.ctx` session events inline (which tools ran, `turn/end` completed) instead of a second committed log golden: the persisted-log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence, and re-pinning it here would double refresh cost against the tier discipline. `refresh` is the sole golden writer — a missing golden in replay mode fails with the healing command rather than self-bootstrapping. + +The typecheck plane split is structural: `apps/web/tests/{harness,support,replay-round-trip.e2e,seeded-history.e2e}.ts` are host-plane programs (they boot the host spine), so they are excluded from the client-registered `apps/web` project and included file-by-file in `tsconfig.host.json` — one program cannot hold both sides of the cordis `Context` merges. + +### Modes and fixtures + +`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless) as inline spec branches — the TUI shape, not a suite factory: at two scenarios the acp-snapshot factory machinery has no owner, and the genuinely shared parts are already exported (`scrubRequestHeaders`, `parseSessionLog`, `installLlmReplay`). Each spec splits into drive steps (type, send, `whenTurnSettled` — run in all modes, never waiting on model-content selectors, so record cannot hang on a live model answering differently) and assertion steps (replay/refresh only). Record = drive live through the real composer + harvest the in-memory `session.header`/`session.events` (the TUI `rawSessionLog` shape — no file decompression) + `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}` tokenization; a follow-up keyless refresh regenerates `ui.expected.md`. Both scenarios' fixtures were recorded against this assembly through this flow. A drift guard ties each spec's drive prompt to the fixture's recorded `user/message`. A fixture-inventory guard holds each scenario directory closed (exact file set, every JSONL a scrub fixed-point). Web fixtures scrub headers everywhere and pin no header class, following the TUI precedent over the strict [pinned-header](2026-07-06-pin-request-header-content-in-one-scenario.md) reading — see Deferred. + +### Scenarios + +1. **`replay-round-trip`** — new session, prompt through the real composer, replay streams reasoning + a `bash` tool call that really executes in the temp workspace + final text (paced 15ms). Asserts settled markdown, the aria golden, and inline world state (bash `tool/call`, completed `turn/end`, >10 chunk events). +2. **`seeded-history`** — a recorded session seeded cold; the sidebar lists it (group row → session row, collapsed by default), opening renders tool cards and text purely from the log through the implicit cold-resume attach inside `session.history` — zero model calls in replay, so no binding constraints; record mode drives the same turn live (real `read` tool against seeded workspace files) to produce the seed. + +### CI stance + +The lane ships gate-exempt inside `pnpm run test:web`, exactly as that config's header records. Adding chromium to CI would reverse the "no browser infrastructure in CI" premise in the [GUI testing note](../process/2026-07-20-gui-testing-system.md) and therefore requires its own Agent Note cross-linked from there, staged as a non-required job first with measured promotion criteria (consecutive green runs, wall time, zero-retry flake budget, runner browser-cache strategy). `TODO(ci-browser)` marks the seam. Scenarios are POSIX-oriented (the lane is not in the Windows matrix). + +## Prior art + +Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot + AI SDK, lobe-chat, open-webui, OpenHands, Chainlit, continue, cline, langfuse, gradio/streamlit; Playwright HAR/route, MSW, Polly/nock, WireMock, aimock). The dominant proven architecture for apps that own their backend is an in-process fake/replay model behind the real backend seam with everything downstream real (LibreChat's `LIBRECHAT_TEST_RUN_HOOK` fake model; ai-chatbot's `MockLanguageModelV3` + `simulateReadableStream`; continue's scripted mock provider classes) — which is what `dsh-llm-replay` already is. Browser-level SSE interception cannot exercise incremental rendering (`route.fulfill` delivers the whole body at once; playwright#33564) and leaves the server SSE stack untested, so projects use it only for edge cases. Chunk pacing as a fixture parameter recurs everywhere (LibreChat 10ms default with slow profiles; ai-chatbot 500ms); real models in CI rot (open-webui's suite grew 120-second timeouts, was disabled, then deleted); sessions are seeded at the persistence layer with controlled timestamps (LibreChat inserts backdated Mongo documents; langfuse seeds its DB). No surveyed project replays a recorded agent-event log through the real backend for UI tests — the closest are provider-level recorded fixtures (aimock) and frontend-level socket history emission (OpenHands MSW) — so the session-log-as-fixture design goes one step beyond prior art along the axis this repo's model-visible ⟺ logged invariant makes natural. + +## Alternatives considered + +**Browser-network SSE interception (`page.route`).** Rejected: `route.fulfill` cannot stream, so incremental token rendering is unexercisable and the server-side SSE/backpressure/close path — where both confirmed P0s hid — goes untested. + +**Mock HTTP provider at `DEEPSEEK_BASE_URL`.** Rejected as the lane's mechanism (kept for the one existing workspace-probe smoke): fixtures become hand-authored OpenAI SSE byte scripts, a second fixture format that drifts from the session-log format the rest of the repo records and replays; the adapter's real HTTP path is with-key e2e's job. + +**Growing the `?fixture` client.** Rejected: tier separation — `FixtureApiClient` exists to test the client shell without a server; everything below the client API seam stays untested by construction. + +**Placeholder `DEEPSEEK_API_KEY` + replay interception instead of the `llm: false` seam.** Rejected despite zero product change and two in-tree precedents: it satisfies `llm-deepseek`'s fail-loud key check with a lie and leaves a dead adapter mounted-but-intercepted; the seam matches an existing option shape and fails loud at the earliest resolvable point. + +**A `packages/support/web-snapshot` package with a `defineWebSnapshotSuite` factory.** Rejected: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and at two scenarios a factory generalizes from one consumer while the genuinely shared logic is already exported from gated packages. Re-entry trigger: a second web-shaped consumer or ≥6 scenarios with demonstrably drifting inline branches; the package boundary would then be drawn browser-free. + +**A committed normalized-session-log golden as a second expected surface.** Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers. Inline world-state assertions on `host.ctx` events keep the world-verification duty. + +**Spawning the `dsh web` bin with a `DSH_SNAPSHOT` replay branch.** Rejected: it needs a test-mode branch plus env plumbing in the product bin where the in-process route uses exported production functions; the bin's thin glue is covered by the keyless CLI smokes. Becomes free only if the web host is ever Loader-ized — declined in review, with the app-assembly ruling reaffirmed. + +**Changing the wire protocol for testability.** Rejected: the contract already has a first-class keyless isomorphic seam (`InProcessApiClient(toFetchHandler(api))`), the per-event unbatched SSE is exactly what makes replay observable in a browser, and testing a wire we no longer ship would invert the tier's purpose. + +**Real-model browser tests as the keyless lane.** Rejected: nondeterministic by construction; the surveyed cautionary case (open-webui) grew unbounded timeouts and was deleted. The with-key W5 smoke stays as the live-model complement. + +**A client `data-dsh-busy` settled signal.** Deferred: the multi-condition settled polls proved sufficient at two scenarios and the host-side `whenIdle` barrier does the heavy lifting. Re-entry trigger: the first settled-poll flake, or a scenario needing a state the DOM does not expose. + +## Testing + +The lane itself: `pnpm run test:web` runs both scenarios keylessly alongside the existing smoke pair; `DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/` re-records a scenario's fixture against the live model; `DSH_SNAPSHOT=refresh` rewrites both aria goldens keylessly. The `llm: false` seam is pinned by `packages/host/runtime/tests/host-runtime.spec.ts` (keyless boot, NO_ADAPTER at first stream, embedder fill through ctx); `paceMs` validation, pacing floor, abort-during-pace, and both `assertConsumed` failure shapes are pinned in `packages/support/llm-replay/tests/llm-replay.spec.ts`. + +## Deferred + +- **Web header-class pin**: web fixtures tokenize `{{system}}`/`{{tools}}` everywhere and no scenario pins bootHost's composed prompt/tool schemas (`TODO(web-header-pin)` — the harness `recordFixture` JSDoc marks it). Following the TUI scrub-everywhere precedent; revisit when the web assembly's header diverges from the repl composition it mirrors. +- **CI browser provisioning**: reversal of the no-browser-in-CI ruling, staged criteria above (`TODO(ci-browser)`). +- **Follow-up-prompt-after-resume scenario**: the history/live stitch path over the real wire; add as its own scenario when that code changes or regresses. + +## Consequences + +The web surface gains its record-once/replay-forever tier: the real chromium → SSE → apiproxy → loop → tools → persistence chain runs keylessly in ~10-30s, deterministic across repeat runs, with fixtures owned and re-recordable by the lane itself. Costs accepted: every intentional conversation-UI change ends with a keyless `DSH_SNAPSHOT=refresh` (golden churn is reviewed diff, anchors keep semantic green); the aria format is Playwright-owned — the one committed snapshot format the repo does not control — so playwright version bumps must be deliberate bump-and-refresh commits (the dependency floats `^1.49.0` in `apps/web/package.json`; pin exactly if churn bites); replay's first-call-order binding constrains scenarios to one prompting session each, with the consumption assertion as the tripwire; `compact-basic` shares the session's replay cursor and stays inert only under the published 128k catalog window; and the lane guards regressions only where it runs (locally, `test:web`) until the CI reversal is separately decided. diff --git a/.agents/notes/proposed/testing/2026-07-24-web-gui-browser-e2e-lane.md b/.agents/notes/proposed/testing/2026-07-24-web-gui-browser-e2e-lane.md deleted file mode 100644 index cc7504dae7..0000000000 --- a/.agents/notes/proposed/testing/2026-07-24-web-gui-browser-e2e-lane.md +++ /dev/null @@ -1,92 +0,0 @@ -# Agent Note: Keyless browser e2e lane for the web GUI - -Status: proposed - -## Problem - -The web GUI ships as a real assembled chain — chromium page → nine client plugin bundles → HTTP unary RPC + two SSE streams → `toFetchHandler`/apiproxy → `bootHost`'s agent loop, tools, and JSONL persistence — and no test exercises that chain keylessly and deterministically. The [GUI testing system](../../implemented/process/2026-07-20-gui-testing-system.md) covers tier 1 (wire isomorphism in node), tier 2 (object-layer state machines), and a tier-3 smoke pair, but the keyless smoke (`apps/web/tests/smoke-fixture.e2e.ts`) drives `FixtureApiClient` behind `?fixture` — no host, no wire, no agent loop — while the full-chain smoke (`smoke-real.e2e.ts`) needs `DEEPSEEK_API_KEY` and a live model, so it is nondeterministic and self-skips in keyless CI. The snapshot philosophy of [docs/testing.md](../../../../docs/testing.md) — record once with a key, replay forever keyless, refresh on format churn — already covers the ACP, headless `stream-json`, and TUI transcript surfaces; the web surface is the one assembled product shape without it. The gap is exactly where the two confirmed GUI P0s hid: the wire carriage chain the fixture client short-circuits. - -## Proposal - -Add a keyless, deterministic browser e2e lane under `apps/web/tests/`, driven by recorded session-log fixtures replayed through `@deepseek-ai/dsh-llm-replay`, asserting the rendered accessibility tree plus in-process world state. No new package; no product-code change except (open question 1) an LLM composition knob. - -### Harness: `apps/web/tests/harness.ts` - -A plain shared-fixture module (the [testing-policy sanctioned shape](../../../../docs/testing.md)), not a package: the gate-worthy logic this lane needs — replay derivation, session parsing, log scrubbing, persistence — already lives in gated packages (`dsh-llm-replay`, `dsh-acp-snapshot`, `dsh-session-persistence-jsonl`); what remains is boot wiring and browser glue, and chromium-driving code cannot hold per-file 100% coverage on the browserless coverage runners. - -`launchWebHarness()` boots the real web assembly in-process from the exported production functions — `startHost({ boot: { persistenceRoot: , workspaceContext: false, cwd: } })`, `installLlmReplay(host.ctx, { file, childFiles, providers })`, `mountWebPlugins(host.ctx)`, `createHostWebPluginRegistry`, `startWebServer({ port: 0, distIndex, apiHandler: host.handler, webPlugins })` — and returns `{ baseUrl, host, workspaceCwd, close }`. This is the web analog of the TUI suite mounting the production bundle in-process ([TUI snapshots](../../implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)): the real entry boundary (`dsh web` bin arg-parsing and dist resolution) stays held by the existing keyless CLI smokes in `smoke-real.e2e.ts`, and the web surface has no `cordis.yml` to bypass — assembly is written in the app per the [GUI layering decision](../../implemented/architecture/2026-07-19-gui-layering-and-rpc-protocol.md). Replay runs in providers-catalog mode with a `contextWindow` (the TUI suite's `PROVIDERS` shape), never catch-all: with no adapter registered, catch-all mode would make `compact-basic`'s `resolveModelContext` throw into its post-step catch every step, spamming warnings and silently disabling the pressure path instead of proving it inert. - -`seedSession(host, fixtureText)` seeds cold sessions through the real persistence API — a throwaway `Context` mounting `SessionStore` + `SessionPersistenceJsonl` against the host's root, `create()` + `append()`, one `utimes` for deterministic sidebar order (the precedent is `examples/acp-agent/tests/semantic-checkpoint.snapshot.ts`) — never raw file writes, so the seeder needs no knowledge of bucket hashing, filename encoding, or compression. Seeds are validated at seed time (parseable, `seq`-contiguous, ending in `turn/end`) so fixture drift fails loud at the earliest resolvable point rather than as silently dropped frames in the client; a seed not ending in `turn/end` would be mutated by resume's crash repair. - -### Determinism rules - -The barrier stack, in order, for a prompted turn: (1) host-side `await agent.whenIdle()` under a timeout — the idle flip happens after both the `turn/end` append and the persistence flush, so one await covers turn completion and durability; (2) browser settled poll — streaming node detached, composer restored, final text visible; (3) log harvest only after `host.dispose()`. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync), and file polling is banned (slow on NFS, superseded by `whenIdle`). For history-open scenarios the barrier is a poll for the last expected message's text, then a poll-until-equal aria capture. `networkidle` is banned outright — it never resolves while an SSE stream is open. - -No single-shot transient-DOM assertions: every hop from replay yield to React commit can coalesce chunks, so sampling `[data-streaming]` is a race by construction. Streaming incrementality is asserted from the persisted `assistant/chunk` events (model-visible ⟺ logged makes the log the authoritative proof), optionally corroborated by an in-page MutationObserver latch armed before send — observers cannot miss a commit; polls can. `dsh-llm-replay` gains an opt-in `paceMs` config field (default absent = today's instant yield) as a realism knob so the browser observes genuinely incremental SSE; correctness never leans on the pace. - -Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Harness `close()` asserts every replay script was fully consumed (all scripts bound, every cursor at end), converting silent underruns and shifted bindings into crisp diagnostics; this is a small additive stats handle on `installLlmReplay`. No vitest retry on the lane — a retried-green race is a flake deferred, and only the chromium launch itself may retry inside the harness, logged. One chromium per file, fresh browser context per scenario, one host per scenario; viewport pinned; selectors anchor on roles, `data-*` attributes, and visible text only. - -### Expected outputs - -One committed golden per scenario: a normalized `ariaSnapshot()` of the conversation region only (`ui.expected.md`) — uuid/cwd/duration tokens normalized, sidebar and other time-bearing chrome structurally excluded, captured poll-until-equal at the settled milestone. The accessibility tree is the mechanization of the client rule "assert what the user would see, never class names": it survives CSS-module hash churn, styling rewrites, and DOM restructuring, and a wholesale component rewrite refreshes it keylessly. Alongside the golden, three or four targeted role/text anchor assertions (heading level, `pre code` content, tool-row accessible name) keep green anchors under a semantics-preserving rewrite so a churned golden diff is reviewable against surviving anchors. World-state assertions ride `host.ctx` session events inline (which tools ran, `turn/end` completed, no error) instead of a second committed log golden: the persisted-log surface is already pinned by the ACP/headless/TUI suites through the same loop and persistence plugins, and re-pinning it here would double refresh cost for no new regression class. `playwright` gets pinned exactly in `apps/web/package.json` — the aria format is Playwright-owned, the one snapshot format in this repo we do not own, so version bumps must be deliberate bump-and-refresh commits. - -### Modes and fixtures - -`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless), as inline branches in the specs — the TUI suite's shape, not a suite-factory: at two scenarios the acp-snapshot factory machinery (scenario tables, pinning classes, Windows sidecars, packed-row stabilization) has no owner, and the genuinely shared parts are already exported (`scrubRequestHeaders`, `normalizeSessionLog`, `parseSessionLog`, `installLlmReplay`). Each scenario script splits into drive steps (type, send, `whenIdle`-generic waits — run in all modes, never waiting on model-content selectors) and interaction/assertion steps (expand reasoning, aria capture — replay/refresh only), so record mode cannot hang on a live model answering with a different tool count. Record = drive + harvest the in-memory `session.header` + `session.events` (the TUI `rawSessionLog` shape — no file decompression, so `bootHost` needs no compression knob) + scrub via `scrubRequestHeaders` + a mandatory keyless refresh to regenerate `ui.expected.md`. Web fixtures scrub headers everywhere and pin nowhere, matching the TUI precedent; whether the web surface must instead own a header-class pin per the [pinned-header discipline](../../implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md) is open question 3. Seeds are recorded fixtures under the same inventory and refresh discipline as replay fixtures, never hand-authored one-offs, so `DSH_SNAPSHOT=refresh` heals every committed surface after intentional shape churn and only `assistant/chunk`-shape churn escalates to re-record. A TUI-style `afterAll` fixture guard holds the inventory closed (expected files present, every fixture scrub-fixed-point, no orphan directories). - -### Demo scenarios - -1. **`fresh-round-trip`** — new session, prompt, replay streams reasoning + markdown + a `bash` tool call that really executes (`echo` in the temp workspace) + final text. Asserts settled markdown semantics (heading, code block), the tool card row, composer restore, the aria golden, and inline world state (bash `tool/call` + completed `turn/end` in the session events). The keyless version of the with-key W5 flows. -2. **`seeded-history`** — a recorded session seeded cold; the sidebar lists it, opening it renders tool cards and collapsible reasoning purely from the log. This exercises the implicit cold-resume attach (`session.history` resumes via `agentFor`), cold summaries, history pagination views, and the client fold of historical events — the surface nothing else covers — with zero model calls, so no replay-binding constraints at all. A follow-up-prompt-after-resume scenario is deliberately deferred until the history/live stitch path changes or regresses. - -### Lane wiring and CI stance - -The lane rides `vitest.web.config.ts` (`pnpm run test:web`, serial), which stays gate-exempt exactly as its header comment records. Adding chromium to CI would reverse the "no browser infrastructure in CI" premise recorded in the [GUI testing system note](../../implemented/process/2026-07-20-gui-testing-system.md) and therefore requires its own Agent Note cross-linked from that note, staged as: non-required CI job first, promotion criteria measured (consecutive green runs, wall time, zero-retry flake budget, browser cache strategy on the enterprise runners, whether the runner images carry the chromium system libraries). Deferred out of this proposal; a `TODO(ci-browser)` marks the seam. Scenarios are `posixOnly` initially. Docs updated in the same implementation PR: the [testing policy](../../../../docs/testing.md) names `apps/web/tests/snapshots/` as the web surface's snapshot home with its divergent `DSH_SNAPSHOT=… pnpm run test:web` commands, the GUI testing note's tier map gains the lane (and drops its stale references to the deleted `missions/scripts/verify-*` files), `packages/client/AGENTS.md`'s check ladder mentions it, and the `dsh-acp-snapshot` README's "ACP-specific by design" sentence is corrected — this lane is the third consumer of its normalizers. - -### Open questions - -1. **LLM seam.** Two viable shapes. (a) `BootHostOptions.llm?: 'deepseek' | false` — an assembly toggle in the one module that owns assembly, matching the existing `workspaceContext: Config | false` shape and the reserved-knob sentence in `start.ts`; `llm: false` mounts no adapter, the harness fills the open seam on `host.ctx`, misuse fails loud at the first stream with `NO_ADAPTER`, and keyless boots stop needing any key. (b) Zero product change — a placeholder `DEEPSEEK_API_KEY` env var satisfies `llm-deepseek`'s load-time presence check (twice-precedented in-tree) and replay intercepts ahead of the mounted adapter. (a) is cleaner semantics and honest keylessness at the cost of a test-motivated product field; (b) is free but satisfies a fail-loud check with a lie and leaves a dead adapter mounted. Recommendation: (a), shaped minimal. -2. **Loader-izing `dsh web`.** Making the web host `cordis.yml`-driven like every example would give the ACP-style `cordis.snapshot.yml` replay overlay for free and align with "everything is a plugin", but it reverses the settled "assembly is written in the app" ruling and is a product-architecture decision on its own merits — its own proposal if wanted; this lane does not need it. -3. **Header-class pin.** Strict reading of the pinned-header discipline wants one web scenario pinning `bootHost`'s composed prompt + tool schemas (a header class no ACP scenario covers); the TUI precedent scrubs everywhere and pins nowhere. Cheap middle: pin sidecars on `fresh-round-trip` at record time. Recommendation: follow TUI now (scrub-only), revisit when the web assembly's header diverges further from the repl composition it mirrors. -4. **Golden breadth.** Full conversation-region aria golden (adopted above) versus targeted assertions only. The golden is the "assembled transcript" duty for user-visible changes; the cost is a keyless refresh on every component rewrite. Recommendation: keep the golden + anchors. -5. **Client settled signal.** A `data-dsh-busy` attribute derived from the object layer's pending-RPC/active-stream state would replace multi-condition settled polls with one selector. Presentation-plane observability, no session-log leak — but the current polls suffice for two scenarios. Recommendation: defer until a settled-poll flake actually appears. - -## Prior art - -Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot + AI SDK, lobe-chat, open-webui, OpenHands, Chainlit, continue, cline, langfuse, gradio/streamlit; Playwright HAR/route, MSW, Polly/nock, WireMock, aimock). The dominant proven architecture for apps that own their backend is an in-process fake/replay model behind the real backend seam with everything downstream real (LibreChat's `LIBRECHAT_TEST_RUN_HOOK` fake model; ai-chatbot's `MockLanguageModelV3` + `simulateReadableStream`; continue's scripted mock provider classes) — which is what `dsh-llm-replay` already is. Browser-level SSE interception cannot exercise incremental rendering (`route.fulfill` delivers the whole body at once; playwright#33564) and leaves the server SSE stack untested, so projects use it only for edge cases. Chunk pacing as a fixture parameter recurs everywhere (LibreChat 10ms default with slow profiles; ai-chatbot 500ms); real models in CI rot (open-webui's suite grew 120-second timeouts, was disabled, then deleted); sessions are seeded at the persistence layer with controlled timestamps (LibreChat inserts backdated Mongo documents; langfuse seeds its DB). No surveyed project replays a recorded agent-event log through the real backend for UI tests — the closest are provider-level recorded fixtures (aimock) and frontend-level socket history emission (OpenHands MSW) — so the session-log-as-fixture design goes one step beyond prior art along the axis this repo's model-visible ⟺ logged invariant makes natural. - -## Alternatives considered - -**Browser-network SSE interception (`page.route`).** Rejected: `route.fulfill` cannot stream, so incremental token rendering is unexercisable and the server-side SSE/backpressure/close path — where both confirmed P0s hid — goes untested. - -**Mock HTTP provider at `DEEPSEEK_BASE_URL`.** Rejected as the lane's mechanism (kept for the one existing workspace-probe smoke): fixtures become hand-authored OpenAI SSE byte scripts, a second fixture format that drifts from the session-log format the rest of the repo records and replays; the adapter's real HTTP path is with-key e2e's job. - -**Growing the `?fixture` client.** Rejected: tier separation — `FixtureApiClient` exists to test the client shell without a server; everything below the client API seam stays untested by construction. - -**A `packages/support/web-snapshot` package with a `defineWebSnapshotSuite` factory.** Rejected for now: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and at two scenarios the factory generalizes from one consumer while the genuinely shared logic already lives exported in `dsh-llm-replay`/`dsh-acp-snapshot`. Re-entry trigger: a second web-shaped consumer or ≥6 scenarios with demonstrably drifting inline branches; the package boundary would then be drawn browser-free. - -**A committed normalized-session-log golden as a second expected surface.** Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers, against the tier discipline. Inline world-state assertions on `host.ctx` events keep the world-verification duty. - -**Spawning the `dsh web` bin with a `DSH_SNAPSHOT` replay branch.** Rejected for now: it needs a test-mode branch plus env plumbing in the product bin where the in-process route uses exported production functions with zero product change; the bin's thin glue is covered by the keyless CLI smokes. Becomes free if the web host is ever Loader-ized (open question 2). - -**Changing the wire protocol for testability.** Rejected: the contract already has a first-class keyless isomorphic seam (`InProcessApiClient(toFetchHandler(api))`), the per-event unbatched SSE is exactly what makes replay observable in a browser, and testing a wire we no longer ship would invert the tier's purpose. - -**Real-model browser tests as the keyless lane.** Rejected: nondeterministic by construction; the surveyed cautionary case (open-webui) grew unbounded timeouts and was deleted. The with-key W5 smoke stays as the live-model complement. - -## Acceptance criteria - -- `pnpm run test:web` keyless passes the two scenarios deterministically (no vitest retry), alongside the existing smoke pair, on a checkout with built client bundles and the frontend dist. -- Replay asserts: aria golden equality at the settled milestone, anchor role/text assertions, inline world-state event assertions, zero pageerrors, zero connection-loss/gap-repair console warnings, all replay scripts fully consumed at teardown. -- `DSH_SNAPSHOT=record` with a key re-records `fresh-round-trip` (drive steps only), rewrites its `session.jsonl` scrubbed, and a follow-up `DSH_SNAPSHOT=refresh` regenerates `ui.expected.md` keylessly; `refresh` alone heals goldens after intentional non-chunk shape churn. -- The seeded scenario renders history through the real cold-resume path with zero model calls and leaves the seed fixture byte-identical (closedness validated at seed time). -- Fixture guard holds the snapshot inventory closed; failure produces a bundle under `.artifacts/` (screenshot, console, pageerrors, persistence copy, actual-vs-expected aria). -- Docs land in the same PR: testing.md web-lane entry, GUI testing note tier map + stale verify-script cleanup, client AGENTS.md ladder, acp-snapshot README correction, this note moved to `implemented/` rewritten in present tense. - -## Risks - -- **The aria format is Playwright-owned** — the one committed snapshot format the repo does not control; a version bump can churn every golden. Mitigated by an exact version pin in `apps/web` and a documented bump-and-refresh procedure; residual risk accepted. -- **Replay's first-call-order binding** stays fragile under concurrent browser-driven sessions; the lane constrains scenarios to one prompting session each (the seeded scenario prompts none), and the teardown consumption assertion turns violations into diagnostics rather than surreal transcripts. -- **`compact-basic` shares the session's replay cursor** — a pressure-triggered summarize would consume a script entry; inert for small fixtures under the 128k catalog window, and the consumption assertion catches it if a fixture ever grows past the threshold. -- **CI remains browserless for now**, so the lane guards regressions only where it is run (locally and in any future non-required job) until the CI reversal is separately decided; the runner images' chromium-library situation is unverified. -- **Record-mode nondeterminism** is contained but not eliminated by the drive/assert split: a live model may still produce a transcript whose replay violates a scenario's assertions, requiring prompt tuning at record time (bounded by terse prompts and a chunk-count warning in record mode). -- **jsdom-lane overlap**: component-level rendering is already covered per-plugin; this lane must stay at assembled-transcript altitude (whole-region golden + anchors) or it starts re-testing tier 2 and paying double maintenance. diff --git a/apps/web/tests/harness.ts b/apps/web/tests/harness.ts index 6f885da0c8..71c3ac9344 100644 --- a/apps/web/tests/harness.ts +++ b/apps/web/tests/harness.ts @@ -243,11 +243,11 @@ export function rawSessionLog(session: Session): string { /** * Record-mode fixture write-back: harvest the live session, scrub request - * headers to {{system}}/{{tools}} (the web lane pins no header class — a - * deliberate deviation logged in the Agent Note's deferred work), tokenize - * the run-local session id and cwd ({{sessionId}}/{{cwd}}, the committed ACP - * fixture convention — re-records then diff only on real content), and write - * the committed fixture. + * headers to {{system}}/{{tools}} (TODO(web-header-pin): the web lane pins no + * header class — a deliberate deviation logged in the Agent Note's deferred + * work), tokenize the run-local session id and cwd ({{sessionId}}/{{cwd}}, + * the committed ACP fixture convention — re-records then diff only on real + * content), and write the committed fixture. * @param harness - the record-mode harness. * @param sessionId - the driven session. * @param fixturePath - the committed session.jsonl / seed.jsonl target. diff --git a/docs/testing.md b/docs/testing.md index 85798cee3e..5841cd5f29 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -8,6 +8,7 @@ How this repo tests, tier by tier, and the rules that keep a green suite meaning - **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped. - **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e Agent Note](../.agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.md)). - **Snapshot** (`pnpm run test:snapshot`): transport-specific keyless expected outputs cover external presentation. ACP suites boot the real example subprocess, replay a recorded session, and diff normalized JSON-RPC plus the re-persisted log ([ACP snapshot Agent Note](../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md)); the headless suite independently pins `stream-json` through its real one-shot subprocess. TUI completed journeys replay recorded primary/child JSONL through the real agent loop and tools before projecting ANSI into semantic terminal-state expected outputs; package-local snapshots retain transient renderer states, and a real PTY conversation covers the process boundary ([TUI snapshot Agent Note](../.agents/notes/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)). Use `pnpm run test:snapshot:record` when a model transcript must change and `pnpm run test:snapshot:refresh` when committed replay input remains correct; review every JSONL and expected-output diff. System-prompt/tool-schema content is pinned by one ACP scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header Agent Note](../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)). +- **Web browser snapshot** (inside `pnpm run test:web`, gate-exempt like the rest of that lane): the web GUI's keyless assembled-transcript tier — a real chromium over the real in-process web assembly (`llm: false` + `dsh-llm-replay`), scenario fixtures and normalized conversation aria goldens under `apps/web/tests/snapshots/`. Its record/refresh commands diverge from `test:snapshot` (`DSH_SNAPSHOT=record pnpm run test:web` re-records against the live model; `DSH_SNAPSHOT=refresh` rewrites goldens keylessly); the [web e2e lane Agent Note](../.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md) owns the design, and CI browser provisioning is deferred there. ## The with-key policy: inference is cheap here @@ -41,4 +42,4 @@ An e2e assertion re-runs the command or re-reads the file externally; a keyword ## When a snapshot test is required -Every non-trivial model- or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP surfaces use `examples//tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the `stream-json` snapshot and replay fixtures. Completed interactive-terminal journeys use JSONL-driven scenarios under `examples/tui-agent/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. New capability seams, lifecycle shapes, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation. +Every non-trivial model- or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP surfaces use `examples//tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the `stream-json` snapshot and replay fixtures. Completed interactive-terminal journeys use JSONL-driven scenarios under `examples/tui-agent/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. Browser-rendered web GUI journeys use `apps/web/tests/snapshots/` through the web e2e lane's harness. New capability seams, lifecycle shapes, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation. diff --git a/packages/client/AGENTS.md b/packages/client/AGENTS.md index 5bde15dc2c..3c96ac1e85 100644 --- a/packages/client/AGENTS.md +++ b/packages/client/AGENTS.md @@ -65,7 +65,7 @@ The GUI test structure (three tiers, lane map) is settled in the [GUI testing sy Run the narrowest rung that covers what you touched; escalate only when the change surface demands it. 1. **Every GUI code change** — `pnpm run test:gui` (seconds; no browser, no server): the client suites plus the host-side GUI packages. This is the inner loop; run it as freely as a typecheck. -2. **Changes to the build surface, boot wiring, or static serving** (`apps/web`, vite config, `dsh-host-webserver`) — additionally `pnpm run test:web`: rebuilds the frontend dist, then runs the browser smoke pair (the real-host case self-skips without `DEEPSEEK_API_KEY`). +2. **Changes to the build surface, boot wiring, static serving, or the wire carriage** (`apps/web`, vite config, `dsh-host-webserver`, connection/handler/SSE) — additionally `pnpm run test:web`: rebuilds the frontend dist, then runs the browser smoke pair (the real-host case self-skips without `DEEPSEEK_API_KEY`) plus the keyless replayed e2e scenarios (`DSH_SNAPSHOT=refresh` rewrites their aria goldens after an intentional conversation-UI change; `DSH_SNAPSHOT=record` re-records fixtures with a key). 3. **Before a PR** — `pnpm run check:pre-push` (the repo-wide gate ladder). Between PR windows this rung is not expected on every commit. If `test:gui` is red on code you did not touch, neither silently fix nor ignore it: note it in your handoff so it lands in the next PR window's sweep. diff --git a/packages/support/acp-snapshot/README.md b/packages/support/acp-snapshot/README.md index 05a4974c19..1243cd5026 100644 --- a/packages/support/acp-snapshot/README.md +++ b/packages/support/acp-snapshot/README.md @@ -42,7 +42,7 @@ Every scenario compares `stdout.expected.jsonl` with cwd-rooted separators canon The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config Agent Note](../../../.agents/notes/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log expected outputs, and each pin's prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md). -Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). Session config options are scriptable too: the `setConfigOption` step switches a knob over `session/set_config_option`, and `setConfigOptionExpectError` asserts the bridge rejects an unknown id or out-of-vocabulary value (the error frame stays in the transcript). +Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). The launcher and suite factory are ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection` — while the normalizers are transport-neutral session-log/text helpers also consumed by the TUI snapshot suite and the web browser e2e lane. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). Session config options are scriptable too: the `setConfigOption` step switches a knob over `session/set_config_option`, and `setConfigOptionExpectError` asserts the bridge rejects an unknown id or out-of-vocabulary value (the error frame stays in the transcript). ## Model Experience diff --git a/vitest.web.config.ts b/vitest.web.config.ts index de220c9f12..bcca93e5dd 100644 --- a/vitest.web.config.ts +++ b/vitest.web.config.ts @@ -1,10 +1,13 @@ import tsconfigPaths from 'vite-tsconfig-paths' import { defineConfig } from 'vitest/config' -// Web smoke lane (GUI, gate-exempt — not part of the CI sequence yet): built -// page + real chromium, so it lives outside the unit/e2e includes. The -// real-host test self-skips without DEEPSEEK_API_KEY; the fixture test is -// keyless and deterministic. +// Web browser lane (GUI, gate-exempt — not part of the CI sequence yet): +// built page + real chromium, so it lives outside the unit/e2e includes. The +// real-host smoke self-skips without DEEPSEEK_API_KEY; the fixture smoke and +// the replayed e2e scenarios are keyless and deterministic. +// TODO(ci-browser): running this lane in CI requires chromium provisioning +// and reverses the no-browser-in-CI ruling — staged criteria in +// .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md. try { // Node >= 21.7 native; throws when the file does not exist. process.loadEnvFile(new URL('.env', import.meta.url).pathname)