diff --git a/docs/testing.i18n.yaml b/docs/testing.i18n.yaml index 3d7569c620..a5160543fb 100644 --- a/docs/testing.i18n.yaml +++ b/docs/testing.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/testing.md -testing.md: d460d9ba76eab84b79e8c66454325d03d7c5908e -testing.zh.md: 821e49b6593b924e754d5ed0d588dd1b5b566051 +testing.md: 5e766bb684d9d76a1c9e854a2aef95796a31ec12 +testing.zh.md: 21b3f98e9ddbbb6f77621cff9ce206ae17125bc3 diff --git a/docs/testing.md b/docs/testing.md index d460d9ba76..5e766bb684 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -7,7 +7,7 @@ How this repo tests, tier by tier, and the rules that keep a green suite meaning ## Tiers - **Unit** (`pnpm run test`): vitest over package and example specs under their `tests/**` directories plus repository script specs under `scripts/**/*.spec.ts`; tests stay with the code area they exercise. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Prefer edge cases, error paths, event ordering, concurrency races, and permanent contract regressions (see `packages/core/agent-loop/tests/contract-regressions.spec.ts`). -- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped. +- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped. Per-file 100% on `packages/bash/pwsh-local/src` needs a real `pwsh`: without one its executor suites self-skip and `vitest.config.ts` exempts the file so pwsh-less hosts stay green, while CI runners ship pwsh and enforce the full bar. - **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e Agent Note](../.agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.md)). - **Snapshot** (`pnpm run test:snapshot`): keyless expected outputs cover external behavior — transport contracts and presentation, while persisted logs pin assembled backend behavior. ACP boots the real automation-server example, replays a recorded session, and diffs normalized JSON-RPC plus the re-persisted log ([ACP snapshot Agent Note](../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md)); headless pins `stream-json` through its real one-shot process. TUI journeys replay primary/child JSONL through the real loop and tools, then project ANSI into semantic terminal-state outputs; package snapshots retain transient states and a real PTY covers the process boundary ([TUI snapshot Agent Note](../.agents/notes/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)). Use `pnpm run test:snapshot:record` when a model transcript changes and `pnpm run test:snapshot:refresh` when replay input remains valid; review every JSONL and expected-output diff. One ACP scenario (`text-turn`) pins full system-prompt/tool-schema content; other fixtures tokenize it so an edit churns one line ([pinned-header Agent Note](../.agents/notes/archived/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)). - **Web browser snapshot** (`pnpm run test:web`; required Linux PR gate): Chromium compares replayed browser output with `apps/web/tests/snapshots/`. CI forces read-only `DSH_SNAPSHOT=replay`, never writing expected outputs; record/refresh stay local and every diff is reviewed ([web e2e lane](../.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md), [CI gate decision](../.agents/notes/implemented/testing/2026-07-30-web-browser-snapshot-ci-gate.md)). `test:web` [builds first](../.agents/notes/implemented/bug-fix/2026-07-28-themed-scrollbars-and-reserved-gutter.md) for plugin CSS. diff --git a/docs/testing.zh.md b/docs/testing.zh.md index 821e49b659..21b3f98e9d 100644 --- a/docs/testing.zh.md +++ b/docs/testing.zh.md @@ -7,7 +7,7 @@ ## 层级 - **单元测试**(`pnpm run test`):vitest 运行包(package)和示例各自的 `tests/**` 目录下的测试,以及匹配 `scripts/**/*.spec.ts` 的仓库脚本测试;测试文件与其所覆盖的代码区域放在一起。每个注册表都有一个 HMR(热模块替换)安全测试(dispose(资源释放)贡献的 fiber,断言清理完成)。优先覆盖边界情况、错误路径、事件顺序、并发竞态,以及永久性契约回归(见 `packages/core/agent-loop/tests/contract-regressions.spec.ts`)。 -- **覆盖率门禁**(`pnpm run test:coverage`):门禁级运行,对 `packages/*/*/src` 按文件 100% 覆盖。未覆盖的行往往是门禁正确标记出的死代码(应删除),而非需要补写的测试。行覆盖率是必要条件,但永远不是充分条件:它证明行被执行过,不证明功能按交付预期工作。 +- **覆盖率门禁**(`pnpm run test:coverage`):门禁级运行,对 `packages/*/*/src` 按文件 100% 覆盖。未覆盖的行往往是门禁正确标记出的死代码(应删除),而非需要补写的测试。行覆盖率是必要条件,但永远不是充分条件:它证明行被执行过,不证明功能按交付预期工作。`packages/bash/pwsh-local/src` 的按文件 100% 覆盖需要真实的 `pwsh`:缺少它时其 executor 套件会自动跳过,`vitest.config.ts` 会豁免该文件以使无 pwsh 的主机保持绿色,而 CI runner 自带 pwsh,仍按完整标准执行门禁。 - **真实 API e2e**(`pnpm run test:e2e`):带密钥测试调用真实提供方 API,包括 DeepSeek 模型以及各提供方特有的冒烟测试;这些测试各自由自己的密钥控制(`EXA_API_KEY`、`PERPLEXITY_API_KEY` 等),缺少密钥时套件会自动跳过,使 keyless CI 保持绿色([真实 API e2e Agent Note](../.agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.md))。 - **快照**(`pnpm run test:snapshot`):无密钥预期输出覆盖对外行为(传输契约与呈现),持久化日志则固定组装后的后端行为。ACP 启动真实的自动化服务器示例、回放录制会话,并对归一化 JSON-RPC 与重新持久化的日志执行 diff([ACP 快照 Agent Note](../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md));headless 通过真实单次运行进程固定 `stream-json`。TUI 旅程通过真实循环与工具回放主会话与子会话 JSONL,再将 ANSI 投影为语义化终端状态输出;包级快照保留瞬态状态,真实 PTY 覆盖进程边界([TUI 快照 Agent Note](../.agents/notes/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md))。当模型 transcript(文本记录)发生变化时使用 `pnpm run test:snapshot:record`,回放输入仍然有效时使用 `pnpm run test:snapshot:refresh`;请审查每一处 JSONL 与预期输出差异。一个 ACP 场景(`text-turn`)固定完整的系统提示词与工具 schema 内容;其他 fixture(测试前置数据)将其 token 化,因此修改只会扰动一行([pinned-header Agent Note](../.agents/notes/archived/testing/2026-07-06-pin-request-header-content-in-one-scenario.md))。 - **Web 浏览器快照**(`pnpm run test:web`;必需的 Linux PR(Pull Request)门禁):Chromium 将回放后的浏览器输出与 `apps/web/tests/snapshots/` 比较。CI 强制只读的 `DSH_SNAPSHOT=replay`,绝不写入预期输出;record/refresh 留在本地,每处 diff 都须评审([web e2e 车道](../.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md)、[CI 门禁决策](../.agents/notes/implemented/testing/2026-07-30-web-browser-snapshot-ci-gate.md))。`test:web` 会[先构建](../.agents/notes/implemented/bug-fix/2026-07-28-themed-scrollbars-and-reserved-gutter.md)以交付插件 CSS。 diff --git a/examples/acp-agent/tests/snapshots/cancel-tool-calls/session.jsonl b/examples/acp-agent/tests/snapshots/cancel-tool-calls/session.jsonl index a05cd1c28f..d66020b594 100644 --- a/examples/acp-agent/tests/snapshots/cancel-tool-calls/session.jsonl +++ b/examples/acp-agent/tests/snapshots/cancel-tool-calls/session.jsonl @@ -16,7 +16,7 @@ {"type":"assistant/chunk","seq":14,"time":1785487611319,"data":{"turn":1,"step":1,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}} {"type":"assistant/message","seq":15,"time":1785487611319,"data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"call_wait","name":"bash","arguments":"{\"command\":\"node -e \\\"require('node:fs').writeFileSync('started.txt', 'started'); setInterval(() => {}, 1000)\\\"\",\"description\":\"Wait until cancellation\"}"},{"type":"tool-call","id":"call_skipped","name":"bash","arguments":"{\"command\":\"printf skipped > skipped.txt\",\"description\":\"Write skipped marker\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash"},"id":"bd630f41-b45d-4183-a785-1ff6e7049b62"},"usage":{"inputTokens":10,"outputTokens":10}},"sourceEventSeqs":[7,8,9,10,11,12,13,14],"surfaceOp":"append"} {"type":"tool/call","seq":16,"time":1785487611319,"data":{"turn":1,"step":1,"callId":"call_wait","name":"bash","arguments":"{\"command\":\"node -e \\\"require('node:fs').writeFileSync('started.txt', 'started'); setInterval(() => {}, 1000)\\\"\",\"description\":\"Wait until cancellation\"}"}} -{"type":"tool/result","seq":17,"time":1785487611378,"data":{"turn":1,"step":1,"message":{"source":{"kind":"tool","callId":"call_wait"},"content":[{"type":"tool-result","toolCallId":"call_wait","content":[{"type":"text","text":"Error: tool call aborted"}],"isError":true}],"role":"user","id":"252903b2-b4e1-4a33-81d8-d5befefcb27e","error":{"name":"AbortError","code":"ABORTED"}},"sourceEventSeqs":[16],"surfaceOp":"append"} +{"type":"tool/result","seq":17,"time":1785487611378,"data":{"turn":1,"step":1,"message":{"source":{"kind":"tool","callId":"call_wait"},"content":[{"type":"tool-result","toolCallId":"call_wait","content":[{"type":"text","text":"Error: tool call aborted"}],"isError":true}],"role":"user","id":"252903b2-b4e1-4a33-81d8-d5befefcb27e"},"error":{"name":"AbortError","code":"ABORTED"}},"sourceEventSeqs":[16],"surfaceOp":"append"} {"type":"tool/call","seq":18,"time":1785487611378,"data":{"turn":1,"step":1,"callId":"call_skipped","name":"bash","arguments":"{\"command\":\"printf skipped > skipped.txt\",\"description\":\"Write skipped marker\"}"}} {"type":"tool/result","seq":19,"time":1785487611378,"data":{"turn":1,"step":1,"message":{"source":{"kind":"tool","callId":"call_skipped"},"content":[{"type":"tool-result","toolCallId":"call_skipped","content":[{"type":"text","text":"Error: tool call aborted before dispatch"}],"isError":true}],"role":"user","id":"282c5c8c-7296-4545-8014-e6393b351436"},"error":{"name":"AbortError","code":"ABORTED_BEFORE_DISPATCH"}},"sourceEventSeqs":[18],"surfaceOp":"append"} {"type":"step/end","seq":20,"time":1785487611378,"data":{"turn":1,"step":1}} diff --git a/scripts/doc-budgets.manifest.json b/scripts/doc-budgets.manifest.json index 4387a894f1..9e8d6076ec 100644 --- a/scripts/doc-budgets.manifest.json +++ b/scripts/doc-budgets.manifest.json @@ -4,7 +4,7 @@ "docs/architecture.md": 2160, "docs/cordis-primer.md": 600, "docs/defensive-patterns.md": 550, - "docs/testing.md": 1120, + "docs/testing.md": 1150, "examples/AGENTS.md": 310, "packages/AGENTS.md": 675, "packages/README.md": 920 diff --git a/vitest.config.ts b/vitest.config.ts index 14d5be103f..458b1134b1 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -1,3 +1,4 @@ +import { spawnSync } from 'node:child_process' import tsconfigPaths from 'vite-tsconfig-paths' import { defineConfig } from 'vitest/config' import { vitestExecArgv } from './vitest.shared.ts' @@ -39,6 +40,18 @@ const windowsCoverageExclusions = process.platform === 'win32' ] : [] +// Mirrors windowsCoverageExclusions: pwsh-local's run/start/lifecycle suites +// self-skip without a real pwsh (executor.spec.ts hasPwsh), leaving this file +// far below per-file 100% on pwsh-less hosts; the exemption keeps those hosts +// green while CI runners ship pwsh and still enforce the full bar. The probe +// is deliberately PATH-only (narrower than the suites' resolvePwshPath): a +// win32 host where only install-location pwsh or 5.1 resolves forfeits the +// exemption while the suites still run, so the gate can only get stricter, +// never falsely green. +const pwshCoverageExclusions = spawnSync('pwsh', ['-NoLogo', '-NoProfile', '-NonInteractive', '-Command', '$true'], { encoding: 'utf8' }).status === 0 + ? [] + : ['packages/bash/pwsh-local/src/index.ts'] + const testIncludes = [ 'packages/*/*/tests/**/*.spec.{ts,tsx}', 'apps/*/tests/**/*.spec.ts', @@ -203,6 +216,7 @@ export default defineConfig({ 'packages/ui/tui/src/index.ts', ...windowsUnsupportedPackages.map(path => `${path}/src/**/*.ts`), ...windowsCoverageExclusions, + ...pwshCoverageExclusions, ], // 100% or it doesn't merge (docs/testing.md: excessive tests are welcome). // Per-file so a well-covered big file can't subsidize a bare one.