Turtle
f290a8b851
`dsh` shipped two config trees that were 43 rows the same: apps/cli/cordis.yml composed web as 74 flat rows, while the TUI booted examples/tui-agent/cordis.yml whose single `@deepseek-ai/dsh-tui-demo` row mounted twelve plugins behind a twenty-key pass-through Config. Neither file was what its location claimed — apps/cli hardcoded the "example" as the product default and the "demo" bundle was the application — and every capability change had to be made twice. - apps/cli/base.cordis.yml holds the 43 shared rows; tui.cordis.yml and web.cordis.yml are patch lists stating only what differs per surface - overlays apply as SIBLING patch lists at one include level, because include patches never cross an include boundary. Precedence: base < surface < (--config | personal ~/.dsh/config.yaml) < launcher flag/profile patches - `--config` now applies an overlay INSTEAD OF the personal one, so a demo or test tree never inherits the user's route; new `--config-replace` boots a file as the entire tree (the old `--config` behaviour). Both survive /resume - vendor/include: index each `insert`ed row as it is added so a later patch can configure or disable it. Upstream built the id index once before the patch loop, leaving every surface-only row — the whole TUI front door — silently unpatchable from user config. Logged as local modification 8 - session identity moves to dsh-agent-loop's CONFIGURED_AGENT_IDENTITIES_KEY; dsh-tui's MAIN_SESSION_ID_KEY is deleted (only the bundle read it) - delete examples/tui-agent, examples/cordis-agent, packages/examples/tui-demo; TUI tests → apps/cli/tests, cordis e2e → packages/cordis/tool-cordis/tests, examples/code-mode survives as an overlay leaf - `dsh web` gains --config, threaded into AppCLIEntry as an extra overlay Three latent defects surfaced and are fixed here: the TUI captured the optional sessionQuery service once at construction and could permanently disable /resume when it won the mount race; the session-store root silently reverted to a project-local ./.sessions; --config-replace was dropped by the resume handoff. Verified by booting each tree through the real Loader (TUI 55 entries, web 75, zero unsettled) rather than reading YAML. All eight terminal snapshots replay byte-identically; 14/14 PTY smoke, 112/112 snapshots, 25/25 doc-sync, hygiene and lint clean.
50 lines
8.7 KiB
Markdown
50 lines
8.7 KiB
Markdown
# Testing policy
|
|
|
|
English | [中文](testing.zh.md)
|
|
|
|
How this repo tests, tier by tier, and the rules that keep a green suite meaningful. Commands live in root [AGENTS.md](../AGENTS.md); linked Agent Notes carry the rationale.
|
|
|
|
## Tiers
|
|
|
|
- **Unit** (`pnpm run test`): vitest over package and example specs under their `tests/**` directories plus repository script specs under `scripts/**/*.spec.ts`; tests stay with the code area they exercise. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Prefer edge cases, error paths, event ordering, concurrency races, and permanent contract regressions (see `packages/core/agent-loop/tests/contract-regressions.spec.ts`).
|
|
- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped.
|
|
- **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e Agent Note](../.agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.md)).
|
|
- **Snapshot** (`pnpm run test:snapshot`): keyless expected outputs cover external behavior — transport contracts and presentation, while persisted logs pin assembled backend behavior. ACP boots the real automation-server example, replays a recorded session, and diffs normalized JSON-RPC plus the re-persisted log ([ACP snapshot Agent Note](../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md)); headless pins `stream-json` through its real one-shot process. TUI journeys replay primary/child JSONL through the real loop and tools, then project ANSI into semantic terminal-state outputs; package snapshots retain transient states and a real PTY covers the process boundary ([TUI snapshot Agent Note](../.agents/notes/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)). Use `pnpm run test:snapshot:record` when a model transcript changes and `pnpm run test:snapshot:refresh` when replay input remains valid; review every JSONL and expected-output diff. One ACP scenario (`text-turn`) pins full system-prompt/tool-schema content; other fixtures tokenize it so an edit churns one line ([pinned-header Agent Note](../.agents/notes/archived/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
|
- **Web browser snapshot** (gate-exempt `pnpm run test:web`): real chromium over the in-process web composition replays recorded fixtures against conversation aria goldens (`apps/web/tests/snapshots/`); record/refresh semantics and the deferred CI browser decision: [web e2e lane Agent Note](../.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md). [Runs `build` first](../.agents/notes/implemented/bug-fix/2026-07-28-themed-scrollbars-and-reserved-gutter.md): plugin CSS ships per plugin.
|
|
|
|
Committed session-format JSONL uses the canonical packed-row layout, and the keyless snapshot gate discovers every such fixture by its `session` header. In-flight branches carrying older fixture edits merge current `master` and run the [temporary migrator](../scripts/migrate-packed-session-fixtures.ts) through `pnpm run migrate:packed-session-fixtures`; the [removal proposal](../.agents/notes/proposed/process/2026-07-26-remove-packed-session-fixture-migrator.md) retires that command and these links after all affected branches converge.
|
|
|
|
## The with-key policy: inference is cheap here
|
|
|
|
We are DeepSeek — do not ration real-API tests. A no-key test proves plumbing; only a with-key run proves the agent works against a real model. Cover file-writing prompts, multi-turn conversations, tool use, and mid-stream cancellation. Highest-value are **smoke tests** that boot the real example, send one prompt, and check the world — they catch the "green unit tests, broken product" class that mocks cannot ([postmortem 0001](postmortem/0001-acp-default-export-drops-inject.md)). Self-skip keeps secretless CI and keyless contributors unblocked; it is not a cost signal. Every example ships keyless and with-key smokes ([examples/AGENTS.md](../examples/AGENTS.md)).
|
|
|
|
## Prefer the real implementation over a mock
|
|
|
|
Mock only the expensive or non-deterministic boundary (LLM adapter, network, clock); keep everything downstream real. A hand-rolled stand-in proves the bridge moves bytes, not that the shipping tool behaves as asserted. Bridge tool-call tests use the scripted mock model with the real tool and executor: `makeBridgeHarness({ withBash: true })` plugs in `dsh-bash-local` and `dsh-tool-bash`, then runs `echo`.
|
|
|
|
Recovery tests separate pre/post-chunk failures by step and prove failed chunks derive no message or tool side effect. Cover exhaustion, cancellation, policy composition, persistence, status, wire counts, transport-closing idle timeouts, and shipping Loader composition.
|
|
|
|
## Verify the world, not the self-report
|
|
|
|
An e2e assertion re-runs the command or re-reads the file externally; a keyword probe on the agent's own output lets a cheating agent pass. Assert untouched files are byte-identical. e2e tests own their resources: create the harness in the test, dispose in `afterEach` (even on failure/retry/timeout); shared fixtures live in a plain `tests/harness.ts`, never another `*.e2e.ts` (importing a spec re-registers its `describe` and duplicates real API calls).
|
|
|
|
## Test the real entry path
|
|
|
|
- Product-visible plugins require a non-unit REAL-composition test. Hand-built `ctx.plugin(...)` suites are insufficient: boot test-only `cordis.yml` through Loader and app/process, mock only external/nondeterministic boundaries, and assert model-visible request/log, durable state, or user-visible output. Keep opt-ins out of shipped defaults.
|
|
- A guard only guards if the regression actually fails it. For a plugin without `inject` (bundle/composition plugins), a Loader smoke stays green under a broken export shape — add an explicit `expect('default' in mod).toBe(false)` plus an `unwrapExports` round-trip assertion, and prove it: introduce the regression, watch red, revert.
|
|
- "Real entry path" means the published artifact: a package `bin` runs built `lib/bin.js` under plain `node`, exposing failures tsx masks (settle races, module resolution, swallowed load failures). The same applies to non-index runtime entries (the worker-thread sibling `lib/worker.cjs`) and singleton modules shared across bundles (`packages/ui/jsonrpc/tests/built-scope-carrier.e2e.ts`). Keep the built-artifact smokes green (`packages/ui/*/tests/built-bin.e2e.ts`, `packages/code-runtime/code-runtime-worker/tests/built-lib.e2e.ts`), and assert a genuinely-missing config exits non-zero.
|
|
|
|
## Test resolution: source plane only
|
|
|
|
- Every vitest config points vite-tsconfig-paths at `tsconfig.base.json`; bare workspace imports resolve to `src` ([layout](development.md#typescript-project-layout)), never through package `exports` to built `lib/` — stale artifacts there load a second copy of module singletons. Built artifacts are consumed only explicitly: `lib`-mode subprocesses and the built smokes below.
|
|
|
|
## Test subprocess launch modes
|
|
|
|
- CI and build-having test lanes run every example or Cordis-config subprocess from built `lib/` through the shared dual-mode launcher. Do not hand-write `--import tsx` for these subprocesses.
|
|
- Protocol and operating-system fixtures that do not load Cordis run erasable `.ts` directly with Node, without tsx or the root paths map.
|
|
- Only a test whose subject is source-path resolution may select `src`; state that contract in the test.
|
|
|
|
## When a snapshot test is required
|
|
|
|
Every non-trivial model-, protocol-, or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP automation scenarios use `examples/<name>/tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the `stream-json` snapshot and replay fixtures. Completed interactive-terminal journeys use JSONL-driven scenarios under `apps/cli/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. Browser-rendered web GUI journeys use `apps/web/tests/snapshots/`. New capability seams, lifecycle shapes, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation.
|