deepseek-harness/docs/testing.md

Testing policy

How this repo tests, tier by tier, and the rules that keep a green suite meaningful. Commands live in root AGENTS.md; linked RFCs carry the rationale.

Tiers

• Unit (pnpm run test): vitest over packages|examples/*/tests/**/*.spec.ts, colocated with what they test. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Prefer edge cases, error paths, event ordering, concurrency races, and permanent contract regressions (see packages/core/agent-loop/tests/contract-regressions.spec.ts).
• Coverage gate (pnpm run test:coverage): the gating run, per-file 100% on packages/*/*/src. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped.
• Real-API e2e (pnpm run test:e2e): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (EXA_API_KEY, PERPLEXITY_API_KEY, …); each suite self-skips without its key so keyless CI stays green (real-API e2e RFC).
• Snapshot (pnpm run test:snapshot): real example subprocesses replay recorded model sessions keylessly and compare normalized stdout plus re-persisted logs (snapshot RFC). The primary suite pins ACP JSON-RPC; the headless projection reuses advanced-toolchain for stream-json. Use pnpm run test:snapshot:record when the model transcript changes and pnpm run test:snapshot:refresh when only replay outputs change; review the golden diff. One scenario per header class pins system-prompt/tool-schema content; other fixtures tokenize it (pinned-header RFC).

The with-key policy: inference is cheap here

We are DeepSeek — do not ration real-API tests. A no-key test proves plumbing; only a with-key run proves the agent works against a real model. Write many: file-writing prompts, multi-turn conversations, tool use, cancellation mid-stream. Highest-value are smoke tests that boot the real example, send one real prompt, and check the world — they catch the "green unit tests, broken product" class that mocks structurally cannot (postmortem 0001). The self-skip exists only so secretless CI and keyless contributors aren't blocked; it is not a cost signal. Every example ships a keyless smoke and — unless keyless-by-nature — a with-key smoke (examples/AGENTS.md).

Prefer the real implementation over a mock

Mock only the genuinely expensive or non-deterministic boundary (the LLM adapter, the network, the clock); keep everything downstream real. A hand-rolled stand-in proves the bridge moves bytes, not that the shipping tool behaves as asserted — the two drift while the test stays green. Example: bridge tool-call tests run the scripted mock MODEL but the real tool + real executor (makeBridgeHarness({ withBash: true }) plugs dsh-bash-local + dsh-tool-bash and runs an actual echo).

Verify the world, not the self-report

An e2e assertion re-runs the command or re-reads the file externally; a keyword probe on the agent's own output lets a cheating agent pass. Assert untouched files are byte-identical. e2e tests own their resources: create the harness in the test, dispose in afterEach (even on failure/retry/timeout); shared fixtures live in a plain tests/harness.ts, never another *.e2e.ts (importing a spec re-registers its describe and duplicates real API calls).

Test the real entry path

• Product-visible plugins require a non-unit REAL-composition test. Hand-built ctx.plugin(...) suites are insufficient: boot test-only cordis.yml through Loader and app/process, mock only external/nondeterministic boundaries, and assert model-visible request/log, durable state, or user-visible output. Keep opt-ins out of shipped defaults.
• A guard only guards if the regression actually fails it. For a plugin without inject (bundle/composition plugins), a Loader smoke stays green under a broken export shape — add an explicit expect('default' in mod).toBe(false) plus an unwrapExports round-trip assertion, and prove it: introduce the regression, watch red, revert.
• "Real entry path" means the published artifact: the package bin points at built lib/bin.js under plain node, which tsx masks (settle races, module resolution, a swallowed load failure exiting 0). The same applies to any non-index runtime entry the built package resolves at run time (the worker-thread runtime's sibling lib/worker.cjs). Keep the built-artifact smokes green (packages/ui/*/tests/built-bin.e2e.ts, packages/code-runtime/code-runtime-worker/tests/built-lib.e2e.ts), and assert a genuinely-missing config exits non-zero.
• An e2e that spawns an example from a temp cwd sets TSX_TSCONFIG_PATH to the repo-root tsconfig, or it silently falls back to stale built lib/ (examples/AGENTS.md).

When a snapshot test is required

A change affecting an editor transcript, headless event stream, or agent UX adds or updates the owning examples/<name>/tests/snapshots/ scenario, or explains its omission in the PR. examples/acp-agent hosts the primary dsh-acp-snapshot table and the headless stream-json projection. Plans for new capability seams, lifecycle shapes, or transcript surfaces identify every test tier and any required harness work before implementation.