diff --git a/.agents/skills/dsh-code-review/SKILL.md b/.agents/skills/dsh-code-review/SKILL.md index ac59587e4d..d97bb1bd77 100644 --- a/.agents/skills/dsh-code-review/SKILL.md +++ b/.agents/skills/dsh-code-review/SKILL.md @@ -31,9 +31,10 @@ These define the conventions and gates this repo is checked against, and they ar These come straight from the source docs above. They are not discretionary; absence is a blocking gap. -1. **Docs in sync.** If the PR changes a config key, default, error code, wire field, or event name, it must update the package README + module/JSDoc in the same diff. The `doc-sync` gate (check #3) does not catch prose drift in config keys, defaults, error codes, or wire fields — that is on the reviewer, but it is still required, not optional. -2. **HMR-safety test.** Any new registry/registration needs a test that disposes the contributing fiber and asserts cleanup (packages/AGENTS.md). Its absence blocks merge. -3. **Quality gates pass.** typecheck, lint, test, test:coverage (100% per-file on `packages/*/src`), knip, build, publint, constraints, `doc-sync` (doc-typecheck + verify-event-taxonomy + verify-md-wrap), module-graph freshness (the quality-gates RFC). Don't re-review what a gate already enforces — trust the gate and spend attention on what it can't check. Note that the `doc-sync` gate only covers compilable `ts` blocks, the event-taxonomy table, and markdown wrapping; prose drift (check #1) is *additional* manual review on top of it, not covered by it. +1. **Docs in sync.** If the PR changes a config key, default, error code, wire field, or event name, it must update the package README + module/JSDoc in the same diff. The `doc-sync` gate (check #4) does not catch prose drift in config keys, defaults, error codes, or wire fields — that is on the reviewer, but it is still required, not optional. +2. **Core-data-structures catalog in sync.** If the PR adds, removes, or reshapes a type the [core-data-structures catalog](../../../docs/core-data-structures/core.md) documents — a new `…Map` variant, a new content-block/session-event type, a field on `GenerateOptions`/`Agent`/`ToolDefinition`/a bash type, or a whole new core/seam type — it must update that catalog in the same diff (prose + any verbatim ` ```ts type-equiv ` block + the 1:1 `scripts/type-equiv.manifest.json`). The `verify-type-equiv` gate (part of `doc-sync`) catches a *drifted paste* of an already-documented type, but it cannot tell you a brand-new core type went undocumented — that judgment is yours. Confirm a genuinely spine-level type landed in core.md and a new capability's vocabulary on a sub-page, per the spine-vs-seam line in [core.md § What counts as "core"](../../../docs/core-data-structures/core.md#what-counts-as-core). A pure internal type with no cross-package reach needs no catalog entry — say so if it's a judgment call. +3. **HMR-safety test.** Any new registry/registration needs a test that disposes the contributing fiber and asserts cleanup (packages/AGENTS.md). Its absence blocks merge. +4. **Quality gates pass.** typecheck, lint, test, test:coverage (100% per-file on `packages/*/src`), knip, build, publint, constraints, `doc-sync` (doc-typecheck + verify-cordis-catalog + verify-md-wrap + verify-md-links + verify-type-equiv), module-graph freshness (the quality-gates RFC). Don't re-review what a gate already enforces — trust the gate and spend attention on what it can't check. Note that the `doc-sync` gate only covers compilable `ts` blocks, the generated cordis events/services catalog, markdown wrapping/links, and verbatim type-equiv blocks; prose drift (checks #1 and #2) is *additional* manual review on top of it, not covered by it. ## Reviewer-only checks (gates can't catch these — judgment required) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5b9a9fca4f..562ffca1bc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -45,10 +45,11 @@ jobs: # Doc-sync gates (doc-sync-enforcement RFC). doc-typecheck compiles the fenced ts blocks in # the docs and resolves vendor packages via their built declarations, which - # the typecheck step above emits — so it runs after typecheck. The event - # taxonomy check and the markdown wrap check only read source. Same - # `doc-sync` script the pre-push hook runs (quality-gates RFC: one source of truth). - - name: Doc-sync gates (doc code blocks + event taxonomy + markdown wrap) + # the typecheck step above emits — so it runs after typecheck. The cordis + # catalog freshness check, type-equiv check, and markdown wrap/link checks + # only read source. Same `doc-sync` script the pre-push hook runs + # (quality-gates RFC: one source of truth). + - name: Doc-sync gates (doc code blocks + cordis catalog + type-equiv + markdown wrap/links) run: pnpm run doc-sync # Module-graph freshness: regenerate docs/module-graph.md from the diff --git a/AGENTS.md b/AGENTS.md index 424780bf74..8b8319ade0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -103,12 +103,13 @@ pnpm run knip # dead-code / unused-dependency check pnpm run publint # package.json publish-correctness check (publishable packages/*) pnpm run hygiene # knip + publint + workspace constraints pnpm run doc-typecheck # typecheck every ```ts block in README.md, docs/**/*.md, - # packages/*/README.md (doc/code drift gate) -pnpm run verify-event-taxonomy # assert the event-taxonomy table in docs/architecture.md - # matches the interface Events declarations in source + # packages/*/*.md (doc/code drift gate) +pnpm run gen-cordis-catalog # regenerate docs/cordis-catalog/events-and-services.md + # (events + services) from the interface Events / Context source +pnpm run verify-cordis-catalog # assert that generated catalog is not stale pnpm run verify-md-wrap # assert no hard-wrapped prose paragraphs in README.md, - # docs/**/*.md, packages/*/README.md, AGENTS.md (one line per paragraph) -pnpm run doc-sync # doc-typecheck + verify-event-taxonomy + verify-md-wrap (CI runs this) + # docs/**/*.md, packages/*/*.md, AGENTS.md (one line per paragraph) +pnpm run doc-sync # doc-typecheck + verify-cordis-catalog + verify-md-wrap + verify-md-links + verify-type-equiv (CI runs this) pnpm run demo:echo # run examples/echo-agent (no API key; type "echo hi" to # see a tool call) — the mock skeleton pnpm run demo:coding # run examples/coding-agent — the real agent (needs @@ -174,11 +175,15 @@ This codebase aims to be **very type-safe and well documented** for maintainabil In the **core** packages (`packages/llm`, `packages/tools`, `packages/agent`, `packages/agent-loop`, `packages/session`, `packages/system-prompt`), **type gymnastics are acceptable when they improve the DX of plugin authors** for common plugin types. The `defineTool` typed schema DSL in `dsh-tools` is the canonical example: the `SchemaSpec` to `InferArgs` type-level mapping gives tool authors zero-cast typed `execute` args, and the cost of the conditional types stays inside the core package. -Verbose documentation is fine **as long as docs and code stay strictly in sync**. Out-of-sync docs are worse than no docs. **When you change code, update its docs in the SAME change** — grep the package README and the module/JSDoc comments for the old behavior (config keys, defaults, error codes, wire field names, event names) and fix every hit. CI runs `pnpm run doc-sync` (`doc-typecheck` + `verify-event-taxonomy` + `verify-md-wrap` + `verify-md-links`), which typechecks every fenced `ts` block in `README.md`, `docs/**/*.md`, and `packages/*/README.md`, verifies the event-taxonomy table against source, asserts no hard-wrapped prose paragraphs, and checks that every relative Markdown cross-link resolves — across those files plus `AGENTS.md` / `packages/AGENTS.md` — but that scope does NOT catch prose drift in `AGENTS.md` / `packages/AGENTS.md` / `packages/README.md` (config keys, defaults, error codes), so keeping those in sync remains on the author. Every module has a module-level doc comment explaining its role. Every exported class, interface, type, function, and non-obvious method has a JSDoc that explains semantics (not just the name) — contracts (what events fire when), disposal behavior, error behavior, and extension intent. Internal helpers get docs only where non-obvious. Prefer one-liners when one line suffices. +Verbose documentation is fine **as long as docs and code stay strictly in sync**. Out-of-sync docs are worse than no docs. **When you change code, update its docs in the SAME change** — grep the package README and the module/JSDoc comments for the old behavior (config keys, defaults, error codes, wire field names, event names) and fix every hit. CI runs `pnpm run doc-sync` (`doc-typecheck` + `verify-cordis-catalog` + `verify-md-wrap` + `verify-md-links` + `verify-type-equiv`), which typechecks every fenced `ts` block in `README.md`, `docs/**/*.md`, and `packages/*/*.md`, regenerates the cordis events/services catalog from source and fails if the committed copy is stale, asserts no hard-wrapped prose paragraphs, checks that every relative Markdown cross-link resolves, and checks that every ` ```ts type-equiv ` doc block still matches its source type — across those files plus `AGENTS.md` / `packages/AGENTS.md` — but that scope does NOT catch prose drift in `AGENTS.md` / `packages/AGENTS.md` / `packages/README.md` (config keys, defaults, error codes), so keeping those in sync remains on the author. Every module has a module-level doc comment explaining its role. Every exported class, interface, type, function, and non-obvious method has a JSDoc that explains semantics (not just the name) — contracts (what events fire when), disposal behavior, error behavior, and extension intent. Internal helpers get docs only where non-obvious. Prefer one-liners when one line suffices. + +**Tag every new event with `@mode`.** The cordis events/services catalog ([docs/cordis-catalog/events-and-services.md](docs/cordis-catalog/events-and-services.md)) is GENERATED from source by `scripts/gen-cordis-catalog.ts` — never hand-edit it; run `pnpm run gen-cordis-catalog` and commit the result. When you add an event to an `interface Events` block, its JSDoc MUST carry a `@mode emit|waterfall|parallel` tag (the generator hard-errors without it): use `waterfall` when the signature ends with a `next: () => …` parameter (the listener transforms or vetoes via `next()`), `parallel` when the loop awaits a fan-out with no veto (e.g. an awaited `Promise | void` checkpoint like `session/flush`), and `emit` for plain fire-and-forget notifications. The generator also cross-checks the tag against the signature where the shape is conclusive (a trailing `next` ⇒ waterfall) and hard-errors on a contradiction. Write the rest of the event's JSDoc to stand alone — it is the catalog entry's prose. + +**The core-data-structures catalog is a maintained surface, not a write-once artifact.** [docs/core-data-structures/](docs/core-data-structures/core.md) catalogs the spine vocabulary (core.md) and the per-seam types (sub-pages). When a change adds, removes, or reshapes a type the catalog documents — a new `…Map` variant, a new content-block or session-event type, a field on `GenerateOptions`/`Agent`/`ToolDefinition`/a bash type, or a whole new core/seam type — update the catalog in the SAME change: edit the prose, and for a pasted ` ```ts type-equiv ` block, re-copy it verbatim and keep `scripts/type-equiv.manifest.json` 1:1 with the blocks. The `verify-type-equiv` gate catches a *drifted paste* of an already-documented type, but it canNOT tell you a brand-new core type was never documented — that judgment is on the author and the reviewer. The definition of "core" (the spine-vs-seam line) is in [core.md § What counts as "core"](docs/core-data-structures/core.md#what-counts-as-core); a genuinely spine-level new type belongs in core.md, a new capability's vocabulary on a sub-page. See [development.md](docs/development.md#documenting-types-verbatim-ts-type-equiv) for the `ts type-equiv` mechanics. **Document the CURRENT state — the "what" and "why" — never the PROCESS or HISTORY of how it got there.** A comment, JSDoc, or doc paragraph describes what the code *is* and why it is that way, as if it had always been so. Do NOT narrate the change that produced it: no "previously X, now Y", "changed from", "used to", "this replaces", "the old map", "renamed", "moved here", "as of this PR", or "(was …)". **In particular, NEVER name the change unit a reader cannot see — the PR, commit, or stack position that introduced the code — in a comment, JSDoc, OR a test name/description.** A `// (PR D's per-agent teardown)` aside, a `* Tests for the cancel primitive (PR C).` module doc, or an `it('… identity no longer matters')` title that only makes sense relative to a prior design are all the same violation: the reader of the current tree has no "PR D" or "old design" to anchor against, and the reference rots the moment the stack merges. Name the *mechanism* (`the session's AgentHandle teardown`), not the PR. Such phrasing rots the instant the next change lands, and a reader of the current code does not need the diff narrated in prose — that belongs in the commit message, the PR description, or an RFC (the durable home for "why we moved away from X"). Write "the owner token lives on the task in the executor" — not "ownership *now* lives on the executor instead of a plugin-local map". When a contrast genuinely aids understanding (a non-obvious choice between live alternatives), frame it against the alternative as a standing fact ("stored on the executor, NOT the tool plugin, so it survives an HMR reload"), not against the codebase's past. The same rule governs review-fix commits: the *commit message* records what the review caught; the *code comment* it touches states only the resulting truth. RFCs (`docs/rfc/`, grouped into `proposed/` / `implemented/` / `rejected/`) record the *why* behind choices a future reader would otherwise re-litigate (the vendoring policy, event-sourcing, the schema DSL are the existing examples). A PR that introduces such a decision — a new third-party runtime dependency over the vendoring default, a cross-package contract, a security/isolation model, a deviation from a documented architecture rule — writes the RFC in `implemented/` **in the same PR**, and links it from the relevant code. A proposal for future work not yet built goes in `proposed/`. A PR whose changes are mechanical, self-evident, or already covered by an existing RFC needs none — do not manufacture an RFC for a routine change. When unsure, the test is: would a competent maintainer six months from now ask "why was it done this way?" and be unable to answer from the code alone? If yes, write it. See [docs/rfc/README.md](docs/rfc/README.md) for the naming scheme and [docs/AGENTS.md](docs/AGENTS.md) for the cross-link convention. -**Markdown is not hard-wrapped**: write one line per paragraph and let the editor soft-wrap. Hard line breaks mid-paragraph make docs harder to edit and diff — a one-word change reflows and re-diffs the whole paragraph. This applies to prose only: leave fenced code blocks, tables, and list structure intact (a wrapped list item folds to one line per bullet). Code comments / JSDoc are exempt — they stay under the linter's column limit. `pnpm run verify-md-wrap` (part of `doc-sync`) enforces this across `README.md`, `docs/**/*.md`, `packages/*/README.md`, and `AGENTS.md` / `packages/AGENTS.md`; `pnpm run verify-md-links` (also part of `doc-sync`) checks that every relative cross-link in those files resolves. +**Markdown is not hard-wrapped**: write one line per paragraph and let the editor soft-wrap. Hard line breaks mid-paragraph make docs harder to edit and diff — a one-word change reflows and re-diffs the whole paragraph. This applies to prose only: leave fenced code blocks, tables, and list structure intact (a wrapped list item folds to one line per bullet). Code comments / JSDoc are exempt — they stay under the linter's column limit. `pnpm run verify-md-wrap` (part of `doc-sync`) enforces this across `README.md`, `docs/**/*.md`, `packages/*/*.md`, and `AGENTS.md` / `packages/AGENTS.md`; `pnpm run verify-md-links` (also part of `doc-sync`) checks that every relative cross-link in those files resolves. **Editing these instructions**: `AGENTS.md` is the real file; `CLAUDE.md` is a symlink to it (at the repo root and in `packages/`). Always edit `AGENTS.md` — never write through the `CLAUDE.md` symlink or replace it with a regular file. diff --git a/docs/architecture.md b/docs/architecture.md index 9a81b1c6d3..e4b9014904 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -8,6 +8,8 @@ The harness core is deliberately tiny: a handful of abstract services plus one c Requirement context: [Coding Harness MVP 需求分析][mvp-doc]. +For a catalog of the **data structures** this architecture moves around — the core vocabulary types, their literal shapes, and the seam types grouped by capability — see [core-data-structures/](core-data-structures/core.md). This document covers behavior; that one covers the types. + **Contents:** [Layering](#layering) · [Service map](#service-map) · [Capability seams](#capability-seams-interface--implementation--consumer) · [The vocabulary (dsh-llm)](#the-vocabulary-dsh-llm) · [Event-sourced sessions](#event-sourced-sessions-dsh-session) · [Prompt assembly](#prompt-assembly-dsh-system-prompt) · [Tool pipeline](#tool-pipeline-dsh-tools) · [Agents and the loop](#agents-dsh-agent-and-the-loop-dsh-agent-loop) ([lifecycle](#loop-lifecycle-session--turn--step), [event taxonomy](#event-taxonomy), [waterfall semantics](#cordis-waterfall-semantics-important)) · [Plugin sanity checklist](#plugin-sanity-checklist) · [Extension cookbook](#extension-cookbook) · [Deferred work](#deferred-work-todo) [microkernel-doc]: https://trtgsjkv6r.feishu.cn/wiki/VS9Lw1kQki6mDJk2UHocyuphnsc @@ -54,6 +56,8 @@ Dependency rule: plugins depend on interface packages, never on `dsh-agent-loop` All registrations (`registerAdapter`, `section`, `tools`, `register`, …) go through `ctx.effect()` and return disposers, so plugin hot-reload (vendored HMR) and fiber disposal clean up automatically. +For each service's full public interface (every method signature, generated from source), plus the inherited cordis-core/loader/hmr/timer surface a plugin also sees, see the `## Services` section of [cordis-catalog/events-and-services.md](cordis-catalog/events-and-services.md). This table is the at-a-glance role summary; that catalog is the exhaustive reference. + ## Capability seams: interface / implementation / consumer Swappable capabilities are split into **three packages** so each part evolves independently. The bash capability is the template: @@ -165,26 +169,7 @@ A failure that happens once the turn is already closed has no in-turn position f ### Event taxonomy -The `agent/*` events are declared in `@deepseek-ai/dsh-agent` (so nothing depends on the loop package); each other service declares its own events (`tools/*`, `llm/*`, `system-prompt/*`, `session/*`). The table below is CI-verified against the `interface Events` declarations in source (`scripts/verify-event-taxonomy.ts`). - -| Event | Mode | Purpose | -|---|---|---| -| `agent/created` / `agent/disposed` / `agent/status` / `agent/queued` | emit | lifecycle + inbox notifications | -| `agent/turn-start` / `agent/turn-end` / `agent/step-start` / `agent/step-end` | emit | boundaries | -| `agent/request` | **waterfall** | mutate the final `GenerateOptions` before the model call | -| `agent/stream-chunk` | emit | token-level UI/log feed | -| `agent/step-result` | **waterfall** | post-process the assistant message before tool dispatch | -| `agent/steering` | emit | steering content injected | -| `agent/turn-continuation` | **waterfall** | override the continue/stop decision | -| `agent/error` | emit | step/turn errors | -| `tools/execute` (dsh-tools) | **waterfall** | wrap/veto/sandbox tool execution | -| `tools/change` (dsh-tools) | emit | a tool was registered/unregistered | -| `llm/stream` / `llm/generate` (dsh-llm) | **waterfall** | model-call interception | -| `llm/adapter-change` (dsh-llm) | emit | an adapter was registered/unregistered | -| `system-prompt/assemble` (dsh-system-prompt) | **waterfall** | mutate the assembly | -| `system-prompt/change` (dsh-system-prompt) | emit | a section/tool-provider changed | -| `session/created` / `session/event` (dsh-session) | emit | session lifecycle + log feed | -| `session/flush` (dsh-session) | parallel (awaited) | durability checkpoint | +The `agent/*` events are declared in `@deepseek-ai/dsh-agent` (so nothing depends on the loop package); each other service declares its own events (`tools/*`, `llm/*`, `system-prompt/*`, `session/*`). The full catalog — every event's exact signature, dispatch mode, and prose — is **generated from source** and lives in [cordis-catalog/events-and-services.md](cordis-catalog/events-and-services.md) (the `## Events` section), alongside the `ctx.` service interfaces. That file is regenerated by `scripts/gen-cordis-catalog.ts` and frozen by the `verify-cordis-catalog` freshness gate (part of `doc-sync`), so it cannot drift from the `interface Events` declarations. ### Cordis waterfall semantics (important) diff --git a/docs/cordis-catalog/events-and-services.md b/docs/cordis-catalog/events-and-services.md new file mode 100644 index 0000000000..e34a2321d1 --- /dev/null +++ b/docs/cordis-catalog/events-and-services.md @@ -0,0 +1,481 @@ + + +# Cordis Events & Services Catalog + +An index reference to the **wiring** a plugin author works against: every cordis event you can listen to (exact signature + dispatch mode) and every `ctx.` service you can call (exact public interface). It complements [core-data-structures/](../core-data-structures/core.md), which catalogs the *data structures* these signatures move around — this page is the verbs, that page is the nouns. + +This file is GENERATED from source (`scripts/gen-cordis-catalog.ts`) and verified fresh by `pnpm run verify-cordis-catalog` (part of `doc-sync`) — do not edit it by hand. Signature blocks use a `ts cordis-catalog` fence (skipped by doc-typecheck, since a bare signature is not standalone-compilable). Type names in a signature link to the page that documents them. + +The **harness tier** below (the `@deepseek-ai/dsh-*` packages) is the vocabulary this repo owns. The **inherited tier** at the end is the cordis-core + loader/hmr/timer surface a plugin also sees — pinned vendor source, summarized tersely. + +## Events + +Dispatch modes: **emit** (fire-and-forget), **waterfall** (each listener gets `next()` and may transform or veto — see [waterfall semantics](../architecture.md#cordis-waterfall-semantics-important)), **parallel** (awaited fan-out, no veto). The harness declares 24 events across 5 scopes. + +### `agent/*` + +#### `agent/created` — emit + +An agent was registered in the AgentRegistry and is ready to receive messages. + +```ts cordis-catalog +'agent/created'(agent: Agent): void +``` + +Types: [Agent](../core-data-structures/core.md) + +Source: [`packages/agent/src/types.ts:140`](../../packages/agent/src/types.ts) + +#### `agent/disposed` — emit + +An agent was disposed and removed from the registry; its fiber and any in-flight turn have been torn down. + +```ts cordis-catalog +'agent/disposed'(agent: Agent): void +``` + +Types: [Agent](../core-data-structures/core.md) + +Source: [`packages/agent/src/types.ts:146`](../../packages/agent/src/types.ts) + +#### `agent/error` — emit + +A step or turn errored. The loop reports a failure here (plus the logger) even when the error has no in-turn position for a session `error` event. + +```ts cordis-catalog +'agent/error'(agent: Agent, turn: number, step: number, error: Error): void +``` + +Types: [Agent](../core-data-structures/core.md) + +Source: [`packages/agent/src/types.ts:223`](../../packages/agent/src/types.ts) + +#### `agent/queued` — emit + +A message entered the agent's inbox (queued or steering). `source` is the resolved source (defaults applied), not the caller's raw options. + +```ts cordis-catalog +'agent/queued'(agent: Agent, content: ContentBlock[], info: { source: MessageSource; steering: boolean }): void +``` + +Types: [Agent](../core-data-structures/core.md) · [ContentBlock](../core-data-structures/core.md) · [MessageSource](../core-data-structures/core.md) + +Source: [`packages/agent/src/types.ts:159`](../../packages/agent/src/types.ts) + +#### `agent/request` — waterfall + +Waterfall: mutate the fully-assembled GenerateOptions before the model call (hooks, compaction, model switching, tool filtering, …). Call `next()` to delegate, or return without it to short-circuit. + +```ts cordis-catalog +'agent/request'(agent: Agent, turn: number, step: number, options: GenerateOptions, next: () => Promise): Promise +``` + +Types: [Agent](../core-data-structures/core.md) · [GenerateOptions](../core-data-structures/core.md) + +Source: [`packages/agent/src/types.ts:192`](../../packages/agent/src/types.ts) + +#### `agent/status` — emit + +Agent status changed (`idle` ⇄ `running`, or → `disposed`). Drive lifecycle off this transition, never off a status you just requested — `send()` does not flip status to `running` before it returns. + +```ts cordis-catalog +'agent/status'(agent: Agent, status: AgentStatus): void +``` + +Types: [Agent](../core-data-structures/core.md) + +Source: [`packages/agent/src/types.ts:153`](../../packages/agent/src/types.ts) + +#### `agent/steering` — emit + +Steering content was injected into a running turn. + +```ts cordis-catalog +'agent/steering'(agent: Agent, turn: number, content: ContentBlock[], source: MessageSource): void +``` + +Types: [Agent](../core-data-structures/core.md) · [ContentBlock](../core-data-structures/core.md) · [MessageSource](../core-data-structures/core.md) + +Source: [`packages/agent/src/types.ts:217`](../../packages/agent/src/types.ts) + +#### `agent/step-end` — emit + +A step ended. + +```ts cordis-catalog +'agent/step-end'(agent: Agent, turn: number, step: number): void +``` + +Types: [Agent](../core-data-structures/core.md) + +Source: [`packages/agent/src/types.ts:183`](../../packages/agent/src/types.ts) + +#### `agent/step-result` — waterfall + +Waterfall: post-process the assembled assistant Message before tool dispatch (validation, content rewriting, …). + +```ts cordis-catalog +'agent/step-result'(agent: Agent, turn: number, step: number, message: Message, next: () => Promise): Promise +``` + +Types: [Agent](../core-data-structures/core.md) · [Message](../core-data-structures/core.md) + +Source: [`packages/agent/src/types.ts:198`](../../packages/agent/src/types.ts) + +#### `agent/step-start` — emit + +A step (one model call plus its tool dispatch) began. `step` is 1-based within the turn; a turn runs one or more steps. + +```ts cordis-catalog +'agent/step-start'(agent: Agent, turn: number, step: number): void +``` + +Types: [Agent](../core-data-structures/core.md) + +Source: [`packages/agent/src/types.ts:178`](../../packages/agent/src/types.ts) + +#### `agent/stream-chunk` — emit + +A raw StreamChunk arrived from the model (token-level UI/log feed). + +```ts cordis-catalog +'agent/stream-chunk'(agent: Agent, turn: number, step: number, chunk: StreamChunk): void +``` + +Types: [Agent](../core-data-structures/core.md) · [StreamChunk](../core-data-structures/llm-streaming.md) + +Source: [`packages/agent/src/types.ts:212`](../../packages/agent/src/types.ts) + +#### `agent/turn-continuation` — waterfall + +Waterfall: override the turn-continuation decision. The default (computed by the loop) is `hadToolCalls || steeringInjected`. Listeners can force-continue (/goal, /loop) or force-stop (budget guards). + +```ts cordis-catalog +'agent/turn-continuation'(agent: Agent, turn: number, defaultDecision: boolean, next: () => Promise): Promise +``` + +Types: [Agent](../core-data-structures/core.md) + +Source: [`packages/agent/src/types.ts:205`](../../packages/agent/src/types.ts) + +#### `agent/turn-end` — emit + +A turn ended. `reason` distinguishes a clean stop from a truncated or aborted one (`completed` | `aborted` | `error` | `disposed` | `max-tokens`). + +```ts cordis-catalog +'agent/turn-end'(agent: Agent, turn: number, reason: TurnEndReason): void +``` + +Types: [Agent](../core-data-structures/core.md) · [TurnEndReason](../core-data-structures/session.md) + +Source: [`packages/agent/src/types.ts:172`](../../packages/agent/src/types.ts) + +#### `agent/turn-start` — emit + +A turn began. `turn` is the 1-based turn number within the session. + +```ts cordis-catalog +'agent/turn-start'(agent: Agent, turn: number): void +``` + +Types: [Agent](../core-data-structures/core.md) + +Source: [`packages/agent/src/types.ts:166`](../../packages/agent/src/types.ts) + +### `llm/*` + +#### `llm/adapter-change` — emit + +An adapter was registered or unregistered (the model→adapter map changed). + +```ts cordis-catalog +'llm/adapter-change'(): void +``` + +Source: [`packages/llm/src/index.ts:43`](../../packages/llm/src/index.ts) + +#### `llm/generate` — waterfall + +Waterfall around every non-streaming model call. Bound to the LlmService; call `next()` to delegate to the adapter. + +```ts cordis-catalog +'llm/generate'(this: LlmService, options: GenerateOptions, next: () => Promise): Promise +``` + +Types: [GenerateOptions](../core-data-structures/core.md) · [GenerateResult](../core-data-structures/core.md) + +Source: [`packages/llm/src/index.ts:38`](../../packages/llm/src/index.ts) + +#### `llm/stream` — waterfall + +Waterfall around every streaming model call (retry, caching, routing). Bound to the LlmService; call `next()` to reach the resolved adapter's stream, or yield your own chunks to short-circuit. + +```ts cordis-catalog +'llm/stream'(this: LlmService, options: GenerateOptions, next: () => AsyncIterable): AsyncIterable +``` + +Types: [GenerateOptions](../core-data-structures/core.md) · [StreamChunk](../core-data-structures/llm-streaming.md) + +Source: [`packages/llm/src/index.ts:32`](../../packages/llm/src/index.ts) + +### `session/*` + +#### `session/created` — emit + +A session was created in the store. + +```ts cordis-catalog +'session/created'(session: Session): void +``` + +Source: [`packages/session/src/index.ts:30`](../../packages/session/src/index.ts) + +#### `session/event` — emit + +An event was appended to a session log (sync, fire-and-forget). This is the per-append feed a UI or invariant plugin tails. + +```ts cordis-catalog +'session/event'(session: Session, event: SessionEvent): void +``` + +Types: [SessionEvent](../core-data-structures/core.md) + +Source: [`packages/session/src/index.ts:36`](../../packages/session/src/index.ts) + +#### `session/flush` — parallel + +Awaited durability checkpoint. The agent loop awaits `ctx.parallel('session/flush', session)` at every turn end; persistence plugins (JSONL, SQLite) drain their write-behind buffers here and on fiber dispose. Awaited (parallel), not a waterfall: every listener runs and the loop waits for all of them, but none can veto. + +```ts cordis-catalog +'session/flush'(session: Session): Promise | void +``` + +Source: [`packages/session/src/index.ts:45`](../../packages/session/src/index.ts) + +### `system-prompt/*` + +#### `system-prompt/assemble` — waterfall + +Waterfall around prompt assembly — mutate or extend the PromptAssembly (sections + tool schemas) before it is rendered. Bound to the SystemPrompt service; call `next()` to delegate. + +```ts cordis-catalog +'system-prompt/assemble'(this: SystemPrompt, assembly: PromptAssembly, next: () => Promise): Promise +``` + +Source: [`packages/system-prompt/src/index.ts:24`](../../packages/system-prompt/src/index.ts) + +#### `system-prompt/change` — emit + +A section or tool provider was registered or unregistered (the assembly inputs changed). + +```ts cordis-catalog +'system-prompt/change'(): void +``` + +Source: [`packages/system-prompt/src/index.ts:30`](../../packages/system-prompt/src/index.ts) + +### `tools/*` + +#### `tools/change` — emit + +A tool was registered or unregistered (the available tool set changed). + +```ts cordis-catalog +'tools/change'(): void +``` + +Source: [`packages/tools/src/index.ts:48`](../../packages/tools/src/index.ts) + +#### `tools/execute` — waterfall + +Waterfall around every tool execution — the single seam where sandbox, permission, hook, and plan-mode plugins wrap or veto a call. Listeners receive `(exec, next)`: call `next()` to proceed (possibly around your own logic), or return a ToolExecutionResult without calling `next()` to short-circuit (veto). + +```ts cordis-catalog +'tools/execute'(this: ToolRegistry, exec: ToolExecution, next: () => Promise): Promise +``` + +Types: [ToolExecution](../core-data-structures/tools.md) · [ToolExecutionResult](../core-data-structures/tools.md) + +Source: [`packages/tools/src/index.ts:43`](../../packages/tools/src/index.ts) + +## Services + +The 8 `ctx.` services the harness provides. An abstract seam (e.g. `ctx.bash`) is implemented by a separate package; the interface is what consumers code against. + +### `ctx.agentLoop` — `AgentLoop` + +The agent-loop plugin (`ctx.agentLoop`): creates ReactLoopAgents, runs their loops, and registers them in `ctx.agents`. Also implements the AgentFactory seam, so plugins create/resume agents through `ctx.agents` (the interface) without depending on this concrete package. + +The loop itself is deliberately thin — every behavior beyond "call the model, run the tools, repeat" belongs to plugins listening on the event taxonomy declared in @deepseek-ai/dsh-agent. + +```ts cordis-catalog +create(id: string, options: AgentOptions = {}): ReactLoopAgent +createAgent(options: CreateAgentOptions): AgentHandle +async resume(options: ResumeAgentOptions): Promise +``` + +Source: [`packages/agent-loop/src/index.ts:60`](../../packages/agent-loop/src/index.ts) + +### `ctx.agents` — `AgentRegistry` + +Agent registry (`ctx.agents`): tracks live agents so UI, hook, and orchestrator plugins can find them without depending on the concrete loop package. Agent *creation* is provided by whichever plugin implements the AgentFactory (phase 1: `@deepseek-ai/dsh-agent-loop`), registered via setFactory. + +```ts cordis-catalog +setFactory(factory: AgentFactory): () => void +create(options: CreateAgentOptions): AgentHandle +async resume(options: ResumeAgentOptions): Promise +register(agent: Agent): () => void +get(id: string): Agent | undefined +list(): Agent[] +``` + +Types: [Agent](../core-data-structures/core.md) + +Source: [`packages/agent/src/index.ts:105`](../../packages/agent/src/index.ts) + +### `ctx.bash` — `BashExecutor` (abstract seam) + +Abstract bash execution service. Subclass, implement the abstract methods, and load the subclass as a plugin — it registers as `ctx.bash` (one implementation per context; loading a second throws, which is cordis' standard duplicate-service behavior). + +Semantics every implementation must honor: + +- run REJECTS only for infrastructure failures (unusable workdir, missing shell, pre-aborted signal). Nonzero exits, timeout kills, and abort kills RESOLVE with a descriptive BashRunResult — reporting a failed command is the tool layer's job, not an exception. +- start returns immediately; no timeout applies to background tasks (callers stop them via kill or the spec's AbortSignal). Completion must fire the onTaskDone listeners exactly once per task, and must NOT fire after the service is disposed. +- readOutput is incremental: consecutive reads never re-deliver output. Implementations bound their buffers; reads that lost data flag `lossy` and point at full-stream spill files when available. +- Disposal kills every running task and awaits their exit (no orphan processes survive `fiber.dispose()`). + +```ts cordis-catalog +abstract resolve(request: BashExecRequest): BashExecSpec +abstract run(spec: BashExecSpec): Promise +abstract start(spec: BashExecSpec): BashTask +abstract get(id: string): BashTask | undefined +abstract ownerOf(id: string): string | undefined +abstract list(): BashTask[] +abstract readOutput(id: string): BashTaskRead +abstract kill(id: string): boolean +onTaskDone(listener: BashTaskListener): () => void +``` + +Types: [BashExecRequest](../core-data-structures/bash.md) · [BashExecSpec](../core-data-structures/bash.md) · [BashRunResult](../core-data-structures/bash.md) · [BashTask](../core-data-structures/bash.md) · [BashTaskRead](../core-data-structures/bash.md) + +Source: [`packages/bash/src/index.ts:58`](../../packages/bash/src/index.ts) + +### `ctx.llm` — `LlmService` + +The abstract `llm` service: an adapter registry plus streaming / non-streaming call surfaces, both interceptable via waterfall events. + +```ts cordis-catalog +registerAdapter(models: string[], adapter: LlmAdapter): () => void +models(): string[] +stream(options: GenerateOptions): AsyncIterable +async * streamBlocks(options: GenerateOptions): AsyncIterable +generate(options: GenerateOptions): Promise +``` + +Types: [ContentBlock](../core-data-structures/core.md) · [GenerateOptions](../core-data-structures/core.md) · [GenerateResult](../core-data-structures/core.md) · [StreamChunk](../core-data-structures/llm-streaming.md) + +Source: [`packages/llm/src/index.ts:81`](../../packages/llm/src/index.ts) + +### `ctx.sessionPersistence` — `SessionPersistence` (abstract seam) + +Abstract durable session-persistence service. Subclass, implement the abstract methods, and load the subclass as a plugin — it registers as `ctx.sessionPersistence` (one implementation per context; loading a second throws, cordis' standard duplicate-service behavior). + +Contracts every implementation MUST honor (a DB backend asserts them inside a transaction; a file backend appends at EOF): + +- **Append-only; a crashed turn is closed, not truncated.** Committed events — those at or below a flushed `turn/end` — are never rewritten. A crash can leave an unclosed final turn whose events are real (and possibly large); load preserves them and closes the orphaned turn with synthetic boundary events (see load). Only a never-fully-written torn tail fragment is discarded. +- **Contiguous seq.** A persisted log is contiguous: `events[i].seq === i`. load rejects a parse error or a `seq` gap in the COMMITTED region (unloadable); append's first event `seq` MUST equal the backend's stored next-seq (after `load` has balanced any interrupted turn). +- **JSON-serializable data.** `SessionEventMap` is merge-extensible and `event.data` is typed only as `SessionEventMap[K]`, so append REJECTS non-JSON-serializable data with an error naming the offending event type. A backend snapshots (serializes/clones) each event when it buffers, since `session.events` hands out the live mutable object. +- **Durability.** append returns only once the batch is durable (the file backend fsyncs; a DB commits). create MAY defer the physical write until the first append (lazy materialization). + +```ts cordis-catalog +abstract create(meta: SessionHeader): Promise +abstract append(id: SessionId, events: readonly SessionEvent[]): Promise +abstract load(id: SessionId): Promise<{ meta: SessionHeader; events: SessionEvent[] }> +abstract list(): Promise +abstract has(id: SessionId): Promise +abstract delete(id: SessionId): Promise +``` + +Types: [SessionEvent](../core-data-structures/core.md) + +Source: [`packages/session-persistence/src/index.ts:98`](../../packages/session-persistence/src/index.ts) + +### `ctx.sessions` — `SessionStore` + +In-memory session store (`ctx.sessions`). + +Persistence is intentionally not implemented here — persistence plugins subscribe to `session/event` and flush on `session/flush` / dispose. + +```ts cordis-catalog +create(id?: string, options?: CreateSessionOptions): Session +prepare(id?: string, options?: CreateSessionOptions): Session +enter(session: Session): () => void +announce(session: Session): void +get(id: string): Session | undefined +list(): Session[] +``` + +Source: [`packages/session/src/index.ts:222`](../../packages/session/src/index.ts) + +### `ctx.systemPrompt` — `SystemPrompt` + +Registry service (`ctx.systemPrompt`): plugins contribute ordered text sections and tool-schema providers; the agent loop calls `assemble()` once per step. + +```ts cordis-catalog +section(section: PromptSection): () => void +tools(provider: () => ToolSchema[]): () => void +assemble(): Promise +``` + +Source: [`packages/system-prompt/src/index.ts:71`](../../packages/system-prompt/src/index.ts) + +### `ctx.tools` — `ToolRegistry` + +Tool registry (`ctx.tools`): tool plugins register definitions; the agent loop executes calls through the `tools/execute` waterfall. The registry contributes its schemas into the system-prompt assembly. + +```ts cordis-catalog +register(definition: ToolDefinition): () => void +get(name: string): ToolDefinition | undefined +schemas(): ToolSchema[] +async execute(exec: ToolExecution): Promise +``` + +Types: [ToolDefinition](../core-data-structures/tools.md) · [ToolExecution](../core-data-structures/tools.md) · [ToolExecutionResult](../core-data-structures/tools.md) + +Source: [`packages/tools/src/index.ts:277`](../../packages/tools/src/index.ts) + +## Inherited tier (cordis core + loader/hmr/timer) + +The framework surface every plugin inherits, beyond the harness vocabulary above. This is pinned vendor source ([vendoring policy](../../vendor/README.md)); it is summarized here so the catalog is a complete picture of what `ctx` and the event bus offer, without elevating framework internals to the harness tier's prominence. + +### Inherited events + +- `internal/plugin` — A plugin fiber was created. ([`vendor/cordis/src/events.ts:197`](../../vendor/cordis/src/events.ts)) +- `internal/status` — A fiber changed lifecycle state. ([`vendor/cordis/src/events.ts:198`](../../vendor/cordis/src/events.ts)) +- `internal/service` — Interception hook for a service binding (no core producer). ([`vendor/cordis/src/events.ts:199`](../../vendor/cordis/src/events.ts)) +- `internal/update` — Waterfall: a fiber config update is being applied. ([`vendor/cordis/src/events.ts:200`](../../vendor/cordis/src/events.ts)) +- `internal/get` — Waterfall: a service is being read from the store. ([`vendor/cordis/src/events.ts:201`](../../vendor/cordis/src/events.ts)) +- `internal/set` — Waterfall: a service is being written to the store. ([`vendor/cordis/src/events.ts:202`](../../vendor/cordis/src/events.ts)) +- `internal/listener` — A listener was registered. ([`vendor/cordis/src/events.ts:203`](../../vendor/cordis/src/events.ts)) +- `internal/dispatch` — An event is being dispatched to listeners. ([`vendor/cordis/src/events.ts:204`](../../vendor/cordis/src/events.ts)) +- `hmr/change` — A watched source file changed on disk. ([`vendor/hmr/src/index.ts:20`](../../vendor/hmr/src/index.ts)) +- `hmr/reload` — Plugins are being reloaded after a change. ([`vendor/hmr/src/index.ts:21`](../../vendor/hmr/src/index.ts)) +- `exit` — The process is exiting on a signal. ([`vendor/loader/src/index.ts:23`](../../vendor/loader/src/index.ts)) +- `loader/config-update` — The loader config tree changed. ([`vendor/loader/src/index.ts:24`](../../vendor/loader/src/index.ts)) +- `loader/entry-init` — A config entry is being initialized. ([`vendor/loader/src/index.ts:25`](../../vendor/loader/src/index.ts)) +- `loader/partial-dispose` — An entry is being partially disposed on reload. ([`vendor/loader/src/index.ts:26`](../../vendor/loader/src/index.ts)) +- `loader/patch-context` — A context is being patched during a reload. ([`vendor/loader/src/index.ts:27`](../../vendor/loader/src/index.ts)) + +### Inherited `ctx` members + +- `ctx.on / ctx.once` — Register an event listener (disposable). ([`vendor/cordis/src/events.ts:29`](../../vendor/cordis/src/events.ts)) +- `ctx.emit / ctx.parallel / ctx.serial / ctx.bail / ctx.waterfall` — Dispatch an event (sync / awaited / first-non-nullish / veto-chain). ([`vendor/cordis/src/events.ts:29`](../../vendor/cordis/src/events.ts)) +- `ctx.plugin / ctx.inject` — Load a plugin / declare required services. ([`vendor/cordis/src/registry.ts:144`](../../vendor/cordis/src/registry.ts)) +- `ctx.effect` — Register a disposable side effect tied to the fiber. ([`vendor/cordis/src/fiber.ts:9`](../../vendor/cordis/src/fiber.ts)) +- `ctx.get / ctx.set / ctx.provide / ctx.accessor / ctx.mixin` — Low-level service-store access and binding. ([`vendor/cordis/src/reflect.ts:7`](../../vendor/cordis/src/reflect.ts)) +- `ctx.extend / ctx.isolate / ctx.intercept` — Derive a child context (scoped services / isolation / interception). ([`vendor/cordis/src/context.ts:35`](../../vendor/cordis/src/context.ts)) +- `ctx.root / ctx.scope / ctx.fiber / ctx.registry / ctx.reflect / ctx.events / ctx.logger` — Ambient handles onto the running context graph. ([`vendor/cordis/src/context.ts:16`](../../vendor/cordis/src/context.ts)) +- `ctx.timer (+ interval / timeout / throttle / debounce / setTimeout / setInterval)` — Disposable timer helpers. The `timer` key is provided at runtime; the six helpers are mixed onto ctx directly (declared via Pick). ([`vendor/timer/src/index.ts:4`](../../vendor/timer/src/index.ts)) +- `ctx.loader` — The config Loader that booted the app (present under the loader). ([`vendor/loader/src/index.ts:30`](../../vendor/loader/src/index.ts)) +- `ctx.hmr` — The hot-module-reload watcher (present under the hmr plugin). ([`vendor/hmr/src/index.ts:15`](../../vendor/hmr/src/index.ts)) diff --git a/docs/core-data-structures/bash.md b/docs/core-data-structures/bash.md new file mode 100644 index 0000000000..fc24bbd386 --- /dev/null +++ b/docs/core-data-structures/bash.md @@ -0,0 +1,123 @@ +# Bash Executor + +The bash execution seam — the canonical [capability seam](../rfc/implemented/2026-06-13-capability-seams.md) example, split across three packages: interface ([dsh-bash](../../packages/bash), `ctx.bash`), implementation ([dsh-bash-local](../../packages/bash-local), local subprocesses), and consumer ([dsh-tool-bash](../../packages/tool-bash), the `bash`/`bash_output`/`bash_kill` tool schemas). Bash is **one optional capability**, not part of the agent-loop spine — so its vocabulary lives here, not in [core.md](core.md). A sandboxed, containerized, or remote backend is a sibling package implementing the same interface. + +Source: [`packages/bash/src/types.ts`](../../packages/bash/src/types.ts) + +## Request vs. spec: the `resolve()` split + +The seam separates the **model-/plugin-facing request** (optional `workdir`/`timeoutMs`, filled from config) from the **fully-resolved spec** the executor acts on (those fields required). The tool layer calls `ctx.bash.resolve(request)` between them — this is the repo's "explicit > implicit at package seams" rule made concrete: the reader of a `BashExecSpec` never wonders where the working directory came from. + +```ts type-equiv +interface BashExecRequest { + command: string + /** Working directory override (default: implementation-configured). */ + workdir?: string | undefined + /** Timeout override in milliseconds (implementations cap it). */ + timeoutMs?: number | undefined + /** Abort signal — implementations kill the command when it fires. */ + signal?: AbortSignal | undefined + /** + * Opaque OWNER token for a background task — the consumer's isolation key + * (the tool layer passes the owning agent's `session.header.id`). The + * executor stores it on the task and exposes it via {@link BashExecutor.ownerOf}; + * the executor itself NEVER interprets it (no access policy lives in the + * seam — that is the consumer's job). Absent for foreground runs and for an + * ownerless background start (a non-agent caller). + */ + owner?: string | undefined +} +``` + +```ts type-equiv +interface BashExecSpec { + command: string + workdir: string + timeoutMs: number + /** Abort signal — implementations kill the command when it fires. */ + signal?: AbortSignal | undefined + /** + * Opaque owner token, REQUIRED-but-nullable (mirrors `workdir`/`timeoutMs` + * being required on the resolved spec): {@link BashExecutor.resolve} carries + * the request's `owner` through, defaulting a missing one to `undefined`. A + * required field makes a forgotten owner a VISIBLE `undefined` rather than a + * silently-absent property that yields an unowned (cross-session-readable) + * task. `start()` stores it; `run()` (foreground) ignores it. + */ + owner: string | undefined +} +``` + +The `owner` token is the isolation key: the executor stores it but never interprets it (access policy is the consumer's job), so a background task started by one agent isn't readable cross-session. A required-but-nullable field makes a forgotten owner a visible `undefined` rather than a silently-unowned task. + +## Foreground runs: `BashRunResult` + +The outcome of one completed (or killed) foreground run. Orthogonal outcomes are reported **independently** — a process can both time out AND exit 0 because it trapped the signal — so `timedOut`, `aborted`, `signal`, and `exitCode` are each their own field; a caller never reads a cut-short run as a clean success. + +```ts type-equiv +interface BashRunResult { + /** Exit code; null when the process died from a signal. */ + exitCode: number | null + /** Terminating signal (e.g. 'SIGTERM'); null on normal exit. */ + signal: NodeJS.Signals | null + /** True when the executor's own timeout killed the command. */ + timedOut: boolean + /** True when the caller's AbortSignal killed the command. */ + aborted: boolean + /** The effective timeout applied to this run (after defaulting/capping). */ + timeoutMs: number + stdout: CollectedOutput + stderr: CollectedOutput +} +``` + +Each stream is a `CollectedOutput` — the (possibly truncated) text plus recovery info. When truncated, `text` is the **tail** and the complete stream spills to a private file: + +```ts type-equiv +interface CollectedOutput { + /** Collected text — the TAIL of the stream when truncated. */ + text: string + /** True when bytes were dropped from `text`. */ + truncated: boolean + /** Path to a file holding the COMPLETE stream, when truncated and available. */ + spillPath?: string +} +``` + +## Background tasks: `BashTask` + +A long-running command started with `start()` is tracked as a `BashTask`. `BashTaskStatus` is `'running' | 'completed' | 'killed'`; `done` resolves when the underlying process closes and never rejects. + +```ts type-equiv +interface BashTask { + readonly id: string + readonly command: string + status: BashTaskStatus + /** Exit code once finished (null = killed by signal / still running). */ + exitCode: number | null + /** Terminating signal name, when signal-killed. */ + signal: NodeJS.Signals | null + /** Resolves when the underlying process closes (never rejects). */ + readonly done: Promise +} +``` + +`readOutput()` returns an incremental `BashTaskRead` — the output produced since the previous read, with a `lossy` flag when truncation dropped unread bytes: + +```ts type-equiv +interface BashTaskRead { + task: BashTask + /** Output produced since the previous read (stderr in a marked section). */ + delta: string + /** True when truncation dropped unread bytes the delta cannot include. */ + lossy: boolean + /** Full stdout spill file, when stdout truncation occurred and a safe path is available. */ + stdoutSpillPath?: string + /** Full stderr spill file, when stderr truncation occurred and a safe path is available. */ + stderrSpillPath?: string +} +``` + +## The service + +`BashExecutor` (`ctx.bash`, abstract — defined in [`packages/bash/src/index.ts`](../../packages/bash/src/index.ts)) mirrors the `LlmService`/`LlmAdapter` split: `resolve` (request → spec), `run` (foreground), `start` (background), `get`/`ownerOf`/`list`/`readOutput`/`kill`, and `onTaskDone` (a `BashTaskListener` completion callback). Spawned commands get a **scrubbed env** (dropping `*KEY*`/`*SECRET*`/`*TOKEN*`) and spill files use a private 0700 dir with random names and owner-only opens — model output never gets the ambient environment or a predictable path. The implementation that provides all this is `dsh-bash-local`; the model-facing `bash`/`bash_output`/`bash_kill` schemas that call it are in `dsh-tool-bash` (and present as terminals via the [tool-presentation vocabulary](tools.md#tool-presentation-ui-vocabulary)). diff --git a/docs/core-data-structures/core.md b/docs/core-data-structures/core.md new file mode 100644 index 0000000000..3743e6e0df --- /dev/null +++ b/docs/core-data-structures/core.md @@ -0,0 +1,300 @@ +# Core Data Structures + +This folder catalogs the **data structures** of the DeepSeek Harness — what each core type represents, its literal shape, and where the full detail lives. It complements [architecture.md](../architecture.md), which describes *behavior* (the service map, the session/turn/step lifecycle, the event taxonomy); this page describes the *vocabulary* that behavior moves around. + +## What counts as "core" + +The harness is a microkernel: a tiny core plus many plugins. Most types belong to one plugin or one capability. A handful, though, are the **spine** — the language the agent loop and its events traffic in on *every* turn, no matter which optional plugins are loaded. Those are "core". + +Precisely, a data structure is **core** if either: + +1. it flows through the agent-loop spine — the loop holds it, derives it, streams it, or logs it on every turn (a `Message`, a `StreamChunk`, a `SessionEvent`, the `Agent` handle itself), independent of which plugins are present; **or** +2. it is the single headline type a plugin author writes against a pipeline — `ToolDefinition` (what every tool *is*). + +Everything else is documented on a **sub-page**, not here. The rule that draws the line: *the type you write, hold, or receive is core; the machinery that types it, renders it, or persists it is a sub-page detail.* So `ToolDefinition` is core, but the `SchemaSpec`/`InferArgs` DSL that types it, the `ToolCallPresentation` vocabulary that renders it, and the `SessionPersistence` seam that stores the event log are not — they live on the sub-pages below. + +| Sub-page | Owns | +|---|---| +| [llm-streaming.md](llm-streaming.md) | the `StreamChunk` wire protocol + adapter contract, `BlockAssembler`, the `LlmAdapter` seam | +| [session.md](session.md) | the full `SessionEventMap` variant catalog, `TurnTrigger`/`TurnEndReason`, `deriveMessages()`, the turn-enclosure invariant | +| [persistence.md](persistence.md) | the durability seam: `SessionPersistence`, JSONL + SQLite backends, `session/flush`, crash recovery, `SessionHeader` | +| [tools.md](tools.md) | `ToolDefinition` full fields, the schema DSL, `ToolExecution`/`ToolResult`, tool-presentation UI types, the `tools/execute` waterfall | +| [bash.md](bash.md) | the bash executor seam: `BashExecRequest`/`Spec`, `BashRunResult`, background `BashTask`s | + +> Type definitions on this page are pasted **verbatim** from source and drift-checked by `pnpm run verify-type-equiv` (see [development.md](../development.md#documenting-types-verbatim-ts-type-equiv)). Inline JSDoc is omitted for readability; follow the source link for the full contracts. + +## The `…Map → derived-union` pattern + +Almost every extensible sum type in the harness follows one shape: an interface keyed by a discriminant tag (the `…Map`), from which the union is derived with `keyof`. Plugins add variants by **declaration merging** — no edit to the owning package. + +```ts ignore-check +// The pattern, schematically: +interface ThingMap { + 'a': { kind: 'a'; /* … */ } + 'b': { kind: 'b'; /* … */ } +} +type ThingKind = keyof ThingMap // 'a' | 'b' +type Thing = ThingMap[keyof ThingMap] // the discriminated union + +// A plugin extends it without touching the source package: +declare module '@deepseek-ai/dsh-llm' { + interface ThingMap { + 'c': { kind: 'c'; /* … */ } + } +} +``` + +Six canonical maps use this pattern; a plugin author extends these: + +| Map | Package | Derives | Catalog | +|---|---|---|---| +| `ContentBlockMap` | dsh-llm | `ContentBlock` | [below](#content-blocks-and-messages) | +| `MessageSourceMap` | dsh-llm | `MessageSource` | [below](#content-blocks-and-messages) | +| `FinishReasonMap` | dsh-llm | `FinishReason` | [below](#the-model-request-and-result) | +| `TurnTriggerMap` | dsh-session | `TurnTrigger` | [session.md](session.md) | +| `TurnEndReasonMap` | dsh-session | `TurnEndReason` | [session.md](session.md) | +| `SessionEventMap` | dsh-session | `SessionEvent` | [session.md](session.md) | + +Two large discriminated unions are the ones consumers `switch` over most: **`StreamChunk`** (the streaming protocol) and **`SessionEvent`** (the log entry). Per the repo convention, `switch` on the tag — don't chain `if`s — so each arm narrows and a typo'd tag fails to compile. + +## Branded IDs + +IDs that cross package boundaries are **branded** — structurally strings, but non-interchangeable at the type level (an `AgentId` can't be passed where a `CallId` is expected). Construction goes through a per-type factory; comparison, logging, and JSON behave as ordinary strings. + +Source: [`packages/llm/src/brand.ts`](../../packages/llm/src/brand.ts) + +```ts type-equiv +type Branded = string & { readonly [BRAND]: B } +``` + +The three core IDs: `CallId` (correlates a tool call with its result; dsh-llm), `SessionId` (dsh-session), `AgentId` (dsh-agent). Each is `Branded<'CallId'>` etc. plus a same-named factory function. + +## Content blocks and messages + +A conversation is `Message`s; a message is an array of typed **content blocks**. The block union derives from `ContentBlockMap`. + +Source: [`packages/llm/src/types.ts`](../../packages/llm/src/types.ts) + +```ts type-equiv +interface ContentBlockMap { + 'text': TextBlock + 'reasoning': ReasoningBlock + 'tool-call': ToolCallBlock + 'tool-result': ToolResultBlock + 'image': ImageBlock +} +``` + +The block interfaces (full fields in source): `TextBlock` (`text`), `ReasoningBlock` (thinking, distinct from visible text), `ToolCallBlock` (`id: CallId`, `name`, raw-JSON `arguments`), `ToolResultBlock` (`toolCallId`, nested `content: ContentBlock[]`, `isError?`), `ImageBlock` (`url`, `mimeType?`). `ContentBlock = ContentBlockMap[ContentBlockType]`. + +A `Message` is a role plus blocks: + +```ts type-equiv +interface Message { + role: 'system' | 'user' | 'assistant' + content: ContentBlock[] +} +``` + +Where a message came from is itself a merge-extensible sum type: + +```ts type-equiv +interface MessageSourceMap { + user: { kind: 'user' } + plugin: { kind: 'plugin'; plugin: string } + agent: { kind: 'agent'; agentId: string } +} +``` + +## Streaming + +Adapters emit a raw **chunk** protocol; the loop logs the chunks (replay fidelity) while feeding the same chunks through a `BlockAssembler` to rebuild blocks and messages. `StreamChunk` is a closed discriminated union over `type` — `block-start`, `text-delta`, `reasoning-delta`, `tool-call-delta`, `block-end`, `usage`, `finish`. + +The full union, the adapter contract (usage-before-finish, raw-JSON tool arguments, the two sanctioned error paths), and `BlockAssembler` live on **[llm-streaming.md](llm-streaming.md)**. + +## The model request and result + +One model call is a fully-assembled `GenerateOptions`; the non-streaming result is `GenerateResult`. + +Source: [`packages/llm/src/types.ts`](../../packages/llm/src/types.ts) + +```ts type-equiv +interface GenerateOptions { + model: string + messages: Message[] + /** System prompt text (adapters map to the provider's system slot). */ + system?: string + /** Tool schemas (adapters map to the provider's `tools` field). */ + tools?: ToolSchema[] + /** Assistant prefix continuation (prefill). */ + prefill?: ContentBlock[] + temperature?: number + maxTokens?: number + /** + * Stop sequences: generation halts as soon as the model produces any one of + * these strings (adapters map to the provider's stop field, e.g. OpenAI + * `stop`). The stop string itself is not included in the output. + */ + stop?: string[] + signal?: AbortSignal +} +``` + +```ts type-equiv +interface GenerateResult { + message: Message + usage?: TokenUsage + finish: FinishReason +} +``` + +Why a model response stopped is a merge-extensible reason: + +```ts type-equiv +interface FinishReasonMap { + 'stop': { kind: 'stop' } + 'tool-calls': { kind: 'tool-calls' } + 'max-tokens': { kind: 'max-tokens' } + 'aborted': { kind: 'aborted' } + 'error': { kind: 'error'; message: string; code?: string } +} +``` + +`FinishReason = FinishReasonMap[keyof FinishReasonMap]`. `TokenUsage` (per-call accounting with disjoint cache fields) is detailed on [llm-streaming.md](llm-streaming.md). + +`GenerateOptions.tools` carries `ToolSchema` — the JSON-schema description of a tool, as sent to the model. It is declared in dsh-llm (not dsh-tools) precisely because it is part of the request the loop assembles every step: + +```ts type-equiv +interface ToolSchema { + name: string + description: string + /** JSON Schema object for the arguments. */ + parameters: Record + strict?: boolean +} +``` + +The model-facing `ToolSchema` is the wire shape; the registered `ToolDefinition` that produces it (schema + `execute`) is on [tools.md](tools.md). + +## Sessions + +A `Session` is an **append-only log** of typed `SessionEvent`s — the single source of truth. The LLM message history is *derived* from the log (`deriveMessages()`), not stored separately. The event vocabulary derives from `SessionEventMap`: + +Source: [`packages/session/src/types.ts`](../../packages/session/src/types.ts) + +```ts type-equiv +type SessionEvent = { + [K in SessionEventType]: { + type: K + /** Monotonic sequence number within the session. */ + seq: number + /** Unix epoch milliseconds. */ + time: number + data: SessionEventMap[K] + } +}[T] +``` + +The thirteen event variants (`turn/start`, `turn/end`, `step/start`, `step/end`, `user/message`, `context/message`, `assistant/chunk`, `assistant/message`, `tool/call`, `tool/result`, `steering/message`, `usage`, `error`), the `deriveMessages()` projection rules, the `TurnTrigger`/`TurnEndReason` reasons, and the turn-enclosure invariant are on **[session.md](session.md)**. How the log is made durable — the `SessionPersistence` seam, JSONL/SQLite backends, the `session/flush` checkpoint, crash recovery, and `SessionHeader` — is on **[persistence.md](persistence.md)**. + +## The agent handle + +`Agent` is the surface every plugin (UI, hooks, orchestrators) programs against. The concrete implementation is `ReactLoopAgent` in dsh-agent-loop; nothing outside the loop depends on the implementation. + +Source: [`packages/agent/src/types.ts`](../../packages/agent/src/types.ts) + +```ts type-equiv +interface Agent { + readonly id: AgentId + readonly options: AgentOptions + readonly session: Session + readonly status: AgentStatus + + /** Queue a user message. Starts a turn when idle; otherwise waits for the next turn. */ + send(content: ContentBlock[], options?: SendOptions): void + + /** + * Steer a running turn: content is injected between steps of the current + * turn. When idle, behaves like {@link send}. + */ + steer(content: ContentBlock[], options?: SendOptions): void + + /** + * Inject in-session context (file-change notices, skill content, cron + * notifications, …): appends a `context/message` session event the next model + * request sees at its chronological position, rendered as tagged synthetic + * context rather than a user prompt. Does not run the model. + * + * Turn-enclosure (the turn-enclosure RFC): an inject while a turn is open joins that turn; + * an inject while idle wraps its `context/message` in a one-shot `injection` + * turn (`turn/start` → `context/message` → `turn/end`) and checkpoints it for + * durability, so every event stays inside a turn and a persistence backend + * never loses a between-turn notice. The idle checkpoint is fire-and-forget + * (inject is synchronous): a failing flush is reported via `agent/error` + * (step `0`) and the logger, never thrown into the caller. + * + * Live-adapter review has validated the tagged-envelope rendering against + * current DeepSeek behavior; provider-specific mismatches belong in that + * adapter, not in the canonical session vocabulary. + */ + inject(content: ContentBlock[], options?: SendOptions): void + + /** Abort the in-flight step (if any); the turn ends with reason 'aborted'. */ + abort(reason?: string): void + + /** + * Cancel ALL pending work for the agent — the narrower {@link abort} kills + * only the in-flight step. `cancel()`: + * + * - clears the queued FIFO (un-started prompts never run) and the steering + * FIFO (steering for the cancelled turn is dropped, not re-enqueued); + * - aborts the in-flight step if one is running (the turn ends `aborted`); + * - drops a turn that is about to start (a `cancel()` landing in the + * pre-step window — after a `send()` queued but before the loop flips to + * `running`, or after `running` is emitted but before the first step) so + * that queued prompt does not run and cannot be batched into the cancelled + * turn. + * + * After `cancel()`, `whenIdle()` resolves on the post-cancel quiescent state. + * `cancel()` on an idle agent with nothing queued or running is a safe no-op + * — it does NOT arm anything that would drop a later legitimate prompt. + */ + cancel(reason?: string): void + + /** + * Resolve once the agent has reached quiescence after settling out of + * `running`, or immediately if it is already idle with no queued work. The + * quiescence signal a teardown awaits: `agent.abort()` then + * `await agent.whenIdle()` guarantees queued/running work has fully stopped + * before the caller proceeds (a closing ACP connection, a disposing UI + * plugin), rather than returning while the driver is still streaming or about + * to start a queued turn. + * + * "Quiescence", not merely "status changed": a disposed agent emits + * `agent/status('disposed')` from inside its disposer, BEFORE the driver loop + * has unwound — so `whenIdle()` resolving on `disposed` must wait for the loop + * to actually exit (the implementation chains the loop-exit promise), not just + * observe the status flip. A mid-step disposal that never reaches `idle` still + * unblocks the await this way. + * + * Distinct from disposal: `whenIdle()` observes the transition WITHOUT tearing + * the agent down. A consumer that owns the agent's lifecycle disposes it + * separately. + */ + whenIdle(): Promise + + // TODO(sub-agents): spawn/fork seams — semantics deliberately deferred. + // The intended shape: a creation option referencing a parent agent + // (fork = seed the child Session with the parent's event log; spawn = + // fresh Session), with the child returned as an Agent handle so steer() + // and event subscription work uniformly. See docs/architecture.md. +} +``` + +`AgentStatus` is `'idle' | 'running' | 'disposed'`. `AgentId` is a branded string. `AgentOptions` (`model?`, `systemPrompt?`) is merge-extensible — plugins add creation options by declaration merging. The `agent/*` event taxonomy (lifecycle, turn/step boundaries, the `agent/request`/`agent/step-result`/`agent/turn-continuation` waterfalls) is in [architecture.md § Event taxonomy](../architecture.md#event-taxonomy). + +## `ToolDefinition` + +The one pipeline-authoring type that is core: what every registered tool *is* — a model-facing `ToolSchema` plus an `execute` function and optional UI presenters. A tool author rarely constructs it by hand (the `defineTool` DSL builds it with typed args), but it is the contract the registry holds and the loop dispatches through. + +Its full fields, the `defineTool`/`SchemaSpec`/`InferArgs` typed schema DSL, the `ToolExecution`/`ToolExecutionResult` waterfall shapes, and the tool-presentation UI vocabulary are on **[tools.md](tools.md)**. diff --git a/docs/core-data-structures/llm-streaming.md b/docs/core-data-structures/llm-streaming.md new file mode 100644 index 0000000000..2e3d4a0838 --- /dev/null +++ b/docs/core-data-structures/llm-streaming.md @@ -0,0 +1,66 @@ +# LLM Streaming + +The wire-level streaming vocabulary of [dsh-llm](../../packages/llm). [core.md](core.md) introduces `StreamChunk`, `Message`, and `ContentBlock`; this page owns the full chunk protocol, the adapter contract every adapter must obey, and the shared assembler. + +Source: [`packages/llm/src/types.ts`](../../packages/llm/src/types.ts) + +## `StreamChunk` — the raw protocol + +A streaming response interleaves several typed blocks (text, reasoning, multiple tool calls). `index` ties each delta to its block; `block-end` carries the fully-assembled `ContentBlock` so consumers don't have to re-assemble deltas themselves. It is a **closed** discriminated union — a `switch` over `type` ends with `assertNever`, so adding a variant breaks compilation at every consumer that must handle it. + +```ts type-equiv +type StreamChunk = + | { type: 'block-start'; index: number; blockType: ContentBlockType } + | { type: 'text-delta'; index: number; text: string } + | { type: 'reasoning-delta'; index: number; text: string } + | { type: 'tool-call-delta'; index: number; id: CallId; name?: string; argumentsDelta: string } + | { type: 'block-end'; index: number; block: ContentBlock } + | { type: 'usage'; usage: TokenUsage } + | { type: 'finish'; reason: FinishReason } +``` + +## The adapter contract + +Every adapter MUST obey these, and every consumer may rely on them: + +- **`usage` before `finish`, nothing after `finish`.** Defer both to the provider's end-of-stream marker so a trailing usage-only chunk can't violate the ordering. +- **Tool-call `arguments` stay raw JSON strings end-to-end.** Partial fragments stream via `argumentsDelta`; a provider that hands back parsed objects re-stringifies at `block-end`. +- **Two sanctioned error paths.** A failure may either THROW from `stream()` (transport/protocol errors) **or** end the stream with `finish {kind:'error'|'aborted'}` (provider in-band errors, for adapters that can't throw mid-stream). Consumers must handle *both*. The agent loop translates a finish-error/aborted into a turn error — it never logs a normal completed assistant message for a failed step. + +This contract is why two adapters exist as a deliberate pair: `dsh-llm-deepseek` (hand-rolled fetch/SSE) and `dsh-llm-pi-ai` (the same endpoint through `@earendil-works/pi-ai`). Two independent internals over one contract is what pinned the protocol down — the library-backed adapter can't throw mid-stream, so it exercises the finish-chunk error path the hand-rolled one might not. + +## `TokenUsage` + +Per-call token accounting. Counts are **disjoint**: `inputTokens` is uncached input only; cached input is reported separately, and billed input is the sum of the three. Adapters whose providers fold cache hits into a single prompt total (DeepSeek's `prompt_tokens`) subtract them back out. + +```ts type-equiv +interface TokenUsage { + inputTokens: number + outputTokens: number + cacheReadTokens?: number + cacheWriteTokens?: number + reasoningTokens?: number +} +``` + +## `BlockAssembler` + +`BlockAssembler` ([`packages/llm/src/assembler.ts`](../../packages/llm/src/assembler.ts)) is the single shared implementation that folds a `StreamChunk` stream back into `ContentBlock`s and a final `Message`. The loop logs the raw chunks (for replay fidelity) while feeding the same chunks through an assembler — so the canonical log keeps token-level detail and the derived message is rebuilt deterministically. A consumer that needs the assembled result without re-implementing the fold uses this. + +## The seam + +`LlmAdapter` is the provider seam: subclass, implement `stream()`, register with `ctx.llm.registerAdapter(models, adapter)`. The `block-start` / `block-end` `index` correlation and the assembler together mean an adapter only has to emit well-formed chunks — block reassembly is not each adapter's problem. The consumer surface (`ctx.llm.stream()` / `streamBlocks()` / `generate()`) and the `llm/stream` waterfall are described in [architecture.md § The vocabulary](../architecture.md#the-vocabulary-dsh-llm). + +`ContentBlockType` (the key set the `index`-correlated blocks carry) derives from `ContentBlockMap`: + +```ts type-equiv +interface ContentBlockMap { + 'text': TextBlock + 'reasoning': ReasoningBlock + 'tool-call': ToolCallBlock + 'tool-result': ToolResultBlock + 'image': ImageBlock +} +``` + +See [core.md § Content blocks and messages](core.md#content-blocks-and-messages) for the block interfaces. diff --git a/docs/core-data-structures/persistence.md b/docs/core-data-structures/persistence.md new file mode 100644 index 0000000000..f232fae1c9 --- /dev/null +++ b/docs/core-data-structures/persistence.md @@ -0,0 +1,63 @@ +# Session Persistence + +The **durability seam** for the event log. [session.md](session.md) describes the in-memory `Session` — the append-only `SessionEvent` log that is the source of truth. This page describes how that log is made durable: the abstract `SessionPersistence` service, its backends, the flush checkpoint, crash recovery, and the metadata header that travels alongside the log. + +The seam is a textbook [capability seam](../rfc/implemented/2026-06-13-capability-seams.md): one abstract service ([dsh-session-persistence](../../packages/session-persistence), `ctx.sessionPersistence`) defining create/append/load/list/has/delete over the existing `SessionEvent` — **no parallel persisted type** — and two interchangeable backends that pass the same `runPersistenceContract` suite. See the [session-persistence RFC](../rfc/implemented/2026-06-14-session-persistence.md). + +## The flush checkpoint + +`session/event` is a *synchronous* notification; persistence plugins buffer it (write-behind) and drain at the awaited `session/flush` checkpoint the loop fires at every turn end. Flush is `ctx.parallel` (awaited): a turn's events are durably committed before the next turn starts, and the turn boundary is the commit boundary. A rejecting flush is reported via `agent/error` and the logger — never as a session event (it would land past the commit boundary), so the backend keeps its buffered events for the next flush. + +## Crash recovery preserves an interrupted turn + +A backend that reloads a log crashed mid-turn finds an open `turn/start` with no `turn/end`. It does **not** truncate — a single turn can be huge in a long-horizon task (many steps, large tool output), and those events were durably appended before the crash. Instead it closes the orphaned turn with a synthetic `turn/end { reason: { kind: 'interrupted' } }`, keeping the log balanced and the turn-enclosure invariant intact. `interrupted` is the one `TurnEndReason` no loop emits (see [session.md](session.md#why-a-turn-ended-turnendreasonmap)). + +## `SessionHeader` — metadata beside the log + +Per-session metadata travels **separately** from the event log: format version, cwd, and lineage are storage concerns, not conversation events, so they stay out of `SessionEventMap` and never reach `deriveMessages()`. The header is attached to a `Session` via `session.header`. + +Source: [`packages/session/src/types.ts`](../../packages/session/src/types.ts) + +```ts type-equiv +interface SessionHeader { + /** On-disk format version; a persistence backend rejects unknown versions. */ + version: number + /** The session's id (mirrors the {@link Session}'s id). */ + id: SessionId + /** Unix epoch milliseconds when the session was created. */ + createdAt: number + /** Absolute working directory the session was created in (if any). */ + cwd?: string + /** The session this one was forked from (seed lineage), if any. */ + parentSession?: SessionId +} +``` + +## `CreateSessionOptions` — seeding and metadata + +Creating a `Session` through the store takes a `seed` (replay/fork an existing event log) and `meta` (the storage-level fields the store folds into a `SessionHeader`). The store fills in `version`/`id` and defaults `createdAt`; the caller supplies the validated absolute `cwd`, the `parentSession` lineage, and — only when reconstructing a persisted session — the original `createdAt` to preserve it. + +```ts type-equiv +interface CreateSessionOptions { + /** Events to seed the new session with (replay/fork). */ + seed?: SessionEvent[] + /** + * Creation metadata. The store fills in `version`/`id` and defaults + * `createdAt` to now; the caller supplies the storage-level fields (validated + * absolute `cwd`, `parentSession` lineage, and — when reconstructing a + * persisted session — the original `createdAt` to preserve it). + */ + meta?: { cwd?: string; parentSession?: SessionId; createdAt?: number } +} +``` + +Replay/fork is therefore `ctx.sessions.create(id, { seed: seedEvents })`; resuming a *persisted* session into a live agent is `ctx.agents.resume({ resumeSessionId })`. + +## The backends + +Both implement the same abstract `SessionPersistence` (create/append/load/list/has/delete over `SessionEvent`) and pass `runPersistenceContract`, proving the seam is genuinely backend-agnostic: + +- **[dsh-session-persistence-jsonl](../../packages/session-persistence-jsonl)** — an append-only JSONL log per session with crash-safe atomic writes, the interrupted-turn crash recovery above, and a read/replay path. +- **[dsh-session-persistence-sqlite](../../packages/session-persistence-sqlite)** — `node:sqlite`, one row per `SessionEvent`. The row shape `(session_id, seq, type, time, data)` maps 1:1 onto the event, so there is no parallel persisted schema to keep in sync. + +Multiple backends sharing one on-disk session coordinate writes through the [shared persistence write-coordinator](../rfc/implemented/2026-06-18-shared-persistence-write-coordinator.md). diff --git a/docs/core-data-structures/session.md b/docs/core-data-structures/session.md new file mode 100644 index 0000000000..5803641c27 --- /dev/null +++ b/docs/core-data-structures/session.md @@ -0,0 +1,119 @@ +# Sessions + +The in-memory, event-sourced model of [dsh-session](../../packages/session). A `Session` is an **append-only log** of typed `SessionEvent`s — the single source of truth for an agent's whole interaction history. The LLM message history is *derived* from the log, never stored separately; replay is re-derivation from the same events. How the log is made **durable** (the persistence seam, backends, crash recovery) is the sibling concern on [persistence.md](persistence.md). + +Source: [`packages/session/src/types.ts`](../../packages/session/src/types.ts) + +## `SessionEventMap` — the event vocabulary + +The append-only event types. Merge-extensible: a plugin (e.g. compaction) declares extra event types via declaration merging. + +```ts type-equiv +interface SessionEventMap { + 'turn/start': { turn: number; trigger: TurnTrigger } + 'turn/end': { turn: number; reason: TurnEndReason } + 'step/start': { turn: number; step: number } + 'step/end': { turn: number; step: number } + /** A user-visible prompt (queued message drained at turn start). */ + 'user/message': { content: ContentBlock[]; source: MessageSource } + /** + * In-session context injection (file-change notices, subdir AGENTS.md, + * skill content, cron notifications, …). Rendered into the derived history + * as tagged synthetic context — NOT a user prompt. + */ + 'context/message': { content: ContentBlock[]; source: MessageSource } + /** Raw stream chunk — token-level replay fidelity. */ + 'assistant/chunk': { turn: number; step: number; chunk: StreamChunk } + /** Assembled assistant message for one step (derived history uses this). */ + 'assistant/message': { turn: number; step: number; content: ContentBlock[] } + 'tool/call': { turn: number; step: number; callId: CallId; name: string; arguments: string } + 'tool/result': { turn: number; step: number; callId: CallId; content: ContentBlock[]; isError: boolean; error?: { name: string; code: string } } + /** Steering content injected between steps of a running turn. */ + 'steering/message': { turn: number; content: ContentBlock[]; source: MessageSource } + 'usage': { turn: number; step: number; usage: TokenUsage } + 'error': { turn: number; step: number; message: string; code?: string } +} +``` + +## `SessionEvent` — one log entry + +A proper discriminated union over `type` (not independent `type`/`data` unions), so `switch (event.type)` narrows `event.data` without casts. `seq` is the monotonic position in the log (`seq = log.length`); `time` is epoch ms. + +```ts type-equiv +type SessionEvent = { + [K in SessionEventType]: { + type: K + /** Monotonic sequence number within the session. */ + seq: number + /** Unix epoch milliseconds. */ + time: number + data: SessionEventMap[K] + } +}[T] +``` + +`SessionEventType = keyof SessionEventMap`. Because `SessionEventMap` is merge-extensible, switches over `SessionEvent` must NOT use `assertNever` — a plugin-added variant is a valid unknown value; handle the known cases and fall through `default`. + +## Derived history: `deriveMessages()` + +`Session.deriveMessages()` projects the event log into the `Message[]` the model sees. The projection rules: + +- `user/message` → a user message. +- `assistant/message` → an assistant message. Raw `assistant/chunk` events are replay/UI data and are **skipped** in derivation (the assembled message is authoritative). +- `tool/result` → a user message carrying a `tool-result` block. +- `context/message`, `steering/message` → user-role messages wrapped in a tagged envelope (`…`) at their chronological position — the "system-reminder" pattern; the model distinguishes them from real prompts by the envelope. + +Everything else (`turn/*`, `step/*`, `usage`, `error`) is structural/telemetry and does not project into a message. + +## What started a turn: `TurnTriggerMap` + +```ts type-equiv +interface TurnTriggerMap { + message: { kind: 'message'; source: MessageSource } + continuation: { kind: 'continuation' } + /** + * An out-of-band context injection (`agent.inject()`) made while the agent + * was idle. The loop wraps the injected `context/message` in a one-shot turn + * (`turn/start` → `context/message` → `turn/end`) so every event in the log + * stays turn-enclosed — the durability/replay boundary is the turn, and a + * bare event between turns would otherwise be indistinguishable from a crash + * tail on reload. + */ + injection: { kind: 'injection'; source: MessageSource } +} +``` + +## Why a turn ended: `TurnEndReasonMap` + +```ts type-equiv +interface TurnEndReasonMap { + completed: { kind: 'completed' } + aborted: { kind: 'aborted'; reason?: string } + error: { kind: 'error'; message: string; code?: string } + disposed: { kind: 'disposed' } + 'max-tokens': { kind: 'max-tokens' } + /** + * The turn never ended on its own: the process crashed mid-turn and a + * persistence backend later closed the orphaned (open) turn on reload so the + * log stays balanced. SYNTHESIZED by the backend's crash-recovery repair — no + * loop ever emits this. Its events are real (they were durably appended before + * the crash) and are PRESERVED, not discarded: a single turn can be huge in a + * long-horizon task (many steps, large tool output), so truncating it would + * lose real work. The marker records that the turn was cut short, not that the + * model completed it. See the session-persistence RFC. + */ + interrupted: { kind: 'interrupted' } +} +``` + +`max-tokens` mirrors the model-call `FinishReason` of the same name: any `max-tokens` step in a turn makes the whole turn end `max-tokens` (the cut-short fact wins over a later continuation), so a consumer can tell a clean stop from a truncated one. `interrupted` is the one reason no loop emits — it is synthesized by crash recovery (see [persistence.md](persistence.md)). Both maps are merge-extensible. + +## The turn-enclosure invariant + +Every session event lives **inside** a turn (between a `turn/start` and its `turn/end`). The loop appends queued `user/message` events *after* `turn/start`, and an idle `agent.inject()` wraps its `context/message` in a one-shot `injection` turn. This makes the turn the single durability/replay boundary: a backend can treat anything after the last `turn/end` as an interrupted-crash tail without risking the loss of legitimately-recorded between-turn context. The `dsh-invariants` plugin enforces it in dev (a message event outside an open turn throws). See [the turn-enclosure invariant RFC](../rfc/implemented/2026-06-15-turn-enclosure-invariant.md). + +## Durability contract + +What a persistence backend relies on: the durable log persists every event verbatim, **including** `assistant/chunk` — `seq` must stay contiguous, so chunks cannot be filtered out of the canonical log. All `event.data` must be JSON-serializable; `Session.append` enforces this at the source (throwing on non-serializable data), so a bad event never enters the log and `session.events` always equals what a backend can persist. Adding an event type that carries non-serializable data, or that breaks the turn/step nesting the invariants plugin checks, is a breaking change to the on-disk format. + +The backends that consume this contract are on [persistence.md](persistence.md). diff --git a/docs/core-data-structures/tools.md b/docs/core-data-structures/tools.md new file mode 100644 index 0000000000..3fdc70c1d5 --- /dev/null +++ b/docs/core-data-structures/tools.md @@ -0,0 +1,112 @@ +# Tools + +The tool pipeline of [dsh-tools](../../packages/tools). [core.md](core.md) introduces `ToolDefinition` as the one pipeline-authoring type promoted to the spine and `ToolSchema` as the model-facing wire shape. This page owns the full `ToolDefinition`, the typed schema DSL that builds it, the waterfall execution shapes, and the UI-presentation vocabulary. + +Source: [`packages/tools/src/index.ts`](../../packages/tools/src/index.ts) · [`packages/tools/src/schema.ts`](../../packages/tools/src/schema.ts) + +## `ToolDefinition` — a registered tool + +A `ToolSchema` (the model-facing fields) plus the `execute` function and optional UI presenters. The registry holds these; the loop dispatches calls through them. The registry's `schemas()` builds the model-facing `ToolSchema[]` by an explicit allowlist — `execute`/`presentCall`/`presentResult` must never leak into a model request. + +```ts type-equiv +interface ToolDefinition extends ToolSchema { + execute(args: unknown, exec: ToolExecution): Promise + /** + * Optional: how to present the PENDING state of one call in a UI, derived + * from the call's `args` (parsed arguments, `unknown` — the tool validates/ + * narrows its own input). Returning `undefined` (or omitting the method) tells + * a UI to fall back to a generic presentation (title = tool name, raw args as + * input). Pure and side-effect-free: a UI may call it during live streaming + * AND a session-log replay, so it must depend only on `args`. + */ + presentCall?(args: unknown): ToolCallPresentation | undefined + /** + * Optional: how to present the COMPLETED state, given the same `args` and the + * `result` (`execute`'s content + whether it errored). Returning `undefined` + * (or omitting the method) tells a UI to keep the pending title and render the + * raw result content. Pure and side-effect-free for the same replay reason. + */ + presentResult?(args: unknown, result: ToolResult): ToolResultPresentation | undefined +} +``` + +`execute` receives `args: unknown` — a raw `ToolDefinition` validates its own input. First-party tools don't write that by hand; they use `defineTool`, which validates and narrows for them. + +## The typed schema DSL + +Plugin authors write per-property specs with a boolean `required: true`, and a type-level helper maps the spec to the `execute` argument type — zero casts. The DSL is *machinery that types* `ToolDefinition`; it is intentionally a sub-page detail, not core. + +Source: [`packages/tools/src/schema.ts`](../../packages/tools/src/schema.ts) + +```ts type-equiv +interface SchemaProp { + type: SchemaType + /** Per-property required flag (NOT the JSON Schema top-level required array). */ + required?: true + /** Human-readable description, surfaced in the JSON Schema as well. */ + description?: string + /** Enum of allowed values (strings only). */ + enum?: string[] + /** Default value. */ + default?: unknown + /** Nested properties for type: 'object'. */ + properties?: SchemaSpec + /** Items schema for type: 'array'. */ + items?: SchemaProp +} +``` + +```ts type-equiv +type SchemaSpec = Record +``` + +`SchemaType` is the primitive union `'string' | 'number' | 'boolean' | 'object' | 'array'`. `InferArgs` maps a `SchemaSpec` to the TS argument type — `required: true` props become required keys, everything else genuinely optional: + +```ts type-equiv +type InferArgs = Simplify< + & { [K in RequiredKeys]: InferPropValue } + & { [K in Exclude>]?: InferPropValue } +> +``` + +`defineTool({ name, description, parameters, execute, … })` ties it together: `parameters` is a `SchemaSpec`, `execute(args, exec)` gets `args: InferArgs`, and the helper converts the spec to JSON Schema (`schemaSpecToJsonSchema`) for the wire and validates model-generated args (`validateArgs`) before the typed body runs. A mismatch throws `ToolArgsError` (`code: 'INVALID_ARGS'`), which the registry turns into an `isError` result so the model can self-correct. Why a custom DSL and not schemastery: tool parameters need JSON Schema (the LLM wire format), not validation/transformation — the lightweight DSL gives the best authoring DX with the smallest surface. + +## Execution: the `tools/execute` waterfall shapes + +`ctx.tools.execute()` runs each call through the `tools/execute` waterfall — the single seam where sandbox, permission, hook, and plan-mode plugins wrap or veto. The pending call is a `ToolExecution`; the outcome is a `ToolExecutionResult`. + +```ts type-equiv +interface ToolExecution { + callId: CallId + name: string + /** Parsed JSON arguments (unknown — tools validate their own input). */ + arguments: unknown + /** The agent on whose behalf the call runs (set by the agent loop). */ + agent?: Agent + signal?: AbortSignal +} +``` + +```ts type-equiv +interface ToolExecutionResult { + callId: CallId + content: ContentBlock[] + isError: boolean + /** + * Set when the call failed with a {@link HarnessError}: machine-routable + * `{ name, code }` for retry/sandbox plugins and replay. The model-facing + * text in `content` is always present; this is extra structure for code. + */ + error?: ToolErrorInfo +} +``` + +A waterfall listener receives `(exec, next)`: call `next()` to proceed (possibly around your own logic), or return a `ToolExecutionResult` without calling `next()` to veto. An unregistered tool routes through the same catch as a tool-thrown error, so both failure classes get a structured `{ name, code }` (`ToolNotFoundError` → `UNKNOWN_TOOL`) — the loop records a failed tool call instead of failing the whole turn. + +## Tool-presentation UI vocabulary + +How a tool wants its call shown in a UI (an editor tool-call card, a CLI log line), provider-neutral so a tool describes itself without depending on any client protocol. `presentCall` returns a `ToolCallPresentation` (pending state: `title`, `kind`, `rawInput`, `content`, optional `terminal`); `presentResult` returns a `ToolResultPresentation` (completed state: replacement `title`, reformatted `content`, terminal `output`/exit). `ToolCallKind` (`'read' | 'edit' | 'delete' | 'move' | 'search' | 'execute' | 'fetch' | 'other'`) picks an icon. A `ToolTerminal` asks a capable UI to render the call as a terminal card (cwd header, output, exit-status pill). + +> These shapes carry a `FIXME(tool-presentation)` in source: they grew incrementally and the call-vs-result terminal split is muddy. Before more tools/UIs depend on them, they will be redesigned (a tagged union over card kinds) and pinned in an RFC, migrating `dsh-tool-bash` and the ACP bridge together. Treat the field-level shapes here as provisional; the source is authoritative. + +The full presentation field docs live in [`packages/tools/src/index.ts`](../../packages/tools/src/index.ts). The bash tool's own schemas (`bash`/`bash_output`/`bash_kill`) and the executor they drive are on [bash.md](bash.md). diff --git a/docs/development.md b/docs/development.md index 2270aa0436..3199893c3d 100644 --- a/docs/development.md +++ b/docs/development.md @@ -93,16 +93,18 @@ pnpm run typecheck # build declarations, then typecheck source, tests, and pnpm run lint # eslint . pnpm run lint:fix # eslint . --fix pnpm run doc-typecheck # compile checked TypeScript snippets in Markdown docs -pnpm run verify-event-taxonomy # compare docs/architecture.md event names with source +pnpm run gen-cordis-catalog # regenerate docs/cordis-catalog/events-and-services.md from source +pnpm run verify-cordis-catalog # fail if the cordis events/services catalog is stale pnpm run verify-md-wrap # fail on hard-wrapped prose paragraphs in docs/README markdown -pnpm run doc-sync # doc-typecheck, event taxonomy, and markdown wrap verification +pnpm run verify-type-equiv # fail if a ```ts type-equiv doc block drifts from its source type +pnpm run doc-sync # doc-typecheck, cordis-catalog freshness, markdown wrap/link, and type-equiv verification pnpm run gen-module-graph # regenerate docs/module-graph.md from package peerDeps pnpm run verify-module-graph # fail if docs/module-graph.md is stale pnpm run build # build declarations and JS bundles pnpm run hygiene # knip, publint, and workspace constraints ``` -When changing package public behavior, update the relevant README or JSDoc in the same change. `pnpm run doc-sync` catches checked TypeScript snippets, event-taxonomy drift, and hard-wrapped markdown prose, but broader prose/API sync still needs review. +When changing package public behavior, update the relevant README or JSDoc in the same change. `pnpm run doc-sync` catches checked TypeScript snippets, cordis events/services catalog drift, and hard-wrapped markdown prose, but broader prose/API sync still needs review. ## Demos @@ -128,6 +130,16 @@ Use one of three comment tags to flag known issues in the code, ordered by urgen Pick the tag that matches the urgency so anyone scanning the code can tell a release blocker from a someday-maybe. +## Documenting types verbatim (`ts type-equiv`) + +The [core data structures](core-data-structures/core.md) docs paste real type definitions so a reader sees the exact shape. To keep a paste from drifting when source changes, fence it as ` ```ts type-equiv ` (instead of ` ```ts `) and register it in `scripts/type-equiv.manifest.json` with the source file and symbol it mirrors: + +```json +{ "doc": "docs/core-data-structures/session.md", "symbol": "SessionEvent", "source": "packages/session/src/types.ts" } +``` + +`pnpm run verify-type-equiv` (part of `doc-sync`) then extracts that symbol's declaration from source via the TypeScript parser and asserts the block matches it (whitespace- and comment-insensitive, so a doc block may show a clean definition and the prose can carry the semantics). It also enforces a 1:1 correspondence: every `ts type-equiv` block has exactly one manifest entry and vice-versa, so a block can't go silently unchecked and a stale entry can't linger. `doc-typecheck` skips `ts type-equiv` blocks (they aren't standalone-compilable) and excludes them from its opt-out ratio. When you change a documented type, the gate fails until you update the paste; when you add or remove a block, update the manifest in the same change. + ## Architecture context Read `docs/architecture.md` before changing anything under `packages/`. The codebase is built around Cordis plugins, event-sourced sessions, typed service seams, and explicit extension points. diff --git a/docs/rfc/README.md b/docs/rfc/README.md index 7098ea216e..c1299bb23a 100644 --- a/docs/rfc/README.md +++ b/docs/rfc/README.md @@ -73,6 +73,8 @@ Do NOT write one for a mechanical or local choice (a variable name, a one-file r | [Drop the mutable session summary](implemented/2026-06-19-drop-mutable-session-summary.md) | 2026-06-19 | | [Shared persistence write coordinator](implemented/2026-06-18-shared-persistence-write-coordinator.md) | 2026-06-18 | | [Agent lifecycle and ownership seams](implemented/2026-06-18-agent-lifecycle-and-ownership-seams.md) | 2026-06-18 | +| [Core-data-structures catalog and the `ts type-equiv` drift gate](implemented/2026-06-20-core-data-structures-catalog.md) | 2026-06-20 | +| [Generated cordis events + services catalog](implemented/2026-06-20-generated-cordis-catalog.md) | 2026-06-20 | ## Rejected diff --git a/docs/rfc/implemented/2026-06-11-doc-sync-enforcement.md b/docs/rfc/implemented/2026-06-11-doc-sync-enforcement.md index 68e6f0a03d..9abc7c08b0 100644 --- a/docs/rfc/implemented/2026-06-11-doc-sync-enforcement.md +++ b/docs/rfc/implemented/2026-06-11-doc-sync-enforcement.md @@ -13,7 +13,7 @@ AGENTS.md promises that docs and code stay strictly in sync, but the promise was Two gates, mirroring the existing `scripts/` style (tsx ESM, one job each): 1. **`doc-typecheck`** extracts every fenced ` ```ts ` block from `README.md`, `docs/**`, and `packages/*/README.md`, writes them to a temp project, and compiles with `tsc --noEmit`. The temp tsconfig copies only resolution-relevant options and the workspace `paths` map from `tsconfig.typecheck.json` (vendor → built `lib`, harness → `src`) — resolving vendor to `lib` is essential, or tsc type-checks raw vendor source and floods the run. A block that is a deliberate sketch opts out with an explicit ` ```ts ignore-check ` info string; the script reports the opt-out ratio and fails if it exceeds half, so the escape hatch can't quietly become the norm. -2. **`verify-event-taxonomy`** extracts the event names from the `interface Events` blocks across `packages/*/src` and from the taxonomy table in `docs/architecture.md`, and asserts the two sets match exactly. Verify, don't generate: the table keeps its hand-written Mode/Purpose columns; only the set of names is checked. (Landing this surfaced three events the table had been missing — `tools/change`, `llm/adapter-change`, `system-prompt/change`.) +2. **`verify-event-taxonomy`** extracts the event names from the `interface Events` blocks across `packages/*/src` and from the taxonomy table in `docs/architecture.md`, and asserts the two sets match exactly. Verify, don't generate: the table keeps its hand-written Mode/Purpose columns; only the set of names is checked. (Landing this surfaced three events the table had been missing — `tools/change`, `llm/adapter-change`, `system-prompt/change`.) **Superseded** by [the generated cordis catalog](2026-06-20-generated-cordis-catalog.md): this gate and its `architecture.md` table are retired in favor of a fully-generated `docs/cordis-catalog/events-and-services.md` and its `verify-cordis-catalog` freshness gate. The other gates here (`doc-typecheck`, and the `verify-md-wrap` amendment below) are unaffected. Both run via a shared `doc-sync` package.json script that the lefthook pre-push hook and CI both invoke ([mechanical quality gates](2026-06-11-quality-gates.md): hooks and CI call the same scripts, so the gate fires locally before a push — not only after it). They run after `pnpm run typecheck` (which emits the vendor `lib/` that doc-typecheck resolves against). API-extractor golden reports ([the deferred API-extractor-reports proposal](../proposed/2026-06-11-api-extractor-reports.md)) were deliberately **deferred** — low value for an internal monorepo where reviewers already see the source diff, and a heavy, finicky dependency. diff --git a/docs/rfc/implemented/2026-06-20-core-data-structures-catalog.md b/docs/rfc/implemented/2026-06-20-core-data-structures-catalog.md new file mode 100644 index 0000000000..5d4209b732 --- /dev/null +++ b/docs/rfc/implemented/2026-06-20-core-data-structures-catalog.md @@ -0,0 +1,56 @@ +# RFC: Core-data-structures catalog and the `ts type-equiv` drift gate + +Status: implemented (accepted 2026-06-20) + + + +## Context + +A reader trying to understand the harness could find its *behavior* in [architecture.md](../../architecture.md) (the service map, the session/turn/step lifecycle, the event taxonomy) but had no single place describing its *vocabulary* — the data structures that behavior moves around. The type shapes lived only in source, scattered across `packages/*/src/types.ts`, so understanding "what is a `Message`, a `SessionEvent`, a `StreamChunk`" meant reading the declarations directly. A prose catalog would help, but a catalog that paraphrases or paste-copies type definitions rots the instant a field changes — and an out-of-sync type doc is worse than none, because a reader trusts it. + +So the work had two intertwined questions: **what belongs in such a catalog** (the scoping problem — a harness has dozens of cross-package types and dumping all of them helps no one), and **how to keep pasted type definitions from drifting** (the durability problem). This RFC records both decisions. Its sibling, [the generated cordis events + services catalog](2026-06-20-generated-cordis-catalog.md), is the *wiring*-axis complement: this one catalogs the data structures, that one the events and services that move them. + +## Decision + +A new `docs/core-data-structures/` folder catalogs the vocabulary, with a new `verify-type-equiv` doc-sync gate that keeps every pasted type definition byte-identical to its source. + +### What counts as "core" — the spine-vs-seam line + +The scoping line was not picked top-down; it was discovered by testing candidate definitions against concrete borderline types until one rule survived every case. The decisive test was `BashExecRequest`/`BashExecSpec`/`BashRunResult`: bash is a capability *seam*, not part of the agent-loop spine, so if those are "core" then "core" means *all cross-package vocabulary* and the catalog is a flat dump; if they are not, "core" means *the central spine* and bash vocabulary belongs on a sub-page. The latter won, which set the whole structure: a **tiered folder**, not a flat document. + +The rule that settled the remaining cases: ***the type you write, hold, or receive is core; the machinery that types it, renders it, or persists it is a sub-page detail.*** Worked through: + +- A data structure is **core** if it flows through the agent-loop spine — the loop holds, derives, streams, or logs it on every turn regardless of which plugins load (`Message`, `StreamChunk`, `SessionEvent`, the `Agent` handle) — **or** it is the single headline type a plugin author writes against a pipeline (`ToolDefinition`). +- `ToolDefinition` is core (it is what every tool author writes) **even though the loop never holds one** — authoring-importance overrides the strict flows-through-spine rule for this one headline type. But its typing machinery — the `SchemaSpec`/`InferArgs` DSL — is a sub-page detail (you write a `ToolDefinition`; the type-level machinery that types it you do not). That is the spine-vs-seam line made sharp. +- `ToolSchema` is core (it is a field of `GenerateOptions`, the model request that flows through every step) even though it is conceptually part of the tool pipeline — *flows through the spine* wins over *conceptual home* when they conflict. +- The tool-presentation vocabulary (`ToolCallPresentation`, …, carrying a `FIXME(tool-presentation)` redesign marker), the `SessionPersistence` durability seam, and bash vocabulary are sub-pages. + +`core.md` is a **self-contained spine doc**: it states the exact type definition of each spine structure with minimal prose and links to sub-pages for the per-seam detail. The sub-pages are `llm-streaming.md`, `session.md`, `persistence.md` (split from session along the in-memory-model vs. durability-seam line), `tools.md`, and `bash.md`. + +### The `ts type-equiv` mechanism — literal AND drift-proof + +The durability requirement was specific: the doc should show the **literal** current type definition (so a reader sees the real shape, not a paraphrase) **and** be mechanically guaranteed to match source. The repo already compiles fenced ` ```ts ` blocks (`doc-typecheck`), but a real typechecked block needs import noise and proves only *assignability*, not *byte-equality* — a renamed field with the same type would pass. So: + +- Type definitions are pasted verbatim into a dedicated ` ```ts type-equiv ` fence. `doc-typecheck` recognizes the fence and skips it (a bare definition is not standalone-compilable), and **excludes it from the opt-out ratio** — it is a separately-checked category, not an unchecked sketch. +- A new `scripts/verify-type-equiv.ts` extracts each block via the TypeScript parser and asserts a **verbatim source match** against the declared symbol — chosen over a compiled `_Check` assertion precisely because byte-equality, not assignability, is the property we want. +- Provenance lives in a central `scripts/type-equiv.manifest.json` (`{ doc, symbol, source }` entries), **not** in directive comments in the prose. The script enforces a **1:1 correspondence**: every type-equiv block has exactly one manifest entry and vice versa, so a block can never be silently unchecked and an entry can never rot. +- Wired into `doc-sync`, so it runs in the same lefthook pre-push and CI paths as the other doc gates. + +### Maintenance is the author's job, with a gate backstop + +`verify-type-equiv` catches a *drifted paste* of an already-documented type, but it cannot tell you a brand-new core type went undocumented. So AGENTS.md and the `dsh-code-review` skill were updated to require keeping the catalog in sync when a change adds or reshapes a documented type — the gate handles drift, the human handles new surface. + +## Process + +The design was driven entirely by a one-question-at-a-time grilling that walked the scoping decision tree through concrete examples (`BashExecRequest`, `ToolSchema`, `ToolDefinition`, the schema DSL, the presentation types, the session/persistence split) before committing to the spine-vs-seam rule — the rule was the *output* of the examples, not an a-priori axiom. The implementation landed as four commits mirroring the structure of the work: the gate (`e97f94b`), the catalog (`7e33c7b`), the maintenance-guard updates (`53e01a0`), and a review-fix commit (`6da7a0f`). + +That last commit is why the process is worth recording: an independent Codex review (gpt-5.5:xhigh) found a real **scan-gap bug** — `verify-type-equiv` only scanned the docs the manifest named, so a type-equiv block added to an *unmanifested* doc was silently skipped, defeating the 1:1 guarantee in one direction. The fix scans every doc in the markdown scope and reports an unmanifested block as an orphan. The same review corrected a `SessionPersistence` surface-listing prose error (`has`/`delete`) and the `doc-sync` command summary. The bug is the point: a drift gate that silently skips part of its input is worse than no gate, and only an adversarial reader caught it. + +This decision shipped in #71 **without** an RFC at the time — the judgment was that the `ts type-equiv` convention was small enough to document in `development.md`. This RFC is the retroactive record: the spine-vs-seam scoping rule and the verbatim-match-over-assignability choice are exactly the kind of "why was it done this way?" decisions a future maintainer would otherwise re-litigate, and its sibling catalog ([generated cordis events + services](2026-06-20-generated-cordis-catalog.md)) does carry an RFC, so the pair should be documented symmetrically. + +## Consequences + +- The vocabulary now has a single home that **cannot silently drift**: a field rename in source fails `verify-type-equiv` in the pre-push hook and CI until the paste is refreshed. +- The spine-vs-seam line is a reusable scoping tool, not a one-off: the same "the thing you write/hold/receive is core; the machinery that types/renders/persists it is a detail" rule is what later scoped the events/services catalog's harness-vs-inherited tiering. +- The `ts type-equiv` fence is a third doc-block category alongside ` ```ts ` (compiled) and ` ```ts ignore-check ` (sketch). A later sibling added a fourth, ` ```ts cordis-catalog ` (generated signature), reusing the same skip-and-exclude treatment. +- Adding or reshaping a core type now carries a documentation obligation the author must honor (the gate cannot detect a missing *new* type), backstopped by the `dsh-code-review` checklist. diff --git a/docs/rfc/implemented/2026-06-20-generated-cordis-catalog.md b/docs/rfc/implemented/2026-06-20-generated-cordis-catalog.md new file mode 100644 index 0000000000..ca57c2eadb --- /dev/null +++ b/docs/rfc/implemented/2026-06-20-generated-cordis-catalog.md @@ -0,0 +1,35 @@ +# RFC: Generated cordis events + services catalog + +Status: implemented (accepted 2026-06-20) + + + +## Context + +A plugin author needs two reference surfaces that no single document gave them: every cordis **event** they can listen to (with its exact signature and dispatch mode) and every `ctx.` **service** they can call (with its exact interface). The pieces existed but were scattered — a hand-maintained event-taxonomy *table* in `docs/architecture.md` (names + prose Mode/Purpose, name-set-checked by `verify-event-taxonomy`), a Service-map table (8 rows of role prose), and the `interface Events` / `interface Context` declarations themselves. The taxonomy table also could not catch a brand-new *undocumented* event: a name-set verifier only checks the names that are already in the table on both sides. + +This is the wiring-axis complement to the [core-data-structures catalog](../../core-data-structures/core.md) ([its RFC](2026-06-20-core-data-structures-catalog.md)): that one catalogs the *data structures* the loop moves around (verified hand-pastes); this one catalogs the *events and services* that move them. + +## Decision + +Generate the catalog from source instead of hand-maintaining a table and verifying a subset. + +`scripts/gen-cordis-catalog.ts` walks the `interface Events` and `interface Context` declarations (plus the service classes) with the TypeScript compiler API and emits `docs/cordis-catalog/events-and-services.md` — one `## Events` section (grouped by scope, each event rendered as signature + mode badge + its source JSDoc) and one `## Services` section (each `ctx.` with its public method signatures + class JSDoc). It mirrors the `gen-module-graph` pattern exactly: `--write` regenerates, `--check` fails if the committed file is stale, output is deterministic (sorted), and the file is a build artifact that is never hand-edited. `verify-cordis-catalog` (the `--check`) runs inside `doc-sync`, so the freshness gate fires in the same lefthook pre-push and CI paths as every other doc gate. + +Pure generation is correct here because the codebase is disciplined enough that the AST is the whole truth: every event/service name is a string literal that round-trips to a static declaration — there are no dynamically-named events and no runtime-only services. So a generated doc cannot be wrong, and it closes the undocumented-event gap structurally (generation enumerates source rather than checking a hand-written subset). + +Specific choices: + +- **`@mode` tag, cross-checked.** Each harness event's JSDoc carries an explicit `@mode emit|waterfall|parallel` tag; the generator hard-errors on a missing tag. Where the signature shape is conclusive — a trailing `next: () => …` parameter is structurally a waterfall — it asserts the tag agrees and hard-errors on a contradiction. The emit-vs-parallel distinction is not structurally visible (`session/flush` returns `Promise | void` with no `next`), so it is trusted from the tag. The authoring rule lives in [AGENTS.md](../../../AGENTS.md). +- **Tiered scope.** The harness tier (the 8 `@deepseek-ai/dsh-*` services + their events) is rendered in full from source. The inherited tier (cordis-core `ctx.on/emit/effect/provide/…` + the `internal/*` events + loader/hmr/timer) is pinned vendor source a plugin also sees; it is rendered tersely (name + one-line + source pointer) from a curated table in the generator, NOT walked from the vendor AST — the cordis-core `Context` mixes true ctx members with non-service fields (`root`, `baseUrl`, `logger`), and the vendor surface changes only on a deliberate vendor sync. +- **Cross-links to the data-structure catalog.** A type name in a signature (`GenerateOptions`, `StreamChunk`, `ToolDefinition`, …) links to the core-data-structures page that documents it. The map is a small hand-curated const in the generator — NOT `type-equiv.manifest.json`, which documents the `…Map` symbols while signatures reference the derived union names, and lists a few symbols on two pages. +- **A dedicated fence.** Signature blocks use a ` ```ts cordis-catalog ` info string that `doc-typecheck` recognizes and skips (a bare signature fragment is not standalone-compilable), excluded from the opt-out ratio — the same treatment `type-equiv` blocks get. + +This **supersedes the event-taxonomy half** of [doc-sync enforcement](2026-06-11-doc-sync-enforcement.md): `verify-event-taxonomy` and its `docs/architecture.md` table are retired (the architecture.md heading stays, its body now points at the catalog; the Service-map role table stays as curated prose). The verify-don't-generate principle that RFC chose for the taxonomy is reversed *for this surface only* — the data here is mechanically complete, so generation is strictly stronger (full signatures, cannot drift, catches undocumented events) than a name-set check of a hand-table. doc-typecheck, verify-md-wrap, verify-md-links, and verify-type-equiv are unchanged. + +## Consequences + +- The catalog cannot drift: a source change that the committed file doesn't reflect fails `verify-cordis-catalog` in the pre-push hook and CI. A new event with no `@mode` tag, or a tag that contradicts its signature, fails the generator outright. +- Event prose now has a single home — the JSDoc at the declaration. Thin JSDoc yields a thin catalog entry, which pressures authors to document at the source (the generator is a forcing function for the AGENTS.md "every export has a semantic JSDoc" rule). +- The inherited tier is hand-summarized, so a vendor sync that adds/renames a cordis-core event or `ctx` member needs a matching edit to the curated table in `gen-cordis-catalog.ts`. This is the deliberate cost of not walking pinned vendor source; it changes rarely and is called out in the generator. +- `verify-event-taxonomy.ts` is deleted and the `docs/architecture.md` event table is gone; anyone who linked to a specific table row now lands on the generated catalog instead. diff --git a/package.json b/package.json index 50319023cf..eeb5dd7d07 100644 --- a/package.json +++ b/package.json @@ -24,13 +24,15 @@ "knip": "knip", "publint": "tsx scripts/publint-all.ts", "doc-typecheck": "tsx scripts/doc-typecheck.ts", - "verify-event-taxonomy": "tsx scripts/verify-event-taxonomy.ts", "verify-md-wrap": "tsx scripts/verify-md-wrap.ts", "verify-md-links": "tsx scripts/verify-md-links.ts", + "verify-type-equiv": "tsx scripts/verify-type-equiv.ts", + "gen-cordis-catalog": "tsx scripts/gen-cordis-catalog.ts", + "verify-cordis-catalog": "tsx scripts/gen-cordis-catalog.ts --check", "gen-module-graph": "tsx scripts/gen-module-graph.ts", "verify-module-graph": "tsx scripts/gen-module-graph.ts --check", "constraints": "tsx scripts/check-workspace-constraints.ts", - "doc-sync": "pnpm run doc-typecheck && pnpm run verify-event-taxonomy && pnpm run verify-md-wrap && pnpm run verify-md-links", + "doc-sync": "pnpm run doc-typecheck && pnpm run verify-cordis-catalog && pnpm run verify-md-wrap && pnpm run verify-md-links && pnpm run verify-type-equiv", "hygiene": "pnpm run knip && pnpm run publint && pnpm run constraints", "demo:echo": "node --expose-internals --import tsx examples/echo-agent/start.ts", "demo:coding": "node --expose-internals --import tsx examples/coding-agent/start.ts", diff --git a/packages/AGENTS.md b/packages/AGENTS.md index 31f42c26c9..735e23a156 100644 --- a/packages/AGENTS.md +++ b/packages/AGENTS.md @@ -13,6 +13,6 @@ Naming notes: - A *service* `src/index.ts` exports the service class as `export default` + all public types; a *function/namespace plugin* `src/index.ts` exports `name`/`inject`/`Config`/`apply` as named exports and NO default (see the plugin-export-shape rule above) - `src/types.ts` contain only types — no runtime code - Tests live at package level under `tests/`, not `src/__tests__/` -- A package's README and module/JSDoc comments are part of the change: when you alter behavior (config keys, defaults, error codes, wire fields), update them in the same commit. CI runs `pnpm run doc-sync`, which typechecks fenced `ts` blocks in `packages/*/README.md`, verifies the event-taxonomy table, and checks markdown wrapping across this file too — but it does NOT catch prose drift (config keys, defaults, error codes), so those stay on the author. +- A package's README and module/JSDoc comments are part of the change: when you alter behavior (config keys, defaults, error codes, wire fields), update them in the same commit. CI runs `pnpm run doc-sync`, which typechecks fenced `ts` blocks in `packages/*/*.md`, regenerates the cordis events/services catalog from the `interface Events` / `interface Context` declarations (failing if the committed copy is stale), and checks markdown wrapping across this file too — but it does NOT catch prose drift (config keys, defaults, error codes), so those stay on the author. A new event needs an `@mode` tag on its JSDoc (the catalog generator hard-errors without it — see the root AGENTS.md). Read the per-package README.md for package-specific details: service API, events, extension points, TODOs. diff --git a/packages/acp/acp-feature-support.md b/packages/acp/acp-feature-support.md new file mode 100644 index 0000000000..04b1a2f258 --- /dev/null +++ b/packages/acp/acp-feature-support.md @@ -0,0 +1,162 @@ +# ACP feature support checklist + +A structured inventory of [Agent Client Protocol](https://agentclientprotocol.com) (ACP) features and where the harness's ACP bridge ([`@deepseek-ai/dsh-acp`](README.md)) stands on each. The bridge exposes the harness agent as an ACP **server** (the agent side of an editor↔agent connection), so "supported" below means *the bridge implements the agent's half* — answering an agent method, advertising a capability, or calling a client method. + +## Scope + +This tracks the **stable** ACP v1 surface (schema `1.14.0`, `schema/v1/schema.json`) PLUS the **unstable/draft** features that the two reference adapters — [`claude-agent-acp`](https://github.com/zed-industries/claude-code-acp) (Claude Code) and [`codex-acp`](https://github.com/zed-industries/codex-acp) (OpenAI Codex) — actually ship. A purely-unstable feature that neither reference adapter uses is omitted (see [Out of scope](#out-of-scope)). + +Legend: ✅ supported · ⚠️ partial / fallback · ❌ not yet · — n/a. The **Stable** column marks whether the feature is in the released v1 schema (S) or only the unstable schema (U). The **Claude** / **Codex** columns record whether each reference adapter ships it, as a maturity signal. + +## At a glance + +The bridge implements the **core prompt-turn loop** for N concurrent sessions: initialize, session new/load, prompt, cancel, streamed assistant/thought chunks, tool-call rendering (including Zed terminal cards), and resumable session replay. The largest **unbuilt** areas are the **permission gate** (`session/request_permission`), **MCP passthrough**, **session modes / config options / model selection**, **slash commands**, and **agent plans** — all of which both reference adapters ship — plus the client **filesystem** and **terminal** method families (which the adapters mostly do NOT drive either — see rows 43-49). See [Gap summary](#gap-summary). + +## 1. Agent methods (client → agent) + +| Method | Stable | Bridge | Claude | Codex | Notes | +|---|---|---|---|---|---| +| `initialize` | S | ✅ | ✅ | ✅ | Negotiates `PROTOCOL_VERSION`; advertises `loadSession` + baseline prompt caps. Snapshots the Zed `_meta.terminal_output` client cap. | +| `authenticate` | S | ⚠️ | ✅ | ✅ | No-op stub; the bridge advertises no `authMethods`, so there is nothing to authenticate. | +| `logout` | S | ❌ | ✅ | ✅ | Gated by `agentCapabilities.auth.logout`; not advertised. | +| `session/new` | S | ✅ | ✅ | ✅ | Maps to `agents.create`; requires an absolute `cwd` (becomes the session workspace); rejects non-empty `additionalDirectories` / `mcpServers`. | +| `session/load` | S | ✅ | ✅ | ✅ | Maps to `agents.resume` + full event-log replay; validates persisted `cwd` before constructing the agent. | +| `session/resume` | S | ❌ | ✅ | ✅ | Reconnect WITHOUT replay; gated by `sessionCapabilities.resume`. Not advertised. | +| `session/close` | S | ❌ | ✅ | ✅ | No `session/close` handler — the SDK dispatch returns `method_not_found`. The bridge tears sessions down on client disconnect / Cordis disposal (cross-cutting, see [§8](#8-cross-cutting)), but that is not the on-demand per-session method. | +| `session/prompt` | S | ✅ | ✅ | ✅ | Maps to `agent.send`; one in-flight prompt per session; settles on the owning turn's end. | +| `session/cancel` | S | ✅ | ✅ | ✅ | Queue-aware `agent.cancel`; settles the in-flight prompt `cancelled`, scoped to the one session. | +| `session/set_mode` | S | ❌ | ✅ | ✅ | Session modes not modeled (see [§6 Modes](#6-session-modes--config-options--models)). | +| `session/set_config_option` | S | ❌ | ✅ | ✅ | Config options not modeled. | +| model selection | S | ❌ | ✅ | ✅ | No distinct stable `session/set_model` — model is the `model`-category `session/set_config_option`. The bridge fixes the model per-bridge via config; no runtime switch. Codex still uses the legacy `unstable_setSessionModel` ext method. | +| `session/list` | S | ❌ | ✅ | ✅ | Gated by `sessionCapabilities.list`. The harness HAS `sessionPersistence.list()` (used internally for load-cwd validation) but does not expose it over ACP. | +| `session/delete` | S | ❌ | ✅ | ✅ | Gated by `sessionCapabilities.delete`. | +| `session/fork` | U | ❌ | ✅ | ❌ | Claude ships `unstable_forkSession`; Codex does not. | + +## 2. Client methods the agent CALLS (agent → client) + +These are capabilities the bridge would *drive* on the editor. The harness runs tools in-process (its own `dsh-bash` executor, direct file I/O), so it does not yet delegate to the editor for any of these. + +| Method | Stable | Bridge | Claude | Codex | Notes | +|---|---|---|---|---|---| +| `session/update` | S | ✅ | ✅ | ✅ | The bridge's primary output channel (see [§4](#4-sessionupdate-variants)). | +| `session/request_permission` | S | ❌ | ✅ | ✅ | **The biggest gap.** Tools run with the executor's full authority; no user authorization round-trip. The `agent→sessionId` reverse map is already in place to route a future permission request. Tracked `TODO(rfc010-permission-gate)`. | +| `fs/read_text_file` | S | ❌ | ✅ | ❌ | The harness reads files directly (it does not see the editor's unsaved buffer state). Claude delegates; Codex does not. | +| `fs/write_text_file` | S | ❌ | ✅ | ❌ | Same — direct writes, no editor delegation. | +| `terminal/create` | S | ❌ | ❌ | ❌ | Neither reference adapter drives the client terminal API either — both, like the bridge, render shell output as tool-call content + a `_meta` channel (see [§5 Terminal](#terminal-rendering)). | +| `terminal/output` | S | ❌ | ❌ | ❌ | As above. | +| `terminal/wait_for_exit` | S | ❌ | ❌ | ❌ | As above. | +| `terminal/kill` | S | ❌ | ❌ | ❌ | As above. | +| `terminal/release` | S | ❌ | ❌ | ❌ | As above. | +| `elicitation/create` · `elicitation/complete` | U | ❌ | ✅ | ⚠️ | Structured user-input forms. Claude calls the `unstable_*` elicitation methods (to surface MCP server elicitations); Codex does NOT — its `CodexElicitationHandler` maps elicitations onto `session/request_permission` instead. | + +## 3. Capabilities + +### 3a. `agentCapabilities` (advertised by the bridge) + +| Capability | Stable | Bridge | Claude | Codex | Notes | +|---|---|---|---|---|---| +| `loadSession` | S | ✅ | ✅ | ✅ | Advertised `true`; backs `session/load`. | +| `promptCapabilities.image` | S | ❌ | ✅ | ✅ | Bridge advertises `image: false`; image prompt blocks are rejected. | +| `promptCapabilities.audio` | S | ❌ | ❌ | ❌ | `audio: false`; neither adapter accepts audio either. | +| `promptCapabilities.embeddedContext` | S | ❌ | ✅ | ✅ | `embeddedContext: false`; embedded `resource` blocks rejected. | +| `mcpCapabilities.{http,sse}` | S | ❌ | ✅ | ⚠️ | No MCP passthrough; `mcpServers` is rejected. Claude advertises http+sse, Codex http only. | +| `sessionCapabilities.*` | S | ❌ | ✅ | ✅ | None advertised (list/delete/resume/close/additionalDirectories/fork all off). | +| `auth.logout` | S | ❌ | ✅ | ✅ | Not advertised. | +| `authMethods[]` | S | ⚠️ | ✅ | ✅ | Advertised as empty (no auth required to reach the model). | +| `agentInfo` (name/version) | S | ✅ | ✅ | ✅ | From `agentName` / `agentVersion` config. | +| `_meta` custom caps | S | ❌ | ✅ | — | E.g. Claude's `claudeCode.promptQueueing`. The bridge advertises no custom `_meta`. | + +### 3b. `clientCapabilities` (consumed by the bridge) + +| Capability | Stable | Bridge | Notes | +|---|---|---|---| +| `fs.{readTextFile,writeTextFile}` | S | ❌ | Not consulted (the bridge never calls `fs/*`). | +| `terminal` | S | ❌ | Not consulted; the bridge keys terminal rendering off the Zed `_meta.terminal_output` cap instead. | +| `_meta.terminal_output` (Zed) | S (`_meta`) | ✅ | Snapshotted per session at create/load; gates terminal-card rendering. | + +## 4. `session/update` variants + +| `sessionUpdate` | Stable | Bridge | Claude | Codex | Notes | +|---|---|---|---|---|---| +| `agent_message_chunk` | S | ✅ | ✅ | ✅ | From `assistant/chunk` text-delta. | +| `agent_thought_chunk` | S | ✅ | ✅ | ✅ | From `assistant/chunk` reasoning-delta. | +| `user_message_chunk` | S | ✅ | ✅ | ✅ | Emitted during `session/load` replay to reconstruct the user side. | +| `tool_call` | S | ✅ | ✅ | ✅ | Tool-owned presentation (`presentCall`); see [§5](#5-tool-call-rendering). | +| `tool_call_update` | S | ✅ | ✅ | ✅ | From `tool/result` via `presentResult`. | +| `plan` | S | ❌ | ✅ | ✅ | No agent plan emitted. Both adapters emit real plan entries (Codex's `CodexEventHandler.updatePlan` maps `turn/plan/updated` → `{ sessionUpdate: 'plan', entries }`). | +| `available_commands_update` | S | ❌ | ✅ | ✅ | No slash commands advertised. | +| `current_mode_update` | S | ❌ | ✅ | ✅ | No session modes. | +| `config_option_update` | S | ❌ | ✅ | ✅ | No config options. | +| `usage_update` | S | ❌ | ✅ | ✅ | Token/cost reporting not surfaced (the harness HAS usage events internally). | +| `session_info_update` | S | ❌ | ⚠️ | ⚠️ | Session title/metadata not pushed. | + +## 5. Tool-call rendering + +Tool-call presentation is **owned by each tool** (`presentCall` / `presentResult` on the `dsh-tools` definition), not special-cased in the bridge — see the [terminal-and-tool-rendering RFC](../../docs/rfc/implemented/2026-06-18-acp-terminal-and-tool-rendering.md). + +| Feature | Stable | Bridge | Claude | Codex | Notes | +|---|---|---|---|---|---| +| `ToolCallKind` mapping | S | ✅ | ✅ | ✅ | `execute`/`read`/`edit`/`other` inferred from the tool; richer mapping possible. | +| `ToolCallStatus` | S | ✅ | ✅ | ✅ | `in_progress` → `completed`/`failed`. | +| `content` blocks | S | ✅ | ✅ | ✅ | Text content; the description renders above the card. | +| `diff` content | S | ❌ | ✅ | ✅ | No structured diff rendering for edits (would need a diffing edit tool + presenter). | +| `terminal` content | S | ✅ | ✅ | ✅ | Via the Zed `_meta` terminal convention (see below), not the spec `terminal/*` sub-protocol. | +| `locations` (follow-along) | S | ❌ | ✅ | ✅ | No file-location hints emitted. | +| `rawInput` | S | ✅ | ⚠️ | ✅ | Parsed tool args surfaced as `rawInput`. | +| `rawOutput` | S | ❌ | ⚠️ | ✅ | Not emitted. | + +### Terminal rendering + +⚠️ Implemented via the **Zed `_meta` convention** (`terminal_info` / `terminal_output` / `terminal_exit`), gated on the client advertising `_meta.terminal_output` — NOT the spec's `terminal/create` sub-protocol (which would make the editor execute the command, bypassing `dsh-bash`'s sandbox / env-scrub / ownership / cwd). Both reference adapters take the same `_meta` approach. Live incremental streaming (`terminal_output_delta`, which Codex negotiates) is a follow-up — the bridge currently sends the full captured output once on the result. + +## 6. Session modes / config options / models + +❌ None modeled. Both reference adapters ship modes (Claude: a "plan" auto-mode; Codex: read-only / agent / agent-full-access mapping to its approval+sandbox policy), the newer config-option surface, and runtime model selection. The harness fixes the model per-bridge via `AcpConfig.model`. These are coupled to the unbuilt **permission gate** (a mode often selects an approval policy), so they are natural follow-ups to it. + +## 7. Content blocks + +| Block | Stable | In prompts | In updates | Notes | +|---|---|---|---|---| +| `text` | S | ✅ | ✅ | Baseline. | +| `resource_link` | S | ✅ | ⚠️ | Accepted in prompts and rendered into text (`acpPromptToText`); not emitted as a structured update block. | +| `image` | S | ❌ | ❌ | Rejected in prompts (`promptCapabilities.image: false`). | +| `audio` | S | ❌ | ❌ | Rejected. | +| `resource` (embedded) | S | ❌ | ❌ | Rejected (`embeddedContext: false`). | + +The bridge rejects unsupported prompt blocks rather than silently dropping them (`promptHasUnsupportedContent`), per the "explicit over implicit" convention. + +## 8. Cross-cutting + +| Feature | Stable | Bridge | Notes | +|---|---|---|---| +| `StopReason` mapping | S | ✅ | `turnEndToStopReason` is total over harness turn-end reasons → `end_turn`/`max_tokens`/`cancelled`. | +| Multi-session (N per connection) | S | ✅ | Strict per-session demux; concurrent streams never interleave. See the [multi-session RFC](../../docs/rfc/proposed/2026-06-14-acp-multi-session.md). | +| Disconnect / disposal teardown | S | ✅ | Quiesces every live session on client disconnect or Cordis disposal. | +| `_meta` extensibility | S | ⚠️ | Consumed (Zed terminal cap) and emitted (terminal `_meta`); no other custom extensions. | +| Background-task ownership isolation | — | ✅ | `bash_output`/`bash_kill` reject another session's task via an opaque owner token. | +| stdout-is-the-protocol guarantee | S | ✅ | The bridge runs in an example with no stdout logger. | + +## Gap summary + +Ranked by how commonly the reference adapters ship them and how much UX they unlock: + +1. **Permission gate** — `session/request_permission` + permission options. Tracked `TODO(rfc010-permission-gate)`; the reverse map is already wired. Foundational, and a prerequisite for modes. +2. **Session lifecycle** — `session/list` + `session/delete` (the persistence layer already lists), then `session/resume` / `session/close`. +3. **Modes / config options / model selection** — coupled to the permission gate. +4. **Agent plan** (`sessionUpdate: 'plan'`) — surface the loop's plan as structured entries. +5. **Slash commands** (`available_commands_update`). +6. **MCP passthrough** (`mcpServers` on `session/new` + `mcpCapabilities`). +7. **Richer prompt content** — image / embedded `resource` blocks (needs a multimodal model path). +8. **Diff + location tool rendering** — `diff` content and `locations` for edit tools. +9. **Usage reporting** (`usage_update`) — the harness already has the internal usage events. +10. **Editor filesystem delegation** (`fs/read_text_file` / `fs/write_text_file`) — lets the agent see unsaved buffers; lower priority since the harness has direct disk access. + +## Out of scope + +Unstable/draft ACP features that **neither** reference adapter ships are not tracked above: `providers/*` (LLM provider selection), `mcp/connect`·`mcp/message`·`mcp/disconnect` (client-side MCP passthrough), `nes/*` (Next Edit Suggestion), `document/did*` (LSP-style document sync), the v2 plan model (`plan_update` / `plan_removed`), boolean config options, `$/cancel_request`, and the draft Streamable-HTTP transport. They can be added if a target editor adopts them. + +## Sources + +- Stable spec: `schema/v1/schema.json` (schema `1.14.0`) and `docs/protocol/v1/*.mdx` in the [agent-client-protocol](https://github.com/agentclientprotocol/agent-client-protocol) repo. +- Reference adapters: [`claude-agent-acp`](https://github.com/zed-industries/claude-code-acp) and [`codex-acp`](https://github.com/zed-industries/codex-acp). +- Bridge: [`README.md`](README.md), [`src/index.ts`](src/index.ts), and the ACP RFCs under [`docs/rfc/`](../../docs/rfc/README.md). diff --git a/packages/agent/src/types.ts b/packages/agent/src/types.ts index 0ebf582bda..d6b35b97a1 100644 --- a/packages/agent/src/types.ts +++ b/packages/agent/src/types.ts @@ -132,48 +132,94 @@ export interface Agent { declare module 'cordis' { interface Events { // ---- lifecycle (emit) ---- - /** An agent was registered. */ + /** + * An agent was registered in the {@link AgentRegistry} and is ready to + * receive messages. + * @mode emit + */ 'agent/created'(agent: Agent): void - /** An agent was disposed. */ + /** + * An agent was disposed and removed from the registry; its fiber and any + * in-flight turn have been torn down. + * @mode emit + */ 'agent/disposed'(agent: Agent): void - /** Agent status changed (idle/running/disposed). */ + /** + * Agent status changed (`idle` ⇄ `running`, or → `disposed`). Drive + * lifecycle off this transition, never off a status you just requested — + * `send()` does not flip status to `running` before it returns. + * @mode emit + */ 'agent/status'(agent: Agent, status: AgentStatus): void /** * A message entered the agent's inbox (queued or steering). `source` is * the resolved source (defaults applied), not the caller's raw options. + * @mode emit */ 'agent/queued'(agent: Agent, content: ContentBlock[], info: { source: MessageSource; steering: boolean }): void // ---- turn/step boundaries (emit) ---- + /** + * A turn began. `turn` is the 1-based turn number within the session. + * @mode emit + */ 'agent/turn-start'(agent: Agent, turn: number): void + /** + * A turn ended. `reason` distinguishes a clean stop from a truncated or + * aborted one (`completed` | `aborted` | `error` | `disposed` | `max-tokens`). + * @mode emit + */ 'agent/turn-end'(agent: Agent, turn: number, reason: TurnEndReason): void + /** + * A step (one model call plus its tool dispatch) began. `step` is 1-based + * within the turn; a turn runs one or more steps. + * @mode emit + */ 'agent/step-start'(agent: Agent, turn: number, step: number): void + /** + * A step ended. + * @mode emit + */ 'agent/step-end'(agent: Agent, turn: number, step: number): void // ---- interception seams (waterfall) ---- /** - * Waterfall: mutate the fully-assembled GenerateOptions before the model - * call (hooks, compaction, model switching, tool filtering, …). + * Waterfall: mutate the fully-assembled {@link GenerateOptions} before the + * model call (hooks, compaction, model switching, tool filtering, …). Call + * `next()` to delegate, or return without it to short-circuit. + * @mode waterfall */ 'agent/request'(agent: Agent, turn: number, step: number, options: GenerateOptions, next: () => Promise): Promise /** - * Waterfall: post-process the assembled assistant message before tool - * dispatch (validation, content rewriting, …). + * Waterfall: post-process the assembled assistant {@link Message} before + * tool dispatch (validation, content rewriting, …). + * @mode waterfall */ 'agent/step-result'(agent: Agent, turn: number, step: number, message: Message, next: () => Promise): Promise /** * Waterfall: override the turn-continuation decision. The default * (computed by the loop) is `hadToolCalls || steeringInjected`. Listeners * can force-continue (/goal, /loop) or force-stop (budget guards). + * @mode waterfall */ 'agent/turn-continuation'(agent: Agent, turn: number, defaultDecision: boolean, next: () => Promise): Promise // ---- streaming + tool notifications (emit) ---- - /** A raw stream chunk arrived (token-level UI/log feed). */ + /** + * A raw {@link StreamChunk} arrived from the model (token-level UI/log feed). + * @mode emit + */ 'agent/stream-chunk'(agent: Agent, turn: number, step: number, chunk: StreamChunk): void - /** Steering content was injected into a running turn. */ + /** + * Steering content was injected into a running turn. + * @mode emit + */ 'agent/steering'(agent: Agent, turn: number, content: ContentBlock[], source: MessageSource): void - /** A step or turn errored. */ + /** + * A step or turn errored. The loop reports a failure here (plus the logger) + * even when the error has no in-turn position for a session `error` event. + * @mode emit + */ 'agent/error'(agent: Agent, turn: number, step: number, error: Error): void } } diff --git a/packages/agent/tests/gen-cordis-catalog.spec.ts b/packages/agent/tests/gen-cordis-catalog.spec.ts new file mode 100644 index 0000000000..66edb9983f --- /dev/null +++ b/packages/agent/tests/gen-cordis-catalog.spec.ts @@ -0,0 +1,83 @@ +/** + * Negative-path tests for the cordis catalog generator (`scripts/gen-cordis-catalog.ts`). + * + * The generated catalog is frozen by a regenerate-and-diff freshness gate, so + * the freshness half is exercised by `pnpm run verify-cordis-catalog` in CI. + * What a freshness diff CANNOT prove is that the generator REJECTS malformed + * source the way it promises to — a missing `@mode` tag, or a tag that + * contradicts the signature shape. These tests drive `collectEvents()` against + * synthetic fixture packages to prove each guard fires (and that a well-formed + * event passes), mirroring the drift-guard negative tests for verify-type-equiv. + */ + +import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, describe, expect, it } from 'vitest' +import { collectEvents } from '../../../scripts/gen-cordis-catalog.ts' + +/** Write a fixture package exposing one `interface Events` block and return the + * scan root to hand `collectEvents`. */ +function fixtureRoot(eventsBlock: string): string { + const root = mkdtempSync(join(tmpdir(), 'cordis-catalog-')) + const dir = join(root, 'packages', 'fix', 'src') + mkdirSync(dir, { recursive: true }) + writeFileSync( + join(dir, 'index.ts'), + `declare module 'cordis' {\n interface Events {\n${eventsBlock}\n }\n}\n`, + ) + return root +} + +const roots: string[] = [] +const make = (block: string): string => { + const r = fixtureRoot(block) + roots.push(r) + return r +} + +afterEach(() => { + while (roots.length) rmSync(roots.pop()!, { recursive: true, force: true }) +}) + +describe('gen-cordis-catalog collectEvents', () => { + it('extracts a well-formed event with its @mode and JSDoc', () => { + const events = collectEvents(make( + ' /**\n * A thing happened.\n * @mode emit\n */\n \'fix/happened\'(id: string): void', + )) + expect(events).toHaveLength(1) + expect(events[0]).toMatchObject({ name: 'fix/happened', scope: 'fix', mode: 'emit', doc: 'A thing happened.' }) + }) + + it('classifies a trailing-next signature as a waterfall', () => { + const events = collectEvents(make( + ' /**\n * Intercept it.\n * @mode waterfall\n */\n \'fix/intercept\'(x: number, next: () => Promise): Promise', + )) + expect(events[0]?.mode).toBe('waterfall') + }) + + it('accepts a parallel (awaited, no next) event by trusting the tag', () => { + const events = collectEvents(make( + ' /**\n * Flush.\n * @mode parallel\n */\n \'fix/flush\'(): Promise | void', + )) + expect(events[0]?.mode).toBe('parallel') + }) + + it('hard-errors when an event is missing its @mode tag', () => { + expect(() => collectEvents(make( + ' /** No mode here. */\n \'fix/untagged\'(id: string): void', + ))).toThrow(/missing an @mode tag/) + }) + + it('hard-errors when @mode contradicts a trailing-next (waterfall) shape', () => { + expect(() => collectEvents(make( + ' /**\n * Mislabeled.\n * @mode emit\n */\n \'fix/wrong\'(x: number, next: () => Promise): Promise', + ))).toThrow(/trailing 'next' parameter .* tagged '@mode emit'/) + }) + + it('hard-errors when @mode waterfall has no trailing next to delegate to', () => { + expect(() => collectEvents(make( + ' /**\n * Not actually a waterfall.\n * @mode waterfall\n */\n \'fix/nonext\'(id: string): void', + ))).toThrow(/tagged '@mode waterfall' but has no trailing 'next'/) + }) +}) diff --git a/packages/llm/src/index.ts b/packages/llm/src/index.ts index 460316ea50..aaa0f66460 100644 --- a/packages/llm/src/index.ts +++ b/packages/llm/src/index.ts @@ -23,11 +23,23 @@ declare module 'cordis' { } interface Events { - /** Waterfall around every streaming model call (retry, caching, routing). */ + /** + * Waterfall around every streaming model call (retry, caching, routing). + * Bound to the {@link LlmService}; call `next()` to reach the resolved + * adapter's stream, or yield your own chunks to short-circuit. + * @mode waterfall + */ 'llm/stream'(this: LlmService, options: GenerateOptions, next: () => AsyncIterable): AsyncIterable - /** Waterfall around every non-streaming model call. */ + /** + * Waterfall around every non-streaming model call. Bound to the + * {@link LlmService}; call `next()` to delegate to the adapter. + * @mode waterfall + */ 'llm/generate'(this: LlmService, options: GenerateOptions, next: () => Promise): Promise - /** An adapter was registered or unregistered. */ + /** + * An adapter was registered or unregistered (the model→adapter map changed). + * @mode emit + */ 'llm/adapter-change'(): void } } diff --git a/packages/session/src/index.ts b/packages/session/src/index.ts index 57210c3431..65fbe01015 100644 --- a/packages/session/src/index.ts +++ b/packages/session/src/index.ts @@ -23,15 +23,24 @@ declare module 'cordis' { } interface Events { - /** A session was created in the store. */ + /** + * A session was created in the store. + * @mode emit + */ 'session/created'(session: Session): void - /** An event was appended to a session log (sync, fire-and-forget). */ + /** + * An event was appended to a session log (sync, fire-and-forget). This is + * the per-append feed a UI or invariant plugin tails. + * @mode emit + */ 'session/event'(session: Session, event: SessionEvent): void /** * Awaited durability checkpoint. The agent loop awaits * `ctx.parallel('session/flush', session)` at every turn end; persistence - * plugins (JSONL, SQLite) drain their write-behind - * buffers here and on fiber dispose. + * plugins (JSONL, SQLite) drain their write-behind buffers here and on + * fiber dispose. Awaited (parallel), not a waterfall: every listener runs + * and the loop waits for all of them, but none can veto. + * @mode parallel */ 'session/flush'(session: Session): Promise | void } diff --git a/packages/system-prompt/src/index.ts b/packages/system-prompt/src/index.ts index 25d87d2194..a5e6e4ed78 100644 --- a/packages/system-prompt/src/index.ts +++ b/packages/system-prompt/src/index.ts @@ -15,9 +15,18 @@ declare module 'cordis' { } interface Events { - /** Waterfall around prompt assembly — mutate/extend the assembly. */ + /** + * Waterfall around prompt assembly — mutate or extend the + * {@link PromptAssembly} (sections + tool schemas) before it is rendered. + * Bound to the {@link SystemPrompt} service; call `next()` to delegate. + * @mode waterfall + */ 'system-prompt/assemble'(this: SystemPrompt, assembly: PromptAssembly, next: () => Promise): Promise - /** A section or tool provider was registered or unregistered. */ + /** + * A section or tool provider was registered or unregistered (the assembly + * inputs changed). + * @mode emit + */ 'system-prompt/change'(): void } } diff --git a/packages/tools/src/index.ts b/packages/tools/src/index.ts index eee77445eb..5a17aa2b0c 100644 --- a/packages/tools/src/index.ts +++ b/packages/tools/src/index.ts @@ -36,11 +36,15 @@ declare module 'cordis' { * Waterfall around every tool execution — the single seam where sandbox, * permission, hook, and plan-mode plugins wrap or veto a call. Listeners * receive `(exec, next)`: call `next()` to proceed (possibly around your - * own logic), or return a ToolExecutionResult without calling `next()` - * to short-circuit (veto). + * own logic), or return a {@link ToolExecutionResult} without calling + * `next()` to short-circuit (veto). + * @mode waterfall */ 'tools/execute'(this: ToolRegistry, exec: ToolExecution, next: () => Promise): Promise - /** A tool was registered or unregistered. */ + /** + * A tool was registered or unregistered (the available tool set changed). + * @mode emit + */ 'tools/change'(): void } } diff --git a/scripts/doc-typecheck.ts b/scripts/doc-typecheck.ts index 10068eb792..905adfb630 100644 --- a/scripts/doc-typecheck.ts +++ b/scripts/doc-typecheck.ts @@ -8,7 +8,14 @@ * build is required first). A block that is a deliberate sketch rather than * compilable code opts out with an explicit ` ```ts ignore-check ` info string * — the opt-out is visible in the source, and this script reports the ratio so - * the escape hatch can't quietly become the norm. + * the escape hatch can't quietly become the norm. A third info string, + * doc-typecheck.ts recognizes two more fence variants and skips both (each is a + * separately-checked category, not an unchecked sketch, so neither counts in the + * opt-out ratio): ` ```ts type-equiv ` is a verbatim source-type paste that + * `scripts/verify-type-equiv.ts` drift-checks, and ` ```ts cordis-catalog ` is a + * generated event/service signature fragment in the cordis catalog (a bare + * signature is not standalone-compilable; the catalog is generated and frozen by + * `scripts/gen-cordis-catalog.ts` + its `--check` freshness gate). * * Run: `tsx scripts/doc-typecheck.ts`. */ @@ -20,23 +27,40 @@ import { glob } from 'node:fs/promises' const root = resolve(import.meta.dirname, '..') +/** + * How a fenced block participates in this gate: + * - `check` (` ```ts `) — compiled. + * - `ignore` (` ```ts ignore-check `) — a deliberate sketch; skipped, and + * counted in the opt-out ratio so the escape hatch can't quietly take over. + * - `type-equiv` (` ```ts type-equiv `) — a verbatim paste of a source type + * definition, drift-checked by `scripts/verify-type-equiv.ts` against the + * source symbol. Skipped HERE (it is not standalone-compilable — no imports) + * and EXCLUDED from the opt-out ratio: it is a separate fully-checked + * category, not an unchecked sketch. + * - `cordis-catalog` (` ```ts cordis-catalog `) — a generated event/service + * signature fragment in the cordis catalog. Skipped HERE for the same reason + * (a bare signature fragment has no imports and does not stand alone) and + * EXCLUDED from the opt-out ratio: the catalog is generated and frozen by + * `scripts/gen-cordis-catalog.ts` + its `--check` freshness gate. + */ +type BlockKind = 'check' | 'ignore' | 'type-equiv' | 'cordis-catalog' + /** One extracted code block. */ interface Block { file: string /** 1-based line of the opening fence. */ line: number - /** `true` when the fence is ` ```ts ignore-check ` (skip compilation). */ - ignored: boolean + kind: BlockKind code: string } -/** Extract every ```ts / ```ts ignore-check block from one Markdown file. */ +/** Extract every ts / ts ignore-check / ts type-equiv / ts cordis-catalog block from one Markdown file. */ function extractBlocks(absPath: string): Block[] { const text = readFileSync(absPath, 'utf8') const lines = text.split('\n') const file = relative(root, absPath) const blocks: Block[] = [] - let open: { line: number; ignored: boolean; body: string[] } | null = null + let open: { line: number; kind: BlockKind; body: string[] } | null = null lines.forEach((raw, i) => { const fence = /^```(\s*)(\S.*)?$/.exec(raw) @@ -46,15 +70,19 @@ function extractBlocks(absPath: string): Block[] { } if (open) { // closing fence - blocks.push({ file, line: open.line, ignored: open.ignored, code: open.body.join('\n') }) + blocks.push({ file, line: open.line, kind: open.kind, code: open.body.join('\n') }) open = null return } // opening fence — only care about ts blocks const info = (fence[2] ?? '').trim() - if (info === 'ts' || info === 'ts ignore-check') { - open = { line: i + 1, ignored: info === 'ts ignore-check', body: [] } - } + const kind: BlockKind | null = + info === 'ts' ? 'check' + : info === 'ts ignore-check' ? 'ignore' + : info === 'ts type-equiv' ? 'type-equiv' + : info === 'ts cordis-catalog' ? 'cordis-catalog' + : null + if (kind) open = { line: i + 1, kind, body: [] } }) return blocks } @@ -97,7 +125,7 @@ function tempTsconfig(): string { }) } -const markdownGlobs = ['README.md', 'docs/**/*.md', 'packages/*/README.md'] +const markdownGlobs = ['README.md', 'docs/**/*.md', 'packages/*/*.md'] const files: string[] = [] for (const pattern of markdownGlobs) { @@ -106,8 +134,14 @@ for (const pattern of markdownGlobs) { files.sort() const all = files.flatMap(extractBlocks) -const checked = all.filter(b => !b.ignored) -const ignored = all.filter(b => b.ignored) +const checked = all.filter(b => b.kind === 'check') +const ignored = all.filter(b => b.kind === 'ignore') +// `type-equiv` and `cordis-catalog` blocks are verified elsewhere +// (verify-type-equiv.ts and the gen-cordis-catalog `--check` freshness gate), +// not here: neither compiled nor counted toward the opt-out ratio (each is a +// separate fully-checked category, not an unchecked sketch). The ratio's +// denominator is therefore the compile-eligible blocks only. +const ratioDenominator = checked.length + ignored.length if (checked.length === 0) { console.log('doc-typecheck: no ts code blocks to check.') @@ -139,11 +173,12 @@ try { process.exit(1) } - const ratio = ignored.length / all.length - console.log(`doc-typecheck: ${checked.length} block(s) compiled, ${ignored.length} ignored (${(ratio * 100).toFixed(0)}% opt-out).`) + const ratio = ignored.length / ratioDenominator + const skipped = all.length - ratioDenominator + console.log(`doc-typecheck: ${checked.length} block(s) compiled, ${ignored.length} ignored (${(ratio * 100).toFixed(0)}% opt-out), ${skipped} type-equiv/cordis-catalog (checked elsewhere).`) // Guard against the escape hatch becoming the norm. - if (all.length >= 4 && ratio > 0.5) { - console.error(`doc-typecheck: too many blocks opt out of checking (${ignored.length}/${all.length}). Make them compile or delete them.`) + if (ratioDenominator >= 4 && ratio > 0.5) { + console.error(`doc-typecheck: too many blocks opt out of checking (${ignored.length}/${ratioDenominator}). Make them compile or delete them.`) process.exit(1) } } finally { diff --git a/scripts/gen-cordis-catalog.ts b/scripts/gen-cordis-catalog.ts new file mode 100644 index 0000000000..49c451bc20 --- /dev/null +++ b/scripts/gen-cordis-catalog.ts @@ -0,0 +1,464 @@ +/** + * Generate (and verify) the cordis events + services catalog in + * docs/cordis-catalog/events-and-services.md. + * + * The catalog is the WIRING-axis reference: every cordis event a plugin can + * listen to (exact signature + dispatch mode) and every `ctx.` service it + * can call (exact public interface). It complements the core-data-structures + * catalog (the VOCABULARY axis — the types these signatures move around). + * + * The catalog is FULLY GENERATED from source — never hand-edit it. The codebase + * is disciplined enough that a pure-AST pass captures the whole truthful + * surface: every event/service is a string literal that round-trips to a static + * `interface Events` / `interface Context` declaration (no dynamically-named + * events, no runtime-only services). So the committed file is a build artifact + * and a regenerate-and-diff freshness check (`--check`) makes drift structurally + * impossible. Because generation enumerates source rather than checking a + * hand-written subset, a brand-new event cannot be silently undocumented — it + * appears in the next regenerate, and an un-regenerated file fails `--check`. + * + * `tsx scripts/gen-cordis-catalog.ts` → write the catalog + * `tsx scripts/gen-cordis-catalog.ts --check` → exit 1 if the committed file + * is stale (CI / pre-push gate) + * + * The HARNESS tier (the `@deepseek-ai/dsh-*` events + services) is rendered in + * full from source: signature, the `@mode` badge, and the declaration's JSDoc. + * Every harness event MUST carry an `@mode emit|waterfall|parallel` tag — the + * generator hard-errors on a missing tag, and where the signature shape is + * conclusive (a trailing `next: () => …` parameter is structurally a waterfall) + * it asserts the tag agrees and hard-errors on a contradiction. The INHERITED + * tier (cordis core + loader/hmr/timer) is pinned vendor source a plugin author + * also sees; it is rendered tersely (name + one-line + source pointer) from a + * curated table in this script, NOT elevated to the harness tier's prominence. + * + * Signature fences use the ` ```ts cordis-catalog ` info string: doc-typecheck + * recognizes it and skips compilation (the signatures are fragments, not + * standalone-compilable, like the ` ```ts type-equiv ` blocks). + */ + +import { globSync, readFileSync, writeFileSync } from 'node:fs' +import { resolve } from 'node:path' +import ts from 'typescript' + +const root = resolve(import.meta.dirname, '..') +const OUT = 'docs/cordis-catalog/events-and-services.md' + +/** The fenced-block info string for generated signature blocks (skipped by + * doc-typecheck, since a bare signature fragment is not standalone-compilable). */ +const FENCE = 'ts cordis-catalog' + +/** A dispatch mode, rendered as the badge after an event name. */ +type Mode = 'emit' | 'waterfall' | 'parallel' + +/** + * Cross-link map: a type name that appears in a signature → the + * core-data-structures page that documents it (path relative to OUT's folder). + * Hand-curated and catalog-owned, NOT derived from type-equiv.manifest.json — + * that manifest documents the `…Map` symbols (`ContentBlockMap`) while + * signatures reference the derived UNION names (`ContentBlock`), and it lists a + * few symbols on two pages. Here each name resolves to exactly one PRIMARY page. + */ +const LINK_MAP: Record = { + Agent: 'core.md', + ContentBlock: 'core.md', + Message: 'core.md', + MessageSource: 'core.md', + GenerateOptions: 'core.md', + GenerateResult: 'core.md', + SessionEvent: 'core.md', + StreamChunk: 'llm-streaming.md', + TurnEndReason: 'session.md', + ToolDefinition: 'tools.md', + ToolExecution: 'tools.md', + ToolExecutionResult: 'tools.md', + BashExecRequest: 'bash.md', + BashExecSpec: 'bash.md', + BashRunResult: 'bash.md', + BashTask: 'bash.md', + BashTaskRead: 'bash.md', +} + +/** One harness event, extracted from an `interface Events` block. */ +interface EventEntry { + /** Scoped name, e.g. `agent/request`. */ + name: string + /** The scope prefix, e.g. `agent` (everything before the first `/`). */ + scope: string + /** Full signature text (the method-signature member, JSDoc stripped). */ + signature: string + /** Dispatch mode from the `@mode` tag. */ + mode: Mode + /** Description prose (JSDoc minus the `@mode` tag), one line per paragraph. */ + doc: string + /** Source pointer `packages/…/file.ts:line` of the declaration. */ + source: string +} + +/** One harness service, extracted from an `interface Context` block. */ +interface ServiceEntry { + /** The `ctx.` name, e.g. `llm`. */ + key: string + /** The service class/interface name, e.g. `LlmService`. */ + type: string + /** Whether the service class is abstract (a seam interface). */ + abstract: boolean + /** Class-level JSDoc prose, one line per paragraph. */ + doc: string + /** Public method signatures (bodies stripped), in source order. */ + methods: string[] + /** Source pointer of the class declaration. */ + source: string +} + +/** A terse inherited-tier entry (pinned vendor surface). */ +interface InheritedEntry { + name: string + summary: string + /** Source pointer `vendor/…:line`. */ + source: string +} + +/** Repo-relative source pointer `file:line` for a node's first character. */ +function pointer(rel: string, sf: ts.SourceFile, node: ts.Node): string { + const { line } = sf.getLineAndCharacterOfPosition(node.getStart(sf)) + return `${rel}:${line + 1}` +} + +/** The raw `/** … *​/` JSDoc block immediately preceding a node, or '' if none. */ +function rawJsDoc(text: string, node: ts.Node): string { + const ranges = ts.getLeadingCommentRanges(text, node.getFullStart()) ?? [] + const jsdoc = ranges.filter(r => text.slice(r.pos, r.pos + 3) === '/**').at(-1) + return jsdoc ? text.slice(jsdoc.pos, jsdoc.end) : '' +} + +/** + * Parse a raw JSDoc block into description prose + the `@mode` tag (when + * present). Output obeys the repo's markdown conventions so the generated file + * passes verify-md-wrap: each prose paragraph collapses to ONE physical line, + * and a `-` bullet list is preserved with each item on its own single line + * (continuation lines folded in). `{@link Foo}` unwraps to `Foo`; `@`-tag lines + * other than `@mode` end the current prose run. + */ +function parseJsDoc(raw: string): { doc: string; mode: Mode | null } { + const inner = raw + .replace(/^\/\*\*/, '') + .replace(/\*\/$/, '') + .split('\n') + .map(l => l.replace(/^\s*\*?\s?/, '').replace(/\s+$/, '')) + let mode: Mode | null = null + const blocks: string[] = [] + let para: string[] = [] + let list: string[] = [] + let item: string[] = [] + const join = (parts: string[]): string => parts.join(' ').replace(/\s+/g, ' ').trim() + const flushItem = (): void => { + if (item.length) list.push(join(item)) + item = [] + } + const flushList = (): void => { + flushItem() + if (list.length) blocks.push(list.join('\n')) // one block, items on own lines + list = [] + } + const flushPara = (): void => { + flushList() + if (para.length) blocks.push(join(para)) + para = [] + } + for (const line of inner) { + const m = /^@mode\s+(emit|waterfall|parallel)\s*$/.exec(line) + if (m) { mode = m[1] as Mode; continue } + if (line.startsWith('@')) { flushPara(); continue } // other tags end the prose + if (line.trim() === '') { flushPara(); continue } + if (/^-\s+/.test(line)) { + // A list item starts: a pending paragraph (e.g. an intro line directly + // above the list, no blank between) flushes FIRST so it renders above. + flushItem() + if (para.length) { blocks.push(join(para)); para = [] } + item.push(line) + continue + } + if (item.length) { item.push(line); continue } // continuation of current item + para.push(line) + } + flushPara() + const doc = blocks.join('\n\n').replace(/\{@link\s+([^}]+)\}/g, '$1').trim() + return { doc, mode } +} + +/** Find the `declare module 'cordis'` body in a source file, or null. */ +function cordisModuleBody(sf: ts.SourceFile): ts.ModuleBlock | null { + for (const stmt of sf.statements) { + if (ts.isModuleDeclaration(stmt) && ts.isStringLiteral(stmt.name) && stmt.name.text === 'cordis') { + if (stmt.body && ts.isModuleBlock(stmt.body)) return stmt.body + } + } + return null +} + +/** The signature text of a method-signature member (everything but a body). */ +function memberSignature(member: ts.TypeElement | ts.ClassElement, sf: ts.SourceFile): string { + const full = member.getText(sf) + const body = (member as { body?: ts.Node }).body + const sig = body ? full.slice(0, full.length - body.getText(sf).length) : full + return sig.replace(/\s*;?\s*$/, '').replace(/\s+/g, ' ').trim() +} + +/** Walk every harness `interface Events` block and extract its events. + * `scanRoot` defaults to the repo root; tests pass a fixture dir. */ +export function collectEvents(scanRoot: string = root): EventEntry[] { + const entries: EventEntry[] = [] + for (const rel of globSync('packages/*/src/*.ts', { cwd: scanRoot }).sort()) { + const abs = resolve(scanRoot, rel) + const text = readFileSync(abs, 'utf8') + if (!text.includes('interface Events')) continue + const sf = ts.createSourceFile(abs, text, ts.ScriptTarget.Latest, true) + const body = cordisModuleBody(sf) + if (!body) continue + for (const stmt of body.statements) { + if (!ts.isInterfaceDeclaration(stmt) || stmt.name.text !== 'Events') continue + for (const member of stmt.members) { + if (!ts.isMethodSignature(member)) continue + const name = ts.isStringLiteral(member.name) ? member.name.text : member.name.getText(sf) + const signature = memberSignature(member, sf) + const { doc, mode } = parseJsDoc(rawJsDoc(text, member)) + const src = pointer(rel, sf, member) + if (!mode) { + throw new Error(`gen-cordis-catalog: event '${name}' (${src}) is missing an @mode tag. Add '@mode emit|waterfall|parallel' to its JSDoc (see AGENTS.md).`) + } + // Conclusive structural check: a trailing `next: () => …` parameter is a + // waterfall. (emit vs parallel is not structurally distinguishable, so + // it is trusted from the tag.) + const last = member.parameters.at(-1) + const hasNext = !!last && last.name.getText(sf) === 'next' + if (hasNext && mode !== 'waterfall') { + throw new Error(`gen-cordis-catalog: event '${name}' (${src}) has a trailing 'next' parameter (structurally a waterfall) but is tagged '@mode ${mode}'. Fix the tag or the signature.`) + } + if (!hasNext && mode === 'waterfall') { + throw new Error(`gen-cordis-catalog: event '${name}' (${src}) is tagged '@mode waterfall' but has no trailing 'next' parameter. A waterfall delegates via next().`) + } + entries.push({ name, scope: name.split('/')[0] ?? name, signature, mode, doc, source: src }) + } + } + } + return entries +} + +/** Walk every harness `interface Context` block + its service class. + * `scanRoot` defaults to the repo root; tests pass a fixture dir. */ +export function collectServices(scanRoot: string = root): ServiceEntry[] { + const entries: ServiceEntry[] = [] + for (const rel of globSync('packages/*/src/index.ts', { cwd: scanRoot }).sort()) { + const abs = resolve(scanRoot, rel) + const text = readFileSync(abs, 'utf8') + if (!text.includes('interface Context')) continue + const sf = ts.createSourceFile(abs, text, ts.ScriptTarget.Latest, true) + const body = cordisModuleBody(sf) + if (!body) continue + // The ctx key → type mapping(s) declared in this file's interface Context. + const keyToType = new Map() + for (const stmt of body.statements) { + if (!ts.isInterfaceDeclaration(stmt) || stmt.name.text !== 'Context') continue + for (const member of stmt.members) { + if (!ts.isPropertySignature(member) || !member.type) continue + const key = member.name.getText(sf) + keyToType.set(key, member.type.getText(sf)) + } + } + if (keyToType.size === 0) continue + // Find each service class declared in the same file and emit an entry. + for (const [key, type] of keyToType) { + const cls = sf.statements.find( + (s): s is ts.ClassDeclaration => ts.isClassDeclaration(s) && s.name?.text === type, + ) + if (!cls) continue // a Pick-mixin member (e.g. timer helpers), not a class here + const abstract = cls.modifiers?.some(m => m.kind === ts.SyntaxKind.AbstractKeyword) ?? false + const methods: string[] = [] + for (const member of cls.members) { + if (!ts.isMethodDeclaration(member)) continue + // Only the PUBLIC callable surface a `ctx.` consumer sees. Drop + // private/protected (a protected method like `notifyTaskDone` is a + // subclass hook, not something a plugin calls through `ctx.bash`) and + // static (not reachable through the instance). + const nonPublic = member.modifiers?.some(m => + m.kind === ts.SyntaxKind.PrivateKeyword + || m.kind === ts.SyntaxKind.ProtectedKeyword + || m.kind === ts.SyntaxKind.StaticKeyword) + || ts.isPrivateIdentifier(member.name) + if (nonPublic) continue + const memberName = member.name.getText(sf) + if (memberName.startsWith('[')) continue // computed/symbol members + methods.push(memberSignature(member, sf)) + } + entries.push({ + key, + type, + abstract, + doc: parseJsDoc(rawJsDoc(text, cls)).doc, + methods, + source: pointer(rel, sf, cls), + }) + } + } + return entries.sort((a, b) => a.key.localeCompare(b.key)) +} + +/** + * The inherited tier — cordis core + loader/hmr/timer. Curated, terse, and + * hand-summarized because (a) it is pinned vendor source that changes only on a + * deliberate vendor sync, (b) the cordis-core `Context` mixes true ctx members + * with non-service fields (`root`, `baseUrl`, `logger`) that a blind walk would + * wrongly surface as services, and (c) the internal/* events carry no JSDoc to + * render. Source pointers are verified against vendor by `verify-md-links`' + * sibling check is N/A; keep them current on a vendor bump. + */ +const INHERITED_EVENTS: InheritedEntry[] = [ + { name: 'internal/plugin', summary: 'A plugin fiber was created.', source: 'vendor/cordis/src/events.ts:197' }, + { name: 'internal/status', summary: 'A fiber changed lifecycle state.', source: 'vendor/cordis/src/events.ts:198' }, + { name: 'internal/service', summary: 'Interception hook for a service binding (no core producer).', source: 'vendor/cordis/src/events.ts:199' }, + { name: 'internal/update', summary: 'Waterfall: a fiber config update is being applied.', source: 'vendor/cordis/src/events.ts:200' }, + { name: 'internal/get', summary: 'Waterfall: a service is being read from the store.', source: 'vendor/cordis/src/events.ts:201' }, + { name: 'internal/set', summary: 'Waterfall: a service is being written to the store.', source: 'vendor/cordis/src/events.ts:202' }, + { name: 'internal/listener', summary: 'A listener was registered.', source: 'vendor/cordis/src/events.ts:203' }, + { name: 'internal/dispatch', summary: 'An event is being dispatched to listeners.', source: 'vendor/cordis/src/events.ts:204' }, + { name: 'hmr/change', summary: 'A watched source file changed on disk.', source: 'vendor/hmr/src/index.ts:20' }, + { name: 'hmr/reload', summary: 'Plugins are being reloaded after a change.', source: 'vendor/hmr/src/index.ts:21' }, + { name: 'exit', summary: 'The process is exiting on a signal.', source: 'vendor/loader/src/index.ts:23' }, + { name: 'loader/config-update', summary: 'The loader config tree changed.', source: 'vendor/loader/src/index.ts:24' }, + { name: 'loader/entry-init', summary: 'A config entry is being initialized.', source: 'vendor/loader/src/index.ts:25' }, + { name: 'loader/partial-dispose', summary: 'An entry is being partially disposed on reload.', source: 'vendor/loader/src/index.ts:26' }, + { name: 'loader/patch-context', summary: 'A context is being patched during a reload.', source: 'vendor/loader/src/index.ts:27' }, +] + +const INHERITED_SERVICES: InheritedEntry[] = [ + { name: 'ctx.on / ctx.once', summary: 'Register an event listener (disposable).', source: 'vendor/cordis/src/events.ts:29' }, + { name: 'ctx.emit / ctx.parallel / ctx.serial / ctx.bail / ctx.waterfall', summary: 'Dispatch an event (sync / awaited / first-non-nullish / veto-chain).', source: 'vendor/cordis/src/events.ts:29' }, + { name: 'ctx.plugin / ctx.inject', summary: 'Load a plugin / declare required services.', source: 'vendor/cordis/src/registry.ts:144' }, + { name: 'ctx.effect', summary: 'Register a disposable side effect tied to the fiber.', source: 'vendor/cordis/src/fiber.ts:9' }, + { name: 'ctx.get / ctx.set / ctx.provide / ctx.accessor / ctx.mixin', summary: 'Low-level service-store access and binding.', source: 'vendor/cordis/src/reflect.ts:7' }, + { name: 'ctx.extend / ctx.isolate / ctx.intercept', summary: 'Derive a child context (scoped services / isolation / interception).', source: 'vendor/cordis/src/context.ts:35' }, + { name: 'ctx.root / ctx.scope / ctx.fiber / ctx.registry / ctx.reflect / ctx.events / ctx.logger', summary: 'Ambient handles onto the running context graph.', source: 'vendor/cordis/src/context.ts:16' }, + { name: 'ctx.timer (+ interval / timeout / throttle / debounce / setTimeout / setInterval)', summary: 'Disposable timer helpers. The `timer` key is provided at runtime; the six helpers are mixed onto ctx directly (declared via Pick).', source: 'vendor/timer/src/index.ts:4' }, + { name: 'ctx.loader', summary: 'The config Loader that booted the app (present under the loader).', source: 'vendor/loader/src/index.ts:30' }, + { name: 'ctx.hmr', summary: 'The hot-module-reload watcher (present under the hmr plugin).', source: 'vendor/hmr/src/index.ts:15' }, +] + +/** Render the cross-link "Types:" line for a signature, or '' if none apply. */ +function typeLinks(signature: string): string { + const seen = new Set() + for (const name of Object.keys(LINK_MAP)) { + if (new RegExp(`\\b${name}\\b`).test(signature)) seen.add(name) + } + if (seen.size === 0) return '' + const links = [...seen].sort().map(n => `[${n}](../core-data-structures/${LINK_MAP[n]})`) + return `Types: ${links.join(' · ')}` +} + +/** Render one harness event entry. */ +function renderEvent(e: EventEntry): string[] { + const out = [`#### \`${e.name}\` — ${e.mode}`, ''] + if (e.doc) out.push(e.doc, '') + out.push('```' + FENCE, e.signature, '```', '') + const links = typeLinks(e.signature) + if (links) out.push(links, '') + out.push(`Source: [\`${e.source}\`](../../${e.source.split(':')[0]})`, '') + return out +} + +/** Render one harness service entry. */ +function renderService(s: ServiceEntry): string[] { + const kind = s.abstract ? ' (abstract seam)' : '' + const out = [`### \`ctx.${s.key}\` — \`${s.type}\`${kind}`, ''] + if (s.doc) out.push(s.doc, '') + if (s.methods.length) { + out.push('```' + FENCE, ...s.methods, '```', '') + const links = typeLinks(s.methods.join('\n')) + if (links) out.push(links, '') + } + out.push(`Source: [\`${s.source}\`](../../${s.source.split(':')[0]})`, '') + return out +} + +/** Render the full catalog (pure, deterministic given sorted inputs). */ +function render(events: EventEntry[], services: ServiceEntry[]): string { + const lines: string[] = [ + '', + '', + '# Cordis Events & Services Catalog', + '', + 'An index reference to the **wiring** a plugin author works against: every cordis event you can listen to (exact signature + dispatch mode) and every `ctx.` service you can call (exact public interface). It complements [core-data-structures/](../core-data-structures/core.md), which catalogs the *data structures* these signatures move around — this page is the verbs, that page is the nouns.', + '', + 'This file is GENERATED from source (`scripts/gen-cordis-catalog.ts`) and verified fresh by `pnpm run verify-cordis-catalog` (part of `doc-sync`) — do not edit it by hand. Signature blocks use a `ts cordis-catalog` fence (skipped by doc-typecheck, since a bare signature is not standalone-compilable). Type names in a signature link to the page that documents them.', + '', + 'The **harness tier** below (the `@deepseek-ai/dsh-*` packages) is the vocabulary this repo owns. The **inherited tier** at the end is the cordis-core + loader/hmr/timer surface a plugin also sees — pinned vendor source, summarized tersely.', + '', + '## Events', + '', + `Dispatch modes: **emit** (fire-and-forget), **waterfall** (each listener gets \`next()\` and may transform or veto — see [waterfall semantics](../architecture.md#cordis-waterfall-semantics-important)), **parallel** (awaited fan-out, no veto). The harness declares ${events.length} events across ${new Set(events.map(e => e.scope)).size} scopes.`, + '', + ] + const scopes = [...new Set(events.map(e => e.scope))].sort() + for (const scope of scopes) { + lines.push(`### \`${scope}/*\``, '') + for (const e of events.filter(x => x.scope === scope).sort((a, b) => a.name.localeCompare(b.name))) { + lines.push(...renderEvent(e)) + } + } + lines.push( + '## Services', + '', + `The ${services.length} \`ctx.\` services the harness provides. An abstract seam (e.g. \`ctx.bash\`) is implemented by a separate package; the interface is what consumers code against.`, + '', + ) + for (const s of services) lines.push(...renderService(s)) + lines.push( + '## Inherited tier (cordis core + loader/hmr/timer)', + '', + 'The framework surface every plugin inherits, beyond the harness vocabulary above. This is pinned vendor source ([vendoring policy](../../vendor/README.md)); it is summarized here so the catalog is a complete picture of what `ctx` and the event bus offer, without elevating framework internals to the harness tier\'s prominence.', + '', + '### Inherited events', + '', + ) + for (const e of INHERITED_EVENTS) { + lines.push(`- \`${e.name}\` — ${e.summary} ([\`${e.source}\`](../../${e.source.split(':')[0]}))`) + } + lines.push('', '### Inherited `ctx` members', '') + for (const s of INHERITED_SERVICES) { + lines.push(`- \`${s.name}\` — ${s.summary} ([\`${s.source}\`](../../${s.source.split(':')[0]}))`) + } + lines.push('') + return lines.join('\n') +} + +/** CLI entry: `--write` (default) writes the catalog, `--check` fails if stale. + * Guarded behind an entry-point check so importing this module for tests neither + * regenerates the committed file nor calls process.exit. */ +function main(): void { + const content = render(collectEvents(), collectServices()) + if (process.argv.includes('--check')) { + let committed: string | null = null + try { + committed = readFileSync(resolve(root, OUT), 'utf8') + } catch { + // Only ENOENT (not yet generated) is expected; a present-but-unreadable + // file is not a state this repo produces. Either way the remedy is the + // same — regenerate — so treat a read failure as "stale". + committed = null + } + if (committed === content) { + console.log(`gen-cordis-catalog: ${OUT} is up to date.`) + process.exit(0) + } + console.error(`gen-cordis-catalog: ${OUT} is stale. Run \`pnpm run gen-cordis-catalog\` and commit ${OUT}.`) + process.exit(1) + } + + writeFileSync(resolve(root, OUT), content) + console.log(`gen-cordis-catalog: wrote ${OUT}.`) +} + +// Run only when invoked as a script, not when imported by a test. +if (process.argv[1] && import.meta.filename === resolve(process.argv[1])) { + main() +} diff --git a/scripts/type-equiv.manifest.json b/scripts/type-equiv.manifest.json new file mode 100644 index 0000000000..5ea2916cd4 --- /dev/null +++ b/scripts/type-equiv.manifest.json @@ -0,0 +1,41 @@ +{ + "comment": "Maps each ` ```ts type-equiv ` block (by doc + declared symbol) to the source symbol it must match verbatim. verify-type-equiv.ts enforces a 1:1 correspondence: every type-equiv block has exactly one entry here, and every entry resolves to exactly one block. Add an entry when you add a type-equiv block; remove it when you remove the block.", + "entries": [ + { "doc": "docs/core-data-structures/core.md", "symbol": "Branded", "source": "packages/llm/src/brand.ts" }, + { "doc": "docs/core-data-structures/core.md", "symbol": "ContentBlockMap", "source": "packages/llm/src/types.ts" }, + { "doc": "docs/core-data-structures/core.md", "symbol": "Message", "source": "packages/llm/src/types.ts" }, + { "doc": "docs/core-data-structures/core.md", "symbol": "MessageSourceMap", "source": "packages/llm/src/types.ts" }, + { "doc": "docs/core-data-structures/core.md", "symbol": "FinishReasonMap", "source": "packages/llm/src/types.ts" }, + { "doc": "docs/core-data-structures/core.md", "symbol": "GenerateOptions", "source": "packages/llm/src/types.ts" }, + { "doc": "docs/core-data-structures/core.md", "symbol": "GenerateResult", "source": "packages/llm/src/types.ts" }, + { "doc": "docs/core-data-structures/core.md", "symbol": "ToolSchema", "source": "packages/llm/src/types.ts" }, + { "doc": "docs/core-data-structures/core.md", "symbol": "SessionEvent", "source": "packages/session/src/types.ts" }, + { "doc": "docs/core-data-structures/core.md", "symbol": "Agent", "source": "packages/agent/src/types.ts" }, + + { "doc": "docs/core-data-structures/llm-streaming.md", "symbol": "StreamChunk", "source": "packages/llm/src/types.ts" }, + { "doc": "docs/core-data-structures/llm-streaming.md", "symbol": "TokenUsage", "source": "packages/llm/src/types.ts" }, + { "doc": "docs/core-data-structures/llm-streaming.md", "symbol": "ContentBlockMap", "source": "packages/llm/src/types.ts" }, + + { "doc": "docs/core-data-structures/session.md", "symbol": "SessionEventMap", "source": "packages/session/src/types.ts" }, + { "doc": "docs/core-data-structures/session.md", "symbol": "SessionEvent", "source": "packages/session/src/types.ts" }, + { "doc": "docs/core-data-structures/session.md", "symbol": "TurnTriggerMap", "source": "packages/session/src/types.ts" }, + { "doc": "docs/core-data-structures/session.md", "symbol": "TurnEndReasonMap", "source": "packages/session/src/types.ts" }, + + { "doc": "docs/core-data-structures/persistence.md", "symbol": "SessionHeader", "source": "packages/session/src/types.ts" }, + { "doc": "docs/core-data-structures/persistence.md", "symbol": "CreateSessionOptions", "source": "packages/session/src/types.ts" }, + + { "doc": "docs/core-data-structures/tools.md", "symbol": "ToolDefinition", "source": "packages/tools/src/index.ts" }, + { "doc": "docs/core-data-structures/tools.md", "symbol": "SchemaProp", "source": "packages/tools/src/schema.ts" }, + { "doc": "docs/core-data-structures/tools.md", "symbol": "SchemaSpec", "source": "packages/tools/src/schema.ts" }, + { "doc": "docs/core-data-structures/tools.md", "symbol": "InferArgs", "source": "packages/tools/src/schema.ts" }, + { "doc": "docs/core-data-structures/tools.md", "symbol": "ToolExecution", "source": "packages/tools/src/index.ts" }, + { "doc": "docs/core-data-structures/tools.md", "symbol": "ToolExecutionResult", "source": "packages/tools/src/index.ts" }, + + { "doc": "docs/core-data-structures/bash.md", "symbol": "BashExecRequest", "source": "packages/bash/src/types.ts" }, + { "doc": "docs/core-data-structures/bash.md", "symbol": "BashExecSpec", "source": "packages/bash/src/types.ts" }, + { "doc": "docs/core-data-structures/bash.md", "symbol": "BashRunResult", "source": "packages/bash/src/types.ts" }, + { "doc": "docs/core-data-structures/bash.md", "symbol": "CollectedOutput", "source": "packages/bash/src/types.ts" }, + { "doc": "docs/core-data-structures/bash.md", "symbol": "BashTask", "source": "packages/bash/src/types.ts" }, + { "doc": "docs/core-data-structures/bash.md", "symbol": "BashTaskRead", "source": "packages/bash/src/types.ts" } + ] +} diff --git a/scripts/verify-event-taxonomy.ts b/scripts/verify-event-taxonomy.ts deleted file mode 100644 index f133d9cf46..0000000000 --- a/scripts/verify-event-taxonomy.ts +++ /dev/null @@ -1,114 +0,0 @@ -/** - * Doc-sync gate (doc-sync-enforcement RFC, part 2): verify the event-taxonomy table in - * docs/architecture.md against the events actually declared in source. - * - * The table duplicates the `declare module 'cordis' { interface Events }` - * blocks across packages/* /src. This script extracts both sets of event names - * and asserts they match exactly — every declared event appears in the table, - * and the table names no event that isn't declared. Verify, don't generate - * (per the RFC): the table keeps its hand-written Mode/Purpose columns; only - * the set of names is checked. - * - * Run: `tsx scripts/verify-event-taxonomy.ts`. - */ - -import { readFileSync } from 'node:fs' -import { join, relative, resolve } from 'node:path' -import { glob } from 'node:fs/promises' - -const root = resolve(import.meta.dirname, '..') - -/** - * Remove `/* *​/` block comments and `//` line comments from TS source. Used to - * de-risk the brace walk in {@link declaredEvents} — a JSDoc `{@link}` tag would - * otherwise throw off the `{`/`}` depth counter. Good enough for our own source - * (no string literals contain `//` or comment-like brace sequences in an Events - * block); it is not a general tokenizer. - */ -function stripComments(text: string): string { - return text - .replace(/\/\*[\s\S]*?\*\//g, '') - .replace(/(^|[^:])\/\/.*$/gm, '$1') -} - -/** - * Event names declared in source: the keys inside every `interface Events` - * block under packages/* /src. A declared event is a quoted `'scope/name'(` - * method signature at the start of a line within such a block. - */ -async function declaredEvents(): Promise> { - const found = new Map() - for await (const match of glob('packages/*/src/**/*.ts', { cwd: root })) { - const abs = resolve(root, match) - // Strip comments first so a JSDoc `{@link …}` tag (or a `// {` line) inside - // an Events block can't unbalance the brace walk below. Event names live in - // code, never in comments, so this loses nothing. - const text = stripComments(readFileSync(abs, 'utf8')) - // Walk `interface Events {` blocks brace-balanced and pull quoted keys. - const re = /interface\s+Events\s*\{/g - let m: RegExpExecArray | null - while ((m = re.exec(text)) !== null) { - let depth = 1 - let i = m.index + m[0].length - const start = i - while (i < text.length && depth > 0) { - const ch = text[i] - if (ch === '{') depth++ - else if (ch === '}') depth-- - i++ - } - const body = text.slice(start, i - 1) - // A declaration is a quoted event name followed by `(` (method form). - for (const k of body.matchAll(/['"]([a-z][a-z-]*\/[a-z-]+)['"]\s*\(/g)) { - const name = k[1] - if (name) found.set(name, relative(root, abs)) - } - } - } - return found -} - -/** Event names referenced in the architecture-doc taxonomy table (in `code`). */ -function tableEvents(): Set { - const text = readFileSync(join(root, 'docs/architecture.md'), 'utf8') - const lines = text.split('\n') - const heading = lines.findIndex(l => /^###\s+Event taxonomy/.test(l)) - if (heading === -1) throw new Error('verify-event-taxonomy: "### Event taxonomy" heading not found') - const names = new Set() - for (let i = heading + 1; i < lines.length; i++) { - const line = lines[i] ?? '' - if (/^###\s/.test(line)) break // next section ends the table - if (!line.includes('|')) continue - for (const code of line.matchAll(/`([^`]+)`/g)) { - // A cell may read "`a/b` / `c/d` (pkg)" — pull each scoped name. - for (const name of (code[1] ?? '').matchAll(/[a-z][a-z-]*\/[a-z-]+/g)) names.add(name[0]) - } - } - return names -} - -const declared = await declaredEvents() -const table = tableEvents() - -const declaredNames = new Set(declared.keys()) -const missingFromTable = [...declaredNames].filter(n => !table.has(n)).sort() -const missingFromSource = [...table].filter(n => !declaredNames.has(n)).sort() - -if (missingFromTable.length === 0 && missingFromSource.length === 0) { - console.log(`verify-event-taxonomy: ${declaredNames.size} events match the architecture-doc table.`) - process.exit(0) -} - -if (missingFromTable.length > 0) { - console.error('verify-event-taxonomy: declared in source but MISSING from the docs/architecture.md table:') - for (const n of missingFromTable) { - console.error(` ${n} (declared in ${declared.get(n) ?? '?'})`) - } -} -if (missingFromSource.length > 0) { - console.error('verify-event-taxonomy: named in the table but NOT declared in source (stale doc):') - for (const n of missingFromSource) { - console.error(` ${n}`) - } -} -process.exit(1) diff --git a/scripts/verify-md-links.ts b/scripts/verify-md-links.ts index 35e42452ca..be1a80f86a 100644 --- a/scripts/verify-md-links.ts +++ b/scripts/verify-md-links.ts @@ -48,7 +48,7 @@ const root = resolve(import.meta.dirname, '..') const PATTERNS = [ 'README.md', 'docs/**/*.md', - 'packages/*/README.md', + 'packages/*/*.md', 'AGENTS.md', 'packages/AGENTS.md', '.agents/skills/**/*.md', diff --git a/scripts/verify-md-wrap.ts b/scripts/verify-md-wrap.ts index 169b06b15b..dbb1e69235 100644 --- a/scripts/verify-md-wrap.ts +++ b/scripts/verify-md-wrap.ts @@ -18,7 +18,7 @@ * A wrapped paragraph inside a list item or blockquote is still a `paragraph` * node, so those are caught too. Scope mirrors doc-typecheck plus the two * AGENTS.md files that doc-sync does NOT otherwise cover (the convention itself - * lives there): README.md, docs/** /*.md, packages/* /README.md, AGENTS.md, + * lives there): README.md, docs/** /*.md, packages/* /*.md, AGENTS.md, * packages/AGENTS.md. The root and packages/ CLAUDE.md are symlinks to the * AGENTS.md files, so they are deduped by real path. * @@ -36,7 +36,7 @@ import type { Nodes } from 'mdast' const root = resolve(import.meta.dirname, '..') /** Files to check: doc-typecheck's scope plus the AGENTS.md pair. */ -const PATTERNS = ['README.md', 'docs/**/*.md', 'packages/*/README.md', 'AGENTS.md', 'packages/AGENTS.md'] +const PATTERNS = ['README.md', 'docs/**/*.md', 'packages/*/*.md', 'AGENTS.md', 'packages/AGENTS.md'] /** A located hard-wrap: a prose paragraph spanning more than one source line. */ interface Violation { diff --git a/scripts/verify-type-equiv.ts b/scripts/verify-type-equiv.ts new file mode 100644 index 0000000000..55250db23f --- /dev/null +++ b/scripts/verify-type-equiv.ts @@ -0,0 +1,227 @@ +/** + * Doc-sync gate: verify every ` ```ts type-equiv ` block in the docs is a + * VERBATIM copy of the source type definition it documents. + * + * The core-data-structures docs paste real type definitions so a reader sees + * the exact shape. A paste drifts the moment source changes — this script is + * the drift guard. For each block it extracts the documented symbol's + * declaration from source via the TypeScript compiler API, whitespace- + * normalizes both the source text and the block, and asserts they are equal. + * + * Provenance lives in a central manifest (`scripts/type-equiv.manifest.json`), + * NOT in the doc prose: each entry names `{ doc, symbol, source }`. The script + * enforces a 1:1 correspondence — every type-equiv block in the docs has + * exactly one manifest entry (keyed by doc + declared symbol), and every + * manifest entry resolves to exactly one block. An orphan on either side fails, + * so a block can never be silently unchecked and an entry can never rot. + * + * doc-typecheck.ts recognizes the same ` ```ts type-equiv ` fence and skips it + * (it is not standalone-compilable and is not counted in the opt-out ratio); + * the two scripts share the fence, this one owns the verification. + * + * Run: `tsx scripts/verify-type-equiv.ts`. + */ + +import { readFileSync, existsSync } from 'node:fs' +import { resolve } from 'node:path' +import { glob } from 'node:fs/promises' +import ts from 'typescript' + +const root = resolve(import.meta.dirname, '..') + +/** + * Markdown globs scanned for ` ```ts type-equiv ` blocks — the SAME scope + * doc-typecheck uses. Scanning every doc (not only the docs the manifest names) + * is what makes the 1:1 guarantee real in both directions: a type-equiv block + * added to a doc with NO manifest entry is still discovered here and reported as + * an orphan, instead of being silently skipped. + */ +const MARKDOWN_GLOBS = ['README.md', 'docs/**/*.md', 'packages/*/*.md'] + +/** One manifest entry: a documented type-equiv block and its source symbol. */ +interface ManifestEntry { + /** Doc file (repo-relative) containing the ` ```ts type-equiv ` block. */ + doc: string + /** The declared symbol the block must match (e.g. `SessionEvent`). */ + symbol: string + /** Source file (repo-relative) that exports the symbol. */ + source: string +} + +/** One extracted ` ```ts type-equiv ` block. */ +interface EquivBlock { + doc: string + /** 1-based line of the opening fence (for diagnostics). */ + line: number + /** Symbol name parsed from the block's declaration. */ + symbol: string + /** Block body (the pasted declaration). */ + code: string +} + +/** Collapse a declaration to its structural form for comparison: drop comments + * (block + line), then collapse all whitespace runs to single spaces. This lets + * a doc block show a CLEAN definition (without source's verbose inline JSDoc) + * while still guaranteeing the field shapes match — drift in a field name or + * type fails; a reworded inline comment does not. Adequate for our own type + * source (no string literal contains `//` or `/* *​/`); not a general tokenizer. */ +function normalize(code: string): string { + return code + .replace(/\/\*[\s\S]*?\*\//g, '') + .replace(/(^|[^:])\/\/.*$/gm, '$1') + .replace(/\s+/g, ' ') + .trim() +} + +/** Strip a leading `export ` / `export default ` modifier — the doc block shows + * the bare declaration, the source carries the export modifier. */ +function stripExport(code: string): string { + return code.replace(/^export\s+(default\s+)?/, '') +} + +/** Parse the declared symbol name from a type-equiv block body. */ +function blockSymbol(code: string): string | null { + const m = /(?:export\s+(?:default\s+)?)?(?:abstract\s+)?(?:interface|type|class|enum)\s+([A-Za-z0-9_]+)/.exec(code) + return m?.[1] ?? null +} + +/** Extract every ` ```ts type-equiv ` block from one Markdown file. */ +function extractEquivBlocks(docRel: string): EquivBlock[] { + const text = readFileSync(resolve(root, docRel), 'utf8') + const lines = text.split('\n') + const blocks: EquivBlock[] = [] + let open: { line: number; body: string[] } | null = null + + for (let i = 0; i < lines.length; i++) { + const raw = lines[i] ?? '' + const fence = /^```(\s*)(\S.*)?$/.exec(raw) + if (!fence) { + if (open) open.body.push(raw) + continue + } + if (open) { + const code = open.body.join('\n') + const symbol = blockSymbol(code) + if (!symbol) { + throw new Error(`verify-type-equiv: ${docRel}:${open.line} — type-equiv block has no parseable interface/type/class declaration`) + } + blocks.push({ doc: docRel, line: open.line, symbol, code }) + open = null + continue + } + if ((fence[2] ?? '').trim() === 'ts type-equiv') open = { line: i + 1, body: [] } + } + if (open) throw new Error(`verify-type-equiv: ${docRel}:${open.line} — unterminated type-equiv block`) + return blocks +} + +/** The declaration text of `symbol` in `sourceRel`, with `export` stripped, or + * null when the symbol is not declared there. Uses the TS parser so it spans + * interfaces, type aliases (including mapped/generic ones), classes, and enums + * uniformly, and excludes the leading JSDoc (getStart skips leading trivia) + * while keeping inline member comments. */ +function sourceDeclaration(sourceRel: string, symbol: string): string | null { + const abs = resolve(root, sourceRel) + const text = readFileSync(abs, 'utf8') + const sf = ts.createSourceFile(abs, text, ts.ScriptTarget.Latest, /* setParentNodes */ true) + for (const stmt of sf.statements) { + const named = + ts.isInterfaceDeclaration(stmt) || ts.isTypeAliasDeclaration(stmt) + || ts.isClassDeclaration(stmt) || ts.isEnumDeclaration(stmt) + if (named && stmt.name?.text === symbol) { + return stripExport(stmt.getText(sf)) + } + } + return null +} + +const manifestRaw = readFileSync(resolve(root, 'scripts/type-equiv.manifest.json'), 'utf8') +const manifest = JSON.parse(manifestRaw) as { entries: ManifestEntry[] } +const entries = manifest.entries + +// Key a block/entry by doc + symbol (a symbol may be documented in more than one +// doc, but at most once per doc). +const keyOf = (x: { doc: string; symbol: string }): string => `${x.doc}::${x.symbol}` + +// Collect every type-equiv block across ALL docs in scope — not only the docs +// the manifest names — so a block in an unmanifested doc is found and reported +// as an orphan rather than silently skipped. +const docSet = new Set() +for (const pattern of MARKDOWN_GLOBS) { + for await (const match of glob(pattern, { cwd: root })) docSet.add(match) +} +const blocks: EquivBlock[] = [...docSet].sort().flatMap(extractEquivBlocks) + +const errors: string[] = [] +// A manifest entry naming a doc that does not exist (or is outside the scanned +// scope, so no block could ever match it) is an error in its own right. +for (const d of [...new Set(entries.map(e => e.doc))]) { + if (!existsSync(resolve(root, d))) errors.push(`manifest references ${d}, which does not exist`) + else if (!docSet.has(d)) errors.push(`manifest references ${d}, which is outside the scanned markdown scope (${MARKDOWN_GLOBS.join(', ')})`) +} + +// Duplicate-block guard: the same symbol twice in one doc is ambiguous. +const blockByKey = new Map() +for (const b of blocks) { + const k = keyOf(b) + const prior = blockByKey.get(k) + if (prior) { + errors.push(`duplicate type-equiv block for ${b.symbol} in ${b.doc} (lines ${prior.line} and ${b.line})`) + continue + } + blockByKey.set(k, b) +} + +// Duplicate-entry guard in the manifest. +const entryByKey = new Map() +for (const e of entries) { + const k = keyOf(e) + if (entryByKey.has(k)) { + errors.push(`duplicate manifest entry for ${e.symbol} in ${e.doc}`) + continue + } + entryByKey.set(k, e) +} + +// 1:1 correspondence: orphan blocks (no entry) and orphan entries (no block). +for (const b of blocks) { + if (!entryByKey.has(keyOf(b))) { + errors.push(`type-equiv block ${b.symbol} (${b.doc}:${b.line}) has no manifest entry — add one to scripts/type-equiv.manifest.json`) + } +} +for (const e of entries) { + if (!blockByKey.has(keyOf(e))) { + errors.push(`manifest entry ${e.symbol} (${e.doc}) has no matching type-equiv block — remove it or add the block`) + } +} + +// Verbatim check: each matched block must equal its source declaration. +let verified = 0 +for (const e of entries) { + const b = blockByKey.get(keyOf(e)) + if (!b) continue // already reported as an orphan entry + const decl = sourceDeclaration(e.source, e.symbol) + if (decl === null) { + errors.push(`symbol ${e.symbol} not found in ${e.source} (manifest entry for ${e.doc})`) + continue + } + if (normalize(decl) !== normalize(stripExport(b.code))) { + errors.push( + `DRIFT: ${e.doc}:${b.line} — type-equiv block for ${e.symbol} does not match ${e.source}.\n` + + ` source: ${normalize(decl)}\n` + + ` doc: ${normalize(stripExport(b.code))}`, + ) + continue + } + verified++ +} + +if (errors.length === 0) { + console.log(`verify-type-equiv: ${verified} type-equiv block(s) match source (1:1 with manifest).`) + process.exit(0) +} + +console.error('verify-type-equiv: type-equiv verification failed:') +for (const e of errors) console.error(` ${e}`) +console.error(`\n(checked ${blocks.length} block(s) across ${new Set(blocks.map(b => b.doc)).size} doc(s); manifest at scripts/type-equiv.manifest.json)`) +process.exit(1)