diff --git a/docs/AGENTS.md b/docs/AGENTS.md index 7b210df89f..b1df4a02d9 100644 --- a/docs/AGENTS.md +++ b/docs/AGENTS.md @@ -17,7 +17,7 @@ Every fact has exactly one home — the tier whose job it is — and every other | [cookbook/](cookbook/adding-a-package.md) | Step-by-step how-tos with numbered verify steps | Design rationale (→ the RFC each guide links) | | Package README | The per-package contract: config, semantics, limitations, extension points | JSDoc restatement, generated-catalog restatement (event/tool tables), other packages' concerns | | [development.md](development.md) | Human-facing setup and daily workflow; a bilingual pair under the [i18n contract](i18n/README.md) | Gate-by-gate enumerations that drift from `package.json` scripts | -| Generated catalogs: [cordis-catalog](cordis-catalog/events-and-services.md), [tool-catalog](tool-catalog/tools.md), [module-graph.md](module-graph.md) | Exhaustive enumerations regenerated from source, freshness-gated | Hand edits of any kind | +| Generated catalogs: [cordis-catalog](cordis-catalog/events-and-services.md), [tool-catalog](tool-catalog/tools.md), [persistence-catalog](persistence-catalog/log-events.md), [module-graph.md](module-graph.md) | Exhaustive enumerations regenerated from source, freshness-gated | Hand edits of any kind | | Skills (`.agents/skills/`) | Workflows: how to carry out a recurring task against the contracts | The contracts themselves (→ docs) | Placement test: a story about a bug → postmortem. Why we chose X → RFC. How to do task Y → cookbook. What type Z looks like → core-data-structures. What package P promises → its README. A rule every agent must always obey → root AGENTS.md, one line, linking the home that holds the why. diff --git a/docs/core-data-structures/persistence.md b/docs/core-data-structures/persistence.md index 327162792a..ec5589dd48 100644 --- a/docs/core-data-structures/persistence.md +++ b/docs/core-data-structures/persistence.md @@ -1,6 +1,6 @@ # Session Persistence -The **durability seam** for the event log. [session.md](session.md) describes the in-memory `Session` — the append-only `SessionEvent` log that is the source of truth. This page describes how that log is made durable: the abstract `SessionPersistence` service, its backends, the flush checkpoint, crash recovery, and the metadata header that travels alongside the log. +The **durability seam** for the event log. [session.md](session.md) describes the in-memory `Session` — the append-only `SessionEvent` log that is the source of truth. This page describes how that log is made durable: the abstract `SessionPersistence` service, its backends, the flush checkpoint, crash recovery, and the metadata header that travels alongside the log. The event vocabulary the log carries is enumerated, member by member, in the generated [persistence log event catalog](../persistence-catalog/log-events.md). The seam is a textbook [capability seam](../rfc/implemented/architecture/2026-06-13-capability-seams.md): one abstract service ([dsh-session-persistence](../../packages/session-persistence/session-persistence), `ctx.sessionPersistence`) defining create/append/load/list over the existing `SessionEvent` — **no parallel persisted type** — and two interchangeable backends that pass the same `runPersistenceContract` suite. See the [session-persistence RFC](../rfc/implemented/architecture/2026-06-14-session-persistence.md). diff --git a/docs/core-data-structures/session.md b/docs/core-data-structures/session.md index 4d2b47c05c..7c004e8dbd 100644 --- a/docs/core-data-structures/session.md +++ b/docs/core-data-structures/session.md @@ -6,7 +6,7 @@ Source: [`packages/core/session/src/types.ts`](../../packages/core/session/src/t ## `SessionEventMap` — the event vocabulary -The append-only event types. Merge-extensible: a plugin declares extra event types via declaration merging — e.g. the [compaction seam](compaction.md) adds `compact/start` / `compact/summary` / `compact/end`, and `@deepseek-ai/dsh-hook-protocol` adds log-only `hook/invoked` / `hook/result` provenance for a hook bridge. Like `compact/*`, these are NOT `SurfaceEventType`s (no `surfaceOp`). +The append-only event types. Merge-extensible: a plugin declares extra event types via declaration merging — e.g. the [compaction seam](compaction.md) adds `compact/start` / `compact/summary` / `compact/end`, and `@deepseek-ai/dsh-hook-protocol` adds log-only `hook/invoked` / `hook/result` provenance for a hook bridge. Like `compact/*`, these are NOT `SurfaceEventType`s (no `surfaceOp`). The generated [persistence log event catalog](../persistence-catalog/log-events.md) enumerates every member — core and merged — with its payload, surface badge, and declaration site. ```ts type-equiv interface SessionEventMap { @@ -224,14 +224,9 @@ Every session event lives **inside** a turn (between a `turn/start` and its `tur ## Plugin-contributed log-only events -A plugin may declaration-merge extra `SessionEventMap` types. These are **log-only**: NOT `SurfaceEventType`s (they carry no `surfaceOp` and contribute nothing to derived history), but, like every event, they must sit inside an open turn. The compaction seam's `compact/*` are documented on [compaction.md](compaction.md); the hook bridges' `hook/*` provenance (from `@deepseek-ai/dsh-hook-protocol`) are: +A plugin may declaration-merge extra `SessionEventMap` types. These are **log-only**: NOT `SurfaceEventType`s (they carry no `surfaceOp` and contribute nothing to derived history), but, like every event, they must sit inside an open turn. The full per-event enumeration — core and plugin-contributed alike, with payloads and provenance — is the generated [persistence log event catalog](../persistence-catalog/log-events.md); the compaction seam's `compact/*` semantics are discussed on [compaction.md](compaction.md). -| Event | Payload | Role | -|---|---|---| -| `hook/invoked` | `{ turn, point, dialect, matcher?, handlerId }` | A hook command was invoked at a hook `point` (`PreToolUse`, `Stop`, …). `dialect` is the bridge (`claude`/`codex`/`native`); `matcher` the matcher-group pattern that selected it (absent for match-all); `handlerId` correlates with the result. | -| `hook/result` | `{ turn, point, handlerId, decision, exitCode?, stderrSummary?, durationMs }` | The decided outcome, paired by `handlerId`. `decision` is the resolved neutral outcome (`deny`/`allow`/`block`/`stop`/`pass`/…); `exitCode` absent when the hook could not run; `stderrSummary` the truncated block-reason source. | - -The mid-turn hook points (`PreToolUse`/`PostToolUse`/`UserPromptSubmit`/`Stop`) fire inside the loop's open turn, so their `hook/*` records are turn-enclosed by construction. `SessionStart` gets no `hook/*` record — its injected `context/message` is the durable evidence — because it has no open turn to enclose one (see the hooks RFC). +The hook bridges' `hook/invoked` / `hook/result` provenance pairs (from `@deepseek-ai/dsh-hook-protocol`) correlate by `handlerId`. The mid-turn hook points (`PreToolUse`/`PostToolUse`/`UserPromptSubmit`/`Stop`) fire inside the loop's open turn, so their `hook/*` records are turn-enclosed by construction. `SessionStart` gets no `hook/*` record — its injected `context/message` is the durable evidence — because it has no open turn to enclose one (see [the hook-bridges RFC](../rfc/implemented/feature/2026-06-30-hook-bridges.md)). ## Durability contract diff --git a/docs/persistence-catalog/log-events.md b/docs/persistence-catalog/log-events.md new file mode 100644 index 0000000000..4bdda95f40 --- /dev/null +++ b/docs/persistence-catalog/log-events.md @@ -0,0 +1,240 @@ + + +# Persistence Log Event Catalog + +Every event type that can appear in a session's durable event log: each member of the merge-extensible `SessionEventMap` — the owning vocabulary in `@deepseek-ai/dsh-session` plus every plugin declaration merge in this repo — with the payload it carries, its surface badge, and the declaration it comes from. It complements [session.md](../core-data-structures/session.md) (the `SessionEvent` envelope, surface list, and `deriveMessages()` projection), [persistence.md](../core-data-structures/persistence.md) (how the log is made durable), and the [cordis catalog](../cordis-catalog/events-and-services.md) (the live bus wiring — a log event is NOT a cordis event; it reaches listeners via the single `session/event` emit). + +This file is GENERATED from source (`scripts/gen-persistence-catalog.ts`) and verified fresh by `pnpm run verify-persistence-catalog` (part of `doc-sync`) — do not edit it by hand. Payload blocks use a `ts persistence-catalog` fence (skipped by doc-typecheck, since a bare payload fragment is not standalone-compilable). Type names in a payload link to the page that documents them. See [the persistence-log-catalog RFC](../rfc/implemented/process/2026-07-04-persistence-log-catalog.md). + +The on-disk envelope around every payload is `SessionEvent` — `type`, monotonic `seq`, epoch-ms `time`, the `data` documented here, plus `surfaceOp`/`sourceEventSeqs` on **surface** events only ([envelope](../core-data-structures/session.md#sessioneventt--one-log-entry)). **surface** marks a `SurfaceEventType` member: it produces an LLM message and declares how it joins the surface list. **log-only** marks everything else: durable, replayable record with no derived-history contribution. Every payload is JSON-serializable (enforced at `Session.append`), and the whole format is pinned at `SESSION_FORMAT_VERSION = 0` — pre-release, no compatibility implied ([the version stance](../core-data-structures/persistence.md)). Scope: the packages in this repo; a downstream plugin can merge further event types, which are outside this catalog by construction. + +## Events + +### `assistant/*` + +#### `assistant/chunk` — log-only + +Raw stream chunk — token-level replay fidelity. + +```ts persistence-catalog +'assistant/chunk': { turn: number; step: number; chunk: StreamChunk } +``` + +Types: [StreamChunk](../core-data-structures/llm-streaming.md) + +Source: [`packages/core/session/src/types.ts:238`](../../packages/core/session/src/types.ts) + +#### `assistant/message` — surface + +Assembled assistant message for one step (derived history uses this). Carries the step's `usage` when the adapter reported token accounting, so the model output and its accounting travel together (there is no separate usage record). `usage` is absent when the adapter reported none. + +```ts persistence-catalog +'assistant/message': { turn: number; step: number; content: ContentBlock[]; usage?: TokenUsage } +``` + +Types: [ContentBlock](../core-data-structures/core.md) · [TokenUsage](../core-data-structures/llm-streaming.md) + +Source: [`packages/core/session/src/types.ts:245`](../../packages/core/session/src/types.ts) + +### `compact/*` + +#### `compact/end` — log-only + +Marks the end of a compaction — log-only, releases the lock. `error` set if summarization failed. + +```ts persistence-catalog +'compact/end': { turn: number; error?: string } +``` + +Source: [`packages/compact/compact/src/types.ts:37`](../../packages/compact/compact/src/types.ts) + +#### `compact/start` — log-only + +Marks the start of a compaction — log-only, holds the lock until `compact/end`. + +```ts persistence-catalog +'compact/start': { turn: number } +``` + +Source: [`packages/compact/compact/src/types.ts:23`](../../packages/compact/compact/src/types.ts) + +#### `compact/summary` — log-only + +Provenance record of a completed summarization — log-only, no surfaceOp. The summary content is in `data.summary`; the actual surface replacement is performed by a subsequent `user/message` event that shadows the compacted range. + +```ts persistence-catalog +'compact/summary': { summary: ContentBlock[]; shadowedRange: { start: number; end: number }; shadowedSeqs: number[]; shadowedTokenCount: number } +``` + +Types: [ContentBlock](../core-data-structures/core.md) + +Source: [`packages/compact/compact/src/types.ts:30`](../../packages/compact/compact/src/types.ts) + +### `context/*` + +#### `context/message` — surface + +In-session context injection (file-change notices, subdir AGENTS.md, skill content, cron notifications, …). Rendered into the derived history as tagged synthetic context — NOT a user prompt. + +```ts persistence-catalog +'context/message': { content: ContentBlock[]; source: MessageSource } +``` + +Types: [ContentBlock](../core-data-structures/core.md) · [MessageSource](../core-data-structures/core.md) + +Source: [`packages/core/session/src/types.ts:236`](../../packages/core/session/src/types.ts) + +### `hook/*` + +#### `hook/invoked` — log-only + +A hook command was invoked at a hook point — log-only provenance (like `compact/*`; NOT a SurfaceEventType, carries no `surfaceOp`). `dialect` is the bridge that ran it (`claude`/`codex`/`native`), `point` the hook point (`PreToolUse`, `Stop`, …), `matcher` the matcher-group pattern that selected it (absent for match-all), `handlerId` a stable id for the command (so an invoked/result pair correlates). `turn` is the open turn the invocation lives inside. + +```ts persistence-catalog +'hook/invoked': { turn: number; point: string; dialect: HookDialect; matcher?: string; handlerId: string } +``` + +Source: [`packages/hooks/hook-protocol/src/types.ts:27`](../../packages/hooks/hook-protocol/src/types.ts) + +#### `hook/result` — log-only + +A hook command's outcome — log-only, paired with a prior `hook/invoked` (same `handlerId`). `decision` is the resolved dialect-neutral outcome the bridge mapped it to (`allow`/`deny`/`ask`/`block`/`continue`/`stop`/`pass`), `exitCode` the process exit (absent if it never ran), `stderrSummary` a truncated stderr (the block reason source on exit 2), `durationMs` the wall time. `turn` matches the `hook/invoked`. + +```ts persistence-catalog +'hook/result': { turn: number; point: string; handlerId: string; decision: string; exitCode?: number; stderrSummary?: string; durationMs: number } +``` + +Source: [`packages/hooks/hook-protocol/src/types.ts:42`](../../packages/hooks/hook-protocol/src/types.ts) + +### `prompt/*` + +#### `prompt/blocked` — log-only + +A queued prompt an `agent/prompt-submit` listener VETOED — the durable record of a blocked prompt and why. Appended in place of the `user/message` the prompt would have become, so the block survives replay even in a MIXED batch where another queued prompt is allowed (there the turn does not end `rejected`, so the boundary reason alone would not preserve it). `content` is the original prompt the listener rejected; `reason` is the veto text (PromptDecision `block.reason`). NOT a SurfaceEventType: a blocked prompt produces no LLM message and never reaches `deriveMessages()`. + +```ts persistence-catalog +'prompt/blocked': { content: ContentBlock[]; source: MessageSource; reason: string } +``` + +Types: [ContentBlock](../core-data-structures/core.md) · [MessageSource](../core-data-structures/core.md) + +Source: [`packages/core/session/src/types.ts:230`](../../packages/core/session/src/types.ts) + +### `steering/*` + +#### `steering/message` — surface + +Steering content injected between steps of a running turn. + +```ts persistence-catalog +'steering/message': { turn: number; content: ContentBlock[]; source: MessageSource } +``` + +Types: [ContentBlock](../core-data-structures/core.md) · [MessageSource](../core-data-structures/core.md) + +Source: [`packages/core/session/src/types.ts:263`](../../packages/core/session/src/types.ts) + +### `step/*` + +#### `step/end` — log-only + +Closes step `step` of turn `turn`. + +```ts persistence-catalog +'step/end': { turn: number; step: number } +``` + +Source: [`packages/core/session/src/types.ts:217`](../../packages/core/session/src/types.ts) + +#### `step/start` — log-only + +Opens step `step` of turn `turn` — one model call plus the tool executions it requested. + +```ts persistence-catalog +'step/start': { turn: number; step: number } +``` + +Source: [`packages/core/session/src/types.ts:215`](../../packages/core/session/src/types.ts) + +### `todo/*` + +#### `todo/write` — log-only + +The agent's whole todo list, carried as a full snapshot and replaced wholesale on each write — the current list is the most recent `todo/write` (last-write-wins on replay, no fold). Appended by an owning agent via `session.append('todo/write', { todos })`. + +NOT a SurfaceEventType: it produces no LLM message and never reaches `deriveMessages()`, so it carries no `surfaceOp` and stays off the surface — it is durable, replayable UI state, distinct from the conversation history. It is a `SessionEventMap` member riding the existing `session/event` emit, not a first-class Cordis `interface Events` notification, so it has no cordis-catalog row. + +```ts persistence-catalog +'todo/write': { todos: TodoItem[] } +``` + +Types: [TodoItem](../core-data-structures/session.md) + +Source: [`packages/core/session/src/types.ts:277`](../../packages/core/session/src/types.ts) + +### `tool/*` + +#### `tool/call` — log-only + +The model requested one tool invocation: `name` with the raw `arguments` JSON string exactly as the model produced it (unparsed). `callId` pairs the call with its `tool/result`. + +```ts persistence-catalog +'tool/call': { turn: number; step: number; callId: CallId; name: string; arguments: string } +``` + +Types: [CallId](../core-data-structures/core.md) + +Source: [`packages/core/session/src/types.ts:251`](../../packages/core/session/src/types.ts) + +#### `tool/result` — surface + +A completed tool call's model-facing result, plus an optional tool-private `meta` presentation payload. `meta` is opaque to the core (`unknown` — the producing tool owns its shape and reads it back in `presentResult`) but MUST be JSON-serializable: `Session.append` runtime-validates all event data with `isJsonValue`, so a non-serializable `meta` is rejected at the source, and the durable log reproduces the identical card on replay. Absent unless the tool attaches one (e.g. `dsh-tool-fs` carries its result-time contextual diff here). + +```ts persistence-catalog +'tool/result': { turn: number; step: number; callId: CallId; content: ContentBlock[]; isError: boolean; error?: { name: string; code: string }; meta?: unknown } +``` + +Types: [CallId](../core-data-structures/core.md) · [ContentBlock](../core-data-structures/core.md) + +Source: [`packages/core/session/src/types.ts:261`](../../packages/core/session/src/types.ts) + +### `turn/*` + +#### `turn/end` — log-only + +Closes turn `turn` with the TurnEndReason that ended it. The loop fires the awaited `session/flush` checkpoint at every turn end, so the turn boundary is also the durable-commit boundary. + +```ts persistence-catalog +'turn/end': { turn: number; reason: TurnEndReason } +``` + +Types: [TurnEndReason](../core-data-structures/session.md) + +Source: [`packages/core/session/src/types.ts:213`](../../packages/core/session/src/types.ts) + +#### `turn/start` — log-only + +Opens turn `turn`. `trigger` records what started it — a drained message batch, a continuation, or an idle-time injection. The turn is the durability/replay boundary: every event sits between a `turn/start` and its matching `turn/end` (the turn-enclosure invariant). + +```ts persistence-catalog +'turn/start': { turn: number; trigger: TurnTrigger } +``` + +Types: [TurnTrigger](../core-data-structures/session.md) + +Source: [`packages/core/session/src/types.ts:207`](../../packages/core/session/src/types.ts) + +### `user/*` + +#### `user/message` — surface + +A user-visible prompt (queued message drained at turn start). + +```ts persistence-catalog +'user/message': { content: ContentBlock[]; source: MessageSource } +``` + +Types: [ContentBlock](../core-data-structures/core.md) · [MessageSource](../core-data-structures/core.md) + +Source: [`packages/core/session/src/types.ts:219`](../../packages/core/session/src/types.ts) diff --git a/docs/rfc/README.md b/docs/rfc/README.md index 3774371316..38fd9bf1a9 100644 --- a/docs/rfc/README.md +++ b/docs/rfc/README.md @@ -172,6 +172,7 @@ Do NOT write one for a mechanical or local choice (a variable name, a one-file r | [JSDoc completeness gate for the cordis surface](implemented/process/2026-07-04-cordis-jsdoc-completeness-gate.md) | 2026-07-04 | | [Documentation tiers, budgets, and the ceiling gate](implemented/process/2026-07-04-doc-tiers-and-budgets.md) | 2026-07-04 | | [Generate the RFC index tables](implemented/process/2026-07-04-generate-rfc-index-tables.md) | 2026-07-04 | +| [Generated persistence log event catalog](implemented/process/2026-07-04-persistence-log-catalog.md) | 2026-07-04 | ### Testing diff --git a/docs/rfc/implemented/process/2026-07-04-persistence-log-catalog.md b/docs/rfc/implemented/process/2026-07-04-persistence-log-catalog.md new file mode 100644 index 0000000000..80e6117c66 --- /dev/null +++ b/docs/rfc/implemented/process/2026-07-04-persistence-log-catalog.md @@ -0,0 +1,29 @@ +# RFC: Generated persistence log event catalog + +Status: implemented (accepted 2026-07-04) + +## Context + +The session event log is the harness's on-disk contract: every `SessionEventMap` member is a record a persistence backend writes verbatim and a replay reconstructs from, and adding one that breaks the durability rules is a breaking change to the on-disk format. Yet the vocabulary had no single reference. The declarations are split across three files — the owning interface in `@deepseek-ai/dsh-session` plus declaration merges in `@deepseek-ai/dsh-compact` and `@deepseek-ai/dsh-hook-protocol` — and the doc surfaces covered it with hand-copies: a `hook/*` payload table in [session.md](../../../core-data-structures/session.md), a `compact/*` payload table in the compact README, payload bullets in the hook-protocol README, and a name-list in the session README. The name-list's merge note had already drifted (it named the compaction merge and omitted the hook merge entirely), and nothing could catch the next merge going undocumented: a hand-copy only checks the names someone already wrote down. This is the same gap the [cordis catalog](2026-06-20-generated-cordis-catalog.md) closed for bus events and the [tool catalog](2026-07-02-tool-schema-catalog.md) closed for model-facing tools — and log events are covered by neither: a `SessionEventMap` member is not a cordis `Events` declaration (it reaches listeners via the single `session/event` emit), so it has no cordis-catalog row by design. + +## Decision + +Generate `docs/persistence-catalog/log-events.md` from source, with a freshness gate, as the fourth reference surface: the *records* a persisted session log can contain, complementing the cordis catalog (wiring), core-data-structures (vocabulary), and the tool catalog (tools). + +`scripts/gen-persistence-catalog.ts` is a pure TypeScript-AST pass, like `gen-cordis-catalog.ts` and unlike the boot-based tool catalog — the right technique because log events ARE statically knowable: every member is a string-literal-named property with a static type annotation, so the AST is the whole truth. The walk collects every `interface SessionEventMap` declaration under `packages/*/*/src` — the owning top-level interface and every `declare module '@deepseek-ai/dsh-session'` merge — so a brand-new event, core or merged, appears in the next regenerate and an un-regenerated file fails `--check` (`verify-persistence-catalog`, a `doc-sync` member, so pre-push and CI both run it). Each entry renders the member's JSDoc prose, its payload (printed through the TypeScript printer, so a newline-separated multi-line type literal still yields a valid one-line fragment), a surface badge, cross-links into core-data-structures, and the declaration's source pointer, grouped by scope. + +Specific choices: + +- **JSDoc completeness, enforced.** Every member must carry description prose — the JSDoc becomes the catalog entry, the same forcing function the cordis catalog applies to bus events. An `@mode` tag on a member is a hard error: dispatch modes belong to cordis bus events, and a log event has none — the tag would misread as "this fires on the bus with mode X". Violations aggregate into one error listing every offender. +- **The surface badge is derived, not hand-listed.** `SurfaceEventType` — the subset that produces LLM messages and may carry `surfaceOp` — is parsed from its union declaration in the owning package; a union member naming no declared event is a hard error (a stale union member would otherwise silently badge nothing). Everything else renders **log-only**. +- **A dedicated fence.** Payload blocks use a ` ```ts persistence-catalog ` info string that `doc-typecheck` recognizes and skips, excluded from the opt-out ratio — the same treatment as `ts cordis-catalog` (a bare payload fragment is not standalone-compilable). +- **Repo scope.** The catalog enumerates the packages in this repo, matching the siblings' packages-only scope; a downstream plugin can merge further event types, which are outside the catalog by construction. Nothing else in the repo may name an interface `SessionEventMap` — the walk treats every such declaration as the merged vocabulary, and a duplicate member across declarations is a hard error. + +This supersedes the hand-copies: the session.md `hook/*` table, the compact README's event table, the hook-protocol README's payload bullets, and the session README's name-list now link the catalog instead of restating payloads (the surrounding semantics prose stays where it was). The two stray `@mode emit` tags on the hook-protocol merge members are removed — the new gate rejects them as the category error they were. + +## Consequences + +- The catalog cannot drift: a vocabulary change the committed file doesn't reflect fails `verify-persistence-catalog` in the pre-push hook and CI, and a new merged event with no JSDoc fails the generator outright — a plugin can no longer add an undocumented on-disk record type. +- Event prose has a single home, the JSDoc at the declaration; thin JSDoc yields a thin catalog entry, pressuring authors to document at the source. +- The `SurfaceEventType` union is now structurally load-bearing for docs: renaming an event without updating the union (or vice versa) fails the generator, not just the compiler. +- The badge derivation assumes the union stays a closed set of string literals with exactly one owner; a refactor away from that shape must update the generator in the same change. diff --git a/package.json b/package.json index 7d136278c6..25c77e24ff 100644 --- a/package.json +++ b/package.json @@ -39,10 +39,12 @@ "verify-cordis-catalog": "tsx scripts/gen-cordis-catalog.ts --check", "gen-tool-catalog": "tsx scripts/gen-tool-catalog.ts", "verify-tool-catalog": "tsx scripts/gen-tool-catalog.ts --check", + "gen-persistence-catalog": "tsx scripts/gen-persistence-catalog.ts", + "verify-persistence-catalog": "tsx scripts/gen-persistence-catalog.ts --check", "gen-module-graph": "tsx scripts/gen-module-graph.ts", "verify-module-graph": "tsx scripts/gen-module-graph.ts --check", "constraints": "tsx scripts/check-workspace-constraints.ts", - "doc-sync": "pnpm run doc-typecheck && pnpm run verify-cordis-catalog && pnpm run verify-tool-catalog && pnpm run verify-md-wrap && pnpm run verify-md-links && pnpm run verify-doc-refs && pnpm run verify-package-paths && pnpm run verify-rfc-classification && pnpm run verify-type-equiv && pnpm run verify-translation-pairing && pnpm run verify-doc-budgets", + "doc-sync": "pnpm run doc-typecheck && pnpm run verify-cordis-catalog && pnpm run verify-tool-catalog && pnpm run verify-persistence-catalog && pnpm run verify-md-wrap && pnpm run verify-md-links && pnpm run verify-doc-refs && pnpm run verify-package-paths && pnpm run verify-rfc-classification && pnpm run verify-type-equiv && pnpm run verify-translation-pairing && pnpm run verify-doc-budgets", "hygiene": "pnpm run knip && pnpm run publint && pnpm run constraints && pnpm run verify-node-next-types", "demo:echo": "node --expose-internals --import tsx packages/ui/stdio-agent/src/bin.ts examples/echo-agent/cordis.yml", "demo:repl": "node --expose-internals --import tsx packages/ui/stdio-agent/src/bin.ts examples/coding-agent/cordis.yml", diff --git a/packages/compact/compact/README.md b/packages/compact/compact/README.md index 424ee12bcf..b49070c3b5 100644 --- a/packages/compact/compact/README.md +++ b/packages/compact/compact/README.md @@ -43,13 +43,7 @@ Compaction is serialized via a log-recorded lock: `compactRegion` refuses to sta ## Events -The `compact/*` events extend `SessionEventMap` (merge-extensible) via declaration merging — they are session events, not cordis `Events`: - -| Event | Payload | On surface? | -|---|---|---| -| `compact/start` | `{ turn }` | no (log-only) | -| `compact/summary` | `{ summary, shadowedRange, shadowedSeqs, shadowedTokenCount }` | no (log-only) | -| `compact/end` | `{ turn, error? }` | no (log-only) | +The `compact/*` events extend `SessionEventMap` (merge-extensible) via declaration merging — they are session events, not cordis `Events`, and all three are log-only (no `surfaceOp`). Per-event payloads and semantics are in the generated [persistence log event catalog](../../../docs/persistence-catalog/log-events.md). ## Implementing a backend diff --git a/packages/core/session/README.md b/packages/core/session/README.md index 96d267c79b..72ca52bfef 100644 --- a/packages/core/session/README.md +++ b/packages/core/session/README.md @@ -49,9 +49,9 @@ Plain class (not a Cordis Service). Create via `ctx.sessions.create()`. ### Session event vocabulary (`types.ts`) -The append-only log: `turn/start`, `turn/end`, `step/start`, `step/end`, `user/message`, `prompt/blocked`, `assistant/message`, `assistant/chunk`, `tool/call`, `tool/result`, `steering/message`, `context/message`, `todo/write`. Token usage rides on `assistant/message.usage`; an operational error's step is on `turn/end.reason` for `kind: 'error'`. +The append-only log's event types, enumerated member by member — payloads, surface badges, provenance — in the generated [persistence log event catalog](../../../docs/persistence-catalog/log-events.md). Token usage rides on `assistant/message.usage`; an operational error's step is on `turn/end.reason` for `kind: 'error'`. -Merge-extensible via `SessionEventMap` — the compaction seam adds `compact/start`, `compact/summary`, and `compact/end`. +Merge-extensible via `SessionEventMap` — a plugin declaration-merges its own types (the compaction seam's `compact/*`, the hook bridges' `hook/*`); merged members appear in the same catalog. Also defines `TurnTriggerMap` and `TurnEndReasonMap` (merge-extensible sum types for typed turn boundaries — `kind`-tagged instead of strings). diff --git a/packages/core/session/src/types.ts b/packages/core/session/src/types.ts index b9a4c3b89b..38f7d91e67 100644 --- a/packages/core/session/src/types.ts +++ b/packages/core/session/src/types.ts @@ -198,9 +198,22 @@ export interface TodoItem { * the invariants plugin checks, is a breaking change to the on-disk format. */ export interface SessionEventMap { + /** + * Opens turn `turn`. `trigger` records what started it — a drained message + * batch, a continuation, or an idle-time injection. The turn is the + * durability/replay boundary: every event sits between a `turn/start` and its + * matching `turn/end` (the turn-enclosure invariant). + */ 'turn/start': { turn: number; trigger: TurnTrigger } + /** + * Closes turn `turn` with the {@link TurnEndReason} that ended it. The loop + * fires the awaited `session/flush` checkpoint at every turn end, so the turn + * boundary is also the durable-commit boundary. + */ 'turn/end': { turn: number; reason: TurnEndReason } + /** Opens step `step` of turn `turn` — one model call plus the tool executions it requested. */ 'step/start': { turn: number; step: number } + /** Closes step `step` of turn `turn`. */ 'step/end': { turn: number; step: number } /** A user-visible prompt (queued message drained at turn start). */ 'user/message': { content: ContentBlock[]; source: MessageSource } @@ -230,6 +243,11 @@ export interface SessionEventMap { * usage record). `usage` is absent when the adapter reported none. */ 'assistant/message': { turn: number; step: number; content: ContentBlock[]; usage?: TokenUsage } + /** + * The model requested one tool invocation: `name` with the raw `arguments` + * JSON string exactly as the model produced it (unparsed). `callId` pairs the + * call with its `tool/result`. + */ 'tool/call': { turn: number; step: number; callId: CallId; name: string; arguments: string } /** * A completed tool call's model-facing result, plus an optional tool-private diff --git a/packages/core/session/tests/gen-persistence-catalog.spec.ts b/packages/core/session/tests/gen-persistence-catalog.spec.ts new file mode 100644 index 0000000000..567fe0ffd5 --- /dev/null +++ b/packages/core/session/tests/gen-persistence-catalog.spec.ts @@ -0,0 +1,172 @@ +/** + * Negative-path tests for the persistence log catalog generator + * (`scripts/gen-persistence-catalog.ts`). + * + * The generated catalog is frozen by a regenerate-and-diff freshness gate, so + * the freshness half is exercised by `pnpm run verify-persistence-catalog` in + * CI. What a freshness diff CANNOT prove is that the generator REJECTS + * malformed source the way it promises to — a member without description + * prose, a forbidden `@mode` tag, a non-literal member name, a duplicate event + * declaration, a missing or ambiguous `SurfaceEventType` union, a stale union + * member. These tests drive the exported collectors against synthetic fixture + * packages to prove each guard fires (and that well-formed declarations pass), + * mirroring the gen-cordis-catalog negative tests. + */ + +import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, describe, expect, it } from 'vitest' +import { + annotateSurface, + collectLogEvents, + collectSurfaceEventTypes, + render, +} from '../../../../scripts/gen-persistence-catalog.ts' + +/** Create a fixture scan root; `files` maps `packages/…`-relative paths to source. */ +function fixtureRoot(files: Record): string { + const root = mkdtempSync(join(tmpdir(), 'persistence-catalog-')) + for (const [rel, source] of Object.entries(files)) { + const abs = join(root, rel) + mkdirSync(join(abs, '..'), { recursive: true }) + writeFileSync(abs, source) + } + return root +} + +const roots: string[] = [] +const make = (files: Record): string => { + const r = fixtureRoot(files) + roots.push(r) + return r +} + +/** A merge-form declaration file wrapping `members` in the session module. */ +const merge = (members: string): string => + `declare module '@deepseek-ai/dsh-session' {\n interface SessionEventMap {\n${members}\n }\n}\n` + +afterEach(() => { + while (roots.length) rmSync(roots.pop()!, { recursive: true, force: true }) +}) + +describe('gen-persistence-catalog collectLogEvents', () => { + it('extracts a documented member of the owning top-level interface', () => { + const events = collectLogEvents(make({ + 'packages/core/fix/src/types.ts': + 'export interface SessionEventMap {\n /** A thing was recorded. */\n \'fix/happened\': { turn: number }\n}\n', + })) + expect(events).toHaveLength(1) + expect(events[0]).toMatchObject({ + name: 'fix/happened', + scope: 'fix', + doc: 'A thing was recorded.', + payload: '{ turn: number }', + source: 'packages/core/fix/src/types.ts:3', + }) + }) + + it('extracts a member declaration-merged via the session module', () => { + const events = collectLogEvents(make({ + 'packages/group/fix/src/types.ts': merge(' /** Merged provenance. */\n \'fix/merged\': { id: string }'), + })) + expect(events).toHaveLength(1) + expect(events[0]).toMatchObject({ name: 'fix/merged', doc: 'Merged provenance.' }) + }) + + it('collapses a newline-separated multi-line payload to a valid one-line fragment', () => { + const events = collectLogEvents(make({ + 'packages/group/fix/src/types.ts': merge( + ' /** Wide payload. */\n \'fix/wide\': {\n alpha: string[]\n range: { start: number; end: number }\n count: number\n }', + ), + })) + expect(events[0]?.payload).toBe('{ alpha: string[]; range: { start: number; end: number }; count: number }') + }) + + it('hard-errors on a member with no description prose', () => { + expect(() => collectLogEvents(make({ + 'packages/group/fix/src/types.ts': merge(' \'fix/undocumented\': { turn: number }'), + }))).toThrow(/no description prose/) + }) + + it('hard-errors on an @mode tag (a log event has no dispatch mode)', () => { + expect(() => collectLogEvents(make({ + 'packages/group/fix/src/types.ts': merge(' /**\n * Documented, but mistagged.\n * @mode emit\n */\n \'fix/tagged\': { turn: number }'), + }))).toThrow(/carries an @mode tag/) + }) + + it('hard-errors on a non-literal member name', () => { + expect(() => collectLogEvents(make({ + 'packages/group/fix/src/types.ts': merge(' /** Not a literal. */\n unquoted: { turn: number }'), + }))).toThrow(/non-literal name/) + }) + + it('hard-errors when the same event is declared twice', () => { + expect(() => collectLogEvents(make({ + 'packages/group/fix/src/a.ts': merge(' /** First. */\n \'fix/dup\': { turn: number }'), + 'packages/group/fix/src/b.ts': merge(' /** Second. */\n \'fix/dup\': { turn: number }'), + }))).toThrow(/already declared at packages\/group\/fix\/src\/a\.ts/) + }) + + it('aggregates every violation into one error instead of failing fast', () => { + expect(() => collectLogEvents(make({ + 'packages/group/fix/src/types.ts': merge(' \'fix/one\': { turn: number }\n \'fix/two\': { turn: number }'), + }))).toThrow(/2 JSDoc completeness violation\(s\)[\s\S]*fix\/one[\s\S]*fix\/two/) + }) +}) + +describe('gen-persistence-catalog collectSurfaceEventTypes', () => { + it('parses the literal union', () => { + const types = collectSurfaceEventTypes(make({ + 'packages/core/fix/src/types.ts': 'export type SurfaceEventType = \'fix/a\' | \'fix/b\'\n', + })) + expect(types).toEqual(['fix/a', 'fix/b']) + }) + + it('hard-errors when no union is declared', () => { + expect(() => collectSurfaceEventTypes(make({ + 'packages/core/fix/src/types.ts': 'export const unrelated = 1\n', + }))).toThrow(/no SurfaceEventType union found/) + }) + + it('hard-errors when the union is declared more than once', () => { + expect(() => collectSurfaceEventTypes(make({ + 'packages/core/fix/src/a.ts': 'export type SurfaceEventType = \'fix/a\'\n', + 'packages/core/fix/src/b.ts': 'export type SurfaceEventType = \'fix/b\'\n', + }))).toThrow(/declared more than once/) + }) + + it('hard-errors on a non-string-literal union member', () => { + expect(() => collectSurfaceEventTypes(make({ + 'packages/core/fix/src/types.ts': 'export type SurfaceEventType = \'fix/a\' | number\n', + }))).toThrow(/non-string-literal member/) + }) +}) + +describe('gen-persistence-catalog annotateSurface + render', () => { + const entry = (name: string) => ({ + name, + scope: name.split('/')[0] ?? name, + payload: '{ turn: number }', + doc: `Records ${name}.`, + source: 'packages/core/fix/src/types.ts:3', + }) + + it('badges union members surface and everything else log-only', () => { + const annotated = annotateSurface([entry('fix/message'), entry('fix/marker')], ['fix/message']) + expect(annotated.map(e => [e.name, e.surface])).toEqual([['fix/message', true], ['fix/marker', false]]) + }) + + it('hard-errors on a union member naming no declared event', () => { + expect(() => annotateSurface([entry('fix/marker')], ['fix/ghost'])) + .toThrow(/'fix\/ghost' name no declared log event/) + }) + + it('renders badges, payload fences, and the generated-file header', () => { + const out = render(annotateSurface([entry('fix/message'), entry('fix/marker')], ['fix/message'])) + expect(out).toContain('Generated by scripts/gen-persistence-catalog.ts') + expect(out).toContain('#### `fix/message` — surface') + expect(out).toContain('#### `fix/marker` — log-only') + expect(out).toContain('```ts persistence-catalog\n\'fix/marker\': { turn: number }\n```') + }) +}) diff --git a/packages/hooks/hook-protocol/README.md b/packages/hooks/hook-protocol/README.md index 8478f8aa74..8371c1d388 100644 --- a/packages/hooks/hook-protocol/README.md +++ b/packages/hooks/hook-protocol/README.md @@ -23,10 +23,7 @@ Why a shared lib at all: Codex deliberately reimplements a *subset* of the Claud ## `hook/*` session events -Declaration-merged into `SessionEventMap` (log-only, like `compact/*` — NOT a `SurfaceEventType`, no `surfaceOp`): - -- `hook/invoked` — `{ turn, point, dialect, matcher?, handlerId }`: a hook command ran. -- `hook/result` — `{ turn, point, handlerId, decision, exitCode?, stderrSummary?, durationMs }`: its outcome, paired by `handlerId`. +Declaration-merged into `SessionEventMap` (log-only, like `compact/*` — NOT a `SurfaceEventType`, no `surfaceOp`): `hook/invoked` (a hook command ran) and `hook/result` (its outcome, paired by `handlerId`). Payloads and per-event JSDoc are in the generated [persistence log event catalog](../../../docs/persistence-catalog/log-events.md). Like every event they must sit inside an open turn. The mid-turn points (`PreToolUse`/`PostToolUse`/`UserPromptSubmit`/`Stop`) fire inside the loop's open turn by construction; `SessionStart` gets no `hook/*` record (its injected `context/message` is the durable evidence) — see the hooks RFC. diff --git a/packages/hooks/hook-protocol/src/types.ts b/packages/hooks/hook-protocol/src/types.ts index c3b75e7c08..2c445c4652 100644 --- a/packages/hooks/hook-protocol/src/types.ts +++ b/packages/hooks/hook-protocol/src/types.ts @@ -23,7 +23,6 @@ declare module '@deepseek-ai/dsh-session' { * pattern that selected it (absent for match-all), `handlerId` a stable id * for the command (so an invoked/result pair correlates). `turn` is the open * turn the invocation lives inside. - * @mode emit */ 'hook/invoked': { turn: number @@ -39,7 +38,6 @@ declare module '@deepseek-ai/dsh-session' { * `exitCode` the process exit (absent if it never ran), `stderrSummary` a * truncated stderr (the block reason source on exit 2), `durationMs` the wall * time. `turn` matches the `hook/invoked`. - * @mode emit */ 'hook/result': { turn: number diff --git a/scripts/doc-typecheck.ts b/scripts/doc-typecheck.ts index 7422d6ac77..c6a5510948 100644 --- a/scripts/doc-typecheck.ts +++ b/scripts/doc-typecheck.ts @@ -9,13 +9,15 @@ * opts out with an explicit ` ```ts ignore-check ` info string — the opt-out * is visible in the source, and this script reports the ratio so the escape * hatch can't quietly become the norm. A third info string, - * doc-typecheck.ts recognizes two more fence variants and skips both (each is a - * separately-checked category, not an unchecked sketch, so neither counts in the - * opt-out ratio): ` ```ts type-equiv ` is a verbatim source-type paste that - * `scripts/verify-type-equiv.ts` drift-checks, and ` ```ts cordis-catalog ` is a + * doc-typecheck.ts recognizes three more fence variants and skips all three (each + * is a separately-checked category, not an unchecked sketch, so none counts in + * the opt-out ratio): ` ```ts type-equiv ` is a verbatim source-type paste that + * `scripts/verify-type-equiv.ts` drift-checks, ` ```ts cordis-catalog ` is a * generated event/service signature fragment in the cordis catalog (a bare * signature is not standalone-compilable; the catalog is generated and frozen by - * `scripts/gen-cordis-catalog.ts` + its `--check` freshness gate). + * `scripts/gen-cordis-catalog.ts` + its `--check` freshness gate), and + * ` ```ts persistence-catalog ` is a generated log-event payload fragment in the + * persistence catalog (same reasoning, frozen by `scripts/gen-persistence-catalog.ts`). * * Run: `tsx scripts/doc-typecheck.ts`. */ @@ -43,8 +45,12 @@ const root = resolve(import.meta.dirname, '..') * (a bare signature fragment has no imports and does not stand alone) and * EXCLUDED from the opt-out ratio: the catalog is generated and frozen by * `scripts/gen-cordis-catalog.ts` + its `--check` freshness gate. + * - `persistence-catalog` (` ```ts persistence-catalog `) — a generated + * log-event payload fragment in the persistence catalog. Same treatment for + * the same reason; frozen by `scripts/gen-persistence-catalog.ts` + its + * `--check` freshness gate. */ -type BlockKind = 'check' | 'ignore' | 'type-equiv' | 'cordis-catalog' +type BlockKind = 'check' | 'ignore' | 'type-equiv' | 'cordis-catalog' | 'persistence-catalog' /** One extracted code block. */ interface Block { @@ -55,7 +61,8 @@ interface Block { code: string } -/** Extract every ts / ts ignore-check / ts type-equiv / ts cordis-catalog block from one Markdown file. */ +/** Extract every ts / ts ignore-check / ts type-equiv / ts cordis-catalog / + * ts persistence-catalog block from one Markdown file. */ function extractBlocks(absPath: string): Block[] { const text = readFileSync(absPath, 'utf8') const lines = text.split('\n') @@ -82,7 +89,8 @@ function extractBlocks(absPath: string): Block[] { : info === 'ts ignore-check' ? 'ignore' : info === 'ts type-equiv' ? 'type-equiv' : info === 'ts cordis-catalog' ? 'cordis-catalog' - : null + : info === 'ts persistence-catalog' ? 'persistence-catalog' + : null if (kind) open = { line: i + 1, kind, body: [] } }) return blocks @@ -131,11 +139,11 @@ files.sort() const all = files.flatMap(extractBlocks) const checked = all.filter(b => b.kind === 'check') const ignored = all.filter(b => b.kind === 'ignore') -// `type-equiv` and `cordis-catalog` blocks are verified elsewhere -// (verify-type-equiv.ts and the gen-cordis-catalog `--check` freshness gate), -// not here: neither compiled nor counted toward the opt-out ratio (each is a -// separate fully-checked category, not an unchecked sketch). The ratio's -// denominator is therefore the compile-eligible blocks only. +// `type-equiv`, `cordis-catalog`, and `persistence-catalog` blocks are verified +// elsewhere (verify-type-equiv.ts and each catalog generator's `--check` +// freshness gate), not here: neither compiled nor counted toward the opt-out +// ratio (each is a separate fully-checked category, not an unchecked sketch). +// The ratio's denominator is therefore the compile-eligible blocks only. const ratioDenominator = checked.length + ignored.length if (checked.length === 0) { @@ -171,7 +179,7 @@ try { const ratio = ignored.length / ratioDenominator const skipped = all.length - ratioDenominator - console.log(`doc-typecheck: ${checked.length} block(s) compiled, ${ignored.length} ignored (${(ratio * 100).toFixed(0)}% opt-out), ${skipped} type-equiv/cordis-catalog (checked elsewhere).`) + console.log(`doc-typecheck: ${checked.length} block(s) compiled, ${ignored.length} ignored (${(ratio * 100).toFixed(0)}% opt-out), ${skipped} type-equiv/catalog (checked elsewhere).`) // Guard against the escape hatch becoming the norm. if (ratioDenominator >= 4 && ratio > 0.5) { console.error(`doc-typecheck: too many blocks opt out of checking (${ignored.length}/${ratioDenominator}). Make them compile or delete them.`) diff --git a/scripts/gen-persistence-catalog.ts b/scripts/gen-persistence-catalog.ts new file mode 100644 index 0000000000..31a1e99ac8 --- /dev/null +++ b/scripts/gen-persistence-catalog.ts @@ -0,0 +1,388 @@ +/** + * Generate (and verify) the persistence log event catalog in + * docs/persistence-catalog/log-events.md. + * + * The catalog is the ON-DISK-vocabulary reference: every event type that can + * appear in a session's durable event log — every member of the + * merge-extensible `SessionEventMap`, across the owning declaration in + * `@deepseek-ai/dsh-session` and every plugin declaration merge. It complements + * the cordis events/services catalog (the live bus wiring — a log event is NOT + * a cordis event; it reaches listeners via the single `session/event` emit) and + * the core-data-structures session page (the `SessionEvent` envelope and + * derivation semantics): this page is the RECORDS a persisted log can contain. + * + * `tsx scripts/gen-persistence-catalog.ts` → write the catalog + * `tsx scripts/gen-persistence-catalog.ts --check` → exit 1 if the committed + * file is stale (CI / + * pre-push gate) + * + * Like its AST sibling `gen-cordis-catalog.ts` (and unlike the boot-based + * `gen-tool-catalog.ts`), this is a pure source pass: every log event is a + * string-literal-named property with a static type annotation, so the AST is + * the whole truth and a brand-new event (core or merged) appears in the next + * regenerate — an un-regenerated file fails `--check`. The walk enforces JSDoc + * COMPLETENESS on the whole vocabulary: every member carries description prose + * (it becomes the catalog entry), and an `@mode` tag on a member is a hard + * error — dispatch modes belong to cordis bus events, and a log event has none + * (see docs/rfc/implemented/process/2026-07-04-persistence-log-catalog.md). + * Violations aggregate into ONE error listing every offender. + * + * The surface/log-only badge is parsed from the `SurfaceEventType` union in the + * owning package (never hand-listed here), and every union member must name a + * collected event — a stale union member is a hard error. + * + * Payload fences use the ` ```ts persistence-catalog ` info string: + * doc-typecheck recognizes it and skips compilation (a bare payload fragment is + * not standalone-compilable), excluded from the opt-out ratio. + */ + +import { globSync, readFileSync, writeFileSync } from 'node:fs' +import { resolve } from 'node:path' +import ts from 'typescript' + +const root = resolve(import.meta.dirname, '..') +const OUT = 'docs/persistence-catalog/log-events.md' + +/** The fenced-block info string for generated payload blocks (skipped by + * doc-typecheck, since a bare payload fragment is not standalone-compilable). */ +const FENCE = 'ts persistence-catalog' + +/** The package whose module id plugin merges augment (`declare module '…'`). */ +const SESSION_MODULE = '@deepseek-ai/dsh-session' + +/** + * Cross-link map: a type name that appears in a payload → the + * core-data-structures page that documents it (path relative to OUT's folder). + * Hand-curated and catalog-owned, same policy as the cordis catalog's map: each + * name resolves to exactly one PRIMARY page. A payload type with no + * core-data-structures home (e.g. `HookDialect`, documented in its package) + * simply gets no link. + */ +const LINK_MAP: Record = { + CallId: 'core.md', + ContentBlock: 'core.md', + MessageSource: 'core.md', + StreamChunk: 'llm-streaming.md', + TokenUsage: 'llm-streaming.md', + TodoItem: 'session.md', + TurnTrigger: 'session.md', + TurnEndReason: 'session.md', +} + +/** One log event, extracted from a `SessionEventMap` declaration. */ +export interface LogEventEntry { + /** Scoped name, e.g. `turn/start`. */ + name: string + /** The scope prefix, e.g. `turn` (everything before the first `/`). */ + scope: string + /** Payload type text (the member's type annotation, whitespace-collapsed). */ + payload: string + /** Description prose (the member's JSDoc), one line per paragraph. */ + doc: string + /** Source pointer `packages/…/file.ts:line` of the declaration. */ + source: string +} + +/** A {@link LogEventEntry} plus its surface-eligibility badge. */ +export interface AnnotatedLogEventEntry extends LogEventEntry { + /** Whether the type is a `SurfaceEventType` member (may carry `surfaceOp`). */ + surface: boolean +} + +/** Repo-relative source pointer `file:line` for a node's first character. */ +function pointer(rel: string, sf: ts.SourceFile, node: ts.Node): string { + const { line } = sf.getLineAndCharacterOfPosition(node.getStart(sf)) + return `${rel}:${line + 1}` +} + +const printer = ts.createPrinter({ removeComments: true }) + +/** + * One-line payload text for a member's type annotation. Printed through the + * TypeScript printer (not sliced from source text): the printer emits `;` + * member separators regardless of how the source separated them, so a + * multi-line newline-separated type literal still collapses to a VALID + * single-line fragment. The trailing `;` the printer puts before every `}` is + * dropped to match the repo's inline-literal style. + */ +function payloadText(type: ts.TypeNode, sf: ts.SourceFile): string { + return printer.printNode(ts.EmitHint.Unspecified, type, sf) + .replace(/\s+/g, ' ') + .replace(/;\s*\}/g, ' }') + .trim() +} + +/** The raw `/** … *​/` JSDoc block immediately preceding a node, or '' if none. */ +function rawJsDoc(text: string, node: ts.Node): string { + const ranges = ts.getLeadingCommentRanges(text, node.getFullStart()) ?? [] + const jsdoc = ranges.filter(r => text.slice(r.pos, r.pos + 3) === '/**').at(-1) + return jsdoc ? text.slice(jsdoc.pos, jsdoc.end) : '' +} + +/** + * Parse a raw JSDoc block into description prose, flagging whether any `@mode` + * tag is present (forbidden on log events). Output obeys the repo's markdown + * conventions so the generated file passes verify-md-wrap: each prose paragraph + * collapses to ONE physical line, and a `-` bullet list is preserved with each + * item on its own single line (continuation lines folded in). `{@link Foo}` + * unwraps to `Foo`. Description prose ends at the FIRST block tag (standard + * JSDoc semantics): tag lines and their continuation lines are never prose. + */ +function parseJsDoc(raw: string): { doc: string; hasMode: boolean } { + const inner = raw + .replace(/^\/\*\*/, '') + .replace(/\*\/$/, '') + .split('\n') + .map(l => l.replace(/^\s*\*?\s?/, '').replace(/\s+$/, '')) + let hasMode = false + let inTags = false + const blocks: string[] = [] + let para: string[] = [] + let list: string[] = [] + let item: string[] = [] + const join = (parts: string[]): string => parts.join(' ').replace(/\s+/g, ' ').trim() + const flushItem = (): void => { + if (item.length) list.push(join(item)) + item = [] + } + const flushList = (): void => { + flushItem() + if (list.length) blocks.push(list.join('\n')) // one block, items on own lines + list = [] + } + const flushPara = (): void => { + flushList() + if (para.length) blocks.push(join(para)) + para = [] + } + for (const line of inner) { + if (/^@mode\b/.test(line)) { hasMode = true; flushPara(); inTags = true; continue } + if (line.startsWith('@')) { flushPara(); inTags = true; continue } + if (inTags) continue // block-tag territory: continuations are never prose + if (line.trim() === '') { flushPara(); continue } + if (/^-\s+/.test(line)) { + // A list item starts: a pending paragraph (e.g. an intro line directly + // above the list, no blank between) flushes FIRST so it renders above. + flushItem() + if (para.length) { blocks.push(join(para)); para = [] } + item.push(line) + continue + } + if (item.length) { item.push(line); continue } // continuation of current item + para.push(line) + } + flushPara() + const doc = blocks.join('\n\n').replace(/\{@link\s+([^}]+)\}/g, '$1').trim() + return { doc, hasMode } +} + +/** + * Throw one aggregate error for every completeness violation a walk collected. + * Aggregation is deliberate: a remediation pass sees the whole list at once + * instead of replaying the gate once per offender. + */ +function reportViolations(violations: string[]): void { + if (violations.length === 0) return + throw new Error( + `gen-persistence-catalog: ${violations.length} JSDoc completeness violation(s):\n` + + violations.map(v => ` ${v}`).join('\n'), + ) +} + +/** + * Every `interface SessionEventMap` declaration in a source file: the owning + * top-level declaration (in `@deepseek-ai/dsh-session`) and any declaration + * merge inside a `declare module '@deepseek-ai/dsh-session'` block. Both forms + * declare members of the SAME merged interface, so both are catalogued + * uniformly; nothing else in the repo may name an interface `SessionEventMap`. + */ +function sessionEventMapDecls(sf: ts.SourceFile): ts.InterfaceDeclaration[] { + const decls: ts.InterfaceDeclaration[] = [] + for (const stmt of sf.statements) { + if (ts.isInterfaceDeclaration(stmt) && stmt.name.text === 'SessionEventMap') decls.push(stmt) + if (ts.isModuleDeclaration(stmt) && ts.isStringLiteral(stmt.name) && stmt.name.text === SESSION_MODULE + && stmt.body && ts.isModuleBlock(stmt.body)) { + for (const inner of stmt.body.statements) { + if (ts.isInterfaceDeclaration(inner) && inner.name.text === 'SessionEventMap') decls.push(inner) + } + } + } + return decls +} + +/** + * Walk every `SessionEventMap` declaration (the owning interface plus every + * plugin declaration merge) and extract its events, hard-erroring (aggregated) + * on any completeness violation: a member without description prose, an + * `@mode` tag (a category error — log events have no dispatch mode), a + * non-literal member name, or the same event declared twice. + * `scanRoot` defaults to the repo root; tests pass a fixture dir. + */ +export function collectLogEvents(scanRoot: string = root): LogEventEntry[] { + const entries: LogEventEntry[] = [] + const violations: string[] = [] + const seen = new Map() + for (const rel of globSync('packages/*/*/src/**/*.ts', { cwd: scanRoot }).sort()) { + const abs = resolve(scanRoot, rel) + const text = readFileSync(abs, 'utf8') + if (!text.includes('SessionEventMap')) continue + const sf = ts.createSourceFile(abs, text, ts.ScriptTarget.Latest, true) + for (const decl of sessionEventMapDecls(sf)) { + for (const member of decl.members) { + if (!ts.isPropertySignature(member) || !member.type) continue + const src = pointer(rel, sf, member) + if (!ts.isStringLiteral(member.name)) { + violations.push(`log event at ${src} has a non-literal name; the catalog needs string-literal event names.`) + continue + } + const name = member.name.text + const where = `log event '${name}' (${src})` + const prior = seen.get(name) + if (prior) { + violations.push(`${where} is already declared at ${prior}; an event type has exactly one declaration.`) + continue + } + seen.set(name, src) + const payload = payloadText(member.type, sf) + const { doc, hasMode } = parseJsDoc(rawJsDoc(text, member)) + if (hasMode) { + violations.push(`${where} carries an @mode tag, but a log event has no dispatch mode (it is not a cordis bus event — it rides the 'session/event' emit). Remove the tag.`) + } + if (!doc) { + violations.push(`${where} has no description prose. Say what the event records and what its payload means — the JSDoc becomes the catalog entry.`) + } + entries.push({ name, scope: name.split('/')[0] ?? name, payload, doc, source: src }) + } + } + } + reportViolations(violations) + return entries +} + +/** + * Parse the `SurfaceEventType` union — the surface-eligible subset of event + * types — from source. Hard-errors when the alias is missing, declared more + * than once, or contains a non-string-literal member: the badge derivation + * relies on the union being a closed set of literal event names. + * `scanRoot` defaults to the repo root; tests pass a fixture dir. + */ +export function collectSurfaceEventTypes(scanRoot: string = root): string[] { + const found: { names: string[]; source: string }[] = [] + for (const rel of globSync('packages/*/*/src/**/*.ts', { cwd: scanRoot }).sort()) { + const abs = resolve(scanRoot, rel) + const text = readFileSync(abs, 'utf8') + if (!text.includes('SurfaceEventType')) continue + const sf = ts.createSourceFile(abs, text, ts.ScriptTarget.Latest, true) + for (const stmt of sf.statements) { + if (!ts.isTypeAliasDeclaration(stmt) || stmt.name.text !== 'SurfaceEventType') continue + const src = pointer(rel, sf, stmt) + const members = ts.isUnionTypeNode(stmt.type) ? [...stmt.type.types] : [stmt.type] + const names: string[] = [] + for (const m of members) { + if (ts.isLiteralTypeNode(m) && ts.isStringLiteral(m.literal)) names.push(m.literal.text) + else throw new Error(`gen-persistence-catalog: SurfaceEventType (${src}) has a non-string-literal member; the badge derivation needs a closed literal union.`) + } + found.push({ names, source: src }) + } + } + const only = found[0] + if (!only) throw new Error('gen-persistence-catalog: no SurfaceEventType union found under packages/*/*/src.') + if (found.length > 1) throw new Error(`gen-persistence-catalog: SurfaceEventType is declared more than once (${found.map(f => f.source).join(', ')}); the surface subset has exactly one owner.`) + return only.names +} + +/** + * Attach the surface/log-only badge to each event. Hard-errors when a + * `SurfaceEventType` union member names no collected event — a stale union + * member would otherwise silently badge nothing. + */ +export function annotateSurface(events: LogEventEntry[], surfaceTypes: string[]): AnnotatedLogEventEntry[] { + const names = new Set(events.map(e => e.name)) + const stale = surfaceTypes.filter(t => !names.has(t)) + if (stale.length > 0) { + throw new Error(`gen-persistence-catalog: SurfaceEventType member(s) ${stale.map(t => `'${t}'`).join(', ')} name no declared log event (stale union member?).`) + } + const surface = new Set(surfaceTypes) + return events.map(e => ({ ...e, surface: surface.has(e.name) })) +} + +/** Render the cross-link "Types:" line for a payload, or '' if none apply. */ +function typeLinks(payload: string): string { + const seen = new Set() + for (const name of Object.keys(LINK_MAP)) { + if (new RegExp(`\\b${name}\\b`).test(payload)) seen.add(name) + } + if (seen.size === 0) return '' + const links = [...seen].sort().map(n => `[${n}](../core-data-structures/${LINK_MAP[n]})`) + return `Types: ${links.join(' · ')}` +} + +/** Render one log event entry. */ +function renderEvent(e: AnnotatedLogEventEntry): string[] { + const out = [`#### \`${e.name}\` — ${e.surface ? 'surface' : 'log-only'}`, ''] + if (e.doc) out.push(e.doc, '') + out.push('```' + FENCE, `'${e.name}': ${e.payload}`, '```', '') + const links = typeLinks(e.payload) + if (links) out.push(links, '') + out.push(`Source: [\`${e.source}\`](../../${e.source.split(':')[0]})`, '') + return out +} + +/** Render the full catalog (pure, deterministic given the collected inputs). */ +export function render(events: AnnotatedLogEventEntry[]): string { + const lines: string[] = [ + '', + '', + '# Persistence Log Event Catalog', + '', + 'Every event type that can appear in a session\'s durable event log: each member of the merge-extensible `SessionEventMap` — the owning vocabulary in `@deepseek-ai/dsh-session` plus every plugin declaration merge in this repo — with the payload it carries, its surface badge, and the declaration it comes from. It complements [session.md](../core-data-structures/session.md) (the `SessionEvent` envelope, surface list, and `deriveMessages()` projection), [persistence.md](../core-data-structures/persistence.md) (how the log is made durable), and the [cordis catalog](../cordis-catalog/events-and-services.md) (the live bus wiring — a log event is NOT a cordis event; it reaches listeners via the single `session/event` emit).', + '', + 'This file is GENERATED from source (`scripts/gen-persistence-catalog.ts`) and verified fresh by `pnpm run verify-persistence-catalog` (part of `doc-sync`) — do not edit it by hand. Payload blocks use a `ts persistence-catalog` fence (skipped by doc-typecheck, since a bare payload fragment is not standalone-compilable). Type names in a payload link to the page that documents them. See [the persistence-log-catalog RFC](../rfc/implemented/process/2026-07-04-persistence-log-catalog.md).', + '', + 'The on-disk envelope around every payload is `SessionEvent` — `type`, monotonic `seq`, epoch-ms `time`, the `data` documented here, plus `surfaceOp`/`sourceEventSeqs` on **surface** events only ([envelope](../core-data-structures/session.md#sessioneventt--one-log-entry)). **surface** marks a `SurfaceEventType` member: it produces an LLM message and declares how it joins the surface list. **log-only** marks everything else: durable, replayable record with no derived-history contribution. Every payload is JSON-serializable (enforced at `Session.append`), and the whole format is pinned at `SESSION_FORMAT_VERSION = 0` — pre-release, no compatibility implied ([the version stance](../core-data-structures/persistence.md)). Scope: the packages in this repo; a downstream plugin can merge further event types, which are outside this catalog by construction.', + '', + '## Events', + '', + ] + const scopes = [...new Set(events.map(e => e.scope))].sort() + for (const scope of scopes) { + lines.push(`### \`${scope}/*\``, '') + for (const e of events.filter(x => x.scope === scope).sort((a, b) => a.name.localeCompare(b.name))) { + lines.push(...renderEvent(e)) + } + } + return lines.join('\n') +} + +/** CLI entry: default writes the catalog, `--check` fails if the committed copy + * is stale. Guarded behind an entry-point check so importing this module for + * tests neither regenerates the committed file nor calls process.exit. */ +function main(): void { + const content = render(annotateSurface(collectLogEvents(), collectSurfaceEventTypes())) + if (process.argv.includes('--check')) { + let committed: string | null = null + try { + committed = readFileSync(resolve(root, OUT), 'utf8') + } catch { + // Only ENOENT (not yet generated) is expected; a present-but-unreadable + // file is not a state this repo produces. Either way the remedy is the + // same — regenerate — so treat a read failure as "stale". + committed = null + } + if (committed === content) { + console.log(`gen-persistence-catalog: ${OUT} is up to date.`) + process.exit(0) + } + console.error(`gen-persistence-catalog: ${OUT} is stale. Run \`pnpm run gen-persistence-catalog\` and commit ${OUT}.`) + process.exit(1) + } + + writeFileSync(resolve(root, OUT), content) + console.log(`gen-persistence-catalog: wrote ${OUT}.`) +} + +// Run only when invoked as a script, not when imported by a test. +if (process.argv[1] && import.meta.filename === resolve(process.argv[1])) { + main() +} diff --git a/scripts/translation-pairing.manifest.json b/scripts/translation-pairing.manifest.json index 8c2708d48d..c9a88aa4e0 100644 --- a/scripts/translation-pairing.manifest.json +++ b/scripts/translation-pairing.manifest.json @@ -11,6 +11,7 @@ "docs/module-graph.md", "docs/cordis-catalog/", "docs/tool-catalog/", + "docs/persistence-catalog/", "docs/i18n/terminology.md" ] }