kevin/deepseek-harness
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document the package hierarchy and finalize the RFC

Browse Source 
Add a README to each group dir (core/llm/bash/session-persistence/ui/
support) stating its role and product-vs-support classification, and
rewrite packages/README.md around the hierarchy (group table, grouped
"what goes where", removed the package-hierarchy FIXME).

Move the package-hierarchy RFC to implemented/architecture/ and rewrite
it to describe what shipped (placement rationale, the paths-wildcard and
publint dedup, the two new guardrail gates). Fold the remaining
tsconfig.build.json references dedup into the discover-package-inventory
proposal and fix its cross-link.

Update AGENTS.md: regrouped repo-layout map, depth-2 globs, the new
verify-package-paths gate in the doc-sync listing, and a note that we
lean toward stricter lint in the agentic-coding era (machine-caught
errors and a consistent foundation outweigh the one-time cost).
This commit is contained in:
Tianyi Cui
2026-06-20 23:25:33 +08:00
parent ca2207e26c
commit 5024cd5757
13 changed files with 219 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+42 -26
View File
@@ -34,25 +34,37 @@ vendor/ Vendored Cordis framework source (original npm names, private).
See vendor/README.md for the manifest, local-modification log,
and the upstream sync procedure. Do NOT edit casually — every
divergence must be logged there.
packages/ Harness packages, all named @deepseek-ai/dsh-<name>:
llm/ abstract LLM service + content-block vocabulary
llm-deepseek/ DeepSeek API adapter (hand-rolled fetch/SSE)
llm-pi-ai/ DeepSeek adapter via @earendil-works/pi-ai (design twin)
session/ event-sourced session log + in-memory store
system-prompt/ prompt-section + tool-schema assembly registry
tools/ tool registry + tools/execute waterfall
agent/ Agent interface, registry, agent/* event vocabulary
agent-loop/ THE concrete plugin: ReactLoopAgent + the loop driver
invariants/ dev-mode event-contract invariants + session-log freeze
bash/ abstract bash executor seam (ctx.bash) — interface only
bash-local/ local-subprocess BashExecutor implementation
tool-bash/ model-facing bash/bash_output/bash_kill tool schemas
acp/ Agent Client Protocol bridge: drive the agent from an ACP
editor (Zed) over JSON-RPC stdio
ui-stdio/ minimal stdio (readline) UI plugin: renders agent/* events,
feeds stdin lines to the agent (shared by the demos)
llm-replay/ record/replay adapter: short-circuits llm/stream from a
recorded session JSONL (keyless snapshot tests)
packages/ Harness packages, grouped by role at packages/<group>/<pkg>/.
Every package is named @deepseek-ai/dsh-<pkg>; the group dir is a
pure container (no package.json). See packages/README.md and each
group's README.md for the product-vs-support split.
core/ product API spine
session/ event-sourced session log + in-memory store
system-prompt/ prompt-section + tool-schema assembly registry
tools/ tool registry + tools/execute waterfall
agent/ Agent interface, registry, agent/* event vocabulary
agent-loop/ THE concrete plugin: ReactLoopAgent + the loop driver
llm/ LLM capability family
llm/ abstract LLM service + content-block vocabulary
llm-deepseek/ DeepSeek API adapter (hand-rolled fetch/SSE)
llm-pi-ai/ DeepSeek adapter via @earendil-works/pi-ai (design twin)
bash/ bash capability family
bash/ abstract bash executor seam (ctx.bash) — interface only
bash-local/ local-subprocess BashExecutor implementation
tool-bash/ model-facing bash/bash_output/bash_kill tool schemas
session-persistence/ persistence capability family
session-persistence/ durable persistence seam + write coordinator
session-persistence-jsonl/ JSONL-sidecar backend
session-persistence-sqlite/ SQLite backend
ui/ product integration surfaces
acp/ Agent Client Protocol bridge: drive the agent from an ACP
editor (Zed) over JSON-RPC stdio
support/ dev/test/example infrastructure (lower compat expectations)
invariants/ dev-mode event-contract invariants + session-log freeze
ui-stdio/ minimal stdio (readline) UI plugin: renders agent/* events,
feeds stdin lines to the agent (shared by the demos)
llm-replay/ record/replay adapter: short-circuits llm/stream from a
recorded session JSONL (keyless snapshot tests)
examples/ Runnable demos (not workspaces; see examples/AGENTS.md). echo-agent
= mock model + echo tool + stdio UI + JSONL persistence, wired via
cordis.yml. coding-agent = the real thing: DeepSeek V4 + bash tools
@@ -83,7 +95,7 @@ scripts/ repo maintenance scripts (vendor-manifest guard, publint runner).
```sh
pnpm install # pnpm workspaces, node >= 24
pnpm run test # vitest run (packages|examples/*/tests/**/*.spec.ts)
pnpm run test:coverage # vitest run --coverage (per-file 100% gate on packages/*/src)
pnpm run test:coverage # vitest run --coverage (per-file 100% gate on packages/*/*/src)
pnpm run test:e2e # real-API tests (packages|examples/*/tests/**/*.e2e.ts);
# self-skips without DEEPSEEK_API_KEY — see Secrets below
pnpm run test:snapshot # ACP snapshot tests (examples/*/tests/**/*.snapshot.ts):
@@ -102,10 +114,10 @@ pnpm run lint # eslint .
pnpm run lint:fix # eslint . --fix
pnpm run build # tsc -b tsconfig.build.json && tsdown (JS bundles into lib/)
pnpm run knip # dead-code / unused-dependency check
pnpm run publint # package.json publish-correctness check (publishable packages/*)
pnpm run publint # package.json publish-correctness check (every packages/*/* package)
pnpm run hygiene # knip + publint + workspace constraints
pnpm run doc-typecheck # typecheck every ```ts block in README.md, docs/**/*.md,
# packages/*/*.md (doc/code drift gate)
# packages/*/*.md + packages/*/*/*.md (doc/code drift gate)
pnpm run gen-cordis-catalog # regenerate docs/cordis-catalog/events-and-services.md
# (events + services) from the interface Events / Context source
pnpm run verify-cordis-catalog # assert that generated catalog is not stale
@@ -113,10 +125,12 @@ pnpm run verify-md-wrap # assert no hard-wrapped prose paragraphs in README.md,
# docs/**/*.md, packages/*/*.md, AGENTS.md (one line per paragraph)
pnpm run verify-doc-refs # assert every docs/*.md path cited in a packages|examples
# TypeScript comment resolves (catches a moved/renamed doc)
pnpm run verify-package-paths # assert every packages/<path> cited in Markdown or a
# TypeScript comment resolves when it names a real (moved) package
pnpm run verify-rfc-classification # assert every RFC lives in a valid
# {lifecycle}/{class}/ folder and docs/rfc/README.md lists it
# under the matching heading (closed class set + index completeness)
pnpm run doc-sync # doc-typecheck + verify-cordis-catalog + verify-md-wrap + verify-md-links + verify-doc-refs + verify-rfc-classification + verify-type-equiv (CI runs this)
pnpm run doc-sync # doc-typecheck + verify-cordis-catalog + verify-md-wrap + verify-md-links + verify-doc-refs + verify-package-paths + verify-rfc-classification + verify-type-equiv (CI runs this)
pnpm run demo:echo # run examples/echo-agent (no API key; type "echo hi" to
# see a tool call) — the mock skeleton
pnpm run demo:coding # run examples/coding-agent — the real agent (needs
@@ -158,7 +172,7 @@ Dev/test/demo run **unbuilt** via tsx + the `paths` map in the root `tsconfig.js
- **Symmetry is usually more correct**: when two related values play parallel roles (a test fixture and its expected output, a request shape and its response shape, a buggy input and the test that checks the fix), give them parallel form — both named consts, or both inline, not one each way. Asymmetry is a smell that usually points at a missed extraction.
- **Merging PRs**: always merge with a **merge commit** (`gh pr merge --merge`), never squash or rebase. The per-PR commit history is intentional — review-fix commits, regression-test commits, and the reasoning in each message are part of the record — and squashing flattens it away.
- **TODO markers**: use `FIXME`/`TODO`/`XXX` to flag known issues by urgency — see [docs/development.md](docs/development.md) for the semantics of each.
- **Tests**: vitest, colocated under `packages/<name>/tests/*.spec.ts`. Every registry needs an HMR-safety test (dispose the contributing fiber, assert cleanup). **Excessive tests are welcome** — when in doubt, write the test; err on the side of covering edge cases, error paths, event ordering, and concurrency races even if they seem unlikely. Review findings get regression tests (see `packages/core/agent-loop/tests/review-fixes.spec.ts`). The same generosity applies to **real-API (with-key) e2e tests — inference is cheap here (we are DeepSeek), so do not ration them**: cover the agent's real flows (a real prompt that writes a file, multi-turn, tool use, cancellation) and run them frequently while developing, especially cheap **smoke tests** that boot the real example and check the world. A green mock/no-key suite proves the plumbing, not the product — the with-key smoke test is what catches "green units, broken product". See § Secrets / .env for the with-key policy and why self-skip is a CI accommodation, not a verdict that real-API tests are expensive.
- **Tests**: vitest, colocated under `packages/<group>/<pkg>/tests/*.spec.ts`. Every registry needs an HMR-safety test (dispose the contributing fiber, assert cleanup). **Excessive tests are welcome** — when in doubt, write the test; err on the side of covering edge cases, error paths, event ordering, and concurrency races even if they seem unlikely. Review findings get regression tests (see `packages/core/agent-loop/tests/review-fixes.spec.ts`). The same generosity applies to **real-API (with-key) e2e tests — inference is cheap here (we are DeepSeek), so do not ration them**: cover the agent's real flows (a real prompt that writes a file, multi-turn, tool use, cancellation) and run them frequently while developing, especially cheap **smoke tests** that boot the real example and check the world. A green mock/no-key suite proves the plumbing, not the product — the with-key smoke test is what catches "green units, broken product". See § Secrets / .env for the with-key policy and why self-skip is a CI accommodation, not a verdict that real-API tests are expensive.
- **Prefer the REAL implementation over a mock/stand-in in tests.** When the genuine collaborator is available in the repo, wire it up instead of hand-rolling a fake — a test that registers an inline `defineTool({ name: 'bash', … })` to stand in for `dsh-tool-bash` proves the *bridge* moves bytes but not that the *shipping tool* renders the way the test asserts; the two drift and the test passes while the product is wrong. Mock only the genuinely expensive/non-deterministic boundary (the LLM adapter, the network, the clock) and keep everything downstream real: a bridge tool-call test runs the scripted mock MODEL but the REAL tool + REAL executor (e.g. `makeBridgeHarness({ withBash: true })` plugs `dsh-bash-local` + `dsh-tool-bash` and runs an actual `echo`), so it verifies the actual `presentCall`/`presentResult` an editor sees. This is the unit-test echo of "verify the world, not a synthetic stand-in" (see § Defensive patterns) — a fake you wrote will agree with whatever you assumed; the real thing won't.
- **A change that affects the editor-facing transcript or end-to-end agent UX needs a snapshot test (or an explicit note in the PR why none applies).** The snapshot tier (`examples/*/tests/**/*.snapshot.ts`, `pnpm run test:snapshot`) boots the real example subprocess, replays a recorded session JSONL deterministically (keyless), and diffs the normalized stdout transcript + re-persisted session log against committed goldens — the full-transcript regression net that mock-level unit tests structurally cannot be (it is what catches a bridge-translation or loop-structure regression that leaves every unit green). When you change the ACP bridge, the agent loop's observable output, tool presentation, or anything an editor renders, add or update a scenario under `examples/acp-agent/tests/snapshots/` and re-record with `pnpm run test:snapshot:record`. Reviewing the golden diff is part of the review. The rule is scoped to transcript/UX-affecting changes — a pure internal refactor with no observable-output change does not need one, but say so. See [docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md](docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
@@ -178,11 +192,13 @@ Each bullet is a bug class that bit us; the rule prevents the reoccurrence.
## Type Safety and Documentation
This codebase aims to be **very type-safe and well documented** for maintainability. Code that fails to compile under `strict: true` (with `noImplicitAny` enabled for all `packages/*` source) is not acceptable. Every `any` that remains must have a specific justification (a comment explaining why a narrower type is infeasible).
This codebase aims to be **very type-safe and well documented** for maintainability. Code that fails to compile under `strict: true` (with `noImplicitAny` enabled for all `packages/*/*` source) is not acceptable. Every `any` that remains must have a specific justification (a comment explaining why a narrower type is infeasible).
**Almost always lean toward the stricter lint rule.** In the agentic-coding era the cost/benefit of strictness has inverted: a machine writes and reads most of the code, so the one-time cost of satisfying a stricter rule is cheap and paid by a tool, while the benefit — a whole class of error caught mechanically, a consistent foundation every agent can rely on, less reviewer attention spent on what a linter could have caught — compounds across every future change. When choosing whether to enable a rule, tighten an existing one, or add a new gate (a `verify-*` script, a constraint check), default to YES unless it has a concrete, recurring false-positive problem. Prefer a narrowly-scoped escape hatch (a justified inline disable with a reason, a per-path override) over leaving the rule off globally. The same reasoning motivates this repo's many bespoke gates (`doc-sync`, `verify-package-paths`, the workspace-shape constraint): encode the invariant in a check so no human or agent has to remember it.
In the **core** packages (`packages/llm/llm`, `packages/core/tools`, `packages/core/agent`, `packages/core/agent-loop`, `packages/core/session`, `packages/core/system-prompt`), **type gymnastics are acceptable when they improve the DX of plugin authors** for common plugin types. The `defineTool` typed schema DSL in `dsh-tools` is the canonical example: the `SchemaSpec` to `InferArgs<S>` type-level mapping gives tool authors zero-cast typed `execute` args, and the cost of the conditional types stays inside the core package.
Verbose documentation is fine **as long as docs and code stay strictly in sync**. Out-of-sync docs are worse than no docs. **When you change code, update its docs in the SAME change** — grep the package README and the module/JSDoc comments for the old behavior (config keys, defaults, error codes, wire field names, event names) and fix every hit. CI runs `pnpm run doc-sync` (`doc-typecheck` + `verify-cordis-catalog` + `verify-md-wrap` + `verify-md-links` + `verify-doc-refs` + `verify-rfc-classification` + `verify-type-equiv`), which typechecks every fenced `ts` block in `README.md`, `docs/**/*.md`, and `packages/*/*.md`, regenerates the cordis events/services catalog from source and fails if the committed copy is stale, asserts no hard-wrapped prose paragraphs, checks that every relative Markdown cross-link resolves, checks that every `docs/*.md` path cited in a source comment resolves, checks that every RFC is filed under a valid class folder and listed in its index, and checks that every ` ```ts type-equiv ` doc block still matches its source type — across those files plus `AGENTS.md` / `packages/AGENTS.md` — but that scope does NOT catch prose drift in `AGENTS.md` / `packages/AGENTS.md` / `packages/README.md` (config keys, defaults, error codes), so keeping those in sync remains on the author. Every module has a module-level doc comment explaining its role. Every exported class, interface, type, function, and non-obvious method has a JSDoc that explains semantics (not just the name) — contracts (what events fire when), disposal behavior, error behavior, and extension intent. Internal helpers get docs only where non-obvious. Prefer one-liners when one line suffices.
Verbose documentation is fine **as long as docs and code stay strictly in sync**. Out-of-sync docs are worse than no docs. **When you change code, update its docs in the SAME change** — grep the package README and the module/JSDoc comments for the old behavior (config keys, defaults, error codes, wire field names, event names) and fix every hit. CI runs `pnpm run doc-sync` (`doc-typecheck` + `verify-cordis-catalog` + `verify-md-wrap` + `verify-md-links` + `verify-doc-refs` + `verify-package-paths` + `verify-rfc-classification` + `verify-type-equiv`), which typechecks every fenced `ts` block in `README.md`, `docs/**/*.md`, and `packages/*/*.md`, regenerates the cordis events/services catalog from source and fails if the committed copy is stale, asserts no hard-wrapped prose paragraphs, checks that every relative Markdown cross-link resolves, checks that every `docs/*.md` path cited in a source comment resolves, checks that every `packages/<path>` reference naming a real package resolves, checks that every RFC is filed under a valid class folder and listed in its index, and checks that every ` ```ts type-equiv ` doc block still matches its source type — across those files plus `AGENTS.md` / `packages/AGENTS.md` — but that scope does NOT catch prose drift in `AGENTS.md` / `packages/AGENTS.md` / `packages/README.md` (config keys, defaults, error codes), so keeping those in sync remains on the author. Every module has a module-level doc comment explaining its role. Every exported class, interface, type, function, and non-obvious method has a JSDoc that explains semantics (not just the name) — contracts (what events fire when), disposal behavior, error behavior, and extension intent. Internal helpers get docs only where non-obvious. Prefer one-liners when one line suffices.
**Tag every new event with `@mode`.** The cordis events/services catalog ([docs/cordis-catalog/events-and-services.md](docs/cordis-catalog/events-and-services.md)) is GENERATED from source by `scripts/gen-cordis-catalog.ts` — never hand-edit it; run `pnpm run gen-cordis-catalog` and commit the result. When you add an event to an `interface Events` block, its JSDoc MUST carry a `@mode emit|waterfall|parallel` tag (the generator hard-errors without it): use `waterfall` when the signature ends with a `next: () => …` parameter (the listener transforms or vetoes via `next()`), `parallel` when the loop awaits a fan-out with no veto (e.g. an awaited `Promise<void> | void` checkpoint like `session/flush`), and `emit` for plain fire-and-forget notifications. The generator also cross-checks the tag against the signature where the shape is conclusive (a trailing `next` ⇒ waterfall) and hard-errors on a contradiction. Write the rest of the event's JSDoc to stand alone — it is the catalog entry's prose.
docs/rfc/README.md
+1 -1
View File
@@ -64,7 +64,6 @@ Do NOT write one for a mechanical or local choice (a variable name, a one-file r
| [Runtime schemas for the event vocabulary (Zod vs the merge-extensible-map pattern)](proposed/architecture/2026-06-16-typed-event-schemas.md) | 2026-06-16 |
| [Extract a generic long-running tool runtime](proposed/architecture/2026-06-20-generic-long-running-tool-runtime.md) | 2026-06-20 |
| [Make the shared example base providerless](proposed/architecture/2026-06-20-providerless-example-base.md) | 2026-06-20 |
| [Reorganize packages into a modular hierarchy](proposed/architecture/2026-06-20-package-hierarchy.md) | 2026-06-20 |
### Process
@@ -115,6 +114,7 @@ Do NOT write one for a mechanical or local choice (a variable name, a one-file r
| [Every session event is enclosed in a turn](implemented/architecture/2026-06-15-turn-enclosure-invariant.md) | 2026-06-15 |
| [Shared persistence write coordinator](implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md) | 2026-06-18 |
| [Agent lifecycle and ownership seams](implemented/architecture/2026-06-18-agent-lifecycle-and-ownership-seams.md) | 2026-06-18 |
| [Reorganize packages into a modular hierarchy](implemented/architecture/2026-06-20-package-hierarchy.md) | 2026-06-20 |
### Process
docs/rfc/implemented/architecture/2026-06-20-package-hierarchy.md
+67
View File
@@ -0,0 +1,67 @@
# RFC: Reorganize packages into a modular hierarchy
Status: implemented
## Problem
`packages/` was flat: 18 packages all sat at `packages/<name>/`, so a package's location said nothing about whether it was core product API, a swappable capability seam, a provider adapter, a product integration, or example/test support. The package README carried a `FIXME(package-hierarchy)` and `scripts/publint-all.ts` a `TODO(package-inventory)` flagging exactly this. Core packages, provider integrations, capability seams, example UI support, and snapshot-only replay support all looked equally foundational.
This was not just cosmetic. Because every top-level package looked like part of the same public surface, future removal was harder, and publish/lint/doc scripts had to encode intent through comments or hand-maintained static lists rather than reading it off the layout.
## What landed
Packages are grouped by modular role at a uniform `packages/<group>/<pkg>/` depth. Group directories are pure containers (no `package.json`); every package keeps its `@deepseek-ai/dsh-<pkg>` name — this is repo structure and maintenance policy, not package renaming.
```text
packages/
core/ (product API spine)
session/
system-prompt/
tools/
agent/
agent-loop/
llm/ (product — capability family)
llm/
llm-deepseek/
llm-pi-ai/
bash/ (product — capability family)
bash/
bash-local/
tool-bash/
session-persistence/ (product — capability family)
session-persistence/
session-persistence-jsonl/
session-persistence-sqlite/
ui/ (product integration)
acp/
support/ (dev/test/example infrastructure)
invariants/
ui-stdio/
llm-replay/
```
### Placement decisions
- **Same-name nesting for capability families.** A family's interface package sits at `packages/<group>/<group>/` (`llm/llm`, `bash/bash`, `session-persistence/session-persistence`), with implementations and consumers as flat siblings. There is no extra `adapters/`/`impls/` sub-tier — every package is exactly depth 2, which keeps the workspace glob a clean `packages/*/*` and lets one `@deepseek-ai/dsh-*` tsconfig wildcard resolve every package (unique dir names make first-on-disk-wins unambiguous).
- **`session` stays in `core/`; persistence is its own family.** The session log is core product API. Its storage backends form a parallel capability family (`session-persistence/`) mirroring `llm/` and `bash/`, rather than nesting under `core/session/`.
- **`agent-loop` is in `core/`.** It is the one concrete implementation of the `agent` seam, but it ships as the harness's default product loop, so it lives with the core spine. Plugins still depend on the `agent` vocabulary, never on `agent-loop`, so the loop stays swappable.
- **`invariants` and `ui-stdio` are `support/`, not product.** `invariants` is dev-mode contract checking. `ui-stdio` was extracted from the examples for reuse and the coverage gate — it is example-coupled, so it sits in `support/` alongside `llm-replay` (the snapshot-test replay adapter). `acp` is the only `ui/` member because it is a real product surface (the ACP bridge an editor drives), structurally distinct from the readline demo helper.
### Deduplicating the package lists
The package list had been enumerated in five places. The uniform depth-2 layout collapsed most of them:
- `tsconfig.base.json` and `tsconfig.typecheck.json` each replaced their 18 per-package `paths` entries with a single `@deepseek-ai/dsh-*` wildcard listing one candidate per group.
- `scripts/publint-all.ts` derives its list by reading the hierarchy (`packages/<group>/<pkg>`), resolving the `TODO(package-inventory)`.
- `tsconfig.build.json`'s project `references` stay an explicit list — TypeScript project references have no wildcard form. Generating these from a manifest is left to a follow-up (see [discover package inventories](../../proposed/process/2026-06-20-discover-package-inventory.md)).
### Guardrails added
Two doc-sync/hygiene gates keep the structure and its references honest, so the manual checks this restructure required do not have to be repeated by hand:
- `scripts/verify-package-paths.ts` flags a `packages/<path>` reference (in Markdown or a `.ts` comment/string) that does not resolve **and** names a real package in a segment — i.e. a stale path to a moved package. A path naming a package that exists nowhere (a forward-looking proposal) is left alone, so the gate applies uniformly across proposed/implemented/rejected.
- `scripts/check-workspace-constraints.ts` asserts the `packages/<group>/<pkg>` shape: group dirs carry no `package.json`, and no package sits flat at the root or nests deeper. Group names stay open — a new group may be added without editing the gate; only the depth-2 shape is fixed.
## What we gave up
The restructure churned imports, workspace globs, doc links, build references, and package paths in one coordinated move. That churn is acceptable pre-release (per the AGENTS.md foundation-over-blast-radius stance) because it stops the flat layout from fossilizing support packages as product contracts, and it is a one-time cost: the wildcard `paths`, the glob-derived publint list, and the shape gate mean a new package needs no further structural edits.
docs/rfc/proposed/architecture/2026-06-20-package-hierarchy.md
-59
View File
@@ -1,59 +0,0 @@
# RFC: Reorganize packages into a modular hierarchy
Status: proposed
## Problem
`packages/` is flat. Core product packages, provider integrations, capability seams, example UI support, and snapshot-only replay support all sit at the same level and look equally foundational. The [package README](../../../../packages/README.md) already has a `FIXME(package-hierarchy)` noting that `ui-stdio` and `llm-replay` were extracted from examples mostly for reuse and coverage. The flat layout makes support packages appear more product-shaped than they are and forces publish/lint/doc scripts to encode intent through comments or static lists.
This is not just cosmetic. A package's location currently says little about whether it is core API, a swappable capability, an adapter integration, an example harness helper, or test infrastructure. That makes future removal harder because every top-level package looks like part of the same public surface.
## Proposal
Move packages into a deliberate hierarchy under `packages/`. The exact layout is deferred to the implementing PR, but it should group packages by modular role rather than keep every package at one flat level.
One plausible shape:
```text
packages/
core/
session/
system-prompt/
tools/
agent/
agent-loop/
invariants/
llm/
llm/
adapters/
llm-deepseek/
llm-pi-ai/
bash/
bash/
bash-local/
tool-bash/
session-persistence/
session-persistence/
session-persistence-jsonl/
session-persistence-sqlite/
acp/
support/
ui-stdio/
llm-replay/
```
The final implementation may choose different names or groupings, but it should keep the same intent: core APIs, package families such as LLM/bash/session persistence, standalone integrations such as ACP, and support/test/example packages are distinguishable from the filesystem alone. Npm package names can stay `@deepseek-ai/dsh-*`; the hierarchy is about repo structure and maintenance policy, not public package renaming.
This proposal does not delete `llm-replay` or `ui-stdio` by itself. It makes their status honest: either they graduate into product packages with documented consumers, or they live under a support/testing/example classification where release and compatibility expectations are lower.
## Acceptance criteria
- Packages move from the flat `packages/<name>/` layout into a documented modular hierarchy.
- The implementing PR chooses the exact hierarchy and updates workspace globs, TypeScript paths, package docs, generated module graphs, `cordis.yml` package paths, build scripts, and publish/lint scripts in one coordinated move.
- Scripts that publish, lint publishability, or generate package inventories use the hierarchy instead of an ad hoc static list where the hierarchy is enough to express the policy.
- Docs explain which package groups are part of the product API and which groups are support/test/example infrastructure.
- New package guidance tells authors where to place a package and discourages new one-off top-level groups.
## What we give up
The restructure churns imports, workspace globs, docs links, and package paths. That churn is acceptable pre-release if it prevents the flat layout from fossilizing support packages as product contracts.
docs/rfc/proposed/process/2026-06-20-discover-package-inventory.md
+6 -4
View File
@@ -4,20 +4,22 @@ Status: proposed
## Problem
Package and gate inventories are repeated by hand. [scripts/publint-all.ts](../../../../scripts/publint-all.ts) has a static list of publishable packages. The [package cookbook](../../../cookbook/adding-a-package.md) tells authors to update several files. The [package README](../../../../packages/README.md) carries a hand-written dependency graph. [CI](../../../../.github/workflows/ci.yml) and [development docs](../../../development.md) can drift from the actual `doc-sync` subcommands when new gates are added. These lists are small today, but every new package or gate creates another manual synchronization point.
Package and gate inventories are repeated by hand. The [package cookbook](../../../cookbook/adding-a-package.md) tells authors to update several files. The [package README](../../../../packages/README.md) carries a hand-written dependency graph. [CI](../../../../.github/workflows/ci.yml) and [development docs](../../../development.md) can drift from the actual `doc-sync` subcommands when new gates are added. `tsconfig.build.json` lists all 18 packages as explicit project `references`. These lists are small today, but every new package or gate creates another manual synchronization point.
The [package hierarchy](../../implemented/architecture/2026-06-20-package-hierarchy.md) already removed several of these by hand: `scripts/publint-all.ts` now derives its list from the `packages/<group>/<pkg>` layout, and the two `tsconfig` `paths` maps collapsed to one `@deepseek-ai/dsh-*` wildcard. What remains is the inventory that cannot be globbed away — chiefly `tsconfig.build.json`'s project `references`, which TypeScript requires as an explicit array (no wildcard form).
Static lists are appropriate when they encode policy; they are needless friction when they duplicate manifest data or layout facts that already exist in `package.json`, workspace globs, or the package hierarchy.
## Proposal
Make package/gate inventories discoverable. Publishability should come from the deliberate [package hierarchy](../architecture/2026-06-20-package-hierarchy.md) plus package manifests, not from a static array in a script or the npm `private` flag. Module graph generation should read package manifests. `doc-sync` should be the one command that defines and prints its sub-gates, with docs linking to that command rather than restating a second list.
Make the remaining package/gate inventories discoverable. A single canonical source — the `packages/<group>/<pkg>` hierarchy plus package manifests — should drive `tsconfig.build.json`'s `references`, the module graph, and any other full-package list, with a generate-and-verify step (the existing `gen-module-graph` / `gen-cordis-catalog` pattern: a generator writes the artifact, a `--check` mode in `hygiene`/`doc-sync` fails on a stale committed copy). Module graph generation already reads package manifests. `doc-sync` should be the one command that defines and prints its sub-gates, with docs linking to that command rather than restating a second list.
The hierarchy does not need to encode every fact about a package, but it should encode the broad maintenance policy: core/product packages, integrations, capability seams, and support/test/example packages should not all require a hand-maintained exception list before scripts can tell them apart.
## Acceptance criteria
- `publint-all` discovers publishable packages from the hierarchy plus manifests instead of a hard-coded array.
- Adding a package does not require editing a static package list for every gate.
- `tsconfig.build.json` project `references` are generated from the hierarchy (a generator emits them; a `--check` gate fails when the committed copy is stale), rather than hand-maintained.
- Adding a package does not require editing a static package list for any gate.
- Docs describe the source of truth rather than repeating generated inventories.
- CI invokes the aggregate commands and lets those commands own their sub-gate lists.
packages/AGENTS.md
+2 -2
View File
@@ -7,12 +7,12 @@ This directory contains all `@deepseek-ai/dsh-*` harness packages. When editing
- **Waterfall semantics**: `ctx.waterfall` listeners receive `(...args, next)`; call `next()` to delegate, or return without it to short-circuit (veto). Never call `next()` after returning.
- **Plugin export shape — namespace OR default, never both.** A *service* package exports the service class as `export default` (the Loader instantiates it). A *function/namespace* plugin exports `name` / `inject` / `Config` / `apply` as separate named exports and **must NOT add `export default`** — the cordis Loader's `unwrapExports` does `exports.default ?? exports`, so a stray default export collapses the module to the bare `apply` function and silently discards the `inject`/`name`/`Config` namespace, leaving the plugin with no injected services (it then throws `cannot get property … without inject` at load). See [docs/postmortem/0001](../docs/postmortem/0001-acp-default-export-drops-inject.md).
- **Read an optional (non-injected) service via `ctx.get(name)`, not `ctx.<name>`.** For a service a plugin reads opportunistically but deliberately leaves out of `static inject` (e.g. `AgentLoop` reading `sessionPersistence`), the `ctx.<name>` property proxy resolves by an ancestor-only fiber walk that throws when the call arrives through a foreign traceable shadow (the service lives on a sibling fiber). `ctx.get(name)` is the topology-independent global-store lookup, strict by default (an inactive/absent backend reads as `undefined` — prefer it over the `ctx.get(name, false)` overload, which also skips the active-state check). Services that ARE in `static inject` resolve fine via `ctx.<name>`. See [docs/postmortem/0001](../docs/postmortem/0001-acp-default-export-drops-inject.md).
- **Tests**: vitest in `packages/<name>/tests/*.spec.ts`. Every registry needs an HMR-safety test (register a plugin, dispose its fiber, assert cleanup). Err on the side of more tests — edge cases, error paths, event ordering, races. A plugin shipped via `cordis.yml` also needs at least one test that drives it through the REAL Loader/export path (hand-built `ctx.plugin({...})` mounts bypass `unwrapExports` and cannot catch a broken export shape) — see AGENTS.md § Defensive patterns "Line coverage is not behavior coverage". Real-API (with-key) e2e tests are cheap here (we are DeepSeek) and welcome — write many, especially smoke tests; see AGENTS.md § Secrets / .env.
- **Tests**: vitest in `packages/<group>/<pkg>/tests/*.spec.ts`. Every registry needs an HMR-safety test (register a plugin, dispose its fiber, assert cleanup). Err on the side of more tests — edge cases, error paths, event ordering, races. A plugin shipped via `cordis.yml` also needs at least one test that drives it through the REAL Loader/export path (hand-built `ctx.plugin({...})` mounts bypass `unwrapExports` and cannot catch a broken export shape) — see AGENTS.md § Defensive patterns "Line coverage is not behavior coverage". Real-API (with-key) e2e tests are cheap here (we are DeepSeek) and welcome — write many, especially smoke tests; see AGENTS.md § Secrets / .env.
Naming notes:
- A *service* `src/index.ts` exports the service class as `export default` + all public types; a *function/namespace plugin* `src/index.ts` exports `name`/`inject`/`Config`/`apply` as named exports and NO default (see the plugin-export-shape rule above)
- `src/types.ts` contain only types — no runtime code
- Tests live at package level under `tests/`, not `src/__tests__/`
- A package's README and module/JSDoc comments are part of the change: when you alter behavior (config keys, defaults, error codes, wire fields), update them in the same commit. CI runs `pnpm run doc-sync`, which typechecks fenced `ts` blocks in `packages/*/*.md`, regenerates the cordis events/services catalog from the `interface Events` / `interface Context` declarations (failing if the committed copy is stale), and checks markdown wrapping across this file too — but it does NOT catch prose drift (config keys, defaults, error codes), so those stay on the author. A new event needs an `@mode` tag on its JSDoc (the catalog generator hard-errors without it — see the root AGENTS.md).
- A package's README and module/JSDoc comments are part of the change: when you alter behavior (config keys, defaults, error codes, wire fields), update them in the same commit. CI runs `pnpm run doc-sync`, which typechecks fenced `ts` blocks in `packages/*/*.md` and `packages/*/*/*.md`, regenerates the cordis events/services catalog from the `interface Events` / `interface Context` declarations (failing if the committed copy is stale), and checks markdown wrapping across this file too — but it does NOT catch prose drift (config keys, defaults, error codes), so those stay on the author. A new event needs an `@mode` tag on its JSDoc (the catalog generator hard-errors without it — see the root AGENTS.md).
Read the per-package README.md for package-specific details: service API, events, extension points, TODOs.
packages/README.md
+35 -25
View File
@@ -2,13 +2,20 @@
Harness packages, all under the `@deepseek-ai/dsh-*` scope. Each package is a Cordis plugin (microkernel-style): it exports either a default `Service` subclass or a functional plugin that gets registered via `ctx.plugin()`, declares its ctx key/events where applicable through declaration merging, and exposes extension points through `ctx.effect()`, `ctx.on()`, and `ctx.waterfall()`.
<!-- FIXME(package-hierarchy): packages/ is currently FLAT, mixing product
packages (llm, session, agent, agent-loop, …) with example-coupled support
packages (ui-stdio, llm-replay — extracted from examples/ for the coverage
gate). ALL packages should eventually be regrouped into a deliberate
hierarchy, e.g. packages/{core,examples,…}/, so the workspace-glob and
tsconfig-paths churn happens ONCE rather than per extraction. Deferred to a
dedicated restructure PR; do not add new top-level subgroups piecemeal. -->
## Hierarchy
Packages are grouped by modular role at `packages/<group>/<pkg>/`. The group directory is a pure container (no `package.json` of its own); the package name stays `@deepseek-ai/dsh-<pkg>` regardless of group. Each group has a `README.md` describing its role and whether it is product or support infrastructure.
| Group | Role | Release expectation |
|---|---|---|
| [`core/`](core/README.md) | Product API spine: session, system-prompt, tools, agent, and the concrete loop | Product — stable surface |
| [`llm/`](llm/README.md) | LLM capability family: the abstract service + provider adapters | Product — stable surface |
| [`bash/`](bash/README.md) | Bash capability family: the executor seam, a local impl, and the model-facing tool | Product — stable surface |
| [`session-persistence/`](session-persistence/README.md) | Persistence capability family: the seam + JSONL/SQLite backends | Product — stable surface |
| [`ui/`](ui/README.md) | Editor/client integration surfaces (the ACP bridge) | Product — stable surface |
| [`support/`](support/README.md) | Dev/test/example infrastructure (invariants, stdio UI, replay adapter) | Support — lower compatibility expectations |
The split is the point: a package's group says whether it is part of the product API or support/test/example infrastructure, so release and removal decisions do not have to treat every package as an equal public contract. New packages join an existing group; adding a new top-level group is a deliberate act (extend the group READMEs and the hierarchy docs).
## Dependency graph
@@ -34,23 +41,26 @@ The rule: plugins depend on interfaces, never on the concrete loop. `dsh-agent-l
## What goes where
| Package | Role | ctx key |
|---|---|---|
| `llm/` | Abstract LLM service + content-block vocabulary + chunk assembler | `ctx.llm` |
| `session/` | Event-sourced session log + in-memory store | `ctx.sessions` |
| `system-prompt/` | Prompt-section + tool-schema assembly registry | `ctx.systemPrompt` |
| `tools/` | Tool registry + `tools/execute` waterfall | `ctx.tools` |
| `agent/` | Agent interface, registry, `agent/*` event vocabulary | `ctx.agents` |
| `agent-loop/` | THE concrete loop plugin: `ReactLoopAgent` + the loop driver | `ctx.agentLoop` |
| `bash/` | Abstract bash executor seam (interface + vocabulary) | `ctx.bash` |
| `bash-local/` | Local-subprocess `BashExecutor` implementation | (registers `ctx.bash`) |
| `tool-bash/` | Model-facing `bash`/`bash_output`/`bash_kill` tool schemas | (registers on `ctx.tools`) |
| `llm-deepseek/` | DeepSeek API adapter (hand-rolled fetch/SSE) | (registers on `ctx.llm`) |
| `llm-pi-ai/` | DeepSeek adapter via `@earendil-works/pi-ai` (design twin) | (registers on `ctx.llm`) |
| `invariants/` | Dev-mode event-contract invariants + session-log freeze | (listens on `session/*`, `agent/*`) |
| `acp/` | Agent Client Protocol bridge: serves the agent to an ACP editor over JSON-RPC stdio | (drives `ctx.agents`/`ctx.sessions`) |
| `ui-stdio/` | Minimal stdio (readline) UI plugin: renders `agent/*` events, feeds stdin lines to the agent | (drives `ctx.agents`) |
| `llm-replay/` | Record/replay adapter: short-circuits `llm/stream` with chunks from a recorded session JSONL (keyless snapshot tests) | (listens on `llm/stream`) |
| Package | Group | Role | ctx key |
|---|---|---|---|
| `llm/` | `llm` | Abstract LLM service + content-block vocabulary + chunk assembler | `ctx.llm` |
| `session/` | `core` | Event-sourced session log + in-memory store | `ctx.sessions` |
| `system-prompt/` | `core` | Prompt-section + tool-schema assembly registry | `ctx.systemPrompt` |
| `tools/` | `core` | Tool registry + `tools/execute` waterfall | `ctx.tools` |
| `agent/` | `core` | Agent interface, registry, `agent/*` event vocabulary | `ctx.agents` |
| `agent-loop/` | `core` | THE concrete loop plugin: `ReactLoopAgent` + the loop driver | `ctx.agentLoop` |
| `bash/` | `bash` | Abstract bash executor seam (interface + vocabulary) | `ctx.bash` |
| `bash-local/` | `bash` | Local-subprocess `BashExecutor` implementation | (registers `ctx.bash`) |
| `tool-bash/` | `bash` | Model-facing `bash`/`bash_output`/`bash_kill` tool schemas | (registers on `ctx.tools`) |
| `llm-deepseek/` | `llm` | DeepSeek API adapter (hand-rolled fetch/SSE) | (registers on `ctx.llm`) |
| `llm-pi-ai/` | `llm` | DeepSeek adapter via `@earendil-works/pi-ai` (design twin) | (registers on `ctx.llm`) |
| `session-persistence/` | `session-persistence` | Persistence seam + write coordinator | `ctx.sessionPersistence` |
| `session-persistence-jsonl/` | `session-persistence` | JSONL-sidecar persistence backend | (registers `ctx.sessionPersistence`) |
| `session-persistence-sqlite/` | `session-persistence` | SQLite persistence backend | (registers `ctx.sessionPersistence`) |
| `invariants/` | `support` | Dev-mode event-contract invariants + session-log freeze | (listens on `session/*`, `agent/*`) |
| `acp/` | `ui` | Agent Client Protocol bridge: serves the agent to an ACP editor over JSON-RPC stdio | (drives `ctx.agents`/`ctx.sessions`) |
| `ui-stdio/` | `support` | Minimal stdio (readline) UI plugin: renders `agent/*` events, feeds stdin lines to the agent | (drives `ctx.agents`) |
| `llm-replay/` | `support` | Record/replay adapter: short-circuits `llm/stream` with chunks from a recorded session JSONL (keyless snapshot tests) | (listens on `llm/stream`) |
Each package has its own `README.md` with purpose, service API, events, extension points, and deliberate non-goals (TODOs).
@@ -61,4 +71,4 @@ Each package has its own `README.md` with purpose, service API, events, extensio
- **Waterfall semantics**: `ctx.waterfall` listeners receive `(...args, next)` and MUST call `next()` to delegate; returning without it short-circuits (the veto mechanism).
- **Extensible unions**: `ContentBlockMap`, `MessageSourceMap`, `FinishReasonMap`, `TurnTriggerMap`, `TurnEndReasonMap`, and `SessionEventMap` use the merge-extensible-map pattern so plugins can add variants via declaration merging.
- **ESM everywhere**; imports use package names across package boundaries, `.ts` extensions within a package.
- **Tests**: vitest, colocated under `packages/<name>/tests/*.spec.ts`. Every registry needs an HMR-safety test. Err on the side of more tests.
- **Tests**: vitest, colocated under `packages/<group>/<pkg>/tests/*.spec.ts`. Every registry needs an HMR-safety test. Err on the side of more tests.
packages/bash/README.md
+11
View File
@@ -0,0 +1,11 @@
# bash/ — bash capability family
The canonical three-package capability seam (see [capability seams](../../docs/rfc/implemented/architecture/2026-06-13-capability-seams.md)): an abstract executor interface, a concrete local implementation, and the model-facing tool that consumes it. All **product** packages.
| Package | Role | ctx key |
|---|---|---|
| `bash/` | Abstract bash executor seam (interface + vocabulary) | `ctx.bash` |
| `bash-local/` | Local-subprocess `BashExecutor` implementation | (registers `ctx.bash`) |
| `tool-bash/` | Model-facing `bash`/`bash_output`/`bash_kill` tool schemas | (registers on `ctx.tools`) |
The interface lives at `bash/bash/`. A sandboxed executor would replace `bash-local` without touching the interface or the tool — the split is what makes that possible.
packages/core/README.md
+13
View File
@@ -0,0 +1,13 @@
# core/ — product API spine
The packages every harness build is assembled from: the session log, the system-prompt assembly, the tool registry, the agent vocabulary, and the one concrete loop that drives them. These are **product** packages — the stable surface plugins and consumers build against.
| Package | Role | ctx key |
|---|---|---|
| `session/` | Event-sourced session log + in-memory store | `ctx.sessions` |
| `system-prompt/` | Prompt-section + tool-schema assembly registry | `ctx.systemPrompt` |
| `tools/` | Tool registry + `tools/execute` waterfall | `ctx.tools` |
| `agent/` | Agent interface, registry, `agent/*` event vocabulary | `ctx.agents` |
| `agent-loop/` | The concrete loop plugin: `ReactLoopAgent` + the loop driver | `ctx.agentLoop` |
`agent-loop` is the one concrete implementation of the `agent` seam and lives here because it is the harness's default product loop; everything else in `core/` is interface/vocabulary. Plugins depend on the `agent` vocabulary, never on `agent-loop` directly, so the loop stays swappable.
packages/llm/README.md
+11
View File
@@ -0,0 +1,11 @@
# llm/ — LLM capability family
The LLM seam and its provider adapters. The interface package (`llm`) owns the abstract service, the content-block vocabulary, and the stream-chunk assembler; the adapters are concrete implementations that register on `ctx.llm`. All **product** packages.
| Package | Role | ctx key |
|---|---|---|
| `llm/` | Abstract LLM service + content-block vocabulary + chunk assembler | `ctx.llm` |
| `llm-deepseek/` | DeepSeek API adapter (hand-rolled fetch/SSE) | (registers on `ctx.llm`) |
| `llm-pi-ai/` | DeepSeek adapter via `@earendil-works/pi-ai` (design twin) | (registers on `ctx.llm`) |
The interface lives at `llm/llm/`; adapters are flat siblings under the group. A new provider adapter joins here and registers on `ctx.llm` without touching the interface. See [twin LLM adapters](../../docs/rfc/implemented/architecture/2026-06-13-twin-llm-adapters.md) for why two adapters exist.
packages/session-persistence/README.md
+11
View File
@@ -0,0 +1,11 @@
# session-persistence/ — persistence capability family
The durable session-persistence seam and its storage backends. The interface package owns the abstract `SessionPersistence` service and the shared write coordinator; the backends are concrete implementations that register on `ctx.sessionPersistence`. All **product** packages.
| Package | Role | ctx key |
|---|---|---|
| `session-persistence/` | Persistence seam + shared write coordinator | `ctx.sessionPersistence` |
| `session-persistence-jsonl/` | JSONL-sidecar persistence backend | (registers `ctx.sessionPersistence`) |
| `session-persistence-sqlite/` | SQLite persistence backend | (registers `ctx.sessionPersistence`) |
The interface lives at `session-persistence/session-persistence/`; backends are flat siblings. A new storage backend joins here and registers on `ctx.sessionPersistence`. See [session persistence](../../docs/rfc/implemented/architecture/2026-06-14-session-persistence.md).
packages/support/README.md
+11
View File
@@ -0,0 +1,11 @@
# support/ — dev/test/example infrastructure
Packages that exist to serve development, testing, and the examples rather than to ship as product API. They are real workspace packages (typed, tested, under the coverage gate), but they carry **lower compatibility expectations**: they may change or be removed when the development need behind them does, without the deprecation care a product package would warrant.
| Package | Role | ctx key |
|---|---|---|
| `invariants/` | Dev-mode event-contract invariants + session-log freeze | (listens on `session/*`, `agent/*`) |
| `ui-stdio/` | Minimal stdio (readline) UI plugin: renders `agent/*` events, feeds stdin lines to the agent | (drives `ctx.agents`) |
| `llm-replay/` | Record/replay adapter: short-circuits `llm/stream` from a recorded session JSONL (keyless snapshot tests) | (listens on `llm/stream`) |
`invariants` runs only in dev mode (contract checks, not runtime behavior). `ui-stdio` and `llm-replay` were extracted from the examples for reuse and to bring them under the per-file coverage gate; they back the demos and the snapshot test tier. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
packages/ui/README.md
+9
View File
@@ -0,0 +1,9 @@
# ui/ — editor/client integration surfaces
Integrations that expose the agent to an external editor or client. These are **product** packages: a real surface a user drives the harness through.
| Package | Role | ctx key |
|---|---|---|
| `acp/` | Agent Client Protocol bridge: serves the agent to an ACP editor (Zed) over JSON-RPC stdio | (drives `ctx.agents`/`ctx.sessions`) |
A UI integration is a client-driver plugin, not a loop change and not a capability seam: it consumes the existing `agent/*` event taxonomy and the `dsh-agent` factory. The readline `ui-stdio` plugin is the unstructured analogue but lives in `support/` because it exists chiefly for the examples and the coverage gate — `ui/` is reserved for surfaces shipped as product.