deepseek-harness/docs/rfc/009-session-persistence-and-resumability.md

# RFC 009: Durable session persistence — an abstract, append-only, event-based store

Status: proposed

## Problem

Sessions live only in memory. The example `session-jsonl.ts` plugin (duplicated byte-for-byte in both `examples/coding-agent` and `examples/echo-agent`) is write-only telemetry: it buffers `session/event` and appends JSON lines, but has no read/replay path, no crash-safety (no fsync, no atomic write, and a fire-and-forget dispose drain), no listing, and no format versioning. [ADR 0003](../adr/0003-event-sourced-sessions.md) and [docs/architecture.md](../architecture.md) both park "real persistence backends (JSONL session dirs, sqlite)" and the session-event-vocabulary review as deferred TODOs "once the loop and the first persistence plugin coexist" — that time is now.

Because nothing can rehydrate a past session from disk into a live agent, durable resume ("continue yesterday's task"), durable forking, and the ACP `session/load` method (RFC 010) are all impossible. (In-memory replay/fork via `ctx.sessions.create(id, seed)` already exists and is tested; what is missing is the durable store behind it and a first-class agent-loop resume path.)

The event-sourced model (ADR 0003) makes the log the single source of truth and derives LLM history from it. Persistence must stay faithful to that: it should persist the existing `SessionEvent` directly — there must be no parallel "persisted message" type that the log has to be converted to and from. We also want the backend to be swappable: a file store now, a database store later, behind one interface.

## Proposal

Mirror the codebase's capability-seam pattern ([ADR 0009](../adr/0009-capability-seams.md), the `bash` template: an abstract `Service` interface, a concrete implementation, and consumers) for persistence.

**1. Abstract service `SessionPersistence`** — a new interface package `@deepseek-ai/dsh-session-persistence` owning `ctx.sessionPersistence`, depending only on `cordis` and `dsh-session`. The `SessionHeader`/`SessionSummary`/`SessionMeta` types are owned by **`dsh-session`** (they live beside `SessionId` because `Session.header` is typed by them — see item 3a); the persistence package imports/re-exports them. Owning them in the persistence package would force `dsh-session` to depend back on it to type `Session.header`, a package cycle. Its persisted unit IS `SessionEvent` (`{ type, seq, time, data }`), reused verbatim — no conversion type. The method surface:

- `create(meta: SessionMeta): Promise<void>` — register a new session's header. The backend MAY defer the physical write until the first `append` (lazy materialization); `has`/`list` semantics for a zero-event session are specified, not left implicit.
- `append(id, events: readonly SessionEvent[]): Promise<void>` — durably persist a batch (called from the flush drain). Committed events (at or below a flushed `turn/end`) are append-only and never rewritten; the only exception is the one-time truncation-repair of a never-committed crash tail on the first `append` after a `load` (see `load`). **Contract**: the first event's `seq` MUST equal the backend's stored next-seq after any such repair (a DB impl asserts this inside a transaction; the file impl appends at EOF). All persisted `event.data` MUST be JSON-serializable.
- `load(id): Promise<{ meta: SessionMeta; events: SessionEvent[] }>` — replay header plus the event log up to the last durable checkpoint. Returns `meta` AND `events` so the live session is reconstructed with its `cwd`/lineage, not just its log. **Validation/repair**: the returned events MUST be contiguous (`events[i].seq === i`); a parse error or `seq` gap in the *middle* of the log makes the session unloadable (reject). The loop only flushes at `turn/end`, so a crash can leave a half-written final turn *below* the last committed checkpoint — `load` returns events only up to the **last complete `turn/end`**, and a subsequent `append` runs the **truncation-repair** step (see impl) that physically discards the orphaned tail before writing. This keeps the append-only contract honest: only the never-committed crash tail is ever removed; events at or below a flushed `turn/end` are never rewritten.
- `list(): Promise<SessionMeta[]>` — lightweight listing from headers, no full-log parse.
- `has(id)` / `delete(id)` — existence and removal.
- `update(id, summary: Partial<SessionSummary>): Promise<void>` — update mutable header fields without touching the append-only event log.

The new `SessionMeta` splits into an immutable `SessionHeader` (`{ id, version, createdAt, cwd?, parentSession? }`) and a mutable `SessionSummary` (`{ updatedAt, title?, firstPrompt? }`); `SessionMeta = SessionHeader & SessionSummary`. Every reference system writes such a header (pi's `version: 3` header line, Codex's `SessionMeta`, Claude Code's tail metadata). It is kept *separate from the event log* deliberately: format-version, cwd, and lineage are storage concerns, not conversation events, so they stay out of `SessionEventMap` and never reach `deriveMessages()`. The alternative — a merge-extensible `session/meta` event as log line 0 — was considered: an in-log event would ride along with a seeded/forked session for free, whereas an out-of-log header must be threaded through a seam (item 3a). It was rejected because metadata is not replayable conversation state; the explicit metadata seam is the cleaner cost.

**2. Concrete impl `SessionPersistenceJsonl`** — a new package `@deepseek-ai/dsh-session-persistence-jsonl`. Per session: an append-only `.jsonl` event log (a `SessionHeader` line — `{ type: 'session', version, id, cwd, createdAt, parentSession? }` — followed by one `SessionEvent` JSON per line), plus a small sidecar `.<id>.summary.json` holding the mutable `SessionSummary` (`updatedAt`, `title?`, `firstPrompt?`). The split keeps committed events untouched: `update(id, summary)` rewrites only the tiny sidecar (atomic temp-write + rename), never the log; `load`/`list` read the header line from the log and merge the sidecar to return a full `SessionMeta` (sidecar absent → summary fields default). On disk: a configured root with per-cwd subdirectories (pi-style `--encoded-cwd--/<timestamp>_<id>.jsonl`) so sessions group by project. `list()` reads only each file's header line plus its sidecar. Resilience over the example: append plus explicit flush; **truncation-repair on the first append after a crash** — `load` computes the byte offset of the last complete `turn/end`, and the impl truncates the file to that offset (`ftruncate`, then `fsync`) before its first append, atomically discarding the never-committed tail. Only the uncommitted crash tail is ever removed. Lazy materialization (no file until the first real event, so abandoned sessions leave nothing behind).

**2a. `assistant/chunk` persistence policy** (decided here, not deferred). The loop appends one `assistant/chunk` per raw stream chunk, but `deriveMessages()` skips chunks entirely — the assembled `assistant/message` is authoritative for history. It is tempting to drop chunks from the durable log (Codex's `policy.rs` filters deltas from its rollout). But `seq = log.length` and the load-validation `events[i].seq === i` require a *contiguous* log: filtering chunks out would leave holes (`[0,1,4,6,8]`) and break both the contract and resume. **Decision: the canonical durable log persists every `SessionEvent` verbatim, including `assistant/chunk`** — this keeps `seq` contiguous, keeps "persist `SessionEvent` directly" literally true, and lets RFC 010 replay streamed turns on `session/load`. A chunk-filtered *projection* (for export or a compacted listing) is possible later as a derived view with its own renumbering, but it is NOT the canonical log and NOT the default. The round-trip test asserts byte-identical events.

**3. Write path** lives in the impl plugin, generalizing the example: subscribe to `session/event` (buffer write-behind, keyed by session), drain to `append()` at the awaited `session/flush` checkpoint and on dispose — the seam the loop already fires at every turn end. The loop's write path needs no change.

**3a. Metadata seam** (the one `dsh-session` change). Today `Session` has only `id` plus the log, and `session/event` carries `(session, event)` — there is nowhere for `cwd`/lineage to live, so a plugin listening to events alone cannot know a session's `cwd`. Add a minimal seam: `SessionStore.create(id, { seed?, meta? })` attaches a `SessionHeader` to the `Session` (a new readonly `session.header`), and the persistence plugin captures it on `session/created`. This is additive; `deriveMessages()` and the log are untouched.

**4. Resume path** — an async helper, NOT a change to the synchronous `create`. `AgentLoop.create(agentId, options)` is synchronous (the `AgentLoop` constructor calls it for configured agents), so it cannot `await` persistence. Add a separate `async resume(agentId, resumeSessionId, options?): Promise<LoopAgent>` that awaits `ctx.sessionPersistence.load(resumeSessionId)`, then calls `ctx.sessions.create(resumeSessionId, { seed: events, meta })`, then constructs/registers/starts the `LoopAgent` on that session. Three distinct identities are kept separate: the `agentId` (the handle), the live `sessionId` (here the resumed one, NOT `${agentId}-session`), and the `resumeSessionId` being loaded. Downstream already works: `Session`'s constructor shallow-copies the seed, `lastTurnNumber()` in `loop.ts` continues turn numbering, and `deriveMessages()` rebuilds history.

Seed handling has two cases that the plugin must distinguish, and neither is the naive "re-append on flush" hazard. Seed events are copied into `Session` by the constructor *without* emitting `session/event` (the store installs `onAppend` only after construction), so the write-behind buffer never sees them — there is no double-write on a plain resume. (1) **Resume / adopt** an existing on-disk session: the events are already persisted, so the plugin initializes its per-session write cursor to the loaded length and appends only events with `seq >= loadedLength`. (2) **Fork** a brand-new session whose seed came from another session: that seed is NOT yet on disk under the new id, so the plugin must persist the full seed once (on `session/created`, via `create(meta)` + an initial `append`) and then set the cursor to the seed length. The `append` seq-contract makes both safe — a re-append of a stored seq is rejected, never silently duplicated.

**5. DB-backend feasibility** (proven by the design, implemented later). `SessionEvent` maps 1:1 onto a row `(session_id TEXT, seq INTEGER, type TEXT, time INTEGER, data JSON, PRIMARY KEY(session_id, seq))` — `seq` already exists and is monotonic. `append` is INSERT (in a transaction asserting the contiguous-seq contract), `load` is SELECT … ORDER BY seq, `list` is SELECT from a `sessions` header table. A future `@deepseek-ai/dsh-session-persistence-sqlite` is a drop-in `SessionPersistence` subclass with no interface change (opencode runs exactly this `session_message(session_id, seq, type, data)` shape on SQLite/WAL). Because `SessionEventMap` is merge-extensible and `data` is typed only as `SessionEventMap[K]`, the interface requires all persisted `event.data` to be JSON-serializable; `append` rejects non-serializable data with an error naming the offending event type, and the plugin snapshots (serializes/clones) each event when it buffers on `session/event`, since `session.events` hands out the live mutable object. A canonical SQLite backend is one such drop-in; a Codex-style derived search/listing index over the JSONL files would instead be a separate projector service, NOT a `SessionPersistence` replacement. `SessionId` is an unvalidated branded string, so the file impl MUST sanitize/encode it before using it in a path (no traversal, no collision).

## Plan

1. Interface package `packages/session-persistence/` per [the cookbook](../cookbook/adding-a-package.md): abstract `SessionPersistence extends Service` (`super(ctx, 'sessionPersistence')`), the `declare module 'cordis'` ctx key, the `SessionHeader`/`SessionSummary`/`SessionMeta` types, and method contracts documented in JSDoc (durability, append-only, contiguous-seq, JSON-serializable, error semantics).
2. `dsh-session` changes: add the three meta types beside `SessionId`; add the metadata seam. `SessionStore.create(id?, seed?)` becomes `create(id?, options?: { seed?; meta? })` — a breaking signature change (callers pass `seed` positionally today: `AgentLoop.create`, and ~20+ call sites across `session`/`invariants`/`agent-loop` tests), so either migrate every caller or keep a deprecated overload during transition. Add a readonly `session.header`. Persistence captures the header on `session/created` (a synchronous event), so the impl must hold a per-session init promise that every `session/flush` awaits before `append`, and must seed existing live sessions via `ctx.sessions.list()` on plugin apply (HMR does not replay `session/created`, mirroring `dsh-invariants`). Do NOT add meta to `SessionEventMap`.
3. JSONL impl `packages/session-persistence-jsonl/`: append-only event log (header line + all events verbatim — see 2a) plus an atomic `.summary.json` sidecar for mutable fields, sanitized per-cwd dirs and filenames, lazy materialize (header + first batch written atomically), append plus flush, a `load` that returns events up to the last complete `turn/end` and computes its byte offset; the first post-load `append` runs truncation-repair (`ftruncate` to that offset + `fsync`, discarding only the uncommitted crash tail) before writing (rejects mid-log gaps), `list` from header + sidecar, the per-session write cursor, and snapshot-on-buffer. `static Config` for root dir and flush policy.
4. Generalize the write-path plugin: the impl subscribes to `session/created` (capture header, persist any seed for forks), `session/event` (snapshot + buffer), and `session/flush`/dispose (drain), replacing the per-example `session-jsonl.ts`; both examples load the shared plugin.
5. Resume seam: the async `AgentLoop.resume(agentId, resumeSessionId, options?)`; initialize the write cursor to the loaded length; verify `lastTurnNumber`/`deriveMessages` continuity. `AgentLoop` does NOT hard-inject `sessionPersistence` (that would break non-persistent examples) — `resume` checks for the service and throws a typed "persistence not configured" error; consumers that need resume (ACP) load the persistence plugin.
6. Tests (event-sourcing makes these strong): a round-trip property (persist an arbitrary log → reload → byte-identical events and identical `deriveMessages()` output — the replay equivalence ADR 0003 promises); resume vs fork (resume appends no duplicate seqs; a fork persists its seed once); contiguous-seq enforcement (mid-log gap rejected, re-append of a stored seq rejected); crash tolerance (a truncated final turn truncates back to the last `turn/end`); JSON-serializability rejection for a plugin-added event carrying non-serializable data; mutation-after-`session/event` does not corrupt the persisted snapshot; SessionId path-traversal is neutralized; lazy materialization (no file until the first event); `has`/`list` semantics for a zero-event session; HMR-safety (dispose drains buffers and closes file handles; apply seeds existing live sessions); concurrent sessions do not cross buffers.
7. Docs: update the "Event-sourced sessions" durability-seam paragraph and the "Deferred work" list in [docs/architecture.md](../architecture.md) (persistence is no longer deferred); sync the affected package READMEs/JSDoc (`dsh-session` for the `create`/`session.header` change); add a [cookbook](../cookbook/) note on writing a persistence backend; resolve the `TODO(review)` on the event vocabulary now that a real persistence plugin coexists with the loop. On implementation this likely graduates to an ADR — "persistence is an abstract service over the existing `SessionEvent`; verbatim append-only log (committed events never rewritten; only an uncommitted crash tail is truncation-repaired); file canonical, DB drop-in" is durable, contested, and surprising enough to record.

## Risks

Format versioning: the header carries a `version`; `load` must reject or migrate unknown versions (pi rejects, Codex relies on serde forward-compat). Fix the policy before shipping so v1 files stay loadable.

Crash-safety bounds: append-only plus flush is robust to partial trailing writes (tolerated on load) but not to fsync-less power loss mid-line. State the guarantee honestly; a DB/WAL backend is the stronger option later.

`SessionMeta` placement is a small public-surface decision (the immutability concern from [ADR 0012](../adr/0012-dev-invariants-over-deep-readonly.md)); pick its owning package deliberately and freeze the shape.

Event-vocabulary churn: persisting the log freezes its shape more firmly, so this is the moment to complete ADR 0003's `TODO(review)` — especially the `assistant/chunk` fidelity question — before committing to an on-disk format.