fix: reserve cold loads across repair

This commit is contained in:
_Kerman
2026-07-23 20:58:20 +08:00
parent 816907ba2d
commit 4e5166828b
10 files changed
+79 -15

No files matched your search

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
2026-07-23-collapse-persistence-flush-state.md: 3a7bc4d832d9c139614a8ecd85f7b29a6b5afe51
2026-07-23-collapse-persistence-flush-state.zh.md: b99906f1051d2dc1369a92072d0e8b98402bba9f
2026-07-23-collapse-persistence-flush-state.md: 21e99b0fc37f97e441f6d92eb02636767cabf51e
2026-07-23-collapse-persistence-flush-state.zh.md: 0b11c45ed7d5291daed4726b9dd7fb8878d4a783
@@ -16,7 +16,7 @@ Each live `Session` has one controller containing `pending`, `init`, and the opt
Initialization now enters the existing per-id operation chain once and calls the unserialized core operations while it owns that turn. The chain remains separate from the live controller because detached public `create`/`append`/`load` calls can race without a `Session` object and still require identity-level serialization.
Crash repair is cold-only. For a live identity, `load(id)` snapshots the authoritative in-memory header and events before awaiting their flush; it returns that durable snapshot when balanced and rejects an open turn without reading or repairing storage. A cold identity follows the stored-prefix repair path. HMR adoption remains separate through `loadStored` plus the coordinator's cwd check and truncates torn storage without closing the authoritative live turn.
Crash repair is cold-only. For a live identity, `load(id)` snapshots the authoritative in-memory header and events before awaiting their flush; it returns that durable snapshot when balanced and rejects an open turn without reading or repairing storage. A cold load reserves its identity synchronously inside the per-id chain before awaiting stored-prefix reads or repair writes; the `session/created` publication boundary rejects and rolls back a same-id live session until the reservation clears. HMR adoption remains separate through `loadStored` plus the coordinator's cwd check and truncates torn storage without closing the authoritative live turn.
The live-controller map is also the retirement registry. Successful retirement drains and removes its controller; failed retirement leaves it in the map. Backend teardown stops event admission, flushes every controller still present, awaits remaining per-id operations, and closes the backend. No separate retirement set is needed to rediscover unfinished work.
@@ -37,6 +37,7 @@ The live-controller map is also the retirement registry. Successful retirement d
- Failure and teardown tests keep rejected batches pending, retry them before close, and prove an in-flight controller delays backend close.
- The shared backend contract persists an open live turn, proves `load` rejects without writing synthetic closers, completes and retires the owner, then reloads the exact completed turn.
- An AgentLoop regression races `resume()` against a live open turn and proves the original agent can still durably complete it without an injected `interrupted` boundary.
- A controlled backend blocks `loadStored`, attempts same-id session publication while repair owns the reservation, and proves rollback leaves no ghost controller before a balanced resume succeeds.
## Consequences
@@ -16,7 +16,7 @@ Status: implemented
初始化只进入现有的按 id 操作链一次,并在占有该轮执行权时调用未串行化的核心操作。该操作链与活跃控制器保持分离,因为公共 `create``append``load` 调用即使没有 `Session` 对象仍可能发生竞态,依然需要按标识串行执行。
崩溃修复仅适用于冷态标识。对于活跃标识,`load(id)` 会在等待刷新完成前,先对内存中的权威标头和事件生成快照;若快照闭合,则返回这个已持久化的快照;若轮次仍打开,则在不读取或修复存储的情况下拒绝该次加载。冷态标识沿用已存储前缀的修复路径。HMR 接管仍由 `loadStored` 与协调器的 cwd 检查独立处理,会截断撕裂的存储,但不会闭合权威的活跃轮次。
崩溃修复仅适用于冷态标识。对于活跃标识,`load(id)` 会在等待刷新完成前,先对内存中的权威标头和事件生成快照;若快照闭合,则返回这个已持久化的快照;若轮次仍打开,则在不读取或修复存储的情况下拒绝该次加载。冷态加载会先在按 id 操作链内同步占用对应标识,再等待读取已存储前缀或执行修复写入;在这项占用解除前,`session/created` 发布边界会拒绝同 id 活跃会话的发布并将其回滚。HMR 接管仍由 `loadStored` 与协调器的 cwd 检查独立处理,会截断撕裂的存储,但不会闭合权威的活跃轮次。
活跃控制器映射同时也是退役注册表。退役成功时,系统排空并移除其控制器;退役失败时,控制器保留在映射中。后端资源销毁会停止接纳事件,刷新所有仍存在的控制器,等待其余按 id 操作完成,然后关闭后端。无需另设退役集合来重新发现未完成的工作。
@@ -37,6 +37,7 @@ Status: implemented
- 失败和资源销毁测试会让写入失败的批次保持待处理,在关闭前重试这些批次,并证明尚在执行的控制器会延迟后端关闭。
- 共享后端契约会持久化一个仍打开的活跃轮次,证明 `load` 会拒绝且不会写入合成闭合事件,随后完成该轮次并让其所有者退役,最后重新加载完全相同的已完成轮次。
- AgentLoop 回归测试让 `resume()` 与一个仍打开的活跃轮次发生竞态,并证明原有的 agent(智能体)仍能完成该轮次并将其持久化,其间不会注入 `interrupted` 边界。
- 一个受控后端会阻塞 `loadStored`,在修复操作持有标识占用期间尝试发布同 id 会话,并证明回滚不会留下残留控制器,之后可以成功恢复一个闭合会话。
## 后果
+2
View File
@@ -920,6 +920,8 @@ abstract append(id: SessionId, events: readonly SessionEvent[]): Promise<void>
* versions and corruption in the committed prefix reject. Implementations
* MUST NOT crash-repair an identity still bound to a live Session: a balanced
* live log may return as a durable snapshot, while an open live turn rejects.
* A coordinator-backed cold load reserves the identity across storage awaits,
* so concurrent publication of a same-id live Session rejects.
* @param id - the persisted session to reload.
* @returns the header and a log ending on a balanced `turn/end`.
*/
+1 -1
View File
@@ -12,7 +12,7 @@ The seam is a textbook [capability seam](../../.agents/notes/implemented/archite
A backend that reloads a log crashed mid-turn finds an open `turn/start` with no `turn/end`. It does **not** truncate — a single turn can be huge in a long-horizon task (many steps, large tool output), and those events were durably appended before the crash. Instead it closes the orphaned turn with a synthetic `turn/end { reason: { kind: 'interrupted' } }`, keeping the log balanced and the turn-enclosure invariant intact. `interrupted` is the one `TurnEndReason` no loop emits (see [session.md](session.md#why-a-turn-ended-turnendreasonmap)).
Repair applies only to cold sessions. For a live id, `SessionPersistence.load(id)` snapshots the in-memory log, waits until that snapshot is durable, and returns it only when balanced; an open live turn rejects rather than receiving synthetic interruption boundaries. HMR also adopts a live prefix without closing its active turn.
Repair applies only to cold sessions. For a live id, `SessionPersistence.load(id)` snapshots the in-memory log, waits until that snapshot is durable, and returns it only when balanced; an open live turn rejects rather than receiving synthetic interruption boundaries. A coordinator-backed cold load reserves the id across backend reads and repair writes, so concurrent publication of a same-id live session rejects and rolls back. HMR also adopts a live prefix without closing its active turn.
## `SessionLocation` — optional per-session artifact target
@@ -462,7 +462,7 @@ export const SERVICE_API: readonly ServiceApiEntry[] = [
},
{
signature: 'abstract load(id: SessionId): Promise<{ meta: SessionHeader; events: SessionEvent[] }>',
jsDoc: '/**\n * Load a header and balanced contiguous log. A complete interrupted final\n * turn is preserved and durably closed with missing tool errors plus any open\n * step and turn boundaries; only a torn final record is discarded. Unknown\n * versions and corruption in the committed prefix reject. Implementations\n * MUST NOT crash-repair an identity still bound to a live Session: a balanced\n * live log may return as a durable snapshot, while an open live turn rejects.\n * @param id - the persisted session to reload.\n * @returns the header and a log ending on a balanced `turn/end`.\n */',
jsDoc: '/**\n * Load a header and balanced contiguous log. A complete interrupted final\n * turn is preserved and durably closed with missing tool errors plus any open\n * step and turn boundaries; only a torn final record is discarded. Unknown\n * versions and corruption in the committed prefix reject. Implementations\n * MUST NOT crash-repair an identity still bound to a live Session: a balanced\n * live log may return as a durable snapshot, while an open live turn rejects.\n * A coordinator-backed cold load reserves the identity across storage awaits,\n * so concurrent publication of a same-id live Session rejects.\n * @param id - the persisted session to reload.\n * @returns the header and a log ending on a balanced `turn/end`.\n */',
},
{
signature: 'abstract list(): Promise<SessionHeader[]>',
@@ -27,7 +27,7 @@ The persisted unit IS the existing `SessionEvent` (event-sourced model — the l
Each `session/event` copies its event into the session controller and starts an eager drain without blocking the producer. Concurrent notifications share the current drain; events admitted during a write remain pending and trigger the next batch. `session/flush` is an observation barrier that waits until the controller has no current or pending batch. An eager failure is logged and retains the batch; the next explicit flush or backend teardown retries it and surfaces failure to its caller.
Crash repair is cold-only. For a live id, `load(id)` snapshots the authoritative in-memory log, waits for that snapshot to become durable, and returns it only when balanced; an open live turn rejects instead of receiving synthetic interruption closers. A cold id follows storage repair normally. HMR adoption likewise reads through `loadStored`, applies the coordinator's cwd check, and never closes the active turn.
Crash repair is cold-only. For a live id, `load(id)` snapshots the authoritative in-memory log, waits for that snapshot to become durable, and returns it only when balanced; an open live turn rejects instead of receiving synthetic interruption closers. A cold load reserves its id across backend reads and repair writes, so concurrent publication of a same-id live `Session` rejects and rolls back. HMR adoption reads through `loadStored`, applies the coordinator's cwd check, and never closes the active turn.
When a live session emits `session/disposed`, the coordinator waits for its controller, serializes a final drain, then releases state owned by that exact `Session` object. Failed retirement leaves the controller in the live-session map, so backend teardown can retry it. Backend teardown stops event admission first, flushes every remaining controller, awaits per-id operations, and only then closes the storage handle.
@@ -154,6 +154,8 @@ export class PersistenceCoordinator<TornMarker = unknown> {
private states = new Map<SessionId, SessionState>()
/** Lifecycle and write-behind state keyed by the exact live Session. */
private live = new Map<Session, LiveSessionState>()
/** Cold loads currently reserving an id across backend reads and repair writes. */
private coldLoads = new Set<SessionId>()
/**
* Per-session serialization: every operation chains onto the prior one for the
* same id, so writes for one session never interleave. Keyed by session id.
@@ -251,7 +253,12 @@ export class PersistenceCoordinator<TornMarker = unknown> {
const selected = await this.serialize(id, async () => {
const live = this.ctx.sessions.get(id)
if (live !== undefined) return { live }
return { loaded: await this.loadCore(id) }
this.coldLoads.add(id)
try {
return { loaded: await this.loadCore(id) }
} finally {
this.coldLoads.delete(id)
}
})
return 'loaded' in selected ? selected.loaded : this.loadLiveSnapshot(selected.live)
}
@@ -372,7 +379,12 @@ export class PersistenceCoordinator<TornMarker = unknown> {
}, `${this.backend.name} write path`)
// Capture the header on creation and persist a fork's seed once.
ctx.on('session/created', (session) => { void this.initFor(session) })
ctx.on('session/created', (session) => {
if (this.coldLoads.has(session.id)) {
throw new Error(`cannot publish session "${session.id}" while its persisted history is loading`)
}
void this.initFor(session)
})
// Keep a persistence-owned copy of each frozen event and start an eager drain.
ctx.on('session/event', (session, event) => {
@@ -394,6 +406,7 @@ export class PersistenceCoordinator<TornMarker = unknown> {
/** Start and observe one disposed session's final drain. */
private retire(session: Session): void {
if (!this.live.has(session)) return
void this.retireCore(session).catch((error: unknown) => {
this.ctx.logger.warn(`${this.backend.name}: session "${session.id}" retirement failed: ${String(error)}`)
})
@@ -79,6 +79,8 @@ export abstract class SessionPersistence extends Service {
* versions and corruption in the committed prefix reject. Implementations
* MUST NOT crash-repair an identity still bound to a live Session: a balanced
* live log may return as a durable snapshot, while an open live turn rejects.
* A coordinator-backed cold load reserves the identity across storage awaits,
* so concurrent publication of a same-id live Session rejects.
* @param id - the persisted session to reload.
* @returns the header and a log ending on a balanced `turn/end`.
*/
@@ -293,6 +293,48 @@ describe('PersistenceCoordinator stored identity', () => {
await ctx.fiber.dispose()
}
})
it('reserves a cold id across asynchronous storage repair', async () => {
const ctx = new Context()
await ctx.plugin(SessionStore)
const backend = new ControlledBackend()
const id = SessionId('cold-load-reservation')
const header = meta(id)
const start: SessionEvent = {
type: 'turn/start',
seq: 0,
time: 1,
data: { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } },
}
backend.store.set(id, { meta: header, events: [start] })
const loadGate = Promise.withResolvers<boolean>()
backend.beforeLoadStored = async () => { await loadGate.promise }
let coordinator!: PersistenceCoordinator<never>
const fiber = await ctx.plugin(Object.assign((inner: Context) => {
coordinator = new PersistenceCoordinator(inner, backend)
}, { inject: ['sessions'] }))
try {
const loading = coordinator.load(id)
await vi.waitFor(() => { expect(backend.loadAttempts).toBe(1) })
await expect(ctx.plugin(Object.assign((inner: Context) => {
inner.sessions.create(id, { seed: [start], meta: header })
}, { inject: ['sessions'] }))).rejects.toThrow(/persisted history is loading/)
expect(ctx.sessions.get(id)).toBeUndefined()
loadGate.resolve(true)
const loaded = await loading
expect(loaded.events.map(event => event.type)).toEqual(['turn/start', 'turn/end'])
const resumed = ctx.sessions.create(id, { seed: loaded.events, meta: loaded.meta })
await expect(ctx.sessions.flush(resumed)).resolves.toBeUndefined()
} finally {
loadGate.resolve(true)
await fiber.dispose()
await ctx.fiber.dispose()
}
})
})
describe('PersistenceCoordinator retirement', () => {
@@ -399,17 +441,20 @@ describe('PersistenceCoordinator retirement', () => {
appendGate.resolve(true)
await vi.waitFor(() => { expect(backend.loadAttempts).toBe(baselineLoads + 1) })
let reuse!: Session
await ctx.plugin(Object.assign((inner: Context) => {
reuse = inner.sessions.create(id)
}, { inject: ['sessions'] }))
const reuseFlush = ctx.sessions.flush(reuse)
await expect(ctx.plugin(Object.assign((inner: Context) => {
inner.sessions.create(id)
}, { inject: ['sessions'] }))).rejects.toThrow(/persisted history is loading/)
loadGate.resolve(true)
await expect(coldLoad).resolves.toMatchObject({
events: [{ seq: 0 }, { seq: 1 }],
})
await expect(reuseFlush).rejects.toThrow(/id collision/)
let reuse!: Session
await ctx.plugin(Object.assign((inner: Context) => {
reuse = inner.sessions.create(id)
}, { inject: ['sessions'] }))
await expect(ctx.sessions.flush(reuse)).rejects.toThrow(/id collision/)
await vi.waitFor(() => {
expect(backend.store.get(id)?.events.map(event => event.seq)).toEqual([0, 1])
})