diff --git a/.agents/notes/implemented/feature/2026-08-05-durable-web-schedule.md b/.agents/notes/implemented/feature/2026-08-05-durable-web-schedule.md index b5e5703c2e..e5afff7d8e 100644 --- a/.agents/notes/implemented/feature/2026-08-05-durable-web-schedule.md +++ b/.agents/notes/implemented/feature/2026-08-05-durable-web-schedule.md @@ -19,7 +19,7 @@ The user-visible boundary is `session-local`: the original Session runs an on-ti | Scenario | Durable fact | Live behavior | User-visible result | | --- | --- | --- | --- | | Create and manage | `schedule/change` create/delete events in the original Session | Agent-scoped tools checkpoint before reading and after mutations | Stable id, UTC target, `scheduled`/`overdue`, and `session-local` disclosure | -| Due while busy | Active create remains in the fold | Owner waits for `whenIdle()`, reserves admission, queues one followup, then appends dispatch | One replayable reminder receipt; model failure does not retract it | +| Due while busy | Active create remains in the fold | Owner waits for `whenIdle()`, claims idle maintenance, queues one followup, then appends dispatch | One replayable reminder receipt; model failure does not retract it | | Process stopped or Session cold | Active create remains in persistence | No timer or background scan exists; resume rebuilds the owner | Future target waits again; overdue target is attempted once | | Fork | Parent events remain in the inherited prefix | Child fold starts at `seedLength` | Parent receipt may appear in history, but no parent reminder becomes active child work | @@ -41,21 +41,21 @@ The persistence coordinator supplies that acknowledgement only after its write p ### Live delivery lifecycle -The Agent-scoped owner derives its earliest target from the durable fold. Long targets use bounded timer segments, and every wake reads the wall clock again, so a rollback cannot fire early and a forward jump becomes overdue. An unavailable `reserveTurnAdmission()` leaves the record active and installs one `whenIdle()` wait before retrying. +The Agent-scoped owner derives its earliest target from the durable fold. Long targets use bounded timer segments, and every wake reads the wall clock again, so a rollback cannot fire early and a forward jump becomes overdue. If a turn or another maintenance task already owns the Agent, `runMaintenance()` rejects the claim; the record stays active and one `whenIdle()` wait triggers a later retry. A rejected persistence preflight also leaves the record active, but no private retry timer runs; later Agent activity reaching idle or a successful Schedule management preflight asks the owner to try again. -The accepted path first clears pending persistence, reserves turn admission, samples the decision clock once, and constructs the complete fixed reminder frame with JSON-escaped id and prompt. It synchronously queues one `followup()`, appends the id-only dispatch, and releases the reservation in `finally`; only then does it wait for the dispatch barrier. A framing or synchronous enqueue failure appends no dispatch. An append failure faults that owner because the message may already be queued. A later prompt-admission, request-checkpoint, or model failure cannot retract a dispatch. +The accepted path first clears pending persistence and claims the true idle phase through `runMaintenance()`. Inside that task it refolds the exact Session suffix so a direct management mutation that won the claim race cannot be followed by a stale dispatch, samples the decision clock once, constructs the complete fixed reminder frame with JSON-escaped id and prompt, synchronously queues one `followup()`, and appends the id-only dispatch. Waking input remains parked until maintenance settles, so the driver cannot claim the message before dispatch enters the log; only after the task releases the phase does the owner wait for the dispatch barrier. A framing or synchronous enqueue failure is contained and appends no dispatch. An append failure faults that owner because the message may already be queued. A later prompt-admission, request-checkpoint, or model failure cannot retract a dispatch. Agent or plugin disposal cancels timers, stops new work, unwinds the three tool registrations, and waits for in-flight preflights or idle waits. It never deletes durable records during teardown. The narrow crash interval after synchronous followup admission and before durable dispatch may repeat the reminder after recovery; the design prefers a visible duplicate over silent loss and makes no model-success, user-read, external-effect, or exactly-once promise. ### Commit-aware Web receipt -The Schedule package owns `scheduleReminderPresentation()`, which derives `{ scheduleId, prompt, occurrenceAt, deliveryMode }` from create plus dispatch. A dispatch inside an inherited fork prefix folds that parent segment for history display; a child-owned dispatch folds only the child suffix. Presentation therefore never changes live ownership. +The Schedule package owns `scheduleReminderPresentation()`, which derives `{ scheduleId, prompt, occurrenceAt, deliveryMode }` from create plus dispatch. A dispatch inside an inherited fork prefix folds from its nearest preceding `session/end-seed` boundary, preserving nested-generation id reuse; a child-owned dispatch folds only the child suffix. Presentation therefore never changes live ownership. The Host continues to send every raw event on append. It keeps one monotonic watermark per exact live `Session` in a `WeakMap`; only `session/flushed` advancement makes it redeliver newly covered dispatch events with the generic `{ for: 'event', view }` sidecar. The durable `schedule/change` type selects the client renderer. Taking the maximum contains reversed concurrent flush completion, and exact object identity prevents a reused Session id from inheriting another lifecycle's cursor. Attached history independently inspects persistence and adds views only to a stored event prefix whose header identity and every event match the live Session. Persistence canonically writes absent top-level `delegationDepth` as zero, so those two forms are identity-equivalent; cwd, lineage, origin, timestamps, version, id, and every event still match exactly. Missing, failed, divergent, or longer inspection withholds the view while returning raw history. Detached history is already a persisted prefix. A parent dispatch copied into a fork seed therefore appears in child history only after child storage proves that prefix. -The browser Session accepts a repeated seq only when the durable event is deeply identical, then upgrades the sidecar immediately without appending another event. Tail loading and true gap repair retain uncovered events in the existing `liveBuffer`; ordinary older-page pagination keeps receiving live tail events in the current arrays, while a sidecar below the current window stays with the in-flight page and attaches only when that page returns the identical event. Reconnect generations prevent stale page or repair results and `finally` blocks from touching the rebuilt window. `TranscriptAdapter` creates a generic `PresentedEventNode` keyed by the durable event type. `ui-conversation` dispatches it through `conversation.chat.eventview` and retains an expandable JSON fallback, while `ui-schedule` owns the bilingual `schedule/change` reminder row. +The browser Session accepts a repeated seq only when the durable event is deeply identical, then upgrades the sidecar immediately without appending another event. Tail loading and true gap repair retain uncovered events in the existing `liveBuffer`; an accepted repair snapshot starts another pull when it advanced the tail but left a later buffered gap, while an identity conflict triggers a full resync. Ordinary older-page pagination keeps receiving live tail events in the current arrays, while a sidecar below the current window stays with the in-flight page and attaches only when that page returns the identical event. Reconnect generations prevent stale page or repair results and `finally` blocks from touching the rebuilt window. `TranscriptAdapter` creates a generic `PresentedEventNode` keyed by the durable event type. `ui-conversation` dispatches it through `conversation.chat.eventview` and retains an expandable JSON fallback, while `ui-schedule` owns the bilingual `schedule/change` reminder row. ```text schedule_create → Session create event → persistence diff --git a/.agents/notes/implemented/feature/2026-08-05-durable-web-schedule.zh.md b/.agents/notes/implemented/feature/2026-08-05-durable-web-schedule.zh.md index 02d16a55aa..39fe02f96b 100644 --- a/.agents/notes/implemented/feature/2026-08-05-durable-web-schedule.zh.md +++ b/.agents/notes/implemented/feature/2026-08-05-durable-web-schedule.zh.md @@ -19,7 +19,7 @@ Status: implemented | 场景 | 持久事实 | live 行为 | 用户可见结果 | | --- | --- | --- | --- | | 创建与管理 | 原 Session 中的 `schedule/change` create/delete event | Agent-scoped 工具在读取前、变更后执行 checkpoint | 稳定 id、UTC 目标、`scheduled`/`overdue` 与 `session-local` 说明 | -| 到期时繁忙 | 活动 create 仍在 fold 中 | owner 等待 `whenIdle()`、预留准入、排入一次 followup,再追加 dispatch | 一条可回放提醒回执;模型失败不会撤回它 | +| 到期时繁忙 | 活动 create 仍在 fold 中 | owner 等待 `whenIdle()`、认领 idle maintenance、排入一次 followup,再追加 dispatch | 一条可回放提醒回执;模型失败不会撤回它 | | 进程停止或 Session cold | 活动 create 仍在 persistence 中 | 不存在 timer 或后台扫描;resume 重建 owner | 未来目标继续等待;overdue 目标尝试一次 | | fork | 父 event 留在继承前缀 | child fold 从 `seedLength` 开始 | history 可显示父回执,但父提醒不会成为 child 活动工作 | @@ -41,21 +41,21 @@ persistence coordinator 只有在写路径完全停稳后才给出该确认。li ### Live 交付生命周期 -Agent-scoped owner 从持久 fold 派生最早目标。超长目标使用有界 timer 分段,每次 wake 都重新读取墙钟,因此回拨不会提前触发,前跳则会形成 overdue。`reserveTurnAdmission()` 不可用时,record 保持活动,并安装一个 `whenIdle()` wait 后再重试。 +Agent-scoped owner 从持久 fold 派生最早目标。超长目标使用有界 timer 分段,每次 wake 都重新读取墙钟,因此回拨不会提前触发,前跳则会形成 overdue。如果 agent 已被某个轮次或另一项 maintenance task 占用,`runMaintenance()` 会拒绝此次认领;record 保持活动,并由一个 `whenIdle()` wait 触发稍后的重试。被拒绝的 persistence preflight 同样会让 record 保持活动,但不会运行私有重试 timer;后续 agent 活动进入 idle,或成功的 Schedule 管理 preflight 会要求 owner 再次尝试。 -获得准入的路径会先清空 pending persistence、预留 turn admission、只采样一次 decision clock,并使用 JSON-escaped id 与 prompt 构造完整固定 reminder frame。它同步排入一次 `followup()`,追加只含 id 的 dispatch,并在 `finally` 中释放 reservation;之后才等待 dispatch barrier。framing 或同步入队失败不会追加 dispatch。append 失败会使该 owner fault,因为消息可能已经入队。后续 prompt admission、request checkpoint 或模型失败都不能撤回 dispatch。 +获得准入的路径会先清空 pending persistence,并通过 `runMaintenance()` 认领真正的 idle phase。该任务会重新折叠确切的 Session 后缀,从而确保在认领竞态中胜出的直接管理变更之后不会跟随陈旧 dispatch;随后只采样一次 decision clock,使用 JSON-escaped id 与 prompt 构造完整固定 reminder frame,同步排入一次 `followup()`,再追加只含 id 的 dispatch。触发唤醒的 input 会保持 parked,直到 maintenance 结束,因此 driver 无法在 dispatch 进入 log 前认领消息;只有该任务释放 phase 后,owner 才会等待 dispatch barrier。framing 或同步入队失败会被收容,且不会追加 dispatch。append 失败会使该 owner fault,因为消息可能已经入队。后续 prompt admission、request checkpoint 或模型失败都不能撤回 dispatch。 Agent 或插件 dispose 会取消 timer、停止新工作、撤销三个工具注册,并等待进行中的 preflight 或 idle wait。teardown 绝不会删除持久 record。同步 followup 获得准入后、durable dispatch 前的狭窄崩溃窗口可能在恢复后重复提醒;本设计选择可见重复而非静默丢失,不承诺模型成功、用户阅读、外部副作用或 exactly-once。 ### Commit-aware Web 回执 -Schedule package 拥有 `scheduleReminderPresentation()`,从 create 加 dispatch 派生 `{ scheduleId, prompt, occurrenceAt, deliveryMode }`。位于继承 fork 前缀中的 dispatch 会折叠该 parent segment 用于 history 显示;child 自有 dispatch 只折叠 child 后缀。因此 presentation 永远不会改变 live ownership。 +Schedule package 拥有 `scheduleReminderPresentation()`,从 create 加 dispatch 派生 `{ scheduleId, prompt, occurrenceAt, deliveryMode }`。位于继承 fork 前缀中的 dispatch 会从其最近的前置 `session/end-seed` 边界开始折叠,保留嵌套 generation 的 id 复用;child 自有 dispatch 只折叠 child 后缀。因此 presentation 永远不会改变 live ownership。 Host 在 append 时继续发送所有 raw event。它在 `WeakMap` 中按 exact live `Session` 保存一个单调 watermark;只有 `session/flushed` 前进时,才会用通用 `{ for: 'event', view }` sidecar 重投新覆盖的 dispatch event。持久 `schedule/change` 类型用于选择 client renderer。取最大值可以收容反序完成的并发 flush,按对象身份键控则阻止复用的 Session id 继承另一个生命周期的 cursor。 已附加 history 会独立 inspect persistence,只有 stored event prefix 的 header identity 与每个 event 都和 live Session 匹配时才添加 view。persistence 会把顶层缺失的 `delegationDepth` 规范写成零,因此两种形式在身份上等价;cwd、lineage、origin、时间戳、版本、id 与每个 event 仍必须精确匹配。inspect 缺失、失败、分歧或比 live 更长时,只会省略 view,raw history 仍然返回。已分离 history 本身就是持久前缀。因此复制进 fork seed 的 parent dispatch 只有在 child storage 证明该前缀后才会显示。 -浏览器 Session 只有在 durable event 深度一致时才接受重复 seq,随后立即升级 sidecar,不再追加 event。只有尾部加载与真正的 gap repair 才会将尚未覆盖的事件保留在既有 `liveBuffer` 中;普通旧页分页会让当前数组继续接收 live tail 事件,当前 window 以下的 sidecar 则由 in-flight page 自身暂存,只有该页返回身份完全相同的事件时才附着。重连 generation 会阻止陈旧的 page 或 repair 结果以及 `finally` 块触碰重建后的 window。`TranscriptAdapter` 创建按持久事件类型键控的通用 `PresentedEventNode`。`ui-conversation` 通过 `conversation.chat.eventview` 分发,并保留可展开 JSON fallback;`ui-schedule` 则拥有双语 `schedule/change` 提醒行。 +浏览器 Session 只有在 durable event 深度一致时才接受重复 seq,随后立即升级 sidecar,不再追加 event。只有尾部加载与真正的 gap repair 才会将尚未覆盖的事件保留在既有 `liveBuffer` 中;已接受的 repair 快照在推进 tail 但仍留下后续已缓冲的 gap 时会启动另一次 pull,身份冲突则会触发全量重新同步。普通旧页分页会让当前数组继续接收 live tail 事件,当前 window 以下的 sidecar 则由 in-flight page 自身暂存,只有该页返回身份完全相同的事件时才附着。重连 generation 会阻止陈旧的 page 或 repair 结果以及 `finally` 块触碰重建后的 window。`TranscriptAdapter` 创建按持久事件类型键控的通用 `PresentedEventNode`。`ui-conversation` 通过 `conversation.chat.eventview` 分发,并保留可展开 JSON fallback;`ui-schedule` 则拥有双语 `schedule/change` 提醒行。 ```text schedule_create → Session create event → persistence diff --git a/docs/subsystems/persistence.md b/docs/subsystems/persistence.md index 991efe39a0..603a5858ce 100644 --- a/docs/subsystems/persistence.md +++ b/docs/subsystems/persistence.md @@ -10,6 +10,8 @@ The seam is a textbook [capability seam](../../.agents/notes/implemented/archite `session/event` is a *synchronous* notification; persistence plugins copy the event into a per-session controller without blocking the producer. The first pending event starts a fixed batching window, and later events join without resetting its deadline. Expiry starts one durable batch; events admitted during that write receive their own deadline and form a follow-up batch. `session/flush` cancels the wait and drains through quiescence, so the loop still uses it as the ordering and error-observation checkpoint before claiming the next ordinary turn. A rejected background write retains its events and pauses automatic retry; a new event starts a fresh window, while explicit flush retries immediately and reports failure through `agent/error` and the logger, never as a session event past the closed turn. Disposal performs the same final drain. The configured maximum bounds only intentional batching wait, not event-loop scheduling or backend durability latency ([decision](../../.agents/notes/implemented/architecture/2026-08-08-bounded-session-persistence-write-batching.md)). +A `session/flush` listener returns literal `true` only after completing durability work; observe-only listeners return void. Once every listener settles, `SessionStore.flush()` returns `true` and publishes contained `session/flushed(session, throughSeq)` only when at least one listener acknowledged durability and none failed. `throughSeq` is the exclusive event boundary captured at call entry, so events appended during the checkpoint require a later success; concurrent checkpoints may publish boundaries out of order. An empty or observe-only checkpoint returns `false`, and a rejection publishes no success observation. + ## Crash recovery preserves an interrupted turn A backend that reloads a log crashed mid-turn finds an open `turn/start` with no `turn/end`. It does **not** truncate — a single turn can be huge in a long-horizon task (many steps, large tool output), and those events were durably appended before the crash. Instead it closes the orphaned turn with a synthetic `turn/end { reason: { kind: 'interrupted' } }`, keeping the interrupted execution balanced without changing any standalone events before or after it. `interrupted` is the one `TurnEndReason` no loop emits (see [session.md](session.md#why-a-turn-ended-turnendreasonmap)). diff --git a/docs/subsystems/persistence.zh.md b/docs/subsystems/persistence.zh.md index 2391eeb2a7..ee65a98c99 100644 --- a/docs/subsystems/persistence.zh.md +++ b/docs/subsystems/persistence.zh.md @@ -10,6 +10,8 @@ `session/event` 是一个*同步*通知;持久化插件会将事件复制到逐会话控制器,而不阻塞生产方。第一个待处理事件会开启固定批处理窗口,后续事件会加入但不会重置截止时间。窗口到期后会启动一个持久化批次;该次写入期间接纳的事件会获得自己的截止时间,并形成后续批次。`session/flush` 会取消等待并排空至完全停稳,因此循环仍将其用作在领取下一个普通轮次之前的顺序与错误观察检查点。后台写入被拒绝时会保留对应事件并暂停自动重试;新事件会开启新的固定窗口,而显式 flush 会立即重试,并通过 `agent/error` 和 logger 报告失败,绝不会把失败记录成已关闭轮次之后的会话事件。dispose(资源释放)会执行同样的最终排空。配置的最大值只限制有意的批处理等待,不限制事件循环调度或后端完成持久化的延迟([决策](../../.agents/notes/implemented/architecture/2026-08-08-bounded-session-persistence-write-batching.md))。 +`session/flush` 监听器只有在完成持久性工作后才返回字面量 `true`;仅观察监听器返回 void。每个监听器都结算后,仅当至少一个监听器确认持久性且没有监听器失败时,`SessionStore.flush()` 才返回 `true`,并以失败收容方式发布 `session/flushed(session, throughSeq)`。`throughSeq` 是调用入口捕获的事件排他边界,因此检查点期间追加的事件需要后续另一次成功;并发检查点可能不按顺序发布边界。空检查点或仅观察检查点返回 `false`,出现拒绝时不会发布成功观测。 + ## 崩溃恢复保留被中断的轮次 后端重新加载一个在轮次中途崩溃的日志时,会发现一个已打开的 `turn/start` 却没有 `turn/end`。它**不会**截断日志:在长周期任务中,单个轮次可能非常庞大(许多步骤、大量工具输出),而这些事件在崩溃前已被持久追加。后端改为用一个合成的 `turn/end { reason: { kind: 'interrupted' } }` 关闭这个遗留轮次,在不改变其前后任何独立事件的情况下配平被中断的执行。`interrupted` 是唯一一个不由循环发出的 `TurnEndReason`(见 [session.md](session.md#why-a-turn-ended-turnendreasonmap))。 diff --git a/docs/subsystems/subagent.md b/docs/subsystems/subagent.md index a14e9ec5fa..a7ff02d408 100644 --- a/docs/subsystems/subagent.md +++ b/docs/subsystems/subagent.md @@ -156,7 +156,7 @@ type SubagentInterruptAuthority = Every Activation owns its `AgentHandle` and an `ownedChildren: Set`; because one Session has at most one live Activation, the child Session id identifies the live child without another runtime-incarnation reference. Starting a child or submitting parent-originated work registers the child in a continuation-managed parent's set before the child can run, and that parent cannot settle while the set is non-empty. A top-level or other non-continuation Agent has no Activation and stays outside the waiting graph. Child release happens only after the child Agent is quiescent, every child of that child is disposed, the best-effort final session flush settles, and the child's `AgentHandle` completes disposal. -Final settlement awaits `ctx.sessions.flush(session)` but ignores its participation boolean because an arbitrary listener cannot prove that a persistence backend stored the state. Rejection is logged without failing the Activation, and the manager still disposes the handle and releases ownership; the persisted child state may then be missing or stale on a later resume. Manager unload invokes an internal manager-wide drain that closes admission and disposes every live forest; `drainContinuableDescendants(parents)` closes admission only below exact live host-owned Agents and disposes their continuable descendants while unrelated forests remain live. Both await already-admitted materializations in their scope, propagate cancellation top-down, release handles child-first, and await every selected branch despite individual failures. Durable child Sessions survive that process-local teardown. +Final settlement awaits `ctx.sessions.flush(session)` but deliberately does not make its durability acknowledgement a release condition because continuation teardown is best effort. A `false` result still disposes the handle and releases ownership; rejection is logged without failing the Activation, and the persisted child state may then be missing or stale on a later resume. Manager unload invokes an internal manager-wide drain that closes admission and disposes every live forest; `drainContinuableDescendants(parents)` closes admission only below exact live host-owned Agents and disposes their continuable descendants while unrelated forests remain live. Both await already-admitted materializations in their scope, propagate cancellation top-down, release handles child-first, and await every selected branch despite individual failures. Durable child Sessions survive that process-local teardown. ```ts type-equiv /** Attribution for a model coordinator's follow-up to one of its children. */ diff --git a/docs/subsystems/subagent.zh.md b/docs/subsystems/subagent.zh.md index 7bf3595324..933508dd3f 100644 --- a/docs/subsystems/subagent.zh.md +++ b/docs/subsystems/subagent.zh.md @@ -156,7 +156,7 @@ type SubagentInterruptAuthority = 每个 Activation 都拥有自己的 `AgentHandle` 和一个 `ownedChildren: Set`;由于一份会话至多有一个存活 Activation,子会话 id 无需另一个运行时化身引用即可标识存活的子 agent。启动子 agent 或提交源自 parent 的工作,会在子 agent 能够运行之前将其注册到受继续执行管理的父级集合中;只要该集合非空,该父级就无法 settle。顶层或其他非继续执行的 Agent 没有 Activation,处于 waiting 图之外。只有当子 Agent 已完全停稳、该子 agent 的每个子级都已 dispose、best-effort 的最终会话 flush 结算完毕,且子 agent 的 `AgentHandle` 完成 dispose 之后,才会释放子 agent。 -最终结算会等待 `ctx.sessions.flush(session)`,但会忽略其参与布尔值,因为任意 listener 都无法证明某个持久化后端已存储该状态。rejection 会被记录,但不会使 Activation 失败;管理器仍会 dispose 该 handle 并释放所有权,此后持久化的子 agent 状态在后续恢复时可能缺失或陈旧。管理器卸载会调用内部的管理器全局 drain,关闭准入并 dispose 每片在线森林;`drainContinuableDescendants(parents)` 只关闭由 host 确切拥有的在线 Agent 之下的准入,并 dispose 其可继续后代,而无关森林保持在线。两者都会等待各自作用域内已获准的物化过程,自顶向下传播取消,按 child-first 顺序释放 handle,并且即使个别分支失败也会等待所有选中分支。持久化子会话不受该进程内拆卸的影响。 +最终结算会等待 `ctx.sessions.flush(session)`,但由于继续执行拆卸采用 best-effort,明确不把其持久性确认作为释放条件。结果为 `false` 时仍会 dispose 该 handle 并释放所有权;rejection 会被记录,但不会使 Activation 失败,此后持久化的子 agent 状态在后续恢复时可能缺失或陈旧。管理器卸载会调用内部的管理器全局 drain,关闭准入并 dispose 每片在线森林;`drainContinuableDescendants(parents)` 只关闭由 host 确切拥有的在线 Agent 之下的准入,并 dispose 其可继续后代,而无关森林保持在线。两者都会等待各自作用域内已获准的物化过程,自顶向下传播取消,按 child-first 顺序释放 handle,并且即使个别分支失败也会等待所有选中分支。持久化子会话不受该进程内拆卸的影响。 ```ts type-equiv /** Attribution for a model coordinator's follow-up to one of its children. */ diff --git a/packages/client/runtime/README.md b/packages/client/runtime/README.md index d6130e6d9f..63e4bc2090 100644 --- a/packages/client/runtime/README.md +++ b/packages/client/runtime/README.md @@ -40,7 +40,7 @@ SlotsService gives the renderer separate bare observables for `useSessions` and Because the projection is log-ordered, the node array is seq-monotonic by construction: log-only `command/run` / `command/done` nodes splice in by seq, `Session` merges interrupted frozen nodes by their fractional seqs, and a window whose checkpoint cites a shadowed range outside it renders the marker with nothing logged. The marker's summary text, replaced-item count, and estimated shadowed-token count come from the checkpoint's cited `compact/summary` event; a window cut that left that event outside makes those fields unavailable, and a later page that supplies it resolves them. `CommandNode.outcome.sourceEventSeq` preserves a successful command's explicit reference to that summary event, allowing the presentation layer to pair `/compact` with its checkpoint without parsing settlement copy or assuming the two rows are adjacent. Performance contract: one append materializes at most one node and copies the projection only when it adds that node; an event that changes no node keeps the previous array reference (a chunk storm costs nothing), and unchanged nodes keep their object identity. -A Host may redeliver the same Session event seq with a new or changed non-persistent view after the event reaches its presentation commit point. `Session` first requires deep event identity, then upgrades only the sidecar; a generic event view becomes one `PresentedEventNode` keyed by the durable event type. Tail loading and true gap repair continue to use the existing `liveBuffer`. Ordinary `loadOlder` leaves live-tail appends in the current window and prepends its page after the await, while an overlapping late sidecar upgrades immediately. Reconnect advances the generation and clears page or repair ownership, so an older request's result or `finally` cannot mutate or block the rebuilt window. +A Host may redeliver the same Session event seq with a new or changed non-persistent view after the event reaches its presentation commit point. `Session` first requires deep event identity, then upgrades only the sidecar; a generic event view becomes one `PresentedEventNode` keyed by the durable event type. Tail loading and true gap repair continue to use the existing `liveBuffer`; repair continues while each accepted snapshot advances the tail and a buffered gap remains, while an identity-conflicting snapshot triggers a full resync. Ordinary `loadOlder` leaves live-tail appends in the current window and prepends its page after the await, while an overlapping late sidecar upgrades immediately. Reconnect advances the generation and clears page or repair ownership, so an older request's result or `finally` cannot mutate or block the rebuilt window. ## Request inspection diff --git a/packages/client/runtime/README.zh.md b/packages/client/runtime/README.zh.md index 5c0775d480..941f035ab8 100644 --- a/packages/client/runtime/README.zh.md +++ b/packages/client/runtime/README.zh.md @@ -40,7 +40,7 @@ SlotsService 分别为 renderer 提供 `useSessions` 与 `useWorkspaces` 的裸 由于投影按日志顺序,节点数组天然按 seq 单调:仅日志的 `command/run` / `command/done` 节点按 seq 插入,`Session` 按分数 seq 归并被打断的冻结节点,而检查点所引范围落在窗口之外的窗口会渲染出标记且不打印任何日志。标记的摘要文本、被替换条目数量和估算的被遮蔽 token 数量都来自检查点引用的 `compact/summary` 事件;窗口切分把该事件留在窗口外时这些字段不可用,后续包含该事件的分页会解析出它们。`CommandNode.outcome.sourceEventSeq` 保留成功命令对该摘要事件的显式引用,使呈现层能够配对 `/compact` 与其检查点,而无须解析结算文案或假定两行相邻。性能约定:一次追加最多物化一个节点,并且仅在加入该节点时复制投影;不改变任何节点的事件保持上一次的数组引用(分片风暴零成本),未变化的节点保持其对象标识。 -一个 Session event 到达其 presentation 提交点后,Host 可以用同一 seq 重新投递完全相同的事件,并携带新增或变化的非持久 view。`Session` 会先要求事件深度一致,再只升级 sidecar;通用 event view 会按持久事件类型形成一个 `PresentedEventNode`。`liveBuffer` 仍只用于尾部加载与真正的 gap repair。普通 `loadOlder` 会将 live-tail 追加项留在当前窗口中,并在 await 后前插所取页面;重叠的迟到 sidecar 则会立即升级。重连会推进 generation 并清除 page/repair 的所有权,因此旧请求的结果或 `finally` 既不能改写,也不能阻塞重建后的窗口。 +一个 Session event 到达其 presentation 提交点后,Host 可以用同一 seq 重新投递完全相同的事件,并携带新增或变化的非持久 view。`Session` 会先要求事件深度一致,再只升级 sidecar;通用 event view 会按持久事件类型形成一个 `PresentedEventNode`。`liveBuffer` 仍只用于尾部加载与真正的 gap repair;每当已接受的快照推进 tail 后仍留有已缓冲的 gap,repair 就会继续;身份冲突的快照则会触发全量重新同步。普通 `loadOlder` 会将 live-tail 追加项留在当前窗口中,并在 await 后前插所取页面;重叠的迟到 sidecar 则会立即升级。重连会推进 generation 并清除 page/repair 的所有权,因此旧请求的结果或 `finally` 既不能改写,也不能阻塞重建后的窗口。 ## 请求检查 diff --git a/packages/client/runtime/src/client/sessions/session.ts b/packages/client/runtime/src/client/sessions/session.ts index 15e9825dfe..f259eb6f6c 100644 --- a/packages/client/runtime/src/client/sessions/session.ts +++ b/packages/client/runtime/src/client/sessions/session.ts @@ -700,11 +700,16 @@ export class Session implements SessionFace { * overwrite a newer push frame); the window events themselves are never * folded — the host is the only computation site. */ - private installWindow(entries: HistoryEntry[], hasMore: boolean, projections?: ProjectionsBaseline): void { - this.mergeWindow(entries) + private installWindow( + entries: HistoryEntry[], + hasMore: boolean, + projections?: ProjectionsBaseline, + ): { changed: boolean; hasGap: boolean } { + const merged = this.mergeWindow(entries) this.hasMore = hasMore if (projections !== undefined) this.projections.seed(projections) this.notifier.markDirty() + return merged } /** @@ -918,16 +923,32 @@ export class Session implements SessionFace { if (this.stitching) return this.stitching = true const generation = this.openGeneration + let retryGap = false + let acceptedHistory = false try { const { result } = await this.history({ maxMessages: PAGE_MESSAGES }) if (generation !== this.openGeneration || this.openState !== 'open') return if (result.ok) { - this.installWindow(result.value.events, result.value.hasMore, result.value.projections) + acceptedHistory = true + const previousTail = this.windowTailSeq() + const { hasGap } = this.installWindow( + result.value.events, + result.value.hasMore, + result.value.projections, + ) + const repairedTail = this.windowTailSeq() + retryGap = hasGap && repairedTail !== null + && (previousTail === null || repairedTail > previousTail) } else { this.mergeWindow() } } catch (error) { if (generation === this.openGeneration) { + if (acceptedHistory) { + console.error('[web-runtime] gap repair snapshot failed validation:', error) + void this.resync() + return + } console.error('[web-runtime] gap repair failed:', error) try { this.mergeWindow() @@ -940,6 +961,7 @@ export class Session implements SessionFace { if (generation === this.openGeneration) { this.stitching = false this.notifier.markDirty() + if (retryGap) void this.repairGap() } } } diff --git a/packages/client/runtime/tests/session.spec.ts b/packages/client/runtime/tests/session.spec.ts index 5cadcab058..d9e5a5f470 100644 --- a/packages/client/runtime/tests/session.spec.ts +++ b/packages/client/runtime/tests/session.spec.ts @@ -821,6 +821,64 @@ describe('live event path', () => { const seqs = session.getSnapshot().nodes.map(n => n.seq) expect(seqs).toEqual([1, 3, 7, 9]) // both turns' user/assistant, no hole, no duplicate 9 }) + + it('continues repair when one tail snapshot leaves a later buffered gap', async () => { + const initial = logRange(0, 6) + const firstGap = ev.user(9, 'first repaired event') + const laterGap = ev.user(12, 'later buffered event') + const firstSnapshot = [...initial, ...logRange(6, 9), firstGap] + const completeSnapshot = [...firstSnapshot, ...logRange(10, 12), laterGap] + const firstRepair = deferred>>() + const secondRepair = deferred>>() + const { api, session } = await opened(initial) + let repairs = 0 + api.onHistory = () => ++repairs === 1 ? firstRepair.promise : secondRepair.promise + + session.handleMuxEnvelope('first-gap' as never, { + type: 'session/event', sessionId: SID, event: firstGap, + }) + session.handleMuxEnvelope('later-gap' as never, { + type: 'session/event', sessionId: SID, event: laterGap, + }) + firstRepair.resolve(ok({ events: entries(firstSnapshot) as never[], hasMore: false })) + + await vi.waitFor(() => { expect(repairs).toBe(2) }) + secondRepair.resolve(ok({ events: entries(completeSnapshot) as never[], hasMore: false })) + await vi.waitFor(() => { + expect(session.getSnapshot().nodes.map(node => node.seq)).toEqual([9, 12]) + }) + }) + + it('resyncs when a successful gap snapshot conflicts with a buffered event identity', async () => { + const initial = logRange(0, 6) + const live = ev.user(9, 'live identity') + const conflicting = ev.user(9, 'conflicting history identity') + const consistent = [...initial, ...logRange(6, 9), live] + const { api, session } = await opened(initial) + let repairs = 0 + api.onHistory = () => { + repairs++ + return repairs === 1 + ? histResponse([...initial, ...logRange(6, 9), conflicting]) + : histResponse(consistent) + } + const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => undefined) + try { + session.handleMuxEnvelope('gap' as never, { + type: 'session/event', sessionId: SID, event: live, + }) + await vi.waitFor(() => { + expect(repairs).toBe(2) + expect(session.getSnapshot().nodes.map(node => node.seq)).toEqual([9]) + }) + expect(errorSpy).toHaveBeenCalledWith( + '[web-runtime] gap repair snapshot failed validation:', + expect.objectContaining({ message: 'session event identity mismatch at seq 9' }), + ) + } finally { + errorSpy.mockRestore() + } + }) }) describe('paging', () => { diff --git a/packages/core/session/src/index.ts b/packages/core/session/src/index.ts index 69e8a0e8a6..d210351e36 100644 --- a/packages/core/session/src/index.ts +++ b/packages/core/session/src/index.ts @@ -109,6 +109,8 @@ declare module 'cordis' { * Observe a successful durability checkpoint. `throughSeq` is the exclusive * event boundary captured when {@link SessionStore.flush} began; events * appended while its listeners run require a later successful checkpoint. + * Concurrent checkpoints may publish their boundaries out of order, so a + * consumer retaining progress must advance by the maximum observed value. * No notification is published when no durability listener participated or * any listener failed. Observer failures are logged and contained. * Scope-filtered dispatch (`@deepseek-ai/dsh-scope`) reuses the session's diff --git a/packages/schedule/tool-schedule/README.md b/packages/schedule/tool-schedule/README.md index eb169d4d42..9bd3e5a151 100644 --- a/packages/schedule/tool-schedule/README.md +++ b/packages/schedule/tool-schedule/README.md @@ -16,7 +16,7 @@ The package owns the strict version-1 `schedule/change` create, delete, and disp Replay rejects unknown versions, extra fields, reused ids, and delete or dispatch transitions against inactive records. Normal sessions fold the complete log. A fork folds only `session.events.slice(session.header.seedLength ?? 0)`, so it does not inherit its parent's reminders. The package's `./invariant` companion applies the same policy to existing logs and candidate events. -`scheduleReminderPresentation(events, dispatchSeq, seedLength)` is the pure Host-facing receipt projection. It pairs a dispatch with the active create in the same ownership segment and returns `scheduleId`, prompt, occurrence, and `session-local` mode. A dispatch inside a persisted fork prefix folds that parent prefix for history display; a child-owned dispatch folds only the child suffix, so presentation never changes live ownership. +`scheduleReminderPresentation(events, dispatchSeq, seedLength)` is the pure Host-facing receipt projection. It pairs a dispatch with the active create in the same ownership segment and returns `scheduleId`, prompt, occurrence, and `session-local` mode. A dispatch inside a persisted fork prefix folds from its nearest preceding `session/end-seed` boundary, so nested generations may reuse session-local ids without hiding ancestor receipts; a child-owned dispatch folds only the child suffix, so presentation never changes live ownership. ## Management tools @@ -79,6 +79,7 @@ The reminder appends after existing history and preserves its reusable prefix. I ## Known Limitations and Deferred Work - **Session-local delivery only** — a reminder runs on time only while its original session is live; a cold session receives no external notification and processes an overdue record only after resume. +- **Activity-driven persistence retry** — a rejected due preflight leaves the overdue record active but starts no private retry timer; the owner retries after later Agent activity reaches idle or a successful Schedule management preflight asks it to recompute. - **After-only protocol** — version 1 rejects `at`, `every_seconds`, `cron`, and `time_zone`; those rules require later protocol variants rather than hidden compatibility fields. - **Narrow crash duplicate window** — a crash after synchronous followup admission but before the dispatch checkpoint can repeat the reminder after recovery; the package does not claim model completion, user acknowledgement, or exactly-once external effects. - **Load-order boundary** — the plugin does not scan or adopt agents that were already live when it loaded. diff --git a/packages/schedule/tool-schedule/README.zh.md b/packages/schedule/tool-schedule/README.zh.md index b090c01d15..2c63e97f35 100644 --- a/packages/schedule/tool-schedule/README.zh.md +++ b/packages/schedule/tool-schedule/README.zh.md @@ -16,7 +16,7 @@ 回放会拒绝未知版本、额外字段、重复使用的 id,以及针对非活动记录的 delete 或 dispatch 转换。普通会话折叠完整日志。fork 只折叠 `session.events.slice(session.header.seedLength ?? 0)`,因此不会继承父会话的提醒。此包的 `./invariant` 配套项会对现有日志和候选事件应用相同策略。 -`scheduleReminderPresentation(events, dispatchSeq, seedLength)` 是供 Host 使用的纯回执投影。它把 dispatch 与同一 ownership segment 中的活动 create 配对,并返回 `scheduleId`、prompt、occurrence 和 `session-local` 模式。位于已持久 fork 前缀中的 dispatch 会折叠对应 parent 前缀用于 history 显示;child 自有 dispatch 只折叠 child 后缀,因此 presentation 绝不会改变 live ownership。 +`scheduleReminderPresentation(events, dispatchSeq, seedLength)` 是供 Host 使用的纯回执投影。它把 dispatch 与同一 ownership segment 中的活动 create 配对,并返回 `scheduleId`、prompt、occurrence 和 `session-local` 模式。位于已持久 fork 前缀中的 dispatch 会从最近的前置 `session/end-seed` 边界开始折叠,因此嵌套 generation 可以复用会话本地 id,而不会隐藏祖先回执;child 自有 dispatch 只折叠 child 后缀,因此 presentation 绝不会改变 live ownership。 ## 管理工具 @@ -79,6 +79,7 @@ reminder_prompt_json: ## 已知限制与暂缓事项 - **仅限会话本地交付**:提醒只有在原会话 live 时才能准时运行;cold 会话不会收到外部通知,只有恢复后才会处理 overdue 记录。 +- **活动驱动的持久化重试**:到期 preflight 被拒绝后,overdue 记录仍保持活动,但不会启动私有重试 timer;后续 agent 活动进入 idle,或成功的 Schedule 管理 preflight 要求 owner 重新计算后,owner 会重试。 - **仅支持 after 协议**:版本 1 拒绝 `at`、`every_seconds`、`cron` 和 `time_zone`;这些规则需要后续协议变体,而不是隐藏的兼容字段。 - **存在狭窄的崩溃重复窗口**:同步 `followup` 获得准入后、dispatch 检查点完成前发生崩溃,可能使提醒在恢复后重复;此包不承诺模型完成、用户确认或外部副作用恰好一次。 - **加载顺序边界**:插件不会扫描或接管加载时已经 live 的 agent。 diff --git a/packages/schedule/tool-schedule/src/domain.ts b/packages/schedule/tool-schedule/src/domain.ts index 290af3fdea..f03aec4c54 100644 --- a/packages/schedule/tool-schedule/src/domain.ts +++ b/packages/schedule/tool-schedule/src/domain.ts @@ -289,10 +289,10 @@ export function scheduleView(record: AfterScheduleRecord, now: number): Schedule /** * Derive the Web receipt for one dispatch from its owning stream segment. - * A dispatch inside an inherited fork prefix folds that original prefix; a - * child-owned dispatch folds only the child suffix, preserving the same - * `seedLength` ownership rule as the live runtime while still allowing a - * persisted parent receipt to render in child history. + * A dispatch inside an inherited fork prefix folds from its nearest preceding + * `session/end-seed` boundary; a child-owned dispatch folds only the child + * suffix. Nested forks can therefore reuse session-local ids without hiding a + * persisted ancestor receipt in descendant history. * @param events - Complete contiguous Session log. * @param dispatchSeq - Exact event seq to present. * @param seedLength - Inherited fork prefix length. @@ -317,7 +317,9 @@ export function scheduleReminderPresentation( const dispatch = decodeScheduleChange(event.data) if (dispatch.operation !== 'dispatch') return undefined - const segmentStart = dispatchSeq < seedLength ? 0 : seedLength + const segmentStart = dispatchSeq < seedLength + ? events.slice(0, dispatchSeq).findLastIndex(candidate => candidate.type === 'session/end-seed') + 1 + : seedLength const before = foldScheduleEvents(events.slice(segmentStart, dispatchSeq)) const record = before.active.find(candidate => candidate.id === dispatch.id) if (record === undefined) { diff --git a/packages/schedule/tool-schedule/src/runtime.ts b/packages/schedule/tool-schedule/src/runtime.ts index 7b615bc6d0..ebbe46d6f5 100644 --- a/packages/schedule/tool-schedule/src/runtime.ts +++ b/packages/schedule/tool-schedule/src/runtime.ts @@ -156,6 +156,22 @@ export class ScheduleOwner { ) } + /** Fold the current exact owner suffix and contain a corrupt durable stream. */ + private readEarliest(): AfterScheduleRecord | undefined { + try { + const folded = foldScheduleEvents( + this.agent.session.events, + this.agent.session.header.seedLength ?? 0, + ) + return earliest(folded.active) + } catch (error: unknown) { + this.faulted = true + const detail = error instanceof ScheduleLogError ? error.message : renderThrown(error) + this.ctx.logger.warn(`tool-schedule: corrupt schedule log for agent "${this.agent.id}": ${detail}`) + return undefined + } + } + /** Preflight, fold, arm, or dispatch the next active one-shot reminder. */ private async driveOnce(): Promise { this.clearTimer() @@ -171,19 +187,7 @@ export class ScheduleOwner { // oxlint-disable-next-line typescript/no-unnecessary-condition -- disposal or replacement can win while persistence is awaited. if (this.stopping || !this.isLive()) return - let record: AfterScheduleRecord | undefined - try { - const folded = foldScheduleEvents( - this.agent.session.events, - this.agent.session.header.seedLength ?? 0, - ) - record = earliest(folded.active) - } catch (error: unknown) { - this.faulted = true - const detail = error instanceof ScheduleLogError ? error.message : renderThrown(error) - this.ctx.logger.warn(`tool-schedule: corrupt schedule log for agent "${this.agent.id}": ${detail}`) - return - } + const record = this.readEarliest() if (record === undefined) return const target = Date.parse(record.scheduledAt) @@ -193,47 +197,50 @@ export class ScheduleOwner { return } - const release = this.agent.reserveTurnAdmission() - if (release === undefined) { - this.waitForIdle() + let maintenance: Promise + try { + maintenance = this.agent.runMaintenance(() => { + if (this.stopping || !this.isLive()) return Promise.resolve(false) + const claimedRecord = this.readEarliest() + if (claimedRecord === undefined) return Promise.resolve(false) + const claimedTarget = Date.parse(claimedRecord.scheduledAt) + const decisionNow = Date.now() + if (decisionNow < claimedTarget) { + this.arm(claimedTarget, decisionNow) + return Promise.resolve(false) + } + try { + const message = createUserMessage({ + content: [{ type: 'text', text: renderReminderFraming(claimedRecord) }], + source: { kind: 'plugin', plugin: 'tool-schedule' }, + }) + this.agent.followup(message) + } catch (error: unknown) { + if (this.isLive()) { + this.ctx.logger.warn(`tool-schedule: framing or followup failed for agent "${this.agent.id}": ${renderThrown(error)}`) + } + return Promise.resolve(false) + } + try { + this.agent.session.append('schedule/change', { + version: 1, + operation: 'dispatch', + id: claimedRecord.id, + }) + } catch (error: unknown) { + this.faulted = true + this.clearTimer() + this.ctx.logger.warn(`tool-schedule: dispatch append failed for agent "${this.agent.id}": ${renderThrown(error)}`) + return Promise.resolve(false) + } + return Promise.resolve(true) + }) + } catch (_busy: unknown) { + // `runMaintenance` rejects synchronously only while another agent activity owns the idle phase. + if (this.isLive()) this.waitForIdle() return } - - try { - // oxlint-disable-next-line typescript/no-unnecessary-condition -- reservation can invalidate the owner. - if (this.stopping || !this.isLive()) return - const decisionNow = Date.now() - if (decisionNow < target) { - this.arm(target, decisionNow) - return - } - const message = createUserMessage({ - content: [{ type: 'text', text: renderReminderFraming(record) }], - source: { kind: 'plugin', plugin: 'tool-schedule' }, - }) - try { - this.agent.followup(message) - } catch (error: unknown) { - if (this.isLive()) { - this.ctx.logger.warn(`tool-schedule: followup failed for agent "${this.agent.id}": ${renderThrown(error)}`) - } - return - } - try { - this.agent.session.append('schedule/change', { - version: 1, - operation: 'dispatch', - id: record.id, - }) - } catch (error: unknown) { - this.faulted = true - this.clearTimer() - this.ctx.logger.warn(`tool-schedule: dispatch append failed for agent "${this.agent.id}": ${renderThrown(error)}`) - return - } - } finally { - release() - } + if (!await maintenance) return try { await flushSchedulePersistence(this.ctx, this.agent.session) diff --git a/packages/schedule/tool-schedule/tests/domain.spec.ts b/packages/schedule/tool-schedule/tests/domain.spec.ts index 5b8ecae638..cf09ad1c48 100644 --- a/packages/schedule/tool-schedule/tests/domain.spec.ts +++ b/packages/schedule/tool-schedule/tests/domain.spec.ts @@ -109,6 +109,19 @@ describe('version-1 Schedule decoding and folding', () => { occurrenceAt: '2026-08-05T12:00:00.000Z', deliveryMode: 'session-local', }) + const nested = [ + scheduleEvent(createData('same-id', 'grandparent prompt'), 0), + scheduleEvent({ version: 1, operation: 'dispatch', id: 'same-id' }, 1), + { type: 'session/end-seed', seq: 2, time: 1, data: {} } as SessionEvent, + scheduleEvent(createData('same-id', 'parent prompt'), 3), + scheduleEvent({ version: 1, operation: 'dispatch', id: 'same-id' }, 4), + ] + expect(scheduleReminderPresentation(nested, 4, 5)).toEqual({ + scheduleId: 'same-id', + prompt: 'parent prompt', + occurrenceAt: '2026-08-05T12:00:00.000Z', + deliveryMode: 'session-local', + }) expect(scheduleReminderPresentation(events, 2, 2)).toBeUndefined() expect(scheduleReminderPresentation([ { type: 'session/end-seed', seq: 0, time: 1, data: {} }, diff --git a/packages/schedule/tool-schedule/tests/runtime.spec.ts b/packages/schedule/tool-schedule/tests/runtime.spec.ts index 1abe0dfe82..8c276f45e7 100644 --- a/packages/schedule/tool-schedule/tests/runtime.spec.ts +++ b/packages/schedule/tool-schedule/tests/runtime.spec.ts @@ -277,6 +277,30 @@ describe('Schedule timer and admission runtime', () => { expect(test.followed).toHaveLength(1) await owner.dispose() }) + + it('rechecks the durable fold after claiming maintenance', async () => { + const test = await harness() + appendAfter(test, 'schedule-1', 1, Date.now() - 1_000) + test.controls.onReserve = () => { + test.controls.onReserve = undefined + test.agent.session.append('schedule/change', { + version: 1, + operation: 'delete', + id: ScheduleId('schedule-1'), + }) + } + const owner = ownerFor(test) + owner.start() + await settle() + + expect(test.controls.releaseCount).toBe(1) + expect(test.followed).toEqual([]) + expect(test.agent.session.events.at(-1)?.data).toMatchObject({ operation: 'delete' }) + owner.requestDrive() + await settle() + expect(test.followed).toEqual([]) + await owner.dispose() + }) }) describe('Schedule runtime failure and teardown boundaries', () => { @@ -459,7 +483,7 @@ describe('Schedule runtime failure and teardown boundaries', () => { expect(unreadable.followed).toEqual([]) }) - it('contains owner startup and run failures', async () => { + it('contains owner startup, maintenance, and framing failures', async () => { const startup = await harness() const startSpy = vi.spyOn(startup.ctx.agents, 'withoutInitiator') .mockImplementation(() => { throw new Error('initiator closing') }) @@ -477,6 +501,29 @@ describe('Schedule runtime failure and teardown boundaries', () => { expect(departedStartup.controls.flushCount).toBe(0) departedStartSpy.mockRestore() + const maintenanceFailure = await harness() + appendAfter(maintenanceFailure, 'schedule-1', 1, Date.now() - 1_000) + const maintenanceSpy = vi.spyOn(maintenanceFailure.agent, 'runMaintenance') + .mockImplementation(() => Promise.reject(new Error('maintenance failed'))) + const maintenanceOwner = ownerFor(maintenanceFailure) + maintenanceOwner.start() + await settle() + expect(maintenanceFailure.followed).toEqual([]) + maintenanceOwner.requestDrive() + await settle() + expect(maintenanceSpy).toHaveBeenCalledOnce() + + const departedMaintenance = await harness() + appendAfter(departedMaintenance, 'schedule-1', 1, Date.now() - 1_000) + vi.spyOn(departedMaintenance.agent, 'runMaintenance').mockImplementation(() => { + departedMaintenance.disposeAgent() + return Promise.reject(new Error('maintenance failed after detach')) + }) + const departedMaintenanceOwner = ownerFor(departedMaintenance) + departedMaintenanceOwner.start() + await settle() + expect(departedMaintenance.followed).toEqual([]) + const runFailure = await harness() appendAfter(runFailure, 'schedule-1', 1, Date.now() - 1_000) const uuidSpy = vi.spyOn(globalThis.crypto, 'randomUUID').mockImplementation(() => { throw 'message failed' }) @@ -486,7 +533,7 @@ describe('Schedule runtime failure and teardown boundaries', () => { uuidSpy.mockRestore() failingOwner.requestDrive() await settle() - expect(runFailure.followed).toEqual([]) + expect(runFailure.followed).toHaveLength(1) const departedRun = await harness() appendAfter(departedRun, 'schedule-1', 1, Date.now() - 1_000)