Merge updated origin/master into feat/tui-master-port
This commit is contained in:
Tianyi Cui
118 files changed
+2540
-848
No files matched your search
+2
-2
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-06-21-bounded-llm-request-recovery.md
|
||||
2026-06-21-bounded-llm-request-recovery.md: 9c9d8a02595b988535158c9aa0ec43d6f1fa0c89
|
||||
2026-06-21-bounded-llm-request-recovery.zh.md: bb9e430eaf87789452fd4cc89085d7d635f54ed1
|
||||
2026-06-21-bounded-llm-request-recovery.md: 83d47e3a7d91bbcd2ceaf7b11cf13316142eb3ed
|
||||
2026-06-21-bounded-llm-request-recovery.zh.md: 00dcbad3d1023ad33a22297bfe938b94bce839d4
|
||||
@@ -4,6 +4,8 @@ Status: implemented
|
||||
|
||||
English | [中文](2026-06-21-bounded-llm-request-recovery.zh.md)
|
||||
|
||||
The [per-provider request retry policy](../feature/2026-07-24-provider-retry-policies.md) extends this foundation with exact-provider configuration and an explicit unbounded mode. This note continues to own structured failure facts, the closed-step recovery boundary, normal mode's transient defaults, visible single attempts, and durable retry status.
|
||||
|
||||
## Problem
|
||||
|
||||
`dsh-llm` can report provider failures either by throwing during adapter dispatch or iteration or by ending with `finish { kind: 'error' | 'aborted' }`. The final adapter boundary tags thrown failures so `dsh-agent-loop` can distinguish them from middleware and result-processing defects, and the loop normalizes both delivery forms into `agent/request-error` after closing the failed step. An unhandled failure is terminal; a handling listener repairs policy-owned state, returns `{ kind: 'retry' }`, and stops waterfall delegation. The [retry-action decision](../simplification/2026-07-27-request-error-retry-action.md) owns this return contract.
|
||||
@@ -16,7 +18,7 @@ The prior boundary left three narrower gaps.
|
||||
- Retry ownership differs by adapter. The hand-written DeepSeek adapter makes one attempt, while pi-ai profiles can enable opaque library retries. Combining hidden transport retries with an `agent/request-error` listener would multiply attempts and omit intermediate failures from the session log.
|
||||
- A recovered failure has no durable status fact. The failed step and chunks remain reconstructable, but an observer cannot tell whether the agent is deliberately backing off, for how long, or why. A long silent wait looks like a stalled loop.
|
||||
|
||||
The goal is bounded recovery from transient failures of the same explicit provider/model request. Provider or model failover, response splicing, and semantic output repair are different problems and have no current consumer.
|
||||
The default policy provides bounded recovery from transient failures of the same explicit provider/model request. Provider or model failover, response splicing, and semantic output repair are different problems and have no current consumer.
|
||||
|
||||
## Decision
|
||||
|
||||
@@ -50,31 +52,19 @@ The shared transient-code set is intentionally small: adapter mappings for `RATE
|
||||
|
||||
`@deepseek-ai/dsh-llm-retry` is a function plugin that listens to `agent/request-error`. It introduces no service or new loop branch; the agent-loop package changes only the data carried through its existing failed-step recovery control flow.
|
||||
|
||||
The `agent/request-error` seam carries only the current `LlmFailure`; the loop owns no retry policy or attempt history. Each recovery plugin keeps a private per-agent counter for its own handled failures and clears it at terminal `agent/settled`. Alternating transient and context-overflow failures therefore consume the `dsh-llm-retry` and compact-basic budgets independently; the maximum request count is one plus the sum of the finite budgets of the loaded recovery policies.
|
||||
The `agent/request-error` seam carries the current `LlmFailure`, an immutable list of prior failures that authorized retry turns in the consecutive recovery sequence, and the serving registration's immutable retry policy. The loop transports but does not interpret that policy, owns the consecutive failure history, and clears it after a successful model request. Normal `dsh-llm-retry` policy counts durable retry records scheduled by the same exact-provider policy, while `dsh-compact-basic` keeps its own context-overflow budget. Alternating transient and context-overflow failures therefore consume their owning finite budgets independently; the maximum request count is one plus the sum of the loaded finite budgets.
|
||||
|
||||
The plugin resolves and validates this deployment configuration at load:
|
||||
|
||||
```ts ignore-check
|
||||
interface Config {
|
||||
maxTransientRetries?: number
|
||||
initialDelayMs?: number
|
||||
maxDelayMs?: number
|
||||
jitterRatio?: number
|
||||
retryableCodes?: string[]
|
||||
}
|
||||
```
|
||||
|
||||
The defaults are two transient retries, a 500 millisecond initial delay, a 10 second delay cap, 10 percent jitter, and the five transient codes above (`RATE_LIMIT`, `SERVER`, `TIMEOUT`, `TRANSPORT`, and `EMPTY_RESPONSE`). The count and delay bounds match the conservative edge of the inspected implementations: [OpenCode uses two request retries with 500 ms/10 s bounds](https://github.com/anomalyco/opencode/blob/9976269ab1accfc9f9dc98a4a688c516934de422/%70ackages/llm/src/route/executor.ts#L36-L39), [Pi separates three agent-level retries from provider retries and defaults provider retries to zero](https://github.com/earendil-works/pi/blob/3da591ab74ab9ab407e72ed882600b2c851fae21/%70ackages/coding-agent/docs/settings.md#L139-L147), and [Codex uses finite request/stream budgets plus a five-minute idle timeout](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/model-provider-info/src/lib.rs#L25-L33). Ten percent follows [Codex's bounded jitter](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/codex-client/src/retry.rs#L40-L47). Two retries mean at most three provider requests when no other recovery policy applies. `maxTransientRetries` is a non-negative integer, delays are positive finite numbers with `initialDelayMs <= maxDelayMs`, `jitterRatio` is in `[0, 1]`, and codes are non-empty and unique. These are Cordis config fields rather than hidden constants so deployments can choose different cost and latency budgets.
|
||||
The [provider-policy decision](../feature/2026-07-24-provider-retry-policies.md) owns the current configuration shape. Provider adapters register their nested `retryPolicy`; omission uses normal defaults: two transient retries, a 500 millisecond initial delay, a 10 second delay cap, 10 percent jitter, and the five transient codes above. The count and delay bounds match the conservative edge of the inspected implementations: [OpenCode uses two request retries with 500 ms/10 s bounds](https://github.com/anomalyco/opencode/blob/9976269ab1accfc9f9dc98a4a688c516934de422/%70ackages/llm/src/route/executor.ts#L36-L39), [Pi separates three agent-level retries from provider retries and defaults provider retries to zero](https://github.com/earendil-works/pi/blob/3da591ab74ab9ab407e72ed882600b2c851fae21/%70ackages/coding-agent/docs/settings.md#L139-L147), and [Codex uses finite request/stream budgets plus a five-minute idle timeout](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/model-provider-info/src/lib.rs#L25-L33). Ten percent follows [Codex's bounded jitter](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/codex-client/src/retry.rs#L40-L47).
|
||||
|
||||
For an eligible failure with budget remaining, the one-based transient retry count uses bounded exponential backoff. A valid `providerRetryAfterMs` replaces exponential backoff only when it does not exceed `maxDelayMs`; a longer provider delay causes delegation instead of an earlier retry that violates the provider instruction. Local backoff multiplies by an injected random factor in `[1 - jitterRatio, 1 + jitterRatio]` and clamps the final value to `maxDelayMs`; provider delay is not jittered.
|
||||
|
||||
The plugin owns a lifetime `AbortController` and tracks every active backoff callback. Each wait fuses the waterfall's turn signal with that lifetime signal. Effect cleanup first unregisters the listener, then aborts and awaits the active callbacks; a captured callback whose lifetime signal aborts returns without retrying or entering the rest of its captured waterfall. This makes HMR disposal quiescent even though Cordis has already captured the listener.
|
||||
The plugin owns a lifetime `AbortController` and tracks every active recovery callback, including delegated waterfall work and backoff. Effect cleanup first unregisters the listener, then aborts and awaits the active callbacks; abort wins over a late delegated retry decision, and a captured callback can neither retry nor enter the rest of its waterfall after disposal. This makes HMR disposal quiescent even though Cordis has already captured the listener.
|
||||
|
||||
Before sleeping, `dsh-llm-retry` appends one non-surface `llm/retry` session event containing the turn, failed step, one-based transient retry number, configured maximum, scheduled delay, and `LlmFailure`. The plugin owns the `SessionEventMap` augmentation; `dsh-session` remains generic persistence and does not absorb the optional policy's vocabulary. The event says what was scheduled, not that the next request completed; cancellation during the delay is subsequently visible on `turn/end`. The event ships only with a production renderer and replay/snapshot coverage, because its purpose is operational state rather than trace collection.
|
||||
Before sleeping, `dsh-llm-retry` appends one non-surface `llm/retry` session event containing the turn, failed step, provider, policy mode, complete resolved-policy key, provider-policy retry number, mode-specific finite maximum when present, scheduled delay, and `LlmFailure`. The key sorts the code set and separates retry histories when a provider route is replaced by a behaviorally different same-mode policy. The plugin owns the `SessionEventMap` augmentation; `dsh-session` remains generic persistence and does not absorb the optional policy's vocabulary. The event says what was scheduled, not that the next request completed; cancellation during the delay is subsequently visible on `turn/end`. The event ships only with a production renderer and replay/snapshot coverage, because its purpose is operational state rather than trace collection.
|
||||
|
||||
The listener calls `next()` for a non-transient code, an exhausted policy budget, or an over-cap provider delay. This preserves composition with context-overflow recovery and later policy plugins. For an owned failure it records and awaits the delay, then returns `{ kind: 'retry' }` without delegating. Turn cancellation and plugin disposal end the wait without returning a retry; the loop's cancellation/disposal checks remain authoritative.
|
||||
|
||||
The agent-spine demo bundle loads the plugin so the shared stdio/TUI, one-shot CLI, and ACP example compositions use the same bounded policy. Library consumers retain explicit plugin composition: omitting the plugin leaves request failures terminal.
|
||||
The agent-spine demo bundle loads the plugin so the shared stdio/TUI, one-shot CLI, and ACP example compositions use the same provider-routed policy. Library consumers retain explicit plugin composition: omitting the plugin leaves request failures terminal.
|
||||
|
||||
### Make one layer own visible attempts
|
||||
|
||||
@@ -101,7 +91,7 @@ If recovery is exhausted, the final failure is stored once on `turn/end.reason`
|
||||
- Automatic provider or model failover. Requests already select one explicit provider and model, and the provider registry deliberately has one adapter owner per provider.
|
||||
- Retrying or continuing after a successful terminal finish, or splicing chunks from two attempts into one assistant message.
|
||||
- Repairing malformed tool arguments, refusals, content filters, or other semantic model output.
|
||||
- Unbounded retries, unattended retry-until-cancelled behavior, circuit breakers, shared provider health, or cross-agent retry budgets.
|
||||
- Circuit breakers, shared provider health, or cross-agent retry budgets.
|
||||
- Changing `llm/stream` into a response lifecycle or adding convenience generation APIs without a production consumer.
|
||||
|
||||
## Alternatives considered
|
||||
@@ -110,7 +100,7 @@ If recovery is exhausted, the final failure is stored once on `turn/end.reason`
|
||||
- **Add response start, interrupted, discarded, failed, and committed events to `dsh-llm`** — rejected because the agent log already separates raw chunks, successful messages, and numbered attempts. A second state machine would duplicate ownership without enabling the bounded same-route retry.
|
||||
- **Add logical routes, capability matrices, and failover selection** — rejected because current requests already name provider and model explicitly, one adapter owns each provider, and no current consumer requires automatic fallback or can prove semantic compatibility.
|
||||
- **Put `retryable` or `failover` on `LlmFailure`** — rejected because adapters report facts while deployment policy decides action. The same 429 may be retried in an interactive bundle and rejected in a cost-capped batch.
|
||||
- **Retry forever while the caller remains active** — rejected because it gives one request unbounded cost and latency. Visible status makes bounded waiting understandable; it does not make an unlimited budget safe.
|
||||
- **Retry forever while the caller remains active** — the [per-provider policy](../feature/2026-07-24-provider-retry-policies.md) supersedes this rejection for explicit `always` entries while retaining bounded normal mode as the default.
|
||||
- **Log retry status only through the process logger** — rejected because process logs do not reconstruct session behavior and cannot drive replayed UI state.
|
||||
- **Keep only flat codes** — rejected because retry delay and provider request id are structured provider facts, and HTTP status is necessary for diagnosis when different wire failures share one stable code.
|
||||
|
||||
@@ -120,11 +110,11 @@ If recovery is exhausted, the final failure is stored once on `turn/end.reason`
|
||||
- An adapter-thrown `Error` reaches `agent/request-error` as the exact same object while its sidecar `LlmFailure` reaches the adjacent argument; tests retain the existing identity assertion for extensible and frozen third-party errors.
|
||||
- DeepSeek and pi-ai adapter tests cover representative 400, 401/403, 429, 5xx, connection, malformed/truncated stream, timeout, abort, retry-after seconds/date, request-id, and unknown-SDK-error paths without recovery policy parsing message text.
|
||||
- Pi-ai pins the SDK option to zero retries and performs one observed wire attempt for a retryable provider response; separate tests make removing either boundary fail.
|
||||
- `agent/request-error` carries only current failure facts; each plugin clears its private per-agent counter at terminal idle, and alternating transient/context-overflow integration tests prove the two policies consume only their own finite budgets.
|
||||
- `dsh-llm-retry` validates every config field at Loader startup, delegates all ineligible paths with `next()`, and makes at most `maxTransientRetries + 1` provider requests when no other policy applies.
|
||||
- HMR-during-backoff tests prove disposal unregisters the listener, aborts and awaits its captured callbacks, makes no retry request after disposal, and leaves no timer or promise alive.
|
||||
- `agent/request-error` carries current failure facts, immutable prior-retried failure facts, and the serving registration's immutable retry policy; a success clears the history, and alternating transient/context-overflow integration tests prove the two policies consume only their own finite budgets.
|
||||
- Each provider adapter validates its nested retry policy at Loader startup, and `ctx.llm` captures it with the route; normal mode delegates ineligible paths and makes at most `maxRetries + 1` provider requests when no other policy applies.
|
||||
- HMR-during-backoff tests prove disposal unregisters the listener, aborts and awaits its captured callbacks, emits no retry decision after disposal, and leaves no timer or promise alive.
|
||||
- Pure unit tests cover transient-code selection, exponential backoff and jitter bounds, valid and over-cap `Retry-After`, exhausted budgets, deterministic timer/random seams, and abort during backoff.
|
||||
- Real agent-loop tests cover failure before chunks, partial chunks then failure, thrown and in-band failures, retry to success in a new step, exhaustion to structured `turn/end.reason`, and composition with `dsh-compact-basic` context-overflow recovery.
|
||||
- Real agent-loop tests cover failure before chunks, partial chunks then failure, thrown and in-band failures, retry to success in a new turn, exhaustion to structured `turn/end.reason`, and composition with `dsh-compact-basic` context-overflow recovery.
|
||||
- The partial-chunk integration test proves failed chunks remain attributed to the failed step, no assistant message or tool side effect is committed for that step, and the successful retry has distinct provenance.
|
||||
- The plugin-owned `llm/retry` event is non-surface, survives JSONL and SQLite round trips, is ignored by message derivation, and drives TUI retraction plus scheduled-retry rendering. Keyless snapshots cover scheduling, cancellation, success, and exhaustion; ACP automation snapshots confirm that a discarded attempt stays off the wire while the recovered reply is emitted.
|
||||
- Idle-watchdog tests prove the stable signal is rearmed only while `next()` is outstanding, disarmed during consumer think time and in `finally`, and classified separately from a total-call deadline and an earlier caller abort; adapter tests prove the signal stops the underlying request rather than merely detaching it.
|
||||
@@ -132,12 +122,12 @@ If recovery is exhausted, the final failure is stored once on `turn/end.reason`
|
||||
|
||||
## Consequences
|
||||
|
||||
- Every transient recovery attempt is visible as a closed failed turn plus `llm/retry`, and the bounded policy prevents hidden SDK retries from multiplying cost. A retry can still duplicate provider billing even when no chunk arrived; the finite attempt budget limits but cannot remove that risk.
|
||||
- Every retry attempt is visible as a closed failed turn plus `llm/retry`, and adapter-level single-attempt behavior prevents hidden SDK retries from multiplying policy decisions. A retry can still duplicate provider billing even when no chunk arrived; normal mode limits that risk, while explicit always mode accepts it until cancellation or success.
|
||||
- Provider SDKs may hide status or retry headers. Those adapters retain the stable facts they expose and otherwise use a coarse code rather than letting recovery policy parse fragile text.
|
||||
- Durable retry events expand the session protocol and UI state machine. Shipping the event and its consumer together prevents an unused telemetry vocabulary, but later schema changes still require persistence and replay work.
|
||||
- Clearing a failed step's live chunks can visibly retract output. That is preferable to presenting discarded text or partial tool JSON as committed history, and snapshots pin the transition.
|
||||
- Adapter-local idle enforcement stops stalled transports without counting consumer think time. Contract tests at each transport boundary guard against SDK drift.
|
||||
- Multiple recovery plugins add their finite budgets. Their classifiers remain disjoint here; an overlapping classifier would be registration-order policy and must be documented and tested by the plugins that introduce it.
|
||||
- Multiple normal recovery plugins add their finite budgets. Always mode delegates first and then supplies an unbounded fallback; overlapping classifiers remain registration-order policy and must be documented and tested by the plugins that introduce them.
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
+16
-26
@@ -4,6 +4,8 @@ Status: implemented
|
||||
|
||||
[English](2026-06-21-bounded-llm-request-recovery.md) | 中文
|
||||
|
||||
[按提供方配置的请求重试策略](../feature/2026-07-24-provider-retry-policies.md)在此基础上增加了确切提供方配置与显式无界 mode。本说明继续负责结构化失败事实、已关闭步骤的恢复边界、normal mode 的暂时性默认值、可见的单次尝试和持久重试状态。
|
||||
|
||||
## 问题
|
||||
|
||||
`dsh-llm` 可能在适配器分发或迭代时抛出异常，也可能以 `finish { kind: 'error' | 'aborted' }` 结束，以这两种形式报告提供方失败。最终适配器边界会标记抛出的失败，使 `dsh-agent-loop` 能将其与中间件和结果处理缺陷区分开。循环关闭失败步骤后，会把两种交付形式统一规范化为 `agent/request-error`。未被处理的失败是终态；处理失败的监听器修复策略自有状态，返回 `{ kind: 'retry' }`，并停止 waterfall 委托。[重试动作决策](../simplification/2026-07-27-request-error-retry-action.md)规定这一返回契约。
|
||||
@@ -16,7 +18,7 @@ Status: implemented
|
||||
- 重试的归属因适配器而异。手写 DeepSeek 适配器只尝试一次，pi-ai profile 则可以启用库内部的不透明重试。如果把隐藏的传输重试与 `agent/request-error` 监听器结合，尝试次数会成倍增加，中间失败也不会记入会话日志。
|
||||
- 恢复后的失败没有持久状态事实。失败的步骤和分片仍可重建，但观察者无法得知 agent（智能体）是否在有意退避、将等待多久，以及等待原因。长时间的静默等待看起来与循环停滞无异。
|
||||
|
||||
本决策的目标是从同一个显式提供方／模型请求的暂时性失败中进行有界恢复。提供方或模型故障转移、响应拼接和语义输出修复都属于其他问题，目前没有消费方。
|
||||
默认策略的目标是从同一个显式提供方／模型请求的暂时性失败中进行有界恢复。提供方或模型故障转移、响应拼接和语义输出修复都属于其他问题，目前没有消费方。
|
||||
|
||||
## 决策
|
||||
|
||||
@@ -50,31 +52,19 @@ agent loop（智能体循环）会保留 `RequestError` 作为该精确的错误
|
||||
|
||||
`@deepseek-ai/dsh-llm-retry` 是监听 `agent/request-error` 的函数插件。它不引入服务或新的循环分支；agent-loop 包仅会更改通过现有失败步骤恢复控制流携带的数据。
|
||||
|
||||
`agent/request-error` seam 只携带当前 `LlmFailure`；循环不拥有重试策略或尝试历史。每个恢复插件为自身处理的失败维护一个逐 agent 的私有计数器，并在终态 `agent/settled` 时清零。因此，暂时性失败与上下文溢出交替出现时，`dsh-llm-retry` 与 compact-basic 的预算独立消耗；最大请求数等于 1 加上已加载恢复策略的有限预算总和。
|
||||
`agent/request-error` seam 携带当前 `LlmFailure`、在连续恢复序列中授权重试轮次的不可变先前失败列表，以及实际服务注册所对应的不可变重试策略。循环只传递而不解释该策略；它拥有连续失败历史，并在模型请求成功后清除。`dsh-llm-retry` 的 normal 策略统计由同一项确切提供方策略安排的持久重试记录，`dsh-compact-basic` 则维护自己的上下文溢出预算。因此，暂时性失败与上下文溢出交替出现时，会各自独立消耗其有限预算；最大请求数等于 1 加上所有已加载有限预算之和。
|
||||
|
||||
该插件在加载时解析并验证以下部署配置：
|
||||
|
||||
```ts ignore-check
|
||||
interface Config {
|
||||
maxTransientRetries?: number
|
||||
initialDelayMs?: number
|
||||
maxDelayMs?: number
|
||||
jitterRatio?: number
|
||||
retryableCodes?: string[]
|
||||
}
|
||||
```
|
||||
|
||||
默认值为两次暂时性重试、500 毫秒初始延迟、10 秒延迟上限、10% 抖动，以及上述五个暂时性 code（`RATE_LIMIT`、`SERVER`、`TIMEOUT`、`TRANSPORT` 和 `EMPTY_RESPONSE`）。计数与延迟边界参考了所调查实现中较保守的一端：[OpenCode 使用两次请求重试，延迟边界为 500 毫秒／10 秒](https://github.com/anomalyco/opencode/blob/9976269ab1accfc9f9dc98a4a688c516934de422/%70ackages/llm/src/route/executor.ts#L36-L39)；[Pi 将三次 agent 级重试与提供方重试分开，且提供方重试默认为零](https://github.com/earendil-works/pi/blob/3da591ab74ab9ab407e72ed882600b2c851fae21/%70ackages/coding-agent/docs/settings.md#L139-L147)；[Codex 使用有限请求／流预算以及五分钟空闲超时](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/model-provider-info/src/lib.rs#L25-L33)。10% 抖动参考 [Codex 的有界抖动](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/codex-client/src/retry.rs#L40-L47)。在没有其他恢复策略时，两次重试表示最多发起三次提供方请求。`maxTransientRetries` 是非负整数，延迟是正的有限数且满足 `initialDelayMs <= maxDelayMs`，`jitterRatio` 位于 `[0, 1]`，code 非空且不重复。这些都是 Cordis 配置字段，而不是隐藏常量，使部署能够选择不同的成本与延迟预算。
|
||||
当前配置形状由[提供方策略决策](../feature/2026-07-24-provider-retry-policies.md)规定。提供方适配器会注册嵌套的 `retryPolicy`；省略时使用 normal 默认值：两次暂时性重试、500 毫秒初始延迟、10 秒延迟上限、10% 抖动，以及上述五个暂时性 code。计数与延迟边界参考了所调查实现中较保守的一端：[OpenCode 使用两次请求重试，延迟边界为 500 毫秒／10 秒](https://github.com/anomalyco/opencode/blob/9976269ab1accfc9f9dc98a4a688c516934de422/%70ackages/llm/src/route/executor.ts#L36-L39)；[Pi 将三次 agent 级重试与提供方重试分开，且提供方重试默认为零](https://github.com/earendil-works/pi/blob/3da591ab74ab9ab407e72ed882600b2c851fae21/%70ackages/coding-agent/docs/settings.md#L139-L147)；[Codex 使用有限请求／流预算以及五分钟空闲超时](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/model-provider-info/src/lib.rs#L25-L33)。10% 抖动参考 [Codex 的有界抖动](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/codex-client/src/retry.rs#L40-L47)。
|
||||
|
||||
对于预算未耗尽的合格失败，从 1 开始的暂时性重试计数使用有界指数退避。有效的 `providerRetryAfterMs` 只有在不超过 `maxDelayMs` 时才会取代指数退避；提供方延迟更长时，系统会委托给下一监听器，而不会违反提供方指令提前重试。本地退避乘以 `[1 - jitterRatio, 1 + jitterRatio]` 内的注入随机因子，并将最终值限制到 `maxDelayMs`；提供方延迟不加抖动。
|
||||
|
||||
插件拥有一个全生命期 `AbortController`，并跟踪每个活跃的退避回调。每次等待都会融合 waterfall（瀑布式事件）的轮次信号与该生命期信号。effect 清理会先注销监听器，再中止并等待活跃回调；被捕获回调的生命期信号中止时，回调会直接返回，不重试，也不进入其捕获 waterfall 的剩余部分。尽管 Cordis 已捕获该监听器，此设计仍能使 HMR（热模块替换）释放达到完全停稳。
|
||||
插件拥有一个全生命期 `AbortController`，并跟踪每个活跃的恢复回调，包括委托的 waterfall（瀑布式事件）工作与退避。effect 清理会先注销监听器，再中止并等待活跃回调；中止会胜过较晚到达的委托重试决策，被捕获的回调在插件释放后既不能重试，也不能进入其 waterfall 的剩余部分。尽管 Cordis 已捕获该监听器，此设计仍能使 HMR（热模块替换）释放达到完全停稳。
|
||||
|
||||
休眠前，`dsh-llm-retry` 会追加一条不进入表层的 `llm/retry` 会话事件，其中包含轮次、失败步骤、从 1 开始的暂时性重试编号、已配置上限、计划延迟和 `LlmFailure`。该插件拥有 `SessionEventMap` 声明合并；`dsh-session` 继续负责通用持久化，不会吸收可选策略的词汇。事件记录已安排的内容，而不是下一个请求已完成；延迟期间取消随后会在 `turn/end` 中可见。因为该事件的目的是表示运行状态，而不是收集跟踪数据，所以它仅与生产渲染器及回放／快照覆盖一起交付。
|
||||
休眠前，`dsh-llm-retry` 会追加一条不进入表层的 `llm/retry` 会话事件，其中包含轮次、失败步骤、提供方、策略 mode、完整的解析策略 key、提供方策略重试编号、该 mode 存在时的有限上限、计划延迟和 `LlmFailure`。该 key 会对 code 集排序，并在提供方路由被行为不同但 mode 相同的策略替换时分隔重试历史。该插件拥有 `SessionEventMap` 声明合并；`dsh-session` 继续负责通用持久化，不会吸收可选策略的词汇。事件记录已安排的内容，而不是下一个请求已完成；延迟期间取消随后会在 `turn/end` 中可见。因为该事件的目的是表示运行状态，而不是收集跟踪数据，所以它仅与生产渲染器及回放／快照覆盖一起交付。
|
||||
|
||||
对非暂时性 code、耗尽的策略预算或超出上限的提供方延迟，监听器会调用 `next()`。这保留了与上下文溢出恢复及后续策略插件的组合能力。对自身处理的失败，它会记录并等待延迟，然后在不委托的情况下返回 `{ kind: 'retry' }`。轮次取消和插件释放会结束等待且不返回重试动作，此后仍以循环的取消／释放检查为准。
|
||||
|
||||
agent-spine 演示组合包加载该插件，因此共享的 stdio/TUI、一次性 CLI（命令行界面）和 ACP（Agent Client Protocol）示例组合使用同一有界策略。库消费方仍需显式组合插件：省略该插件时，请求失败保持终态。
|
||||
agent-spine 演示组合包加载该插件，因此共享的 stdio/TUI、一次性 CLI（命令行界面）和 ACP（Agent Client Protocol）示例组合使用同一套按提供方路由的策略。库消费方仍需显式组合插件：省略该插件时，请求失败保持终态。
|
||||
|
||||
### 由单一层负责可见的尝试
|
||||
|
||||
@@ -101,7 +91,7 @@ agent-spine 演示组合包加载该插件，因此共享的 stdio/TUI、一次
|
||||
- 自动提供方或模型故障转移。请求已显式选择一个提供方和模型，提供方注册表也有意规定每个提供方只由一个适配器负责。
|
||||
- 在成功的终止性 finish 后重试或继续，或将两次尝试的分片拼接成一条 assistant 消息。
|
||||
- 修复格式错误的工具参数、拒答、内容过滤或其他语义模型输出。
|
||||
- 无界重试、无人值守地持续重试直至取消、熔断器、共享提供方健康状态或跨 agent 重试预算。
|
||||
- 熔断器、共享提供方健康状态或跨 agent 重试预算。
|
||||
- 在没有生产消费方的情况下，把 `llm/stream` 改造成响应生命周期或增加便利的生成 API。
|
||||
|
||||
## 考虑过的替代方案
|
||||
@@ -110,7 +100,7 @@ agent-spine 演示组合包加载该插件，因此共享的 stdio/TUI、一次
|
||||
- **向 `dsh-llm` 增加响应开始、中断、丢弃、失败和提交事件**：拒绝采用，因为 agent 日志已经分隔原始分片、成功消息和编号尝试。第二套状态机会重复归属关系，又不能支持有界的同路由重试。
|
||||
- **增加逻辑路由、能力矩阵和故障转移选择**：拒绝采用，因为当前请求已经显式指定提供方和模型，每个提供方由一个适配器负责，而且没有当前消费方要求自动回退或能够证明语义兼容性。
|
||||
- **把 `retryable` 或 `failover` 放在 `LlmFailure` 上**：拒绝采用，因为适配器报告事实，部署策略决定动作。同一个 429 可以在交互式组合包中重试，也可以在成本受限的批处理中被拒绝。
|
||||
- **只要调用方仍处于活跃状态就无限重试**：拒绝采用，因为这会让一次请求产生无界成本和延迟。可见状态能使有界等待易于理解，却不能让无限预算变得安全。
|
||||
- **只要调用方仍处于活跃状态就无限重试**：[按提供方配置的策略](../feature/2026-07-24-provider-retry-policies.md)对显式 `always` 配置项推翻了这项拒绝，同时保留有界的 normal mode 作为默认值。
|
||||
- **只通过进程 logger 记录重试状态**：拒绝采用，因为进程日志无法重建会话行为，也不能驱动回放后的 UI 状态。
|
||||
- **只保留扁平 code**：拒绝采用，因为重试延迟和提供方请求 id 是结构化的提供方事实，而当不同协议失败共用一个稳定 code 时，诊断还需要 HTTP 状态。
|
||||
|
||||
@@ -120,11 +110,11 @@ agent-spine 演示组合包加载该插件，因此共享的 stdio/TUI、一次
|
||||
- 适配器抛出的 `Error` 会以完全相同的对象抵达 `agent/request-error`，其伴随的 `LlmFailure` 则抵达相邻参数；测试保留针对可扩展及冻结第三方错误的现有对象标识断言。
|
||||
- DeepSeek 和 pi-ai 适配器测试覆盖具有代表性的 400、401/403、429、5xx、连接、格式错误／截断流、超时、中止、Retry-After 秒数／日期、请求 id 和未知 SDK 错误路径，恢复策略无需解析消息文本。
|
||||
- Pi 将 SDK 选项固定为零次重试，并针对可重试的提供方响应执行一次可观测的实际网络请求；独立测试确保移除任一边界都会失败。
|
||||
- `agent/request-error` 只携带当前失败事实；每个插件在终态空闲时清零其逐 agent 私有计数器，暂时性失败／上下文溢出交替发生的集成测试证明两种策略只消耗各自的有限预算。
|
||||
- `dsh-llm-retry` 在 Loader 启动时验证每个配置字段，使用 `next()` 委托所有不合格路径，而且在没有其他策略时最多发起 `maxTransientRetries + 1` 次提供方请求。
|
||||
- 退避期间执行 HMR 的测试证明：释放过程会注销监听器、中止并等待其捕获的回调，释放后不发起重试请求，也不留下存活的定时器或 promise。
|
||||
- `agent/request-error` 携带当前失败事实、不可变的先前已重试失败事实，以及实际服务注册所对应的不可变重试策略；成功会清除历史，暂时性失败／上下文溢出交替发生的集成测试证明两种策略只消耗各自的有限预算。
|
||||
- 每个提供方适配器都在 Loader 启动时验证其嵌套重试策略，`ctx.llm` 则将该策略与路由一同捕获；normal mode 会委托不合格路径，而且在没有其他策略时最多发起 `maxRetries + 1` 次提供方请求。
|
||||
- 退避期间执行 HMR 的测试证明：释放过程会注销监听器、中止并等待其捕获的回调，释放后不发出重试决策，也不留下存活的定时器或 promise。
|
||||
- 纯单元测试覆盖暂时性 code 选择、指数退避和抖动边界、有效及超出上限的 `Retry-After`、耗尽的预算、确定性定时器／随机数 seam，以及退避期间中止。
|
||||
- 真实 agent-loop 测试覆盖分片前失败、部分分片后失败、抛出及带内失败、在新步骤中重试至成功、耗尽后写入结构化 `turn/end.reason`，以及与 `dsh-compact-basic` 上下文溢出恢复的组合。
|
||||
- 真实 agent-loop 测试覆盖分片前失败、部分分片后失败、抛出及带内失败、在新轮次中重试至成功、耗尽后写入结构化 `turn/end.reason`，以及与 `dsh-compact-basic` 上下文溢出恢复的组合。
|
||||
- 部分分片集成测试证明：失败分片仍归属于失败步骤，该步骤不会提交 assistant 消息或工具副作用，成功的重试具有不同的来源信息。
|
||||
- 插件拥有的不进入表层的 `llm/retry` 事件可在 JSONL 和 SQLite 往返后保留，被消息派生忽略，并驱动 TUI 撤回和计划重试渲染。无密钥快照覆盖调度、取消、成功和耗尽；ACP 自动化快照确认，被丢弃的尝试不会通过协议发出，而恢复后的回复会正常发出。
|
||||
- 空闲看门狗测试证明：只有 `next()` 尚未完成时才会重新布防稳定信号；在消费方思考期间及 `finally` 中会解除布防；它与总调用 deadline 以及更早发生的调用方中止分开分类。适配器测试证明该信号会终止底层请求，而不只是与其脱离。
|
||||
@@ -132,12 +122,12 @@ agent-spine 演示组合包加载该插件，因此共享的 stdio/TUI、一次
|
||||
|
||||
## 后果
|
||||
|
||||
- 每次暂时性恢复尝试都以一个已关闭失败轮次加 `llm/retry` 的形式可见，有界策略还会防止隐藏的 SDK 重试成倍增加成本。即使没有分片到达，重试仍可能造成提供方重复计费；有限的尝试预算只能限制而无法消除此风险。
|
||||
- 每次重试尝试都以一个已关闭失败轮次加 `llm/retry` 的形式可见，适配器级的单次尝试行为会防止隐藏的 SDK 重试成倍增加策略决策。即使没有分片到达，重试仍可能造成提供方重复计费；normal mode 会限制此风险，而显式 always mode 会接受它，直至取消或成功。
|
||||
- 提供方 SDK 可能隐藏状态或重试标头。适配器会保留 SDK 公开的稳定事实，否则使用粗粒度 code，而不会让恢复策略解析脆弱的文本。
|
||||
- 持久重试事件扩展了会话协议和 UI 状态机。事件与其消费方一同交付，可避免产生无人使用的遥测词汇；但以后更改 schema 仍需要同步完成持久化和回放工作。
|
||||
- 清除失败步骤的实时分片可能会明显撤回输出。与把丢弃的文本或不完整工具 JSON 呈现为已提交历史相比，这是更好的选择；快照固定这一转换。
|
||||
- 适配器局部的空闲强制机制可以终止停滞的传输，而不会计入消费方思考时间。每个传输边界的契约测试会防止 SDK 漂移。
|
||||
- 多个恢复插件会叠加各自的有限预算。此处它们的分类器互不重叠；重叠的分类器会形成依赖注册顺序的策略，必须由引入它们的插件记录并测试。
|
||||
- 多个 normal 恢复插件会叠加各自的有限预算。always mode 会先委托，再提供无界回退；重叠的分类器仍会形成依赖注册顺序的策略，必须由引入它们的插件记录并测试。
|
||||
|
||||
## 相关资料
|
||||
|
||||
|
||||
+2
-2
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-14-provider-routed-llm-adapters.md: 98205d18d07752e0cdba86d7cba80368d45fd816
|
||||
2026-07-14-provider-routed-llm-adapters.zh.md: c35225a86baf4c2d09732b5940abbc8046d365fb
|
||||
2026-07-14-provider-routed-llm-adapters.md: 1bd9197667f6e49c5025c98b4a77500f78595c2b
|
||||
2026-07-14-provider-routed-llm-adapters.zh.md: 4d57f2cb33ac296500a4a19771ea493621ff93f6
|
||||
@@ -28,7 +28,7 @@ A provider has exactly one adapter owner in a Cordis context. `dsh-llm-deepseek`
|
||||
|
||||
### Explicit pi-ai provider profiles
|
||||
|
||||
`dsh-llm-pi-ai` takes one non-empty list of provider profiles. Provider names must be unique within the list and present in pi-ai's `getProviders()` result. Each profile contains the provider name plus optional `apiKey`, `baseURL`, headers, reasoning level and budgets, cache retention, transport, SDK timeouts, and a Harness stream-idle timeout. Provider retry fields are deliberately absent: the adapter forces pi-ai's `maxRetries` to zero so one `stream()` call makes one visible provider attempt, while `dsh-llm-retry` owns bounded agent-level recovery. Credentials are never global: an explicit key applies only to its profile, while an absent key lets pi-ai resolve its standard environment variable, OAuth token, AWS credential chain, Google ADC, or other provider-native ambient authentication. An explicitly empty key is invalid configuration rather than an environment fallback.
|
||||
`dsh-llm-pi-ai` takes one non-empty list of provider profiles. Provider names must be unique within the list and present in pi-ai's `getProviders()` result. Each profile contains the provider name plus optional `apiKey`, `baseURL`, headers, reasoning level and budgets, cache retention, transport, SDK timeouts, a Harness stream-idle timeout, and a provider-owned `retryPolicy`. The adapter forces pi-ai's `maxRetries` to zero so one `stream()` call makes one visible provider attempt, while `dsh-llm-retry` executes the resolved policy at the agent failed-step seam. Credentials are never global: an explicit key applies only to its profile, while an absent key lets pi-ai resolve its standard environment variable, OAuth token, AWS credential chain, Google ADC, or other provider-native ambient authentication. An explicitly empty key is invalid configuration rather than an environment fallback.
|
||||
|
||||
The plugin registers all configured provider names against one `PiAiAdapter` in one all-or-nothing call. A request uses its provider to select the matching profile and finds its model in `getModels(provider)` to obtain the catalog descriptor. An unknown provider fails at plugin load; an unknown model fails before network I/O with `UNKNOWN_MODEL`. The catalog object is never mutated. When a profile supplies `baseURL`, the adapter clones the selected descriptor and overrides only `baseUrl`, so a private endpoint can retain pi-ai's API, capabilities, compatibility flags, context limits, and reasoning map. The private endpoint must implement the selected provider's protocol, and the model id must still exist in the installed pi-ai catalog.
|
||||
|
||||
|
||||
+1
-1
@@ -28,7 +28,7 @@ Status: implemented
|
||||
|
||||
### 显式 pi-ai 提供方配置
|
||||
|
||||
`dsh-llm-pi-ai` 接受一个非空的提供方配置列表。列表内的提供方名称必须唯一，并且存在于 pi-ai 的 `getProviders()` 结果中。每项配置包含提供方名称，以及可选的 `apiKey`、`baseURL`、headers、推理级别和预算、缓存保留设置、传输方式、SDK 超时和 Harness 流空闲超时。配置中有意不提供重试字段：适配器强制将 pi-ai 的 `maxRetries` 设为零，使一次 `stream()` 调用只发起一次可见的提供方请求；有界的 agent 层恢复由 `dsh-llm-retry` 负责。凭据不设全局值：显式密钥仅对所属配置生效；未提供密钥时，pi-ai 使用标准环境变量、OAuth token、AWS 凭据链、Google ADC 或其他提供方原生环境认证。显式空密钥属于无效配置，不会回退到环境认证。
|
||||
`dsh-llm-pi-ai` 接受一个非空的提供方配置列表。列表内的提供方名称必须唯一，并且存在于 pi-ai 的 `getProviders()` 结果中。每项配置包含提供方名称，以及可选的 `apiKey`、`baseURL`、headers、推理级别和预算、缓存保留设置、传输方式、SDK 超时、Harness 流空闲超时，以及由提供方拥有的 `retryPolicy`。适配器强制将 pi-ai 的 `maxRetries` 设为零，使一次 `stream()` 调用只发起一次可见的提供方请求；`dsh-llm-retry` 则在 agent 失败步骤 seam 上执行解析后的策略。凭据不设全局值：显式密钥仅对所属配置生效；未提供密钥时，pi-ai 使用标准环境变量、OAuth token、AWS 凭据链、Google ADC 或其他提供方原生环境认证。显式空密钥属于无效配置，不会回退到环境认证。
|
||||
|
||||
插件通过一次全有或全无调用，将所有已配置的提供方名称注册到同一个 `PiAiAdapter`。请求按 provider 选择对应配置，并在 `getModels(provider)` 中查找模型以取得目录描述符。未知提供方会在插件加载时失败；未知模型会在网络 I/O 前以 `UNKNOWN_MODEL` 失败。适配器不会修改目录对象。当配置提供 `baseURL` 时，适配器复制选中的描述符，仅覆盖 `baseUrl`，使私有端点保留 pi-ai 的 API、能力、兼容标志、上下文限制与推理映射。私有端点必须实现所选提供方的协议，模型 ID 也仍须存在于已安装的 pi-ai 目录中。
|
||||
|
||||
|
||||
+2
-2
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-24-empty-model-response-is-retryable.md: f4a6373178efd5ca1ba5882fb2aaf97dffb2526b
|
||||
2026-07-24-empty-model-response-is-retryable.zh.md: 4c3afe44140c029d274f34ade97803b958c6d669
|
||||
2026-07-24-empty-model-response-is-retryable.md: 3ecb106fc3a53070f66d1de351120aaf99d4d0de
|
||||
2026-07-24-empty-model-response-is-retryable.zh.md: 91ce4105ebe60b71f12667ccf28ea905566353d5
|
||||
@@ -15,7 +15,7 @@ An adapter classifies a completed empty response as a provider-boundary failure,
|
||||
- `dsh-llm` exports the canonical code `EMPTY_RESPONSE_CODE` (`'EMPTY_RESPONSE'`) beside `CONTEXT_WINDOW_EXCEEDED_CODE`/`QUOTA_EXCEEDED_CODE`.
|
||||
- `dsh-llm-pi-ai` (`mapStopReason`): a terminal `stop` whose assistant message has no content blocks becomes a `finish {kind: 'error'}` with that code. Context-overflow detection still wins where it applies (it is checked first and is the more actionable classification).
|
||||
- `dsh-llm-deepseek` (`translate`): at `[DONE]`, a `stop` (or absent) finish with no opened blocks becomes the same error finish. Reasoning-only streams count as content and stay successful.
|
||||
- `dsh-llm-retry` adds `EMPTY_RESPONSE` to `DEFAULT_RETRYABLE_CODES`: the attempt produced nothing durable, so repeating it is safe; deployments can still remove it via `retryableCodes`.
|
||||
- The provider-owned normal retry default includes `EMPTY_RESPONSE`: the attempt produced nothing durable, so repeating it is safe; deployments can still remove it via `retryableCodes`, and `dsh-llm-retry` executes the resolved policy.
|
||||
|
||||
Detection is scoped to `stop` finishes only. `max-tokens` with empty content keeps its existing meaning (pi-ai already normalizes the zero-output overflow case), `tool-calls` cannot be block-empty in practice, and error/aborted finishes already fail.
|
||||
|
||||
|
||||
+1
-1
@@ -15,7 +15,7 @@ Status: implemented
|
||||
- `dsh-llm` 在 `CONTEXT_WINDOW_EXCEEDED_CODE`/`QUOTA_EXCEEDED_CODE` 之外，导出规范代码 `EMPTY_RESPONSE_CODE`（`'EMPTY_RESPONSE'`）。
|
||||
- `dsh-llm-pi-ai`（`mapStopReason`）：当终止性 `stop` 所对应的 assistant 消息没有内容块时，它会变成一个携带该代码的 `finish {kind: 'error'}`。上下文溢出检测在其适用场景中仍然优先（它先被检查，也是更具可操作性的归类）。
|
||||
- `dsh-llm-deepseek`（`translate`）：在 `[DONE]` 处，若 `stop`（或缺失）结束且没有打开过任何块，则同样变成该错误结束。仅含 reasoning 的流算作有内容，仍视为成功。
|
||||
- `dsh-llm-retry` 把 `EMPTY_RESPONSE` 加入 `DEFAULT_RETRYABLE_CODES`：这次尝试没有产生任何持久内容，因此重复它是安全的；部署方仍可通过 `retryableCodes` 将其移除。
|
||||
- 提供方拥有的 normal 重试默认策略包含 `EMPTY_RESPONSE`：这次尝试没有产生任何持久内容，因此重复它是安全的；部署方仍可通过 `retryableCodes` 将其移除，而 `dsh-llm-retry` 会执行解析后的策略。
|
||||
|
||||
检测仅限于 `stop` 结束。内容为空的 `max-tokens` 保持其既有含义（pi-ai 已经把零输出的溢出场景归一化处理），`tool-calls` 在实践中不可能是空块，而 error／aborted 结束本身已经算失败。
|
||||
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-24-provider-retry-policies.md: 1831ce6b96178d11e7c9927ceccbe07ea578cd2c
|
||||
2026-07-24-provider-retry-policies.zh.md: 788f1f1963861e1b46ff0d8798e53e01bd9892b4
|
||||
@@ -0,0 +1,65 @@
|
||||
# Agent Note: Per-provider request retry policies
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-07-24-provider-retry-policies.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
One process may route model requests to providers with different reliability and cost constraints. A single transient classifier and finite retry budget cannot express a deployment that wants bounded recovery for most providers but requires one provider to keep retrying every model-request failure until the request succeeds or the caller cancels it.
|
||||
|
||||
Provider policy must follow the request that actually failed, including a route selected by `agent/request`, rather than the agent's initial options. Unbounded policy also cannot store JavaScript `Infinity` in the durable session event, and neither provider error text nor discarded partial output may enter the next model request.
|
||||
|
||||
## Decision
|
||||
|
||||
Each concrete adapter accepts an optional `retryPolicy` inside its provider configuration. The adapter validates and resolves the policy, and `ctx.llm` captures it when that exact provider route registers. When a call enters its final adapter boundary, `ctx.llm` binds the serving registration's immutable policy to that call; the agent loop passes it to closed-step recovery even if the route is disposed or replaced while the request is in flight. `@deepseek-ai/dsh-llm-retry` combines that call-local policy with the failed step's durable provider identity. A call that never reaches a final adapter has no serving policy and delegates. A provider without `retryPolicy` uses the normal defaults.
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- provider: deepseek
|
||||
retryPolicy:
|
||||
mode: normal
|
||||
maxRetries: 2
|
||||
retryableCodes: [EMPTY_RESPONSE, RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT]
|
||||
backoff:
|
||||
initialDelayMs: 500
|
||||
maxDelayMs: 10000
|
||||
jitterRatio: 0.1
|
||||
- provider: internal
|
||||
retryPolicy:
|
||||
mode: always
|
||||
backoff:
|
||||
initialDelayMs: 1000
|
||||
maxDelayMs: 30000
|
||||
jitterRatio: 0.2
|
||||
```
|
||||
|
||||
The listener reads the provider from the durable `request/header` in force when the failed step closed, excluding later recovery mutations, but never re-resolves policy from the mutable provider registry. It derives a canonical key from every field of the resolved serving policy, sorting `retryableCodes` because eligibility uses set membership, and continues retry history only for the same provider and key. Replacing a route with different limits, code membership, or backoff therefore starts a new count and initial delay even when the mode is unchanged. Normal mode retains the bounded transient behavior: it retries configured codes up to `maxRetries` and otherwise delegates.
|
||||
|
||||
Always mode asks downstream recovery first so a specialized policy such as context-overflow compaction can make progress. A downstream retry wins. A downstream failure decision or thrown recovery error falls back to an unbounded retry of the same provider request; the thrown error is logged. The retry listener owns and drains delegated recovery before cancellation or plugin disposal can finish, then applies the abort instead of a late downstream decision. Success, turn cancellation, and plugin disposal are the only termination paths.
|
||||
|
||||
Both modes use exponential local delays from `initialDelayMs` to `maxDelayMs`. `jitterRatio` multiplies each target by a uniform sample in `[1 - jitterRatio, 1 + jitterRatio]`, then applies the cap. A positive provider `Retry-After` within the cap remains exact and unjittered. An over-cap provider delay makes normal mode delegate; always mode retains its guarantee by using the configured local backoff.
|
||||
|
||||
Each scheduled retry appends a non-surface `llm/retry` event with the failed provider, policy mode, canonical resolved-policy key, provider-policy retry number, delay, and failure facts. Normal events carry finite `maxRetries`; always events omit it, and UIs render the limit as `∞`. The event and failed `assistant/chunk` records do not contribute surface messages, so the next request contains the same derived context as the failed request unless another recovery policy deliberately changes the surface.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**One global `always` switch** — rejected because it cannot isolate the unbounded cost and latency risk to the provider that needs it and can silently apply after runtime rerouting.
|
||||
|
||||
**A separate exact-provider list on `dsh-llm-retry`** — rejected because it duplicates provider route names outside their owning adapter configuration and lets provider registration drift from recovery policy.
|
||||
|
||||
**A very large finite retry count** — rejected because it eventually violates the requested keep-retrying contract and serializes an arbitrary operational limit as if it were meaningful.
|
||||
|
||||
**Provider-SDK retries** — rejected because hidden attempts multiply agent-level budgets, cannot use the closed-step durability boundary, and may splice or discard streamed output without a reconstructable retry record.
|
||||
|
||||
**Put the error into model context** — rejected because a transport or provider diagnostic is operational state, not conversation content. It can expose sensitive provider details and changes the retried request instead of repeating the failed request.
|
||||
|
||||
## Verification
|
||||
|
||||
Adapter tests validate nested policies at provider load, prove registration captures configured and default policies, and retain the serving policy across in-flight route replacement. Unit tests select policies from the failed request's serving registration, separate provider and changed-policy histories, exercise always mode beyond the normal budget, pin jitter and delay caps, prove downstream recovery ordering, prove cancellation and disposal drain delegated recovery before reaching quiescence, and prove both abort active backoff waits. Request-level coverage compares the complete messages of failed and retried attempts and rejects both provider error text and discarded partial output. A keyless headless `stream-json` snapshot runs failure, retry, and success through the assembled app, pins the complete `llm/retry` record, and rejects any model-message change between attempts. JSONL and SQLite tests round-trip an always event without `Infinity`; invariant tests bind provider identity to the request header, validate failure and mode-specific timer bounds, and bind retry numbers to provider-policy keys; TUI tests render finite and infinite limits.
|
||||
|
||||
## Consequences
|
||||
|
||||
Normal mode remains a finite default, while an explicit always policy can spend unbounded requests and time on permanent authentication, quota, invalid-request, protocol, or context failures. Operators must pair always mode with a cancellable caller and provider-specific cost controls. Retry state stays observable and durable without becoming model-visible, and serving-registration capture prevents adapter lifecycle changes from retroactively changing an in-flight request's recovery contract.
|
||||
|
||||
This decision extends the closed-step recovery, single visible adapter attempt, structured failure, and durable status design in [bounded recovery for transient LLM request failures](../architecture/2026-06-21-bounded-llm-request-recovery.md).
|
||||
@@ -0,0 +1,65 @@
|
||||
# Agent Note: 逐提供方请求重试策略
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-07-24-provider-retry-policies.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
同一进程可能把模型请求路由到可靠性和成本约束各不相同的提供方。单一的瞬态错误分类器与有限重试预算无法表达这种部署需求：大多数提供方只需有界恢复，但其中一个提供方必须持续重试每次模型请求失败，直到请求成功或调用方取消。
|
||||
|
||||
提供方策略必须跟随实际失败的请求，包括 `agent/request` 选择的路由，而不能跟随 agent（智能体）的初始选项。无界策略也不能把 JavaScript `Infinity` 存入持久会话事件；提供方错误文本与丢弃的部分输出都不得进入下一次模型请求。
|
||||
|
||||
## 决策
|
||||
|
||||
每个具体适配器都在其提供方配置中接受可选的 `retryPolicy`。适配器负责校验并解析策略，`ctx.llm` 则在该精确提供方路由注册时捕获策略。当调用进入最终适配器边界时，`ctx.llm` 会把实际提供服务的注册项所持不可变策略绑定到该调用；即使路由在请求进行期间被 dispose 或替换，agent loop 仍会把该策略传给已关闭步骤恢复。`@deepseek-ai/dsh-llm-retry` 会把绑定到该调用的策略与失败步骤的持久提供方标识结合起来。未到达最终适配器的调用没有实际提供服务的策略，因而会委托后续处理。未配置 `retryPolicy` 的提供方使用 normal 默认值。
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- provider: deepseek
|
||||
retryPolicy:
|
||||
mode: normal
|
||||
maxRetries: 2
|
||||
retryableCodes: [EMPTY_RESPONSE, RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT]
|
||||
backoff:
|
||||
initialDelayMs: 500
|
||||
maxDelayMs: 10000
|
||||
jitterRatio: 0.1
|
||||
- provider: internal
|
||||
retryPolicy:
|
||||
mode: always
|
||||
backoff:
|
||||
initialDelayMs: 1000
|
||||
maxDelayMs: 30000
|
||||
jitterRatio: 0.2
|
||||
```
|
||||
|
||||
监听器从失败步骤关闭时生效的持久 `request/header` 读取提供方，后续恢复产生的改动不参与选择，但绝不会从可变的提供方注册表重新解析策略。它会根据已解析实际服务策略的所有字段生成规范键；由于错误资格按集合成员判断，生成时会对 `retryableCodes` 排序。重试历史只会对同一提供方和同一规范键延续。因此，即使模式未变，只要路由替换后的次数上限、错误代码成员或退避不同，重试计数与初始延迟都会重新开始。normal 模式保留有界瞬态错误处理行为：它重试配置的错误代码，次数不超过 `maxRetries`；其他情况委托后续处理。
|
||||
|
||||
always 模式先请求下游恢复，使上下文溢出压缩（compaction）之类的专用策略有机会取得进展。下游若决定重试，则以该决定为准。下游若决定失败或恢复过程抛出错误，则回退为无界重试同一提供方请求；抛出的错误会写入日志。重试监听器会持有并排空已委托的恢复，轮次取消或插件 dispose（资源释放）只能在其结束后完成；随后监听器会应用取消，而不会采用迟到的下游决定。成功、轮次取消和插件 dispose 是仅有的终止路径。
|
||||
|
||||
两种模式的本地延迟都按指数增长，从 `initialDelayMs` 增至 `maxDelayMs`。`jitterRatio` 用 `[1 - jitterRatio, 1 + jitterRatio]` 区间内的均匀随机样本乘以每次目标值，再应用上限。提供方给出的正数 `Retry-After` 若未超过上限，则保持精确且不加抖动。若提供方延迟超过上限，normal 模式会委托后续处理；always 模式则改用配置的本地退避，以维持无限重试保证。
|
||||
|
||||
每次安排重试都会追加一条不进入表层的 `llm/retry` 事件，其中包含失败的提供方、策略模式、已解析策略的规范键、提供方策略内的重试编号、延迟和失败事实。normal 事件包含有限的 `maxRetries`；always 事件省略该字段，UI 将上限渲染为 `∞`。该事件与失败的 `assistant/chunk` 记录都不会生成表层消息，因此除非其他恢复策略有意改变表层，否则下一次请求包含的派生上下文与失败请求相同。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
**单一全局 `always` 开关**：不予采纳，因为它无法把无界成本与延迟风险限制在确有需要的提供方，还可能在运行时重新路由后悄然生效。
|
||||
|
||||
**在 `dsh-llm-retry` 上维护单独的精确提供方列表**：不予采纳，因为它会在所属适配器配置之外重复提供方路由名称，并让提供方注册与恢复策略发生偏差。
|
||||
|
||||
**设置很大的有限重试次数**：不予采纳，因为它最终仍会违反持续重试的契约，并把任意选取的运维上限序列化成看似有意义的数值。
|
||||
|
||||
**使用提供方 SDK 重试**：不予采纳，因为隐藏尝试会叠加 agent 层预算，无法利用已关闭 step 的持久性边界，还可能在没有可重建重试记录的情况下拼接或丢弃流式输出。
|
||||
|
||||
**把错误放入模型上下文**：不予采纳，因为传输或提供方诊断信息属于运维状态，而非对话内容。它可能暴露敏感的提供方细节，并会改变重试请求，无法重复原本失败的请求。
|
||||
|
||||
## 验证
|
||||
|
||||
适配器测试会在提供方加载时校验嵌套策略，证明注册流程会捕获已配置策略和默认策略，并证明请求进行期间替换路由后仍会保留实际提供服务的策略。单元测试根据失败请求实际使用的注册项选择策略、分离不同提供方和策略变更后的重试历史、验证 always 模式可越过 normal 预算、固定抖动和延迟上限、证明下游恢复顺序、证明取消与 dispose 会先排空已委托的恢复再达到完全停稳，并证明二者都会停止正在进行的退避等待。请求级覆盖会比较失败尝试与重试尝试的完整消息，并排除提供方错误文本和丢弃的部分输出。一个无密钥 headless `stream-json` 快照会通过组装后的应用执行失败、重试与成功流程，固定完整的 `llm/retry` 记录，并拒绝各次尝试之间出现任何模型消息变化。JSONL 与 SQLite 测试会往返读写不含 `Infinity` 的 always 事件；不变式测试会将提供方标识绑定到请求头、校验失败事实和各模式的计时器边界，并将重试编号绑定到提供方策略键；TUI 测试会渲染有限和无限上限。
|
||||
|
||||
## 后果
|
||||
|
||||
normal 模式仍是有限的默认策略；显式的 always 策略可能在永久性的身份验证、配额、无效请求、协议或上下文错误上耗费无限次请求和无限时间。运维方必须为 always 模式配备可取消的调用方和针对提供方的成本控制。重试状态保持可观察且持久，但不会对模型可见；捕获实际提供服务的注册项，也能防止适配器生命周期变化反过来改变进行中请求的恢复契约。
|
||||
|
||||
本决策扩展了[瞬态 LLM（大语言模型）请求失败的有界恢复](../architecture/2026-06-21-bounded-llm-request-recovery.md)中确定的已关闭 step 恢复、单次可见适配器尝试、结构化失败与持久状态设计。
|
||||
Reference in New Issue
Block a user