fix(llm): isolate retry policy histories

This commit is contained in:
Turtle
2026-07-25 15:38:58 +08:00
parent 38ce422f71
commit efc725b7e4
31 changed files with 666 additions and 28 deletions
@@ -56,9 +56,9 @@ The [provider-policy decision](../feature/2026-07-24-provider-retry-policies.md)
For an eligible failure with budget remaining, the one-based transient retry count uses bounded exponential backoff. A valid `providerRetryAfterMs` replaces exponential backoff only when it does not exceed `maxDelayMs`; a longer provider delay causes delegation instead of an earlier retry that violates the provider instruction. Local backoff multiplies by an injected random factor in `[1 - jitterRatio, 1 + jitterRatio]` and clamps the final value to `maxDelayMs`; provider delay is not jittered.
The plugin owns a lifetime `AbortController` and tracks every active backoff callback. Each wait fuses the waterfall's turn signal with that lifetime signal. Effect cleanup first unregisters the listener, then aborts and awaits the active callbacks; a captured callback whose lifetime signal aborts returns `fail` and can neither retry nor enter the rest of its captured waterfall after disposal. This makes HMR disposal quiescent even though Cordis has already captured the listener.
The plugin owns a lifetime `AbortController` and tracks every active recovery callback, including delegated waterfall work and backoff. Effect cleanup first unregisters the listener, then aborts and awaits the active callbacks; abort wins over a late delegated retry decision, and a captured callback can neither retry nor enter the rest of its waterfall after disposal. This makes HMR disposal quiescent even though Cordis has already captured the listener.
Before sleeping, `dsh-llm-retry` appends one non-surface `llm/retry` session event containing the turn, failed step, provider, policy mode, provider-policy retry number, mode-specific finite maximum when present, scheduled delay, and `LlmFailure`. The plugin owns the `SessionEventMap` augmentation; `dsh-session` remains generic persistence and does not absorb the optional policy's vocabulary. The event says what was scheduled, not that the next request completed; cancellation during the delay is subsequently visible on `turn/end`. The event ships only with a production renderer and replay/snapshot coverage, because its purpose is operational state rather than trace collection.
Before sleeping, `dsh-llm-retry` appends one non-surface `llm/retry` session event containing the turn, failed step, provider, policy mode, complete resolved-policy key, provider-policy retry number, mode-specific finite maximum when present, scheduled delay, and `LlmFailure`. The key sorts the code set and separates retry histories when a provider route is replaced by a behaviorally different same-mode policy. The plugin owns the `SessionEventMap` augmentation; `dsh-session` remains generic persistence and does not absorb the optional policy's vocabulary. The event says what was scheduled, not that the next request completed; cancellation during the delay is subsequently visible on `turn/end`. The event ships only with a production renderer and replay/snapshot coverage, because its purpose is operational state rather than trace collection.
The listener calls `next()` for a non-transient code, an exhausted policy budget, or an over-cap provider delay. This preserves composition with context-overflow recovery and later policy plugins. It returns `{ action: 'retry' }` only after the delay completes under both signals; turn cancellation and plugin disposal return `fail`, after which the loop's cancellation/disposal checks remain authoritative.
@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
2026-07-24-provider-retry-policies.md: 3799dbee9658c883f4084ac9c792d4552cc8e2fb
2026-07-24-provider-retry-policies.zh.md: 8024ea707343686d2d1b1351cd7c81ecb8767b85
2026-07-24-provider-retry-policies.md: 06d21d10d31cc277e998ff6006bb01f10fe0b6b9
2026-07-24-provider-retry-policies.zh.md: d8351bd94d444b116483c81dc70bf0f6768dc37d
@@ -34,13 +34,13 @@ providers:
jitterRatio: 0.2
```
The listener reads the provider from the durable `request/header` in force when the failed step closed, excluding later recovery mutations, but never re-resolves policy from the mutable provider registry. Normal mode retains the bounded transient behavior: it retries configured codes up to `maxRetries`, counts retries scheduled by the same provider policy in the current consecutive failure sequence, and otherwise delegates.
The listener reads the provider from the durable `request/header` in force when the failed step closed, excluding later recovery mutations, but never re-resolves policy from the mutable provider registry. It derives a canonical key from every field of the resolved serving policy, sorting `retryableCodes` because eligibility uses set membership, and continues retry history only for the same provider and key. Replacing a route with different limits, code membership, or backoff therefore starts a new count and initial delay even when the mode is unchanged. Normal mode retains the bounded transient behavior: it retries configured codes up to `maxRetries` and otherwise delegates.
Always mode asks downstream recovery first so a specialized policy such as context-overflow compaction can make progress. A downstream retry wins. A downstream failure decision or thrown recovery error falls back to an unbounded retry of the same provider request; the thrown error is logged. The retry listener owns and drains delegated recovery before cancellation or plugin disposal can finish, then applies the abort instead of a late downstream decision. Success, turn cancellation, and plugin disposal are the only termination paths.
Both modes use exponential local delays from `initialDelayMs` to `maxDelayMs`. `jitterRatio` multiplies each target by a uniform sample in `[1 - jitterRatio, 1 + jitterRatio]`, then applies the cap. A positive provider `Retry-After` within the cap remains exact and unjittered. An over-cap provider delay makes normal mode delegate; always mode retains its guarantee by using the configured local backoff.
Each scheduled retry appends a non-surface `llm/retry` event with the failed provider, policy mode, provider-policy retry number, delay, and failure facts. Normal events carry finite `maxRetries`; always events omit it, and UIs render the limit as `∞`. The event and failed `assistant/chunk` records do not contribute surface messages, so the next request contains the same derived context as the failed request unless another recovery policy deliberately changes the surface.
Each scheduled retry appends a non-surface `llm/retry` event with the failed provider, policy mode, canonical resolved-policy key, provider-policy retry number, delay, and failure facts. Normal events carry finite `maxRetries`; always events omit it, and UIs render the limit as `∞`. The event and failed `assistant/chunk` records do not contribute surface messages, so the next request contains the same derived context as the failed request unless another recovery policy deliberately changes the surface.
## Alternatives considered
@@ -56,7 +56,7 @@ Each scheduled retry appends a non-surface `llm/retry` event with the failed pro
## Verification
Adapter tests validate nested policies at provider load, prove registration captures configured and default policies, and retain the serving policy across in-flight route replacement. Unit and real-Loader composition tests select policies from the failed request's serving registration, exercise always mode beyond the normal budget, pin jitter and delay caps, prove downstream recovery ordering, prove cancellation and disposal drain delegated recovery before reaching quiescence, and prove both abort active backoff waits. Request-level coverage compares the complete messages of failed and retried attempts and rejects both provider error text and discarded partial output. JSONL and SQLite tests round-trip an always event without `Infinity`; invariant tests bind its provider to the request header and its retry number to the active provider policy; TUI tests render finite and infinite limits.
Adapter tests validate nested policies at provider load, prove registration captures configured and default policies, and retain the serving policy across in-flight route replacement. Unit and plugin-validation tests select policies from the failed request's serving registration, reject top-level `llmRetry` at the spine, CLI, TUI, and ACP schemas, separate different same-mode policies while preserving histories across reordered code sets, exercise always mode beyond the normal budget, pin jitter and delay caps, prove downstream recovery ordering, prove cancellation and disposal drain delegated recovery before reaching quiescence, and prove both abort active backoff waits. Published Loader fixtures reject the invalid app-level key in CLI and ACP and the invalid bundle-level key when loading the spine directly. Request-level coverage compares the complete messages of failed and retried attempts and rejects both provider error text and discarded partial output. A keyless headless `stream-json` snapshot runs failure, retry, and success through the assembled app, pins the complete `llm/retry` record, and rejects any model-message change between attempts. JSONL and SQLite tests round-trip an always event without `Infinity`; invariant tests validate the canonical policy tuple, bind its provider to the request header, bind its failure code and delay to the encoded policy, and bind its retry number to the active provider policy key; TUI tests render finite and infinite limits.
## Consequences
@@ -34,13 +34,13 @@ providers:
jitterRatio: 0.2
```
监听器从失败步骤关闭时生效的持久 `request/header` 读取提供方,后续恢复产生的改动不参与选择,但绝不会从可变的提供方注册表重新解析策略。normal 模式保留有界瞬态错误处理行为:它重试配置的错误代码,次数不超过 `maxRetries`在当前连续失败序列中,同一提供方策略安排的重试都计入次数;其他情况委托后续处理。
监听器从失败步骤关闭时生效的持久 `request/header` 读取提供方,后续恢复产生的改动不参与选择,但绝不会从可变的提供方注册表重新解析策略。它会根据已解析实际服务策略的所有字段生成规范键;由于错误资格按集合成员判断,生成时会对 `retryableCodes` 排序。重试历史只会对同一提供方和同一规范键延续。因此,即使模式未变,只要路由替换后的次数上限、错误代码成员或退避不同,重试计数与初始延迟都会重新开始。normal 模式保留有界瞬态错误处理行为:它重试配置的错误代码,次数不超过 `maxRetries`;其他情况委托后续处理。
always 模式先请求下游恢复,使上下文溢出压缩(compaction)之类的专用策略有机会取得进展。下游若决定重试,则以该决定为准。下游若决定失败或恢复过程抛出错误,则回退为无界重试同一提供方请求;抛出的错误会写入日志。重试监听器会持有并排空已委托的恢复,轮次取消或插件 dispose(资源释放)只能在其结束后完成;随后监听器会应用取消,而不会采用迟到的下游决定。成功、轮次取消和插件 dispose 是仅有的终止路径。
两种模式的本地延迟都按指数增长,从 `initialDelayMs` 增至 `maxDelayMs``jitterRatio``[1 - jitterRatio, 1 + jitterRatio]` 区间内的均匀随机样本乘以每次目标值,再应用上限。提供方给出的正数 `Retry-After` 若未超过上限,则保持精确且不加抖动。若提供方延迟超过上限,normal 模式会委托后续处理;always 模式则改用配置的本地退避,以维持无限重试保证。
每次安排重试都会追加一条不进入表层的 `llm/retry` 事件,其中包含失败的提供方、策略模式、提供方策略内的重试编号、延迟和失败事实。normal 事件包含有限的 `maxRetries`;always 事件省略该字段,UI 将上限渲染为 `∞`。该事件与失败的 `assistant/chunk` 记录都不会生成表层消息,因此除非其他恢复策略有意改变表层,否则下一次请求包含的派生上下文与失败请求相同。
每次安排重试都会追加一条不进入表层的 `llm/retry` 事件,其中包含失败的提供方、策略模式、已解析策略的规范键、提供方策略内的重试编号、延迟和失败事实。normal 事件包含有限的 `maxRetries`;always 事件省略该字段,UI 将上限渲染为 `∞`。该事件与失败的 `assistant/chunk` 记录都不会生成表层消息,因此除非其他恢复策略有意改变表层,否则下一次请求包含的派生上下文与失败请求相同。
## 曾考虑的替代方案
@@ -56,7 +56,7 @@ always 模式先请求下游恢复,使上下文溢出压缩(compaction)之
## 验证
适配器测试会在提供方加载时校验嵌套策略,证明注册流程会捕获已配置策略和默认策略,并证明请求进行期间替换路由后仍会保留实际提供服务的策略。单元测试与真实 Loader 组合测试根据失败请求实际使用的注册项选择策略、验证 always 模式可越过 normal 预算、固定抖动和延迟上限、证明下游恢复顺序、证明取消与 dispose 会在达到静止状态前排空已委托的恢复,并证明二者都会停止正在进行的退避等待。请求级覆盖会比较失败尝试与重试尝试的完整消息,并排除提供方错误文本和丢弃的部分输出。JSONL 与 SQLite 测试会往返读写不含 `Infinity` 的 always 事件;不变式测试会将事件中的提供方绑定到请求头,并将重试编号绑定到活跃的提供方策略;TUI 测试会渲染有限和无限上限。
适配器测试会在提供方加载时校验嵌套策略,证明注册流程会捕获已配置策略和默认策略,并证明请求进行期间替换路由后仍会保留实际提供服务的策略。单元测试与插件校验测试根据失败请求实际使用的注册项选择策略、在主干、CLI、TUI 与 ACP schema 拒绝顶层 `llmRetry`、分离模式相同但策略不同的替换路由历史,同时在错误代码集合仅顺序不同时延续历史、验证 always 模式可越过 normal 预算、固定抖动和延迟上限、证明下游恢复顺序、证明取消与 dispose 会排空已委托的恢复再达到完全停稳,并证明二者都会停止正在进行的退避等待。发布版 Loader fixture(测试前置数据)会在 CLI 与 ACP 中拒绝无效的应用级配置键,并在直接加载主干时拒绝无效的 bundle 级配置键。请求级覆盖会比较失败尝试与重试尝试的完整消息,并排除提供方错误文本和丢弃的部分输出。一个无密钥 headless `stream-json` 快照会通过组装后的应用执行失败、重试与成功流程,固定完整的 `llm/retry` 记录,并拒绝各次尝试之间出现任何模型消息变化。JSONL 与 SQLite 测试会往返读写不含 `Infinity` 的 always 事件;不变式测试会校验规范策略元组、将事件中的提供方绑定到请求头、将失败代码与延迟绑定到编码后的策略,并将重试编号绑定到活跃的提供方策略TUI 测试会渲染有限和无限上限。
## 后果
+9 -1
View File
@@ -73,6 +73,8 @@ export interface Config {
toolTasks?: NonNullable<agentCore.Config['toolTasks']>
/** Persisted same-session goals; owner defaults enable them, or false disables the stack and tools. */
goals?: agentCore.GoalConfig | false
/** Invalid at app level; configure `retryPolicy` under each provider. */
llmRetry?: never
}
```
@@ -160,6 +162,8 @@ export interface Config {
invariants?: InvariantConfig
/** Opt-in persisted same-session goal stack; set false or omit to leave it unmounted. */
goals?: GoalConfig | false
/** Invalid at bundle level; configure `retryPolicy` under each provider. */
llmRetry?: never
}
/** Skill bundle config forwarded to the registry, local provider, and model-facing consumer. */
@@ -261,6 +265,8 @@ export interface Config {
toolTasks?: NonNullable<agentCore.Config['toolTasks']>
/** Controls automatic AGENTS.md/CLAUDE.md loading; configure a byte budget or set `false`. */
workspaceContext: agentCore.Config['workspaceContext']
/** Invalid at app level; configure `retryPolicy` under each provider. */
llmRetry?: never
}
```
@@ -696,7 +702,7 @@ Requires: `agents`
export type Config = Readonly<Record<string, never>>
```
Source: [`packages/llm/llm-retry/src/index.ts:43`](../packages/llm/llm-retry/src/index.ts)
Source: [`packages/llm/llm-retry/src/index.ts:46`](../packages/llm/llm-retry/src/index.ts)
## `@deepseek-ai/dsh-lsp-local`
@@ -1794,6 +1800,8 @@ export interface Config {
resumeSessionId?: string
/** Controls automatic AGENTS.md/CLAUDE.md loading; configure a byte budget or set `false`. */
workspaceContext: agentCore.Config['workspaceContext']
/** Invalid at app level; configure `retryPolicy` under each provider. */
llmRetry?: never
}
```
+3 -1
View File
@@ -277,6 +277,7 @@ Source: [`packages/hooks/hook-protocol/src/types.ts:31`](../packages/hooks/hook-
step: number
provider: string
mode: 'normal'
policyKey: string
retry: number
maxRetries: number
delayMs: number
@@ -286,13 +287,14 @@ Source: [`packages/hooks/hook-protocol/src/types.ts:31`](../packages/hooks/hook-
step: number
provider: string
mode: 'always'
policyKey: string
retry: number
delayMs: number
failure: LlmFailure
}
```
Source: [`packages/llm/llm-retry/src/index.ts:18`](../packages/llm/llm-retry/src/index.ts)
Source: [`packages/llm/llm-retry/src/index.ts:19`](../packages/llm/llm-retry/src/index.ts)
### `permission/*`
@@ -0,0 +1,12 @@
# Keyless provider-retry composition for the headless stream-json snapshot.
- id: base
name: '@cordisjs/plugin-include'
config:
path: ./cordis.yml
patches:
- id: llm-deepseek
name: '@deepseek-ai/dsh-llm-deepseek'
disabled: true
- insert:
- id: retry-snapshot-backend
name: './tests/fixtures/retry-snapshot-backend.mjs'
@@ -0,0 +1,53 @@
/** Deterministic provider adapter for the headless retry-policy snapshot. */
import {
LlmAdapter,
LlmError,
resolveRetryPolicy,
} from '@deepseek-ai/dsh-llm'
class RetrySnapshotAdapter extends LlmAdapter {
requests = 0
firstMessages
policy = resolveRetryPolicy({
mode: 'normal',
maxRetries: 1,
retryableCodes: ['RATE_LIMIT'],
backoff: { initialDelayMs: 1, maxDelayMs: 1, jitterRatio: 0 },
}, 'retry-snapshot-backend.retryPolicy')
providerRetryPolicy() {
return this.policy
}
async * stream(options) {
const messages = JSON.stringify(options.messages)
this.requests++
if (this.requests === 1) {
this.firstMessages = messages
throw new LlmError('snapshot transient failure', 'RATE_LIMIT', { status: 429 })
}
if (this.requests === 2 && messages !== this.firstMessages) {
throw new Error('retry snapshot changed the model-visible messages')
}
const text = 'RETRY_OK'
yield { type: 'block-start', index: 0, blockType: 'text' }
yield { type: 'text-delta', index: 0, text }
yield { type: 'block-end', index: 0, block: { type: 'text', text } }
yield { type: 'usage', usage: { inputTokens: 4, outputTokens: 2 } }
yield { type: 'finish', reason: { kind: 'stop' } }
}
}
/** Cordis plugin name. */
export const name = 'retry-snapshot-backend'
/** Required LLM registry service. */
export const inject = ['llm']
/**
* Register the deterministic provider adapter.
* @param {import('cordis').Context} ctx - plugin context carrying the LLM service.
*/
export function apply(ctx) {
ctx.llm.registerAdapter(['deepseek'], new RetrySnapshotAdapter())
}
@@ -24,6 +24,8 @@ const ptyStreamExpected = join(ptyScenarioDir, 'stream-json.expected.jsonl')
const ptyConfigPath = fileURLToPath(new URL('../pty.cordis.snapshot.yml', import.meta.url))
const goalScenarioDir = join(snapshotsDir, 'goal-tools')
const goalConfigPath = fileURLToPath(new URL('../goal.cordis.snapshot.yml', import.meta.url))
const retryScenarioDir = join(snapshotsDir, 'provider-retry')
const retryConfigPath = fileURLToPath(new URL('../retry.cordis.snapshot.yml', import.meta.url))
const ralphScenarioDir = join(snapshotsDir, 'ralph-loop')
const ralphConfigPath = fileURLToPath(new URL('../ralph.cordis.snapshot.yml', import.meta.url))
const binScript = fileURLToPath(new URL('../../../packages/examples/cli-demo/src/bin.ts', import.meta.url))
@@ -124,6 +126,46 @@ async function persistedLogs(cwd: string): Promise<PersistedLog[]> {
}
describe('headless stream-json snapshots', () => {
it('retries a transient provider failure through the one-shot app', async () => {
const prompt = await scenarioPrompt(retryScenarioDir, 'provider-retry')
const streamExpected = join(retryScenarioDir, 'stream-json.expected.jsonl')
let runCwd = ''
const result = await runLoaderSmoke({
label: 'provider retry headless stream-json snapshot',
tempDirPrefix: 'headless-snapshot-provider-retry-',
binScript,
configPath: retryConfigPath,
binArgs: ['--config', retryConfigPath, '--output-format', 'stream-json', prompt],
tsconfigPath,
env: {
DSH_SNAPSHOT: 'replay',
NODE_OPTIONS: [process.env.NODE_OPTIONS, '--disable-warning=ExperimentalWarning'].filter(Boolean).join(' '),
},
prepare: (cwd) => { runCwd = cwd },
inspect: async (cwd) => {
const logs = await persistedLogs(cwd)
expect(logs).toHaveLength(1)
const records = parseJsonl(logs[0]?.content ?? '')
const retries = records.filter(record => record.type === 'llm/retry')
expect(retries).toHaveLength(1)
expect(retries[0]?.data).toMatchObject({
provider: 'deepseek',
mode: 'normal',
policyKey: '["normal",1,["RATE_LIMIT"],1,1,0]',
retry: 1,
maxRetries: 1,
delayMs: 1,
failure: { message: 'snapshot transient failure', code: 'RATE_LIMIT', status: 429 },
})
},
})
expect(result.stderr).toBe('')
const normalized = normalizeHeadlessStream(result.stdout, runCwd)
if (refreshing) await writeFile(streamExpected, normalized)
expect(normalized).toBe(await readFile(streamExpected, 'utf8'))
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
it('replays the advanced toolchain through the one-shot app', async () => {
const prompt = await scenarioPrompt(advancedScenarioDir, 'advanced-toolchain')
const fixtureFiles = [
@@ -0,0 +1,8 @@
{
"steps": [
{
"op": "prompt",
"text": "retry the transient provider failure"
}
]
}
@@ -0,0 +1,17 @@
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"turn/start","seq":0,"time":0,"data":{"turn":1,"trigger":{"kind":"message","source":{"kind":"user"}}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"user/message","seq":1,"time":0,"data":{"content":[{"type":"text","text":"retry the transient provider failure"}],"source":{"kind":"user"}},"surfaceOp":"append"}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"session/title","seq":2,"time":0,"data":{"title":"retry the transient provider failure","messageSeqs":[1],"source":{"kind":"fallback"}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"step/start","seq":3,"time":0,"data":{"turn":1,"step":1}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"request/header","seq":4,"time":0,"data":{"header":{"config":{"provider":"deepseek","model":"deepseek-v4-flash"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"step/end","seq":5,"time":0,"data":{"turn":1,"step":1}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"llm/retry","seq":6,"time":0,"data":{"turn":1,"step":1,"provider":"deepseek","mode":"normal","policyKey":"[\"normal\",1,[\"RATE_LIMIT\"],1,1,0]","retry":1,"maxRetries":1,"delayMs":1,"failure":{"message":"snapshot transient failure","code":"RATE_LIMIT","status":429}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"step/start","seq":7,"time":0,"data":{"turn":1,"step":2}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":8,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"block-start","index":0,"blockType":"text"}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":9,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"text-delta","index":0,"text":"RETRY_OK"}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":10,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"RETRY_OK"}}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":11,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"usage","usage":{"inputTokens":4,"outputTokens":2}}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":12,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"finish","reason":{"kind":"stop"}}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/message","seq":13,"time":0,"data":{"turn":1,"step":2,"content":[{"type":"text","text":"RETRY_OK"}],"provenance":{"provider":"deepseek","model":"deepseek-v4-flash"},"usage":{"inputTokens":4,"outputTokens":2}},"sourceEventSeqs":[8,9,10,11,12],"surfaceOp":"append"}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"step/end","seq":14,"time":0,"data":{"turn":1,"step":2}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"turn/end","seq":15,"time":0,"data":{"turn":1,"reason":{"kind":"completed"}}}}
{"type":"result","success":true,"sessionId":"{{sessionId}}","turn":1,"result":"RETRY_OK","reason":{"kind":"completed"},"usage":{"inputTokens":4,"outputTokens":2}}
+4
View File
@@ -67,6 +67,8 @@ export interface Config {
toolTasks?: NonNullable<agentCore.Config['toolTasks']>
/** Persisted same-session goals; owner defaults enable them, or false disables the stack and tools. */
goals?: agentCore.GoalConfig | false
/** Invalid at app level; configure `retryPolicy` under each provider. */
llmRetry?: never
}
// Each front door owns a complete, directly readable config schema; extracting
@@ -92,6 +94,8 @@ export const Config: z<Config> = z.object({
toolBash: agentCore.ToolBashConfigSchema,
toolTasks: z.union([z.const(false), agentCore.ToolTasksConfigSchema]),
goals: z.union([z.const(false), agentCore.GoalConfigSchema]),
// Provider retryPolicy makes a top-level llmRetry invalid.
llmRetry: z.never(),
})
/* jscpd:ignore-end */
@@ -76,6 +76,16 @@ async function withIsolatedSkillHomes<T>(run: () => Promise<T>): Promise<T> {
}
describe('dsh-acp-demo composition', () => {
it('rejects app-level llmRetry config through plugin validation', async () => {
const ctx = new Context()
await expect(ctx.plugin(acpAgent, {
provider: 'mock',
model: 'mock',
workspaceContext: false,
llmRetry: { maxTransientRetries: 2 },
} as never)).rejects.toThrow(/llmRetry/)
})
it('brings up the spine + persistence + the ACP bridge', async () => {
const ctx = await mount({
provider: 'mock',
@@ -207,6 +207,20 @@ describe.skipIf(!existsSync(acpBin))('dsh-acp-demo BUILT bin (node lib/bin.js, n
expect(code).not.toBe(0)
expect(stderr).toContain('config file not found')
}, 30_000)
it('rejects legacy app-level llmRetry through the published Loader path', async () => {
consumer = await makeConsumer()
const configPath = join(consumer, 'cordis.yml')
const config = await readFile(configPath, 'utf8')
await writeFile(configPath, config.replace(
' workspaceContext: false',
' workspaceContext: false\n llmRetry:\n maxTransientRetries: 2',
))
const { code, stderr } = await runBinExpectingExit('./cordis.yml', consumer)
expect(code).not.toBe(0)
expect(stderr).toContain('llmRetry')
}, 30_000)
})
/** Spawn the built acp bin against `configArg` and resolve with its exit code + stderr. */
@@ -112,6 +112,8 @@ export interface Config {
invariants?: InvariantConfig
/** Opt-in persisted same-session goal stack; set false or omit to leave it unmounted. */
goals?: GoalConfig | false
/** Invalid at bundle level; configure `retryPolicy` under each provider. */
llmRetry?: never
}
/** The skill config schema exported for app packages that forward `skills`. */
@@ -152,6 +154,9 @@ export const Config = z.intersect([
toolTasks: z.union([z.const(false), ToolTasksConfigSchema]),
invariants: InvariantService.Config,
goals: z.union([z.const(false), GoalConfigSchema]),
// Schemastery preserves unknown object properties. A top-level llmRetry is
// known-but-impossible because provider retryPolicy owns this configuration.
llmRetry: z.never(),
}) as unknown as z<Pick<Config, 'tools' | 'dshHome' | 'sessionTitle' | 'skills' | 'workspaceContext' | 'toolBash' | 'toolTasks' | 'invariants' | 'goals'>>,
]) as unknown as z<Config>
@@ -594,6 +594,14 @@ describe('dsh-agent-spine-demo bundle', () => {
expect(agentCore.name).toBe('agent-spine-demo')
})
it('rejects bundle-level llmRetry config through plugin validation', async () => {
const ctx = new Context()
await expect(ctx.plugin(agentCore, {
workspaceContext: false,
llmRetry: { maxTransientRetries: 2 },
} as never)).rejects.toThrow(/llmRetry/)
})
it('has the namespace-plugin export shape (no stray default) so the Loader keeps name/Config/apply', () => {
// A default export would make `unwrapExports` collapse this inject-less namespace and silently
// drop `name`/`Config`. Apps import the bundle directly, so this is its Loader-shape guard.
+4
View File
@@ -52,6 +52,8 @@ export interface Config {
toolTasks?: NonNullable<agentCore.Config['toolTasks']>
/** Controls automatic AGENTS.md/CLAUDE.md loading; configure a byte budget or set `false`. */
workspaceContext: agentCore.Config['workspaceContext']
/** Invalid at app level; configure `retryPolicy` under each provider. */
llmRetry?: never
}
// Each front door keeps a complete Loader schema so its deployment contract is
@@ -73,6 +75,8 @@ export const Config: z<Config> = z.object({
toolBash: agentCore.ToolBashConfigSchema,
toolTasks: z.union([z.const(false), agentCore.ToolTasksConfigSchema]),
workspaceContext: z.union([z.const(false), workspaceContext.Config]).required(),
// Provider retryPolicy makes a top-level llmRetry invalid.
llmRetry: z.never(),
})
/* jscpd:ignore-end */
@@ -192,6 +192,39 @@ describe.skipIf(!existsSync(cliBin))('dsh-cli-demo BUILT bin', () => {
}
}, 30_000)
it('rejects legacy app-level llmRetry through the published Loader path', async () => {
consumer = await makeConsumer()
const configPath = join(consumer, 'cordis.yml')
const config = await readFile(configPath, 'utf8')
await writeFile(configPath, config.replace(
' workspaceContext: false',
' workspaceContext: false\n llmRetry:\n maxTransientRetries: 2',
))
const result = await runBuiltBin(consumer, ['--config', './cordis.yml', 'task'])
expect(result.code).not.toBe(0)
expect(result.stdout).toBe('')
expect(result.stderr).toContain('llmRetry')
}, 30_000)
it('rejects legacy bundle-level llmRetry when the published spine is loaded directly', async () => {
consumer = await makeConsumer()
await writeFile(join(consumer, 'cordis.yml'), [
'- id: spine',
" name: '@deepseek-ai/dsh-agent-spine-demo'",
' config:',
' workspaceContext: false',
' llmRetry:',
' maxTransientRetries: 2',
'',
].join('\n'))
const result = await runBuiltBin(consumer, ['--config', './cordis.yml', 'task'])
expect(result.code).not.toBe(0)
expect(result.stdout).toBe('')
expect(result.stderr).toContain('llmRetry')
}, 30_000)
describe.skipIf(process.platform === 'win32')('POSIX signal delivery', () => {
it.each([
['SIGINT', 130],
@@ -173,6 +173,16 @@ afterEach(async () => {
})
describe('parseCliArgs', () => {
it('rejects app-level llmRetry config through plugin validation', async () => {
const ctx = new Context()
await expect(ctx.plugin(cliDemo, {
provider: 'mock',
model: 'mock',
workspaceContext: false,
llmRetry: { maxTransientRetries: 2 },
} as never)).rejects.toThrow(/llmRetry/)
})
it('parses defaults, explicit options, spaces, and an option-like task after --', () => {
expect(parseCliArgs(['task with spaces'])).toEqual({
kind: 'run', configPath: './cordis.yml', outputFormat: 'text', task: 'task with spaces',
+4
View File
@@ -82,6 +82,8 @@ export interface Config {
resumeSessionId?: string
/** Controls automatic AGENTS.md/CLAUDE.md loading; configure a byte budget or set `false`. */
workspaceContext: agentCore.Config['workspaceContext']
/** Invalid at app level; configure `retryPolicy` under each provider. */
llmRetry?: never
}
export const Config: z<Config> = z.object({
@@ -106,6 +108,8 @@ export const Config: z<Config> = z.object({
goals: z.union([z.const(false), agentCore.GoalConfigSchema]),
resumeSessionId: z.string(),
workspaceContext: z.union([z.const(false), workspaceContext.Config]).required(),
// Provider retryPolicy makes a top-level llmRetry invalid.
llmRetry: z.never(),
})
/* jscpd:ignore-end */
@@ -1,6 +1,6 @@
import { describe, expect, it } from 'vitest'
import { join } from 'node:path'
import type { Context } from 'cordis'
import { Context } from 'cordis'
import Loader from '@cordisjs/plugin-loader'
import { TOOL_ORDER_REST } from '@deepseek-ai/dsh-system-prompt'
import * as tuiAgent from '../src/index.ts'
@@ -21,6 +21,16 @@ function recordingContext(): { readonly ctx: Context; readonly calls: PluginCall
}
describe('dsh-tui-demo app', () => {
it('rejects app-level llmRetry config through plugin validation', async () => {
const ctx = new Context()
await expect(ctx.plugin(tuiAgent, {
provider: 'mock',
model: 'mock',
workspaceContext: false,
llmRetry: { maxTransientRetries: 2 },
} as never)).rejects.toThrow(/llmRetry/)
})
it('composes the TUI cluster around one fresh exact session identity', () => {
const { ctx, calls } = recordingContext()
tuiAgent.composeTuiApp(ctx, {
+2 -2
View File
@@ -6,9 +6,9 @@ Each provider adapter owns an optional nested `retryPolicy`, captured when its r
Both modes use bounded exponential backoff with symmetric jitter. A valid `providerRetryAfterMs` at or below `maxDelayMs` replaces local backoff without jitter. An over-cap provider delay makes normal mode delegate, while always mode uses its configured local backoff so it cannot terminate on that instruction.
Before waiting, the plugin appends a non-surface `llm/retry` event with the provider, mode, failure, and scheduled delay. Normal events include the finite maximum; always events omit it, and UIs render `∞`. Cancellation and plugin disposal abort active backoff, drain active delegated recovery before applying the abort, and make a callback captured before disposal fail closed.
Before waiting, the plugin appends a non-surface `llm/retry` event with the provider, mode, canonical resolved-policy key, failure, and scheduled delay. The key includes every behavior-affecting field and sorts normal-mode codes because eligibility uses set membership. Retry numbers continue only across events with the same provider and complete policy key, so a route replacement with different limits, code membership, or backoff starts its own history. Normal events include the finite maximum; always events omit it, and UIs render `∞`. Cancellation and plugin disposal abort active backoff, drain active delegated recovery before applying the abort, and make a callback captured before disposal fail closed.
The separately published `./invariant` companion checks that every retry record names the current open turn and latest closed step, matches the failed request's durable provider, has a unique step record and correct provider-policy retry number, and carries a valid mode-specific budget and bounded timer delay. Full jitter may schedule zero milliseconds at its lower boundary.
The separately published `./invariant` companion checks that every retry record names the current open turn and latest closed step, matches the failed request's durable provider, carries a producer-canonical policy key consistent with its mode and finite budget, binds normal failures and every scheduled delay to that policy, has a unique step record and correct provider-policy retry number, and carries a bounded timer delay. Full jitter may schedule zero milliseconds at its lower boundary.
```yaml
- name: '@deepseek-ai/dsh-llm-deepseek'
+9 -2
View File
@@ -11,6 +11,7 @@ import type { Agent, RequestError, RequestErrorDecision } from '@deepseek-ai/dsh
import type { LlmFailure, ResolvedRetryPolicy } from '@deepseek-ai/dsh-llm'
import type { SessionEvent } from '@deepseek-ai/dsh-session'
import { providerForClosedStep } from './history.ts'
import { retryPolicyKey } from './policy-key.ts'
declare module '@deepseek-ai/dsh-session' {
interface SessionEventMap {
@@ -20,6 +21,7 @@ declare module '@deepseek-ai/dsh-session' {
step: number
provider: string
mode: 'normal'
policyKey: string
retry: number
maxRetries: number
delayMs: number
@@ -29,6 +31,7 @@ declare module '@deepseek-ai/dsh-session' {
step: number
provider: string
mode: 'always'
policyKey: string
retry: number
delayMs: number
failure: LlmFailure
@@ -121,6 +124,7 @@ export function apply(ctx: Context, config: Config = {}, internals: RetryInterna
failure: LlmFailure,
provider: string,
policy: ResolvedRetryPolicy,
policyKey: string,
retry: number,
delayMs: number,
signal: AbortSignal,
@@ -133,6 +137,7 @@ export function apply(ctx: Context, config: Config = {}, internals: RetryInterna
step,
provider,
mode: policy.mode,
policyKey,
retry,
maxRetries: policy.maxRetries,
delayMs,
@@ -143,6 +148,7 @@ export function apply(ctx: Context, config: Config = {}, internals: RetryInterna
step,
provider,
mode: policy.mode,
policyKey,
retry,
delayMs,
failure,
@@ -192,6 +198,7 @@ export function apply(ctx: Context, config: Config = {}, internals: RetryInterna
return next()
}
const policyKey = retryPolicyKey(policy)
const firstPriorStep = step - priorFailures.length
const priorPolicyRetry = agent.session.events.findLast((event): event is SessionEvent<'llm/retry'> =>
event.type === 'llm/retry'
@@ -199,7 +206,7 @@ export function apply(ctx: Context, config: Config = {}, internals: RetryInterna
&& event.data.step >= firstPriorStep
&& event.data.step < step
&& event.data.provider === provider
&& event.data.mode === policy.mode,
&& event.data.policyKey === policyKey,
)
const previousRetry = priorPolicyRetry?.data.retry ?? 0
if (policy.mode === 'normal' && previousRetry >= policy.maxRetries) return next()
@@ -218,7 +225,7 @@ export function apply(ctx: Context, config: Config = {}, internals: RetryInterna
delayMs = localDelay(policy, retry, random)
}
return backoff(agent, turn, step, failure, provider, policy, retry, delayMs, signal)
return backoff(agent, turn, step, failure, provider, policy, policyKey, retry, delayMs, signal)
}
const disposeListener = ctx.on('agent/request-error', (
+21 -5
View File
@@ -2,9 +2,9 @@
import type { Context } from 'cordis'
import type { Session, SessionEvent } from '@deepseek-ai/dsh-session'
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
import type { InvariantFailure, InvariantInstaller } from '@deepseek-ai/dsh-invariants'
import { providerForClosedStep } from './history.ts'
import { parseRetryPolicyKey } from './policy-key.ts'
import type {} from './index.ts'
const PACKAGE_NAME = '@deepseek-ai/dsh-llm-retry'
@@ -20,30 +20,46 @@ function validateRetry(
event: SessionEvent<'llm/retry'>,
fail: InvariantFailure,
): void {
const { turn, step, provider, mode, retry, delayMs } = event.data
const { turn, step, provider, mode, policyKey, retry, delayMs } = event.data
if (!Number.isSafeInteger(retry) || retry < 1) {
fail('llm/retry retry must be a positive safe integer')
}
if (typeof provider !== 'string' || provider.length === 0) {
fail('llm/retry provider must be non-empty string')
}
const keyedPolicy = parseRetryPolicyKey(policyKey)
if (keyedPolicy === undefined) {
fail('llm/retry policyKey must encode a canonical resolved policy')
}
switch (mode) {
case 'normal': {
const { maxRetries } = event.data
if (!Number.isSafeInteger(maxRetries) || maxRetries < 1 || retry > maxRetries) {
fail(`llm/retry retry ${retry} must not exceed a positive safe maxRetries ${maxRetries}`)
}
if (keyedPolicy.mode !== 'normal') {
fail(`llm/retry mode normal must match policyKey mode ${keyedPolicy.mode}`)
}
if (keyedPolicy.maxRetries !== maxRetries) {
fail(`llm/retry maxRetries ${maxRetries} must match policyKey`)
}
if (!keyedPolicy.retryableCodes.includes(event.data.failure.code)) {
fail(`llm/retry failure code ${event.data.failure.code} must be eligible under policyKey`)
}
break
}
case 'always':
if (keyedPolicy.mode !== 'always') {
fail(`llm/retry mode always must match policyKey mode ${keyedPolicy.mode}`)
}
if ('maxRetries' in event.data) fail('llm/retry always mode must omit maxRetries')
break
default:
fail(`llm/retry mode must be normal or always, got ${String(mode)}`)
}
if (typeof delayMs !== 'number' || !Number.isFinite(delayMs)
|| delayMs < 0 || delayMs > MAX_TIMER_DELAY_MS) {
fail(`llm/retry delayMs must be a finite number within 0..${MAX_TIMER_DELAY_MS}`)
|| delayMs < 0 || delayMs > keyedPolicy.maxDelayMs) {
fail(`llm/retry delayMs must be a finite number within policyKey range 0..${keyedPolicy.maxDelayMs}`)
}
const turnStartIndex = history.findLastIndex(prior =>
@@ -86,7 +102,7 @@ function validateRetry(
index > lastSuccessIndex
&& prior.type === 'llm/retry'
&& prior.data.provider === provider
&& prior.data.mode === mode
&& prior.data.policyKey === policyKey
))
const expectedRetry = (priorPolicyRetry?.data.retry ?? 0) + 1
if (retry !== expectedRetry) {
+100
View File
@@ -0,0 +1,100 @@
/** Canonical durable identity for resolved retry policies. @module @deepseek-ai/dsh-llm-retry/policy-key */
import type { ResolvedRetryBackoff, ResolvedRetryPolicy } from '@deepseek-ai/dsh-llm'
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
function parseBackoff(
tuple: readonly unknown[],
offset: number,
): ResolvedRetryBackoff | undefined {
const initialDelayMs = tuple[offset]
const maxDelayMs = tuple[offset + 1]
const jitterRatio = tuple[offset + 2]
if (typeof initialDelayMs !== 'number' || !Number.isFinite(initialDelayMs)
|| initialDelayMs <= 0 || initialDelayMs > MAX_TIMER_DELAY_MS
|| typeof maxDelayMs !== 'number' || !Number.isFinite(maxDelayMs)
|| maxDelayMs <= 0 || maxDelayMs > MAX_TIMER_DELAY_MS
|| initialDelayMs > maxDelayMs
|| typeof jitterRatio !== 'number' || !Number.isFinite(jitterRatio)
|| jitterRatio < 0 || jitterRatio > 1) {
return undefined
}
return { initialDelayMs, maxDelayMs, jitterRatio }
}
/**
* Derive the canonical durable key for one fully resolved provider policy.
* Retryable-code order is normalized because eligibility uses set membership.
* @param policy - immutable policy captured from the serving registration.
* @returns canonical JSON tuple containing every behavior-affecting field.
*/
export function retryPolicyKey(policy: ResolvedRetryPolicy): string {
if (policy.mode === 'always') {
return JSON.stringify([
policy.mode,
policy.initialDelayMs,
policy.maxDelayMs,
policy.jitterRatio,
])
}
return JSON.stringify([
policy.mode,
policy.maxRetries,
[...policy.retryableCodes].sort(),
policy.initialDelayMs,
policy.maxDelayMs,
policy.jitterRatio,
])
}
/**
* Parse a producer-canonical policy key from durable input.
* @param value - untrusted persisted event field.
* @returns the resolved policy encoded by the key, or `undefined` for any non-canonical value.
*/
export function parseRetryPolicyKey(value: unknown): ResolvedRetryPolicy | undefined {
if (typeof value !== 'string' || value.length === 0) return undefined
let tuple: unknown
try {
tuple = JSON.parse(value) as unknown
} catch (_invalidPolicyKeyJson) {
return undefined
}
if (!Array.isArray(tuple)) return undefined
const items = tuple as readonly unknown[]
const mode = items[0]
let policy: ResolvedRetryPolicy
switch (mode) {
case 'always': {
if (items.length !== 4) return undefined
const backoff = parseBackoff(items, 1)
if (backoff === undefined) return undefined
policy = Object.freeze({ mode, ...backoff })
break
}
case 'normal': {
if (items.length !== 6) return undefined
const maxRetries = items[1]
const retryableCodes = items[2]
const backoff = parseBackoff(items, 3)
if (!Number.isSafeInteger(maxRetries) || (maxRetries as number) < 0
|| !Array.isArray(retryableCodes) || retryableCodes.length === 0
|| (retryableCodes as readonly unknown[])
.some(code => typeof code !== 'string' || code.length === 0)
|| new Set(retryableCodes).size !== retryableCodes.length
|| backoff === undefined) {
return undefined
}
policy = Object.freeze({
mode,
maxRetries: maxRetries as number,
retryableCodes: Object.freeze(retryableCodes as string[]),
...backoff,
})
break
}
default:
return undefined
}
return retryPolicyKey(policy) === value ? policy : undefined
}
+94 -6
View File
@@ -27,7 +27,10 @@ function closeStep(ctx: Context, id: string, turn = 1, step = 1) {
}
const failure = { message: 'provider busy', code: 'RATE_LIMIT', status: 429 }
const normal = { provider: 'mock', mode: 'normal' as const }
const normalPolicyKey = (maxRetries: number): string =>
`["normal",${maxRetries},["RATE_LIMIT"],1,10000,0]`
const alwaysPolicyKey = '["always",1,10000,0]'
const normal = { provider: 'mock', mode: 'normal' as const, policyKey: normalPolicyKey(2) }
describe('llm-retry invariants', () => {
it('has no provider without the requested closed step', () => {
@@ -65,7 +68,8 @@ describe('llm-retry invariants', () => {
})
const zeroDelay = closeStep(ctx, 'retry-invariant-zero-delay')
zeroDelay.append('llm/retry', {
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 1, delayMs: 0, failure,
turn: 1, step: 1, ...normal, policyKey: normalPolicyKey(1),
retry: 1, maxRetries: 1, delayMs: 0, failure,
})
}).not.toThrow()
expect(() => { ctx.emit('tools/change') }).not.toThrow()
@@ -80,6 +84,7 @@ describe('llm-retry invariants', () => {
step: 1,
provider: 'mock',
mode: 'always',
policyKey: alwaysPolicyKey,
retry: 1,
delayMs: 500,
failure,
@@ -91,6 +96,7 @@ describe('llm-retry invariants', () => {
step: 1,
provider: 'mock',
mode: 'always',
policyKey: alwaysPolicyKey,
retry: 1,
maxRetries: 2,
delayMs: 500,
@@ -99,6 +105,65 @@ describe('llm-retry invariants', () => {
}).toThrow(/always mode must omit maxRetries/)
})
it('binds event mode and finite budget to the canonical policy key', async () => {
const ctx = await setup()
const normalModeMismatch = closeStep(ctx, 'retry-invariant-normal-mode-key')
expect(() => {
normalModeMismatch.append('llm/retry', {
turn: 1, step: 1, ...normal, policyKey: alwaysPolicyKey,
retry: 1, maxRetries: 2, delayMs: 1, failure,
})
}).toThrow(/mode normal must match policyKey mode always/)
const alwaysModeMismatch = closeStep(ctx, 'retry-invariant-always-mode-key')
expect(() => {
alwaysModeMismatch.append('llm/retry', {
turn: 1,
step: 1,
provider: 'mock',
mode: 'always',
policyKey: normalPolicyKey(2),
retry: 1,
delayMs: 1,
failure,
})
}).toThrow(/mode always must match policyKey mode normal/)
const budgetMismatch = closeStep(ctx, 'retry-invariant-budget-key')
expect(() => {
budgetMismatch.append('llm/retry', {
turn: 1, step: 1, ...normal, policyKey: normalPolicyKey(3),
retry: 1, maxRetries: 2, delayMs: 1, failure,
})
}).toThrow(/maxRetries 2 must match policyKey/)
})
it('binds the failure code and scheduled delay to the canonical policy key', async () => {
const ctx = await setup()
const ineligibleFailure = closeStep(ctx, 'retry-invariant-failure-code-key')
expect(() => {
ineligibleFailure.append('llm/retry', {
turn: 1, step: 1, ...normal,
retry: 1, maxRetries: 2, delayMs: 1,
failure: { message: 'authentication failed', code: 'AUTH', status: 401 },
})
}).toThrow(/failure code AUTH must be eligible under policyKey/)
const overPolicyDelay = closeStep(ctx, 'retry-invariant-delay-key')
expect(() => {
overPolicyDelay.append('llm/retry', {
turn: 1,
step: 1,
provider: 'mock',
mode: 'always',
policyKey: '["always",1,1,0]',
retry: 1,
delayMs: 2,
failure,
})
}).toThrow(/within policyKey range 0\.\.1/)
})
it('rejects empty providers and unknown modes from hostile durable input', async () => {
const ctx = await setup()
const emptyProvider = closeStep(ctx, 'retry-invariant-empty-provider')
@@ -108,6 +173,7 @@ describe('llm-retry invariants', () => {
step: 1,
provider: '',
mode: 'always',
policyKey: alwaysPolicyKey,
retry: 1,
delayMs: 1,
failure,
@@ -121,11 +187,26 @@ describe('llm-retry invariants', () => {
step: 1,
provider: 'mock',
mode: 'sometimes',
policyKey: alwaysPolicyKey,
retry: 1,
delayMs: 1,
failure,
} as never)
}).toThrow(/mode must be normal or always/)
const emptyPolicyKey = closeStep(ctx, 'retry-invariant-empty-policy-key')
expect(() => {
emptyPolicyKey.append('llm/retry', {
turn: 1,
step: 1,
provider: 'mock',
mode: 'always',
policyKey: '',
retry: 1,
delayMs: 1,
failure,
})
}).toThrow(/policyKey must encode a canonical resolved policy/)
})
it.each([
@@ -206,6 +287,7 @@ describe('llm-retry invariants', () => {
step: 1,
provider: 'mock',
mode: 'always',
policyKey: alwaysPolicyKey,
retry: 1,
delayMs: 1,
failure,
@@ -219,6 +301,7 @@ describe('llm-retry invariants', () => {
step: 1,
provider: 'other',
mode: 'always',
policyKey: alwaysPolicyKey,
retry: 1,
delayMs: 1,
failure,
@@ -239,6 +322,7 @@ describe('llm-retry invariants', () => {
step: 1,
provider: 'mock',
mode: 'always',
policyKey: alwaysPolicyKey,
retry: 1,
delayMs: 1,
failure,
@@ -260,23 +344,27 @@ describe('llm-retry invariants', () => {
const ctx = await setup()
const duplicate = closeStep(ctx, 'retry-invariant-duplicate')
duplicate.append('llm/retry', {
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 3, delayMs: 1, failure,
turn: 1, step: 1, ...normal, policyKey: normalPolicyKey(3),
retry: 1, maxRetries: 3, delayMs: 1, failure,
})
expect(() => {
duplicate.append('llm/retry', {
turn: 1, step: 1, ...normal, retry: 2, maxRetries: 3, delayMs: 1, failure,
turn: 1, step: 1, ...normal, policyKey: normalPolicyKey(3),
retry: 2, maxRetries: 3, delayMs: 1, failure,
})
}).toThrow(/duplicates the retry record/)
const nonIncreasing = closeStep(ctx, 'retry-invariant-non-increasing')
nonIncreasing.append('llm/retry', {
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 3, delayMs: 1, failure,
turn: 1, step: 1, ...normal, policyKey: normalPolicyKey(3),
retry: 1, maxRetries: 3, delayMs: 1, failure,
})
nonIncreasing.append('step/start', { turn: 1, step: 2 })
nonIncreasing.append('step/end', { turn: 1, step: 2 })
expect(() => {
nonIncreasing.append('llm/retry', {
turn: 1, step: 2, ...normal, retry: 1, maxRetries: 3, delayMs: 1, failure,
turn: 1, step: 2, ...normal, policyKey: normalPolicyKey(3),
retry: 1, maxRetries: 3, delayMs: 1, failure,
})
}).toThrow(/must equal provider policy retry 2/)
})
@@ -44,6 +44,7 @@ describe.each(['jsonl', 'sqlite'] as const)('%s retry-event persistence', (kind)
step: 1,
provider: 'mock',
mode: 'always',
policyKey: '["always",500,10000,0.1]',
retry: 1,
delayMs: 750,
failure: { message: 'provider busy', code: 'RATE_LIMIT', status: 429 },
@@ -0,0 +1,71 @@
import { describe, expect, it } from 'vitest'
import { resolveRetryPolicy } from '@deepseek-ai/dsh-llm'
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
import { parseRetryPolicyKey, retryPolicyKey } from '../src/policy-key.ts'
describe('retry policy durable key', () => {
it('includes every policy field while normalizing code-set order', () => {
const first = resolveRetryPolicy({
mode: 'normal',
maxRetries: 4,
retryableCodes: ['SERVER', 'RATE_LIMIT'],
backoff: { initialDelayMs: 3, maxDelayMs: 9, jitterRatio: 0.25 },
}, 'first')
const reordered = resolveRetryPolicy({
mode: 'normal',
maxRetries: 4,
retryableCodes: ['RATE_LIMIT', 'SERVER'],
backoff: { initialDelayMs: 3, maxDelayMs: 9, jitterRatio: 0.25 },
}, 'reordered')
const key = retryPolicyKey(first)
expect(key).toBe('["normal",4,["RATE_LIMIT","SERVER"],3,9,0.25]')
expect(retryPolicyKey(reordered)).toBe(key)
expect(parseRetryPolicyKey(key)).toEqual(reordered)
})
it('round-trips always mode', () => {
const policy = resolveRetryPolicy({
mode: 'always',
backoff: { initialDelayMs: 2, maxDelayMs: 8, jitterRatio: 1 },
}, 'always')
const key = retryPolicyKey(policy)
expect(key).toBe('["always",2,8,1]')
expect(parseRetryPolicyKey(key)).toEqual(policy)
})
it.each([
undefined,
'',
'{',
'{}',
'["sometimes",1,2,0]',
'["always",1,2]',
'["always","1",2,0]',
'["always",1e400,2,0]',
'["always",0,2,0]',
`["always",${MAX_TIMER_DELAY_MS + 1},${MAX_TIMER_DELAY_MS + 1},0]`,
'["always",1,"2",0]',
'["always",1,1e400,0]',
'["always",1,0,0]',
`["always",1,${MAX_TIMER_DELAY_MS + 1},0]`,
'["always",2,1,0]',
'["always",1,2,"0"]',
'["always",1,2,1e400]',
'["always",1,2,-0.1]',
'["always",1,2,1.1]',
'["normal",2,["SERVER"],1,2]',
'["normal","2",["SERVER"],1,2,0]',
'["normal",-1,["SERVER"],1,2,0]',
'["normal",2,"SERVER",1,2,0]',
'["normal",2,[],1,2,0]',
'["normal",2,[1],1,2,0]',
'["normal",2,[""],1,2,0]',
'["normal",2,["SERVER","SERVER"],1,2,0]',
'["normal",2,["SERVER"],2,1,0]',
'["normal",2,["SERVER","RATE_LIMIT"],1,2,0]',
])('rejects non-canonical durable input %#', (value) => {
expect(parseRetryPolicyKey(value)).toBeUndefined()
})
})
+105
View File
@@ -184,6 +184,7 @@ describe('provider-routed retry policy', () => {
step: 1,
provider: 'mock',
mode: 'normal',
policyKey: '["normal",2,["RATE_LIMIT","SERVER","TIMEOUT","TRANSPORT"],500,10000,0.1]',
retry: 1,
maxRetries: 2,
delayMs: 500,
@@ -561,6 +562,110 @@ describe('provider-routed retry policy', () => {
},
)
it('starts a new retry history when a same-mode route replacement changes policy', async () => {
vi.useFakeTimers()
const oldAdapter = new ScriptedAdapter([
new LlmError('old route failed', 'AUTH'),
])
const mounted = await harness(oldAdapter, { mock: alwaysConfig({
initialDelayMs: 1,
maxDelayMs: 1,
}) })
context = mounted.ctx
const agent = context.agentLoop.create(SessionId('retry-policy-replacement'), {
provider: 'mock',
model: 'mock',
})
const first = waitForRetry(context, agent, 1)
agent.followup([{ type: 'text', text: 'replace policy between attempts' }])
expect((await first).data).toMatchObject({
mode: 'always',
retry: 1,
delayMs: 1,
})
mounted.disposeAdapter()
const replacement = new ScriptedAdapter([
new LlmError('replacement failed', 'AUTH'),
textResponse('replacement recovered'),
])
replacement.configureRetryPolicies({ mock: alwaysConfig({
initialDelayMs: 3,
maxDelayMs: 9,
}) })
context.llm.registerAdapter(['mock'], replacement)
const second = waitForRetry(context, agent, 1)
await vi.advanceTimersByTimeAsync(1)
expect((await second).data).toMatchObject({
mode: 'always',
retry: 1,
delayMs: 3,
})
const idle = waitForIdle(context, agent)
await vi.advanceTimersByTimeAsync(3)
await idle
expect(oldAdapter.requests).toHaveLength(1)
expect(replacement.requests).toHaveLength(2)
expect(agent.session.events.filter(event => event.type === 'llm/retry').map(event => ({
policyKey: event.data.policyKey,
retry: event.data.retry,
}))).toEqual([
{ policyKey: '["always",1,1,0]', retry: 1 },
{ policyKey: '["always",3,9,0]', retry: 1 },
])
})
it('continues retry history when a replacement only reorders retryable codes', async () => {
vi.useFakeTimers()
const oldAdapter = new ScriptedAdapter([
new LlmError('old route failed', 'SERVER'),
])
const mounted = await harness(oldAdapter, { mock: normalConfig({
maxRetries: 2,
retryableCodes: ['SERVER', 'RATE_LIMIT'],
backoff: { initialDelayMs: 1, maxDelayMs: 4 },
}) })
context = mounted.ctx
const agent = context.agentLoop.create(SessionId('retry-policy-code-order'), {
provider: 'mock',
model: 'mock',
})
const first = waitForRetry(context, agent, 1)
agent.followup([{ type: 'text', text: 'replace equivalent policy between attempts' }])
const firstEvent = await first
expect(firstEvent.data.delayMs).toBe(1)
mounted.disposeAdapter()
const replacement = new ScriptedAdapter([
new LlmError('replacement failed', 'SERVER'),
textResponse('replacement recovered'),
])
replacement.configureRetryPolicies({ mock: normalConfig({
maxRetries: 2,
retryableCodes: ['RATE_LIMIT', 'SERVER'],
backoff: { initialDelayMs: 1, maxDelayMs: 4 },
}) })
context.llm.registerAdapter(['mock'], replacement)
const second = waitForRetry(context, agent, 2)
await vi.advanceTimersByTimeAsync(1)
const secondEvent = await second
expect(secondEvent.data).toMatchObject({ retry: 2, delayMs: 2 })
expect(secondEvent.data.policyKey).toBe(firstEvent.data.policyKey)
const idle = waitForIdle(context, agent)
await vi.advanceTimersByTimeAsync(2)
await idle
expect(oldAdapter.requests).toHaveLength(1)
expect(replacement.requests).toHaveLength(2)
})
it('keeps always mode unbounded while preserving cancellable jittered backoff', async () => {
vi.useFakeTimers()
const adapter = new ScriptedAdapter([
+2
View File
@@ -271,6 +271,7 @@ describe('TUI terminal-state snapshots', () => {
step: 1,
provider: 'mock',
mode: 'normal',
policyKey: '["normal",2,["RATE_LIMIT"],1,10000,0]',
retry: 1,
maxRetries: 2,
delayMs: 500,
@@ -301,6 +302,7 @@ describe('TUI terminal-state snapshots', () => {
step: 1,
provider: 'mock',
mode: 'always',
policyKey: '["always",1,10000,0]',
retry: 1,
delayMs: 1_000,
failure: { message: 'temporary transport failure', code: 'TRANSPORT' },
+4
View File
@@ -1296,6 +1296,7 @@ describe('pi-tui chat lifecycle and transcript', () => {
step: 1,
provider: 'mock',
mode: 'normal',
policyKey: '["normal",2,["RATE_LIMIT"],1,10000,0]',
retry: 1,
maxRetries: 2,
delayMs: 500,
@@ -1330,6 +1331,7 @@ describe('pi-tui chat lifecycle and transcript', () => {
step: 1,
provider: 'mock',
mode: 'normal',
policyKey: '["normal",2,["RATE_LIMIT"],1,10000,0]',
retry: 1,
maxRetries: 2,
delayMs: 500,
@@ -1340,6 +1342,7 @@ describe('pi-tui chat lifecycle and transcript', () => {
step: 2,
provider: 'mock',
mode: 'normal',
policyKey: '["normal",2,["RATE_LIMIT"],1,10000,0]',
retry: 2,
maxRetries: 2,
delayMs: 1_000,
@@ -1350,6 +1353,7 @@ describe('pi-tui chat lifecycle and transcript', () => {
step: 3,
provider: 'mock',
mode: 'always',
policyKey: '["always",1,10000,0]',
retry: 1,
delayMs: 2_000,
failure: { message: 'retry without limit', code: 'AUTH', status: 401 },