feat(llm): add scriptable mock fault server
This commit is contained in:
22 files changed
+1972
-1
No files matched your search
+6
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-25-scriptable-llm-wire-fault-server.md: 92f7d6aad8e7b4dc8bb08e98bb5847ff27470229
|
||||
2026-07-25-scriptable-llm-wire-fault-server.zh.md: 2f5fcc1321b0e4f501f3814e5e96d4b26cf18ec6
|
||||
@@ -0,0 +1,41 @@
|
||||
# Agent Note: Scriptable LLM wire fault server
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-07-25-scriptable-llm-wire-fault-server.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
Adapter unit tests use local HTTP servers to classify individual provider failures, while retry tests use an in-process scripted `LlmAdapter` to prove closed-step recovery. Neither boundary provides a reusable server for running the shipping HTTP adapter, agent loop, and retry policy together, and neither lets a developer point an existing app at deterministic transport faults by changing only its base URL and API key.
|
||||
|
||||
Connection refusal, a reset before the first event, clean EOF without `[DONE]`, a valid content-less completion, and a reset after partial output have different adapter and recovery outcomes. Treating them as one generic mock failure hides whether the provider boundary preserved the distinction and whether failed chunks remained outside committed model history.
|
||||
|
||||
## Decision
|
||||
|
||||
`@deepseek-ai/dsh-llm-mock-server` is a support package with an importable Node HTTP server and a standalone CLI. It accepts OpenAI-compatible root and `/v1` chat-completions paths, validates an optional bearer token, captures requests, and consumes one explicit behavior per accepted request. Script exhaustion fails loud; repetition requires `repeatLast`.
|
||||
|
||||
Request behaviors cover socket reset, post-header disconnect, partial disconnect, stall, valid empty completion, clean truncated streams, malformed payloads, representative HTTP failures, complete text/reasoning/tool-call responses, slow streaming, and max-token completion. A true `connection_refused` is a CLI listener-lifecycle phase because a bound request handler cannot refuse its own TCP connection.
|
||||
|
||||
The `random` script entry performs a new weighted selection for every request. The server exposes and logs its unsigned 32-bit seed, accepts caller-supplied relative weights, and ships a success-heavy stress profile that mixes transport, protocol, provider, timeout, and semantic-empty outcomes. The profile is configurable test pressure rather than an estimate of production incident frequency; `connection_refused` remains outside the request-level pool.
|
||||
|
||||
The server reports wire facts only and does not classify retryability. Real-composition tests route it through `dsh-llm-deepseek`, `dsh-agent-loop`, and `dsh-llm-retry`: connection refusal, hard disconnect, partial reset, and idle timeout recover under the existing default policy; a valid content-less completion succeeds without retry; clean partial EOF remains `STREAM_CLOSED` and is not retried by default. The package does not change those policies.
|
||||
|
||||
## Verification
|
||||
|
||||
Package tests exercise every request behavior, HTTP validation without script consumption, script exhaustion/repetition, stalled-connection teardown, CLI parsing, random seed reproducibility, weight validation, telemetry, lifecycle cleanup, and the invariant companion under the per-file coverage gate. The retry integration suite proves exact request counts, numbered retry steps, request-body identity, failed partial-chunk isolation, empty-success semantics, clean-EOF classification, timeout recovery, true refused-connection recovery after delayed listener startup, and bounded exhaustion through the real HTTP/SSE adapter.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Implement the server in Python** — rejected because Node's standard HTTP and socket APIs expose every required fault, while TypeScript keeps the server, CLI parser, tests, package build, lint, and coverage inside the repository's existing toolchain. A second runtime would add environment and subprocess dependencies without increasing wire isolation.
|
||||
|
||||
**Keep separate inline mock servers in adapter tests** — rejected because those fixtures cannot be launched by an existing app and would duplicate behavior sequencing, randomization, telemetry, and connection cleanup across suites. A support package gives tests a shared implementation without promoting it to product API.
|
||||
|
||||
**Use only an in-process `LlmAdapter` mock** — rejected because it bypasses fetch, HTTP status/header parsing, SSE framing, socket termination, and the adapter idle watchdog: the exact boundaries this test infrastructure exists to exercise.
|
||||
|
||||
**Change retry defaults with the server** — rejected because the server reveals existing semantics rather than deciding policy. Adding `STREAM_CLOSED` or semantic-empty recovery requires a separate decision with its own cost, latency, and duplicate-generation trade-offs.
|
||||
|
||||
## Consequences
|
||||
|
||||
Developers can reproduce fault sequences by changing only provider URL/key configuration, and automated tests can keep socket-level failures deterministic through explicit scripts and seeds. The same wire fixture now exposes gaps between hard resets, clean truncation, and successful empty completions without splicing attempts or modifying model history.
|
||||
|
||||
The server adds a support package, executable build entry, and behavior vocabulary that must remain compatible with both direct tests and CLI examples. Arrival-ordered scripts are intentionally shared across clients, random defaults are stress weights rather than operational truth, and exact connection refusal requires coordinating the client attempt with the pre-listen interval.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Agent Note: 可脚本控制的 LLM(大语言模型)协议层故障服务器
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-07-25-scriptable-llm-wire-fault-server.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
适配器单元测试使用本地 HTTP 服务器对各类提供方故障逐一分类,重试测试则使用进程内的脚本化 `LlmAdapter` 证明已关闭步骤的恢复能力。这两个边界都无法提供可复用的服务器,以便同时运行交付版本的 HTTP 适配器、agent loop(智能体循环)和重试策略;开发者也无法仅修改现有应用的 base URL 与 API key,就让应用连接到确定性的传输故障。
|
||||
|
||||
连接遭拒、首个事件前连接被重置、未收到 `[DONE]` 即正常 EOF、合法但无内容的完成,以及输出部分内容后连接被重置,会产生不同的适配器与恢复结果。把它们统一视为普通 mock 故障,会掩盖提供方边界是否保留了这些区别,以及失败请求的分片是否确实没有进入已提交的模型历史。
|
||||
|
||||
## 决策
|
||||
|
||||
`@deepseek-ai/dsh-llm-mock-server` 是一个支持包(package),提供可导入的 Node HTTP 服务器和独立 CLI(命令行界面)。它接受兼容 OpenAI 的根路径和 `/v1` chat-completions 路径,校验可选的 bearer token,捕获请求,并对每个已接受请求消耗一个显式行为。脚本耗尽时快速失败;只有设置 `repeatLast` 才会重复最后一个行为。
|
||||
|
||||
请求行为覆盖 socket 重置、发送 header 后断开、发送部分内容后断开、停滞、合法空完成、正常关闭但被截断的流、畸形 payload、典型 HTTP 故障、完整的文本/推理/工具调用响应、慢速流式输出以及达到 token 上限的完成。真正的 `connection_refused` 由 CLI 的监听器生命周期阶段实现,因为已经绑定端口的请求处理器无法拒绝自身的 TCP 连接。
|
||||
|
||||
脚本项 `random` 会为每个请求重新执行一次加权选择。服务器公开并记录其无符号 32 位 seed,允许调用方提供相对权重,并内置一套偏重成功结果的压力测试配置,将传输、协议、提供方、超时和语义空结果混合在一起。该配置用于提供可调的测试压力,并非对生产事故发生频率的估算;`connection_refused` 仍不进入请求级随机池。
|
||||
|
||||
服务器只报告协议层事实,不判断是否可重试。真实组合测试让请求依次经过 `dsh-llm-deepseek`、`dsh-agent-loop` 和 `dsh-llm-retry`:在现有默认策略下,连接遭拒、硬断开、部分输出后重置以及空闲超时均可恢复;合法的无内容完成无需重试即可成功;正常关闭的部分输出 EOF 仍归类为 `STREAM_CLOSED`,默认不重试。该包不会改变这些策略。
|
||||
|
||||
## 验证
|
||||
|
||||
包测试覆盖所有请求行为、不消耗脚本的 HTTP 校验、脚本耗尽与重复、停滞连接清理、CLI 解析、随机 seed 可复现性、权重校验、遥测、生命周期清理,以及逐文件覆盖率门禁下的配套不变式插件。重试集成套件通过真实 HTTP/SSE(Server-Sent Events)适配器,验证准确的请求次数、带编号的重试步骤、请求体完全一致、失败的部分分片不会泄漏、空完成成功语义、正常 EOF 分类、超时恢复、监听器延迟启动后从真实连接遭拒中恢复,以及有界重试耗尽。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
**使用 Python 实现服务器**:不予采纳。Node 的标准 HTTP 与 socket API 足以暴露所有所需故障,而 TypeScript 可以让服务器、CLI 解析器、测试、包构建、lint 和覆盖率全部留在仓库现有工具链中。引入第二套运行时会增加环境与子进程依赖,却不能增强协议隔离。
|
||||
|
||||
**在适配器测试中继续使用各自独立的内联 mock 服务器**:不予采纳。这些 fixture(测试前置数据)无法作为独立服务器启动并供现有应用连接,还会让不同测试套件重复实现行为编排、随机化、遥测和连接清理。支持包让测试共享同一套实现,又不会将其提升为产品 API。
|
||||
|
||||
**仅使用进程内的 `LlmAdapter` mock**:不予采纳。它会绕过 fetch、HTTP 状态与 header 解析、SSE 分帧、socket 终止以及适配器的空闲看门狗,而这正是这套测试基础设施要覆盖的边界。
|
||||
|
||||
**随服务器一起修改默认重试策略**:不予采纳。服务器用于揭示既有语义,而非决定策略。是否为 `STREAM_CLOSED` 或语义空结果增加恢复能力,需要单独决策,并权衡成本、延迟和重复生成风险。
|
||||
|
||||
## 后果
|
||||
|
||||
开发者只需修改提供方 URL/key 配置即可复现故障序列;自动化测试则可通过显式脚本和 seed,让 socket 层故障保持确定性。同一套协议 fixture 现在可以暴露硬重置、正常截断与成功空完成之间的差异,而不会拼接多次尝试的内容或修改模型历史。
|
||||
|
||||
服务器新增了一个支持包、可执行构建入口和行为词汇,三者必须同时兼容直接测试与 CLI 示例。按请求到达顺序执行的脚本有意由所有客户端共享;随机模式的默认值代表压力测试权重,而非实际运行规律;精确模拟连接遭拒时,需要让客户端尝试与监听开始前的时间区间协调一致。
|
||||
Reference in New Issue
Block a user