feat(llm): add scriptable mock fault server
This commit is contained in:
Yichen Jiang
+6
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-25-scriptable-llm-wire-fault-server.md: 92f7d6aad8e7b4dc8bb08e98bb5847ff27470229
|
||||
2026-07-25-scriptable-llm-wire-fault-server.zh.md: 2f5fcc1321b0e4f501f3814e5e96d4b26cf18ec6
|
||||
@@ -0,0 +1,41 @@
|
||||
# Agent Note: Scriptable LLM wire fault server
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-07-25-scriptable-llm-wire-fault-server.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
Adapter unit tests use local HTTP servers to classify individual provider failures, while retry tests use an in-process scripted `LlmAdapter` to prove closed-step recovery. Neither boundary provides a reusable server for running the shipping HTTP adapter, agent loop, and retry policy together, and neither lets a developer point an existing app at deterministic transport faults by changing only its base URL and API key.
|
||||
|
||||
Connection refusal, a reset before the first event, clean EOF without `[DONE]`, a valid content-less completion, and a reset after partial output have different adapter and recovery outcomes. Treating them as one generic mock failure hides whether the provider boundary preserved the distinction and whether failed chunks remained outside committed model history.
|
||||
|
||||
## Decision
|
||||
|
||||
`@deepseek-ai/dsh-llm-mock-server` is a support package with an importable Node HTTP server and a standalone CLI. It accepts OpenAI-compatible root and `/v1` chat-completions paths, validates an optional bearer token, captures requests, and consumes one explicit behavior per accepted request. Script exhaustion fails loud; repetition requires `repeatLast`.
|
||||
|
||||
Request behaviors cover socket reset, post-header disconnect, partial disconnect, stall, valid empty completion, clean truncated streams, malformed payloads, representative HTTP failures, complete text/reasoning/tool-call responses, slow streaming, and max-token completion. A true `connection_refused` is a CLI listener-lifecycle phase because a bound request handler cannot refuse its own TCP connection.
|
||||
|
||||
The `random` script entry performs a new weighted selection for every request. The server exposes and logs its unsigned 32-bit seed, accepts caller-supplied relative weights, and ships a success-heavy stress profile that mixes transport, protocol, provider, timeout, and semantic-empty outcomes. The profile is configurable test pressure rather than an estimate of production incident frequency; `connection_refused` remains outside the request-level pool.
|
||||
|
||||
The server reports wire facts only and does not classify retryability. Real-composition tests route it through `dsh-llm-deepseek`, `dsh-agent-loop`, and `dsh-llm-retry`: connection refusal, hard disconnect, partial reset, and idle timeout recover under the existing default policy; a valid content-less completion succeeds without retry; clean partial EOF remains `STREAM_CLOSED` and is not retried by default. The package does not change those policies.
|
||||
|
||||
## Verification
|
||||
|
||||
Package tests exercise every request behavior, HTTP validation without script consumption, script exhaustion/repetition, stalled-connection teardown, CLI parsing, random seed reproducibility, weight validation, telemetry, lifecycle cleanup, and the invariant companion under the per-file coverage gate. The retry integration suite proves exact request counts, numbered retry steps, request-body identity, failed partial-chunk isolation, empty-success semantics, clean-EOF classification, timeout recovery, true refused-connection recovery after delayed listener startup, and bounded exhaustion through the real HTTP/SSE adapter.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Implement the server in Python** — rejected because Node's standard HTTP and socket APIs expose every required fault, while TypeScript keeps the server, CLI parser, tests, package build, lint, and coverage inside the repository's existing toolchain. A second runtime would add environment and subprocess dependencies without increasing wire isolation.
|
||||
|
||||
**Keep separate inline mock servers in adapter tests** — rejected because those fixtures cannot be launched by an existing app and would duplicate behavior sequencing, randomization, telemetry, and connection cleanup across suites. A support package gives tests a shared implementation without promoting it to product API.
|
||||
|
||||
**Use only an in-process `LlmAdapter` mock** — rejected because it bypasses fetch, HTTP status/header parsing, SSE framing, socket termination, and the adapter idle watchdog: the exact boundaries this test infrastructure exists to exercise.
|
||||
|
||||
**Change retry defaults with the server** — rejected because the server reveals existing semantics rather than deciding policy. Adding `STREAM_CLOSED` or semantic-empty recovery requires a separate decision with its own cost, latency, and duplicate-generation trade-offs.
|
||||
|
||||
## Consequences
|
||||
|
||||
Developers can reproduce fault sequences by changing only provider URL/key configuration, and automated tests can keep socket-level failures deterministic through explicit scripts and seeds. The same wire fixture now exposes gaps between hard resets, clean truncation, and successful empty completions without splicing attempts or modifying model history.
|
||||
|
||||
The server adds a support package, executable build entry, and behavior vocabulary that must remain compatible with both direct tests and CLI examples. Arrival-ordered scripts are intentionally shared across clients, random defaults are stress weights rather than operational truth, and exact connection refusal requires coordinating the client attempt with the pre-listen interval.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Agent Note: 可脚本控制的 LLM（大语言模型）协议层故障服务器
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-07-25-scriptable-llm-wire-fault-server.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
适配器单元测试使用本地 HTTP 服务器对各类提供方故障逐一分类，重试测试则使用进程内的脚本化 `LlmAdapter` 证明已关闭步骤的恢复能力。这两个边界都无法提供可复用的服务器，以便同时运行交付版本的 HTTP 适配器、agent loop（智能体循环）和重试策略；开发者也无法仅修改现有应用的 base URL 与 API key，就让应用连接到确定性的传输故障。
|
||||
|
||||
连接遭拒、首个事件前连接被重置、未收到 `[DONE]` 即正常 EOF、合法但无内容的完成，以及输出部分内容后连接被重置，会产生不同的适配器与恢复结果。把它们统一视为普通 mock 故障，会掩盖提供方边界是否保留了这些区别，以及失败请求的分片是否确实没有进入已提交的模型历史。
|
||||
|
||||
## 决策
|
||||
|
||||
`@deepseek-ai/dsh-llm-mock-server` 是一个支持包（package），提供可导入的 Node HTTP 服务器和独立 CLI（命令行界面）。它接受兼容 OpenAI 的根路径和 `/v1` chat-completions 路径，校验可选的 bearer token，捕获请求，并对每个已接受请求消耗一个显式行为。脚本耗尽时快速失败；只有设置 `repeatLast` 才会重复最后一个行为。
|
||||
|
||||
请求行为覆盖 socket 重置、发送 header 后断开、发送部分内容后断开、停滞、合法空完成、正常关闭但被截断的流、畸形 payload、典型 HTTP 故障、完整的文本/推理/工具调用响应、慢速流式输出以及达到 token 上限的完成。真正的 `connection_refused` 由 CLI 的监听器生命周期阶段实现，因为已经绑定端口的请求处理器无法拒绝自身的 TCP 连接。
|
||||
|
||||
脚本项 `random` 会为每个请求重新执行一次加权选择。服务器公开并记录其无符号 32 位 seed，允许调用方提供相对权重，并内置一套偏重成功结果的压力测试配置，将传输、协议、提供方、超时和语义空结果混合在一起。该配置用于提供可调的测试压力，并非对生产事故发生频率的估算；`connection_refused` 仍不进入请求级随机池。
|
||||
|
||||
服务器只报告协议层事实，不判断是否可重试。真实组合测试让请求依次经过 `dsh-llm-deepseek`、`dsh-agent-loop` 和 `dsh-llm-retry`：在现有默认策略下，连接遭拒、硬断开、部分输出后重置以及空闲超时均可恢复；合法的无内容完成无需重试即可成功；正常关闭的部分输出 EOF 仍归类为 `STREAM_CLOSED`，默认不重试。该包不会改变这些策略。
|
||||
|
||||
## 验证
|
||||
|
||||
包测试覆盖所有请求行为、不消耗脚本的 HTTP 校验、脚本耗尽与重复、停滞连接清理、CLI 解析、随机 seed 可复现性、权重校验、遥测、生命周期清理，以及逐文件覆盖率门禁下的配套不变式插件。重试集成套件通过真实 HTTP/SSE（Server-Sent Events）适配器，验证准确的请求次数、带编号的重试步骤、请求体完全一致、失败的部分分片不会泄漏、空完成成功语义、正常 EOF 分类、超时恢复、监听器延迟启动后从真实连接遭拒中恢复，以及有界重试耗尽。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
**使用 Python 实现服务器**：不予采纳。Node 的标准 HTTP 与 socket API 足以暴露所有所需故障，而 TypeScript 可以让服务器、CLI 解析器、测试、包构建、lint 和覆盖率全部留在仓库现有工具链中。引入第二套运行时会增加环境与子进程依赖，却不能增强协议隔离。
|
||||
|
||||
**在适配器测试中继续使用各自独立的内联 mock 服务器**：不予采纳。这些 fixture（测试前置数据）无法作为独立服务器启动并供现有应用连接，还会让不同测试套件重复实现行为编排、随机化、遥测和连接清理。支持包让测试共享同一套实现，又不会将其提升为产品 API。
|
||||
|
||||
**仅使用进程内的 `LlmAdapter` mock**：不予采纳。它会绕过 fetch、HTTP 状态与 header 解析、SSE 分帧、socket 终止以及适配器的空闲看门狗，而这正是这套测试基础设施要覆盖的边界。
|
||||
|
||||
**随服务器一起修改默认重试策略**：不予采纳。服务器用于揭示既有语义，而非决定策略。是否为 `STREAM_CLOSED` 或语义空结果增加恢复能力，需要单独决策，并权衡成本、延迟和重复生成风险。
|
||||
|
||||
## 后果
|
||||
|
||||
开发者只需修改提供方 URL/key 配置即可复现故障序列；自动化测试则可通过显式脚本和 seed，让 socket 层故障保持确定性。同一套协议 fixture 现在可以暴露硬重置、正常截断与成功空完成之间的差异，而不会拼接多次尝试的内容或修改模型历史。
|
||||
|
||||
服务器新增了一个支持包、可执行构建入口和行为词汇，三者必须同时兼容直接测试与 CLI 示例。按请求到达顺序执行的脚本有意由所有客户端共享；随机模式的默认值代表压力测试权重，而非实际运行规律；精确模拟连接遭拒时，需要让客户端尝试与监听开始前的时间区间协调一致。
|
||||
@@ -1953,6 +1953,7 @@ Imported as libraries by other packages; a `cordis.yml` cannot load them.
|
||||
- `@deepseek-ai/dsh-host-runtime` ([`packages/host/runtime/src/index.ts`](../packages/host/runtime/src/index.ts))
|
||||
- `@deepseek-ai/dsh-host-webserver` ([`packages/host/webserver/src/index.ts`](../packages/host/webserver/src/index.ts))
|
||||
- `@deepseek-ai/dsh-jsonrpc-demo` ([`packages/examples/jsonrpc-demo/src/index.ts`](../packages/examples/jsonrpc-demo/src/index.ts))
|
||||
- `@deepseek-ai/dsh-llm-mock-server` ([`packages/support/llm-mock-server/src/index.ts`](../packages/support/llm-mock-server/src/index.ts))
|
||||
- `@deepseek-ai/dsh-loader-smoke` ([`packages/support/loader-smoke/src/index.ts`](../packages/support/loader-smoke/src/index.ts))
|
||||
- `@deepseek-ai/dsh-paths` ([`packages/util/paths/src/index.ts`](../packages/util/paths/src/index.ts))
|
||||
- `@deepseek-ai/dsh-retention` ([`packages/util/retention/src/index.ts`](../packages/util/retention/src/index.ts))
|
||||
|
||||
@@ -95,6 +95,7 @@
|
||||
"demo:cordis": "node --import tsx packages/examples/tui-demo/src/bin.ts examples/cordis-agent/cordis.yml",
|
||||
"demo:acp": "node --import tsx packages/examples/acp-demo/src/bin.ts --config examples/acp-agent/cordis.yml",
|
||||
"demo:web": "npm run build && npm run build:web && node --import tsx apps/cli/src/bin.ts web",
|
||||
"mock:llm": "node --import tsx packages/support/llm-mock-server/src/bin.ts",
|
||||
"dev:web": "tsx scripts/dev-web.ts --poll",
|
||||
"postinstall": "node scripts/install-lefthook.mjs"
|
||||
},
|
||||
|
||||
@@ -41,8 +41,11 @@
|
||||
"@cordisjs/plugin-loader": "workspace:^",
|
||||
"@deepseek-ai/dsh-agent": "workspace:^",
|
||||
"@deepseek-ai/dsh-agent-loop": "workspace:^",
|
||||
"@deepseek-ai/dsh-agent-loop-testkit": "workspace:^",
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm-deepseek": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm-mock-server": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence-jsonl": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence-sqlite": "workspace:^",
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
import { createServer } from 'node:http'
|
||||
import type { AddressInfo } from 'node:net'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import type { Agent } from '@deepseek-ai/dsh-agent'
|
||||
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
|
||||
import { mountAgentLoopTestDependencies } from '@deepseek-ai/dsh-agent-loop-testkit'
|
||||
import * as LlmDeepSeek from '@deepseek-ai/dsh-llm-deepseek'
|
||||
import type { MockLlmBehavior, MockLlmServer } from '@deepseek-ai/dsh-llm-mock-server'
|
||||
import { startMockLlmServer } from '@deepseek-ai/dsh-llm-mock-server'
|
||||
import { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import * as Retry from '../src/index.ts'
|
||||
|
||||
let context: Context | undefined
|
||||
const servers: MockLlmServer[] = []
|
||||
|
||||
afterEach(async () => {
|
||||
await context?.fiber.dispose()
|
||||
context = undefined
|
||||
await Promise.all(servers.splice(0).map(server => server.close()))
|
||||
})
|
||||
|
||||
async function start(
|
||||
sequence: readonly MockLlmBehavior[],
|
||||
options: Omit<Parameters<typeof startMockLlmServer>[0], 'sequence'> = {},
|
||||
): Promise<MockLlmServer> {
|
||||
const server = await startMockLlmServer({ sequence, ...options })
|
||||
servers.push(server)
|
||||
return server
|
||||
}
|
||||
|
||||
async function harness(
|
||||
baseURL: string,
|
||||
options: { streamIdleTimeoutMs?: number; initialDelayMs?: number } = {},
|
||||
): Promise<Context> {
|
||||
const ctx = new Context()
|
||||
await mountAgentLoopTestDependencies(ctx)
|
||||
await ctx.plugin(LlmDeepSeek, {
|
||||
apiKey: 'mock-key',
|
||||
baseURL,
|
||||
streamIdleTimeoutMs: options.streamIdleTimeoutMs ?? 1_000,
|
||||
})
|
||||
await ctx.plugin(Retry, {
|
||||
maxTransientRetries: 2,
|
||||
initialDelayMs: options.initialDelayMs ?? 10,
|
||||
maxDelayMs: options.initialDelayMs ?? 10,
|
||||
jitterRatio: 0,
|
||||
})
|
||||
await ctx.plugin(AgentLoop, { agents: [] })
|
||||
return ctx
|
||||
}
|
||||
|
||||
function waitForIdle(ctx: Context, agent: Agent): Promise<void> {
|
||||
return new Promise((resolve) => {
|
||||
const dispose = ctx.on('agent/status', (subject, status) => {
|
||||
if (subject !== agent || status !== 'idle') return
|
||||
dispose()
|
||||
resolve()
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
function sendAndWait(ctx: Context, agent: Agent): Promise<void> {
|
||||
const idle = waitForIdle(ctx, agent)
|
||||
agent.followup([{ type: 'text', text: 'recover through the provider boundary' }])
|
||||
return idle
|
||||
}
|
||||
|
||||
function finalAssistantText(agent: Agent): string | undefined {
|
||||
const message = agent.session.deriveMessages().at(-1)
|
||||
if (message?.role !== 'assistant') return undefined
|
||||
return message.content
|
||||
.filter(block => block.type === 'text')
|
||||
.map(block => block.text)
|
||||
.join('')
|
||||
}
|
||||
|
||||
async function unusedPort(): Promise<number> {
|
||||
const server = createServer()
|
||||
await new Promise<void>((resolve) => { server.listen(0, '127.0.0.1', resolve) })
|
||||
const port = (server.address() as AddressInfo).port
|
||||
await new Promise<void>((resolve) => { server.close(() => { resolve() }) })
|
||||
return port
|
||||
}
|
||||
|
||||
describe('bounded retry through the real DeepSeek HTTP/SSE adapter', () => {
|
||||
it('recovers from a true refused connection after the endpoint starts during backoff', async () => {
|
||||
const port = await unusedPort()
|
||||
context = await harness(`http://127.0.0.1:${port}`, { initialDelayMs: 100 })
|
||||
const agent = context.agentLoop.create(SessionId('wire-refused'), {
|
||||
provider: 'deepseek',
|
||||
model: 'mock-model',
|
||||
})
|
||||
let recoveryServer: Promise<MockLlmServer> | undefined
|
||||
context.on('session/event', (session, event) => {
|
||||
if (session !== agent.session || event.type !== 'llm/retry' || event.data.retry !== 1) return
|
||||
recoveryServer = start(['success'], { port, apiKey: 'mock-key', successText: 'connected after retry' })
|
||||
})
|
||||
|
||||
await sendAndWait(context, agent)
|
||||
const server = await recoveryServer
|
||||
|
||||
expect(server).toBeDefined()
|
||||
expect(server?.requests).toHaveLength(1)
|
||||
expect(agent.session.events.filter(event => event.type === 'step/start').map(event => event.data.step))
|
||||
.toEqual([1, 2])
|
||||
expect(agent.session.events.filter(event => event.type === 'llm/retry').map(event => event.data.failure.code))
|
||||
.toEqual(['TRANSPORT'])
|
||||
expect(finalAssistantText(agent)).toBe('connected after retry')
|
||||
})
|
||||
|
||||
it.each([
|
||||
['stream_disconnect', 0] as const,
|
||||
['partial_disconnect', 2] as const,
|
||||
])('retries %s without committing failed chunks', async (behavior, failedChunkCount) => {
|
||||
const server = await start([behavior, 'success'], {
|
||||
apiKey: 'mock-key',
|
||||
partialText: 'discard me',
|
||||
chunkSize: 100,
|
||||
disconnectDelayMs: 20,
|
||||
successText: 'recovered response',
|
||||
})
|
||||
context = await harness(server.baseURL)
|
||||
const agent = context.agentLoop.create(SessionId(`wire-${behavior}`), {
|
||||
provider: 'deepseek',
|
||||
model: 'mock-model',
|
||||
})
|
||||
|
||||
await sendAndWait(context, agent)
|
||||
|
||||
expect(server.requests).toHaveLength(2)
|
||||
expect(server.requests[0]?.body).toEqual(server.requests[1]?.body)
|
||||
expect(agent.session.events.filter(event =>
|
||||
event.type === 'assistant/chunk' && event.data.step === 1,
|
||||
)).toHaveLength(failedChunkCount)
|
||||
expect(agent.session.events.filter(event => event.type === 'assistant/message').map(event => event.data.step))
|
||||
.toEqual([2])
|
||||
expect(agent.session.events.filter(event => event.type === 'llm/retry').map(event => event.data.failure.code))
|
||||
.toEqual(['TRANSPORT'])
|
||||
expect(finalAssistantText(agent)).toBe('recovered response')
|
||||
})
|
||||
|
||||
it('treats a wire-valid content-less completion as success without retrying', async () => {
|
||||
const server = await start(['empty', 'success'], { apiKey: 'mock-key' })
|
||||
context = await harness(server.baseURL)
|
||||
const agent = context.agentLoop.create(SessionId('wire-empty'), {
|
||||
provider: 'deepseek',
|
||||
model: 'mock-model',
|
||||
})
|
||||
|
||||
await sendAndWait(context, agent)
|
||||
|
||||
expect(server.requests).toHaveLength(1)
|
||||
expect(agent.session.events.some(event => event.type === 'llm/retry')).toBe(false)
|
||||
expect(agent.session.events.find(event => event.type === 'assistant/message')).toMatchObject({
|
||||
data: { turn: 1, step: 1, content: [] },
|
||||
})
|
||||
expect(agent.session.events.at(-1)).toMatchObject({
|
||||
type: 'turn/end',
|
||||
data: { reason: { kind: 'completed' } },
|
||||
})
|
||||
expect(finalAssistantText(agent)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('exposes a clean partial EOF as non-default-retryable STREAM_CLOSED', async () => {
|
||||
const server = await start(['partial_eof', 'success'], {
|
||||
apiKey: 'mock-key',
|
||||
partialText: 'discarded clean eof',
|
||||
chunkSize: 100,
|
||||
})
|
||||
context = await harness(server.baseURL)
|
||||
const agent = context.agentLoop.create(SessionId('wire-partial-eof'), {
|
||||
provider: 'deepseek',
|
||||
model: 'mock-model',
|
||||
})
|
||||
|
||||
await sendAndWait(context, agent)
|
||||
|
||||
expect(server.requests).toHaveLength(1)
|
||||
expect(agent.session.events.filter(event =>
|
||||
event.type === 'assistant/chunk' && event.data.step === 1,
|
||||
)).toHaveLength(2)
|
||||
expect(agent.session.events.some(event => event.type === 'assistant/message')).toBe(false)
|
||||
expect(agent.session.events.some(event => event.type === 'llm/retry')).toBe(false)
|
||||
expect(agent.session.events.at(-1)).toMatchObject({
|
||||
type: 'turn/end',
|
||||
data: { reason: { kind: 'error', failure: { code: 'STREAM_CLOSED' } } },
|
||||
})
|
||||
})
|
||||
|
||||
it('turns a stalled body into TIMEOUT and succeeds on the next request', async () => {
|
||||
const server = await start(['stall', 'success'], {
|
||||
apiKey: 'mock-key',
|
||||
successText: 'recovered after timeout',
|
||||
})
|
||||
context = await harness(server.baseURL, { streamIdleTimeoutMs: 30 })
|
||||
const agent = context.agentLoop.create(SessionId('wire-stall'), {
|
||||
provider: 'deepseek',
|
||||
model: 'mock-model',
|
||||
})
|
||||
|
||||
await sendAndWait(context, agent)
|
||||
|
||||
expect(server.requests.map(record => record.behavior)).toEqual(['stall', 'success'])
|
||||
expect(agent.session.events.filter(event => event.type === 'llm/retry').map(event => event.data.failure.code))
|
||||
.toEqual(['TIMEOUT'])
|
||||
expect(finalAssistantText(agent)).toBe('recovered after timeout')
|
||||
})
|
||||
|
||||
it('stops after the configured transport retry budget is exhausted', async () => {
|
||||
const server = await start(['connection_reset', 'connection_reset', 'connection_reset'], {
|
||||
apiKey: 'mock-key',
|
||||
})
|
||||
context = await harness(server.baseURL)
|
||||
const agent = context.agentLoop.create(SessionId('wire-exhausted'), {
|
||||
provider: 'deepseek',
|
||||
model: 'mock-model',
|
||||
})
|
||||
|
||||
await sendAndWait(context, agent)
|
||||
|
||||
expect(server.requests).toHaveLength(3)
|
||||
expect(agent.session.events.filter(event => event.type === 'step/start')).toHaveLength(3)
|
||||
expect(agent.session.events.filter(event => event.type === 'llm/retry')).toHaveLength(2)
|
||||
expect(agent.session.events.at(-1)).toMatchObject({
|
||||
type: 'turn/end',
|
||||
data: { reason: { kind: 'error', failure: { code: 'TRANSPORT' } } },
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -8,6 +8,7 @@ Packages that exist to serve development, testing, and the examples rather than
|
||||
| `agent-loop-testkit/` | Shared prerequisite mounting for tests that exercise the concrete agent loop | (library — imported by AgentLoop integration tests) |
|
||||
| `invariants/` | Runtime event-contract assertions for development diagnostics | (listens on `session/*`, `agent/*`) |
|
||||
| `loader-smoke/` | Shared real-Loader subprocess harness for keyless example smokes | (library — imported by example e2e suites) |
|
||||
| `llm-mock-server/` | Scriptable OpenAI-compatible HTTP/SSE fault server + CLI for LLM recovery tests | (standalone server and test library) |
|
||||
| `llm-replay/` | Record/replay adapter: short-circuits `llm/stream` from a recorded session JSONL (keyless snapshot tests) | (listens on `llm/stream`) |
|
||||
|
||||
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `agent-loop-testkit` centralizes the mandatory service spine for hand-built AgentLoop tests without owning their loop or scenario. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the ACP subprocess/client boundary plus the snapshot harness, normalizers, and suite machinery, while `loader-smoke` owns the parallel real-Loader launch boundary used by keyless example e2e suites. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
|
||||
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `agent-loop-testkit` centralizes the mandatory service spine for hand-built AgentLoop tests without owning their loop or scenario. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate, while `llm-mock-server` drives real provider adapters through deterministic HTTP/SSE faults. `acp-snapshot` carries the ACP subprocess/client boundary plus the snapshot harness, normalizers, and suite machinery, while `loader-smoke` owns the parallel real-Loader launch boundary used by keyless example e2e suites. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
|
||||
@@ -0,0 +1,84 @@
|
||||
# `@deepseek-ai/dsh-llm-mock-server`
|
||||
|
||||
A scriptable OpenAI-compatible HTTP/SSE server for exercising real LLM adapters, the agent loop, and recovery policy without a provider key. It accepts `POST /chat/completions` and `POST /v1/chat/completions`; each accepted request consumes one configured behavior in arrival order. Invalid methods, paths, bearer tokens, and JSON do not consume the script.
|
||||
|
||||
The library entry exports `startMockLlmServer(options)`, behavior and telemetry types, the default random stress weights, and a running handle with the bound `baseURL`, generated or configured `randomSeed`, captured requests, and idempotent `close()`. Closing force-terminates stalled connections.
|
||||
|
||||
## Standalone use
|
||||
|
||||
Run the source entry from this repository:
|
||||
|
||||
```sh
|
||||
pnpm run mock:llm -- \
|
||||
--port 8000 \
|
||||
--api-key mock-key \
|
||||
--sequence partial_disconnect,success \
|
||||
--partial-text "discard this half"
|
||||
```
|
||||
|
||||
Point the shipping DeepSeek adapter at the server; it appends `/chat/completions` to the configured base:
|
||||
|
||||
```sh
|
||||
DEEPSEEK_BASE_URL=http://127.0.0.1:8000/v1 \
|
||||
DEEPSEEK_API_KEY=mock-key \
|
||||
pnpm run demo:headless "test provider recovery"
|
||||
```
|
||||
|
||||
The built package also exposes `dsh-llm-mock-server`. Stdout is JSONL: a `ready` record carries the `/v1` base URL and random seed, followed by request/result records that name both the scripted behavior and the concrete selected behavior.
|
||||
|
||||
## Behavior script
|
||||
|
||||
`--sequence` is a comma-separated FIFO. Exhaustion returns a structured HTTP 500; `--repeat-last` explicitly reuses the last entry.
|
||||
|
||||
| Behavior | Wire result |
|
||||
|---|---|
|
||||
| `connection_reset` | Destroy the socket before HTTP headers |
|
||||
| `stream_disconnect` | Send SSE headers, then reset before the first event |
|
||||
| `partial_disconnect` | Send text deltas, then reset the socket |
|
||||
| `stall` | Send SSE headers and remain idle until client/server cancellation |
|
||||
| `empty` | Send a valid content-less stop and `[DONE]` |
|
||||
| `empty_body` / `stream_eof` / `partial_eof` | End cleanly without the required `[DONE]` boundary |
|
||||
| `malformed_json` / `malformed_event` | Send invalid SSE JSON or an invalid provider chunk shape |
|
||||
| `rate_limit` / `server_error` / `service_unavailable` | Return retry-oriented 429/500/503 JSON errors |
|
||||
| `auth_error` / `invalid_request` / `context_overflow` / `quota_exceeded` | Return terminal or separately recovered provider errors |
|
||||
| `success` / `slow_success` / `reasoning_success` | Stream a complete text response, optionally delayed or preceded by reasoning |
|
||||
| `tool_call_success` / `max_tokens` | Complete with a tool call or `length` finish |
|
||||
| `wrong_content_type` | Send a valid SSE body under `application/json` |
|
||||
| `random` | Select a concrete request behavior from weighted seeded randomness |
|
||||
|
||||
`connection_refused` is CLI-only and must be the first entry. It delays binding a caller-specified nonzero port, so requests during `--listen-delay-ms` receive a real TCP refusal; the remaining entries begin after the listener starts.
|
||||
|
||||
## Random mode
|
||||
|
||||
Use a repeating `random` entry for an open-ended mixed run:
|
||||
|
||||
```sh
|
||||
pnpm run mock:llm -- \
|
||||
--port 8000 \
|
||||
--sequence random \
|
||||
--repeat-last \
|
||||
--seed 42 \
|
||||
--random-weights 'success=60,slow_success=10,connection_reset=5,stream_disconnect=5,partial_disconnect=10,empty=5,server_error=5'
|
||||
```
|
||||
|
||||
Omitting `--seed` generates one and prints it in the `ready` record. `--random-weights` accepts non-negative relative `behavior=weight` entries and requires at least one positive concrete behavior. The exported default is a success-heavy stress profile containing reset, disconnect, partial output, empty completion, stall, 429/5xx, clean truncation, and malformed JSON; it is test pressure, not an estimate of production incident frequency. `connection_refused` is excluded because a bound request handler cannot produce a true refusal.
|
||||
|
||||
When random weights include `stall`, configure the client under test with a short stream-idle timeout so the scenario terminates promptly.
|
||||
|
||||
## Timing and content controls
|
||||
|
||||
The CLI exposes `--success-text`, `--partial-text`, `--reasoning-text`, `--chunk-size`, `--chunk-delay-ms`, `--disconnect-delay-ms`, `--retry-after-ms`, `--request-id`, `--tool-name`, and `--tool-arguments`. The library accepts the same camel-case options. An optional exact `apiKey` validates `Authorization: Bearer <token>`; omission accepts any token.
|
||||
|
||||
## Model Experience
|
||||
|
||||
None, as this test server substitutes provider wire behavior without invoking a real model.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
None; requests terminate locally and never reach a provider cache.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Random weights model test pressure, not production incidence** — callers that want an environment-specific distribution must provide measured weights and record the emitted seed.
|
||||
- **Request scripts are arrival-ordered** — concurrent callers share one cursor, so deterministic per-session fault assignment requires separate server instances.
|
||||
- **True connection refusal is a listener lifecycle phase** — the CLI delay must overlap the client attempt; request-level random selection can only reset an accepted connection.
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-llm-mock-server",
|
||||
"description": "Scriptable OpenAI-compatible HTTP/SSE fault server for LLM recovery tests",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"bin": {
|
||||
"dsh-llm-mock-server": "lib/bin.js"
|
||||
},
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./invariant": {
|
||||
"types": "./lib/types/invariant.d.ts",
|
||||
"default": "./lib/invariant.js"
|
||||
},
|
||||
"./bin": {
|
||||
"types": "./lib/types/bin.d.ts",
|
||||
"default": "./lib/bin.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/bin.js",
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/types/**/*.d.ts.map",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Standalone process wrapper for the scriptable mock LLM server.
|
||||
* @module @deepseek-ai/dsh-llm-mock-server/bin
|
||||
*/
|
||||
|
||||
import { setTimeout as delay } from 'node:timers/promises'
|
||||
import { MOCK_LLM_CLI_USAGE, parseMockLlmCliArgs } from './cli.ts'
|
||||
import { startMockLlmServer } from './index.ts'
|
||||
|
||||
/* v8 ignore start -- thin process/signal glue; parser and server behavior are covered directly */
|
||||
try {
|
||||
const parsed = parseMockLlmCliArgs(process.argv.slice(2))
|
||||
if (parsed.kind === 'help') {
|
||||
process.stdout.write(MOCK_LLM_CLI_USAGE)
|
||||
} else {
|
||||
const { server: serverOptions, listenDelayMs, startsUnavailable } = parsed.config
|
||||
const host = serverOptions.host ?? '127.0.0.1'
|
||||
const port = serverOptions.port ?? 8_000
|
||||
if (startsUnavailable) {
|
||||
process.stdout.write(`${JSON.stringify({
|
||||
type: 'unavailable',
|
||||
baseURL: `http://${host}:${port}/v1`,
|
||||
listenDelayMs,
|
||||
})}\n`)
|
||||
await delay(listenDelayMs)
|
||||
}
|
||||
const server = await startMockLlmServer({
|
||||
...serverOptions,
|
||||
onEvent: (event) => { process.stdout.write(`${JSON.stringify(event)}\n`) },
|
||||
})
|
||||
process.stdout.write(`${JSON.stringify({
|
||||
type: 'ready',
|
||||
baseURL: `${server.baseURL}/v1`,
|
||||
randomSeed: server.randomSeed,
|
||||
})}\n`)
|
||||
let closing = false
|
||||
const close = (code: number): void => {
|
||||
if (closing) return
|
||||
closing = true
|
||||
void server.close().finally(() => { process.exit(code) })
|
||||
}
|
||||
process.on('SIGINT', () => { close(130) })
|
||||
process.on('SIGTERM', () => { close(143) })
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n\n${MOCK_LLM_CLI_USAGE}`)
|
||||
process.exitCode = 1
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
@@ -0,0 +1,212 @@
|
||||
/**
|
||||
* Dependency-free CLI parsing for the standalone mock LLM server.
|
||||
* @module @deepseek-ai/dsh-llm-mock-server/cli
|
||||
*/
|
||||
|
||||
import { MOCK_LLM_BEHAVIORS } from './index.ts'
|
||||
import type {
|
||||
ConcreteMockLlmBehavior,
|
||||
MockLlmBehavior,
|
||||
MockLlmRandomWeights,
|
||||
MockLlmServerOptions,
|
||||
} from './index.ts'
|
||||
|
||||
/** Listener lifecycle behavior understood only by the standalone CLI. */
|
||||
export const CONNECTION_REFUSED_BEHAVIOR = 'connection_refused'
|
||||
|
||||
/** Parsed CLI configuration, including a pre-listen unavailable interval. */
|
||||
export interface MockLlmCliConfig {
|
||||
/** Server options after removing the lifecycle-only `connection_refused` entry. */
|
||||
readonly server: MockLlmServerOptions
|
||||
/** Delay before binding the model port; zero starts immediately. */
|
||||
readonly listenDelayMs: number
|
||||
/** Whether the original sequence requested a true pre-listen refusal phase. */
|
||||
readonly startsUnavailable: boolean
|
||||
}
|
||||
|
||||
/** Result of parsing `dsh-llm-mock-server` arguments. */
|
||||
export type MockLlmCliParseResult =
|
||||
| { readonly kind: 'help' }
|
||||
| { readonly kind: 'run'; readonly config: MockLlmCliConfig }
|
||||
|
||||
const BEHAVIORS = new Set<string>(MOCK_LLM_BEHAVIORS)
|
||||
const DEFAULT_LISTEN_DELAY_MS = 750
|
||||
|
||||
/** Command usage written for `--help` and invalid arguments. */
|
||||
export const MOCK_LLM_CLI_USAGE = `Usage: dsh-llm-mock-server [options]
|
||||
|
||||
Required:
|
||||
--sequence <a,b,...> Ordered behaviors; connection_refused is allowed first
|
||||
|
||||
Listener:
|
||||
--host <host> Default 127.0.0.1
|
||||
--port <port> Default 8000; required and nonzero for connection_refused
|
||||
--api-key <token> Validate exact Bearer token when present
|
||||
--listen-delay-ms <ms> Unavailable interval (default 750 with connection_refused)
|
||||
--repeat-last Repeat the final request behavior after exhaustion
|
||||
--seed <uint32> Reproduce random selections
|
||||
--random-weights <a=n,...> Relative weights for concrete behaviors
|
||||
|
||||
Response:
|
||||
--success-text <text>
|
||||
--partial-text <text>
|
||||
--reasoning-text <text>
|
||||
--chunk-size <count>
|
||||
--chunk-delay-ms <ms>
|
||||
--disconnect-delay-ms <ms>
|
||||
--retry-after-ms <ms>
|
||||
--request-id <id>
|
||||
--tool-name <name>
|
||||
--tool-arguments <json>
|
||||
|
||||
Other:
|
||||
--help
|
||||
`
|
||||
|
||||
function optionValue(argv: readonly string[], index: number, option: string): string {
|
||||
const value = argv[index + 1]
|
||||
if (value === undefined || value.startsWith('--')) {
|
||||
throw new Error(`dsh-llm-mock-server: ${option} requires a value`)
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
function numberValue(option: string, value: string): number {
|
||||
const parsed = Number(value)
|
||||
if (!Number.isFinite(parsed)) throw new Error(`dsh-llm-mock-server: ${option} must be a finite number`)
|
||||
return parsed
|
||||
}
|
||||
|
||||
function parseSequence(raw: string): { startsUnavailable: boolean; sequence: MockLlmBehavior[] } {
|
||||
const entries = raw.split(',').map(entry => entry.trim())
|
||||
if (entries.some(entry => entry.length === 0)) {
|
||||
throw new Error('dsh-llm-mock-server: --sequence must contain non-empty comma-separated behaviors')
|
||||
}
|
||||
const startsUnavailable = entries[0] === CONNECTION_REFUSED_BEHAVIOR
|
||||
if (entries.slice(1).includes(CONNECTION_REFUSED_BEHAVIOR)) {
|
||||
throw new Error('dsh-llm-mock-server: connection_refused is allowed only as the first behavior')
|
||||
}
|
||||
const requestEntries = startsUnavailable ? entries.slice(1) : entries
|
||||
if (requestEntries.length === 0) {
|
||||
throw new Error('dsh-llm-mock-server: connection_refused must be followed by a request behavior')
|
||||
}
|
||||
for (const entry of requestEntries) {
|
||||
if (!BEHAVIORS.has(entry)) throw new Error(`dsh-llm-mock-server: unknown behavior ${JSON.stringify(entry)}`)
|
||||
}
|
||||
return { startsUnavailable, sequence: requestEntries as MockLlmBehavior[] }
|
||||
}
|
||||
|
||||
function parseRandomWeights(raw: string): MockLlmRandomWeights {
|
||||
const weights: MockLlmRandomWeights = {}
|
||||
for (const entry of raw.split(',')) {
|
||||
const [behavior, rawWeight, ...extra] = entry.split('=')
|
||||
if (behavior === undefined || behavior === '' || rawWeight === undefined || rawWeight === '' || extra.length > 0) {
|
||||
throw new Error('dsh-llm-mock-server: --random-weights expects behavior=weight comma-separated entries')
|
||||
}
|
||||
if (!BEHAVIORS.has(behavior) || behavior === 'random') {
|
||||
throw new Error(`dsh-llm-mock-server: random weight requires a concrete behavior, got ${JSON.stringify(behavior)}`)
|
||||
}
|
||||
if (Object.hasOwn(weights, behavior)) {
|
||||
throw new Error(`dsh-llm-mock-server: duplicate random weight for ${JSON.stringify(behavior)}`)
|
||||
}
|
||||
weights[behavior as ConcreteMockLlmBehavior] = numberValue('--random-weights', rawWeight)
|
||||
}
|
||||
return weights
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse standalone server arguments without starting a process or listener.
|
||||
* @param argv - arguments after the executable name.
|
||||
* @returns help or validated run configuration.
|
||||
*/
|
||||
export function parseMockLlmCliArgs(argv: readonly string[]): MockLlmCliParseResult {
|
||||
if (argv.includes('--help')) return { kind: 'help' }
|
||||
|
||||
let sequenceRaw: string | undefined
|
||||
let host: string | undefined
|
||||
let port = 8_000
|
||||
let apiKey: string | undefined
|
||||
let listenDelayMs: number | undefined
|
||||
let repeatLast = false
|
||||
let randomSeed: number | undefined
|
||||
let randomWeights: MockLlmRandomWeights | undefined
|
||||
let successText: string | undefined
|
||||
let partialText: string | undefined
|
||||
let reasoningText: string | undefined
|
||||
let chunkSize: number | undefined
|
||||
let chunkDelayMs: number | undefined
|
||||
let disconnectDelayMs: number | undefined
|
||||
let retryAfterMs: number | undefined
|
||||
let requestId: string | undefined
|
||||
let toolName: string | undefined
|
||||
let toolArguments: string | undefined
|
||||
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const option = argv[index] as string
|
||||
if (option === '--repeat-last') {
|
||||
repeatLast = true
|
||||
continue
|
||||
}
|
||||
const value = optionValue(argv, index, option)
|
||||
index += 1
|
||||
switch (option) {
|
||||
case '--sequence': sequenceRaw = value; break
|
||||
case '--host': host = value; break
|
||||
case '--port': port = numberValue(option, value); break
|
||||
case '--api-key': apiKey = value; break
|
||||
case '--listen-delay-ms': listenDelayMs = numberValue(option, value); break
|
||||
case '--seed': randomSeed = numberValue(option, value); break
|
||||
case '--random-weights': randomWeights = parseRandomWeights(value); break
|
||||
case '--success-text': successText = value; break
|
||||
case '--partial-text': partialText = value; break
|
||||
case '--reasoning-text': reasoningText = value; break
|
||||
case '--chunk-size': chunkSize = numberValue(option, value); break
|
||||
case '--chunk-delay-ms': chunkDelayMs = numberValue(option, value); break
|
||||
case '--disconnect-delay-ms': disconnectDelayMs = numberValue(option, value); break
|
||||
case '--retry-after-ms': retryAfterMs = numberValue(option, value); break
|
||||
case '--request-id': requestId = value; break
|
||||
case '--tool-name': toolName = value; break
|
||||
case '--tool-arguments': toolArguments = value; break
|
||||
default: throw new Error(`dsh-llm-mock-server: unknown option ${JSON.stringify(option)}`)
|
||||
}
|
||||
}
|
||||
|
||||
if (sequenceRaw === undefined) throw new Error('dsh-llm-mock-server: --sequence is required')
|
||||
const parsedSequence = parseSequence(sequenceRaw)
|
||||
if (parsedSequence.startsUnavailable && port === 0) {
|
||||
throw new Error('dsh-llm-mock-server: connection_refused requires an explicit nonzero --port')
|
||||
}
|
||||
if (!parsedSequence.startsUnavailable && listenDelayMs !== undefined) {
|
||||
throw new Error('dsh-llm-mock-server: --listen-delay-ms requires connection_refused first in --sequence')
|
||||
}
|
||||
if (!parsedSequence.sequence.includes('random') && (randomSeed !== undefined || randomWeights !== undefined)) {
|
||||
throw new Error('dsh-llm-mock-server: --seed and --random-weights require random in --sequence')
|
||||
}
|
||||
|
||||
return {
|
||||
kind: 'run',
|
||||
config: {
|
||||
server: {
|
||||
sequence: parsedSequence.sequence,
|
||||
port,
|
||||
repeatLast,
|
||||
...randomSeed === undefined ? {} : { randomSeed },
|
||||
...randomWeights === undefined ? {} : { randomWeights },
|
||||
...host === undefined ? {} : { host },
|
||||
...apiKey === undefined ? {} : { apiKey },
|
||||
...successText === undefined ? {} : { successText },
|
||||
...partialText === undefined ? {} : { partialText },
|
||||
...reasoningText === undefined ? {} : { reasoningText },
|
||||
...chunkSize === undefined ? {} : { chunkSize },
|
||||
...chunkDelayMs === undefined ? {} : { chunkDelayMs },
|
||||
...disconnectDelayMs === undefined ? {} : { disconnectDelayMs },
|
||||
...retryAfterMs === undefined ? {} : { retryAfterMs },
|
||||
...requestId === undefined ? {} : { requestId },
|
||||
...toolName === undefined ? {} : { toolName },
|
||||
...toolArguments === undefined ? {} : { toolArguments },
|
||||
},
|
||||
listenDelayMs: parsedSequence.startsUnavailable ? listenDelayMs ?? DEFAULT_LISTEN_DELAY_MS : 0,
|
||||
startsUnavailable: parsedSequence.startsUnavailable,
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,723 @@
|
||||
/**
|
||||
* Scriptable OpenAI-compatible HTTP/SSE server for transport, protocol, and
|
||||
* semantic-empty LLM recovery tests. Each accepted chat-completions request
|
||||
* consumes one behavior; the server never retries or interprets harness policy.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-llm-mock-server
|
||||
*/
|
||||
|
||||
import { createServer } from 'node:http'
|
||||
import type { IncomingHttpHeaders, IncomingMessage, ServerResponse } from 'node:http'
|
||||
import { randomBytes } from 'node:crypto'
|
||||
import type { AddressInfo } from 'node:net'
|
||||
import { setTimeout as delay } from 'node:timers/promises'
|
||||
|
||||
/** Request-scoped behaviors accepted by {@link startMockLlmServer}. */
|
||||
export const MOCK_LLM_BEHAVIORS = [
|
||||
'connection_reset',
|
||||
'stream_disconnect',
|
||||
'empty',
|
||||
'empty_body',
|
||||
'stream_eof',
|
||||
'partial_eof',
|
||||
'partial_disconnect',
|
||||
'stall',
|
||||
'malformed_json',
|
||||
'malformed_event',
|
||||
'wrong_content_type',
|
||||
'rate_limit',
|
||||
'server_error',
|
||||
'service_unavailable',
|
||||
'auth_error',
|
||||
'invalid_request',
|
||||
'context_overflow',
|
||||
'quota_exceeded',
|
||||
'success',
|
||||
'reasoning_success',
|
||||
'tool_call_success',
|
||||
'max_tokens',
|
||||
'slow_success',
|
||||
'random',
|
||||
] as const
|
||||
|
||||
/** One scripted mock behavior name; `random` selects a concrete behavior per request. */
|
||||
export type MockLlmBehavior = typeof MOCK_LLM_BEHAVIORS[number]
|
||||
|
||||
/** One concrete request behavior after resolving a `random` script entry. */
|
||||
export type ConcreteMockLlmBehavior = Exclude<MockLlmBehavior, 'random'>
|
||||
|
||||
/** Relative non-negative weights for random request behavior selection. */
|
||||
export type MockLlmRandomWeights = Partial<Record<ConcreteMockLlmBehavior, number>>
|
||||
|
||||
/**
|
||||
* Default stress profile for `random`. Weights are configurable test pressure,
|
||||
* not a claim about production incident frequency.
|
||||
*/
|
||||
export const DEFAULT_MOCK_LLM_RANDOM_WEIGHTS: Readonly<MockLlmRandomWeights> = Object.freeze({
|
||||
success: 48,
|
||||
slow_success: 10,
|
||||
max_tokens: 2,
|
||||
connection_reset: 5,
|
||||
stream_disconnect: 5,
|
||||
partial_disconnect: 10,
|
||||
empty: 5,
|
||||
stall: 2,
|
||||
rate_limit: 5,
|
||||
server_error: 4,
|
||||
service_unavailable: 2,
|
||||
partial_eof: 1,
|
||||
malformed_json: 1,
|
||||
})
|
||||
|
||||
/** How one accepted request ended at the mock boundary. */
|
||||
export type MockLlmRequestOutcome = 'completed' | 'reset' | 'stalled' | 'client_closed' | 'server_error'
|
||||
|
||||
/** Immutable telemetry emitted when a request starts or reaches an outcome. */
|
||||
export type MockLlmServerEvent =
|
||||
| {
|
||||
readonly type: 'request'
|
||||
readonly attempt: number
|
||||
readonly scriptBehavior: MockLlmBehavior | 'script_exhausted'
|
||||
readonly behavior: ConcreteMockLlmBehavior | 'script_exhausted'
|
||||
readonly path: string
|
||||
}
|
||||
| {
|
||||
readonly type: 'result'
|
||||
readonly attempt: number
|
||||
readonly scriptBehavior: MockLlmBehavior | 'script_exhausted'
|
||||
readonly behavior: ConcreteMockLlmBehavior | 'script_exhausted'
|
||||
readonly outcome: MockLlmRequestOutcome
|
||||
readonly chunksSent: number
|
||||
}
|
||||
|
||||
/** Captured wire request and its final server-side outcome. */
|
||||
export interface MockLlmRequestRecord {
|
||||
/** One-based accepted chat-completions request number. */
|
||||
readonly attempt: number
|
||||
/** Script entry consumed for this request before random resolution. */
|
||||
readonly scriptBehavior: MockLlmBehavior | 'script_exhausted'
|
||||
/** Concrete behavior selected for this request, or exhaustion after the configured script. */
|
||||
readonly behavior: ConcreteMockLlmBehavior | 'script_exhausted'
|
||||
/** Original request path, including a `/v1` prefix when the client supplied one. */
|
||||
readonly path: string
|
||||
/** Detached request headers. */
|
||||
readonly headers: Readonly<IncomingHttpHeaders>
|
||||
/** Parsed JSON request body. */
|
||||
readonly body: unknown
|
||||
/** Number of SSE `data:` events handed to Node before the outcome. */
|
||||
chunksSent: number
|
||||
/** Final server-side outcome; absent while a stalled request remains open. */
|
||||
outcome?: MockLlmRequestOutcome
|
||||
}
|
||||
|
||||
/** Configuration for one mock server instance. */
|
||||
export interface MockLlmServerOptions {
|
||||
/** Loopback host by default. */
|
||||
readonly host?: string
|
||||
/** TCP port; zero requests an OS-assigned port. */
|
||||
readonly port?: number
|
||||
/** Optional exact bearer token; omission accepts any authorization header. */
|
||||
readonly apiKey?: string
|
||||
/** Ordered request behaviors; exhaustion fails loud unless `repeatLast` is true. */
|
||||
readonly sequence: readonly MockLlmBehavior[]
|
||||
/** Reuse the final behavior after the sequence is consumed. */
|
||||
readonly repeatLast?: boolean
|
||||
/** Optional deterministic unsigned 32-bit seed; omission generates and exposes one. */
|
||||
readonly randomSeed?: number
|
||||
/** Relative weights used whenever a script entry is `random`. */
|
||||
readonly randomWeights?: Readonly<MockLlmRandomWeights>
|
||||
/** Complete text returned by success-shaped behaviors. */
|
||||
readonly successText?: string
|
||||
/** Text emitted before partial EOF/reset behaviors terminate. */
|
||||
readonly partialText?: string
|
||||
/** Reasoning text emitted by `reasoning_success`. */
|
||||
readonly reasoningText?: string
|
||||
/** Unicode code-point count per text or reasoning SSE delta. */
|
||||
readonly chunkSize?: number
|
||||
/** Inter-chunk delay for `slow_success`, in milliseconds. */
|
||||
readonly chunkDelayMs?: number
|
||||
/** Delay after headers/deltas before a forced disconnect, in milliseconds. */
|
||||
readonly disconnectDelayMs?: number
|
||||
/** Provider retry delay; the wire `Retry-After` value rounds up to whole seconds. */
|
||||
readonly retryAfterMs?: number
|
||||
/** Optional provider request id returned on HTTP failures. */
|
||||
readonly requestId?: string
|
||||
/** Tool name emitted by `tool_call_success`. */
|
||||
readonly toolName?: string
|
||||
/** Raw JSON arguments emitted by `tool_call_success`. */
|
||||
readonly toolArguments?: string
|
||||
/** Optional observer for JSONL CLI telemetry; observer failures never affect wire behavior. */
|
||||
readonly onEvent?: (event: MockLlmServerEvent) => void
|
||||
}
|
||||
|
||||
/** Running mock server and captured request state. */
|
||||
export interface MockLlmServer {
|
||||
/** Base URL without `/v1`; both root and `/v1` chat-completions paths are accepted. */
|
||||
readonly baseURL: string
|
||||
/** Actual bound port, including an OS-assigned value. */
|
||||
readonly port: number
|
||||
/** Seed used for random behavior selection, including the generated default. */
|
||||
readonly randomSeed: number
|
||||
/** Live request records in arrival order. */
|
||||
readonly requests: readonly MockLlmRequestRecord[]
|
||||
/** Stop accepting requests and force-close stalled/streaming connections; idempotent. */
|
||||
close(): Promise<void>
|
||||
}
|
||||
|
||||
interface ResolvedOptions {
|
||||
readonly host: string
|
||||
readonly port: number
|
||||
readonly apiKey?: string
|
||||
readonly sequence: readonly MockLlmBehavior[]
|
||||
readonly lastBehavior: MockLlmBehavior
|
||||
readonly repeatLast: boolean
|
||||
readonly randomSeed: number
|
||||
readonly randomWeights: readonly (readonly [ConcreteMockLlmBehavior, number])[]
|
||||
readonly successText: string
|
||||
readonly partialText: string
|
||||
readonly reasoningText: string
|
||||
readonly chunkSize: number
|
||||
readonly chunkDelayMs: number
|
||||
readonly disconnectDelayMs: number
|
||||
readonly retryAfterMs: number
|
||||
readonly requestId?: string
|
||||
readonly toolName: string
|
||||
readonly toolArguments: string
|
||||
readonly onEvent?: (event: MockLlmServerEvent) => void
|
||||
}
|
||||
|
||||
const MAX_TIMER_DELAY_MS = 2_147_483_647
|
||||
const DEFAULT_SUCCESS_TEXT = 'mock response recovered'
|
||||
const DEFAULT_PARTIAL_TEXT = 'discarded partial response'
|
||||
const DEFAULT_REASONING_TEXT = 'mock reasoning'
|
||||
const CONCRETE_BEHAVIORS = new Set<string>(MOCK_LLM_BEHAVIORS.filter(behavior => behavior !== 'random'))
|
||||
|
||||
function boundedInteger(name: string, value: number, min: number, max: number): number {
|
||||
if (!Number.isInteger(value) || value < min || value > max) {
|
||||
throw new Error(`llm-mock-server: ${name} must be an integer between ${min} and ${max}`)
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
function resolveOptions(options: MockLlmServerOptions): ResolvedOptions {
|
||||
const host = options.host ?? '127.0.0.1'
|
||||
const port = boundedInteger('port', options.port ?? 0, 0, 65_535)
|
||||
const chunkSize = boundedInteger('chunkSize', options.chunkSize ?? 8, 1, Number.MAX_SAFE_INTEGER)
|
||||
const chunkDelayMs = boundedInteger('chunkDelayMs', options.chunkDelayMs ?? 25, 0, MAX_TIMER_DELAY_MS)
|
||||
const disconnectDelayMs = boundedInteger(
|
||||
'disconnectDelayMs',
|
||||
options.disconnectDelayMs ?? 10,
|
||||
0,
|
||||
MAX_TIMER_DELAY_MS,
|
||||
)
|
||||
const retryAfterMs = boundedInteger('retryAfterMs', options.retryAfterMs ?? 1_000, 1, MAX_TIMER_DELAY_MS)
|
||||
const randomSeed = boundedInteger(
|
||||
'randomSeed',
|
||||
options.randomSeed ?? randomBytes(4).readUInt32LE(0),
|
||||
0,
|
||||
0xffff_ffff,
|
||||
)
|
||||
const successText = options.successText ?? DEFAULT_SUCCESS_TEXT
|
||||
const partialText = options.partialText ?? DEFAULT_PARTIAL_TEXT
|
||||
const reasoningText = options.reasoningText ?? DEFAULT_REASONING_TEXT
|
||||
const toolName = options.toolName ?? 'mock_tool'
|
||||
const toolArguments = options.toolArguments ?? '{"value":"mock"}'
|
||||
|
||||
if (host.length === 0) throw new Error('llm-mock-server: host must not be empty')
|
||||
if (options.sequence.length === 0) throw new Error('llm-mock-server: sequence must not be empty')
|
||||
const lastBehavior = options.sequence.reduce((_previous, behavior) => behavior)
|
||||
if (options.apiKey === '') throw new Error('llm-mock-server: apiKey must not be empty')
|
||||
if (successText.length === 0) throw new Error('llm-mock-server: successText must not be empty')
|
||||
if (partialText.length === 0) throw new Error('llm-mock-server: partialText must not be empty')
|
||||
if (reasoningText.length === 0) throw new Error('llm-mock-server: reasoningText must not be empty')
|
||||
if (toolName.length === 0) throw new Error('llm-mock-server: toolName must not be empty')
|
||||
if (options.requestId === '') throw new Error('llm-mock-server: requestId must not be empty')
|
||||
try {
|
||||
JSON.parse(toolArguments)
|
||||
} catch {
|
||||
throw new Error('llm-mock-server: toolArguments must be valid JSON')
|
||||
}
|
||||
|
||||
const configuredWeights = options.randomWeights ?? DEFAULT_MOCK_LLM_RANDOM_WEIGHTS
|
||||
const randomWeights: Array<readonly [ConcreteMockLlmBehavior, number]> = []
|
||||
for (const [behavior, weight] of Object.entries(configuredWeights)) {
|
||||
if (!CONCRETE_BEHAVIORS.has(behavior)) {
|
||||
throw new Error(`llm-mock-server: randomWeights contains unknown concrete behavior ${JSON.stringify(behavior)}`)
|
||||
}
|
||||
if (!Number.isFinite(weight) || weight < 0) {
|
||||
throw new Error(`llm-mock-server: random weight for ${behavior} must be a non-negative finite number`)
|
||||
}
|
||||
if (weight > 0) randomWeights.push([behavior as ConcreteMockLlmBehavior, weight])
|
||||
}
|
||||
if (randomWeights.length === 0) {
|
||||
throw new Error('llm-mock-server: randomWeights must contain at least one positive weight')
|
||||
}
|
||||
|
||||
return {
|
||||
host,
|
||||
port,
|
||||
...options.apiKey === undefined ? {} : { apiKey: options.apiKey },
|
||||
sequence: [...options.sequence],
|
||||
lastBehavior,
|
||||
repeatLast: options.repeatLast ?? false,
|
||||
randomSeed,
|
||||
randomWeights,
|
||||
successText,
|
||||
partialText,
|
||||
reasoningText,
|
||||
chunkSize,
|
||||
chunkDelayMs,
|
||||
disconnectDelayMs,
|
||||
retryAfterMs,
|
||||
...options.requestId === undefined ? {} : { requestId: options.requestId },
|
||||
toolName,
|
||||
toolArguments,
|
||||
...options.onEvent === undefined ? {} : { onEvent: options.onEvent },
|
||||
}
|
||||
}
|
||||
|
||||
function emit(options: ResolvedOptions, event: MockLlmServerEvent): void {
|
||||
try {
|
||||
options.onEvent?.(Object.freeze(event))
|
||||
} catch (_telemetryObserverFailure) {
|
||||
// Test telemetry is observational; a broken observer cannot change provider wire behavior.
|
||||
}
|
||||
}
|
||||
|
||||
async function readJsonBody(request: IncomingMessage): Promise<unknown> {
|
||||
let body = ''
|
||||
for await (const chunk of request) body += Buffer.from(chunk).toString('utf8')
|
||||
return body.length === 0 ? undefined : JSON.parse(body)
|
||||
}
|
||||
|
||||
function splitText(text: string, size: number): string[] {
|
||||
const points = Array.from(text)
|
||||
const chunks: string[] = []
|
||||
for (let index = 0; index < points.length; index += size) chunks.push(points.slice(index, index + size).join(''))
|
||||
return chunks
|
||||
}
|
||||
|
||||
function openSse(response: ServerResponse, contentType = 'text/event-stream; charset=utf-8'): void {
|
||||
response.writeHead(200, {
|
||||
'content-type': contentType,
|
||||
'cache-control': 'no-cache',
|
||||
'connection': 'keep-alive',
|
||||
})
|
||||
response.flushHeaders()
|
||||
}
|
||||
|
||||
function writeSse(record: MockLlmRequestRecord, response: ServerResponse, payload: unknown): void {
|
||||
response.write(`data: ${typeof payload === 'string' ? payload : JSON.stringify(payload)}\n\n`)
|
||||
record.chunksSent += 1
|
||||
}
|
||||
|
||||
function writeDone(record: MockLlmRequestRecord, response: ServerResponse): void {
|
||||
writeSse(record, response, '[DONE]')
|
||||
}
|
||||
|
||||
function finishRecord(
|
||||
options: ResolvedOptions,
|
||||
record: MockLlmRequestRecord,
|
||||
outcome: MockLlmRequestOutcome,
|
||||
): void {
|
||||
record.outcome = outcome
|
||||
emit(options, {
|
||||
type: 'result',
|
||||
attempt: record.attempt,
|
||||
scriptBehavior: record.scriptBehavior,
|
||||
behavior: record.behavior,
|
||||
outcome,
|
||||
chunksSent: record.chunksSent,
|
||||
})
|
||||
}
|
||||
|
||||
function httpError(
|
||||
options: ResolvedOptions,
|
||||
record: MockLlmRequestRecord,
|
||||
response: ServerResponse,
|
||||
status: number,
|
||||
message: string,
|
||||
code: string,
|
||||
type = 'mock_error',
|
||||
): void {
|
||||
const headers: Record<string, string> = { 'content-type': 'application/json' }
|
||||
if (record.behavior === 'rate_limit') {
|
||||
headers['retry-after'] = String(Math.ceil(options.retryAfterMs / 1_000))
|
||||
}
|
||||
if (options.requestId !== undefined) headers['x-request-id'] = options.requestId
|
||||
response.writeHead(status, headers)
|
||||
response.end(JSON.stringify({ error: { message, type, code } }))
|
||||
finishRecord(options, record, 'completed')
|
||||
}
|
||||
|
||||
function terminalChunk(reason: string, outputTokens: number): unknown {
|
||||
return {
|
||||
choices: [{ index: 0, delta: { content: '' }, finish_reason: reason }],
|
||||
usage: { prompt_tokens: 3, completion_tokens: outputTokens },
|
||||
}
|
||||
}
|
||||
|
||||
async function pause(milliseconds: number, response: ServerResponse): Promise<boolean> {
|
||||
if (milliseconds === 0) return !response.destroyed
|
||||
const controller = new AbortController()
|
||||
const stop = (): void => { controller.abort() }
|
||||
response.once('close', stop)
|
||||
try {
|
||||
await delay(milliseconds, undefined, { signal: controller.signal })
|
||||
return true
|
||||
} catch (_responseClosed) {
|
||||
// The timer only receives this response-owned abort signal; closing the response cancels its wait.
|
||||
return false
|
||||
} finally {
|
||||
response.off('close', stop)
|
||||
}
|
||||
}
|
||||
|
||||
async function streamText(
|
||||
options: ResolvedOptions,
|
||||
record: MockLlmRequestRecord,
|
||||
response: ServerResponse,
|
||||
text: string,
|
||||
delayMs: number,
|
||||
): Promise<boolean> {
|
||||
for (const chunk of splitText(text, options.chunkSize)) {
|
||||
writeSse(record, response, { choices: [{ index: 0, delta: { content: chunk }, finish_reason: null }] })
|
||||
if (!await pause(delayMs, response)) return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
async function completeText(
|
||||
options: ResolvedOptions,
|
||||
record: MockLlmRequestRecord,
|
||||
response: ServerResponse,
|
||||
reason: 'stop' | 'length',
|
||||
delayMs: number,
|
||||
): Promise<void> {
|
||||
if (!await streamText(options, record, response, options.successText, delayMs)) {
|
||||
finishRecord(options, record, 'client_closed')
|
||||
return
|
||||
}
|
||||
writeSse(record, response, terminalChunk(reason, Array.from(options.successText).length))
|
||||
writeDone(record, response)
|
||||
response.end()
|
||||
finishRecord(options, record, 'completed')
|
||||
}
|
||||
|
||||
async function disconnect(
|
||||
options: ResolvedOptions,
|
||||
record: MockLlmRequestRecord,
|
||||
response: ServerResponse,
|
||||
): Promise<void> {
|
||||
if (!await pause(options.disconnectDelayMs, response)) {
|
||||
finishRecord(options, record, 'client_closed')
|
||||
return
|
||||
}
|
||||
finishRecord(options, record, 'reset')
|
||||
response.destroy()
|
||||
}
|
||||
|
||||
function toolCallChunks(options: ResolvedOptions): readonly unknown[] {
|
||||
const midpoint = Math.max(1, Math.floor(options.toolArguments.length / 2))
|
||||
return [
|
||||
{
|
||||
choices: [{
|
||||
index: 0,
|
||||
delta: {
|
||||
tool_calls: [{
|
||||
index: 0,
|
||||
id: 'mock-call-1',
|
||||
type: 'function',
|
||||
function: { name: options.toolName, arguments: options.toolArguments.slice(0, midpoint) },
|
||||
}],
|
||||
},
|
||||
finish_reason: null,
|
||||
}],
|
||||
},
|
||||
{
|
||||
choices: [{
|
||||
index: 0,
|
||||
delta: { tool_calls: [{ index: 0, function: { arguments: options.toolArguments.slice(midpoint) } }] },
|
||||
finish_reason: null,
|
||||
}],
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
async function runBehavior(
|
||||
options: ResolvedOptions,
|
||||
record: MockLlmRequestRecord,
|
||||
request: IncomingMessage,
|
||||
response: ServerResponse,
|
||||
): Promise<void> {
|
||||
switch (record.behavior) {
|
||||
case 'script_exhausted':
|
||||
httpError(options, record, response, 500, 'mock script exhausted', 'MOCK_SCRIPT_EXHAUSTED')
|
||||
return
|
||||
case 'connection_reset':
|
||||
finishRecord(options, record, 'reset')
|
||||
request.socket.destroy()
|
||||
return
|
||||
case 'stream_disconnect':
|
||||
openSse(response)
|
||||
await disconnect(options, record, response)
|
||||
return
|
||||
case 'empty':
|
||||
openSse(response)
|
||||
writeSse(record, response, terminalChunk('stop', 0))
|
||||
writeDone(record, response)
|
||||
response.end()
|
||||
finishRecord(options, record, 'completed')
|
||||
return
|
||||
case 'empty_body':
|
||||
openSse(response)
|
||||
response.end()
|
||||
finishRecord(options, record, 'completed')
|
||||
return
|
||||
case 'stream_eof':
|
||||
openSse(response)
|
||||
writeSse(record, response, { choices: [{ index: 0, delta: { role: 'assistant' }, finish_reason: null }] })
|
||||
response.end()
|
||||
finishRecord(options, record, 'completed')
|
||||
return
|
||||
case 'partial_eof':
|
||||
openSse(response)
|
||||
await streamText(options, record, response, options.partialText, 0)
|
||||
response.end()
|
||||
finishRecord(options, record, 'completed')
|
||||
return
|
||||
case 'partial_disconnect':
|
||||
openSse(response)
|
||||
if (!await streamText(options, record, response, options.partialText, options.chunkDelayMs)) return
|
||||
await disconnect(options, record, response)
|
||||
return
|
||||
case 'stall':
|
||||
openSse(response)
|
||||
finishRecord(options, record, 'stalled')
|
||||
return
|
||||
case 'malformed_json':
|
||||
openSse(response)
|
||||
writeSse(record, response, '{not-json')
|
||||
writeDone(record, response)
|
||||
response.end()
|
||||
finishRecord(options, record, 'completed')
|
||||
return
|
||||
case 'malformed_event':
|
||||
openSse(response)
|
||||
writeSse(record, response, { choices: [null] })
|
||||
writeDone(record, response)
|
||||
response.end()
|
||||
finishRecord(options, record, 'completed')
|
||||
return
|
||||
case 'wrong_content_type':
|
||||
openSse(response, 'application/json')
|
||||
await completeText(options, record, response, 'stop', 0)
|
||||
return
|
||||
case 'rate_limit':
|
||||
httpError(options, record, response, 429, 'mock rate limit', 'rate_limit')
|
||||
return
|
||||
case 'server_error':
|
||||
httpError(options, record, response, 500, 'mock server error', 'server_error')
|
||||
return
|
||||
case 'service_unavailable':
|
||||
httpError(options, record, response, 503, 'mock service unavailable', 'service_unavailable')
|
||||
return
|
||||
case 'auth_error':
|
||||
httpError(options, record, response, 401, 'mock authentication failed', 'invalid_api_key')
|
||||
return
|
||||
case 'invalid_request':
|
||||
httpError(options, record, response, 400, 'mock invalid request', 'invalid_request')
|
||||
return
|
||||
case 'context_overflow':
|
||||
httpError(
|
||||
options,
|
||||
record,
|
||||
response,
|
||||
400,
|
||||
'mock input exceeds the model context window',
|
||||
'context_length_exceeded',
|
||||
'invalid_request_error',
|
||||
)
|
||||
return
|
||||
case 'quota_exceeded':
|
||||
httpError(options, record, response, 429, 'mock insufficient quota', 'insufficient_quota')
|
||||
return
|
||||
case 'success':
|
||||
openSse(response)
|
||||
await completeText(options, record, response, 'stop', 0)
|
||||
return
|
||||
case 'reasoning_success':
|
||||
openSse(response)
|
||||
for (const chunk of splitText(options.reasoningText, options.chunkSize)) {
|
||||
writeSse(record, response, {
|
||||
choices: [{ index: 0, delta: { reasoning_content: chunk }, finish_reason: null }],
|
||||
})
|
||||
}
|
||||
await completeText(options, record, response, 'stop', 0)
|
||||
return
|
||||
case 'tool_call_success':
|
||||
openSse(response)
|
||||
for (const chunk of toolCallChunks(options)) writeSse(record, response, chunk)
|
||||
writeSse(record, response, terminalChunk('tool_calls', 2))
|
||||
writeDone(record, response)
|
||||
response.end()
|
||||
finishRecord(options, record, 'completed')
|
||||
return
|
||||
case 'max_tokens':
|
||||
openSse(response)
|
||||
await completeText(options, record, response, 'length', 0)
|
||||
return
|
||||
case 'slow_success':
|
||||
openSse(response)
|
||||
await completeText(options, record, response, 'stop', options.chunkDelayMs)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
function seededRandom(seed: number): () => number {
|
||||
let state = seed
|
||||
return () => {
|
||||
state = (state + 0x6d2b_79f5) >>> 0
|
||||
let mixed = state
|
||||
mixed = Math.imul(mixed ^ mixed >>> 15, mixed | 1)
|
||||
mixed ^= mixed + Math.imul(mixed ^ mixed >>> 7, mixed | 61)
|
||||
return ((mixed ^ mixed >>> 14) >>> 0) / 0x1_0000_0000
|
||||
}
|
||||
}
|
||||
|
||||
function chooseRandomBehavior(
|
||||
weights: readonly (readonly [ConcreteMockLlmBehavior, number])[],
|
||||
random: () => number,
|
||||
): ConcreteMockLlmBehavior {
|
||||
const total = weights.reduce((sum, entry) => sum + entry[1], 0)
|
||||
let draw = random() * total
|
||||
for (const [behavior, weight] of weights) {
|
||||
if (draw < weight) return behavior
|
||||
draw -= weight
|
||||
}
|
||||
// Floating-point subtraction can only leave a rounding residue at the upper boundary.
|
||||
/* v8 ignore next -- seededRandom is strictly less than one; this guards floating-point residue only */
|
||||
return (weights.at(-1) as readonly [ConcreteMockLlmBehavior, number])[0]
|
||||
}
|
||||
|
||||
/**
|
||||
* Start a local chat-completions server that consumes one configured behavior
|
||||
* per accepted request. Only a `POST` path ending in `/chat/completions` consumes the script;
|
||||
* invalid routes, methods, authorization, and JSON receive ordinary 4xx
|
||||
* responses. Closing the handle terminates stalled connections.
|
||||
*
|
||||
* @param options - listener, script, response content, timing, and telemetry options.
|
||||
* @returns the listening handle after the port is bound.
|
||||
*/
|
||||
export async function startMockLlmServer(options: MockLlmServerOptions): Promise<MockLlmServer> {
|
||||
const resolved = resolveOptions(options)
|
||||
const requests: MockLlmRequestRecord[] = []
|
||||
const random = seededRandom(resolved.randomSeed)
|
||||
let cursor = 0
|
||||
|
||||
const selectBehavior = (): {
|
||||
scriptBehavior: MockLlmBehavior | 'script_exhausted'
|
||||
behavior: ConcreteMockLlmBehavior | 'script_exhausted'
|
||||
} => {
|
||||
const selected = resolved.sequence[cursor]
|
||||
cursor += 1
|
||||
const scriptBehavior = selected
|
||||
?? (resolved.repeatLast ? resolved.lastBehavior : 'script_exhausted')
|
||||
return {
|
||||
scriptBehavior,
|
||||
behavior: scriptBehavior === 'random'
|
||||
? chooseRandomBehavior(resolved.randomWeights, random)
|
||||
: scriptBehavior,
|
||||
}
|
||||
}
|
||||
|
||||
const handle = async (request: IncomingMessage, response: ServerResponse): Promise<void> => {
|
||||
/* v8 ignore next -- node:http server requests always carry a URL despite the shared optional type */
|
||||
const path = new URL(request.url ?? '/', 'http://mock.invalid').pathname
|
||||
if (request.method !== 'POST') {
|
||||
response.writeHead(405, { allow: 'POST' }).end()
|
||||
return
|
||||
}
|
||||
if (!path.endsWith('/chat/completions')) {
|
||||
response.writeHead(404).end()
|
||||
return
|
||||
}
|
||||
if (resolved.apiKey !== undefined && request.headers.authorization !== `Bearer ${resolved.apiKey}`) {
|
||||
response.writeHead(401, { 'content-type': 'application/json' })
|
||||
response.end(JSON.stringify({ error: { message: 'invalid mock bearer token', code: 'invalid_api_key' } }))
|
||||
return
|
||||
}
|
||||
|
||||
let body: unknown
|
||||
try {
|
||||
body = await readJsonBody(request)
|
||||
} catch {
|
||||
response.writeHead(400, { 'content-type': 'application/json' })
|
||||
response.end(JSON.stringify({ error: { message: 'request body must be valid JSON', code: 'invalid_json' } }))
|
||||
return
|
||||
}
|
||||
|
||||
const selected = selectBehavior()
|
||||
const record: MockLlmRequestRecord = {
|
||||
attempt: requests.length + 1,
|
||||
scriptBehavior: selected.scriptBehavior,
|
||||
behavior: selected.behavior,
|
||||
path,
|
||||
headers: { ...request.headers },
|
||||
body,
|
||||
chunksSent: 0,
|
||||
}
|
||||
requests.push(record)
|
||||
response.once('close', () => {
|
||||
if (!response.writableFinished && record.outcome === undefined) {
|
||||
finishRecord(resolved, record, 'client_closed')
|
||||
}
|
||||
})
|
||||
emit(resolved, {
|
||||
type: 'request',
|
||||
attempt: record.attempt,
|
||||
scriptBehavior: record.scriptBehavior,
|
||||
behavior: record.behavior,
|
||||
path,
|
||||
})
|
||||
await runBehavior(resolved, record, request, response)
|
||||
}
|
||||
|
||||
const server = createServer((request, response) => {
|
||||
/* v8 ignore start -- last-resort containment for Node response failures after validated test inputs */
|
||||
handle(request, response).catch((error: unknown) => {
|
||||
const record = requests.at(-1)
|
||||
if (record !== undefined) finishRecord(resolved, record, 'server_error')
|
||||
if (response.headersSent) {
|
||||
response.destroy(error instanceof Error ? error : new Error(String(error)))
|
||||
return
|
||||
}
|
||||
response.writeHead(500, { 'content-type': 'application/json' })
|
||||
response.end(JSON.stringify({ error: { message: 'mock server handler failed', code: 'MOCK_HANDLER_FAILED' } }))
|
||||
})
|
||||
/* v8 ignore stop */
|
||||
})
|
||||
|
||||
let closing: Promise<void> | undefined
|
||||
const close = (): Promise<void> => (closing ??= new Promise((resolveClose) => {
|
||||
server.close(() => { resolveClose() })
|
||||
server.closeAllConnections()
|
||||
}))
|
||||
|
||||
await new Promise<void>((resolveListen, rejectListen) => {
|
||||
server.once('error', rejectListen)
|
||||
server.listen(resolved.port, resolved.host, () => {
|
||||
server.off('error', rejectListen)
|
||||
resolveListen()
|
||||
})
|
||||
})
|
||||
|
||||
const address = server.address() as AddressInfo
|
||||
return {
|
||||
baseURL: `http://${resolved.host}:${address.port}`,
|
||||
port: address.port,
|
||||
randomSeed: resolved.randomSeed,
|
||||
requests,
|
||||
close,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
/**
|
||||
* Package-owned invariant companion for `@deepseek-ai/dsh-llm-mock-server`.
|
||||
* @module @deepseek-ai/dsh-llm-mock-server/invariant
|
||||
*/
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from 'cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-llm-mock-server'
|
||||
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'llm-mock-server-invariant'
|
||||
/** Service required before the companion can reserve package ownership. */
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: this standalone test server owns no Cordis event stream or shared data;
|
||||
* its wire behavior and lifecycle are exercised through direct HTTP and assembled-loop tests.
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
/**
|
||||
* Register this package's invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the installed registration's disposer after setup succeeds.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> =>
|
||||
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
@@ -0,0 +1,121 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
MOCK_LLM_CLI_USAGE,
|
||||
parseMockLlmCliArgs,
|
||||
} from '../src/cli.ts'
|
||||
|
||||
describe('mock LLM server CLI parser', () => {
|
||||
it('returns help without requiring a sequence', () => {
|
||||
expect(parseMockLlmCliArgs(['--help'])).toEqual({ kind: 'help' })
|
||||
expect(MOCK_LLM_CLI_USAGE).toContain('--sequence')
|
||||
})
|
||||
|
||||
it('parses every request and listener option', () => {
|
||||
expect(parseMockLlmCliArgs([
|
||||
'--sequence', 'connection_refused,partial_disconnect,success',
|
||||
'--host', 'localhost',
|
||||
'--port', '9010',
|
||||
'--api-key', 'mock-key',
|
||||
'--listen-delay-ms', '100',
|
||||
'--repeat-last',
|
||||
'--success-text', 'done',
|
||||
'--partial-text', 'half',
|
||||
'--reasoning-text', 'think',
|
||||
'--chunk-size', '2',
|
||||
'--chunk-delay-ms', '3',
|
||||
'--disconnect-delay-ms', '4',
|
||||
'--retry-after-ms', '5000',
|
||||
'--request-id', 'request-1',
|
||||
'--tool-name', 'lookup',
|
||||
'--tool-arguments', '{"id":1}',
|
||||
])).toEqual({
|
||||
kind: 'run',
|
||||
config: {
|
||||
startsUnavailable: true,
|
||||
listenDelayMs: 100,
|
||||
server: {
|
||||
sequence: ['partial_disconnect', 'success'],
|
||||
host: 'localhost',
|
||||
port: 9010,
|
||||
apiKey: 'mock-key',
|
||||
repeatLast: true,
|
||||
successText: 'done',
|
||||
partialText: 'half',
|
||||
reasoningText: 'think',
|
||||
chunkSize: 2,
|
||||
chunkDelayMs: 3,
|
||||
disconnectDelayMs: 4,
|
||||
retryAfterMs: 5000,
|
||||
requestId: 'request-1',
|
||||
toolName: 'lookup',
|
||||
toolArguments: '{"id":1}',
|
||||
},
|
||||
},
|
||||
})
|
||||
})
|
||||
|
||||
it('uses standalone defaults for an ordinary sequence', () => {
|
||||
expect(parseMockLlmCliArgs(['--sequence', 'success'])).toEqual({
|
||||
kind: 'run',
|
||||
config: {
|
||||
startsUnavailable: false,
|
||||
listenDelayMs: 0,
|
||||
server: {
|
||||
sequence: ['success'],
|
||||
port: 8000,
|
||||
repeatLast: false,
|
||||
},
|
||||
},
|
||||
})
|
||||
})
|
||||
|
||||
it('uses the default unavailable interval', () => {
|
||||
const result = parseMockLlmCliArgs(['--sequence', 'connection_refused,success', '--port', '8001'])
|
||||
expect(result).toMatchObject({
|
||||
kind: 'run',
|
||||
config: { startsUnavailable: true, listenDelayMs: 750 },
|
||||
})
|
||||
})
|
||||
|
||||
it('parses a reproducible weighted random profile', () => {
|
||||
expect(parseMockLlmCliArgs([
|
||||
'--sequence', 'random',
|
||||
'--repeat-last',
|
||||
'--seed', '42',
|
||||
'--random-weights', 'success=8,partial_disconnect=2',
|
||||
])).toEqual({
|
||||
kind: 'run',
|
||||
config: {
|
||||
startsUnavailable: false,
|
||||
listenDelayMs: 0,
|
||||
server: {
|
||||
sequence: ['random'],
|
||||
port: 8000,
|
||||
repeatLast: true,
|
||||
randomSeed: 42,
|
||||
randomWeights: { success: 8, partial_disconnect: 2 },
|
||||
},
|
||||
},
|
||||
})
|
||||
})
|
||||
|
||||
it.each([
|
||||
[[], /--sequence is required/],
|
||||
[['--wat'], /requires a value/],
|
||||
[['--wat', 'x'], /unknown option/],
|
||||
[['--port', 'NaN', '--sequence', 'success'], /finite number/],
|
||||
[['--sequence', 'success,'], /non-empty/],
|
||||
[['--sequence', 'success,connection_refused'], /only as the first/],
|
||||
[['--sequence', 'connection_refused'], /must be followed/],
|
||||
[['--sequence', 'unknown'], /unknown behavior/],
|
||||
[['--sequence', 'connection_refused,success', '--port', '0'], /nonzero/],
|
||||
[['--sequence', 'success', '--listen-delay-ms', '5'], /requires connection_refused/],
|
||||
[['--sequence', 'success', '--seed', '1'], /require random/],
|
||||
[['--sequence', 'random', '--random-weights', 'success'], /expects behavior=weight/],
|
||||
[['--sequence', 'random', '--random-weights', 'random=1'], /concrete behavior/],
|
||||
[['--sequence', 'random', '--random-weights', 'success=1,success=2'], /duplicate/],
|
||||
[['--sequence', 'random', '--random-weights', 'success=nope'], /finite number/],
|
||||
])('rejects invalid argv %#', (argv, expected) => {
|
||||
expect(() => parseMockLlmCliArgs(argv)).toThrow(expected)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,18 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import InvariantService from '@deepseek-ai/dsh-invariants'
|
||||
import * as MockServerInvariant from '../src/invariant.ts'
|
||||
|
||||
describe('mock LLM server invariant companion', () => {
|
||||
it('registers its explained empty runtime invariant', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(InvariantService)
|
||||
const fiber = await ctx.plugin(MockServerInvariant)
|
||||
|
||||
expect(() => {
|
||||
ctx.invariants.register('@deepseek-ai/dsh-llm-mock-server', () => {})
|
||||
}).toThrow(/already registered/)
|
||||
await fiber.dispose()
|
||||
await ctx.fiber.dispose()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,312 @@
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import type { MockLlmBehavior, MockLlmServer, MockLlmServerEvent } from '../src/index.ts'
|
||||
import { startMockLlmServer } from '../src/index.ts'
|
||||
|
||||
const running: MockLlmServer[] = []
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(running.splice(0).map(server => server.close()))
|
||||
})
|
||||
|
||||
async function start(
|
||||
sequence: readonly MockLlmBehavior[],
|
||||
options: Omit<Parameters<typeof startMockLlmServer>[0], 'sequence'> = {},
|
||||
): Promise<MockLlmServer> {
|
||||
const server = await startMockLlmServer({ sequence, ...options })
|
||||
running.push(server)
|
||||
return server
|
||||
}
|
||||
|
||||
function chat(
|
||||
server: MockLlmServer,
|
||||
options: { path?: string; key?: string; body?: string; signal?: AbortSignal } = {},
|
||||
): Promise<Response> {
|
||||
return fetch(`${server.baseURL}${options.path ?? '/v1/chat/completions'}`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'content-type': 'application/json',
|
||||
...options.key === undefined ? {} : { authorization: `Bearer ${options.key}` },
|
||||
},
|
||||
body: options.body ?? JSON.stringify({ model: 'mock', messages: [], stream: true }),
|
||||
...options.signal === undefined ? {} : { signal: options.signal },
|
||||
})
|
||||
}
|
||||
|
||||
describe('mock LLM server wire behaviors', () => {
|
||||
it('streams a complete text response and captures the request', async () => {
|
||||
const events: MockLlmServerEvent[] = []
|
||||
const server = await start(['success'], {
|
||||
apiKey: 'mock-key',
|
||||
successText: 'recovered',
|
||||
chunkSize: 3,
|
||||
onEvent: (event) => { events.push(event) },
|
||||
})
|
||||
|
||||
const response = await chat(server, { key: 'mock-key' })
|
||||
const body = await response.text()
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(response.headers.get('content-type')).toContain('text/event-stream')
|
||||
expect(body).toContain('"content":"rec"')
|
||||
expect(body).toContain('"content":"ove"')
|
||||
expect(body).toContain('"content":"red"')
|
||||
expect(body).toContain('"finish_reason":"stop"')
|
||||
expect(body).toContain('data: [DONE]')
|
||||
expect(server.requests).toEqual([expect.objectContaining({
|
||||
attempt: 1,
|
||||
behavior: 'success',
|
||||
path: '/v1/chat/completions',
|
||||
body: { model: 'mock', messages: [], stream: true },
|
||||
chunksSent: 5,
|
||||
outcome: 'completed',
|
||||
})])
|
||||
expect(events).toEqual([
|
||||
{
|
||||
type: 'request',
|
||||
attempt: 1,
|
||||
scriptBehavior: 'success',
|
||||
behavior: 'success',
|
||||
path: '/v1/chat/completions',
|
||||
},
|
||||
{
|
||||
type: 'result',
|
||||
attempt: 1,
|
||||
scriptBehavior: 'success',
|
||||
behavior: 'success',
|
||||
outcome: 'completed',
|
||||
chunksSent: 5,
|
||||
},
|
||||
])
|
||||
})
|
||||
|
||||
it('supports root paths and intentionally ignores telemetry observer failures', async () => {
|
||||
const server = await start(['empty'], {
|
||||
onEvent() {
|
||||
throw new Error('observer failed')
|
||||
},
|
||||
})
|
||||
const response = await chat(server, { path: '/chat/completions' })
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(await response.text()).toContain('data: [DONE]')
|
||||
expect(server.requests[0]).toMatchObject({ path: '/chat/completions', outcome: 'completed' })
|
||||
})
|
||||
|
||||
it.each([
|
||||
['empty_body', 0, ''] as const,
|
||||
['stream_eof', 1, '"role":"assistant"'] as const,
|
||||
['partial_eof', 1, 'discarded partial response'] as const,
|
||||
['malformed_json', 2, 'data: {not-json'] as const,
|
||||
['malformed_event', 2, '"choices":[null]'] as const,
|
||||
])('serves %s without inventing a terminal completion', async (behavior, chunks, marker) => {
|
||||
const server = await start([behavior], { chunkSize: 100 })
|
||||
const response = await chat(server)
|
||||
const body = await response.text()
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(body).toContain(marker)
|
||||
if (behavior !== 'malformed_json' && behavior !== 'malformed_event') {
|
||||
expect(body).not.toContain('[DONE]')
|
||||
}
|
||||
expect(server.requests[0]).toMatchObject({ behavior, chunksSent: chunks, outcome: 'completed' })
|
||||
})
|
||||
|
||||
it.each([
|
||||
['connection_reset', false] as const,
|
||||
['stream_disconnect', true] as const,
|
||||
['partial_disconnect', true] as const,
|
||||
])('forces the %s transport boundary', async (behavior, receivesHeaders) => {
|
||||
const server = await start([behavior], { disconnectDelayMs: 20, partialText: 'half' })
|
||||
|
||||
let headersReceived = false
|
||||
await expect((async () => {
|
||||
const response = await chat(server)
|
||||
headersReceived = true
|
||||
await response.text()
|
||||
})()).rejects.toThrow()
|
||||
|
||||
expect(headersReceived).toBe(receivesHeaders)
|
||||
expect(server.requests[0]).toMatchObject({
|
||||
behavior,
|
||||
chunksSent: behavior === 'partial_disconnect' ? 1 : 0,
|
||||
outcome: 'reset',
|
||||
})
|
||||
})
|
||||
|
||||
it('holds a stalled stream until the client aborts and server close remains idempotent', async () => {
|
||||
const server = await start(['stall'])
|
||||
const controller = new AbortController()
|
||||
const response = await chat(server, { signal: controller.signal })
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(server.requests[0]).toMatchObject({ behavior: 'stall', outcome: 'stalled' })
|
||||
controller.abort()
|
||||
await expect(response.text()).rejects.toThrow()
|
||||
await server.close()
|
||||
await server.close()
|
||||
})
|
||||
|
||||
it.each([
|
||||
['slow_success', 100] as const,
|
||||
['stream_disconnect', 100] as const,
|
||||
['partial_disconnect', 100] as const,
|
||||
])('records a client that closes during %s', async (behavior, delayMs) => {
|
||||
const server = await start([behavior], {
|
||||
chunkDelayMs: delayMs,
|
||||
disconnectDelayMs: delayMs,
|
||||
chunkSize: 1,
|
||||
})
|
||||
const controller = new AbortController()
|
||||
const response = await chat(server, { signal: controller.signal })
|
||||
controller.abort()
|
||||
await expect(response.text()).rejects.toThrow()
|
||||
await new Promise((resolve) => { setTimeout(resolve, 5) })
|
||||
|
||||
expect(server.requests[0]).toMatchObject({ behavior, outcome: 'client_closed' })
|
||||
})
|
||||
|
||||
it('emits reasoning, tool calls, max-token finishes, slow chunks, and a wrong content type', async () => {
|
||||
const server = await start([
|
||||
'reasoning_success',
|
||||
'tool_call_success',
|
||||
'max_tokens',
|
||||
'slow_success',
|
||||
'wrong_content_type',
|
||||
], {
|
||||
successText: 'answer',
|
||||
reasoningText: 'think',
|
||||
toolName: 'lookup',
|
||||
toolArguments: '{"id":7}',
|
||||
chunkDelayMs: 1,
|
||||
chunkSize: 2,
|
||||
})
|
||||
|
||||
const bodies: string[] = []
|
||||
const contentTypes: Array<string | null> = []
|
||||
for (let index = 0; index < 5; index += 1) {
|
||||
const response = await chat(server)
|
||||
contentTypes.push(response.headers.get('content-type'))
|
||||
bodies.push(await response.text())
|
||||
}
|
||||
|
||||
expect(bodies[0]).toContain('"reasoning_content":"th"')
|
||||
expect(bodies[1]).toContain('"name":"lookup"')
|
||||
expect(bodies[1]).toContain('"arguments":"{\\"id"')
|
||||
expect(bodies[1]).toContain('"finish_reason":"tool_calls"')
|
||||
expect(bodies[2]).toContain('"finish_reason":"length"')
|
||||
expect(bodies[3]).toContain('"finish_reason":"stop"')
|
||||
expect(contentTypes[4]).toBe('application/json')
|
||||
expect(server.requests).toHaveLength(5)
|
||||
expect(server.requests.every(record => record.outcome === 'completed')).toBe(true)
|
||||
})
|
||||
|
||||
it.each([
|
||||
['rate_limit', 429, 'mock rate limit'] as const,
|
||||
['server_error', 500, 'mock server error'] as const,
|
||||
['service_unavailable', 503, 'mock service unavailable'] as const,
|
||||
['auth_error', 401, 'mock authentication failed'] as const,
|
||||
['invalid_request', 400, 'mock invalid request'] as const,
|
||||
['context_overflow', 400, 'context_length_exceeded'] as const,
|
||||
['quota_exceeded', 429, 'insufficient_quota'] as const,
|
||||
])('serves %s as a structured HTTP error', async (behavior, status, marker) => {
|
||||
const server = await start([behavior], { retryAfterMs: 1_001, requestId: 'mock-request-1' })
|
||||
const response = await chat(server)
|
||||
const body = await response.text()
|
||||
|
||||
expect(response.status).toBe(status)
|
||||
expect(body).toContain(marker)
|
||||
expect(response.headers.get('x-request-id')).toBe('mock-request-1')
|
||||
if (behavior === 'rate_limit') expect(response.headers.get('retry-after')).toBe('2')
|
||||
else expect(response.headers.get('retry-after')).toBeNull()
|
||||
expect(server.requests[0]?.outcome).toBe('completed')
|
||||
})
|
||||
|
||||
it('fails loud on script exhaustion and can explicitly repeat the final behavior', async () => {
|
||||
const exhausted = await start(['success'], { successText: 'once' })
|
||||
await (await chat(exhausted)).text()
|
||||
const exhaustedResponse = await chat(exhausted)
|
||||
expect(exhaustedResponse.status).toBe(500)
|
||||
expect(await exhaustedResponse.text()).toContain('mock script exhausted')
|
||||
expect(exhausted.requests.map(record => record.behavior)).toEqual(['success', 'script_exhausted'])
|
||||
|
||||
const repeating = await start(['empty'], { repeatLast: true })
|
||||
await (await chat(repeating)).text()
|
||||
await (await chat(repeating)).text()
|
||||
expect(repeating.requests.map(record => record.behavior)).toEqual(['empty', 'empty'])
|
||||
})
|
||||
|
||||
it('selects weighted random behaviors reproducibly and reports the concrete choice', async () => {
|
||||
const options = {
|
||||
sequence: ['random'] as const,
|
||||
repeatLast: true,
|
||||
randomSeed: 42,
|
||||
randomWeights: { success: 1, empty: 1 },
|
||||
successText: 'random success',
|
||||
}
|
||||
const first = await startMockLlmServer(options)
|
||||
const second = await startMockLlmServer(options)
|
||||
running.push(first, second)
|
||||
|
||||
for (let attempt = 0; attempt < 12; attempt += 1) {
|
||||
await (await chat(first)).text()
|
||||
await (await chat(second)).text()
|
||||
}
|
||||
|
||||
const firstChoices = first.requests.map(record => record.behavior)
|
||||
expect(first.randomSeed).toBe(42)
|
||||
expect(second.randomSeed).toBe(42)
|
||||
expect(firstChoices).toEqual(second.requests.map(record => record.behavior))
|
||||
expect(new Set(firstChoices)).toEqual(new Set(['success', 'empty']))
|
||||
expect(first.requests.every(record => record.scriptBehavior === 'random')).toBe(true)
|
||||
})
|
||||
|
||||
it('rejects invalid method, route, bearer token, and JSON without consuming the script', async () => {
|
||||
const server = await start(['success'], { apiKey: 'expected' })
|
||||
const method = await fetch(`${server.baseURL}/v1/chat/completions`)
|
||||
const route = await fetch(`${server.baseURL}/v1/other`, { method: 'POST', body: '{}' })
|
||||
const auth = await chat(server, { key: 'wrong' })
|
||||
const json = await chat(server, { key: 'expected', body: '{' })
|
||||
|
||||
expect(method.status).toBe(405)
|
||||
expect(method.headers.get('allow')).toBe('POST')
|
||||
expect(route.status).toBe(404)
|
||||
expect(auth.status).toBe(401)
|
||||
expect(json.status).toBe(400)
|
||||
expect(server.requests).toHaveLength(0)
|
||||
|
||||
const emptyRequest = await fetch(`${server.baseURL}/v1/chat/completions`, {
|
||||
method: 'POST',
|
||||
headers: { authorization: 'Bearer expected' },
|
||||
})
|
||||
expect(emptyRequest.status).toBe(200)
|
||||
expect(server.requests[0]?.behavior).toBe('success')
|
||||
expect(server.requests[0]?.body).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe('mock LLM server option validation', () => {
|
||||
it.each([
|
||||
[{ sequence: [] }, /sequence/],
|
||||
[{ sequence: ['success'], host: '' }, /host/],
|
||||
[{ sequence: ['success'], port: -1 }, /port/],
|
||||
[{ sequence: ['success'], port: 65_536 }, /port/],
|
||||
[{ sequence: ['success'], apiKey: '' }, /apiKey/],
|
||||
[{ sequence: ['success'], successText: '' }, /successText/],
|
||||
[{ sequence: ['success'], partialText: '' }, /partialText/],
|
||||
[{ sequence: ['success'], reasoningText: '' }, /reasoningText/],
|
||||
[{ sequence: ['success'], chunkSize: 0 }, /chunkSize/],
|
||||
[{ sequence: ['success'], chunkDelayMs: -1 }, /chunkDelayMs/],
|
||||
[{ sequence: ['success'], disconnectDelayMs: Number.POSITIVE_INFINITY }, /disconnectDelayMs/],
|
||||
[{ sequence: ['success'], retryAfterMs: 0 }, /retryAfterMs/],
|
||||
[{ sequence: ['success'], requestId: '' }, /requestId/],
|
||||
[{ sequence: ['success'], toolName: '' }, /toolName/],
|
||||
[{ sequence: ['success'], toolArguments: '{' }, /toolArguments/],
|
||||
[{ sequence: ['random'], randomSeed: -1 }, /randomSeed/],
|
||||
[{ sequence: ['random'], randomWeights: { random: 1 } }, /unknown concrete behavior/],
|
||||
[{ sequence: ['random'], randomWeights: { success: -1 } }, /non-negative/],
|
||||
[{ sequence: ['random'], randomWeights: { success: 0 } }, /positive weight/],
|
||||
] as const)('rejects invalid options %#', async (options, expected) => {
|
||||
await expect(startMockLlmServer(options as Parameters<typeof startMockLlmServer>[0]))
|
||||
.rejects.toThrow(expected)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../support/invariants"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
import { defineConfig } from 'tsdown'
|
||||
|
||||
/** Builds each public entry as a self-contained file admitted by the package whitelist. */
|
||||
export default defineConfig([
|
||||
{
|
||||
entry: ['lib/types/index.js'], outDir: 'lib', format: ['esm'], platform: 'node', target: 'es2024',
|
||||
fixedExtension: false, outputOptions: { codeSplitting: false }, dts: false, clean: false,
|
||||
},
|
||||
{
|
||||
entry: ['lib/types/invariant.js'], outDir: 'lib', format: ['esm'], platform: 'node', target: 'es2024',
|
||||
fixedExtension: false, outputOptions: { codeSplitting: false }, dts: false, clean: false,
|
||||
},
|
||||
{
|
||||
entry: ['lib/types/bin.js'], outDir: 'lib', format: ['esm'], platform: 'node', target: 'es2024',
|
||||
fixedExtension: false, outputOptions: { codeSplitting: false }, dts: false, clean: false,
|
||||
},
|
||||
])
|
||||
Generated
+18
@@ -2290,12 +2290,21 @@ importers:
|
||||
'@deepseek-ai/dsh-agent-loop':
|
||||
specifier: workspace:^
|
||||
version: link:../../core/agent-loop
|
||||
'@deepseek-ai/dsh-agent-loop-testkit':
|
||||
specifier: workspace:^
|
||||
version: link:../../support/agent-loop-testkit
|
||||
'@deepseek-ai/dsh-invariants':
|
||||
specifier: workspace:^
|
||||
version: link:../../support/invariants
|
||||
'@deepseek-ai/dsh-llm':
|
||||
specifier: workspace:^
|
||||
version: link:../llm
|
||||
'@deepseek-ai/dsh-llm-deepseek':
|
||||
specifier: workspace:^
|
||||
version: link:../llm-deepseek
|
||||
'@deepseek-ai/dsh-llm-mock-server':
|
||||
specifier: workspace:^
|
||||
version: link:../../support/llm-mock-server
|
||||
'@deepseek-ai/dsh-session':
|
||||
specifier: workspace:^
|
||||
version: link:../../core/session
|
||||
@@ -3455,6 +3464,15 @@ importers:
|
||||
specifier: ^4.0.0-rc.7
|
||||
version: 4.0.0-rc.7(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.5)
|
||||
|
||||
packages/support/llm-mock-server:
|
||||
devDependencies:
|
||||
'@deepseek-ai/dsh-invariants':
|
||||
specifier: workspace:^
|
||||
version: link:../invariants
|
||||
cordis:
|
||||
specifier: ^4.0.0-rc.7
|
||||
version: 4.0.0-rc.7(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.5)
|
||||
|
||||
packages/support/llm-replay:
|
||||
devDependencies:
|
||||
'@deepseek-ai/dsh-invariants':
|
||||
|
||||
@@ -90,6 +90,7 @@ const SENTENCE_MODEL_EXPERIENCE: Readonly<Record<string, SentenceContract>> = {
|
||||
'packages/support/agent-loop-testkit': { kind: 'none', reason: 'The test helper mounts services but neither drives nor modifies model requests.' },
|
||||
'packages/support/invariants': { kind: 'none', reason: 'The observer validates requests but never rewrites their context.' },
|
||||
'packages/support/loader-smoke': { kind: 'none', reason: 'The test harness observes child-process streams without changing live requests.' },
|
||||
'packages/support/llm-mock-server': { kind: 'none', reason: 'The test server substitutes provider wire behavior without invoking a real model.' },
|
||||
'packages/support/llm-replay': { kind: 'none', reason: 'The keyless adapter invokes no provider model.' },
|
||||
'packages/tasks/tasks': { kind: 'indirect', reason: 'Producer and control-surface plugins own all model rendering over the task registry.' },
|
||||
'packages/examples/acp-demo': { kind: 'indirect', reason: 'The app bundle delegates request composition to dsh-agent-spine-demo and dsh-acp.' },
|
||||
|
||||
@@ -115,6 +115,7 @@
|
||||
{ "path": "./packages/support/llm-replay" },
|
||||
{ "path": "./packages/support/acp-snapshot" },
|
||||
{ "path": "./packages/support/loader-smoke" },
|
||||
{ "path": "./packages/support/llm-mock-server" },
|
||||
{ "path": "./packages/subagent/subagent" },
|
||||
{ "path": "./packages/subagent/tool-subagent" },
|
||||
{ "path": "./packages/subagent/subagent-inprocess" },
|
||||
|
||||
Reference in New Issue
Block a user