feat(tools): validate model-generated tool args at the boundary (RFC 005 pt 1)

defineTool now runs validateArgs against the SchemaSpec before execute, so a
malformed model call returns a self-correctable isError result listing the
violations instead of reaching the typed body untyped-in-practice. The
validator mirrors schemaSpecToJsonSchema semantics exactly (required from
required:true only, extra keys allowed, default not applied, object/array
without properties/items only type-checks, enum membership).

tool-bash's hand-rolled type/required checks (carrying the TODO(RFC 005)
stopgap note) are slimmed to just the value constraints the DSL can't express
(non-empty strings, positive timeout). Graduates RFC 005 pt 1 to ADR 0011.
This commit is contained in:
Tianyi Cui
2026-06-13 23:00:42 +08:00
parent 39b3db4b9c
commit 36a30180b8
10 changed files with 325 additions and 44 deletions
+20
View File
@@ -0,0 +1,20 @@
# ADR 0011: Runtime arg validation at the model boundary
Status: accepted (2026-06-13)
## Context
`defineTool` (ADR 0005) gives tool authors a typed `execute(args)` via the `InferArgs<S>` mapping. But that type is a compile-time claim about a value that arrives at runtime as model-generated JSON: nothing forced the model to honor the schema, so a malformed call — missing a required key, a string where a number was declared, an enum value outside the set — reached `execute` typed-in-name-only. The tool body then either crashed on the bad shape (a generic stack trace the model can't act on) or, worse, silently misbehaved. Meanwhile the converter already encodes the exact structure a validator would need to walk.
## Decision
`validateArgs(spec, args): string[]` interprets a `SchemaSpec` over a runtime value, returning human-readable violations (empty = valid), and is total (never throws). `defineTool` runs it before the typed body; on violations it throws `ToolArgsError` (`code: 'INVALID_ARGS'`, message listing the violations), which the registry's existing execute-waterfall catch turns into an `isError` result the model reads and self-corrects from.
The validator mirrors `schemaSpecToJsonSchema` semantics exactly — same structure walked, same rules: top level must be a non-array object; required keys come only from `required: true`; extra keys are allowed (no `additionalProperties: false`); `default` is not applied; an `object`/`array` prop without `properties`/`items` only type-checks; `enum` is membership. Raw-registered (MCP) tools are not touched — they validate their own input.
## Consequences
- The model gets actionable feedback on its own malformed calls instead of an opaque crash, closing the gap between `InferArgs`'s promise and runtime reality.
- The validator and `InferArgs` must stay in agreement; that drift risk is closed by a property test (RFC 001) generating args that satisfy `InferArgs` and asserting they pass `validateArgs`.
- `ToolArgsError` is a plain `Error` with a `code` field for now; if a harness-wide error taxonomy lands it becomes a subclass without changing callers that read `.message`.
- Validation cost is negligible next to a model call.
+1
View File
@@ -22,3 +22,4 @@ Do NOT write an ADR for: a mechanical or local choice (a variable name, a one-fi
| [0008](0008-tsdown-over-dumble.md) | tsdown for JS bundling instead of dumble | accepted |
| [0009](0009-capability-seams.md) | Capability seams — interface / implementation / consumer split | accepted |
| [0010](0010-twin-llm-adapters.md) | Two LLM adapters as a design-verification twin | accepted |
| [0011](0011-runtime-arg-validation.md) | Runtime arg validation at the model boundary | accepted |
@@ -1,6 +1,6 @@
# RFC 005: Runtime validation at the model boundary, error taxonomy, dev-mode invariants
Status: proposed
Status: partially implemented — part 1 (arg validation) → [ADR 0011](../adr/0011-runtime-arg-validation.md); parts 2-3 in progress
## Problem
+1 -1
View File
@@ -8,7 +8,7 @@ Proposals for substantial future work — reviewed before implementation, unlike
| [002](002-mutation-testing.md) | Mutation testing as the coverage counterweight | proposed |
| [003](003-deterministic-and-stress-testing.md) | Deterministic tests + replay invariant fixture + race stress | proposed |
| [004](004-architectural-conformance.md) | Architectural rules: dependency-cruiser, adapter conformance kit | proposed |
| [005](005-runtime-validation-and-error-taxonomy.md) | Runtime arg validation, structured error taxonomy, dev-mode invariants | proposed |
| [005](005-runtime-validation-and-error-taxonomy.md) | Runtime arg validation, structured error taxonomy, dev-mode invariants | partially implemented |
| [006](006-doc-sync-and-api-reports.md) | Doc-sync enforcement and API extractor reports | proposed |
| [007](007-supply-chain-and-vendor-drift.md) | Supply chain checks and vendor drift verification | proposed |
| [008](008-immutable-public-surfaces.md) | Deep-readonly public surfaces | proposed |
+15 -21
View File
@@ -28,14 +28,11 @@ export const name = 'tool-bash'
export const inject = ['tools', 'bash']
/**
* Validate model-produced arguments. `defineTool`'s `InferArgs` typing is
* compile-time only — at runtime `arguments` is whatever JSON the model
* emitted, so every field is checked before it reaches the executor.
*
* TODO(RFC 005): this hand-rolled validation is the per-tool stopgap until
* `defineTool` validates parsed args against the SchemaSpec itself (the
* converter already encodes the structure). When that lands, delete this and
* let the registry reject malformed calls — see docs/rfc/005.
* Validate the constraints the SchemaSpec can't express. `defineTool` now
* validates parsed args against the SchemaSpec before `execute` runs (RFC 005
* → ADR 0011), so type/required/enum checks are already done and `args` is
* the validated `InferArgs` shape here. What remains are value constraints the
* DSL has no vocabulary for: non-empty strings and a positive, finite timeout.
*/
function validateBashArgs(args: {
command: string
@@ -44,27 +41,24 @@ function validateBashArgs(args: {
workdir?: string
run_in_background?: boolean
}): void {
if (typeof args.command !== 'string' || args.command.trim().length === 0) {
if (args.command.trim().length === 0) {
throw new Error('invalid command: expected a non-empty string')
}
if (typeof args.description !== 'string' || args.description.trim().length === 0) {
if (args.description.trim().length === 0) {
throw new Error('invalid description: expected a non-empty string')
}
if (args.timeoutMs !== undefined
&& (typeof args.timeoutMs !== 'number' || !Number.isFinite(args.timeoutMs) || args.timeoutMs <= 0)) {
if (args.timeoutMs !== undefined && (!Number.isFinite(args.timeoutMs) || args.timeoutMs <= 0)) {
throw new Error(`invalid timeoutMs: expected a positive number, got ${JSON.stringify(args.timeoutMs)}`)
}
if (args.workdir !== undefined && typeof args.workdir !== 'string') {
throw new Error(`invalid workdir: expected a string, got ${JSON.stringify(args.workdir)}`)
}
if (args.run_in_background !== undefined && typeof args.run_in_background !== 'boolean') {
throw new Error(`invalid run_in_background: expected a boolean, got ${JSON.stringify(args.run_in_background)}`)
}
}
/** Require a string `task_id` (model-produced, so runtime-checked). */
function validateTaskId(value: unknown): string {
if (typeof value !== 'string' || value.length === 0) {
/**
* Reject an empty `task_id`. Type and presence are guaranteed by the
* SchemaSpec validation (ADR 0011); only the non-empty constraint, which the
* DSL can't express, is left to check here.
*/
function validateTaskId(value: string): string {
if (value.length === 0) {
throw new Error(`invalid task_id: expected a string, got ${JSON.stringify(value)}`)
}
return value
+26 -15
View File
@@ -118,19 +118,30 @@ describe('bash tool', () => {
expect(text(result)).toMatch(/aborted/)
})
// Type and required-key violations are now rejected by the harness
// (defineTool validates against the SchemaSpec — ADR 0011) before execute.
it.each([
[{}, /invalid command/],
[{ command: 42 }, /invalid command/],
[{ command: ' ' }, /invalid command/],
[{ command: 'x' }, /invalid description/],
[{ command: 'x', description: '' }, /invalid description/],
[{ command: 'x', description: 7 }, /invalid description/],
[{ command: 'x', description: 'd', timeoutMs: 'soon' }, /invalid timeoutMs/],
[{}, /missing required property "command"/],
[{ command: 42, description: 'd' }, /"command" must be a string/],
[{ command: 'x' }, /missing required property "description"/],
[{ command: 'x', description: 7 }, /"description" must be a string/],
[{ command: 'x', description: 'd', timeoutMs: 'soon' }, /"timeoutMs" must be a number/],
[{ command: 'x', description: 'd', workdir: 7 }, /"workdir" must be a string/],
[{ command: 'x', description: 'd', run_in_background: 'yes' }, /"run_in_background" must be a boolean/],
])('rejects schema-invalid args %j', async (args, pattern) => {
const ctx = await setup()
const result = await call(ctx, 'bash', args)
expect(result.isError).toBe(true)
expect(text(result)).toMatch(pattern)
})
// Value constraints the SchemaSpec can't express stay in the tool body.
it.each([
[{ command: ' ', description: 'd' }, /invalid command/],
[{ command: 'x', description: ' ' }, /invalid description/],
[{ command: 'x', description: 'd', timeoutMs: -1 }, /invalid timeoutMs/],
[{ command: 'x', description: 'd', timeoutMs: Number.NaN }, /invalid timeoutMs/],
[{ command: 'x', description: 'd', workdir: 7 }, /invalid workdir/],
[{ command: 'x', description: 'd', run_in_background: 'yes' }, /invalid run_in_background/],
])('rejects invalid args %j', async (args, pattern) => {
])('rejects value-invalid args %j', async (args, pattern) => {
const ctx = await setup()
const result = await call(ctx, 'bash', args)
expect(result.isError).toBe(true)
@@ -241,14 +252,14 @@ describe('background tools', () => {
})
it.each([
['bash_output', {}],
['bash_output', { task_id: 9 }],
['bash_kill', { task_id: '' }],
])('%s rejects invalid task_id %j', async (tool, args) => {
['bash_output', {}, /missing required property "task_id"/],
['bash_output', { task_id: 9 }, /"task_id" must be a string/],
['bash_kill', { task_id: '' }, /invalid task_id/],
])('%s rejects invalid task_id %j', async (tool, args, pattern) => {
const ctx = await setup()
const result = await call(ctx, tool, args)
expect(result.isError).toBe(true)
expect(text(result)).toMatch(/invalid task_id/)
expect(text(result)).toMatch(pattern)
})
it('injects a completion notice into the owning agent', async () => {
+3 -1
View File
@@ -59,7 +59,9 @@ ctx.tools.register(defineTool({
The helper converts the author-facing `SchemaSpec` (with `required: true` as a per-property boolean) to standard JSON Schema for the wire format. Raw JSON-Schema tool definitions (from MCP servers) are still accepted by the registry directly.
See `defineTool`, `SchemaSpec`, `InferArgs`, and `schemaSpecToJsonSchema` in the public API for details.
A `defineTool` tool also **validates the model-generated arguments against its `SchemaSpec` before `execute` runs** (`validateArgs`). The model's JSON is untrusted — `InferArgs<S>` is a compile-time claim, not a runtime guarantee — so on a mismatch (missing required key, wrong primitive, bad enum member, nested violation) the tool throws a `ToolArgsError` (`code: 'INVALID_ARGS'`); the registry turns it into an `isError` result whose text lists the violations, which the model sees and self-corrects from. Validation mirrors the JSON Schema conversion exactly: extra keys are allowed, `default` is not applied, and an `object`/`array` prop without `properties`/`items` only type-checks. Raw-registered tools (MCP) are **not** validated by the harness — they validate their own input.
See `defineTool`, `validateArgs`, `ToolArgsError`, `SchemaSpec`, `InferArgs`, and `schemaSpecToJsonSchema` in the public API for details.
### What is NOT here (TODO)
+2
View File
@@ -15,6 +15,8 @@ import type {} from '@deepseek-ai/dsh-system-prompt'
export {
defineTool,
schemaSpecToJsonSchema,
validateArgs,
ToolArgsError,
type SchemaSpec,
type SchemaProp,
type SchemaType,
+107 -4
View File
@@ -167,6 +167,101 @@ export function schemaSpecToJsonSchema(spec: SchemaSpec): JsonSchemaObject {
return result
}
// ---------------------------------------------------------------------------
// Runtime validation: model-generated args ↔ SchemaSpec
// ---------------------------------------------------------------------------
/**
* Thrown by a {@link defineTool} tool when the model-generated arguments don't
* match the declared {@link SchemaSpec}. The registry's execute waterfall
* catches it and returns an `isError` result so the model can self-correct.
*
* Plain `Error` for now (carries a `code` field); a later change promotes the
* harness error taxonomy and this extends a common base.
*/
export class ToolArgsError extends Error {
/** Machine-routable code; stable across the message wording. */
readonly code = 'INVALID_ARGS'
/** The individual violation messages, in declaration order. */
readonly violations: string[]
constructor(violations: string[]) {
super(`invalid arguments: ${violations.join('; ')}`)
this.name = 'ToolArgsError'
this.violations = violations
}
}
/** Whether a value is a non-null, non-array object (a JSON Schema `object`). */
function isPlainObject(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null && !Array.isArray(value)
}
/** Collect violations for one property value against its {@link SchemaProp}. */
function checkValue(prop: SchemaProp, value: unknown, path: string): string[] {
switch (prop.type) {
case 'string': {
if (typeof value !== 'string') return [`"${path}" must be a string`]
if (prop.enum && !prop.enum.includes(value)) {
return [`"${path}" must be one of ${JSON.stringify(prop.enum)}`]
}
return []
}
case 'number': {
return typeof value === 'number' ? [] : [`"${path}" must be a number`]
}
case 'boolean': {
return typeof value === 'boolean' ? [] : [`"${path}" must be a boolean`]
}
case 'object': {
if (!isPlainObject(value)) return [`"${path}" must be an object`]
// Mirror the converter: an object without `properties` only type-checks.
return prop.properties ? checkSpec(prop.properties, value, path) : []
}
case 'array': {
if (!Array.isArray(value)) return [`"${path}" must be an array`]
// Mirror the converter: an array without `items` only type-checks.
if (!prop.items) return []
const items = prop.items
return value.flatMap((el, i) => checkValue(items, el, `${path}[${i}]`))
}
// No default: SchemaType is a closed union; every case is handled above.
}
}
/** Collect violations for an object value against a {@link SchemaSpec}. */
function checkSpec(spec: SchemaSpec, value: unknown, path: string): string[] {
if (!isPlainObject(value)) return [`"${path || 'arguments'}" must be an object`]
const violations: string[] = []
for (const [key, prop] of Object.entries(spec)) {
const propPath = path ? `${path}.${key}` : key
const v = value[key]
if (v === undefined) {
// A required key absent OR present-but-undefined is a violation; an
// optional absent key is fine. `default` is NOT applied (validation only).
if (prop.required === true) violations.push(`missing required property "${propPath}"`)
continue
}
violations.push(...checkValue(prop, v, propPath))
}
return violations
}
/**
* Validate model-generated `args` against a {@link SchemaSpec}, returning a
* list of human-readable violation messages (empty = valid). Total — never
* throws, regardless of how malformed `args` is.
*
* Semantics mirror {@link schemaSpecToJsonSchema} exactly: the top level must
* be a non-array object; required keys come only from `required: true`; extra
* keys are allowed (no `additionalProperties: false`); `default` is not
* applied; an `object`/`array` prop without `properties`/`items` only
* type-checks; `enum` is membership (strings only).
*/
export function validateArgs(spec: SchemaSpec, args: unknown): string[] {
return checkSpec(spec, args, '')
}
// ---------------------------------------------------------------------------
// defineTool — typed helper for first-party plugin authors
// ---------------------------------------------------------------------------
@@ -219,14 +314,22 @@ export interface DefineToolOptions<S extends SchemaSpec> {
* first-party plugin authors.
*/
export function defineTool<S extends SchemaSpec>(options: DefineToolOptions<S>): ToolDefinition {
// Object-literal execute methods don't use `this`; the reference is safe.
// eslint-disable-next-line @typescript-eslint/unbound-method
const userExecute = options.execute
return {
name: options.name,
description: options.description,
parameters: schemaSpecToJsonSchema(options.parameters) as unknown as Record<string, unknown>,
...options.strict !== undefined ? { strict: options.strict } : {},
// Object-literal execute methods don't use `this`; passing the reference
// through is safe.
// eslint-disable-next-line @typescript-eslint/unbound-method
execute: options.execute,
async execute(args: unknown, exec: ToolExecution): Promise<ContentBlock[]> {
// Validate the model-generated args before the typed body runs. On
// mismatch we throw ToolArgsError; the registry turns it into an
// isError result so the model can self-correct. After this guard, the
// cast to InferArgs<S> reflects the validated shape.
const violations = validateArgs(options.parameters, args)
if (violations.length > 0) throw new ToolArgsError(violations)
return userExecute(args as InferArgs<S>, exec)
},
}
}
+149 -1
View File
@@ -3,7 +3,7 @@ import { Context } from 'cordis'
import { CallId } from '@deepseek-ai/dsh-llm'
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
import ToolRegistry, {
defineTool, schemaSpecToJsonSchema,
defineTool, schemaSpecToJsonSchema, validateArgs, ToolArgsError,
type InferArgs, type SchemaSpec, type ToolExecutionResult,
} from '@deepseek-ai/dsh-tools'
@@ -574,3 +574,151 @@ describe('ToolRegistry.get', () => {
expect(ctx.tools.get('nope')).toBeUndefined()
})
})
describe('validateArgs (RFC 005 part 1)', () => {
it('returns [] for valid args and is total over malformed input', () => {
const spec = {
path: { type: 'string', required: true },
limit: { type: 'number' },
} satisfies SchemaSpec
expect(validateArgs(spec, { path: '/tmp' })).toEqual([])
expect(validateArgs(spec, { path: '/tmp', limit: 5 })).toEqual([])
// never throws regardless of shape
expect(validateArgs(spec, null)).toHaveLength(1)
expect(validateArgs(spec, 'nope')).toHaveLength(1)
expect(validateArgs(spec, [])).toHaveLength(1)
})
it('flags a missing required key and a required key present as undefined', () => {
const spec = { path: { type: 'string', required: true } } satisfies SchemaSpec
expect(validateArgs(spec, {})).toEqual(['missing required property "path"'])
expect(validateArgs(spec, { path: undefined })).toEqual(['missing required property "path"'])
})
it('allows extra keys (no additionalProperties:false) and omitted optionals', () => {
const spec = { path: { type: 'string', required: true } } satisfies SchemaSpec
expect(validateArgs(spec, { path: '/tmp', extra: 1 })).toEqual([])
})
it('does not apply defaults (validation only)', () => {
const spec = { limit: { type: 'number', default: 25 } } satisfies SchemaSpec
// absent optional is valid, and validation does not synthesize the default
expect(validateArgs(spec, {})).toEqual([])
})
it('type-checks primitives', () => {
const spec = {
s: { type: 'string' },
n: { type: 'number' },
b: { type: 'boolean' },
} satisfies SchemaSpec
expect(validateArgs(spec, { s: 1 })).toEqual(['"s" must be a string'])
expect(validateArgs(spec, { n: 'x' })).toEqual(['"n" must be a number'])
expect(validateArgs(spec, { b: 'x' })).toEqual(['"b" must be a boolean'])
})
it('checks enum membership', () => {
const spec = { color: { type: 'string', enum: ['red', 'green'] } } satisfies SchemaSpec
expect(validateArgs(spec, { color: 'red' })).toEqual([])
expect(validateArgs(spec, { color: 'blue' })).toEqual(['"color" must be one of ["red","green"]'])
})
it('recurses into nested objects (and an object without properties only type-checks)', () => {
const spec = {
config: {
type: 'object',
required: true,
properties: { host: { type: 'string', required: true }, port: { type: 'number' } },
},
bag: { type: 'object' },
} satisfies SchemaSpec
expect(validateArgs(spec, { config: { host: 'h' }, bag: { anything: true } })).toEqual([])
expect(validateArgs(spec, { config: { port: 9 }, bag: 5 })).toEqual([
'missing required property "config.host"',
'"bag" must be an object',
])
})
it('recurses into array items (and an array without items only type-checks)', () => {
const spec = {
tags: { type: 'array', items: { type: 'string' } },
raw: { type: 'array' },
} satisfies SchemaSpec
expect(validateArgs(spec, { tags: ['a', 'b'], raw: [1, {}, 'x'] })).toEqual([])
expect(validateArgs(spec, { tags: ['a', 2] })).toEqual(['"tags[1]" must be a string'])
// a non-array value for an array-typed prop
expect(validateArgs(spec, { tags: 'nope' })).toEqual(['"tags" must be an array'])
})
it('validates arrays of objects element-wise', () => {
const spec = {
servers: {
type: 'array',
items: { type: 'object', properties: { host: { type: 'string', required: true } } },
},
} satisfies SchemaSpec
expect(validateArgs(spec, { servers: [{ host: 'a' }, {}] })).toEqual([
'missing required property "servers[1].host"',
])
})
})
describe('defineTool validation (RFC 005 part 1)', () => {
it('returns an isError result with the violations when the model sends bad args', async () => {
const ctx = await setup()
ctx.tools.register(defineTool({
name: 'reader',
description: 'reads a path',
parameters: { path: { type: 'string', required: true } },
async execute(args) {
return [{ type: 'text', text: args.path }]
},
}))
const result = await ctx.tools.execute({ callId: CallId('c1'), name: 'reader', arguments: {} })
expect(result.isError).toBe(true)
expect(result.content[0]).toMatchObject({
text: 'Error: invalid arguments: missing required property "path"',
})
})
it('runs execute normally when args are valid', async () => {
const ctx = await setup()
ctx.tools.register(defineTool({
name: 'reader',
description: 'reads a path',
parameters: { path: { type: 'string', required: true } },
async execute(args) {
return [{ type: 'text', text: `read ${args.path}` }]
},
}))
const result = await ctx.tools.execute({ callId: CallId('c1'), name: 'reader', arguments: { path: '/x' } })
expect(result).toEqual({ callId: CallId('c1'), content: [{ type: 'text', text: 'read /x' }], isError: false })
})
it('ToolArgsError carries a stable code and the violation list', () => {
const err = new ToolArgsError(['missing required property "a"', '"b" must be a number'])
expect(err).toBeInstanceOf(Error)
expect(err.name).toBe('ToolArgsError')
expect(err.code).toBe('INVALID_ARGS')
expect(err.violations).toEqual(['missing required property "a"', '"b" must be a number'])
expect(err.message).toBe('invalid arguments: missing required property "a"; "b" must be a number')
})
it('raw-registered tools are NOT validated by defineTool (MCP keeps its own)', async () => {
const ctx = await setup()
// A raw ToolDefinition: no defineTool wrapping, so no validateArgs guard.
ctx.tools.register({
name: 'raw',
description: 'raw tool',
parameters: { type: 'object', properties: { path: { type: 'string' } }, required: ['path'] },
async execute(args: unknown) {
return [{ type: 'text', text: typeof args }]
},
})
// Missing the "required" path — but raw tools validate their own input, so
// this reaches execute rather than being rejected by the harness.
const result = await ctx.tools.execute({ callId: CallId('c1'), name: 'raw', arguments: {} })
expect(result.isError).toBe(false)
})
})