fix(compact): size the compaction e2e to actually cross threshold; sync stale docs (CBR-007)
The compaction e2e never exercised compaction: its window/fixture combo (contextWindow 8000, thresholdRatio 0.5 → threshold 4000; four small files) peaked at ~1389 estimated tokens, so compactIfNeeded declined every pre-step and compact/start never landed. Shrink the window (contextWindow 2400 → threshold 1200; retainTokens 500 + summarizationMaxTokens 300 = 800 < 1200, convergence holds) and grow the fixture to six files so a couple of bash steps reliably cross the threshold. Verified compaction fires and the suite passes across repeated real-API runs. Sync docs left stale by the landed compaction work: list compaction.e2e.ts and keyless-smoke.e2e.ts in the coding-agent README (and fix the wrong "Both self-skip" count), add compaction to the examples with-key inventory, and replace the hypothetical compaction/marker / "future plugin" naming in the session README, session types JSDoc, and the core-data-structures catalog with the real compact/start, compact/summary, compact/end events.
This commit is contained in:
Hypatia May
@@ -205,7 +205,7 @@ type SessionEvent<T extends SessionEventType = SessionEventType> = {
|
||||
/**
|
||||
* Seq numbers of events that are provenance sources of this event
|
||||
* (e.g. the `assistant/chunk` seqs that built an `assistant/message`,
|
||||
* or the surface nodes shadowed by a compaction marker).
|
||||
* or the surface nodes shadowed by a compaction replace node).
|
||||
*/
|
||||
sourceEventSeqs?: number[]
|
||||
/** How this event entered the surface; absent for non-surface events. */
|
||||
|
||||
@@ -55,7 +55,7 @@ type SessionEvent<T extends SessionEventType = SessionEventType> = {
|
||||
/**
|
||||
* Seq numbers of events that are provenance sources of this event
|
||||
* (e.g. the `assistant/chunk` seqs that built an `assistant/message`,
|
||||
* or the surface nodes shadowed by a compaction marker).
|
||||
* or the surface nodes shadowed by a compaction replace node).
|
||||
*/
|
||||
sourceEventSeqs?: number[]
|
||||
/** How this event entered the surface; absent for non-surface events. */
|
||||
|
||||
+1
-1
@@ -20,7 +20,7 @@ A keyless smoke that spawns the example from a temp cwd must set `TSX_TSCONFIG_P
|
||||
| Example | Keyless smoke | With-key smoke |
|
||||
|---|---|---|
|
||||
| `echo-agent` | `tests/echo.e2e.ts` — boots the real `cordis.yml`, drives the echo tool round-trip and the direct canned reply | **N/A — keyless by nature** (the `mock-echo` model has no real provider) |
|
||||
| `coding-agent` | `tests/keyless-smoke.e2e.ts` — boots the full real tree (dummy key, no prompt → no model call), asserts banner + clean exit | `tests/{full-loop,coding-task,resume}.e2e.ts` — real model + real bash, world-verified |
|
||||
| `coding-agent` | `tests/keyless-smoke.e2e.ts` — boots the full real tree (dummy key, no prompt → no model call), asserts banner + clean exit | `tests/{full-loop,coding-task,resume,compaction}.e2e.ts` — real model + real bash, world-verified |
|
||||
| `acp-agent` | `pnpm run test:snapshot` — boots the real ACP subprocess and replays a recorded session keyless; `tests/acp.e2e.ts` also asserts stdout purity without a key | `tests/acp.e2e.ts` — real ACP prompt, verifies a file the agent wrote |
|
||||
|
||||
See [the root AGENTS.md](../AGENTS.md) for repo-wide conventions and [docs/architecture.md](../docs/architecture.md) for the design.
|
||||
@@ -48,5 +48,6 @@ This example is a thin leaf `cordis.yml`: it picks the swappable backends and lo
|
||||
- `tests/full-loop.e2e.ts` — the canary: real model runs `echo e2e-ok` through the real bash tool; asserts `tool/call`/`tool/result` session events and the final answer.
|
||||
- `tests/coding-task.e2e.ts` — the swebench-style smoke: a temp dir holds `add.js` (with `a - b` where `a + b` belongs) and a failing `add.test.js`; the agent must fix the bug and verify. The test re-runs `node add.test.js` ITSELF and inspects the files — agent claims are not trusted.
|
||||
- `tests/resume.e2e.ts` — durable continuity across processes: run 1 tells the real model a secret code and persists the turn to a temp JSONL root, then the whole context is disposed; run 2 is a fresh context over the same root that RESUMES the session id and asks the model to recall the code. The recall can only come from the rehydrated log.
|
||||
- `tests/compaction.e2e.ts` — the compaction smoke: a real multi-step bash task runs with a deliberately tiny context window so the auto-compaction listener fires MID-SESSION. Verifies the WORLD — a `compact/start…end` pair landed in the real log, the surface shrank (a replace node shadowed older nodes), and the agent still produced a correct final answer after compaction.
|
||||
|
||||
Both self-skip without `DEEPSEEK_API_KEY`.
|
||||
All four self-skip without `DEEPSEEK_API_KEY`. The keyless boot smoke is `tests/keyless-smoke.e2e.ts` (boots the full real tree with a dummy key and no prompt, so no model call), which runs in the default e2e gate.
|
||||
@@ -33,21 +33,23 @@ afterEach(async () => {
|
||||
describe.skipIf(!process.env.DEEPSEEK_API_KEY)('compaction: a long session compacts mid-flight and keeps running', () => {
|
||||
it('summarizes older history into a checkpoint without breaking the task', async () => {
|
||||
workdir = await mkdtemp(join(tmpdir(), 'dsh-compaction-'))
|
||||
// A few files for the model to read, so multiple bash steps accumulate
|
||||
// surface nodes (tool calls + results) and grow the history.
|
||||
for (let i = 1; i <= 4; i++) {
|
||||
// A handful of files for the model to read, so multiple bash steps
|
||||
// accumulate surface nodes (tool calls + results) and grow the history past
|
||||
// the (deliberately tiny) window.
|
||||
for (let i = 1; i <= 6; i++) {
|
||||
await writeFile(join(workdir, `file${i}.txt`), `This is file number ${i}. `.repeat(40))
|
||||
}
|
||||
|
||||
// Tiny window so a handful of steps crosses the threshold. The convergence
|
||||
// invariant requires summarizationMaxTokens + retainTokens <= window *
|
||||
// ratio = floor(8000 * 0.5) = 4000; 1500 + 2000 = 3500 <= 4000.
|
||||
// Tiny window so a couple of steps crosses the threshold. The convergence
|
||||
// invariant requires summarizationMaxTokens + retainTokens to be strictly
|
||||
// BELOW the threshold = floor(contextWindow * thresholdRatio) =
|
||||
// floor(2400 * 0.5) = 1200; 300 + 500 = 800 < 1200.
|
||||
ctx = await codingHarness(workdir, {
|
||||
compact: {
|
||||
contextWindow: 8000,
|
||||
contextWindow: 2400,
|
||||
thresholdRatio: 0.5,
|
||||
retainTokens: 2000,
|
||||
summarizationMaxTokens: 1500,
|
||||
retainTokens: 500,
|
||||
summarizationMaxTokens: 300,
|
||||
},
|
||||
})
|
||||
const agent = ctx.agentLoop.create(AgentId('e2e-compaction'), {
|
||||
@@ -57,9 +59,9 @@ describe.skipIf(!process.env.DEEPSEEK_API_KEY)('compaction: a long session compa
|
||||
|
||||
agent.send([{
|
||||
type: 'text',
|
||||
text: 'Read file1.txt, file2.txt, file3.txt, and file4.txt one at a time using cat '
|
||||
+ '(a separate bash command for each). After reading all four, tell me how many '
|
||||
+ 'files you read and the number mentioned in file1.txt.',
|
||||
text: 'Read file1.txt, file2.txt, file3.txt, file4.txt, file5.txt, and file6.txt one at a '
|
||||
+ 'time using cat (a separate bash command for each). After reading all six, tell me how '
|
||||
+ 'many files you read and the number mentioned in file1.txt.',
|
||||
}])
|
||||
await waitForIdle(ctx, agent)
|
||||
|
||||
@@ -87,9 +89,9 @@ describe.skipIf(!process.env.DEEPSEEK_API_KEY)('compaction: a long session compa
|
||||
expect(summaryData.shadowedSeqs.length).toBeGreaterThan(0)
|
||||
|
||||
// The conversation survived compaction: the agent produced a final answer
|
||||
// that reflects the work (it read four files).
|
||||
// that reflects the work (it read six files).
|
||||
const answer = finalText(events).toLowerCase()
|
||||
expect(answer.length).toBeGreaterThan(0)
|
||||
expect(answer).toMatch(/\b(4|four)\b/)
|
||||
expect(answer).toMatch(/\b(6|six)\b/)
|
||||
}, 240_000)
|
||||
})
|
||||
@@ -51,13 +51,13 @@ Plain class (not a Cordis Service). Create via `ctx.sessions.create()`.
|
||||
|
||||
The append-only log: `turn/start`, `turn/end`, `step/start`, `step/end`, `user/message`, `assistant/message`, `assistant/chunk`, `tool/call`, `tool/result`, `steering/message`, `context/message`. Token usage rides on `assistant/message.usage`; an operational error's step is on `turn/end.reason` for `kind: 'error'`.
|
||||
|
||||
Merge-extensible via `SessionEventMap` — a compaction plugin adds `compaction/marker`, etc.
|
||||
Merge-extensible via `SessionEventMap` — the compaction plugin (`dsh-compact-basic`) adds `compact/start`, `compact/summary`, `compact/end`, etc.
|
||||
|
||||
Also defines `TurnTriggerMap` and `TurnEndReasonMap` (merge-extensible sum types for typed turn boundaries — `kind`-tagged instead of strings).
|
||||
|
||||
Every `SessionEvent` carries two optional top-level fields (structural metadata):
|
||||
|
||||
- `sourceEventSeqs?: number[]` — seq numbers of provenance sources (e.g., the `assistant/chunk` seqs behind an `assistant/message`, or the shadowed nodes behind a compaction marker).
|
||||
- `sourceEventSeqs?: number[]` — seq numbers of provenance sources (e.g., the `assistant/chunk` seqs behind an `assistant/message`, or the shadowed nodes behind a compaction replace node).
|
||||
- `surfaceOp?: SurfaceOp` — how this event entered the surface. Absent for non-surface events (boundaries, chunks, usage, errors).
|
||||
|
||||
### Metadata types (`types.ts`)
|
||||
@@ -68,7 +68,7 @@ Every `SessionEvent` carries two optional top-level fields (structural metadata)
|
||||
|
||||
- Persistence plugins: subscribe to `session/event` (write-behind) and drain on `session/flush` (awaited) and fiber dispose. A durable backend reads the log and reloads it into a live session; the metadata seam (`SessionHeader`, `session.header`) is what such a backend stores beside the log.
|
||||
- Replay/fork: `ctx.sessions.create(id, { seed })` seeds a new session with an existing event log. The surface rebuilds deterministically from `surfaceOp` markers in the seeded events. The seed is validated to the SAME invariants `append` enforces — including that every surface-eligible event (`SurfaceEventType`) carries a `surfaceOp` marker — so a marker-less message event is rejected at construction rather than silently vanishing from `deriveMessages()` (the surface is the sole derivation path) on resume.
|
||||
- Compaction: a future plugin appends a new event with `surfaceOp: { op: 'replace', start, end }` to shadow old surface nodes.
|
||||
- Compaction: the `dsh-compact-basic` plugin appends a `user/message` with `surfaceOp: { op: 'replace', start, end }` to shadow old surface nodes behind a summary checkpoint.
|
||||
|
||||
### What is NOT here (TODO)
|
||||
|
||||
|
||||
@@ -156,7 +156,8 @@ export type TurnEndReason = TurnEndReasonMap[keyof TurnEndReasonMap]
|
||||
* same events; trace/telemetry = subscribe to the log.
|
||||
*
|
||||
* Merge-extensible: plugins declare extra event types via declaration merging
|
||||
* (e.g. a compaction plugin adds `'compaction/marker'`).
|
||||
* (e.g. the compaction plugin adds `'compact/start'`, `'compact/summary'`,
|
||||
* `'compact/end'`).
|
||||
*
|
||||
* Durability contract (what a persistence backend relies on): the durable log
|
||||
* persists every event verbatim, INCLUDING `assistant/chunk` — `seq` must stay
|
||||
@@ -279,7 +280,7 @@ export type SessionEvent<T extends SessionEventType = SessionEventType> = {
|
||||
/**
|
||||
* Seq numbers of events that are provenance sources of this event
|
||||
* (e.g. the `assistant/chunk` seqs that built an `assistant/message`,
|
||||
* or the surface nodes shadowed by a compaction marker).
|
||||
* or the surface nodes shadowed by a compaction replace node).
|
||||
*/
|
||||
sourceEventSeqs?: number[]
|
||||
/** How this event entered the surface; absent for non-surface events. */
|
||||
|
||||
Reference in New Issue
Block a user