diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml index f261b041a6..3b0096576d 100644 --- a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md -2026-07-24-web-gui-browser-e2e-lane.md: 4bdfa511b18a1e12199c1235179420afd903018a -2026-07-24-web-gui-browser-e2e-lane.zh.md: f0f1957ad8e64f4cdb557f93102758a51ff7c946 +2026-07-24-web-gui-browser-e2e-lane.md: 51277846f7e796430081501a06c3164fc3e2cdd0 +2026-07-24-web-gui-browser-e2e-lane.zh.md: 0a625ca98257df813e3413c90cc47c3be563f3be diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md index 4bdfa511b1..51277846f7 100644 --- a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md @@ -48,7 +48,7 @@ The lane covers three behavior families. Live-turn scenarios pin ordinary tool e The lane is a required compare-only gate for Linux pull requests under the [browser snapshot CI decision](2026-07-30-web-browser-snapshot-ci-gate.md). The `node 24 / snapshots and artifacts` consumer job owns the [single Linux build](../process/2026-07-30-independent-ci-consumer-build.md), installs the lockfile-selected Chromium, restores its OS-and-lockfile-keyed cache, and runs the lane with `DSH_SNAPSHOT=replay`. This is an intentional plane split: the host and specs use the [tsx source-launch contract](../architecture/2026-07-29-dsh-source-launch-tsx-esm.md), while the browser consumes `apps/web/dist` and package `lib/client.js` artifacts, so the gate depends on `built-package-invariants` for those client artifacts. The hosted and self-hosted default-branch Linux serial jobs run the same gate; the hosted job produces the browser cache consumed by pull requests, while the persistent self-hosted pool needs no hosted cache. CI never records or refreshes goldens. Scenarios remain POSIX-oriented and stay outside the Windows and macOS matrices. -High-cardinality performance diagnostics use the separate opt-in `apps/web/tests/**/*.perf.ts` inventory selected only by `vitest.web.perf.config.ts`. The `complex-history.perf.ts` case reuses the real scaffold, seeds 1,000 compact sessions plus one 500-turn history containing 500 tool calls, and reports Chromium main-thread, DOM, listener, heap, paging, search, and Trajectory measurements. It carries structural assertions for the intended load shape but no timing thresholds because machine speed is not a correctness contract. The required `vitest.web.config.ts` inventory remains limited to `*.e2e.ts` and `*.snapshot.ts`, so neither `test:web:built` nor its CI gate collects performance cases. +High-cardinality performance diagnostics use the separate opt-in `apps/web/tests/**/*.perf.ts` inventory selected only by `vitest.web.perf.config.ts`. The `complex-history.perf.ts` case reuses the real scaffold, seeds 1,000 compact sessions plus one 500-turn history containing 500 tool calls, and reports Chromium main-thread, DOM, listener, heap, paging, search, and Trajectory measurements. It then opens a blank session, fills a deterministic mixed-language prompt through the real composer, clicks Send, and replays 120 paced text deltas through the real loop and SSE wire; the report includes click-to-user-echo, click-to-first-chunk, stream completion, browser mutation, and persisted-chunk measurements. It carries structural assertions for the intended load and stream shapes but no timing thresholds because machine speed is not a correctness contract. The required `vitest.web.config.ts` inventory remains limited to `*.e2e.ts` and `*.snapshot.ts`, so neither `test:web:built` nor its CI gate collects performance cases. ## Prior art diff --git a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md index f0f1957ad8..0a625ca982 100644 --- a/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md +++ b/.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md @@ -48,7 +48,7 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu 根据[浏览器快照 CI 决策](2026-07-30-web-browser-snapshot-ci-gate.md),该车道是 Linux 拉取请求必需的只比较门禁。`node 24 / snapshots and artifacts` 消费方任务在[消费方独立构建](../process/2026-07-30-independent-ci-consumer-build.md)中负责唯一一次 Linux 构建,安装锁文件选定的 Chromium,恢复以操作系统和锁文件为键的缓存,并用 `DSH_SNAPSHOT=replay` 运行该车道。这是有意的平面切分:host 与 spec 使用 [tsx 源码启动契约](../architecture/2026-07-29-dsh-source-launch-tsx-esm.md),浏览器则消费 `apps/web/dist` 和包的 `lib/client.js` 产物,因此门禁依赖 `built-package-invariants` 提供这些客户端产物。托管和自托管的默认分支 Linux 串行任务运行同一门禁;托管任务生成供 PR 消费的浏览器缓存,持久化自托管池则不需要托管侧缓存。CI 从不录制或刷新预期输出。场景仍面向 POSIX,并继续置于 Windows 和 macOS 矩阵之外。 -高基数性能诊断使用单独按需启用的 `apps/web/tests/**/*.perf.ts` 清单,并且只由 `vitest.web.perf.config.ts` 选中。`complex-history.perf.ts` 用例复用真实 scaffold,播种 1,000 个紧凑会话以及一份包含 500 次工具调用的 500 轮次历史,并报告 Chromium 主线程、DOM、监听器、堆内存、分页、搜索和 Trajectory 测量结果。它对预期负载形状设有结构性断言,但不设时间阈值,因为机器速度不属于正确性契约。必需的 `vitest.web.config.ts` 清单仍仅限 `*.e2e.ts` 和 `*.snapshot.ts`,因此 `test:web:built` 及其 CI 门禁都不会收集性能用例。 +高基数性能诊断使用单独按需启用的 `apps/web/tests/**/*.perf.ts` 清单,并且只由 `vitest.web.perf.config.ts` 选中。`complex-history.perf.ts` 用例复用真实 scaffold,播种 1,000 个紧凑会话以及一份包含 500 次工具调用的 500 轮次历史,并报告 Chromium 主线程、DOM、监听器、堆内存、分页、搜索和 Trajectory 测量结果。随后它打开一个空白会话,通过真实输入框填入一条确定性的混合语言提示词,点击 Send,并通过真实循环和 SSE wire 回放 120 个带节奏的文本增量;报告包括点击到用户消息回显、点击到首个分片、流完成、浏览器变更和持久化分片测量结果。它对预期负载和流形状设有结构性断言,但不设时间阈值,因为机器速度不属于正确性契约。必需的 `vitest.web.config.ts` 清单仍仅限 `*.e2e.ts` 和 `*.snapshot.ts`,因此 `test:web:built` 及其 CI 门禁都不会收集性能用例。 ## 业界先例 diff --git a/apps/web/tests/complex-history.perf.ts b/apps/web/tests/complex-history.perf.ts index 084767312d..8c677663fa 100644 --- a/apps/web/tests/complex-history.perf.ts +++ b/apps/web/tests/complex-history.perf.ts @@ -2,16 +2,22 @@ // rendering. It reports measurements without timing assertions because host // speed is not a correctness contract; structural assertions keep the load // shape from silently shrinking. +import { mkdtemp, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' import { performance } from 'node:perf_hooks' import type { Browser, CDPSession, Locator, Page } from 'playwright' import { chromium } from 'playwright' import { afterAll, beforeAll, describe, expect, it } from 'vitest' +import type { StreamChunk } from '@deepseek-ai/dsh-llm' import { CallId, createAssistantMessage, createToolResultMessage, createUserMessage, } from '@deepseek-ai/dsh-llm' +import type { ReplayOverrideDoc } from '@deepseek-ai/dsh-llm-replay' +import type { SessionEvent } from '@deepseek-ai/dsh-session' import { SESSION_FORMAT_VERSION, Session, @@ -23,9 +29,10 @@ import { launchWebScaffold, seedSession, watchConsole, + webSnapshotMode, type WebScaffold, } from './scaffold.ts' -import { newEnglishPage } from './support.ts' +import { connectFreshWorkspace, newEnglishPage } from './support.ts' const SIDEBAR_SESSION_COUNT = 1_000 const LONG_SESSION_ID = 'perf-long-history' @@ -35,6 +42,33 @@ const TOOL_TURN_INTERVAL = 10 const TOOLS_PER_TOOL_TURN = 10 const EXPECTED_TOOL_CALLS = LONG_HISTORY_TURNS / TOOL_TURN_INTERVAL * TOOLS_PER_TOOL_TURN const EXPECTED_TRAJECTORY_ROWS = 2_100 +const STREAM_PACE_MS = 8 +const STREAM_DELTA_COUNT = 120 +const LIVE_PROMPT_MARKER = 'STREAM_PERF_USER_INPUT' +const STREAM_FIRST_MARKER = 'STREAM_PERF_FIRST' +const STREAM_DONE_MARKER = 'STREAM_PERF_DONE' +const LIVE_PROMPT = [ + LIVE_PROMPT_MARKER, + 'Analyze the following mixed-language project context and return a concise diagnostic.', + ...Array.from( + { length: 48 }, + (_, index) => + `Context ${String(index + 1).padStart(2, '0')}: 用户正在检查长会话中的增量渲染性能。` + + ` Preserve item ${String(index)} and compare ${'payload'.repeat(8)}.`, + ), + '```ts', + ...Array.from( + { length: 40 }, + (_, index) => `const sample_${String(index)} = ${JSON.stringify(`value-${String(index)}-${'x'.repeat(32)}`)}`, + ), + '```', +].join('\n') +const STREAM_DELTAS = Array.from({ length: STREAM_DELTA_COUNT }, (_, index) => { + if (index === 0) return `${STREAM_FIRST_MARKER} ` + if (index === STREAM_DELTA_COUNT - 1) return `${STREAM_DONE_MARKER}.` + return `chunk-${String(index).padStart(3, '0')} ${'response'.repeat(3)} ` +}) +const STREAM_RESPONSE = STREAM_DELTAS.join('') interface ChromiumMetrics { readonly [name: string]: number @@ -54,6 +88,11 @@ interface Measurement { readonly heapMb: number } +interface MutationProbeResult { + readonly batches: number + readonly records: number +} + function text(value: string): { type: 'text'; text: string }[] { return [{ type: 'text', text: value }] } @@ -255,6 +294,31 @@ function longHistoryFixture(): string { return fixtureLog(session) } +function streamingReplayOverride(): ReplayOverrideDoc { + const chunks: StreamChunk[] = [ + { type: 'block-start', index: 0, blockType: 'text' }, + ...STREAM_DELTAS.map(text => ({ + type: 'text-delta' as const, + index: 0, + text, + })), + { + type: 'block-end', + index: 0, + block: { type: 'text', text: STREAM_RESPONSE }, + }, + { + type: 'usage', + usage: { + inputTokens: Math.ceil(LIVE_PROMPT.length / 4), + outputTokens: Math.ceil(STREAM_RESPONSE.length / 4), + }, + }, + { type: 'finish', reason: { kind: 'stop' } }, + ] + return [{ kind: 'chunks', chunks }] +} + function rounded(value: number): number { return Math.round(value * 1_000) / 1_000 } @@ -308,6 +372,43 @@ async function measure( return { measurement: metricDelta(before, after, wallMs), value } } +async function startMutationProbe(page: Page): Promise { + await page.evaluate(() => { + const target = document.querySelector('[class*="centerCol"]') + if (target === null) throw new Error('stream mutation probe target is unavailable') + const probe = { + batches: 0, + records: 0, + observer: undefined as MutationObserver | undefined, + } + const observer = new MutationObserver((records) => { + probe.batches += 1 + probe.records += records.length + }) + probe.observer = observer + observer.observe(target, { + attributes: true, + attributeFilter: ['data-streaming'], + characterData: true, + childList: true, + subtree: true, + }) + Reflect.set(globalThis, '__dshPerfMutationProbe', probe) + }) +} + +async function stopMutationProbe(page: Page): Promise { + return page.evaluate(() => { + const probe = Reflect.get(globalThis, '__dshPerfMutationProbe') as + | { batches: number; records: number; observer: MutationObserver } + | undefined + if (probe === undefined) throw new Error('stream mutation probe was not started') + probe.observer.disconnect() + Reflect.deleteProperty(globalThis, '__dshPerfMutationProbe') + return { batches: probe.batches, records: probe.records } + }) +} + async function stableCount( locator: Locator, accepts: (count: number) => boolean, @@ -341,10 +442,27 @@ describe('manual web performance: complex workspace and history', () => { let page: Page let setupMs = 0 let tripwire: ReturnType + let replayDir: string | undefined + const sessionEvents: SessionEvent[] = [] beforeAll(async () => { + if (webSnapshotMode() === 'record') { + throw new Error('manual web performance runs only with deterministic replay') + } const setupStarted = performance.now() - scaffold = await launchWebScaffold({}) + replayDir = await mkdtemp(join(tmpdir(), 'dsh-web-perf-replay-')) + const replayOverride = join(replayDir, 'replay.override.json') + await writeFile(replayOverride, JSON.stringify(streamingReplayOverride())) + scaffold = await launchWebScaffold({ + // A whole-script override needs no committed JSONL, but replayFixture + // selects the replay adapter and supplies its diagnostic identity. + replayFixture: join(replayDir, 'override-only.jsonl'), + replayOverride, + paceMs: STREAM_PACE_MS, + }) + scaffold.ctx.on('session/event', (_session, event: SessionEvent) => { + sessionEvents.push(event) + }) const small = smallSidebarFixture() for (let index = 0; index < SIDEBAR_SESSION_COUNT; index += 1) { await seedSession(scaffold, small, `perf-sidebar-${String(index).padStart(4, '0')}`) @@ -358,8 +476,17 @@ describe('manual web performance: complex workspace and history', () => { }) afterAll(async () => { - await browser?.close() - await scaffold?.close() + const failures: unknown[] = [] + await browser?.close().catch((error: unknown) => failures.push(error)) + await scaffold?.close().catch((error: unknown) => failures.push(error)) + if (replayDir !== undefined) { + await rm(replayDir, { recursive: true, force: true }) + .catch((error: unknown) => failures.push(error)) + } + if (failures.length === 1) throw failures[0] + if (failures.length > 1) { + throw new AggregateError(failures, 'web performance teardown failed') + } }) it('reports sidebar, paging, and trajectory rendering costs', async () => { @@ -446,6 +573,62 @@ describe('manual web performance: complex workspace and history', () => { }) expect(warmTrajectory.value).toBe(EXPECTED_TRAJECTORY_ROWS) + const newSession = await measure(cdp, async () => { + await page.getByRole('button', { name: 'New session', exact: true }).last().click() + await connectFreshWorkspace(page, 'stream-perf') + const input = page.locator( + 'textarea:enabled[placeholder="Describe what you want to build"]', + ) + await input.waitFor({ timeout: 15_000 }) + return input + }) + + const composerInput = await measure(cdp, async () => { + await newSession.value.fill(LIVE_PROMPT) + await expect.poll(() => newSession.value.inputValue()).toBe(LIVE_PROMPT) + return (await newSession.value.inputValue()).length + }) + expect(composerInput.value).toBe(LIVE_PROMPT.length) + + await startMutationProbe(page) + const streamBefore = await chromiumMetrics(cdp) + const streamStarted = performance.now() + const settled = scaffold.whenTurnSettled(60_000) + await page.getByRole('button', { name: 'Send message', exact: true }).click() + await page.getByText(LIVE_PROMPT_MARKER, { exact: false }).last() + .waitFor({ timeout: 15_000 }) + const clickToUserEchoMs = performance.now() - streamStarted + await page.getByText(STREAM_FIRST_MARKER, { exact: false }).last() + .waitFor({ timeout: 15_000 }) + const clickToFirstChunkMs = performance.now() - streamStarted + await settled + await expect.poll( + () => page.locator('[data-streaming="true"]').count(), + { timeout: 15_000 }, + ).toBe(0) + await page.getByText(STREAM_DONE_MARKER, { exact: false }).last() + .waitFor({ timeout: 15_000 }) + const clickToSettledMs = performance.now() - streamStarted + const streamAfter = await chromiumMetrics(cdp) + const streamMutations = await stopMutationProbe(page) + const streamMeasurement = metricDelta( + streamBefore, + streamAfter, + clickToSettledMs, + ) + + const assistantChunks = sessionEvents.filter( + event => event.type === 'assistant/chunk', + ) + expect(assistantChunks).toHaveLength(STREAM_DELTA_COUNT + 4) + const liveUser = sessionEvents.filter(event => event.type === 'user/message').at(-1) + if (liveUser?.type !== 'user/message') throw new Error('live performance prompt was not logged') + const liveUserText = liveUser.data.content + .filter(block => block.type === 'text') + .map(block => block.text) + .join('') + expect(liveUserText).toBe(LIVE_PROMPT) + const report = { fixture: { sidebarSessions: SIDEBAR_SESSION_COUNT, @@ -484,6 +667,22 @@ describe('manual web performance: complex workspace and history', () => { rows: warmTrajectory.value, ...warmTrajectory.measurement, }, + liveInput: { + promptChars: LIVE_PROMPT.length, + newSession: newSession.measurement, + composerFill: composerInput.measurement, + stream: { + paceMs: STREAM_PACE_MS, + deltaChunks: STREAM_DELTA_COUNT, + persistedChunks: assistantChunks.length, + clickToUserEchoMs: rounded(clickToUserEchoMs), + clickToFirstChunkMs: rounded(clickToFirstChunkMs), + firstChunkToSettledMs: rounded(clickToSettledMs - clickToFirstChunkMs), + mutationBatches: streamMutations.batches, + mutationRecords: streamMutations.records, + ...streamMeasurement, + }, + }, } console.info(`WEB_PERF_RESULT ${JSON.stringify(report, null, 2)}`) expect(tripwire.warnings).toEqual([]) diff --git a/package.json b/package.json index d5c9aaa9f8..038a9f7316 100644 --- a/package.json +++ b/package.json @@ -34,7 +34,7 @@ "test:web:refresh": "npm run build && DSH_SNAPSHOT=refresh vitest run --config vitest.web.config.ts", "test:web:built": "vitest run --config vitest.web.config.ts", "test:web:perf": "npm run build && npm run test:web:perf:built", - "test:web:perf:built": "vitest run --config vitest.web.perf.config.ts", + "test:web:perf:built": "DSH_SNAPSHOT=replay vitest run --config vitest.web.perf.config.ts", "test:gui": "vitest run packages/client packages/host", "check:all": "tsx scripts/run-gates.ts check-all", "check:ci": "tsx scripts/run-gates.ts ci-primary",