test: measure live web streaming performance

2026-08-04 14:23:34 +08:00
parent a435aae538
commit 7f0b8a7e87
5 changed files with 208 additions and 9 deletions
@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md
-2026-07-24-web-gui-browser-e2e-lane.md: 4bdfa511b18a1e12199c1235179420afd903018a
-2026-07-24-web-gui-browser-e2e-lane.zh.md: f0f1957ad8e64f4cdb557f93102758a51ff7c946
+2026-07-24-web-gui-browser-e2e-lane.md: 51277846f7e796430081501a06c3164fc3e2cdd0
+2026-07-24-web-gui-browser-e2e-lane.zh.md: 0a625ca98257df813e3413c90cc47c3be563f3be
@@ -48,7 +48,7 @@ The lane covers three behavior families. Live-turn scenarios pin ordinary tool e

 The lane is a required compare-only gate for Linux pull requests under the [browser snapshot CI decision](2026-07-30-web-browser-snapshot-ci-gate.md). The `node 24 / snapshots and artifacts` consumer job owns the [single Linux build](../process/2026-07-30-independent-ci-consumer-build.md), installs the lockfile-selected Chromium, restores its OS-and-lockfile-keyed cache, and runs the lane with `DSH_SNAPSHOT=replay`. This is an intentional plane split: the host and specs use the [tsx source-launch contract](../architecture/2026-07-29-dsh-source-launch-tsx-esm.md), while the browser consumes `apps/web/dist` and package `lib/client.js` artifacts, so the gate depends on `built-package-invariants` for those client artifacts. The hosted and self-hosted default-branch Linux serial jobs run the same gate; the hosted job produces the browser cache consumed by pull requests, while the persistent self-hosted pool needs no hosted cache. CI never records or refreshes goldens. Scenarios remain POSIX-oriented and stay outside the Windows and macOS matrices.

-High-cardinality performance diagnostics use the separate opt-in `apps/web/tests/**/*.perf.ts` inventory selected only by `vitest.web.perf.config.ts`. The `complex-history.perf.ts` case reuses the real scaffold, seeds 1,000 compact sessions plus one 500-turn history containing 500 tool calls, and reports Chromium main-thread, DOM, listener, heap, paging, search, and Trajectory measurements. It carries structural assertions for the intended load shape but no timing thresholds because machine speed is not a correctness contract. The required `vitest.web.config.ts` inventory remains limited to `*.e2e.ts` and `*.snapshot.ts`, so neither `test:web:built` nor its CI gate collects performance cases.
+High-cardinality performance diagnostics use the separate opt-in `apps/web/tests/**/*.perf.ts` inventory selected only by `vitest.web.perf.config.ts`. The `complex-history.perf.ts` case reuses the real scaffold, seeds 1,000 compact sessions plus one 500-turn history containing 500 tool calls, and reports Chromium main-thread, DOM, listener, heap, paging, search, and Trajectory measurements. It then opens a blank session, fills a deterministic mixed-language prompt through the real composer, clicks Send, and replays 120 paced text deltas through the real loop and SSE wire; the report includes click-to-user-echo, click-to-first-chunk, stream completion, browser mutation, and persisted-chunk measurements. It carries structural assertions for the intended load and stream shapes but no timing thresholds because machine speed is not a correctness contract. The required `vitest.web.config.ts` inventory remains limited to `*.e2e.ts` and `*.snapshot.ts`, so neither `test:web:built` nor its CI gate collects performance cases.

 ## Prior art

@@ -48,7 +48,7 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu

 根据[浏览器快照 CI 决策](2026-07-30-web-browser-snapshot-ci-gate.md)，该车道是 Linux 拉取请求必需的只比较门禁。`node 24 / snapshots and artifacts` 消费方任务在[消费方独立构建](../process/2026-07-30-independent-ci-consumer-build.md)中负责唯一一次 Linux 构建，安装锁文件选定的 Chromium，恢复以操作系统和锁文件为键的缓存，并用 `DSH_SNAPSHOT=replay` 运行该车道。这是有意的平面切分：host 与 spec 使用 [tsx 源码启动契约](../architecture/2026-07-29-dsh-source-launch-tsx-esm.md)，浏览器则消费 `apps/web/dist` 和包的 `lib/client.js` 产物，因此门禁依赖 `built-package-invariants` 提供这些客户端产物。托管和自托管的默认分支 Linux 串行任务运行同一门禁；托管任务生成供 PR 消费的浏览器缓存，持久化自托管池则不需要托管侧缓存。CI 从不录制或刷新预期输出。场景仍面向 POSIX，并继续置于 Windows 和 macOS 矩阵之外。

-高基数性能诊断使用单独按需启用的 `apps/web/tests/**/*.perf.ts` 清单，并且只由 `vitest.web.perf.config.ts` 选中。`complex-history.perf.ts` 用例复用真实 scaffold，播种 1,000 个紧凑会话以及一份包含 500 次工具调用的 500 轮次历史，并报告 Chromium 主线程、DOM、监听器、堆内存、分页、搜索和 Trajectory 测量结果。它对预期负载形状设有结构性断言，但不设时间阈值，因为机器速度不属于正确性契约。必需的 `vitest.web.config.ts` 清单仍仅限 `*.e2e.ts` 和 `*.snapshot.ts`，因此 `test:web:built` 及其 CI 门禁都不会收集性能用例。
+高基数性能诊断使用单独按需启用的 `apps/web/tests/**/*.perf.ts` 清单，并且只由 `vitest.web.perf.config.ts` 选中。`complex-history.perf.ts` 用例复用真实 scaffold，播种 1,000 个紧凑会话以及一份包含 500 次工具调用的 500 轮次历史，并报告 Chromium 主线程、DOM、监听器、堆内存、分页、搜索和 Trajectory 测量结果。随后它打开一个空白会话，通过真实输入框填入一条确定性的混合语言提示词，点击 Send，并通过真实循环和 SSE wire 回放 120 个带节奏的文本增量；报告包括点击到用户消息回显、点击到首个分片、流完成、浏览器变更和持久化分片测量结果。它对预期负载和流形状设有结构性断言，但不设时间阈值，因为机器速度不属于正确性契约。必需的 `vitest.web.config.ts` 清单仍仅限 `*.e2e.ts` 和 `*.snapshot.ts`，因此 `test:web:built` 及其 CI 门禁都不会收集性能用例。

 ## 业界先例

@@ -2,16 +2,22 @@
 // rendering. It reports measurements without timing assertions because host
 // speed is not a correctness contract; structural assertions keep the load
 // shape from silently shrinking.
+import { mkdtemp, rm, writeFile } from 'node:fs/promises'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
 import { performance } from 'node:perf_hooks'
 import type { Browser, CDPSession, Locator, Page } from 'playwright'
 import { chromium } from 'playwright'
 import { afterAll, beforeAll, describe, expect, it } from 'vitest'
+import type { StreamChunk } from '@deepseek-ai/dsh-llm'
 import {
  CallId,
  createAssistantMessage,
  createToolResultMessage,
  createUserMessage,
 } from '@deepseek-ai/dsh-llm'
+import type { ReplayOverrideDoc } from '@deepseek-ai/dsh-llm-replay'
+import type { SessionEvent } from '@deepseek-ai/dsh-session'
 import {
  SESSION_FORMAT_VERSION,
  Session,
@@ -23,9 +29,10 @@ import {
  launchWebScaffold,
  seedSession,
  watchConsole,
+  webSnapshotMode,
  type WebScaffold,
 } from './scaffold.ts'
-import { newEnglishPage } from './support.ts'
+import { connectFreshWorkspace, newEnglishPage } from './support.ts'

 const SIDEBAR_SESSION_COUNT = 1_000
 const LONG_SESSION_ID = 'perf-long-history'
@@ -35,6 +42,33 @@ const TOOL_TURN_INTERVAL = 10
 const TOOLS_PER_TOOL_TURN = 10
 const EXPECTED_TOOL_CALLS = LONG_HISTORY_TURNS / TOOL_TURN_INTERVAL * TOOLS_PER_TOOL_TURN
 const EXPECTED_TRAJECTORY_ROWS = 2_100
+const STREAM_PACE_MS = 8
+const STREAM_DELTA_COUNT = 120
+const LIVE_PROMPT_MARKER = 'STREAM_PERF_USER_INPUT'
+const STREAM_FIRST_MARKER = 'STREAM_PERF_FIRST'
+const STREAM_DONE_MARKER = 'STREAM_PERF_DONE'
+const LIVE_PROMPT = [
+  LIVE_PROMPT_MARKER,
+  'Analyze the following mixed-language project context and return a concise diagnostic.',
+  ...Array.from(
+    { length: 48 },
+    (_, index) =>
+      `Context ${String(index + 1).padStart(2, '0')}: 用户正在检查长会话中的增量渲染性能。`
+      + ` Preserve item ${String(index)} and compare ${'payload'.repeat(8)}.`,
+  ),
+  '```ts',
+  ...Array.from(
+    { length: 40 },
+    (_, index) => `const sample_${String(index)} = ${JSON.stringify(`value-${String(index)}-${'x'.repeat(32)}`)}`,
+  ),
+  '```',
+].join('\n')
+const STREAM_DELTAS = Array.from({ length: STREAM_DELTA_COUNT }, (_, index) => {
+  if (index === 0) return `${STREAM_FIRST_MARKER} `
+  if (index === STREAM_DELTA_COUNT - 1) return `${STREAM_DONE_MARKER}.`
+  return `chunk-${String(index).padStart(3, '0')} ${'response'.repeat(3)} `
+})
+const STREAM_RESPONSE = STREAM_DELTAS.join('')

 interface ChromiumMetrics {
  readonly [name: string]: number
@@ -54,6 +88,11 @@ interface Measurement {
  readonly heapMb: number
 }

+interface MutationProbeResult {
+  readonly batches: number
+  readonly records: number
+}
+
 function text(value: string): { type: 'text'; text: string }[] {
  return [{ type: 'text', text: value }]
 }
@@ -255,6 +294,31 @@ function longHistoryFixture(): string {
  return fixtureLog(session)
 }

+function streamingReplayOverride(): ReplayOverrideDoc {
+  const chunks: StreamChunk[] = [
+    { type: 'block-start', index: 0, blockType: 'text' },
+    ...STREAM_DELTAS.map(text => ({
+      type: 'text-delta' as const,
+      index: 0,
+      text,
+    })),
+    {
+      type: 'block-end',
+      index: 0,
+      block: { type: 'text', text: STREAM_RESPONSE },
+    },
+    {
+      type: 'usage',
+      usage: {
+        inputTokens: Math.ceil(LIVE_PROMPT.length / 4),
+        outputTokens: Math.ceil(STREAM_RESPONSE.length / 4),
+      },
+    },
+    { type: 'finish', reason: { kind: 'stop' } },
+  ]
+  return [{ kind: 'chunks', chunks }]
+}
+
 function rounded(value: number): number {
  return Math.round(value * 1_000) / 1_000
 }
@@ -308,6 +372,43 @@ async function measure<T>(
  return { measurement: metricDelta(before, after, wallMs), value }
 }

+async function startMutationProbe(page: Page): Promise<void> {
+  await page.evaluate(() => {
+    const target = document.querySelector('[class*="centerCol"]')
+    if (target === null) throw new Error('stream mutation probe target is unavailable')
+    const probe = {
+      batches: 0,
+      records: 0,
+      observer: undefined as MutationObserver | undefined,
+    }
+    const observer = new MutationObserver((records) => {
+      probe.batches += 1
+      probe.records += records.length
+    })
+    probe.observer = observer
+    observer.observe(target, {
+      attributes: true,
+      attributeFilter: ['data-streaming'],
+      characterData: true,
+      childList: true,
+      subtree: true,
+    })
+    Reflect.set(globalThis, '__dshPerfMutationProbe', probe)
+  })
+}
+
+async function stopMutationProbe(page: Page): Promise<MutationProbeResult> {
+  return page.evaluate(() => {
+    const probe = Reflect.get(globalThis, '__dshPerfMutationProbe') as
+      | { batches: number; records: number; observer: MutationObserver }
+      | undefined
+    if (probe === undefined) throw new Error('stream mutation probe was not started')
+    probe.observer.disconnect()
+    Reflect.deleteProperty(globalThis, '__dshPerfMutationProbe')
+    return { batches: probe.batches, records: probe.records }
+  })
+}
+
 async function stableCount(
  locator: Locator,
  accepts: (count: number) => boolean,
@@ -341,10 +442,27 @@ describe('manual web performance: complex workspace and history', () => {
  let page: Page
  let setupMs = 0
  let tripwire: ReturnType<typeof watchConsole>
+  let replayDir: string | undefined
+  const sessionEvents: SessionEvent[] = []

  beforeAll(async () => {
+    if (webSnapshotMode() === 'record') {
+      throw new Error('manual web performance runs only with deterministic replay')
+    }
    const setupStarted = performance.now()
-    scaffold = await launchWebScaffold({})
+    replayDir = await mkdtemp(join(tmpdir(), 'dsh-web-perf-replay-'))
+    const replayOverride = join(replayDir, 'replay.override.json')
+    await writeFile(replayOverride, JSON.stringify(streamingReplayOverride()))
+    scaffold = await launchWebScaffold({
+      // A whole-script override needs no committed JSONL, but replayFixture
+      // selects the replay adapter and supplies its diagnostic identity.
+      replayFixture: join(replayDir, 'override-only.jsonl'),
+      replayOverride,
+      paceMs: STREAM_PACE_MS,
+    })
+    scaffold.ctx.on('session/event', (_session, event: SessionEvent) => {
+      sessionEvents.push(event)
+    })
    const small = smallSidebarFixture()
    for (let index = 0; index < SIDEBAR_SESSION_COUNT; index += 1) {
      await seedSession(scaffold, small, `perf-sidebar-${String(index).padStart(4, '0')}`)
@@ -358,8 +476,17 @@ describe('manual web performance: complex workspace and history', () => {
  })

  afterAll(async () => {
-    await browser?.close()
-    await scaffold?.close()
+    const failures: unknown[] = []
+    await browser?.close().catch((error: unknown) => failures.push(error))
+    await scaffold?.close().catch((error: unknown) => failures.push(error))
+    if (replayDir !== undefined) {
+      await rm(replayDir, { recursive: true, force: true })
+        .catch((error: unknown) => failures.push(error))
+    }
+    if (failures.length === 1) throw failures[0]
+    if (failures.length > 1) {
+      throw new AggregateError(failures, 'web performance teardown failed')
+    }
  })

  it('reports sidebar, paging, and trajectory rendering costs', async () => {
@@ -446,6 +573,62 @@ describe('manual web performance: complex workspace and history', () => {
    })
    expect(warmTrajectory.value).toBe(EXPECTED_TRAJECTORY_ROWS)

+    const newSession = await measure(cdp, async () => {
+      await page.getByRole('button', { name: 'New session', exact: true }).last().click()
+      await connectFreshWorkspace(page, 'stream-perf')
+      const input = page.locator(
+        'textarea:enabled[placeholder="Describe what you want to build"]',
+      )
+      await input.waitFor({ timeout: 15_000 })
+      return input
+    })
+
+    const composerInput = await measure(cdp, async () => {
+      await newSession.value.fill(LIVE_PROMPT)
+      await expect.poll(() => newSession.value.inputValue()).toBe(LIVE_PROMPT)
+      return (await newSession.value.inputValue()).length
+    })
+    expect(composerInput.value).toBe(LIVE_PROMPT.length)
+
+    await startMutationProbe(page)
+    const streamBefore = await chromiumMetrics(cdp)
+    const streamStarted = performance.now()
+    const settled = scaffold.whenTurnSettled(60_000)
+    await page.getByRole('button', { name: 'Send message', exact: true }).click()
+    await page.getByText(LIVE_PROMPT_MARKER, { exact: false }).last()
+      .waitFor({ timeout: 15_000 })
+    const clickToUserEchoMs = performance.now() - streamStarted
+    await page.getByText(STREAM_FIRST_MARKER, { exact: false }).last()
+      .waitFor({ timeout: 15_000 })
+    const clickToFirstChunkMs = performance.now() - streamStarted
+    await settled
+    await expect.poll(
+      () => page.locator('[data-streaming="true"]').count(),
+      { timeout: 15_000 },
+    ).toBe(0)
+    await page.getByText(STREAM_DONE_MARKER, { exact: false }).last()
+      .waitFor({ timeout: 15_000 })
+    const clickToSettledMs = performance.now() - streamStarted
+    const streamAfter = await chromiumMetrics(cdp)
+    const streamMutations = await stopMutationProbe(page)
+    const streamMeasurement = metricDelta(
+      streamBefore,
+      streamAfter,
+      clickToSettledMs,
+    )
+
+    const assistantChunks = sessionEvents.filter(
+      event => event.type === 'assistant/chunk',
+    )
+    expect(assistantChunks).toHaveLength(STREAM_DELTA_COUNT + 4)
+    const liveUser = sessionEvents.filter(event => event.type === 'user/message').at(-1)
+    if (liveUser?.type !== 'user/message') throw new Error('live performance prompt was not logged')
+    const liveUserText = liveUser.data.content
+      .filter(block => block.type === 'text')
+      .map(block => block.text)
+      .join('')
+    expect(liveUserText).toBe(LIVE_PROMPT)
+
    const report = {
      fixture: {
        sidebarSessions: SIDEBAR_SESSION_COUNT,
@@ -484,6 +667,22 @@ describe('manual web performance: complex workspace and history', () => {
        rows: warmTrajectory.value,
        ...warmTrajectory.measurement,
      },
+      liveInput: {
+        promptChars: LIVE_PROMPT.length,
+        newSession: newSession.measurement,
+        composerFill: composerInput.measurement,
+        stream: {
+          paceMs: STREAM_PACE_MS,
+          deltaChunks: STREAM_DELTA_COUNT,
+          persistedChunks: assistantChunks.length,
+          clickToUserEchoMs: rounded(clickToUserEchoMs),
+          clickToFirstChunkMs: rounded(clickToFirstChunkMs),
+          firstChunkToSettledMs: rounded(clickToSettledMs - clickToFirstChunkMs),
+          mutationBatches: streamMutations.batches,
+          mutationRecords: streamMutations.records,
+          ...streamMeasurement,
+        },
+      },
    }
    console.info(`WEB_PERF_RESULT ${JSON.stringify(report, null, 2)}`)
    expect(tripwire.warnings).toEqual([])
@@ -34,7 +34,7 @@
    "test:web:refresh": "npm run build && DSH_SNAPSHOT=refresh vitest run --config vitest.web.config.ts",
    "test:web:built": "vitest run --config vitest.web.config.ts",
    "test:web:perf": "npm run build && npm run test:web:perf:built",
-    "test:web:perf:built": "vitest run --config vitest.web.perf.config.ts",
+    "test:web:perf:built": "DSH_SNAPSHOT=replay vitest run --config vitest.web.perf.config.ts",
    "test:gui": "vitest run packages/client packages/host",
    "check:all": "tsx scripts/run-gates.ts check-all",
    "check:ci": "tsx scripts/run-gates.ts ci-primary",