diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml index da19d5558b..d5b558837f 100644 --- a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write -2026-07-22-evidence-based-larger-hosted-runners.md: 8d3852c56e2289bd222a260f2685f6a941a0cb19 -2026-07-22-evidence-based-larger-hosted-runners.zh.md: 4e7f846db1be2d68a080381943012583636ed2c4 +2026-07-22-evidence-based-larger-hosted-runners.md: 7d38552d23f7a3fb8abca45cbb4cf4e9c82fdd78 +2026-07-22-evidence-based-larger-hosted-runners.zh.md: f16db8eeb9000fe9a7671345efa9b1fd7fc744e1 diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md index 8d3852c56e..7d38552d23 100644 --- a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md @@ -14,13 +14,13 @@ Larger runners make it possible to pay setup once and parallelize inside the rep The organization keeps twelve x64 larger-runner pools in the repo-restricted `dsh-larger-ci` group: Ubuntu 24.04 and Windows 2025 at 4, 8, 16, 32, 64, and 96 cores. Public IPs are disabled. Each pool has an autoscaling ceiling of 256; the ceiling does not allocate idle machines or remove the need to bound workflow demand. -Production CI uses three larger-runner jobs and keeps Node compatibility, Python, and the final aggregator on standard runners: +Production CI uses three coarse larger-runner jobs and keeps Node compatibility, Python, and the final aggregator on standard runners: -- `node 24 / complete` uses the 96-core Linux pool. One checkout, setup, cache restore, and install feeds the complete unsharded 40-gate primary inventory. `run-gates` starts up to 32 independent gates, ESLint uses 32 workers, snapshots use up to 32 subprocesses, and coverage uses 16 forks. Build starts beside typecheck; snapshot and artifact consumers still wait for emitted output. Coverage stays below 32 forks because that setting twice caused Node 24's CJS lexer to terminate a Vitest worker and invalidate coverage. -- `windows / blocking builds` uses the 16-core Windows pool. Build and the production VitePress site run concurrently after one setup. -- `windows node 24 / observational` uses the 32-core Windows pool. Its complete unsharded 37-gate static, lint, and artifact inventory runs with 32 outer scheduler slots and remains non-blocking. ESLint itself stays single-threaded because 16 ESLint worker threads increased full-lint time to 174.54 seconds; with outer concurrency and no ESLint workers, the same full lint took 31.67 seconds. +- `node 24 / core` uses the 96-core Linux pool. One checkout, setup, cache restore, and install feeds the unsharded 39-gate primary inventory other than the production site build. `run-gates` starts up to 32 independent gates, ESLint uses 32 workers, snapshots use up to 32 subprocesses, and coverage uses 16 forks. Build starts beside typecheck; snapshot and artifact consumers still wait for emitted output. Coverage stays below 32 forks because that setting twice caused Node 24's CJS lexer to terminate a Vitest worker and invalidate coverage. +- `node 24 / production site` uses the 16-core Linux pool for the longest independent primary gate. This is one coarse split, not a shard matrix: the job performs one setup and one production VitePress build. +- `windows node 24 / complete` uses the 32-core Windows pool. One setup feeds the required package build, the required production site build, and the complete observational portability inventory. The outer scheduler has 32 slots. Required failures fail the job; observational failures are printed as non-blocking and preserve their former advisory status. ESLint itself stays single-threaded because 16 ESLint worker threads increased full-lint time to 174.54 seconds; outer gate concurrency uses the runner without multiplying Windows worker startup and TypeScript project loading. -The two Windows jobs deliberately use different pools. A first candidate put both on the 16-core pool and GitHub took 93 seconds to provision the second same-label runner despite the configured autoscaling ceiling. Using the 16-core and 32-core pools let every production job begin within two seconds in the final validation run. +The final shape followed two cold-path observations. A first candidate used two 16-core Windows jobs, and GitHub took 93 seconds to provision the second same-label runner despite the configured autoscaling ceiling. A later [documentation-head validation](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29900502413) took 64 seconds on the combined Linux job because its production site gate lasted 29.05 seconds, while a separate Windows blocking job took 266 seconds after spending 138 seconds restoring a 153 MB pnpm cache. Moving that independent Linux gate to one coarse 16-core lane and combining all Windows work on one 32-core box removed both duplicate setup waves. The workflow retains four manual diagnostics. `suite=larger-runner-benchmark` compares isolated critical lanes across every size, `suite=consolidated-runner-benchmark` compares whole aggregates, `suite=sharded-reference` preserves the former production shard topology, and `suite=serial-reference` remains the unsharded cross-platform completeness oracle. `suite=optimized-larger-runners` runs the exact production topology against a branch ref when a pull request cannot form a merge commit. @@ -33,23 +33,23 @@ The first [twelve-size critical-lane benchmark](https://github.com/deepseek-harn Those isolated results showed that setup dominated but did not identify the production size. A [whole-aggregate benchmark without native ESLint concurrency](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29897826082) found a 69-second single-threaded Linux lint gate. After enabling native Linux ESLint concurrency, the [second whole-aggregate benchmark](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29898331705) produced these active job times: -| Aggregate job | 4 cores | 8 cores | 16 cores | 32 cores | 64 cores | 96 cores | +| Job | 4 cores | 8 cores | 16 cores | 32 cores | 64 cores | 96 cores | |---|---:|---:|---:|---:|---:|---:| | Linux complete primary | 147 s | 104 s | 95 s | failed at 57 s | 66 s | 60 s | | Windows blocking builds | 137 s | 127 s | 113 s | 107 s | 105 s | 131 s | -The Linux 32-core failure was the first CJS-lexer worker crash. The 96-core aggregate was the only successful all-size result at the one-minute boundary. Windows gained little above 16 cores, so the blocking job uses 16 cores; the observational job uses a separate 32-core pool to avoid same-label provisioning delay and to start all outer gates together. +The Linux 32-core failure was the first CJS-lexer worker crash. The 96-core aggregate was the only successful all-size result at the one-minute boundary. Although Windows repository work gained little above 16 cores, the 32-core pool can start the complete outer inventory together and, more importantly, removes an entire paid setup from production. -The exact production [validation run](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29899733584) passed every job at the tested branch head: +The exact [reduced-fanout validation run](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29901539360) passed every job at the tested branch head: | Production job | Active time | Repository work | Result | |---|---:|---:|---:| -| Linux complete primary | 50 s | 40 gates in 23.23 s | passed | -| Slowest standard non-Windows job | 40 s | Node 26 compatibility | passed | -| Windows blocking builds | 91 s | 2 gates in 28.69 s | passed | -| Windows observational | 153 s | 37 gates in 31.83 s | passed | +| Linux core | 52 s | 39 gates in 25.84 s | passed | +| Linux production site | 44 s | VitePress in 22.42 s | passed | +| Slowest standard non-Windows job | 43 s | Node 26 compatibility | passed | +| Windows complete | 96 s | 37 gates in 31.13 s | passed | -The Windows observational job spent 57 seconds restoring its pnpm cache, so its remaining margin measures hosted setup variance as well as repository work. The final run still stays below one minute for every non-Windows job and below three minutes for both Windows jobs. +All three paid jobs began together. The Windows job spent 21 seconds restoring its pnpm cache and 25 seconds installing dependencies, so its margin measures hosted setup variance as well as repository work. Every non-Windows job stays below one minute and the sole Windows job stays below three minutes. ## Alternatives considered @@ -57,16 +57,20 @@ The Windows observational job spent 57 seconds restoring its pnpm cache, so its **Select a production size from the critical-lane benchmark.** Four cores looked cost-effective for isolated typecheck and site builds, but the full aggregate found repository-wide lint and dependent artifact work that those commands did not represent. -**Prebuild before starting the Linux aggregate.** This moved build onto the setup path and produced a 66-second candidate. Starting build eagerly inside `run-gates` preserves artifact dependencies while overlapping it with unrelated checks; the final aggregate completed in 23.23 seconds. +**Run the production site inside the Linux core aggregate.** This reached 50 seconds with warm hosted setup, then crossed the threshold at 64 seconds when the site gate took 29.05 seconds. One coarse independent site job protects the target without returning to gate-level sharding. + +**Keep required and observational Windows checks in separate jobs.** The split preserved status semantics at the workflow level but paid setup twice, and a cold cache pushed the required job to 266 seconds. `run-gates` now preserves those semantics inside one process: build and production site are required, while the remaining inventory is explicitly non-blocking. + +**Prebuild before starting the Linux aggregate.** This moved build onto the setup path and produced a 66-second candidate. Starting build eagerly inside `run-gates` preserves artifact dependencies while overlapping it with unrelated checks. **Use native ESLint worker concurrency on Windows.** Sixteen workers made lint more than five times slower than the final single-threaded result. Outer gate parallelism uses the 32-core runner without multiplying ESLint's Windows worker startup and TypeScript project loading. -**Move compatibility, Python, and aggregation to larger runners.** These standard-runner jobs all completed in 40 seconds or less. Paid capacity would not shorten the critical path. +**Move compatibility, Python, and aggregation to larger runners.** These standard-runner jobs completed in 43 seconds or less. Paid capacity would not shorten the critical path. ## Consequences -The final production validation consumed one billed 96-core Linux minute, two billed 16-core Windows minutes, and three billed 32-core Windows minutes. At the configured larger-runner rates, its larger-runner cost was $0.902. The all-size critical benchmark cost $2.936. GitHub rounds each larger-runner job up to a whole minute, so reducing paid job count from 49 to three matters as much as shortening repository work. +The reduced-fanout validation consumed one billed 96-core Linux minute, one billed 16-core Linux minute, and two billed 32-core Windows minutes. At the configured larger-runner rates, its larger-runner cost was $0.618. The all-size critical benchmark cost $2.936. GitHub rounds each larger-runner job up to a whole minute, so reducing paid job count from 49 to three matters as much as shortening repository work. The existing zero-dollar Actions budget did not block larger-runner jobs. The repo-only runner group, bounded workflow topology, manual benchmark triggers, and job timeouts are the observed cost controls; the budget is not treated as an execution guard. -Production CI depends on the organization-owned runner names in this note and in `.github/workflows/ci.yml`. Missing or renamed pools leave jobs queued instead of falling back to standard capacity. Manual all-size and former-shard suites remain available so image, dependency, scheduler, or pricing changes can be remeasured before changing production labels. +Production CI depends on the organization-owned runner names in this note and in `.github/workflows/ci.yml`. Missing or renamed pools leave jobs queued instead of falling back to standard capacity. Manual all-size, consolidated, former-shard, and serial suites remain available so image, dependency, scheduler, or pricing changes can be remeasured before changing production labels. diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md index 4e7f846db1..f16db8eeb9 100644 --- a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md @@ -14,13 +14,13 @@ Status: implemented 组织在仅限本仓库使用的 `dsh-larger-ci` 运行器组中保留 12 个 x64 大型运行器池:Ubuntu 24.04 和 Windows 2025 各设 4、8、16、32、64、96 核规格。公网 IP 已禁用。每个池的自动扩缩容上限为 256;该上限既不会分配闲置机器,也不能免除限制工作流需求的必要性。 -生产 CI 使用 3 个大型运行器作业,并让 Node 兼容性、Python 和最终聚合作业继续使用标准运行器: +生产 CI 使用 3 个粗粒度大型运行器作业,并让 Node 兼容性、Python 和最终聚合作业继续使用标准运行器: -- `node 24 / complete` 使用 96 核 Linux 池。只需执行一次代码检出、设置、缓存恢复和安装,即可供完整且未分片的 40 项主门禁清单使用。`run-gates` 最多同时启动 32 项相互独立的门禁,ESLint 使用 32 个工作线程,快照最多使用 32 个子进程,覆盖率运行使用 16 个 fork。构建与类型检查同时启动;快照和产物消费方仍会等待生成的输出。覆盖率运行的 fork 数保持低于 32,因为将其设为 32 曾两次导致 Node 24 的 CJS 词法分析器终止一个 Vitest 工作进程,使覆盖率结果失效。 -- `windows / blocking builds` 使用 16 核 Windows 池。一次设置完成后,构建任务与生产版 VitePress 网站任务并发运行。 -- `windows node 24 / observational` 使用 32 核 Windows 池。其完整且未分片的 37 项静态、lint 和产物门禁清单使用 32 个外层调度器槽位运行,并保持非阻塞。ESLint 本身仍采用单线程,因为启用 16 个 ESLint 工作线程会把完整 lint 的耗时增加至 174.54 秒;在启用外层并发且不使用 ESLint 工作线程时,同一项完整 lint 只需 31.67 秒。 +- `node 24 / core` 使用 96 核 Linux 池。只需执行一次代码检出、设置、缓存恢复和安装,即可供生产网站构建以外、未分片的 39 项主门禁清单使用。`run-gates` 最多同时启动 32 项相互独立的门禁,ESLint 使用 32 个工作线程,快照最多使用 32 个子进程,覆盖率运行使用 16 个 fork。构建与类型检查同时启动;快照和产物消费方仍会等待生成的输出。覆盖率运行的 fork 数保持低于 32,因为将其设为 32 曾两次导致 Node 24 的 CJS 词法分析器终止一个 Vitest 工作进程,使覆盖率结果失效。 +- `node 24 / production site` 使用 16 核 Linux 池运行耗时最长的独立主门禁。这是一项粗粒度拆分,而非分片矩阵:该作业执行一次设置和一次生产版 VitePress 构建。 +- `windows node 24 / complete` 使用 32 核 Windows 池。一次设置供必需的包构建、必需的生产网站构建以及完整的观测性可移植性清单共用。外层调度器设有 32 个槽位。任何必需项失败都会使作业失败;观测项失败会明确打印为非阻塞,并保留其原有的建议性状态。ESLint 本身仍采用单线程,因为启用 16 个 ESLint 工作线程会把完整 lint 的耗时增加至 174.54 秒;外层门禁并发能够利用该运行器,同时不会成倍增加 Windows 工作线程的启动和 TypeScript 项目加载开销。 -两个 Windows 作业有意使用不同的运行器池。首个候选方案让二者都使用 16 核池;尽管已配置自动扩缩容上限,GitHub 仍花费 93 秒才预配好第二台同标签运行器。分别使用 16 核池和 32 核池后,最终验证运行中的每个生产作业都在 2 秒内开始运行。 +最终结构由两项冷路径观测结果确定。首个候选方案使用两个 16 核 Windows 作业;尽管已配置自动扩缩容上限,GitHub 仍花费 93 秒才预配好第二台同标签运行器。随后一次[文档分支头验证](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29900502413)中,合并后的 Linux 作业耗时 64 秒,因为其中的生产网站门禁耗时 29.05 秒;另一个独立的 Windows 阻塞性作业则耗时 266 秒,其中恢复 153 MB pnpm 缓存就用了 138 秒。把这项独立的 Linux 门禁移入一个粗粒度 16 核通道,并把所有 Windows 工作合并到一台 32 核机器上,消除了这两轮重复设置。 工作流保留 4 项手动诊断。`suite=larger-runner-benchmark` 比较所有规格下相互独立的关键通道,`suite=consolidated-runner-benchmark` 比较完整聚合流程,`suite=sharded-reference` 保留原生产分片拓扑,`suite=serial-reference` 则继续作为未分片的跨平台完整性判定基准。当拉取请求无法生成合并提交时,`suite=optimized-larger-runners` 会直接针对分支引用运行与生产环境完全相同的拓扑。 @@ -33,23 +33,23 @@ Status: implemented 这些单项结果表明设置开销占主导地位,却无法确定生产环境应选用的规格。一项[未启用 ESLint 原生并发的完整聚合基准测试](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29897826082)发现,Linux 单线程 lint 门禁耗时 69 秒。启用 Linux ESLint 原生并发后,[第二次完整聚合基准测试](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29898331705)得到了以下作业活动耗时: -| 聚合作业 | 4 核 | 8 核 | 16 核 | 32 核 | 64 核 | 96 核 | +| 作业 | 4 核 | 8 核 | 16 核 | 32 核 | 64 核 | 96 核 | |---|---:|---:|---:|---:|---:|---:| | Linux 完整主流程 | 147 秒 | 104 秒 | 95 秒 | 57 秒时失败 | 66 秒 | 60 秒 | | Windows 阻塞性构建 | 137 秒 | 127 秒 | 113 秒 | 107 秒 | 105 秒 | 131 秒 | -Linux 32 核作业的失败是首次发生的 CJS 词法分析器工作进程崩溃。在所有规格的结果中,96 核聚合作业是唯一成功达到 1 分钟边界的结果。Windows 超过 16 核后的收益很小,因此阻塞性作业使用 16 核;观测作业则使用单独的 32 核池,以避免同标签运行器的预配延迟,并让全部外层门禁同时启动。 +Linux 32 核作业的失败是首次发生的 CJS 词法分析器工作进程崩溃。在所有规格的结果中,96 核聚合作业是唯一成功达到 1 分钟边界的结果。尽管 Windows 仓库工作在超过 16 核后收益很小,32 核池可以让完整的外层清单同时启动;更重要的是,它能从生产环境中消除整整一次付费设置。 -生产环境的精确[验证运行](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29899733584)在受测分支头通过了所有作业: +精确的[低扇出验证运行](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29901539360)在受测分支头通过了所有作业: | 生产作业 | 活动耗时 | 仓库工作 | 结果 | |---|---:|---:|---:| -| Linux 完整主流程 | 50 秒 | 40 项门禁耗时 23.23 秒 | 通过 | -| 最慢的标准非 Windows 作业 | 40 秒 | Node 26 兼容性 | 通过 | -| Windows 阻塞性构建 | 91 秒 | 2 项门禁耗时 28.69 秒 | 通过 | -| Windows 观测作业 | 153 秒 | 37 项门禁耗时 31.83 秒 | 通过 | +| Linux 核心 | 52 秒 | 39 项门禁耗时 25.84 秒 | 通过 | +| Linux 生产网站 | 44 秒 | VitePress 耗时 22.42 秒 | 通过 | +| 最慢的标准非 Windows 作业 | 43 秒 | Node 26 兼容性 | 通过 | +| Windows 完整流程 | 96 秒 | 37 项门禁耗时 31.13 秒 | 通过 | -Windows 观测作业花费 57 秒恢复 pnpm 缓存,因此其剩余余量既反映托管环境的设置波动,也反映仓库工作耗时。最终运行中每个非 Windows 作业仍低于 1 分钟,两个 Windows 作业也都低于 3 分钟。 +3 个付费作业同时开始运行。Windows 作业花费 21 秒恢复 pnpm 缓存、25 秒安装依赖,因此其余量既反映托管环境的设置波动,也反映仓库工作耗时。每个非 Windows 作业均低于 1 分钟,唯一的 Windows 作业也低于 3 分钟。 ## 曾考虑的替代方案 @@ -57,16 +57,20 @@ Windows 观测作业花费 57 秒恢复 pnpm 缓存,因此其剩余余量既 **根据关键通道基准测试选择生产规格。** 对单独的类型检查和网站构建而言,4 核看起来具备成本效益,但完整聚合流程发现了这些命令未覆盖的全仓库 lint 和存在依赖关系的产物工作。 -**在启动 Linux 聚合流程前预先构建。** 此方案让构建成为设置路径的一部分,并产生了一个耗时 66 秒的候选结果。在 `run-gates` 内尽早启动构建,既能保留产物依赖关系,又能让构建与无关检查重叠执行;最终聚合流程耗时 23.23 秒。 +**在 Linux 核心聚合流程内运行生产网站。** 托管环境设置处于热状态时,此方案用时 50 秒;当网站门禁耗时 29.05 秒时,总耗时增至 64 秒,超过了目标。一项独立的粗粒度网站作业能够保障目标,而无需重新采用门禁级分片。 + +**将必需的 Windows 检查和观测性 Windows 检查保留在不同作业中。** 这种拆分在工作流层保留了状态语义,却要支付两次设置开销,而且冷缓存让必需作业的耗时达到 266 秒。`run-gates` 目前在一个进程内保留这些语义:构建和生产网站属于必需项,其余清单则明确为非阻塞。 + +**在启动 Linux 聚合流程前预先构建。** 此方案让构建成为设置路径的一部分,并产生了一个耗时 66 秒的候选结果。在 `run-gates` 内尽早启动构建,既能保留产物依赖关系,又能让构建与无关检查重叠执行。 **在 Windows 上使用 ESLint 原生工作线程并发。** 16 个工作线程让 lint 比最终的单线程结果慢 5 倍以上。外层门禁并发能够利用 32 核运行器,同时不会成倍增加 ESLint 在 Windows 上启动工作线程和加载 TypeScript 项目的开销。 -**将兼容性、Python 和聚合作业迁移到大型运行器。** 这些标准运行器作业都在 40 秒以内完成。付费容量不会缩短关键路径。 +**将兼容性、Python 和聚合作业迁移到大型运行器。** 这些标准运行器作业都在 43 秒以内完成。付费容量不会缩短关键路径。 ## 后果 -最终生产验证产生的计费时长为:96 核 Linux 1 分钟、16 核 Windows 2 分钟和 32 核 Windows 3 分钟。按已配置的大型运行器费率计算,其大型运行器成本为 $0.902。全规格关键通道基准测试的成本为 $2.936。GitHub 会把每个大型运行器作业向上取整到整分钟计费,因此把付费作业数从 49 个减少到 3 个,与缩短仓库工作耗时同样重要。 +低扇出验证产生的计费时长为:96 核 Linux 1 分钟、16 核 Linux 1 分钟和 32 核 Windows 2 分钟。按已配置的大型运行器费率计算,其大型运行器成本为 $0.618。全规格关键通道基准测试的成本为 $2.936。GitHub 会把每个大型运行器作业向上取整到整分钟计费,因此把付费作业数从 49 个减少到 3 个,与缩短仓库工作耗时同样重要。 现有的零美元 Actions 预算并未阻止大型运行器作业。仅限本仓库的运行器组、有界的工作流拓扑、手动基准测试触发和作业超时限制才是经实测有效的成本控制机制;该预算不被视为执行防护措施。 -生产 CI 依赖本 Agent Note 和 `.github/workflows/ci.yml` 中由组织持有的运行器名称。池缺失或改名会让作业一直排队,不会回退到标准容量。手动全规格套件和原分片套件均予以保留,以便在映像、依赖、调度器或定价发生变化后重新测量,再调整生产标签。 +生产 CI 依赖本 Agent Note 和 `.github/workflows/ci.yml` 中由组织持有的运行器名称。池缺失或改名会让作业一直排队,不会回退到标准容量。手动全规格、完整聚合、原分片和串行套件均予以保留,以便在映像、依赖、调度器或定价发生变化后重新测量,再调整生产标签。