docs: record all-pool validation

2026-07-22 16:21:17 +08:00
parent 9d81078f2f
commit 2bd466a4a6
3 changed files with 44 additions and 26 deletions
@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write
-2026-07-22-evidence-based-larger-hosted-runners.md: 7d38552d23f7a3fb8abca45cbb4cf4e9c82fdd78
-2026-07-22-evidence-based-larger-hosted-runners.zh.md: f16db8eeb9000fe9a7671345efa9b1fd7fc744e1
+2026-07-22-evidence-based-larger-hosted-runners.md: 5390cce001f5d493fc4fb3eaee9203c34af1175d
+2026-07-22-evidence-based-larger-hosted-runners.zh.md: 04a32f9ac1e0b131a695695b8f6cece3a8ffd3fa
@@ -14,13 +14,17 @@ Larger runners make it possible to pay setup once and parallelize inside the rep

 The organization keeps twelve x64 larger-runner pools in the repo-restricted `dsh-larger-ci` group: Ubuntu 24.04 and Windows 2025 at 4, 8, 16, 32, 64, and 96 cores. Public IPs are disabled. Each pool has an autoscaling ceiling of 256; the ceiling does not allocate idle machines or remove the need to bound workflow demand.

-Production CI uses three coarse larger-runner jobs and keeps Node compatibility, Python, and the final aggregator on standard runners:
+Production CI assigns each of the six Linux pool sizes exactly once, assigns one 32-core Windows pool, and keeps only the final aggregator on a standard runner. The version and language jobs are environment contracts rather than slices of one gate inventory; the primary Node work has three coarse lanes instead of a gate-level shard matrix:

- `node 24 / core` uses the 96-core Linux pool. One checkout, setup, cache restore, and install feeds the unsharded 39-gate primary inventory other than the production site build. `run-gates` starts up to 32 independent gates, ESLint uses 32 workers, snapshots use up to 32 subprocesses, and coverage uses 16 forks. Build starts beside typecheck; snapshot and artifact consumers still wait for emitted output. Coverage stays below 32 forks because that setting twice caused Node 24's CJS lexer to terminate a Vitest worker and invalidate coverage.
+- `node 24 / core` uses the 96-core Linux pool. One checkout, setup, cache restore, and install feeds 36 unsharded static, lint, documentation, hygiene, build, and artifact gates. `run-gates` starts up to 32 independent gates and ESLint uses 32 workers. Build starts eagerly; its artifact consumers still wait for emitted output.
+- `node 24 / cpu` uses the 64-core Linux pool for six CPU- or dependency-critical gates: typecheck, coverage, build followed by snapshot replay, and two Node 24 compatibility smokes. Coverage and snapshot each use at most 16 workers. This lane builds separately so snapshot replay consumes same-lane output. Coverage stays below 32 forks because that setting twice caused Node 24's CJS lexer to terminate a Vitest worker and invalidate coverage.
 - `node 24 / production site` uses the 16-core Linux pool for the longest independent primary gate. This is one coarse split, not a shard matrix: the job performs one setup and one production VitePress build.
+- Node 22.19 compatibility, Python 3.10, and Node 26 compatibility use the 4-, 8-, and 32-core Linux pools respectively. Distinct labels avoid both standard-runner setup outliers and the delayed second allocation observed when two jobs shared one pool.
 - `windows node 24 / complete` uses the 32-core Windows pool. One setup feeds the required package build, the required production site build, and the complete observational portability inventory. The outer scheduler has 32 slots. Required failures fail the job; observational failures are printed as non-blocking and preserve their former advisory status. ESLint itself stays single-threaded because 16 ESLint worker threads increased full-lint time to 174.54 seconds; outer gate concurrency uses the runner without multiplying Windows worker startup and TypeScript project loading.

-The final shape followed two cold-path observations. A first candidate used two 16-core Windows jobs, and GitHub took 93 seconds to provision the second same-label runner despite the configured autoscaling ceiling. A later [documentation-head validation](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29900502413) took 64 seconds on the combined Linux job because its production site gate lasted 29.05 seconds, while a separate Windows blocking job took 266 seconds after spending 138 seconds restoring a 153 MB pnpm cache. Moving that independent Linux gate to one coarse 16-core lane and combining all Windows work on one 32-core box removed both duplicate setup waves.
+The Windows shape followed two cold-path observations. A first candidate used two 16-core Windows jobs, and GitHub took 93 seconds to provision the second same-label runner despite the configured autoscaling ceiling. A later [documentation-head validation](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29900502413) took 266 seconds on a separate Windows blocking job after spending 138 seconds restoring a 153 MB pnpm cache. Combining all Windows work on one 32-core box removed the duplicate setup wave.
+
+Two later runs set the Linux boundaries. A [standard-runner validation](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29902209492) took 67 seconds for Node 26 even though repository work took five seconds, because GitHub spent 36 seconds in `Set up job`. Moving the environment contracts to distinct larger pools removed that lottery. The next [all-larger-runner validation](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29902541203) took 68 seconds on the 96-core primary job: repository work remained 26 seconds, but setup, cache, install, and finalization consumed 42 seconds. Moving typecheck, coverage, and the build-to-snapshot dependency chain to one coarse 64-core lane reduced the 96-core lane's repository critical path to 14.81 seconds without returning to per-gate shards.

 The workflow retains four manual diagnostics. `suite=larger-runner-benchmark` compares isolated critical lanes across every size, `suite=consolidated-runner-benchmark` compares whole aggregates, `suite=sharded-reference` preserves the former production shard topology, and `suite=serial-reference` remains the unsharded cross-platform completeness oracle. `suite=optimized-larger-runners` runs the exact production topology against a branch ref when a pull request cannot form a merge commit.

@@ -40,16 +44,19 @@ Those isolated results showed that setup dominated but did not identify the prod

 The Linux 32-core failure was the first CJS-lexer worker crash. The 96-core aggregate was the only successful all-size result at the one-minute boundary. Although Windows repository work gained little above 16 cores, the 32-core pool can start the complete outer inventory together and, more importantly, removes an entire paid setup from production.

-The exact [reduced-fanout validation run](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29901539360) passed every job at the tested branch head:
+The exact [all-pool validation run](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29903067274) passed every job at the tested branch head:

-| Production job | Active time | Repository work | Result |
+| Production job (pool) | Active time | Repository work | Result |
 |---|---:|---:|---:|
-| Linux core | 52 s | 39 gates in 25.84 s | passed |
-| Linux production site | 44 s | VitePress in 22.42 s | passed |
-| Slowest standard non-Windows job | 43 s | Node 26 compatibility | passed |
-| Windows complete | 96 s | 37 gates in 31.13 s | passed |
+| Node 22.19 compatibility (Linux 4) | 26 s | compatibility smokes | passed |
+| Python 3.10 (Linux 8) | 22 s | complete keyless SDK suite | passed |
+| Production site (Linux 16) | 39 s | VitePress in 22.57 s | passed |
+| Node 26 compatibility (Linux 32) | 24 s | compatibility smokes | passed |
+| Primary CPU (Linux 64) | 53 s | 6 gates in 23.70 s | passed |
+| Primary core (Linux 96) | 47 s | 36 gates in 14.81 s | passed |
+| Windows complete (Windows 32) | 109 s | 37 gates in 32.71 s | passed |

-All three paid jobs began together. The Windows job spent 21 seconds restoring its pnpm cache and 25 seconds installing dependencies, so its margin measures hosted setup variance as well as repository work. Every non-Windows job stays below one minute and the sole Windows job stays below three minutes.
+All seven paid jobs began in the same second. The slowest non-Windows job finished in 53 seconds. The Windows job spent 20 seconds restoring its pnpm cache and 15 seconds installing dependencies, so its 109-second active time measures hosted setup variance as well as repository work. Every non-Windows job stays below one minute and the sole Windows job stays below three minutes.

 ## Alternatives considered

@@ -59,17 +66,19 @@ All three paid jobs began together. The Windows job spent 21 seconds restoring i

 **Run the production site inside the Linux core aggregate.** This reached 50 seconds with warm hosted setup, then crossed the threshold at 64 seconds when the site gate took 29.05 seconds. One coarse independent site job protects the target without returning to gate-level sharding.

+**Keep every primary gate on the 96-core Linux runner.** Repository work completed in 26 seconds, but a 42-second cold path still pushed the job to 68 seconds. The 64-core CPU lane owns the three longest independent or dependency-critical paths; the remaining 36-gate core inventory completes its repository work in 14.81 seconds.
+
 **Keep required and observational Windows checks in separate jobs.** The split preserved status semantics at the workflow level but paid setup twice, and a cold cache pushed the required job to 266 seconds. `run-gates` now preserves those semantics inside one process: build and production site are required, while the remaining inventory is explicitly non-blocking.

 **Prebuild before starting the Linux aggregate.** This moved build onto the setup path and produced a 66-second candidate. Starting build eagerly inside `run-gates` preserves artifact dependencies while overlapping it with unrelated checks.

 **Use native ESLint worker concurrency on Windows.** Sixteen workers made lint more than five times slower than the final single-threaded result. Outer gate parallelism uses the 32-core runner without multiplying ESLint's Windows worker startup and TypeScript project loading.

-**Move compatibility, Python, and aggregation to larger runners.** These standard-runner jobs completed in 43 seconds or less. Paid capacity would not shorten the critical path.
+**Keep compatibility and Python on standard runners.** Warm runs completed in 43 seconds or less, but one Node 26 job later spent 36 seconds in GitHub setup and crossed the target despite only five seconds of repository work. Distinct larger pools stabilize those environment contracts; the three-second final aggregator remains on a standard runner because it begins only after the paid jobs release capacity.

 ## Consequences

-The reduced-fanout validation consumed one billed 96-core Linux minute, one billed 16-core Linux minute, and two billed 32-core Windows minutes. At the configured larger-runner rates, its larger-runner cost was $0.618. The all-size critical benchmark cost $2.936. GitHub rounds each larger-runner job up to a whole minute, so reducing paid job count from 49 to three matters as much as shortening repository work.
+The all-pool validation consumed one billed minute at each Linux size and two billed 32-core Windows minutes. At the configured larger-runner rates, its larger-runner cost was $0.896. The all-size critical benchmark cost $2.936. GitHub rounds each larger-runner job up to a whole minute, so reducing paid job count from 49 to seven matters as much as shortening repository work.

 The existing zero-dollar Actions budget did not block larger-runner jobs. The repo-only runner group, bounded workflow topology, manual benchmark triggers, and job timeouts are the observed cost controls; the budget is not treated as an execution guard.

@@ -14,13 +14,17 @@ Status: implemented

 组织在仅限本仓库使用的 `dsh-larger-ci` 运行器组中保留 12 个 x64 大型运行器池：Ubuntu 24.04 和 Windows 2025 各设 4、8、16、32、64、96 核规格。公网 IP 已禁用。每个池的自动扩缩容上限为 256；该上限既不会分配闲置机器，也不能免除限制工作流需求的必要性。

-生产 CI 使用 3 个粗粒度大型运行器作业，并让 Node 兼容性、Python 和最终聚合作业继续使用标准运行器：
+生产 CI 对 6 种 Linux 池规格各使用一次，另使用一个 32 核 Windows 池，只有最终聚合作业继续使用标准运行器。版本和语言作业属于环境契约，并非从同一门禁清单切出的分片；主 Node 工作采用 3 个粗粒度通道，而非门禁级分片矩阵：

- `node 24 / core` 使用 96 核 Linux 池。只需执行一次代码检出、设置、缓存恢复和安装，即可供生产网站构建以外、未分片的 39 项主门禁清单使用。`run-gates` 最多同时启动 32 项相互独立的门禁，ESLint 使用 32 个工作线程，快照最多使用 32 个子进程，覆盖率运行使用 16 个 fork。构建与类型检查同时启动；快照和产物消费方仍会等待生成的输出。覆盖率运行的 fork 数保持低于 32，因为将其设为 32 曾两次导致 Node 24 的 CJS 词法分析器终止一个 Vitest 工作进程，使覆盖率结果失效。
+- `node 24 / core` 使用 96 核 Linux 池。只需执行一次代码检出、设置、缓存恢复和安装，即可供 36 项未分片的静态、lint、文档、hygiene、构建和产物门禁使用。`run-gates` 最多同时启动 32 项相互独立的门禁，ESLint 使用 32 个工作线程。构建会尽早启动；其产物消费方仍会等待生成的输出。
+- `node 24 / cpu` 使用 64 核 Linux 池运行 6 项 CPU 或依赖关键门禁：类型检查、覆盖率、构建后回放快照，以及两项 Node 24 兼容性冒烟测试。覆盖率和快照各最多使用 16 个工作进程。该通道单独构建，使快照回放使用同一通道生成的输出。覆盖率运行的 fork 数保持低于 32，因为将其设为 32 曾两次导致 Node 24 的 CJS 词法分析器终止一个 Vitest 工作进程，使覆盖率结果失效。
 - `node 24 / production site` 使用 16 核 Linux 池运行耗时最长的独立主门禁。这是一项粗粒度拆分，而非分片矩阵：该作业执行一次设置和一次生产版 VitePress 构建。
+- Node 22.19 兼容性、Python 3.10 和 Node 26 兼容性分别使用 4 核、8 核和 32 核 Linux 池。不同标签既避免了标准运行器设置耗时的异常值，也避免了两个作业共享一个池时曾出现的第二次分配延迟。
 - `windows node 24 / complete` 使用 32 核 Windows 池。一次设置供必需的包构建、必需的生产网站构建以及完整的观测性可移植性清单共用。外层调度器设有 32 个槽位。任何必需项失败都会使作业失败；观测项失败会明确打印为非阻塞，并保留其原有的建议性状态。ESLint 本身仍采用单线程，因为启用 16 个 ESLint 工作线程会把完整 lint 的耗时增加至 174.54 秒；外层门禁并发能够利用该运行器，同时不会成倍增加 Windows 工作线程的启动和 TypeScript 项目加载开销。

-最终结构由两项冷路径观测结果确定。首个候选方案使用两个 16 核 Windows 作业；尽管已配置自动扩缩容上限，GitHub 仍花费 93 秒才预配好第二台同标签运行器。随后一次[文档分支头验证](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29900502413)中，合并后的 Linux 作业耗时 64 秒，因为其中的生产网站门禁耗时 29.05 秒；另一个独立的 Windows 阻塞性作业则耗时 266 秒，其中恢复 153 MB pnpm 缓存就用了 138 秒。把这项独立的 Linux 门禁移入一个粗粒度 16 核通道，并把所有 Windows 工作合并到一台 32 核机器上，消除了这两轮重复设置。
+Windows 结构由两项冷路径观测结果确定。首个候选方案使用两个 16 核 Windows 作业；尽管已配置自动扩缩容上限，GitHub 仍花费 93 秒才预配好第二台同标签运行器。随后一次[文档分支头验证](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29900502413)中，一个独立的 Windows 阻塞性作业耗时 266 秒，其中恢复 153 MB pnpm 缓存就用了 138 秒。把所有 Windows 工作合并到一台 32 核机器上，消除了这轮重复设置。
+
+随后两次运行确定了 Linux 的边界。一次[标准运行器验证](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29902209492)中，Node 26 作业耗时 67 秒，而仓库工作仅耗时 5 秒，因为 GitHub 在 `Set up job` 上花费了 36 秒。把各项环境契约迁移到不同的大型运行器池后，这种随机波动随之消失。接下来的[全大型运行器验证](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29902541203)中，96 核主作业耗时 68 秒：仓库工作仍为 26 秒，但设置、缓存、安装和收尾共占用 42 秒。把类型检查、覆盖率以及构建至快照的依赖链迁移到一个粗粒度 64 核通道后，96 核通道的仓库关键路径缩短至 14.81 秒，且无需恢复按门禁分片。

 工作流保留 4 项手动诊断。`suite=larger-runner-benchmark` 比较所有规格下相互独立的关键通道，`suite=consolidated-runner-benchmark` 比较完整聚合流程，`suite=sharded-reference` 保留原生产分片拓扑，`suite=serial-reference` 则继续作为未分片的跨平台完整性判定基准。当拉取请求无法生成合并提交时，`suite=optimized-larger-runners` 会直接针对分支引用运行与生产环境完全相同的拓扑。

@@ -40,16 +44,19 @@ Status: implemented

 Linux 32 核作业的失败是首次发生的 CJS 词法分析器工作进程崩溃。在所有规格的结果中，96 核聚合作业是唯一成功达到 1 分钟边界的结果。尽管 Windows 仓库工作在超过 16 核后收益很小，32 核池可以让完整的外层清单同时启动；更重要的是，它能从生产环境中消除整整一次付费设置。

-精确的[低扇出验证运行](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29901539360)在受测分支头通过了所有作业：
+精确的[全运行器池验证运行](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29903067274)在受测分支头通过了所有作业：

-| 生产作业 | 活动耗时 | 仓库工作 | 结果 |
+| 生产作业（运行器池） | 活动耗时 | 仓库工作 | 结果 |
 |---|---:|---:|---:|
-| Linux 核心 | 52 秒 | 39 项门禁耗时 25.84 秒 | 通过 |
-| Linux 生产网站 | 44 秒 | VitePress 耗时 22.42 秒 | 通过 |
-| 最慢的标准非 Windows 作业 | 43 秒 | Node 26 兼容性 | 通过 |
-| Windows 完整流程 | 96 秒 | 37 项门禁耗时 31.13 秒 | 通过 |
+| Node 22.19 兼容性（Linux 4） | 26 秒 | 兼容性冒烟测试 | 通过 |
+| Python 3.10（Linux 8） | 22 秒 | 完整无密钥 SDK 套件 | 通过 |
+| 生产网站（Linux 16） | 39 秒 | VitePress 耗时 22.57 秒 | 通过 |
+| Node 26 兼容性（Linux 32） | 24 秒 | 兼容性冒烟测试 | 通过 |
+| 主流程 CPU（Linux 64） | 53 秒 | 6 项门禁耗时 23.70 秒 | 通过 |
+| 主流程核心（Linux 96） | 47 秒 | 36 项门禁耗时 14.81 秒 | 通过 |
+| Windows 完整流程（Windows 32） | 109 秒 | 37 项门禁耗时 32.71 秒 | 通过 |

-3 个付费作业同时开始运行。Windows 作业花费 21 秒恢复 pnpm 缓存、25 秒安装依赖，因此其余量既反映托管环境的设置波动，也反映仓库工作耗时。每个非 Windows 作业均低于 1 分钟，唯一的 Windows 作业也低于 3 分钟。
+7 个付费作业均在同一秒开始运行。最慢的非 Windows 作业耗时 53 秒。Windows 作业花费 20 秒恢复 pnpm 缓存、15 秒安装依赖，因此其 109 秒的活动耗时既反映托管环境的设置波动，也反映仓库工作耗时。每个非 Windows 作业均低于 1 分钟，唯一的 Windows 作业也低于 3 分钟。

 ## 曾考虑的替代方案

@@ -59,17 +66,19 @@ Linux 32 核作业的失败是首次发生的 CJS 词法分析器工作进程崩

 **在 Linux 核心聚合流程内运行生产网站。** 托管环境设置处于热状态时，此方案用时 50 秒；当网站门禁耗时 29.05 秒时，总耗时增至 64 秒，超过了目标。一项独立的粗粒度网站作业能够保障目标，而无需重新采用门禁级分片。

+**将所有主门禁都保留在 96 核 Linux 运行器上。** 仓库工作在 26 秒内完成，但 42 秒的冷路径仍把作业总耗时推高至 68 秒。64 核 CPU 通道负责 3 条耗时最长的独立或依赖关键路径；其余 36 项核心门禁清单的仓库工作耗时为 14.81 秒。
+
 **将必需的 Windows 检查和观测性 Windows 检查保留在不同作业中。** 这种拆分在工作流层保留了状态语义，却要支付两次设置开销，而且冷缓存让必需作业的耗时达到 266 秒。`run-gates` 目前在一个进程内保留这些语义：构建和生产网站属于必需项，其余清单则明确为非阻塞。

 **在启动 Linux 聚合流程前预先构建。** 此方案让构建成为设置路径的一部分，并产生了一个耗时 66 秒的候选结果。在 `run-gates` 内尽早启动构建，既能保留产物依赖关系，又能让构建与无关检查重叠执行。

 **在 Windows 上使用 ESLint 原生工作线程并发。** 16 个工作线程让 lint 比最终的单线程结果慢 5 倍以上。外层门禁并发能够利用 32 核运行器，同时不会成倍增加 ESLint 在 Windows 上启动工作线程和加载 TypeScript 项目的开销。

-**将兼容性、Python 和聚合作业迁移到大型运行器。** 这些标准运行器作业都在 43 秒以内完成。付费容量不会缩短关键路径。
+**让兼容性和 Python 继续使用标准运行器。** 热运行均在 43 秒以内完成，但后来一个 Node 26 作业在 GitHub 设置上花费 36 秒，即使仓库工作仅耗时 5 秒，仍超过了目标。不同的大型运行器池能够稳定这些环境契约；最终聚合作业耗时 3 秒，仍使用标准运行器，因为它只会在付费作业释放容量后启动。

 ## 后果

-低扇出验证产生的计费时长为：96 核 Linux 1 分钟、16 核 Linux 1 分钟和 32 核 Windows 2 分钟。按已配置的大型运行器费率计算，其大型运行器成本为 $0.618。全规格关键通道基准测试的成本为 $2.936。GitHub 会把每个大型运行器作业向上取整到整分钟计费，因此把付费作业数从 49 个减少到 3 个，与缩短仓库工作耗时同样重要。
+全运行器池验证中，每种 Linux 规格各计费 1 分钟，32 核 Windows 计费 2 分钟。按已配置的大型运行器费率计算，其大型运行器成本为 $0.896。全规格关键通道基准测试的成本为 $2.936。GitHub 会把每个大型运行器作业向上取整到整分钟计费，因此把付费作业数从 49 个减少到 7 个，与缩短仓库工作耗时同样重要。

 现有的零美元 Actions 预算并未阻止大型运行器作业。仅限本仓库的运行器组、有界的工作流拓扑、手动基准测试触发和作业超时限制才是经实测有效的成本控制机制；该预算不被视为执行防护措施。