doc: record the coverage-exempt heavy suites decision

Agent Note (both languages, pairing recorded): what is exempted, the
per-entry reconciliation showing no threshold input changes, the
membership contract, and why the per-file 100% thresholds police the
roster automatically.
This commit is contained in:
imccyu
2026-07-31 02:55:37 +08:00
parent e9ed7193d8
commit c368a89dc1
3 changed files with 128 additions and 0 deletions
@@ -0,0 +1,6 @@
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-31-coverage-exempt-heavy-suites.md
2026-07-31-coverage-exempt-heavy-suites.md: 7235a5193554947ecf71f62d522d09f4e21cb1da
2026-07-31-coverage-exempt-heavy-suites.zh.md: b739e4494ae8d240b0e35109920a49876ebd222d
@@ -0,0 +1,61 @@
# Agent Note: Coverage-exempt heavy suites
Status: implemented
English | [中文](2026-07-31-coverage-exempt-heavy-suites.zh.md)
## Problem
The CI coverage lane (`check:ci:coverage`) had its wall clock pinned by a handful of heavy test files: in a local 6-worker full-suite profile, 555 test files aggregated 1595 seconds, with `packages/typert/generator/tests/type-model.spec.ts` alone at 885 seconds and the top 10 files holding 84% of the aggregate. These suites share one shape — every case performs whole-workspace compiler analysis or drives real subprocess fixtures — and v8 instrumentation multiplies exactly that kind of runtime.
The decisive waste: the instrumentation tax these suites paid contributed **nothing** to the per-file 100% thresholds — the measured code they execute in-process is either outside the threshold scope already or independently fully covered by other suites. Running them instrumented traded lane time for zero information.
## Decision
The `ci-coverage` aggregate splits into two parallel gates; every test still runs, and only the heavy suites stop paying the instrumentation tax:
- **Instrumented gate** (`test:coverage`): sets `DSH_COVERAGE_EXEMPT_HEAVY=1`, which makes `vitest.config.ts` drop the exempt suites from both projects' excludes; every remaining file runs instrumented and carries the entire threshold proof. The variable is injected through the gate's own env (the existing `Gate.env` mechanism), not the workflow-global environment, so the uninstrumented gate beside it and any local `vitest run` never see it and behave unchanged.
- **Uninstrumented gate** (`test:coverage-exempt-heavy`): runs exactly the exempt suites through paired positional filters, keeping the correctness signal whole.
`scripts/coverage-exempt.ts` is the single roster point, holding the membership contract and the filter/exclude pairs so the two sides cannot drift.
### The roster, reconciled entry by entry
A suite contributes to coverage exactly when it executes measured files in-process (`coverage.include` spans the package src trees). The current roster, audited:
| Exempt suite | Measured code executed in-process | Who carries the coverage |
| --- | --- | --- |
| All 6 typert generator specs | The generator's own src | Generator src is threshold-excluded as a package (`vitest.config.ts`) — outside the threshold scope to begin with |
| tools-catalog.spec additionally imports | `typert-registry` and `tool-cordis` src | Each package's own tests cover them fully (verified with focused coverage runs, zero threshold errors) |
| `scripts/install-lefthook.spec.ts`, `scripts/oxlint-contract.spec.ts`, `scripts/change-scope.spec.ts` | None — they test `scripts/` sources (never in `coverage.include`) and work by spawning child processes | Nothing to carry |
### Membership contract
A new exemption must satisfy both: every measured file the suite executes in-process is already fully covered by other suites (or threshold-excluded), and the filter and exclude select exactly the same file set. The contract text lives beside the roster in the same file.
### The gate polices the roster automatically
The per-file 100% thresholds are themselves the roster's guard; a wrong roster cannot pass silently:
- If a future exempt suite in fact solely covers some measured file, the instrumented gate goes red on the spot (that file drops below 100%).
- The converse holds too: new code covered only by an exempt suite turns the gate red immediately.
Coverage-result invariance therefore does not rest on humans maintaining the roster, in line with the misconfiguration-fails-loud convention. The only thing given up is that the exempt suites' own execution no longer produces coverage data — the table above shows that data was entirely redundant, so the final report is file-for-file identical in threshold terms.
## Alternatives considered
- **CLI `--exclude` to drop the exempt suites from the instrumented gate.** Proven ineffective: vitest 4's `cliExclude` does not participate in per-project include resolution, so under a multi-project config the exempt suites stayed selected; the env + config route replaced it.
- **Lowering worker counts or raising gate concurrency.** Measured ineffective during the incident: the lane's wall clock was pinned by the longest tail files (aggregate/wall ≈ 4× effective parallelism), and the concurrency knobs moved nothing in either direction.
- **Cross-runner sharding (`--shard` + blob merge).** Would compress the wall clock further but adds matrix, artifact-pipeline, and merge-job complexity; with the split landed the lane sits near 2 minutes, which does not justify the cost. Revisit if the suite grows substantially.
- **Deleting or skipping the heavy suites.** Rejected: they are the sole correctness evidence for the typert generator and the scripts tooling; running them uninstrumented in parallel preserves the full signal.
## Verification
Measured on CI (16-core runner): the gate segment went from 424 seconds to the two gates in parallel — `test:coverage` 95.9 s + `test:coverage-exempt-heavy` 71.1 s — with the lane converging on the slower at about 96 seconds; the instrumented gate reported zero threshold errors both before and after the split. `vitest list` verifies the env toggle adds and removes exactly the exempt set; `run-gates.spec.ts` covers the aggregate graph construction.
## Consequences
- The coverage lane's gate segment drops from about 7 minutes to about 96 seconds with no change in threshold outcome or executed test set.
- `DSH_GATE_CONCURRENCY` has two schedulable gates in this lane again, so the aggregate scheduler is no longer a pass-through.
- Adding a heavy suite to the roster requires the membership audit above; a wrong entry fails the instrumented gate loudly rather than eroding coverage silently.
- The exempt suites no longer appear in the coverage report's file list of contributors; their correctness signal lives solely in the uninstrumented gate's pass/fail.
@@ -0,0 +1,61 @@
# Agent Note: 覆盖率豁免重型套件
Status: implemented
[English](2026-07-31-coverage-exempt-heavy-suites.md) | 中文
## Problem
CI 覆盖率 lane`check:ci:coverage`)的墙钟被少数几个重型测试文件钉死:本地 6-worker 全量剖析中,555 个测试文件聚合 1595 秒,其中 `packages/typert/generator/tests/type-model.spec.ts` 一个文件占 885 秒,前 10 个文件占聚合时长的 84%。这类套件的共同点是每个用例都做全工作区编译器分析或真实子进程 fixture,v8 插桩把这类代码的运行时间放大数倍。
关键的浪费在于:这些套件缴纳的插桩税对 per-file 100% 阈值**没有任何贡献**——它们进程内执行的被度量代码,要么本来就不在阈值口径内,要么已由其他套件独立满覆盖。继续在插桩下运行它们,纯粹是用 lane 时长换零信息。
## Decision
`ci-coverage` 聚合拆成两个并行 gate,全部测试仍然执行,只有重型套件不再交插桩税:
- **插桩 gate**`test:coverage`):设 `DSH_COVERAGE_EXEMPT_HEAVY=1``vitest.config.ts` 据此从两个 project 的 exclude 中剔除豁免套件,其余全部文件照旧插桩并承担全部阈值证明。经 gate 自带 env 注入(既有 `Gate.env` 机制),不进 workflow 全局环境,因此并排的无插桩 gate 和本地直跑 `vitest run` 都看不到该变量、行为不变。
- **无插桩 gate**`test:coverage-exempt-heavy`):用配对的 positional filter 恰好运行豁免套件,保证正确性信号不缩水。
`scripts/coverage-exempt.ts` 是唯一名单点,集中持有成员资格契约与 filter/exclude 配对,防止两侧漂移。
### 豁免名单与逐项对账
一个套件对覆盖率有贡献,当且仅当它在进程内执行了被度量的文件(`coverage.include` = 包 src 树)。现行名单逐项核对:
| 豁免套件 | 进程内执行的被度量代码 | 覆盖由谁接住 |
| --- | --- | --- |
| typert generator 全部 6 个 spec | generator 自身 src | generator src 已整包 threshold-excluded`vitest.config.ts`),本不在阈值口径内 |
| 其中 tools-catalog.spec 额外 import | `typert-registry``tool-cordis` 的 src | 两包各自的测试独立满覆盖(focused coverage 实测无阈值错误) |
| `scripts/install-lefthook.spec.ts``scripts/oxlint-contract.spec.ts``scripts/change-scope.spec.ts` | 无——被测对象是 `scripts/` 源码(从不在 coverage.include),执行方式是 spawn 子进程 | 无需接 |
### 成员资格契约
新增豁免必须同时满足:套件进程内执行的每个被度量文件都已由其他套件满覆盖(或在阈值排除名单内);filter 与 exclude 选中完全相同的文件集。契约文本随名单同文件维护。
### 门禁自动守卫名单正确性
per-file 100% 阈值本身就是豁免名单的守卫,名单错误无法静默通过:
- 若未来某个豁免套件实际独家覆盖着某个被度量文件,插桩 gate 当场红(该文件跌破 100%);
- 反向同理:出现"只有豁免套件才覆盖"的新代码,同样立刻红。
因此覆盖率结果的不变性不依赖人工维护名单,符合"misconfiguration fails loud"约定。唯一失去的是豁免套件自身的执行不再产出覆盖数据——由上表可知这些数据全部冗余,最终报告在阈值意义上逐文件相同。
## Alternatives considered
- **CLI `--exclude` 从插桩 gate 剔除豁免套件。** 实证无效:vitest 4 的 `cliExclude` 不参与 per-project include 解析,多 project 配置下豁免套件仍被选中,故改走 env + config。
- **降低 worker 数或提高 gate 并发。** 事故期间实测无效:lane 墙钟被尾部最长文件钉死(聚合/墙钟 ≈ 4× 有效并行),并发旋钮两个方向都动不了尾巴。
- **跨 runner 分片(`--shard` + blob 合并)。** 能进一步压墙钟但引入 matrix、artifact 管道与合并 job 的复杂度;拆分落地后 lane 已到约 2 分钟,不值得付。若未来套件规模再涨可重新评估。
- **直接删除或跳过重型套件。** 拒绝:它们是 typert generator 与 scripts 工具的唯一正确性证据,无插桩并排执行保住全部信号。
## Verification
CI 实测(16 核 runner):拆分前 gate 段 424 秒,拆分后两 gate 并行 `test:coverage` 95.9 秒 + `test:coverage-exempt-heavy` 71.1 秒,lane 收敛于较慢者约 96 秒;拆分前后插桩 gate 阈值错误均为零。`vitest list` 验证 env 开关两态恰好增删豁免集;`run-gates.spec.ts` 覆盖聚合图构造。
## Consequences
- 覆盖率 lane 的 gate 段从约 7 分钟降到约 96 秒,阈值结果与执行测试集均无变化。
- `DSH_GATE_CONCURRENCY` 在本 lane 重新拥有两个可调度对象,聚合调度器不再是直通。
- 向名单新增重型套件必须完成上述成员资格对账;错误条目会让插桩 gate 大声失败,而不是静默侵蚀覆盖率。
- 豁免套件不再出现在覆盖率报告的贡献文件列表中;其正确性信号完全由无插桩 gate 的红绿承载。