ci: gate static lane's cache restore under failover; fix runbook recovery steps

Review round on the pivoted design:

- node-24 (static) kept an unconditional hosted pnpm cache restore while the
  coverage and consumers lanes skip it under failover. On the self-hosted VM
  that restore downloads ~180 MB into /home/runner, a path pnpm never reads
  there, adding latency and contention during an outage. Gate it with the same
  `vars.DSH_CI_FAILOVER != 'selfhosted'` condition so all three lanes match.

- Runbook switch step 2 said "Re-run failed jobs", but the documented
  indefinite-queue outage leaves jobs queued (not failed), which cannot be
  re-run in place and do not retarget on variable change. Correct both language
  sides to cancel the run and re-run all jobs, or push a new commit.

- The standby-lane comment still described the switch as a one-line runs-on
  change; it is now setting the admin-only DSH_CI_FAILOVER variable.
This commit is contained in:
Chinesezjc
2026-07-26 05:44:35 +08:00
parent 68e280ce4f
commit 498df1d8de
4 files changed
+9 -6

No files matched your search

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
2026-07-26-ci-failover-runbook.md: 9100cf226467d06835478b13c41904bc50270b78
2026-07-26-ci-failover-runbook.zh.md: 4ec80ae36411335a378f7979b9bca704c17732d0
2026-07-26-ci-failover-runbook.md: db8e0676ecc6eeaea16438e7868ccf9ac43887cc
2026-07-26-ci-failover-runbook.zh.md: b3b4149f460784e88ce03458fc556f402c38fa2f
@@ -19,7 +19,7 @@ Each of the three required Linux jobs resolves its runner pool through the `DSH_
### Switch (repo admin, ~1 minute, no merge)
1. Repository **Settings → Secrets and variables → Actions → Variables → New repository variable**: name `DSH_CI_FAILOVER`, value `selfhosted`.
2. Re-run the failed/queued required jobs (Re-run failed jobs on affected PRs, or let new pushes pick it up).
2. Retrigger the required jobs so they re-resolve their pool. Jobs already **queued** for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued.
3. That is the entire switch. Under failover the workflow also, automatically: halves `DSH_COVERAGE_MAX_WORKERS` to 12 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 16 (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs).
### Capacity during failover
@@ -19,7 +19,7 @@ Status: implemented
### 切换步骤(仓库管理员,约 1 分钟,无需合并)
1. 仓库 **Settings → Secrets and variables → Actions → Variables → New repository variable**:名称 `DSH_CI_FAILOVER`,值 `selfhosted`
2. 对受影响 PR 的失败/排队作业点 Re-run failed jobs(或等新推送自然触发)
2. 重新触发必需作业,使其重新解析运行器池。已经为托管标签**排队**的作业不会重定向,也无法原地 re-run,因此对于本手册所述的无限排队故障,应取消卡住的运行并 re-run all jobs,或推送一个新提交;“Re-run failed jobs”只有在作业真正失败(而非仍在排队)时才有用
3. 切换到此完成。故障切换状态下工作流还会自动:把 `DSH_COVERAGE_MAX_WORKERS` 降为 12、`DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 16(共享虚拟机的争抢上限),并跳过托管路径的 pnpm 缓存恢复(虚拟机的持久 store 直接提供热安装)。
### 切换期间的容量
+5 -2
View File
@@ -55,8 +55,10 @@ jobs:
persist-credentials: false
# Pull requests consume the default-branch cache but do not put cache
# compression and upload on the paid latency-critical path.
# compression and upload on the paid latency-critical path. Skipped
# under failover — see the coverage lane's identical rationale.
- uses: actions/cache/restore@v4
if: vars.DSH_CI_FAILOVER != 'selfhosted'
with:
path: /home/runner/.local/share/pnpm/store/v11
key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
@@ -398,7 +400,8 @@ jobs:
# Hot-standby drill for the in-house self-hosted pool: every master move
# re-runs the complete unsharded aggregate on the persistent 64-core VM,
# continuously proving that environment can take over a required lane if
# the hosted pools degrade (the switch is then a one-line runs-on change).
# the hosted pools degrade (the switch is then setting the admin-only
# DSH_CI_FAILOVER variable — see the failover runbook, no merge required).
# Push-triggered, so it always executes the base branch's own workflow
# definition — no PR-editable path selects these runners. Non-blocking for
# pull requests; no cache steps because the VM's persistent pnpm store and