ci: gate static lane's cache restore under failover; fix runbook recovery steps
Review round on the pivoted design: - node-24 (static) kept an unconditional hosted pnpm cache restore while the coverage and consumers lanes skip it under failover. On the self-hosted VM that restore downloads ~180 MB into /home/runner, a path pnpm never reads there, adding latency and contention during an outage. Gate it with the same `vars.DSH_CI_FAILOVER != 'selfhosted'` condition so all three lanes match. - Runbook switch step 2 said "Re-run failed jobs", but the documented indefinite-queue outage leaves jobs queued (not failed), which cannot be re-run in place and do not retarget on variable change. Correct both language sides to cancel the run and re-run all jobs, or push a new commit. - The standby-lane comment still described the switch as a one-line runs-on change; it is now setting the admin-only DSH_CI_FAILOVER variable.
This commit is contained in:
4 files changed
+9
-6
No files matched your search
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-26-ci-failover-runbook.md: 9100cf226467d06835478b13c41904bc50270b78
|
||||
2026-07-26-ci-failover-runbook.zh.md: 4ec80ae36411335a378f7979b9bca704c17732d0
|
||||
2026-07-26-ci-failover-runbook.md: db8e0676ecc6eeaea16438e7868ccf9ac43887cc
|
||||
2026-07-26-ci-failover-runbook.zh.md: b3b4149f460784e88ce03458fc556f402c38fa2f
|
||||
@@ -19,7 +19,7 @@ Each of the three required Linux jobs resolves its runner pool through the `DSH_
|
||||
### Switch (repo admin, ~1 minute, no merge)
|
||||
|
||||
1. Repository **Settings → Secrets and variables → Actions → Variables → New repository variable**: name `DSH_CI_FAILOVER`, value `selfhosted`.
|
||||
2. Re-run the failed/queued required jobs (Re-run failed jobs on affected PRs, or let new pushes pick it up).
|
||||
2. Retrigger the required jobs so they re-resolve their pool. Jobs already **queued** for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued.
|
||||
3. That is the entire switch. Under failover the workflow also, automatically: halves `DSH_COVERAGE_MAX_WORKERS` to 12 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 16 (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs).
|
||||
|
||||
### Capacity during failover
|
||||
|
||||
@@ -19,7 +19,7 @@ Status: implemented
|
||||
### 切换步骤(仓库管理员,约 1 分钟,无需合并)
|
||||
|
||||
1. 仓库 **Settings → Secrets and variables → Actions → Variables → New repository variable**:名称 `DSH_CI_FAILOVER`,值 `selfhosted`。
|
||||
2. 对受影响 PR 的失败/排队作业点 Re-run failed jobs(或等新推送自然触发)。
|
||||
2. 重新触发必需作业,使其重新解析运行器池。已经为托管标签**排队**的作业不会重定向,也无法原地 re-run,因此对于本手册所述的无限排队故障,应取消卡住的运行并 re-run all jobs,或推送一个新提交;“Re-run failed jobs”只有在作业真正失败(而非仍在排队)时才有用。
|
||||
3. 切换到此完成。故障切换状态下工作流还会自动:把 `DSH_COVERAGE_MAX_WORKERS` 降为 12、`DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 16(共享虚拟机的争抢上限),并跳过托管路径的 pnpm 缓存恢复(虚拟机的持久 store 直接提供热安装)。
|
||||
|
||||
### 切换期间的容量
|
||||
|
||||
@@ -55,8 +55,10 @@ jobs:
|
||||
persist-credentials: false
|
||||
|
||||
# Pull requests consume the default-branch cache but do not put cache
|
||||
# compression and upload on the paid latency-critical path.
|
||||
# compression and upload on the paid latency-critical path. Skipped
|
||||
# under failover — see the coverage lane's identical rationale.
|
||||
- uses: actions/cache/restore@v4
|
||||
if: vars.DSH_CI_FAILOVER != 'selfhosted'
|
||||
with:
|
||||
path: /home/runner/.local/share/pnpm/store/v11
|
||||
key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
|
||||
@@ -398,7 +400,8 @@ jobs:
|
||||
# Hot-standby drill for the in-house self-hosted pool: every master move
|
||||
# re-runs the complete unsharded aggregate on the persistent 64-core VM,
|
||||
# continuously proving that environment can take over a required lane if
|
||||
# the hosted pools degrade (the switch is then a one-line runs-on change).
|
||||
# the hosted pools degrade (the switch is then setting the admin-only
|
||||
# DSH_CI_FAILOVER variable — see the failover runbook, no merge required).
|
||||
# Push-triggered, so it always executes the base branch's own workflow
|
||||
# definition — no PR-editable path selects these runners. Non-blocking for
|
||||
# pull requests; no cache steps because the VM's persistent pnpm store and
|
||||
|
||||
Reference in New Issue
Block a user