ci: standby fetches full history; runbook — writer wording throughout, master-ref pinning incompatibility
- serial-linux-selfhosted checks out fetch-depth 0: depth 2 misses github.event.before on multi-commit or force pushes, failing the archive verifier on a valid tree. Full fetch is cheap against the VM's local mirror. - Runbook (both languages): every remaining admin phrasing (problem statement, switch heading, alternatives, consequences) now says writer; and the 'composes with this mechanism' claim about a master-ref-pinned runner group is replaced with the truth observed live on 2026-07-27 — master-ref pinning blocks PR failover, and the shipped posture is repository-scoped all-workflow group access. Static gate green locally: 32 passed, 0 failed.
This commit is contained in:
Chinesezjc
Ubuntu
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md
|
||||
2026-07-26-ci-failover-runbook.md: 05014454fa3e38045b89a857c346db0f897ab5a6
|
||||
2026-07-26-ci-failover-runbook.zh.md: e106a0de40799ca1c218217ea66c24068697dc53
|
||||
2026-07-26-ci-failover-runbook.md: ca4349661d03ff4e28d7c3c2b6e910106ff4aa30
|
||||
2026-07-26-ci-failover-runbook.zh.md: 1d59bd537879f531c9075e833c9e1dbfbd4bb0a2
|
||||
@@ -6,7 +6,7 @@ English | [中文](2026-07-26-ci-failover-runbook.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) and the required verdict job that aggregates them (`all checks passed`) run on the hosted enterprise 32-core pools. When those pools degrade — jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. An outage therefore needs a switch a repository admin can throw without merging anything.
|
||||
The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) and the required verdict job that aggregates them (`all checks passed`) run on the hosted enterprise 32-core pools. When those pools degrade — jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. An outage therefore needs a switch any responder with repository write access can throw without merging anything.
|
||||
|
||||
## Decision
|
||||
|
||||
@@ -16,7 +16,7 @@ Each of the three required Linux worker jobs — and the `all checks passed` ver
|
||||
|
||||
`vm-backup`: one 64-core VM, six always-on systemd-managed runner instances. Check the latest `serial / linux (self-hosted standby)` run before switching: a green standby is verified-yesterday capacity.
|
||||
|
||||
### Switch (repo admin, ~1 minute, no merge)
|
||||
### Switch (any repository writer, ~1 minute, no merge)
|
||||
|
||||
1. Repository **Settings → Secrets and variables → Actions → Variables → New repository variable**: name `DSH_CI_FAILOVER`, value `selfhosted`.
|
||||
2. Retrigger the required jobs so they re-resolve their pool. Jobs already **queued** for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued.
|
||||
@@ -37,14 +37,14 @@ Delete the `DSH_CI_FAILOVER` variable (or set it to anything other than `selfhos
|
||||
|
||||
### Trust boundary
|
||||
|
||||
The variable is writer-manageable repository state; a pull request event itself can neither set it nor read a different value into effect, and the selector expressions live in workflow definitions. Note that under failover, `pull_request` runs execute the PR merge ref's own workflow definition — the boundary against untrusted code is repository membership (private, forking disabled, Dependabot excluded by the selectors), not the variable. Runner-side enforcement — an org-level runner group restricting these runners to the master-ref workflow — is tracked separately and composes with this mechanism.
|
||||
The variable is writer-manageable repository state; a pull request event itself can neither set it nor read a different value into effect, and the selector expressions live in workflow definitions. Note that under failover, `pull_request` runs execute the PR merge ref's own workflow definition — the boundary against untrusted code is repository membership (private, forking disabled, Dependabot excluded by the selectors), not the variable. Note on runner-group policy: pinning the runner group to the master-ref workflow is **incompatible** with this failover — the four failover jobs are `pull_request` runs evaluated from PR merge refs, and a master-pinned group leaves them queued (observed live on 2026-07-27; the group was widened to all workflows of this repository to unblock the switch). A stricter runner-side policy therefore costs PR failover; the shipped posture accepts repository-scoped, all-workflow group access.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Merge a workflow change to switch pools.** Rejected because the outage that motivates the switch is exactly the state in which no PR can merge: the required checks are the ones failing. A repository variable is admin-controlled state that takes effect on re-run without a merge.
|
||||
**Merge a workflow change to switch pools.** Rejected because the outage that motivates the switch is exactly the state in which no PR can merge: the required checks are the ones failing. A repository variable is writer-manageable state that takes effect on re-run without a merge.
|
||||
|
||||
**Keep the self-hosted pool always in the required path.** Rejected because it trades hosted-pool availability for the in-house VM's, moving a single point of failure rather than adding a fallback. The variable keeps the hosted pools primary and the self-hosted pool a proven, one-action standby.
|
||||
|
||||
## Consequences
|
||||
|
||||
Recovering from a hosted-pool outage is a single admin variable plus a re-run, with no merge on the critical path. The cost is a second runner topology to keep working: the standby lane exercises it on every master push so the failover target never goes stale, and the concurrency and cache-restore branches in `ci.yml` carry a `selfhosted` leg that must stay in step with the hosted leg.
|
||||
Recovering from a hosted-pool outage is a single variable (any writer) plus a re-run, with no merge on the critical path. The cost is a second runner topology to keep working: the standby lane exercises it on every master push so the failover target never goes stale, and the concurrency and cache-restore branches in `ci.yml` carry a `selfhosted` leg that must stay in step with the hosted leg.
|
||||
@@ -6,7 +6,7 @@ Status: implemented
|
||||
|
||||
## 问题
|
||||
|
||||
[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 工作作业（`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`）以及聚合它们的必需判定作业（`all checks passed`）运行在托管的企业级 32 核池上。当这些托管池发生故障——作业无限排队、企业标签消失或 GitHub 侧容量故障——所有开启的拉取请求都无法合并，而"合并一个修复"这一常规恢复手段本身正被那些无法运行的必需检查死锁。因此故障需要一个仓库管理员无需合并任何代码即可触发的开关。
|
||||
[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 工作作业（`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`）以及聚合它们的必需判定作业（`all checks passed`）运行在托管的企业级 32 核池上。当这些托管池发生故障——作业无限排队、企业标签消失或 GitHub 侧容量故障——所有开启的拉取请求都无法合并，而"合并一个修复"这一常规恢复手段本身正被那些无法运行的必需检查死锁。因此故障需要一个任何具备仓库写权限的响应者都能在不合并任何代码的情况下触发的开关。
|
||||
|
||||
## 决策
|
||||
|
||||
@@ -16,7 +16,7 @@ Status: implemented
|
||||
|
||||
`vm-backup`：一台 64 核虚拟机，6 个常驻 systemd 管理的运行器实例。切换前先看 `serial / linux (self-hosted standby)` 最近一次运行：绿色 = 这套环境昨天刚被全量验证过。
|
||||
|
||||
### 切换步骤（仓库管理员，约 1 分钟，无需合并）
|
||||
### 切换步骤（任何具备写权限的协作者，约 1 分钟，无需合并）
|
||||
|
||||
1. 仓库 **Settings → Secrets and variables → Actions → Variables → New repository variable**：名称 `DSH_CI_FAILOVER`，值 `selfhosted`。
|
||||
2. 重新触发必需作业，使其重新解析运行器池。已经为托管标签**排队**的作业不会重定向，也无法原地 re-run，因此对于本手册所述的无限排队故障，应取消卡住的运行并 re-run all jobs，或推送一个新提交；“Re-run failed jobs”只有在作业真正失败（而非仍在排队）时才有用。
|
||||
@@ -37,14 +37,14 @@ Status: implemented
|
||||
|
||||
### 信任边界
|
||||
|
||||
该变量是写者可管理的仓库状态；`pull_request` 事件本身既不能设置它，也不能让不同的值生效，选择器表达式存在于工作流定义中。需要注意：故障切换期间，`pull_request` 运行执行的是 PR merge 引用自带的工作流定义——抵御不可信代码的边界是仓库成员资格（私有、禁 fork、选择器排除 Dependabot），而非该变量。（运行器侧的组织级 runner group 约束另行跟踪，与本机制互补。）
|
||||
该变量是写者可管理的仓库状态；`pull_request` 事件本身既不能设置它，也不能让不同的值生效，选择器表达式存在于工作流定义中。需要注意：故障切换期间，`pull_request` 运行执行的是 PR merge 引用自带的工作流定义——抵御不可信代码的边界是仓库成员资格（私有、禁 fork、选择器排除 Dependabot），而非该变量。关于 runner group 策略的说明：把 runner group 绑定到 master 引用的工作流与本故障切换机制**不兼容**——四个故障切换作业是从 PR merge 引用求值的 `pull_request` 运行，master 绑定的组会让它们持续排队（2026-07-27 实际故障中亲历；当时将组放宽为本仓库全部工作流才疏通了切换）。更严格的运行器侧策略以牺牲 PR 故障切换为代价；当前采用的形态是仓库范围、全工作流的组访问。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
**通过合并一次工作流改动来切换池。** 否决，因为触发切换的故障状态恰恰是任何 PR 都无法合并的状态：必需检查正是失败的那些。仓库变量是管理员控制的状态，重跑即生效，无需合并。
|
||||
**通过合并一次工作流改动来切换池。** 否决，因为触发切换的故障状态恰恰是任何 PR 都无法合并的状态：必需检查正是失败的那些。仓库变量是写者可管理的状态，重跑即生效，无需合并。
|
||||
|
||||
**让自托管池长期处于必需路径中。** 否决，因为这是拿托管池的可用性去换自有虚拟机的可用性，只是搬移了单点故障而非增加回退。该变量让托管池保持主路径，自托管池作为一个经过验证、一步即可启用的热备。
|
||||
|
||||
## 后果
|
||||
|
||||
从托管池故障中恢复只需一个管理员变量加一次重跑，关键路径上没有合并。代价是要维护第二套运行器拓扑：热备通道在每次 master 推送时都运行它，使故障切换目标永不失效；而 `ci.yml` 中的并发与缓存恢复分支带有一条 `selfhosted` 支路，必须与托管支路保持同步。
|
||||
从托管池故障中恢复只需一个变量（任何写者可设）加一次重跑，关键路径上没有合并。代价是要维护第二套运行器拓扑：热备通道在每次 master 推送时都运行它，使故障切换目标永不失效；而 `ci.yml` 中的并发与缓存恢复分支带有一条 `selfhosted` 支路，必须与托管支路保持同步。
|
||||
@@ -427,12 +427,13 @@ jobs:
|
||||
name: serial / linux (self-hosted standby)
|
||||
runs-on: [self-hosted, linux, x64, vm-backup]
|
||||
steps:
|
||||
# fetch-depth 2 + DSH_ARCHIVE_BASE_REF below: same frozen-archive
|
||||
# comparison as serial-linux — without the prior commit the archive
|
||||
# verifier defaults to HEAD and compares the new manifest with itself.
|
||||
# Full history + DSH_ARCHIVE_BASE_REF below: same frozen-archive
|
||||
# comparison as serial-linux. Depth 2 would miss github.event.before
|
||||
# on multi-commit or force pushes; full fetch is cheap here because
|
||||
# checkout resolves against the VM's local mirror.
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 2
|
||||
fetch-depth: 0
|
||||
|
||||
- uses: actions/setup-node@v6
|
||||
with:
|
||||
|
||||
Reference in New Issue
Block a user