From 498df1d8de66d3f17ed52ec93d7ffa863604cde8 Mon Sep 17 00:00:00 2001 From: Chinesezjc Date: Sun, 26 Jul 2026 05:44:35 +0800 Subject: [PATCH] ci: gate static lane's cache restore under failover; fix runbook recovery steps Review round on the pivoted design: - node-24 (static) kept an unconditional hosted pnpm cache restore while the coverage and consumers lanes skip it under failover. On the self-hosted VM that restore downloads ~180 MB into /home/runner, a path pnpm never reads there, adding latency and contention during an outage. Gate it with the same `vars.DSH_CI_FAILOVER != 'selfhosted'` condition so all three lanes match. - Runbook switch step 2 said "Re-run failed jobs", but the documented indefinite-queue outage leaves jobs queued (not failed), which cannot be re-run in place and do not retarget on variable change. Correct both language sides to cancel the run and re-run all jobs, or push a new commit. - The standby-lane comment still described the switch as a one-line runs-on change; it is now setting the admin-only DSH_CI_FAILOVER variable. --- .../process/2026-07-26-ci-failover-runbook.i18n.yaml | 4 ++-- .../implemented/process/2026-07-26-ci-failover-runbook.md | 2 +- .../process/2026-07-26-ci-failover-runbook.zh.md | 2 +- .github/workflows/ci.yml | 7 +++++-- 4 files changed, 9 insertions(+), 6 deletions(-) diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml index 7a65ff5479..a2725da1b2 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write -2026-07-26-ci-failover-runbook.md: 9100cf226467d06835478b13c41904bc50270b78 -2026-07-26-ci-failover-runbook.zh.md: 4ec80ae36411335a378f7979b9bca704c17732d0 +2026-07-26-ci-failover-runbook.md: db8e0676ecc6eeaea16438e7868ccf9ac43887cc +2026-07-26-ci-failover-runbook.zh.md: b3b4149f460784e88ce03458fc556f402c38fa2f diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md index 9100cf2264..db8e0676ec 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md @@ -19,7 +19,7 @@ Each of the three required Linux jobs resolves its runner pool through the `DSH_ ### Switch (repo admin, ~1 minute, no merge) 1. Repository **Settings → Secrets and variables → Actions → Variables → New repository variable**: name `DSH_CI_FAILOVER`, value `selfhosted`. -2. Re-run the failed/queued required jobs (Re-run failed jobs on affected PRs, or let new pushes pick it up). +2. Retrigger the required jobs so they re-resolve their pool. Jobs already **queued** for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued. 3. That is the entire switch. Under failover the workflow also, automatically: halves `DSH_COVERAGE_MAX_WORKERS` to 12 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 16 (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs). ### Capacity during failover diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md index 4ec80ae364..b3b4149f46 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md @@ -19,7 +19,7 @@ Status: implemented ### 切换步骤（仓库管理员，约 1 分钟，无需合并） 1. 仓库 **Settings → Secrets and variables → Actions → Variables → New repository variable**：名称 `DSH_CI_FAILOVER`，值 `selfhosted`。 -2. 对受影响 PR 的失败/排队作业点 Re-run failed jobs（或等新推送自然触发）。 +2. 重新触发必需作业，使其重新解析运行器池。已经为托管标签**排队**的作业不会重定向，也无法原地 re-run，因此对于本手册所述的无限排队故障，应取消卡住的运行并 re-run all jobs，或推送一个新提交；“Re-run failed jobs”只有在作业真正失败（而非仍在排队）时才有用。 3. 切换到此完成。故障切换状态下工作流还会自动：把 `DSH_COVERAGE_MAX_WORKERS` 降为 12、`DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 16（共享虚拟机的争抢上限），并跳过托管路径的 pnpm 缓存恢复（虚拟机的持久 store 直接提供热安装）。 ### 切换期间的容量 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0be7655193..95b54b44f9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -55,8 +55,10 @@ jobs: persist-credentials: false # Pull requests consume the default-branch cache but do not put cache - # compression and upload on the paid latency-critical path. + # compression and upload on the paid latency-critical path. Skipped + # under failover — see the coverage lane's identical rationale. - uses: actions/cache/restore@v4 + if: vars.DSH_CI_FAILOVER != 'selfhosted' with: path: /home/runner/.local/share/pnpm/store/v11 key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }} @@ -398,7 +400,8 @@ jobs: # Hot-standby drill for the in-house self-hosted pool: every master move # re-runs the complete unsharded aggregate on the persistent 64-core VM, # continuously proving that environment can take over a required lane if - # the hosted pools degrade (the switch is then a one-line runs-on change). + # the hosted pools degrade (the switch is then setting the admin-only + # DSH_CI_FAILOVER variable — see the failover runbook, no merge required). # Push-triggered, so it always executes the base branch's own workflow # definition — no PR-editable path selects these runners. Non-blocking for # pull requests; no cache steps because the VM's persistent pnpm store and