docs(ci): six always-on instances, no pre-registered spares
The spare tier is retired. Steady-state pool load is one serial standby job per master push, so six always-on instances already are the failover capacity; pre-registered offline runners are a silently expiring guarantee (GitHub garbage-collects them after 30 days offline). Incident-time extra capacity is a one-minute org-token registration, now documented in the runbook.
This commit is contained in:
Chinesezjc
+3
-3
@@ -1,6 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-22-evidence-based-larger-hosted-runners.md: 21e602b2b5850176df981dcf448f4f827b756719
|
||||
2026-07-22-evidence-based-larger-hosted-runners.zh.md: ba49ff18ac304f4078d4c8ebfd00bb1a85ada0b3
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md
|
||||
2026-07-22-evidence-based-larger-hosted-runners.md: 5b399be5571ddaf1f775ba43a2233198b8e09b18
|
||||
2026-07-22-evidence-based-larger-hosted-runners.zh.md: 40970ec33c1a16af85ea47be3fc932209efdd654
|
||||
+1
-1
@@ -52,7 +52,7 @@ The process-bound coverage project contains exactly five suite files. Thirty-two
|
||||
|
||||
Complete serial Linux, macOS, and Windows references run only when `master` moves. Pull requests use the enterprise required path plus standard-hosted compatibility jobs, while other larger-runner sizes run only by manual dispatch.
|
||||
|
||||
An additional serial Linux reference runs on the in-house self-hosted pool (`vm-backup` label: a 64-core VM with four always-on systemd-managed runner instances plus four registered spares) on every `master` push. It is a hot-standby drill, not a required check: each run re-proves that the persistent VM can execute the complete unsharded aggregate. The actual switch is pre-wired: the three required Linux jobs resolve their pool through the admin-only `DSH_CI_FAILOVER` repository variable, so an outage response is setting one variable and re-running — no merge, which would be deadlocked behind the failing checks themselves ([runbook](2026-07-26-ci-failover-runbook.md)). Because the standby lane is push-triggered, it always executes the base branch's workflow definition — no pull-request-editable path can route code to these runners, and the repository additionally keeps forking disabled.
|
||||
An additional serial Linux reference runs on the in-house self-hosted pool (`vm-backup` label: a 64-core VM with six always-on systemd-managed runner instances) on every `master` push. It is a hot-standby drill, not a required check: each run re-proves that the persistent VM can execute the complete unsharded aggregate. The actual switch is pre-wired: the three required Linux jobs resolve their pool through the admin-only `DSH_CI_FAILOVER` repository variable, so an outage response is setting one variable and re-running — no merge, which would be deadlocked behind the failing checks themselves ([runbook](2026-07-26-ci-failover-runbook.md)). Because the standby lane is push-triggered, it always executes the base branch's workflow definition — no pull-request-editable path can route code to these runners, and the repository additionally keeps forking disabled.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
|
||||
+1
-1
@@ -52,7 +52,7 @@ Windows 仓库工作在超过 16 核后收益很小，但 32 核池可以让完
|
||||
|
||||
只有在 `master` 移动时，才运行完整的 Linux、macOS 和 Windows 串行参考。拉取请求使用企业级运行器必需路径和标准托管兼容性作业，其他大型运行器规格仅通过手动触发运行。
|
||||
|
||||
另有一条串行 Linux 参考在每次 `master` 推送时运行于公司自有的自托管池（`vm-backup` 标签：一台 64 核虚拟机，运行 4 个常驻的 systemd 管理运行器实例，另有 4 个已注册备用位）。它是热备演练而非必需检查：每次运行都重新证明这台持久化虚拟机能够执行完整的未分片聚合流程。实际切换机制已预先布线：三个必需 Linux 作业通过仅限管理员的仓库变量 `DSH_CI_FAILOVER` 解析运行器池，因此故障响应就是设置一个变量并重跑——无需合并（合并本身会被正在失败的检查死锁）（[切换手册](2026-07-26-ci-failover-runbook.md)）。该热备通道由 push 触发，执行的始终是基线分支自身的工作流定义——不存在任何可由拉取请求编辑的路径能把代码路由到这些运行器上；此外仓库继续保持禁用 fork。
|
||||
另有一条串行 Linux 参考在每次 `master` 推送时运行于公司自有的自托管池（`vm-backup` 标签：一台 64 核虚拟机，运行 6 个常驻的 systemd 管理运行器实例）。它是热备演练而非必需检查：每次运行都重新证明这台持久化虚拟机能够执行完整的未分片聚合流程。实际切换机制已预先布线：三个必需 Linux 作业通过仅限管理员的仓库变量 `DSH_CI_FAILOVER` 解析运行器池，因此故障响应就是设置一个变量并重跑——无需合并（合并本身会被正在失败的检查死锁）（[切换手册](2026-07-26-ci-failover-runbook.zh.md)）。该热备通道由 push 触发，执行的始终是基线分支自身的工作流定义——不存在任何可由拉取请求编辑的路径能把代码路由到这些运行器上；此外仓库继续保持禁用 fork。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-26-ci-failover-runbook.md: db8e0676ecc6eeaea16438e7868ccf9ac43887cc
|
||||
2026-07-26-ci-failover-runbook.zh.md: b3b4149f460784e88ce03458fc556f402c38fa2f
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md
|
||||
2026-07-26-ci-failover-runbook.md: 0bce83e0f9c842fa3dd73ae9c0a3eefc0975cdae
|
||||
2026-07-26-ci-failover-runbook.zh.md: 4bc6c67bab754ad3f0127557b0d5e04f7934c8a2
|
||||
@@ -14,7 +14,7 @@ Each of the three required Linux jobs resolves its runner pool through the `DSH_
|
||||
|
||||
### What the in-house pool is
|
||||
|
||||
`vm-backup`: one 64-core VM, four always-on systemd-managed runner instances, four registered spares. Check the latest `serial / linux (self-hosted standby)` run before switching: a green standby is verified-yesterday capacity.
|
||||
`vm-backup`: one 64-core VM, six always-on systemd-managed runner instances. Check the latest `serial / linux (self-hosted standby)` run before switching: a green standby is verified-yesterday capacity.
|
||||
|
||||
### Switch (repo admin, ~1 minute, no merge)
|
||||
|
||||
@@ -24,15 +24,12 @@ Each of the three required Linux jobs resolves its runner pool through the `DSH_
|
||||
|
||||
### Capacity during failover
|
||||
|
||||
Four always-on instances absorb normal PR traffic. If queues build, bring the four registered spares online on the VM (no token needed — they are already registered):
|
||||
Six always-on instances absorb normal PR traffic (the pool's steady-state load is one serial standby job per master push, so failover capacity is effectively the full pool). If queues still build, register additional instances with an org registration token (org Settings → Actions → Runners → New runner) — cloning an existing runner directory and running `config.sh` takes about a minute per instance.
|
||||
|
||||
```bash
|
||||
for i in 7 8 9 10; do cd /data_local/actions-runner-$i && sudo ./svc.sh install ubuntu && sudo ./svc.sh start; done
|
||||
```
|
||||
|
||||
### Switch back
|
||||
|
||||
Delete the `DSH_CI_FAILOVER` variable (or set it to anything other than `selfhosted`). New runs resolve back to the hosted enterprise pools. Stop the spare instances if they were started.
|
||||
Delete the `DSH_CI_FAILOVER` variable (or set it to anything other than `selfhosted`). New runs resolve back to the hosted enterprise pools. Remove any extra instances that were registered during the incident.
|
||||
|
||||
### Trust boundary
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ Status: implemented
|
||||
|
||||
### 自有池是什么
|
||||
|
||||
`vm-backup`：一台 64 核虚拟机，4 个常驻 systemd 管理的运行器实例，另有 4 个已注册备用位。切换前先看 `serial / linux (self-hosted standby)` 最近一次运行：绿色 = 这套环境昨天刚被全量验证过。
|
||||
`vm-backup`：一台 64 核虚拟机，6 个常驻 systemd 管理的运行器实例。切换前先看 `serial / linux (self-hosted standby)` 最近一次运行：绿色 = 这套环境昨天刚被全量验证过。
|
||||
|
||||
### 切换步骤（仓库管理员，约 1 分钟，无需合并）
|
||||
|
||||
@@ -24,15 +24,12 @@ Status: implemented
|
||||
|
||||
### 切换期间的容量
|
||||
|
||||
4 个常驻实例可承接正常 PR 流量。若出现排队，在虚拟机上把 4 个已注册的备用位拉起（无需 token——它们已注册）：
|
||||
6 个常驻实例可承接正常 PR 流量（该池平时唯一的稳态负载是每次 master 推送一个串行热备作业，故障切换时几乎全池可用）。若仍出现排队，用组织级注册 token（组织 Settings → Actions → Runners → New runner）追加注册实例——复制现有 runner 目录再跑 `config.sh`，每个约一分钟。
|
||||
|
||||
```bash
|
||||
for i in 7 8 9 10; do cd /data_local/actions-runner-$i && sudo ./svc.sh install ubuntu && sudo ./svc.sh start; done
|
||||
```
|
||||
|
||||
### 切回
|
||||
|
||||
删除 `DSH_CI_FAILOVER` 变量（或改为 `selfhosted` 以外的任何值），新的运行即解析回托管企业池。若启动过备用实例，将其停止。
|
||||
删除 `DSH_CI_FAILOVER` 变量（或改为 `selfhosted` 以外的任何值），新的运行即解析回托管企业池。若故障期间追加注册过实例，将其移除。
|
||||
|
||||
### 信任边界
|
||||
|
||||
|
||||
Reference in New Issue
Block a user