18 个请求就够拍板吗
2026-10-31 22:05，arch-lab 订单服务 1.5.0 开始金丝雀，先接 5% 流量。90 秒内这个组只收到 18 个请求，失败 0 个，控制器判定“健康”，把流量推到 50%。两分钟后库存预占超时率升到 7%，但 HTTP 错误率只有 0.4%，总错误率仍低于 1% 的阈值，自动回滚没有触发，直到仓库侧告警才人工切回。
复盘时三个问题同时成立：样本太小，18 个请求连一个 9% 的失败率都未必能观察到；指标太粗，库存超时被计成“业务降级”而不是失败；判定太快，单个窗口健康就被当成整体健康。金丝雀发布不是“先放一点流量看看”，而是一套有样本下限、有守卫指标、有连续窗口和明确退出条件的状态机。基础的金丝雀与蓝绿对比可以对照滚动、蓝绿与金丝雀 和部署策略 ，本节只讨论控制器如何做决定。
判据要分五层
把“健康”压成一个错误率，必然会在某个维度上失明。arch-lab 采用五层判据：样本量决定“能不能判”，窗口时长决定“抖动算不算”，可用性与延迟决定“服务是否变差”，业务守卫决定“钱和库存有没有错”。前两层不满足时只能等待，后三层任一越界才允许回滚。
| 层次 | 指标 | 示例阈值 | 未满足时 | 要防的问题 |
|---|---|---|---|---|
| 样本 | 窗口请求数 | 每窗 ≥ 100 | 保持当前比例 | 小样本噪声 |
| 时长 | 窗口持续 | 每窗 ≥ 3 分钟 | 保持当前比例 | 瞬时抖动 |
| 可用性 | 失败率 | ≤ 2%，连续 2 窗 | 回滚 | 整体错误上升 |
| 延迟 | P99 | ≤ 300ms，连续 2 窗 | 回滚 | 长尾拖垮调用方 |
| 业务守卫 | 超卖、重复扣款、越权 | 必须为 0 | 立即回滚 | 聚合指标掩盖资金错误 |
业务守卫必须独立于错误率。库存预占超时、支付重复回调、订单状态非法跳转，这些事件在 HTTP 层可能都是 200，在总错误率里完全看不见；它们一旦发生，继续放量的期望损失远高于一次误回滚。反过来，单次失败率越界也不必立刻回滚，先用连续窗口过滤噪声，这就是“守卫立即、统计连续”的分工。
状态机而不是一次性判断
控制器只有四个动作：等待、晋级下一阶段、全量、回滚。等待不是失败，它表示样本或时间还不够；晋级要重置窗口，因为新比例下的流量结构已经变了；全量是最后一个阶段通过后的终态；回滚是终态，之后不再接受观测。状态机把“再等等”和“有问题”分开，避免把样本不足误判成健康，也避免把一次抖动误判成故障。
实现上还要拒绝倒序观测。真实监控回填、时钟漂移或重复投递都可能让窗口时间倒退，如果控制器照单全收，连续窗口的计数就会失真。Observe 要求时间严格前进，计划校验要求比例严格递增、样本下限为正、错误率在 0 到 1 之间；这些拒绝不是防御性代码，而是让判据在坏输入下仍然可解释。
可运行实现
下面的 release 包与 9.1 共用，Clock 由 9.1 定义。CanaryPlan.Validate 检查阶段与阈值；Canary.Observe 按“守卫立即回滚、统计连续两窗”的规则返回动作；终态之后的观测返回 ErrCanaryClosed，防止回滚后又被旧数据推回运行态。
// file: internal/platform/release/canary.go
// tier: L1
// verify: go test ./internal/platform/release/ -count=1
package release
import (
"errors"
"fmt"
"time"
)
// Action is what the release controller should do after one observation.
type Action string
const (
ActionWait Action = "wait"
ActionAdvance Action = "advance"
ActionPromote Action = "promote"
ActionRollback Action = "rollback"
)
// Stage is one canary step: route Percent, then collect enough evidence.
type Stage struct {
Percent int
MinSamples int
MinDuration time.Duration
MaxErrorRate float64
MaxP99 time.Duration
}
// CanaryPlan is an ordered list of stages plus a streak requirement.
type CanaryPlan struct {
Stages []Stage
Consecutive int
}
// Validate rejects plans that cannot produce a defensible decision.
func (p CanaryPlan) Validate() error {
if len(p.Stages) == 0 {
return errors.New("release: canary plan has no stages")
}
prev := 0
for i, s := range p.Stages {
if s.Percent <= prev || s.Percent > 100 {
return fmt.Errorf("release: stage %d percent %d not in (%d,100]", i, s.Percent, prev)
}
if s.MinSamples <= 0 {
return fmt.Errorf("release: stage %d has no minimum sample", i)
}
if s.MaxErrorRate < 0 || s.MaxErrorRate > 1 {
return fmt.Errorf("release: stage %d error rate %v outside 0..1", i, s.MaxErrorRate)
}
prev = s.Percent
}
if p.Consecutive < 1 {
return errors.New("release: consecutive windows must be at least 1")
}
return nil
}
// Observation is one measurement window of the canary group.
type Observation struct {
At time.Time
Requests int
Failures int
P99 time.Duration
GuardrailBreaches int
}
// Decision is the controller's verdict for one observation.
type Decision struct {
At time.Time
Stage int
Action Action
Reason string
}
// Canary advances only with enough samples, enough time and no guardrail
// breach; otherwise it holds the current stage or rolls back.
type Canary struct {
plan CanaryPlan
clock Clock
state string
stage int
windowAt time.Time
lastAt time.Time
samples int
failures int
goodStreak int
badStreak int
}
// ErrCanaryClosed means a terminal decision was already made.
var ErrCanaryClosed = errors.New("release: canary is closed")
// NewCanary validates the plan and starts in the pending state.
func NewCanary(plan CanaryPlan, clock Clock) (*Canary, error) {
if err := plan.Validate(); err != nil {
return nil, err
}
if clock == nil {
return nil, errors.New("release: canary needs a clock")
}
return &Canary{plan: plan, clock: clock, state: "pending"}, nil
}
// Start opens the first observation window.
func (c *Canary) Start() error {
if c.state != "pending" {
return fmt.Errorf("release: cannot start from %s", c.state)
}
c.state = "running"
c.windowAt = c.clock()
c.lastAt = c.windowAt
return nil
}
// Observe consumes one window and returns the next action.
func (c *Canary) Observe(o Observation) (Decision, error) {
if c.state == "promoted" || c.state == "rolled_back" {
return Decision{}, ErrCanaryClosed
}
if c.state != "running" {
return Decision{}, errors.New("release: canary not started")
}
if o.Requests <= 0 || o.Failures < 0 || o.Failures > o.Requests {
return Decision{}, errors.New("release: invalid observation counts")
}
if !o.At.After(c.lastAt) {
return Decision{}, errors.New("release: observation time must advance")
}
c.lastAt = o.At
if o.GuardrailBreaches > 0 {
return c.decide(o.At, ActionRollback,
fmt.Sprintf("guardrail breached %d time(s) at %d%%", o.GuardrailBreaches, c.plan.Stages[c.stage].Percent)), nil
}
c.samples += o.Requests
c.failures += o.Failures
st := c.plan.Stages[c.stage]
if c.samples < st.MinSamples || o.At.Sub(c.windowAt) < st.MinDuration {
return c.decide(o.At, ActionWait,
fmt.Sprintf("need %d samples/%s, have %d/%s", st.MinSamples, st.MinDuration, c.samples, o.At.Sub(c.windowAt))), nil
}
rate := float64(c.failures) / float64(c.samples)
if rate > st.MaxErrorRate || (st.MaxP99 > 0 && o.P99 > st.MaxP99) {
c.goodStreak = 0
c.badStreak++
if c.badStreak >= c.plan.Consecutive {
return c.decide(o.At, ActionRollback,
fmt.Sprintf("error rate %.3f > %.3f for %d window(s)", rate, st.MaxErrorRate, c.badStreak)), nil
}
return c.decide(o.At, ActionWait, fmt.Sprintf("error rate %.3f over limit, holding", rate)), nil
}
c.badStreak = 0
c.goodStreak++
if c.goodStreak < c.plan.Consecutive {
return c.decide(o.At, ActionWait, "healthy, waiting for a second window"), nil
}
if c.stage == len(c.plan.Stages)-1 {
return c.decide(o.At, ActionPromote, "all stages healthy"), nil
}
c.stage++
c.samples, c.failures = 0, 0
c.windowAt = o.At
c.goodStreak, c.badStreak = 0, 0
return c.decide(o.At, ActionAdvance, fmt.Sprintf("advanced to %d%%", c.plan.Stages[c.stage].Percent)), nil
}
func (c *Canary) decide(at time.Time, a Action, reason string) Decision {
if a == ActionPromote {
c.state = "promoted"
}
if a == ActionRollback {
c.state = "rolled_back"
}
return Decision{At: at, Stage: c.stage, Action: a, Reason: reason}
}
代码里有四个刻意选择。第一，守卫越界在累计样本之前判断，所以哪怕只有 20 个请求，一次超卖也能立即回滚。第二，统计越界要连续 Consecutive 个窗口，单窗抖动只进入等待。第三，晋级时清零窗口并重置时间起点，避免把前一阶段的数据算进新比例。第四，终态之后拒绝观测，防止重复投递把已经回滚的发布重新“判活”。
测试块覆盖正常晋级、守卫立即回滚、样本与时长不足、连续坏窗口以及计划校验：
// file: internal/platform/release/canary_test.go
// tier: L1
// verify: go test ./internal/platform/release/ -count=1
package release
import (
"testing"
"time"
)
func started(t *testing.T, p CanaryPlan, base time.Time) *Canary {
t.Helper()
c, err := NewCanary(p, func() time.Time { return base })
if err != nil {
t.Fatalf("NewCanary: %v", err)
}
if err := c.Start(); err != nil {
t.Fatalf("Start: %v", err)
}
return c
}
func TestCanaryHappyPath(t *testing.T) {
base := time.Date(2026, 10, 31, 4, 0, 0, 0, time.UTC)
c := started(t, CanaryPlan{Stages: []Stage{
{Percent: 5, MinSamples: 100, MaxErrorRate: 0.02},
{Percent: 50, MinSamples: 100, MaxErrorRate: 0.02},
}, Consecutive: 2}, base)
var got []Action
for i := 0; i < 4; i++ {
d, err := c.Observe(Observation{At: base.Add(time.Duration(i+1) * time.Minute), Requests: 100})
if err != nil {
t.Fatalf("observe %d: %v", i, err)
}
got = append(got, d.Action)
}
want := []Action{ActionWait, ActionAdvance, ActionWait, ActionPromote}
for i := range want {
if got[i] != want[i] {
t.Fatalf("actions = %v, want %v", got, want)
}
}
t.Logf("actions=%v stage=%d", got, c.stage)
}
func TestCanaryGuardrailRollsBackImmediately(t *testing.T) {
base := time.Date(2026, 10, 31, 4, 10, 0, 0, time.UTC)
c := started(t, CanaryPlan{Stages: []Stage{{Percent: 5, MinSamples: 1000, MaxErrorRate: 0.01}}, Consecutive: 2}, base)
d, err := c.Observe(Observation{At: base.Add(time.Second), Requests: 20, GuardrailBreaches: 1})
if err != nil || d.Action != ActionRollback {
t.Fatalf("guardrail window = %+v, %v", d, err)
}
if _, err := c.Observe(Observation{At: base.Add(2 * time.Second), Requests: 1}); err != ErrCanaryClosed {
t.Fatalf("closed canary returned %v", err)
}
t.Logf("guardrail action=%s reason=%q", d.Action, d.Reason)
}
func TestCanaryWaitsForSampleAndTime(t *testing.T) {
base := time.Date(2026, 10, 31, 4, 20, 0, 0, time.UTC)
c := started(t, CanaryPlan{
Stages: []Stage{{Percent: 10, MinSamples: 100, MinDuration: time.Minute, MaxErrorRate: 0.01}},
Consecutive: 2,
}, base)
first, err := c.Observe(Observation{At: base.Add(30 * time.Second), Requests: 100})
if err != nil || first.Action != ActionWait {
t.Fatalf("small window = %+v, %v", first, err)
}
second, err := c.Observe(Observation{At: base.Add(61 * time.Second), Requests: 100})
if err != nil || second.Action != ActionWait {
t.Fatalf("healthy first window = %+v, %v", second, err)
}
t.Logf("first=%s second=%s reason=%q", first.Action, second.Action, second.Reason)
}
func TestCanaryNeedsConsecutiveBadWindows(t *testing.T) {
base := time.Date(2026, 10, 31, 4, 30, 0, 0, time.UTC)
c := started(t, CanaryPlan{Stages: []Stage{{Percent: 5, MinSamples: 100, MaxErrorRate: 0.05}}, Consecutive: 2}, base)
first, err := c.Observe(Observation{At: base.Add(time.Minute), Requests: 100, Failures: 10})
if err != nil || first.Action != ActionWait {
t.Fatalf("first bad window = %+v, %v", first, err)
}
second, err := c.Observe(Observation{At: base.Add(2 * time.Minute), Requests: 100, Failures: 10})
if err != nil || second.Action != ActionRollback {
t.Fatalf("second bad window = %+v, %v", second, err)
}
t.Logf("first=%s second=%s", first.Action, second.Action)
}
func TestCanaryPlanValidation(t *testing.T) {
cases := []CanaryPlan{
{},
{Stages: []Stage{{Percent: 50}, {Percent: 10}}, Consecutive: 2},
{Stages: []Stage{{Percent: 0}}, Consecutive: 2},
{Stages: []Stage{{Percent: 10, MinSamples: 10, MaxErrorRate: 1.5}}, Consecutive: 2},
{Stages: []Stage{{Percent: 10, MinSamples: 10}}, Consecutive: 0},
}
for i, p := range cases {
if err := p.Validate(); err == nil {
t.Fatalf("case %d accepted", i)
}
}
t.Logf("rejected=%d", len(cases))
}
本机实测
环境为 macOS darwin/arm64、Go 1.26.0；gofmt -l . 与 go vet ./... 均无输出。运行本节相关测试：
$ go test ./internal/platform/release/ -run 'TestCanary' -v -count=1
=== RUN TestCanaryHappyPath
canary_test.go:43: actions=[wait advance wait promote] stage=1
--- PASS: TestCanaryHappyPath (0.00s)
=== RUN TestCanaryGuardrailRollsBackImmediately
canary_test.go:56: guardrail action=rollback reason="guardrail breached 1 time(s) at 5%"
--- PASS: TestCanaryGuardrailRollsBackImmediately (0.00s)
=== RUN TestCanaryWaitsForSampleAndTime
canary_test.go:73: first=wait second=wait reason="healthy, waiting for a second window"
--- PASS: TestCanaryWaitsForSampleAndTime (0.00s)
=== RUN TestCanaryNeedsConsecutiveBadWindows
canary_test.go:87: first=wait second=rollback
--- PASS: TestCanaryNeedsConsecutiveBadWindows (0.00s)
=== RUN TestCanaryPlanValidation
canary_test.go:103: rejected=5
--- PASS: TestCanaryPlanValidation (0.00s)
PASS
ok archlab/internal/platform/release 0.277s
读数要按“决策序列”看，而不是只看 PASS。正常路径的动作是 wait → advance → wait → promote：第一窗健康但还不够连续，第二窗才晋级；进入 50% 后重新计数，再连续两窗才全量。守卫路径在只有 20 个请求、远低于 1000 样本下限时仍然立即回滚，说明业务守卫不参与样本门槛，也说明回滚之后再次观测会得到 ErrCanaryClosed。坏窗口路径第一窗只等待，第二窗才回滚；这正是“统计连续”要过滤的噪声。
边界如下：这是固定时钟、合成观测值的单进程测试，0.277 秒不代表真实发布耗时；真实流量里“窗口”由监控系统切分，窗口边界、指标聚合延迟和样本归属都会影响结果。上线前要用真实指标重放一次历史发布，确认失败率、P99 与业务守卫的口径和控制器一致。错误预算与发布门禁的配合可以对照错误预算 ，熔断与限流在发布期间的保护可以对照熔断与限流 。
常见错误与边界
- 只有总错误率，没有业务守卫。超卖、重复扣款、越权在 HTTP 层可能是 200，必须单独计数并设为“非零即回滚”。
- 样本不足就晋级。18 个请求观察不到 5% 的失败率；样本下限应按基线失败率和希望检出的最小劣化计算，而不是拍一个固定数字。
- 一个窗口越界就回滚。单窗抖动会制造大量误回滚；用连续窗口过滤，但守卫必须保留立即回滚通道。
- 回滚只切流量，不回滚配置与数据。新版本可能已经写入新字段或新状态，回滚前要确认旧版本能读；数据兼容窗口见回滚与兼容窗口 。
- 按请求随机分桶。同一用户会在新旧版本间闪烁，指标也被污染；分桶单位应是有稳定标识的实例、租户或会话。
小结
金丝雀的核心不是“放多少流量”，而是“在什么证据下允许继续”。样本量与窗口时长回答“能不能判”，可用性与延迟回答“服务有没有变差”，业务守卫回答“有没有产生不可接受的业务后果”。把守卫设为立即回滚、把统计指标设为连续窗口，控制器才能在噪声和风险之间做出可解释的选择。
本节的状态机只处理单个服务的流量比例，没有处理后端依赖的容量与数据库变更。下一节把发布与 schema 变更放在同一条时间线上：先 expand 让新旧代码都能跑，再回填，最后 contract；任何一步缺少前置条件，发布都应该停住。
阅读导航：上一节：9.1 制品、环境与不可变部署 · 下一节：9.3 数据库变更与发布协同：expand-contract 。
继续阅读
探索更多技术文章
浏览归档，发现更多关于系统设计、工具链和工程实践的内容。