熔断不是限流的替代品
限流保护自己不被流量打垮，熔断保护自己不被下游拖垮。通用模式在 架构中的熔断与限流 里已经讲过；本节只做一件事：把 arch-lab 支付下游的熔断器写成可注入时钟、可并发测试、可手工干预的状态机，并把“什么时候重开、重开时放多少探测”变成明确规则。
三态与两个判定
closed 正常放行并记录结果；open 直接返回 CIRCUIT_OPEN，不再产生真实调用；half-open 只放少量探测，探测成功足够多次才回到 closed，任何一次探测失败立即回到 open。状态迁移只由三类事件触发：窗口统计越线、open 时长到期、人工强制操作。把这三类事件都写进 Transition 日志，排障时才能回答“它为什么打开”。
滑窗成功率、最少请求与慢调用
用时间滑窗而不是固定计数窗口：每次调用记录时间戳、是否失败、是否慢调用，prune 丢掉窗口外的记录。只有窗口内样本数达到 MinRequests 才评估，避免“1 个请求失败 = 100% 失败率”的误熔断。失败率和慢调用率是两个独立判定：失败率看错误，慢调用率看“没报错但已经不可用”的请求。任一条越线都打开，并记录原因。
// file: internal/platform/circuitbreaker/breaker.go
// tier: L1
// verify: go test -race ./internal/platform/circuitbreaker/
package circuitbreaker
import (
"errors"
"sync"
"time"
)
var ErrOpen = errors.New("CIRCUIT_OPEN")
type State string
const (
Closed State = "closed"
Open State = "open"
HalfOpen State = "half-open"
)
type Config struct {
Window time.Duration
MinRequests int
FailureRatio float64
SlowCall time.Duration
SlowRatio float64
OpenDuration time.Duration
HalfOpenMaxProbes int
HalfOpenSuccesses int
}
type Clock interface{ Now() time.Time }
type ManualClock struct{ now time.Time }
func NewManualClock(t time.Time) *ManualClock { return &ManualClock{now: t} }
func (c *ManualClock) Now() time.Time { return c.now }
func (c *ManualClock) Advance(d time.Duration) { c.now = c.now.Add(d) }
type Transition struct {
At time.Time
From, To State
Reason string
}
type outcome struct {
at time.Time
fail, slow bool
}
type Breaker struct {
mu sync.Mutex
cfg Config
clock Clock
state State
openedAt time.Time
window []outcome
probes int
successes int
forced bool
forcedUntil time.Time
log []Transition
}
func New(clock Clock, cfg Config) *Breaker {
return &Breaker{clock: clock, cfg: cfg, state: Closed}
}
func (b *Breaker) set(now time.Time, to State, reason string) {
if b.state == to {
return
}
b.log = append(b.log, Transition{At: now, From: b.state, To: to, Reason: reason})
b.state = to
}
func (b *Breaker) Allow() (*Permit, error) {
b.mu.Lock()
defer b.mu.Unlock()
now := b.clock.Now()
if b.forced {
if b.forcedUntil.IsZero() || now.Before(b.forcedUntil) {
return nil, ErrOpen
}
b.forced = false
b.set(now, HalfOpen, "manual open expired")
b.probes, b.successes = 0, 0
}
if b.state == Open {
if now.Sub(b.openedAt) < b.cfg.OpenDuration {
return nil, ErrOpen
}
b.set(now, HalfOpen, "open duration elapsed")
b.probes, b.successes = 0, 0
}
if b.state == HalfOpen {
if b.probes >= b.cfg.HalfOpenMaxProbes {
return nil, ErrOpen
}
b.probes++
}
return &Permit{breaker: b, halfOpen: b.state == HalfOpen}, nil
}
func (b *Breaker) trip(now time.Time, reason string) {
b.set(now, Open, reason)
b.openedAt = now
b.probes, b.successes = 0, 0
}
func (b *Breaker) close(now time.Time, reason string) {
b.set(now, Closed, reason)
b.window = nil
b.probes, b.successes = 0, 0
b.forced = false
}
func (b *Breaker) prune(now time.Time) {
cutoff := now.Add(-b.cfg.Window)
kept := b.window[:0]
for _, o := range b.window {
if o.at.After(cutoff) {
kept = append(kept, o)
}
}
b.window = kept
}
func (b *Breaker) evaluate(now time.Time) {
if b.state != Closed || len(b.window) < b.cfg.MinRequests {
return
}
fail, slow := 0, 0
for _, o := range b.window {
if o.fail {
fail++
}
if o.slow {
slow++
}
}
n := float64(len(b.window))
if float64(fail)/n >= b.cfg.FailureRatio {
b.trip(now, "failure ratio")
return
}
if b.cfg.SlowCall > 0 && float64(slow)/n >= b.cfg.SlowRatio {
b.trip(now, "slow call ratio")
}
}
type Permit struct {
breaker *Breaker
halfOpen bool
}
func (p *Permit) Done(err error, slow bool) {
b := p.breaker
b.mu.Lock()
defer b.mu.Unlock()
now := b.clock.Now()
if p.halfOpen {
b.probes--
if err != nil || (b.cfg.SlowCall > 0 && slow) {
b.trip(now, "half-open probe failed")
return
}
b.successes++
if b.successes >= b.cfg.HalfOpenSuccesses {
b.close(now, "half-open probes succeeded")
}
return
}
b.window = append(b.window, outcome{at: now, fail: err != nil, slow: slow})
b.prune(now)
b.evaluate(now)
}
func (b *Breaker) ForceOpen(until time.Time, reason string) {
b.mu.Lock()
defer b.mu.Unlock()
now := b.clock.Now()
b.forced = true
b.forcedUntil = until
b.set(now, Open, "manual: "+reason)
b.openedAt = now
}
func (b *Breaker) ForceClose(reason string) {
b.mu.Lock()
defer b.mu.Unlock()
b.close(b.clock.Now(), "manual: "+reason)
}
func (b *Breaker) State() State {
b.mu.Lock()
defer b.mu.Unlock()
return b.state
}
一个容易忽略的细节：half-open 的探测也是一次真实调用，必须走 Done 归还配额。probes 在 Allow 时加、在 Done 时减；如果调用方拿了 permit 却不调 Done，探测配额会被永久占住，熔断器再也回不到 closed。接口只能约定“一次 permit 必须且只能 Done 一次”。
并发测试：20 个请求只放 3 个探测
测试先在 closed 状态打 10 个失败请求（MinRequests=10，失败率 100%）触发 open；推进 5 秒后，20 个 goroutine 同时 Allow，断言只有 3 个拿到探测许可；让其中一个探测失败，状态必须立刻回到 open；再等 5 秒，连续 3 次成功探测后才 closed。
// file: internal/platform/circuitbreaker/breaker_test.go
// tier: L1
// verify: go test -race ./internal/platform/circuitbreaker/
package circuitbreaker
import (
"errors"
"sync"
"testing"
"time"
)
var base = time.Date(2026, 10, 18, 12, 0, 0, 0, time.UTC)
func testConfig() Config {
return Config{Window: 10 * time.Second, MinRequests: 10, FailureRatio: 0.5,
SlowCall: 100 * time.Millisecond, SlowRatio: 0.6,
OpenDuration: 5 * time.Second, HalfOpenMaxProbes: 3, HalfOpenSuccesses: 3}
}
func mustCall(t *testing.T, b *Breaker, err error, slow bool) {
t.Helper()
p, perr := b.Allow()
if perr != nil {
t.Fatalf("allow rejected: %v", perr)
}
p.Done(err, slow)
}
func TestBreakerTransitions(t *testing.T) {
clock := NewManualClock(base)
b := New(clock, testConfig())
for i := 0; i < 10; i++ {
mustCall(t, b, errors.New("down"), false)
}
if b.State() != Open {
t.Fatalf("state=%s want open", b.State())
}
clock.Advance(5 * time.Second)
var (
wg sync.WaitGroup
mu sync.Mutex
permits []*Permit
)
for i := 0; i < 20; i++ {
wg.Add(1)
go func() {
defer wg.Done()
if p, err := b.Allow(); err == nil {
mu.Lock()
permits = append(permits, p)
mu.Unlock()
}
}()
}
wg.Wait()
if len(permits) != 3 {
t.Fatalf("half-open permits=%d want 3", len(permits))
}
permits[0].Done(errors.New("probe failed"), false)
if b.State() != Open {
t.Fatalf("probe failure state=%s want open", b.State())
}
clock.Advance(5 * time.Second)
for i := 0; i < 3; i++ {
mustCall(t, b, nil, false)
}
if b.State() != Closed {
t.Fatalf("state=%s want closed", b.State())
}
for _, tr := range b.log {
t.Logf("%s %s -> %s (%s)", tr.At.Format("15:04:05"), tr.From, tr.To, tr.Reason)
}
}
func TestSlowCalls(t *testing.T) {
clock := NewManualClock(base)
c := testConfig()
c.FailureRatio = 1
b := New(clock, c)
for i := 0; i < 10; i++ {
mustCall(t, b, nil, true)
}
if b.State() != Open {
t.Fatalf("slow calls state=%s want open", b.State())
}
clock.Advance(5 * time.Second)
p, err := b.Allow()
if err != nil {
t.Fatal(err)
}
p.Done(nil, true)
if b.State() != Open {
t.Fatalf("slow probe state=%s want open", b.State())
}
}
func TestManualOverride(t *testing.T) {
clock := NewManualClock(base)
b := New(clock, testConfig())
b.ForceOpen(time.Time{}, "redis failover")
if _, err := b.Allow(); err != ErrOpen {
t.Fatalf("forced open err=%v", err)
}
b.ForceClose("failover done")
if _, err := b.Allow(); err != nil {
t.Fatalf("after force close err=%v", err)
}
b.ForceOpen(base.Add(time.Second), "short maintenance")
clock.Advance(2 * time.Second)
if _, err := b.Allow(); err != nil {
t.Fatalf("expired force open err=%v", err)
}
}
本机 Go 1.26.0（darwin/arm64），命令 go test -race -v ./... -count=1，真实输出保存在 .local/reports/be-arch/be-arch-perf-06-2-circuit-breaker-state-machine-tests.txt。下面的并发配额与状态迁移是确定性结果，耗时属于本机单次运行：
=== RUN TestBreakerTransitions
breaker_test.go:69: 12:00:00 closed -> open (failure ratio)
breaker_test.go:69: 12:00:05 open -> half-open (open duration elapsed)
breaker_test.go:69: 12:00:05 half-open -> open (half-open probe failed)
breaker_test.go:69: 12:00:10 open -> half-open (open duration elapsed)
breaker_test.go:69: 12:00:10 half-open -> closed (half-open probes succeeded)
--- PASS: TestBreakerTransitions (0.00s)
=== RUN TestSlowCalls
--- PASS: TestSlowCalls (0.00s)
=== RUN TestManualOverride
--- PASS: TestManualOverride (0.00s)
PASS
ok archlab/internal/platform/circuitbreaker 1.485s
测试同时覆盖三件事：20 个并发 Allow 恰好只产生 3 个探测许可；探测失败把状态从 half-open 拉回 open，而不是“再试一次”；慢调用在两个位置都计入——closed 阶段 10 个慢调用按 60% 慢调用率打开，half-open 阶段一个慢探测同样立即重开。-race 下没有报告数据竞争。
支付下游的阈值
下面是 arch-lab 的配置目标值，不是线上实测：
| 参数 | 值 | 依据 |
|---|---|---|
| Window | 10s | 覆盖一次渠道抖动，又不至于记住太久 |
| MinRequests | 20 | 支付 5 QPS 时约 4 秒样本，避免小样本误判 |
| FailureRatio | 0.5 | 一半失败已明显偏离正常错误率 |
| SlowCall / SlowRatio | 300ms / 0.6 | 超时预算里支付单次 120ms，300ms 已是三倍 |
| OpenDuration | 5s | 给下游一次自愈窗口，5 QPS 下最多拒绝 25 单 |
| HalfOpenMaxProbes / Successes | 3 / 3 | 3 个并发探测足以覆盖冷启动，成功 3 次才恢复 |
窗口 10 秒、最少 20 个样本意味着：突发 10 个请求全失败也不会立刻熔断，因为样本不足。这是有意的——支付链路的单次失败更可能是参数问题，而不是下游整体故障。
误熔断的代价与手工入口
误熔断的代价可以直接算：支付 5 QPS、打开 5 秒，约 25 笔订单收到 CIRCUIT_OPEN；如果客户端按 RetryAfter 退避，这些订单大多能在恢复后成功，损失的是转化率和一次用户等待。反过来，下游真实失败率 30% 时若迟迟不熔断，5 秒内约 7–8 笔失败，每笔还可能重试 2 次，把 300ms 超时放大成近 1 秒的线程占用。阈值就是这两笔成本之间的取舍，必须按业务流量算，不能照抄。
运维入口有两个，都必须带原因和审计：POST /admin/circuit/payment/open 接受 {"ttl":"60s","reason":"provider incident"}，到期自动进入 half-open；POST /admin/circuit/payment/close 立即回到 closed 并清空窗口。强制打开适合“已知下游维护”，强制关闭只适合“确认误判”，因为下一次真实失败仍会重新打开；不要用它长期绕过熔断。
常见错误与边界
第一，没有最少请求数。窗口里只有 1 个请求且失败，失败率就是 100%，熔断器会被一次偶发错误打开；MinRequests 是必需参数。
第二，half-open 不限制并发。只判断“时间到了”就全部放行，探测会变成一次小型流量洪峰，把刚恢复的下游再次打挂；必须用 probes 配额。
第三，探测成功一次就关闭。单次成功可能只是缓存命中或下游刚好空闲；恢复条件应是连续 N 次成功，N 按下游冷启动时间设定。
第四，把慢调用同时算进失败率和慢调用率。两套判定会叠加放大敏感度；要么只用一个，要么明确两者都开启并分别调阈值。
第五，强制关闭当成永久开关。没有 TTL、没有审计的强制关闭会让熔断器形同虚设，下一次故障直接穿透。
第六，open 时长拍脑袋。太短会在下游还没恢复时反复探测，太长会让可恢复的抖动变成长时间不可用；5 秒是 arch-lab 的目标值，必须用故障演练验证。
小结
熔断器是一个三态状态机：closed 用时间滑窗统计失败率和慢调用率，open 在 OpenDuration 内直接拒绝，half-open 用有限的并发探测决定恢复还是重开。arch-lab 的支付下游目标配置是 10 秒窗口、最少 20 个样本、50% 失败率、300ms 慢调用阈值、5 秒打开、3 个探测且连续 3 次成功才关闭；并发测试证明 20 个请求只放 3 个探测，探测失败立即重开。下一节讨论熔断打开之后，订单查询和支付回调分别怎样降级、哪些步骤绝对不能降级。
阅读导航：上一节：6.1 限流算法与分布式限流 · 下一节：6.3 降级、兜底与用户体验 。
继续阅读
探索更多技术文章
浏览归档，发现更多关于系统设计、工具链和工程实践的内容。