一次“回滚”回到了另一个版本
2026-10-30 21:40,arch-lab 订单服务 1.4.2 因空指针触发回滚。发布系统按 orders:1.4.2 重新拉取镜像,三台实例启动后错误率没有下降。排查发现当天下午构建缓存失效,1.4.2 这个标签被重新构建并覆盖,回滚拉到的不是上周那份字节;审计日志只记了标签,没人能回答“上周生产实际运行的到底是哪份内容”。
这不是回滚脚本写错,而是发布系统把“名字”当成了“身份”。只要标签可以被覆盖,回滚、复现和审计就都失去基准。本节把制品、环境与部署拆成三个可验证的概念,并给出一个只依赖标准库的实现。回滚窗口本身的设计可以对照回滚与兼容窗口 ,制品来源与证明链路可以对照制品仓库与来源证明 ;本节只处理“部署引用什么”这一层。
标签是名字,摘要才是身份
标签是给人看的,它可以指向任何一次构建;摘要是由内容算出来的,同样的字节永远得到同样的值。把两者混用,发布记录里写的是标签,实际运行的却是另一份内容,回滚就变成了重新抽奖。
| 引用方式 | 可被覆盖 | 回滚语义 | 审计能回答 | 适用位置 |
|---|---|---|---|---|
可变标签 orders:latest | 是 | 重新拉取当前指向,可能已变 | 只知名字 | 本地试验 |
构建号 orders#1841 | 通常不可变 | 需依赖构建系统保留记录 | 构建记录 | 流水线内部 |
内容摘要 sha256:... | 否 | 精确回到同一份字节 | 内容本身 | 生产部署与回滚 |
摘要还有第二个作用:把“校验”从信任变成计算。注册表在写入时重新计算内容摘要,与声明的摘要不一致就拒绝;同一个摘要第二次写入也拒绝,因为不可变意味着不能悄悄替换。这样即使构建系统、对象存储或传输链路某一环出错,错误也会在进入环境之前暴露。
环境是“摘要 + 配置版本”的声明
环境不应该是一个可以随意打标签的命名空间,而应该是一条声明:生产当前运行摘要 D、配置版本 C,由谁在什么时间部署。部署动作只接受摘要,配置版本必须为正数,操作者必须留下身份;晋级则要求来源环境合法、制品已经烘焙足够长时间。
这条声明还带来一个可观测判据:实例上报自己实际加载的摘要,与环境声明的摘要比对。不一致的实例就是漂移,可能来自滚动发布未完成、手工热更新或节点回退。漂移检测不需要知道发布系统的内部状态,只需要“声明”和“实际”两个值。
| 观测 | 判据 | 动作 | 退出条件 |
|---|---|---|---|
| 部署请求带标签 | 不是 sha256: 摘要 | 直接拒绝 | 流水线改为解析摘要 |
| 内容与声明摘要不符 | 重算结果不同 | 拒绝入库并告警 | 重新构建并生成新摘要 |
| 晋级早于烘焙窗口 | 小于目标环境 MinBake | 保持候选,不切流量 | 达到窗口且指标平稳 |
| 运行实例摘要漂移 | 任一实例不等于声明 | 停止晋级,排查实例 | 全部实例回到声明摘要 |
可运行实现
下面的 release 包只依赖标准库。Registry 以摘要为键保存制品并校验内容;Environment.Deploy 只接受摘要;CanPromote 检查晋级路径与烘焙时间;Drift 比对声明摘要与实例上报值。本章三节共用同一个包,后续两节会继续加入金丝雀状态机与 expand-contract 检查。
// file: internal/platform/release/artifact.go
// tier: L1
// verify: go test ./internal/platform/release/ -count=1
package release
import (
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"slices"
"sort"
"strings"
"time"
)
// Clock lets deployment checks run against a fixed timeline in tests.
type Clock func() time.Time
// Digest is the content address of one artifact: "sha256:" plus 64 hex chars.
type Digest string
// DigestOf returns the content address of b.
func DigestOf(b []byte) Digest {
sum := sha256.Sum256(b)
return Digest("sha256:" + hex.EncodeToString(sum[:]))
}
// ParseDigest accepts only a lowercase sha256 content address, never a tag.
func ParseDigest(s string) (Digest, error) {
body, ok := strings.CutPrefix(s, "sha256:")
if !ok || len(body) != 64 {
return "", fmt.Errorf("release: %q is not a 64-char digest", s)
}
for _, r := range body {
if (r < '0' || r > '9') && (r < 'a' || r > 'f') {
return "", errors.New("release: digest is not lowercase hex")
}
}
return Digest(s), nil
}
// Artifact is an immutable build output.
type Artifact struct {
Name, Version string
Digest Digest
Size int64
BuiltAt time.Time
}
// Registry stores artifacts by content address. A digest is never overwritten,
// so a rollback returns exactly the bytes that ran before.
type Registry struct{ artifacts map[Digest]Artifact }
// NewRegistry returns an empty registry.
func NewRegistry() *Registry { return &Registry{artifacts: map[Digest]Artifact{}} }
// Push verifies content and metadata before recording the artifact.
func (r *Registry) Push(a Artifact, content []byte) error {
if a.Name == "" || a.Version == "" || a.Size <= 0 || a.BuiltAt.IsZero() {
return errors.New("release: artifact metadata is incomplete")
}
if _, err := ParseDigest(string(a.Digest)); err != nil {
return err
}
if got := DigestOf(content); got != a.Digest {
return fmt.Errorf("release: content hashes to %s, declared %s", got, a.Digest)
}
if a.Size != int64(len(content)) {
return fmt.Errorf("release: size %d, content %d", a.Size, len(content))
}
if _, exists := r.artifacts[a.Digest]; exists {
return fmt.Errorf("release: digest %s already stored", a.Digest)
}
r.artifacts[a.Digest] = a
return nil
}
// Resolve returns the stored artifact for a digest.
func (r *Registry) Resolve(d Digest) (Artifact, error) {
a, ok := r.artifacts[d]
if !ok {
return Artifact{}, fmt.Errorf("release: digest %s not found", d)
}
return a, nil
}
// Deployment is one pinned release in one environment.
type Deployment struct {
Environment, Version, Actor string
Digest, Previous Digest
Config uint64
At time.Time
}
// DeployRequest is what a pipeline may ask an environment to run.
type DeployRequest struct {
Digest Digest
Config uint64
Actor string
}
// Environment holds the current deployment.
type Environment struct {
Name string
Accepts []string
MinBake time.Duration
Current *Deployment
}
// Deploy accepts a registry digest, never a tag such as "orders:latest".
func (e *Environment) Deploy(clock Clock, reg *Registry, req DeployRequest) (Deployment, error) {
if e.Name == "" {
return Deployment{}, errors.New("release: environment name is empty")
}
if _, err := ParseDigest(string(req.Digest)); err != nil {
return Deployment{}, err
}
art, err := reg.Resolve(req.Digest)
if err != nil {
return Deployment{}, err
}
if req.Config == 0 || req.Actor == "" {
return Deployment{}, errors.New("release: config and actor are required")
}
d := Deployment{Environment: e.Name, Digest: art.Digest, Version: art.Version,
Config: req.Config, Actor: req.Actor, At: clock()}
if e.Current != nil {
d.Previous = e.Current.Digest
}
e.Current = &d
return d, nil
}
// CanPromote checks the allowed path and the bake time.
func CanPromote(source, target Environment, clock Clock) error {
if !slices.Contains(target.Accepts, source.Name) {
return fmt.Errorf("release: %s does not accept promotion from %s", target.Name, source.Name)
}
if source.Current == nil {
return fmt.Errorf("release: %s has no deployment", source.Name)
}
if elapsed := clock().Sub(source.Current.At); elapsed < target.MinBake {
return fmt.Errorf("release: baked %s, want %s", elapsed, target.MinBake)
}
return nil
}
// Instance is one process reporting the digest it actually loaded.
type Instance struct {
ID string
Digest Digest
}
// DriftItem is one instance running something other than the declared digest.
type DriftItem struct {
Instance string
Running, Expected Digest
}
// Drift returns instances not running the declared digest, sorted by ID.
func Drift(expected Digest, instances []Instance) []DriftItem {
var out []DriftItem
for _, in := range instances {
if in.Digest != expected {
out = append(out, DriftItem{in.ID, in.Digest, expected})
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Instance < out[j].Instance })
return out
}
实现里有三个刻意选择。第一,注册表以摘要为键,重复写入直接报错,避免“同摘要不同内容”这种自相矛盾的状态。第二,部署请求先过 ParseDigest,标签在进入环境之前就被挡住,而不是等到拉取时才发现。第三,晋级同时检查路径和烘焙时间,因为“制品在测试环境跑过”和“制品在测试环境跑够久”不是一回事。
测试块覆盖摘要校验、注册表拒绝内容不符与重复写入、部署拒绝标签、晋级窗口以及漂移检测:
// file: internal/platform/release/artifact_test.go
// tier: L1
// verify: go test ./internal/platform/release/ -count=1
package release
import (
"strings"
"testing"
"time"
)
func fixedClock(t time.Time) Clock { return func() time.Time { return t } }
func registryWith(t *testing.T, content []byte, version string) (*Registry, Digest) {
t.Helper()
d := DigestOf(content)
reg := NewRegistry()
err := reg.Push(Artifact{Name: "orders", Version: version, Digest: d,
Size: int64(len(content)), BuiltAt: time.Unix(0, 0).UTC()}, content)
if err != nil {
t.Fatalf("push: %v", err)
}
return reg, d
}
func TestDigestAndRegistry(t *testing.T) {
content := []byte("orders image v1")
reg, d := registryWith(t, content, "1.4.2")
if _, err := ParseDigest(string(d)); err != nil {
t.Fatalf("DigestOf produced invalid digest: %v", err)
}
for _, bad := range []string{"orders:latest", "sha256:abc", "sha256:" + strings.Repeat("A", 64)} {
if _, err := ParseDigest(bad); err == nil {
t.Fatalf("accepted %q", bad)
}
}
art, _ := reg.Resolve(d)
if err := reg.Push(art, []byte("orders image v2")); err == nil {
t.Fatal("content/digest mismatch accepted")
}
if err := reg.Push(art, content); err == nil {
t.Fatal("duplicate digest overwrote an immutable artifact")
}
t.Logf("digest=%s version=%s", d, art.Version)
}
func TestDeployRequiresDigest(t *testing.T) {
content := []byte("orders image v1")
reg, d := registryWith(t, content, "1.4.2")
now := time.Date(2026, 10, 31, 0, 0, 0, 0, time.UTC)
env := Environment{Name: "prod"}
if _, err := env.Deploy(fixedClock(now), reg, DeployRequest{Digest: "orders:latest", Config: 7, Actor: "pipeline"}); err == nil {
t.Fatal("tag accepted as a deployable reference")
}
if _, err := env.Deploy(fixedClock(now), reg, DeployRequest{Digest: d, Config: 0, Actor: "pipeline"}); err == nil {
t.Fatal("zero config version accepted")
}
dep, err := env.Deploy(fixedClock(now), reg, DeployRequest{Digest: d, Config: 7, Actor: "pipeline"})
if err != nil || dep.Version != "1.4.2" || dep.Config != 7 {
t.Fatalf("Deploy = %+v, %v", dep, err)
}
t.Logf("deployed digest=%s config=%d", dep.Digest, dep.Config)
}
func TestPromotionBakeTime(t *testing.T) {
content := []byte("orders image v1")
reg, d := registryWith(t, content, "1.4.2")
base := time.Date(2026, 10, 31, 0, 0, 0, 0, time.UTC)
staging := Environment{Name: "staging"}
if _, err := staging.Deploy(fixedClock(base), reg, DeployRequest{Digest: d, Config: 7, Actor: "pipeline"}); err != nil {
t.Fatal(err)
}
prod := Environment{Name: "prod", Accepts: []string{"staging"}, MinBake: 30 * time.Minute}
if err := CanPromote(staging, prod, fixedClock(base.Add(10*time.Minute))); err == nil {
t.Fatal("promoted before the bake window")
}
if err := CanPromote(staging, prod, fixedClock(base.Add(31*time.Minute))); err != nil {
t.Fatalf("promotion after bake window: %v", err)
}
if err := CanPromote(Environment{Name: "dev"}, prod, fixedClock(base.Add(time.Hour))); err == nil {
t.Fatal("dev promoted directly to prod")
}
t.Logf("staging baked=%s prod_min=%s", 31*time.Minute, prod.MinBake)
}
func TestDriftDetection(t *testing.T) {
declared := DigestOf([]byte("orders image v2"))
old := DigestOf([]byte("orders image v1"))
instances := []Instance{{"orders-2", declared}, {"orders-1", old}, {"orders-3", old}, {"orders-0", declared}}
drift := Drift(declared, instances)
if len(drift) != 2 || drift[0].Instance != "orders-1" || drift[1].Instance != "orders-3" {
t.Fatalf("drift = %+v", drift)
}
t.Logf("instances=%d drifting=%d first=%s", len(instances), len(drift), drift[0].Instance)
}
本机实测
命令与环境:macOS darwin/arm64、Go 1.26.0。先跑 gofmt -l . 与 go vet ./...,两者均无输出,再跑本节相关测试:
$ go test ./internal/platform/release/ -run 'TestDigestAndRegistry|TestDeployRequiresDigest|TestPromotionBakeTime|TestDriftDetection' -v -count=1
=== RUN TestDigestAndRegistry
artifact_test.go:44: digest=sha256:69aaeb8913379f079adbb3a18357fb546ca3e878c6410ffec8497af42d11f20c version=1.4.2
--- PASS: TestDigestAndRegistry (0.00s)
=== RUN TestDeployRequiresDigest
artifact_test.go:62: deployed digest=sha256:69aaeb8913379f079adbb3a18357fb546ca3e878c6410ffec8497af42d11f20c config=7
--- PASS: TestDeployRequiresDigest (0.00s)
=== RUN TestPromotionBakeTime
artifact_test.go:83: staging baked=31m0s prod_min=30m0s
--- PASS: TestPromotionBakeTime (0.00s)
=== RUN TestDriftDetection
artifact_test.go:94: instances=4 drifting=2 first=orders-1
--- PASS: TestDriftDetection (0.00s)
PASS
ok archlab/internal/platform/release 0.252s
读数分三组。摘要组,同一份内容得到同一个 sha256:69aa…f20c,标签、短摘要和大写十六进制都被拒绝,说明“名字”进不了部署入口。注册表组,内容与声明摘要不一致、重复写入同一摘要都被拒绝,回滚只能取回原始字节。晋级组,10 分钟时被挡下,31 分钟时通过,dev 直接晋级 prod 被路径规则拒绝;漂移组在 4 个实例里找到 2 个仍运行旧摘要,并按实例名排序输出。
边界要说明:这是单进程、固定时钟、合成内容的测试,0.252 秒只代表本机这组用例。真实镜像的摘要由仓库计算,本节示例对任意字节做 SHA-256;生产上应确认流水线、仓库和运行时三处使用同一算法与编码,否则“摘要一致”只是字符串一致。
常见错误与边界
- 用标签部署生产。标签可以被重新指向,回滚和复现都会失去确定性;部署入口应在解析阶段就拒绝非摘要引用。
- 只校验元数据,不校验内容。声明了摘要却不重算,等于把摘要当注释;注册表写入时必须对内容做一次哈希。
- 把环境当成命名空间。环境如果只是“一堆可变标签的集合”,就无法回答生产当前运行什么;它应该是“摘要 + 配置版本”的固定声明。
- 回滚只改发布系统记录,不核对实例实际加载值。记录回滚成功但实例仍跑旧摘要,属于静默漂移,必须用运行态上报做交叉验证。
- 烘焙时间按“部署完成”而不是“开始接流量”计算。制品启动成功但尚未经过真实流量,不能算已烘焙。
小结
发布系统真正需要保证的不是“名字对上”,而是“字节对上”。摘要把制品变成内容寻址的对象,注册表保证同一摘要只对应一份内容,环境把“摘要 + 配置版本”固化成可审计的声明,晋级规则再要求这份声明先经过足够长的真实运行。四者合起来,回滚才是一次确定的动作,而不是重新抽奖。
本节实现只覆盖单环境的声明与漂移检测,没有涉及流量比例、指标判定和自动回滚;这些是下一节金丝雀状态机的内容。环境晋级通过后,下一步不是立刻全量,而是让一小部分订单先走新摘要,再用可计算的判据决定继续还是退回。
阅读导航:上一节:8.3 从演练到修复:游戏日、回归与证据留存 · 下一节:9.2 渐进式发布:金丝雀、流量比例与自动回滚 。
继续阅读
探索更多技术文章
浏览归档,发现更多关于系统设计、工具链和工程实践的内容。