《后端架构设计·演进与治理》9.1 制品、环境与不可变部署

从 arch-lab 订单服务按标签回滚却加载了另一份字节的事故切入,拆开标签、制品摘要与环境声明的职责差异,给出不可变注册表、按摘要部署、晋级烘焙时间与运行态漂移检测的 Go 实现,并附本机测试输出、可复现命令、四条判据、边界、取舍与回退。

一次“回滚”回到了另一个版本

2026-10-30 21:40,arch-lab 订单服务 1.4.2 因空指针触发回滚。发布系统按 orders:1.4.2 重新拉取镜像,三台实例启动后错误率没有下降。排查发现当天下午构建缓存失效,1.4.2 这个标签被重新构建并覆盖,回滚拉到的不是上周那份字节;审计日志只记了标签,没人能回答“上周生产实际运行的到底是哪份内容”。

这不是回滚脚本写错,而是发布系统把“名字”当成了“身份”。只要标签可以被覆盖,回滚、复现和审计就都失去基准。本节把制品、环境与部署拆成三个可验证的概念,并给出一个只依赖标准库的实现。回滚窗口本身的设计可以对照回滚与兼容窗口 ,制品来源与证明链路可以对照制品仓库与来源证明 ;本节只处理“部署引用什么”这一层。

标签是名字,摘要才是身份

标签是给人看的,它可以指向任何一次构建;摘要是由内容算出来的,同样的字节永远得到同样的值。把两者混用,发布记录里写的是标签,实际运行的却是另一份内容,回滚就变成了重新抽奖。

引用方式可被覆盖回滚语义审计能回答适用位置
可变标签 orders:latest是重新拉取当前指向,可能已变只知名字本地试验
构建号 orders#1841通常不可变需依赖构建系统保留记录构建记录流水线内部
内容摘要 sha256:...否精确回到同一份字节内容本身生产部署与回滚

摘要还有第二个作用:把“校验”从信任变成计算。注册表在写入时重新计算内容摘要,与声明的摘要不一致就拒绝;同一个摘要第二次写入也拒绝,因为不可变意味着不能悄悄替换。这样即使构建系统、对象存储或传输链路某一环出错,错误也会在进入环境之前暴露。

环境是“摘要 + 配置版本”的声明

环境不应该是一个可以随意打标签的命名空间,而应该是一条声明:生产当前运行摘要 D、配置版本 C,由谁在什么时间部署。部署动作只接受摘要,配置版本必须为正数,操作者必须留下身份;晋级则要求来源环境合法、制品已经烘焙足够长时间。

这条声明还带来一个可观测判据:实例上报自己实际加载的摘要,与环境声明的摘要比对。不一致的实例就是漂移,可能来自滚动发布未完成、手工热更新或节点回退。漂移检测不需要知道发布系统的内部状态,只需要“声明”和“实际”两个值。

观测判据动作退出条件
部署请求带标签不是 sha256: 摘要直接拒绝流水线改为解析摘要
内容与声明摘要不符重算结果不同拒绝入库并告警重新构建并生成新摘要
晋级早于烘焙窗口小于目标环境 MinBake保持候选,不切流量达到窗口且指标平稳
运行实例摘要漂移任一实例不等于声明停止晋级,排查实例全部实例回到声明摘要

可运行实现

下面的 release 包只依赖标准库。Registry 以摘要为键保存制品并校验内容;Environment.Deploy 只接受摘要;CanPromote 检查晋级路径与烘焙时间;Drift 比对声明摘要与实例上报值。本章三节共用同一个包,后续两节会继续加入金丝雀状态机与 expand-contract 检查。

// file: internal/platform/release/artifact.go
// tier: L1
// verify: go test ./internal/platform/release/ -count=1
package release

import (
	"crypto/sha256"
	"encoding/hex"
	"errors"
	"fmt"
	"slices"
	"sort"
	"strings"
	"time"
)

// Clock lets deployment checks run against a fixed timeline in tests.
type Clock func() time.Time

// Digest is the content address of one artifact: "sha256:" plus 64 hex chars.
type Digest string

// DigestOf returns the content address of b.
func DigestOf(b []byte) Digest {
	sum := sha256.Sum256(b)
	return Digest("sha256:" + hex.EncodeToString(sum[:]))
}

// ParseDigest accepts only a lowercase sha256 content address, never a tag.
func ParseDigest(s string) (Digest, error) {
	body, ok := strings.CutPrefix(s, "sha256:")
	if !ok || len(body) != 64 {
		return "", fmt.Errorf("release: %q is not a 64-char digest", s)
	}
	for _, r := range body {
		if (r < '0' || r > '9') && (r < 'a' || r > 'f') {
			return "", errors.New("release: digest is not lowercase hex")
		}
	}
	return Digest(s), nil
}

// Artifact is an immutable build output.
type Artifact struct {
	Name, Version string
	Digest        Digest
	Size          int64
	BuiltAt       time.Time
}

// Registry stores artifacts by content address. A digest is never overwritten,
// so a rollback returns exactly the bytes that ran before.
type Registry struct{ artifacts map[Digest]Artifact }

// NewRegistry returns an empty registry.
func NewRegistry() *Registry { return &Registry{artifacts: map[Digest]Artifact{}} }

// Push verifies content and metadata before recording the artifact.
func (r *Registry) Push(a Artifact, content []byte) error {
	if a.Name == "" || a.Version == "" || a.Size <= 0 || a.BuiltAt.IsZero() {
		return errors.New("release: artifact metadata is incomplete")
	}
	if _, err := ParseDigest(string(a.Digest)); err != nil {
		return err
	}
	if got := DigestOf(content); got != a.Digest {
		return fmt.Errorf("release: content hashes to %s, declared %s", got, a.Digest)
	}
	if a.Size != int64(len(content)) {
		return fmt.Errorf("release: size %d, content %d", a.Size, len(content))
	}
	if _, exists := r.artifacts[a.Digest]; exists {
		return fmt.Errorf("release: digest %s already stored", a.Digest)
	}
	r.artifacts[a.Digest] = a
	return nil
}

// Resolve returns the stored artifact for a digest.
func (r *Registry) Resolve(d Digest) (Artifact, error) {
	a, ok := r.artifacts[d]
	if !ok {
		return Artifact{}, fmt.Errorf("release: digest %s not found", d)
	}
	return a, nil
}

// Deployment is one pinned release in one environment.
type Deployment struct {
	Environment, Version, Actor string
	Digest, Previous            Digest
	Config                      uint64
	At                          time.Time
}

// DeployRequest is what a pipeline may ask an environment to run.
type DeployRequest struct {
	Digest Digest
	Config uint64
	Actor  string
}

// Environment holds the current deployment.
type Environment struct {
	Name    string
	Accepts []string
	MinBake time.Duration
	Current *Deployment
}

// Deploy accepts a registry digest, never a tag such as "orders:latest".
func (e *Environment) Deploy(clock Clock, reg *Registry, req DeployRequest) (Deployment, error) {
	if e.Name == "" {
		return Deployment{}, errors.New("release: environment name is empty")
	}
	if _, err := ParseDigest(string(req.Digest)); err != nil {
		return Deployment{}, err
	}
	art, err := reg.Resolve(req.Digest)
	if err != nil {
		return Deployment{}, err
	}
	if req.Config == 0 || req.Actor == "" {
		return Deployment{}, errors.New("release: config and actor are required")
	}
	d := Deployment{Environment: e.Name, Digest: art.Digest, Version: art.Version,
		Config: req.Config, Actor: req.Actor, At: clock()}
	if e.Current != nil {
		d.Previous = e.Current.Digest
	}
	e.Current = &d
	return d, nil
}

// CanPromote checks the allowed path and the bake time.
func CanPromote(source, target Environment, clock Clock) error {
	if !slices.Contains(target.Accepts, source.Name) {
		return fmt.Errorf("release: %s does not accept promotion from %s", target.Name, source.Name)
	}
	if source.Current == nil {
		return fmt.Errorf("release: %s has no deployment", source.Name)
	}
	if elapsed := clock().Sub(source.Current.At); elapsed < target.MinBake {
		return fmt.Errorf("release: baked %s, want %s", elapsed, target.MinBake)
	}
	return nil
}

// Instance is one process reporting the digest it actually loaded.
type Instance struct {
	ID     string
	Digest Digest
}

// DriftItem is one instance running something other than the declared digest.
type DriftItem struct {
	Instance          string
	Running, Expected Digest
}

// Drift returns instances not running the declared digest, sorted by ID.
func Drift(expected Digest, instances []Instance) []DriftItem {
	var out []DriftItem
	for _, in := range instances {
		if in.Digest != expected {
			out = append(out, DriftItem{in.ID, in.Digest, expected})
		}
	}
	sort.Slice(out, func(i, j int) bool { return out[i].Instance < out[j].Instance })
	return out
}

实现里有三个刻意选择。第一,注册表以摘要为键,重复写入直接报错,避免“同摘要不同内容”这种自相矛盾的状态。第二,部署请求先过 ParseDigest,标签在进入环境之前就被挡住,而不是等到拉取时才发现。第三,晋级同时检查路径和烘焙时间,因为“制品在测试环境跑过”和“制品在测试环境跑够久”不是一回事。

测试块覆盖摘要校验、注册表拒绝内容不符与重复写入、部署拒绝标签、晋级窗口以及漂移检测:

// file: internal/platform/release/artifact_test.go
// tier: L1
// verify: go test ./internal/platform/release/ -count=1
package release

import (
	"strings"
	"testing"
	"time"
)

func fixedClock(t time.Time) Clock { return func() time.Time { return t } }

func registryWith(t *testing.T, content []byte, version string) (*Registry, Digest) {
	t.Helper()
	d := DigestOf(content)
	reg := NewRegistry()
	err := reg.Push(Artifact{Name: "orders", Version: version, Digest: d,
		Size: int64(len(content)), BuiltAt: time.Unix(0, 0).UTC()}, content)
	if err != nil {
		t.Fatalf("push: %v", err)
	}
	return reg, d
}

func TestDigestAndRegistry(t *testing.T) {
	content := []byte("orders image v1")
	reg, d := registryWith(t, content, "1.4.2")
	if _, err := ParseDigest(string(d)); err != nil {
		t.Fatalf("DigestOf produced invalid digest: %v", err)
	}
	for _, bad := range []string{"orders:latest", "sha256:abc", "sha256:" + strings.Repeat("A", 64)} {
		if _, err := ParseDigest(bad); err == nil {
			t.Fatalf("accepted %q", bad)
		}
	}
	art, _ := reg.Resolve(d)
	if err := reg.Push(art, []byte("orders image v2")); err == nil {
		t.Fatal("content/digest mismatch accepted")
	}
	if err := reg.Push(art, content); err == nil {
		t.Fatal("duplicate digest overwrote an immutable artifact")
	}
	t.Logf("digest=%s version=%s", d, art.Version)
}

func TestDeployRequiresDigest(t *testing.T) {
	content := []byte("orders image v1")
	reg, d := registryWith(t, content, "1.4.2")
	now := time.Date(2026, 10, 31, 0, 0, 0, 0, time.UTC)
	env := Environment{Name: "prod"}
	if _, err := env.Deploy(fixedClock(now), reg, DeployRequest{Digest: "orders:latest", Config: 7, Actor: "pipeline"}); err == nil {
		t.Fatal("tag accepted as a deployable reference")
	}
	if _, err := env.Deploy(fixedClock(now), reg, DeployRequest{Digest: d, Config: 0, Actor: "pipeline"}); err == nil {
		t.Fatal("zero config version accepted")
	}
	dep, err := env.Deploy(fixedClock(now), reg, DeployRequest{Digest: d, Config: 7, Actor: "pipeline"})
	if err != nil || dep.Version != "1.4.2" || dep.Config != 7 {
		t.Fatalf("Deploy = %+v, %v", dep, err)
	}
	t.Logf("deployed digest=%s config=%d", dep.Digest, dep.Config)
}

func TestPromotionBakeTime(t *testing.T) {
	content := []byte("orders image v1")
	reg, d := registryWith(t, content, "1.4.2")
	base := time.Date(2026, 10, 31, 0, 0, 0, 0, time.UTC)
	staging := Environment{Name: "staging"}
	if _, err := staging.Deploy(fixedClock(base), reg, DeployRequest{Digest: d, Config: 7, Actor: "pipeline"}); err != nil {
		t.Fatal(err)
	}
	prod := Environment{Name: "prod", Accepts: []string{"staging"}, MinBake: 30 * time.Minute}
	if err := CanPromote(staging, prod, fixedClock(base.Add(10*time.Minute))); err == nil {
		t.Fatal("promoted before the bake window")
	}
	if err := CanPromote(staging, prod, fixedClock(base.Add(31*time.Minute))); err != nil {
		t.Fatalf("promotion after bake window: %v", err)
	}
	if err := CanPromote(Environment{Name: "dev"}, prod, fixedClock(base.Add(time.Hour))); err == nil {
		t.Fatal("dev promoted directly to prod")
	}
	t.Logf("staging baked=%s prod_min=%s", 31*time.Minute, prod.MinBake)
}

func TestDriftDetection(t *testing.T) {
	declared := DigestOf([]byte("orders image v2"))
	old := DigestOf([]byte("orders image v1"))
	instances := []Instance{{"orders-2", declared}, {"orders-1", old}, {"orders-3", old}, {"orders-0", declared}}
	drift := Drift(declared, instances)
	if len(drift) != 2 || drift[0].Instance != "orders-1" || drift[1].Instance != "orders-3" {
		t.Fatalf("drift = %+v", drift)
	}
	t.Logf("instances=%d drifting=%d first=%s", len(instances), len(drift), drift[0].Instance)
}

本机实测

命令与环境:macOS darwin/arm64、Go 1.26.0。先跑 gofmt -l . 与 go vet ./...,两者均无输出,再跑本节相关测试:

$ go test ./internal/platform/release/ -run 'TestDigestAndRegistry|TestDeployRequiresDigest|TestPromotionBakeTime|TestDriftDetection' -v -count=1
=== RUN   TestDigestAndRegistry
    artifact_test.go:44: digest=sha256:69aaeb8913379f079adbb3a18357fb546ca3e878c6410ffec8497af42d11f20c version=1.4.2
--- PASS: TestDigestAndRegistry (0.00s)
=== RUN   TestDeployRequiresDigest
    artifact_test.go:62: deployed digest=sha256:69aaeb8913379f079adbb3a18357fb546ca3e878c6410ffec8497af42d11f20c config=7
--- PASS: TestDeployRequiresDigest (0.00s)
=== RUN   TestPromotionBakeTime
    artifact_test.go:83: staging baked=31m0s prod_min=30m0s
--- PASS: TestPromotionBakeTime (0.00s)
=== RUN   TestDriftDetection
    artifact_test.go:94: instances=4 drifting=2 first=orders-1
--- PASS: TestDriftDetection (0.00s)
PASS
ok  	archlab/internal/platform/release	0.252s

读数分三组。摘要组,同一份内容得到同一个 sha256:69aa…f20c,标签、短摘要和大写十六进制都被拒绝,说明“名字”进不了部署入口。注册表组,内容与声明摘要不一致、重复写入同一摘要都被拒绝,回滚只能取回原始字节。晋级组,10 分钟时被挡下,31 分钟时通过,dev 直接晋级 prod 被路径规则拒绝;漂移组在 4 个实例里找到 2 个仍运行旧摘要,并按实例名排序输出。

边界要说明:这是单进程、固定时钟、合成内容的测试,0.252 秒只代表本机这组用例。真实镜像的摘要由仓库计算,本节示例对任意字节做 SHA-256;生产上应确认流水线、仓库和运行时三处使用同一算法与编码,否则“摘要一致”只是字符串一致。

常见错误与边界

  • 用标签部署生产。标签可以被重新指向,回滚和复现都会失去确定性;部署入口应在解析阶段就拒绝非摘要引用。
  • 只校验元数据,不校验内容。声明了摘要却不重算,等于把摘要当注释;注册表写入时必须对内容做一次哈希。
  • 把环境当成命名空间。环境如果只是“一堆可变标签的集合”,就无法回答生产当前运行什么;它应该是“摘要 + 配置版本”的固定声明。
  • 回滚只改发布系统记录,不核对实例实际加载值。记录回滚成功但实例仍跑旧摘要,属于静默漂移,必须用运行态上报做交叉验证。
  • 烘焙时间按“部署完成”而不是“开始接流量”计算。制品启动成功但尚未经过真实流量,不能算已烘焙。

小结

发布系统真正需要保证的不是“名字对上”,而是“字节对上”。摘要把制品变成内容寻址的对象,注册表保证同一摘要只对应一份内容,环境把“摘要 + 配置版本”固化成可审计的声明,晋级规则再要求这份声明先经过足够长的真实运行。四者合起来,回滚才是一次确定的动作,而不是重新抽奖。

本节实现只覆盖单环境的声明与漂移检测,没有涉及流量比例、指标判定和自动回滚;这些是下一节金丝雀状态机的内容。环境晋级通过后,下一步不是立刻全量,而是让一小部分订单先走新摘要,再用可计算的判据决定继续还是退回。

阅读导航:上一节:8.3 从演练到修复:游戏日、回归与证据留存 · 下一节:9.2 渐进式发布:金丝雀、流量比例与自动回滚 。

继续阅读

探索更多技术文章

浏览归档,发现更多关于系统设计、工具链和工程实践的内容。

全部文章 返回首页

「books」更多文章

  1. 《互联网产品的 AI 架构·平台化与治理》附录 C 平台化评审清单
  2. 《互联网产品的 AI 架构·平台化与治理》附录 B 能力目录与配额字段速查
  3. 《互联网产品的 AI 架构·平台化与治理》附录 A 治理角色与职责速查