一次“加个非空列”引发的下单失败
2026-11-02 02:10,arch-lab 订单服务发布 2.0.0,需要记录 courier_id。DBA 在同一窗口执行 ALTER TABLE orders ADD COLUMN courier_id BIGINT NOT NULL,发布采用滚动方式,旧版本 1.9.8 还有 12 个实例在接单。旧代码的 INSERT 不带这一列,数据库直接拒绝,下单失败持续了 6 分钟。
计划里写着“先加列,再发布”,字面上顺序没错,但加的是 NOT NULL:它对旧代码并不安全。回填没有做,contract 也没有等待旧实例退出。三个独立条件被压成了一个动作,失败几乎是必然的。数据库变更与发布的协同可以对照数据库迁移流水线
和开关与数据兼容
,本节把重点放在“哪个阶段允许做什么、什么时候允许进入下一阶段”。
三个阶段各自回答一个问题
expand 回答“新旧代码能不能同时跑”,migrate 回答“数据有没有补齐”,contract 回答“旧代码是不是已经彻底退出”。每个阶段只做自己那类操作,跨阶段的操作即使语法上能执行,也应该被门禁拒绝。
| 阶段 | 要回答的问题 | 允许的操作 | 禁止的操作 | 退出条件 |
|---|---|---|---|---|
| expand | 新旧代码能否同时运行 | 加可空列、加索引、开始双写 | 加非空约束、删列、改名、收窄类型 | 新旧版本都已部署且能读写 |
| migrate | 历史数据是否补齐 | 回填、校验、修差异 | 删除旧列、改旧列语义 | 回填完成且差异为 0 |
| contract | 旧代码是否已退出 | 加非空约束、删旧列、删旧索引 | 再改新列语义 | 无旧版本流量,且安静期已满 |
改名必须拆成“加新列 + 回填 + 删旧列”,不能一步 RENAME。原因不是数据库不支持,而是滚动发布期间新旧代码会同时存在:旧代码读旧列名,新代码读新列名,任何一步改名都会让其中一侧立即失败。收窄类型同理,旧代码可能写入超出新范围的值,只有等旧写入完全停止并清理完历史数据,才能收紧。
顺序是门禁,不是建议
阶段顺序只是第一道门。真正的运行时门禁有四条:migrate 阶段必须处于双写,否则回填期间的新写入会丢失;contract 阶段必须等所有运行版本都不低于 MinAppVersion;回填必须完成;旧代码最后一次写入之后必须经过 QuietPeriod。四条缺一不可。
版本比较不能用字符串。v1.10.0 按字典序小于 v1.9.0,但按语义版本它更大;如果用字符串比较,控制器会误以为旧版本还在运行,或者更糟,误以为已经退出。安静期也不能从“发布开始”算,而要从“最后一次观察到旧代码写入”算,否则回填刚结束就删列,仍可能撞上迟到的旧请求。
order.Branch 这类状态字段的收窄尤其危险:旧代码可能写入 "cancelled_by_user",新枚举只接受 "CANCELLED"。正确做法是先把旧值映射到新值并双写,确认没有旧值再收紧。跨服务的双写与幂等可以对照双写与分支抽象
。
可运行实现
下面的 release 包与 9.1、9.2 共用。Check 先做结构检查:计划非空、阶段不倒序、expand 操作对旧代码安全、改名必须拆分;再做运行时检查:migrate 需要双写,contract 需要版本下限、回填完成和安静期。所有问题一次性返回,而不是遇到第一条就停,便于发布评审一次看到全部阻塞项。
// file: internal/platform/release/migrate.go
// tier: L1
// verify: go test ./internal/platform/release/ -count=1
package release
import (
"errors"
"fmt"
"strconv"
"strings"
"time"
)
// Phase is one step of an expand-contract migration.
type Phase string
const (
PhaseExpand Phase = "expand"
PhaseMigrate Phase = "migrate"
PhaseContract Phase = "contract"
)
// Op is a schema operation whose code compatibility is known up front.
type Op string
const (
OpAddNullableColumn Op = "add_nullable_column"
OpAddIndex Op = "add_index"
OpBackfill Op = "backfill"
OpSetNotNull Op = "set_not_null"
OpDropColumn Op = "drop_column"
OpRenameColumn Op = "rename_column"
)
// Change is one schema operation tied to a phase.
type Change struct {
ID string
Column string
Phase Phase
Op Op
}
// Plan is the ordered migration plan plus the release constraints.
type Plan struct {
Changes []Change
MinAppVersion string
QuietPeriod time.Duration
}
// State is what the release controller can observe right now.
type State struct {
Now time.Time
AppVersions []string
Backfilled map[string]time.Time // backfill change ID -> completion
StoppedWriting map[string]time.Time // column -> last old-code write
DualWrite bool
}
func phaseRank(p Phase) int {
switch p {
case PhaseExpand:
return 0
case PhaseMigrate:
return 1
case PhaseContract:
return 2
}
return -1
}
func oldCodeSafe(op Op) bool {
return op == OpAddNullableColumn || op == OpAddIndex || op == OpBackfill
}
// Check applies structural rules first, then the runtime gates: dual write,
// old readers, completed backfill and the quiet period after last old write.
func Check(p Plan, st State) []error {
var errs []error
if len(p.Changes) == 0 {
return []error{errors.New("release: migration plan is empty")}
}
if p.MinAppVersion == "" {
errs = append(errs, errors.New("release: plan has no minimum app version"))
}
if p.QuietPeriod <= 0 {
errs = append(errs, errors.New("release: plan has no quiet period"))
}
if st.Now.IsZero() {
errs = append(errs, errors.New("release: state has no clock"))
}
seen := map[string]bool{}
rank := -1
for i, ch := range p.Changes {
where := fmt.Sprintf("change[%d] %s", i, ch.ID)
if ch.ID == "" {
errs = append(errs, fmt.Errorf("change[%d]: id is empty", i))
} else if seen[ch.ID] {
errs = append(errs, fmt.Errorf("%s: duplicate id", where))
}
seen[ch.ID] = true
if ch.Column == "" {
errs = append(errs, fmt.Errorf("%s: column is empty", where))
}
r := phaseRank(ch.Phase)
if r < 0 {
errs = append(errs, fmt.Errorf("%s: unknown phase %q", where, ch.Phase))
continue
}
if r < rank {
errs = append(errs, fmt.Errorf("%s: phase %s after a later phase", where, ch.Phase))
}
rank = r
if ch.Op == OpRenameColumn {
errs = append(errs, fmt.Errorf("%s: rename must be add+backfill+drop", where))
}
if ch.Phase == PhaseExpand && !oldCodeSafe(ch.Op) {
errs = append(errs, fmt.Errorf("%s: %s is not safe for old code", where, ch.Op))
}
}
for _, ch := range p.Changes {
if ch.Phase == PhaseMigrate && !st.DualWrite {
errs = append(errs, fmt.Errorf("%s: backfill needs dual write", ch.ID))
}
if ch.Phase != PhaseContract {
continue
}
if !versionsAtLeast(st.AppVersions, p.MinAppVersion) {
errs = append(errs, fmt.Errorf("%s: old code still serves: %v < %s", ch.ID, st.AppVersions, p.MinAppVersion))
}
if missing := missingBackfills(p, st); len(missing) > 0 {
errs = append(errs, fmt.Errorf("%s: contract waits for backfill %v", ch.ID, missing))
}
stopped, ok := st.StoppedWriting[ch.Column]
if !ok {
errs = append(errs, fmt.Errorf("%s: no stop-writing record for %s", ch.ID, ch.Column))
} else if quiet := st.Now.Sub(stopped); quiet < p.QuietPeriod {
errs = append(errs, fmt.Errorf("%s: quiet %s < %s", ch.ID, quiet, p.QuietPeriod))
}
}
return errs
}
func missingBackfills(p Plan, st State) []string {
var missing []string
for _, ch := range p.Changes {
if ch.Phase != PhaseMigrate {
continue
}
if _, ok := st.Backfilled[ch.ID]; !ok {
missing = append(missing, ch.ID)
}
}
return missing
}
func versionsAtLeast(have []string, want string) bool {
for _, v := range have {
if compareVersion(v, want) < 0 {
return false
}
}
return true
}
// compareVersion compares dotted versions numerically: v1.10.0 > v1.9.0.
func compareVersion(a, b string) int {
as := strings.Split(strings.TrimPrefix(a, "v"), ".")
bs := strings.Split(strings.TrimPrefix(b, "v"), ".")
n := len(as)
if len(bs) > n {
n = len(bs)
}
for i := 0; i < n; i++ {
x, y := 0, 0
if i < len(as) {
x, _ = strconv.Atoi(as[i])
}
if i < len(bs) {
y, _ = strconv.Atoi(bs[i])
}
if x != y {
if x < y {
return -1
}
return 1
}
}
return 0
}
两个细节值得单独说明。第一,oldCodeSafe 只放行加可空列、加索引和回填;set_not_null、drop_column、rename_column 都被归到 contract 或直接拒绝。第二,版本比较逐段转整数,缺位补 0,所以 v1.10.0 大于 v1.9.0、2 等于 v2.0.0;这比字符串比较更接近发布系统实际使用的版本语义。
测试块覆盖合法计划、阶段倒序、expand 中做非空约束、改名、旧版本仍在运行、回填缺失、安静期不足和双写关闭:
// file: internal/platform/release/migrate_test.go
// tier: L1
// verify: go test ./internal/platform/release/ -count=1
package release
import (
"strings"
"testing"
"time"
)
func validPlan() Plan {
return Plan{
Changes: []Change{
{ID: "E1", Column: "courier_id", Phase: PhaseExpand, Op: OpAddNullableColumn},
{ID: "M1", Column: "courier_id", Phase: PhaseMigrate, Op: OpBackfill},
{ID: "C1", Column: "legacy_status", Phase: PhaseContract, Op: OpDropColumn},
},
MinAppVersion: "v1.10.0",
QuietPeriod: 72 * time.Hour,
}
}
func readyState(now time.Time) State {
return State{
Now: now,
AppVersions: []string{"v1.10.0", "v1.10.1"},
Backfilled: map[string]time.Time{"M1": now.Add(-80 * time.Hour)},
StoppedWriting: map[string]time.Time{"legacy_status": now.Add(-73 * time.Hour)},
DualWrite: true,
}
}
func TestValidPlanPasses(t *testing.T) {
now := time.Date(2026, 10, 31, 8, 20, 0, 0, time.UTC)
if errs := Check(validPlan(), readyState(now)); len(errs) != 0 {
t.Fatalf("ready plan rejected: %v", errs)
}
t.Logf("changes=%d min_app=%s quiet=%s", len(validPlan().Changes), validPlan().MinAppVersion, validPlan().QuietPeriod)
for _, tc := range []struct {
a, b string
want int
}{{"v1.10.0", "v1.9.0", 1}, {"1.9.0", "v1.10.0", -1}} {
if got := compareVersion(tc.a, tc.b); got != tc.want {
t.Fatalf("compare(%s,%s)=%d", tc.a, tc.b, got)
}
}
}
func TestPlanValidationRejectsUnsafeOrderAndOps(t *testing.T) {
cases := map[string]Plan{
"contract_first": {Changes: []Change{
{ID: "C1", Column: "a", Phase: PhaseContract, Op: OpDropColumn},
{ID: "E1", Column: "b", Phase: PhaseExpand, Op: OpAddNullableColumn},
}, MinAppVersion: "v1.0.0", QuietPeriod: time.Hour},
"not_null_in_expand": {Changes: []Change{
{ID: "E1", Column: "a", Phase: PhaseExpand, Op: OpSetNotNull},
}, MinAppVersion: "v1.0.0", QuietPeriod: time.Hour},
"rename": {Changes: []Change{
{ID: "E1", Column: "a", Phase: PhaseExpand, Op: OpRenameColumn},
}, MinAppVersion: "v1.0.0", QuietPeriod: time.Hour},
"empty": {},
}
now := time.Date(2026, 10, 31, 8, 0, 0, 0, time.UTC)
for name, p := range cases {
if errs := Check(p, State{Now: now}); len(errs) == 0 {
t.Fatalf("%s accepted", name)
} else {
t.Logf("%s: %v", name, errs[0])
}
}
}
func TestContractRuntimeGates(t *testing.T) {
now := time.Date(2026, 10, 31, 8, 20, 0, 0, time.UTC)
cases := []struct {
name string
mutate func(*State)
expect string
}{
{"old_app", func(s *State) { s.AppVersions = []string{"v1.9.0"} }, "old code still serves"},
{"no_dual_write", func(s *State) { s.DualWrite = false }, "backfill needs dual write"},
{"not_backfilled", func(s *State) { delete(s.Backfilled, "M1") }, "contract waits for backfill"},
{"quiet_too_short", func(s *State) { s.StoppedWriting["legacy_status"] = now.Add(-71 * time.Hour) }, "quiet"},
{"no_stop_record", func(s *State) { delete(s.StoppedWriting, "legacy_status") }, "no stop-writing record"},
}
for _, tc := range cases {
st := readyState(now)
tc.mutate(&st)
errs := Check(validPlan(), st)
found := false
for _, err := range errs {
found = found || strings.Contains(err.Error(), tc.expect)
}
if !found {
t.Fatalf("%s: errors %v missing %q", tc.name, errs, tc.expect)
}
t.Logf("%s: %v", tc.name, errs[0])
}
}
本机实测
环境为 macOS darwin/arm64、Go 1.26.0;gofmt -l . 与 go vet ./... 均无输出。运行本节相关测试:
$ go test ./internal/platform/release/ -run 'TestValidPlanPasses|TestPlanValidationRejectsUnsafeOrderAndOps|TestContractRuntimeGates' -v -count=1
=== RUN TestValidPlanPasses
migrate_test.go:39: changes=3 min_app=v1.10.0 quiet=72h0m0s
--- PASS: TestValidPlanPasses (0.00s)
=== RUN TestPlanValidationRejectsUnsafeOrderAndOps
migrate_test.go:69: empty: release: migration plan is empty
migrate_test.go:69: contract_first: change[1] E1: phase expand after a later phase
migrate_test.go:69: not_null_in_expand: change[0] E1: set_not_null is not safe for old code
migrate_test.go:69: rename: change[0] E1: rename must be add+backfill+drop
--- PASS: TestPlanValidationRejectsUnsafeOrderAndOps (0.00s)
=== RUN TestContractRuntimeGates
migrate_test.go:98: old_app: C1: old code still serves: [v1.9.0] < v1.10.0
migrate_test.go:98: no_dual_write: M1: backfill needs dual write
migrate_test.go:98: not_backfilled: C1: contract waits for backfill [M1]
migrate_test.go:98: quiet_too_short: C1: quiet 71h0m0s < 72h0m0s
migrate_test.go:98: no_stop_record: C1: no stop-writing record for legacy_status
--- PASS: TestContractRuntimeGates (0.00s)
PASS
ok archlab/internal/platform/release 0.274s
读数分三组。合法计划有三个变更、三个版本约束,Check 返回空错误,说明门禁全部满足;compareVersion 同时验证了 v1.10.0 > v1.9.0,字符串比较会在这里给出相反结论。结构组里,contract 排在 expand 之前被 phase expand after a later phase 拦住,expand 中做 set_not_null 被“不安全”拦住,改名被要求拆成三步。运行时组里,旧版本仍在服务、双写关闭、回填缺失、安静期只有 71 小时、没有停止写入记录,五种情况分别给出对应原因。
边界如下:这是固定时钟、手工构造状态的单元测试,0.274 秒只代表本机。真实环境还要处理 DDL 锁、在线加索引的执行时间、回填批次对主库的压力、复制延迟导致的读旧值,以及跨服务同时写同一列的顺序问题。Check 只判断“计划与观测是否满足门禁”,不执行 DDL,也不替代数据库侧的锁与超时保护。
常见错误与边界
- 在 expand 阶段加非空约束、删列或改名。旧代码仍在运行时,这些操作会让旧写入或旧读取立即失败;expand 只做旧代码能理解的加法。
- 用字符串比较版本。
v1.10.0按字典序小于v1.9.0,会把“旧版本已退出”判成“仍在线”,或者反过来提前放行 contract。 - 回填没完成就进入 contract。回填是唯一能把历史行补齐的动作,缺了它,新代码读到 NULL 或旧值,非空约束也无法加上。
- 安静期从发布开始算。旧实例可能在新版本上线后仍处理迟到请求,安静期必须从最后一次观察到旧代码写入算起。
- 只改 schema 不改双写。回填期间新写入仍走旧列,补完历史数据后新数据又缺,差异永远清不干净;双写与校验要一起上线。
小结
expand-contract 的价值不在于把一次 DDL 拆成三次,而在于让每一步都有明确的兼容对象和退出条件。expand 保证新旧代码能同时跑,migrate 保证历史数据补齐,contract 保证旧代码已经退出;顺序、版本下限、回填完成和安静期是四条独立门禁,任何一条缺失都不应该放行。
本节实现只检查计划与观测,不执行迁移,也不处理锁与复制延迟;这些需要数据库侧的工具和真实流量验证。三节合起来,发布工程回答的是同一个问题:这次变更引用的是哪份确定的内容,它在什么证据下被允许继续,以及当旧代码、旧数据和新 schema 同时存在时,谁必须先退场。
阅读导航:上一节:9.2 渐进式发布:金丝雀、流量比例与自动回滚 · 下一节:10.1 租户模型:隔离级别与数据分区 。
继续阅读
探索更多技术文章
浏览归档,发现更多关于系统设计、工具链和工程实践的内容。