《MMO 服务器开发》7.2 命令、补偿与重试

权限决定了谁能下令,但一条 GM 命令从控制台下发到真正落地,中间要穿过网络重发、进程崩溃、发奖半途而废这些失败点。这一节用命令指纹与票据做幂等,让同一条命令重复执行只生效一次;用补偿把发奖失败的钱退回去,守住账目守恒;用重试加去重让暂时性失败自动恢复,并给出可跑的幂等与补偿断言。

7.2 命令、补偿与重试

上一节 7.1 权限与审计 解决了「谁能下令」和「下过什么令」的问题。但一条命令被批准、被下发之后,故事才刚开始:网关和世界节点之间可能超时,客户端可能因为没收到回包而重发,双人审批通过后操作员可能手抖又点了一次,世界节点在处理到一半时可能崩掉重启。这些都不是异常,而是分布式系统里的常态。

如果对这些常态视而不见,后果很具体:一次「发 100 元宝」的重发会变成发 200,一次「发道具」失败会让玩家钱扣了道具却没到。这一节把 GM 命令总线要解决的三件事讲清楚——幂等(同一条命令只生效一次)、补偿(失败的操作怎么补回来)、重试与去重(暂时性失败自动恢复但不能重复生效),并落成可跑的 Go 代码。

幂等:为什么「重复执行」是常态

先接受一个前提:在分布式环境里,一条命令被投递多次是默认情况,投递一次才是幸运。所以幂等不能靠「客户端别重发」来保证,只能靠服务端自己认得「这条命令我处理过了」。

识别一条命令需要两个东西。一个是票据(Ticket)——由控制台在创建命令时生成,同一条逻辑命令的所有重发共享同一个票据。另一个是指纹(Fingerprint)——把命令号、操作人、目标玩家、票据和参数一起哈希出来的稳定标识。为什么票据之外还要指纹?因为票据可能被复用,把参数也纳入哈希,才能保证「改了参数的另一个命令」不被误判成同一条;参数先排序再哈希,否则 {item:5, count:2} 和 {count:2, item:5} 会算出两个指纹。

有了指纹,CommandStore.Execute 的逻辑就一句话:这个指纹我见过吗?见过就直接返回上次的结果,没见过去真正执行。 关键细节是:只有执行成功才把指纹写进 store;执行失败不写,于是重试时它会被当成一条全新的命令重新执行。这也解释了为什么「失败」和「成功」在幂等语义里必须区分对待。

补偿:发奖失败怎么补

幂等解决的是「别重复」,但解决不了「做了一半」。最典型的场景是发奖:运营给玩家发一份礼包,系统先给玩家加上元宝,再去投递道具邮件——结果邮件服务挂了。这时候钱已经加了,道具没到。如果什么都不做,玩家白拿元宝;如果粗暴回滚整个操作,可能把玩家本来正常的状态也搅乱。

补偿(compensation)的思路是:为每一个「已经发生的副作用」准备一个反向操作。这里副作用是「加了元宝」,反向操作就是「退回元宝」。DeliverGrant 把这两步写在一起——先加钱,再尝试发货;发货失败就把钱退回去,返回一个带原因的错误。这样无论成功还是失败,账目都是守恒的。

什么叫账目守恒?Ledger 维护三个量:每个玩家的余额、累计发放 granted、累计退回 refunded,守恒式就是 granted - refunded == 所有玩家余额之和。每一条发奖路径都必须守住它:漏了一次退款,或者重试时多发了一次,这个等式立刻不成立——这正是它作为断言的价值。

重试与去重

有了幂等和补偿,「重试」就变得安全了。CommandStore.Retry 把执行包进一个最多 N 次的循环:只要某次成功就返回,全部失败才报错。因为 Execute 自带去重,重试在成功之后再被调用一次也不会重复生效——它只会命中 store 里已有的指纹,返回 Applied=false。没有去重的重试是灾难,没有重试的去重又挡不住暂时性抖动,两者必须成对出现。

下面这段是 L1,go test 可以直接跑。它把命令号、指纹、幂等存储、账本与补偿装进同一个文件 internal/gm/command.go:

// file: internal/gm/command.go
// tier: L1
// verify: go test ./internal/gm/
package gm

import (
	"crypto/sha256"
	"encoding/hex"
	"fmt"
	"sort"
)

// Operations-domain command numbers, shared with the wire protocol of volume
// one. The fixed header is unchanged: length(4, BE) | cmd(2, BE) | seq(4, BE) |
// flags(1), HeaderSize 11, MaxPayload 1<<20.
const (
	CmdGmCommandReq  uint16 = 0x1101
	CmdGmCommandResp uint16 = 0x1102
	CmdGmAuditNotify uint16 = 0x1103
)

// PlayerID is the cross-volume alias for a globally unique character id.
type PlayerID uint64

// GmCommand is one operator-issued action. Ticket is the idempotency key: two
// requests that carry the same ticket are the same command, however many times
// the client retries them.
type GmCommand struct {
	CmdID    string
	Operator string
	Target   PlayerID
	Ticket   string
	Args     map[string]int64
}

// Fingerprint is the stable identity of a command. It hashes the command id,
// operator, target, ticket and a sorted view of the arguments, so retries of
// the same logical command collide on purpose while a genuinely different
// command never does.
func (c GmCommand) Fingerprint() string {
	keys := make([]string, 0, len(c.Args))
	for k := range c.Args {
		keys = append(keys, k)
	}
	sort.Strings(keys)
	h := sha256.New()
	fmt.Fprintf(h, "%s|%s|%d|%s", c.CmdID, c.Operator, c.Target, c.Ticket)
	for _, k := range keys {
		fmt.Fprintf(h, "|%s=%d", k, c.Args[k])
	}
	return hex.EncodeToString(h.Sum(nil))
}

// Result records what a command did, so a duplicate can return the same answer
// without re-applying it.
type Result struct {
	Fingerprint string
	Applied     bool
	Detail      string
}

// CommandStore dedups commands by fingerprint. It is the memory of "I already
// did this", and it is the reason a retry is safe.
type CommandStore struct{ done map[string]Result }

// NewCommandStore returns an empty store.
func NewCommandStore() *CommandStore { return &CommandStore{done: map[string]Result{}} }

// Execute applies a command exactly once. The second call with the same
// fingerprint returns the stored result with Applied=false and does not call
// apply again.
func (s *CommandStore) Execute(c GmCommand, apply func(GmCommand) error) (Result, error) {
	fp := c.Fingerprint()
	if prev, ok := s.done[fp]; ok {
		prev.Applied = false
		return prev, nil
	}
	if err := apply(c); err != nil {
		return Result{}, err
	}
	r := Result{Fingerprint: fp, Applied: true, Detail: c.CmdID}
	s.done[fp] = r
	return r, nil
}

// Retry runs apply up to maxAttempts times, stopping at the first success.
// Because Execute dedups by fingerprint, a retry after a success is a no-op
// rather than a second grant; a retry after a failure starts clean.
func (s *CommandStore) Retry(c GmCommand, maxAttempts int, apply func(GmCommand) error) (Result, int, error) {
	for attempt := 1; attempt <= maxAttempts; attempt++ {
		if r, err := s.Execute(c, apply); err == nil {
			return r, attempt, nil
		}
	}
	return Result{}, maxAttempts, fmt.Errorf("gm: command %s failed after %d attempts", c.CmdID, maxAttempts)
}

// Ledger is the currency balance of the world. Grant, compensation and retry
// all move the same two counters, so a conservation check is meaningful: the
// sum of every balance plus the total refunded must equal the total granted.
type Ledger struct {
	balances map[PlayerID]int64
	granted  int64
	refunded int64
}

// NewLedger returns an empty ledger.
func NewLedger() *Ledger { return &Ledger{balances: map[PlayerID]int64{}} }

// Balance returns a player's current currency.
func (l *Ledger) Balance(id PlayerID) int64 { return l.balances[id] }

// Grant moves amount to the player and counts it as granted.
func (l *Ledger) Grant(id PlayerID, amount int64) {
	l.balances[id] += amount
	l.granted += amount
}

// Refund moves amount back from the player and counts it as refunded. It is the
// compensation half of a grant that must be undone.
func (l *Ledger) Refund(id PlayerID, amount int64) {
	l.balances[id] -= amount
	l.refunded += amount
}

// Total is the sum of every player balance.
func (l *Ledger) Total() int64 {
	var sum int64
	for _, v := range l.balances {
		sum += v
	}
	return sum
}

// TotalGranted and TotalRefunded expose the running counters.
func (l *Ledger) TotalGranted() int64  { return l.granted }
func (l *Ledger) TotalRefunded() int64 { return l.refunded }

// Conserved reports the invariant every grant path must keep: granted minus
// refunded equals the sum of all balances. A compensation that forgets to debit
// the player, or a retry that double-credits, breaks this.
func (l *Ledger) Conserved() bool { return l.granted-l.refunded == l.Total() }

// DeliverGrant applies a grant and, on failure, compensates it. This is the
// "发奖失败怎么补" path: the currency is added first, then the item is delivered;
// if delivery fails the currency is refunded so the ledger stays conserved and
// the player is left exactly where they started.
func (s *CommandStore) DeliverGrant(l *Ledger, c GmCommand, amount int64, deliver func(PlayerID) error) (Result, error) {
	return s.Execute(c, func(cmd GmCommand) error {
		l.Grant(cmd.Target, amount)
		if err := deliver(cmd.Target); err != nil {
			l.Refund(cmd.Target, amount)
			return fmt.Errorf("gm: grant for %d failed and was compensated: %w", cmd.Target, err)
		}
		return nil
	})
}

再配一个测试 internal/gm/command_test.go,断言幂等、补偿与账目守恒:

// file: internal/gm/command_test.go
// tier: L1
// verify: go test ./internal/gm/
package gm

import (
	"errors"
	"testing"
)

func TestCommandNumbersAreStable(t *testing.T) {
	if CmdGmCommandReq != 0x1101 || CmdGmCommandResp != 0x1102 || CmdGmAuditNotify != 0x1103 {
		t.Fatalf("gm command numbers drifted: %#x %#x %#x", CmdGmCommandReq, CmdGmCommandResp, CmdGmAuditNotify)
	}
}

func TestSameTicketAppliesOnce(t *testing.T) {
	store := NewCommandStore()
	calls := 0
	cmd := GmCommand{CmdID: "grant_item", Operator: "alice", Target: 1001, Ticket: "t-1", Args: map[string]int64{"count": 5}}
	apply := func(GmCommand) error { calls++; return nil }

	first, err := store.Execute(cmd, apply)
	if err != nil || !first.Applied {
		t.Fatalf("first execute: applied=%v err=%v", first.Applied, err)
	}
	second, err := store.Execute(cmd, apply)
	if err != nil {
		t.Fatalf("second execute: %v", err)
	}
	if second.Applied || second.Fingerprint != first.Fingerprint {
		t.Fatal("a duplicate ticket must not re-apply, but must return the stored result")
	}
	if calls != 1 {
		t.Fatalf("apply called %d times, want 1", calls)
	}
}

func TestRetryDedupsAfterSuccess(t *testing.T) {
	store := NewCommandStore()
	ledger := NewLedger()
	cmd := GmCommand{CmdID: "grant_currency", Operator: "alice", Target: 1001, Ticket: "t-2", Args: map[string]int64{"amount": 100}}
	calls := 0
	apply := func(c GmCommand) error { calls++; ledger.Grant(c.Target, c.Args["amount"]); return nil }

	if _, attempts, err := store.Retry(cmd, 3, apply); err != nil || attempts != 1 {
		t.Fatalf("first retry loop: attempts=%d err=%v", attempts, err)
	}
	// A second retry loop with the same ticket must not grant again.
	if _, attempts, err := store.Retry(cmd, 3, apply); err != nil || attempts != 1 {
		t.Fatalf("second retry loop: attempts=%d err=%v", attempts, err)
	}
	if ledger.Balance(1001) != 100 || calls != 1 {
		t.Fatalf("balance=%d calls=%d, want 100 and 1 (no double grant)", ledger.Balance(1001), calls)
	}
	if !ledger.Conserved() {
		t.Fatal("ledger must stay conserved")
	}
}

func TestGrantCompensationConservesLedger(t *testing.T) {
	store := NewCommandStore()
	ledger := NewLedger()
	cmd := GmCommand{CmdID: "grant_currency", Operator: "alice", Target: 1001, Ticket: "t-4", Args: map[string]int64{"amount": 500}}
	// Delivery fails: the grant must be compensated so no currency leaks.
	if _, err := store.DeliverGrant(ledger, cmd, 500, func(PlayerID) error { return errors.New("mailbox full") }); err == nil {
		t.Fatal("a failed delivery must surface an error")
	}
	if ledger.Balance(1001) != 0 {
		t.Fatalf("balance = %d, want 0 after compensation", ledger.Balance(1001))
	}
	if ledger.TotalGranted() != 500 || ledger.TotalRefunded() != 500 {
		t.Fatalf("granted=%d refunded=%d, want 500/500", ledger.TotalGranted(), ledger.TotalRefunded())
	}
	if !ledger.Conserved() {
		t.Fatal("ledger must stay conserved after compensation")
	}
}

func TestGrantSuccessThenDuplicateKeepsBooks(t *testing.T) {
	store := NewCommandStore()
	ledger := NewLedger()
	cmd := GmCommand{CmdID: "grant_currency", Operator: "alice", Target: 1001, Ticket: "t-5", Args: map[string]int64{"amount": 200}}
	deliver := func(PlayerID) error { return nil }
	if _, err := store.DeliverGrant(ledger, cmd, 200, deliver); err != nil {
		t.Fatalf("first grant: %v", err)
	}
	if r, err := store.DeliverGrant(ledger, cmd, 200, deliver); err != nil || r.Applied {
		t.Fatalf("duplicate grant: applied=%v err=%v", r.Applied, err)
	}
	if ledger.Balance(1001) != 200 || !ledger.Conserved() {
		t.Fatalf("balance=%d conserved=%v, want 200 and true", ledger.Balance(1001), ledger.Conserved())
	}
}

本机实跑(go test ./internal/gm/ -v)五条断言全绿:

=== RUN   TestCommandNumbersAreStable
--- PASS: TestCommandNumbersAreStable (0.00s)
=== RUN   TestSameTicketAppliesOnce
--- PASS: TestSameTicketAppliesOnce (0.00s)
=== RUN   TestRetryDedupsAfterSuccess
--- PASS: TestRetryDedupsAfterSuccess (0.00s)
=== RUN   TestGrantCompensationConservesLedger
--- PASS: TestGrantCompensationConservesLedger (0.00s)
=== RUN   TestGrantSuccessThenDuplicateKeepsBooks
--- PASS: TestGrantSuccessThenDuplicateKeepsBooks (0.00s)
PASS
ok  	aegis/internal/gm	0.527s

TestGrantCompensationConservesLedger 是本节的核心断言:它故意让发货失败,然后同时检查玩家余额回到 0、累计发放与累计退回都是 500、Conserved() 为真。这三条一起,才把「补偿真的把账做平了」钉死;谁把 DeliverGrant 里的 l.Refund 删掉「省一次写」,Conserved() 立刻为假。

本节小结

这一节把 GM 命令总线的三件事落成了可执行代码:

  1. 幂等靠指纹加票据:参数排序后哈希成指纹,Execute 对同一指纹只真正执行一次,重复调用返回 Applied=false 的旧结果。
  2. 补偿靠反向操作:DeliverGrant 先加钱再发货,发货失败就退款,Ledger 的守恒式 granted - refunded == Σ余额 是每次发奖路径都要守住的底线。
  3. 重试靠去重兜底:Retry 循环最多 N 次,成功即返回;因为 Execute 自带去重,重试在成功之后也不会重复生效。

幂等、补偿、重试让「发钱」这类命令变得可控,但运营事故远不止发钱。下一节 7.3 客服工具与数据修复流程 把镜头对准客服工单:一张工单从提交到归档要走哪些状态、修复脚本有哪些护栏、修完怎么对账。

阅读导航:上一节:7.1 权限与审计 · 下一节:7.3 客服工具与数据修复流程 。

继续阅读

探索更多技术文章

浏览归档,发现更多关于系统设计、工具链和工程实践的内容。

全部文章 返回首页

「mmo」更多文章

  1. 《MMO 服务器开发》工程化与运营 · 目录
  2. 《MMO 服务器开发》9.3 上线检查清单
  3. 《MMO 服务器开发》9.2 性能预算达标