写入高峰在结算后的十秒
《裂隙战场》的赛季最后十分钟,每局结束都会同时提交十个玩家的分数;《群岛纪元》的跨服战按小时结算,一个战区一次涌入几万条行军结果;《泡泡工坊》的好友榜写入不多,但也不能让每个玩家点一次结算就打一次数据库。三款产品的共同点是:写入是突发的,读取是持续的,而榜单的权威版本只能有一个。
9.1 的 Board 每收到一条写入就改一次内存,单进程几万条还能撑住;把同样的写法接到结算服务上,问题不在 CPU,而在顺序和边界。结算请求会重试、会乱序、会跨过赛季截止时刻。缓冲层的职责不是“攒一批再写”,而是把这三个问题收敛到一个可验证的状态机里。
缓冲模型:先决定能丢什么
| 模型 | 可见延迟 | 崩溃丢分 | 实现成本 | 适用 |
|---|---|---|---|---|
| 逐条直写 | 无 | 无 | 低 | 低频、强一致 |
| 微批(100 ms 到 1 s) | 一个批次 | 未刷的部分 | 中 | 大多数赛季榜 |
| 延迟可见(5 到 30 s) | 秒级 | 窗口内写入 | 中 | 能接受榜单慢半拍 |
| 内存 + WAL | 窗口内 | 不丢,可重放 | 高 | 高价值奖励 |
北辰的选择是微批加内存快照:常规分数允许丢掉最后一个未刷批次,因为玩家下一局会刷新更高分;赛季奖励名单不允许,所以封榜时先落一次持久化快照,再进入结算。把“可丢”和“不可丢”分开,比笼统地说“榜单要可靠”更有用。
合并的前提是单调
缓冲能按玩家合并写入,是因为赛季分数只增不减。一个玩家在同一赛季里提交十次,只有最高分有意义;低分和同分但更晚的写入可以直接丢弃。这个前提一旦不成立,例如天梯分或 ELO 会上下浮动,缓冲就必须保留最后一次写入,不能再用“保留最优”。
better 的三级比较在 9.1 和本节各写了一份,这是有意的重复:两个包可以独立抽取,但规则必须逐字一致。排序规则是跨服务的契约,不是某个包的私有实现。
截止时刻:事件时间优先
| 时间 | 含义 | 用途 |
|---|---|---|
事件时间 At | 结算产生分数的时刻 | 判断赛季归属 |
| 接收时间 | 缓冲收到写入的时刻 | 监控积压和延迟 |
| 封榜时间 | 调用 Seal 的时刻 | 生成不可变快照 |
| 发奖时间 | 调用 Settle 的时刻 | 写幂等回执 |
赛季归属只看事件时间。一场在 23:59:59 结束、00:00:01 才到缓冲的比赛,属于旧赛季;如果按接收时间判断,玩家会因为网络延迟丢掉整个赛季的分数。Submit 在入口检查 at >= cutoff 并返回 ErrLateWrite,把这条规则放在唯一入口,而不是让每个调用方自己判断。
事件时间依赖结算服务和应用服务器的时钟。允许的偏差要写进契约,通常用 NTP 加一个几十秒的宽限窗口;宽限窗口内的迟到写入仍然按事件时间归属,超过窗口才拒绝。
在途写入必须进入快照
封榜不是“停止接收”这么简单。调用 Seal 之前已经通过校验、但还留在 pending 里的写入,属于旧赛季的合法分数;如果封榜只读取已刷盘的数据,这些玩家会凭空少分。
本节的实现把 pending 和 best 分开:pending 是尚未刷盘的增量,best 是本赛季已接受的全量。Seal 从 best 建快照,因此无论刷盘进行到哪一步,快照都包含所有已确认写入。单进程下 Seal 与 Submit 由同一个调度协程串行调用;多进程部署时,需要一个“截止屏障”:先广播截止,等待最大在途时间,再各自封榜。
结算幂等:回执不是日志
结算服务会重试,运营也可能手动补跑。幂等的判据不是“跑过没有”,而是“这次输入和上次是否产生同一份奖励”。本节的 Settler 按赛季保存回执,回执摘要把赛季、截止时间、封榜时间和每一条奖励一起哈希;输入相同就返回原回执并标记 fresh=false,输入不同就返回 ErrSettled,让人工介入而不是重复发奖。
奖励服务自身还要有第二道幂等键,通常是 赛季:玩家:奖项。榜单回执解决“整批不重复”,奖励键解决“单条不重复”;两层都做,才能在重试和补跑同时发生时仍然只发一次。
切换期读一致性
赛季切换时同时存在三份数据:旧赛季的封榜快照、新赛季的空榜、以及正在结算的奖励名单。读接口必须显式带赛季参数,不能用“最新榜”这种默认值;旧快照只读,新榜从空开始,跨赛季查询走归档存储。
玩家在切换瞬间看到“旧榜还在、新榜没分”是正常的。客户端要显示赛季标识和结算状态,而不是把两张榜拼在一起。9.3 的缓存会在这个窗口里遇到版本跳变,处理方式同样是显式版本而不是猜测。
可运行片段:缓冲、封榜与幂等结算
下面的 internal/rankbuffer 把微批合并、截止校验、封榜快照和按赛季幂等的结算写成可测试的包。它不依赖存储:Flush 的返回值用于刷盘,Seal 生成的快照由调用方负责持久化后再发奖。
// file: internal/rankbuffer/rankbuffer.go
// tier: L1
// verify: go test ./internal/rankbuffer/
package rankbuffer
import (
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"sort"
)
var (
ErrSeason = errors.New("rankbuffer: empty season")
ErrCutoff = errors.New("rankbuffer: cutoff must be positive")
ErrLimit = errors.New("rankbuffer: limit must be positive")
ErrPlayer = errors.New("rankbuffer: empty player id")
ErrScore = errors.New("rankbuffer: negative score")
ErrTime = errors.New("rankbuffer: negative timestamp")
ErrLateWrite = errors.New("rankbuffer: write arrived after the season cutoff")
ErrSealed = errors.New("rankbuffer: season is sealed")
ErrNilSnapshot = errors.New("rankbuffer: nil snapshot")
ErrAmounts = errors.New("rankbuffer: reward amounts must be non-negative")
ErrSettled = errors.New("rankbuffer: season was already settled with a different result")
)
// Entry is one accepted write. Seq is assigned by the buffer and is the id a
// durable log can use to deduplicate replays.
type Entry struct {
PlayerID string
Score int64
At int64
Seq uint64
}
// better must match the ordering used by the board model. Duplicating the
// three comparisons keeps this package extractable without importing the
// write-model package, but the rule itself is a shared contract.
func better(a, b Entry) bool {
if a.Score != b.Score {
return a.Score > b.Score
}
if a.At != b.At {
return a.At < b.At
}
return a.PlayerID < b.PlayerID
}
// Buffer coalesces writes for one season. At is event time supplied by the
// game server, not the wall clock of the flush loop; a write with at >= cutoff
// belongs to no season and is rejected even before Seal runs.
type Buffer struct {
season string
cutoff int64
limit int
seq uint64
pending map[string]Entry
best map[string]Entry
sealed bool
sealedAt int64
}
// New creates an open buffer. limit is the number of reward ranks the season
// promises; it also caps the snapshot's displayed top list.
func New(season string, cutoff int64, limit int) (*Buffer, error) {
if season == "" {
return nil, ErrSeason
}
if cutoff <= 0 {
return nil, ErrCutoff
}
if limit <= 0 {
return nil, ErrLimit
}
return &Buffer{
season: season,
cutoff: cutoff,
limit: limit,
pending: make(map[string]Entry),
best: make(map[string]Entry),
}, nil
}
// Submit accepts a score into the write buffer. Writes are coalesced per
// player: only the best score and its sequence number survive until the next
// flush. The returned Entry is always the candidate with its assigned Seq; a
// false-looking no-op is not an error, because the caller only needs the ack.
func (b *Buffer) Submit(playerID string, score, at int64) (Entry, error) {
if b.sealed {
return Entry{}, ErrSealed
}
if playerID == "" {
return Entry{}, ErrPlayer
}
if score < 0 {
return Entry{}, ErrScore
}
if at < 0 {
return Entry{}, ErrTime
}
if at >= b.cutoff {
return Entry{}, ErrLateWrite
}
b.seq++
candidate := Entry{PlayerID: playerID, Score: score, At: at, Seq: b.seq}
if old, ok := b.best[playerID]; ok && !better(candidate, old) {
return candidate, nil
}
b.best[playerID] = candidate
b.pending[playerID] = candidate
return candidate, nil
}
// Flush drains pending writes in player-id order. The result is a delta for
// durable storage; the buffer keeps the full best map so a crash between
// flushes loses only the pending delta, never already flushed scores.
func (b *Buffer) Flush() ([]Entry, error) {
if b.sealed {
return nil, ErrSealed
}
out := make([]Entry, 0, len(b.pending))
for _, e := range b.pending {
out = append(out, e)
}
sort.Slice(out, func(i, j int) bool { return out[i].PlayerID < out[j].PlayerID })
b.pending = make(map[string]Entry)
return out, nil
}
// Seal freezes the season and returns an immutable snapshot. Pending writes
// are included: they were accepted before the cutoff, so dropping them would
// lose legitimate scores that the player already saw acknowledged.
func (b *Buffer) Seal(at int64) (*Snapshot, error) {
if b.sealed {
return nil, ErrSealed
}
if at < 0 {
return nil, ErrTime
}
b.sealed = true
b.sealedAt = at
entries := make([]Entry, 0, len(b.best))
for _, e := range b.best {
entries = append(entries, e)
}
sort.Slice(entries, func(i, j int) bool { return better(entries[i], entries[j]) })
return &Snapshot{
season: b.season,
cutoff: b.cutoff,
sealedAt: at,
limit: b.limit,
entries: entries,
}, nil
}
// Snapshot is the frozen result of one season. It holds every ranked entry so
// rank queries stay exact while Top still truncates to the promised limit.
type Snapshot struct {
season string
cutoff int64
sealedAt int64
limit int
entries []Entry
}
// Total is the number of ranked players, including those outside Top.
func (s *Snapshot) Total() int { return len(s.entries) }
// Top returns at most min(n, limit) entries. n <= 0 means the display limit.
func (s *Snapshot) Top(n int) []Entry {
if n <= 0 || n > s.limit {
n = s.limit
}
if n > len(s.entries) {
n = len(s.entries)
}
out := make([]Entry, n)
copy(out, s.entries[:n])
return out
}
// Rank returns the 1-based rank and whether the player appears at all.
func (s *Snapshot) Rank(playerID string) (int, bool) {
for i, e := range s.entries {
if e.PlayerID == playerID {
return i + 1, true
}
}
return 0, false
}
// Award is one reward line produced by settlement.
type Award struct {
PlayerID string
Rank int
Amount int64
}
// Receipt is the idempotency record for a settled season. Digest covers the
// season, cutoff, seal time and every award, so a replayed settlement with a
// different result is detectable instead of silently paying twice.
type Receipt struct {
Season string
Cutoff int64
SealedAt int64
Awards []Award
Digest string
}
// Settler records one receipt per season. It is not safe for concurrent use;
// the scheduler that owns a season calls Settle from a single goroutine.
type Settler struct {
receipts map[string]Receipt
}
// NewSettler creates an empty settlement ledger.
func NewSettler() *Settler {
return &Settler{receipts: make(map[string]Receipt)}
}
// Settle computes awards from a frozen snapshot and records the receipt by
// season. Replaying the same input returns the stored receipt and fresh=false;
// replaying a different result for an already settled season is a conflict.
func (st *Settler) Settle(s *Snapshot, amounts []int64) (Receipt, bool, error) {
if s == nil {
return Receipt{}, false, ErrNilSnapshot
}
for _, amount := range amounts {
if amount < 0 {
return Receipt{}, false, ErrAmounts
}
}
awards := make([]Award, 0, len(amounts))
for i := 0; i < len(s.entries) && i < len(amounts); i++ {
if amounts[i] == 0 {
continue
}
awards = append(awards, Award{PlayerID: s.entries[i].PlayerID, Rank: i + 1, Amount: amounts[i]})
}
digest := digestOf(s, awards)
if prev, ok := st.receipts[s.season]; ok {
if prev.Digest != digest {
return Receipt{}, false, ErrSettled
}
return cloneReceipt(prev), false, nil
}
receipt := Receipt{
Season: s.season,
Cutoff: s.cutoff,
SealedAt: s.sealedAt,
Awards: awards,
Digest: digest,
}
st.receipts[s.season] = receipt
return cloneReceipt(receipt), true, nil
}
func digestOf(s *Snapshot, awards []Award) string {
h := sha256.New()
fmt.Fprintf(h, "season=%s\ncutoff=%d\nsealedAt=%d\n", s.season, s.cutoff, s.sealedAt)
for _, a := range awards {
fmt.Fprintf(h, "%d,%s,%d\n", a.Rank, a.PlayerID, a.Amount)
}
return hex.EncodeToString(h.Sum(nil))
}
func cloneReceipt(r Receipt) Receipt {
out := r
out.Awards = make([]Award, len(r.Awards))
copy(out.Awards, r.Awards)
return out
}
几个实现取舍值得说明:
Submit在封榜后返回ErrSealed,截止后返回ErrLateWrite。两个错误必须分开,前者是生命周期问题,后者是数据归属问题。Flush只清空pending,保留best。这样封榜快照不会因为“刚刷过盘”而漏掉已确认写入。Settle的fresh返回值让调用方区分“本次真的发了奖”和“重放拿到旧回执”,奖励服务据此决定是否调用发放接口。- 回执里的
Awards在返回前复制,避免调用方修改切片后污染下一次重放。
失败模式与兜底
| 失败 | 表现 | 兜底 |
|---|---|---|
| 缓冲进程崩溃 | 最后一个批次丢失 | 常规分可接受;奖励季先落持久化快照 |
| 重复结算 | 同一玩家收到两份奖励 | 赛季回执摘要 + 奖励幂等键 |
| 迟到写入 | 旧赛季分数记到新赛季 | 按事件时间校验,超过宽限窗口拒绝 |
| 封榜与刷盘并发 | 快照漏掉在途写入 | 从全量 best 建快照,封榜串行化 |
| 时钟偏差 | 合法写入被判迟到 | NTP 加宽限窗口,记录偏差指标 |
| 切换期读默认榜 | 玩家看到空榜或旧榜混排 | 读接口强制带赛季,客户端显示状态 |
常见错误
第一,用接收时间判赛季。网络抖动和结算排队会让合法写入跨过截止线,玩家丢分且很难申诉。
第二,封榜只读已刷盘数据。在途写入已经向玩家确认过,漏掉它们比延迟可见更难解释。
第三,把幂等做成“查一下发过没有”。查询和发放之间的竞态仍然会重复;用带回执摘要的原子记录,或者奖励服务自己的幂等键。
第四,切换时保留“最新榜”接口。没有赛季参数的读接口在切换瞬间必然产生歧义,事后靠客户端补丁修不回来。
第五,缓冲层自己决定排序。缓冲只负责合并和截止,排序规则必须和榜单模型共用同一份契约。
小结
写缓冲解决的是突发写入和在途边界,赛季结算解决的是截止与幂等,两者共同保证“旧赛季的分数只进旧榜、奖励只发一次”。当写入被收敛、快照被冻结之后,剩下的压力全部来到读侧:一张几万人的榜会被每个玩家反复刷新。
阅读导航:上一节:9.1 榜单模型与分段 · 下一节:9.3 榜单的读模型与缓存 。
继续阅读
探索更多技术文章
浏览归档,发现更多关于系统设计、工具链和工程实践的内容。