事故:崩溃率降了,投诉却多了
《潮汐消消》某次活动版本上线后,质量看板上的崩溃率低于基线 0.18%,一片绿色;客服侧关于「启动就闪退」「打完一局卡住」的反馈却同时上升。排查发现两件事:采集 SDK 在首帧之后才初始化,启动期的原生崩溃没有被捕获;另一张报表把 OOM 击杀和后台清理都计入了崩溃,分子与分母同时被改写。
这不是 SDK 不稳定,而是采集边界没有定义。能采的、只能采样的、根本采不到的,必须分开建模;比率必须先写清分母,再讨论高低。 案例设定里消消的 Android 占 60%、4GB RAM 以下低端机占 35%,启动期恰恰是低端机最容易出问题的阶段,晚初始化会把最需要的数据漏掉。
三类数据,三种采集窗口
崩溃、ANR 与卡顿的采集能力不同,不能用一个「异常上报」接口糊在一起。
| 类型 | 采集窗口 | 进程内可完成 | 计入崩溃率 | 主要丢失原因 |
|---|---|---|---|---|
| 托管异常 | 崩溃当刻 | 是 | 是 | 处理器安装晚于异常 |
| 原生信号 | 崩溃当刻 | 是,但受限 | 是 | 信号处理器里分配内存或调用业务代码 |
| 脚本未捕获异常 | 当刻或下次启动 | 是 | 否 | 按采样丢弃,或引擎先于脚本处理 |
| ANR | 下次启动 | 否 | 否 | 主线程无响应,应用没有机会上传 |
| OOM 击杀 | 下次启动 | 否 | 否 | 没有回调,只能靠退出标记推断 |
| 后台清理 | 下次启动 | 否 | 否 | 与崩溃共用退出标记,容易误判 |
原生信号处理器运行在受限上下文,只能做异步信号安全的操作:写固定大小的转储、记录信号号与最小寄存器信息,不能分配内存、加锁或调用引擎。把它写成「什么都采集」的入口,往往会引入二次崩溃。ANR 更直接:主线程被卡住时,应用无法执行上报,只能依赖系统写入的 trace,在下次启动时补报。
OOM 击杀与后台清理都表现为「上一次会话没有正常结束」。用退出标记推断时,最多只能得到「非正常退出」,不能直接算崩溃;把它们混入崩溃率,会让低内存机型的数字虚高,也会掩盖真正的崩溃。案例设定里消消稳态内存 420MB、水位 520MB,低端机上越接近水位,越容易发生 OOM 击杀,这类数据应当单独看,而不是塞进崩溃率。
比率的分母
同一份原始数据,换一个分母就是另一个故事。三类比率的最小口径如下。
| 指标 | 分子 | 分母 | 最小样本 | 案例设定 |
|---|---|---|---|---|
| 崩溃率(会话) | 发生致命崩溃的会话 | 已启动会话 | 1000 会话 | 0.18% |
| ANR 率(会话) | 发生 ANR 的会话 | 已启动会话 | 1000 会话 | 0.05% |
| Jank 率(帧) | 超过 100ms 的帧 | 渲染帧总数 | 30000 帧 | 2.1% |
| 崩溃自由会话 | 未发生崩溃的会话 | 已启动会话 | 1000 会话 | 由崩溃率换算 |
崩溃率的分母是「已启动会话」,不是「已正常结束会话」。若只用正常结束的会话作分母,崩溃会话会被排除,崩溃率永远接近零。代价是未解析会话会带来不确定性:模型把未解析会话占比超过 10% 的样本标记为不精确,提醒读者这是下界而不是点估计。
Jank 率的分母是渲染帧数,不是会话数。案例设定里,消消默认 30fps,每帧预算 33.3ms,超过 100ms 的帧约等于连续丢掉三帧;方舟默认 60fps,每帧预算 16.7ms,同样 100ms 的阈值对应更多帧。阈值相同,帧预算不同,玩家感知也不同,所以 Jank 率必须和帧率档位一起看。
SDK 初始化时序与失败降级
采集 SDK 的初始化不是一次调用,而是三个阶段的权限递增。
| 阶段 | 安装处理器 | 采集致命 | 采集非致命 | 上传 | 个人字段 |
|---|---|---|---|---|---|
| 进程启动 | 是 | 是 | 否 | 否 | 否 |
| 隐私同意后 | 是 | 是 | 是 | 是 | 哈希后 |
| 首帧之后 | 是 | 是 | 是 | 是 | 哈希后 |
进程启动阶段就可以安装崩溃处理器,因为它只在本机落盘,不发生传输;但上传、非致命采集与个人字段必须等隐私同意。把「安装处理器」和「上传数据」绑在一起,是启动期丢崩溃的常见原因。
失败必须降级,不能阻塞游戏。处理器装不上时退化为空实现;存储写满时只保留内存中的最近一条;网络不可用时落盘等待;策略关闭或未同意时整体禁用。崩溃循环还要限制上传:同一天同一设备,前 3 次传完整转储,之后只传摘要,超过 20 次直接抑制上传,但本地计数继续。
字段也要最小化。会话标识与设备型号可以在启动期保留;用户标识、设备标识、IP 在同意前丢弃,同意后只上传加盐哈希;广告标识与自由文本永远不上传。崩溃堆栈用于定位必须保留,但要在采集侧检查是否混入了玩家路径或输入内容。
可运行模型:crash-model.ts
下面的模型把采集窗口、字段策略、初始化时序、降级状态与三类比率放在同一个文件里。它不接触设备,也不处理真实信号转储;崩溃处理器安装通过 CrashHost 接口表达,测试注入 mock,ANR trace 与帧统计仍需要原生 Host 才能落地,这里是模型,不是设备侧采集结果。
// file: src/obs/crash-model.ts
// tier: L1
// verify: node --test test/obs/crash-model.test.ts
export type CrashClass =
| "managed-exception"
| "native-signal"
| "script-error"
| "anr"
| "oom-kill"
| "background-kill";
export type CollectWindow = "at-crash" | "next-launch" | "sampled" | "never";
export type Severity = "fatal" | "degraded" | "informational";
export interface CollectionRule {
readonly crashClass: CrashClass;
readonly window: CollectWindow;
readonly severity: Severity;
readonly countsInCrashRate: boolean;
readonly note: string;
}
export const COLLECTION_RULES: readonly CollectionRule[] = [
{ crashClass: "managed-exception", window: "at-crash", severity: "fatal", countsInCrashRate: true, note: "进程内处理器捕获,先落盘,下次启动上传" },
{ crashClass: "native-signal", window: "at-crash", severity: "fatal", countsInCrashRate: true, note: "信号处理器只写最小转储,不能分配内存或调用业务代码" },
{ crashClass: "script-error", window: "sampled", severity: "degraded", countsInCrashRate: false, note: "脚本未捕获异常按会话采样,避免同一错误刷满配额" },
{ crashClass: "anr", window: "next-launch", severity: "fatal", countsInCrashRate: false, note: "主线程无响应时应用无法执行上传,只能下次启动补报" },
{ crashClass: "oom-kill", window: "next-launch", severity: "informational", countsInCrashRate: false, note: "系统杀进程没有回调,只能靠退出标记推断,不能算崩溃" },
{ crashClass: "background-kill", window: "next-launch", severity: "informational", countsInCrashRate: false, note: "用户或系统清理后台,不属于质量事故" },
];
export function collectionRuleFor(crashClass: CrashClass): CollectionRule {
const rule = COLLECTION_RULES.find((item) => item.crashClass === crashClass);
if (!rule) {
throw new Error(`unknown crash class: ${crashClass}`);
}
return rule;
}
export type InitPhase = "process-start" | "after-consent" | "after-first-frame";
export type FieldClass = "operational" | "quasi-identifier" | "personal" | "secret";
export type FieldMode = "keep" | "hash" | "drop";
export interface FieldRule {
readonly field: string;
readonly fieldClass: FieldClass;
readonly beforeConsent: FieldMode;
readonly afterConsent: FieldMode;
}
export const FIELD_RULES: readonly FieldRule[] = [
{ field: "crashClass", fieldClass: "operational", beforeConsent: "keep", afterConsent: "keep" },
{ field: "appVersion", fieldClass: "operational", beforeConsent: "keep", afterConsent: "keep" },
{ field: "osVersion", fieldClass: "operational", beforeConsent: "keep", afterConsent: "keep" },
{ field: "stack", fieldClass: "operational", beforeConsent: "keep", afterConsent: "keep" },
{ field: "deviceModel", fieldClass: "quasi-identifier", beforeConsent: "keep", afterConsent: "keep" },
{ field: "sessionId", fieldClass: "quasi-identifier", beforeConsent: "hash", afterConsent: "hash" },
{ field: "userId", fieldClass: "personal", beforeConsent: "drop", afterConsent: "hash" },
{ field: "deviceId", fieldClass: "personal", beforeConsent: "drop", afterConsent: "hash" },
{ field: "ip", fieldClass: "personal", beforeConsent: "drop", afterConsent: "hash" },
{ field: "advertisingId", fieldClass: "personal", beforeConsent: "drop", afterConsent: "drop" },
{ field: "freeText", fieldClass: "secret", beforeConsent: "drop", afterConsent: "drop" },
];
export interface InitPlan {
readonly phase: InitPhase;
readonly consent: boolean;
readonly installHandler: boolean;
readonly collectFatal: boolean;
readonly collectNonFatal: boolean;
readonly upload: boolean;
readonly spoolToDisk: boolean;
readonly personalData: boolean;
}
export function planInit(phase: InitPhase, consent: boolean): InitPlan {
const consented = consent && phase !== "process-start";
return {
phase,
consent,
installHandler: true,
collectFatal: true,
collectNonFatal: consented,
upload: consented,
spoolToDisk: true,
personalData: consented,
};
}
export function fnv1a(text: string): number {
let hash = 0x811c9dc5;
for (let index = 0; index < text.length; index += 1) {
hash ^= text.charCodeAt(index);
hash = Math.imul(hash, 0x01000193) >>> 0;
}
return hash >>> 0;
}
export function pseudonym(value: string, salt: string): string {
const left = fnv1a(`${salt}:${value}`).toString(16).padStart(8, "0");
const right = fnv1a(`${value}:${salt}`).toString(16).padStart(8, "0");
return `${left}${right}`;
}
export interface SanitizedDraft {
readonly fields: Readonly<Record<string, string>>;
readonly dropped: readonly string[];
}
export function sanitizeDraft(
draft: Readonly<Record<string, unknown>>,
phase: InitPhase,
consent: boolean,
salt = "mga",
): SanitizedDraft {
const fields: Record<string, string> = {};
const dropped: string[] = [];
const consented = consent && phase !== "process-start";
for (const [key, value] of Object.entries(draft)) {
const rule = FIELD_RULES.find((item) => item.field === key);
if (!rule) {
dropped.push(key);
continue;
}
const mode = consented ? rule.afterConsent : rule.beforeConsent;
if (mode === "drop") {
dropped.push(key);
continue;
}
const text = String(value);
fields[key] = mode === "hash" ? pseudonym(text, salt) : text;
}
return { fields, dropped };
}
export type SdkStatus = "ok" | "handler-missing" | "storage-full" | "network-blocked" | "disabled";
export type Fallback = "none" | "memory-only" | "disk-spool" | "noop";
export interface SdkHealth {
readonly status: SdkStatus;
readonly canRecord: boolean;
readonly canUpload: boolean;
readonly fallback: Fallback;
readonly note: string;
}
export function healthOf(status: SdkStatus): SdkHealth {
switch (status) {
case "ok": return { status, canRecord: true, canUpload: true, fallback: "none", note: "采集与上传均可用" };
case "handler-missing": return { status, canRecord: false, canUpload: false, fallback: "noop", note: "无法安装处理器,游戏必须继续运行" };
case "storage-full": return { status, canRecord: true, canUpload: false, fallback: "memory-only", note: "只保留内存中的最近一条,重启即丢失" };
case "network-blocked": return { status, canRecord: true, canUpload: false, fallback: "disk-spool", note: "落盘等待下一次可用网络" };
case "disabled": return { status, canRecord: false, canUpload: false, fallback: "noop", note: "被策略关闭或隐私未同意" };
}
}
export interface CrashHost {
installHandler(crashClass: CrashClass): boolean;
writeSpool(payload: string): boolean;
}
export interface HostReport {
readonly status: SdkStatus;
readonly installed: readonly CrashClass[];
}
export function installCrashHandlers(host: CrashHost): HostReport {
const wanted: readonly CrashClass[] = ["managed-exception", "native-signal"];
const installed = wanted.filter((crashClass) => host.installHandler(crashClass));
return {
status: installed.length === wanted.length ? "ok" : "handler-missing",
installed,
};
}
export type UploadTier = "full" | "summary" | "suppressed";
export function uploadTier(reportsToday: number, fullLimit = 3, summaryLimit = 20): UploadTier {
if (!Number.isInteger(reportsToday) || reportsToday < 0) {
throw new RangeError("reportsToday must be a non-negative integer");
}
if (reportsToday < fullLimit) {
return "full";
}
if (reportsToday < summaryLimit) {
return "summary";
}
return "suppressed";
}
export type SessionOutcome = "normal" | "crash" | "anr" | "kill" | "unresolved";
export interface SessionRecord {
readonly sessionId: string;
readonly outcome: SessionOutcome;
readonly renderedFrames: number;
readonly jankyFrames: number;
}
export interface RateReport {
readonly numerator: number;
readonly denominator: number;
readonly value: number;
readonly unresolvedShare: number;
readonly precise: boolean;
}
export const MIN_SESSION_SAMPLE = 1000;
export const MIN_FRAME_SAMPLE = 30_000;
export const MAX_UNRESOLVED_SHARE = 0.1;
function makeReport(
numerator: number,
denominator: number,
unresolved: number,
minDenominator: number,
): RateReport {
if (denominator <= 0) {
throw new RangeError("denominator must be positive");
}
const unresolvedShare = unresolved / denominator;
return {
numerator,
denominator,
value: numerator / denominator,
unresolvedShare,
precise: unresolvedShare <= MAX_UNRESOLVED_SHARE && denominator >= minDenominator,
};
}
export function crashRate(sessions: readonly SessionRecord[]): RateReport {
const numerator = sessions.filter((session) => session.outcome === "crash").length;
const unresolved = sessions.filter((session) => session.outcome === "unresolved").length;
return makeReport(numerator, sessions.length, unresolved, MIN_SESSION_SAMPLE);
}
export function anrRate(sessions: readonly SessionRecord[]): RateReport {
const numerator = sessions.filter((session) => session.outcome === "anr").length;
const unresolved = sessions.filter((session) => session.outcome === "unresolved").length;
return makeReport(numerator, sessions.length, unresolved, MIN_SESSION_SAMPLE);
}
export function crashFreeSessions(sessions: readonly SessionRecord[]): RateReport {
const base = crashRate(sessions);
return {
...base,
numerator: base.denominator - base.numerator,
value: 1 - base.value,
};
}
export function jankRate(sessions: readonly SessionRecord[]): RateReport {
let rendered = 0;
let janky = 0;
for (const session of sessions) {
if (session.jankyFrames > session.renderedFrames) {
throw new RangeError(`session ${session.sessionId} reports more janky frames than rendered frames`);
}
rendered += session.renderedFrames;
janky += session.jankyFrames;
}
return makeReport(janky, rendered, 0, MIN_FRAME_SAMPLE);
}
与相邻章节的关系
11.3 的区域化配置决定不同区域需要哪些同意项,本节只消费「是否同意」这一个布尔值。12.2 会把同一套字段策略扩展到普通日志,12.3 则把这里的比率接入告警、切片与灰度止损。崩溃与卡顿的通用讨论可以参考崩溃与卡顿 ,低端机降级策略可以参考低端机降级 。
小结
- 崩溃、ANR、OOM 击杀与后台清理的采集窗口不同,只有真正的致命崩溃进入崩溃率。
- 崩溃率与 ANR 率的分母是已启动会话,Jank 率的分母是渲染帧数;分母写错,结论必然错。
- 未解析会话超过一成时只能给下界,不能把缺失样本当作正常样本。
- 崩溃处理器可以在进程启动时安装,但上传个人字段必须等隐私同意;安装与上传是两件事。
- SDK 失败要降级为内存、落盘或空实现,游戏继续运行;崩溃循环用分级上传而不是无限重传。
阅读导航:上一节:11.3 区域化配置与内容分级 · 下一节:12.2 日志分级、采样与隐私脱敏 。
常见错误与边界
- 把 ANR 计入崩溃率。ANR 有独立的分母与补报窗口,混入崩溃率会让主线程问题变成「崩溃」问题,掩盖真实原因。
- 用正常结束会话作分母。崩溃会话恰好不会正常结束,这样算出的崩溃率会系统性偏低。
- 把 OOM 击杀当崩溃。系统杀进程没有回调,退出标记无法区分崩溃、OOM 与用户清理;这类数据只能单列。
- SDK 在首帧后才初始化。低端机启动期最容易崩,晚初始化等于主动丢掉最关键的样本;安装处理器必须前置。
- 处理器里调用业务代码。信号处理器只能做异步信号安全操作,分配内存或加锁可能造成二次崩溃,转储也会失真。
- 未同意就上传个人字段。安装本机处理器与上传是两件事;用户标识、设备标识与广告标识必须按同意状态分级处理。
- 崩溃循环无限上传。同一设备反复崩溃会打爆配额与带宽,应使用完整、摘要、抑制三级策略。
继续阅读
探索更多技术文章
浏览归档,发现更多关于系统设计、工具链和工程实践的内容。