The Fire Drill You Schedule: Proving OTA Recovery Before an Incident Does

Why the unscheduled failure is a poor teacher

Most OTA teams first test rollback during an incident. That is the worst time to learn that the signing service has no failover, the CDN origin switch is undocumented, or the runbook references a dashboard that was retired. A chaos day moves that learning into a scheduled window with a known cohort, an abort plan, and people watching.

The goal is not to break production for its own sake. The goal is to prove that specific recovery mechanisms work under realistic failure and to produce evidence that operators, auditors, and executives can inspect. A drill without success criteria is a stunt. A drill with criteria is reliability engineering.

Four failure domains to exercise

A useful OTA drill covers at least four domains. Signing and key management: revoke or make unavailable the primary signing key or KMS endpoint and verify that the backup path works, that manifests remain verifiable, and that operators know who can authorize a switch. Delivery: remove or misroute a CDN origin, expire a cache rule, or introduce a DNS failure and observe fallback and retry behavior. Device check-in: throttle or reject check-ins, or introduce a clock skew, and verify backoff, jitter, and resumption without a thundering herd. Telemetry and control plane: drop the metrics pipeline or the update status API and confirm that rollout decisions fail safe rather than fail open.

Each domain should have a hypothesis. For signing, the hypothesis might be that a secondary signer can issue a valid manifest within 20 minutes without manual key export. For CDN, it might be that devices fall back to the secondary origin within two check-in intervals and that egress stays within budget.

Designing the drill

Choose a cohort that is small, known, and recoverable. Internal lab devices, employee test units, or a one percent canary are appropriate. Do not use devices that control safety, access, or revenue unless the drill has been explicitly approved at that level. Freeze unrelated releases for the window. Assign a facilitator, an operator, an observer, and a communications lead. The facilitator injects; the operator follows the runbook; the observer records timestamps; communications tells stakeholders what is expected and what would count as an abort.

Define abort criteria before injection. Examples: more than a stated percentage of the cohort unreachable for more than two intervals, any device in a boot loop that cannot be recovered remotely, or loss of the ability to sign a rollback. The abort path should be tested as part of the drill, not improvised.

A quarterly drill plan

Two weeks before, select the failure domain and write the hypothesis. One week before, review the runbook with the on-call team and confirm access to backup credentials, secondary origins, and recovery tooling. The day before, snapshot dashboards and notify stakeholders.

On drill day, inject one fault at a time. Record detection time, first operator action, time to mitigation, and time to full recovery. If the first fault reveals a gap, do not stack a second fault on top; capture the finding and reschedule the compound scenario. After recovery, hold a short hot wash while memory is fresh. Within five business days, publish a written after-action report with owners and dates for each corrective action.

Success criteria that matter

Measure mean time to detect, mean time to mitigate, and mean time to recover for the cohort. Measure the percentage of devices that returned to a healthy state without manual intervention. Verify that no device outside the cohort was affected. Check that the rollback path produced a signed, verifiable image and that the device reported the expected version after recovery. Confirm that telemetry gaps were visible and bounded, not silently ignored.

A drill can pass even if a component fails, provided the recovery mechanism works and the failure is detected within the stated threshold. A drill fails if the team discovers an unknown dependency, cannot execute the runbook, or cannot prove the cohort returned to a known good state.

Make it routine

Schedule the drill on a recurring calendar, rotate the facilitator, and track corrective actions to closure. Publish a short summary to the wider engineering organization. The point is not to create a culture of blame; it is to make recovery a rehearsed capability. In 2026, buyers and auditors increasingly ask for evidence of recovery exercises. A dated after-action report with metrics is stronger evidence than a statement that rollback is supported.

The pipeline that survives an incident is usually the one that has already survived a Tuesday afternoon with the team watching.