Rollback procedure

Definition

A rollback procedure is written before the change is made, not after the incident. It records the specific settings before the change, the sequence to reverse them, any dependencies that also need to be reverted, and a verification step. For a platform without native versioning, the rollback is the only recovery path.

Why it matters

Most incidents are caused by recent changes. A tested rollback turns a revenue-affecting incident from hours to minutes. No rollback means diagnosing the change from live effects, which is expensive to do under pressure and often produces the wrong fix.

How it fails in practice

The rollback is theoretical — written for compliance, never rehearsed. When the incident happens, the steps assume a state the platform is no longer in, or reference a UI that has since changed. The team improvises, and the improvisation itself becomes the next incident.