Rollback restores a version; it may not undo its effects
A proposed incident exercise for an AI workflow: a new prompt produces a wrong client-update draft. The team restores the previous prompt. Is the incident over?
Split the recovery into two tracks.
- Restore useful service. Record the last-known prompt, model configuration, source inputs, and affected time window. Return to a reviewed version if that is the agreed bounded recovery step.
- Account for effects already produced. List which drafts were generated, which were reviewed, and whether any were sent. Restoring the old prompt does not retract a message or correct a commitment someone already read. Give a person responsibility for reviewing and correcting those effects.
Try a fictional tabletop exercise with three outputs: one unsent draft, one sent update, and one draft copied into another document. For each, write an owner, a correction path, and an observable completion check. Preserve the evidence without copying unnecessary private information into the incident record.
Then rerun the failure case on both versions. If the older version also fails, “rollback complete” is a service-state description, not evidence that the problem is solved.
This exercise does not prescribe a universal incident process. Its useful question is narrower: can the team distinguish restoring the system from repairing what the system already changed?
Which downstream effect would be easiest to overlook in your workflow?